Migrate to mcpyrate macro expander

[Technologicat]

Migrate to the new mcpyrate macro expander, because:

• correct test coverage reporting (less need for # noqa)
• identifier macros, to make magic variables such as it that may only appear in certain contexts (and raise an error at macro expansion time otherwise)
• advanced quasiquotes
  • don't auto-expand macros inside quotes; may allow us to clean up some macro code
  • can hygienically unquote macro names, too
• can define macros in the REPL, too
• integrated agile tools and dialect system (like imacropy and pydialect, but developed slightly further)

It's a 3rd-gen expander (after macropy and mcpy), and by design, not drop-in compatible, so this will have to wait until 0.15.0.

[Technologicat]

mcpyrate is now on PyPI.

[Technologicat]

Two steps:

1. Port existing code to use mcpyrate with minimal changes. (To be done in 0.15.0.)
2. Exploit the new opportunities afforded by mcpyrate. (To be done later, possibly in 0.15.x or in 0.16.0.)

[Technologicat]

As of this writing, the latest released macropy3 (1.1.0b2) does not seem to run on Python 3.8 at all, so this migration just became top priority for the future development of unpythonic.

[Technologicat]

Status (❌ = no, ✅ = yes):

✅ Boots up on Python 3.8 and mcpyrate
✅ Passes test suite on Python 3.8 and mcpyrate
✅ Final touches for initial version running on mcpyrate

As of ab9c836, done so far:

• Drop Python 3.4 and 3.5 support, mcpyrate runs on 3.6+.
• Add support for Python 3.8 and 3.9.
• Drop macropy3 bootstrapper. The solution is to use macropython from mcpyrate instead.
• Remove most mentions of pydialect; this technology is now part of mcpyrate.
• Raise exception normally instead of assert when an error occurs during macro expansion (custom macro-related syntax errors, mostly).
• Change macro interface call signatures to mcpyrate format.
• Recurse explicitly in the syntax transformers.
  • The code to pass in the expander argument via dyn, so that the syntax transformer can expander.visit(tree), is already in place in the macro interfaces.
• Convert MacroPy Walkers into mcpyrate AST walkers.
  • Port the scope analyzer.
• expose_unhygienic does not exist in mcpyrate. Store dbgprint_expr in dyn.
• Update quasiquote code, especially parts using hygienic quasiquotes (mcpyrate's work slightly differently).
• Implement lazy, which was previously provided by MacroPy.
• Implement the f[] macro of quick_lambda, which was previously provided by MacroPy.
• Update quicklambda documentation to match new implementation.
• @macro_stub does not exist in mcpyrate. Make a regular macro that raises SyntaxError.
• Make some magic variables (such as anaphoric if's it), that error out at compile time when the name appears outside a meaningful context.
  • the in the testingtools module
  • call_cc in continuations
  • local, delete in do
  • it in aif
  • block and expr in let_syntax/abbrev
  • q, u, kw in prefix
• block and expr in let_syntax/abbrev: for syntactic consistency, would be nice to pass their arguments using brackets. These are not actually macros, but AST patterns matched by let_syntax/abbrev itself...
• mcpyrate does not auto-expand macros in quasiquoted code. Consider when we should do so, and when let the expander iterate on its own. (We don't emit many macro invocations; the ones we do, should be correct.)
  • If macros need to invoke other macros in their output, use hygienic macro capture in the quasiquoted code.
• Investigate: do we still need to set ctx in our macros?
  • No, but we need to call mcpyrate.astfixers.fix_ctx when it is critical for the operation of some macro that the ctx are filled in correctly. Eliminated manual ctx filling and added those calls where needed.
• Update macro docs: first pass → outside in and second pass → inside out. mcpyrate doesn't use a two-pass system, but performs outside-in iteration on each statement in the module until no macros remain; inside-out expansion must be explicitly asked for (by recursing explicitly).
• Grep for any other interface changes for 0.15 I've forgotten about. Maybe move some of these to 0.16.
• Replace obsolete function calls:
  • gen_sym switched to mcpyrate.gensym
  • unpythonic.syntax.util.splice removed and switched to mcpyrate.splicing.splice_expression
  • macropy.tracing.show_expanded switched to mcpyrate.debug.step_expansion
• Grep for any remaining mentions of MacroPy.
  • Ones of historical interest left in; ones of current interest changed to point to mcpyrate.
• Update setup.py.
• Update contribution guidelines.

Items postponed until later:

• Consider moving isx and getname from unpythonic.syntax.nameutil into mcpyrate.
  • This is hardly the only project that needs to detect names and/or values of hygienic captures in the AST, so beside the low-level workhorse to get the actual data (mcpyrate.quotes.is_captured_value), maybe mcpyrate should present a friendly high-level interface, too.
  • Now there's also is_unexpanded_expr_macro and is_unexpanded_block_macro. This is starting to look like the beginnings of a macro destructuring subsystem...
  • Also, consider making a simple AST pattern-matching utility; this would simplify AST analysis in macros.
  • This stuff is important, but not for 0.15. Moving for later.
• Consider renaming 0.15 as 1.0.0.
  • Semantically, it's definitely 1.0, but the up-and-coming version has been known as "0.15" for such a long time (since 2019, maybe?) that maybe the number shouldn't be changed at this point.
  • Decided: not changing the version at this time. Maybe "0.16" could be 1.0.

Remaining items have been documented in the TODO comments at the start of the module unpythonic.syntax.__init__. It's a living development note that might not exactly match this occasionally updated one here in the issue comments.

[Technologicat]

As of ab9c836, we're "done enough" for a first version that runs on mcpyrate and Python 3.6 through 3.9.