Whitebox def macros

If I understand correctly, what Quill needs to do specifically is to pass data between macro callsites, data created from the AST of one callsite, and used to create the AST of another callsite.

If you split out whitebox macroing into “type-level computation” and “AST computation”, you won’t be able to generate the necessary data Quill needs during the type-level computation because it depends on the exact AST captured by the macro.

What quill needs is for ways for the AST computations to communicate, which is currently hacked together by shoving data onto the types and unpickling it later, but could plausibly be done with a dedicate mechanism to support that. In which case the bodies of each blackbox macro callsite will have some “side channel” to pass information to each other, but would be guaranteed to not affect the rest of the typer.

Perhaps inlining of ASTs is one such side-channel, or perhaps instead of each AST node having a Type, it would have a tuple of (Type, SideChannelData), where `SideChannelData can be seen an acted upon by macro callsites, but is guaranteed to be ignored by the main typechecker. Hence it could possibly be used to customize codegen, or perform additional validations (in other macros), but it could never e.g. make a typechecking that would otherwise fail, pass because of this data.

An AST computation -> AST computation “side channel” data flow may seem ad-hoc, but nevertheless avoids all the problems that people don’t like about whitebox macros: tooling support, separation of typechecking & codegen, etc… Tools that ignore the SideChannelData would nevertheless be able to typecheck everything successfully; perhaps only missing out on additional errors that macros may generate when using this side channel data for validation.

If we want to expose this side channel data in an IDE, they can be taught how to recognize it, while IDEs which do not recognize it can ignore it and still generate a complete understanding of the “rest” of the code.

Notably, I remember the Parboiled guys wanted to do similar things to optimize parsers across multiple parser rule(...) calls. IIRC were exploring a type-refinement-based mechanism similar to what Quill uses (for some reason I cannot dig up the references right now) as well as build-time code generation (https://github.com/alexander-myltsev/sbt-parboiled2-boost).
They, too, “just” need AST computation -> AST computation data flow: they want their parsers to be able to optimize based on other parsers they call. The “rest of the world” can typecheck without knowing the details inside each parser, just like how it can typecheck withot know the details inside each Quill query

1 Like

Emma uses whitebox macros in a similar way to Quill.

storm-enroute coroutines uses whitebox macros; although I’m not entirely sure if they couldn’t do with blackbox as well. I asked on the Gitter channel, because they break analysis in IntelliJ, but got no answer.

Oh, no. Many of my libraries will have gone then.

2 Likes

I agree to remove whitebox macro in 2.12.4, as long as compiler plugins have the ability to hack the typer.

I actually found that AnalyzerPlugin is a replacement of whitebox macro. Please keep AnalyzerPlugin if whitebox macro will gone.

So I talked to Flavio and it’s pretty clear that we don’t need side-channel data to implement Quill. The current capabilities with inline should be enough. What would really useful for us is a quasai-quoting mechanism.

2 Likes