I believe it won’t be possible without the whitebox macros.
Can you explain why it won’t be possible without whitebox macros? quill is surely a good use case so it’s worth figuring out what’s needed to support it.
1 Like
Quill propagates refinement types along with their quoted SQL snippets in order to perform SQL construction & optimization at compile time. Greatly simplified, it basically takes
val x = quote{1}
val y = quote{x + 2}
val z = quote{y + 3}
And the quote refines the types being returned so the contents of each call is visible in the type as an annotation:
val x: Quote[Int] @ast("1") = quote{1}
val y: Quote[Int] @ast("1 + 2") = quote{x + 2}
val z: Quote[Int] @ast("1 + 2 + 3") = quote{y + 3}
And then proceeds to perform optimizations at compile-time based on the annotated AST:
val x: Quote[Int] @ast("1") = Quote(1)
val y: Quote[Int] @ast("1 + 2") = Quote(3)
val z: Quote[Int] @ast("1 + 2 + 3") = Quote(6)
Quill doesn’t need all the power of whitebox macros. The annotations it uses to pass data around between invocations is “side channel”-ish: they never affect the “primary” type e.g. Quote[Int], but only the contents of downstream annotations.
Without the ability to pass annotations between calls via their types, Quill falls back to performing the optimisations and transformations at runtime. This works, but pushes computation-overhead and failure-reporting to runtime, whereas with the annotations present it can do all that and report any errors before you ever run any code. This would be the case if whitebox macros did not exist (blackbox macros cannot refine the types they return based on the AST captured).
5 Likes
Thanks for the explanations!
Note that in the proposed new meta programming framework, we still can create new types after typing, it’s just that those types are unavailable for the main type checking itself. So refinement types could be created as a side channel during macro expansion.
However, there’s another problem: We need to solve the problem of dependencies and separate compilation. Refinement types depend on other refinement types in fairly arbitrary ways. This has to be handled. A large part of the more sophisticated parts of Scala’s typer are there to deal with this problem. It seems a later phase of macro expansion has to implement something similar.
1 Like
It seems that it’d be possible to implement Quill’s compile-time query generation with inline. @odersky will the tree of the method marked with inline be visible if it is used as a parameter of a macro?
will the tree of the method marked with
inlinebe visible if it is used as a parameter of a macro?
Yes. You can get at it using Tasty reflection. @nicolasstucki can give more details. He just presented a paper at the Scala Symposium that shows a key technique for doing this.
I think it will be great if scala support language injection.
Something like :
for(rs<-Sql”select a,b,c from table”){
println(rs.a())
}
Is it really difficult for string interpolation and java interfaces?
That paper will not be much help. The short answer is that there are two levels of abstraction/complexities in the macros with inline. The simplest abstraction is just having inline, quotes ' and splices ~ and the second add reflection capabilities.
Quotes and splices allow the implementation of generative macros. The macro will receive parameters as Expr[T] or Type[T] which can be spliced in some quoted code. These cannot inspect the trees of the parameters unless the parameters are marked as inline, in which case the value of the tree will be passed to the macro instead of an Expr[T]. This currently works with primitive value types and I am experimenting with case classes and case objects. This will not be enough for Quill.
With TASTy reflect it is possible to take some Expr[T] or Type[T] and open them to allow reflection. This is quite similar to the reflection in Scala reflect. It is implemented using abstract types, abstract extractors and abstract extension methods (the paper https://dl.acm.org/citation.cfm?id=3241658 talks about this kind of design). Currently the API does not expose all it could, we have been conservative to avoid adding unnecessary functionality in the core. It would be interesting to know what exactly is required by Quill.
Another interesting new feature that comes with TASTy reflect macros is that it is possible to inspect the trees of the code that is outside the macro itself. This might be useful to be able to get statically know information about an argument you receive even though it is defined outside.
That sounds quite neat – are there any examples of TASTy reflect macros in the tests yet?
Cool – thanks!
https://github.com/lampepfl/dotty/pull/5096 is a good example of how to interact with TASTy trees within a macro.
Additionally, ShowSourceCode is a decompiler that takes a TASTy tree and recreates source code. It is probably the most comprehensive use of TASTy. It is not used within a macro, it uses the same interface on the tree of a complete file.
https://github.com/lampepfl/dotty/blob/master/library/src/scala/tasty/util/ShowSourceCode.scala
Neat – thanks!
I hope the new low-level TASTy-based reflection API will support quasiquotes, as in scala-reflect, which can construct and match code. They have proven to be really useful in that context, and were definitely one of the good parts of scala-reflect. It would be sad if the lessons learned with the scala-reflect experiment were forgotten.
It is part of the plan. Ideally, I would like to be able to support it using the quotes and splices syntax (rather than string interpolated quasiquotes) and make it a middle ground between plain generative quotes and splices and full TASTy reflection (i.e. something similar to Squid, but implemented natively).
That is fine, but I think we also need untyped quasiquotes for manipulating definitions such as classes etc.
Will TASTy have a quasaiquote interpolator from any library anyone is develping? Quill uses quasaiquoting extensively in a few areas. Porting something like the parser would be waay easier. I heard scalameta was going to read TASTy trees, is this still happening?
(scalameta#147 is superseded by scalameta#435 which is closed and Eugene is gone!)
This post should explain it Whitebox def macros - #19 by fwbrasil (sorry for the late response).
It would be a shame if the new macros would completely kill the concept of Quill, it is probably the best LINQ implementation in any language (yes even better than the C# one)
We’re definitely going to work on porting Quill! Falvio’s idea is to use inline to get all of the Scala ASTs together into the run macro call-site. That way we don’t need to do any dark-magic with reified types (storing intermediate AST trees inside an annotation etc…) and you’d be able to add type-annotations to variables/methods that are holding quoted blocks without forcing them to produce dynamic queries. That could also open us up to the possibility of using typelevel-transformations on Queries.
I know that if we have something that works similar to quasai-quoting, re-writing the parser will be very doable. There are some other things we are using whitebox macros for releated to massaging the intermediate quoted blocks but that stuff can be done at runtime.
Hello? Has there been any forward movement in this regard? @nicolasstucki is tasty-reflect any closer to having a quasi-quote interpolator? I’d like to be able to prototype parts of Quill for Scala 3 at some point!
I think these macros are using quasiquotes https://github.com/bishabosha/spire-scala3/blob/master/src/main/scala/spire/syntax/macros/cforMacros.scala
1 Like