At this point, I would like to share some detail about Quill’s path forward. For the past several months I’ve been heavily focused on prototyping Quill in the Dotty macro system. With the initial POC being almost complete, I can confidently say that Dotty macros provide the needed functionality for Quill to move forward.
Here are a few architecture notes.
- Whitebox macros are no longer needed because Dotty’s ‘inline’ construct provides the feature set we need. The ‘inline’ construct is also extensively used (and will be used more!) in the implementation of Dotty-Quill itself. This will allow us to have type-annotations on variables holding Quill Quotes (these will typically be ‘inline def’). Additionally, this will allow us to do many more operations on compile-time queries then we were able to do before. This includes the ability to pass compile-time queries into methods which even includes recursive methods! This allows recursive construction of various query logics during compile-time and well as many other new things!
- Quoted is now a case-class and Quotation is static. De-coupling it from a specific context removes the need for the annoying cake patterns that are needed today. Based on my conversations with @fwbrasil and @lihaoyi this will significantly increase Quill useability.
- However, this requires that ‘lift’ sections delay summoning encoders until the
runmethod. Consequently, the type-tags of the lifts need to be propagated through the Quoted blocks into the
runcall. For compile-time (i.e.
inline) Queries this is doable since the Dotty tree can be made available to the run macro. For runtime Queries, however, Dotty currently does not support runtime-passable type-tags (@pshirshov and the ZIO folks have expressed that they intend to implement the Izumi LightTypeTag in Dotty which could be a solution to this issue). For now, the solution is to differentiate “lazy lifts” which can be done from a context-less Quoted section and “eager lifts” which are done once a context is imported, “lazy lifts” delay encoder summoning while “eager lifts” summon an encoder immediately. The semantics for the two will be equivalent and in most cases, a user will not need to know the difference. Once created, “lazy lifts” will be useable only with compile-time queries, while “eager lifts” will be useable everywhere.
- The mechanism for lifting, encoding, and decoding is significantly changed. Instead of manually constructing an object-encoding lattice via ValueComputation, Dotty’s Typeclass-Derivation is used. This is a very powerful approach that will allow us to implement handling of enums and embedded types much more transparently than in the past. Something like Hibernate’s table-per-hierarchy strategy may now be possible with co-product embedded classes.
- The parser will be implemented using Dotty’s quote matching. This mechanism is about as powerful as quasi-quoting but has stricter requirements about typing.
- Despite these things, Quill will still largely be able to have the same API with the exception that compile-time queries will need to have an ‘inline’ keyword in their declaration.
I will be working on the following things within the timeframe of a few months:
Parts of Quill in quill-core and quill-sql that are compatible with dotty will be moved out to quill-core-portable and quill-sql-portable. Originally I had planned to move out the handling of macros itself but Scala 2 macros are very deeply embedded into Quill and at this point, it would be impossible to move them out without causing major disruption to Quill’s external API (and hence the ecosystem). quill-sql-portable will only depend on quill-core-portable. This will allow Dotty-Quill to reuse the AST transformations.
Currently the Dotty-Quill design is being chrystallized on the dotty_test project. Once this is complete, I will create a project called ‘Proto-Quill’ (in a new codebase) that will be the initial implementation of Quill-Dotty (if anyone has a better name, please let me know!). Quill features will be migrated into Proto-Quill one by one. Initially, implementation of the Parser and the Metas will be very primitive and they will be fleshed-out module by module.
As time passes, quill-core-portable and quill-sql-portable will need adjustments in order to make Quill more tractable in Dotty. Small changes will be made to Quill’s API (e.g. Encoders/Decoders need to be traits instead of Types). These will be introduced slowly in order to prevent disruption to Quill’s ecosystem.
Proto-Quill will be around for several years as it matures and as Scala 3 becomes productionalized. Eventually, Proto-Quill will be merged to the proper Quill codebase and become Quill 4. The Scala 2 Quill (i.e. Quill 3) will then be moved in a different repo and called Exo-Quill which will eventually be deprecated.
Overall, I find the task ahead of us daunting but also inspiring. Scala 3 macros are substantially better behaved then Scala 2 macros and the Scala 3 Typeclass-Derivation mechanisms are really, good. There are many, many, many things that we will be able to do with Quill in Scala 3 that are completely impossible today. Amongst these are the ability re-use inline code between Quill transpiled expressions and regular Scala code. The ability to use Quill quotes inside of Typelevel code (e.g. Aux-Pattern returning Quoted blocks), recursive construction of Quill queries (since inlined blocks can be recursive), and many, many other things.