Macro annotations are instrumentals for frameworks like Freestyle and as far as I can tell what we do today is not possible with codegen,
Freestyle modifies the definitions of the annotated classes to remove boilerplate and make those classes implement the patterns it needs for programs to remain compatible in vanilla scala contexts where Freestyle is not in use.
This could not be done with codegen without creating additional classes and types.
In the context of Free and Tagless carrying over a F[_] representing in the former the total Coproduct of Algebras and in the later the target runtime to which programs are interpreted is just boilerplate to most users and removing that and using the FS dependent type materialized by EffectLike is one of Freestyle features and foundations in which the entire library and companion libraries are built.
There is many other frameworks that modify in place annotated classes doing trivial things like adding companions or members to companions. Not being able to do that in place and just code generating things will change not only the usage but also semantics as to where implicits are placed and potentially result in other undesired behaviors that users will be responsible to fix.
Not supporting annotation macros would be a major breaking change and will break a ton of user code not only from a compilation stand point but also semantics and usage of these project public APIs.
There are a few points I’d like to clarify about my understanding of what the community requirements are, just so we’re all on the same page:
annotation macros are currently provided by a third party plugin and are notoriously broken in build tooling (presentation compiler, intellij, coverage, etc) and have some very bizarre behaviour in many cases when types get involved.
We have many existing macro libraries using annotation macros, e.g. simulacrum, deriving.
meta paradise offered an improved API over macro paradise, but unfortunately the tooling breakages are very bad (doesn’t support for comprehensions, crashing the presentation compiler, scaladocs, etc) with technical challenges mounting.
compiler plugins allow placement of meta programming at a specific phase in the compiler and are therefore more stable for tooling (only intellij requires custom support, and it is not hard to write it). But unfortunately quasiquotes (and the meta API) are not supported for compiler plugins and the ability to typecheck arbitrary trees is not available (which is only really “best efforts” in macros anyway, discovered through trial and error).
the difference for a scala developer to use a compiler plugin vs a macro is completely negligible. It’s a one liner in both cases and involves no code changes.
we should strive to use codegen where we can (e.g. the creation of new ADTs), but there are many places where we simply cannot (e.g. when modifying a compilation unit to have access to knownSubTypes, existing classes, and companions)
many current macro annotations require access to type information that is not available from early phases in a compiler plugin.
The obvious options seem to be:
add support for annotation macros to the Production Ready Macros, including full support for all tooling (including the presentation compiler, intellij, etc).
or, make a meta-like API available to compiler plugin authors and add some earlier typechecking / fqn / dealiasing phases (and the testkit!), such that existing plugins like simulacrum, freestyle and deriving can all be easily ported.
Synthesising members that need to be visible to other code currently being typechecked is a very delicate procedure, e.g. our implemenation of case classes in the Namer is pretty tough to understand. The hard part is that you want to base the logic of the macro on Types, rather than just syntax, but to get types you might trigger typechecking of code that will observe a scope before your macro has added/amended it. This could manifest as a fatal cyclic error, or as a failure to typecheck code that expects the synthetic members to be available.
How can we providing API/infrastructure support to macro authors who try to walk this tightrope? This is a core question, but one that requires a serious investment of research.
This question is orthoganal to the question of what API should be used to interrogate or synthesize trees or types, or the whether macros can be executed in the IntelliJ presentation compiler (which are both Hard Problems™️ in and of themselves!)
the last time I looked at Java annotation processors, they didn’t work in IntelliJ and maintaining a separate impl of each expansion was necessary. e.g. Lombok.
However, I’m coming around to the thinking of having a separate impl for the IDE and the compiler, for perf reasons. For example, in stalactite (the deriving macro) we don’t do any implicit derivations in IntelliJ… which dramatically speeds up the dev cycle. It’s only during the actual compile that errors will be discovered, and a lot of people prefer that because it doesn’t get in the way and then it means you’re interacting with the real compiler to fix tricky implicit problems. I’m in favour of less false red squigglies, and letting more through to the real compiler to catch.
To add to this, Scalameta annotation macros and the inline/meta proposal don’t expose semantic APIs to annotation macros. They are purely syntactic. This limits the capabilities of annotation macros but avoids introducing a cyclic dependency between typechecking and macro expansion. It seems that syntactic APIs are still sufficient to accomplish many impressive applications.
I’m using macro annotations in my scala-js-preact library. This is the scala.js facade for the js library. Macro annotations help me to reduce boilerplate code in the public API. Resulting API looks a lot more clean and simple.
The first implementation of the library API was without macro annotations. It contains a few objects with factory methods for creating entities. And I was not happy with this API. It requires a lot more description in the docs and was a lot more complex. Macro annotations contain all this logic inside and at the same time its effect is still understandable (not too much magic).
I’m not sure that I can switch current API to another metaprogramming technique. As far as I understand, atm I have 2 alternatives:
Using code generation with build tool. Actually, in my case, it could work, but I don’t want to provide my library with extra sbt plugin. It could confuse users and it makes the library less convenient for getting started.
Compiler plugin. It could work too, but it has the same disadvantages as sbt plugin. And also, it makes the code less portable.
Just wanted to chime in here to plea for macro annotations to not be abandoned. As mentioned before, libraries like freestyle allow to make complex patterns easy, otherwise requiring quite some boilerplate.
For my personal project i use them to build interfaces between scala (jvm) server and a scala jsclient, in the js client i use it to, for example, convert scala futures to js promises, scala lists to js arrays, so that i can easily call the methods in scala-js from from plain vanilla js.
Replacing this by compiler plugins would be way over my head, macro annotations are easy. Some other plain old codegen seems reinventing the wheel to me, it would be way more cumbersome, as it would require parsing the annotated code one way or another.
One of the big plusses (hate to mentioned it here) of scala over kotlin are its meta programming capabilities and the amount of boilerplate that they can reduce. Loosing macro annotations would hurt the language in this respect. I understand that making things work nicely in the ide is hard, but i’d be perfectly happy with macro annotations that have no ide support. Ide support for fancy scala is abysmal as is, red squiggly lines everywhere. In the case of annotation based code gen i’m not bothered by the ide borking, no big deal.
We use them for grafter which is a dependency-injection library
why are macro annotations important for you and your users?
Because they allow us to modify the dependency graph of an application by basically doing nothing but add a new dependency to a case class. This is lots of productivity gained.
can you use alternative metaprogramming techniques such as
code generation scripts or compiler plugins to achieve the same
functionality? How would that refactoring impact your whitebox macro?
I have no idea if I would know how to replace them but I would be extremely annoyed to have to change this code, which would have a major impact on our applications and components.
I make use of frees.io annotations to eliminate the error prone boilerplate needed to explicitly define Free DSLs
why are macro annotations important for you and your users?
Without macro annotations using Free becomes too cumbersome and especially too complex for new developers. Macro annoations allow me to create powerful abstractions that elegantly separate the concerns of what my program is doing and how it is doing it, in a concise manner with very little overhead.
can you use alternative metaprogramming techniques such as
code generation scripts or compiler plugins to achieve the same
functionality? How would that refactoring impact your whitebox macro?
I’m not experienced enough for answer this. I’m not defining the macro annotations in frees.io, just making use of them.
Whats the plan with scala meta @ast macros if macro annotations disappear? Will that switch to code generation? How would the type of AST node be specified? Via a tagging annotation? If infrastructure is provided by scalameta to support syntactic code generation directly perhaps as a pre-parse compiler plugin, I would fully support the decision to remove annotation macros. I still feel a lot is possible without semantic information, it would definitely keep the API smaller and easier to maintain. I should imagine it would also make it much easier for IDE’s etc. to analyze.
In fact, they actually got scala-meta macro annotations working nicely in the IDE at some point (it could expand on click, etc). So it’s not even like it’s unfeasible – it totally is.
I don’t want to use code generation for two reasons:
(as someone mentioned before), code generation requires another sbt plugin or shell script which must be manually executed. That’s annoying.
Generated code COULD be modified, either intentionally or accidentally. It may become a cause of bugs which are difficult to find. If you use macro annotation, code is generated internally, thus the code is protected from any modifications.
Compiler plugins are possible option. However, there must be a nice tool or library which supports plugin development process.
You will pleased to know work has started on scalagen a scalameta powered code generation library and SBT plugin.
All macro annotation’s that were supported in scalameta/paradise should be supported via codegen in scalagen. In fact, the api’s are so similar, that the bulk of your macro code, should work as code generation. But we can do more then a compiler plugin can.
The primary focus is performance (We all know Scala compiles slowly, lets not make it worse).
I don’t want to use code generation for two reasons:
(as someone mentioned before), code generation requires another sbt plugin or shell script which must be manually executed. That’s annoying.
Generated code COULD be modified, either intentionally or accidentally. It may become a cause of bugs which are difficult to find. If you use macro annotation, code is generated internally, thus the code is protected from any modifications.
Compiler plugins are possible option. However, there must be a nice tool or library which supports plugin development process.
Code generation is scala version and backend agnostic. This means the same generator will work for SN/SJS code with scala 2.10/2.11/2.12/2.13 and even Dotty. Provided those you dont use platform specfic features.
Compiler plugins are scala version specific and can cause issues with other compiler plugins.
Also if using a compiler plugin, any libraries used must be compatible with that compiler and backend.
Code generation has the downside of being build-tool specific. But for most annotation macro authors the ability to upgrade without waiting on a compiler plugin update, should far outweigh the downsides.
If generated code is modified (and you are using a plugin for a build tool such as SBT) it will be overriden. Although to be honest, it would suprise me if a generated file in the target directory was accidentally modified with any sort of frequency.
We could add a pre-parse compiler plugin to scalameta/scalagen. However at this stage, given the cross version requirements of that plugin. I would suggest that we do not do this unless absolutely necessary.
For the first point, I had forgotten the version-lock problem when I wrote the previous post. Now I understand code generation is superior to compiler plugins on this point.
For the second point, I should have explained about the situation I thought.
scalikejdbc has a code generator which generates some boilerplate to access RDB contents. A problem of this generator is that it generates code under the main src tree, thus generated code are not separated from other code and it can easily be modified. (And that’s why I re-implemented this feature by macro annotation to prevent generated code from being modified)
However, by reading @olafurpg’s comment on this issue, now I understand that code generated by scalagen will be put in another sbt project, so it won’t occur such kind of problems.
Happens to me all the time when working on the scala/scala integration branch for the new collections. It currently uses a git submodule with the collection strawman whose sources get translated from strawman,collection to scala.collection. When I click through to a definition in IntelliJ I inevitably wind up editing the generated version of the file.