Pre-SIP: Export Macros

One of the more powerful features of Scala 2 is obviously macro annotations. Scala 3 has greatly dialed things back on the metaprogramming front, and understandably so, but expressive metaprogramming capabilities are something that I’d really like to have access to in some form. On that note, I’ve been working on language feature that I hope offers a compromise.

For me the primary use-cases I have around metaprogramming would be to add new declarations at compile-time to existing traits, classes, and objects. For example, I should at compile-time be able to write code that declares a case class, declares a type-level transformation of the shape of that case class, and applies that transformation to create a source-code-level new case class with that new shape. This would help eliminate duplicate copy/paste data models that differ in mechanically-expressible ways. In the language of Shapeless, if I can go from a case class to a Repr, why can’t I go from a Repr to a case class?

I’m sure the folks here could express another hundred different use-cases that require the same capabilities of the compiler.

I believe the Scala 3 export keyword could be leveraged for such use in metaprogramming.

I’ve put together a fork of the dotty compiler and implemented the feature to prove it can work (see links at the end). The basic idea is to allow export to expand out the content of a macro and splice in the generated definitions. In some sense this isn’t so different than what export already does, and as proof of this many of the same code paths are re-used in my fork.

Here is some sample code of the feature taken from a simple unit test I have:

     // In file A.scala
       import scala.quoted._
       object TestMacro {
           def dothis(b: Boolean)(using Quotes): List[quotes.reflect.Definition] = {
             import quotes.reflect.*
             if (b) {
                val helloSymbol = Symbol.newVal(Symbol.spliceOwner, "hello", TypeRepr.of[String], Flags.EmptyFlags, Symbol.noSymbol)
                val helloVal = ValDef(helloSymbol, Some(Literal(StringConstant("Hello, World!"))))
                List(helloVal)
              } else {
                val holaSymbol = Symbol.newVal(Symbol.spliceOwner, "hola", TypeRepr.of[String], Flags.EmptyFlags, Symbol.noSymbol)
                val holaVal = ValDef(holaSymbol, Some(Literal(StringConstant("Hola, World!"))))
                List(holaVal)
              }
           }
         }

     // In file B.scala
         class Foo {
           export ${TestMacro.dothis(false)}._
           // expands to: val hola = "Hola, World!"
         }

Depending on the value of the boolean argument to the macro dothis, one of two different definitions will end up spliced in to the resulting class. And because the expansion is done as part of the typer phase (which is where export is processed normally), these definitions are visible to other code and are part of the type signature of the surrounding class/object/trait.

One huge benefit of re-using export in this way is that user-code as written remains untouched, and in fact cannot be altered. The definitions produced by the macro have to play by the same rules as the normal ones would.

What do folks think?

Links:

  • Dotty Fork: GitHub - littlenag/dotty at export-macro
    ** I want to stress that the fork (in branch export-macro) isn’t synced with the latest from upstream and was created a long while back. It also isn’t very clean in its implementation, and there are debug printlns galore.
  • Longer write up: Expressive Metaprogramming for Scala 3 · GitHub
    ** I’ve been thinking about this feature for a while now. Linked is a much more in-depth write up.
11 Likes

How should tooling work with this? If you auto-generate definitions and then refer to them, how does navigation and hyperlinking work? What about incremental compilation? How can I keep track of what changed and what needs to be recompiled if definitions are auto-generated?

Paint me skeptical. I believe Scala had too many efforts like this, where meta facilities for language dialects were created that then fell short for tooling purposes.

In general, the Scala 3 motto is NOT to enable dialects. We allowed that in Scala 2 and it caused much pain and I believe is one of the main causes for the backlash against the language that we are seeing. Scala 3 is powerful enough as it is, generally. I admit there are always niche cases were someone wants more power, but we have to realize that’s a tradeoff. There’s value in standardization, too.

The choice to reuse export was in part driven by the fact that Scala 3 tooling will already have to integrate with export regardless and that the lift between what export already does, which is a very limited form of meta-programming already, and macro generated definitions hopefully wouldn’t be that large.

For tools that leverage the pickled type signatures, or can, then the story should be identical between this proposal and what already exists around export. It’s possible some tools might already have “support” for this feature. I think that’s pretty cool.

Re incremental compilation, the compiler already has to track upstream targets changing and affecting an export, so I don’t see how this feature makes that any different.

I do agree that there are issues around discoverability. When a developer sees a class name or method name in code they should be able to quickly navigate either to its macro invocation in source or to a more useful definition. It isn’t immediately clear how that could or should work, but I’m not sure that’s a problem unique to this proposal, or within the scope of this proposal to fix. I’m happy to outline ideas, but that feels like something with a very long tail.

If you have specific tools in mind that you would want to see this feature tested with, or have an integration story for, then I’m happy to investigate more deeply. Certainly I would need to ensure compatibility with what we use at $DAY_JOB.

2 Likes

There is a fundamental difference. Currently, exports are decided by specified, declarative rules. With macros, what gets exported is decided by a Turing-complete program. That completely breaks all the assumptions that the incremental compiler is built on.

Perhaps. I would be surprised if the lift was that high, given that the incremental compiler already has to think about export and macros in general for each file that gets compiled. I’ll investigate and see what the story is.

1 Like

There’s a very similar issue with type- and “term-aliases”.

So nothing new here, imho.

In case I understand the proposal correctly there is no big problem with tooling. The generated code would end up in some form on disk, wouldn’t it? Or is the generated code purely “virtual”?

Nevertheless there is imho a need for such a feature. Scala is boldly lacking an unrestricted code gen feature!

Languages like Java have that: Annotation pre-processors. (Don’t try to use that form Scala though if you’re not keen on much pain in the *piep*).

It’s imho quite “funny” that such a powerful language like Scala has no mechanism to cut down mechanical boilerplate when even a still boilerplate-hell language like Java can do that.

The current “solutions” in Scala emit concatenated Strings as code files… That approach couldn’t be more primitive and error prone! :face_exhaling:

Please @odersky consider some form of proper code gen for Scala! I know your gut feelings are mostly right, but this issue at hand needs a solution. A solution that’s worth being in such a great language as Scala. Show the world how an outstanding feature in such regard could look like instead of leaving it to hacks like “emitting code as plain strings to disk”.

This proposal here looks really promising. (Almost) nobody complained, but it got instantly 11 Likes, which is quite a lot for this forum. (I read this as: The proposal is so convincing everybody is just shouting “take my money”. Usually you hear “Bedenkenträger” even in case of otherwise great proposals). It’s even already implemented half way… @nicolasstucki the macro maintainer seems to like the idea presented here also as I infer form some GitHub comments.

Btw., regarding tooling for code gen: Maybe Scala could get some inspiration form C# here. They introduced a feature called “partial classes” to aid with auto-generated compilation units.

  • When working with automatically generated source, code can be added to the class without having to recreate the source file. Visual Studio uses this approach when it creates Windows Forms, Web service wrapper code, and so on. You can create code that uses these classes without having to modify the file created by Visual Studio.
  • When using source generators to generate additional functionality in a class.

I’m not sure this should be imitated verbatim. But it’s interesting food for thought.

Everybody involved here: Thanks for finally looking into such an important feature like code gen!

I’m not sure if this is possible yet, but if macro annotations are allowed to fill in methods of a trait, I think I’ve come around to the idea that you should avoid generating declarations that typer relies on. That is, val hola = ${MyMacro.dothis(false)} is great, as is

trait HasSomeMethod {
  def classNameAndMethodCount(): (String, Int)
}

@fillInHasSomeMethod
class Foo extends HasSomeMethod {
  def method1(): Int = 5
  def method2(): Boolean = false
   // `@fillInHasSomeMethod quietly generates 
   // def classNameAndMethodCount(): (String, Int) = ("HasSomeMethod", 2)
}

Tooling might still get a little confused about where to bring you for the definition of Foo. classNameAndMethodCount, but it’s no longer totally magical – it could just bring you to the trait.