Roadmap for the TASTy Reader For Scala 2

Scala Center Updates
1

(The Scala Center team is dedicated to providing regular and transparent community updates about project plans & progress. In this forum we created a new topic category to allow quicker orientation going forward: “Scala Center Updates”. Even though all feedback is welcome, we keep the right to make executive decisions about projects we lead. Overview of all our activities can always be found at https://scala.epfl.ch/records.html)

Dear Scala Contributors,

In this article, we will provide some updates on progress on the TASTy Reader for Scala 2 project and outline some milestones for our future work.

How you can try it out

While we wait for the support to land in the official Scala release, you can still try it out by building for yourself the Scala 2.13.2 compiler with support for reading TASTy. Clone the repository at tasty-reader-compat-testkit and adapt it to try out with your own Scala 3 dependencies.

History

Initiated in September 2019, the primary goal of this project is to allow a gradual migration to Scala 3, which will arrive in late 2020, with many changes to source compatibility with Scala 2.13, and some new APIs added to its standard library. This project will enable users of Scala 2.13 to continue to use libraries that have migrated to Scala 3.0, giving a long window for projects to migrate one by one.

We achieve this forward compatibility by having bidirectional compatibility with two interfaces of the Scala language: the binary runtime interface, handled by the Dotty team at LAMP in addition to help by the Scala Center; and signatures of library dependencies, handled by this project. TASTy Reader enables Scala 2.13.x to read signatures of compiled code produced by Scala 3.0, so that 3.0 libraries may be freely consumed by Scala 2.13.x, provided they only use API’s that can be projected to the common language subset, explained further on. More information about how this works can be seen in my January 2020 talk Taste the Difference with Scala 3.

As of May 4, 2020, we have already opened a PR to scala/scala, where we support all of the common language between Scala 3.0 and Scala 2.13 in API signatures, excluding annotations, which may only have simple expression arguments. Libraries published for Scala 3.0 are able to take advantage of some new features and remain consumable from Scala 2.13, with some caveats:

  • Enums (equivalent to case classes and Java enum values)
  • Intersection types (the same as compound types)
  • Higher Kinded type lambdas (as Scala 2 style type lambdas)
  • Opaque type aliases (opacity is lost)
  • Scala 3 extension methods (as ordinary methods)
  • New syntax for implicits (as its desugared form)
  • Inheritance of open classes (the modifier is ignored)
  • Exported definitions (used as ordinary definitions)

Looking forward, there are still some core features we would like to achieve in the coming months:

Milestone 1: Use Scala 2 macro definitions from a library published for Scala 3.0

Why

A Scala 2.13 library that provides macros needs to rewrite the implementations to be compatible with Scala 3.0. However, currently, there is no way to provide a macro definition that can link in 2.13 with a library built by Scala 3, so a project must cross build. We would like to enable Scala 3 to provide definitions of macros to Scala 2.13 and 3.0 using the same API and avoid the need to cross build in this scenario.

How

Enable Scala 3.0 to provide a definition of a scala 2 macro in TASTy, alongside a scala 3 macro definition with the same signature. Scala 2.13 will instead link with the Scala 2.13 version. Scala 2.13 macro implementations should then be isolated to a library with a namespace that does not clash with the Scala 3 implementations, and be provided on the same classpath. Or the tooling can be adapted to provide both implementations in a single jar with jar versioning and the compiler selects the correct partition to search for implementations (how does this work with plain directory dependencies)

Caveats

Extra burden is placed onto library maintainers to duplicate definitions of macros in the Scala 3.0 version

Milestone 2: Support arbitrary definitions in annotations in the cross compatible subset of Scala 2.13/3.0

Why

Annotations can embed arbitrary trees, which may be used for metaprogramming, and it is an arbitrary restriction to stop at simple expressions.

How

Enable unpickling definitions to provide trees, not just symbols.

Caveats

It is not yet known how much would have to change to support this feature, e.g. Dotty compiler relies on lazy reading of class/method bodies, which is not possible in the Scala 2 compiler.

Milestone 3: Support Union types in Scala 2

Why

Union Types allow more precision in an API than a simple least upper bound. In most cases it should be possible to approximate something compatible with Dotty

How

Edit: Repurpose a similar type to Scala.js Union in the frontend of Scala 2.13 and erase it to match the JVM signature produced by Dotty.

Caveats

So far Scala.js Union can not derive all constraints necessary for full equivalence with Scala 3.0. Is it possible to adapt it? Are the restrictions good enough?

Maintenance Goal: Remain compatible with TASTy format changes

Why

With each Dotty public release, there may be changes to the TASTy format, codified by an explicit version change. Changes to the format may be made for many reasons, including changing the representation of an existing feature, with no effect on semantics. We need to maintain compatibility by adopting changes to the TASTy format.

22 Likes
2

This is very exciting!

Some comments:

  1. Re macros, this might be more work but wouldn’t it be possible to add dotty macro support to scala 2? After all, dotty macros aren’t tied to compiler internals. If that were done then perhaps a subset of dotty macros could be used in libraries built with scala 3 and be consumed by scala 2 code – but it also might mean scala 2 libraries could begin using dotty macros before upgrading entirely to scala 3 (making them usable by scala 3 code).

  2. I didn’t understand milestone 2 (about annotations) – what is an example of this or who is it relevant to?

  3. I’m not sure I understood milestone 3. It sounds like the plan is that when the scala 2 tasty reader reads dotty APIs using union types, it will transform signatures into equivalent ones that would use scalajs | (which would somehow be added to the classpath). The signatures would be different but would be usable in mostly the same way as from scala 3 code.

4 Likes
3

@nafg Hi, thanks for responding, I can say that on milestone 2, this goal doesn’t have a high priority. Until a practical need arises this goal is simply a technical bar to reach that would mark the reader as complete as possible.

4

It is very unlikely that we can add Scala 3 macro support to Scala 2. None of the core compiler developers and contributors believe it’s possible. So who would even attempt? The thing is that, although they’re independent of the compiler internals (or very close to that, still improving), they require very involved support in several core parts of the compiler, including the type checker. The Scala 3 macro system includes fundamental additions to the type system, which would need to be added. Changing the type checker so deeply is simply not in the cards for Scala 2.

It wouldn’t be Scala.js’ scala.scalajs.js.|. That wouldn’t be possible on the JVM since it’s not there. It would be a (synthetic) compile-time-only addition to the regular Scala stdlib (perhaps as scala.runtime.|), which would be written as a copy of Scala.js’ |. And we’d make that copy a bit more magic than Scala.js’ so that it can be entirely erased at compile-time (necessary for the forward bincompat policy of Scala 2.13) and erase to the LUB of the two sides (whereas the one in Scala.js erases to Object in the binaries).

So the signatures would be different, somewhat, but they would have the same erasure, ensure compatibility at the binary level.

1 Like
5

Scala.js union relies on implicits to emulate subtyping relationships, is the plan to keep it that way or to try to enhance the scalac subtype checker a bit to support them more directly? The implicit-based approach of scala.js seems to have trouble with singleton types: Scastie - An interactive playground for Scala.. The encoding also seems to have some performance issues, quoting Encoding · ScalablyTyped by @oyvindberg:

We’ve had some issues where unions of many types, among other issues like compile time, bumps into the JVMs limit for how many string literals can be referenced in a class/method.

(though maybe those issues only manifest themselves with the kind of huge union types which appear in typescript bindings and are not a problem in practice with the kind of code people write in Scala)

6

The plan is emphatically not to add true union types to the type system. That’s hardly more practical than adding support for the inline stuff to the type system. So yes it would use the same implicits-based system that Scala.js uses. As you say, AFAICT performance issues only arise for very large unions, which you wouldn’t find in typical Scala code.

The only “deep” change in the compiler would be to treat that type specially at erasure.

8

Heh, can you add “Milestone 4 - Add ‘inline’ to Scala 2”? That would be really great for Quill users. (just kidding of course! :yum:)

In all seriousness, this looks like a great plan! I didn’t expect Scala 3 compiled macros to ever be cross-portable to Scala 2 codebases so this definitely comes as a pleasant surprise to me, although I’m not exactly sure what to do with this capability yet. In Quill’s case specifically, Scala 3 Quill will no longer be relying on Whitebox macros that write AST trees into Java annotations; the annotation-passing mechanism will rely on Dotty’s ‘Inline’ construct. This means that all compile-time generated Scala3-Quill queries/quotes (i.e. everything… quote { in here! }) will have to be assigned into inline defs as opposed to vals or vars. Since Scala 2 does not have an inline construct, using Scala3-Quill cross-ported into Scala 2 would only allow Runtime query generation, not compile-time. For some use-cases, this might be reasonable, for other use-cases it might not. I need to think about it further.

The alternative, of course, is to write a ScalaFix component that would re-write vals that carry quotations into inline defs and try to get a cross-build working that way. I don’t know the full consequences of doing it like that but perhaps Quill’s own testing code could be adapted to this pattern. However since, Scala-Quill3 has a different component-model this might not actually make sense (at least for a large subset of the tests).

I expect many similar issues will be encountered by authors of Macro-based libraries as they take the plunge into Scala 3 meta. Library authors who decide to go through the pain of porting their macro-based code into Scala 3 will want to be able to use the new capabilities of Scala 3 meta to give their users new features so that their users themsevles have an incentive to switch. That’s the only way that the entire process is worth it. The ‘inline’ construct is the foundation of most of these features.

Trying to look at things holistically, I think its about what combinations of tools to use and about what reasonable usage patterns can emerge. ScalaFix will be required for some parts of the problem, but it cannot be relied upon to refactor a massively large codebases. Cross-portability can try to solve the problem from the other end but Scala 3 macros might not sensibly be structured the same ways as their Scala 2 equivalents. That means a codebase may itself need to be wrangled into some sort of Scala 2 “compatibility mode”. Ultimately a combination of approaches will be necessary.

Thank you for your post @bishabosha, hopefully this will be the start of a larger conversation.

9

Hi thank you for your response. Currently it would seem providing Scala 2 macros side-by-side Scala 3 macros, (i.e. in the same source file with the same signature) only makes sense if the way clients use the macro does not change with Scala 3 - unless you want to port the Scala 2 macros to the new scheme. I guess in this situation you could provide a different namespace for each macro variant.

10

It occurs to me that Quill’s use-cases of inline on the client side of the story could be emulated by two instructions. These would be quoteR that takes an existing tree and stores it into a java-annotation of refined type Record[R], and unquoteR which takes a Record[R] extracts the tree from the annotation splicing it back into the call-point. This is what quote/unquote in Quill presently does aside from all the domain-specific stuff such as AST parsing lifting/unlifting etc… so if we create an external mechanism to do this, it could potentially allow propagation of compile-time quotes on Quill-Scala3 macros enabling the compile-time query generation.

Unfortunately, this makes for a very bloated API:

val q = quoteR(quote { query[Person].filter(p => p.name == "Joe") })
// returns: Record[Quoted[Query[Person]]] { @stored(value=[query[Person].filter(p => p.name == "Joe")]) }
run(unquoteR(q))

Also, all the limitations of the current Quotation mechanism apply to this approach. The type Record[T] { ... } cannot be widened into Record[T] without loss of the compile-time information which dissalows type-annotations on methods/values containg Record[R] { ... } and passing Record[T] { ... } values into “proper” functions also cannot be done. These numerous limitations are why we decided to embrace inline in the first place.

I am wondering if this will be the case for other macro library authors. Anyone else planning to have clients use inline to propagate compile-time state? What are your use-cases?

11

I think you could implement Dotty’s inline with a Scala 2 macro annotation, no?

This way, you would get exactly the same system in Scala 2 — but with an additional @ in front:

@inline val q = quote { blah }

run( ... q ... )

would expand into something like:

def q = macro q_macro
def q_macro(c: Context) =
  c.reify { quote { blah } }
  // ^ reify will type-check the quote at the macro-definition site

run( ... quote[T] { blah } ... ) // you'll get type-checked trees inlined

The downside is that @inline values will have to live in top-level objects, and not classes.

EDIT: Of course, the other big downside is that you’ll have to have run in a separate project. Maybe that’s enough to make my solution impractical.
As an alternative, the annotation could expand into the code you proposed.

1 Like
12

This would be really, really cool! I’ll have to look into annotation-based macros. It would be really awesome if there was a library that did just this.

13

I think that’s really only a couple of lines of code, something like this probably works:

@compileTimeOnly("Enable macro paradise to expand macro annotations.")
class inline extends StaticAnnotation {
  def macroTransform(annottees: Any*): Any = macro InlineExpander.inlineImpl
}
object InlineExpander {
  def inlineImpl(c: whitebox.Context)(annottees: c.Tree*): c.Tree = {
    import c.universe._
    annottees match {
      case ValDef(...) :: Nil => ValDef(...)
      case _ => ???
    }
  }
}

The important thing to keep in mind is that these macros work on untyped trees and so really shouldn’t try to do anything semantic (they should keep to syntactic expansion).

14

When will this land in scala 2.13.x? I guess not yet with 2.13.3, right?
What’s the way to go for now if I want to write a library in dotty but provide it also to users using 2.13.x?

15

I can’t answer when TASTy reader will be released in Scala 2, but the current milestone on the scala/scala Github repo for my initial pull request is 2.13.4. Right now, if you want to publish for dotty and 2.13 your only option is to use cross Scala versions and not use any dotty features in the public API.
There is intended to be a dotty compiler option to help with cross compilation but does not exist yet, (something like -source 2x3)

16

Hello, I would just like to post an update here to say that we have completed a working prototype of Scala 2 macro definitions compiled by Dotty and being consumed in Scala 2. This was supported by Nicolas Stucki in the Dotty compiler team who completed in Dotty 0.25.0-RC2 support for compilation of def macros and macro bundles.

The following files illustrate a scenario where a single Scala 3 source file defines macros that can be consumed from both Scala 2 and Scala 3, using the same API (so usage is the same), but using different implementations. The restriction still follows that macro implementations can only be compiled by the language that executes them.

Macro Definitions for Scala 2/3 sharing a source file and API:

Macro implementations compiled with Scala 2:


Use Site in Scala 2:

This work is still in an experimental fork of the Scala compiler, which can be built locally to try out. I would be interested to see feedback on this feature, which will have an impact on how a project is structured compared to cross building of macro APIs.

9 Likes
17

Hello, I would like to post an update to say that a PR has been merged at scala/scala which contains the support for reading code compiled with Dotty 0.23.0-RC1. A follow up pull request will add support for 0.25.0-RC2 and macros (see milestone 1 above). We will continue to update the tasty support in Scala with each coming Dotty release.

8 Likes
18

Hello everyone, I would like to update this thread to demonstrate something exciting: I have produced a self contained repository demonstrating milestone 1, i.e. how to structure an sbt project so that a library built by Scala 3 built can ship with macros usable from Scala 2.

The sample repository can be found at scalacenter/mix-macros-scala-2-and-3 and demonstrates implementations of both def macros and macro bundles where the macro definitions are in a Scala 3 library, and consumed from a Scala 2 application.

It would be interesting to see which scenarios can’t be covered at present by adapting this repository to your own needs.

11 Likes
19

Have you tried whitebox macros?

20

Hi, I’ve just updated the demo with a white box macro, it is admittedly a simple example

1 Like
21

Hi everyone, I would like to point out that support for Dotty 0.25 and higher (up to 0.27.0-RC1) has been merged in scala/scala repository. This means that we now have support for consuming code built by 0.27.0-RC1 in 2.13.4 nightly releases. With sbt 1.4.0-RC1 it is now super easy to mix subprojects with different Scala versions, e.g. Scala 2 and 3 subprojects, just change scalaVersion - I have updated the example build below and hopefully you can see how simple it is:

10 Likes