Context: F(by) Conference 2020
On the 25th January 2020, Darja Jovanovic and I, representing the Scala Center, were invited speakers at the 2020 F(by) conference in Minsk, Belarus. It is a conference focussed on all aspects of functional programming, and now for the first time for the 6th edition, the organisers introduced a Scala specific track.
We decided to address some of the concerns we’ve been faced with in the Scala community towards the migration from Scala 2 to 3, from two different but equally important points of view: Community and Communication as well as Technical.
Community and Communication Point of View
Darja’s talk illustrated the wider context leading up to the announcement of Scala 3 in April 2018, and underlined that each time Scala community was faced with a “problem” it managed to observe it as an “opportunity” and develop what is now seen as an economy surrounding and based upon Scala, such as:
- Founding Lightbend in 2010 (formerly known as Scala Solutions and Typesafe) to keep Scala compiler production ready as a response to new-found industry adoption;
- Creating online courses (MOOCs) on Coursera from 2013 onwards to facilitate teaching and training Scala engineers around the world to be able to keep up with the new demand for Scala engineers.
- Founding the Scala Center in 2016 as a response to growing community’s need to be represented and heard.
She focused on the importance of understanding that the pain-points of Scala 2 to 3 transition are very similar across the different (and sometimes opposing by interest) actors/stakeholders - Academia, Industry, Community.
Finally, she encouraged attendees to observe the new situation through the “opportunity” lens and get involved in discussions, meetups, to teach and to contribute to projects migrating to Scala 3.
For the full talk, please check the text here.
Technical Point of View
My talk then followed, addressing the large concern of how maintainers of Scala 2 projects can respond when Scala 3 arrives and how the Scala tooling will support the transition, as detailed below.
Tooling to Help The Migration
The Scala Center is committed to supporting two scenarios during the migration from Scala 2 to 3:
-
Remaining on Scala 2.13 and use the latest release of dependencies published for Scala 3.0
-
Migrating code to Scala 3 while using dependencies that remain on Scala 2.13
To assist in realising this vision, we are working on several projects:
-
TASTy Reader for Scala 2: This will allow the Scala 2 compiler to understand binaries that have been compiled with Scala 3, which supports the scenario of depending on published artifacts that are only available for Scala 3.0. This will also allow Scala 2 to invoke a
scala.reflectmacro where the implementation exists in a binary compiled with Scala 3. Try it out -
Scalafix: we will produce rewrite rules tailored for the transition.
-
Metals will add IDE support for Scala 3.
Questions From The Attendees (and more)
What we couldn’t cover in our 30 min presentation we tried answering during Q&A, and still, time limited what we could answer even then. It is important to hear the concerns so we can do a better job going forward. Here we flesh out some of the questions asked by attendees and are adding some that we’ve been hearing about in the community.
Please use this thread to keep asking us questions, and we will try our best to answer or seek knowledge if we don’t have it.
We also encourage you to write up answers if we didn’t get it right or you just simply want to volunteer, that will help us a lot! Here we go:
Q: There are so many changes happening in the Dotty project every month, is there a process to control this?
A: Scala 3 as a language specification is separate from the Dotty project. The Scala Improvement Process exists to stabilise what can and can not become a feature of the language. Each month, the SIP committee meets to evaluate changes to the language, and will review each new feature in Dotty before it can be included in the final release. In addition, the committee does not make isolated decisions: each proposal under review also goes through a period of evaluation by the community, with a dedicated thread on this forum to seek comments.
Q: When will Dotty be stable enough for me to use? I thought it was feature complete?
A: It is correct that Dotty is feature complete, this means that new proposed language features won’t be worked on by the core team until after Scala 3.0. However, experimentation is still allowed in existing changes to try and find the best version to present to the SIP committee. The dotty team will be coordinating with the Scala Center and Lightbend to provide a release candidate for Scala 3.0 in December 2020. Please stay tuned to this forum, or the scala-lang and dotty official websites for more updates.
Q: What steps are being taken to ease the transition to Scala 3?
A: Please read the announcements here
Q: What is the story for moving all the Scala 2 macros to Scala 3? Why do we need to if Scala 3 will compile macros for Scala 2? Why do we need yet another macro system at all?
A: Scala 2 macro implementations can be compiled by Scala 3 because they are implemented as library code using standard Scala features. However, Scala 3 will not be able to run these Scala 2 macros, because the Scala 3 compiler can not provide the required Context to execute and inline a Scala 2 macro at the callsite, (the Context API is too specific to internals of the Scala 2 compiler). What Scala 3 can do however is provide a Scala 2 macro implementation in a library and supply the necessary information in TASTy that Scala 2 needs to execute that macro from Scala 2 dependents of that library.
In Scala 3, a choice was made to provide a safer API to users of macros. Macros expand after type checking, so all the syntax trees are typed, in contrast to the prior macro system which mixed typed and untyped trees. Additionally, unsafe operations are restricted, such as adding public members to a class. Macros that relied on such functionality will currently need to be replaced with a bespoke solution, such as code generation.
This means that all existing Scala 2 macros, where possible, need to be ported to be executed from Scala 3 consumers of a library. Libraries such as lihaoyi/utest, lihaoyi/sourcecode and scalatest have already shown that it is possible to rewrite macros for Scala 3, using separate sources with cross compilation for the different implementations. We are working so that in the future, a single artifact can contain all macro code. Additionally, in a lot of cases, Scala 2 macros can be ported to language features of Scala 3, such as inline methods, the new framework to support type class derivation, or generic tuples and match types for type level programming.
Q: How is TASTy different to Scala.js IR or NIR for Scala Native, don’t they all contain the full program representation?
A: The representation of a program in TASTy is at a different level of abstraction to the other two formats. Scala.js IR and NIR are at the level of abstraction of a JVM class file, and therefore many of the details contained in a Scala source file are left out, such as arguments to generic type parameters. These formats are concerned with replicating JVM semantics, where possible, in different runtime environments, and preserving the same compatibility story as Scala on the JVM.
TASTy is at a much higher level of abstraction than either class files, Scala.js IR, or NIR. Unlike those formats, all the type information about the program is preserved, additionally, high level features such as lazy vals and pattern matching are preserved, which are converted to other constructs at the level of a class file.
Q: When will Spark make a transition to Scala 3, will it?
A: We don’t have such an answer but we recognise the importance in seeking for it, please help us find it. In the meantime, until the maintainers make the move to Scala 3, active work is taking place to make a release for Scala 2.13, which you will be able to use it as a dependency from Scala 3.
Q: How can a user be sure that a library dependency will still link at runtime if they switch to a future version?
A: You can use sbt-missinglink, a plugin for sbt that detects binary incompatibilities between your code and any transitive dependencies. Read the blog article
Thanks for checking this post out, please feel free to ask questions and give answers!