Scala 3: Crossing the starting line

japgolly · December 16, 2020, 11:02pm

I read https://www.scala-lang.org/blog/2020/12/15/scala-3-crossing-the-finish-line.html this morning, and wrote up a response on reddit. I was just planning on having a bit of semi-playful rant but now I’m thinking it’s actually appropriate to post here and have a hopefully-productive discussion. I’d also really like to hear from other library authors as well to know their experiences and perspectives on this situation.

So here it is. Although my frustration is genuine, the tone I’m going for is lighthearted, reactionary, conversational and honest to my point of view. It’s not sardonic or scathing or anything like that so although I hope it’s already obvious, if not, please try to see it through my lens.

I’m not annoyed at any of the people, but I am annoyed at the process and how it’s played out with Scala 3.

The very title frustrates me. “Scala 3: Crossing the finish line”. Finish line? What are you talking about? The race hasn’t even begun. From the perspective of the Scala 3 team, I think they think someone fired the gun and the race started years ago, and now the goal is in sight. Nah man, you spent years getting ready for the race. Races don’t take years with athletes starting out incapable and then doing all their training in the race itself. Scala 3 is approaching the starting line, not the finishing line. Announce that you’re ready for the race to begin, then let the race occur with the community/ecosystem running beside you. (To be clear, IMO the 3.0 RC process is the metaphorical race and 3.0.0 GA/final is the finishing line.)

> “japgolly, are you just nitpicking the wording in their choice of metaphor?”

No no no. I’m sticking with the metaphor and highlighting it because it accurately explains the process and expectations I’m seeing, and those would be problematic regardless of the metaphor or wording. The issue is basically “hey we’re gonna give the community a few weeks starting now then it’s 3.0.0 GA and cake time.”

I’ve got like 10 OSS Scala libraries that I need to start cross-compiling for Scala 3, not to mention I should be a good citizen and patiently smile, investigate, minify and report all the compiler bugs I’m going to encounter, then I’ll of course continue being a good citizen and trying to respond to all the github emails I get it in response, and regularly revise my failed migration attempts with workarounds etc etc. All in a month. Over xmas and new years when I have friends staying with me and finally plan to take some time off to avoid my burnout becoming catastrophic.

> “But japgolly, they’ve been publishing Dotty milestones every ~6 weeks, why haven’t you already started?”

Because, in line with the regular Dotty messaging, I was aware that nothing was stable, things were changing all the time and I’m not just talking about things being a little buggy and then stabilising. No, I’m talking about milestones being incomplete, features not being fleshed out and changing between milestones, functionality required by me to be able to cross-compile my stuff not being in place yet (Scala.JS support for example was only added very recently), even now I’m not sure if things are stable. For example is the metaprogramming API all done now? Doesn’t seem to be from what I read (and macros need to be completely re-written for Scala 3).

Even if the milestones were usable to me and had the functionality I need, I don’t have the luxury of so much free time that I can afford to:

learn how everything works in milestone X
modify all of my code across all my libraries
wait for next milestone
figure out everything that’s changed
modify all of my code across all my libraries (again)
repeat repeat

It’s very, very time consuming and I don’t have the time. I don’t think it’s fair to expect library authors to deal with so much churn, at least not in a short timeframe.

My plan was to wait until all the experimentation is done and all the features I need are there, then migrate everything once.

So basically I haven’t migrated my libraries sooner because

I couldn’t because the required functionality wasn’t there
Even if it was there, I can’t afford to deal with lots of churn so I need the API to be at least tentatively stable.

> “Ok japgolly. they’re there now and you have a month. All gud rite?”

No not all good. In my opinion, a month is unreasonable.

That’s very little time to do all the work to migrate my libraries (which I have to do for free in my own time)
The timing with holidays is terrible. I can’t keep living on the cusp of burnout and although covid isn’t the cause for me, I suspect covid will mean there are many others who are wfh all the time and close to burn out too.
My libraries integrate with many others in the ecosystem so I have to wait around for those libraries to be ready first
When issues are inevitably found, it takes time and a significant amount of back and forth to get it resolved and resume progress

> “Hey japgolly. what about the SIP process? … Why am I asking you? Why, for this heading of course!”

Btw hey does anyone remember the SIP process and how all Scala 3 changes (except optional-braces which the ref gave a pass) would go through the SIP process? So I guess that’s happening in the next month too with all discussion concluding and all resulting changes being made in Scala 3 and all library authors updating their code bases to work with the new changes. That’s gonna be a pretty intense month.

> “Ok ok japgolly, much complain. Do you have any suggestions?”

Yes, I suggest this: go through the SIP process properly. After that, release RC1 and let the RC process go for around 6 months. Give the community time. Let the poor people at EPFL have a well deserved break too. Understand that most libraries require unpaid work and are anyway often blocked by dependencies. Get a big community build working before we lock in 3.0 final in stone, with those consequences to last for many years.

soronpo · December 16, 2020, 11:11pm

I agree. IMO, the milestone period realistically STARTS when all the changes are done. And changes are still happening. Once that’s over, then and only then, the industry and community can take this process seriously enough to start migration and needs at least several months to do so. I can even forgive the lack of SIP process, but a month is nowhere near enough.

I mean look at the size of the dotty community build vs. the Scala community build. Not even all the top upstream libraries have migrated. To me this sends a clear warning sign. I have no idea what is the reason to rush this, but community/industry/technically-wise it doesn’t seem so wise.

jeantil · December 16, 2020, 11:28pm

I mostly understand the position except for this specific bit

I follow the scala 3 development and discussion from pretty far but the promise was and the blog post concurs:

, Scala 3 is backward binary compatible with Scala 2.13, as well as forward compatible under the -Ytasty-reader flag of Scala 2.13. We will keep this bidirectional compatibility until an unknown major or minor version of Scala 3 (excluded).

Doesn’t that mean that you should be able to add dependencies against the libraries you depend on even if they haven’t been ported yet ?
is it because all the libs you depend on offer macros which won’t be executable when compiling with dotty ? (as an application author, not even a library author; my dependency tree is insanely huge and I am genuinely worried by this cross compatibility thing with and without macros if the compat is not there I probably won’t be able to use scala 3 for years not just months)

soronpo · December 16, 2020, 11:39pm

While that is true, it’s not a real measure of Scala 3. The point is to be able to migrate the source if we wish to deem the language “worthy”. If the message is “you can use the Scala 2 binaries as long as you don’t use any special Scala 3 features and/or Scala 2 macros”, then I’d say “well, I’ll just stay in Scala 2 land then”. We want to encourage migration, not a split in the community. Partial binary compatibility only helps some of the migration process. We cannot rely on it completely, because then we are not really migrating. We are just putting a Scala 3 paint-job on a Scala 2 body.

mpilquist · December 17, 2020, 12:16am

@jagpolly I saw your request for feedback from other library authors so I figured I’d add my two cents.

I’ve been supporting Dotty in some capacity in many of my libraries for almost a year now. This includes scodec-{bits, core, stream, protocols, cats}, fs2 for both cats-effect 2 and cats-effect 3, ip4s, munit-cats-effect, scalacheck-effect, and then various Typelevel projects I help maintain like cats-effect.

Overall the experience has been very positive. Both scodec-bits and scodec-core use lots of new Scala 3 features like macros, inline methods, type class derivation, match types, and arity-polymorphic tuples. During the Dotty 0.2x releases, APIs changed frequently and getting those projects added to the Dotty community build was very important, as the Dotty team then started including the necessary changes to scodec when the language changed. This has mostly settled down in the final 0.2x releases and the 3.0.0-Mx releases.

Timeline wise, 3.0 final in “early-mid 2021”, as stated in the blog post, makes a lot of sense to me.

One final note: thank you to everyone working to get Scala 3.0 released! It’s an enormous project representing the work of dozens (hundreds?) of people and I’m really looking forward to it.

nafg · December 17, 2020, 5:38am

As Daniel pointed out in his talk, withDottyCompat should not be used in libraries, otherwise you create a diamond problem where an application can end up having on its transitive classpath a scala 3 version of a library as well as the scala 2 version on the same library via withDottyCompat.

So libraries that want to publish for scala 3 need their upstream libraries to also publish for scala 3.

Sciss · December 17, 2020, 10:11am

Hi there,

I want to share here my process as a library author getting started with Dotty migration.

First of all, I disagree with some of the points japgolly is raising; especially the point that “things were changing all the time” and you would have to “modify all of my code across all my libraries” for every milestone. I believe this is only true if you wrote genuine Scala 3 code – which probably almost nobody was doing. As library author, I was primarily interested in making sure that once Scala 3 is released, I know my libraries cross-compile. So, I am currently not using any feature that is exclusively available to Dotty (with very few exceptions where I had to fork the source).

I have looked briefly at Dotty 0.27 when it came out, and started cross-compiling a few smaller libraries of mine. That is, I simply began working around the restrictions of the intersection of valid Scala 2 and valid Scala 3 code. Fortunately, almost everything can stay in place if you do not use explicitly dropped features such as type projections, and latest Scala 2.13 makes an effort to warn you about things that you should fix for future compatibility. My own “ecosystem” consists of two or three dozen libraries that constitute a larger system ; if I can trust Scaladex, I have now ensured that 38 projects are published for Scala 3.0.0-M2.

I always hit a wall with one of my “most downstream” projects ‘Lucre’ which is a system for representing reactive objects in a transparently accessed ACID database, in a nutshell. All the small libraries I wrote that it depends on never caused big issues with the Dotty transition, but ‘Lucre’ was hit hard from the beginning, because it was designed around type-projections, which, well, they were basically abolished. I had to set up a separate project to try to model the API almost from scratch. I started in early August, and being a project in “my free time”, it took until end of September, when I thought the approach was robust enough to start migrating the actual project, which then took another week. I could have learned scalafix and these tools, but I decided that the net costs would be less if I just went ahead with manual find-replace operations, along with IntelliJ’s excellent rename-refactoring and quick fixes (IntelliJ still can’t properly handle Scala 3 projects, so I worked in this from the Scala 2.13 perspective). This went relatively smooth, it took time, but it was a quite mechanical process. Essentially I replaced a type parameter S <: Sys[S] and projections on that type S#Acc, S#Tx, S#Id, S#Var by a new type parameter T <: Txn[T] and types Access[T], T, Ident[T], Var[T, _] after finding a few tricks how to “tie the types together” in the end. This was a bit more involved than it sounds in this short description, the process was a lot of effort, with no functional gain for the project except that now I removed type projections and could theoretically cross-compile in Scala 2 and Scala 3. On the other hand, the new design is better in my opinion, so I am not unhappy having done this refactoring. (Although a couple of papers published now have obsolete code in them, oh well).

Still, the project wouldn’t compile – at the time it was Scala 3.0.0-M1, as I was running into compiler crashes from incomplete support of F-bounded types. Indeed I submitted a number of bug reports involving F-bounded types, and it turned out that everyone but me hated them, including the Dotty team, with the suggestion that I “get rid” of them (another two months of API redesign and refactoring?), and Martin Odersky even said that it was definitely not him who would spend time fixing these bugs in the compiler. At least, and I am grateful for that, he returned to the topic and offered a possible way to make a similarly functioning design in Dotty (something along the lines of replacing T <: Txn[T] with T & Txn[T]). As for now, I am not intending to drop Scala 2 cross-building in the next two years or so, this remains a bookmark for me, and it doesn’t directly solve the problem of cross-compiling my project.

When 3.0.0-M2 came out, I made a new attempt to compile ‘Lucre’, again hitting crashes related to F-bounded types. In the meantime, the old bugs had been patched up by adding a new “Cyclic reference” compiler error instead of just crashing. Step by step, I managed to make the code base compile, with the final move being to split one of my modules into two parts, so the compiler would not longer get trapped in “cyclic reference” errors. The work took from beginning of November to beginning of December, and finally I could compile the project successfully, making way for another dozen or so projects of mine that depended on it. Those in turn were relatively simple to adjust to cross-compilation.

With my experience in ‘Lucre’, and knowing that some of the work-arounds with F-bounded types produced a fragile situation, I thought it would be wise to avoid breakage in the next milestone, and making a case for inclusion in Dotty’s Community Build (CB). It works, because I had control over most dependencies so they could also be build inside the CB. Thankfully, my project was accepted in the CB, and I am confident that the project is now future-proof when 3.0.0 final arrives.

So the main obstacles for me were the missing type projections, and the still fragile support of F-bounded types. All other rewritings I had to do were relatively mundane, such as (these are all mentioned in the migration guide):

fixing indentation errors. These were always were clear cases where I had bad indentation, and I’m actually grateful to have them fixed. Only very few cases occurred.
putting parens around lambda parameters, like map { x: Foo => becomes map { (x: Foo) =>. Not a big fan of this, as I can write map { x =>, so why do I need to add those parens when I have a type acription?
giving names to self-types, like trait Foo { _: Bar => becomes trait Foo { self: Bar =>, not a biggie
fixing a number of new “shadowing” errors, like trait Outer { def cursor = ???; object Inner { def cursor = ??? ; def foo = cursor ; here you need to be extra careful to make the right explicit reference, I think this doesn’t fix any problems, I had never any issues in Scala 2 with shadowing being unclear to me.
some stuff that is already reported in the latest Scala 2.13, like application of unit values, procedure syntax (this was weird for me in constructors)
removed reflection; suddenly OptManifest was no longer implicitly generated; this caused some rewrites and forked Scala 2/3 API necessary in Scala-STM.
removed structural type support; this caused some rewrites/changes necessary to Scala-Swing.
support for do { body } while(cond) dropped. There is a mechanical rewrite as while({ body; cond }) (). not pretty, but no biggie
a few changes with access visibility; I had a few calls to private constructors that were “ok” in Scala 2, but no longer in Scala 3; easy to increase the visibility when it occurs.
in serialization, when you read Byte or Short, you often have to explicitly call toInt to keep working with these values

Things I still find annoying:

the compiler got too opinionated IMHO. your average project will be full of warnings, including things like implicitly casting from Float to Double or Int to Float, which I think is silly. When I find the time, I will disable these warnings with compiler options. Also tons of pat-mat exhaustive warnings that I don’t want. If I write val Some(foo) = bar, I know that it will fail for None, if I write val (foo :: Nil) = bar I know this will fail if the list doesn’t have size one. I don’t need the compiler to tell me this. So either you disable the warnings, or you’ll put many @unchecked into your code.
additional warnings that don’t exist in Scala 2, e.g. cannot-emit-switch-statements for pat-mat with two or so cases. I guess these will vanish at some point.
no support for @elide. You’ll have to fork the source and use inline in Dotty perhaps to have a work-around. If you have elided debug statements, be aware that these will always remain and possibly be called when compiling for Scala 3.

Some other things to watch out for:

Scala 3 more or less no longer cares about package objects. If you had implicits there, be prepared that they are no longer found for objects living in that package. Better put them into the companion objects or other search paths. There are also some subtle changes of visibility and shadowing and priority in imports, which mainly comes into play if you cross-compile Scala 2.12 and 3.0.

I had relatively few dependencies beyond my own control, including for example Scala-Test and Scallop. When new milestones come out, you’ll have to wait perhaps a few weeks until your dependencies are available, too. This extra delay should be taken into consideration when calculating the final “release path” of Scala 3. As others have mentioned, withDottyCompat should not be used when you publish a library. Right now I am stuck with missing Scala 3 artefacts for Akka (Stream), and it seems that Akka will be one of the projects that take long time to migrate to Dotty cross-compilation ; there haven’t been released any artefacts for any milestone so far.

All in all, I would say that in 8 out of 10 projects, getting to cross-compilation was super simple, in 1 out of 10 cases, it was slight work but not diffult, and in the remaining 1 out of 10 cases, it was a big effort. I have to note that almost nowhere did I use macros, so if your project uses macros, that may be an effort similar to getting rid of type projections, i.e. requiring basically rewrites from bottom to top.

For the overall Scala ecosystem, in my opinion the transition will be relatively painless. Not everything will happen in one iteration, there will be projects that never migrate and thus other projects that have to rethink on which libraries they want to depend. But in the majority of cases, you can assume that by mid of 2021, they will cross-compile for Scala 2 and 3.

hepin1989 · December 17, 2020, 7:45pm

I hope that we first check through the issues in https://github.com/scala/bug/issues and https://github.com/lampepfl/dotty/issues, try to fix most of them in dotty, and get more projects on board before we release the first RC1.