Library and Compiler Contributions in Scala 2

#1

As announced in the recent Scala 2 Roadmap Update, Scala 2.13 will be the last major release in the Scala 2 series. In this thread I’d like to summarize what impact this has for contributors of the Scala 2 compiler and library.

Generally speaking, Scala 2.13 is no different than earlier Scala 2 releases in the kind of changes that can go into a minor 2.13.x release. In fact, 2.13 will quite likely see more evolution compared to previous Scala 2 releases, given that it will be maintained for a long time. In particular, we highly welcome changes that focus on user friendliness and ergonomics, such as improvements to the REPL, and bug fixes.

The limiting factors for the evolution of Scala 2.13 are binary compatibility, avoiding breaking changes and the risk of introducing regressions. In particular, changes to type inference and implicit resolution have been proven difficult to test in the past, even with the existence of the community build, and they have a limited scope because Scala 3 has an entirely rewritten type checker. On a less technical side, the focus is on improving the usability of Scala 2, changes that help with the migration to Scala 3 and bug fixes. The evolution of the language is happening in Scala 3.

In the past, we always created a new major branch (2.13.x) soon after a new major release (2.12.0) that could be used to merge changes for the next major release. This is different this time: there is no 2.14.x branch. To give two examples of my own, I had PRs to skip getters for private fields and to emit constructor defaults as static. Both changes are not binary compatible and the PRs were therefore closed.

What about the standard library?

The Scala 2.13 standard library is backwards and forwards binary compatible, i.e., a program or library compiled against 2.13.1 works with both the 2.13.0 and the 2.13.2 standard libraries. This implies that no publicly accessible classes or methods can be removed (backwards compatibility) or added (forwards compatibility) to the standard library in a minor release.

The reason for enforcing forwards compatibility is that sbt pins the standard library to the specific scalaVersion specified in the build definition. For other dependencies, sbt’s conflict manager selects the latest version of a library if the dependency graph brings in multiple version. The special case for the standard library is historic: it was considered that the compiler specified by scalaVersion should be used, and that the compiler and standard library versions need to be the same.

We plan to eliminate this special case for Scala 3, which means that sbt will always select the latest standard library in the dependency graph. This will allow lifting the forwards binary compatibility restriction. Unfortunately, I believe making this change already in a 2.13 minor release is too risky. Assume that 2.13.2 has a new class that doesn’t exist in 2.13.1, and some dependency-1.0.1 uses this new class. If a project defines scalaVersion := 2.13.1 and uses a current version of sbt that pins the standard library version, using dependency-1.0.1 will break.

Note that Scala 3.0 will ship without its own standard library and use the one from 2.13 instead, which reduces the migration cost for users, and allows binary interoperability between Scala 2.13 and 3.0. Only a later release, perhaps 3.1 or 3.2, will ship with its own standard library which can contain new additions.

So, what should happen to binary incompatible PRs to the standard library?

Additions to existing classes in the standard library will be possible with the next library release, which will likely be 3.1 or 3.2. Currently there is no branch in the dotty repository where such changes could be accepted.

Adding new classes to the Scala 3.0 standard library is technically possible by including them in Scala 3’s library extensions - however, I don’t know if such changes are encouraged or not (maybe @smarter or @sjrd can comment). In case this is done, it would be possible to create a new artifact that brings the same new classes to Scala 2.13; however, splitting packages across multiple artifacts is considered problematic.

Finally, if you want to propose a change for the next standard library now to make sure your code is “stored away” somewhere, feel free to create a pull request for the 2.13.x branch on scala/scala. We will mark the PR (for example with the 2.14.0-M1 milestone, or some labels) and eventually close (instead of merge) it. When the time comes, these marked PRs will be re-visited.

6 Likes
#2

dotty-library should really only be for things that are needed for dotty to work at all, if people want to experiment with new collections or new operations for existing collections, proposing them for inclusions in https://github.com/scala/scala-collection-contrib seems more appropriate.

4 Likes
#3

Would it be worth creating a “scala-library-future” for working on things which may end up in a future version of Scala, but are not forward binary compatible with the current stdlib? For example, to work on fixing Ordering

3 Likes
#4

Do you mean a git branch, or a released artifact?

#5

A scala-library-future or scala-library-updates backwards compatible artifact would be really nice I think. But if it’s not backwards compatible, the pain of releasing libraries cross-versioned against scala-version/updates-version is probably worse than the pain it’s trying to remedy.

#6

@martijnhoekstra are you thinking of a drop-in replacement artifact that could be used instead of the standard library? The problem I see with this idea is that sbt would have to know about it, so that the replacement artifact “wins” over the 2.13 standard library in dependency resolution if both are pulled into the dependency graph. So we’re back to the issue described in the original post: if some of your dependencies brings in the replacement artifact, you need to be using an up-to-date version of sbt (that doesn’t exist at this point), otherwise things break. (I assume there’s no way to make this work with current sbt releases.)

#7

More of a drop-in-augmentation library, like scala-collection-compat, but broader in scope.

1 Like
#8

I agree this makes sense to investigate. The scope is limited to new classes though, things like https://github.com/scala/scala/pull/8510 are not possible. Also changes that are source compatbile, but the encoding is not binary compatible. Related: https://github.com/scala/bug/issues/11804.

#9

A lot can be done with enrichments though.

implicit class ClampingOps[A](underlying: A) {
  def clamp(lowerBound: A, upperBound: A)(implicit n: Numeric[A]): A = n.max(lowerBound, n.min(underlying, upperBound))
}

in a package scala.backports or something like that would accomplish the same, at the cost of import scala.backports._.

1 Like
#10

I meant a released artifact, exactly as Martijn describes. It would have its own dedicated package (sub-package?) to mirror the stdlib (e.g. scala.future => scala.future.collection.immutable, scala.future.math.Ordering).

To deal with issues of people depending on different, incompatible versions of scala-library-future during its evolution (since Scala 3.1/Scala 4 are a long way off), we could go a bit further and version the (sub-?)package(s) (e.g. scala.future.v1, scala.future.v2 => scala.future.v1.collection.immutable, etc.) if needed. We can even start at v2 (since it’s an unlikely name for a package anyway) if we don’t want to start down that path, but decide to later.