Sbt 1.12.1 and unmoored scala-reflect / scala-compiler issue

This is a public service announcement / question regarding sbt 1.12.1 + Scala 2.13 dependency resolution. Summary is that sbt 1.12.1 allows scala-reflect and scala-compiler on the classpath be lower than scalaVersion in its attempt to accommodate Scala 3.8, but it ends up causing strange errors.

Background 1: forceScalaVersion

sbt 1.x traditionally has forced scala-library, scala-compiler, and scala-reflect libraries to align to the same version in Apache Ivy, and similar mechanism forceScalaVersion was implemented on Coursier to achieve the same effect for Scala 2.12.x, 2.13.x, etc.

Background 2: Scala 2.13-3.x sandwich (2021)

In 2021, Scala 3.0.0 was released using Scala 2.13 as its scala-library. Soon afterwards Scala Center contributed a feature to sbt 1.5.0, which I’ve been calling Scala 2.13-3.x sandwich, a new set of operands .cross(CrossVersion.for3Use2_13) and .cross(CrossVersion.for2_13Use3), which allows Scala 3.x libraries to be mixed into Scala 2.13 classpath.

This is somewhat analogous to %% operator in a sense that scala-library started keeping the binary compatibility starting 2.9.x, and sbt added a mechanism to allow end users to take advantage of the compatibility. For Scala 2.13, forceScalaVersion remained in effect, so scala-library, -reflect, and -compiler versions aligned.

Background 3: SIP-51 Unfreeze scala-library (2024)

In 2024, @lrytz contributed sbt#7480, which removed forceScalaVersion so scala-library can be discovered from the classpath. As a substitute mechanism Lukas introduced csrSameVersions constraint, which forced scala-library, -reflect, and -compiler versions to be the same version. This combined with the build-time error to enforce that scalaVersion is same or greater than the discovered scala-library version effectively kept the same power of enforcement as forceScalaVersion.

Background 4: Scala 3.8.1 ships its own scala-library (2026)

In 2026, Scala 3.8 shipped, and it started building its own scala-library in Scala 3.x. This means that if we go to Maven Central there is now Central Repository: org/scala-lang/scala-library/3.8.1 . However, it does not come with scala-reflect or scala-compiler (Scala 3 compiler is called scala3-compiler_3).

Error downloading org.scala-lang:scala-reflect:3.8.1

Recall that we have Scala 2.13-3.x sandwich, which allows interoperability of Scala 2.13 and 3.x ecosystem. When a subproject using 2.13.x started using a Scala 3.8.x library via for2_13Use3, the csrSameVersions constraint kicked in, attempting to align scala-library:3.8.1 with scala-reflect, and caused ResolveException (sbt#8533) :

[error] (projectC / update) sbt.librarymanagement.ResolveException: Error downloading org.scala-lang:scala-reflect:3.8.0
[error]   not found: https://repo1.maven.org/maven2/org/scala-lang/scala-reflect/3.8.0/scala-reflect-3.8.0.pom

Unmoored scala-reflect and scala-compiler

To workaround the scala-reflect “not found” issue, in sbt 1.12.1 I’ve dropped the csrSameVersions constraint for Scala 2.13. This should allow the use of Scala 3.8.x libraries on Scala 2.13 subprojects, but at the same time, the resolution now behaves in a strange way.

Problem 1: ClassNotFound: scala.reflect.internal.tpe.TypeMaps$ContainsAnyCollector

sbt#8661:

[info]   Cause: java.lang.ClassNotFoundException: scala.reflect.internal.tpe.TypeMaps$ContainsAnyCollector
[info]   at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
[info]   at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
[info]   at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
[info]   at scala.tools.nsc.typechecker.Typers$Typer.<init>(Typers.scala:211)

In the reported case, scala-compiler and scala-reflect on the classpath were downgraded from 2.13.18 to 2.13.14 and 2.13.16.

Problem 2: JLine was compiled by a more recent version

sbt#8689

An exception or error caused a run to abort: org/jline/terminal/impl/ffm/CLibrary$termios has been compiled by a more recent version of the Java Runtime (class file version 66.0), this version of the Java Runtime only recognizes class file versions up to 65.0
  java.lang.UnsupportedClassVersionError: org/jline/terminal/impl/ffm/CLibrary$termios has been compiled by a more recent version of the Java Runtime (class file version 66.0), this version of the Java Runtime only recognizes class file versions up to 65.0
  at java.base/java.lang.ClassLoader.defineClass1(Native Method)

This strange error is also related likely to scala-compiler downgrading since scala-compiler 2.13.15, depended by refine library, brings in different version of JLine from that of scala-compiler 2.13.18.

Discussion: What should we do?

  • Scala 2.13-3.x sandwich may have been useful during the initial introduction of Scala 3.x, but I’m now wondering if I should revert sbt#8633 and reintroduce csrSameVersions, and effectively drop the sandwich support for 3.8+. I am actually not even sure if Scala 2.13 macros can safely execute using Scala 3.8.1 scala-library.
  • Not having the alignment of scala-library, scala-reflect, and scala-compiler on Scala 2.13 causes confusing issues.
  • If some folks are truly inclined to use Scala 3.8.x libraries from 2.13 application, they could manually drop csrSameVersions, and also deal with strangeness if they have to.
3 Likes

Actually, this seems like a problem that could be solved on Scala 2.13 side. e.g. have Scala 2.13.19 build against/ship with scala-library-3.8.1 - invert the previous relationship where Scala 3 reused 2.13 stdlib to the exact opposite - 2.13 forward compatibility has been revoked anyway, so this ought to be possible?

Actually, this seems like a problem that could be solved on Scala 2.13 side. e.g. have Scala 2.13.19 build against/ship with scala-library-3.8.1 - invert the previous relationship where Scala 3 reused 2.13 stdlib to the exact opposite - 2.13 forward compatibility has been revoked anyway, so this ought to be possible?

Does this risk turning into a boat anchor? Since 2.13 can’t use a bunch of Scala 3 features, this feels like it would exert pressure for the Scala 3 stdlib to not use those features, which seems potentially unfortunate in the medium term.

Do you mean capabilities? Would they have to be explicit parameters when viewed from Scala 2? That’s a concern, but I also thought capabilities were still experimental, so there shouldn’t be non-experimental additions to Scala 3 stdlib that would touch its 2.13 subset. Scala 2.13 wouldn’t be able to use some definitions properly, e.g. extension methods, but if those are new extension methods, that would still be the case if Scala 2.13 keeps using old stdlib. Replacing old implicit methods with extension methods would probably have to be done ala chore: deprecate `scala.util.ChainingOps` by hamzaremmal · Pull Request #24725 · scala/scala3 · GitHub - deprecating old implicit classes and using implicit priority to make extension methods have higher priority than old implicit classes - old implicit classes would still be callable from 2.13.

Union types were the first thing that came to mind when I commented above, but I suspect there are lots of different features that might be useful in stdlib but I think aren’t consumable from Scala 2.13.

Honestly, a part of me would love to see the stdlib fork – allow Scala 3 to go its own way and start really rethinking how we might improve the ergonomics based on the new functionality. I’m very conscious of this now that I’m using Scala 3 at work – I’m finding lots of little ways to make my old patterns work better by leveraging stuff like Context Functions.

I suspect that’s impractical due to eviction concerns so long as Scala 3 can use Scala 2.13 libraries, but I’m wondering if there are pragmatic ways to split the difference. (Eg, a “supplementary” Scala 3 only stdlib, or something like that.) Just some food for thought…

(post deleted by author)

I think a more official statements would be forthcoming from Scala 3 team, but I’ll note here that I’ve gotten a confirmation from @hamzaremmal that:

  1. Using a Scala 2.13.x library in a Scala 3 subproject is still supported in 3.8.x.
  2. Using a Scala >= 3.8.x library in a Scala 2.13 subproject is no longer supported.

TASTy Reader can’t handle union types and cannot handle transparent inline methods for explicit nulls, and likely they will be adding more things in the future that are not supported.

I think this clears the way for sbt to reintroduce csrSameVersions constraints for 2.13 scala-reflect and scala-compiler.

3 Likes

Right now we don’t even have a doc page on the TASTy Reader, a place where we could document what versions it’s expected to work (or not work) with. I think we can fix that under the Sovereign doc project umbrella.

1 Like

Also Scala 2 should error early if someone even tries to use it with Scala 3.8+. I’ve opened TASTy reader should error early on unsupported Scala 3 version · Issue #13152 · scala/bug · GitHub to track this on the Scala 2 side.

1 Like

Scala 3 Migration Guide has a section called Compatibility Reference, and under it there’s Classpath Level page that covers both sides of the sandwich, including Scala 2.13 TASTy Reader.