Improving Scala 3 forward compatibility

[nafg]

Why did --scala-target become -Yscala-release? The latter name is a lot less self-documenting. There’s no way to know what it does without having learned about it, while --scala-target is self-explanatory.

The other thing I’m confused about is I thought TASTy was supposed to be a lot more stable than classfile encodings and hardly ever have to change in a breaking way. And as for non-breaking changes (like new AST node types) why can’t that work dynamically? Why can’t the consuming scala compiler worry about it if and when it finds such a node? That way the producing compiler wouldn’t need a flag at all unless it could generate the same thing in an older encoding, or if I want to statically guarantee that I support a certain version (so if I accidentally use a newer feature I’ll know to take it out).

It seems like that would make life simpler for a lot of people.

A slightly different take is if the producing compiler knows what version of Scala introduced each TASTy node, it can at the end put in the file the minimum required TASTy version to read that particular file.

As for the standard library annotations, I’m confused why the standard library can’t be treated like just another library here. If my library depends on a newer version than some application using it, why can’t the normal eviction rules apply? Especially if my previous suggestion were implemented, that would enable me to use anything from the standard library that I got bumped to unless it uses a newer TASTy feature, at which point I would be told to upgrade my compiler version.

[Kordyjan]

@nafg

Why did --scala-target become -Yscala-release? The latter name is a lot less self-documenting. There’s no way to know what it does without having learned about it, while --scala-target is self-explanatory.

The flag is named -scala-release to make it look similar to -release flag that is doing exactly the same but on bytecode instead of TASTy level. Y is here to mark the flag as unstable, and will be dropped in 3.2.

@nafg

As for the standard library annotations, I’m confused why the standard library can’t be treated like just another library here.

For now, the standard library is treated in a special way by the compiler. In the future, it may be possible to treat stdlib as any other dependency, and then annotations will be only used for documentation purposes. We are thinking about that.

[Jasper-M]

@Kordyjan

The flag is named -scala-release to make it look similar to -release flag that is doing exactly the same but on bytecode instead of TASTy level.

I thought that the target flag specifies which version of bytecode to emit, and release specifies which version of the java std library to compile against. Or is that different in Scala 3 vs Scala 2?

[prolativ]

For javac both -target and -release allow you to specify the target version of JDK to produce bytecode for. The difference is that -release additionally checks that the parts of the stdlib API that you reference actually exist in this JDK version. -target is more like YOLO mode, hence it was renamed to -Xtarget in scala 3 and it’s usage is discouraged - -release should be used instead whenever possible. As for compiling to older TASTy we also perform the checks, it’s more like -release rather than -Xtarget.

[smarter]

@Jasper-M

Or is that different in Scala 3 vs Scala 2?

Yeah I think Scala 2 -release doesn’t set the target, but both Scala 3 and javac -release do, IMO the behavior of Scala 2 should be changed to align with the others.

[eed3si9n]

@nafg

Why did --scala-target become -Yscala-release? The latter name is a lot less self-documenting. There’s no way to know what it does without having learned about it, while --scala-target is self-explanatory.

I also find the term -Yscala-release confusing, even though I think I am relatively familiar with JDK tooling. In JDK, the idea of release is (and has been) a thing – JDK Releases for example lists all JDK releases and their associated release type, release support timeline.

JEP 238: Multi-Release JAR Files and JEP 247: Compile for Older Platform Versions extended Java compiler toolchain to support multi-release JAR file that contains multiple A.class against the Java class A, but targeting JDK 8 and JDK 11 etc. In that context the flag --release, kind of makes sense because

javac --release 8 -d classes src\main\java\A.java
javac --release 11 -d classes-11 src\main\java11\A.java

are denoting JDK releases, and often compiled against different source code with one using Unsafe etc.

Since 2018 (Scala 2.12.5) Scala 2.x has supported --release 8 in the same semantics (https://github.com/scala/scala/pull/6362), which allows downgrading of JDK target and generating multiple *.class.

-Yscala-release 3.0 I don’t think translates into Scala version context, chiefly because we don’t use the term “release” to describe minor versions within a binary compatible series, and it might also be confusing especially because the idea of forward compatibility was something Scala 2.x library authors took for granted.

I am commenting here because there’s now a proposal that translates -Yscala-release name into an sbt setting (Add support for Scala 3 -scala-output-version flag by prolativ · Pull Request #6814 · sbt/sbt · GitHub):

ThisBuild / scalaVersion := "3.1.2-RC1"
ThisBuild / scalaReleaseVersion := "3.0.2"

@prolativ

-target is more like YOLO mode, hence it was renamed to -Xtarget in scala 3 and it’s usage is discouraged - -release should be used instead whenever possible.

This notion that the word release implies better checking of API compatibility but target doesn’t is an implementation detail I don’t most people appreciate or care since most libraries probably just use older JDK to build and not create multi-release JAR.

As alternative, I think something like

ThisBuild / scalaVersion := "3.1.2-RC1"
ThisBuild / targetScalaVersion := "3.0.2"

or

ThisBuild / scalaVersion := "3.1.2-RC1"
ThisBuild / compatibleScalaVersion := "3.0.2"

would be more intuitive. Alternatively:

ThisBuild / compileTimeScalaVersion := "3.1.2-RC1"
ThisBuild / scalaVersion := "3.0.2"

I think also should be considered. Primary objection (Add support for Scala 3 -scala-output-version flag by prolativ · Pull Request #6814 · sbt/sbt · GitHub) to that is that now we’d have to rewrite builds that uses scalaVersion to pick compiler options, which is fair.

[Kordyjan]

I really like scalaVersion / compatibleScalaVersion pair for an sbt setting. I find it much more intuitive than any other proposed option. Now I’m thinking about what should be used as a name for the compiler flag.
-compatible 3.0, -compatibility 3.0, -compatibility-version 3.0, -compatible-version 3.0? Any other ideas?

[smarter]

Whatever we choose, I think we should consider having a matching setting for setting -release X in scala 3 / javac and -release X -target X in scala 2 (because in scala 2 only -release doesn’t imply -target as discussed above). So for example if we go with compatibleScalaVersion I would call that setting compatibleJavaVersion.

[prolativ]

How about outputScalaVersion or scalaOutputVersion setting and -scala-output compiler flag?

[eed3si9n]

I am ok with either

ThisBuild / scalaVersion := "3.1.2-RC1"
ThisBuild / outputScalaVersion := "3.0.2"

or

ThisBuild / scalaVersion := "3.1.2-RC1"
ThisBuild / compatibleScalaVersion := "3.0.2"

For compiler flags:

scalac --compatible-java 8 --compatible-scala 3.0 -d classes src\main\scala\A.scala

scalac --output-java 8 --output-scala 3.0 -d classes src\main\scala\A.scala

Both work well here as well I think.

[soronpo]

Instead of compatible, I propose compat

[prolativ]

One argument against compat(-ible/-ibility/-) is that people might think that the compiled code will either only work with this particular version or will work only with versions which are chronologically equal or later, which is not entirely true if we take patches into account - code compiled with compatScalaVersion := "3.0.2" would work with code compiled with 3.0.0 compiler but it would bump the version of the stdlib to 3.0.2

[mushtaq]

outputScalaVersion is much easier to understand IMO.

[prolativ]

After some more discussions (including the core compiler team) here’s my suggestion how to make the naming (in the compiler and in sbt, but also potentially in other build tools) consistent and less cryptic. If multiple entries exist in the same cell then they’re interchangeable (that’s in order not to break compatibility).

	compiler (current)	compiler (suggested)	sbt (current)	sbt (suggested)
compiler version (e.g. for `sbt compile` or `sbt test:compile`)	scalac -version	scalac -version	scalaVersion := "3.1.2-RC1" ++3.1.2-RC1	scalaVersion := "3.1.2-RC1" ++3.1.2-RC1
version of produced TASTy files (expressed with corresponding scala version; validates references to scala stdlib API)	-Yscala-release 3.0	-scala-output-version 3.0	scalaVersion := "3.1.2-RC1" ++3.1.2-RC1	scalaOutputVersion := "3.0.2"
stdlib version used at runtime (e.g. for `sbt run` or `sbt test`)	scala -version	scala -version	scalaVersion := "3.1.2-RC1" ++3.1.2-RC1	scalaOutputVersion := "3.0.2"
stdlib version declared as transitive dependency in Ivy and POM files (for compile time and runtime of dependent projects)	-	-	scalaVersion := "3.1.2-RC1" ++3.1.2-RC1	scalaOutputVersion := "3.0.2"
version of produced JVM bytecode (validates references to JDK API)	-release 8	-java-output-version 8 -release 8	scalacOptions ++= Seq( "java-output-version", "8") scalacOptions ++= Seq( "release", "8")	scalacOptions ++= Seq( "-java-output-version", "8") scalacOptions ++= Seq("-release", "8")
version of produced JVM bytecode (doesn't validate references to JDK API; discouraged)	-Xtarget 8	-Xunchecked-java-output-version 8 -Xtarget 8	scalacOptions ++= Seq( "-Xunchecked-java-output-version", "8") scalacOptions ++= Seq( "-Xtarget", "8")	scalacOptions ++= Seq( "-Xunchecked-java-output-version", "8") scalacOptions ++= Seq("-Xtarget", "8")

[prolativ]

Hopefully a more readable version as an image

[eed3si9n]

@prolativ

After some more discussions (including the core compiler team) here’s my suggestion how to make the naming (in the compiler and in sbt, but also potentially in other build tools) consistent and less cryptic. If multiple entries exist in the same cell then they’re interchangeable (that’s in order not to break compatibility).

Thanks for the update. This looks good to me.

[eed3si9n]

I was randomly skimming through my GitHub notifications and came across an interesting pull request by Paweł https://github.com/scala/scala-lang/pull/1387.

More importantly, during work on 3.2, we realized that maintaining this flag may be more challenging than anticipated. With time, it will be increasingly harder to be sure that our handling of it is correct. Thus we have decided to remove the possibility of configuring the output version altogether in Scala 3.2.

I wish you well for the LTS / Next approach, but in the meantime, should I remove the newly added settings from sbt since I’ve only released a milestone thus far?

[Kordyjan]

@eed3si9n

should I remove the newly added settings from sbt since I’ve only released a milestone thus far?

Yes, feel free to remove it. We were going to make a PR reverting support for the flag as soon as we deal with the communication of the decision.

[prolativ]

Besides directly supporting -scala-output-version that PR also added a more general possibility to use different versions of Scala for compilation and for running the code. I’m still wondering whether someone might find this part useful in some situations so maybe the PR should be reverted only partially?

[eed3si9n]

I am sympathetic to keeping working code, but I’m not convinced if it’s worth keeping it around. For one, the triggering mechanism hinges on having scalaOutputVersion being different and dependency graph having different scala-library.jar in Runtime. The problem is, without the compiler actually supporting forward compatibility someone compiling with 3.2.0 etc and declaring it at 3.1.2 in POM is unsafe. One potential use as a library author might be to use a nightly compiler to circumvent @experimental , but I don’t know if you y’all want ppl doing that.

In general, we should try to make illegal states unrepresentable, and that applies to builds as well.