Hi, I was wondering if there will be any sort of mandate set forth for Scala 3 (dotty) to maintain binary compatibility between version increments? It’s been a bit of a pain point in the Scala 2.1x world for me, especially when maintainers of external dependencies fail to keep up with cross-compilation efforts.
If we could remove the need for cross publishing libraries in scala 3 that would be a huge win for library maintainers.
I would be willing to accept much slower evolution of standard libraries in exchange for not needing to move the ecosystem in step every two years when a new scala compiler is released.
2 Likes
Binary incompatibilities arise also from changes in bytecode and scala pickle, such as changes in lambda encoding, changes in lazy val encoding, etc. So it impacts more than just standard library evolution.
However, I also think it’s a costly tax on the ecosystem and one I’ve been thinking about. TASTY might help, but it’s not straightforward.
1 Like
One of the things I admire about the Java language designers (Brian Goetz comes to mind) is their ability to evolve the language whilst keeping binary compatibility in check. Not trying to open myself up to a debate on Java vs. Scala. I’m aware of the trade-offs of course, but still, they make a go of it even with the binary compatibility constraints they face and the language is evolving forward.
My main Scala project is locked to Scala v2.11. One of the fundamental libraries it was built with was never cross-compiled to 2.12 by the maintainers. The solution of course is to either fork the library and try to get it to be compatible with 2.12, or find another library and integrate that instead. But my point is, I shouldn’t have to.
But please don’t take my statement as me placing any sort of entitlement on the backs of many people’s hard work on Scala. I just want to be able to confidently recommend Scala 3, because as it stands I always have to include the caveat “but just be aware, upgrading minor versions can be a big pain!”
So, I feel that Scala 3.0 needs to be much more cognizant of binary compatibility. It would be really nice if there was a mandate, or statement to that effect. I don’t think it’s a desirable goal to have a repo full of artifact_3.0-.1.0.jar, artifact_3.1-.1.0.jar, artifact_3.2-.1.0.jar… It’d be great if we only needed the _3 and could leave it at that.
@dwijnand I gather from your TASTY comment that you feel as though macros may be the savior here?
1 Like
Another thought, and this may be super naive on my part, but is it not possible to have a compiler that recognizes dependencies are binary incompatible but then somehow weaves in code that bridges the gaps? I’m kind of thinking in terms of containerization. In this way, applications wouldn’t cease to work just because of laggard dependencies, albeit they might not perform that well until they are cross-compiled.
Just a thought, and I’m sure it’s not an original (or even valid?) one.
Theoretically they can use java instrumentation and asm to replace method call to dynamic call.
https://docs.oracle.com/javase/8/docs/api/java/lang/instrument/ClassFileTransformer.html
I personally think It would be great if a class can be annotated as “scripted”, in order to all call to object of that class will be implemented via invokedynamic.
1 Like
Wasn’t thinking of macros at all. I was thinking: we could rebuilt the Scala ecosystem to be based on TASTY libraries, and only at application time create bytecode. But that wouldn’t work for Java users.
3 Likes
My main project is locked to Scala 2.11 too (and it is not related to Spark at all). Hope to move in next week but still, problem remains. If theoretically simple migration (2.11-> 2.12) is sometimes so big deal what’ll happen when real code changes will be needed?
There is also Fury project that tries to simplify compilation libraries directly from sources, and this is another option. If we’ll unify build process enough, then this is another way to go.
Not sure if this is good idea. Sebastien Doeraene has talk where he argues with this concept.
1 Like
Wasn’t thinking of macros at all. I was thinking: we could rebuilt the Scala ecosystem to be based on TASTY libraries, and only at application time create bytecode. But that wouldn’t work for Java users.
I am not very knowledgable on this topic so I apologize if this is a dumb suggestion, but for Java users couldn’t we publish jars as well? If Scala projects were published as TASTY libraries, it would then be presumably fairly easy to publish updated jars even for old, unmaintained projects. I grant that the main problem here is probably rebuilding the Scala ecosystem, but if we can do so and remove the issue of binary incompatibility, that would be amazing.
Another (kind of crazy) idea is that we publish TASTY libraries as resources in a normal Java project with some simple code in the normal published jar that can “bootstrap” the TASTY resources into class files and dynamically load them. I imagine that would be a pretty high startup cost though and not particularly elegant.
Java has all that backward binary compatibility because it runs on the java virtual machine, and the java virtual machine support all of the Java features in one way or another (erasure being an example of “another”).
Scala does not have that backward binary compatibility because it runs on the java virtual machine, and the java virtual machine does not support many of the features that make Scala the powerful language it is.
The “bridge” between Scala and the JVM is where binary compatibility breaks down. TASTY should help out with that bridge, making Dotty able to use Scala 2.12+ libraries and vice versa (not sure what Scala versions are these features planned for atm). And, presumably, it ought to help Scala 3+. See https://www.slideshare.net/Odersky/preparing-for-scala-3 for example (I don’t have a newer reference).
But it must be said that this relies on using the Scala tooling, as far as I know. If you have a Java application using Scala libraries, I’m not sure it can be helped.
Well said. And truly I’m just hoping for compatibility between versions of Scala 3+ compiled libraries, Java->Scala isn’t even on my radar.
I didn’t even dream that Scala 3+ would be able to use Scala 2.12 libs. If that does turn out to be the case, it’s a pretty huge win.
If we’re publishing jars for Java users, then what are saving time on?
I guess one possible improvement is if we could distinguish changes in the “surface language” (source to tasty) vs “mid-level language” (tasty to bytecode)…
You can imagine that maintaining compatibility for Java’s view of compiled scala code is easier, since the java type system is simpler, than maintaining compatibility for scala mapped into class files.
1 Like