Backwards compatibility in Scala

This thread has been split from a previous discussion.


And which problem is that? This has been discussed at length in other mediums, and I don’t think it’s a problem – it’s how the language is evolved, with obvious advantages and disadvantages.

Also, let’s remember how easy cross-compilation is for library authors. Your complaints on 2.10.x are valid – they were the first version to include Scala Macros, so of course that wasn’t polished and has improved in the next major versions. But I’d say this is an issue of this particular Scala version for your particular use case – not a problem of cross-compilation per se.

Well, I feel that this isn’t the right topic to discuss it, but here’s a small rant …

You can take any Java library that hasn’t been updated in 10 years and still use it and compile it without problems on the latest JVM. You can take most libraries developed for Clojure 1.0 and still use them on the latest Clojure 1.8.0. You can almost say the same thing about JavaScript as well. Of course, all these aforementioned platforms have issues with libraries breaking compatibility in newer versions. Scala does too, in addition to Scala itself breaking compatibility in every major version.

So in the words of Rich Hickey, there’s accretion and there’s breakage and breaking backwards compatibility is breakage. I would have loved for us to come up with a better convention, instead of using that dreadful “semantic versioning”, which is basically a recipe for breakage. We could choose for example to never, ever break backwards compatibility without changing the namespace.

But we don’t do that, instead we are hoping for some magical unicorn (e.g. something OSGi-like) to save us from our sins.

I really believe there are absolutely no advantages to how Scala is in this regard. We tolerate it because we are getting lovely features instead. But we are already seeing old codebases stuck on old Scala versions with old library versions, because there’s really no good return of investment on upgrading everything, since the costs are huge and upgrading it piecemeal is mission impossible.

So in my mind, even if getting new features is great, even if we got cross-compilation tools instead, Scala is a terrible, terrible option for long lived codebases.

1 Like

@alexandru While I understand your general frustration, I think that you’re oversimplifying the problem at hand. Decisions on this topic are not simple, they have both advantages and disadvantages. This discussion is contentious in a lot of other programming language communities. You’ve cited the most common programming languages that do have a notion of binary compatibility built-in into the language. But there are many others that don’t – and compromise similarly to how Scala does.

I disagree with this idea of “semantic versioning being a recipe for breakage”. Semantic versioning is necessary to have a “semi-formal” way of reasoning about software changes and its implications to their ecosystem. It’s a concept useful on its own – it lacks precision, but it’s better than nothing.

I believe that the whole Scala community should use a refinement of semantic versioning adapted to Scala – where source and binary compatibility are taken into account. But that’s another topic.

I agree this is a good idea in general. There are, however, certain scenarios in which this is not possible – the standard library is one of them. The standard library is part of the language. Using different namespaces would create a ridiculous overhead for those interfacing with libraries whose interfaces use previous versions of the collections library, for example.

As you probably know, the Scala standard library will be overhauled by the Scala Core + Scala Platform combo, allowing libraries to evolve much, much faster in the Scala Platform, where most of the utils will be maintained. So I see this move as a nice, future improvement.

There are indeed benefits. Scala doesn’t break bincompat just because the Scala team is keen on giving users new features. Oftentimes, bincompat is broken because the encoding of certain features changes, either for better performance or correctness. Evolving a language means polishing the rough edges, too. I would not be happy coping with decisions that were made several years ago and were afterwards deemed bad ideas and changed. Under your proposal, I would need to cope with them.

I don’t believe this is mission impossible. Take the 2.11.x-2.12.x migration. From my OSS point of view, it hasn’t been so painful – I agree this may not be the case for certain industrial software. There are indeed libraries that have slowed the process down, some of them for lack of economic incentive, some others because of technical problems (Spark).


Languages like Java (and JS) have a substantial advantage over Scala because they control the runtime – they can afford themselves to implement lambdas in a bincompat way in both the VM and the language. Clojure is an interesting exception and won’t comment on it since my understanding is limited. However, I venture to say that Clojure is much simpler than Scala (it needs less syntactic sugar and encodings, which makes it significantly easier to keep bincompat across major versions).

If you want to move this discussion forward and my arguments don’t convince you, I believe it’s better to discuss concrete, technical details on how you would evolve the language without breaking binary compatibity and what implications that would have for language designers, library authors and users.

FWIW, there is an ongoing SIP to improve binary compatibility in Scala releases, which I think it’s relevant for this discussion: http://docs.scala-lang.org/sips/pending/binary-compatibility.html.html.

Would the overhead of assigning different versions to different
namespaces be large?

Let’s say package scala is the current version, and then we also have
packages scala210, scala211, scala212, scala213 etc.

If you have code written for Scala 2.10, and you want to compile it in a
Scala 2.11 environment, couldn’t you just do so by adding the following
line on top of each file:

import scala210 => scala

1 Like

A post was merged into an existing topic: Continuing or dropping Scala 2.10 maintenance in the ecosystem?

I feel the same way. Ironically, Sbt decided to keep binary compatibility by sticking to Scala 2.10 for so many releases. I think nobody questions the success of that approach (though people complain, the Sbt plugin ecosystem exploded). Same for Spark. Maybe there’s something to consider there, as neither Spark nor Sbt lack user adoption.

From my limited glimpse into commercial Scala development, less than 10% of our customers and leads are in Scala 2.12 today. This seems to contradict the “zero cost upgrade” that we (open-source devs) believe in. While the cost is perceived to be very small for open-source projects, it’s obviously not so for commercial projects. Having tried to move a Play codebase from 2.10 to 2.11, I have to say I gave up since the web of dependencies was too complex (for example, the minimal Akka release for 2.11 was incompatible with the rest of the project). If anyone doubts me, you can feel for yourself: GitHub - softwaremill/codebrag: Your daily code review tool.

This has a lot of downsides as well. Many features (or problems) remain in future releases, because removing them would break backward compatibility:

  • Raw types still compile
  • Poorly designed APIs remain in Java
    • APIs are stuck with bad design and can’t ever be fixed
      • Integer.getInteger (and related)
      • Properties extends Hashtable
      • Clonable interface (with no public methods)
      • Iterable is in java.lang instead of java.util with other collections-related stuff (including Iterator)
      • Interfaces couldn’t have static methods until Java 8
        • Static methods for all collections are in Collections, rather than the appropriate interface (List, Map, etc.)
        • Static methods for interface Thing are in Things
      • Date is mutable
    • New, improved APIs have to use new, often less intuitive (and more verbose) names or namespaces
      • CompletionStage (similar to Scala’s Future)
      • NavigableMap and NavigableSet (couldn’t change public API for SortedMap and SortedSet)
      • java.nio
      • LocalDateTime
        • Immutable alternative to Date
        • I didn’t even know this existed until just now
    • It’s often difficult to figure out which API to use (bad/old vs new/good)
      • URL vs URI
      • Enumeration vs Iterator
      • Stack vs Deque
      • Vector vs ArrayList
  • Older classes and interfaces are all dumped in java.util, and can’t be better categorized
    • New classes tend to be in more specific packages (e.g. java.time.temporal)
    • Collections framework is stuck in java.util, and can’t be moved to java.util.collection (or something)

The list above is by no mean exhaustive.

Additionally, new features added to the language have to work around existing syntax, often leading to significantly increased verbosity and an uglier language (e.g. value types with generics may require Parametric<any T>).

There are a lot of downsides to maintaining backward compatibility as well. It’s a trade-off. Java’s decision to keep everything backward compatibility is not strictly better than Scala’s decision not to. It has advantages and disadvantages.

3 Likes

It seems there are two separate issues here: upgrading Scala and upgrading a dependency. Upgrading Scala would have (probably) been trivial if all dependencies were available for 2.11. Upgrading Akka to a new version was painful because it requires you to find new matching versions of your dependencies. But would that be any easier if Scala releases were binary compatible? It seems the difficulty comes from the dependency graph, upgrading one thing requires you to upgrade the rest in lock step.

If Scala releases were binary incompatible, upgrading to 2.11 would not have required you to upgrade Akka. But is that really a problem? If a project stays with old libraries, it can probably also stay with an older Scala version. And if a project stays on an old Scala version with old dependencies, that doesn’t affect oss libraries in moving forward (and dropping support for older Scala releases).

1 Like

The types in the standard library define (de facto) a standard for data exchange between libraries. If one of your libraries gives you a scala212.List but the other one asks for a scala211.List you need a conversion.

I find the Java issues you mention not so bad and a small price to pay
for backwards compatibility. They have been irrelevant to my decision to
switch from Java to Scala.

That raw types compile is embarrassing, but you should get a warning, and
you should be able to configure your development environment to upgrade it
to error, if you desire.

I don’t see the problem with Integer.getInteger. In fact, I wish Scala
had a Number type from which Int, Long, Float, Double etc inherit.

When you say Clonable is poor design, I guess you mean the fact that
there is a method java.lang.Object.clone(). But since scala.AnyRef is just
a synonym for java.lang.Object, Scala is stuck with the same method,
despite all the breaking of backwards compatibility.

Regarding URL vs URI and Vector vs ArrayList: these are not examples of
old versus replacement, but they address different use cases.

That types or methods a located in awkward or unexpected places is in my
mind not that big of a deal.

The real selling points of Scala, in my mind, are type inference,
operators as identifiers, uniform access principle, merging of constructor
and type declaration, auto-generation of getters and setters, case classes,
easy declaration of functions (including closure), etc.

Many of these Java could not adopt easily, because it would have required
changes of syntax or semantics, and some could only be adopted awkwardly
(e.g. functions). Changing syntax or semantics is basically creating a new
language.

On the other hand, Java could also have adopted many innovations but
didn’t (like renaming imports).

 Best, Oliver

You can call them two separate issues if you wish, but it’s really the deadly combination of the two that makes it too hard to upgrade. The fact that every major Scala release requires a reboot of the ecosystem is the root cause, and if Scala was binary compatible there wouldn’t be a need to upgrade both the Scala version and the dependent library. It’s really the exception rather than the rule that one could upgrade them independently. Unless I miss something, it’s really Scala that forces this lockstep evolution.

Yes, people can stay on an old Scala version (as many actually do), but that can’t be what the Scala team wants (i.e. longer support for old versions).

Of course, it’s a policy that the Scala team decides on, and the downsides may be carefully weighted and the decision may still be to keep the status quo. But every time this comes up I’m told how easy it is to upgrade, and that anyone saying otherwise is doing it wrong. So I’m just adding my 2c saying that it’s not really like that, just trying to make sure you are aware of this.

To put this another way, why do you think Spark and Sbt decided to stay so long on a deprecated Scala version? Binary compatibility must be important enough for building their ecosystem.

1 Like

On the other hand, the fact that Scala evolved into what it is now was probably only possible because it hasn’t been stuck in eternal compatibility mode.

4 Likes

You are aware that it returns the Integer value of a system property with the specified name, right?

I mean the fact that clone() is defined for Object, not Clonable, and that the Clonable interface contains no methods. Unlike every other interface (e.g. List), Clonable does not require you to define the relevant method. The clone() method ought to be defined (publicly?) in the Clonable interface, or something.

From the javadocs for URL.equals(Object):

Since hosts comparison requires name resolution, this operation is a blocking operation.

Note: The defined behavior for equals is known to be inconsistent with virtual hosting in HTTP.

From Vector:

Unlike the new collection implementations, Vector is synchronized. If a thread-safe implementation is not needed, it is recommended to use ArrayList in place of Vector.

Neither of these behaviors is expected or particularly desirable in most cases. For Vector, I don’t know that there is any advantage over Collections.synchronizedList(new ArrayList()) anyway.


In Scala, if an API is poorly designed, it can be fixed in a later version. For example, the collections framework is being rewritten again (to simplify it and improve ease of use). In Java, you are stuck with the poor APIs.

1 Like

Ok, you got me on Integer.getInteger, which I confused with
Integer.intValue. I agree Integer.getInteger is a wart, but I wouldn’t
consider removing it such a priority that I would break backwards
compatibility for it.

If you have two parts of the standard library have similar functionality
such that it is not easy to tell which to use, I’m not such that this is
primarily an API issue?