The new collections API (to be released with 2.13) is not always source compatible with the current collections API. We should be able to implement scalafix migration rules so that upgrading an application from 2.12 to 2.13 will be simple. However, this might not be enough for libraries, which aim to support several major Scala versions. Consequently, we have to find solutions to make it possible to cross-build between 2.12 and 2.13 despite the important API differences. The purpose of this thread is to list the use cases for which we need cross compatibility and brainstorm about possible solutions.
Once we will have a working scala-library that uses the new collections it will be simpler to try it on a corpus of open source libraries to get a precise idea of where breakages come from, but we already know some of them:
Some operations are still supported but have been moved around or renamed. For instance, toIterator has been replaced with iterator(). In such cases, we can either just not break the API and use the same method name as before (toIterator) or introduce a deprecated alias.
Some operations have the same name but a different type signature. For instance, the to method (like in myList.to[Vector]) now takes a value instead of a type constructor as parameter (myList.to(Vector)). Unfortunately, if we try to add an overload of to with the old type signature we sometimes end up with overloading resolution ambiguities.
In last resort, library authors can use separate source directories for parts of code that are specific to a major scala version.
I have done proof of concepts of cross project building while applying scalafix rules to the build output with CBT. I don’t have anything production ready but it all worked out fine so my suggestion would be to add support for adding scalafix rule support to cross-project building to allow scalafix to run on the source for 2.13 -> 2.12. In general it would be a nice feature for library authors to be able to work with the latest versions of Scala while easily supporting older versions.
We should have a “how to maintain separate source directories with SBT” tutorial and/or sample build file, if that is part of the standard solution.
Also, w.r.t. ShaneDelmore’s suggestion, there are some features in the new library (views) that just don’t have a sensible transformation for the old library, and there are CBF tricks you can play with the old library that you simply can’t do in the new one (that was kind of the point of moving away from CBF), so scalafix should go as far as it can in both directions, but it won’t catch everything.
I would be extremely wary of any situation where I am required to scalafix my library code on-the-fly to cross-compile it. I would never trust it enough (note that tests don’t help, since they would require the same rewritings, so I cannot trust my tests either). scalafix is great as long as I can see and review the diff as part of my commit. I would not trust it to be an automated, invisible part of my CI.
IMO being able to cross-build the (some) same source code is critical to the viability of the new collections library. I had already expressed my concerns on this issue in some GitHub PR or issue, and in particular about the case of .to. IMO having code that just can’t cross-compile ever is a mistake that will endanger our ecosystem.
I would also advise against deprecating stuff that cannot be written in a different way in the previous releases. In Scala.js we already have to deal with this situation for the internal compiler APIs, and we have to constantly live with deprecations in our compiler codebase, which means we cannot enable -Xfatal-warnings, which in turn causes genuine problems to sneak in without our CI detecting it. Ideally, I would like to be able to write code that cross-compiles without any warnings on all versions. But at the very least, I must be able to write code that compiles without any warning in the newest version, and also compiles (possibly with warnings) in the old version.
We should indeed have a consistent story for cross compiling Scala code with all the build tools, and aim to provide better tooling — it can be dangerous for our ecosystem not to do so IMHO.
@julienrf How difficult do you think it would be to make the collections rework source compatible? Is there any technical detail hampering such compatibility?
@jvican As I explained in my first post there are cases where methods have a completely different type signature. An important example is transformation methods such as map, flatMap and collect. In the current collection they take an implicit CanBuildFrom parameter. This is not the case anymore and this is one of the purposes of having new collections. If you want your code to cross-compile and if you use such transformation methods then you must not explicitly supply the CanBuildFrom instance (ie. you can not use breakOut). But I’m not sure how much of an issue that would be in practice (I expect most people to let this CanBuildFrom instance be implicitly filled in by the compiler).
My intuition is that to might be the method that causes most problems. In such a case, we could change it to match the current type signature.
Would it be possible to create a facade library with the 2.13 API that delegates to the 2.12- collections? For 2.13+ this library would just forward method calls, for 2.12- it would perform any necessary translations.
This might even allow the library to be omitted in 2.13+ builds, but I think this needs extra predef/imports customization options in the compiler for the face to also use the scala.collections namespace.
Or this could be provided as a -Y2.13-collections-api flag in a future version of the 2.12 compiler. This is the easiest solution for library authors, but does mean extra work for the scala team and limits cross-compatibility to 2.12+.
Yeah, exactly, but doesn’t this prove my point that it would be source compatible, or at least close to it? I mean, the CanBuildFrom instances are almost always implicitly injected.
Also, one question: have JavaConverters been changed in the new collections rework? Have you finally removed JavaConversions?