Yes, this is exactly what I’m talking about.
I know a lot of contributors, who are not using this forum, and they will admit in private that this is becoming a problem.
Yes that is correct. As somebody with no personal or professional interest in scalajs or native I find it hard to understand why I need to support it when creating a new library. Given that I mostly write tooling, this ends up being an even larger burden. It’s fair to say, for example, that I’ve dedicated a lot of time to getting it “barely working” in ensime and I’m not maintaining that so it will rot and be “not working” very shortly.
Btw, I encourage its use in ensime-atom and ensime-vscode because I think it’s a great tech match there, and I’ve used it in the past and been very impressed. I just don’t think I should have to pay the tax for scalajs and native users.
I’d prefer it if there were no separate projects at all, with separate compileJs/testJs tasks but I appreciate that it could be a huge change.
To be honest I found it a bit hard to integrate, and i.e. this approach is hard to scale (i.e. you cannot apply it through a sbt-plugin but you have to keep this configuration in end projects).
Regarding libraries and “std-lib” is actually a really long story, I do not think this is a proper thread to discuss this topic.
2 Likes
I agree that this thread is getting into a bit of a digression… can somebody please split it off into its own discussion (perhaps even the first post in this thread) so that we can focus on scalajs/native distribution mechanisms.
On topic in regards to the scala-js/scala-native to provide a clearer picture of my personal problem. I am currently trying to release scalajson (https://github.com/mdedetrich/scalajson). The idea of this project is its meant to be ultra stable (i.e. release as little as possible) with zero compile time dependencies.
The project had support for Scala.js for quite some time but because it was never released I had the privalige of just bumping to the latest Scala.js version all of the time. However now that I am going to get to a point where I need to release the libary, I have this predicament where a lot of the Scala.js community is on 0.6.x and some is moving to 1.0.0x. If scala-js was the only other backend, just using environment variables as a workaround would have been tolerable.
The issue that is now happening however is that I want to support scala-native. The reason why I want to support scala-native is that argonaut (http://argonaut.io/) and very likely other JSON libraries in the future (such as circe) are going to be publishing to scala-native. This is where the environment variables workaround starts really getting messy, because I will need to publish scalajson for scalajs 0.6.x, scalajs 1.0.0 and scala-native 0.3.x and 0.4.x (when it gets released) as a matrix. This is going to get worse as more scala-native versions come out (hopefully Scala-js will continue to be binary compatible for 1.0.0.x).
The situation is even more complicated with scala-native/scala-js because each of these plugins target specific versions of Scalac. For example Scalajs 0.6.18 only targets Scala 2.12.3 and 2.11.11, 2.10.6 and earlier versions (I think, I may be missing some versions), where as earlier versions of Scalajs 0.6.x (lets say 0.6.14) target Scala 2.12.1, 2.11.8 and 2.10.6 with earlier versions NOTE: I can’t remember the exact versions of the top of my head but the point is the scala-js/scala-native plugins have to be compiled for specific scala versions).
Again if this was just Scala.js it may not be an issue (currently its fine for my use case because I haven’t released yet), but what happens if I am targeting Scala version 2.12.3 (for the 2.12.x version) and there are Scala.js 0.6.x and 1.0.0.x versions for this Scala version, but there isn’t a scala-native version for 2.12.3 (because I am targeting an older version of scala-native which was only compiled for, lets say, Scala 2.12.1).
Now I have to put my scala versions also as environment variables and I get this really complicated matrix. Whats even worse for my personal use case, since I am targeting the Scala platform I apparently have to use SBT (see https://github.com/mdedetrich/scalajson/issues/34 ), supposedly CBT should have better support for this scenario.
Due to these issues (and others) I have kind of lost motivation for scalajson. I definitely wan’t to release scalajson soon, but for the past few months I was just kind of contemplating where to proceed from here. Actually at this point in time, I can’t really think of a sane way to fix my issue. I could just only support scala-js and not scala-native, but then If I want to get argonaut to either integrate scalajson, its going to be really stupid/hard for them if they can only provide one version of argonaut (with scalajson) for scala/scala-js and another (without scalajson) for some version of scala-native because I chose not to target the platform.
I think you can avoid that issue if you look at things in the following way. At any given time, for a particular commit in the history of scalajson, you can choose to use the latest version of Scala that is supported by the versions of Scala.js and Scala Native that you use. Note that for that commit, you can always choose to upgrade Scala.js and Scala Native to a newer version, since it is a new commit and it will therefore be part of a new release. Therefore, for that commit, all platforms can be built with the same Scala version, and there is no need for an additional environment variable.
If, in the future, a new major (aka binary breaking) version of Scala Native comes out, and you’d like to back-publish scalajson for that version, you can come back the tagged commit of your release, switch the version of Scala Native, and rebuild and republish. Since the newer Scala Native should still support the older Scala version that you used at the time you made that release, you won’t have a problem with the Scala version either, in this scenario. (Note: I am not sure whether Scala Native actually has a policy of always keeping support for earlier versions of Scala. Scala.js has that policy: every new version of Scala.js is published for all the minor versions of Scala available at the time of that release, back to 2.10.2).
It seems to me that this strategy should work in your specific case. Did I miss something?
The problem is that this may not always be the case. If the version of scala-js is too new (lets suppose scala-js 1.1.0 is targeting scala 2.12.5 for some versions in the future) and I happen to be supporting the current latest scala-native 0.3.3, there isn’t a single Scala release that can satisfy both of these requirements.
As an alternative I can have a seperate branch just for scala-js and another one for scala-native so that I never load both plugins at the same time, but again this workaround complicates matters. I mean at the end of the day, I have so support a matrix of all of the binary compatible release versions that satisfy scalac and all of the other backends.
Well the thing is, the publishing is meant to be handled by the scala platform SBT plugin, its not meant to be done manually (currently its being done manually but this is because the scala platform plugin is not ready for SBT and SBT 1.0.0.x needed a newer version of Scalajson). Point is that in the future, I am not going to be doing +publishSigned, instead there is going to be some Task provided by the scala platform sbt-plugin which I need to run.
Being forced to use the scala platform SBT plugin is also the reason why I can’t do this Explore CBT · Issue #34 · mdedetrich/scalajson · GitHub (I actually wanted to see if CBT has a much cleaner way of doing this)
That does not happen. Scala.js 1.1.0 would still support 2.12.3 in addition to 2.12.5, as well as any previous version of Scala that was supported by earlier versions of Scala.js. We don’t stop supporting old versions of Scala. For example Scala.js 0.6.20 supports [2.10.2, 2.10.6], [2.11.0, 2.11.11], [2.12.0, 2.12.3]. Scala.js 0.6.21 will support at least all those versions.
It is possible for an older version of Scala.js not to support a newer version of Scala. But a newer version of Scala.js always supports all the older versions of Scala.
I am not familiar with the constraints imposed by the Scala platform sbt plugin. Perhaps the necessary scenario can be built in this sbt plugin? I guess it would apply to other projects as well.
The Scala Platform sbt plugin does not have any constraints in this regard to the best of my knowledge.
The task you’ll need to run is releaseEarly, which internally calls publishSigned.
Yes, I just don’t know whether or not scala-native will have the same guarantee. I am not saying I have this problem now (I haven’t released yet!) but afaik scala-native versions are not binary compatible (i.e 0.2.x is not binary compatible with 0.3.3) and the releases happen fairly often. So when I do release scalajson (with the latest scala-native version at the time, lets say 0.3.x) then I am going to be stuck to the latest version Scala version that scala-native compiled against for 0.3.3).
This isn’t purely just about Scala.js (if scala-native didn’t exist then it wouldn’t be too much of an issue) however this isn’t the case and libraries are already publishing for scala-native. I also don’t want to impose binary restrictions onto scala-native as I think the project is too young and it would be detrimental to scala-native’s development (they don’t have a stable runtime, and they shouldn’t have one until they decide how to handle multi-threaded programming)
Okay so I think this will work with +releaseEarly as well.
I guess I will see if I can set up a build matrix for scalajson with environment variables to handle the issues I am having, its going to be a bit nasty but I don’t have another choice right now.
And scala-native?
Scala Native has the same design principle. However, since it is less mature, we still sometimes discover “design bug
s” that cause NIR not to be binary compatible when it should. However, they are being fixed. Therefore, modulo bugs, sbt-mima also does the correct thing for Scala Native.
We’ve found a mima-incompatible flaws in NIR design in 0.3.x, that’s going to be addressed in 0.4.x.
As for Scala Native, it is less mature, so yes, these things will still happen for Scala Native. However, precisely because it is less mature, I would advise library authors against cross-compile for different versions of Scala Native at the same time. This is mental. Just upgrade to the latest Scala Native version and only publish for that one. Forget about older versions. There isn’t a large ecosystem on things stuck on old versions of Scala Native that you still need to support.
This is the best way to go at the moment. Scala Native supports only one (latest) version at the time. It’s advised to upgrade to the latest version.
Yes, I just don’t know whether or not scala-native will have the same guarantee. I am not saying I have this problem now (I haven’t released yet!) but afaik scala-native versions are not binary compatible (i.e 0.2.x is not binary compatible with 0.3.3) and the releases happen fairly often. So when I do release scalajson (with the latest scala-native version at the time, lets say 0.3.x) then I am going to be stuck to the latest version Scala version that scala-native compiled against for 0.3.3).
In Scala Native we do releases once a month. While we don’t have any exact timeline for 0.4.0, 0.3.x series is likely to last at least up until the end of the year, making 0.3.x series stay compatible for 6 months or more, before 0.4.x happens. I don’t think we can guarantee more than 6 month compatibility window for older releases at this stage of the project.
We’ve been slow to address 2.12 support in Scala Native, but the list of supporting version is strictly monotonically increasing. Once the version is supported it’s going to stay that way with newer releases of Scala Native. (i.e. once 2.12 supported is released we are not going to drop 2.11.)
Unfortunately this conflicts with one of the goals for scalajson, so I will have to see how this plays out.
My main issue is with the build matrix that is expanding, although this gets to the earlier point keeping updates for scala-native.
p.s. The point of scalajson also is that it should rarely (ideally never) go past v1.0.0. So if new versions of scala-native/scala-js/scala get relased, we aren’t going to bump the version of the library
We already have rather solid compatibility for the core Scala language (given that large projects like scalac run without modifications.) If library doesn’t use any of the language extensions, changes necessary to support newer versions of Scala Native are limited to bumping the version in the build and re-releasing the library once per major (i.e. 0.x.y to 0.(x+1).y) release of Scala Native. Minor releases (i.e. 0.x.y to 0.x.(y+1)) are one-way binary compatible and don’t require any changes from the library author.
Going back to Heather’s question: I’m happy that @sjrd could find a solution to all technical issues. I also respect @mdedetrich’s skills. But if it takes Scala.JS creator to figure this out, publishing for Scala.js seems still a bit challenging. Not sure what to do, since cloning @sjrd’s not an option. (If you know me, you should expect the word “docs” and it might be the right one—but I can’t tell).
1 Like
To be clear, if you only support Scala.js then its really not that much of an issue (at worst you have to use a single environment variable for the scalaJS plugin version since you can’t crossbuild against scala plugins). Things however get a lot more complicated if you want to support Scala.js and scala-native and you are publishing a library that is designed to be very stable and takes binary compatibility very seriously (i.e. if we suppose that akka was cross compiling against scala-js and scala-native, it would have the exact same issues).
This is where you have to form some sought of a build matrix or do very tedious manual publishing (which will probably work fine in my case because I never expect to publish after 1.0.0 of scalajson is released, but for other libraries which are expected to release more then one version this is going to be a PITA)
1 Like
Hello,
It is a social rather than a technical problem. As some one else said
earlier: if you have something valuable, people will come and adapt it to
different targets.
If you want to start a new library, the best is start it for one Scala
version and platform, and instead of worrying too much about new targets,
put the focus on making sure the library does well what it wants to do.
And, make sure others can easily join as contributors.
Libraries usually don’t fail due to lack of supported platforms, but due
to lack of features, bugs or other limitations.
A few years ago, I wanted to use Scala Swing, a wrapper for Java Swing.
The problem: Scala Swing did not wrap all Java Swing widgets, so it turned
out to be useless to me.
Other libraries over time I have avoided because they were incomplete,
hard to understand, cumbersome, poorly documented or they made code take
insane times to compile.
Best, OliverOver the past two days, I have hacked up together this sbt plugin:
As the readme explains in detail, this sbt plugin, named sbt-dynscalajs (for dynamic Scala.js), allows to dynamically switch the target of projects in a build from JVM to JS, and also allows to switch between versions of Scala, based on the value of a setting. In a nutshell, all projects are JVM by default. And with a simple
set dynScalaJSVersion := Some("1.0.0-M1")
(which is basically the same difficulty, if not verbosity, as ++2.12.3), they are magically turned into Scala.js projects, using Scala.js 1.0.0-M1. fastOptJS, fullOptJS, scalaJSStage, run, test, and all those good things are available as in normal Scala.js project. Other Scala.js versions that are supported are 0.6.19, as well as 0.6.20 but without test support at the moment.
This plugin effectively makes it as easy as ++2.12.3 to switch between targets and between versions of Scala.js. However, it has caveats (see readme). It should for example definitely not be used by developers who are doing Scala.js-specific things. It is only intended for library maintainers who don’t want to deal with Scala.js, yet support it nevertheless.
Also, it obviously does not support Scala Native. So don’t use this if you also want to support Scala Native.
Make sure to read the readme, especially about the expectations you should (and should not) have about this plugin and its maintenance.
Also, I suggest you actually read the source code if you want to invest in the discussion on this thread, as it will provide you with a relative clear picture of the challenges 
7 Likes