(Or perhaps scratch “Windows-only” from that — the core of the suggestion is to handle initial setup only, rather than ongoing package management.)
Windows 10 and 11 have a package manager built-in (winget): Windows Package Manager - Wikipedia. There’s is probably a cultural difference, using a package manager on Windows is less common. However, we’re talking about installing CLI tools, so I wonder if a winget command would really be a hurdle, compared to a double-click installer.
I don’t know who maintains / maintained https://winget.run/pkg/Scala/Scala.2 and https://winget.run/pkg/sbt/sbt.
2 Likes
Scala CLI 1.0.0 is released: Release v1.0.0 · VirtusLab/scala-cli · GitHub
Having addressed all the SIP-46 requirements, this version is going to become the official
scalarunner, replacing the oldscalacommand.
5 Likes
This horse is likely already far far away from its barn, but I was encouraged to post some concerns regarding ScalaCLI, at least as a record or “things to have heard about for future choices”.
I believe this is the right place, but if not please let me know, I will move things.
These concerns are not informed by use of scala CLI in prod yet, but I have similar experience with other lang which chose similar approach (build tool directive in source code, like #! in bash, go directive, c preprocessor, etc) in several decades in prod env and things that go the worst possible way when given the opportunity.
So here the list:
- not principled: comments are not meant to hold semantic instruction, so nothing is built with that on mind which leads to fractal of creative problems because of surprise effect or ecosystem level support of the thing,
- security. It’s an open door to supply chain attacks, hard to notice if your not familiar with the “comments change build/import” idea, and the kind of attack vector that is in full boom lately (so lots of motivated malicious actor learning how to us it)
- tooling. Comments are already hard to deal with, it’s a new level of complexity that is imposed on all tools and the whole ecosystem
- discoverability : hard to grep, hard to find - which file do what ?
- complexity / learning: hard to build a mental model of a build system scattered around, in comments
- refactoring: copy/paste source from one project to another
- consistency: developers doing things in different ways, junior vs senior
I know that some concerns are already partialy addressed with restriction in latter version of the SIP, or are in the process of being limited (like authorizing only one file with build directives, restricting to one line of directive, having the directive parser being much stricter, etc)
I also know that there is some fundamental divergence about what people feel/wants in a program, and that my aversion to unprincipled shortcuts is not wildely share, see for ex: https://twitter.com/lukasz_bialy/status/1659155187292930051
Hope it helps anyway, at least a bit in some ways.
1 Like
Thanks for laying down your concerns!
- not principled: comments are not meant to hold semantic instruction, so nothing is built with that on mind which leads to fractal of creative problems because of surprise effect or ecosystem level support of the thing,
I wonder if as a more principled approach you would suggest having a build tool with a full configuration? If so then, this is exactly what we wanted to avoid, to be more friendly to beginners and for smaller scale use cases. It’s really hard to always tell people that they need to use a build tool while things like python really don’t care.
The comments in current form are quite easy to handle from the tooling perspective actually and we’ve been already able to do some work related to them such as automatic suggestions of libraries.
Anything else we tried was actually harder to support and had a much bigger impact on the amount of work needed to be done. And the scope is really having a couple of files and being able to work on it quickly.
For larger projects I agree this might become a nightmare sometimes, but on the other hand I’ve seen sbt builds which were a nightmare.
Anyway, we’ll keep in mind your concerns and try to minimize any negative fallout that might relate to them. Thanks!
For discussing the getting-started / Coursier angle, Julien has started a separate discussion at Streamlining the Scala installation procedure - Question - Scala Users
(sorry, I answered by email and I thought that at some point it used to put answer here, too, but I may have dream of that)
I wonder if as a more principled approach you would suggest having a build tool with a full configuration? If so then, this is exactly what we wanted to avoid, to be more friendly to beginners and for smaller scale use cases. It’s really hard to always tell people that they need to use a build tool while things like python really don’t care.
A more principled approach would be at least in a semantically meaningful token in place of a comment, like at least an annotation.
But yes, a separate config file would be even better in the separation of concern realm - but really, I’m not even asking for that, just to avoid “this comment is changing my build and importing illicit dependencies and executing foreign code”
Here,
#> ... and # >... change what is executed on the machine, which has a massive suprising effect for people not aware of the possibility.
Anything else we tried was actually harder to support and had a much bigger impact on the amount of work needed to be done. And the scope is really having a couple of files and being able to work on it quickly.
For larger projects I agree this might become a nightmare sometimes, but on the other hand I’ve seen sbt builds which were a nightmare.
If you are already able to forsee how things can go horribly wrong, please give the tools to set a very strict framework for the one willing to, like option to globally disable the feature without possibility to override it locally for CI env where there’s absolutely good reasons to forbid it, or limit the number of files authorized to have a build directive to at most one, or only for file with a “.scalacli” extension, or whatever clear and strict way to control the scope of things able to go wrong. Someone, on the morning of a CVE publication with a CISO humming behind his neck will thank you.
We need to step back for a while and think about why this is so. Python (like Shell and GCC/C) lives in Unix. There, you do not need to resolve dependencies, as this is done on the OS level. Libraries are supposed to be already available and that’s it. So, what we need is not a Nth build tool to solve this problem over and over again, but a true Java OS, maybe on top of Linux or the like (something like Android but with normal Java instead of Dalvik + Android API). I have a project along these lines called linoleum. There, one first installs any needed libraries, which are resolved transitively (by Ivy), like with apt in Linux. Then the build process is really simple : juste run javac/scalac, and you’re done.
The only area where build tools remain relevant is cross-compiling. But it is also a complicated process in GCC.
[To the moderators : maybe this message is more suited to the “Almighty standard build tool” thread. Do not hesitate to move it there if needed.]
Well, I think the real issue here is that people have an extremely different and disjunctive perception about what actually constitutes a “build tool”. (That’s actually part of what I wanted to bring into attention in the related build tool thread, but still missed to do so.)
For example regarding Linux one could rightfully declare everything being a dependency of the “build-essential” Debian package to be part of “the build tooling”. (This package would pull in a full Linux system in case you would try to install it separately! Unix / Linux is an extremely tightly coupled monolith, so more or less everything on a minimal system is in the end part of “the build tooling”. In case someone likes to argue: I’m using Linux exclusively for over 20 years and back in the day I used to build my it form scratch. So I think I my opinion here isn’t made out of thin air.)
But if you just declare a tool that can convert a directory and its sub-directories full of Scala source files and some resources into some form of publishable artifact (like a JAR file) a “build tool” should be strictly unnecessary. (Additional packaging options like the creation of an APT package, an AppImage, or maybe even an .exe file could be added on top of this basic packing of course). The whole rigmarole of mapping a directory structure which describes your project and it’s modules already perfectly fine on some “virtual project structure” through obscure configs and tooling makes things just unnecessarily complex, imho. Especially for newcomers! (But not only for them.)
When looking on both above examples I guess we need to start to distinguish different requirements and task-scopes for “build tools”. Form “just give me that JAR file” to the creation of integration tested docker containers there are very different things people want out of a “build tool”. All those requirements / expectations are valid of course.
So imho a divide and conquer approach would be the right here. A bunch of simple tools, each with just narrow focus, would be ideal. Together such tools could still form arbitrary complex build “pipelines”.
I think tools like Bazel do a lot of things right: Just concentrate on one task, but do it really well. Having dependency management, change management, packaging, all kinds of (integration) testing, creating and pulling down whole environments, etc. in one tool is not a good approach, imho. That just leads to infinite feature creep because people have really very different opinions about / requirements for “a build”…
OTOH you need than some high level tool that combines the dedicated and specialized tools to work together to model arbitrary complex “builds” (consisting of everything someone deems to be part of “a build”).
I would love it if scala-cli could be that simple no-config “base tool” that is able to assemble a source directory structure into JARs. But everything else should be kept out of it. “Orchestrating” everything around what one could possibly consider part of a build would be better done by some dedicated tooling, like task runners. (Of course scala-cli could integrate an dependency manager and even some plugins in some form of a distribution. Task runners can be build in Scala. But scala-cli shouldn’t be hard coded to anything like that, imho. Scala’s dependencies are already entangled to a point where you actually can’t build anything of it in a clean way form scratch! That’s why there are no proper Debian packages of the language or its libs since many years – which is an absolutely terrible and miserable state for a software that claims to be OpenSource…)
Getting back to the main topic of the SIP-46, I wanted to mention the current work being done and our plans.
After much back and forth trying to figure out the best way of making ScalaCLI work with all the package managers we finally realised that the best and most legitimate looking way of doing it would be to include Scala CLI in the the actual Scala releases.
This means that:
- starting with most likely Scala 3.3.2 we will exchange the default scala runner with our own jar launcher that will be able to start Scala CLI irrelevant of JAVA_HOME used. Since ScalaCLI requires JDK 17 (uses extensively feature introduced in JDK 16 namely the new socket support), the launcher will download new JDK if it was run with anything lower than JDK 17.
- we will additionally add native image launchers to the releases, which will be the recommended way of using the Scala runner as it will be much faster than the jar launcher
- the main benefit of this approach is that the package managers will only need an updated URL in most places to use the new native image launcher. Any package manager that we will not be able to update in time will use the default Scala CLI jar runner, which makes the transition as smooth as possible.
- the jars and native images will be build together with the release, so that the default Scala version used will be exactly the same one as the release one. So if anyone installs a specific version of Scala there will be no surprise there.
- there will be a separate Scala CLI artifact for most package managers for the time being, but that will be for power users and can be removed at some point (will, for sure stay on coursier indefinitely)
We can do exactly the same for Scala 2 if needed, but if anyone installs newest Scala they will be able to use Scala 2 anyway by specifying the using directive.
5 Likes
While I’m not opposed to the proposal, I would like to express the importance of a simple Scala compiler/runner that only compiles/runs what is specified on the command line without trying to magically download anything: dependencies, compilers, JDKs. That simple compiler/runner could well be scala-cli with some command line option, but something like that is needed either within scala-cli or as a separate script.
The use case is sandboxed scripted server environments where things cannot be downloaded and the runner must use predictable already installed dependencies, compilers, JDKs. My specific use case is in teaching a course, where an automatic grading server tests students’ code.
The last time I taught my course, I had updated everything in the course to Scala 3. At the last minute before the start of the course, I could not get the testing server working with Scala 3, and I had to make the difficult choice to downgrade the course back to Scala 2.13. The problem turned out to be a minor issue in the Scala runner (#17489), but I did not have time to debug it before the course started. I now have a workaround for next time, but that issue was the sole reason for one year of the course running with Scala 2.13 instead of 3.
I’m fine with scala-cli doing many things for convenience, but it needs to have a mode where it simply does what the current scala/scalac do without any magic.
3 Likes
Should we keep the old runner under something like scala-legacy, anyone objects to that? Which should be safe to use for run/compiler in the limited capacity.
That simple compiler/runner could well be scala-cli with some command line option, but something like that is needed either within scala-cli or as a separate script.
Ideally, for those cases Scala CLI should work with --server=false, though I think we currently would at least download scala compiler jars. We can add a fallback that searches for compiler jars in local directory, so that we don’t even download the Scala compiler jars.
1 Like
I would think that the nicest experience would be an --offline flag where any checks for new versions and such are omitted (just use what you’ve got), and where instead of downloading a missing critical dependency, you get an error message saying exactly what you would have tried to download if only you had not been offline.
9 Likes
Thanks for the update on what is coming! Scala CLI is indeed really valuable, esp. to beginners!
Yes, I think it is good if the old runner can be executed under some different name such as scala-legacy or scala-old-runner or similar that is self-explanatory. It is good if it is installed under a name starting with scala- so that it is discoverable using tab completion. People can then make alias to it if they wish.
I also think it is important that cs keep installing the scala-cli runner also under the name scala-cli when doing a scala install from the official install page, so that documents written assuming that cs has installed scala-cli still works for a long while. (For instance, I will send hundreds of books to print on paper soon relying on the scala-cli runner installed by cs and I guess more teachers are in my situation as the fall semester is approaching…)
Also, I think if users install Scala CLI from its home page I think they should get the runner (also) under the name scala-cli as I guess many have written instructions relating to the Scala CLI home page: Install Scala CLI | Scala CLI
1 Like
I’ve been enjoying scala-cli for convenience of trying out bug reports. Much nicer than creating an sbt project!
In addition, Scala 2 tests recognize //> using options and Scala 3 has a ticket for it.
Until now, Scala 2 tests have used // scalac: -options syntax.
I did not realize that syntax was introduced ten years ago. Another missed anniversary.
However, the other day while using scala-cli casually, I got the weird “can’t start compilation server” or whatever, “use clean”, which also didn’t work. So I’m a bit set back in my total embrace.
Scala 2 REPL used to have a server mode, fsc, that was always breaking and eventually deprecated. I think the “no compilation daemon” option -nc became the default, but maybe you can still ask for -nc:false. That was the source of many headaches.
I hope scala-cli has a mode which is no-server, no-downloads, just give me a basic REPL that reflects “what the batch compiler would do”, to the best of its ability. As usual, any friction in this basic function is problematic.
Also, Bjorn, thanks for your other post where you note that ampersand is a fish. As a Pisces, I’m embarrassed to say that escaped me until now. (Edit: I neglected to pun on && and Pisces, except one ampersand should be upside-down.)
1 Like
No worries: you have only lazily short-circuited that until now.
1 Like