SIP-46 - Scala CLI as default Scala command

Tooling
61

I am using scoop on windows.

62

Yes, most developer is using Windows including me and we can add a help guide for ChromeBook too.

63

I think the current //> using ... is not that clear as //>import dep:... or something like //>import url:... , //>import git:... and //>import mvn:...

64

I think the discussion about what installation methods we recommend to newcomers is valuable, but it’s also mostly orthogonal to Scala CLI and to the SIP. Regardless, I’d like to state my opinions.

I doubt our ability as an organization to maintain an entire package manager. Coursier is a one-person project that encompasses both a dependency manager and a package manager. The dependency management aspect is a big and undermaintained project unto itself…!

On MacOS and Linux, I see little value in having yet another package manager. I personally have stopped using cs at all on these OSs. I’d much rather use a standard, robust package manager that I’m already familiar with, such as Homebrew, to install Scala CLI and sbt. And I’d much rather manage my JDKs some standard way, too.

Now that scala-cli can do things like run scalafmt and scalafix, the package management aspect of Coursier no longer provides any value for me. (It certainly doesn’t provide value in proportion to the amount of development effort needed to maintain it, going forward.)

On Windows, sadly, the question is more complicated. cs setup has real value there. Perhaps we should retire the cross-platform package management aspect of Coursier and concentrate on providing a Windows-only tool that does the equivalent of cs setup.

6 Likes
65

For a long time I wasn’t sure what I thought. After reviewing the discussions above, I now agree that keeping the Scala CLI version numbering separate is best.

2 Likes
66

We found a number of issues in 1.0.0-RC1 so we will be releasing another one before working on the package managers.

2 Likes
67

(Or perhaps scratch “Windows-only” from that — the core of the suggestion is to handle initial setup only, rather than ongoing package management.)

68

Windows 10 and 11 have a package manager built-in (winget): Windows Package Manager - Wikipedia. There’s is probably a cultural difference, using a package manager on Windows is less common. However, we’re talking about installing CLI tools, so I wonder if a winget command would really be a hurdle, compared to a double-click installer.

I don’t know who maintains / maintained https://winget.run/pkg/Scala/Scala.2 and https://winget.run/pkg/sbt/sbt.

2 Likes
69

image
image1236×788 171 KB

WOW! thanks

70

Scala CLI 1.0.0 is released: Release v1.0.0 · VirtusLab/scala-cli · GitHub

Having addressed all the SIP-46 requirements, this version is going to become the official scala runner, replacing the old scala command.

5 Likes
71

Congrats! I can’t wait to try out the new build tool.

72

3 posts were split to a new topic: Almighty Standard Build Tool

73

This horse is likely already far far away from its barn, but I was encouraged to post some concerns regarding ScalaCLI, at least as a record or “things to have heard about for future choices”.
I believe this is the right place, but if not please let me know, I will move things.
These concerns are not informed by use of scala CLI in prod yet, but I have similar experience with other lang which chose similar approach (build tool directive in source code, like #! in bash, go directive, c preprocessor, etc) in several decades in prod env and things that go the worst possible way when given the opportunity.

So here the list:

  • not principled: comments are not meant to hold semantic instruction, so nothing is built with that on mind which leads to fractal of creative problems because of surprise effect or ecosystem level support of the thing,
  • security. It’s an open door to supply chain attacks, hard to notice if your not familiar with the “comments change build/import” idea, and the kind of attack vector that is in full boom lately (so lots of motivated malicious actor learning how to us it)
  • tooling. Comments are already hard to deal with, it’s a new level of complexity that is imposed on all tools and the whole ecosystem
  • discoverability : hard to grep, hard to find - which file do what ?
  • complexity / learning: hard to build a mental model of a build system scattered around, in comments
  • refactoring: copy/paste source from one project to another
  • consistency: developers doing things in different ways, junior vs senior

I know that some concerns are already partialy addressed with restriction in latter version of the SIP, or are in the process of being limited (like authorizing only one file with build directives, restricting to one line of directive, having the directive parser being much stricter, etc)

I also know that there is some fundamental divergence about what people feel/wants in a program, and that my aversion to unprincipled shortcuts is not wildely share, see for ex: https://twitter.com/lukasz_bialy/status/1659155187292930051

Hope it helps anyway, at least a bit in some ways.

1 Like
74

Thanks for laying down your concerns!

  • not principled: comments are not meant to hold semantic instruction, so nothing is built with that on mind which leads to fractal of creative problems because of surprise effect or ecosystem level support of the thing,

I wonder if as a more principled approach you would suggest having a build tool with a full configuration? If so then, this is exactly what we wanted to avoid, to be more friendly to beginners and for smaller scale use cases. It’s really hard to always tell people that they need to use a build tool while things like python really don’t care.

The comments in current form are quite easy to handle from the tooling perspective actually and we’ve been already able to do some work related to them such as automatic suggestions of libraries.

Anything else we tried was actually harder to support and had a much bigger impact on the amount of work needed to be done. And the scope is really having a couple of files and being able to work on it quickly.

For larger projects I agree this might become a nightmare sometimes, but on the other hand I’ve seen sbt builds which were a nightmare.

Anyway, we’ll keep in mind your concerns and try to minimize any negative fallout that might relate to them. Thanks!

75

For discussing the getting-started / Coursier angle, Julien has started a separate discussion at Streamlining the Scala installation procedure - Question - Scala Users

76

(sorry, I answered by email and I thought that at some point it used to put answer here, too, but I may have dream of that)

I wonder if as a more principled approach you would suggest having a build tool with a full configuration? If so then, this is exactly what we wanted to avoid, to be more friendly to beginners and for smaller scale use cases. It’s really hard to always tell people that they need to use a build tool while things like python really don’t care.

A more principled approach would be at least in a semantically meaningful token in place of a comment, like at least an annotation.
But yes, a separate config file would be even better in the separation of concern realm - but really, I’m not even asking for that, just to avoid “this comment is changing my build and importing illicit dependencies and executing foreign code”

Here,

#> ... and # >... change what is executed on the machine, which has a massive suprising effect for people not aware of the possibility.

Anything else we tried was actually harder to support and had a much bigger impact on the amount of work needed to be done. And the scope is really having a couple of files and being able to work on it quickly.
For larger projects I agree this might become a nightmare sometimes, but on the other hand I’ve seen sbt builds which were a nightmare.

If you are already able to forsee how things can go horribly wrong, please give the tools to set a very strict framework for the one willing to, like option to globally disable the feature without possibility to override it locally for CI env where there’s absolutely good reasons to forbid it, or limit the number of files authorized to have a build directive to at most one, or only for file with a “.scalacli” extension, or whatever clear and strict way to control the scope of things able to go wrong. Someone, on the morning of a CVE publication with a CISO humming behind his neck will thank you.

77

We need to step back for a while and think about why this is so. Python (like Shell and GCC/C) lives in Unix. There, you do not need to resolve dependencies, as this is done on the OS level. Libraries are supposed to be already available and that’s it. So, what we need is not a Nth build tool to solve this problem over and over again, but a true Java OS, maybe on top of Linux or the like (something like Android but with normal Java instead of Dalvik + Android API). I have a project along these lines called linoleum. There, one first installs any needed libraries, which are resolved transitively (by Ivy), like with apt in Linux. Then the build process is really simple : juste run javac/scalac, and you’re done.

The only area where build tools remain relevant is cross-compiling. But it is also a complicated process in GCC.

[To the moderators : maybe this message is more suited to the “Almighty standard build tool” thread. Do not hesitate to move it there if needed.]

78

Well, I think the real issue here is that people have an extremely different and disjunctive perception about what actually constitutes a “build tool”. (That’s actually part of what I wanted to bring into attention in the related build tool thread, but still missed to do so.)

For example regarding Linux one could rightfully declare everything being a dependency of the “build-essential” Debian package to be part of “the build tooling”. (This package would pull in a full Linux system in case you would try to install it separately! Unix / Linux is an extremely tightly coupled monolith, so more or less everything on a minimal system is in the end part of “the build tooling”. In case someone likes to argue: I’m using Linux exclusively for over 20 years and back in the day I used to build my it form scratch. So I think I my opinion here isn’t made out of thin air.)

But if you just declare a tool that can convert a directory and its sub-directories full of Scala source files and some resources into some form of publishable artifact (like a JAR file) a “build tool” should be strictly unnecessary. (Additional packaging options like the creation of an APT package, an AppImage, or maybe even an .exe file could be added on top of this basic packing of course). The whole rigmarole of mapping a directory structure which describes your project and it’s modules already perfectly fine on some “virtual project structure” through obscure configs and tooling makes things just unnecessarily complex, imho. Especially for newcomers! (But not only for them.)

When looking on both above examples I guess we need to start to distinguish different requirements and task-scopes for “build tools”. Form “just give me that JAR file” to the creation of integration tested docker containers there are very different things people want out of a “build tool”. All those requirements / expectations are valid of course.

So imho a divide and conquer approach would be the right here. A bunch of simple tools, each with just narrow focus, would be ideal. Together such tools could still form arbitrary complex build “pipelines”.

I think tools like Bazel do a lot of things right: Just concentrate on one task, but do it really well. Having dependency management, change management, packaging, all kinds of (integration) testing, creating and pulling down whole environments, etc. in one tool is not a good approach, imho. That just leads to infinite feature creep because people have really very different opinions about / requirements for “a build”…

OTOH you need than some high level tool that combines the dedicated and specialized tools to work together to model arbitrary complex “builds” (consisting of everything someone deems to be part of “a build”).

I would love it if scala-cli could be that simple no-config “base tool” that is able to assemble a source directory structure into JARs. But everything else should be kept out of it. “Orchestrating” everything around what one could possibly consider part of a build would be better done by some dedicated tooling, like task runners. (Of course scala-cli could integrate an dependency manager and even some plugins in some form of a distribution. Task runners can be build in Scala. But scala-cli shouldn’t be hard coded to anything like that, imho. Scala’s dependencies are already entangled to a point where you actually can’t build anything of it in a clean way form scratch! That’s why there are no proper Debian packages of the language or its libs since many years – which is an absolutely terrible and miserable state for a software that claims to be OpenSource…)

79

Getting back to the main topic of the SIP-46, I wanted to mention the current work being done and our plans.

After much back and forth trying to figure out the best way of making ScalaCLI work with all the package managers we finally realised that the best and most legitimate looking way of doing it would be to include Scala CLI in the the actual Scala releases.

This means that:

  • starting with most likely Scala 3.3.2 we will exchange the default scala runner with our own jar launcher that will be able to start Scala CLI irrelevant of JAVA_HOME used. Since ScalaCLI requires JDK 17 (uses extensively feature introduced in JDK 16 namely the new socket support), the launcher will download new JDK if it was run with anything lower than JDK 17.
  • we will additionally add native image launchers to the releases, which will be the recommended way of using the Scala runner as it will be much faster than the jar launcher
  • the main benefit of this approach is that the package managers will only need an updated URL in most places to use the new native image launcher. Any package manager that we will not be able to update in time will use the default Scala CLI jar runner, which makes the transition as smooth as possible.
  • the jars and native images will be build together with the release, so that the default Scala version used will be exactly the same one as the release one. So if anyone installs a specific version of Scala there will be no surprise there.
  • there will be a separate Scala CLI artifact for most package managers for the time being, but that will be for power users and can be removed at some point (will, for sure stay on coursier indefinitely)

We can do exactly the same for Scala 2 if needed, but if anyone installs newest Scala they will be able to use Scala 2 anyway by specifying the using directive.

5 Likes
80

While I’m not opposed to the proposal, I would like to express the importance of a simple Scala compiler/runner that only compiles/runs what is specified on the command line without trying to magically download anything: dependencies, compilers, JDKs. That simple compiler/runner could well be scala-cli with some command line option, but something like that is needed either within scala-cli or as a separate script.

The use case is sandboxed scripted server environments where things cannot be downloaded and the runner must use predictable already installed dependencies, compilers, JDKs. My specific use case is in teaching a course, where an automatic grading server tests students’ code.

The last time I taught my course, I had updated everything in the course to Scala 3. At the last minute before the start of the course, I could not get the testing server working with Scala 3, and I had to make the difficult choice to downgrade the course back to Scala 2.13. The problem turned out to be a minor issue in the Scala runner (#17489), but I did not have time to debug it before the course started. I now have a workaround for next time, but that issue was the sole reason for one year of the course running with Scala 2.13 instead of 3.

I’m fine with scala-cli doing many things for convenience, but it needs to have a mode where it simply does what the current scala/scalac do without any magic.

3 Likes