Pre SIP: scala-cli as new scala command

Developers can access Scala through an installed compiler/REPL or build tools, such as Mill or sbt. The compiler and REPL are pretty limited as neither can resolve dependencies nor easily switch between different versions of the language. The build tool approach is relatively heavy for beginners; most importantly, it means learning the basics of the build tool first, which makes learning Scala harder.

Scala’s pace of adoption accelerates whenever it becomes easier to learn. Scala 3 has introduced significant language changes, and we believe it is past time for tooling and the ecosystem to catch up with those.

The main factors that make learning easier are:

  • One-click installation supporting the most popular operating systems and hardware combinations. Ideally, most users should find a way to install Scala using their preferred installation method.
  • Solid defaults within the tool, so that users are not faced with too many choices too early.
  • The learning curve of the tool should be minimal. People learning Scala should focus on the language, not the tools around it (there will be time to learn a build tool like sbt later)
  • The tool should have rock-solid support in IDEs and tools such as scastie and notebooks.

Scala has multiple runtimes (JVM, JS, and Native), and having consistent support for all of them would make learning easier.

Existing tooling for Scala is often insufficient for rapid prototyping and issue reproduction. The key functionalities needed for those aspects are:

  • Fast feedback loops
  • Ability to quickly change configuration like dependencies, Scala versions or JVMs
  • Flexible ways to provide configuration, sometimes the whole configuration should come directly from a command line, while at other times, self-contained examples are better

We propose that Scala CLI replaces the current ‘scala’ command as the default method for installing and learning Scala.

Because Scala CLI includes numerous installation methods native to most operating systems (such as brew or apt-get), most users will not need to install Scala using another package manager, such as coursier or sdkman. For most functionalities, there are no additional prerequisites (like having a specific version of JVM installed) and Scala CLI manages necessary dependencies.

Scala CLI is designed to balance powerful features, like dependency management, incremental compilation or support for Scala.js / Scala Native, with ease of use. Well-known and powerful tools like sbt or Mill offer everything above and more; however, both are relatively hard to learn compared to Scala CLI.

Running a Scala file with Scala CLI is as easy as running scala-cli A.scala. Scala CLI uses native binaries, compilation server and aggressive caching to provide as fast feedback loops as possible.

Scala CLI is designed with a command-line-first approach in mind. This means that everything within Scala CLI, including dependencies, libraries or even JVMs to run a program, is configurable from the command line. This leads to super-fast feedback loops, especially when prototyping or reproducing bugs. This approach also forces a simple configuration model (key-value or key-values) without inheritance, aliasing, grouping and additional advanced features that make other tools less intuitive.

Scala CLI offers a persistent configuration in using directives (a special type of comment) within .scala files. It may sound confusing for seasoned Scala developers, but it is pretty intuitive within the scope of projects that Scala CLI is designed for. Additionally, the in-source configuration creates self-contained examples (code and configuration) that are perfect for bug reports or education scenarios.

Scala CLI is backed by a VirtusLab team and is closely developed along with core teams of the Scala ecosystem, including those working on the Scala compiler and Metals. This provides the necessary resources to complete such an ambitious project within a reasonable time. As of now, the Scala CLI project is not even a year old.

Scala CLI has been available to the public for a few months and it is being adopted at a fast pace. We are pleased to see it getting used more often to reproduce bugs. In addition, it is being used for running examples in blog posts (e.g. here or here) or as a foundation of other tools (like snunit-cli or thingy), to name some of the use cases.

Scala CLI is still under development. It needs to be improved, especially in terms of IDE integration, particularly Intellij. This is one of our main focus points. Currently, scala-cli behaves differently from the current scala command. For example, running just scala starts an empty REPL session when scala-cli will just print the help message. However, given that scala-cli has much a broader scope than the current scala command, matching the exact behaviour of the current scala command is likely to cause more harm than good in the long run.

The use of Scala CLI doesn’t stop at prototyping and learning. Scala CLI turns Scala into a proper scripting language with support for shebang, piping input and more (see here). Scala CLI can be used to build small projects and features like packaging for docker images or self-contained executables that are really useful in such a context. In the future, it will allow for the creation and publishing of micro-libraries.

Too many features?

Scala CLI contains many valuable features that may seem too much for just a scala command. To make Scala CLI into a battery-included tool and at the same time keep scala simple and accessible, we decided to implement a mechanism to limit the accessible commands and options depending on whether Scala CLI is run as scala or scala-cli (or maybe even a shorter form like scl). In this way, we can experiment with new features and, at the same time, keep the scala command accessible. While compile, run and repl are pretty obvious choices, we suggest that format, test and clean should also be included in the new scala command. Option-wise, we do not think any of the existing options for the commands above should be disabled when running the new scala command.

As Scala CLI would install multiple binaries anyway (scala and scala-cli), we may consider including coursier to provide users with an option to install other tools from the Scala ecosystem, such as sbt or scalafix.

Conclusion

We learned a lot during the development of Scala CLI. We are now prepared to discuss whether Scala CLI should become a scala command and, if so, what the scope of this new ‘scala’ command should be. This post only briefly touches on what Scala CLI is and can do, and we recommend you to check out our documentation to learn more.

We do not fear criticism, yet we highly recommend you try out Scala CLI before commenting, as we have seen many cases where people who were initially sceptical about the project changed their opinion after using it for a few days, for a use-case it was designed for :wink:

25 Likes

This is great! 2022 is the year I’m going to follow up on all my New Year’s resolutions, and I think the new scala command will be a big part of that. (“3. Use scala-cli.”) Probably I’m not the only person who thinks, “scala-clippy”, but no one said naming was easy.

It seems that “scala beginner tool” is already taken?

I hope the Swiss army knife tool becomes scala and that repl remains an unadulterated tool for exercising the compiler. For example, scastie recently showed a parse error that made someone think, “Oh, it must be a REPL thing,” when really it was a different parser tool. Similarly, someone reported that an expected warning went missing, but it was just an interaction with sbt. Every feature-laden level above scalac is an opportunity for mystification, with a possible pun on the island adventure game and the similar word in German.

I can’t wait to start using the Virtus command for my snippeting and testing. The gist support is a great idea. I will retire both my directory of snippets and previous attempts to tame it with git. I imagine it will also change how tutorials are delivered. I have always made heavy use of REPL to interact with data and exercise domain code; I hope the new tool makes that mode of work more broadly appealing.

I will follow up with more feedback after I have used it for a few days.

6 Likes

Totally, nice idea.

I fell in love with scala-cli instantly, and have been using as the default CLI - repl, watch etc. Also, able to do everything without much fuss (like running gist etc.) has been very useful to me.

scala-cli - A Breeze

5 Likes

I’m supportive, scala-cli makes many things easier and better, and can let Scala grow in some areas.

I would like us to think about current uses of command-line scala / scalac that will change or go away, and help people migrate. A few random examples:

  • launching a jar with scala
  • things like scalac -help (tried scala-cli compile -S 2.13 Hello.scala -O -help, works but needs a file argument, and exits with an error), or scalac -Vphases (tried scala-cli compile -S 2.13 Hello.scala -O -Vphases, crashes)
  • compile something to inspect a classfile, they end up in a different place
  • bug isolation, for example, scalac A.scala; scalac B.scala to test a bug in separate compilation; scalac A.scala B.java to test mixed compilation
  • should there be a scaladoc command?

When scala-cli becomes the official scala, we need to consolidate / remove other ways to install Scala:

  • brew install scala, cs install scala, cs install scala3, apt install scala (if possible…), sdkman, msi / deb / rpm packages
  • the downloadable distributions (website for 2.x, github releases for 3.x)
  • others?

I feel quite strongly that there should be a single scala command out there, not multiple that have different behaviors. Maybe it makes sense to keep the lgacy scripts (scala, scalac, scaladoc, scalap) in the downloadable distribution and in the source repo for compiler devs, but then we should rename the one currently called scala.

Should installing scala also install sbt, cs?

Finally, we should have easy to find documentation about the magic behind scala-cli, I think it’s important that users can understand how what it does.

  • what files it leaves behind (scala-cli clean needs a file argument?)
  • how it uses coursier
  • how it uses bloop, what happens with the process (https://github.com/scalameta/metals-feature-requests/issues/129)
  • how jvm selection / installation works (cs java --installed on my machine fails with Unrecognized option: --installed)
9 Likes

@lrytz
Thank you very much for your support for the Scala CLI project!

Regarding the first proposed example:

Do you want scala-cli --jar name.jar?

I created the issue https://github.com/VirtusLab/scala-cli/issues/752 on the repository.

Your contribution to the discussion, there would be very much appreciated!

@zmerr I personally don’t use scala x.jar, I was just trying to make a few examples of things that will change or disappear when we exchange scala by scala-cli. My list of examples is also not exhaustive.

It’s certainly OK for some features to go away, running a jar might be one of them. What I’m trying to say: we need to make sure migration is handled very carefully, we should discover all features and use cases of the old scripts and provide good documentation for things that change.

2 Likes

Thank you for the further elaboration! It makes sense!

A couple of years ago, in that past life, I had routine tasks that I trusted only to

scala the-thing.jar some-arg

Also, recently a user wanted classes named Con and Aux on Windows, which is benign under

-d the-thing.jar

I hope the tool remains open to less-fancy use cases.

Another evocative idea for naming is clima, which suggests both cli and ecosystem.

From a beginner perspective I think it is really important to not confuse with to many concepts. Now we have scala, cs, scala-cli, sbt, scastie, ammonite, sdkman, brew, …

I’m totally in favor of consolidating scala-cli and cs this into a single scala command that can start the repl, compile, run and install things, and be the recommended way to getting started.

The current getting-started page is quite daunting for a beginner and delegates visitors to the coursier page that seems more directed to professionals than beginners. With a single-click install of a install+run thing called scala that “just works” , the situation would be much improved for beginners.

I have heard several colleagues at my own and other universities mention difficulties in installing Scala (esp. on Windows), as an argument against using Scala in introductory programming…

4 Likes

I hear this a lot, but I’m not convinced. I give my students an sbt file (which they don’t need to understand), and they’re all set. No Scala to install and, if they use VSCode or IntelliJ, they don’t even need to install sbt! It also makes it easy to add a dependency if needed, like ScalaTest or a Json library.

(I’ll agree that my Windows-using students tend to face more problems, but they’re not specific to Scala. They have problems with Java, with git, with IntelliJ, with everything…)

I’m sorry that the experience isn’t the best for you on the getting started page. We try and provide specialised instructions for each OS, but it seems you are seeing the default (which delegates to coursier’s website) would you mind sharing your browser and OS setup to see if we can improve the situation? - also requires javascript enabled

Also on Windows the user is meant to see a link to directly download a windows GUI based installer wizard that runs cs setup.

Without disregarding the enormous efforts the Scala team has made towards making it more easily accessible, we are still lagging.
Just for reference, this is how a download page supposed to look like

And this is how it looks like:

4 Likes

The scala download page looks good, but it requires another (redundant) click to the getting started guide which is completely obscure for newcomers. When someone searches to download Scala, they want to download Scala and not try out scastie.

2 Likes

Currently it is quite a problem to get right - I think the getting started page should be where scala-lang.org/download is currently, rather than the docs.scala-lang website, however we do not have the infrastructure set-up for translations on the main website yet. Another problem to get right is where do we put the link to point advanced users how to download a specific release

1 Like

See the python download page.

See the rust download page

The Scala download page is better then the Haskell download page. Do we want Haskell adoption or Python and Rust adoption?

3 Likes

would you mind sharing your browser and OS setup

I use Firefox on Linux with javascript enabled. @bishabosha

specialised instructions for each OS

Yeah, well this is not so easy to always get right… I have seen the guessing of what the user wants go wrong before. I think the most robust solution is to have buttons for each platform but highlight the platform buttons according to an automatic guess of platform and possibly make that default button a special color and the other ones grey. Then the user is guided but you don’t hinder those who want to select another platform (such as special cases when you are on Windows but want the Linux thing beacuse of WSL or what not).

This is how scala-cli has done it and it seems to work well:

When running on Linux that button is yellow, when you run it from Windows that button i yellow and the other ones grey etc.

Many thanks for all your hard work at the Scala Center!

2 Likes

I agree that the rumor of Scala’s install difficulties is exaggerated. When I show them scala-cli they get impressed and some of them changes their mind…

I give my students an sbt file

Yes that’s an option, but still they need to install one more thing if they want several terminals with the REPL in one window separate from sbt. think the scala-cli “not another a build tool”-approach with one swiss army knife for repl, compile and run including dependency management is the smoothest for a learner.

Ideally, I think it should be called just scala and not the longish and, to a beginner somewhat cryptic, scala-cli

2 Likes

Absolutely agree @soronpo!

I actually raised a similar issue earlier. Windows & Scala 3 UX - consistency · Issue #12550 · lampepfl/dotty · GitHub

Thanks @som-snytt for referencing this discussion.

Another major hurdle to Scala adoption is that SBT plug-ins do not work out of the box with Artifactory in enterprise environments due to not following POM consistency (more detail - The maven pattern for sbt plugins is invalid based on the spec · Issue #3410 · sbt/sbt (github.com)). It’s nice in the open source/academic world that you have direct access to Maven Central, but in large scale enterprises, they cannot afford to give that level of unrestricted access due to severe security challenges (, so place everything behind a proxy which works fine for standard Maven artifacts.

Having worked in several large-scale enterprises (think >10k developers alone), I see a very clear pattern of Scala abandonment entirely because SBT cannot work there. Scala is not that fun to use in Gradle or Maven, and the standard docs are all showing how to get your magic dust with SBT. It’s not that it is demanding, it is actually painstakingly difficult to make it work consistently (3 sets of credentials and repository settings and SBT settings to set up). I massively love SBT (having written articles like Essential SBT) and it is very sad.

While I have not checked out scala-cli in detail, to make it official, it would definitely not hurt to consider 2 aspects:

  • to reuse existing Maven settings.xml (credentials/proxy/repo definitions) in fetching artifacts
  • ensuring scala-cli can itself be downloaded via Maven (so you can even fetch it with mvn dependency commands)
1 Like

the Rust download page duplicates some content from their getting started page, which is very similar to our own (minus style)

Getting started - Rust Programming Language

do you think its also a problem that the Rust getting started page recommends their online playground first?

That’s a different thing than what happens on scala-lang.org. When you search scala download in search engine it takes you to the Scala download which directs to the getting started page, which is not a download page. The rust download page is a download page.

If a user searches for “getting started with Scala”, then fine, give them other options than downloading. But if they search for download scala, give them the option to download Scala. And in any case, even the getting-started of rust is much clearer than the Scala one.

2 Likes