SIP-46 - Scala CLI as default Scala command

Tooling
9 
$ scala uninstall-completions
Problem occurred while uninstalling scala-cli completions
Updated /home/bjornr/.bashrc
scala-cli completions uninstalled successfully
$ scala install-completions
Updated /home/bjornr/.bashrc
It is recommended to reload your shell, or source /home/bjornr/.bashrc in the current session, for its changes to be taken into account.
$ scala set<TAB> #nothing happens
10 
$ tail ~/.bashrc 
# <<< scala-cli completions <<<

# >>> .scala.aux completions >>>
_.scala.aux_completions() {
  local IFS=$'\n'
  eval "$(.scala.aux complete bash-v1 "$(( $COMP_CWORD + 1 ))" "${COMP_WORDS[@]}")"
}

complete -F _.scala.aux_completions .scala.aux
# <<< .scala.aux completions <<<
11

What about doing what it says:

$ . ~/.bashrc
$ scala set<TAB>
12

It seems to work for me correctly from bash after reloading it. Could you try again @bjornregnell ?

Interestingly enough I get:

# >>> scala completions >>>
_scala_completions() {
  local IFS=$'\n'
  eval "$(scala complete bash-v1 "$(( $COMP_CWORD + 1 ))" "${COMP_WORDS[@]}")"
}

complete -F _scala_completions scala
# <<< scala completions <<<

So maybe .scala.aux is the issue?

Btw. currently the best reference of what scala-experimental can do might be found at Commands | Scala CLI since it list everything in a much more compact matter. I haven’t previously linked to that directly.

13

Edit: I posted a more elaborated comment below that overlaps with this one.

I’ve tried to use the new scala command to run an old script I have that used to work with Scala 2.13.

I noticed the following compatibility issues:

  • the shebang header is different. It now has to be the following:

    #!/usr/bin/env scala -S shebang

    I wonder if it would be possible to remove the shebang command at all (like in the Scala 2 runner)? Otherwise, there should be a good documentation explaining how to migrate from the Scala 2 runner. I wonder if this issue is related to that as well.

  • self-executable scripts without extension are not supported. You have to add either the .sc or .scala extension. When I looked at that, I noticed that the file extensions have different meanings for Scala CLI (ie, .sc files are interpreted as a “worksheet” whereas .scala files are interpreted as regular Scala program— ie, with a “main” method). It seems that the Scala 2 runner supports both use cases regardless of the file extension: if a standard main method is detected, that method is called, otherwise we fallback to the “worksheet” mode (which is similar to the .sc mode of Scala CLI). There is an open issue related to that (but there is no discussion about the differences between .sc and .scala scripts).

  • the -save option is not supported anymore. My understanding is that Scala CLI now always behaves like the Scala 2 runner with the -save option. I would suggest recognizing it but showing a warning like “Unnecessary option: -save. This option is always enabled.”

7 Likes
14

Hello,

We just had a SIP meeting where we discussed the action plan for Scala CLI to become the default scala command. Here is the outcome of the discussion, with links to the corresponding issues or discussions in the scala-cli repository.

  • Users should ultimately have only one tool on their machine. They will still be able to use the “restricted” features via the --power runner argument, so there is no need to have both scala and scala-cli. Related discussion: #351.
  • The documentation should be updated to reflect that change of perspective: instead of presenting scala as a subset of scala-cli, it would present scala --power as a way to get extra features (with no stability guarantees) over scala. This means that the website should primarily document the stable features, and then have extra sections to cover the --power features. Related issue: #1700.
  • The fact that the new runner requires a special shebang command to be used in scripts is an annoying incompatibility with the current runner. Would it be possible to fix it (ie, to remove the need to use shebang)? Related issue: #603.
  • The fact that the new runner differentiates “scripts” from “programs” based on their file extension is an incompatibility with the current runner, and it is also added complexity for the users (who have to reason about .sc vs .scala files). Would it be possible to follow the same approach as with the current Scala 2 runner, which is to fall back to the script mode if there is no “main method” defined in the program? Related discussion #1668.
  • As a by-product, solving the above issue would also solve the current impossibility of defining a script with no file extension. Related issue: #466.
  • There should be a mechanism to ensure that any “option” that can be supplied via a command-line argument can also be consistently supplied via a “using directive”. There are several possible strategies here, which are discussed in #1524.
6 Likes
15

In the discussions @tgodzik proposed the more self-explanatory --experimental instead of --power.

5 Likes
16

Coming back to the conversation after some work on the ScalaCLI side I am happy to report that most of the issues were resolved.

  • Users should ultimately have only one tool on their machine. They will still be able to use the “restricted” features via the --power runner argument, so there is no need to have both scala and scala-cli. Related discussion: #351 .

We now have the same behaviour regardless of the name of the binary. Advanced features are only available after --power or after specifying scala config power true

  • The documentation should be updated to reflect that change of perspective: instead of presenting scala as a subset of scala-cli, it would present scala --power as a way to get extra features (with no stability guarantees) over scala. This means that the website should primarily document the stable features, and then have extra sections to cover the --power features. Related issue: #1700 .

We now have a single documentation website with power commands being clearly marked and accompanied by warning sections.

  • The fact that the new runner requires a special shebang command to be used in scripts is an annoying incompatibility with the current runner. Would it be possible to fix it (ie, to remove the need to use shebang)? Related issue: #603 .

The discussion concluded with us still needing the shebang command for scripts. We would otherwise need to change the default command to shebang instead of run, which differs in its syntax from all the other commands and would thus be confusing to users. Shebang will only accept one file/directory as input and the rest will be arguments for the script. For run you can add multiple files and -- separates the program arguments, which allows us to be more flexible with sources. -- is a known pattern used for example in rust.

  • The fact that the new runner differentiates “scripts” from “programs” based on their file extension is an incompatibility with the current runner, and it is also added complexity for the users (who have to reason about .sc vs .scala files). Would it be possible to follow the same approach as with the current Scala 2 runner, which is to fall back to the script mode if there is no “main method” defined in the program? Related discussion #1668 .
  • As a by-product, solving the above issue would also solve the current impossibility of defining a script with no file extension. Related issue: #466

This would require some additional steps (probably double compilation) to determine if something can be treated as a script. We decided against it and instead only when using shebang command we will default to script, so it’s now possible to define a no extension or a custom extension script. That behaviour is also not supported by the old Scala 3 runner.

  • There should be a mechanism to ensure that any “option” that can be supplied via a command-line argument can also be consistently supplied via a “using directive”. There are several possible strategies here, which are discussed in #1524 .

Almost all the options can be both supplied by a using directive as well as in the command line. If there is any that is not available and would make sense to be included in both, we can fix it promptly. I am against having a using directive to use any cmd option since that would make it less obvious for users what is going on. If there is an option from command line that is needed we have to make sure it’s available in using directives.

Let me know if that answered your questions and worries before the next SiP.

As for the current plan for switching to Scala CLI backed scala launcher is as follows:

  1. As soon as the SIP is accepted, switch the scala app in coursier to Scala CLI the same as it’s done when installing scala-experimental and raise PR to change the formula under https://github.com/Homebrew/homebrew-core/blob/master/Formula/scala.rb

These might require changing ScalaCLI version to be compatible with Scala releases (starting with 3.3.0) as the package managers might not update it properly otherwise.

  1. Create or update other possible installation methods which are not currently well supported by Scala:

  • (Windows) SDKMAN, Chocolatey, Scoop

  • (Linux) Apt, Deb, Yum, Rpm, Alpine, Nix, SDKMAN

  • (MacOS) Nix, SDKMAN

5 Likes
17

These might require changing ScalaCLI version to be compatible with Scala releases (starting with 3.3.0) as the package managers might not update it properly otherwise.

What does compatible mean? In general, I’m concerned by the fact that both the scala compiler and scala-cli have their own release schedule. How do we decide which scala version ships with which scala-cli version? Is there a way with using directives to specify a minimum scala-cli version supported by the script?

I wonder if it wouldn’t make more sense to have scala-cli use the same version number and release schedule as scala itself. This would match how cargo works in Rust and should be less confusing.

1 Like
18

What does compatible mean? In general, I’m concerned by the fact that both the scala compiler and scala-cli have their own release schedule. How do we decide which scala version ships with which scala-cli version? Is there a way with using directives to specify a minimum scala-cli version supported by the script?

I wonder if it wouldn’t make more sense to have scala-cli use the same version number and release schedule as scala itself. This would match how cargo works in Rust and should be less confusing.

Sorry, I wasn’t very clear. By compatible I meant that we would most likely need to release together with the Scala release. So after 3.3.0 version is released we release scala-cli 3.3.0 etc. The default version would then match the Scala version.

I anyone wants to use a specific Scala version they can always set that in the script, otherwise we would just have the same version.

6 Likes
19

Great!

So if I have scala-cli from Scala 3.3.0 installed, and my script requires Scala 3.3.1 via a using directive, will scala-cli itself take care of downloading the new scala-cli from Scala 3.3.1 and use that to run my script?

20

So if I have scala-cli from Scala 3.3.0 installed, and my script requires Scala 3.3.1 via a using directive, will scala-cli itself take care of downloading the new scala-cli from Scala 3.3.1 and use that to run my script?

Since ScalaCLI works with any compiler version, it will only download the compiler jars etc and compile. New ScalaClI version will need to be updated via your package manager. Otherwise, it would be a nightmare to figure out.

21

Even though it works with any compiler version, an updated scala-cli might be necessary if my script relies on a using directive introduced in a new scala-cli version. If the scala version using directive ensured that the scala and scala-cli versions were guaranteed to be in sync, this wouldn’t be an issue.

1 Like
22

Even though it works with any compiler version, an updated scala-cli might be necessary if my script relies on a using directive introduced in a new scala-cli version. If the scala version using directive ensured that the scala and scala-cli versions were guaranteed to be in sync, this wouldn’t be an issue.

It would probably be better in some cases, but that clashes with the whole idea of package managers if we did a behind the scenes update.

We can do that though I think via parameter --scala-cli-version so that would theoretically be possible.

23

This can work currently via for example

scala --cli-version 0.2.0 version

1 Like
24

This proposal was discussed during the SIP meeting today. There was a long discussion (see below), but ultimately the Committee voted to accept moving the proposal from the experimental stage to the completed stage. Here are the points that were raised by the Committee:

  • The content of the proposal should be updated to reflect the changes that happened during the experimental phase (e.g. drop of multi-line using directives in the syntax specification, possibly express their syntax without relying on the grammar of Scala).
  • The versioning adopted by Scala CLI has complex implications (in addition to the points raised by @smarter above):
    • following the same versioning as Scala 3 may foster some confusion because the users will believe that they need to update Scala CLI to use a newer version of Scala although what they need to do really is to bump the version of Scala in their using directive (ie, they may think that Scala and Scala CLI are the same things, although they are not exactly the same).
    • on the other hand, with separate versioning, beginners may perceive some unnecessary complexity (“why do I need a tool besides the compiler itself?”). Also, if Scala CLI gets installed via a package named scala (as in cs install scala or brew install scala), it has to use a version that is higher than the current version of the existing scala packages (someone suggested using a version prefixed by the year, like 2023.1.0.0).
3 Likes
25

This proposal was discussed during the SIP meeting today. There was a long discussion (see below), but ultimately the Committee voted to accept moving the proposal from the experimental stage to the completed stage. Here are the points that were raised by the Committee:

This is great to hear! I finally have some time to put things in motion.

  • The content of the proposal should be updated to reflect the changes that happened during the experimental phase (e.g. drop of multi-line using directives in the syntax specification, possibly express their syntax without relying on the grammar of Scala).

Next ScalaCLI version will only support one line using directives, which has already been merged. I am also working on removing any leftovers of the previous approaches.

As for versioning, I don’t think we have a better choice than using the current Scala version and adjusting release cycle to resemble it. This is because the scala command will be expected to be the same version and anything else can cause confusion. This might not always work ideal, but will probably cause least amount of confusion. The default used Scala version in each Scala CLI will match that same version.

The next release of Scala CLI will be 3.3.0-RC4 to match the next RC of Scala 3 which will be most likely released. We will be able to do some last checks at that point, prepare PRs and once 3.3.0 is out we can release that and officially become stable.

Let us know if you have any doubts or questions!

26

Maybe I’m misunderstanding, but if the release versions will be tied to one another, what happens if scala-cli needs to make a patch release in-between the Scala releases?

4 Likes
27

This hopefully shouldn’t happen, but we could do 3.3.0-P1 or something similar as a workaround. Or just wait until the next release of Scala. We could make RC versions follow Scala RCs and this way we would have a ton of testing time for the final version.

28

Another option we could consider is having date based versioning such as 2023.04.01, however this has some drawbacks:

  • decoupled versioning scheme, user installs newest version and the package manager says the version is not what they could expect. This also breaks the current contract for coursier and brew.
  • it’s not semantic versioning, which might not be necessary for a CLI tool, but one could imagine users of certain jar components of the runner.

While I think the drawbacks of the suggested solutions are mostly to do with possible breakages:

  • we might have a bug inside ScalaCLI that needs fix right away (should be solved by releasing RC versions)
  • if a bug doesn’t need a quick fix, we can wait 6/8/10 weeks until the next Scala version is released

I think the new workflow with the aligned versioning would be:

  • Scala x.y.z-RC1 is released
  • Scala CLI branches x.y.z
  • Scala CLI releases RC1
  • if Scala and Scala CLI both work correctly next versions will be also released together
  • in case there are breakages in Scala CLI we could do more RCs (would be very rare)
  • if no breakages happen we just release RCs along with the compiler until the full version is out.