Simplify the Getting Started experience with Coursier, roadmap

(The Scala Center team is dedicated to providing regular and transparent community updates about project plans & progress. In this forum we created a new topic category to allow quicker orientation going forward: “Scala Center Updates”. Even though all feedback is welcome, we keep the right to make executive decisions about projects we lead. Overview of all our activities can always be found at https://scala.epfl.ch/records.html)

Coursier has become a tool that every Scala developer depends on, either directly or indirectly. It packs an incredibly impressive feature set. With the addition of the cs setup functionality it’s no surprise that Coursier is becoming an increasingly popular option, not only to set up the common Scala toolchain, but also to maintain Java versions and to launch applications. Coursier has transitioned from a power-user tool to a tool that is now possibly the first Scala tool encountered by a newcomer to the language.

With that in mind, contrasting the user experience with other common tools used for similar purposes, Coursier is lagging behind. With an overwhelming help menu (see cs java -h), varying installation instructions found on the Scala website, and unclear details about what exact versions and specific launchers (sbt for example) are being installed, it brings unnecessary confusion to the Getting Started experience.

<intro from the SCP-026 Solify Getting Started with Coursier>

Problem

One of the main blockers to actively recommend coursier to newcomers is that it does not install the original distribution of some software, in particular sbt and the Scala 3 REPL launcher. Instead, it installs a custom distribution that better fits its model. This comes at a cost: it duplicates the amount of maintenance that is required when new features are developed, or, more often, it causes desynchronization between the various distributions. The latter has another consequence: the maintainers and other expert users of these software cannot help beginners to troubleshoot issues, since environmental factors differ.

Another issue is that, as software for beginners, its help menu is overwhelming. coursier is a powerful tool with many configuration options, but most of them should not be shown to beginners by default.

Roadmap

In order to address these issues, at the Scala Center we will work on the following milestones.

M1, Oct 15 - Fix the binary version of Scala 3 artifacts as seen by coursier

See coursier#2194. This is a simple issue, but which currently prevents resolution of a number of artifacts when using Scala 3 applications through the coursier launcher.

M2, Nov 5 - Install the original distribution of sbt, Scala 3 runner

Initial work to support the installation of original distributions (in general), was done by Alexandre Archambault in coursier#2188 and coursier#2207. We will build on this foundation to use the official distribution of software installed by coursier, when they have one, and when its behavior differs from the one installed by coursier. In particular, for sbt and the Scala 3 runners.

M3, Nov 12 - Document running/installing specific versions of software

This is simple documentation work, but is a prerequisite for the following milestone.

M4, Nov 26 - Merge the setup of Scala 3 apps with Scala 2 apps

Currently, coursier installs Scala 2 core applications (such as the compiler and REPL) and Scala 3 applications as distinct applications, with distinct names. This causes confusion when following tutorials or other sources of documentation for Scala 3, as the names of the commands are not the same, and by default are associated with Scala 2 only.

This milestone consists in merging Scala 2 apps and Scala 3 apps as a single set of applications, where the appropriate version is nothing more than a different version of the software. As a side effect, this means that Scala 3 will be installed by default (since its version number is higher). Users will be able to select Scala 2 by forcing the version of the software, which we will have documented in M3.

M5, Dec 10 - Improve the help menu

As previously mentioned, the help menu of coursier is overwhelming to a newcomer. We will adapt it to use a layered approach, where only frequently used options are shown by default. Specific options will request help for the advanced/rarely used options. This is similar to the way -X, -Y and -W options work in the scalac command.

In addition, we will better document what each of the top-level commands does, such as “setup” or “launch”.

M6, Dec 24 - Revisit the Getting Started pages on the official website

With all the above milestones taken care of, we will be ready to actively recommend coursier as the official way to install Scala and related software. We will therefore revisit the Getting Started and Download pages of the website to reflect that new reality.

28 Likes

In addition, all the default software that coursier is installing (that the community has control of), should refer to coursier as its preferred installation method. E.g.: sbt, ammonite

4 Likes

Rob suggested I comment here about a basic auth problem I am having with coursier.

The situation is that coursier will not handle a redirect that involves basic auth. All straight through requests work. This affects both sbt and metals/bloop. Where coursier interacts with metals/bloop is unknown to me. It is pretty easy to see the redirects in the errors attached.

What we do. We create an environment that has JAVA_OPTS with -Dhttp/https.proxyHost, proxyPort, proxyUser, proxyPassword. This is by far the best way to pass these to the JVM via the properties. I know Basic Auth is silly but it is what it is.

I tried to address the issue myself but was unable to figure out exactly how to work on coursier and I could get no support on the forums so I could make progress. I have been involved with this before which got basic auth working with no redirect. See this issue and linked issues. https://github.com/coursier/coursier/issues/930

Below is an example of the errors from sbt and Metals respectively.

Hoping this can finally be resolved. I am sure this has stopped others from using Scala in this situation. Falling back to Ivy works for sbt but not Metals.

Thanks,
Eric

sbt:sn-test> reload
[info] welcome to sbt 1.5.5 (Private Build Java 1.8.0_292)
[info] loading settings for project sn-test-build from metals.sbt,plugins.sbt ...
[info] loading project definition from /home/eric/workspace/sn-test/project
[info] Updating
[info] Resolved dependencies
[warn]
[warn] Note: Some unresolved dependencies have extra attributes. Check that these dependencies exist with the requested attributes.
[warn] org.scala-native:sbt-scala-native:0.4.0 (sbtVersion=1.0, scalaVersion=2.12)
[warn]
[warn] Note: Unresolved dependencies path:
[error] stack trace is suppressed; run last update for the full output
[error] (update) sbt.librarymanagement.ResolveException: Error downloading org.scala-native:sbt-scala-native;sbtVersion=1.0;scalaVersion=2.12:0.4.0
[error] Not found
[error] Not found
[error] not found: https://repo1.maven.org/maven2/org/scala-native/sbt-scala-native_2.12_1.0/0.4.0/sbt-scala-native-0.4.0.pom
[error] not found: /home/eric/.ivy2/local/org.scala-native/sbt-scala-native/scala_2.12/sbt_1.0/0.4.0/ivys/ivy.xml
[error] download error: Caught java.io.IOException: Unable to tunnel through proxy. Proxy returns "HTTP/1.1 407 Proxy Authentication Required" (Unable to tunnel through proxy. Proxy returns "HTTP/1.1 407 Proxy Authentication Required") while downloading https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/org.scala-native/sbt-scala-native/scala_2.12/sbt_1.0/0.4.0/ivys/ivy.xml
[error] download error: Caught java.io.IOException: Unable to tunnel through proxy. Proxy returns "HTTP/1.1 407 Proxy Authentication Required" (Unable to tunnel through proxy. Proxy returns "HTTP/1.1 407 Proxy Authentication Required") while downloading https://repo.typesafe.com/typesafe/ivy-releases/org.scala-native/sbt-scala-native/scala_2.12/sbt_1.0/0.4.0/ivys/ivy.xml

Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore?

Java home: C:\Program Files\Amazon Corretto\jdk1.8.0_302

Metals version: 0.10.7

Downloading https://repo1.maven.org/maven2/org/scalameta/metals_2.12/0.10.7/metals_2.12-0.10.7.pom

Downloaded https://repo1.maven.org/maven2/org/scalameta/metals_2.12/0.10.7/metals_2.12-0.10.7.pom

Downloading https://repo1.maven.org/maven2/org/scalameta/metals_2.12/0.10.7/metals_2.12-0.10.7.pom.sha1

Downloaded https://repo1.maven.org/maven2/org/scalameta/metals_2.12/0.10.7/metals_2.12-0.10.7.pom.sha1

Downloading https://dl.bintray.com/scalacenter/releases/org/scalameta/metals_2.12/0.10.7/metals_2.12-0.10.7.pom

Downloaded https://dl.bintray.com/scalacenter/releases/org/scalameta/metals_2.12/0.10.7/metals_2.12-0.10.7.pom

Downloading https://dl.bintray.com/scalacenter/releases/org/scalameta/metals_2.12/0.10.7/metals_2.12-0.10.7.pom.sha1

Downloaded https://dl.bintray.com/scalacenter/releases/org/scalameta/metals_2.12/0.10.7/metals_2.12-0.10.7.pom.sha1

Downloading https://oss.sonatype.org/content/repositories/public/org/scalameta/metals_2.12/0.10.7/metals_2.12-0.10.7.pom

Downloaded https://oss.sonatype.org/content/repositories/public/org/scalameta/metals_2.12/0.10.7/metals_2.12-0.10.7.pom

Downloading https://oss.sonatype.org/content/repositories/public/org/scalameta/metals_2.12/0.10.7/metals_2.12-0.10.7.pom.sha1

Downloaded https://oss.sonatype.org/content/repositories/public/org/scalameta/metals_2.12/0.10.7/metals_2.12-0.10.7.pom.sha1

Downloading https://oss.sonatype.org/content/repositories/snapshots/org/scalameta/metals_2.12/0.10.7/metals_2.12-0.10.7.pom

Downloaded https://oss.sonatype.org/content/repositories/snapshots/org/scalameta/metals_2.12/0.10.7/metals_2.12-0.10.7.pom

Downloading https://oss.sonatype.org/content/repositories/snapshots/org/scalameta/metals_2.12/0.10.7/metals_2.12-0.10.7.pom.sha1

Downloaded https://oss.sonatype.org/content/repositories/snapshots/org/scalameta/metals_2.12/0.10.7/metals_2.12-0.10.7.pom.sha1

Downloading https://repo1.maven.org/maven2/org/scalameta/metals_2.12/0.10.7/metals_2.12-0.10.7.pom

Downloaded https://repo1.maven.org/maven2/org/scalameta/metals_2.12/0.10.7/metals_2.12-0.10.7.pom

Downloading https://repo1.maven.org/maven2/org/scalameta/metals_2.12/0.10.7/metals_2.12-0.10.7.pom.sha1

Downloaded https://repo1.maven.org/maven2/org/scalameta/metals_2.12/0.10.7/metals_2.12-0.10.7.pom.sha1

Downloading https://dl.bintray.com/scalacenter/releases/org/scalameta/metals_2.12/0.10.7/metals_2.12-0.10.7.pom

Downloaded https://dl.bintray.com/scalacenter/releases/org/scalameta/metals_2.12/0.10.7/metals_2.12-0.10.7.pom

Downloading https://dl.bintray.com/scalacenter/releases/org/scalameta/metals_2.12/0.10.7/metals_2.12-0.10.7.pom.sha1

Downloaded https://dl.bintray.com/scalacenter/releases/org/scalameta/metals_2.12/0.10.7/metals_2.12-0.10.7.pom.sha1

Downloading https://oss.sonatype.org/content/repositories/public/org/scalameta/metals_2.12/0.10.7/metals_2.12-0.10.7.pom

Downloaded https://oss.sonatype.org/content/repositories/public/org/scalameta/metals_2.12/0.10.7/metals_2.12-0.10.7.pom

Downloading https://oss.sonatype.org/content/repositories/public/org/scalameta/metals_2.12/0.10.7/metals_2.12-0.10.7.pom.sha1

Downloaded https://oss.sonatype.org/content/repositories/public/org/scalameta/metals_2.12/0.10.7/metals_2.12-0.10.7.pom.sha1

Downloading https://oss.sonatype.org/content/repositories/snapshots/org/scalameta/metals_2.12/0.10.7/metals_2.12-0.10.7.pom

Downloaded https://oss.sonatype.org/content/repositories/snapshots/org/scalameta/metals_2.12/0.10.7/metals_2.12-0.10.7.pom

Downloading https://oss.sonatype.org/content/repositories/snapshots/org/scalameta/metals_2.12/0.10.7/metals_2.12-0.10.7.pom.sha1

Downloaded https://oss.sonatype.org/content/repositories/snapshots/org/scalameta/metals_2.12/0.10.7/metals_2.12-0.10.7.pom.sha1

Resolution error: Error downloading org.scalameta:metals_2.12:0.10.7

not found: C:\Users\ekr32456\.ivy2\local\org.scalameta\metals_2.12\0.10.7\ivys\ivy.xml

download error: Caught java.net.ConnectException (Connection refused: connect) while downloading https://repo1.maven.org/maven2/org/scalameta/metals_2.12/0.10.7/metals_2.12-0.10.7.pom

forbidden: https://dl.bintray.com/scalacenter/releases/org/scalameta/metals_2.12/0.10.7/metals_2.12-0.10.7.pom

download error: Caught java.net.ConnectException (Connection refused: connect) while downloading https://oss.sonatype.org/content/repositories/public/org/scalameta/metals_2.12/0.10.7/metals_2.12-0.10.7.pom

download error: Caught java.net.ConnectException (Connection refused: connect) while downloading https://oss.sonatype.org/content/repositories/snapshots/org/scalameta/metals_2.12/0.10.7/metals_2.12-0.10.7.pom

Downloading Metals failed with the following:

Process exited with code 1
1 Like

@ekrich Is there a new issue on the coursier repo with this problem of combining redirect + basic auth? Is there some info somewhere on how to reproduce that locally?

thanks for the awesome information.

Hello all. It seems I forgot to come back here and post the updates after all we did.

We completed everything mentioned in the original post. The core changes were already released at the end of November in coursier 2.1.0-M1. The latest version is coursier v2.1.0-M5. (Don’t be misled by the -Mx suffixes to coursier versioning; -Mx versions indicate that the internal API might not be stable, but the CLI is.) Since February 1st, coursier installs the original sbt by default, and considers Scala 2 and 3 to be the same application, only with different version numbers. The documentation and command-line help commands have been overhauled to be (we hope) more beginner-friendly.

After all this work on coursier itself, we updated the main Scala website to unify the getting started experience, notably on the front page and the Getting Started page linked from its big blue “Get Started” button.


In some cases, upgrading cs across these changes has led to issues. If you experience some trouble, it is worth trying to completely uninstall cs, and reinstall it from the latest installation instructions.

11 Likes

This looks good! Two remarks:

1 Like

Another thing that is likely to become more and more important is Apple M1 support: Coursier does not run on Apple M1 Silicon · Issue #2335 · coursier/coursier · GitHub. All these issues make me think it might be worth having alternative instructions for when coursier fails.

Thank you @smarter for the detailed report with all the links. I agree that these issues are important to deliver a smooth user experience to newcomers, and we need to address them. I will have a look at how we can work on them, and I will report back.

1 Like

If ammonite doesn’t have complete Scala 3 support yet then the confusion can be avoided without not installing it simply by changing the executable name. For instance, it could be called amm2 or ammonite-scala-2.

I realize this question is really old but there was a container used to get the basic auth working but the redirect still doesn’t work - my assumption is that the basic auth headers are not forwarded.

Hey everyone, just a heads up that I posted a follow-up roadmap here: Bugfixes and seamless installation process for Coursier.

2 Likes