Sbt 2 production-ready roadmap


This thread is part of the projects covered by Sovereign Tech Fund’s investment in Scala.

For the full context, please read the announcement blog post: The Sovereign Tech Fund invests in Scala | The Scala Programming Language

On this forum, each project supported through this investment has its own dedicated thread.

This thread covers sbt 2 migration effort project and will be used to share the project overview, a roadmap with key milestones, ongoing progress updates, and opportunities to engage—so we can hear ideas from the community and encourage contributions.


As part of the Sovereign Tech Agency program, we are working on sbt a goal to bring sbt 2 release behind the finish line and make it reliable and production-ready.

The scope of work is:

  • Prepare and ship a stable sbt 2 release, fixing any blocker issues.
  • Migrate ecosystem projects builds from sbt 1 to sbt 2 to encourage the wider migration effort and stress-test the release.
  • Post-release maintenance, fixing any critical issues discovered in the process of migration.
  • Update essential sbt 1 plugins to support sbt 2

This thread is for:

  • Migration pain points
  • Reports from sbt 2 early adopters adopters
  • Plugin maintainer coordination
  • Ecosystem-level issues that deserve priority

The goal of the project is to spearhead the migration of the Scala ecosystem of libraries and sbt plugins to sbt 2 - but certainly there is so many more projects out there than this project can cover. Therefore, we encourage the ecosystem maintainers to migrate their builds and projects to sbt 2. We can use this thread for migration experiences exchange, prompt detection of critical issues and streamlining of migration process.

Stay tuned for the updates on the work progress here!

11 Likes

Not sure how up to date this page is:
sbt 2.x plugin migration · sbt/sbt Wiki · GitHub
But scala js and scala-native support missing are currently my first step blockers for trying out sbt 2. At least they were missing last time I tried, and the page above has not changed for them since.

2 Likes

I’m going to work on Scala Native sbt 2 plugin this week. I was, and I’m still waiting for mima-plugin (I can see PR was merged, but no new release), however it probably it’s not required to be published just to enable cross-compilation

11 Likes

As part of this program, I’ve started working on the Scala.js sbt2 plugin migration: https://github.com/scala-js/scala-js/pull/5314

I haven’t tested it with any serious projects yet, but I’ve at least succeeded in getting many of the scripted tests to pass on sbt2! :slight_smile:

12 Likes

sbt 2.x support was merged for Scala Native, and is currently available in the nightlies (first version is 0.5.11-20260208-6024c6e-SNAPSHOT) We’ve identified some issues (e.g. sbt/sbt#8665) in the sbt itself, these were fixed and would be available in sbt 2.0.0-RC9

7 Likes

My next stumbling point would be cross building documentation for jvm/native.
The doc says nothing yet:

sbt crossproject is not supported, and I assume superseded by the in sourced projectmatrix?
but sbt projectmatrix doc says native is not supported

I tried anyways and got an exception:
java.lang.RuntimeException: no rows were found in rdts matching ProjectRow(true, List(PlatformAxis(native,Native,native), ScalaVersionAxis(3.8.1,3))): List()

So helpful answers for further testing would be:
• Is projectmatrix now the assumed way? Should this already work, or do I need to wait. If so, what would be an example setup, specifically, for multiple modules depending on each other.

It’s clearly a missing/scattered documentation issue:

  • sbt-projectmatrix mention that it got archived because, it got “sourced into sbt 2.0”. sbt has it mentioned… as a very short ticket. It’s mentioned in a few placed I think, but not with any great announcement
  • I migrated all my projects from sbt-crossproject to sbt-projectmatrix and I use it for Scala.js/Scala Native as well, so I can confirm first hand that it work
  • however I am also using sbt-commandmatrix and I think its builder is what helps me avoid seeing your missing rows errors

There is a need for some documentation for this feature, a guide how to migrate from sbt-crossproject, common pitfalls and how to avoid them, etc.

3 Likes

Just got a native project working with sbt2:

4 Likes

Breakdown of the wes build.sbt. As it’s short but demonstrates various techniques. Perhaps worth making into a doc page?

Breakdown of the important bits in build.sbt from build.sbt · main · Corey O'Connor / wes · GitLab :

version := Versions.self

Sets the version of all the projects contained here to self from project/Versions.scala.

val EnableNative = sys.env.get("ENABLE_NATIVE").map(_ == "true").getOrElse(false)

This build only enables the native build as an option. (Faster development cycle working only with JVM.) This build uses the environment variable ENABLE_NATIVE=true to turn on all scala native builds.

def wesProject(pm0: ProjectMatrix): ProjectMatrix = {
  val pm1 =
    if (EnableNative)
      pm0.nativePlatform(
        Seq(Versions.scala),
        Seq(
          nativeConfig ~= { c =>
            c.withMode(Mode.releaseFast)
              .withLTO(LTO.thin)
              .withMultithreading(true)
          }
        )
      )
    else
      pm0

  pm1.jvmPlatform(
    Seq(Versions.scala)
  ).enablePlugins(JavaAppPackaging)
}

This sets up a standard project for this system wes. This also abstracts away enable/disable of the scala-native build. Presuming scala native is enabled this:

  1. enables the scala native build by calling nativePlatform with the desired scala version.
  2. configures nativeConfig for all native builds in this system.
  3. configures the jvm platform to use the same scala version and the JavaAppPackaging plugin.
    1. Note: This should only be applied to the JVM project but this is not currently supported.
def pkgConfig(pkg: String, arg: String) = {
  import sys.process.*
  s"pkg-config --$arg $pkg".!!.trim.split(" ").toList
}

useful to use pkg-config to provide the library&header paths to native dependencies. This helper is used later.

scalaVersion := Versions.scala
scalacOptions += "-explain"
scalacOptions += "-experimental"
scalacOptions += "-Wunused:imports"
scalacOptions += "-Yexplicit-nulls"
[...]
outputStrategy := Some(StdoutOutput)
fork := true

Sets up the scala standard for all projects. Adjust fork and output strategy for nicer support for CLI applications from run.

libraryDependencies += "org.scodec" %% "scodec-core" % "2.3.3"
libraryDependencies += "com.lihaoyi" %% "fansi" % "0.5.1"

Adds direct dependencies to this libraries to all projects. This is a style choice to have these libraryDependencies defined here. This whole build would still work if these dependencies were only added to lib.

libraryDependencies += "org.scalameta" %% "munit" % "1.2.0" % Test
testFrameworks += new TestFramework("munit.Framework")

Select a test framework that supports native.

val sdlLinkSettings = if (EnableNative) {
  Seq(
    nativeConfig ~= { c =>
      c.withCompileOptions(_ ++ pkgConfig("sdl2-compat", "cflags"))
       .withLinkingOptions(_ ++ pkgConfig("sdl2-compat", "libs"))
    }
  )
} else {
  Seq.empty
}

The SDL link settings are only relevant if native is enabled. This checks the enabled flag and either adjusts the native config or does nothing.

lazy val lib = wesProject(projectMatrix in file("lib"))
  .settings(
    Compile / mainClass := Some("wes.nes.tui.Main")
  )

The lib project the gui uses. The projectMatrix is not in wesProject so the auto derived named from the lazy val lib still applies. Plus this pattern enables chaining project constructors.

lazy val gui = wesProject(projectMatrix in file("gui"))
  .settings(
    Compile / mainClass := Some("wes.gui.Main"),
    sdlLinkSettings,
    libraryDependencies += "eu.joaocosta" %% "minart" % "0.6.5",
    libraryDependencies += "eu.joaocosta" %% "minart-image" % "0.6.5",
    executableScriptName := "wes-gui"
  )
  .dependsOn(lib)

gui depends on lib. Has it’s own dependencies plus the SDL dependencies (Which are non-empty if native is enabled).

def guiJVM = gui.jvm(true)
def libJVM = lib.jvm(true)

def guiNative = gui.native(true)
def libNative = lib.native(true)

LocalRootProject / Test / test := {
  (libJVM / Test/ test).toTask("").value
}

I want to refer to the JVM and native builds specifically later. These are def so they are not evaluated immediately. The project finders jvm(true) and native(true) pick the singular jvm and native projects defined above.

LocalRootProject is the top level project wes-root that contains lib and gui. This automatically aggregates the subprojects. However, for testing I only want to run the JVM based tests. This selects all of the JVM based tests via .toTask(““). A more thorough aggregate would use this to also run a subset of native tests - perhaps only the native integration tests.

LocalRootProject / stage := Def.taskDyn {
  val outDir = rootOutputDirectory.value.toFile / "stage"
  sbt.IO.createDirectory(outDir)
  sbt.IO.createDirectory(outDir / "bin")
  sbt.IO.createDirectory(outDir / "lib")

  val tuiExeJVM = (libJVM / stage).value / "bin" / (libJVM / executableScriptName).value
  val targetTuiExeJVM = outDir / "bin" / "wes-tui-jvm"

  val guiExeJVM = (guiJVM / stage).value / "bin" / (guiJVM / executableScriptName).value
  val targetGuiExeJVM = outDir / "bin" / "wes-gui-jvm"

  val guiLibJVM = (guiJVM / stage).value / "lib"
  val targetGuiLibJVM = outDir / "lib"

  sbt.IO.copyFile(guiExeJVM, targetGuiExeJVM)
  sbt.IO.copyDirectory(guiLibJVM, targetGuiLibJVM)
  sbt.IO.copyFile(tuiExeJVM, targetTuiExeJVM)

  Def.taskIf {
    if (EnableNative) {
      val guiExeNative = (guiNative / Compile / nativeLinkReleaseFull).value
      val targetGuiExeNative = outDir / "bin" / "wes-gui"

      val tuiExeNative = (libNative / Compile / nativeLinkReleaseFull).value
      val targetTuiExeNative = outDir / "bin" / "wes-tui"

      sbt.IO.copyFile(guiExeNative, targetGuiExeNative)
      sbt.IO.copyFile(tuiExeNative, targetTuiExeNative)
      target.value
    } else {
      target.value
    }
  }

}.value

Since this uses the sbt-native-packager sbt plugin

// in project/plugins.sbt
addSbtPlugin("com.github.sbt" % "sbt-native-packager" % "1.11.7")

there is a stage task declared. For the top level project I define this as:

  1. collect the JVM binaries (bin and lib)
  2. If native is enabled: collect the native executables from bin

Note the use of Def.taskIf to handle the EnableNative flag. If this was not applied then the build.sbt would fail to compile with an error like:

-- [E007] Type Mismatch Error: /home/coconnor/Documents/areas/wes/build.sbt:139:11                                                            
139 |                                                                                                                                         
    |    ^^^^^^^^^^^^                                                                                                                         
    |    Found:    java.io.File                                                                                                               
    |    Required: sbt.Def.Initialize[sbt.Task[A1]]                                                                                           
    |                                                                                                                                         
    |    where:    A1 is a type variable                                                                                                      
    |                                                                                                                                         
    | longer explanation available when compiling with `-explain`                                                                             

This stage task is what the flake.nix uses to produce the distribution.

1 Like

Okay, I think I got enough of my test build working to share some more insights.

For context, the repo is Scala 3 only, but contains ~15 projects half of which are libraries that cross build on native/js/jvm and the other half are case studies and apps that usually only build on one of the platforms and make use of the libraries.

I think, my main conclusion is that the migration from crossplugin to projectmatrix is very painful. Otherwise sbt2 seems fine, though I’ll wait with more thorough testing until I can also compile the JS projects.

The thing with crossplugin → projectmatrix has the following notes.

There seems to be barely any documentation on projectmatrix. The repo itself is not really documented, and the sbt2 documentation does not really talk about different platforms.
Helpful things that I miss in the documentation:
use myProject.jvm(true) or myProject.native(“3.8.2”) instead of just myProject.jvm
Platform specific settings have to be defined like this:

  .jvmPlatform(
    scalaVersions = Settings.scalaVersions,
    settings = Seq(Test / fork := true)
  )

The source layout for platform specific sources is different and looks like this as far as I can tell

src/main/
  scala
  scalajvm
  scalanative
  scalajs?
src/test/
  scala
  scalajvm
  scalanative
  scalajs?
2 Likes

We’ve recently released the sbt2-compat plugin that abstracts the API broken between sbt 1 and sbt 2 into functions available for both versions with the same signature. Currently covering the changes related to different handling of files by the two sbt versions. Migrating sbt plugins to sbt 2 with sbt2-compat plugin | The Scala Programming Language - contributions welcome.

Thank you everyone for sharing your experiences migrating the ecosystem! As we get more feedback, we can use it to improve the docs and sbt2-compat to cover pain points.

7 Likes