Try out pipelined builds in Scala 3.5 nightly

Hello, I am happy to announce that from the latest Scala Next nightly (3.5.0-RC1-bin-20240411-4f17e25-NIGHTLY) you can now try pipelined builds in sbt.

[Edit 11 April] - updated post with a newer nightly with better diagnostics for problematic macro code under pipelining.

As shown in some benchmarks from my Scala Days 2023 talk, projects such as lichess-org/lila could reduce build times by 25% from a single setting change.

As a reminder you need sbt > 1.4, and to see any difference, you need at least two “scopes” to build - that can be test scope and main scope on a single project, or two subprojects.

you also need to set ThisBuild / usePipelining := true.

Try it out! (also you can try it in Scala 2 projects since last few years)

Thanks to everyone whose work made this possible!

Note: as far as I know only sbt supports pipelined builds, not bloop or Mill. (other tools not verified)


How much time is reduced when building the scala/scala3 itself?

Tried it on my library.

[error] Cyclic macro dependencies in ...
[error] Compilation stopped since no further progress can be made.
[error] To fix this, place macros in one set of files and their callers in another.
[error] Compile with -Xprint-suspension for information.
[error] one error found

What are the known limitations? Was there an attempt to run the OpenCB with pipelining on for all projects?

1 Like

I have a test specifically where macros cause a suspension, including in both inlining and typer phases:

Cyclic macro dependencies in …

I’d like to see which library this was, because this shouldn’t cause more suspensions than already are present.

Edit: Also it seems it is quite important how you set the ThisBuild/usePipelining setting to avoid problems. For best results write it directly in the build.sbt file

not yet, although we are investigating it

Edit: we are now running one - it seems there are projects that raise new problems not seen previously
Edit 2: a lot of those errors seem like misconfiguration of the build when running

Created a branch for you.


Thanks for making the branch! I can reproduce


Sorry it’s a big library. I won’t be able to assist in minimizing this month. Maybe next month, if it’s relevant.

there’s the same issue in html.scala library, which seems much smaller

@soronpo you should be able to fix the problem by turning off pipelining in compiler_ir

 lazy val compiler_ir = (project in file("compiler/ir"))
     name := s"$projectName-compiler-ir",
-    settings
+    settings,
+    Compile / exportPipelining := false

you can see an explanation here, where I implement a better error message:

Edit 1: there should be a better error message in the next nightly (releasing 11th April)
Edit 2: now released at 3.5.0-RC1-bin-20240411-4f17e25-NIGHTLY

in general, all transitive dependencies of a macro should not be pipelined (and sbt automatically turns off pipelining on a project with macro definitions). UNLESS you never call the macro within any project (you probably are if you have adequate tests).

This unfortunately means that for a lot of projects, you can’t just “flip the switch” and hope it to work. In this case its best to have macros as close to the top of the root of the build as possible


As of April 17th the new nightly (3.5.0-RC1-bin-20240416-63810dc-NIGHTLY) will write pipeline outputs in parallel with the main compiler thread, this should give a (very) small boost to performance.

(previously we blocked the main compiler thread until these pipeline outputs were written)


Tried out pipelining in my new app, gone from 32s (3.4.1) to 35s (3.5.0-RC1-bin-20240416-63810dc-NIGHTLY) :sweat_smile:

I did see the main module compiling before its dependency completed, though.

1 Like

Did you try the nightly before/after enabling pipelining?

After. I guess I should’ve tried it before as well :sweat_smile:

It fails to build my work project that has a subproject cross-compiling Java and Scala.

The compiler emits [E008] Not Found Error for Java classes even under compileOrder := CompileOrder.JavaThenScala config. I guess it tries to compile Scala code before Java compilation is done.

  • Compiler version: 3.5.0-RC1-bin-20240416-63810dc-NIGHTLY
  • sbt version: 1.9.9, 1.10.0-RC2

For a project with 23KLOC my compile time goes up from 46s (3.4.1) to 54s (3.5.0-RC1-bin-20240415-c47138c-NIGHTLY). (Both are second compilations after clean). This is only one experiment, more are needed to be certain this is a real effect of course. But, how can you verify that the pipelining is really active? I see no difference in the messages.

Please share the results for 3.5.x with and without the pipeline flag on.

does this happen only with pipelining turned on? either way it would be good to get a link I can try on.

For the versions:

  • scala: 3.5.0-RC1-bin-20240416-63810dc-NIGHTLY
  • sbt 1.9.4

I performed stb followed by five cycles of clean, compile with and without pipelining active. The first value should be discarded as it is influenced by loading caches etc. Also, i made sure i quickly repeated the commands, thereby keeping the core temperature as stable as possible.


  • with pipelining: 72s, 44s, 40s, 38s, 37s.
  • without pipelining: 67s, 44s, 41s, 40s, 39s.

So pipelining seems to speed up the process marginally. I should do more experiments to see if this survives the test of statistical significance, but its an indication alright.

pipelining benefits most when you have many levels of (edit: subproject) dependencies - do you have a description of the projects layout? Also it can’t do much if you have not enough cpu cores compared to the number of projects that compile concurrently.

Also, if a project has macros, then sbt will turn off pipelining invisibly.

I’m thinking of adding an extra flag you can turn on that will hard error any time pipelining is probably not effectively used.

1 Like