Try out pipelined builds in Scala 3.5 nightly

bishabosha · April 5, 2024, 8:52am

Hello, I am happy to announce that from the latest Scala Next nightly (3.5.0-RC1-bin-20240411-4f17e25-NIGHTLY) you can now try pipelined builds in sbt.

[Edit 11 April] - updated post with a newer nightly with better diagnostics for problematic macro code under pipelining.

As shown in some benchmarks from my Scala Days 2023 talk, projects such as lichess-org/lila could reduce build times by 25% from a single setting change.

As a reminder you need sbt > 1.4, and to see any difference, you need at least two “scopes” to build - that can be test scope and main scope on a single project, or two subprojects.

you also need to set ThisBuild / usePipelining := true.

Try it out! (also you can try it in Scala 2 projects since last few years)

Thanks to everyone whose work made this possible!

Note: as far as I know only sbt supports pipelined builds, not bloop or Mill. (other tools not verified)

hepin1989 · April 5, 2024, 9:16am

Thank~!
How much time is reduced when building the scala/scala3 itself?

https://youtrack.jetbrains.com/issue/SCL-21787/investigate-support-for-sbt-usePipelining

soronpo · April 5, 2024, 10:47am

Tried it on my library.

[error] Cyclic macro dependencies in ...
[error] Compilation stopped since no further progress can be made.
[error]
[error] To fix this, place macros in one set of files and their callers in another.
[error]
[error] Compile with -Xprint-suspension for information.
[error] one error found
java.util.concurrent.CancellationException

What are the known limitations? Was there an attempt to run the OpenCB with pipelining on for all projects?

bishabosha · April 5, 2024, 11:46am

I have a test specifically where macros cause a suspension, including in both inlining and typer phases:

Cyclic macro dependencies in …

I’d like to see which library this was, because this shouldn’t cause more suspensions than already are present.

Edit: Also it seems it is quite important how you set the ThisBuild/usePipelining setting to avoid problems. For best results write it directly in the build.sbt file

bishabosha · April 5, 2024, 12:11pm

not yet, although we are investigating it

Edit: we are now running one - it seems there are projects that raise new problems not seen previously
Edit 2: a lot of those errors seem like misconfiguration of the build when running

soronpo · April 5, 2024, 6:39pm

Created a branch for you.

bishabosha · April 5, 2024, 10:49pm

Thanks for making the branch! I can reproduce

soronpo · April 5, 2024, 11:09pm

Sorry it’s a big library. I won’t be able to assist in minimizing this month. Maybe next month, if it’s relevant.

bishabosha · April 6, 2024, 1:36pm

there’s the same issue in html.scala library, which seems much smaller

bishabosha · April 9, 2024, 3:34pm

@soronpo you should be able to fix the problem by turning off pipelining in compiler_ir

 lazy val compiler_ir = (project in file("compiler/ir"))
   .settings(
     name := s"$projectName-compiler-ir",
-    settings
+    settings,
+    Compile / exportPipelining := false
   ).dependsOn(internals)

you can see an explanation here, where I implement a better error message:

github.com/scala/scala3

Detect macro dependencies that are missing from the classloader

scala:main ← dotty-staging:fix-pipelining-cyclic-macros

opened 03:05PM - 09 Apr 24 UTC

bishabosha

+227 -37

So the situation is basically that `DFiant` and `html.scala` projects do not wor…k "out of the box" with pipelining, and will need to tune their builds if they want some pipelining. However, the compiler reports an error that is not helpful to the user, so in this PR we report a better one. Previously, it was assumed that a missing class (that is valid in current run) during macro evaluation was due to the symbol being defined in the same project. If this condition is met, then compilation is suspended. This assumption breaks when the symbol comes from the classpath, but without a corresponding class file, leading a situation where the same file is always suspended, until it is the only one left, leading to the "cyclic macro dependencies" error. In this case we should assume that the class file will never become available because class path entries are supposed to be immutable. Therefore we should not suspend in this case. This commit therefore detects this situation. Instead of suspending the unit, the compiler aborts the macro expansion, reporting an error that the user will have to deal with - likely by changing the build definition/ In the end, users will see something like: ```scala [error] -- Error: /Users/jamie/workspace/DFiant/core/src/main/scala/dfhdl/core/DFVector.scala:163:25 [error] 163 | val idxVal = d"$i".asConstOf[DFUInt[Int]] [error] | ^^^^^ [error] |Macro code depends on trait DFType in package dfhdl.compiler.ir found on the classpath, but could not be loaded while evaluating the macro. [error] | This is likely because class files could not be found in the classpath entry for the symbol. [error] | [error] | A possible cause is if the origin of this symbol was built with pipelined compilation; [error] | in which case, this problem may go away by disabling pipelining for that origin. [error] | [error] | trait DFType is defined in file /Users/jamie/workspace/DFiant/compiler/ir/target/scala-3.5.0-RC1-bin-SNAPSHOT/early/dfhdl-compiler-ir_3-0.3.7+15-05c25215+20240409-1626-SNAPSHOT.jar(dfhdl/compiler/ir/DFType.tasty) [error] |--------------------------------------------------------------------------- [error] |Inline stack trace [error] |- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - [error] |This location contains code that was inlined from DFDecimal.scala:411 [error] 411 | ${ applyMacro('sc, 'args) } [error] | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ [error] --------------------------------------------------------------------------- ``` and ```scala [error] -- Error: /Users/jamie/workspace/html.scala/html/target/scala-3.5.0-RC1-bin-SNAPSHOT/src_managed/test/sbt-example-generated.scala:15:52 [error] 15 | val span: Binding.Stable[HTMLSpanElement] = html"<span>The quick ${color.bind} fox jumps over the lazy $animal</span>" [error] | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [error] |Macro code depends on missing object Definitions in package com.yang_bo.html found on the classpath, but could not be loaded while evaluating the macro. [error] | This is likely because class files could not be found in the classpath entry for the symbol. [error] | [error] | A possible cause is if the origin of this symbol was built with pipelined compilation; [error] | in which case, this problem may go away by disabling pipelining for that origin. [error] | [error] |object Definitions is defined in file /Users/jamie/workspace/html.scala/html-Definitions/target/scala-3.5.0-RC1-bin-SNAPSHOT/early/html-definitions_sjs1_3-3.0.3+0-d627afe0+20240409-1532.jar(com/yang_bo/html/Definitions.tasty) [error] |--------------------------------------------------------------------------- [error] |Inline stack trace [error] |- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - [error] |This location contains code that was inlined from package.scala:712 [error] 712 | ${ [error] | ^ [error] 713 | Macros.html('stringContext, 'args) [error] 714 | } [error] --------------------------------------------------------------------------- ``` which is more actionable. Note that sbt already automatically disables pipelining on projects that define macros, but this is not useful if the macro itself depends on upstream projects that do not define macros. This is probably a hard problem to detect automatically - so this is good compromise. We also fix `-Xprint-suspension`, which appeared to swallow a lot of diagnostic information. Also make `-Yno-suspended-units` behave better. fixes #20119

Edit 1: there should be a better error message in the next nightly (releasing 11th April)
Edit 2: now released at 3.5.0-RC1-bin-20240411-4f17e25-NIGHTLY

in general, all transitive dependencies of a macro should not be pipelined (and sbt automatically turns off pipelining on a project with macro definitions). UNLESS you never call the macro within any project (you probably are if you have adequate tests).

This unfortunately means that for a lot of projects, you can’t just “flip the switch” and hope it to work. In this case its best to have macros as close to the top of the root of the build as possible

bishabosha · April 17, 2024, 8:08am

As of April 17th the new nightly (3.5.0-RC1-bin-20240416-63810dc-NIGHTLY) will write pipeline outputs in parallel with the main compiler thread, this should give a (very) small boost to performance.

(previously we blocked the main compiler thread until these pipeline outputs were written)

kubukoz · April 17, 2024, 9:05pm

Tried out pipelining in my new app, gone from 32s (3.4.1) to 35s (3.5.0-RC1-bin-20240416-63810dc-NIGHTLY)

I did see the main module compiling before its dependency completed, though.

soronpo · April 17, 2024, 10:29pm

Did you try the nightly before/after enabling pipelining?

kubukoz · April 18, 2024, 1:22am

After. I guess I should’ve tried it before as well

guersam · April 18, 2024, 1:41am

It fails to build my work project that has a subproject cross-compiling Java and Scala.

The compiler emits [E008] Not Found Error for Java classes even under compileOrder := CompileOrder.JavaThenScala config. I guess it tries to compile Scala code before Java compilation is done.

Compiler version: 3.5.0-RC1-bin-20240416-63810dc-NIGHTLY
sbt version: 1.9.9, 1.10.0-RC2

devlaam · April 18, 2024, 9:14am

For a project with 23KLOC my compile time goes up from 46s (3.4.1) to 54s (3.5.0-RC1-bin-20240415-c47138c-NIGHTLY). (Both are second compilations after clean). This is only one experiment, more are needed to be certain this is a real effect of course. But, how can you verify that the pipelining is really active? I see no difference in the messages.

soronpo · April 18, 2024, 9:16am

Please share the results for 3.5.x with and without the pipeline flag on.

bishabosha · April 18, 2024, 9:39am

does this happen only with pipelining turned on? either way it would be good to get a link I can try on.

devlaam · April 18, 2024, 9:40am

For the versions:

scala: 3.5.0-RC1-bin-20240416-63810dc-NIGHTLY
sbt 1.9.4

I performed stb followed by five cycles of clean, compile with and without pipelining active. The first value should be discarded as it is influenced by loading caches etc. Also, i made sure i quickly repeated the commands, thereby keeping the core temperature as stable as possible.

Results:

with pipelining: 72s, 44s, 40s, 38s, 37s.
without pipelining: 67s, 44s, 41s, 40s, 39s.

So pipelining seems to speed up the process marginally. I should do more experiments to see if this survives the test of statistical significance, but its an indication alright.

bishabosha · April 18, 2024, 9:41am

pipelining benefits most when you have many levels of (edit: subproject) dependencies - do you have a description of the projects layout? Also it can’t do much if you have not enough cpu cores compared to the number of projects that compile concurrently.

Also, if a project has macros, then sbt will turn off pipelining invisibly.

I’m thinking of adding an extra flag you can turn on that will hard error any time pipelining is probably not effectively used.