Community build progress report (August 2019)

Scala Standard Library
1

Here’s what’s happened with the Scala community build in the last 6 months or so.

(For older news, see https://www.scala-lang.org/2019/01/18/community-build.html. If you’re not sure what the community build even is, see https://www.scala-lang.org/2018/01/16/community-build-growth.html.)

I’ve recently closed out a round of work on the community build, with a focus on JDK 11+ and on being ready for these upcoming releases: sbt 1.3 (now at RC3), Scala 2.12.9 (now imminent), Scala 2.13.1 (expected to follow a few weeks afterwards).

Number of green projects in the various versions of the build are currently as follows:

  • 2.11.x: 180 (JDK 8), 167 (JDK 11)
  • 2.12.x: 196 (JDK 8), 179 (JDK 11), 175 (JDK 12)
  • 2.13.x: 146 (JDK 8), 131 (JDK 11), 125 (JDK 12)

new projects: added finagle, claimant, mockito, prog-scala-examples, expression-evaluator, quicklens, kits, scala-pet-store

We aren’t adding many new projects anymore, as the build is now big enough, overall, to accomplish its main goal, that is, to guard against regressions in the Scala compiler and standard library. Nonetheless, more additions are welcome, especially if we expect they will improve coverage in specific areas.

Scala 2.13: As more and more of the Scala ecosystem supports 2.13, the 2.13 community build has grown as well. It’s now up to 146 green projects, which is about 3/4 of the 2.12 build.

JDK 11 and 12: one of the focuses of the 2.12.9 release is support for JDK 11, 12, and 13. We already had JDK 11 builds on 2.12.x and 2.13.x, but they were smaller and less actively maintained than the JDK 8 versions. We recently added JDK 12 builds on both 2.12.x and 2.13.x and we’ve stepped up maintenance. On JDK 11 and 12, about 75% of the builds are now green.

sbt 1.3: As each sbt 1.3 release candidate is released, Eugene Yokota and I have been upgrading the community build to use on as many repos as we can. This has already uncovered some sbt regressions that will be addressed for 1.3 final.

Scala 2.11: We have revived the Scala 2.11 community build, which had become moribund. For a long time, it was much smaller than the 2.12 build (for historical reasons), and it ran only on JDK 6 and 8. In the reboot, we dropped JDK 6, added JDK 11, discarded the old projects list, and backported the 2.12 projects list and config instead. The reboot runs at https://scala-ci.typesafe.com/view/scala-2.11.x/job/scala-2.11.x-integrate-community-build/ and includes 90% of the 2.12 build. The main goal of this effort is to shake out any compatibility issues on JDK 11.

What’s next?

  • We plan to add Jenkins jobs to run the build on JDK 13 and on Graal.
  • We may need a strategy for supporting more build tools. Currently the community build is sbt-only. Recently some included projects have switched to Mill. For now, we were able to keep the projects in by forking them and adding minimal sbt builds. That’s fine for now, but more change may lie ahead.
  • It’s time to start thinking more about Scala 3. I’m looking forward to discussing plans for this with the Dotty team and engineers from the Scala Center.
10 Likes
2

Maybe a shared community build between Scala 2.14 and Scala 3?

3

Just wanted to take the time to express my thanks for this extraordinary work.

To have cross check and regression test across 4 Language versions, and up to 5 different Runtime targets is a massive undertaking, and an example to the wider JVM community.

Particularly since i mostly rely on Scala 2.11 still due to Spark, it’s really encouraging to see that even though the Language is relentlessly moving forward, it still takes care of supporting versions that are now 5 years old.

Fantastic work !

4 Likes
4

quite possibly, I’ve been exploring a trial at explore the possibility of supporting Dotty · Issue #927 · scala/community-build · GitHub , haven’t had time lately to push it farther, but I’ll come back to it

another possibility, not mutually exclusive with continuing with dbuild, is to spin up something like the community build that just compiles each project independently, without doing the dependency-rewiring thing. this would catch most classes of bugs

5

This is what scala3/community-build/test/scala/dotty/communitybuild at main · scala/scala3 · GitHub already does, but it’s going to stop working as soon as dotty projects start depending on other dotty projects given that we still regularly break the ABI. So I’m very much hoping for explore the possibility of supporting Dotty · Issue #927 · scala/community-build · GitHub to come to fruition (I don’t think a proper alternative to dbuild will materialize any time soon).

6

If I were correct, breaking of ABI or type-binary-interface(TBI, e.g. Tasty) by compilers is not a regularity in Scala compiler development.

  1. It only happens relatively often in a pre-production stage of a compiler, or
  2. It may happen between big version changes of a production compiler [1].

Let’s call the first bootstrap problem, and the second migration problem.

It seems that the bootstrap problem is just temporary. Maybe a temporary solution is good enough for a temporary problem.

The migration problem is rare if it does occur. Maybe here we can isolate the problem from the problem of community build, and propose a temporary solution too?

Given the complexity of community build, I’d say we should optimize for reducing maintenance efforts. If it saves efforts on 90% of days and over 70% of the matrix, it’s already a win.

As a side effect, it will give us scalability — that is the possibility to add more projects and test more behaviors of compilers, that aligns with the primary goal of community build.

  1. But Dotty artifacts are bin-compat with Scala2, and Dotty can compile with Scala2 libs (except for macros). I assume it’s more or less the case for major Scala2 compiler versions. ↩︎

1 Like
7

But standard library changes that break binary compatibility are common.

Note that it’s also possible to add projects to the community build and just pull their dependencies from Maven Central, rather than use locally built dependencies. dbuild can do that. Doing this could help address the “add more projects” axis. (Of course, this only works in contexts where the binary dependencies are available and no ABI changes and no binary-incompatible stdlib changes have been made.)

1 Like
8

I’ve recently made some changes to improve the maintainability of the community build:

  • All projects are now built from fixed SHAs.
    • This is so results are deterministic and don’t change every day as the included projects evolve.
    • This makes maintenance easier because the number of possible causes of a failure is much reduced.
  • New script advance moves projects to new fixed SHAs.
    • This updates the SHAs either build-wide (./advance), or for selected projects only (./advance akka akka-http playframework).
    • (Unlike the old freeze script, it talks to GitHub directly rather than using the results from a previous Jenkins run, which was awkward.)
  • Each included project now has its own separate config file.
    • Example: https://github.com/scala/community-build/blob/2.12.x/proj/playframework.conf
    • This makes it easier to involve project maintainers in maintenance and troubleshooting.
    • It also makes my job easier by making it possible to use git and other command line tools (that handle multiple files well) in more maintenance tasks, rather than having to do everything in a text editor.
    • Note that the file shows both the current fixed SHA, and the branch or tag that will be used to advance the SHA.
  • New script narrow shrinks the build to just one or more projects and their dependencies.
    • Example: e.g. if you’re working on Play, you might ./narrow play-webgoat
    • …and now you don’t have to wait for 200 repos to extract. That takes an hour or more the first time, and 5 minutes subsequent times. Even 5 minutes is painful when you’re testing changes. Now, you can just extract (and then build) the 20 repos actually needed.
    • Enabling this was one of the big reasons I gave each project its own config file.
  • Build logs are now per-project, too.
10 Likes
9

A big build-wide upgrade to the Scala 2.13 community build has landed:

  • ScalaTest 3.1 is now supported
  • Major projects unfrozen
  • Formerly failing repos now pass
  • New repos added
  • Some repos dropped
  • Some notable repos still fail
  • No more “spaces”

These are likely the last major changes the Scala 2 community build will see.

Details on the PR, here: https://github.com/scala/community-build/pull/1040

4 Likes
10

more status at https://www.scala-lang.org/2020/02/20/community-build.html

6 Likes