Bloop Next Steps

Bloop is a Scala build server, whose main purpose is to provide fast and reliable compilation for other build tools and editors. It can be used by Metals, Intellij, and Scala CLI or via a standalone CLI. It has proven to be a useful tool for making developers productive, however, because of the complicated build of its code base it has fallen by the wayside recently.

Bloop has a lot of great custom solutions for a number of issues such as shadowing or doing additional release tasks, however it is also a bit complicated and hard to understand. These can also cause a lot of maintenance burden and scare away potential contributors with consequences such as:

  • slow start of sbt due to additional configuration to be generated
  • complex build definition, with a lot of custom solutions that might break easily when modified
  • no automatic updates for dependencies, mostly connected to the previous point

There has recently been work done to extract the main functionality of Bloop to a separate repository for Scala CLI which is being used since the 0.1.0 version. However, the situation with the fork will also not help the Bloop maintenance, so we should aim to have a single core repository of Bloop, where it interacts with Zinc and contains the most important parts plus the ability to run tests and benchmarks.

We are currently working on adopting the mainline Zinc instead of a Bloop custom fork, which would make sure that we get all the most recent improvements and avoid any unpleasant surprises such as with Scala 2.13.7 that needed a separate fix. We will also be able to drop support for older Zinc versions in Scala 3 and get all recent improvements and fixes. We will later integrate pipelining supported by Zinc, which helps us to reduce some custom logic we previously had for it.

Some of the intended work to be done:

  • remove custom shading plugin and use something like GitHub - coursier/sbt-shading instead.
  • move integrations to separate repositories, where they can be released independently. Configuration has been stable for the last 2 years, so there is no need to keep it all there.
  • create a community suite (most likely some modification of buildpress) that would test out some of the more popular repositories.
  • make benchmarks easier to run and use.
  • rewrite the generate test projects to be created during tests
  • migrate from sbt-release-early to sbt-ci-release
  • add Scala steward

These changes should simplify working with Bloop and ease the maintenance burden on any current and future contributors. Later we would also be able to synchronize with the Scala CLI fork and make sure that we only have a single Bloop fork.

We are also considering to ship Bloop with its own JDK 17 (this is currently being tested by the Scala CLI fork), which has better support for named sockets and would allow more secure and much less flaky connection. A change like that might be seen as controversial, however: dealing with multiple Java versions, detecting the Java home and making sure that everything works on each possible version takes a toll on the amount of time used for maintenance without providing clear advantages.

We would welcome any comments or suggestions, nothing is 100% decided yet.

11 Likes

It would be great to upgrade bloop to monix 3 (wip pr which is now apparently closed: WIP: Monix 3.x.x by kpodsiad · Pull Request #1544 · scalacenter/bloop · GitHub), because I’ve been told that this is what’s preventing the metals codebase from being build with a more recent version of Scala (currently stuck on Scala 2.12: metals/build.sbt at 34d9f2803d87845a0aa46f994179075362b56a26 · scalameta/metals · GitHub). My intuition is that if metals itself could be developed using Scala 3, this would naturally lead to better Scala 3 support.

1 Like

We tried and failed utterly upgrading to Monix 3. We did also have help from the monix developers, but unfortunately we didn’t manage to get it all mergable.

The good news is that we managed to compile the launcher module, which is needed for Metals and there is nothing currently blocking from upgrading Metals to 2.13.

3 Likes

That’s good to know, but then what’s the long term plan for bloop here? Eventually it will also need to move on from Scala 2.12.

Note that it’s cross compiling, but still not published. The release process also needs to account for this if I’m not mistaken for a bloop-launcher_2.13 to be seen in the wild.

For now this isn’t decided, we might attempt to migrate to Monix 3 again at some point, but other things on the list will take priority.

Note that it’s cross compiling, but still not published. The release process also needs to account for this if I’m not mistaken for a bloop-launcher_2.13 to be seen in the wild.

It’s not yet, but we should be able to do that soon!

If the code base is really too unwieldy to reasonably upgrade, have you considered doing a (partial) rewrite? I really don’t know whether that’s the way to go but it seems like something worth considering given how much effort it takes to maintain right now.

The main issue here is cancellation, which seems to work differently in Monix 3. We might end up doing some kind of rewrite because of that, but I have not looked at it properly yet.

Since project reboots need a 3-char suffix, I suggest bloop-bns per the title.

Perhaps, “big next steps”.

1 Like