Build Server Protocol in sbt

RichType · May 8, 2020, 1:02pm

I don’t know if this the right place to put this, but one thing I’ve wondered for a while, is if it would be possible to do an incremental build and and a build from scratch in parallel at the same time. I’m sure these days most developers have got idle cores sitting around nearly all of the time.

adpi2 · May 12, 2020, 9:38am

The compile finished notification is required for Metals to be able to properly restart the presentation compiler (which influences the correctness of completions among other things)

I have tried to reproduce the limitation you are mentioning but without success. Maybe the json-rpc response of the buildTarget/compile do restart the presentation compiler.

Do you have in mind a scenario in which the completion would not be correct?

olafurpg · May 12, 2020, 10:23am

The relevant code in Metals is here

github.com

scalameta/metals/blob/c5c3a90f2594b72c720813dfff4b3f5306d76a8a/metals/src/main/scala/scala/meta/internal/metals/ForwardingMetalsBuildClient.scala#L144




@JsonNotification("build/taskFinish")
def buildTaskFinish(params: TaskFinishParams): Unit = {
  params.getDataKind match {
    case TaskDataKind.COMPILE_REPORT =>
      for {
        report <- params.asCompileReport
        compilation <- compilations.remove(report.getTarget)
      } {
        diagnostics.onFinishCompileBuildTarget(report.getTarget)
        didCompile(report)
        val target = report.getTarget
        compilation.promise.trySuccess(report)
        val name = buildTargets.info(report.getTarget) match {
          case Some(i) => i.getDisplayName
          case None => report.getTarget.getUri
        }
        val isSuccess = report.getErrors == 0
        val icon =
          if (isSuccess) clientConfig.initialConfig.icons.check
          else clientConfig.initialConfig.icons.alert

github.com

scalameta/metals/blob/c5c3a90f2594b72c720813dfff4b3f5306d76a8a/metals/src/main/scala/scala/meta/internal/metals/Compilers.scala#L202


def didCompile(report: CompileReport): Unit = {
  if (report.getErrors > 0) {
    cache.get(report.getTarget).foreach(_.restart())
  } else {
    // Restart PC for all build targets that depend on this target since the classfiles
    // may have changed.
    for {
      target <- buildTargets.allInverseDependencies(report.getTarget)
      compiler <- cache.get(target)
    } {
      compiler.restart()
    }
  }
}


def completionItemResolve(
    item: CompletionItem,
    token: CancelToken
): Future[CompletionItem] = {
  for {
    data <- item.data

Here’s an example scenario where you may be able to reproduce an issue without task finish notifications.

Step 1. Trigger completion in B.scala, which is in a different project than A.scala (for example main and test)

// src/main/scala/a/A.scala
package a
object A {
  def foo: Int = 42
}
// src/test/scala/b/B.scala
package b
object B {
  def bar = a.A.foo // completion here shows type Int
}

Step 2, save the following diff, let the compilation succeed

// src/main/scala/a/A.scala
package a
object A {
-  def foo: Int = 42
+ def foo: String = ""
}

Step 3, trigger completion in B.scala again. It may still show Int when it should show the updated String signature. It’s important that B.scala is in a different project than A.scala because otherwise there’s a chance that the presentation compiler may already have updated the signature for a.A.foo because Metals reuses the compiler instance within the same project.

I’m not 100% sure this scenario reproduces the issue, but in general Metals needs to compile finished notification to tell the presentation compiler to load fresh symbols from the classpath. Otherwise users risk seeing stale signatures in completions, parameter hints, type at point, etc.

adpi2 · May 13, 2020, 4:03pm

Thank you Olaf for the detailed explanation! It behaves exactly the way you described.

olafurpg · May 13, 2020, 5:16pm

Do you mean that you’re able to reproduce the issue or the completions work as expected (show the new String signature)?

adpi2 · May 25, 2020, 12:50pm

Hello everyone,

You will find a detailed status of the BSP implementation in sbt in this post.

The implemented endpoints are:

build/initialize
workspace/buildTargets
buildTarget/sources
buildTarget/dependencySources
buildTarget/scalacOptions
buildTarget/compile

The compiler diagnostic notifications, as well as the compilation taskStart and taskFinish notifications are also implemented.

adpi2 · June 25, 2020, 12:14pm

EDIT: sbt 1.4.0 is now released

Partial support for the BSP protocol has been shipped in sbt milestone 1.4.0-M1. And so it is now possible for any BSP client, Metals and IntelliJ for instance, to connect to sbt server.

Both Metals and IntelliJ are used to relying on Bloop for importing an sbt project. Bloop is a build server that manages compilation, test and run of several workspaces concurrently. It supports BSP and much more.

The combination of Bloop and IDEs has proved to be reliable and efficient. However, the integration with sbt is still burdened by the injection of a plugin (sbt-bloop). For this reason we decided to implement BSP in sbt.

In this post I propose you try to import your project in Metals or IntelliJ by connecting to the sbt server directly, using BSP.

The good aspect of it is that the IDE uses the exact same compiler than you do when you run sbt compile, thereby:

you are immune to inconsistencies (false error reports)
you avoid duplicated compilation which consume resources
when you run sbt test or sbt run your code is already compiled

Metals

Metals is a BSP client which means that it relies on a BSP server to delegate compilation, run and test.

By default Metals connects to Bloop. Here we want to prevent that because we want Metals to connect to sbt server instead of Bloop.

Close VS Code and remove the .bloop folder, which contains the Bloop configuration files. You can optionally remove the project/metals.sbt file that is not needed anymore.
Open VS Code again. Metals will propose you to import the build. Deny it by clicking on Not now.
Set sbt version to 1.4.0 in the project/build.properties file.
You are good to start the sbt shell.

Once sbt is loaded you should see a log saying that sbt server started. You can also check that a .bsp/sbt.json file has been created. This file contains the information that Metals needs to connect to sbt.

In the Metals panel click the Connect to build server button and see the magic happens.

Metals sends several requests to sbt to extract the build structure out of it. You can see that in the sbt shell as well as in the Metals log (View > Output then choose Metals in the dropdown list).

Open a source file and start coding.

Each time you save a file, Metals sends a compilation request to sbt. Every compile error or warning is reported by sbt to Metals so that it can highlight them to you.

About code navigation

Code Navigation in Metals relies on the semanticdb files that are generated by the semanticdb compiler plugin. You can enable this plugin by setting semanticdbEnabled := true in your build.sbt file.

Do not forget to reload sbt and click Connect to build server again.

BSP discovery

Since the .bsp/sbt.json file exists, Metals will always try to connect to sbt. This is not very convenient because it starts an sbt process in the background that you cannot interact with. I recommend to always start sbt in a shell before opening VS Code.

EDIT: The new sbtn thin client in sbt 1.4.0 makes it very easy to interact with the sbt server in the
background.

IntelliJ IDEA

The IntelliJ Scala plugin offers the ability to import a Scala project using BSP. Like Metals, it is used to relying on Bloop and that’s why it creates the Bloop configuration file.

In the following tutorial I use IntelliJ IDEA version 2020.1.2.

Set sbt version to 1.4.0 in the project/build.properties file.
Start the sbt shell.

Once sbt is loaded you should see a log saying that sbt server started. You can also check that a .bsp/sbt.json file has been created. This file contains the information that IntelliJ needs to connect to sbt. We are ready to go.

We need to import the project from scratch, so remove the .idea folder.
Open IntelliJ in the project folder (in Linux or Mac OS you can run idea . command) and choose BSP.
Don’t bother if it creates the Bloop configuration files, it will then connect to sbt and not Bloop.

In the sbt shell you should see the requests that IntelliJ sent to extract the build structure. After that you are ready to start coding in IntelliJ.

You can improve the experience by customizing the BSP settings. Go to File > Settings > Build, Execution, Deployment > Build Tools > BSP. Uncheck export sbt projects to Bloop before import and check build automatically on file save.

On file save, IntelliJ will use sbt to compile the changes and it will report the compiler diagnostics sent back by sbt.

BSP discovery

Since the .bsp/sbt.json file exists, IntelliJ will always try to connect to sbt. This is not very convenient because it starts an sbt process in the background that you cannot interact with. I recommend to always start sbt in a shell before opening your IntelliJ project.

EDIT: The new sbtn thin client in sbt 1.4.0 makes it very easy to interact with the sbt server in the
background.

Jasper-M · June 25, 2020, 1:40pm

That’s not very convenient. If I run sbt for a project in VS Code I usually do so in a shell in VS Code.

tgodzik · June 25, 2020, 2:04pm

The exact user experience will most likely be fixed. There is still some more work to be done on that front.

adpi2 · June 25, 2020, 2:46pm

That’s not very convenient. If I run sbt for a project in VS Code I usually do so in a shell in VS Code.

I agree this is not convenient at all and we have to work on it.

One solution would be to delegate the task of lauching the sbt shell to the IDE. But I don’t think this is good solution for the long term. My expectations are for the thin client mode to become the solution. So that you can have one sbt server running in a terminal or in the background and that you can connect to it from a terminal in your IDE.

ckipp01 · June 25, 2020, 6:35pm

Really neat to see the progress on this. Great work!

eyalroth · June 26, 2020, 12:18pm

This is great to hear!

The work here can also provide the much needed reference for more build tools on how to integrate with BSP.

jasticE · June 26, 2020, 6:08pm

Thanks for this writeup! First tests will IntelliJ look good!
We’ll soon be working on a better sbt integration in the IntelliJ Scala plugin based on BSP!

kamilkloch · October 28, 2020, 10:57am

Thanks for the amazing work!
I am having one gripe with IntelliJ-BSP, though. Presentation compiler is still showing false compiler errors (the infamous red squibbles). With BSP integration I was expecting no more presentation compler errors, like in VSCode.

jasticE · October 28, 2020, 11:22am

To address this, you can try enabling the IntelliJ registry key (open from Action search) scala.highlighting.compiler.errors.in.editor. This will use compiler errors either from the built-in Scala compiler or BSP to do the editor error highlighting. Please note that this mode isn’t officially supported in any way.

smiklos · November 2, 2020, 12:05pm

I’ve tried this out using sbt 1.4.2 and IntellJ idea 2020.2.3 Community Edition.

I’m facing a couple of issues so here’s my feedback:

Using akka in a multi-subproject setup results in runtime errors when starting the server. Somehow the dependencies are mixed up and bsp/sbt resolves both old and newer version of akka-core (both being referenced transiently and the newer explicitly in build.sbt).
When introducing a change in the dependencies of sbt, there seem to be no way to force bsp projects to be reloaded and therefor I need to completely delete the bsp setup and redo the whole thing. Essentially a simple sbt reload doesn’t communicate the update to Intellj.
How does this approach differ from enabling Intellj to use the scala shell for building instead of its own compiler? To me that worked most of the times to it also does mix up some dependencies unlike the vanilla sbt setup

ckipp01 · November 10, 2020, 7:35pm

Worth noting that this previous release of Metals, v.0.9.5, has much better built-in support for sbt BSP. I’ve written a blog post to go along with our release notes here on the Metals site to give some extra details about it. We’d love any feedback people may have!

I also want to give another big shout out to the Scala Center for making this happen and for @adpi2’s quick responses to issues we came across when working on this!

V-Lamp · November 20, 2020, 2:10pm

Hello!

When loading a project in Intellij via BSP, the build.sbt and plugins.sbt are not recognised, all sbt syntax is marked red. Am I missing something here?

jasticE · November 20, 2020, 3:25pm

Not missing anything, it’s not supported yet in BSP.