Stable Presentation Compiler API

Scala Center Updates
1

We started an initiative focusing on creation of the new API of the Presentation Compiler to be used by Scala tooling. The main idea behind that project is to create a public, stable API that would stay binary compatible.

Problem

Currently, Metals, scalafix and other tools, rely on the interactive package implemented inside the compiler. It provides a base implementation for the various IDE capabilities such as code completions, hovers, signature help or even go to definition. Secondly, it provides an API to manually implement new logic on abstract syntax trees such as Interactive.pathTo which finds the enclosing tree for the given position.

Then, when the new version of the Scala compiler is published, we need to make a new release in those tools that hopefully don’t require any changes or, on the other hand, need special handling to make them work after changes in the compiler. As compiler developers are working on the new version of the compiler on every release (even the release candidates), we must either republish tools with the newest version, or make a workaround such as fallback to the previous version (which could break, and not work at all due to binary incompatible changes). This leads to a situation, where tools such as Metals support only a subset of Scala Versions.

As you can notice, it also introduces the next problem: some bugs, won’t be detected in the dotty repository until they are tested in dependent repositories, as interactive test coverage of IDE capabilities is not as vast as for e.g. Metals.

Solution

We intend to solve those issues by moving the Stable Presentation Compiler API to the dotty repository. Its definition is available in metals repository: metals/mtags-interfaces/src/main/java/scala/meta/pc at main · scalameta/metals · GitHub

Moving the implementation of this interface will force every fix to happen in the compiler while making it easy to always support any Scala 3 version, by releasing it along the compiler. It will solve the issue with only a subset of supported versions. Its implementation will be based on mtags implementation, as it already handles most of the use cases.

The stability will also be improved, as instead of relying on tests defined in DottyLanguageServer we will have a new test suite ported from Metals to dotty. With this change, we will be able to detect errors and bugs before they are merged. Previously, tools such as Metals relied on nightly releases of the compiler, to run tests suite manually in their repository which sometimes led to failures, which couldn’t be resolved for some time and every CI action was failing because of that change.

Roadmap

  • (done) Set up an initial meeting to clarify the goals of the projects and the role of each team member
    • (done) Onboard the Metals team (they will maintain the implementation)
    • (done) Agree on the proposed architecture and next steps.
  • (done) Add MIMa for Presentation Compiler Interfaces
  • (done) Separate shared Mtags into a new module mtags-shared GitHub PR
  • (done) Add architecture to test Metals with a presentation compiler published from dotty
  • (WIP) Add mtags-shared as compile from sources dependency.
  • (WIP) Move mtags implementation to the dotty repository as a new module
  • (WIP) Port mtags tests to dotty
  • Add a nightly release pipeline for presentation compiler

Questions

Q: How is this going to work with Scala versions that existed before the stable interface?
A: Older versions of Scala will not provide a stable interface for the presentation compiler. The first version with official support will be the Scala 3.3.0. Tools which are using presentation compiler already, should remain using it the same for older versions.

Q: What will happen to Metals support for the older versions of the Scala 3 compiler?
A: All new features for Scala 3 IDE will be implemented firstly inside the new stable presentation compiler implementation, and then they can also be implemented in mtags for versions before LTS. We plan on gradually dropping the support for those versions and gradually removing mtags completely from Metals in favor of compiler provided API.

6 Likes
2

Hopefully the API used by ENSIME:TNG ENSIME: The Next Generation continues to work as expected. I would appreciate if you were to take a look at the use of the API and ensure that it is not broken, or send a patch so that it works for this version of the compiler going forward.

3

We are not changing the internal API which you are probably using, such as interactive package, so you shouldn’t experience any issues. We’re only adding a new module which provides a stable, binary compatible API which will be recommended to get completions, hovers and other capabilities from. It will still rely on interactive and changing this is not planned for now.

1 Like
4

To add a bit more to what @rochala said.

There is a possibility that there will be some breakages in the future, since those are the compiler internal APIs and there are no compatibility guarantees. I’ve actually already had to do some changes related to sealed descendants in Metals, so it would probably best to migrate to the new Java API if possible.

Of course I am not 100% sure you would be able to migrate for Ensime, but would probably be cool to share the implementation this way. It only uses the compiler and it’s APIs at this point and if you feel some completions are not necessary then we could for sure turn those off via a presentation compiler config.