Thanks for writing this! I fully agree with the sentiment here, but have some different opinions on the specifics :).
Historically, externalizing our tooling hasn’t worked well for Scala (look at the state of scala-ide). Tools maintained outside of the compiler tend to have regression (because they’re not part of the compiler CI), take time to catch up to new compiler versions, need to duplicate a lot of logic from the compiler, and sometimes just stop being maintained, leaving everyone scrambling for a replacement. This is why for example the Dotty Language Server and the zinc-specific compiler phases are maintained in the Dotty repo instead of somewhere else. The same reasoning applies to the REPL, the Dotty REPL:
- Uses JLine3 to provide a nice cross-platform editing experience.
- Has tests which are part of our CI, catching issues regularly.
- Has syntax highlighting based on the tree that the compiler parser outputs, the implementation is thus both simple and robust, and when we do find syntax highlighting bugs (e.g. because the parser set the wrong position for some tree node), fixing them benefits every other part of the compiler that relies on our parser.
- Has completions based on the same APIs we use for the Language Server (http://guillaume.martres.me/ide_paper.pdf and reveal.js talk about this a bit), the same code sharing benefits we get for reusing the parser apply.
- Has a
:doccommand to display scaladoc/javadoc, again using APIs shared with the Language Sever.
And there’s a lot more planned in the same spirit:
- Now that we have top-level definitions, we can probably simplify some of the REPL logic concerned with wrapping user code into objects.
- The Dotty Language Server already has a Worksheet mode based on .sc files. With top-level defs we can get rid of a lot of logic here too, we just need the compiler to understand that you can write top-level statements in .sc files.
- Coursier is getting a pure Java API, which means the compiler can depend on it to support
import $ivy:...in the REPL and .sc files. - I’ve had a student work on integrating the REPL with Jupyter, to make this reusable this is based on the IDE of having a “REPL Server Protocol” akin to the Language Server Protocol allowing different frontends (terminal, Jupyter, worksheet) to the REPL.
I think there really is an opportunity for us to get great tooling in Scala 3, just by sharing as much code and infrastructure as possible between the tools and the compiler!