(The Scala Org aims to release Scala 3 by the end of fall 2020. We are about 15 employees (some of whom work part-time), spread in 4 organisations (+ active community members), focusing on finalising 52 essential projects in 6 months. Project leads publish the road-maps under the category “Scala 3 release projects” to share with you what is to be expected and hopefully get your advice & contributions as well. All the projects’ road-maps come after an extensive feedback gathering, rounds of discussion, and involvement of major stakeholders, we now need the community to help push this effort over the line. Your collaboration is highly appreciated, thank you in advance!)
Porting Ammonite to Dotty
At LAMP, we are coordinating efforts to port the Ammonite Scala REPL and related libraries to Dotty.
Ammonite is a tool widely used by Scala developers across all ecosystems and sub-communities, and offers an excellent interactive experience for both newbies and experienced developers.
Ammonite’s suite of dependencies and related libraries also provides a self-contained platform for writing simple Scala applications that we hope will help people immediately start doing useful work using Scala 3
Ammonite’s related libraries exercise the Scala language in a bunch of interesting ways: implicit constructors, typeclasses, macros for typeclass derivation, for tree-transformation, and for inlining and performance. This will help provide a diverse and realistic set of use cases to exercise the Scala 3 language and compiler.
Red circle – that the library contains compiler plugins
Current status
Libraries
The following libraries are ported & merged to upstream:
utest
Sourcecode
Fansi
PPrint
You can already use the above libraries in your Dotty projects – they are available on Maven for Dotty 0.26.0-RC1.
The following libraries are included in the Dotty community build but are not yet merged upstream:
Geny
oslib
Requests (not yet in CB but PR submitted to upstream)
The following libraries are being ported:
Scalatags
Cask
upickle
Team
We are a team of 5 active porters:
Anatolii Kmetiuk
Yilin Wei
Jacob Odersky
Manohar Jonnalagedda
Lars Hupel
Roadmap
Merge Fansi & PPrint into upstream in early August – done by Yilin
Merge Geny, Scalatags, uPickle, Requests, Cask into upstream by October – done by Jacob and Manohar
Get rudimentary port of Ammonite done by October – done by Anatolii
Ammonite technical debt cleanup – by December. So far it is unclear what will be needed for this step, but most probably some technical debt & non-implemented features will be generated by the previous step.
Mill – by December. This deadline is a preliminary estimate – it is not yet clear what the underwater stones there are. Done by Anatolii.
In parallel with the above effort, we are developing & maintaining tooling to manage the ecosystem we port. This is the Dotty Ecosystem project which aims to manage the moving parts of the porting effort. This is an internal effort, however, may be interesting to people who port their own libraries to Dotty.
How can I help?
Fastparse is not yet covered by anyone. If you’re interested in learning more about the Dotty’s metaprogramming framework, Fastparse may be a good project to undertake.
In future, we may need help with Ammonite and Mill. So far, it is not clear what needs to be done there exactly. However, it’s possible that after the rudimentary port is done, the effort of quality-of-life-features implementation and tech debt cleanup will be parallelisable. So, if you want to learn more about compiler internals and compiler plugins, you may be interested in helping us out.
If you are interested in helping, get in touch with me – collaboration is always welcome!
Props for including technical debt cleanup into the planning :^)
I’d really like to see fastparse ported, but I’d also like to see how the new metaprogramming capabilities of Dotty could be used to make the API less surprising and error-prone. It’s currently full of gotchas and corner cases because of its unusual macro-based approach. For instance, see this PR I made in January 2019 (not yet merged), which was only a stop-gap measure to report some obviously-wrong usages at compile time — but the fundamental problem is still there, and leads to ClassCastExceptions which are especially hard to debug since they come from generated code.
To be clear, these gotchas and corner cases have nothing to do with macros. Instead, they have to do with the fact that FastParse uses mutation and casting in order to avoid allocations and improve performance; while it usually works, when something goes wrong we get ClassCastExceptions, as is the case when casting normally goes wrong. Fastparse could be rewritten without macros, just using higher-order functions and by-name parameters, and the same gotchas and corner cases would exist.
The solution to this does exist, but not in Scala. Basically we would need Rust-style linear types and borrow checking in order to enforce that the lifetimes of people using the mutable state are well-formed and do not overlap. I too would like to see such a solution to help avoid these pitfalls, but implementing Rust-style linear types and borrow checking would be a large research topic outside the scope of this effort
Thanks for the clarification. I don’t have time to look into this again right now, but I’m sure a less error-prone API could be devised. Currently, there is no indication that parser arguments should only be passed by name; users find out the hard way by getting a very low-level error which does not hint towards the actual problem — or worse, a parser with surprising non-compositional semantics. This seems like a serious usability flaw to me.
Wouldn’t there be a way to hide the state handling behind a pure interface? Similarly to how Seq and SeqView implementations work with iterators behind the scenes, but are still pure abstractions on top of the imperative low-level Iterator abstraction. (You could say the same about the need for linearity to ensure Iterator is used safely, like in Rust, but that doesn’t prevent us from hiding it behind the simpler pure interface.)
Sure there is, but at the cost of performance. Allocating all those Parsed.Success and Parsed.Failure objects gets expensive real quick. If the JVM had inline structs which didn’t need heap allocation, perhaps the calculus would be different, but for now the idea of a “value” that all pure functional programming is built on top of is closely tied to a minimum 24-byte object allocation, and FastParse parsers are hot enough code paths that these 24-byte object allocations can easily dominate the cost of all the other logic involved in parsing structured text
I think that’s where Dotty’s new metaprogramming constructs could come into play. Staging away parser combinators is an old idea, and could well be done at compile time.
The scope is already quite impressive but have you considered migrating almond (https://almond.sh/) as well? It is based on ammonite and jupyter is a very popular tool among data scientists. Having support for dotty there from day one would be a very nice thing.
So far we do not plan such an effort for Scala 3 release. But if someone is willing to contribute their time working on it, we’d be happy to collaborate!
Nice to see such an effort being coordinated and getting underway!
Just wanted to mention that an alternative way of handling that could be to abstract away the scala version in Ammonite, so that Ammonite can use a scala version internally, but expose another one to its users.
That way, only a subset of these libraries would need to be ported (namely only the user-facing ones, so pprint and its dependencies; and fastparse would need to be able to parse dotty code, while not necessarily being itself compiled with dotty right now). sbt does that already: sbt itself uses scala 2.12, but is able to compile, run, and start a console, for projects in any scala version.
This could possibly require more significant changes in the Ammonite code base though, which would make the port less trivial than simply porting the existing dependencies and components of Ammonite over to dotty.
Mill already supports building Dotty projects even if Mill itself isn’t build with Dotty (https://github.com/lampepfl/dotty-example-project/tree/mill). I’m not sure if mill supports Dotty+Scala.js projects but that’s unrelated to this project and something you should ask to ask to the Mill maintainers instead.
Currently blocked on me for review. I’m a bit swamped by work, but will hopefully have some time opening up in the coming days to focus on this change. Sorry for the delays!
Hi. I was just looking into using Ammonite with Scala 3, and I was wondering what the status of this effort is? If you need a tester, I’m glad to volunteer for that.
On my machine that is.