Plan for @main and mainargs in Scala3?

Language Design
1

Scala3’s @main methods are a stripped down version of the @main methods provided by Ammonite (docs). Ammonite’s @main methods, as well as it’s own CLI flags, and those of Mill and Sjsonnet, have since been extracted into https://github.com/com-lihaoyi/mainargs: a shared library providing enough functionality to satisfy all the use cases of these widely-used tools.

The world of CLI argument parsing is messy and does not lend itself to simplistic APIs. Here’s some of the things that the mainargs library has to support that Scala3’s @main methods do not:

  1. --help support
  2. Default values for parameters
  3. Default values in combination with varargs (String* does not work, so we needed a custom Leftover[T] type)
  4. Custom short flags e.g. -f, custom long flags e.g. --my-num, automatic selection of short v.s. long flags
  5. Flag parameters, which are passed or not but do not take a value
  6. docs for each parameter, docs for the main method as a whole
  7. Support for multiple main methods
  8. Support for constructing a case class rather than call ing a method directly
  9. Re-using argument sets between @main methods and case classes by embedding a case class in the argument list
  10. Option[T] and Seq[T] parameter support for repeated parameters
  11. Choice of positional v.s. name CLI args, or mixed (e.g. lots of applications use a positional CLI arg for the script name and named CLI args for configuration)
  12. Usage in non-entrypoint positions (e.g. inside Mill’s background daemon, or inside Ammonite’s script-launcher)

There are other use cases that mainargs still does not support, e.g. space-less short flags, POSIX-style multi-flags like -xzvf, etc. While mainargs is already useful and has a rich API, it is by no means done, and I expect it to have to continue to evolve over time.

What’s the plan for Scala3’s @main methods then? As implemented, they are sufficiently stripped down and under-specified that they as-described are unlikely to be usable for most “real” use cases. At the same time, the Scala3 implementation seems prettymuch hardcoded as to what it does. Scala3’s @main methods seem like they will be useful for intro-to-Scala level teaching use cases and not much beyond that. In particular, ~none of the use cases that mainargs currently supports that inspired the Scala3 @main method implementation can be satisfied by the Scala3 language feature.

Some possible options of what to do are:

  1. We accept that the built-in @main is a Scala-101 teaching tool and no more, similar to Scala2’s built-in scripts

  2. We try to generalize it and make it extensible so that it can serve as a platform for more advanced use cases

  3. We just vendor the entirety of mainargs into the Scala3 standard library

  4. We move @main methods into a separate standalone launcher, that can be installed separately and used specifically for learning Scala (similar to how www.handsonscala.com relies on Ammonite being installed), and can evolve more quickly to satisfy the intro-to-Scala teaching use case (e.g. it could include pretty-printing via PPrint and other things)

  5. Make a separate launcher that is just a thin wrapper around mainargs, which already implements much more functionality than the Scala3 @main methods do

  6. Once Ammonite is ported to Scala3 (currently WIP), we just make Ammonite the “intro to Scala” launcher and dispense with the standard-library @main methods entirely

While mainargs is a small library (currently ~1000 lines split over 12 files, with as much test code) it is a small library filled with domain-specific concerns and edge-cases, of the sort that I would not expect the Scala language-maintainers to have expertise in. We’ve all seen the outcome of projects like scala-xml, scala-actors, scala-parallel-collections, etc., similar domain-specific libraries that were frozen into the language and standard library, dooming them to stagnation, obsolescence, and eventual removal. Others, like scala.sys.process, have not been removed but are clearly showing their age.

Nevertheless, mainargs is small enough and the domain static enough (CLIs conventions don’t change often…) that embedding the whole thing into the language and standard library is not out of the question. Especially if we take care to define proper hooks and extension points, it seems plausible that we could make it “good enough” to cover 99% of use cases for the next 10 years, which is probably as long as most standard library code can be expected to last.

Just going all-in on Ammonite is another approach worth considering. While the Ammonite Scala3 port needs to be completed, and the codebase could use some refurbishment besides, it already covers a huge range of quality-of-life features that the Scala3 built-in functionality can never match. Ammonite can do this because it relies heavily on libraries and tools that the Scala language and standard library is prohibited from depending on. And why should a teaching tool be limited to only what’s in the Scala standard library? It’s not like we demand that the Metals or IntelliJ folk work with only what’s in the Scala standard library with no external dependencies.

I don’t have a strong preference to either, mainargs works and will continue to work for the foreseeable future, and seems straightforward to port to Scala3’s macros. But it would be good to discuss to have clarity and consensus on what the plan going forward is going to be.

20 Likes
2

The plan, after discussion in the SIP committee several months ago, is:

  1. In Scala 3.0.0, ship a minimal @main support, that is a strict subset of mainargs and that the committee could build consensus on.
  2. After 3.0.0, revisit and generalize it to support more use cases.
3

I’ve never made any complex cli apps, but I definitely see the value in at least being extensible.

But first, can we allow the following to comple?

@main def main(): Unit = println("hello")

The opaque compiler error about “shadowing” was my first experience with Scala 3, and will probably be for others. Thankfully, I am persistent, but people will be turned off. I understand this would require a ‘rethink’ of the implementation.

13 Likes
4

How do you think the implementation should be rethought? If you call the method named main from a Java runner, it has to be in class main. But if you generate that class you render the annotation invisible in the future since the class will shadow it (or at least, that might happen; it depends how your classpath is set up).

The best I could come up with would be a hard error like

-- Error: ../new/test.scala:3:12 -----------------------------------------------
3 |  @main def main(): Unit = println("hello")
  |            ^
  |        A @main method cannot be called `main`; please choose another name
1 error found

Would that be preferable?

3 Likes
5

Ideal would be if the compiler pretends that the generated class main doesn’t exist. I would think it’s doable since both classes don’t have the same fully qualified names, so the compiler can see both and is free to ignore one of them. Assuming there’s some way for the compiler to see that a class was generated by a @main annotation.

6

It’s not a question of the compiler. If you declare a method @main def foo you will get a program foo that you can call from Java, i.e. java foo. By Java’s rules there must be a class called foo with a main method so that you can do this.

Rereading your comment, the question is can we somehow hide the generated class from the Scala compiler, so that it is only visible to the runtime? That’s an interesting idea…

7

Should we hide it because it’s synthetic and outside of the language, and in general main methods’ generated classes are not accessible as classes within the language? Or should we just make it smarter in understanding that the annotation of a given method cannot possibly be that method’s generated class because that would be a chicken-and-egg situation and therefore give other symbols (like scala.main) precedence in that particular context?

2 Likes
8

The latter approach might also apply to something like

package my package
import scala.annotation._

@nowarn
class nowarn

(Better examples probably exist)

That is, in general an annotation cannot resolve to the symbol it’s annotating, or at least it would be a last resort to assume that it is, and resolving nowarn to the imported one gets higher priority than resolving it to the one in the current package scope.

On the other hand, the first approach is also more general. For example what does/should this give?

@main def go = ???

def m = new go

So in general are they visible as classes within the language?

9

My gut reaction would be that they should not be. These are runtime scaffolding, and we want to reserve the possibility to add that after typer. Which means that the class would be invisible to
code compiled in the same run as the method, since it does not exist yet when that code is typechecked. But then that code should also be hidden under separate compilation.

There are other constructions that could be treated the same. For instance Java setters and getters that are generated via an annotation.

2 Likes
10

Yeah we already do this for the bean annotations: dotty/compiler/src/dotty/tools/dotc/core/tasty/TreeUnpickler.scala at ce30a1c38042dfd2445d1555bc9e9f4f8ac277d0 · lampepfl/dotty · GitHub (they’re entered after typer so under joint compilation we don’t see them either)

11

Right. But that strategy looks like it could

I wonder whether we can use the Artifact flag as a general way to ignore it? We could still load such definitions, but skip them on findMember if the phase is at Typer.

12

Artifact translates to ACC_SYNTHETIC and javac will completely ignore ACC_SYNTHETIC definitions (they’re meant to only be needed for runtime reasons), so while that might be ok for main classes, it doesn’t work for getters/setters.

13

What’s misleading is to name a class after a method. We could introduce a parameter name to the annotation:

@main("Test") def main(): Unit = println("hello")

The default could be uppercase Main. The method’s name could be anything. The convention would be to always use main.

14

Sounds good to me! Happy to table this discussion until the Scala 3.0.0 dust has settled

2 Likes
15

The current way is to generate helper identifiers with some $ characters. For example, @main def main() = () could result something like class main$main.

16

The class needs to be named after the method, and not under any mangled name, to be usable as the argument to the java command. That is the entire point. So main$main is not an option.

4 Likes
17

How about setting a capital case convention for entry points?

@main def Main(): Unit = println("hello")

This should solve the shadowing problem and produce a Java conventions friendly class.

18

Note that the shadowing problem shouldn’t happen as of Make main methods invisible by odersky · Pull Request #11546 · lampepfl/dotty · GitHub

1 Like
19
  1. After 3.0.0, revisit and generalize it to support more use cases.

Seeing that it’s been quite some time since then, what’s the latest on this? Has this been discussed any further or is there any sort of roadmap?

20

I don’t recall it being discussed recently. I believe the current plan is to build any extension on top of macro annotations, which have an experimental implementation.