Pre SIP: scala-cli as new scala command

Jasper-M · March 1, 2023, 3:03pm

You could use the .properties format.

AMatveev · March 1, 2023, 3:03pm

To weigh in, I think it will be absolutli imposible to guess what it does if you do not known scala-cli.
For examle:

//@scala-cli deps+=foo:bar:baz
//@scala-cli deps+=foo:bar:baz

I can understand that it is an annotation and it will be processed later
It has simple syntax command scala_exspression
I can google documentation
I can even guess that it add a dependency

I have used such gramma because:

@ - It is very common symbol for annotation
+= - It is a common operator for scala expressions
deps - if somebody knows sbt he can try to guess

IMHO: it simple to write and much more simple to read.
edit:
Shorter alternative, it is less readable but still googlable:

//@cli dep foo:bar:baz

tgodzik · March 1, 2023, 3:26pm

This comes dangerously close to having a touring complete configuration language inside the directives, which we do not want for sure. That’s sbt territory and we should keep it this way.

I don’t think it’s unclear if we have a real library shown:

//> lib org.scalameta::munit:1.0.0

I don’t think anyone will need the + to understand that the library is added.

Besides the scripts in the wild should have a shebang statement which will be super easy to google.

We’ll also have a bunch of beginner documentation that will utilize the new scala runner, so if anyone looks up on how to run Scala they should quickly find the right docs. The same way people see .py file they will know to run python.

I can understand that it is an annotation an it will be processed later

I don’t think that would be clear to everyone.

But that is an interesting example, thanks!

EDIT:

Och, and even without the added scala-cli it’s still easy to google it:

Pardon my polish.

EDIT2: Ok some other are not that easy to google, but I wouldn’t think we would be able to make everything googlable. Especially that scala-cli will just be scala in the future.

AMatveev · March 1, 2023, 3:34pm

To be fair:

tgodzik · March 1, 2023, 3:40pm

Yeah, that’s true, so not ideal. But wirtting scala-cli at each line would then tie it directly with Scala CLI despite it being actually just the normal Scala runner

scalway · March 1, 2023, 3:54pm

I’m big fan of current syntax, and I really love that there is only one blessed way of defining things so there is no syntax wars about singleline vs multiline vs json vs yams… just one clean line.

//> using dep "foo:bar:baz"

Agree… so we probably should stick to version that is more verbose but will give us more flexibility in the future (aka keep some kind of using or use and quotes for now and keep directives in comments //>).

Ichoran · March 2, 2023, 12:25am

Hang on here. I think we’re getting overly caught up in the magical one-liner idea. Let’s look at an example from the documentation:

//> using:
//>   scala "2.13"
//>   options "-Xasync"
//>   target {
//>     scope "test"
//>     platform "jvm"
//>   }

This kind of blows all of the comments about the problems of multi-line directives out of the water, doesn’t it? How are you supposed to enter that “target” thing without a multi-line scope, whether in single-line comments or a multi-line comment?

If we’re allowing this, we’re not that far from YAML features.

YAML is a headache, though. Why not just use JSON? It’s not that hard to write by hand. Everyone and their pet turtle can parse it and write it.

So I think we need to take a good hard look at the scope of configuration here, and also how what we type in here is different, if it is, from what we type into a build system or what we type into the compiler.

If we can get away with simple one-word commands followed by a single-line trimmed unquoted string, then //> seems good

//> scala 3.2.1
//> option -Xlint
//> library com.github.pathikrit::better-files:3.9.2

Okay, cool, seems fine. But if we have anything that goes beyond needing to take the first token as the key and anything remaining on the line as the argument, this seems clunky (even clunkier with using over and over again, but still clunky). This reads and types way better to me than the above:

/***{
  "scala": "3.2.1",
  "options": ["-Werror", "-Xlint"],
  "libraries": [
    "com.github.pathikrit::better-files:3.9.2",
    "com.lihaoyi::requests:0.8.0"
  ],
  "target": {
    "scope": "test",
    "platforms": ["jdk17", "js"]
  }
}***/

Basically, I think it’s very important to decide what we are NOT going to do with scala-cli arguments. Then, pick a syntax that supports the most complex thing that we ARE going to do. If it’s not utterly trivial, pick an existing thing so we don’t have to learn new syntax all over again.

If we’re allowing multi-part directives and multiple options, the conception is very different than if we’re supporting only the one-simple-obvious-thing-to-do, and defer to build tools for anything more complicated.

If the build tools don’t handle the more complicated task well, then I’d argue it’s the job of the build tool to make it easier.

lihaoyi · March 2, 2023, 3:13am

//> using target ["test"|"main"]

I think a lot of it boils down to this: it’s been claimed that this is just a key-value pair syntax, but the examples contain tons of stuff that is clearly not just key value pairs. In fact, it looks exactly like a hierarchical configuration format! Just a bespoke, poorly specified, incompatible one.

We don’t need to support YAML multi-line strings if we go with a subset, just like we don’t need to support references and type tags and so on. That’s the point of selecting a (well-specified) subset after all!

These examples written in a YAML subset only allowing single-line quoted strings (for unambiguity) would be

//> scala: "2.13"
//> options: "-Xasync"
//> target:
//>   scope: "test"
//>   platform: "jvm"

//> target: ["test", "main"]

Maybe we don’t like YAML. How about TOML?

//> scala = "2.13"
//> options = "-Xasync"
//> target.scope = "test"
//> target.platform = "jvm"

//> target = ["test", "main"]

It looks almost identical, just with some small tweaks! In exchange for these small tweaks, we get broad tooling compatibility, broad familiarity, well defined (and well known!) rules for quoting and escaping etc. Someone would come from e.g. Rust/Cargo or Markdown/Jekyll and be immediately familiar with to read and write this syntax. Those are exactly the kind of newcomers to Scala that we hope ScalaCLI would reach!

Sometimes you do need a bespoke programming or configuration language, but this does not appear to be such a scenario. It’s clear from the examples that what we want here is some kind of simple lightweight hierarchical configuration format, of which there are plenty we can pick off-the-shelf or adapt. Even a “dialect” of an existing language brings most of these benefits (e.g. see the situation with Markdown) without needing full compatibility for every wart or corner case.

morgen-peschke · March 2, 2023, 3:46am

Ichoran:

/***{
  "scala": "3.2.1",
  "options": ["-Werror", "-Xlint"],
  "libraries": [
    "com.github.pathikrit::better-files:3.9.2",
    "com.lihaoyi::requests:0.8.0"
  ],
  "target": {
    "scope": "test",
    "platforms": ["jdk17", "js"]
  }
}***/

HOCON is also an option that’s familiar (because of Play) and easy to both write and read:

/***scala-cli
  scala: 3.2.1
  options: ["-Werror", "-Xlint"]
  libraries: [
    com.github.pathikrit::better-files:3.9.2
    com.lihaoyi::requests:0.8.0
  ]
  target {
    scope: test
    platforms: ["jdk17", "js"]
  }
***/

hkt · March 2, 2023, 4:06am

There are so many things in Scala that is going to be so new if not intimidating to newcomers. I am surprised we are worried about the lack of similarity / familiarity with things like frontmatter.

While we should look for inspiration elsewhere in other languages and ecosystems, it is alright to choose to do what is natural to Scala or the tool in question.

While yaml sounds attractive, it does not anything advantage over //> other than familiarity for newcomers.

Single line vs multi-line is definitely that we need to brainstorm before making a decision. My gut feel is we will settle with single line //>.

AMatveev · March 2, 2023, 8:10am

I like toml for single line exspressions.
But I think that scala-cli should be optional choice. It means our programmers may not know it.
They must know sbt, it is completly enough.
So > would be too cryptic for these peaple.
May be:

//@buildUse target.platform = "jvm"

buildUse is something like using but with much more cohesion, so it will be easier to google
@ is very common symbol for annotation
it has simple grammar: command tomlExpr
toml is the most common config with hierarchical keys.
IMHO: The following is really important

lihaoyi:

we get broad tooling compatibility, broad familiarity, well defined (and well known!) rules for quoting
and escaping etc.

devlaam · March 2, 2023, 9:40am

IMHO it is completely irrelevant what kind of format is used, //> using, yaml whatever as long as it adheres to few principles:

If i see an example of the format, can i just start using it? (i.e. without reading some lengthy docs’)
If i change something simple, will the effect be predictable?
If i want something simple, can that be written down in a simple manner?

For the long term you could add:

If the format evolves over time will old definitions continue to work?
Will for equal intention there still be one natural way to write things down?

SBT, although being a very powerful tool, fails in every aspect.

tgodzik · March 2, 2023, 9:53am

That’s badly written example. It’s either:
//> using target "test"

or

//> using target "main"

Scala CLI will not interpret |. We will fix that in the docs.

//> using:
//>   scala "2.13"
//>   options "-Xasync"
//>   target {
//>     scope "test"
//>     platform "jvm"
//>   }

This I think should be disallowed, this is another leftover of the previous discussions. If we included using as part of the language it would need optional braces support. But with the current approach this no longer needed. That’s a good catch!

tgodzik · March 2, 2023, 3:47pm

I am removing the multiline syntax here: feature: Remove multiline directives to simplify them by tgodzik · Pull Request #44 · VirtusLab/using_directives · GitHub

It’s not included in the SIP so it should not be supported.

sideeffffect · March 2, 2023, 5:36pm

There is also another point, unrelated to allowing multiline or not. I think that if I were a beginner, I’d be tripped up by the abbreviations like dep or lib. I may not be very well versed yet in the lingo used by veteran software engineers.

I think the benefits of using abbreviations like this are tiny compared to the pain it will cause to the newbies we want to welcome in our Scala community. Especially when we can invest the characters that we save up by dropping using. Btw, we also currently spell it options and not opts, or packaging and not pkging, or platform and not plat.

I’m very happy for the good shape that Scala-CLI is already in and grateful for the effort many people have put into it and hope that the discussion that is taking place here will make it even better, especially for long term future and beginners

prolativ · March 3, 2023, 12:01pm

@sideeffffect We could of course add longer and potentially more self explanatory aliases for different build settings like library or dependency but that would probably introduce a deeper split in the community - newcomers might prefer full names for clarity while people who have already written a lot of scala code and have to write even more might consider these as boilerplate.
Instead I would suggest having a uniform way to express the build configuration in source code and relying more heavily on tooling. E.g. when one starts typing //> using an IDE should provide code completions for available settings with a description of each option and a list of existing aliases. Similarly, when a user hovers over a directive they should see a tooltip with a description etc. An IDE might even make some suggestions (as actionable diagnostics) about renaming one directive into another to make the style consistent when someone uses different aliases for the same setting

AMatveev · March 3, 2023, 2:07pm

Ide is not a Silver Bullet here. I think the killer feature of scala cli is that you can easylly execute snippet of code anywhere, in:

online ide
directive of documentation
etc

IMHO: Such statements should be readable from plain text(github, stackoverflow), and primaraly for newcomers

andreaTP · March 3, 2023, 2:32pm

Since I haven’t seen it mentioned just yet, in Java land Jbang ( https://www.jbang.dev/ ) is starting to have quite some traction (and is an excellent tool too …).

The syntax seems to reflect most of the needs mentioned in this thread and there is additional IDE support being developed specifically for it.

bjornregnell · March 3, 2023, 4:04pm

Hello again Contributors!

Let me first give my love to all people working with scala-cli: it’s a FANTASTIC tool!

I would also like to thank all for providing your thoughts in this thread during this on-going experimental phase of SIP-46 - several interesting input have been raised and its good to hear all the various views: MANY THANKS!

@sjrd gave a very good summary of the history and rational here behind revisiting the using-keyword in the directives syntax. And I’ll in this post try to give a summary and thoughts on what’s next.

The next formal step for SIP-46 in the SIP process is for the committee to take into account lessons learned in the experimental phase and then decide if it can be promoted to “stable”, which will mean that scala-cli is released as the official new scala runner.

Each SIP proposal has 3 assigned reviewers with the responsibility to give feedback to the SIP proposal authors and provide recommendations to the SIP committee in its decision-making. The 3 reviewers of SIP-46, aka “assignees” are @srjd, @chrisandrews-ms and myself. The next SIP meeting is in two weeks and before that the SIP committee chair @julienrf needs to decide if SIP-46 should be up for a vote or not, based on if available decision input is mature enough.

I am very happy for all the input here, and I have drawn these conclusions:

There are arguments for keeping the one-liner magic comment key-value syntax: mainly that it is both human- and tool-friendly.
There are arguments for changing the directives syntax to some existing configuration language: mainly that an existing configuration language would ease adoption.
There are concerns that the current configuration keyword using is clashing with the Scala 3 language keyword using, while many think its not a big deal with the clash.

I don’t think arguments for changing to YAML or X or Y or Z are really that strong, as there are so many different configuration languages out there and on-boarding will only be easier for the ones that happen to know the particular language chosen. Furthermore, the cost of ripping up the whole directives syntax from its roots is pretty high, and the experimental stage of this SIP was entered with a general approval of the current design philosophy of the directives.

I do think that there are merits in the arguments for changing using to something else, as it may be confusing to have overloaded meanings. I also think that now, rather than later, is the best time to change, if we should change it. I think the keyword use is a great replacement for using as it is short and to the point. I also think use scala 3 reads very well as an imperative statement in English (better than e.g. config scala 3).

So currently I lean towards making my recommendation as reviewer to the SIP committee as follows:

Promote SIP-46 from experimental to stable, with the addition that using is changed to use, while using is kept working for a generous grace period of several months with suitable deprecation warnings.

What do you think?

Thanks again to you all for your engagement and contributions to the evolution of Scala!

//B

ekrich · March 3, 2023, 8:23pm

I agree with the single line format as well as they could be key value pairs. The TOML idea is good too. We should use non-jargon/abbreviated based keys as well. Java properties allow a list as just comma separated so maybe that could work for options or scala.options. We don’t really need a keyword in front if we go with the //> comment to start the option. I worked on the property parsing in Scala.js and Scala Native so some of that code could be mined and improved if needed.

Example:

//> scala 3
//> library com.github.pathikrit::better-files:3.9.2
//> other stuff