PRE-SIP: using directives

[romanowski]

Our proposal is to add Using Directives, a key-value pair-based syntax for delivering Scala program configurations. Here’s an example:

using scala "3.1.2"

@main def hello = println(“Hello from Scala 3.1.2”)

Using directives can be placed on top of the .scala file, above imports, package definition and everything else that is not a comment. This proposal does not imply that the Scala compiler would need to resolve dependencies or apply specific options; the role of using directives is to provide a configuration for build tools or IDEs. From the compiler’s point of view, using directives will be treated just like a comment with a fixed syntax.

This proposal does not cover the semantics of the using directives (e.g. their scope or how to resolve conflicts if directives come from multiple files). If such semantics should be a part of the language specification, then we suggest adopting semantics from Scala CLI.

Standardising a set of supported keys across different tools is not a core part of this proposal, yet we believe it would be beneficial. The list of directives used by Scala CLI would be a good starting point for a standardised list of settings.

Motivation

The primary motivation for introducing using directives is the ability to keep configuration together with code and share it as a simple text rather than a project with a structure. In addition, scripts, bug reports, tutorials, education materials, REPL or notebooks would greatly benefit from the ability to include configuration.

Magic imports from Ammonite are a good illustration that in-source configuration is needed in certain use cases.

Additionally, using directives could provide universal syntax shared across build tools, scripts, REPL or code snippets.

Using directives is a core part of Scala CLI; however, at this moment, we recommend placing them in comments using a special syntax: “//> using scala 3.1” and that’s a bit counterintuitive and misleading.

Using directives syntax:

We propose the following syntax for using directives:

UsingDirective ::= "using" (Setting | Settings)
Settings ::= "{" Setting { ";" Setting } [";"] "}"
Setting ::= Ident [Values | Settings]
Ident ::= ScalaIdent { “.” ScalaIdent }
Values ::= Value { "," Value} [","]
Value ::= stringLiteral | ["-"] numericLiteral | true | false

Where:

• A Settings block is similar to the standard Scala code block. Braces and indentation syntax is allowed. e.g.:

someSettings { setting1 value 1; setting2; }

// or

someSettings {
  setting1
  setting2
}

// or

someSettings:
  setting1
  setting2

• Ident is the standard Scala identifier or list of identifiers separated by dots:

foo

foo.bar

`foo-bar`.bazz

• String literals and numeric literals are similar to Scala.
• No value after the identifier is treated as true value: using scalaSettings.fatalWarnings
• Specifying a setting with the same path more than once and specifying the same setting with a list of values are equivalent
• A path created using dot-separated idents is semantically equivalent to a path created by nesting idents:
  foo.bar is equivalent to foo { bar }

Implementation

Scala CLI employs using directives to create a persistent configuration for projects. However, users still can deliver configuration data in the command line, which overrides any values from using directives.

Under the hood, Scala CLI uses a dedicated library called using_directives for parsing. Then it processes the data provided to extract the build configuration values. The library is written in Java, so it can be used in build tools / IDEs regardless of the Scala version. Furthermore, as the library is basically a fragment of the Scala 3 parser rewritten to Java, we believe that adding support for using directives to Scala compilers shouldn’t be difficult.

The library is still under development.

[som-snytt]

Both compiler test suites leverage magic leading comments

// scalac: -Werror -Xlint
// javaVersion: +9
// test: -jvm:+9

where arbitrary config is associated with canonical but arbitrary tool names.

using is just a placeholder word for syntax. Maybe there is another way to say, Here is some metadata for tools. I guess at least Using for resources is uppercased.

I just moved some config from the programmatic test rig to a test file directive. It’s so nice, I hope support arrives in some form, even if just to lower the “activation energy” required to get your favorite tool to use it.

How about #! script support that reads the special config and then, depending on what name was used to invoke the script, supplies the desired arguments to the tool.

It only sounds crazy when I say it.

[julienrf]

Thank you @romanowski for starting this discussion! Here are my thoughts.

About the motivation part, I would like to highlight something that I think is not really visible in your post: currently, distributing a Scala program as a source file (to be executed by scala my-program.scala) relies on a global installation of a Scala interpreter that has to match exactly the version used by the program (or, at least, be compatible with it). That makes it very fragile to distribute Scala programs as source files.

About the general idea of having using directives in the source code, I must say that I am still on the fence because using directives are not part of the program content, but they are meta-information on how to interpret the program content. The fact Scala compilers will have to implement support for parsing them just to ignore them feels odd to me:

@romanowski

From the compiler’s point of view, using directives will be treated just like a comment with a fixed syntax.

I wonder if special comments (e.g. //>) would be more appropriate for that, then.

Next, about the scope of the proposal:

@romanowski

This proposal does not cover the semantics of the using directives

So, if I understand correctly, this proposal is to add bespoke syntax for configuration information that could be processed by third-party tools. I wonder if we should go further and standardize the semantics of such directives to make it consistent across IDEs and tools (e.g., worksheets would have the same semantics in IntelliJ and Metals). I understand that it would be more work, and maybe it’s premature to do it now.

Last, about supporting alternative syntaxes:

@romanowski

someSettings {
  setting1
  setting2
}

// or

someSettings:
  setting1
  setting2

If we introduce new syntax, I strongly believe we should pick just one way of doing things.

[som-snytt]

Since doc gets special comment syntax, it seems natural for other job functions to get comment support as well.

It’s worth asking whether we spend more time writing doc comments or trying to configure stuff?

This is also a good opportunity to find comment syntax with useful mnemonics for teaching purposes. For example,

/*==
 *  setting
 */

would be a “comet comment” because it looks like a comet.

//:setting

would be “fold left” syntax because of the lamented operator.

I look forward to moving my sbt config into embedded comments, after I finish moving my Build.scala to build.sbt.

But since by now I’m used to putting my code in triple backticks, maybe it’s easier to embed code in config. Then after the backticks I could add Scala2 or Scala3 to specify the dialect; ifdef has been much requested for years. I would contribute backtick ScalaX3 to mean -Xsource:3.

[lihaoyi]

Personally I’m strongly opposed to this proposal, as is, for a few reasons:

1. It’s re-purposing an existing keyword in a weird way. using is kinda-sorta tangentially related to one particular use case of what we’re trying to do here, just like import $ is kinda-sorta tangentially related to one particular use case of what Ammonite does. But it’s different enough that it seems like it would cause confusion, not unlike how we used to use underscore _ for a bunch of scenarios that were kinda-sorta related

2. It doesn’t look like Scala. scala "3.1.2" does not look like valid Scala syntax at all. someSettings { setting1 value 1; setting2; } is technically valid Scala syntax, but is in the style of odd DSLs. It looks like Kotlin or Groovy with their convention of builder DSLs, but those aren’t ubiquitous in the Scala ecosystem.

3. The fact that it looks similar to Scala expressions at all is misleading. Can I import stuff? What’s the evaluation order? Can I define vals/defs/classes and so on? I assume not, and that this stuff would get statically inspected as metadata, rather than evaluated as code. This means the syntax is actively misleading people into thinking it’s Scala expressions, when it’s not

Overall, I think it looks about as smoothly integrated as Ammonite’s magic imports, and neither fits well enough into Scala that I’d be happy including them in the language.

TBH, if we want a way to associate metadata with Scala files, that sounds a lot like the existing way to associate metadata with pieces of Scala code: an annotation. I think some annotation-like syntax using @ would be a lot better than the using syntax proposed above. We’d need to finesse the syntax in order to make it unambiguous and ergonomic, but I think it can be done and result in a much more familiar user experience. This applies to people coming from any language, as most languages have some sort of annotation-like syntax for associating metadata with code, and that’s exactly what we want to do here.

[som-snytt]

The only thing I love more than an enthusiastic lihaoyi presentation is when he is strongly opposed.

I think annotations suffer from the problem that they require processing (typing). By the time you’ve typechecked, you already needed the metadata, which may include compile time or run time directives.

I also agree with the sentiment that “it doesn’t look like Scala”, but “Scala 3” also didn’t look like Scala.

In any case, it’s a great pleasure to follow the discussion.

[rjolly]

As I understand it, this would supersede language imports:

using language.postfixOps

Conversely, we could use import for general settings:

import scala "3.1.2"

It would address the objection that using already has its (different) purpose. To avoid name clashes, we could use some separate namespace:

import settings.scala "3.1.2"

Edit: It also adresses the objection that looking similar to Scala expressions is misleading : imports have their own separate syntax and it is not so much of a problem. It would have to undergo a non-trivial extension though. So maybe a different keyword is better after all.

[rssh]

Or without breaking scala syntax:

import language.version.`3.1.2`

[lihaoyi]

Yeah, that’s basically Ammonite magic imports

import $language.version.`3.1.2`

I like Ammonite magic imports. They’re great, do their job, look kinda-sorta OK-ish, and make sense for a limited set of use cases that Ammonite leverages heavily. But they are no doubt clunky, and get even more clunky when used as a general mechanism for annotating metadata outside of their core use cases. Not something I would support baking into the Scala language. And I invented them!

I feel like using directives have basically the same properties as magic imports, and everything above applies to using directives just as much. I wouldn’t support baking them into the Scala language either

[Katrix]

It would from what I understand not supercede language imports, as those can be locally scoped, while these can not.

[som-snytt]

Not to be supersillyass or anything, but (as I learned once the hard way) the spelling is supersede because it’s about sitting (sedere).

Supersedure is replacement of an old queen bee. It’s too bad that long word doesn’t actually have enough distinct letters for the spelling bee puzzle.

There is a word supersedeas but it sounds too legalese for the puzzle. It’s pronounced like super tedious.

[Sporarum]

I believe we can make english spelling easier to learn, and more regular, if we stop correcting people’s “mistakes”, especially in cases where it doesn’t impede comprehension !
Especially given this is a forum, a realm of written spoken english, rather than (formal) written english

Note that this can also discourage people whose first language is not english from posting, which would be a shame !

[som-snytt]

Apologies to anyone who was discouraged by my post, which I intended to be suitably couched in fun and inviting language.

I’m not pedantic about spelling, descriptive not prescriptive and all that, but it’s pleasant when in the rare case the standard English spelling actually has a reason.

Also my spontaneous transliteration of supercilious was a good one.

Since “supersede” is a common word, with PRs said to be superseding one another, I hope anyone who needs to find it in a dictionary can now do so. I learned it in a legal document. Sorry again to derail the discussion of the word “using”.

[amsayk]

I think pragma would fit perfectly as a keyword for this. It is well known for this kind of feature in other programming languages.

[julienrf]

Here is a summary of the main points that have been raised so far. I suggest taking them into account before submitting an actual SIP.

Motivation

@julienrf

About the motivation part, I would like to highlight something that I think is not really visible in your post: currently, distributing a Scala program as a source file (to be executed by scala my-program.scala) relies on a global installation of a Scala interpreter that has to match exactly the version used by the program (or, at least, be compatible with it). That makes it very fragile to distribute Scala programs as source files.

Do you (@romanowski) plan to update the motivation part to include that point?

Alternative Solution: Comments

@som-snytt

Both compiler test suites leverage magic leading comments

@julienrf

The fact Scala compilers will have to implement support for parsing them just to ignore them feels odd to me:

@romanowski

From the compiler’s point of view, using directives will be treated just like a comment with a fixed syntax.

I wonder if special comments (e.g. //>) would be more appropriate for that, then.

@lihaoyi

The fact that it looks similar to Scala expressions at all is misleading. Can I import stuff? What’s the evaluation order? Can I define vals/defs/classes and so on? I assume not, and that this stuff would get statically inspected as metadata, rather than evaluated as code. This means the syntax is actively misleading people into thinking it’s Scala expressions, when it’s not

Could you please expand on why you think “using directives” is the best solution vs comments?

Alternative Solution: Annotations

@lihaoyi

TBH, if we want a way to associate metadata with Scala files, that sounds a lot like the existing way to associate metadata with pieces of Scala code: an annotation. I think some annotation-like syntax using @ would be a lot better than the using syntax proposed above. We’d need to finesse the syntax in order to make it unambiguous and ergonomic, but I think it can be done and result in a much more familiar user experience. This applies to people coming from any language, as most languages have some sort of annotation-like syntax for associating metadata with code, and that’s exactly what we want to do here.

Probably, a discussion on a solution based on annotations should be included. An argument against annotation has been made here already:

@som-snytt

I think annotations suffer from the problem that they require processing (typing). By the time you’ve typechecked, you already needed the metadata, which may include compile time or run time directives.

Alternative Solution: Language Imports

@rjolly

Conversely, we could use import for general settings:

import scala "3.1.2"

It would address the objection that using already has its (different) purpose.

@rssh

Or without breaking scala syntax:

import language.version.`3.1.2`

Maybe consider including language imports in the alternative solutions. There is already an argument against reusing imports:

@Katrix

It would from what I understand not supercede language imports, as those can be locally scoped, while these can not.

Re-purposing the using keyword vs introducing a new keyword

@lihaoyi

It’s re-purposing an existing keyword in a weird way. using is kinda-sorta tangentially related to one particular use case of what we’re trying to do here, just like import $ is kinda-sorta tangentially related to one particular use case of what Ammonite does. But it’s different enough that it seems like it would cause confusion, not unlike how we used to use underscore _ for a bunch of scenarios that were kinda-sorta related

Would you mind elaborating on why you think re-purposing using is the best solution? Should we introduce another keyword instead? E.g. “pragma”?

@amsayk

I think pragma would fit perfectly as a keyword for this. It is well known for this kind of feature in other programming languages.

Syntax does not look like usual Scala

@lihaoyi

It doesn’t look like Scala. scala "3.1.2" does not look like valid Scala syntax at all. someSettings { setting1 value 1; setting2; } is technically valid Scala syntax, but is in the style of odd DSLs. It looks like Kotlin or Groovy with their convention of builder DSLs, but those aren’t ubiquitous in the Scala ecosystem.

[romanowski]

Thank you, everyone, for the comments and suggestions.

After many discussions, I get convinced to stick to the comment syntax for now. Usability gains with dropping special syntax for comments are small, and there is non-zero potential to confuse users. It also seems that early adopters of Scala CLI are content with comment-based syntax.

I think this concludes the following points: Re-purposing the using keyword vs introducing a new keyword, Syntax does not look like usual Scala, Alternative Solution: Comments.

Alternative Solution: Language Imports

We haven’t picked imports since the scope of the import is limited to a single file but using directives applies configuration to the whole compilation unit (e.g. compiler options). I think overloading imports with meanings (import members to the scope and import settings to the build) is not a good direction.

Alternative Solution: Annotations

We were considering annotation initially, but there is a critical semantic problem with annotations: annotation provides metadata for something (a class, method, parameter etc.). In the case of Using Directives, the metadata is provided to the whole compilation unit, a concept that is not materialized within a source file, so there is nothing to annotate. Moreover, similarly to imports, we would use the same constructs and syntax for 2 different purposes and this is something that I think we should avoid.

[SethTisue]

Glad for this outcome.

And then at that point, does it need a SIP any more at all…?

[romanowski]

does it need a SIP any more at all…?

I do not think so.