Unification of sbt shell notation and build.sbt DSL


#1

I’ve long thought that sbt having two notations for setting keys, one for sbt shell and another for build.sbt made the tool harder to learn.

As a recap:

  • in sbt shell the full form of the scoped key is written as <project-id>/config:intask::key
  • in build.sbt, it’s written as key in (proj, Config, intask)

Recently I’ve implemented two solutions for unifying the notation.

solution 1: Unified slash syntax

First implementation is actually a third syntax that can be used for both the shell and build.sbt. Instead of /, :, and ::, the slash syntax uses / for all levels.

> Global / cancelable
> ThisBuild / scalaVersion
> Test / test
> root / Compile / compile / scalacOptions

In build.sbt it would look like this:

lazy val root = (project in file("."))
  .settings(
    Global / cancelable := true,
    ThisBuild / scalaVersion := "2.12.2",
    Test / test := (),
    Compile / console / scalacOptions += "-Ywarn-numeric-widen"
  )

solution 2: Unify to build.sbt DSL

Second implementation is unifying towards the current build.sbt DSL. This accepts the use of key in (proj, Config, intask) notation in sbt shell:

> cancelable in Global
> scalaVersion.in(ThisBuild)
> test in Test
> scalacOptions in (root, Compile, compile)
> scalacOptions in root in Compile in compile

notes

In both solutions, the String representation of scoped keys in commands such as inspect should work both both the shell and build.sbt.

Edit:
Another interesting thing to note is that both the slash syntax and build.sbt parser can overlay on top of the existing 0.13 ( <project-id>/config:intask::key) shell. As demonstrated by the fact that I first sent the PR to 0.13 branch, keeping the old shell notation around or not is another knob we can tweak.

discussion

Now that there are two solutions, the question is which one is better.

@sjrd has argued against the slash syntax:

The mere fact of changing the syntax will have dramatic consequences on the eco-system.

… The last time we had a change of syntax was in sbt 0.13, 4 years ago, we and are still paying the price of changing that syntax, because there is so much documentation and SO answers out there using the old syntax. …

If we’re going to unify the syntaxes of the REPL and .sbt files, I would suggest we don’t touch the syntax of .sbt files, and instead only adapt the syntax of the REPL to be the same.

@lihaoyi is in favor of the slash syntax over build.sbt:

I personally prefer the slash syntax to the build.sbt syntax.

… If we believe that the 0.13 syntax is the problem, rather than the rate-of-change of syntax, then the correct thing to do isn’t to avoid further change but rather to evolve away from the problematic parts of 0.13 syntax as fast as possible.

What are your thoughts? Which solution do you think will reduce the cognitive load of learning sbt?


Improving the compilation error reporting of sbt
#2

Isn’t there any way to make the sbt shell be “just” a scala shell?

BTW the two sides you presented (@sjrd and @lihaoyi) are really broader than any particular syntax proposal:

The fact is that currently the .sbt syntax is pretty verbose and hard to get right. And the current shell syntax is not valid scala. So the problem can be phrased as, can we have a syntax that’s shell-friendly and valid scala, without causing any problems (such as ecosystem disruption, symbol ambiguity, etc. etc.)… and if not, what’s the best tradeoff we can have? (Again, this is not particular to any specific proposal.)


#3

I believe the / syntax is far more intuitive IF configurations are really a hierarchy. I have never met a developer who didn’t understand a tree using / syntax but @sjrd I believe has argued that it is not truly a hierarchy.

When I see
key in (proj, Config, intask) my intuition is to think that this key is being set for proj, Config, and intask, in other words this key is being set in three separate places.

When I see proj / Config / intask / key I believe this key is set in one very specific place, three levels deep, very definitely not for all instances of intask, only the intask that is inside Config, that is inside the specific project proj.


#4

Would either of these support good code completion? Hitting . and seeing what options come up is the most common way people explore code.


#5

Isn’t there any way to make the sbt shell be “just” a scala shell?

I used to wonder the same (for example https://github.com/sbt/sbt/issues/1812 was first written from that angle). There’s a command called consoleProject if you want to experience the shell as plain Scala REPL, and it’s a good exercise to realize the niceties the sbt shell provides that we take for granted.

  • To compile, you’d need to type in (compile in Compile).eval and to test, (test in Test).eval. This is because shell can find the configuration axis that the given key is in.
  • Various commands such as project (multi-project support), ~ (file watch), ; (sequential processing), and + (cross building) would need to be reimplemented.

Technically all these challenges could be overcome, but it would be too much work for an unknown gain in terms of making sbt easier. Changing the shell notation to make it look like Scala expression was something feasible.


#6

The scope axes forms a hierarchy of has-a relationship in my opinion.

  1. At the top level is a build and global.
  2. A build contains multiple subprojects.
  3. A subproject contains a distinct sequence of configurations. Typically they are the same Compile, Runtime, Test, …, but they can be changed for each subproject.
  4. A configuration contains some of the tasks. (but not all)

A setting key or a task key can hang under any levels. That doesn’t necessarily mean that the axes aren’t hierarchical: it just means some of the values can be set to Zero. For example, key in Compile is short for key in (<current project>, Compile, Zero).


#7

I don’t know if either is better in build.sbt. In the sbt shell, the slash syntax will retain the current tab completion. For example:

  • Test/test
  • test in Test

In the above, I can narrow down the candidate after Test/[tab] to only the keys under Test configuration.


#8

I think the / syntax is better in that we are more accustomed to reading hierarchies left-to-right, it always baffled me that I wouldn’t write something more apt to Test.test

The change in syntax might even become a good occasion to change surprising defaults in scoping, e.g. making ThisBuild the default; for instance, idea (not sure about the feasibility) : it would be cool if ThisBuild could be somehow shortened to a leading /, so that one may write /foo to mean ThisBuild/foo


#9
    Global.cancelable = true,
    ThisBuild.scalaVersion = "2.12.2",
    Test.test = (),
    Compile.console.scalacOptions += "-Ywarn-numeric-widen"

As some here already argumented, let us stop reinventing the wheel. There is already a syntax for mutable variables in scala, so why not use that?

Exploring the API is very hard atm and this would make it way easier


#10

I will comment on this thread soon. I need some time to think about the best way to unify both syntaxes, and the consequences of the aforementioned approaches.

It is not reinventing the wheel, and := is a nice way to define the implementation of a task or setting. You cannot use =, it’s a reserved keyword. You would need to define it as a symbol and force users to use backticks for every =. Note that this thread is about unifying the sbt DSL and the shell syntax, which have been historically different.


#11

Is it necessary to flip the order to support / syntax?
How about ‘key in axis in axis in axis’ syntax?
It looks like it will be easier for command line use and still will be compatible with old scala syntax.
TBH different order in command line and scala and requirement to tuple axis specifiers was always tripping me up.


#12

Daxten, using dots really would make for nice shell syntax, but seems to me, at best, to require major hacks and adoption pains to also cover build files. Barring that, it would just expand the gap between shell and build files—not a desirable outcome.

The 0.13 syntax, broadly speaking, is a nice leaky abstraction—users can usually intuit over it almost as easily as they would over mutable state, while still being reminded that there’s something else going on under the hood. That aspect of the 0.13 API isn’t broken.


#13

Can option 2 support the same nice tab completion as the current shell syntax? Tab completion is very important for me to discover/explore builds. I really like the new proposed slash syntax, but I’m afraid the migration cost would only further delay sbt v1.0 adoption.


#14

Maybe a ‘/:’ for the same evaluation order as ‘in’?


#15

What does the DSL desugar to? I want to know what the normal form is before trying to figure out what the sugar needs to look like. What do := and .value mean?


#16

@OlegYch

Is it necessary to flip the order to support / syntax?

As many have noted / has a strong notion of contains-a relationship, so flipping will be confusing.

How about ‘key in axis in axis in axis’ syntax?

Solution 2 #3259 actually supports both key in (proj, Config, intask) notation and key in axis in axis in axis.

Having said that, I generally recommend using in with tuple because you could write nonsensical scopes like scalacOptions in Compile in Test or something order dependent:

  • scalacOptions in Compile in Global
  • scalacOptions in Global in Compile

@olafurpg

Can option 2 support the same nice tab completion as the current shell syntax? Tab completion is very important for me to discover/explore builds.

Solution 2 will have tab completion for unscoped keys:

> tes[tab]
> test
> test[tab]
> test
::                   test:                testForkedParallel   testFrameworks       testGrouping         testListeners        testLoader           testOnly             testOptions
testQuick

I think it covers the case for most usages. The current parser and the slash syntax will have the advantage when we start to do scoping by Test for example, but these are relatively exotic usages, I think.

I really like the new proposed slash syntax, but I’m afraid the migration cost would only further delay sbt v1.0 adoption.

As demonstrated by the fact that I first sent the PRs to 0.13 branch, I was able to overlay these solutions on top of existing shell and build.sbt DSL without breaking source or binary compatibility. The decision to drop the old shell parser and the in method in build.sbt is a knob we can tweak. There seems to be “Keep Old in 1.0” and “Drop Old in 1.0” camps.

@nadavwr

Maybe a ‘/:’ for the same evaluation order as ‘in’?

Not sure what this suggestion is in response for, but /: means foldLeft, and it just makes simple / more cryptic.

@tpolecat

What does the DSL desugar to? I want to know what the normal form is before trying to figure out what the sugar needs to look like.

The topic on this thread is string representation of a data structure called ScopedKey in sbt shell and unifying that with setting/task keys in build.sbt.

The implementation of setting key’s in “operator” is actually a method Structure.scala#L38 that takes a Scope as an argument and returns a new setting key.
There are a bunch of overloads for in methods that creates Scope, which is just a case class consisting of scope axes.


#17

if ‘key in axis in axis in axis’ is supported then the only major difference between 1 and 2 is the position of ‘key’
(i don’t think replacing ‘in’ with ‘/’ is worth the community effort)
I think we have to consider two things:

  1. which position is better for completion
  2. which position provides better command line editing experience, eg makes it easier to inspect same key in multiple configs, or multiple keys in same project
    alternatively, perhaps it’s possible to support both leading and trailing ‘key’?
    so far i’m leaning towards option 2 if it doesn’t inhibit completion too much

#18

just realized ‘axis in key’ would not sound good, so it would be necessary to use / if key is trailing
but i still think we have to consider the above two things first


#19

What about keys scoped by project?

> test in m<TAB>
> test in myProject
// OR
> testOnly in myProject c<TAB>
> testOnly in myProject com.MySpec

The order of arguments to the overloaded in method has always confused me. At a glimpse, the / syntax seems more intuitive, I’d say it deserves a shot as long as the old behavior keeps working. I will probably need hand-on experience with the / syntax to understand the full implications.

The decision to drop the old shell parser and the in method in build.sbt is a knob we can tweak. There seems to be “Keep Old in 1.0” and “Drop Old in 1.0” camps.

Awesome. I’m BTW thrilled to have this discussion on improving sbt user-experience. I’d say “keep old in 1.0” is the right choice for this particular change, practically every sbt build in the wild uses in. I also suspect a few bash scripts rely on the current shell syntax sbt foo/bar::baz:qux.


#20

@olafurpg

Solution 2 will have tab completion for unscoped keys:

What about keys scoped by project?

> test in m<TAB>
> test in myProject
// OR
> testOnly in myProject c<TAB>
> testOnly in myProject com.MySpec

Once you have typed in test or testOnly, the tab completion does function to complete the subproject id or the available test suites.