Unification of sbt shell notation and build.sbt DSL

[nadavwr]

Maybe a ‘/:’ for the same evaluation order as ‘in’?

[tpolecat]

What does the DSL desugar to? I want to know what the normal form is before trying to figure out what the sugar needs to look like. What do := and .value mean?

[eed3si9n]

@OlegYch

Is it necessary to flip the order to support / syntax?

As many have noted / has a strong notion of contains-a relationship, so flipping will be confusing.

How about ‘key in axis in axis in axis’ syntax?

Solution 2 #3259 actually supports both key in (proj, Config, intask) notation and key in axis in axis in axis.

Having said that, I generally recommend using in with tuple because you could write nonsensical scopes like scalacOptions in Compile in Test or something order dependent:

• scalacOptions in Compile in Global
• scalacOptions in Global in Compile

@olafurpg

Can option 2 support the same nice tab completion as the current shell syntax? Tab completion is very important for me to discover/explore builds.

Solution 2 will have tab completion for unscoped keys:

> tes[tab]
> test
> test[tab]
> test
::                   test:                testForkedParallel   testFrameworks       testGrouping         testListeners        testLoader           testOnly             testOptions
testQuick

I think it covers the case for most usages. The current parser and the slash syntax will have the advantage when we start to do scoping by Test for example, but these are relatively exotic usages, I think.

I really like the new proposed slash syntax, but I’m afraid the migration cost would only further delay sbt v1.0 adoption.

As demonstrated by the fact that I first sent the PRs to 0.13 branch, I was able to overlay these solutions on top of existing shell and build.sbt DSL without breaking source or binary compatibility. The decision to drop the old shell parser and the in method in build.sbt is a knob we can tweak. There seems to be “Keep Old in 1.0” and “Drop Old in 1.0” camps.

@nadavwr

Maybe a ‘/:’ for the same evaluation order as ‘in’?

Not sure what this suggestion is in response for, but /: means foldLeft, and it just makes simple / more cryptic.

@tpolecat

What does the DSL desugar to? I want to know what the normal form is before trying to figure out what the sugar needs to look like.

The topic on this thread is string representation of a data structure called ScopedKey in sbt shell and unifying that with setting/task keys in build.sbt.

The implementation of setting key’s in “operator” is actually a method Structure.scala#L38 that takes a Scope as an argument and returns a new setting key.
There are a bunch of overloads for in methods that creates Scope, which is just a case class consisting of scope axes.

[OlegYch]

if ‘key in axis in axis in axis’ is supported then the only major difference between 1 and 2 is the position of ‘key’
(i don’t think replacing ‘in’ with ‘/’ is worth the community effort)
I think we have to consider two things:

1. which position is better for completion
2. which position provides better command line editing experience, eg makes it easier to inspect same key in multiple configs, or multiple keys in same project
   alternatively, perhaps it’s possible to support both leading and trailing ‘key’?
   so far i’m leaning towards option 2 if it doesn’t inhibit completion too much

[OlegYch]

just realized ‘axis in key’ would not sound good, so it would be necessary to use / if key is trailing
but i still think we have to consider the above two things first

[olafurpg]

@eed3si9n

Solution 2 will have tab completion for unscoped keys:

What about keys scoped by project?

> test in m<TAB>
> test in myProject
// OR
> testOnly in myProject c<TAB>
> testOnly in myProject com.MySpec

The order of arguments to the overloaded in method has always confused me. At a glimpse, the / syntax seems more intuitive, I’d say it deserves a shot as long as the old behavior keeps working. I will probably need hand-on experience with the / syntax to understand the full implications.

The decision to drop the old shell parser and the in method in build.sbt is a knob we can tweak. There seems to be “Keep Old in 1.0” and “Drop Old in 1.0” camps.

Awesome. I’m BTW thrilled to have this discussion on improving sbt user-experience. I’d say “keep old in 1.0” is the right choice for this particular change, practically every sbt build in the wild uses in. I also suspect a few bash scripts rely on the current shell syntax sbt foo/bar::baz:qux.

[eed3si9n]

@olafurpg

Solution 2 will have tab completion for unscoped keys:

What about keys scoped by project?

> test in m<TAB>
> test in myProject
// OR
> testOnly in myProject c<TAB>
> testOnly in myProject com.MySpec

Once you have typed in test or testOnly, the tab completion does function to complete the subproject id or the available test suites.

[sjrd]

@eed3si9n

A setting key or a task key can hang under any levels. That doesn’t necessarily mean that the axes aren’t hierarchical: it just means some of the values can be set to Zero. For example, key in Compile is short for key in (<current project>, Compile, Zero).

A setting key or task key can also hang somewhere that can hardly be called a level. For example key in (<current project>, Zero, compile). How do you reconcile that with the hierarchical model? Is it thisProject / Zero / compile / key?

[eed3si9n]

@sjrd

For example key in (<current project>, Zero, compile). How do you reconcile that with the hierarchical model? Is it thisProject / Zero / compile / key?

Yes, that would be the absolute coordinate of a subproject-and-task scoped key, which does occur with plugin settings. In reality, the user would write something like:

assembly / assemblyJarName := "foo.jar"

or type in:

> assembly/assemblyJarName

in the shell. I can deduce this because I am checking that “assembly” in the first segment is not a subproject id.

[nadavwr]

@eed3si9n The ‘/:’ comment was in response to @OlegYch. I meant it as a drop in replacement for ‘in’. In second thought it’s a horrible idea—describing hierarchy in reverse with ‘/:’ won’t make any visual sense.

Do people still use ‘/:’ for foldLeft? I think of it as a relic.

[jducoeur]

@nadavwr

Do people still use ‘/:’ for foldLeft?

All the time, yes. I’m always a bit mystified why it gets so much hate – I’ve always found it intuitive…

[hohonuuli]

+1 for /: hate.

Seriously, SBT has been full of these weird arcane symbols. These are thankfully being deprecated, but I vote for getting rid of symbolic method names as much as possible. After years, I still have build files with <<= in them and I have no real idea what that means.

[lihaoyi]

@Daxten

As some here already argumented, let us stop reinventing the wheel. There is already a syntax for mutable variables in scala, so why not use that?

In many ways, lots of SBT is re-inventing the wheel: re-inventing values, variables, scopes, assignments, etc… I don’t like that either. But even if we wanted dot-notation to work, I don’t see a clean way to implement it in the build files given the constraints of the current SBT implementation, and “fixing” those constraints to avoid SBT re-inventing all those things is probably a huge undertaking outside the scope of this discussion.

Chris Vogt’s CBT (Video) does tons of interesting work in trying to not re-invent stuff, in a direction you’d probably be happy with, but if you watch his talk it’s a total ground-up re-think and re-write of what a build-tool is. We should encourage such efforts, but they can happen in parallel and shouldn’t get in the way of this kind of incremental quality-of-life improvement to SBT: collapsing two distinct, idiosyncratic, equivalent syntaxes into one.

[muuki88]

Thanks @eed3si9n for your work on this.

At this point I favor the new / syntax if I get the direction of this proposal correctly.

(1) No Nonsensical scopes

@eed3si9n

Having said that, I generally recommend using in with tuple because you could write nonsensical scopes like scalacOptions in Compile in Test or something order dependent:

From my understanding the / syntax doesn’t allow arbitrary combinations of scopes, but only a well defined set. E.g.

// okay
root / Compile / compile := ...
Compile / compile := ...

// fails to compile
compile / Compile := ...

Which will get rid of various typos and confusing errors.

(2) Readability & Usability

Tab completion works more naturally IMHO and when all axis are spelled out things get really clear
where settings are applied.

I’m not sure if this is a good idea, but things like this would be possible

// scope multiple keys to a scope
Compile / Seq(
   name := ...,
   test := ...
)

But which is nice for more nested settings, e.g.

Debian / assembly / Seq (
   name := ...,
   assemblyJar := ...
)

(3) Visualizing == Using

Display all values for a key would look as natural as writing it. E.g. with some kind of “inspect scopes” command

$ sbt
> inspectScopes compile
root / Compile / compile := Defined at Defaults.scala:271
root / compile := Defined at Defaults.scala:271
root / Test / compile := Defined at Defaults.scala:271

IMHO this would look a bit strange and unreadable with the in notation.

Looking forward to see this UX improvement in SBT. Thanks a lot

[jvican]

Hey guys,

I am really enthusiastic about these improvements to the sbt UX .

I lean towards the / syntax because:

• Autocompletion works flawlessly and accomodates all the use cases that the current syntax supports now.

  The in syntax does not allow you to inspect / autocomplete keys for a subproject. For example, if you have a key in a subproject foo, you cannot do foo/<TAB> and autocomplete like you would do now – which means users need to know beforehand what keys exist in all the subproject, which is definitely not something that they know. I believe this is a very common use case, so that’s why I think in is not the candidate we’re after.

  Note that the same happens not only for subprojects but for any scope component (that is, configurations like Compile and tasks that can have other keys defined inside). In my opinion, merging the in unification syntax into sbt will render the tool unusable for most of the users since they are not aware of what’s inside what and they rely heavily on sbt to answer their questions.

• Scopes are vertically aligned, which makes them easier to read (in both sbt files and the shell output like the inspect command):

  Compile / compileInputs := { ... }
  Compile / compile := { ... }
  Test / test := { ... }
  Test / cacheDirectory := { ... }
  Test / target := { ... } 
  

• The / syntax gives scope keys the importance they deserve by being placed first. In my opinion, the in syntax makes scopes more difficult to spot and feels less natural. Making scopes more obvious is a win for clarity given their major role in the use of the build tool.

• Adding an extra scope to an existing key is easier. If we have project / key, we just place the cursor before key and type Compile /. With the in syntax, you first have key in project and to modify it we type key in (Compile, project), which is a little bit harder. The tuple is the most common format to represent complex scopes and feels more awkward than project / Compile / key.

What are the disadvantages of the / syntax? Well, it breaks 0.13.x syntax and the way people write sbt code now. If we’re to pick this one, we must ask ourselves: how are we going to improve sbt users’ life without forcing them to rewrite all their code?

To answer this question, I am working on a migration tool called sbtfix that I believe can migrate all or at least most of the uses of in to the new / syntax, both in sbt files and plugins (note that the README is outdated). I will show in the next days a rewrite to prove this claim.

So, this leaves us with one last inconvenience: people need to get used to the new syntax. I think this is the price to pay to improve the usability of any tool.

However, the good thing is that if they get it wrong and write in instead, they will not get a compile error, but a deprecation error. And if they add sbtfix to their global plugins, sbtfix will rewrite the in syntax for free. Of course, existing 0.13 sbt code in docs and StackOverflow questions will still work!

We always complain sbt is hard. This is our chance to make it easier. Sbt 1.0.x can go down in history as the release that significantly simplified the use of the tool. I say let’s be bold and take this chance.

[scf37]

IMO, best sbt devs can do is to:

• separate build.sbt DSL classes from implementation
• document them well using scaladoc

I was able to use sbt freely only after spending several days understanding object model and how it is glued together.

P.S. still have no idea how to get ProjectRef by Project and vice versa.

[eed3si9n]

Thanks everyone for your inputs.

It seems like there are stronger preference towards the slash syntax, but there are some concerns around backward compatibility of existing builds. I think we can address backward compatibility by keeping around the old syntax for the duration of sbt 1.x series.

There’s also been concerns around introducing this change late in the game, so after discussing with Dale, we decided to introduce this as a form of a plugin. This way you can start playing around with it without waiting for sbt 1, and get better feedback. https://github.com/sbt/sbt-slash

[jvican]

Hey guys,

Just wanted to drop by to say that I’m enjoying sbt-slash a lot. I highly encourage you to give it a try. Not having to deal with the shell syntax makes the sbt experience much nicer.

Try it and let us know your feedback!

[eed3si9n]

eugene yokota 🥙 (eed3si9n)

The PR for slash syntax is now merged. Here’s a sneak preview:

[jvican]

Congratulations on the bold move. I’m optimistic this will make the use of sbt much simpler.