other than the 18 open tickets I’ve created at sbt/sbt https://github.com/sbt/sbt/issues/created_by/fommil I feel the most important thing right now is getting a 2.11 or 2.12 build of sbt. The reason is because as long as sbt uses 2.10, the entire scala ecosystem feels compelled to crosscompile to 2.10 and that is becoming harder and harder and slowing down the adoption of technologies like scala.meta.
sbt needs to stick to the original plan of 2 or 3 year cycles allowing for binary breaking changes. For now, I don’t even care about new features, just please move to a new version of scala so we can all let go of 2.10 (we can all still offer commercial support for legacy versions of scala, but remember we are almost all volunteers and it is burning us out).
Single jar download of sbt would also be very very useful and speed up the initial install of scala for beginners.
I assume most of the things on your list would happen in a future 1.0.x or 1.x release, and not hold up the release of 1.0.0? Like Sam, I’m mainly interested (both personally, and in my opinions about what benefits the community at large) in getting on Scala 2.12, and that means making 1.0.0 happen as soon as reasonably possible.
Overall in day to day use SBT is great, I only have a couple of issues that I see come up repeatedly in daily use (which I am keeping separate from implementing new sbt tasks).
No support for putting a cache of dependencies in my repo for true offline mode. Related to https://github.com/sbt/sbt/wiki/User-Stories:--Offline-mode-and-Dependency-Locking I would love to be able to switch to any branch and build offline. I already have a way to get my code to a known state, git. It would be wonderful if I could add a command to keep dependencies used in the project in a cache in my project directory. There has recently been a similar feature putting the cache in the users home directory, which is almost what is needed. If someone clones a scala project, it would be wonderful if they could just build it, always, even if maven was down, or sbt publishing on sonatype was broken.
Error messages from scalac are not obviously actionable for end users as most don’t understand the difference between scalac and sbt. If you run sbt compile and get a message to rerun with -feature for details it is very confusing for them to then run sbt compile -feature and find it doesn’t help.
Separately, extending SBT is made difficult for the average user because it cannot be done in the same way as their day to day programming. If you program line of business applications in intellij in scala every day as I am guessing the overwhelming majority of users do, when something goes wrong you just add a test around the problem, maybe add a breakpoint and then run/debug your test. This model of development falls down quickly with SBT as you cannot easily write a test for it or set a breakpoint on it from IntelliJ which means you have to learn a new way of working to add that task you wanted to SBT, along with of course learning the mental model of SBT. It is a large enough hurdle few do it and instead resort to copying and pasting from any example they can find.
I am not offering any solutions here, just sharing a few common pain points I have suffered and seen.
The release of 1.0.0 has an approximate deadline and we will try to merge as many things as possible into sbt 1.0 respecting such expected release date. More details are to come in the roadmap to be published by the sbt team.
Some of the issues described in this list are ideas that if we don’t make them before sbt 1.0, could be done later on. But it’s unclear if the Scala Center will keep on contributing to sbt in the future. This is why we want to focus on the most urgent things.
Yes, I wholeheartedly agree. I know that for some of the Scala Center projects supporting 2.10 takes a considerable amount of our time. Keep the eyes open to the roadmap that sbt team will announce shortly, it happens that sbt 1.0 will be coming to our hands earlier than expected.
I would like to see one-jar installation, or at least a simplification of the published jars that are resolved. Apparently, the current design is due to the desire of having independent sbt modules that you can independently update (like Zinc). Related discussion here.
Interesting! I was not aware of that document you link to. The dependency lock file is in our list, and I agree it’s important to add it to sbt. You already mention the most direct use case, offline mode, but I think that it would be wonderful for two things more:
Bring build reproducibility. Module resolution will always bring the same result because it’s cached.
Remove unnecessary dependency resolution, which is expensive. Currently, sbt spends time resolving dependencies when:
You change scala versions (cross-compiling) or projects with different scala versions.
You run sbt in your CI.
@eed3si9n and I have been discussing this several times in Gitter and this ticket. Swing by and let us know what you think about our current proposal.
Any idea how could we fix this? It seems that the root of the problem is that they don’t see a clear distinction between sbt and Scala. I haven’t been in that position before, so I cannot imagine what could be done to make it more obvious. Perhaps you can reach out to those users (if you know them) and let us know what they think?
@sjrd also proposed a way to add a dynamic inspect-tree option that would report on all the tasks and settings executed in a given run, opposed to the current inspect-tree that only shows static task dependencies. This task would run the task to “inspect”. It is useful when you start having things like Def.taskDyn that are not capable of showing its dependencies.
My experience has been–which appears to be general consensus also–that it is very difficult to know how to get SBT to do something that isn’t a very slight variant on something you already have a formulaic solution for.
Much of this is dictated by SBT’s overall design, which at this point cannot be substantively changed. But it could be vastly better documented. The existing documentation, long though it is, leaves one exceedingly far from mastery. Documentation that carefully describes how each higher level thing (e.g. project .dependsOn) is implemented at the next lower level (something do with tasks, I’m guessing?), and how to get access to that,and change it, and if one needs to modify existing stuff what the evaluation order is, etc. etc., would enable people who needed to to come to the required level of mastery in a more reasonable amount of time.
Whether we continue working on sbt long-term is something that depends on the Advisory Board. Our current effort is to improve the build tool as much as possible in a concrete timeframe as this is an internal initiative, but we cannot make long-term plans for now – that would probably require either a SCP or direct approval of the board.
Here are some thoughts on documentation (which has already improved a lot in the last year),
I understand that the website isn’t currently auto-generated on check-in. This could be automated.
There are lots of things which when googled get you old versions of the documentation, but no equivalent in the new documentation. There are lots which do, so this would filling in the gaps.
As a (fairly advanced) user it’s very confusing when browsing the docs trying to understand if you should be reading the 0.13 docs or the 1.0 docs (or why there are even 1.0 docs tbh).
There is no official (or semi-official) documentation on plugins. There are a huge number of really good plugins but it can be quite hard to discover them. Scaladex could be made to integrate really nicely. Right now it doesn’t seem to have half the plugins correctly tagged (https://index.scala-lang.org/search?q=keywords:sbt-plugin)
There’s loads of stuff that isn’t documented at all but which really great answers exist in Stackoverflow. I think some sort of triage of those might show where the current docs could be improved. (10 second google produced this example for reference, http://stackoverflow.com/a/25781310/58005)
For me the most urgent issue is the state of the source code, which makes it very hard for people to contribute. The formatting and organization are absolutely batshit and there are very few comments, which leads to instant demoralization and table-flipping, for me at least. So I request some attention to #2944.
Related, I would like to see documentation on every macro and every use of reflection, and how to write a build that doesn’t rely on these. The things that make sbt “easy” also make it unlike normal scala and thus very hard to understand.
Yes, this is a problem, and a big one. IMO, docs are great but they are not teaching sbt, but showing how to do stuff with it (which is conceptually different). It seems that there’s not a reasonable getting started tutorial. I think this was the idea with sbt in Action, but it’s a paid book which is less than ideal for the OSS community around sbt.
This will come with sbt 1.0 soon. We will help you get there by easing the migration between 0.13 and 1.0 with sbt-migration-rewrites.
Oh, this is a good one. It’s true that plugin documentation is poor and could be improved, both for writing plugins and finding them. I feel we sometimes reinvent the wheel many times when there’s a plugin that solves our problem, but we haven’t found it.
Regarding Scaladex, @MasseGuillaume and @julienrf have recently worked on that use case. For now, Scaladex can show sbt plugins published to Maven Central, but cannot support sbt plugins in the sbt Bintray repository because there is no way to link them to GitHub if users don’t fill in the vcs information on the repository (scmInfo does not exist in ivy style artifacts). Users that wish to show their plugins have to manually claim them.
Yes, I feel this too. It would be interesting to create a FAQ with a gist of the solution and links to the original answers. There are valuable answers in Stack Overflow.
I proposed several times to format the sources, and it eventually crystallised in this pull request that was closed by lack of agreement. I hope we can resurrect it some time soon and agree on a solution.
Macro implementations are pretty well documented, so I guess you’re referring to the use of macros. My problem with them is that their contract is not publicly known to users, and the documentation does not do a good job to thoroughly describe their use. Also, macro constructs can often be used for different use cases that are conceptually different, which is confusing.
I echo your suggestion to keep the Scala API alive for users that don’t want to write in a macro style. There are several disadvantages by not using the sbt DSL (for example, you lose task deduplication). But, in my opinion, this is not a problem if users value more Scala semantics over sbt properties and they don’t want to spend time learning another API.
In my opinion, the regular macros-free API should be documented in the website (it’s not documented right now). I feel that the direction that sbt 1.0 is taking goes aggressively towards a DSL style and while I think this is a reasonable approach for them, I believe that the two styles should be supported by sbt, because not all users share their vision.
To me, writing non-macro code has three big advantages:
You don’t need to learn the sbt semantics and DSL API (which is big). Regular Scala code suffices.
You have total control over task execution.
Compilation is faster – sbt macro expansion is expensive.
However, @tpolecat, in order to make that move you need to get in touch with @eed3si9n and @dwijnand. I have already expressed these opinions to them, so I cannot do anything else than voicing them again in public. I suggest that you open a ticket in the sbt issue tracker to discuss it.
Just to clarify: scaladex indexes both sbt plugins published on Maven Central and Bintray. However, for sbt plugins published on Bintray to show up in scaladex you additionally have to manually claim them (it’s just a matter of adding a line like so).
I’d like SBT to reduce the amount of magic going on. To me the DSL looks often opaque and surprising. Here’s a few things I’d probably prefer to go away (this was originally posted to sbt-dev but it received very little feedback)
an empty or missing build.sbt is a valid project (unless you use sbt-extras)
“bare” keys are not scoped per-build, but per-project, even when there is no explicit project definition
when an explicit root project is missing, sub-projects are auto-aggregated. As you introduce a root project, you’ll lose auto-aggregation
the project macro fills in the name of the project by “reading” it from the val identifier in lazy val foo = project. This is surprising because it does not behave like regular code. I’d even argue that just using Project might be clearer
other random personal observations:
the “recursive” project definition is “elegant”, but (to me) hard to explain; moreover, is there really any benefit in being able to recur with arbitrary depth ? wouldn’t just 1 level of meta build suffice ? thus, is it really necessary for it to behave like a distinct build, or would it be better to special case? (cf.: coursier sbt launcher)
scoping is still unclear (to me), because I’d expect subprojects to inherit keys from parents, while we are expected to use inThisBuild or commonSettings
I don’t see clearly the distinction between Global and ThisBuild, which I would expect to behave the same (within the same build.sbt file)
I would prefer SBT to give up its DSL and move to plain Scala code. The
only reason I don’t abandon SBT DSL in my projects in favor of a plain
Scala build file is that most docs and examples are written for the DSL,
not plain Scala code, and I still find it difficult to translate.
Also, the SBT docs read something like this: “It looks like simply
assigning values to keys, but don’t be fooled, it is more complicated than
that! That’s why you need to learn about concepts first to understand the
difference!”
And then I read through the concepts and at the end, I still don’t
understand how it is effectively different from simply assigning values to
keys.
One possible solution might be to detect scalac messages prompting users to add a compiler flag and to insert instructions on how to do that from SBT. JIT instructions. When the user is already frustrated because they can’t get their code to compile the last thing we want is additional frustration because they can’t figure out how to act on the compiler feedback.
I’ve been thinking about this thread over the past day and I keep coming back to the same question. Are we as a community sure that this goal is aligned with the Scala Center’s goal of “Substantially grow our community of open source contributors.”? As an open-source contributor who would like to contribute even more SBT has consistently been one of my biggest impediments to contributing further. This feel like asking people what they want and hearing a faster horse. Sure, if you pool a group of scala developers who use SBT and ask, do you need help using SBT to do basic tasks you will hear 80% of them say “Yes, please help me” but I would love for the Scala-Center to take a longer term view and ask if 80% of experienced scala developers cannot accomplish their goals with a tool, maybe we need more than improving it’s compilation time?
If your problem is
Sub-par performance.
Learning and using sbt is difficult.
Lack of community involvement.
And it consistently resolves around SBT, then maybe more SBT is not the answer. Or at least not in it’s current incarnation. I honestly believe learning how the sausage is made is not going to make more people want to contribute to SBT, or to Scala in general, so I am not trolling here, I just don’t share the belief that educating people on what I see as a failed design experiment in Scala is the best thing we can do to encourage adoption and I would prefer to see a few people take a step back, pretend they had never heard of existing solutions, ask users what they would like to accomplish, and see if they still believe educating them on the design philosophy behind sbt is the correct solution.
I also am not a fan of the idea of the Scala-Center teaching design practices that we as a community no longer support. Do we really want newcomers to learn about using macros and DSLs that don’t behave like regular scala as soon as they pick up Scala. It’s almost polar opposite of teaching Scala - The Simple Parts. I would much rather have Scala-Center teach beginners http://underscore.io/blog/posts/2015/06/25/keeping-scala-simple.html than how macros work.
With regard to Zinc, this all sounds reasonable. The only thing that isn’t explicitly spelled out here is support for fine-grained builds. Folks using Zinc via Pants (and Bazel, presumably) are encouraged to create lots of well-defined, small modules, and to use “strict” (ie, only directly declared) dependencies.
This allows for better cache hit rates against distributed build caches, but requires that Zinc support low overhead per invoke to avoid the actual compilation of ~2000 source modules being dwarfed by the startup overhead.
Finally, achieving incremental compile within jars would be a massive boost as well: ie, compiling directly to output jars, and then supporting incremental compile that consumes the slightly modified jars. The JVM hates classpaths containing loose classes in directories (anecdotally, jarring all inputs improved cold-build performance by 4x in one case due to drastically fewer syscalls and less IO), and this definitely affects Zinc when lots of modules are in play.