Hello everyone,
First of all, I would like to thank you all for your feedback on sbt. The result of this initiative has exceeded my expectations and I am happy to see that so many of you have gone out of your way to reply to the call of this thread. I appreciate that – we’re all busy these days.
Your insights have helped me to understand how sbt is perceived and understood, and which are the flaws that most affect you. This information has been invaluable during the past month to prioritize my work on sbt. We have changed our plan according to your feedback with the spirit of optimizing for impact.
As a result of this effort, I created the following mind map:
Note: if your browser does not render the PNG file correctly or you cannot read it well, you can download the PDF version here.
The mind map sums up this thread and splits your concerns in several major topics. These major topics represent fields on which sbt could do better. We will dive into them later.
Most of you have expressed concerns about a large array of different sbt problems. I guess this variety depends on the different uses of the build tool and the professional environments in which it is leveraged. However, there have been several recurring themes across all your comments:
- Documentation has to improve.
- Sbt should be easier to use.
- Sbt 1.0 on Scala 2.12 should be out as soon as possible.
These problems have led my contributions to sbt in the past month. However I have also focused on some of the other remaining topics, contributing to topics like incremental compilation, dependency management, new features and performance.
Contributions
These are the contributions that the Scala Center has done for the next sbt milestone (1.0.0-M6).
- Contributing experience:
- Suggested a reformat of the sources with scalafmt both in Gitter and GitHub. PRs: #1 and 2. Eugene has done the same for sbt/sbt.
- Addition of new code documentation around major sbt components: sbt parser and compiler, library management, major tasks, incremental compiler, et cetera.
- Reduction of CI build times by 50%.
- Improve Intellij integration for sbt 1.0.x.
- A new redesign of the scripted test framework that has batch mode execution. Scripted now reuses the same sbt instance to run sbt tests. This is helpful both for sbt contributors and plugin authors.
- I have been active in both the issue tracker and the Gitter channel(s) to attract more contributions and answer questions. For example, #1 and #2.
- Usability and Learning experience:
- Report on unused settings and tasks. Indispensable to detect builds and debug scope delegation – makes it easier to reason about scoping.
- Addition of sbt DSL macro checks that block dangerous uses of the DSL:
- Proposal to improve documentation, which include:
- Proposal to improve the use of scopes.
- Performance:
- Speed up startup and reload times by ~15%.
- Discovery of hot spots that can be optimized for faster startup in the future (there are quite a few).
- New features:
-
Work in progress 90%: Addition of the dependency lock file. The dependency lock file has three benefits:
- Brings reproducibility to the dependency management of your build: transitive dependencies are stored in a file named
dependency.json. - Brings performance improvements: several modes in ivy are enabled to increase performance of resolution. It allows both sbt and ivy to assume certain properties about resolution.
- Allows Coursier or other dependency managers to resolve and download dependencies independently of sbt.
- NOTE: This feature will be available in a binary-compatible experimental sbt 1.0 version in the next days. As it’s a feature difficult to get right that required big changes in the sbt-ivy integration, I preferred to wait a little bit more and battle-test it with common builds.
- Brings reproducibility to the dependency management of your build: transitive dependencies are stored in a file named
- Current misbehaviours:
- Discovery of certain sbt bugs in 1.0 in #1, #2, #3 and #4.
- Handle cross-versioned Scala modules exclusion, an issue @dossett and many others have complained about in this thread and outside of it.
- Dependency management:
- Add parallel downloads. This feature makes the CI test execution 40% faster.
- Add a true offline mode. When the offline mode is enabled, ivy will only check the cache and not try to hit the internet unlike the previous offline mode implementation.
- Allow users to map a concrete resolver to a dependency. This feature will be used for the dependency lock file and can be used independently by the user. It brings substantial improvements to builds with several resolvers and ensures that a given jar only comes from a certain repository.
- Allow users specify artifacts with concrete checksums for every module ID.
- Add frozen resolution to ivy. Informal benchmarks report 2x speedup. This mode is used when all transitive dependencies are specified.
- Clean up the old API and document new implementation in #1 and #2.
- Coursier integration: I have looked into ways of adding a coursier backend to our library management API. I did not act on this well-aclaimed issue because I estimate it to last more than two weeks.
- Zinc, the incremental compiler:
- Fixed an undercompilation issue for type projections.
- Created a Java-friendly Zinc 1.0 API. This has been a major overhaul of the Zinc internal API, crucial for a good Scala integration with build tools like Pants, Gradle, CBT, Maven.
- Registered features proposed by @stuhood in the issue tracker. They will be worked on before the sbt release candidate. Those are:
- Add support for fine-grained builds, reducing startup time for Zinc.
- Compile incrementally within jars, which will supposedly bring juicy overall speedups.
- Migration from 0.13.x to 1.0.0x:
- Created an sbt migration tool to help migrate plugins and builds. This tool has been created to ease migration between 0.13.x and 1.0.0x.
Almost all of this work has been done in the last month and is available now in the sbt 1.0.0-M6 release, published to Maven Central.
The Lightbend tooling team will continue cross-publishing core plugins for it, and if you’re a plugin author you’re encouraged to do so as well! To the best of our knowledge, projects like Coursier and Scalafmt already cross-publish to both sbt 1.0 and 0.13. Use our migrator and see how to migrate to 1.0.x here and how to cross-publish here.
A last, positive remark
I was not familiar with sbt internals before all these contributions. I used to think sbt was too slow of a build tool and excessively complex. I have had some kind of love-hate relationship with sbt during my past year, having developed several sbt plugins and making an intensive use of it. Some of you have expressed similar concerns in this thread and outside of it.
However, this opportunity has given me a lot of insight about how sbt works. It has debunked lots of thoughts and opinions I had on sbt. This new understanding of the build tool has helped me to analyze sbt more objectively.
If I have learned something during the previous month, it’s that most of sbt complexity and slowness is accidental. There is room to address almost all of the complaints you have had in this thread: usability issues, difficulty to reason about scoping, unexpected behaviours. I do not believe there is something inherent to the sbt model that makes it impossible for people to learn it. The lack of hands-on documentation and tools to reason about sbt have not helped in this regard, but this could change, and the Scala Center has done a first step towards this direction.
With the help of the community, we can polish the rough edges of sbt. Help us! You can participate in the development of sbt 1.0.x by filing issues, helping in the Gitter channel and opening PRs. If speed bothers you, file an issue pinpointing what you think could be improved. If you think the docs for a concrete issue are not good enough, head over the issue tracker and let us know. The more actionable something is, the more likely is to happen.
I am optimistic about the future of sbt and hope to see it grow in the next months.
See you in the issue tracker
All in all, I am really enthusiastic about the outcome of this thread! The Lightbend team and the Scala Center agreed to repeat this experiment for the next sbt 1.0 releases. We plan to create a Scala Contributors thread to get your feedback on every release, and we hope that you help us steer the boat towards a direction we all feel comfortable with.
I would like to thank the Lightbend tooling team (@eed3si9n and @dwijnand) for all their help. I have been working with them elbow to elbow to ease migration to sbt 1.0 and advance its development for an earlier release. As announced by Eugene, sbt 1.0.0-M6 is already out and the next release will be the release candidate, where sbt API will freeze and will focus on stability. This new release is scheduled to happen in the beginning of July.
Have a good day,