Why does Scala need its own build tool (SBT)?

[eyalroth]

When I first learned Scala 3 years ago, I didn’t think too much about its main build tool – SBT. In fact, my main project throughout this time was a mixed Java-Scala project which started with Maven and later migrated to Gradle (by me). It’s not until recently that I’ve started to really think about this question, and it certainly seems that others have been wondering about it as well; not all that surprising given that there is seemingly no formal addressing to the topic.

Perhaps I am a bit ignorant of the full SBT experience, but I couldn’t easily come up with any major advantages that it has over its Java world alternatives (Maven / Gradle). The incremental compiler (Zinc) surely has no real alternative, but I don’t entirely consider it as an integral part of SBT as a build platform (and I believe this thought is reflected with the separation of Zinc to its own repository). Given no clear advantages, what is the justification for keeping SBT alive and advocating it as the main build tool for Scala?

Scala doesn’t exist on its own - it relies heavily on Java tools and libraries, and a big part of its community consists of developers who either came to Scala from Java, or are involved in Java projects as well (or both). As one of those developers, there is no reality in which I will not be familiar with at least one of the popular build tools in the Java world (Maven / Gradle), and having to learn an entirely new tool just makes my work harder.

This is in fact a criticism I share with other Scala oriented tools which already have alternatives in the Java world, such as ScalaMock trying to replace Mockito. Why implement a whole new library when it is possible to extend over Mockito – a widely used Java tool? In fact, someone has recently started a new Mockito-Scala project in order to do just that. The funny thing is that this new project maintains two build configurations – SBT and Gradle (build tool of the original Mockito project) – and for what?

I would honestly prefer it if the attention of both the Scala community and its main contributors would focus on integrating Scala and its inherently exclusive tools (such as Zinc) with existing build tools. This means adding a full support to both Maven and Gradle, and not just a flaky plugin that doesn’t even integrate with the new Zinc compiler. I find it quite absurd that I have to work on my own example project of Gradle mixed with Java and Scala (still WIP).

This is both an open question to the more experienced members here and (hopefully) the beginning of an honest discussion on the subject.

[curoli]

Would you rather write your build definition in Scala, Groovy, or XML?

[eyalroth]

@curoli

Would you rather write your build definition in Scala, Groovy, or XML?

It really matters very little to me as long as it’s readable, flexible, and isn’t hard to learn, which I believe Gradle’s take on Groovy provides all of these features (and there’s Kotlin DSL for type-safety, but I haven’t tried that). This is why I prefer Gradle over Maven, as Maven is based on XML and thus restricts its users to its DSL.

[jducoeur]

@eyalroth

This is in fact a criticism I share with other Scala oriented tools which already have alternatives in the Java world, such as ScalaMock trying to replace Mockito. Why implement a whole new library when it is possible to extend over Mockito – a widely used Java tool?

This isn’t exactly a good example, IMO. Having just spent some months working with a client that used Mockito a good deal in their code, I (and I believe many of their senior engineers) came to the conclusion that Mockito works disastrously badly for idiomatic Scala code. Its tendency to accidentally inject nulls into Scala code (which tends not to be defensive against that) made the test harness very tricky to maintain.

Which isn’t to say that a simple reimplementation of Mockito in Scala is what I’d recommend. Rather, it suggests to me that Scala wants different test tools, rather than just using the ones from Java. While the languages are compatible, idiomatic Scala is quite different from idiomatic Java, and often calls for somewhat different solutions.

I’m not going to give a full-throated defense of sbt, mind – I think it tends to be annoyingly obscure. (I’ve started to describe it as, “Correct, Comprehensive and Concise, at the expense of Clarity.”) But the reports I’ve heard from people trying to do Scala development with Maven and Gradle have been mixed – I’ve spoken to a fair number who tried it, got annoyed, and decided that sbt is the lesser evil for Scala-centric development.

(What I am seeing genuine enthusiasm for is Mill. Haven’t tried it myself, but I’ve been impressed by the reviews from those who have.)

None of which says that you’re fundamentally wrong – I can sympathize with the desire not to reinvent the wheel, and for mixed-language projects there is a strong case to be made for using less-specialized tools.

But there is a counter-argument: that the primary benefit of open source is the “thousand flowers” effect, and that most evolution and improvement in the industry comes precisely from people reinventing the wheel a little bit better each time.

So I don’t actually mind using a Scala-centric tool for my 100% Scala projects – it’s designed to work well for them, and there is a ton of infrastructure to support weirdnesses like cross-compiling to JS. And I’m actually rather excited about the way that sbt’s weaknesses are leading people to evolve better tools – the competition is lifting the entire ecosystem…

[scottcarey]

Maven supports xml, yaml, groovy, scala, kotllin, clojure, ruby and more. (https://github.com/takari/polyglot-maven).

[eyalroth]

@jducoeur

This isn’t exactly a good example, IMO. Having just spent some months working with a client that used Mockito a good deal in their code, I (and I believe many of their senior engineers) came to the conclusion that Mockito works disastrously badly for idiomatic Scala code. Its tendency to accidentally inject nulls into Scala code (which tends not to be defensive against that) made the test harness very tricky to maintain.

I would argue that Java and Scala tests are very similar and that Mockito fits very well to both of them. I started with ScalaMock but other than a bit of a different syntax and a fair amount of added bugs and missing features, it was pretty much the same. But I believe that’s not the focus of this discussion

@jducoeur

I’m not going to give a full-throated defense of sbt, mind – I think it tends to be annoyingly obscure. (I’ve started to describe it as, “Correct, Comprehensive and Concise, at the expense of Clarity.”) But the reports I’ve heard from people trying to do Scala development with Maven and Gradle have been mixed – I’ve spoken to a fair number who tried it, got annoyed, and decided that sbt is the lesser evil for Scala-centric development.

Exactly, and this stems from the community focus on SBT instead of integrating Scala with existing tools. There is nothing inherently wrong with the existing Java tools as far as I’m aware, and there is no reason why Scala shouldn’t work smoothly with them.

@jducoeur

But there is a counter-argument: that the primary benefit of open source is the “thousand flowers” effect, and that most evolution and improvement in the industry comes precisely from people reinventing the wheel a little bit better each time.

I agree, and this is why I don’t object to having side-projects developed by the community – such as the case with Mill and CBT – but I don’t understand the reasoning behind reinventing widely accepted open-source build tools, especially at the cost of (potentially) other major developments in the Scala world.

[martijnhoekstra]

I’ve very often complained about sbt, everything appears to work by accident, Tasks, and Settings being confusingly similar and confusingly different, and to make matters worse, things are usually called something they aren’t (tasks? keys? settings? task keys?), scoping is confusing, test is too special cased, and key resolution is slow, probably due to keys being copied roughly a million time in a moderately complicated build.

And of course scala doesn’t need its own build tool. There are approximately infinity general purpose build tools that could build scala code.

But sbt exists, and despite that it’s not needed, it’s used very commonly. That is by choice. People are choosing to use sbt. Nobody is putting a gun to anybodies head. Use whatever the hell you want to use.

[eyalroth]

@martijnhoekstra

But sbt exists, and despite that it’s not needed, it’s used very commonly. That is by choice. People are choosing
to use sbt. Nobody is putting a gun to anybodies head. Use whatever the hell you want to use.

But then there may very well be many who choose not to use Scala due to this norm (among other things), and I believe that it’s the Scala contributors’ responsibility to at least address this, if not re-think this entire endeavor and shift the now-mainstream approach from SBT to other tools.

[martijnhoekstra]

They could also just choose to use a different build tool. I have no problem with people choosing whatever toolchain they’re comfortable with.

[scottcarey]

(digression) Mockito and ScalaMock are both anti-patterns IMO.

Your code should be testable without external mocks, or else its API / abstractions aren’t very good.

These tools are OK for mocking third party code that is poorly designed or lacking in useful testing utilities.

(main topic)

SBT is fundamentally flawed IMO – it is ‘inward facing’, defining submodules from a parent, without principled mechanisms to import behavior or base structure off of a template.
Making things DRY across projects is difficult.

With sbt how do you share a common way to generate a doc website across related projects (for example, the cats ecosystem)? Copy paste!
(ok, I guess custom plugins might work?)
I haven’t kept up on recent SBT things, but plugins were the only way to share across independent builds last I looked.

Common things to share across independent builds: Release process, versioning rules, doc site gen, compiler settings, lint rules, banned dependencies, logger backend and settings, ‘ecosystem’ dependency sets: e.g. “If you need hadoop, here are all the versions you have to compile against and all the dependency exclusion rules”, test coverage tool config and minimum thresholds, common docker container build settings, common corporate repo settings, and more. SBT asks me to copy-paste most of those. I can put them in a common place with maven and share them.

Maven has a ton of flaws, but I feel like SBT was designed in a vacuum without any real understanding of what the rest of the world did well.

Unfortunately, nothing is good these days if you have a polyglot jvm shop. Want scala + java + maybe a sprinkle of kotlin or clojure? SBT fails miserably. Gradle? can work with effort and is polyglot, scala is a weakness. Maven? can work but scala pluigin is starting to fail and cross-scala-version is awful.

Honestly, if we replaced all our scala with Kotlin, then Gradle or Maven suddenly work a lot better. Hmmm.

Mill? I don’t know, how is its IDE integration and polyglot support? How is its support for sharing processes and configurations across builds?

[selmank]

I am not completely agree with you. By the way some java developer disagree with you(http://xerial.org/blog/2014/03/24/sbt/)

SBT is a different approach to how a build tool should be. It has both advantages and disadvantages when we compare to other alternatives. It is Odersky and other team members answer to a build tool problem. A different approach, different solution is always a good thing. But that doesn’t mean it is A perfect solution. It can be always better.

At this point, I think what we should think as a community : What are the strengths and weaknesses of SBT. How sbt can be better.

If the arguments is put forward correspondingly to that questions, I think we can think more clearly.

Yours sincerely,

[eyalroth]

@selmank

I am not completely agree with you. By the way some java developer disagree with you(Building Java projects with sbt - Xerial Blog)

SBT is a different approach to how a build tool should be. It has both advantages and disadvantages when we compare to other alternatives. It is Odersky and other team members answer to a build tool problem. A different approach, different solution is always a good thing. But that doesn’t mean it is A perfect solution. It can be always better.

At this point, I think what we should think as a community : What are the strengths and weaknesses of SBT. How sbt can be better.

If the arguments is put forward correspondingly to that questions, I think we can think more clearly.

Yours sincerely,

This blog post is almost 5 years old, which is quite irrelevant nowadays. Besides, it compares Maven with SBT, but there is also Gradle which IMHO overcomes a lot of the difficulties with Maven.

No one is arguing that the Java tools are perfect. The question is given that there are no apparent major advantages for SBT over its alternatives and that it only comprises of a different approach, why should it be developed by Scala’s core team and advocated by its community, when there is a big disadvantage for using it instead of the Java tools - the additional complexity forced upon Java-Scala developers, which makes their work harder and discourages them from adopting Scala (thus keeping its status as a niche language).

[martijnhoekstra]

@eyalroth

why should it be developed by Scala’s core team

I don’t want to sell Dale and Eugene short, but this seems a bit of an overstatement.

[eyalroth]

@martijnhoekstra

@eyalroth

why should it be developed by Scala’s core team

I don’t want to sell Dale and Eugene short, but this seems a bit of an overstatement.

I admit to not know the team behind SBT all that well, but being on the Scala-Center agenda and also being both sponsored and supported by Lightbend, I guess it counts as being a part of the core Scala development.

[odersky]

@selmank

It is Odersky and other team members answer to a build tool problem.

Definitely not my answer. I was always very skeptical of SBT’s approach and remain so.

[AMatveev]

I closely monitor for bloop

I think when bloob has a good intellij idea integration, I will prefer to use gradle or maven because of Occam’s razor

[jvican]

@eyalroth

I would honestly prefer it if the attention of both the Scala community and its main contributors would focus on integrating Scala and its inherently exclusive tools (such as Zinc) with existing build tools. This means adding a full support to both Maven and Gradle, and not just a flaky plugin that doesn’t even integrate with the new Zinc compiler.

Hey there, I’m one of the main maintainers of Zinc and one of the creators of bloop. The rationale behind bloop is just that: create a build server easy to integrate with and that enables any build tool to provide the best Scala developer experience out of the box. The bloop website linked above is rich in details.

We decided to do this after realizing how badly integrated Zinc was in many popular build tools and how it lacked many performance improvements that could be fixed and be provided as a service for every build tool, improving the experience for every Scala developer no matter what build tool they use. For example, Bloop has pioneered an implementation of pipelined compilation that intends to compile your build faster.

So far, bloop is only missing one feature (safe and performant handling of concurrent compilations for the same project; which I have already partly implemented) before I start to gradually improve our build plugins in Maven, gradle and sbt to integrate with bloop via BSP.

The main requirement of these improved integrations is that end users will now know that they are using bloop under the hood – they will only benefit from all the improvements bloop offers (reduced memory consumption, faster compiles, etc) without learning another tool or paying a UX cost.

I believe it’s a matter of days before we can see these build tools using bloop together with IDEs. For the moment, Metals, fury and IntelliJ are already integrating with bloop. I’ll keep you updated @eyalroth if you’re interested in more developments.

A good way to stay up-to-date is to watch the repo or use our scalacenter/bloop gitter channel.

[eed3si9n]

@eyalroth

Perhaps I am a bit ignorant of the full SBT experience, but I couldn’t easily come up with any major advantages that it has over its Java world alternatives (Maven / Gradle).

sbt 0.3.2, the first public release was released December 18, 2008, about ten years ago to workaround idiosyncrasy of Scala and its ecosystem at the time, like slow compilation and lack of IDE.

When Scala 2.9.0 implemented binary compatibility across patch versions, sbt evolved together to implement _2.10 name mangling via %% and cross building support. It’s not uncommon for modern Scala libraries to support not only multiple Scala 2.x versions, but Scala.js cross building as well.

It also set the common interface for test libraries so things like testOnly and testQuick works across ScalaTest, Specs2, and ScalaCheck. In the hindsight we can also question whether Scala needed its own test frameworks too, but sbt provided utility for many users in this regard.

Also there’s plugin ecosystem, which is directly connected to Scala library ecosystem as well. sbt 0.13.x releases held binary compatibility for four years, but there was a demand from the plugin authors for sbt 1.x that uses Scala 2.12.x.

One of the first thing I’ve done was to split Zinc out into its own repository, precisely so other build tools like Maven, Gradle, Bazel and its cousins would have better Scala compilation, and we can evolve Zinc together. But I disagree with the idea that incremental compilation was never the integral part of sbt. Incrementality and parallel processing of tasks is what enable things like ~testQuick across many subprojects. (testQuick reruns test for affected code)

I know people poke at “simple” in sbt, but the build can be simple. Just make a blank build.sbt and run sbt console to get Scala REPL. Throw a few libraryDependencies in and you get something similar to what Maven gives you.

As a sbt maintainer I am a fan of sbt, but as Lightbend Tooling team, I wish for a vibrant Scala ecosystem. For that I have huge respect for folks like Chris Vogt, Li Haoyi and Jon Pretty exploring the build tooling space by putting their ideas to the test; as well as other build tools like Gradle. Where possible, I’d be picking on their ideas to improve sbt since I think there are many things we can improve to get to better existing user’s experience.

Happy hacking.

[Ichoran]

@eed3si9n - You mean 2008, not 2018. Might want to fix that, though I think given the link and the statement it’s pretty clear that it’s a typo.

[eed3si9n]

Yes, thanks. I fixed it.