Asking for your feedback on sbt + Scala Center announcement

For me the most urgent issue is the state of the source code, which makes it very hard for people to contribute. The formatting and organization are absolutely batshit and there are very few comments, which leads to instant demoralization and table-flipping, for me at least. So I request some attention to #2944.

Related, I would like to see documentation on every macro and every use of reflection, and how to write a build that doesn’t rely on these. The things that make sbt “easy” also make it unlike normal scala and thus very hard to understand.

6 Likes

Yes, this is a problem, and a big one. IMO, docs are great but they are not teaching sbt, but showing how to do stuff with it (which is conceptually different). It seems that there’s not a reasonable getting started tutorial. I think this was the idea with sbt in Action, but it’s a paid book which is less than ideal for the OSS community around sbt.

This will come with sbt 1.0 soon. We will help you get there by easing the migration between 0.13 and 1.0 with sbt-migration-rewrites.

Oh, this is a good one. It’s true that plugin documentation is poor and could be improved, both for writing plugins and finding them. I feel we sometimes reinvent the wheel many times when there’s a plugin that solves our problem, but we haven’t found it.

Regarding Scaladex, @MasseGuillaume and @julienrf have recently worked on that use case. For now, Scaladex can show sbt plugins published to Maven Central, but cannot support sbt plugins in the sbt Bintray repository because there is no way to link them to GitHub if users don’t fill in the vcs information on the repository (scmInfo does not exist in ivy style artifacts). Users that wish to show their plugins have to manually claim them.

Yes, I feel this too. It would be interesting to create a FAQ with a gist of the solution and links to the original answers. There are valuable answers in Stack Overflow.

I proposed several times to format the sources, and it eventually crystallised in this pull request that was closed by lack of agreement. I hope we can resurrect it some time soon and agree on a solution.

Macro implementations are pretty well documented, so I guess you’re referring to the use of macros. My problem with them is that their contract is not publicly known to users, and the documentation does not do a good job to thoroughly describe their use. Also, macro constructs can often be used for different use cases that are conceptually different, which is confusing.

I echo your suggestion to keep the Scala API alive for users that don’t want to write in a macro style. There are several disadvantages by not using the sbt DSL (for example, you lose task deduplication). But, in my opinion, this is not a problem if users value more Scala semantics over sbt properties and they don’t want to spend time learning another API.

In my opinion, the regular macros-free API should be documented in the website (it’s not documented right now). I feel that the direction that sbt 1.0 is taking goes aggressively towards a DSL style and while I think this is a reasonable approach for them, I believe that the two styles should be supported by sbt, because not all users share their vision.

To me, writing non-macro code has three big advantages:

  • You don’t need to learn the sbt semantics and DSL API (which is big). Regular Scala code suffices.
  • You have total control over task execution.
  • Compilation is faster – sbt macro expansion is expensive.

However, @tpolecat, in order to make that move you need to get in touch with @eed3si9n and @dwijnand. I have already expressed these opinions to them, so I cannot do anything else than voicing them again in public. I suggest that you open a ticket in the sbt issue tracker to discuss it.

Just to clarify: scaladex indexes both sbt plugins published on Maven Central and Bintray. However, for sbt plugins published on Bintray to show up in scaladex you additionally have to manually claim them (it’s just a matter of adding a line like so).

I’d like SBT to reduce the amount of magic going on. To me the DSL looks often opaque and surprising. Here’s a few things I’d probably prefer to go away (this was originally posted to sbt-dev but it received very little feedback)

  • an empty or missing build.sbt is a valid project (unless you use sbt-extras)
  • “bare” keys are not scoped per-build, but per-project, even when there is no explicit project definition
  • when an explicit root project is missing, sub-projects are auto-aggregated. As you introduce a root project, you’ll lose auto-aggregation
  • the project macro fills in the name of the project by “reading” it from the val identifier in lazy val foo = project. This is surprising because it does not behave like regular code. I’d even argue that just using Project might be clearer

other random personal observations:

  • the “recursive” project definition is “elegant”, but (to me) hard to explain; moreover, is there really any benefit in being able to recur with arbitrary depth ? wouldn’t just 1 level of meta build suffice ? thus, is it really necessary for it to behave like a distinct build, or would it be better to special case? (cf.: coursier sbt launcher)
  • scoping is still unclear (to me), because I’d expect subprojects to inherit keys from parents, while we are expected to use inThisBuild or commonSettings
  • I don’t see clearly the distinction between Global and ThisBuild, which I would expect to behave the same (within the same build.sbt file)
3 Likes

Hello,

I would prefer SBT to give up its DSL and move to plain Scala code. The
only reason I don’t abandon SBT DSL in my projects in favor of a plain
Scala build file is that most docs and examples are written for the DSL,
not plain Scala code, and I still find it difficult to translate.

Also, the SBT docs read something like this: “It looks like simply
assigning values to keys, but don’t be fooled, it is more complicated than
that! That’s why you need to learn about concepts first to understand the
difference!”

And then I read through the concepts and at the end, I still don’t
understand how it is effectively different from simply assigning values to
keys.

 Best, Oliver
2 Likes

3 posts were split to a new topic: Discussion on CBT: Chris’s build tool

Discussion about how CBT does what curoli asks for here: Discussion on CBT: Chris's build tool

Let’s move CBT discussions there, so we don’t highjack this thread.

One possible solution might be to detect scalac messages prompting users to add a compiler flag and to insert instructions on how to do that from SBT. JIT instructions. When the user is already frustrated because they can’t get their code to compile the last thing we want is additional frustration because they can’t figure out how to act on the compiler feedback.

1 Like

I’ve been thinking about this thread over the past day and I keep coming back to the same question. Are we as a community sure that this goal is aligned with the Scala Center’s goal of “Substantially grow our community of open source contributors.”? As an open-source contributor who would like to contribute even more SBT has consistently been one of my biggest impediments to contributing further. This feel like asking people what they want and hearing a faster horse. Sure, if you pool a group of scala developers who use SBT and ask, do you need help using SBT to do basic tasks you will hear 80% of them say “Yes, please help me” but I would love for the Scala-Center to take a longer term view and ask if 80% of experienced scala developers cannot accomplish their goals with a tool, maybe we need more than improving it’s compilation time?

If your problem is
Sub-par performance.
Learning and using sbt is difficult.
Lack of community involvement.

And it consistently resolves around SBT, then maybe more SBT is not the answer. Or at least not in it’s current incarnation. I honestly believe learning how the sausage is made is not going to make more people want to contribute to SBT, or to Scala in general, so I am not trolling here, I just don’t share the belief that educating people on what I see as a failed design experiment in Scala is the best thing we can do to encourage adoption and I would prefer to see a few people take a step back, pretend they had never heard of existing solutions, ask users what they would like to accomplish, and see if they still believe educating them on the design philosophy behind sbt is the correct solution.

I also am not a fan of the idea of the Scala-Center teaching design practices that we as a community no longer support. Do we really want newcomers to learn about using macros and DSLs that don’t behave like regular scala as soon as they pick up Scala. It’s almost polar opposite of teaching Scala - The Simple Parts. I would much rather have Scala-Center teach beginners http://underscore.io/blog/posts/2015/06/25/keeping-scala-simple.html than how macros work.

1 Like

With regard to Zinc, this all sounds reasonable. The only thing that isn’t explicitly spelled out here is support for fine-grained builds. Folks using Zinc via Pants (and Bazel, presumably) are encouraged to create lots of well-defined, small modules, and to use “strict” (ie, only directly declared) dependencies.

This allows for better cache hit rates against distributed build caches, but requires that Zinc support low overhead per invoke to avoid the actual compilation of ~2000 source modules being dwarfed by the startup overhead.

Finally, achieving incremental compile within jars would be a massive boost as well: ie, compiling directly to output jars, and then supporting incremental compile that consumes the slightly modified jars. The JVM hates classpaths containing loose classes in directories (anecdotally, jarring all inputs improved cold-build performance by 4x in one case due to drastically fewer syscalls and less IO), and this definitely affects Zinc when lots of modules are in play.

3 Likes

I wrote up sbt 1.0 roadmap and beta-1 post. Please check it out.

Also we pushed out sbt 1.0.0-M5 today, which is the first beta released.

We’d be happy to get your feedback on the sbt 1.0 roadmap on this thread.

1 Like

I second that. CBT equally encourages lots of well-defined small modules and like everyone else is equally affected by zincs overhead.

1 Like

I thought that ship sailed long ago? .value is a macro, alternatives are deprecated in 0.13 and dropped in 1.0. Same for using .scala over build.sbt. Or do I miss something?

Would you be fine with creating a new JAR replacing the old one? I’m not sure there’s another way to “slightly modify a JAR”—at least not when using JarOutputStream. I guess input libraries are loaded through ClassLoaders, so this change would presumably require more classloader invalidation… though it might well work.

Either way: such an idea, if it makes sense, would probably require its own issue in the SBT tracker.

Yea, it would be a new jar rather than a mutation of an old jar. I meant “slightly modified” in the sense that rather than choosing to invalidate everything that depends on a jar when it changes, you would invalidate based on the fingerprints of classes inside the jar.

Great news.

I would like to see https://github.com/sbt/sbt/issues/2310 move forward. I assume @sjrd has some input for this as well.

As a community we should have a standard sbt build setup, which people can use to start their projects. This should IMHO contain the following:

  • multiple projects
  • cross version build
  • documentation how to use and comments in the build.sbt
1 Like

Another feature I miss with sbt (or maybe just haven’t found yet) is being able to share a bill of materials (BOM for short). This is a common pattern in the maven community (see https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html).
The idea is that projects provide a configuration (pom.xml), which explicitly manages the versions which a certain library depends on. This bill of materials can then be used by the project which uses the library.

For example my project ScalaWebTest depends on ScalaTest, Selenium and HtmlUnit. Chances are the project using ScalaWebTest already has a transitive dependency on HtmlUnit. In this case it is important for me, that I have an easy way to tell my users, which dependency versions should be managed in his project. For maven users, I can do this by providing a bill of materials. See http://www.scalawebtest.org/download.html#maven

I suggest to make use of the Dreyfus model when improving the SBT documentation. The Dreyfus model shows how people have different needs depending on their level of a mastering a skill. “Pragmatic thinking and learing” by Andy Hunt (https://pragprog.com/book/ahptl/pragmatic-thinking-and-learning - see chapter 2 Journey from Novice to Expert) puts this in to perspective of teaching IT skills.

(for those wondering about the multiple posts - max. 2 links for new user :frowning:)

3 Likes

oh, the other huge thing (sbt 1.0+ or a 0.14 on scala 2.12) would be to use NIO everywhere. So many performance benefits if used correctly, especially on Windows and no more FS polling.

3 Likes

Hello @jvican .

For my Scala Projects i currently use Gradle Wrapper with the Scala Plugin.

Now i decide to use sbt.

Some starting points I missed from my point of view:

  1. sbt installation and first start / load of sbt with a Company Artifactory: Here I would like to define the Company Artifactory a) for first sbt start and b) for Project Dependencies (this is already documented : How can I do that ?

  2. I missed a minimal valid .gitignore Definition for a sbt Project (single and multi build). I been confused about the Details in the ‘project’ Subdirectory and it’s content.

  3. Compare the Gradle Wrapper with sbt: Does work the local installed sbt ‘same prinzip’ as the Gradle Wrapper?

As a sbt Beginner
Röbi

1 Like

Resolution of this issue would be most welcome: