Improving the compilation error reporting of sbt

Tooling
21

The best argument to support this point of view is that merging the new error message format into Scalac won’t put it in everyone’s hands – only the users of that specific version, which to this day will be 2.12.x where x > 3. Note that Scala 2.11.x and 2.10.x are discontinued.

However, if we merge it into sbt 1.0, this change will become available to all Scala developers, including those stuck in 2.10.x and 2.11.x. To the best of my knowledge, the proportion of developers and companies stuck in these versions is a significant (if not majority) part of our ecosystem.

4 Likes
22

I also think that we should invest in improving the error messages in the compiler, rather than working around their shortcomings. I can understand the argument about the short-term benefit. But we should not forget about the longer term, and the necessary future investments to keep the plugin working well (the compiler’s error messages and formatting might change in the future).

Is there a good source of data about Scala versions being used? I also think that a majority of projects is currently running on Scala 2.11, but 2.10 is probably less active (except for sbt). For projects on 2.12 it doesn’t matter if an improvement comes in a minor release (2.12.4), using the latest minor release is trivial.

How likely is it that the sbt plugin is going to land in sbt 1.0? It’s not in 1.0.0-RC2, as far as I can tell from the release notes. If it remains a plugin, improving error messages in 2.12.x might have a better reach than the sbt plugin (searching for sbt-errors-summary on github doesn’t give a large number of results, though people might have it in their ~/.sbt). But even if it gets included in sbt 1.0, people might move to Scala 2.12 earlier than to sbt 1.0.

One important aspect where the sbt plugin doesn’t help is the presentation compiler. I think developers see a lot of compiler error messages as they type without actually building the project.

Nowadays we have an sbt plugin that improves error messages, we have a compiler plugin (https://github.com/tek/splain), ensime and the Scala IDE probably have to do quite a bit of work to parse messages from the compiler, and there’s all the work that’s been done in dotty. To me this shows that there’s a lot of interest and workforce available to improve error messages, and the Scala Center is investing in it. We should use this opportunity to work on the source of the issue, collect requirements, design a solution and make the compiler’s error messages more clear, parseable, reusable, maybe extensible.

2 Likes
23

I agree with Lukas.

To expand/reinforce:

  • Putting something in an sbt plugin does technically make it available to the entire community, but it doesn’t mean the whole community will actually get the benefit, since sbt is by no means universally used. (It just feels that way if you mainly hang out with open-source people.) So I don’t agree that the plugin plan puts the improvements “in everyone’s hands”.

  • I’m also concerned about splitting the ecosystem — on the same code, some people will see one thing, other people will see something else. I think it’s better to have a single error reporting format and have everyone on the same page.

I think the best counterargument is the 2.11 argument (2.10 is very nearly dead). But I think we should be playing the long game w/r/t fundamental improvements like this.

2 Likes
24

A small clarification: I’m not saying sbt is the only build tool for Scala, but that the proportion of people using 2.12, 2.11 and 2.10 in sbt is bigger than the proportion of people in 2.12 using any Scala-compatible build tool.

In my opinion, I think it’s fair that people using different build tools see different outputs. Though I admit that Lukas has a point with the presentation compiler, that will not benefit from these improvements.

2 Likes
25

How can we move things forward? If Martin is investing time, we should make sure it’s going towards the right solution. So we should define what that is.

1 Like
26

I still disagree with the idea that this should be put in scalac.

sbt, for >80% of Scala users, is the UI to scalac. No one really uses scalac directly. Therefore, UI work should be done on the UI, and that UI is sbt. @Duhemm has assured that both new and old error reporting would continue to exist in sbt (configurable) so people can leave their tools/scripts/etc that may already be parsing sbt’s error messages as-is, to eliminate ecosystem/tooling breakage from potentially occurring due to an error reporting facelift.

Additional reasons include the following; sbt 1.0 is bringing huge changes to how sbt is used. It’s a good time to also improve the output of sbt. Changing the output of sbt in 1.1 or some later version would just be surprising. We’re making big changes now; let’s not surprise people with a different UI experience down the road. Since we’re changing lots of user-facing things about sbt now (the DSL, the structure of builds, etc) we should also touch something user-facing like sbt output now. It’s only consistent.

Another huge and completely discounted reason we should fasttrack this and get it into sbt is education. The people reading this mailing list/forum are typically grizzled, seasoned, and professional “back-end” people. Most people have been hacking on complicated “back-end stuff” for years. But if I want to teach my 16 year old sister how to program in Scala, her experience will be unmeasurably different with colored/better spaced/easier-to-read output from sbt. Right now, to her, the output of sbt looks like the matrix:

teh-matrix

I see the following happening right now; “I see someone has invested time in something that a lot of people seem to care about (significantly improved and user-visible error reporting), so let’s crowdsource all of the things we would like to see in scalac and request that that’s done rather than optimizing the sbt experience.”

The short answer here is that we don’t have the budget to do this.

Meanwhile, I note that 89% of people we asked want this to go into sbt. This shouldn’t be discounted.

Martin has already invested time. This isn’t an unbounded investment from the Scala Center; Martin has loads of other responsibilities that relate directly to Scala Center initiatives. The goal here was to get this work he already did into peoples’ hands. Not to define an entirely new project to revamp error-reporting in scalac.

Further, the likelihood that a single person will be able to quell all of Adriaan’s, Lukas’s, Jason’s, Seth’s, …, (insert whoever else I neglected to mention here)‘s wishes for The Perfect Error Message ™ to a sufficient degree of “perfect” to have merged into 2.13 or whatever Scala version lands in 1-1.5 years, is simply too much of an investment for us, for a return too far down the line. We don’t have that kind of money or time. The Scala Center exists to improve things that have been languishing for years, and to get those improvements into peoples’ hands as soon as possible. Satisfying a wishlist added-to by a potentially unbounded number of people for The Perfect Error Message ™ in scalac means (1) lots of time, and (2) no one sees this benefit for a minimum of one year. On the one hand, we don’t have the time to invest in this (speaking of SC budget), and on the other, the SC is set up so that the community sees benefit sooner rather than later. This is low-hanging fruit that a lot of people would benefit from now rather than in 1-1.5 years. Think of all the teenage girls or university bachelor students, or ScalaBridge participants, or regular already-trained full-time working Scala developers, etc, that would have a much improved experience this year rather than next. This what the SC is designed to optimize for.

4 Likes
27

I must disagree with this belief that sbt is the interface to scala for most developers.

In my experience, the vast majority of Scala developers use IntelliJ and would struggle to start or use sbt to do basic tasks. Errors are reported in terms of red squiggly lines and (even when they are wrong) are taken as the golden source. If something somehow managed to pass the red squiggly line test, and the compiler rejects it, then maybe the user may look at the compiler logs. From what I’ve seen, since IntelliJ’s typechecker is overly conservative, this situation is very rare (it’s far more likely that the developer will give up assuming scalac is somehow broken, when in fact it is IntelliJ).

Although it is perhaps true that sbt is used by 80%+ of scala projects, the CI will be the primary user of sbt. It’s only weirdos like us lot at ENSIME who use sbt directly… and my protests against changes to the output format are summarily dismissed as holding back innovation. It breaks our workflow, our workflow is pretty awesome already.

In the sbt thread, our biggest feedback is “release sbt for scala 2.12”, we don’t even want or need any new features… indeed the breaking changes are probably going to mean that adoption is very difficult. Certainly I don’t know where we will find the time in ensime to update our sbt plugins… updating the ecosystem to sbt 1.0 is about the most useful thing the Scala Center could do right now (and I know none of you want to hear that because it is not a Big Bang deliverable, it’s a slog, as most engineering is).

2 Likes
28

Disagree strongly with “it’s only weirdos like us lot at ENSIME who use sbt directly.” This is simply not accurate at all.

3 Likes
29

All I have to lean on is personal experience working at 5 very different Scala development teams of varying size from 2 to 200 people, over a 5 year Scala career, plus feedback in the ENSIME community. I’m sure you have access to a wider sample than me, has there been a survey performed of “do you primarily use IntelliJ or sbt for compiler feedback?” (and baring in mind that most people don’t understand that the red squiggly lines in IntelliJ are not, in fact, coming from scalac).

If indeed, the direct sbt userbase is as strong as you say, does it not highlight the need to prioritise rebooting the ecosystem for 1.0?

3 Likes
30

Intellij itself is switching to using sbt directly to compile scala. If I understand the goals listed there correctly they intend to use sbt’s incremental compilation to provide error highlighting too (and possibly drop their own parser/typer)

  • Output of the sbt build should be parsed or delivered to the IDE via server protocol to give error highlighting within the editor and project views (SCL-11079)

I’m from userland, I use sbt and intellij and double check intellij’s squigglies with sbt/scalac when in doubt but I am not a statistically representative sample :slight_smile:

Also I dream of a world where the compilation errors are improved in SBT right now, then fixed once and for all in scalac but I am aware of what limited resources mean.

edit
a few further references about the use of SBT within intellij just hot of the press IntelliJ IDEA Scala plugin 2017.2: SBT 1.0, improved SBT Shell, Play 2.6 and better implicits management | The IntelliJ Scala Plugin Blog

4 Likes
31

I would challenge this statement. The vast majority of people use an IDE and see compilation errors in the IDE, not on the sbt console. That’s how it should be, the best possible experience is to see errors in your editor and have navigation to the corresponding source code. If we think about the error messages for the majority of users, we have to think about IDEs. It seems things are working OK there, maybe we could make their lives a little easier with a more structured format, and probably by improving the actual messages.

That said, there is still value in improving the console output.

I understand your concerns of resource management of course. I think you are over-estimating the friction of getting a change into the compiler. The current implementation for error messages in the compiler is in a pretty decent state. The features of the sbt plugin could be added incrementally (colors, shortening paths, grouping / summarizing by file, what did I forget?). These improvements would be welcome no matter if there is time left (and demand) for other refactorings or not.

I have to say that I find it quite likely that a lot of people responded to the improvements in error messages, not to “in sbt” in particular. We would probably have received the same 89% if we asked about “in scalac”.

1 Like
32

On the contrary @jeantil … what you’ve just said is the common misconception that most IntelliJ users (i.e. the vast majority of Scala developers) have.

The red squiggly lines are provided (and always will be provided) by IntelliJ’s re-implementation of the scala compiler, which performs typing and provides answers to interactive queries such as completions and type-at-point. They did this because a) they wanted interactive support before scalac had the Presentation Compiler (which is what Scala IDE and ENSIME use) b) this is also how IntelliJ reports errors in Java so it was a natural design choice for them. It does, ironically, mean that we - as a community - have more paid developers working on re-implementation of the compiler’s typer than the actual compiler. JetBrains are the biggest investor in Scala’s tooling.

The only time you will get actual scalac errors is if you hit “build project” or run a main (this is known as “batch” or “incremental” compilation, not “interactive”)… if IntelliJ is configured to use sbt, the sbt output tends to be hidden in a log file somewhere, although for a long time IntelliJ has been using zinc directly for incremental compilation, not via sbt. When they talk about closer integration with sbt, it is at this layer. And they may put the errors back onto the editor view but these new changes proposed my Martin actually break that support and require design work to support the new format. Nothing will change here for IntelliJ users.

So when the community is asked questions about sbt incremental compilation and compiler output, I feel that most of them are not fully appreciating the nuances about which tool is doing what thing and are not best positioned to answer questions about how to prioritise the work to achieve what they want. To echo @lrytz’s point: the specific question that Martin asked on twitter has been largely (mis)interpreted as “would you like better scala error messages?”

33

In 2017 (pre IntelliJ switching to using sbt), people use IntelliJ concurrent with sbt. It doesn’t take much more than a toy project to wind up with bogus squigglies. Every student I’ve ever mentored has very quickly learned to use sbt concurrent with IntelliJ for this reason.

To someone out there, sure. But again, almost no one directly uses scalac. I stand by my assertion that sbt is how a vast majority of people interact with scalac. Further, improving scalac’s output isn’t what we’re proposing to do. To repeat:

34

Perhaps we (@jeantil and I) are both misreading SCL-11079, but I also read it as saying that, at the very least, IntelliJ’s highlighting will soon be supplemented/corrected by sbt output (though not instantaneously). I don’t think either of us were saying that that is currently the case.

2 Likes
35

I don’t think I am necessarily representative of most Scala users either; however, I too use sbt to double-check errors reported by IntelliJ.

Though I’m sure it happens, I have never encountered a situation where IntelliJ incorrectly reported an error with Java code. However, I’ve seen it sufficiently many times with Scala that if a piece of code looks like it ought to compile, but IntelliJ marks it as wrong, I will check with sbt to be sure that it is actually invalid.

2 Likes
36

Deleted - someone already said was I was saying here.

37

When I started coding in Java, I learned fairly early on to use Maven for dependency management, and it really isn’t very difficult to run mvn clean install to build and package your project. It is no more difficult to type sbt clean compile or sbt clean packagedArtifacts into a terminal. If programmers’ projects are set up to use sbt, they can probably compile them in a terminal.

38

What is the reason you’re not using cmd+f9 / ctl+f9 (Build Project)? This runs the scala compiler and presents the real compilation errors within IntelliJ, double-clicking will take you to the source location.

1 Like
39

I will echo that understanding. The ticket specifically makes mention of “Output of the sbt build should be parsed or delivered to the IDE via server protocol to give error highlighting within the editor and project views” - and that sounds like changing how the highlighting is being delivered.

40

@lrytz most developers don’t know the difference between the IntelliJ Presentation Compiler and scalac. @NthPortal is already one step ahead of the curve.

And even the people who do know the difference often don’t know the difference between the IntelliJ Presentation Compiler and scalac’s Presentation Compiler. I know people who have exploited this lack of knowledge to our advantage, who’s managers thought they were instructing your team to fix something for all their IntelliJ users but actually it was just helping out the ENSIME and Scala IDE users in the team :speak_no_evil: (insert evil laughter gif)