So… ever since we moved to Scala 3 the coverage on all of our projects is broken, basically, anything within a for comprehension (but in particular zio and other effect systems) won’t mark as done. We went from a 87% coverage to a 35%, and every month or so management asks me where am I on the coverage goals for the quarter and I keep on saying that it’s scala 3’s fault.
There’s a ticket on scoverage, but there’s also one on scala, since my esteemed collegue (@Vijay Ramesh)…
This is holding back moving a lot of other projects to Scala 3. Mostly I don’t understand why nobody seems to see it as a problem, and it totally meshes with the narrative that only new features are worked on. I love, no, I ADORE scala, and hate having to constantly defend it!
I actually don’t want to be that snarky guy, but if fixing the metrics is so important to your company did your company consider investing into that by helping to fix the issues?
Scala is an OpenSource project.
It’s imho always wrong to just show up to complain that nobody is doing the work, btw. completely for free!
I’m sorry if I sounded harsh or rude on my email, I really didn’t mean it that way.
Yes, we’ve tried to do something ourselves about it, as someone else mentioned, Vijay has a PR out, but from my reading of it it doesn’t really fix the issue. Personally, I briefly looked at it and I’m afraid that the issues involved are a bit above my expertise; not necessarily above my capabilities, but it would take me significant time to get up-to-date, believe me, if I had a fix I’d have already pushed a PR. On a different track, I’m trying to figure out if we can offer a bounty for the issue to be fixed.
On the other hand, my question was not meant to be glib, I am genuinely interested in why the lack of accurate coverage results seems to be such a low priority for the community (and I say “it seems”, I’m not privy to any huge discussion that may be happening in Mastodon or otherwise).
I believe it might be that coverage currently falls through the cracks in terms of responsibilities. It’s not part of the core compiler team and not part of IDE tooling either. Maybe it needs to find an organization that takes it under their wing.
Having been on both sides of the conversations I can sympathize with both points of view. I’ve worked in places where code coverage was important enough to dedicate 4-5 person months to pull it through, but not every organization can afford that (this was on Scala 2 with Bazel, so not exactly relevant in this case). My hypothesis for why “nobody cares” is that “all those who care are still on Scala 2”.
We used to have such an entity for code coverage, which is the scoverage project. I haven’t followed closely, so I’m wondering why is code coverage now part of the Scala 3 compiler? I would assume that being part of the Scala compiler should guarantee some level of commitment, and perhaps a higher standard of quality, than an under-funded open-source project. If that’s not the case, it would have been clearer to leave it as a community project.
I don’t know. I have not been involved with coverage at all. It seems to me that it could very well be a third-party compiler plugin instead of a proper phase, but like I said I am not the expert here.
Because compiler plugins need to be republished for every compiler patch release, they tend to block people from upgrading which is especially problematic with the more rapid release schedule of Scala 3. Having it be part of the compiler codebase avoids this problem and means that we continuously test the coverage support against the latest compiler. I don’t think that having it be in a separate repository would help with maintenance given the low commit activity of the scala 2 coverage plugin: Commits · scoverage/scalac-scoverage-plugin · GitHub, we just have more bugs currently because the scala 3 coverage implementation hasn’t had time to mature yet. On the other hand, we have more precise coverage information than Scala 2 in a bunch of situation, and our implementation is also faster, see the report from the initial implementation: http://guillaume.martres.me/code_coverage.pdf
I apologize ahead of time for my harshness here and in any other conversations but please let’s get real.
Code-coverage is a company’s first and only reliable line-of-defense against incidental failure that can be easily understood and measured. Without code coverage metrics, upper management is literally flying blind when it comes to their ability to assess the potential downside of critical systems.
As we speak there are important organizations in the financial sector looking at this thread and making decisions about whether or not to adopt Scala 3 and whether to continue even using Scala for new projects at all (I happen to know of two in particular!). If Scala 3 adoption does not happen now, it will probably not happen at all.
The fact that Scala 3 has been around for over a year and code coverage is still broken is an extremely difficult issue to justify. The engineers and architects who have already stuck their necks out staking their careers and reputations on Scala 3’s viability are waiting patiently to see how this story plays out. For the sake of Scala 3’s future, I sincerely hope this thread does not end with “we’re not sure what to do.”
@smarter It certainly does not matter at all how much more accurate coverage might be with the plugin that exists today because almost all flatMap invocations can falsely report as Not Covered. I don’t mean only effects, I mean Lists, Option, Either etc… from what I’ve seen, the issue is extremely generalized.
Maybe the answer is to abandon SCoverage entirely and go back to JACOCO. In that case, the discussion needs to be about how to decrease the noise of synthesized Scala functions because the filtration criteria of the current JACOCO SBT plugin is insufficient to produce a meaningful report.
Again, I sincerely apologize for any harshness but Scala’s future is in no small way dependant on how this thread is resolved.
Many different things are important to many different organizations, and they can help these things happen by either contributing directly or by sponsoring the work of others, but just pressuring open source developers only leads to burning them out.
This attitude is a good illustration of why corporate adoption of Scala (3) is declining. A person comes here with an observation of a very real, serious issue, and is met with hostility over his “tone.”
His tone is really inconsequential and irrelevant regarding the actual problem. (But I’d also say it’s entirely reasonable given that this problem is not new and is definitely “holding back moving a lot of other projects to Scala 3.” It’s affecting things at my Fortune 500 co, for example.)
It’s the objective situation that is important. Related to that, here are some things to be aware of:
Other, mature languages, that want to achieve solid corporate adoption and have achieved it, don’t have issues like this. Their tooling works - with few exceptions, all of it (all major parts), all the time.
If they did have an issue like this, the community would not react to it - or someone reporting it - by attacking the messenger. The non-fractured, large Node, Rust, Go, etc. ecosystems would not allow a problem like this to exist for very long, if ever.
So, from the point of view of a developer who loves Scala 3, wants to use it in his job, and wants it to succeed, the OP is entirely reasonable.
Dr. Odersky’s information on why scoverage falls through the cracks is certainly helpful.
But don’t have the standard of “Anyone who draws attention to a language-ecosystem issue should also be willing to fix the issue” unless you want to see Scala decline even further on all the language indexes.
When speaking about unprioritized features in any open source project it is a difficult conversation. How do you address urgency, while not offending anyone? I do not think anyone raising the urgency is trying to offend anyone. And its fine if you don’t think its important.
I think it is important that observed impediments to Scala 3 adoption are raised, and some folks on this thread directly or indirectly have awareness of companies who are assessing the future of Scala 3 and/or Scala in their organizations, and they have a bar for any supported language to meet, which includes code scanning metrics and code coverage.
Many of these organizations do not have compiler expertise, but believe in open source. I think the perspective that says, “just contribute if you care so much”, looks past the fact that some would contribute if they felt knowledgeable enough.
There are also organizations who do not contribute to the compiler, but contribute elsewhere in the ecosystem.
I’ll admit, code coverage means less to the general open source and academic communities than it would for a business. But for Scala 3 adoption in the enterprise this is an important factor and it would be great to get this work prioritized.
These are some of the few forums we have to raise these concerns, and if folks are attacked, instead of assuming positive intent then it does the ecosystem a dis-service.
In an attempt to be solution oriented, I would agree with @odersky that it makes sense to give tracking of this issue a proper home.
I see the original poster has attempted a fix or linked to a fix, is there anyway the compiler teams can help the community get up to speed with what needs to be known to take ownership of this area of tooling?
I for one have watched the compiler videos and even signed up for a spree (but scheduling conflicts caused me to be unable to participate after all), but don’t feel knowledgeable enough to even know where to start on this, and I applaud the OP and others for taking this up. I felt encouraged to see someone else decided to take up this effort and would love to help as well.
You are being unfair to people who are actually doing the work here. From my perspective, someone came in to raise a problem, I took a look at the linked PR, suggested a way forward, and made a note to myself to follow up on it. I can’t control everyone’s reaction in a public forum, and I can’t immediately fix every issue.
There are ways for companies to influence the work being done on Scala, for example by sponsoring the Scala Center, but I don’t think that threats of abandoning Scala will achieve anything positive.
The compiler sprees are great for this, but as I mentioned earlier in this thread, anyone is welcome to come on Discord to chat about an issue and we can organize impromptu pair-programming sessions.
I’m not being “unfair” to anyone - I am stating facts. Attacking people who point out that something as basic as test coverage verification hasn’t worked for a year will have a negative effect on the success of Scala, and THAT is what people should be concerned about, primarily, certainly more than feelings.
(The childishness, infighting, and politicization of the Scala ecosystem is a related subject that also works against the success of the language.)
That’s about the last of what I have to say on this particular subject.
No one is trying to threaten. It’s a very real problem that Scala is losing to the competition that many of us are very scared about and bothered by.
I get that Scala is seriously under-resourced. But IMO this is more about the attitude that is often projected. And I’m not talking about you. But all the same facts on the ground could be the same, and there could be someone who projects leadership, talks about how and when this will get done, tries to make connections to speed it up, and sets expectations that make people feel supported, not simply be defensive about why it hasn’t been done yet.
Some of this is just about talking about the full half of the cup instead of the empty half.
Also as alluded, it’s possible there are companies that would love to put money into it but don’t have where to put the money to make it happen. Someone who wants Scala to succeed could take the initiative of trying to make that happen.
A given company, left to their own devices, may not prefer that approach, since they can equally stay on scala 2 and/or invest in other languages. But if someone reaches out to them maybe they could be persuaded.
Every time people that represent Scala reply on the forum, they should see it as a PR situation, because unfortunately that’s what it is. If people see it as a ragtag band of stressed out solo developers they will go elsewhere. That doesn’t mean you should take on too much and get burnt out. It’s about the messaging.
(Also, I hope that tooling is being prioritized over cool new improvements, not for myself but so that Scala can be more popular in the US.)
Hello! I am just scala language user. I have nothing with the compiler team or scala organization. But I noticed that the compiler team is unsure about who is responsible for this task. Could you please put someone on this task that is capable of doing this? And fill in the cracks in the process. Something Unassigned to anyone must be assigned to someone that will know what to do next.
I believe that it’s important issues like this to be taken seriously.
@odersky group, you created a beautiful language, and I think it would be great if we could put it to good use for everybody.
And industry adoption is really REQUIRED for success. Now that Scala 3.3.0 is out, what are the priorities? Is there a way for the community to vote on which priorities should be addressed first? This coverage must be voted up! Is there a mechanism to prioritize tasks like this? JetBrains has YouTrack tool, which looks cool. Does scala compiler team / tooling has one? If not make it.Maybe this voices concerns “why nobody cares”, maybe because there is not mechanism to hear the voice of your userbase, except this form. Listening frustrations here is not a pleasant experience. If scala lang org needs funding let us know(blog post?) and we can donate, you will need to make your compiler team bigger, maybe funding can go in this direction and we can yearly donate what we can. If scala is bringing success to the people why not giving something back. I really am grateful to all open source developer on this project. You are doing fine job with the language, but some work with greese needs to be done otherwise your open source contributions will not see much of a future adoption. And I believe that you(scala organisation) can do it somehow.
One engineer from VirtusLab already started working on coverage bugs.
It is unfortunate that this area fell through the cracks in terms of responsibility. I’ll take care that things like this will not happen in the future.