I used to follow traditional programming methods, but that’s no longer an option this year; everyone’s doing all sorts of hacking. After using it for a few days, I still find it quite good. I think it’s also pretty useful for analyzing and solving bugs; after all, people need to provide ideas and direction. Hopefully, this can also help us locate and fix some Scala bugs.
From our experience, Claude Opus is quite good at code understanding. It can even synthesize working solutions that pass all tests. However, and this is the important part, those solutions are usually unsatisfying unless the problem is extremely trivial. They often miss the bigger picture.
At this point, I would not trust Claude, or any other LLM, to work autonomously on sensitive parts of the compiler such as the typer. For the time being, human judgment remains indispensable.
We have also added policies around LLM usage. We are not categorically opposed to using LLMs, but there has to be a human behind the work who understands it and is accountable for it. (I’m also relieved that we have been mostly spared with total slop PRs).
So regarding the original question, my answer would be no: I do not think we are going to see a noticeably accelerated burndown of issues from this.
12 Likes
Yes, AI tools are only supplementary. We must ensure that the person submitting the PR fully understands and accepts the proposed solution and is satisfied with it. We cannot lower the overall quality standards of the project because of AI. Also, I’ve noticed that the community’s development focus seems to have largely shifted to version 3.x.
That is certainly a reality.
I totally agree with everything @etaconversion said above. A bottleneck is going to be trusted poeple that understand the big picture, have enough background to review contributions. The AI will always tell you with confidence that it got it right (“Great, now I have the full picture!”).
Also, not all bugs are equally important. With AI enabling more contributions, there’s a risk that the bottleneck of reviewers gets stuffed. We cannot shift the responsability for contributed code from the person submitting the code to the reviewers / maintainers.
4 Likes
This is, more generally, a near-existential point for most open source projects right now, far as I can tell: a lot of people are tossing AI-generated PRs (often of low quality, often not even desired) over the wall, and the maintainers are spending too much time triaging those instead of doing the actually-important work.