Move slowly and iterate

Language Design
15

I disagree with this entirely. I think it’s important to understand that Scala 2 is used in real-world applications today, and Scala 3 is not. Employers are not maintaining Dotty code bases today because the code base is experimental. To say that someone should go out and learn Dotty disregards the fact that there are no practical applications for Dotty.

If you want Scala to continue to succeed, it must be looked at beyond an academic eye. You must be willing to work with who your true user base is - business and library authors.

What is this prediction based on? We’ve observed major overhauls in several languages over history, and the “big bang” approach has failed to gain traction with users. Prolonged paralysis was exactly what we saw with Perl 6 and Python 3. Why repeat the exact same mistake?

I predict authors of libraries would be far more eager to adopt an iterative approach. The amount of work to integrate with pieces of Dotty is far less than one that integrates with Dotty entirely. It also allows for the Dotty changes to become hardened through adoption and iteration, which gives users more confidence in the version.

If you release it all at once, an author is less likely to do a major overhaul because the incentive is entirely theoretical. Businesses would look at the amount of work needed to overhaul their libraries, and many would make the decision to switch languages.

1 Like
16

IIUC the proposed feature set for Scala 3 is really going to simplify language and make it easy to use except code assistance.

I wonder why there are so many improvement in language, but there are no improvement in ide code assistance.

I think the real simplicity is when the ide can assist you in every task. No matter how often you use a library. When you can organize work in such way which does not require us to use google or read documentation.

I can only see that situation is going to be worse in such direcion. Because the delayedInit will be droped. And there are no improvment in such area. I do not sugest:

  • remain deleayedInit
  • implement kotlin receive function
  • implement import aliases
  • implement scope injeciton

It have been discussed.

I just wonder why we have so many improvement in language simplicity, but in “scope management automation” the situation is going to be more complex.

just to note:
Class shadowing will be dropped, it will not make “scope management” more easier.

17

So this is the reason why a MVP with only the DOT core features and the new compiler would not fly in my opinion. It would be all pain and no gain. We’d put out the new compiler codebase which would require considerable migration effort for no obvious benefits. My prediction is people won’t do it and it will deepen rather than avoid the split between Scala 2 and Scala 3. Authors of libraries, books, and courseware will also wait until the new features that everybody talks about and that will make Scala simpler and more accessible finally arrive. The result would be prolonged paralysis.

If it’s compatible, there is not considerable migration effort. It shouldn’t be harder than migrating to 2.14 modulo macros. Also, don’t underestimate the draw of faster compilation times. If you also promise compatibility, why wouldn’t everyone immediately migrate to dotty?

You can also keep talking about changes to make to Scala without adding them immediately, or only adding them under certain flags for now.

1 Like
18

Dotty is so named because it is based on the DOT calculus, which, as Martin explains, is a “proven foundation for Scala”, allowing to “make and prove formal statements about it” and “opens the door to do language work with much better confidence than before.”

Does that make the compiler faster? Maybe it does, what do I know. But claiming that that was the primary driving force is clearly historical revisionism.

19

I realize this post is somewhat tongue-in-cheek, but since others might not get it, I just wanted to state that it is quite a comical mis-representation of what Scala is about and what my own motives are. If I was primary interested in teaching and research I would have designed a different language. I’d probably have continued with something like Funnel.

I do believe that simplicity is not just important for teaching but also for industrial scale applications. I also believe that at some point some research results will be part of everyday languages - otherwise we should not call ourselves engineers, since engineering is the application of science to technology.

12 Likes
20

Dotty is released: Scala 3

3 Likes
21

See what happens when I’m not fully paying attention?! :slight_smile:
Thanks.

22

This is a misnomer… Dotty is published, sure. To say that it is released, which has a stigma of being a stable, production-ready library? That’s frustratingly obtuse.

1 Like
23

I believe that these experiences actually strengthen my arguments:

  • Perl 6: Was in announcement state forever. Scala 3 will go into feature freeze this summer, will have developer previews this year and the final release will be out next year.

  • Python 2/3: Suffered exactly the “all pain and no gain problem” I was writing about. People did not get why they should suffer the pain of migration for Unicode strings. Now that Python 3 has more interesting features, Python 2 is going away. Besides, I am confident we can avoid Python’s migration problems through a combination of static typing and binary compatibility.

3 Likes
24

I would personally consider “dotty” as “released” when scala 3.0.0 is released.

2 Likes
25

Is the syntax stable yet?

26

Correct me if I’m wrong, but Scala 3 will have been in development for 7 years if the release goes by the timeline that you suggest. Sure, it’s not the 15 years that Perl 6 went through, but it is significant. Why should we assume that Dotty is immune to this treatment?

27

Large projects take time, and sometimes they take longer than we’d like. The difference is that Dotty was just Dotty, the research project, until last year. We announced Scala 3 then and we will deliver it next year. That seems like a reasonable time frame.

1 Like
28

I think the biggest missed opportunity is compile times: I think that if we can provide significant compilation time wins, people will migrate to Scala 3, and eat the migration pain, without any new features at all! In case people have forgotten, in the recent Scala Developer Survey (marginally) more people wanted faster compile times than every other language feature/improvement combined.

Different work from trying to judge new features on their “elegance” or “simplicity”, but if you can show people Scala code compiling 2x as fast (which is not unreasonable: that was about the improvement from 2.10.0 to 2.12.8) people will jump through as many hoops as you’d like them to. Reactive-Akka developers, Scalaz/Cats developers, Better-Java developers, Scala-JS-React developers may disagree on the elegance of implicit function types or the new typeclass syntax, but everyone would agree that improved compile times would be a huge carrot that is hard to turn down, especially in professional environments with large codebases.

If we think the Dotty Scala codebase is so much easier to work with (I presume we do, because otherwise why would it be worth spending a decade rewriting NSC), and it has not been micro-optimized-to-the-moon like Scalac has, then it should be significantly easier to get significant wins. The fact that the language is simpler for humans to reason about should surely also make it simpler for machines to analyze.

I don’t think metaprogramming migration is going to be that hard. The main difficulty is just getting something that we can migrate to, which has not existed for the past 5 years. I write a lot of macros, in a lot of OSS libraries, and it shouldn’t take me more than a week or two to re-write all my macros using whatever meta-programming library you’d like, as long as it supports the same use cases.

The last “supports the same use cases” line is key. Past work on Metaprogramming and macros (at least post scala-reflect) has been hamstrung by the “let’s think up some elegant setup from first principles” approach. I think that is entirely the wrong approach: a feature is nothing without use cases, and if you want a serious alternative to scala-reflect, you need to start with deeply understanding everyone’s use cases for macros, how you might want to support them, and then consolidating that into a coherent API.

If I was the one working on the project, I would start by taking a dozen or so popular OSS macro libraries and making sure their use cases are fully supported by-hook-or-by-crook, before even considering the elegance of the API. Until that happens, it’s not a “migration” issue, it’s just a “new thing is strictly worse than old thing” issue.

I wouldn’t ask anyone just-getting-started to work with Scala 3, unless they had Martin Odersky and team to teach them. The top priority for someone getting started is to just do what “everyone else” is doing, to minimize the number of irrelevant issues they may encounter while learning, and everyone else is very much not doing Scala 3. “Spark works on Scala 3 (in class at EPFL)” is very different from “Spark works on Scala 3 (at 2am on Saturday night when your biggest customer just called)”

13 Likes
29

Humans are deeply psychological creatures. The pain they feel can have a lot to do with their expectations. If you tell people its going to be an epic change then you’re setting yourself up for trouble. There is a simple way to ease the transition from Scala 2 to 3. Rename 2.13 as Scala 3, 2.14 becomes 4 and the first Dotty becomes 5.

I would like to see us go over to yearly releases. This seems the trend that a lot of software is going. Using this approach if a feature / change is not ready, it just gets put back to the next release. When releases are regular, this is not the end of the world. More frequent releases would make it much easier to implement a proper deprecation cycle. Ideally when a feature is replaced, the old mechanism should not be removed in the release that its replacement is introduced.

I feel the text book issue is unavoidable. Our software is in continual evolution, the teaching of it must be in continual evolution. Surely one thing that the history of Scala has taught us, is that language creators can never predict exactly how the capabilities of the language will be used. Text books start to date before they are even printed.

Written from my Kubuntu 18.04 desktop, kernel 4.15, systemd 237.

1 Like
30

Sorry but I think it’s the other way around, it’s revisionism to say that performance was not a goal. When Dotty was originally announced talks and blog posts about it focused primarily on its performance and removal of features from Scala 2. Since the later Scala 2.12 patch releases matched Dotty’s performance and many new features were added to Dotty since then nobody is trying to sell it that way any more, that’s all.

1 Like
31

Agreed, and we’ve spent a significant amount of our cycles working on performance (recent example: Reduce some allocations by odersky · Pull Request #5748 · lampepfl/dotty · GitHub), but there’s still no free lunch in sight (except maybe [WIP] Parallelize the compiler via two-pass compilation by smarter · Pull Request #4767 · lampepfl/dotty · GitHub if everything works out), and meanwhile there’s a ton of other things to do to get to the feature freeze, so this isn’t a priority currently.

That’s basically the approach we’re taking now, starting with scalatest (much trickier than you’d think!): ScalaTest macros · Issue #5491 · lampepfl/dotty · GitHub, and recently monocle: Implement Monocle macros by liufengyun · Pull Request #5944 · lampepfl/dotty · GitHub.

1 Like
32

What I meant to say was: if you’re new to Scala, you don’t have to worry too much about which exact version you’re starting with, because most of what you’re going to do will be the same across language versions, Spark working with Dotty being an illustration of that. I’m not saying anyone should use Dotty in production today.

33

Aha, the goal of Dotty was to be faster, but it is not actually faster. Thanks for the history lesson!

1 Like
34

Yes and just from a learning perspective its much safer to keep with the herd. The new thing may be better, but virtually all the other early adopters will be fully conversant with the old thing. All the early discussions, early blogs etc are geared to people upgrading them from the old thing. When I came to Scala, I found it quite challenging because familiarity with Java, the JVM and the Java ecosystem was the starting place for most discussions.

I really do feel that gearing Scala 3 transition around teaching Scala to beginners, is putting the cart before the horse. The experienced Scala 2 users and library writers need to become conversant with the new features before we can finalise how to teach Scala 3 from scratch. How to develop students from scratch must follow not lead language development.

Yes I’m sure carrot and stick, pain and gain need to be somewhat balanced in a version upgrade, but that doesn’t mean upgrades can not be broken down into smaller chunks. My favoured compromise would be yearly version releases but no special long term support releases. Promising extended long term support for Scala 2.last would be a mistake in my opinion.

I would suggest again to consider forking the Dotty compiler, one fork’s goal would be the fastest possible convergence with and replacement of Scala 2, the other fork would support few dependencies, give few guarantees, but allow extensive experimentation.

3 Likes