Move slowly and iterate

Language Design
1

I seem to recall that some years ago, there were sentiments on the mailing lists that Scala had become too stabilized, that it seemed there was too much resistance to evolving the language as Scala had become used more widely and in the enterprise. Thankfully, today that is far from the case, for a number of reasons.

One reason is that a few years ago, Dotty became a realistic project, and has since been intensely active. The motivation for Dotty was primarily to reboot scala on solid theoretical foundations. Since it was decided that the Dotty codebase would itself be used for Scala 3, that is another important reason for a “reboot” in some sense: The new codebase is better than the scalac codebase in many ways, so whatever ramifications switching to a rewritten compiler has, are worthwhile for that reason as well.

The Dotty codebase is also the place where a lot of innovation on the language is happening. However I would argue that technically, this is mostly orthogonal to the reboot, since language evolution is a need in of itself. In principle the same language changes could be written against the Scala 2 codebase (and in fact a number are being backported), however there are some practical reason why they’re being written against the Dotty codebase: (1) It’s expected to be their long-term home, so that is their primary target in a sense; (2) Since there is no pressure to cut production-ready releases from the Dotty codebase, it’s a safer place for experiments; and (3) it’s an easier codebase to work with.

So I think in discussions we need to disentangle somewhat the DOT-and-codebase reboot from the broader matter of language evolution.

Now, there are some language changes that are inherently coupled to the new DOT foundations. However many of them are not. So the question is, should we be aiming to work through and implement as many changes as possible in one go? Or should we have more of a slow-and-steady outlook?

For instance, what is our long-term strategy? Three years after Dotty is released will we still be making changes to the language, or at some point will we just declare it “done”? I assume we will think of things by then that we haven’t thought of today, and besides the landscape of other languages will have changed in one respect or another. So I’m sure there will changes we want to make then, one way or another.

Speaking of the languages landscape, a few years ago perhaps there was more of a sense that most languages are what they are, especially Java. But today it looks to me like every major language is evolving and innovating. ES6 was a game changer for the Javascript community but language updates are a yearly thing now. Java is evolving almost rapidly now. Typescript, Rust, C#, Python, C++, all seem today to have a mindset of continuous evolution.

So I think before we discuss what language changes will be in Scala 3.0, we should have a clear strategy in place for how innovation and evolution will happen after 3.0.

The current, almost entirely (in my field of vision) implicit attitude seems to be that Scala 3.0 is our “one big chance.”

I want to make a counterargument to this approach. Here are some issues I have with it:

  1. Right now (or in a year from now, or whatever), what should I tell someone that wants to learn Scala? Why would someone learn a language when the fundamentals are going to change radically soon?
  2. It will be too hard to upgrade. Yes there will be rewrite tools, although we will only know how successful they will be (stated goals notwithstanding) when they are actually tried on the many gigantic private codebases out there, which can only happen after 3.0 is finalized. To be clear the rewrite tools are necessary. But how far they will get us is to me a hope that has not been proven. This applies especially to changes that are not simple mechanical transformations. And I can imagine scenarios where the rewrite tool cannot really be used. People have already said it’s unrealistic to make such big changes to some codebases.
  3. I fear that this mindset leads to pressure on the people making decisions about the features, which may influence things negatively.

The situation reminds me a bit of the old Waterfall approach vs. Agile/iterative approach dichotomy. Not that Dotty development is Waterfall, but the distinction can be applied at different levels. The main idea of the iterative approach is that you get things into real-world use as quickly as possible, even if it means smaller units of delivery, which ideally means into production releases used by end-users.

So here is an alternative proposal.

I said before that the reboot is inherently tied to some langauge changes, namely those that are required by DOT. But really, we can completely decouple the codebase switch from language evolution altogether, since Dotty has a Scala 2 mode. So:

  1. Release an MVP (minimum viable product) from the Dotty codebase as soon as possible, with Scala 2 mode only (at least as exposed to most users). The most radical version of this is to implement 2.14 this way, but that isn’t intrinsic to the point.
  2. The next stable release from the Dotty codebase would be the minimum changes required to implement DOT.
  3. The highest priority should be to get TASTY production ready so that we can put an end to the “separate universe per major scala version” thing. Right now, that is a big restriction on innovation since it boxes in the compiler release cycle a lot. This would take off a lot of the pressure involved in designing features, since you don’t have to worry about missing your chance to get it in.
  4. Freed of extraneous pressure, evolve the language at its own pace, from now on, indefinitely.

Optionally:

  1. We could adopt a timed-releases model (similar to Ubuntu, Java, GitLab, and many others), namely that stable releases are cut at some fixed time period, but what goes into them can vary, depending purely on what got done during that cycle. For instance if the cycle was monthly (like GitLab), it doesn’t matter if during that month there was only a tiny bug fix, or there were seven major language changes that stabilized, you release it.
  2. Adopt semantic versioning (MAJOR.MINOR.PATCH). Bump the first number whenever appropriate. Don’t fear high numbers. A few years ago getting to higher numbers may have seemed weird but now it’s completely normal, for instance Google Chrome’s version is in the 70s and Firefox’s is in the 60s. C# is up to version 8 and Java is up to 12. I for one would not object to having in the future Scala 4.0, 5.0, etc.

The relevance of these last two suggestions is that it dovetails with the idea that in a post-TASTY world, innovation can happen at an unconstrained pace.

One argument made by Martin Odersky for the “big bang” approach is that “You cannot rewrite books continually.” If I understand correctly he applied that mainly to changes where something fundamental was being replaced, as opposed to changes that involve “putting new layers of additional complexity on what is already there” – such changes don’t require rewriting the book I guess, since newcomers can read the book without getting wrong information, and learn whatever was added later separately?

Anyway, I wonder how other languages compare in that regard? In any case I’m not sure the answer, but it seems to me that it needs a different solution somehow.

12 Likes
2

IMHO More thought should be put into preventing Scala from jumping the shark in Scala 3, like Python 3 did. The trend in proposing ever more huge language changes (rewrite all macros, remove new, remake all syntax for implicits) and pushing massive language changes without any discussion whatsoever (deprecation of package object extends) worries me, and likely not just me.

7 Likes
3

I don’t think that’s fair – we’ve been talking in the community about package objects as a language wart to be fixed for as long as I can remember.

And nothing in Dotty goes into Scala 3 automatically: there’s a good deal of process and discussion that has yet to happen before anything becomes final. Hence all of the feature-specific discussions that have been happening here. Those aren’t “this is a fait accompli” – they are “this is something we are currently planning, so now is the time to talk about it before it is officially proposed as a language change”.

I’m sympathetic to @nafg’s main point, that we should take this opportunity to revise the language-revision process to be more evolution-friendly, and should try to avoid feeling like Scala 3.0 is our only chance to make significant changes. But the package object change was one of the less surprising (to me, anyway) proposals for Scala 3…

2 Likes
4

I’d love to see a more iterative approach with more frequent releases where new features can land without waiting for a major revision. TASTY was announced in 2015. If it had been released without waiting for Scala 3, it could have been helping the binary packaging ecosystem for a while now. Similarly, implicit function types were announced at the end of 2016. Assuming they were stabilized in Q1 of 2017, they could have been out and in use for a couple of years now.

1 Like
5

IIRC, deprecation of package object extends has yet to be implemented in Dotty, but it did suddenly get merged into Scala 2.13. This single disruptive and invasive change is a tough pill to swallow – motivation to upgrade to 2.13 has been greatly reduced.

It would be preferable to do one large rewrite in Scala 3 than several rewrites along the way to Scala 3.

6

The issue is that the package object change occurred first in 2.13. Why blow up affected code bases with deprecation warnings when the deprecation can easily wait until 2.14?

Also, there’s no viable replacement on the horizon, so at least affected users have that going for them.

7

The Python 2/3 issue and the Perl 5/6 versions splits have been incredibly damaging to those communities. That cannot be understated and it is a problem that must be proactively addressed. You cannot fix the issues retroactively. Kotlin or someone else steals your Scala 3 ideas and the Scala 2 community because you stalled one without delivering to either party.

The Waterfall problem is real and you need to fix it by delivering the MVP. Here’s how I’d do it:

  • Immediately remove every single feature you hate and don’t want to support. Macros, XML, extending AnyVal, etc
  • Add the MVP pieces that language needs: e.g. New Macro language.
  • Reimplement all the temporary work-arounds, essential features, using your new macro language
  • For all new experiments and ideas for new things: implement them as optional features via macros
  • Get everyone moved across.
  • Replace the backwards compatible macros with best practice macros.
1 Like
8

Thanks for the thoughtful comments. Of course, I have been thinking for a long time what is the best way to move to Scala 3 and I am very interested in other’s opinions on this. Doing what we are trying to do for Scala 3 is very complex, since the goal is to simplify the language, which means both making it more accessible and slimming it down.

Speaking of the languages landscape, a few years ago perhaps there was more of a sense that most languages are what they are, especially Java. But today it looks to me like every major language is evolving and innovating. ES6 was a game changer for the Javascript community but language updates are a yearly thing now. Java is evolving almost rapidly now. Typescript, Rust, C#, Python, C++, all seem today to have a mindset of continuous evolution.

It’s true that language evolution has picked up. But it’s also true that most of the languages in that list are getting bulkier and fatter at a rapid pace. In fact, some of Scala’s close cousins like Typescript or Kotlin or Swift are arguably already bulkier than Scala despite being much younger. Adding features is easy and can be done continuously. Taking features away and restricting them is hard, since it causes migration pain.

So, what to do? To get more data as a basis for a discussion I have made a classification of the currently proposed feature set for Scala 3.

Some high-level summary:

  • Scala-3 is indeed serious about simplifying, restricting usage and slimming the language down. The majority of changes falls into this category.
  • The hardest aspect of language evolution concerns meta programming. Here we pay the price for the organic / unregulated (pick your favorite term) feature spread in various macro extensions and plugins of Scala 2.
  • The highest cost of migration has to do, in this order: Meta programming, the new type checker and inferencer, features to be dropped. Relative to this, the cost of migrating to new or simplified features is small.

So this is the reason why a MVP with only the DOT core features and the new compiler would not fly in my opinion. It would be all pain and no gain. We’d put out the new compiler codebase which would require considerable migration effort for no obvious benefits. My prediction is people won’t do it and it will deepen rather than avoid the split between Scala 2 and Scala 3. Authors of libraries, books, and courseware will also wait until the new features that everybody talks about and that will make Scala simpler and more accessible finally arrive. The result would be prolonged paralysis.

I believe the most promising approach is to build the initial Scala 3 release on

  • the new compiler codebase including the new typechecker and inferencer,
  • the minimal feature set to support a reasonable subset of existing macro usages,
  • the core features suggested by our foundations work (there are just 4 of them),
  • the set of simplifications and restrictions that we believe are beneficial,
  • a small set of new features that are judged to be clear wins.

That’s still a big jump for sure. The advantages are that it gives a clear migration end point to everyone, and also a motivation why one would want to do this. By everyone I mean maintainers of codebases and libraries, as well as people teaching others.

To make the big jump bearable, we absolutely need binary compatibility between Scala 2 and 3, so that codebases can migrate at their own pace, and no-one is blocked by some other piece of software that has not migrated yet. So I completely agree that the stabilization of Tasty and its implementation in Scala 2 is essential and should get highest priority.

Looking beyond Scala 3.0, I believe the next several releases will concentrate on phasing out the old features that are superseded by the simplified ones. There will also be new developments for sure. Refinement types and algebraic effects are both interesting developments IMO, but it’s too early to make concrete plans for them.

Finally, let me respond to your initial question:

  1. Right now (or in a year from now, or whatever), what should I tell someone that wants to learn Scala? Why would someone learn a language when the fundamentals are going to change radically soon?

I’d tell them to start with Dotty / Scala 3 now. We already use Dotty to teach our own students at EPFL.

11 Likes
9

It’s worth noting that we were able to make our students use Dotty with basically no changes to our course material (even Spark works now!), so I don’t think programmers new to Scala need to worry too much about the jump to Scala 3, the only investment I wouldn’t make if I were them would be learning about scala-reflect and macros, but that’s a rather advanced topic.

1 Like
10

Scala is owned by Martin Odersky, an academic teacher and researcher, and therefore its mission is primarily to advance teaching and research. Scala’s proven usefulness outside of academia is greatly appreciated, but not the main priority. Improvements that serve non-academic purposes are very welcome as long as they do not conflict with the academic mission.

Being a research project, change is constant and largely driven by theoretical considerations. If Scala 2 has been stable recently, then that is primarily because it has been in the shadow of Dotty/Scala 3.

As soon as Scala 3 has stabilized and researchers had enough time to draw lessons from it, research focus will again shift towards new shores and how things can be made even better. At that point we can expect Martin and his lab to invent an even better language. I don’t know how they will call it, so let me for now just call it the Most Awesome Programming Language Ever (MAPLE). MAPLE will be so sound, simple and consistent that Scala will be a broken convoluted mess in comparison.

At this point, there are two possibilities.

Scenario one is that MAPLE looks a lot like Scala such that one can dream of migrating easily. In that case, MAPLE will within a few years essentially become Scala 4, and history will repeat itself.

Scenario two is that MAPLE looks so different from Scala that we are not even going to pretend that migration is generally feasible. In that case, MAPLE will draw the attention of the academics and Scala will be left mostly to non-academic users. This will open the floodgates for feature requests that the academics had blocked until then because they deemed them unsound or too complex. And in some cases, the new feature will make the language a bit broken or more complex, but the practical benefits will outweigh the deficits, while in other cases, the community will regret the new feature and remove it again two versions later. For some years to come, Scala will be really cool, but gradually, it will be bogged down by complexity and eventually be superseded.

1 Like
11

For me Scala is very much a practical, real-world language. I use it to do things. It generally does them very well, with some warts (but then all languages have some warts). If it became a primarily research oriented language, I’d jump ship. So far, the changes I see in dotty are all big improvements. It does a lot more type inference so I have to think a lot less about types. The new way of writing implicits interacts beautifully with opaque types and top-level declarations, so I can spend much more time designing my codebase and a lot less of it structuring things to make them palatable to the compiler.

FYI, I’ll start writing production code in dotty once the syntax stabilises enough to expect code I wrote a couple of months before to not have bitrotted against the latest compiler. Right now I’m writing some toys.

2 Likes
12

It is scary phrase, I can only hope that scala will not change its original concepts. It could be very painful to migrate on kotlin or something else :frowning:

3 Likes
13

I think @nafg hit it on the nail with this statement:

So I think in discussions we need to disentangle somewhat the DOT-and-codebase reboot from the broader matter of language evolution.

Personally, I would like to see dotty released soon even if all it did was replicate the 2.12 scalac functionality. Future 2.x versions should then include backward compatible improvements to the language. Why? Because the sooner we get these changes accessible to people, the sooner and more gradually (incrementally) they can evolve their code bases.

When it comes time to do the Scala 3 features that require source changes from Scala 2, then the community will have perhaps years of experience with dotty already which hopefully has matured through the exposure to real world usage.

14

I think you are missing the whole reasoning behind this redesign. It is not purely driven from the research perspective but is actually required in order to substantially improve Scala any further.

There have been a lot of complaints from enterprise Scala developers about various things but one thing comes at top spot - slow compilation of Scala code bases. And from what has been discussed in community in past few years is that scalac is becoming harder and harder to improve.

And some of the these scalac problems have been attributed to some fundamental design choices (which may look bad in hindsight but made sense at that time) which are so deeply entrenched in Scala that those can not be changed without a major rewrite.

The whole reason of this rewrite is to simplify Scala with better design choices (some of which were not possible back when Scala was conceptualised).

1 Like
15

I disagree with this entirely. I think it’s important to understand that Scala 2 is used in real-world applications today, and Scala 3 is not. Employers are not maintaining Dotty code bases today because the code base is experimental. To say that someone should go out and learn Dotty disregards the fact that there are no practical applications for Dotty.

If you want Scala to continue to succeed, it must be looked at beyond an academic eye. You must be willing to work with who your true user base is - business and library authors.

What is this prediction based on? We’ve observed major overhauls in several languages over history, and the “big bang” approach has failed to gain traction with users. Prolonged paralysis was exactly what we saw with Perl 6 and Python 3. Why repeat the exact same mistake?

I predict authors of libraries would be far more eager to adopt an iterative approach. The amount of work to integrate with pieces of Dotty is far less than one that integrates with Dotty entirely. It also allows for the Dotty changes to become hardened through adoption and iteration, which gives users more confidence in the version.

If you release it all at once, an author is less likely to do a major overhaul because the incentive is entirely theoretical. Businesses would look at the amount of work needed to overhaul their libraries, and many would make the decision to switch languages.

1 Like
16

IIUC the proposed feature set for Scala 3 is really going to simplify language and make it easy to use except code assistance.

I wonder why there are so many improvement in language, but there are no improvement in ide code assistance.

I think the real simplicity is when the ide can assist you in every task. No matter how often you use a library. When you can organize work in such way which does not require us to use google or read documentation.

I can only see that situation is going to be worse in such direcion. Because the delayedInit will be droped. And there are no improvment in such area. I do not sugest:

  • remain deleayedInit
  • implement kotlin receive function
  • implement import aliases
  • implement scope injeciton

It have been discussed.

I just wonder why we have so many improvement in language simplicity, but in “scope management automation” the situation is going to be more complex.

just to note:
Class shadowing will be dropped, it will not make “scope management” more easier.

17

So this is the reason why a MVP with only the DOT core features and the new compiler would not fly in my opinion. It would be all pain and no gain. We’d put out the new compiler codebase which would require considerable migration effort for no obvious benefits. My prediction is people won’t do it and it will deepen rather than avoid the split between Scala 2 and Scala 3. Authors of libraries, books, and courseware will also wait until the new features that everybody talks about and that will make Scala simpler and more accessible finally arrive. The result would be prolonged paralysis.

If it’s compatible, there is not considerable migration effort. It shouldn’t be harder than migrating to 2.14 modulo macros. Also, don’t underestimate the draw of faster compilation times. If you also promise compatibility, why wouldn’t everyone immediately migrate to dotty?

You can also keep talking about changes to make to Scala without adding them immediately, or only adding them under certain flags for now.

1 Like
18

Dotty is so named because it is based on the DOT calculus, which, as Martin explains, is a “proven foundation for Scala”, allowing to “make and prove formal statements about it” and “opens the door to do language work with much better confidence than before.”

Does that make the compiler faster? Maybe it does, what do I know. But claiming that that was the primary driving force is clearly historical revisionism.

19

I realize this post is somewhat tongue-in-cheek, but since others might not get it, I just wanted to state that it is quite a comical mis-representation of what Scala is about and what my own motives are. If I was primary interested in teaching and research I would have designed a different language. I’d probably have continued with something like Funnel.

I do believe that simplicity is not just important for teaching but also for industrial scale applications. I also believe that at some point some research results will be part of everyday languages - otherwise we should not call ourselves engineers, since engineering is the application of science to technology.

12 Likes
20

Dotty is released: Scala 3

3 Likes