OSS and Professional thoughts on migrating to Scala 3

Dotty
5

Thanks for taking the time to write this. It seems to me that the software industry as a whole is moving towards a model of steady progressive predictable change, as opposed to the backwards compatibility must be maintained for ever model and the big bang revolutionary model.

A complex system that works is always developed from a simple system that works. We can surely go further now and say that a complex system that works, always develops from a very simple system that works through, multiple steps that all work. Symbolism matters, the nettle needs to be grasped. So I would suggest that the Epic / Major change distinction should be dropped and hence that the aim will be to deliver Scala 4, no more than 2 years after Scala 3, and Scala 5, no more than 4 years after the release of Scala 3.

2 Likes
6

Thanks for reading!

@kjsingh

I don’t think I say that git branches don’t work, rather I think they’re generally an inferior tool to using version-specific source-folders if you expect to keep your library compatible with multiple versions of Scala as it evolves.

We do in fact use git-branches for some of our spark-related-code at work, and it definitely has a tendency for the branches to diverge and for improvements to only land in master, and for PRs targeting multiple branches to get harder over time as the codebases diverge.

It’s not impossible - it works - but from my experience with the process I definitely would not choose git-branch cross-building for my own projects, and at work we have not chosen git-branch based cross-building for other projects and instead added custom support to Bazel to allow source-folder-based cross-building and are very happy with that choice

@julienrf,

Regarding the CrossCompat rule, scala-collection-compat has definitely been great, no doubt. I can’t really speak to the value of the rewrite rule, because the scala-collection-compat shims have been excellent at bridging the gap between 2.12 and 2.13 and making them almost the same. I’ve used scala-collection-compat very heavily, but I haven’t personally used the rewrite rule at all.

Mind you, I’m a big fan of autofixes in general. At work, I have pushed projects that autofix all sorts of things: pull-requests are automatically fmted with the relevant *fmt tool regardless of what language they are in (scalafmt, jsonnet fmt, yapf, etc.), generated files are automatically re-generated, and so on. Autofixes are great; I just don’t they apply to the a potential Scala-3 migration in either of my OSS or Professional contexts, and I suspect other people maintaining open source libraries may have similar constraints.

Regarding the process having already started, you’re right that that’s the case. I’m just bringing this up because I’ve constantly heard “can we autofix this?” and “this will be solved by a scalafix” in threads, with much hope and skepticism is unduly focused on such autofixes, which in the end I do not think will be the savior that people seem to hope. It’s a good technology, but I don’t think it’s a good fit for this particular problem

You’re right about macros being a risk, but I think that argument is for a time years ago. Right now, the current usage of macros is where the community has decided it is worth it, and for us managing the upgrade we have to make do with the current reality.

Furthermore, I think the macros in libraries like Circe (circe-derivation), Scalatest, and SBT are reasonable: I have many of the same macros in my own libraries uPickle, uTest, and Mill. I don’t think any of these libraries are using macros gratuitously.

7 Likes
7

One of the stated aims of Scala 3 is “to be more opinionated.” I have for a little while become concerned about this, but didn’t feel comfortable about speaking up. As @lihaoyi, a very major player in the Scala Community has raised his concerns, I a very minor player within the Scala Community feel able to raise mine.

If Scala 3 is to be more opinionated, the question that arises for me is whose opinion. Martin Odersky’s? EPFL’s? Lightbend’s? The SIP Commitee?, some kind of wide ranging consensus or something else? An attitude of “trust me”, or “trust us” seems to have arisen. Bjarne Stroustrup famously said “There’s languages that people complain about and there’s languages that nobody uses.” So I would argue that its actually a testament to the creator of a language, if when he / she comes to create a 2nd / 3rd version of that language, there are a significant number of people who care enough to not just trust them to get on with it.

I hope my contributions will be experienced as respectful. However I would argue that even disrespectful criticism is a testament to the achievements of the language creator. It was inevitable that once Scala started to have significant success, it would be seen as a threat. In particular many people were happy with Scala as a Haskellator. Haskell has consistently (whether by accident or design) failed to break out of its ghetto. It was inevitable that some of those people would become angry when they realised the creators of Scala were interested in Scala as a language in its own right. It was inevitable that some of them would become angry if and when Scala threatened to eclipse Haskell.

I would suggest that their criticisms of Scala can not be taken in good faith. Their criticisms were not intended to be constructive, they were in fact intended to be destructive of Scala’s success. Added to this is the fact that the more successful a language becomes, the more developers will be required to use it not through choice. Some of them will have been quite a happy with the previous programming languages, this will inevitably produce a growing ground swell of dissatisfaction.

So in particular the criticism of language complexity is not one that can simply be taken in good faith. Managing complexity, reducing incidental complexity is what its all about. There are no silver bullets for simplicity, as we strive to solve ever more complex problems, as we struggle to work in ever more complex problem domains and eco systems. To say that one finds a programming language complicated, really tells us nothing more than that you don’t like the language. I mean who actually ever claims to like the unnecessary complexity of a language? So I would suggest that any mad rushes to “simplify” the language will end in disaster. Sudden lurches into “simplification” will actually lead to the reverse.

We should be very wary of premature simplification. We should be very wary of trying to simplify aspects of the language, before we have fully grasped the fully comlexity of the problem domain, before we have fully grasped the full range and diversity of the eco system that uses that aspect of the language. Above all we need to understand the full capabilities that are required, only then should we focus on accessibility and learning. Learnabilty, the ability to use the language quickly and efficiently for simple purposes is of vital importance. The ability to evolve one’s use of the language, to be able to progress in using the language in more sophisticated and powerful ways in well documented, comprehensible small steps is a key axis of scalabilty. However its important not to put the cart before the horse.

3 Likes
8

I believe the general strategy of the professional / commercial industry has settled around adopting changes as gradually as possible.

By far, the thing I am waiting for the most in Scala 3 is reduced compile times, and I have the feeling that this bothers many other professional developers. Anything that makes the migration harder just for the sake of other features / benefits is one step further away from faster compile times.

I think this lesson can be somewhat learned from Python 3, which took forever to migrate to (and is still going on), and perhaps because it introduced too many breaking changes that only provide minor benefits on their own (this is a nice article about it).

1 Like
9

Scala 3 is introducing some nice things (Union types, enums, nullability) and removing/fixing some broken stuff, but I’m not entirely convinced by some of the new implicits stuff. I thought that it was meant to be simpler than what we have today (which I basically avoid due to the complexity), but objectively I’m not convinced that it necessarily meets that bar of being significantly simpler.

E.g. one example is extension methods. There are multiple different ways that these can be expressed, but I think that the language would be much cleaner if they were all expressed using the Collective Extensions syntax, and all other methods for defining extension methods were dropped.

In fact, I would even suggest using _ if it doesn’t matter what the extension name is.

E.g.

extension stringOps on (xs: Seq[String]) {
  def longestStrings: Seq[String] = {
val maxLength = xs.map(_.length).max
xs.filter(_.length == maxLength)
  }
}

extension listOps on [T](xs: List[T]) {
  def second = xs.tail.head
  def third: T = xs.tail.tail.head
}

extension _ on [T](xs: List[T]) with Ordering[T] {
  def largest(n: Int) = xs.sorted.takeRight(n)
}

My perceived advantages of this syntax over the other methods are:

  1. It clearly identifies what it is (i.e. an extension). It is easy for a newbie to see and google for Scala extension methods.
  2. Methods are defined in exactly the same way as you would find in a regular class.
  3. This is apparently similar to how it is done in other languages, and commonality and familiarity is generally a good thing.
2 Likes
10

This is a bit off-topic in this discussion. I believe the better way to show you concerns are either in the topic for the current implicits proposal (monitored by the SIP committee I think), or in the topic for the alternative proposal (authored by me).

11

The specific comment about extension syntax may be off topic (but raised previously), but I think that the underlying principle that it is attempting to illustrate is on topic. Scala 3 was meant to be simpler than Scala 2, but I’m not sure whether it is going to be simpler or just have different (and perhaps more) overall complexity.

I hope that the folks in charge don’t try and rush Scala 3 out of the door, but take the necessary time to take a critical look at all of the features that are being introduced and harshly evaluate each feature to determine worth the feature is really worth the additional complexity that it inevitably brings. I would suggest, if in doubt leave it out, and introduce it in Scala 3.1 or 3.2, especially since it is much harder to take something out again after it has been introduced. There is an excellent presentation along this theme by Guy Steele: “Growing a language”.

3 Likes
12

Maybe the new rule should be that discussion of Scala 3 implicits syntax is not allowed unless conducted over a beer or other beverage of preference.

That rule could also improve the atmosphere at SIP meetings.

Is there already an XKCD where any mention of Dotty leads inevitably to a personal opinion about syntax?

Also add it to the list of things excluded from dinner conversation: politics, religion, Dotty implicits.

On-topic, I appreciate the patented @lihaoyi tone of thoughtful, fearless optimism. We’ve heard the Python 3 alarm for so long that it’s baked into the Scala 3 sales pitch; I’m not sure I’ve heard “keep calm and migrate on.”

4 Likes
13

Yeah, I was just trying to prevent yet another syntax bikeshedding discussion about implicits. I couldn’t agree more with your underlying message.

2 Likes
14

Thanks for the excellent write up!

These are all things that have been discussed and are top-of-mind in the discussion of Scala 3. You definitely nailed the major concerns when it comes to community, stability + maintenance, and then commercial migration.

I think your concern w/ macros and dependencies on macros is a major pain-point for evolving the ecosystem. It means that the binary-compatibly usage-mode for libraries will basically require a fence between downstream scala 2 code, and scala 3 code. This gives an option for folks either at the beginning of the dependency chain, or the end but really hurts those in the middle (like ammonite).

As you’re aware, the Scala ecosystem has focused on rebuilding from core, and most maintainers have gotten great at rebuilding the ecosystem quick on release. The binary compatibility basically just gives those at the FAR downstream the ability to try out a (limited) Scala 3 with their current tooling (theoretically).

I think ensuring these ‘middle-of-the-graph’ dependencies have a good migration story is critical for the health of the ecosystem (both OSS and commercial), so thank you for calling attention to it!

2 Likes
15

One thing that might help a lot with migration is if Scala 3 macros were implemented for Scala 2. The reverse is impossible because Scala 2 macros are about exposing low-level implementation details which don’t exist in, and can’t be mapped to, Scala 3. However Scala 3 macros are very high level. In principle I expect it would be possible to implement them on Scala 2.

Of course, the development effort would be significant. I don’t know that anyone has the time, ability, and interest to do this. But I think it’s the only way to get to the seamless transition that Li Haoyi is describing.

3 Likes
16

I’d be happy if we could just have extension on syntax. But we can’t since this syntax does not allow to define or implement abstract extension methods. And that capability is the key to do typeclasses. Without it, you are back to simulacrum. Have a look at https://dotty.epfl.ch/docs/reference/contextual/typeclasses-new.html and see whether that could be implemented with just extension on.

1 Like
17

I think this hits the point of what @rgwilton and the rest of us are saying regarding simplicity. The new given system is not simple (and definitely isn’t simpler than before), and it will make it harder for people to migrate (or newly adopt Scala).

There are alternatives for making this much simpler, and I would’ve loved if someone from the SIP committee would take a look at those alternatives and actually consider them.

1 Like
18

Let’s take discussions about individual features in the threads where they belong. I should not to have responded to @rgwilton here, my mistake.

5 Likes
19

I believe it would be a huge undertaking to re-implement Scala 3 macros in Scala 2. A solution that’s simpler to implement was proposed by @dwijnand: Allow parallel Scala 3 and Scala 2 implementations of the same macro in the same source file. The effort to write these dual implementations is not worse than writing everything with Scala 3 macros if we take into account that the Scala 2 versions exist already.

That would then allow cross building libraries without separate version specific files (but maybe such versions are still needed for other reasons). I’d be interested in feedback from library maintainers whether such a feature would help in cross-building.

3 Likes
20

The motivation for this idea was to allow libraries to transition to the Scala 3 compiler, while still keeping Scala 2.13 support for their libraries. Kind of like an interim stage in migrating a library from Scala 2 to Scala 3.

1 Like
21

Where is that proposal

22

Apologies. I have posted my reply to Updated Proposal: Revisiting Implicits

23

I just don’t see how we can avoid almost completely duplicating our code bases for cross compilation due to the implicit -> given changes. This is not a minor detail. Yes we can somewhat avoid this with not changing anything and relying on Scala 2 support, but once we want to make the move into the new syntax, we might as well create a whole new branch for our libraries.

24

I just don’t see how we can avoid almost completely duplicating our code bases for cross compilation due to the implicit → given changes. This is not a minor detail. Yes we can somewhat avoid this with not changing anything and relying on Scala 2 support, but once we want to make the move into the new syntax, we might as well create a whole new branch for our libraries.

implicits are still around, so for cross-building with 3.0 this will not be an issue. The intention is that as long as cross-building is widely needed, the large language subset that is common between 2 and 3 will be maintained.