Questions regarding Dotty's position in the ecosystem

Dotty
1

Originally posted in the Scala subreddit under a throwaway account. I wanted the questions to be as neutral as possible and freed from my persona and opinions, but I can understand why it wasn’t a great move.

I am not sure anymore what Dotty really is, with relation to Scala and its ecosystem. If the Dotty developers are able to, I would appreciate some answers on the following questions so I (and probably many others) can see clearer.

Regarding Dotty and what goes into it :

  • Is it decided that Dotty is to become Scala 3, or is it an experimental sandbox from which Scala 3’s features will be cherry-picked from ?
  • What is a relationship between the SIP commitee and Dotty ?
  • What is the relationship between the Scalacenter and Dotty ?
  • Apart from binary compatibility between Dotty and Scala 2 via the TASTY format, is source compatibility (to an extent) a concern of Dotty ?

Regarding the controversial “signficant-whitespace” PR :

  • I understand that the PR is meant to act as a support for an experiment as to whether it’s a good idea or not. What will be the factors against which the experiment will be declared a success or a failure ? Who decides on those factors ?
  • Are the two syntaxes meant to cohabitate, or is there going to be a decision as to which one will remain ? Who will make that decision ?
  • If the two syntaxes not meant to cohabitate, will the non-default one remain as an opt-in ?

Regarding editor support

The Metals LSP server is promising (I have used vscode as my main editor for a few months, thanks to it). I’d personally like for it to live for as long as possible, as it helps my productivity at work. I suspect it is the case for a growing number of people. I understand Dotty has a built-in LSP support, but Dotty is not a build tool, so features like “jump to definition” when the method is coming from a Java lib are probably still gonna need a tool like Metals.

  • Is there a plan for the Dotty built-in editor support and Metals to converge ?
  • Are dotty-developers going to participate in any way in the upgrade of scalameta so that it can support Dotty’s syntax ?
  • Are dotty-developers going to help with porting the SemanticDB plugin to Dotty ?

There you go. Thank you for your time

Kind regards

10 Likes
2

These are great questions (and thanks for explaining the reasoning behind the reddit account!). Let me try to answer the questions one-by-one:

Is it decided that Dotty will become Scala 3, or is it an experimental sandbox from which Scala 3’s features will be cherry-picked from ?

It’s decided that Dotty will become Scala 3. This was announced last year at ScalaDays.

What is a relationship between the SIP commitee and Dotty ?

The SIP committee reviews all new features that will be part of the Scala 3 language. Dotty is the implementation platform for these features. When Scala 3 is released, the Dotty compiler will become
the reference compiler for Scala 3.

What is the relationship between the Scalacenter and Dotty ?

The Scala Center participates in the design and implementation of tooling for Dotty/Scala 3. The center’s director Sebastien Douraene is on the SIP committee.

Apart from binary compatibility between Dotty and Scala 2 via the TASTY format, is source compatibility (to an extent) a concern of Dotty ?

Very much so, in the sense that

  • we invest in rewrite tools to migrate source automatically,
  • we want to keep source breakage after automatic rewriting as rare as reasonably possible.

Regarding the controversial “signficant-whitespace” PR :
I understand that the PR is meant to act as a support for an experiment as to whether it’s a good idea or not. What will be the factors against which the experiment will be declared a success or a failure ? Who decides on those factors ?

The experiment is open-ended, we have to see what comes up when people use the new syntax, and the new syntax is also still subject to change. I will make a recommendation once more experience is gained. In the end the SIP committee decides whether this will be part of the language standard or not. As is the case for all other new language features there will be a consultation period where everyone is invited to comment on https://contributors.scala-lang.org/.

Are the two syntaxes meant to cohabitate, or is there going to be a decision as to which one will remain ? Who will make that decision ?

There are four things thrown into one pot here which should be treated separately.

  1. New “quiet” control syntax (Dotty PR #7024). For the time being all combinations of old and new syntax are supported. My personal expectation (which is not shared by everyone) is that the new syntax is so much better that the community will move to it quickly. The old syntax will nevertheless be kept around for the foreseeable future.

  2. Enforcing well-indentation: There will be some (fairly flexible) rules that catch indentation mistakes in programs (assuming the current syntax). These will issue warnings at first but I expect that they will be turned into errors at some point.

  3. Optional braces. Depending on the outcome of the experiment, braces { … } might become optional in more situations than now. For instance: Right now, braces around a single statement branch in an if-else are optional. In the future, they might also be optional around a branch consisting of multiple indented statements. This part is really not as dramatic as people make it now. Braces are already optional today in some situations. Coding styles differ as to where braces are put. Nobody has a problem with that, and I do not expect this to cause problems in the future.

  4. Using : (or something else) at ends of lines as a replacement for {...}. This seems to be the most contentious issue. If we introduce it at all, it will clearly have to co-exist with current syntax, and it will likely be put under a language extension flag.

If the two syntaxes not meant to cohabitate, will the non-default one remain as an opt-in ?

This would be an issue only if “noisy” control syntax would be phased out at some point in the far future. We’ll think about the question if and when that issue arises.

Is there a plan for the Dotty built-in editor support and Metals to converge ?

Yes. I will personally work with other people on the Dotty team, with Scala Center and with VirtusLabs to integrate Metals with Tasty and the Dotty presentation compiler.

Are dotty-developers going to participate in any way in the upgrade of scalameta so that it can support Dotty’s syntax ?

I don’t know. This might not be necessary for Metals to work. The current plan is to pull the necessary meta-information out of Tasty, so no source file parsing is needed. Syntax highlighting is already shared between Dotty and Metals.

Are dotty-developers going to help with porting the SemanticDB plugin to Dotty ?

I don’t know that yet either. I believe we have to find out first to what degree a Tasty → ScalaMeta export is feasible and whether something else is needed. Once we know that, we can start planning who does what.

15 Likes
3

Thank you very much for taking the time to answer these questions, this is greatly appreciated.

Godspeed !

5 Likes
4

It is very scaring question for me. I understand that all analogies are false.
But I do not understand how to explain my thought in other way.
There are three interesting styles in:

  • Variant: Stroustrup
  • Variant: Linux kernel
  • Lisp style

When I had seen Lisp style at the first time I thought it was stupid, but when I had start to work with Lisp I understood it is the best style for lisp.

The anology is that a style can vary depends on a domain area.

And I feel that we can easy get situation when they will say:

  • Scala doesn’t need to be a language for everything.

Ok, but we have chosen scala 2. It is very hard to get such changes.While it is optional it will add complexity, if " signficant-whitespace" is obligatory it will add us more troubles.

Most languages have not signficant-whitespace syntax in the world, the community build have even less relevancy. It is ironic that my opinion does not matter at all in such scale :))

3 Likes
5

Note that a tasty → semanticdb prototype already exists: GitHub - lampepfl/dotty-semanticdb but is not actively maintained currently.

2 Likes
6

Hello,

Martin, thanks for the answers.

That Dotty will be the Scala 3 reference compiler, has that decision gone through the Scala Improvement Process?

If yes, which SIP is that? If no, why not? Thanks!

Best, Oliver

7

If the SIP committee votes that Scala 3 should be a given set of changes relative the the previous version why does it matter which codebase it’s built from?

8

I think that is the confusion.
Outsiders see ‘dotty is scala 3’ and they think ‘the features in dotty are scala 3 features’.

But the meaning is ‘the dotc compiler and codebase will be the scala 3 compiler, not the old scalac codebase’.

The language level features of Scala 3 are independent of ‘dotty’, but most observers dont know that.

2 Likes
9

“The Scala Improvement Process (SIP) covers changes to the Scala language, the Scala compiler, and the core of the Scala standard library.”

10

That is not a relevant distinction here, since the Scala Improvement Process also covers changes to the compiler.

As a practical matter, a change in reference compiler in itself will cause changes, even if the two compilers were intended to be equivalent.

1 Like
11

So, every PR to the compiler goes through a SIP?

12

(answering myself) Obviously not. “changes to the compiler” is narrowly defined. Im not sure where the line is drawn.

13

No the SIP committee does not review changes to the compiler. It only reviews changes to the language. Of course some changes to the compiler affect the language, and that can only be done if the language changes have been approved.

The change of compiler codebase, from scalac to dotc, is simply accepted as the only realistic way for the Scala 3 language to have an implementation at all. No one is going to reimplement all Scala 3 features in scalac, even assuming that it’s possible at all.

4 Likes
14

In this case, what does it mean that the SIP covers the compiler?

15

As I said above, for changes in the compiler that affect the language, the SIP committee indirectly validates them, because they can’t be merged without the SIP signing off the language change. That’s what it means.

The SIP definitely does not review every PR to the compiler. That would be completely insane.

1 Like
16

Cherry picking here, but I assume your last sentence to mean: “That would be mentally unhealthy for the members of the SIP Committee”.

17

No, that would be an insanity that envelopes the entire ecosystem.

There are many comments on PRs along the lines of, “Is this change big enough to warrant a SIP?” and sometimes the answer is no. That decision is not vetted by the pre-SIP committee.

You can’t have progress grind to a halt on procedural grounds.

I think the “Scala team” and associated actors such as the Scala Center have done a good job at maintaining this pragmatic goal, of advancing the language without nullilfying previous investments. That is, I’m sure projects have experienced challenges upgrading, but that is not the meme illustrating what has sundered the community.

Personally, I am suspicious of the benevolent dictator model under which dotty was conceived, but on the other hand, it’s working out OK. The devil is in the details, and it turns out to be the devil you know.

I would like to acknowledge that they rewrote collections and, lo, it kinda just works. So I’m leaning toward, they rewrote the language and maybe it will just work.

My only criticism of the dotty effort is that when I knock off my day job, I can still make a contained contribution to Scala 2, but for some reason it was harder to keep up with the evolving Scala 3 discussion, even in terms of how do I build it and try it out. Right now, I’m trying to get my daughter to keep at her flute practice for another quarter hour, it’s so pleasant to hear.

4 Likes
18

No. It would be #3 https://www.dictionary.com/browse/insane

1 Like
19

I’m not entirely sure I follow the conclusion that just because it is impractical to review every single PR by committee, it necessarily follows that no review of any changes whatsoever to the compiler can be subject to the SIP other than through the language definition.

But if this is the reality, why don’t we simply say the SIP covers the language and the core of the standard library, period? Adding the compiler to the list only gives the wrong impression that there is more to it.

20

For one, I didn’t learn Python at all, due to the annoyances of whitespaces. I also think editor support will be harder than braces (e.g. jump to ends of braces) and when you have too many nested level, you get lost which level the code is in, which is less readable to me.

1 Like