Two proposed changes for extension methods

Language Design
1

Extension methods are shaping up to be a great addition to Scala. I have now used them in extensively in both the compiler and in my teaching, so far with few hassles. But there are two small issues that we might want to address before shipping Scala 3. (There’s also a larger one that we want to allow multiple type parameter sections, but this one is for later).

The first issue is the status of extension as a soft keyword. That caused a bit of confusion . For instance you can have a method named extension which you could call like this:

obj.extension(f)

But you can’t call it without a qualifier at the beginning of a statement like

extension(f)

since then the extension is classified as a keyword. See Methods named `extension` cannot be invoked · Issue #10076 · lampepfl/dotty · GitHub for more details.

We could try to put extra effort in disambiguating, but the question is whether it’s worth it. It might be better to make extension a hard keyword, like all other keywords that start a definition.

That would also offer a solution for the second problem: How to call an extension method as a normal method. That’s needed sometimes when we want to be explicit what method should be called. So far, the scheme is that an extension method

object obj:
  extension (x: A) def fun(y: B): C

is encoded as the method

object obj:
  def extension_fun(x: A)(y: B): C

So if fun is defined in object obj, you can call it as obj.extension_fun(a)(b) instead of the usual syntax

a.fun(b)

It turns out that this user-facing naming scheme is a bit awkward. How do you prevent people from writing extension_foo definitions themselves, or how do you prevent the compiler from getting confused by them? Which name do you import? What about overriding rules? And so on. It would be simpler if we could avoid the name-mangling of fun. But then we need special syntax to express a direct call to it, since allowing both prefix and infix usages at the same time would lead to too much ambiguity. With extension as a soft keyword, I was unable to come up with a convincing syntax. But with extension available as a hard keyword there are new possibilities. Here’s one originally proposed by @smarter.

obj.extension(a).fun(b)

This follows the order in which things are declared: obj, then extension, then the first parameter, then fun, then the second parameter, so it should be easy to remember. Also, the use of extension makes it clear that we deal with an extension method, which makes it simpler to figure out what the syntax means. After all, the use case is quite rare, so a lot of readers will be puzzled when they see it.

8 Likes
2

What would something like this evaluate to?

val funExtension = obj.extension(a)

My first guess is that it would be disallowed, but personally I don’t like that, as it would be the only case in the language where you can’t partially apply something in that manner.

Overall though I do like the idea and the goals.

3 Likes
3

It would have to be disallowed.

4

That could always evaluate to some auto-generated object with the right methods.

5

obj.extension(a) would just not be a value, in the same way that packages and Java classes aren’t values even if they’re valid prefix (e.g. System.out is a valid expression, but val x = System doesn’t compile).

3 Likes
6

So we change a corner case that looks ugly (name mangling) for another one (a perfectly fine looking method call that cannot be).
In the spirit of making concessions, some cost we’ll pay, but I don’t find either good enough for the cost.

I didn’t mind the original encoding of extensions, where you had to give them a name and that would generate an equally named object there. Perhaps we can do away with requiring a name but still generate an object called extensions or similar? though I realize that this would also be some form of name mangling from usage perspective, as a user could still declare such an object himself.

The language already does syntactic sugar for things that get desugared to “regular” scala, in the same vein, would it be possible to desugar

object obj:
  extension (x: A) def fun(y: B): C

into

object obj:
  object extensions:
    @extension
    def fun(x: A)(y: B): C

My idea here would be that @extension alters the way that it can be called in the same way as @infix is intended to be used today, permitting it to be called in the desired way. This way the extension syntax would just be syntactic sugar.

Edit: rewording so that my message is less nonconstructive.

7

I can kind of see that, but as I’ve always thought of it. If it’s defined in Scala (and is not package, unless it’s a package object), then I can assign it to a value. This VERY much looks like a function call, and without being pretty familiar with how extension methods are encoded, I would be surprised that it’s not valid. I can already imagine it being a “common” question why in sites like StackOverflow.

I much more like the idea of it returning something that you can actually call the extension methods on, however, making that a real function call runs into all kinds of issues with overloading.

I think the problem with this as it is, is that it looks too much like any other function call, when it’s not that at all. If there was some sort of other clue that it’s not a normal function call, that you could get just from reading the code, then I’d feel better about it. I honestly don’t mind the ugly current syntax that much, as it won’t be used as much either.

2 Likes
8

Ohh, and another thing, how well would the proposed scheme work with cases where you don’t have an object available. Both of these work just fine today.

def covariant[F[+_], A, B >: A](using F: Functor[F]): F[A] => F[B] = F.extension_map(_)(identity)
def curry[F[_], A, B](using F: Functor[F]): F[A] => (A => B) => F[B] = F.extension_map

If you’re calling the raw extension function, you probably don’t have an instance of the value on hand, so why make you call it as if you do?

You probably instead want to pass in parameters to the following parameter blocks. Not sure how, but would be great if the syntax placed focus on those instead.

9

Well, since it’s a keyword it will at least be highlighted in another color by your editor.

1 Like
10

What about a prefix keyword, like in (using myValue)?

object Functor {
  extension [F[_], A](fa: F[A]) def map[B](f: A => B)(using Functor[F]) = fa.map(f)
}

// prefix call
(extension Functor.map)(fa)(f)
3 Likes
11

Could someone point me to where the decision was made to add the extension_ prefix as I argued (in a talk, see scripting.pdf - Google Drive , p. 21) that being able to use the same name in both extension and raw style is desirable.

Edit : I found it Unified extension methods by odersky · Pull Request #9255 · lampepfl/dotty · GitHub

An extension method m is mapped to a normal method with name extension_m instead of just m. This is probably better anyway, since it avoids accidental calling of methods in the wrong mode

I don’t think that’s really a problem, is it ?

1 Like
12

| odersky SIP Committee member
October 27 |

  • | - |

Extension methods are shaping up to be a great addition to Scala. I have now used them in extensively in both the compiler and in my teaching, so far with few hassles. But there are two small issues that we might want to address before shipping Scala 3. (There’s also a larger one that we want to allow multiple type parameter sections, but this one is for later).

The first issue is the status of extension as a soft keyword. That caused a bit of confusion . For instance you can have a method named extension which you could call like this:

obj.extension(f)

But you can’t call it without a qualifier at the beginning of a statement like

extension(f)

since then the extension is classified as a keyword. See https://github.com/lampepfl/dotty/issues/10076 for more details.

We could try to put extra effort in disambiguating, but the question is whether it’s worth it. It might be better to make extension a hard keyword, like all other keywords that start a definition.

I think it’s worth trying harder to disambiguate. If you make it a hard keyword you’re breaking tons of code. Some code will have to be backticked but the less the better.

That would also offer a solution for the second problem: How to call an extension method as a normal method. That’s needed sometimes when we want to be explicit what method should be called. So far, the scheme is that an extension method

object obj:

  extension (x: A) def fun(y: B): C

is encoded as the method

object obj:

  def extension_fun(x: A)(y: B): C

So if fun is defined in object obj, you can call it as obj.extension_fun(a)(b) instead of the usual syntax

a.fun(b)

It turns out that this user-facing naming scheme is a bit awkward. How do you prevent people from writing extension_foo definitions themselves,

Why do we need to? There are a lot of things in Scala that are syntactic sugar, and we’re allowed to use the lower-level form to. Why not allow it?

You could always have a warning that recommends using the preferred syntax.

or how do you prevent the compiler from getting confused by them?

Why would it get confused?

Which name do you import? What about overriding rules? And so on. It would be simpler if we could avoid the name-mangling of fun. But then we need special syntax to express a direct call to it, since allowing both prefix and infix usages at the same time would lead to too much ambiguity. With extension as a soft keyword, I was unable to come up with a convincing syntax. But with extension available as a hard keyword there are new possibilities. Here’s one originally proposed by @smarter.

obj.extension(a).fun(b)

This follows the order in which things are declared: obj, then extension, then the first parameter, then fun, then the second parameter, so it should be easy to remember.

What is obj and what is a?

Can you give concrete examples of the ambiguity you’re trying to resolve?

13

It’s what was shown earlier:

object obj:
  extension (x: A) def fun(y: B): C

Then the directly called version of

a.fun(b)

would be

obj.extension(a).fun(b)
14

I’m in favor of the hard keyword and the second part of the proposal, but I have a question regarding binary/source compatibility for existing code that uses a def extension.
Will def `extension` : be possible (or any term named extension with the use of backticks)? Or alternatively, will it be possible to use @alpha("extension") to allow compatibility?

15

I just checked how many occurrences in the Scala 3 community build would have to be changed: It’s about a 100 lines out of 1.3M lines of code. So, still significant, but maybe it’s tolerable.

Here’s an alternative syntax that would also work with extension as a soft keyword.

a.(extension obj.fun)(b)

as the direct reference to obj.fun in the expression

a.fun(b)

I.e. what we would express today as

obj.extension_fun(a)(b)

The advantage of that syntax is that it allows local changes. The usual scenario where I need a fully qualified extension name is when the compiler for some reason is unable to pick up the right one by itself, or there is an obscure type error, where I want to be more explicit so that i can get a better error message. E.g. I start with

a.fun(b)

and get a type error that demands clarification. I could then just replace the fun part by its fully qualified form (extension obj.fun) and leave the operands a and b (which could be large) as they are,.

3 Likes
16

def `extension` should always be possible

2 Likes
17

It looks like we have gone full circle back to the original extension class syntax. It looks similar when defining it:

  extension(x: A) {
    def fun(y: B): C = ...
  }
  // vs
  implicit class extension(x: A) {
    def fun(y: B): C = ...
  }

and when using it explicitly:

obj.extension(a).fun(b) // in both cases

The main differences are:

  • The new extension form can have some defs left abstract, which is useful for type classes.

  • Implicit classes allocate a wrapper, but this is not the case if we make them extends AnyVal.

  • The new extension form has some annoying limitations: cannot have type parameter clauses in the defs, and cannot add private helpers or values alongside the defs.

The limitations of the new extension are made worse by the fact that they are solely due to the way it’s desugared, not something fundamental. They give worse functionality than implicit classes for no particularly good reason.

Instead, why not just introduce a new extension class construct, as a cleaner and more concise version of implicit class extends AnyVal, but which retains its advantages?

To get the ability to leave methods abstract, any basic encoding of virtual classes can do the trick.

trait Foo {
  extension class Ops(x: A) {
    def fun(y: B): C
  }
}
class Bar {
  extension class Ops(x: A) {
    def fun(y: B): C = ...
  }
}
6 Likes
18

My 2c:

  • make extension a hard keyword
  • mangle names like extension$fun
  • allow obj.extension(_).fun(_) such that it returns (A,B) => C
  • allow partial application such that obj.extension(a).fun returns B => C
  • obj.extension(a) (without the .fun) is a syntax error
  • allow import obj.extension.fun if that’s something you want to allow - notice no (a) obviously
2 Likes
19

Current syntax is import obj.fun, I think that’s better than introducing another way to use extension.

2 Likes
20

I don’t see why val x = obj.extension(a) can’t be made to work, though it would be a bit of work