Proposal to add Extension Methods to the language

Scala Improvement Process
62

One question is whether we want to enforce or imply? I.e.

def (x: Int) to (y: Int): Range

could possibly imply the @infix annotation just by the way it is written, no need to assert it separately?

1 Like
63

100% with @kai here as well. Fewer ways to do things are better. That was the goal of @infix. Now we have a nice way to denote that without even an annotation. Squeaky clean!

64

I’m with @kai on this too. Write it the way you intend to use it, no extra clunky stuff needed!

Edit: I should point out that I am very strongly opposed to the @infix idea at all. Having use-site infix calling is one of the biggest reasons why I think my Scala code is easier to read than my Rust code. (The other three are closures as parameters, better and better-named collections methods, and symbolic method names. Well, and the biggest one: I am still more adept at Scala than Rust.) Someone else deciding for me what some sliver of my code should look like just makes it hard to maintain a consistent style in my code.

However, if we’re going to have this, write-it-as-you-use-it is a big advantage.

5 Likes
65

Moreover, the @infix annotation could be dropped by instead treating extension methods on this.type as special:

def (self: this.type) + (that: Int): Int = ...
def (self: this.type) :: (elem: A): List[A] = ...

I don’t see how this would fit into the way extension methods are defined. That already means something else. Say you have:

class C:
  def (self: this.type) + (that: Int): Int = ...

and assume c: C.

Then c + 1 would typecheck under the rules we have, but only if c was a given instance, or else if there’s an import like

import c._

in scope. Otherwise, we cannot resolve + as an extension method.

A more radical approach would be to require all infix operators to be defined as regular extension methods. So to get + on C you’d need to write something like this:

class C:
  def add(y: Int): C = ...
object C:
  extension:
    def (x: C) + (y: Int) = x.add(y)

This would kill several birds with one stone.

  • No need for @infix, the extension method syntax already determines this.
  • No need for @alpha, since we can then require that all regular methods must be alphanumeric
  • No need for the special rule that operators ending in : are methods of their right operand, so their definitions have to be written backwards.

Infix methods would have to be written as individual extension methods. Collective does not work since then the problem of specifying infix or not would come back. We currently lack nice syntax to define such individual methods so that they are in the context scope of their receiver type. With current syntax we’'d express it like this:

given AnyRef:
  def (x: C) + (y: Int) = x.add(y)

which is definitely not nice. The example above used a putative extension syntax to mean the same thing. So we’d have extension to define a bunch of individual extension methods and make them available for implicit resolution, and extension on to define collective extensions, The second form would map to the first, and the first form would expand given instances like the one above.

Caveat: It remains to be seen whether all the rich structure of current infix operators could be expressed as extension methods without confusing type inference. That’s a rather big if.

5 Likes
66

It does not, that’s why such definitions would need to be handled ‘specially’.

The reason I propose special syntax instead of restricting operators to extension-only is to retain control of whether such a method is a direct instance method for the purpose of overrides, private access, performance, name resolution (direct methods can’t be overriden via implicit scope), type refinement of member methods, etc.

67

The reason I propose special syntax instead of restricting operators to extension-only is to retain control of whether such a method is a direct instance method for the purpose of overrides, private access, performance, name resolution (direct methods can’t be overriden via implicit scope), type refinement of member methods, etc.

I think one could get all this using plain alphanumeric methods and then just define an extension infix operator that forwards to the alphanumeric method. It could even be inline to make sure there’s no performance overhead. My main remaining doubt is what would happen if we map many overloaded variants of an infix operator to extension methods.

68

https://github.com/lampepfl/dotty/pull/8308 implements the scheme where @infix is implied for alphanumeric extension operators. I.e. in

def (x: T) min (y: T): T
def (x: T).max (y: T): T

min is treated as an @infix method, but max is not.

But it’s probably premature to drop @infix and @alpha.

First, there’s the issue of joint Scala 2/Scala 3 compilation. For the foreseeable future a single standard library will serve both Scala 2 and Scala 3. This means we need ways to express infixity and alphanumeric host names without doing large scale refactorings that map normal methods to extension methods which are not available in Scala 2.

Second, @infix is also used for types, where no ready replacement exists.

Third, the rationale for @alpha also applies to symbolic extension operators. They have to map to some host supported name, and they do profit from a searchable alias name. In a situation where a symbolic operator forwards to a regular method, the @alpha name can be the same as the method name. I.e.

class C:
  def add(y: Int): C = ...
object C:
  extension:
    @alpha("add") def (x: C) + (y: Int) = x.add(y)

I find that duplication acceptable.

So, there’s less scope than I hoped for to reduce the feature set of the language. But we have still learned a better way to teach operators. Let’s put the extension syntax front end center and de-emphasize operators as regular methods.

69

For the operator case, why not force that an alphanumeric operator exists, and then just have the infix operator be an alias to that.

E.g. something like this:

 class C:
  def add(y: Int): C = ...

object C:
  extension:
     def (_: C) + (_: Int) = add
2 Likes
70

Really elegant. Love it!

1 Like
71

Why don’t we cleanly separate these two mechanisms:

  • providing extension methods

  • bundling definitions that take the same parameters

The latter could have a syntax like:

with (given config: Config) {
  def foo(x: Int) = ... config ...
  def bar(y: String) = ... summon[Config] ...
}

which would expand into:

def foo(given config: Config)(x: Int) = ... config ...
def bar(given config: Config)(y: String) = ... summon[Config] ...

I think it would make the existing proposed features more regular, replacing several concepts by these two orthogonal ones. And it would allow new interesting capabilities too (such as the Config-passing pattern above).

Then, assuming we now mark extension methods by naming the extended parameter this, we can do grouped extension methods as follows, for example:

with [A](this: A) {
  def pairWith[B](that: B) = (this, that)
  def |>[B](f: A => B): B = f(this)
  ...
}

The extension syntax using this was proposed and discussed before. It has the advantage of already being well established in C#, and Kotlin also rebinds this for its version of extensions methods.

We would place such extension bundles inside the definitions of type class and their instances:

// type class declaration:
trait Foo[A] {
  with (this: A) {
    def + (that: A): A
    def * (that: A): A
  }
}

// type class instance:
case class Bar(x: Int)
object Bar {
  using as Foo[Bar] {
    with (this: Bar) {
      def + (that: Bar): Bar = Bar(this.x + that.x)
      def * (that: Bar): Bar = Bar(this.x * that.x)
    }
  }
  // or just:
  using as Foo[Bar] {
    def + (this: Bar)(that: Bar): Bar = Bar(this.x + that.x)
    def * (this: Bar)(that: Bar): Bar = Bar(this.x * that.x)
  }
}

Or we could have them be plain extensions not related to a type class:

case class Bar(x: Int)
object Bar {
  using {
    with [A](this: Bar) {
      def + (that: Bar): Bar = Bar(this.x + that.x)
      def * (that: Bar): Bar = Bar(this.x * that.x)
    }
  }
}
3 Likes
72

Grouped definition syntax seems entirely redundant in a language with classes - they fill the same role. It really only makes sense for extension methods I think

73

Classes cannot define methods (concrete and abstract) right into their enclosing scope. The grouping syntax also makes sense for the Config example, don’t you think?

74

In this example, does the A after with reference the A defined by Foo, or does it shadow it?

75

Oops, it was a typo. I’ve fixed it, thanks!

1 Like
76

I think the best way to provide this “simple case” extension method is to do it just as a syntactic sugar for a normal method call. The element in the last, singleton parameter list would be designated as the “this”. It would look like so:

case class A(...)
def f(b: B, c: C)(a: A): D = ...
a.f(b, c) // de-sugars to f(b, c)(a)

The current proposal is in some ways already close to this, that it’s just a syntactic sugar. But the advantage of this is that it makes the very same methods methods easily chainable, in both scenarios

  • using . as method application: sequence.map(plus1).filter(isEven)
  • using andThen (or >>>) as method composition: map(plus1) andThen filter(isEven)
class Sequence[A](...)
...
def map(f: A => B)(s: Sequence[A]): Sequence[B] = ???
def filter(p: A => Boolean)(s: Sequence[A]): Sequence[A] = ???
...
sequence.map(plus1)
// de-sugars to `map(f)(sequence)`, since
// * `Sequence` doesn't have a member `map` and
// * `map(...)(s: Sequence[A])` is in scope


sequence.map(plus1).filter(isEven)
// de-sugars to `filter(isEven)(map(f)(sequence))`

// and these same function can also be used on their own very nicely, especially with chaining
map(plus1) andThen filter(isEven)

Currently we have to nest method calls in parentheses, which is cumbersome, e.g.
filter(isEven)(map(f)(sequence))

I think this approach to having “extension methods” would be a good addition to Scala 3 and at the same time a great simplification to many use cases. But I’m not sure how big support this has, is there anyone who would like it too?

(I’ve written about this before here: 1, 2)

77

How would you handle currying with this approach?

def map[A, B](f: A => B)(s: Sequence[A]): Sequence[B] = ???

val strToInt = map[String, Int](_.toInt) _

Would you want to allow calls like someSequence.strToInt? This generalizes to methods with one parameter as well. Further, how about more than two parameter lists:

def foldl[A, B](f: (B, A) => B)(z: B)(s: Sequence[A]): B = ???

Do you want to de-sugar only projections that would match the last parameter list or all intermediate lists as well?

// using the second param. list as receiver
0.foldl(_ + _)(List(1,2,3))

// using the last param. list as receiver
List(1,2,3).foldl(_ + _)(0)

I think it would be incredibly complicated to integrate this syntactic form with currying in general. One way around this would be to only allow fully applied calls to be desugared.

I like the syntactic forms proposed by Martin and the one by @dhoepelman (suggesting to take inspiration from Kotlin’s objectively succinct syntax). They also play more nicely with a post by @Ichoran. I am playing with this concept in an own language and what I have found out is that providing such use-site syntax as seq from 0 until 10 where from and until are the literal parts of the application is best implemented by a 1:1 correspondence to the declaration. Any indirection will confuse the hek out of users looking at the signature of a function. It would quickly become unclear - unless well versed in the language - what applications would be syntactically correct given a distinct syntax for the function’s declaration. This would only further increase the barrier of entry to the language.

1 Like
78

Is there going to be an idiomatic do define certain operations as is the case with C++, where overloaded operators like the arithmetic operators should be defined outside a class and only access a class’ public members?

struct rational_t { int num; int denom; }
rational_t operator * (const rational_t& r1, const rational_t& r2) {
    return rational_t { r1.num * r2.num, r1.denom * r2.denom };
}

The thought arose when reading a post by @jimka2001:

For one, the compiler could force the receiver parameter of an extension method to be called this, though this is a bit inconsistent since it should not provide access to protected and private members of the respective class in my opinion. Then this would hold different access-privileges depending on its context. On the other hand, it would make more sense in a collective extension sceanrio. Instead of writing

extension listOps on [T](xs: List[T]) {
  def second = xs.tail.head
  def third: T = xs.tail.tail.head
}

one could write (somewhat) more succinctly something alon the lines of:

extension listOps on [T](List[T]) {
  def second = this.tail.head
  def third: T = this.tail.tail.head
}

where this could refer to listOps.type, typed a sealed subtype of List[T]. Another problem I am seeing is that this will be ambiguous in a setting where the extensions are defined, for instance, as part of a class or object body so I think it would probably be better to forbid usages of this in extension methods alltogether.

TL;DR:

  • Should a characteristical set of operators, for example, always be defined as extension methods?
  • What will this refer to in the context of an extension method?
79

Currying should work fine. But the de-sugaring, should really work on a parameter which is in a parameter list which is both singleton and last. Otherwise it would be too complicated. If the last parameter list is also the only one, it doesn’t cause any problems. Things like this should work:

def map[A, B](f: A => B)(s: Sequence[A]): Sequence[B] = ???
val strToInt = map[String, Int](_.toInt) // there shouldn't need be any `_`
someSequence.strToInt // strToInt is a valid candidate to extend someSequence, because it fits the criteria
def foldl[A, B](f: (B, A) => B)(z: B)(s: Sequence[A]): B = ???
// using the last param. list as receiver
List(1,2,3).foldl(_ + _)(0)

Exactly, that’s why I would want this to stick to just the last singleton parameter list.

80

Sounds like a good approach. But what are the advantages and disadvantages as opposed to the current solution in dotty? Or would you like to add your syntactic sugar on top of extensions as they exist in dotty? One problem I am seeing is how to implement collective extensions using your notation.

The current form is

extension listOps on [T](xs: List[T]) {
  def second = xs.tail.head
  def third: T = xs.tail.tail.head
}

But how would that look like with the receiver being the last parameter list? I think it could be confusing to mix the current definition for collective extension with your proposal, though it sounds cool to me otherwise.