Taking extensions further

I have been quite intrigued by Lean 4’s sections. They allow to write a list of type parameters once and then all definitions referring to the declared types are implicitly augmented with additional type parameters.

Can we add similar functionality to Scala? The most logical way to do so would be by way of generalizing the extension mechanism. So far, an extension header consists of

  • precisely one value parameter (the prefix)
  • zero or one type parameter clauses
  • zero or more using clauses

Example:

  extension[X](xs: List[X])(using Ctx[X])
    def combine(ys: List[X]) = ...
    def extract = ...

The defined extension methods can be invoked like this

  as.combine(bs)
  as.extract

or like this

  combine(as)(bs)
  extract(as)

Now, how about we make the value parameter of an extension header also optional? I.e: An extension header consists of

  • optionally, a value parameter (the prefix)
  • … (as before)

So what would an extension clause without extended parameter mean? First, all methods in it must be called in prefix form, as in the second example above, since there is no separate object on which they are defined.

Then, every type parameter in the extension header that gets referred from the signature of an extension method gets added as a type parameter to the method. Example:

extension [A, B]
  def f[C](x: A): C = ...
  def g(y: B): B = ...

gets expanded to

  def f[A][C](x: A): C = ...
  def f[B](y: B): B = ...

Note this is more restrictive than Lean’s sections, since we only add type parameters referred to from the signature, not parameters referred to from the body. If a type in the body refers to an extension type parameter that is not also referred to from the signature, this would be an error.,

Passing explicit type arguments to these methods is a bit problematic since we have to do an expansion in our head to figure out what type parameters a method gets in what order. I propose to avoid those mental acrobatics and require that all added type parameters of these extension methods are instantiated by name. I.e. f[SomeA](a) would be illegal. You’d have to write f[A = SomeA](a).

Moving on to using clauses, the idea would be that every using clause in an extension header is added as a using clause to all methods defined in the extension. The using clause appears in its natural place, before all other parameters of the method. Example:

extension (using c: Context)
  def f(x: c.A) = ...
  def g[T <: c.B](x: T) = 

gets expanded to

  def f(using c: Context)(x: c.A) = ...
  def g(using c: Context)[T <: c.B](x: T) = ...

Again we are different from Lean in that using clauses get added unconditionally, not just when they are referred to from the body of a method. This is done in the interest of binary stability. We do not want an implementation detail of a method to influence its binary signature.

That’s it. It’s syntactically a small change (just make one element of extensions optional). Its main benefit is DRYness. It’s a good tool to avoid repetitive type parameters and using clauses in method signatures without forcing methods to be factored out into separate objects.

I’d be interested to get your feedback on this.

7 Likes

On second thought, one correction. I think if we hang this on the extension mechanism then we should have the same rules for type parameters no matter whether there is an extended value parameter or not. So that means all type parameters in the extension head become type parameters of every extension method. The example I gave would then expand as follows:

  def f[A, B][C](x: A): C = ...
  def g[A, B](y: B): B = ...

Named type arguments are still essential to avoid long parameter lists of parameters that are maybe not even used in a method. But they would be optional now. So you could write f[SomeA, SomeB](a)
or alternatively and more clearly f[A = SomeA](a).

So in the end, it means there are no new rules except the fact that the extended value parameter is optional. We are already almost there!

1 Like

I am not such a fan of this change for a syntactic reason, as well as for a problem that arises with any mechanism that allows type clause reuse:

Syntax

In hindsight, I think the following was a mistake:

It is not at all intuitive the second form is allowed, and I think we should (at least) discourage it strongly, in the same way we discourage v1.+(v2)

This invocation freedom also forbids us from doing the following:

trait Show[A]:
  def show(a: A): String
  extension (a: A) def show = show(a) // Error: Double definition

Which would allow new Show[Int]{ def show(i: Int) = ??? } instead of overloading the extension method (which feels weird).

In general, I think we should keep this mechanism as far away from users as possible, so they can focus on “This is a construct that allows me to add methods to (instances of) an existing type”
And this proposal goes in the opposite direction !

Reusing inferrable parameters

I am also afraid (but not that much) it might produces things like this: (to get an alias R inside the signature)

extension [A, B, C, R <: Computation[A,B,C] >: Computation[A,B,C]]
  def foo(a: A, b: B, c: C): List[R] = ???

Which of course are a risk with any partial type inference, but being allowed to write it only once, instead of for every method, makes it much more attractive

Conclusion

All in all, I think we should always look for ways to reduce boilerplate and code duplication, even in method signature, but I am not on board with this particular solution

4 Likes

I second this opinion and I would argue that it complexifies the code without providing much value which is against Scala 3 direction (AFAIK).

I think not. I know many people don’t like it and want something like implicit classes instead. But the idea that extension methods are normal methods in the scope of the extension is the core of the design. And it’s shared by essentially all approaches to extension methods in other languages. Hiding it will not work. My proposal doubles down on that aspect of the design.

1 Like

Current extension methods

But I really like it !
I think it’s much cleaner than anything like implicit classes

And they should behave like methods, for their scoping rules and all

But for me this feels really counter intuitive:

extension (x: Int) def foo(y: Int)
foo(4)(5)

If the method had a different name, it would be obvious something special is going on:

extension$foo(4)(5)

This proposal

My main problem is communicating intent:

extension [A]
  def synthesise: A

It looks like we extend (instances of) A, but it’s actually just a normal type parameter

For me extension and section-like definitions need to be semantically split, even though they use the same mechanism underneath

I would propose to use a different keyword for the latter, for example with:

with [A]:
  def synthesise: A

It seems much clearer here that the intent is to define methods which use A as a type parameter. We can even have explicit parameters, and the aim remains clear:

with [A](a: A):
  def ident: A = a

ident[Int](4) // returns 4
4.ident // error: Not Found

And for methods that are intended to be used both way, we can use both keywords, something like extension with
This has an additional benefit: it checks that the LHS is compatible with extension methods, as section-like methods are more general

(I used the keyword with as an example, because it seems fitting, but there’s probably reasons we can’t use it, and/or a better keyword)

2 Likes

I agree that extension could be misleading, and that it’s kind of hard to see whether we are actually extending anything or just adding a bunch of type parameters and using clauses.

So, maybe same or similar syntax but another keyword? An obvious choice would be section, but we’d have to worry about conflicts with existing code (this holds for any word we choose, and held for extension as well, it’s just that section is fairly common.)

with is also an interesting choice and has the advantage that it’s already a keyword. An objection would be that it might be confusing that it’s used in both infix and prefix. E.g.

   class A extends B
   with [A](using Context):
      def foo...

vs

   class A extends B
   with C[A](using Context):
     def foo...

The two mean very different things.

Also, I would maybe draw the line at factoring out real value parameters like the x: A in your example. That look like a step too far to me.

These could be told apart with whitespace:
(Be careful, I swapped the order for flow)

  class A extends B
  with C[A](using Context):
    def foo...

vs

  class A extends B
    with [A](using Context):
      def foo...

Especially with more space:

  class A extends B

    with [A](using Context):
      def foo...

I don’t think this is ideal, maybe if we impose this blank line and encourage putting at least 1 definition between them, it’s not that bad ?

Also extends A with B is discouraged in favor of extends A, B, no ?

I’m not a fan of section, I don’t think it’s specific enough

I agree, that’s why I don’t like that extension can currently do it !

I think common would work:

common [T](using x: Ctx[T])
  def f...
  def g...

Slightly reminiscent of common blocks in Fortran.

2 Likes

The only potential issue with “common” is that it may be frequent in package names, like com.google.common. I’m not sure if that’s problematic?

common would be a soft modifier, like extension. It would be recognized only if it occurred where a definition could start and if it was followed by “(” or “[”. So I don’t think there’s a possible overlap with package names (or object names, for that matter).

1 Like

Think it would be good to get a more elaborate motivation section. I currently don’t really see where this would really be a benefit and whether it would be worth it to extend the language.

On another note. If we are going to improve extension I would rather see that you could also add (lazy) vals in exports and not just def. In that regard implicit class can do more than extension and I am not sure what would be the best way to migrate away from implicit class now for those use cases.

1 Like

I think lazy val can’t possibly work. How should that be implemented? Extension blocks have no runtime instances, so where would such a val be stored?

1 Like

While lazy vals are possible on an implicit class, AFAICT they’re pretty much pointless because a new instance of the class is created for every method call. Or every time you deference the lazy val. Unless you explicitly create and pass around instances of implicit classes.

1 Like

IMO I’d much more prefer to actually be able to write something like def foo[A][B]: (A, B) = ??? once we have that, we can talk about reducing noise where needed, but AFAIK we currently do not have that.

1 Like

Note that this syntax is sometimes necessary to avoid ambiguity: Confusing interaction between opaque types and extension methods · Issue #9880 · lampepfl/dotty · GitHub

This is being discussed!

Just use braces

class A extends B
with [A](using Context) {
  def foo...
}

vs

class A extends B {
  with C[A](using Context) {
    def foo...
  }
}

It’s almost like this is a solved problem :wink:

2 Likes

You might also consider using instead of with, since it is implicit (zing) in non-extension use cases I think:

using [A]:
  …

using (x: Int):
  …

using [A](x: Int):
  …

instead of

with [A]:
  …

with (using x: Int):
  …

with [A](using x: Int):
  …

Well, unless you ever want to add non-using parameters via a with/common clause.

I prefer with, but using could be less syntactically ambiguous and a little more terse.

From what I understand, non-using parameters are also considered in the original proposal.