Change shadowing mechanism of extension methods for on par implicit class behavior

Language Design
1

We want in the future to remove implicit classes in favor of extension methods, but they have some unique behaviors that extension methods fail to imitate currently. One of these is the shadowing mechanism that affects extension methods like regular methods, but does not affect implicit classes because their behavior is derived from implicit conversions.
Example:

import scala.annotation.targetName
trait Foo[+T]

object Ops:
  extension (foo : Foo[Int])
    @targetName("bazInt")
    def baz : Unit = {}

import Ops.*

extension (foo : Foo[Double])
  @targetName("bazDouble")
  def baz : Unit = {}

val f = new Foo[Int] {}
f.baz //error

The error is:

value baz is not a member of Playground.Foo[Int].
An extension method was tried, but could not be fully constructed:

    Playground.baz(f)    failed with

        Found:    (Playground.f : Playground.Foo[Int])
        Required: Playground.Foo[Double]

In Scala 3 the name shadowing mechanism was disabled for implicits. I think it should be the same for extension methods, not only to match the implicit class behavior, but also because I believe it’s the more intuitive approach. It does not make sense to me that if I import a library that implements a + operator for a type-class via an extension method, then I cannot define my own + for a more specific type-class within the same namespace without having the library + for fallback/different behavior when I need it.

I’m not the only one. Past issues (#14777, #12680) show that others encountered the same problem, suggesting the same intuition.

I don’t know how many libraries were completely converted to Scala 3, but I think that it is likely we haven’t seen enough noise about this since many still use implicit classes for cross-compilation. If this behavior can be fixed, we should do so promptly, because it can lead to breaking changes as more code bases rely on extension methods.

4 Likes
2

It definitely seems odd for the compiler to even attempt applying an extension method that doesn’t match the type of the receiver. But I guess the resolution algorithm tries any extension with matching name.

3

Extension methods are really normal methods, so this behavior is not surprising. Here is the example reformulated with normal method surface syntax:

import scala.annotation.targetName
trait Foo[+T]

object Ops:
    @targetName("bazInt")
    def baz(foo : Foo[Int]) : Unit = {}

import Ops.*

@targetName("bazDouble")
def baz(foo : Foo[Double]) : Unit = {}

val f = new Foo[Int] {}
val _ = baz(f) //error

You get basically the same error:

-- [E007] Type Mismatch Error: test.scala:14:12 --------------------------------
14 |val _ = baz(f) //error
   |            ^
   |            Found:    (f : Foo[Int])
   |            Required: Foo[Double]
   |
   | longer explanation available when compiling with `-explain`
1 error found

So none of this is unexpected. It’s what extension methods are. Tweaking that behavior would be dubious at best.

4

From user perspective that’s an implementation detail and not a language detail, because extension methods are meant to replace the language feature called implicit classes. Implicit classes work for the example I gave above. How do you see we can replace implicit classes if nothing covers their current function?

2 Likes
5

Although such behaviour may be anticipated from the language designers point of view, it still raises my eyebrow as non-guru regular user because all of these examples (where the methods are defined on equal footing) compile just fine:

  1. direct methods (scastie):
import scala.annotation.targetName
trait Foo[+T]

@targetName("bazInt")
def baz(foo : Foo[Int]) : Unit = {}

@targetName("bazDouble")
def baz(foo : Foo[Double]) : Unit = {}

val f = new Foo[Int] {}
val _ = baz(f) //works!
  1. methods via extensions (scastie):
import scala.annotation.targetName
trait Foo[+T]

extension (foo : Foo[Int])
  @targetName("bazInt")
  def baz : Unit = {}

extension (foo : Foo[Double])
  @targetName("bazDouble")
  def baz : Unit = {}

val f = new Foo[Int] {}
f.baz //works!
  1. imported methods via extensions (scastie):
import scala.annotation.targetName
trait Foo[+T]

object Ops:
  extension (foo : Foo[Int])
    @targetName("bazInt")
    def baz : Unit = {}

  extension (foo : Foo[Double])
    @targetName("bazDouble")
    def baz : Unit = {}

import Ops.*
val f = new Foo[Int] {}
f.baz //works!

It’s only when you mix the two the compiler says: ‘no’.

5 Likes
6

Shadowing is useful to remove the surprise factor from global definitions that get imported into the scope. But extensions are intuitively considered relational (to a specific type) and not global.

7

See also:

And the reference therein:

1 Like
8

@odersky I started writing a SIP to change this, but I encountered an odd inconsistent behavior trying to formulate the examples. This behavior shows that extension methods are shadowing or not depending on the path to the extended class, and is different from regular methods.

scastie

object ExtMethodsImportedFoo:
  object lib:
    class Foo[T]
    extension (foo: Foo[Int]) def baz: Int = 0

  import lib.*
  extension (foo: Foo[Double]) def baz(arg: Int): Int = arg

  val x = Foo[Int].baz //works ?!

object ExtMethodsGlobalFoo:
  class Foo[T]
  object lib:
    extension (foo: Foo[Int]) def baz: Int = 0

  import lib.*
  extension (foo: Foo[Double]) def baz(arg: Int): Int = arg

  val x = Foo[Int].baz //error ?!


object RegularMethodsImportedFoo:
  object lib:
    class Foo[T]
    def baz(foo: Foo[Int]): Int = 0

  import lib.*
  def baz(foo: Foo[Double])(arg: Int): Int = arg

  val x = baz(Foo[Int]) // error, as expected

object RegularMethodsGlobalFoo:
  class Foo[T]
  object lib:
    def baz(foo: Foo[Int]): Int = 0

  import lib.*
  def baz(foo: Foo[Double])(arg: Int): Int = arg

  val x = baz(Foo[Int]) // error, as expected
1 Like
9

This has been a massive substantial* annoyance in converting my libraries to Scala 3.

There are workarounds–define the functionality you want somewhere else and write a dispatcher where everything is in the same scope, for instance–but they are annoying.

Addendum: I think it’s a bit dubious that methods with distinct target names should ever shadow each other. If you can dispatch appropriately, it should work; if not, then you should get an error about ambiguous dispatch.

* (Decided it was better to keep a more neutral perspective.)

2 Likes
10

The example that surprisingly //works ?! works because you put the extension in implicit scope.

In Scala 2, you’d look for implicits (to convert things as in implicit classes or to supply implicit args) by looking first in lexical scope, then in implicit scope.

In Scala 3, you’re looking for an extension m(e) for the e.m you wrote. (No conversion is required or desired.) You still look first in lexical scope, which is how “regular methods” work, then in implicit scope. In that sense, nothing has changed. (Additionally, the extension can be squirreled away in a given instance.)

The one change is that packages no longer form part of the implicit scope of types they package. That is point 3 at “changes in implicit resolution”.

If I still agree with my comment on the linked ticket, maybe it’s because I haven’t had a chance to use extension methods in anger, or more anger than usual when working with computers.

I think it’s a matter of expectation (and documentation). We’ve lived in the magic kingdom for so long, we expect magic.

I like that an extension method is “just a method” plus a kind of rewrite mechanism.

I like the elegance of putting them into the hat of implicit scope and pulling out the right one. That is just the improved overload resolution at work. Knowing the trick doesn’t make it less magical.

By “documentation”, I think I meant “show the idioms for the use cases.”

1 Like
11

I don’t understand the addendum, namely, how target names could change the behavior of ordinary identifiers in the language. Would names with targets get a manufactured name, like anonymous givens? Could I continue to refer to f until it conflicted with another f, but I would be able to use its manufactured name? Until now, I thought of the target name as a hidden implementation detail.

12

@som-snytt - I was hoping that manufactured names would get around the arbitrary limitations of overloaded methods.

Under the current rules, for instance, mathematical types are incredibly annoying to use with extension methods.

For example, suppose you have a Vec class. You want to scale your vector, so you add a method

  def *(scale: Double): Vec = ???

Okay, now you can write v * 5. Great! But what about 5 * v? No worries,

extension (scale: Double)
  def *(v: Vec): Vec = v * scale

Great! Now, let’s suppose we want to represent a value with some error. We’ll call it Approx. We want to be able to scale it up and down.

  def *(scale: Double): Approx = ???

Now a * 5 works. Yay! And of course

extension (scale: Double)
  def *(a: Approx): Approx = a * scale

And BOOM! Now neither 5 * v nor 5 * a work.

So, OKAY, okay, you either put things into different namespaces (now they work again), or you factor out the extension of double into its own file and

// ImplementationRestrictedDoubleExtensions.scala
extension (scale: Double) {
  def *(v: Vec): Vec = v * scale
  def *(a: Approx): Approx = a * scale
}

Now it works! Well, almost. Not if Approx and Vec are actually both opaque types over (Double, Double). But then @targetName to the rescue, and it works again.

The shadowing problem described here is just one piece of this larger issue–method overloading and shadowing rules clobber the ability of extension methods to naturally extend things. It doesn’t “just work”; there are finicky rules that you constantly have to keep in mind, some of which won’t even show up at compile-time because the ambiguity doesn’t appear until the use-site (hope you wrote good unit tests!).

But, of course, since the implicit class method could resolve all this stuff completely fine, the issue is a manufactured one: it’s not that it isn’t 1000% clear what to do, it’s that the encoding (with particular method names) and limitations on overloaded methods end up getting in the way. With a target name, the limitation goes away, so everything could work naturally.

Except, right now, it only sometimes does.

Does this mean that you should be able to refer to the target name? I don’t know for sure. I tend to think of it as an implementation detail, so I’d argue no–but it’s an implementation detail that you stick in to avoid other implementation details mucking up your ability to write code in the natural way.

7 Likes
13

targetName is just to overcome ambiguity after type eraser. It has no relevance to this proposal.

14

Thanks. I read or re-read the topics about weaning off implicit conversions or limiting them.

I lost internet connection and the precise links, but the gist I understand is that implicit adaptation of r.m(x) when the arg does not typecheck is not included in search for extensions.

In the previous topics, this came up late in the thread, but was never foregrounded. (Someone says augmenting primitives is why they preferred Scala to Kotlin.)

There is a late comment from odersky that the frontend can be made aware of the backend targetname (with an investment of work).

My expectation was that once extensions consulted implicit scope, then overloading resolution picks results. My misunderstanding was that extensions only cover selection r.m and not the case where there is a member m but the application does not typecheck.

My other observation is that the topics are clear that unconstrained conversions are undesirable. Whether the magnet pattern (with dispatch, which may be what you meant by forwarding) is the implementation, the goal is to create a bottleneck of expression where a library must say which conversions are allowed (for example, extensions on Double). Is it generally agreed that that is a worth goal?

One could imagine a few solutions: as proposed by you and soronpo, use target names for extensions (after failing on nominal names); add arg adaptation to trigger for extension methods; show an example where receiver is augmented to take a magnet pattern arg (if that is feasible).

15

Unfortunately, this violates the Scala 3 principle of focusing on intent rather than mechanism.

Extension methods promise to be like methods, which dispatch based on the type of their first argument (which appears to the left of the method name, on the other side of a dot). For instance, imagine this scenario:

import java.time._
Option(foo).get

[error]  |   Reference to get is ambiguous,
[error]  |   it is both imported by import scala.{Option}
[error]  |   and imported subsequently by import java.time.{Duration}

You can’t conveniently build anything of scale with this–you’re back in the C world of namespace_my_fn_name_hungariantypenotation(foo).

If the intent of extension methods is to act like methods, they need to act like methods. As it is, they’re a booby-trap for unwary developers who see a nice-looking feature and use it according to natural intent.

Extension methods as conceived are fine as a way to define functionality on opaque types (where the extensions are all given in the companion object, and thereby aren’t considered ambiguous).

But as their use expands to actually extend existing classes, the propensity for collisions increases quadratically with scale.

For example, do you want to use / to divide java.time.Duration and also to add to a java.nio.file.Path? Not with extension methods (unless you restructure your code as Divisibles.scala which contains every instance of everything that you might want to divide…and then…perish the thought that anyone else ever wants to divide any other pre-existing thing that you didn’t think of!).

After having converted my personal library to use extension, I’m having to go back and rip out extension and put implicit class back to get around increasing numbers of problems. I cannot in good conscience recommend that anyone use extension to extend anything in a library that might be used by anyone else.

It’s very unfortunate, because aside from being unscalable due to focus on mechanism rather than intent, it’s a really nice way to express intent and to build enabling functionality.

I don’t have the bandwidth now to write a SIP (or change the compiler), if that’s what it would take. I do recommend changing all the documentation to warn everyone upfront not to use this mechanism if it might be consumed by anyone else, until/unless a fix is forthcoming.

9 Likes
16

You can make a PRE-SIP forum post with a few of the motivating examples that lead to namespace collisions - I think an actionable concrete proposal is to change resolution of raw identifiers, e.g. foo(a) to only consider extension methods last, and warn when an extension method is resolved for a raw identifier. Then in a future version to not resolve extension methods unless explicitly selected with .

This proposal would still allow extension methods to be used as “normal methods”, or as extension methods, as long as they have some explicit prefix.

17

That won’t help for the problem @Ichoran was describing, unfortunately. The problem is the extension method resolution mechanism. If x.foo is not found, the compiler tries foo(x) instead. And that fails with an ambiguity error if there are several conflicting imports.

18

I would suggest to explore a proposal that considers conflicting term imports to be overloaded instead.

1 Like
19

That will create its own problems. The compiler is organized around types and their meaning (denotations). A type can be a reference to a term, which consists of a prefix type and a name.
Such a type can refer to several overloaded alternatives in a MultiDenotation. But imports don’t behave this way. An overloaded meaning of an import would instead consist of several types, each with their own prefix, but sharing a name. I believe that would turn overloading into an even bigger ball of mud interacting badly with everything else than it is now.

20

I think the deeper problem here is that extension methods are referenced via their name, but that’s really not enough to distinguish them. This is a “solved” problem in Scala 2, and the same solution seems like it would apply to Scala 3

Consider what we did in the past: with implicit classes providing extension methods, we would give each implicit class a long and useless name like HaoyisIntExtensions, in order to avoid conflicts. This is similar to what we did with many implicit definitions in Scala as well. In both cases, this made sense because the name was never meant to to be used in user code. We expect users to refer to implicits (both classes and definitions) by their type: either by invoking an implicit class via an extension method .foo on a value of the appropriate type, or by referencing an implicit definition via its type implicitly[Foo]. The actual definition names could be randomly-generated UUIDs like implicit def e66d1e3ab64411edafa10242ac120002 and they would serve the same purpose.

In Scala 3, implicit definitions were given their own syntax: you can write given Foo without assigning it a name, and you can import A.{given Foo} without referencing a name either. That is a significant improvement, and in-line with what people were already doing in the wild.

It seems the most elegant solution for extension methods would be to give them the same privilege that we give to given definitions and imports. While given definitions are fully referenced by their type, extension methods are referenced by their type and name. So it would follow that people should import extension methods in a similar fashion, with special syntax similar to what we gave given imports. Perhaps something like

import A.extension
import A.{extension Foo.bar}

To mirror the existing

import A.given
import A.{given TC}