Change shadowing mechanism of extension methods for on par implicit class behavior

We want in the future to remove implicit classes in favor of extension methods, but they have some unique behaviors that extension methods fail to imitate currently. One of these is the shadowing mechanism that affects extension methods like regular methods, but does not affect implicit classes because their behavior is derived from implicit conversions.
Example:

import scala.annotation.targetName
trait Foo[+T]

object Ops:
  extension (foo : Foo[Int])
    @targetName("bazInt")
    def baz : Unit = {}

import Ops.*

extension (foo : Foo[Double])
  @targetName("bazDouble")
  def baz : Unit = {}

val f = new Foo[Int] {}
f.baz //error

The error is:

value baz is not a member of Playground.Foo[Int].
An extension method was tried, but could not be fully constructed:

    Playground.baz(f)    failed with

        Found:    (Playground.f : Playground.Foo[Int])
        Required: Playground.Foo[Double]

In Scala 3 the name shadowing mechanism was disabled for implicits. I think it should be the same for extension methods, not only to match the implicit class behavior, but also because I believe it’s the more intuitive approach. It does not make sense to me that if I import a library that implements a + operator for a type-class via an extension method, then I cannot define my own + for a more specific type-class within the same namespace without having the library + for fallback/different behavior when I need it.

I’m not the only one. Past issues (#14777, #12680) show that others encountered the same problem, suggesting the same intuition.

I don’t know how many libraries were completely converted to Scala 3, but I think that it is likely we haven’t seen enough noise about this since many still use implicit classes for cross-compilation. If this behavior can be fixed, we should do so promptly, because it can lead to breaking changes as more code bases rely on extension methods.

2 Likes

It definitely seems odd for the compiler to even attempt applying an extension method that doesn’t match the type of the receiver. But I guess the resolution algorithm tries any extension with matching name.

Extension methods are really normal methods, so this behavior is not surprising. Here is the example reformulated with normal method surface syntax:

import scala.annotation.targetName
trait Foo[+T]

object Ops:
    @targetName("bazInt")
    def baz(foo : Foo[Int]) : Unit = {}

import Ops.*

@targetName("bazDouble")
def baz(foo : Foo[Double]) : Unit = {}

val f = new Foo[Int] {}
val _ = baz(f) //error

You get basically the same error:

-- [E007] Type Mismatch Error: test.scala:14:12 --------------------------------
14 |val _ = baz(f) //error
   |            ^
   |            Found:    (f : Foo[Int])
   |            Required: Foo[Double]
   |
   | longer explanation available when compiling with `-explain`
1 error found

So none of this is unexpected. It’s what extension methods are. Tweaking that behavior would be dubious at best.

From user perspective that’s an implementation detail and not a language detail, because extension methods are meant to replace the language feature called implicit classes. Implicit classes work for the example I gave above. How do you see we can replace implicit classes if nothing covers their current function?

1 Like

Although such behaviour may be anticipated from the language designers point of view, it still raises my eyebrow as non-guru regular user because all of these examples (where the methods are defined on equal footing) compile just fine:

  1. direct methods (scastie):
import scala.annotation.targetName
trait Foo[+T]

@targetName("bazInt")
def baz(foo : Foo[Int]) : Unit = {}

@targetName("bazDouble")
def baz(foo : Foo[Double]) : Unit = {}

val f = new Foo[Int] {}
val _ = baz(f) //works!
  1. methods via extensions (scastie):
import scala.annotation.targetName
trait Foo[+T]

extension (foo : Foo[Int])
  @targetName("bazInt")
  def baz : Unit = {}

extension (foo : Foo[Double])
  @targetName("bazDouble")
  def baz : Unit = {}

val f = new Foo[Int] {}
f.baz //works!
  1. imported methods via extensions (scastie):
import scala.annotation.targetName
trait Foo[+T]

object Ops:
  extension (foo : Foo[Int])
    @targetName("bazInt")
    def baz : Unit = {}

  extension (foo : Foo[Double])
    @targetName("bazDouble")
    def baz : Unit = {}

import Ops.*
val f = new Foo[Int] {}
f.baz //works!

It’s only when you mix the two the compiler says: ‘no’.

4 Likes

Shadowing is useful to remove the surprise factor from global definitions that get imported into the scope. But extensions are intuitively considered relational (to a specific type) and not global.

See also:

And the reference therein:

1 Like

@odersky I started writing a SIP to change this, but I encountered an odd inconsistent behavior trying to formulate the examples. This behavior shows that extension methods are shadowing or not depending on the path to the extended class, and is different from regular methods.

scastie

object ExtMethodsImportedFoo:
  object lib:
    class Foo[T]
    extension (foo: Foo[Int]) def baz: Int = 0

  import lib.*
  extension (foo: Foo[Double]) def baz(arg: Int): Int = arg

  val x = Foo[Int].baz //works ?!

object ExtMethodsGlobalFoo:
  class Foo[T]
  object lib:
    extension (foo: Foo[Int]) def baz: Int = 0

  import lib.*
  extension (foo: Foo[Double]) def baz(arg: Int): Int = arg

  val x = Foo[Int].baz //error ?!


object RegularMethodsImportedFoo:
  object lib:
    class Foo[T]
    def baz(foo: Foo[Int]): Int = 0

  import lib.*
  def baz(foo: Foo[Double])(arg: Int): Int = arg

  val x = baz(Foo[Int]) // error, as expected

object RegularMethodsGlobalFoo:
  class Foo[T]
  object lib:
    def baz(foo: Foo[Int]): Int = 0

  import lib.*
  def baz(foo: Foo[Double])(arg: Int): Int = arg

  val x = baz(Foo[Int]) // error, as expected
1 Like

This has been a massive substantial* annoyance in converting my libraries to Scala 3.

There are workarounds–define the functionality you want somewhere else and write a dispatcher where everything is in the same scope, for instance–but they are annoying.

Addendum: I think it’s a bit dubious that methods with distinct target names should ever shadow each other. If you can dispatch appropriately, it should work; if not, then you should get an error about ambiguous dispatch.

* (Decided it was better to keep a more neutral perspective.)

1 Like

The example that surprisingly //works ?! works because you put the extension in implicit scope.

In Scala 2, you’d look for implicits (to convert things as in implicit classes or to supply implicit args) by looking first in lexical scope, then in implicit scope.

In Scala 3, you’re looking for an extension m(e) for the e.m you wrote. (No conversion is required or desired.) You still look first in lexical scope, which is how “regular methods” work, then in implicit scope. In that sense, nothing has changed. (Additionally, the extension can be squirreled away in a given instance.)

The one change is that packages no longer form part of the implicit scope of types they package. That is point 3 at “changes in implicit resolution”.

If I still agree with my comment on the linked ticket, maybe it’s because I haven’t had a chance to use extension methods in anger, or more anger than usual when working with computers.

I think it’s a matter of expectation (and documentation). We’ve lived in the magic kingdom for so long, we expect magic.

I like that an extension method is “just a method” plus a kind of rewrite mechanism.

I like the elegance of putting them into the hat of implicit scope and pulling out the right one. That is just the improved overload resolution at work. Knowing the trick doesn’t make it less magical.

By “documentation”, I think I meant “show the idioms for the use cases.”

1 Like

I don’t understand the addendum, namely, how target names could change the behavior of ordinary identifiers in the language. Would names with targets get a manufactured name, like anonymous givens? Could I continue to refer to f until it conflicted with another f, but I would be able to use its manufactured name? Until now, I thought of the target name as a hidden implementation detail.

@som-snytt - I was hoping that manufactured names would get around the arbitrary limitations of overloaded methods.

Under the current rules, for instance, mathematical types are incredibly annoying to use with extension methods.

For example, suppose you have a Vec class. You want to scale your vector, so you add a method

  def *(scale: Double): Vec = ???

Okay, now you can write v * 5. Great! But what about 5 * v? No worries,

extension (scale: Double)
  def *(v: Vec): Vec = v * scale

Great! Now, let’s suppose we want to represent a value with some error. We’ll call it Approx. We want to be able to scale it up and down.

  def *(scale: Double): Approx = ???

Now a * 5 works. Yay! And of course

extension (scale: Double)
  def *(a: Approx): Approx = a * scale

And BOOM! Now neither 5 * v nor 5 * a work.

So, OKAY, okay, you either put things into different namespaces (now they work again), or you factor out the extension of double into its own file and

// ImplementationRestrictedDoubleExtensions.scala
extension (scale: Double) {
  def *(v: Vec): Vec = v * scale
  def *(a: Approx): Approx = a * scale
}

Now it works! Well, almost. Not if Approx and Vec are actually both opaque types over (Double, Double). But then @targetName to the rescue, and it works again.

The shadowing problem described here is just one piece of this larger issue–method overloading and shadowing rules clobber the ability of extension methods to naturally extend things. It doesn’t “just work”; there are finicky rules that you constantly have to keep in mind, some of which won’t even show up at compile-time because the ambiguity doesn’t appear until the use-site (hope you wrote good unit tests!).

But, of course, since the implicit class method could resolve all this stuff completely fine, the issue is a manufactured one: it’s not that it isn’t 1000% clear what to do, it’s that the encoding (with particular method names) and limitations on overloaded methods end up getting in the way. With a target name, the limitation goes away, so everything could work naturally.

Except, right now, it only sometimes does.

Does this mean that you should be able to refer to the target name? I don’t know for sure. I tend to think of it as an implementation detail, so I’d argue no–but it’s an implementation detail that you stick in to avoid other implementation details mucking up your ability to write code in the natural way.

5 Likes

targetName is just to overcome ambiguity after type eraser. It has no relevance to this proposal.

Thanks. I read or re-read the topics about weaning off implicit conversions or limiting them.

I lost internet connection and the precise links, but the gist I understand is that implicit adaptation of r.m(x) when the arg does not typecheck is not included in search for extensions.

In the previous topics, this came up late in the thread, but was never foregrounded. (Someone says augmenting primitives is why they preferred Scala to Kotlin.)

There is a late comment from odersky that the frontend can be made aware of the backend targetname (with an investment of work).

My expectation was that once extensions consulted implicit scope, then overloading resolution picks results. My misunderstanding was that extensions only cover selection r.m and not the case where there is a member m but the application does not typecheck.

My other observation is that the topics are clear that unconstrained conversions are undesirable. Whether the magnet pattern (with dispatch, which may be what you meant by forwarding) is the implementation, the goal is to create a bottleneck of expression where a library must say which conversions are allowed (for example, extensions on Double). Is it generally agreed that that is a worth goal?

One could imagine a few solutions: as proposed by you and soronpo, use target names for extensions (after failing on nominal names); add arg adaptation to trigger for extension methods; show an example where receiver is augmented to take a magnet pattern arg (if that is feasible).