That will create its own problems. The compiler is organized around types and their meaning (denotations). A type can be a reference to a term, which consists of a prefix type and a name.
Such a type can refer to several overloaded alternatives in a MultiDenotation. But imports don’t behave this way. An overloaded meaning of an import would instead consist of several types, each with their own prefix, but sharing a name. I believe that would turn overloading into an even bigger ball of mud interacting badly with everything else than it is now.
I think the deeper problem here is that extension methods are referenced via their name, but that’s really not enough to distinguish them. This is a “solved” problem in Scala 2, and the same solution seems like it would apply to Scala 3
Consider what we did in the past: with implicit classes providing extension methods, we would give each implicit class a long and useless name like HaoyisIntExtensions, in order to avoid conflicts. This is similar to what we did with many implicit definitions in Scala as well. In both cases, this made sense because the name was never meant to to be used in user code. We expect users to refer to implicits (both classes and definitions) by their type: either by invoking an implicit class via an extension method .foo on a value of the appropriate type, or by referencing an implicit definition via its type implicitly[Foo]. The actual definition names could be randomly-generated UUIDs like implicit def e66d1e3ab64411edafa10242ac120002 and they would serve the same purpose.
In Scala 3, implicit definitions were given their own syntax: you can write given Foo without assigning it a name, and you can import A.{given Foo} without referencing a name either. That is a significant improvement, and in-line with what people were already doing in the wild.
It seems the most elegant solution for extension methods would be to give them the same privilege that we give to given definitions and imports. While given definitions are fully referenced by their type, extension methods are referenced by their type and name. So it would follow that people should import extension methods in a similar fashion, with special syntax similar to what we gave given imports. Perhaps something like
import A.extension
import A.{extension Foo.bar}
To mirror the existing
import A.given
import A.{given TC}
Well, that could be, but isn’t this conceptually the cleanest solution? What, realistically, are the options? Especially in a way that is TASTY-compatible?
(1) Status quo: extension methods effectively have no namespace. They clobber each other willy-nilly like everything without a namespace always does, getting quadratically bad the more they are used. Solution: style guide says “never use extension methods instead of implicit classes when writing libraries”. For something that was supposed to replace implicit classes (and has considerably nicer syntax), this is pretty bad.
(2) Generate a synthetic erased implicit class, and use the regular implicit mechanism: when you see x.foo it is code for (new org.whomever.package$Filename$4(x)).foo , where this is the 4th extension in package package from file Filename. Or something like that. Anyway, then you have replaced the implicit class mechanism, which works, with a better-hidden variant of itself.
(3) Allow ad-hoc overloading based on imports at least for extension methods, if not everything. The compiler already figures out the options to generate a workable error message. So, yes, you have a new idea of overloaded dispatch where foo(x) might be bar.baz.foo(x) or bippy.quux.foo(x) depending on whether the type of x is knowable and resolves to one or the other. If you want to fit it into the existing MultiDenotation framework, then you would need to create a synthetic type equivalent to
trait SyntheticMultiDenotation$8 {
inline def foo(x: String): Int = bar.baz.foo(x)
inline def foo(f: Float): Long = bippy.quux.foo(x)
}
and then import of the two extension methods with foo would be equivalent to asking for the two existing foo methods to not be imported, and instead creating an instance of such a trait (or an equivalent object which forwards the extensions) and importing those foos instead. If the synthetic trait or object could not be created, then you’d get a compiler error.
(4) Allow more of the namespace to be specified. Very clunky, but at least you can still use method notation. So, x.foo doesn’t work, but x.baz.foo and x.quux.foo do (assuming no ambiguity on baz or quux). Unfortunately, this would introduce a source of name clashes–full paths vs. method names–that never used to exist. But since it would only be invoked when the class itself didn’t have the method, maybe it would be okay.
(5) Exploit the targetName mechanism for disambiguation, with part of the desugaring of x.foo being resolving foo(x) to stringOverloadFoo(x). I’m not sure that this is actually a solution, because to me it seems like all it does is allow overloads with types that are different when seen by Scala (which should be okay) but not Java (oops), so maybe the compiler can’t use this to invent overloads where none exist.
Maybe there are other options. But to me, it seems like ad-hoc overloading (for extension methods if nothing else) is conceptually the cleanest. There is one place it doesn’t work: when the extensions are defined in different files in the same namespace. Then there is literally no way to disambiguate the two. But that could be worked around with either @targetName or simply saying: sorry guys, but if you really want to do that, you’ve got to stuff it all in the same file. It’s your namespace, so you ought to have that much control, even if it isn’t the nicest organization. (We already have compromises to make in that regard with things in objects, though actually export reduces that problem substantially.)
5 Likes
I see your point, but I also think that the requirement for manually typing out given imports is one of the most pointless wastes of boilerplate left in Scala 3. All they manage to do for me, at least, is make certain random things break because ohhhhh, I forgot to import given! And they provide no clarity when I can’t figure out where a given is coming from. It still doesn’t tell you, just provides a very clunky selection mechanism that might allow you to narrow it down to one namespace. Or not, if it’s part of the automatic given resolutions.
Adding the same hassle-with-no-apparent-benefit to extension doesn’t seem like a win to me. But if other people think that import given is actually a fantastic feature, well, fine, I can get used to typing {given, extension, _} on everything instead of just {given, _}, if it means that I can add Durations and also add strings to Paths.
2 Likes
It would be the cleanest solution locally, at the price of making almost everything else more complicated both spec-wise and implementation-wise. So, I don’t think that’s realistic.
(3) Allow ad-hoc overloading based on imports at least for extension methods, if not everything
As far as I can see, that has a better chance of working. There might be a way to couple imports at the same nesting level with extension methods. I.e. when looking for an extension method, treat imports on the same nesting level as alternatives, check each one individually whether it’s applicable, and rank them according to specificity if there are several. That also complicates matters but here the complication is local. Basically, we need to create another mode of name resolution when searching for extension methods.
5 Likes
A “me too” to this experience report. @Ichoran’s description aligns with my experiences trying to apply extension methods. They are prone to name collisions. It can be a minefield.
Most recently, on an upgrade from 3.3.0-RC2 → 3.3.0-RC3, I started seeing new error msgs: Note that overloaded methods must all be defined in the same group of toplevel definitions. Previously working extension methods in two different compilation units stopped compiling.
1 Like
That’s odd. It didn’t work for me before. I have a file called OverloadedExtensions.scala in one project specifically as a workaround for this. (Originally created for 3.1!)
Maybe you had nominally but not actually circular dependencies that RC3 realized aren’t circular, so that it became possible to not compile everything simultaneously, and that change triggered the behavior?
Ok, I hadn’t seen that particular error before, so assumed it had introduced on 3.3.0-RC3, but apparently not. The 2nd, conflicting, overload was written while on 3.3.0-RC2 and initially compiled, so your incremental compilation explanation seems plausible.
See Support extension methods imported from different objects by odersky · Pull Request #17050 · lampepfl/dotty · GitHub for a possible solution.
8 Likes
There’s now a SIP for this change: SIP-54 - Multi-Source Extension Overloads. by sjrd · Pull Request #60 · scala/improvement-proposals · GitHub
If you have comments, please add them to this PR.
EDIT: Fixed link
3 Likes
Update: the proposal SIP-54 has been implemented and merged into the compiler. It is now available as an experimental feature that you can use as follows:
//> using scala 3.nightly
import scala.language.experimental.relaxedExtensionImports
object A:
extension (s: String)
def wow: Unit = println(s)
object B:
extension (i: Int)
def wow: Unit = println(i)
import A._
import B._
5.wow
"five".wow
(Based on this gist)
Note that you have to use a nightly build of the compiler.
Before we make it a stable feature, we would like to hear from the community if the current design and implementation work for you. In particular, we would be interested to know if there are still use cases where you used implicit classes in Scala 2 that you cannot migrate to extension methods in Scala 3.
8 Likes
It solves, for me, the use-case that could not be worked around, which was that extension method names for unrelated types from unrelated code bases would collide and basically prevent the extension method mechanism from scaling to a nontrivial degree of use. This was really critical, and it’s great that it now works as one would conceptually think it should!
It does not solve the irritating but workable-around problem that all overloaded methods must be defined in the same source file–and that extension methods are encoded as overloaded methods.
So if you have a mathematics library and you have, say, several different vector and matrix classes, all in the same namespace, and you want to be able to multiply a Double by each of them on the left–that is, x * m and x * v should work–you have to create a DoubleExtensions.scala file or somesuch, even though it would be much more natural to have the extension for each vector, matrix, etc., in the file that defined the data type.
This is awkward, but since there is a workaround, it just means that developing libraries is less pleasant than it could be.
As a user, one doesn’t notice that the library designer had to place the extensions all in the same file.
Note that there is no such restriction for extends AnyVal-based implicit class extensions. You can put them wherever it makes sense for them to belong, and they work.
1 Like
I experienced no such limitation. Can you give an example?
Sure. In my personal library, in addition to foreach, which returns unit, and tap, which is not monadic, I have a method I call use. It’s basically just x.use(f) = x.tap(_.foreach(f)).
So, in one file, Flow.scala, I add some things to Option. One would naturally define use there:
extension [A](option: Option[A])
inline def use(inline f: A => Unit): option.type =
(option: Option[A]) match
case Some(a) => f(a)
case _ =>
option
But there’s no reason you can’t define the same thing index-by-index on an array (where you tap the whole array, not the element), except array extensions are in Data.scala:
extension (ai: Array[Int])
inline def use(i: Int)(inline f: Int => Unit): ai.type = { f(ai(i)); ai }
These both live in the kse.flow namespace, but would be in two different files.
Except:
[error] -- [E161] Naming Error: /home/kerrr/Code/s3/kse3/flow/src/Flow.scala:607:13 ----
[error] 607 | inline def use(inline f: A => Unit): option.type =
[error] | ^
[error] |use is already defined as method use in /home/kerrr/Code/s3/kse3/flow/src/Data.scala
[error] |
[error] |Note that overloaded methods must all be defined in the same group of toplevel definitions
So, they all have to go into OverloadedExtensions.scala instead.
Just put the extension methods in some Ops objects in the different files and export them to your public source.
Flow.scala
package personalLib
object Flow:
object Ops:
extension [A](option: Option[A])
inline def use(inline f: A => Unit): option.type =
(option: Option[A]) match
case Some(a) => f(a)
case _ =>
option
Data.scala
package personalLib
object Data:
object Ops:
extension (ai: Array[Int])
inline def use(i: Int)(inline f: Int => Unit): ai.type = { f(ai(i)); ai }
PublicOps.scala
package personalLib
export Flow.Ops.*
export Data.Ops.*
Oh, somehow I missed that inline was kept with export, meaning that as long as the original extension is inline, the export is zero overhead. And you can always create the actual method with some weird name and have the extension be an inline call to that, so it’s completely general.
Consequently, this is a better way to do it. Thanks!
(I had been avoiding export as much as possible to avoid creating overly deep call stacks that can frustrate JIT-based inlining.)
Still a little awkward and still requires the extra file, but at least the logic stays where it makes sense.
2 Likes
One other thing I noticed is that this case still doesn’t work:
//> using scala 3.nightly
import scala.language.experimental.relaxedExtensionImports
object A:
extension (s: String)
def wow(x: String): Unit = println(x)
object B:
extension (s: String)
def wow(x: Int): Unit = println(x)
import A._
import B._
"five".wow("seven") // error
"five".wow(7) // error
It does work if wow are real overloads, both defined in the same object.
Yes that’s to be expected. The argument available for resolving extension overloads is the extension argument itself, which here is String in both cases.
Ok, but that’s still an unfortunate corner case that you can run into.
1 Like
Maybe I misunderstood, but this seems incorrect, as the following works:
extension (s: String)
def wow(x: String): Unit = println(x)
def wow(x: Int): Unit = println(x)
"five".wow("seven")
"five".wow(7)