Updated Proposal: Revisiting Implicits

Extension Methods

I think some more elaborate example would be helpful to get the feeling, like arithmetic progression sum.

// ??? - what givens, extensions, etc are needed here to make below code compile without changes?

def arithmeticProgressionSum[T: Numeric](firstTerm: T, step: T, termsNum: Int): T =
  termsNum * (2 * firstTerm + (termsNum - 1) * step) / 2

val sum = arithmeticProgressionSum(new java.math.BigInteger("3453"), new java.math.BigInteger("245"), 745)
println(sum)

Given Instances

The docs give this example:

given (given outer: Context): Context = outer.withOwner(currentOwner)

This makes anything about existing implicits look like the paragon of clarity in comparison. It looks like the “Buffalo buffalo buffalo” puzzles sometimes given in e.g. philosophy of language classes.

Something should change to make this impossible. For instance, making give provide things and given ask for them would fit naturally into common language usage instead of admitting mindboggling things that read like “given given Foo Foo”.

The syntax of given instances and aliases are both weird special cases. Nothing else in the language works like that. I understand what the point of each is, but the instances look like a weird hybrid of what you’d do for a def and what you’d do for a class, making it anything but clear what the point is. That regular instances use =, e.g. val five = 5, while given instances do not, is especially befuddling. (That object O does not use = is already befuddling enough, but there at least since it isn’t parameterized you can reason that it both declares the singleton class type and automatically gives it a unique name–that is, it’s a sugar for sealed abstract class O.type { ... }; lazy val O: O.type = new O.type {}.)

Since given instances can be parameterized, and aliases can alias to defs, it’s also quite a bit of work to puzzle out what is gained by having both constructs.

Given Clauses

Given clauses are pretty nice, though I found the foo(x) given y syntax vastly superior visually in simple cases (but untenable in complex cases). However, the document doesn’t explain whether the (implicit thing: Thing = Companion.defaultThing) pattern works any longer. It looks like no, given the grammar. It also isn’t clear to me whether def f(given foo: Foo, i: Int) can be called as f(5). I don’t think I missed the example?

It’s not clear why they are called “given clauses”. Parameters aren’t called clauses in other places.

It’s not clear how the values are filled in with multiple relevant givens in scope. Presumably something like implicit resolution, which right now forces awkward tricks like creating superfluous traits just to drive resolution? Or…? Should be spelled out.

Context Bounds

Looks fine! Seems pretty clear to me. Also doesn’t seem to have changed.

Given Imports

This is a solution to only half the problem (actually, only half of half of the problem). Not knowing that import mylib._ brought in implicits is fixed by givens having to be import mylib.{given, _}. So that’s something. But you still don’t actually know what you’re getting from there. So what you’re required to do doesn’t actually solve even this half of the problem; you have to take the explicit-type option and add all that boilerplate.

But import mylib._ not working because you forgot to bring in implicits is even worse because now they can’t conveniently be inside mylib any more. This suggests that to recover parity with the old way, best practice will be to spam import mylib{given. _} everywhere, whether or not you need it…which then gives you the first problem right back again.

Instead, what we need is a less awkward way to specify precedence, along with deprecation messages. So you could

import mylib._

and if you’d get the implicits (givens) that would make things work, but if you marked them all deprecated, you could have a message like, “This invocation uses a default given from mylib. Please import mylib.fast._ to select the fast versions of these implicits, or import mylib.accurate_ to select the accurate ones.”

Maybe it’s too late for a change like this, but I think the new version is strictly worse than the old version as it stands.

It seems like a tooling problem is being hoisted onto the programmer in an unwelcome way. I’d rather the compiler do more for me e.g. by suggesting places to find givens that I need, than force me to manually do that kind of work myself.

Finally, it’s not clear that the givens can be brought in by type with the full types that could appear in the given alias or instance.

Extension Methods

I love the idea, but the syntax is rather ugly. I wouldn’t assume that I’ll just get used to the syntax and eventually love it, because after like a decade of experience with C++ I still find the lambdas impossible, and after a couple of years of Rust I still find |x| bar(x) lambda syntax way less readable than Scala’s x => bar(x).

I’d prefer something like

extend (c: Circle) with {
  def circumference: Double = c.radius * math.Pi
}

which could be shortened to

extend (c: Circle) with circumference: Double = c.radius + math.Pi

in the case where you wanted a one-liner.

If you complain that paring this down to be more refined and elegant will just get you back at the current extension syntax, I plead guilty. Yes, it does. Sometimes the extra fluff helps. I think this is one of those times.

Implicit Conversions

This is so awesome! (Note: still need clarification on how givens are resolved when potentially conflicting.)

Implicit By-Name Parameters

This seems like kind of a weird and ugly hack. Isn’t there some better way to get around the problem so we don’t have to worry about it? Like, what’s wrong with always having the by-name behavior to avoid circularity?

It’s especially weird because by-name suggest lazy runtime behavior, but here it’s being used to drive compile-time differences. This is a fundamental shift in how by-name parameters work.

Relationship with Scala 2 Implicits

Seems reasonable, save for not mentioning how to migrate (implicit thing: Thing = defaultThing), as I brought up earlier.

11 Likes

This was indeed more a “demonstrate the limits of what this can do” example than a piece of recommended code. I propose we remove the example.

What it did is that it allows us to define an inner given using an outer one. As @tarsa asks, it’s not obvious why this should work. In fact it works because trying to resolve the given with itself would produce an implicit divergence. In Scala-3 is a recoverable error, so the outer given would be searched next and that one would succeed. In current Scala the same trick would not work since there divergence errors are fatal. But once can make it work rather perversely by writing this:

implicit val ctx = implicitly[Context].withOwner(currentOwner)

It’s essential that the type of ctx is not given explicitly, since that prevents ctx from being elaborated on its rhs. To my surprise there are actually widely used libraries out there that rely on this trick. But all this is rather too specialized and deep for a casual discussion and it has nothing to do with anonymous instances. So we should just drop the example.

To clarify: I don’t think

given (given Foo): Bar

is a clear and obvious thing either. Having two identical keywords to mean two different features, used in close proximity where they force rapid mental shifts in interpretation, is not good. The given clause has the virtue of being signaled well in advance by def and method name and stuff; here it requires an instant mental flip. This is obvious:

give Bar given Foo

I’m not saying that’s what the syntax should be, necessarily, just that it helps the reader out.

Also, removing the example of given (given outer: Context): Context doesn’t prevent it from being used, or for the problem it illustrates to be not-a-problem. It just seemed particularly egregious to me.

7 Likes

Thanks @Ichoran for the write up. I’d like to second the concerns about the syntax of given instances. I have already explained here why I think we should just have the “given aliases” syntax (like for val and def definitions).

1 Like

EDIT: this is very similar to the link @julienrf posted just above, I didn’t see it until after I posted this message!

I agree that the syntax for given instances looks like a hybrid thing, by contract given aliases seem more natural to me. So could we drop given instances? This would be one less thing to teach! Instead of:

trait Foo { def foo: Int }
given Foo {
  def foo: Int = 1
}

we can use a given alias:

given Foo = new Foo {
  def foo: Int = 1
}

The former translates to an object, the latter translate to a lazy val and an anonymous class, those should be semantically equivalent.

And instead of:

trait Bar[T](x: T) { def foo: T }
given (given x: Int): Bar[Int](x) {
  def bar: Int = 1
}

we can also use a given alias:

given (given x: Int): Bar[Int] = new Bar[Int](x) {
  def bar: Int = 1
}

The former translates to a class and a def, the latter also translates to a class and a def and the semantics are again equivalent as far as I can tell.

Of course, the main issue here is the repetition, so let’s invent even more syntax (sorry!):

given Foo = new {
  def foo: Int = 1
}
given (given x: Int): Bar[Int] = new(x) {
  def bar: Int = 1
}

The unparameterized version actually already works in Dotty, the parameterized one has been suggested before by @propensive in a more general context, and @odersky gave some reasons it might not be worth it in Expunging `new` from scala 3, but I think it’s worth revisiting in the context of givens: it it allows us to drop given instances that would be a huge simplification and more than make up the extra complexity it introduces.

2 Likes

The (non-alias) given instance syntax actually looks better once we allow a with:

given Monoid[String] with
  def (x: String) combine (y: String): String = x.concat(y)
  def unit: String = ""

given Monoid[Int] with
  def (x: Int) combine (y: Int): Int = x + y
  def unit: Int = 0

BYOB (“bring your own braces if you like them”). I don’t like the alias syntax with a new. I’d like to get away from using new – it’s connotations are far too operational! Writing

given Monoid[String] = new {
  def (x: String) combine (y: String): String = x.concat(y)
  def unit: String = ""
}

emphasises the point that we create a new object to be a monoid. But nobody cares about that and it is in fact misleading.

1 Like

This would also provide a possible path to support extension methods that need to be anchored on a generic type, if the extension target can have it’s own type arguments:

extend[A] (a: A) with {
  def pure[C[_]](given Applicative[C]): C[A] = summon[Applicative[C]].pure(a)
}

Which is effectively impossible with the current syntax, as the only way to enable 1.pure[Option] involves implicit conversions which are (justifiably) made painful to use.

2 Likes

extend would be nice to use, but we can’t. It’s too common as a word to take away as a keyword. And in this position the option of making it a soft keyword is out of the question since it would create too many ambiguities.

Having multiple type parameter lists in extension methods would also be great, but there’s another caveat. The way things stand, it’s a huge effort to spec and implement. I do not think we’ll get there for 3.0, but hopefully for a release soon after. In the meantime I did change the Tasty format so that it can accommodate such a change if it happens. https://github.com/lampepfl/dotty/pull/7504

3 Likes

I’m not familiar with the specific token parser limitation but does a word combination reduce the ambiguities if one of them is a keyword? Something like

extend class (c: Circle) with {
  def circumference: Double = c.radius * math.Pi
}

Or maybe even go “silence of the lambs” and use the extends keyword :laughing:

It’s unclear if extension methods are related to given instances or not. The documentation provides three ways to extend a class with a method:

  • New style of def
def (foo : Foo) op (bar : Foo) = ???
  • Using given ... extension
given fooOps : extension (foo : Foo) {
  def op (bar : Bar) = ???
}
  • Using given...AnyRef and the new style for def
given fooOps : AnyRef {
  def (foo : Foo) op (bar : Bar) = ???
}

Problems:

  1. The fact that we have of three different syntactic sugars to do the same thing (in case of a single method) is a concern. How frequent is it to extend with just a single method to merit the first example?

  2. It is unclear how extension methods are imported into scope. Do they follow the same given import rules? They are suppose to be separate mechanism from given instances, so this is highly confusing. The first example does not use the given keyword at all, so is it included in regular imports?

I would like to know. What was so wrong with implicit classes other than using implicit?

2 Likes

In general, to my eyes, all the options for anonymous given instances, clauses, etc., are just more confusing than actually cleaning up the code, because I have to look for and parse two kind of texts (with and without names).

I would much rather have _ : Something for anonymity. Yes, this is an “new” underscore usage, but still in the context applied to declare something “we do not care about”.

Edit: I also wonder how code completion tooling would react to this. A tool will not know if we require suggestions for term or type names.

6 Likes

Does the new implicits have anything better for prioritizing them yet? Now that we’re not bound to the syntax of val/def anymore I can’t see why something like given(n) can’t be used to give a custom priority.Saves you from creating a whole new trait for each priority level.

1 Like

If we want to completely replace implicit, we also need different names or even different mechanisms for:

  • DummyImplicit. Used to differentiate between different methods with the same signature after erasure. To me this always felt like a hack. Why can’t the compiler do it by itself?
  • @implicitNotFound. Can there a better way to provide custom error messages?
  • @implicitAmbiguous
1 Like

I would argue such annotation is redundant. Rust compiler can give suggestions for missing imports (implicit values candidates) itself, by scanning the classpath. Scala compiler could do that too. I’ve raised that issue in the topic about implicits (the topic about “why implicits in Scala aren’t a runaway success, but people love Rust with typeclasses”).

1 Like

Untrue. I use it frequently to provide custom error messages. It’s not just a better suggestion for missing imports.FWIW, I currently use macros to customize the error annotation.

6 Likes

There are good example about postconditions in “implicit function types”.
I like this pattern.
But I sometimes need to use implicit conversion and extension in my practice for such case.
So the code:

import PostConditions.{ensuring, result}

val s = List(1, 2, 3).sum.ensuring(result == 6)

will be converted into something like:

{  //we will need braces if we do not want that isDistinctFrom is visible outside of expression. 
import PostConditions.{given,_}
val s = List(1, 2, 3).sum.ensuring( 6.isDistinctFrom) 
}

It is not very convenient and it need time to accustom. So it seems that more preferable construction will be something like:

val s = new Postcondision(List(1, 2, 3).sum){
  ensuring( 6.isDistinctFrom)
} 

I am not sure that is is good way for scope reusage. But it seems it is still the less error prone and succinct way in such case.
It seems that there are still not enough context reusage in such patterns in general.

1 Like

Implicit Conversions

Unfortunately they don’t have parity with existing implicit defs yet. There are at least three differences:

  1. Conversions do not support path-dependency currently - [1]
  2. Combining them with macros aka inlines is very odd now because they’re “values”. They can be upcast and they can lose their “inline” status through upcasting. Because of that the compiler will emit an error on straightforward definition of a macro Conversion, you’ll have to define two traits and override a runtime method with an inline method. example: [2]
  3. Given and conversion names cannot be overloaded, because they are not methods – even when they take parameters [3]. This gets in the way of implicit punning technique – replacing a typeclass’s companion with a slew of implicit defs to ensure that whenever a typeclass is imported, its syntax is ALWAYS available with it. examples: [usage] [implementation with overloading]. Dotty’s extension methods and the fact that they’re available whenever an implicit is in lexical scope, which happens in strictly more cases than when a type name is imported make this trick unnecessary in most cases, except when extension methods are insufficient like in @morgen-peschke’s example.
1 Like

Honestly I don’t find the syntax great. given doesn’t seem to give us anything that implicit doesn’t: if we find-and-replace all the givens with implicits, we’d get all the benefits and much less disruption. The usage of :, and how strongly it binds left and right, also goes against what someone programming in Scala would expect.

Some more specific issues:

  • This syntax is pretty confusing: it looks like some kind of refinement type, but it isn’t:
given intOrd: Ord[Int] {
  def compare(x: Int, y: Int) =
    if (x < y) -1 else if (x > y) +1 else 0
}

If Scala didn’t have refinement types maybe this could be ok, but as is I find this extremely hard to parse. And the precedence is all wrong: normally { has higher precedence than :, but here it’s reversed.

  • I also think the anonymous given instances look much worse than what we currently have with implicits:
given Ord[Int] { ... }
given [T](given Ord[T]): Ord[List[T]] { ... }

Neither of these look particularly good. The top one has a type expression in a place that looks like no other language construct in existence, while the bottom looks like keyword soup with the two givens that do not match to any normal english usage of the word.

  • Alias givens honestly would look fine as implicit
given global: ExecutionContext = new ForkJoinPool()
implicit global: ExecutionContext = new ForkJoinPool() 

In fact, that’s almost the syntax now! We just need to make the def/val/lazy val optional, and we’re done.

  • I am personally not a huge fan of the extension method syntax. The precedence between given, :, extension and { seems all wrong in the given examples
given stringOps: extension (xs: Seq[String]) {
  def longestStrings: Seq[String] = {
    val maxLength = xs.map(_.length).max
    xs.filter(_.length == maxLength)
  }
}

Overall the usage of : in the given syntax doesn’t work out. It looks too much like something we’re already familiar with, and comes with baggage of the precedence we’re already used to, but is parsed in a totally different way. Using a new keyword would be far better than overloading : in this way

12 Likes

Can you outline (again if I missed it) why extend is too common but given is rare enough? What’s the threshold (in say of scala files with a particular identifier) ? Was this measured on some corpus?

I find extend or extension for making extension methods far more straight forward and important enough to merit a keyword IMO. Reusing keywords was part of what made implicit confusing (conversions vs the static-dependency injections).

4 Likes