Updated Proposal: Revisiting Implicits

Scala Improvement Process
1

Hi !

There’s been a lot of discussions on the Scala 3 proposals related to implicits already, since then the implementation in Dotty has evolved both based on these discussions and usage experience, therefore we’re opening a new thread to gather comments on the latest iteration (which should be more stable given the upcoming Dotty feature freeze). This proposal is composed of the following documents:

In previous discussions, we had one thread per specific feature, but this time we’re opening a single thread to encourage discussion on how all the parts fit together.

Some guidelines to keep the discussion productive:

  • Please avoid getting into a back-and-forth with someone else, this is rarely productive.
  • I think we’ve had more than enough debate on which keyword should be used, there’s a lot more aspects of the proposal that probably haven’t been scrutinized enough yet because of this.

Here’s some examples of possible discussion topics I think might be worth going over:

  • The Given Instances page has a section on instance initialization:

    A given instance without type parameters or given clause is initialized on-demand, the first time it is accessed. It is not required to ensure safe publication, which means that different threads might create different instances for the same given definition. If a given definition has type parameters or a given clause, a fresh instance is created for each reference.

    Does everyone agree that this is the ideal behavior ? Can we think of cases where this might have unexpected consequences ? EDIT: the doc is actually out of date here see Updated Proposal: Revisiting Implicits - #4 by odersky

  • The implicit keyword can be used in many more situations than given, for example it’s possible to create an abstract implicit val or def, to get something equivalent using given, one must create both an abstract val or def and a given alias that forwards to that abstract val or def. Is this acceptable ? How does this impact migration ?

  • Is the distinction between given aliases and given instances clear for everyone ?

Looking forward to hearing from everyone!

3 Likes
2

Yet we still have this given keyword in unrelated (IMO) extension methods declaration, so instead of implicit everywhere we have given everywhere.

4 Likes
3

As for given aliases, it’s unclear if they are evaluated once or on every given inference. The abstraction over vals, lazy vals, or defs is hurting in this case.

The documentation has the following example:

given global: ExecutionContext = new ForkJoinPool()

What will happen with the following:

summon[ExecutionContext]
summon[ExecutionContext]

Is new ForkJoinPool() called twice or not?

3 Likes
4

In fact that’s not true. We discussed this and agreed that given instances should ensure safe publication, and that’s what the implementation does (it uses a lazy val). The page on relationship with Scala-2 explicits is quite explicit that these things map to lazy vals and even includes the ForkJoin example that @soronpo inquires about. It seems the old wording was kept by an oversight in the cited quote.

1 Like
5

Thanks, I remember this was discussed but did not check if the current implementation matched the documentation, I’ve opened an issue: https://github.com/lampepfl/dotty/issues/7590. We can still discuss whether the new implementation is the most appropriate one though :).

6

Isn’t abstract class too heavyweight for a simple conversion? At least a trait would allow for the lightweight Java 8 lambdas encoding. Or maybe Java 8 lambdas can work with abstract classes too?

1 Like
7

Given Instances

How about allowing a given (lazy) vals in addition to plain givens (without val keyword)? It would:

  1. Simplify encoding rules and corresponding documentation. given (lazy) val translates to (lazy) val under the hood. given without following val is translated to def.
  2. Improve performance as plain vals are more efficient than lazy ones.
  3. Allow to have parameterless given defs as they are sometimes useful. One example is https://github.com/scalatest/scalatest/blob/release-3.0.8/scalactic/src/main/scala/org/scalactic/source/Position.scala which has a implicit def here: Position = macro PositionMacro.genPosition for generating implicit instance of current position in the source code.

Also in Dotty documentation there’s following example:

given (given outer: Context): Context = outer.withOwner(currentOwner)

How is that supposed to work? If I invoke a method with given parameter and I have a given instance in scope then that instance should be used immediately, I think. This way the above conversion would be never used. Or not?

given context1: Context = ???
// will it stay as it is or be expanded using the anonymous given instance
given context2: Context = context1
2 Likes
8

Extension Methods

I think some more elaborate example would be helpful to get the feeling, like arithmetic progression sum.

// ??? - what givens, extensions, etc are needed here to make below code compile without changes?

def arithmeticProgressionSum[T: Numeric](firstTerm: T, step: T, termsNum: Int): T =
  termsNum * (2 * firstTerm + (termsNum - 1) * step) / 2

val sum = arithmeticProgressionSum(new java.math.BigInteger("3453"), new java.math.BigInteger("245"), 745)
println(sum)
9

Given Instances

The docs give this example:

given (given outer: Context): Context = outer.withOwner(currentOwner)

This makes anything about existing implicits look like the paragon of clarity in comparison. It looks like the “Buffalo buffalo buffalo” puzzles sometimes given in e.g. philosophy of language classes.

Something should change to make this impossible. For instance, making give provide things and given ask for them would fit naturally into common language usage instead of admitting mindboggling things that read like “given given Foo Foo”.

The syntax of given instances and aliases are both weird special cases. Nothing else in the language works like that. I understand what the point of each is, but the instances look like a weird hybrid of what you’d do for a def and what you’d do for a class, making it anything but clear what the point is. That regular instances use =, e.g. val five = 5, while given instances do not, is especially befuddling. (That object O does not use = is already befuddling enough, but there at least since it isn’t parameterized you can reason that it both declares the singleton class type and automatically gives it a unique name–that is, it’s a sugar for sealed abstract class O.type { ... }; lazy val O: O.type = new O.type {}.)

Since given instances can be parameterized, and aliases can alias to defs, it’s also quite a bit of work to puzzle out what is gained by having both constructs.

Given Clauses

Given clauses are pretty nice, though I found the foo(x) given y syntax vastly superior visually in simple cases (but untenable in complex cases). However, the document doesn’t explain whether the (implicit thing: Thing = Companion.defaultThing) pattern works any longer. It looks like no, given the grammar. It also isn’t clear to me whether def f(given foo: Foo, i: Int) can be called as f(5). I don’t think I missed the example?

It’s not clear why they are called “given clauses”. Parameters aren’t called clauses in other places.

It’s not clear how the values are filled in with multiple relevant givens in scope. Presumably something like implicit resolution, which right now forces awkward tricks like creating superfluous traits just to drive resolution? Or…? Should be spelled out.

Context Bounds

Looks fine! Seems pretty clear to me. Also doesn’t seem to have changed.

Given Imports

This is a solution to only half the problem (actually, only half of half of the problem). Not knowing that import mylib._ brought in implicits is fixed by givens having to be import mylib.{given, _}. So that’s something. But you still don’t actually know what you’re getting from there. So what you’re required to do doesn’t actually solve even this half of the problem; you have to take the explicit-type option and add all that boilerplate.

But import mylib._ not working because you forgot to bring in implicits is even worse because now they can’t conveniently be inside mylib any more. This suggests that to recover parity with the old way, best practice will be to spam import mylib{given. _} everywhere, whether or not you need it…which then gives you the first problem right back again.

Instead, what we need is a less awkward way to specify precedence, along with deprecation messages. So you could

import mylib._

and if you’d get the implicits (givens) that would make things work, but if you marked them all deprecated, you could have a message like, “This invocation uses a default given from mylib. Please import mylib.fast._ to select the fast versions of these implicits, or import mylib.accurate_ to select the accurate ones.”

Maybe it’s too late for a change like this, but I think the new version is strictly worse than the old version as it stands.

It seems like a tooling problem is being hoisted onto the programmer in an unwelcome way. I’d rather the compiler do more for me e.g. by suggesting places to find givens that I need, than force me to manually do that kind of work myself.

Finally, it’s not clear that the givens can be brought in by type with the full types that could appear in the given alias or instance.

Extension Methods

I love the idea, but the syntax is rather ugly. I wouldn’t assume that I’ll just get used to the syntax and eventually love it, because after like a decade of experience with C++ I still find the lambdas impossible, and after a couple of years of Rust I still find |x| bar(x) lambda syntax way less readable than Scala’s x => bar(x).

I’d prefer something like

extend (c: Circle) with {
  def circumference: Double = c.radius * math.Pi
}

which could be shortened to

extend (c: Circle) with circumference: Double = c.radius + math.Pi

in the case where you wanted a one-liner.

If you complain that paring this down to be more refined and elegant will just get you back at the current extension syntax, I plead guilty. Yes, it does. Sometimes the extra fluff helps. I think this is one of those times.

Implicit Conversions

This is so awesome! (Note: still need clarification on how givens are resolved when potentially conflicting.)

Implicit By-Name Parameters

This seems like kind of a weird and ugly hack. Isn’t there some better way to get around the problem so we don’t have to worry about it? Like, what’s wrong with always having the by-name behavior to avoid circularity?

It’s especially weird because by-name suggest lazy runtime behavior, but here it’s being used to drive compile-time differences. This is a fundamental shift in how by-name parameters work.

Relationship with Scala 2 Implicits

Seems reasonable, save for not mentioning how to migrate (implicit thing: Thing = defaultThing), as I brought up earlier.

11 Likes
10

This was indeed more a “demonstrate the limits of what this can do” example than a piece of recommended code. I propose we remove the example.

What it did is that it allows us to define an inner given using an outer one. As @tarsa asks, it’s not obvious why this should work. In fact it works because trying to resolve the given with itself would produce an implicit divergence. In Scala-3 is a recoverable error, so the outer given would be searched next and that one would succeed. In current Scala the same trick would not work since there divergence errors are fatal. But once can make it work rather perversely by writing this:

implicit val ctx = implicitly[Context].withOwner(currentOwner)

It’s essential that the type of ctx is not given explicitly, since that prevents ctx from being elaborated on its rhs. To my surprise there are actually widely used libraries out there that rely on this trick. But all this is rather too specialized and deep for a casual discussion and it has nothing to do with anonymous instances. So we should just drop the example.

11

To clarify: I don’t think

given (given Foo): Bar

is a clear and obvious thing either. Having two identical keywords to mean two different features, used in close proximity where they force rapid mental shifts in interpretation, is not good. The given clause has the virtue of being signaled well in advance by def and method name and stuff; here it requires an instant mental flip. This is obvious:

give Bar given Foo

I’m not saying that’s what the syntax should be, necessarily, just that it helps the reader out.

Also, removing the example of given (given outer: Context): Context doesn’t prevent it from being used, or for the problem it illustrates to be not-a-problem. It just seemed particularly egregious to me.

7 Likes
12

Thanks @Ichoran for the write up. I’d like to second the concerns about the syntax of given instances. I have already explained here why I think we should just have the “given aliases” syntax (like for val and def definitions).

1 Like
13

EDIT: this is very similar to the link @julienrf posted just above, I didn’t see it until after I posted this message!

I agree that the syntax for given instances looks like a hybrid thing, by contract given aliases seem more natural to me. So could we drop given instances? This would be one less thing to teach! Instead of:

trait Foo { def foo: Int }
given Foo {
  def foo: Int = 1
}

we can use a given alias:

given Foo = new Foo {
  def foo: Int = 1
}

The former translates to an object, the latter translate to a lazy val and an anonymous class, those should be semantically equivalent.

And instead of:

trait Bar[T](x: T) { def foo: T }
given (given x: Int): Bar[Int](x) {
  def bar: Int = 1
}

we can also use a given alias:

given (given x: Int): Bar[Int] = new Bar[Int](x) {
  def bar: Int = 1
}

The former translates to a class and a def, the latter also translates to a class and a def and the semantics are again equivalent as far as I can tell.

Of course, the main issue here is the repetition, so let’s invent even more syntax (sorry!):

given Foo = new {
  def foo: Int = 1
}
given (given x: Int): Bar[Int] = new(x) {
  def bar: Int = 1
}

The unparameterized version actually already works in Dotty, the parameterized one has been suggested before by @propensive in a more general context, and @odersky gave some reasons it might not be worth it in Expunging `new` from scala 3 - #124 by odersky, but I think it’s worth revisiting in the context of givens: it it allows us to drop given instances that would be a huge simplification and more than make up the extra complexity it introduces.

2 Likes
14

The (non-alias) given instance syntax actually looks better once we allow a with:

given Monoid[String] with
  def (x: String) combine (y: String): String = x.concat(y)
  def unit: String = ""

given Monoid[Int] with
  def (x: Int) combine (y: Int): Int = x + y
  def unit: Int = 0

BYOB (“bring your own braces if you like them”). I don’t like the alias syntax with a new. I’d like to get away from using new – it’s connotations are far too operational! Writing

given Monoid[String] = new {
  def (x: String) combine (y: String): String = x.concat(y)
  def unit: String = ""
}

emphasises the point that we create a new object to be a monoid. But nobody cares about that and it is in fact misleading.

1 Like
15

This would also provide a possible path to support extension methods that need to be anchored on a generic type, if the extension target can have it’s own type arguments:

extend[A] (a: A) with {
  def pure[C[_]](given Applicative[C]): C[A] = summon[Applicative[C]].pure(a)
}

Which is effectively impossible with the current syntax, as the only way to enable 1.pure[Option] involves implicit conversions which are (justifiably) made painful to use.

2 Likes
16

extend would be nice to use, but we can’t. It’s too common as a word to take away as a keyword. And in this position the option of making it a soft keyword is out of the question since it would create too many ambiguities.

Having multiple type parameter lists in extension methods would also be great, but there’s another caveat. The way things stand, it’s a huge effort to spec and implement. I do not think we’ll get there for 3.0, but hopefully for a release soon after. In the meantime I did change the Tasty format so that it can accommodate such a change if it happens. https://github.com/lampepfl/dotty/pull/7504

3 Likes
17

I’m not familiar with the specific token parser limitation but does a word combination reduce the ambiguities if one of them is a keyword? Something like

extend class (c: Circle) with {
  def circumference: Double = c.radius * math.Pi
}

Or maybe even go “silence of the lambs” and use the extends keyword :laughing:

18

It’s unclear if extension methods are related to given instances or not. The documentation provides three ways to extend a class with a method:

  • New style of def 
def (foo : Foo) op (bar : Foo) = ???
  • Using given ... extension 
given fooOps : extension (foo : Foo) {
  def op (bar : Bar) = ???
}
  • Using given...AnyRef and the new style for def 
given fooOps : AnyRef {
  def (foo : Foo) op (bar : Bar) = ???
}

Problems:

  1. The fact that we have of three different syntactic sugars to do the same thing (in case of a single method) is a concern. How frequent is it to extend with just a single method to merit the first example?

  2. It is unclear how extension methods are imported into scope. Do they follow the same given import rules? They are suppose to be separate mechanism from given instances, so this is highly confusing. The first example does not use the given keyword at all, so is it included in regular imports?

I would like to know. What was so wrong with implicit classes other than using implicit?

2 Likes
19

In general, to my eyes, all the options for anonymous given instances, clauses, etc., are just more confusing than actually cleaning up the code, because I have to look for and parse two kind of texts (with and without names).

I would much rather have _ : Something for anonymity. Yes, this is an “new” underscore usage, but still in the context applied to declare something “we do not care about”.

Edit: I also wonder how code completion tooling would react to this. A tool will not know if we require suggestions for term or type names.

6 Likes
20

Does the new implicits have anything better for prioritizing them yet? Now that we’re not bound to the syntax of val/def anymore I can’t see why something like given(n) can’t be used to give a custom priority.Saves you from creating a whole new trait for each priority level.

1 Like