Proposal To Revise Implicit Parameters

Scala Improvement Process
18

If you allow for consecutive skippable implicit argument lists then you’ll have such ambiguities in both my proposal and Martin’s one (I think). To disambiguate we could use solution proposed by @curoli which is:

f(given a)(given ...)("abc")
f(given ...)(given a)("abc")

... means any number of implicit arguments.

Consecutive implicit parameter lists probably won’t be popular enough to warrant such special syntax.

Probably in the simplest way - assume you always start with first implicit arguments list from the consecutive ones.

19

I think that for someone has no previous background in Scala, something like this would be a lot more intuitive:

def max[T](x: T, y: T, ord: Ord[T] = implicit): T =
  if (ord.compare(x, y) < 1) y else x

max(2, 3, ord = IntOrd)
max(2, 3)
max(List(1, 2, 3), Nil)

I personally do not like the given syntax, and think a default-arg-based syntax would be easier for everyone coming from programming languages with default args (i.e. all of them, these days). But by this point I’m probably sounding like a very small shell script, so I’ll leave it at that. Take from this what you will!

20 Likes
20

Lately I have been using definitions such as the following:

def foo[A, B, C](a: A, b: B)
                given Ctx1, Ctx2, Ctx3: Lifted[C] = ???

where to update one of the Ctx params I would use the implied for Ctx = ??? syntax. Otherwise, if you want to use given at the call site it’s best to define your inferrable parameters in a curried form, abstracted through types, or else you have to supply the entire list.

I do think it feels a bit surprising to someone unfamiliar with the desugared form that the compiler doesn’t generate the other parameters if you only supply one with given at the call site.

21

So I am going to state here what I stated on the initial proposal PR, which is that I am not a fan of the proposal in general because I think its solving the wrong problems. In summary my points are

  • The real issue with implicits is not how they are defined or used, but the fact that the compiler gives completely terrible diagnostics especially with chained implicit resolutions. This is not going to solve this. Its also unclear how implied is going to work when combined with other keywords.
  • The only real problematic part of implicits is implicit conversions, which people have (rightly) complained about and are already deprecated and considered an anti-pattern
  • The current implicit is actually much more familiar with the current mechanics of Scala compared to this proposal. implicit works the same way as keywords like final, i.e. they modify a definition to state that “this variable can be implicitly provided”. Likewise using it in a parameter list states that “I want to get a variable from a type that has been implicitly defined”. These concepts are not hard to understand.
  • This proposal goes against the nature of Scala where we do have control over how variables are defined. There is a difference between an implicit val vs implicit lazy val vs implicit def (just like there is a difference between final def vs final val), i.e.
    • implicit lazy can combined with trait can be used for compile time DI
    • implicit def an be used for constructing DSL’s in a safe and principled manner.

I would rather the effort be spent improving the current implicits because honestly we are already 80% there. This includes

  • Giving sane compiler errors about missing implicits (i.e. https://github.com/tek/splain should be part of the scala compiler, not a plugin)
  • Fixing discrepancies when it comes to implicit function application. I.e. If you have function like
    def doSomething(key: String)(implicit ctx: Context): Map[String, String]
    This is going to behave differently on .apply rather than
    def doSomething(key: String): Map[String, String]
    In the former the .apply is going to be providing the Context where as in the latter the .apply is going to point to the key. In both cases apply should point to key and if you want to explicitly provide the implicit context it should be something like
    doSomething("myKey").explicitly(someContext)
    This (along with some other changes) would fix all of the currying issues that we have with implicits.

There are probably some other things which I have missed but the tl;dr is that we should fix the current implicits rather than making an entirely new proposal which isn’t even in the spirit of the language.

IDE’s also need to provide better inspections, Intellij as of late is doing an excellent job here (it will actually show you implicit chains and it now has the ability to tell you where implicits are being used which is incredibly handy). When metals does the same a lot of the quality of life issues with implicits should be solved.

9 Likes
22

I agree the default argument syntax is attractive because it is familiar. The problem is, it leads to a completely different system of implicits than what we have now.

In particular implicit parameters are tied to normal applications. For instance, we currently have a max extension method on lists, which is roughly defined like this:

    class ListOps[T](xs: List[T]) {
      def max(implicit ord: Ordering[T]) = ...
    }

You call it like this: List(1, 2, 3).max.

With the implicit as default params proposal that would now do something different. It would expand to (ev: Ordering[Int]) => List(1, 2, 3).max(ev), i.e. it would return a function that takes an ordering and produces a number. You have to pass a () argument to max to get the maximum: List(1, 2, 3).max(). That’s just how default arguments work: They fill in missing arguments inside an argument list but if the (...) is missing you get an eta expansion instead.

The example also shows something more fundamental: Current implicits and implicits-as-defaults behave differently under partial application. Given

def max[T](x: T, y: T, ord: Ord[T] = implicit): T = ...

What would max[Int] without further arguments expand to? According to the rules for default parameters it would be:

(x, y, ord) => max[Int](x, y, ord)

But according to current (both old and new-style) implicits it would be:

(x, y) => max[Int](x, y)(IntOrdering)

See the difference? The implicit argument moved from definition to use site! This makes implicits a whole lot more like dynamic scoping. Maybe that would work and maybe it wouldn’t. But it sure is completely different to what we have now! And, I believe it’s fair to say that the behavior of implicits under partial applications works fine as is, at least I can’t recall anybody complaining about it.

One could consider tweaking the rules so that default arguments with implicits behave somehow different from normal default arguments so that we can get back the status quo. But that would be a classical “easy instead of simple” move. We buy the easy familar syntax at the price of complex rules that mix things like implicits, default arguments and partial application that formerly were better separated. So, that’s why, much as I like the syntax. I don’t think implicits as defaults are the right way to go.

6 Likes
23

This has nothing to do with with familiarity, it has to do with consistency (and to a lesser effect orthogonality)

You can do this right now

implicit class MyOrderingListOps[T](xs: List[T])(implicit ord: Ordering[T]) {
  def myMax = ???
}

And you can call it just like normal, i.e.

List(1,2,4).myMax

Then you are passing the environment into the max function and you get all of the properties (as you describe). If you want to the above explicitly, you would do

new MyOrderingListOps(List(1,2,4))(ordering)).max

Which is semantically equivalent to what you mentioned, i.e.

This is what I mean when I talk about consistency, because this is completely consistent with how the language Scala works, and this is the beauty of having something that is orthogonal, depending on where you place the implicit you can control how application works; we don’t need to invent a new language construct, we can already do this! This doesn’t need a new language construct, its already possible.

And the final point is we should have control over this because as you pointed out there is a difference between applying the environment (in this case the Ordering) curried before the application of x and y versus after. Creating an entirely new language construct just to do this is overkill, to say the least.

And yes I am aware that the implicit class syntax will get replaced with extension methods, but it should work in the same manner.

1 Like
24

I’ve written quite a lot of code with the current implied/given syntax (perhaps 10k LOC) and I still stumble over it. The keyword just doesn’t stick in my head, I think because the grammar doesn’t quite work in native english. I would really prefer instance for introducing instances:

instance OptionMonoid[T] given Monoid[T] for Monoid[Option[T]] = {
  case (Some(lhs), Some(rhs)) => the[Monoid[T]].combine(lhs, rhs)
  ...
}

I know this is liable to sound like bikeshedding, but I’ve really, really tried to stick with it and after weeks of coding every day, implied still doesn’t stick - I have to look it up over and over, and half the time I’ve written instance.

The other thing is that I often type the for and given clauses the wrong way around - vis:

instance OptionMonoid[T] for Monoid[Option[T]] given Monoid[T] = {
  case (Some(lhs), Some(rhs)) => the[Monoid[T]].combine(lhs, rhs)
  ...
}
1 Like
25

My previous post was not focusing around the syntax, but that is another thing that this proposal needs a lot of improving. We are overloading the use of keywords in completely different context (i.e. for is also used in for comprehension) and the way that the parenthesis are applied is inconsistent with the rest of the language.

26

instance OptionMonoid[T] of Monoid[Option[T]] given Monoid[T] works for me as well. It’s almost literate, when written like that. And it reads when you skip the name as in instance [T] of Monoid[Option[T]] given Monoid[T]

1 Like
27

@mdedetrich @drdozer This thread is about the implicit parameter proposal presented in https://dotty.epfl.ch/docs/reference/contextual/inferable-params.html whereas some of your comments are related to the separate but related proposal in https://dotty.epfl.ch/docs/reference/contextual/instance-defs.html which will should get its own thread on contributors.scala-lang.org soon enough. Please keep your comments on this thread related to the implicit parameters proposal otherwise things are going to get way too confusing.

1 Like
28

Agreed with @smarter that this belongs elsewhere, but since we are already in it a general question to @drdozer. As someone who has actually written a lot of code in the new system, would you recommend it over old style implicit parameters?

29

See the difference? The implicit argument moved from definition to use site!

You can do this right now

I think we are talking at cross purposes here. I was discussing that switching to default parameter syntax would go much deeper than just syntax: you get a different set of implicit rules that drastically change how the whole system hangs together and how your programs are behaving.

I was not arguing that List#max could only be expressed using given. As far as I know it is expressible in all systems we deal with, only in different ways.

This is what I mean when I talk about consistency, because this is completely consistent with how the language Scala works, and this is the beautify of having something that is orthogonal, depending on where you place the implicit you can control how application works; we don’t need to invent a new language construct, we can already do this! This doesn’t need a new language construct, its already possible.

Exactly the same argument applies to given, minus a number of warts that are removed by it.

30

I agree with this, but i still think it is worth doing. This entire discussion is “how can we make implicits easy and familiar to people without prior experience”, and all the proposals are essentially different syntactic spellings of the same thing rather than any fundamental changes. We’ve literally spent hundreds of posts arguing over what the keyword should be spelled! Buying “easy and familiar” syntax isn’t some side benefit, but is the entire purpose of this exercise.

As far as I can tell, the given syntax achieves this goal not at all. given, instance, etc will have zero familiarity with people not already familiar with typeclasses, which is the vast majority of programmers. Despite the low value, it comes with the same huge up-front cost of changing such a fundamental syntactic construct, and i believe the net value is definitely negative.

The default argument encoding requires more speccing - it is less of a trivial syntactic desugaring, but it actually can solve our problem once and for all! It can make implicits easy and intuitive for people with zero Scala experience! The fact that it requires a bit more finesse in how it is specced, rather than being a dumb desugaring, seems a tiny cost compared to the massive fixed migration overhead either approach would have to pay anyway

There’s also a third road, that I have brought up before: make all the “semantic” or “boilerplate” improvements, without overhauling the core syntax. Things like multiple implicit argument lists, anonymous implicits, better inference for declarations, making the val/def in implicit val/implicit def optional, etc. can be fitted in reasonably nicely without any changes. Avoiding the implicits-vs-currying ambiguity by introducing a keyword like foo(x).explicitly(y) or foo(x)(implicit y) can similarly be done with relatively little change or disruption.

Thus I would very much be in favor of the solution that makes things easy and familiar to users but takes more work to spec and implement. Second place preference would be to do nothing, syntactically: we can support multiple implicit argument lists, anonymous implicits, and other “semantic” improvements with much less invasive syntactic changes. The given/inferred/etc syntaxes definitely seem a distant third, as an expensive non-solution that has similarly huge costs but gives us very little in return.

14 Likes
31

I think it’s questionable whether “for someone has no previous background in Scala” the version

def max[T](x: T, y: T, ord: Ord[T] = implicit): T

is more intuitive than

def max[T](x: T, y: T) given (ord: Ord[T]): T

We are all long term Scala users, so for us probably the implicit thing is more familar, but for someone coming new to the language, you have to explain the concept of implicit/implied in any case. I think Martin’s proposal is more clear in terms of how you read the statement, more analogous how you would make a mathematical statement etc.

I’m not against the = implicit per se, but I don’t see it as a big advantage, either. What happens with multiple implicit arguments? This is actually quite a common case in all my scenarios. So

def foo(x: Int)(implicit tx: Tx, cursor: Cursor): Unit

This doesn’t map at all to the = implicit thing, unless you want it to become

def foo(x: Int, tx: Tx = implicit, cursor: Cursor = implicit): Unit

which is not only more verbose, but also more difficult to parse visually, compared to

def foo(x: Int) given (tx: Tx, cursor: Cursor): Unit
5 Likes
32

Maybe we should try to frame this discussion by mentioning the downsides of the current syntax that we’d like to address, here’s a first attempt:

  1. The current syntax has a concept of implicit parameter list at definition-site, but does not distinguish implicit and explicit parameter lists at use-site. This means it’s easy to accidentally pass an argument to an implicit parameter (in fact this has happened at least once in Dotty itself).
  2. The current syntax does not allow having explicit parameter lists after implicit parameter lists
  3. The current syntax does not allow anonymous implicit parameters (except by using context bounds)

I think the most straight-forward way to address 1. is to keep the definition-site syntax the same but require mirroring it at use-site, e.g.:

def minimum[T](xs: List[T])(implicit ord: Ord[T]) =
  maximum(xs)(implicit descending)

minimum(xs)
maximum(xs)(implicit descending)
maximum(xs)(implicit descending(implicit ListOrd))
maximum(xs)(implicit descending(implicit ListOrd(implicit IntOrd)))

(I’m using implicit here since this is the current syntax, I think we should defer discussing keyword renaming for now since this isn’t central to the main discussion).

Once 1. is address then we can just remove the restriction from 2. which isn’t necessary anymore, however this can only be done once all the existing code has been updated to use the new syntax at use-site, so that means a somewhat long transition period.

Finally, 3. could be addressed by allowing something like this:

def minimum[T](xs: List[T])(implicit Ord[T]) =
  maximum(xs)(implicit descending)

implicit _: Ord[T] could also work but I’d rather not add more uses of underscores in the language. In any case this seems like something that could be discussed separately as its own proposal.

Compared to the original proposal in this thread this has several advantages I think:

  • No existing definition needs to be rewritten
  • The use-site syntax does not require spending precious brain-cycles thinking about where to put parentheses to please the parser.
  • This is a less radical change, so existing users are less likely to see the new syntax as alien and then declare that “Scala 3 is a new language” which at the very least is bad PR I think.

Opinions?

14 Likes
33

I really like @smarter’s proposal !

I have to say I think the current one involving given, instance and for seems to be making thing more complex not less; I find the new syntax more opaque/magical.

The way implicit values and parameters work currently, I think, isn’t that complicated; the keyword implicit merely acts as a marker for certain values.

1 Like
34

I like the clear classification of what is wrong with the current syntax!

But I am afraid I find (implicit x) for an argument an abomination. Scala already suffers from plastering implicit on too many things. We now would do even more of this, and in situations where for a beginner it seems to make no sense at all! Marking an explicit argument as implicit is pretty contorted. It can be understood only by a long-term Scala programmer who has learned that implicit is just a tag, without any particular meaning beyond it. We could just as well call it magic, that would probably improve the situation.

The argument that no existing definitions need to be rewritten looks attractive, until you realize that this means (as you say yourself) a very long, possibly indeterminate migration period. If the definition site syntax does not change you have to wait until all use sites are rewritten before you can change the rules for matching definitions with use sites. This might well be never. The same problem hampers @mdedetrich’s explicitly proposal, which we considered before.

given neatly sidesteps all of these problems. It aligns use site and definition site, has a clear migration strategy, and is immediately familiar. I believe very strongly it’s the single, obvious solution. I say this with conviction, since I have thought so long and hard about alternatives. The only alternative that had a similar “why didn’t I think of this before?” effect on me was @lihaoyi’s proposal of implicits as defaults. But, as I have argued before in this thread, that would lead to a completely different system of implicits. So if one was just starting out designing a new language this would definitely be an interesting candidate. But it can’t be a replacement for what we have.

3 Likes
35

But that’s not the tradeoff. implicits as defaults lead to a different system of implicits. We could try to invest a lot of spec and implementation work to hide this, but all we would achieve is paper over fundamental differences, most likely ruining both implicits and defaults along the way. Good language design is about discovering simple principles of fundamental applicability and find a clear syntax to match them. Form follows function. If we work hard to hide the principles under a syntax that does not fit we have made things worse, not better.

If we would start out with a new language, implicits as defaults could work. We’d have to make the following design choices:

  1. When eta expanding a function, parameters with default arguments are instantiated instead of being lifted out. Given

    def f(x: Int, y: Int = 2)

    then f should expand to x => f(x, 2). In Scala it expands to (x, y) => f(x, y) instead. I believe either design could work, but it’s impossible for us to change now. Too much code would break.

  2. We’d have to accept the fact that every function taking an implicit is applied to some parameters, even if it’s just (). It would not be possible to pass an implicit to a parameterless function. That’s clearly a severe restriction compared to what we have in Scala. Maybe one could get used to it, or maybe it would be too annoying. But, again, I see no way how Scala could migrate to this.

Compared to this, I believe given is equally obvious to a newcomer. And it has the advantage that it is shorter. E.g. in the dotc compiler we repeat the parameter list

def f(x: T)(implicit ctx: Context)

more than 2500 times. There’s lots of code like this, e.g. parallel or concurrent code taking an ExecutionContext. We can reduce that to

def f(x: T) given Context

which is clearly better. The alternative

def f(x: T, ctx: Context = implicit)

is equally clear but adds more bulk. In summary, given has the advantage that it is both familiar and more streamlined than implicits as defaults.

6 Likes
36

This is why we have a -scala2 flag in Dotty and why the proposal is being suggested for Scala3 and not current Scala. This is also something that can be in scope of scalafix.

If the only good reason for making this a completely new feature with alien syntax is to avoid this migration (which is how it appears from my end) then we need to seriously question the proposal. I agree with @lihaoyi here, I would rather fix all of the quality of life issues with the current implicit syntax (which can easily be done) rather than adding a completely new feature (whos only real merit seems to be sidestepping migration which we have to do in Dotty anyways).

This is also works, I like using the keyword explicitly better because it is clear that you are passing an argument in explicitly but I am not going to fight over it.

37

When eta expanding a function, parameters with default arguments are instantiated instead of being lifted out.

Will it work if we do it only for the default params which are implicit? This way, the behavior for the implicits and default params remains consistent with Scala 2, at least when it comes to eta expansion.