Proposal To Revise Implicit Parameters

[smarter]

@odersky

If, a year from now, the consensus is that most people hate the new implicits, we will retire them before going into 3.0 final. We can at that point consider whether we would want to backport some of the improvements to the old implicits. But I bet it won’t come to that.

As a last resort I’m OK with that plan, but it’d be so much better if we agreed on things sooner rather than later. E.g. my proposal would benefit from a gradual migration strategy like the one proposed by @sjrd in Post 63 and Post 65, if we wait until a short time before 3.0 to reconsider it, then every milestone of the migration plan will have to be significantly delayed.

[lihaoyi]

@odersky

No, but, “I never tried working with it but it looks strange to me and therefore I am against” is not a valid argument either.

Literally nobody has said that. You have made a proposal, and gotten a lot of feedback. Detailed, constructive feedback, both positive and negative, including proposed alternatives which are specced out as much as can be without sending a patch for the compiler. Please don’t dismiss that as “It looks strange to me”, because it isn’t.

You are clearly frustrated getting a lot of the same feedback and concerns, over and over, across multiple threads. Not just from me, but from multiple people. But the reason that same feedback keeps appearing is not because people are stubborn and obstinate: it is because you are not responding to it in a sufficiently understandable, convincing manner to win people over, and so the concerns remain.

By putting up a new proposal, people have the implicit/implied/given expectation (ha!) you to want to hear about feedback, concerns, and alternatives. Isn’t that what proposals are for?

Perhaps what we need to do is set expectations for the discussion right:

• What do you want from the discussion? Do you want only minor tweaks to the current proposal? Are you interested in alternative proposals? Are you interested in meta-feedback, e.g. “this proposal could be more convincing if it had XYZ”?

• Is there any actionable outcome? Are we looking for a go/no-go consensus? What’s the expected outcome if people are favorable, unfavorable, or favorable to something else? Are we just trying to improve this specific proposal, with no actionable decision point? Or is there no actionably output except the discussion itself, and work will keep churning along regardless?

• Where do alternatives come in? This thread, or elsewhere? Do alternatives need a detailed, self-contained spec before being worth discussion? Or only alternatives with an implementation in Dotty? Or perhaps we do not have {time,resources,effort} to spare on properly discussing alternatives, and so this proposal is all we’ve got?

• Since we’ve started bringing in social proof in an ad-hoc manner, should we formalize that into a proper survey? That would save a lot of “this guy here likes this design more” “that guy there likes that design more” boilerplate discussion, and leave that to the end under a more rigorous process everyone can agree on beforehand.

• What about repeated feedback: things that have been brought up before. Is that welcome, if the feedback still applies? Or is it unwelcome, because it has already been assimilated? How do we know if something has been assimilated, rather than forgotten?

I think by settling these fundamentals right, we’d be able to have a much more productive and much less frustrating discussion for everyone.

[Ichoran]

@odersky

It might look strange at first. My bet is, after using it a bit, it will be very natural.

I don’t actually need to use it to know how natural the given syntax is going to feel, as I have enough prior art with other things.

This is something that doesn’t feel natural now but very likely I will learn to feel is natural:

def f(x: Double) given RoundingMode: Double
f(x) given HalfUp

This is not:

def f given RoundingMode(x: Double): Double
(f given HalfUp)(x)

The visual precedence is wrong in the definition, the groupings suggest a syntax tree that doesn’t accord with the underlying construct, and even having to consider that this might possibly occur in the code will negatively impact my experience.

The fact that we have prior art in (c + 2)(foo) having foo be an argument to the + method is just, to me, a condemnation of that part of the current rules. The compiler scolds me:

scala> class C { def +(i: Int)(j: Int) = ??? }
defined class C

scala> val c = new C
c: C = C@536bb027

scala> c + 2
<console>:61: error: missing argument list for method + in class C
Unapplied methods are only converted to functions when a function type is expected.
You can make this conversion explicit by writing `+ _` or `+(_)(_)` instead of `+`.
       c + 2
         ^

No conversion unless a function type is expected–yes, please! I want to know.

In other contexts, the presence of a trailing (x) alone isn’t enough to convert something to a function type, so it shouldn’t be here either. (Try (c + 2).pipe(_(3)) for example.)

The rule that what is inside parentheses is evaluated first is one of the most inviolable in mathematical and programming syntax, and it’s broken here. Yes, if the curried and uncurried forms are treated as equivalent, then it’s the same thing, but having to do the mental juggling between different forms is a burden.

[soronpo]

@odersky

I agree we need large scale experience reports before we finalize this. What I will propose is the following: We keep new style implicits by and large as they are implemented in the Dotty codebase. A lot of thought went into them and they have been carefully designed and calibrated. If people have ideas how to improve or fill in some details I am very open to integrate them. But we will not start from zero.

Dotty has indeed made great strides to improving the Scala language, yet regarding this proposal I fear we may be putting the carriage before the horses. The ideal language change trajectory should always be an iteration of “spec->discussion->spec…” and only finally implementation, so we don’t waste time on work to see it rejected later. Indeed in many situations it might be easier to present a working feature to play around with and convince people, which is what the dotty project is all about, but this may also get us emotionally attached to ideas we put a lot of effort in. For me, the “already too much work was done” argument is irrelevant to this (technical/user-experience) discussion. Powers-that-be may decide later (outside of this discussion) that this argument is valid since we don’t live in an ideal world and there are other concerns (budget, timetable, etc.).

@odersky

If, a year from now, the consensus is that most people hate the new implicits, we will retire them before going into 3.0 final.

If Scala 3 books and documentation are a concern, can we reasonably state that?

[DWvanGeest]

I mostly just lurk in here, but I feel rather compelled to say something.

I’ve taught plenty of Scala newbies about the current implicit system. Conceptually, most folks understand it relatively quickly. It’s pretty straightforward and the fact that the implicit keyword is used in parameter lists and also when declaring implied values is a huge clue to what’s going on, even to the uninitiated. Many folks are also familiar with multiple parameter lists (or at least don’t see it as a terribly strange concept), so the idea of one parameter list being “different” is not such a great leap.

I do agree that there are warts, particularly the arbitrary limiting to a single, final implicit param list. I do not agree that these warts warrant such a drastic, complicated syntax change. I’ve read the proposal multiple times and am still struggling with some of the basics. What was appealing about the old implicit system is that it just made slight modifications to already familiar concepts. With the new system, fundamental things (like the definition and calling of a function) have become pretty unfamiliar through the insertion of given. The use of for is even more confusing.

I will echo what @lihaoyi, @mdedetrich and others have said: let’s identify the major pain points in the current implicit system and try to solve them with minimal changes. I think there have been many compelling alternatives presented that deserve serious consideration, alternatives that build on familiar concepts. This is one of the things I love most about Scala - is that you don’t need to understand everything deeply at first, but you can still have pretty good intuition about what it does, largely because it builds on familiar constructs from other languages. The old implicits, even though they’re a relatively unfamiliar concept, actually scaled gradually with the developer, following the Scala philosophy. I do not think the same can be said for the proposed system.

On a more meta level, I am also frustrated that this SIP process appears to be somewhat for show. @lihaoyi has been very diplomatic about pointing out some of the uncertainty around the goals of this process, but I’m afraid I will be more blunt. When highly-qualified contributors from the community spend hours engaging with the material, politely raising legitimate and well-reasoned concerns, and constructively presenting alternatives in good faith, why are they being dismissed so flippantly?

[odersky]

@lihaoyi Thanks for the meta discussion! I believe you identified the problems quite clearly. I was not communicating well enough. Part of the problem was the format of the discussion, where (as a SIP committee) we decided we would split issues into separate threads and push them out individually for discussion. That means the current thread was greatly lacking context. I was amiss in giving a better motivation and framing and was consequently frustrated in seeing many counter-proposals that were already discussed and discarded previously on PRs.

So let me try to give some more background first and then discuss expectations.

Background

A big challenge in this discussion is that we come from widely different assumptions. A year ago I would have had a similar approach to many people here who are pushing back. But at some point last year I tried to step out of my comfort zone and asked myself a hard question:

• If implicits are so good why are they not the run-away success they should be? Why do the great majority of people who are exposed to implicits hate them, yet the same people would love Haskell’s type classes, or Rust’s traits, or Swift’s protocols? The usual answer I get from people who are used to current implicits is that we just need minor tweaks and everything will be fine. I don’t believe that anymore.

• Otherwise put: What can we learn from the other languages? The main distinguishing factor is that their term synthesis is separate from the rest of programming, and that they more or less hide what terms get generated. What terms are generated is an implementation detail, the user should not be too concerned about it.

• By contrast, Scala exposes implementation details completely, and just by adding implicit we get candidates for term inference. The advantage of that approach is that it is very lightweight. We only need one modifier and that’s it. The disadvantage is that it is too low-level. It forces the programmer to think in terms of mechanism instead of intent. It is very confusing to learners. It feels a bit like Forth instead of Pascal. Yes, both languages use a stack for parameter passing and Forth makes that explicit. Forth is in that sense the much simpler language. But Pascal is far easier to learn and harder to abuse. Since I believe that’s an apt analogy I also believe that fiddling with Forth (i.e. current implicits) will not solve the problem.

So that led to a new approach that evolved over time. Along the way many variants were tried and discarded. In the end, after lots of experimentation, I arrived at the following principles:

1. Implicit parameters and arguments should use the same syntax
2. That syntax should be very different from normal parameters and arguments.
   EDIT: In fact it’s better not to think of them as parameters at all, but rather see them as constraints.
3. The new syntax should be clear also to casual readers. No cryptic brackets or symbols are allowed.
4. There should be a single form of implicit instance definition. That syntax must be able to express monomorphic as well parameterized and conditional definitions, and stand-alone instances as well as aliases.
5. The new syntax should not mirror the full range of choices of the other definitions in Scala, e.g. val vs def, lazy vs strict, concrete vs abstract. Instead one should construct these definitions in the normal world and then inject them separately into the implicit world.
6. Imports of implicits should be clearly differentiated from normal imports.
7. Implicit conversions should be derived from the rest, instead of having their own syntax.

I arrived at these principles through lots of experimentation. Most of them were not present from the start but were discovered along the way. I believe these principles are worth keeping up, so I am pretty stubborn when it comes to weaken them. And I also believe that, given the mindshift these principles imply, there is no particular value to keep the syntax close to what it is now. In fact, keeping the syntax close has disadvantages for learning and migration.

Expectations

So, how can we make the discussion more productive and less frustrating for everyone?

• Both meta feedback and proposals to change details would be very valuable. They should be clearly identified as one or the other.

• Alternative proposals are welcome as well, but maybe they are better developed on separate threads.

• Actionable outcomes: I’d be glad if we could come up together with a number of changes to the proposal that we could agree on and that could be implemented in short order. It would be great if we could arrive at a go/no go decision of the whole thing by consensus, but I have the feeling that will be hard to achieve at present, since it would require more practical experience of people working with the new constructs (myself included).

• Alternatives could be worked out on separate threads. I’m happy to give feedback on the iterations. To be fully considered, they’d need to be at the same level of worked out detail as the current proposal. I.e. we need an informal spec and an implementation with which one can experiment.

• I don’t believe in surveys, because of selection bias. This is particularly pronounced here since the people participating in the survey would mostly be used to current implicits. If I had been asked a year ago what I prefer in a survey I would probably have picked current implicits as well!

• Repeated feedback: Is probably unavoidable since not everyone is current on what has been discussed and sometimes it is unclear why previous feedback was not incorporated. On the other hand, given the quantity of issues I am unable to respond to all repeated feedback in undisgested form. So, a proposal: Can we make a collective effort to process repeated feedback. E.g. if , say, explicitly is proposed , can we mine github history, note that it was proposed, try to distil the previous discussion and then continue here? That would make it much easier for me to respond and avoids monopolizing the thread with my comments.

Thanks again for the constructive feedback!

[mpol]

@curoli

Can’t we at least have some syntax that keeps the argument lists together (nothing but whitespace between them)? Just put something inside the parentheses instead of outside, like @Rex Kerr’s suggestion, or mine. Since mine with the dots was hard to read, how about sharps instead:

def m(# a: A, b: B #)(c: C, d: D)(# e: E, f: F #): R

For a discussion about concrete syntax it seems surprising that there was so little said about this or similar variants of marking implicit parameters with some type of “implicit parens”, different from the plain explicit ones.

This seems to give about the same benefits for the abstract syntax as the given keyword while possibly being more immediately recognizable on the surface.

[mpol]

@odersky

• That syntax should be very different from normal parameters and arguments
• The new syntax should be clear also to casual readers. No cryptic brackets or symbols are allowed.

This actually replies to my post even before I sent it, but there are trade-offs here and not everyone here seems to consider the same syntactic elements clear and cryptic.

[odersky]

This actually replies to my post even before I sent it, but there are trade-offs here and not everyone here seems to consider the same syntactic elements clear and cryptic.

See also New implicit parameter & argument syntax · Issue #1260 · lampepfl/dotty · GitHub for context.

[AMatveev]

When I am reading Contextual Abstractions at whole I think it is very good step forword.
I like separation from usual function arguments. It make using library with context bounds much more readable. (I think context bounds is the main feature which should be known by library users)
I like the most things except one:

• Implied Imports

When I think that someone should sometimes use that it seems terrible.
It means that every usage of that import leads a question to google in the best case.

I have only one question how I can easy get rid of the need to use Implied Imports, when I solve libraries integration tasks.

It seems there are no such tools in scala
I do not know the appropriate suggestion. But I think the answer is in scope management.
For example: https://kotlinlang.org/docs/reference/scope-functions.html

[odersky]

@Ichoran

Yes, I agree that is a problem, but I don’t worry too much about it, as it is a special advanced case. Most people would put their given parameters at the end, like they do now. And then it works well I think.

Can we then please limit it to the case where it works well? I don’t see any compelling use case for early given s since implicit arguments are going away and thus the call chain becomes unambiguous. Just create a temporary class or newtype that wraps the given and proceed from there.

I think we could do that. Having leading implicit parameters followed by normal parameters s nice for orthogonality, but there are no use cases that cannot be worked around easily. So I would be prepared to drop them if we get a better syntax for the common case instead. Not sure we still need several given sections in that case, there could be only one.

This restriction is consistent with thinking of implicit parameters as constraints, which is what e.g. Haskell does. If you communicate constraints, you are one level removed from the order in which your constraint evidence should be aligned in a list of curried parameters. So passing all constraint evidence at the end is a sensible thing to do, if we can work around the limitations,

[Jasper-M]

Has it been considered to give up on passing given parameters explicitly?
So you would have to write

implied for Ctx = new Ctx
foo()

to override the Ctx passed to foo. It might be slightly inconvenient sometimes but then at least the separation – between regular parameters which are passed in explicitly and given parameters which are inferred – is complete.

There is one thing I still find lacking in the proposal, even if I completely go along with the philosophy behind it. I think I’ve brought it up before but can’t recall getting any response.

implied impliedCtx for Ctx = new Ctx
def bar() given Ctx = ???
def foo() given (givenCtx: Ctx) = bar()

The Ctx passed to bar is givenCtx (which should definitely stay that way!). With implicits it was obvious that an implicit parameter was itself implicit. But now I find it completely non-obvious, because of the disconnect between the different keywords, that a given parameter is also implied.

[smarter]

@odersky

The main distinguishing factor is that their term synthesis is separate from the rest of programming, and that they more or less hide what terms get generated. What terms are generated is an implementation detail, the user should not be too concerned about it.

I think there’s a more fundamental difference, all these languages (I think ?) guarantee some form of “typeclass coherence”. That is, different instances/impls of typeclasses/traits should not overlap. Once you have that, hiding term inference makes sense since the term will always come from the same place, the same isn’t true in Scala and it seems to me that bringing over nameless instances but not coherence would be the worst of both worlds: how do I do “find all references” on a nameless implied ? And how can the compiler give me a useful error message when two nameless implied instances are ambiguous ?

[bishabosha]

Personally, I am already using this proposal along with the other contextual abstractions in a 5500+ line codebase. I really like the given keyword, I think it makes sense to say it’s some constraint on the current values in scope.

My only complaint is that, having confirmed with a non-scala programmer, that given that the inferrable parameters in def fancy given A, B, C = "Fancy!" do not really look like a parameter list, (which is the intent), why should they be applied as a whole list if you only want to update a single one?

for example, using only new features in this proposal:

trait A
trait B
trait C

def fancy given A, B, C = "Fancy!"

implicit object A_ extends A
implicit object B_ extends B
implicit object C_ extends C

println(fancy)

val newFancy = fancy given (implicitly[A], implicitly[B], new C {})

Now I must summon all the other values to change a single implied argument. However, I don’t think it’s possible to really prove which implied parameter list is the correct one if you allow partial application and generate the other arguments.

This causes me never to use given at the use site and instead update the implied scope in a previous statement in a block before calling, or otherwise restrict myself to only single argument implied parameter lists.

[DWvanGeest]

@odersky

In fact it’s better not to think of them as parameters at all, but rather see them as constraints

I’m not sure I agree with this. The previously mentioned use case of a CorrelationId does not seem like a constraint, just a parameter. Personally, I find this style to be very common and useful. I wonder, then, if this proposal is focusing on too narrow a use case, which may be resulting in the disagreement.

[bishabosha]

it’s not in this proposal, but the other abstraction: Context Queries would probably be most idiomatic for that argument:

type Transactional[O] = given CorrelationId => O

def getUser(id: String): Transactional[Future[User]]

or whichever is the most appropriate name.

[tarsa]

@odersky

If implicits are so good why are they not the run-away success they should be? Why do the great majority of people who are exposed to implicits hate them, yet the same people would love Haskell’s type classes, or Rust’s traits, or Swift’s protocols? The usual answer I get from people who are used to current implicits is that we just need minor tweaks and everything will be fine. I don’t believe that anymore.

One of main Rust’s strengths is friendliness of compiler errors. Rust compiler very often suggest possible corrections and the first one frequently works. If I forget some import (use in Rust parlance) needed for a typeclass to work, Rust compiler often suggest it to me. What will Scala compiler say?

object Main {
  implicit class RichInt(value: Int)(implicit name: String) {
    def print(): Unit =
      println(s"$name: $value")
  }
  def main(args: Array[String]): Unit = {
    // implicit val name: String = "bbb"
    5.print()
  }
}

value print is not a member of Int

Scala compiler doesn’t suggest any possible solution. Rust would search for some, order them by suitability and show e.g. 5 first ones.

Changing implicit to given won’t change the fact that Scala compiler doesn’t try to offer possible corrections.

IntelliJ offers implicits expansion display to help decrypting already working code that uses implicits: IntelliJ Scala plugin 2018.2: advanced “Implicit” support, improved patterns autocompletion, semantic highlighting, scalafmt and more | The IntelliJ Scala Plugin Blog That helps a lot, but works only when code is already correct. When trying to fix problems with incorrect code, compiler suggestions are required for good developer experience.

[odersky]

Are Swift protocol extensions coherent? I have not found anything asserting this. Anyway, I believe coherence is a minor concern, at best. And even with coherence the necessity to somehow identify instances does not go away. You either have to refuse two conflicting instances at the point where they are defined or at the point they are used, that’s all. But you don’t need a user-defined name for that.

[odersky]

I believe we have already made some progress with error messages, but further improvements would definitely help. I believe @olafurpg had some ideas about this. Anyway, any pull requests in that area would be greatly appreciated! But as you write yourself, that issue is orthogonal to the current discussion.

[AMatveev]

@DWvanGeest

I’m not sure I agree with this. The previously mentioned use case of a CorrelationId does not seem like a constraint, just a parameter.

IIUC in such cases a library user do not pass such argument into a function.
In many languages it can be implemented via thread variable.
It is useful in any case. But I think, such pattern is not a killer feature.

There are killer features of implicit parameters. And I agree that it is more like constraints in such cases

maximum(xs) given descending

It is more natural at least for people at our company(we use sql very often )

  select max(xs) over (order by salary)
     from table