Proposal To Revise Implicit Parameters

I mostly just lurk in here, but I feel rather compelled to say something.

I’ve taught plenty of Scala newbies about the current implicit system. Conceptually, most folks understand it relatively quickly. It’s pretty straightforward and the fact that the implicit keyword is used in parameter lists and also when declaring implied values is a huge clue to what’s going on, even to the uninitiated. Many folks are also familiar with multiple parameter lists (or at least don’t see it as a terribly strange concept), so the idea of one parameter list being “different” is not such a great leap.

I do agree that there are warts, particularly the arbitrary limiting to a single, final implicit param list. I do not agree that these warts warrant such a drastic, complicated syntax change. I’ve read the proposal multiple times and am still struggling with some of the basics. What was appealing about the old implicit system is that it just made slight modifications to already familiar concepts. With the new system, fundamental things (like the definition and calling of a function) have become pretty unfamiliar through the insertion of given. The use of for is even more confusing.

I will echo what @lihaoyi, @mdedetrich and others have said: let’s identify the major pain points in the current implicit system and try to solve them with minimal changes. I think there have been many compelling alternatives presented that deserve serious consideration, alternatives that build on familiar concepts. This is one of the things I love most about Scala - is that you don’t need to understand everything deeply at first, but you can still have pretty good intuition about what it does, largely because it builds on familiar constructs from other languages. The old implicits, even though they’re a relatively unfamiliar concept, actually scaled gradually with the developer, following the Scala philosophy. I do not think the same can be said for the proposed system.

On a more meta level, I am also frustrated that this SIP process appears to be somewhat for show. @lihaoyi has been very diplomatic about pointing out some of the uncertainty around the goals of this process, but I’m afraid I will be more blunt. When highly-qualified contributors from the community spend hours engaging with the material, politely raising legitimate and well-reasoned concerns, and constructively presenting alternatives in good faith, why are they being dismissed so flippantly?

3 Likes

@lihaoyi Thanks for the meta discussion! I believe you identified the problems quite clearly. I was not communicating well enough. Part of the problem was the format of the discussion, where (as a SIP committee) we decided we would split issues into separate threads and push them out individually for discussion. That means the current thread was greatly lacking context. I was amiss in giving a better motivation and framing and was consequently frustrated in seeing many counter-proposals that were already discussed and discarded previously on PRs.

So let me try to give some more background first and then discuss expectations.

Background

A big challenge in this discussion is that we come from widely different assumptions. A year ago I would have had a similar approach to many people here who are pushing back. But at some point last year I tried to step out of my comfort zone and asked myself a hard question:

  • If implicits are so good why are they not the run-away success they should be? Why do the great majority of people who are exposed to implicits hate them, yet the same people would love Haskell’s type classes, or Rust’s traits, or Swift’s protocols? The usual answer I get from people who are used to current implicits is that we just need minor tweaks and everything will be fine. I don’t believe that anymore.

  • Otherwise put: What can we learn from the other languages? The main distinguishing factor is that their term synthesis is separate from the rest of programming, and that they more or less hide what terms get generated. What terms are generated is an implementation detail, the user should not be too concerned about it.

  • By contrast, Scala exposes implementation details completely, and just by adding implicit we get candidates for term inference. The advantage of that approach is that it is very lightweight. We only need one modifier and that’s it. The disadvantage is that it is too low-level. It forces the programmer to think in terms of mechanism instead of intent. It is very confusing to learners. It feels a bit like Forth instead of Pascal. Yes, both languages use a stack for parameter passing and Forth makes that explicit. Forth is in that sense the much simpler language. But Pascal is far easier to learn and harder to abuse. Since I believe that’s an apt analogy I also believe that fiddling with Forth (i.e. current implicits) will not solve the problem.

So that led to a new approach that evolved over time. Along the way many variants were tried and discarded. In the end, after lots of experimentation, I arrived at the following principles:

  1. Implicit parameters and arguments should use the same syntax
  2. That syntax should be very different from normal parameters and arguments.
    EDIT: In fact it’s better not to think of them as parameters at all, but rather see them as constraints.
  3. The new syntax should be clear also to casual readers. No cryptic brackets or symbols are allowed.
  4. There should be a single form of implicit instance definition. That syntax must be able to express monomorphic as well parameterized and conditional definitions, and stand-alone instances as well as aliases.
  5. The new syntax should not mirror the full range of choices of the other definitions in Scala, e.g. val vs def, lazy vs strict, concrete vs abstract. Instead one should construct these definitions in the normal world and then inject them separately into the implicit world.
  6. Imports of implicits should be clearly differentiated from normal imports.
  7. Implicit conversions should be derived from the rest, instead of having their own syntax.

I arrived at these principles through lots of experimentation. Most of them were not present from the start but were discovered along the way. I believe these principles are worth keeping up, so I am pretty stubborn when it comes to weaken them. And I also believe that, given the mindshift these principles imply, there is no particular value to keep the syntax close to what it is now. In fact, keeping the syntax close has disadvantages for learning and migration.

Expectations

So, how can we make the discussion more productive and less frustrating for everyone?

  • Both meta feedback and proposals to change details would be very valuable. They should be clearly identified as one or the other.

  • Alternative proposals are welcome as well, but maybe they are better developed on separate threads.

  • Actionable outcomes: I’d be glad if we could come up together with a number of changes to the proposal that we could agree on and that could be implemented in short order. It would be great if we could arrive at a go/no go decision of the whole thing by consensus, but I have the feeling that will be hard to achieve at present, since it would require more practical experience of people working with the new constructs (myself included).

  • Alternatives could be worked out on separate threads. I’m happy to give feedback on the iterations. To be fully considered, they’d need to be at the same level of worked out detail as the current proposal. I.e. we need an informal spec and an implementation with which one can experiment.

  • I don’t believe in surveys, because of selection bias. This is particularly pronounced here since the people participating in the survey would mostly be used to current implicits. If I had been asked a year ago what I prefer in a survey I would probably have picked current implicits as well!

  • Repeated feedback: Is probably unavoidable since not everyone is current on what has been discussed and sometimes it is unclear why previous feedback was not incorporated. On the other hand, given the quantity of issues I am unable to respond to all repeated feedback in undisgested form. So, a proposal: Can we make a collective effort to process repeated feedback. E.g. if , say, explicitly is proposed , can we mine github history, note that it was proposed, try to distil the previous discussion and then continue here? That would make it much easier for me to respond and avoids monopolizing the thread with my comments.

Thanks again for the constructive feedback!

4 Likes

For a discussion about concrete syntax it seems surprising that there was so little said about this or similar variants of marking implicit parameters with some type of “implicit parens”, different from the plain explicit ones.

This seems to give about the same benefits for the abstract syntax as the given keyword while possibly being more immediately recognizable on the surface.

This actually replies to my post even before I sent it, but there are trade-offs here and not everyone here seems to consider the same syntactic elements clear and cryptic.

This actually replies to my post even before I sent it, but there are trade-offs here and not everyone here seems to consider the same syntactic elements clear and cryptic.

See also New implicit parameter & argument syntax · Issue #1260 · lampepfl/dotty · GitHub for context.

When I am reading Contextual Abstractions at whole I think it is very good step forword.
I like separation from usual function arguments. It make using library with context bounds much more readable. (I think context bounds is the main feature which should be known by library users)
I like the most things except one:

  • Implied Imports

When I think that someone should sometimes use that it seems terrible.
It means that every usage of that import leads a question to google in the best case.

I have only one question how I can easy get rid of the need to use Implied Imports, when I solve libraries integration tasks.

It seems there are no such tools in scala :frowning:
I do not know the appropriate suggestion. But I think the answer is in scope management.
For example: https://kotlinlang.org/docs/reference/scope-functions.html

I think we could do that. Having leading implicit parameters followed by normal parameters s nice for orthogonality, but there are no use cases that cannot be worked around easily. So I would be prepared to drop them if we get a better syntax for the common case instead. Not sure we still need several given sections in that case, there could be only one.

This restriction is consistent with thinking of implicit parameters as constraints, which is what e.g. Haskell does. If you communicate constraints, you are one level removed from the order in which your constraint evidence should be aligned in a list of curried parameters. So passing all constraint evidence at the end is a sensible thing to do, if we can work around the limitations,

1 Like

Has it been considered to give up on passing given parameters explicitly?
So you would have to write

implied for Ctx = new Ctx
foo()

to override the Ctx passed to foo. It might be slightly inconvenient sometimes but then at least the separation – between regular parameters which are passed in explicitly and given parameters which are inferred – is complete.

There is one thing I still find lacking in the proposal, even if I completely go along with the philosophy behind it. I think I’ve brought it up before but can’t recall getting any response.

implied impliedCtx for Ctx = new Ctx
def bar() given Ctx = ???
def foo() given (givenCtx: Ctx) = bar()

The Ctx passed to bar is givenCtx (which should definitely stay that way!). With implicits it was obvious that an implicit parameter was itself implicit. But now I find it completely non-obvious, because of the disconnect between the different keywords, that a given parameter is also implied.

I think there’s a more fundamental difference, all these languages (I think ?) guarantee some form of “typeclass coherence”. That is, different instances/impls of typeclasses/traits should not overlap. Once you have that, hiding term inference makes sense since the term will always come from the same place, the same isn’t true in Scala and it seems to me that bringing over nameless instances but not coherence would be the worst of both worlds: how do I do “find all references” on a nameless implied ? And how can the compiler give me a useful error message when two nameless implied instances are ambiguous ?

3 Likes

Personally, I am already using this proposal along with the other contextual abstractions in a 5500+ line codebase. I really like the given keyword, I think it makes sense to say it’s some constraint on the current values in scope.

My only complaint is that, having confirmed with a non-scala programmer, that given that the inferrable parameters in def fancy given A, B, C = "Fancy!" do not really look like a parameter list, (which is the intent), why should they be applied as a whole list if you only want to update a single one?

for example, using only new features in this proposal:

trait A
trait B
trait C

def fancy given A, B, C = "Fancy!"

implicit object A_ extends A
implicit object B_ extends B
implicit object C_ extends C

println(fancy)

val newFancy = fancy given (implicitly[A], implicitly[B], new C {})

Now I must summon all the other values to change a single implied argument. However, I don’t think it’s possible to really prove which implied parameter list is the correct one if you allow partial application and generate the other arguments.

This causes me never to use given at the use site and instead update the implied scope in a previous statement in a block before calling, or otherwise restrict myself to only single argument implied parameter lists.

I’m not sure I agree with this. The previously mentioned use case of a CorrelationId does not seem like a constraint, just a parameter. Personally, I find this style to be very common and useful. I wonder, then, if this proposal is focusing on too narrow a use case, which may be resulting in the disagreement.

it’s not in this proposal, but the other abstraction: Context Queries would probably be most idiomatic for that argument:

type Transactional[O] = given CorrelationId => O

def getUser(id: String): Transactional[Future[User]]

or whichever is the most appropriate name.

One of main Rust’s strengths is friendliness of compiler errors. Rust compiler very often suggest possible corrections and the first one frequently works. If I forget some import (use in Rust parlance) needed for a typeclass to work, Rust compiler often suggest it to me. What will Scala compiler say?

object Main {
  implicit class RichInt(value: Int)(implicit name: String) {
    def print(): Unit =
      println(s"$name: $value")
  }
  def main(args: Array[String]): Unit = {
    // implicit val name: String = "bbb"
    5.print()
  }
}
value print is not a member of Int

Scala compiler doesn’t suggest any possible solution. Rust would search for some, order them by suitability and show e.g. 5 first ones.

Changing implicit to given won’t change the fact that Scala compiler doesn’t try to offer possible corrections.

IntelliJ offers implicits expansion display to help decrypting already working code that uses implicits: IntelliJ Scala plugin 2018.2: advanced “Implicit” support, improved patterns autocompletion, semantic highlighting, scalafmt and more | The IntelliJ Scala Plugin Blog That helps a lot, but works only when code is already correct. When trying to fix problems with incorrect code, compiler suggestions are required for good developer experience.

1 Like

Are Swift protocol extensions coherent? I have not found anything asserting this. Anyway, I believe coherence is a minor concern, at best. And even with coherence the necessity to somehow identify instances does not go away. You either have to refuse two conflicting instances at the point where they are defined or at the point they are used, that’s all. But you don’t need a user-defined name for that.

I believe we have already made some progress with error messages, but further improvements would definitely help. I believe @olafurpg had some ideas about this. Anyway, any pull requests in that area would be greatly appreciated! But as you write yourself, that issue is orthogonal to the current discussion.

IIUC in such cases a library user do not pass such argument into a function.
In many languages it can be implemented via thread variable.
It is useful in any case. But I think, such pattern is not a killer feature.

There are killer features of implicit parameters. And I agree that it is more like constraints in such cases

maximum(xs) given descending

It is more natural at least for people at our company(we use sql very often )

  select max(xs) over (order by salary)
     from table

Thread locals aren’t compatible with asynchronous programming in general and that bitten me a lot when using Futures with LiftWeb, which uses a lot of thread locals. Personally, I use thread locals only as a last resort to avoid headaches and I don’t have much positive experience with them.

You also get no compilation errors or IDE support when a thread local is missing or set to wrong value. It can also be harder to see where thread local comes from as you can set thread local in some very deeply nested method, whereas implicits are passed from higher level method to lower level method directly.

Version with thread locals:

def highLevel() = {
  runSomeArcaneMachineryToSetSingleThreadLocalTo(computeValueUsingHeavyMachinery())
  lowLevel()
}

def lowLevel() = {
  val value = searchDeepSomewhereToGetThreadLocal() // first you need to make sure which thread local container is the correct one
  println(value)
}

Version with implicits:

def highLevel() = {
  // you can't push implicit definition into some deeply hidden method, it has to be in scope here
  implicit val anImplicit = computeValueUsingHeavyMachinery()
  lowLevel()
}

// you don't need to figure out which implicits are available for you, because you have them all in the signature
def lowLevel()(implicit value: Int) = {
  println(value)
}

You could use a local implied instead as @Jasper-M suggests:

{ implied for c = new C
  val newFancy = fancy
}

@Jasper-M:

Yes, we could do without given in applications. But I have the impression it’s a useful functionality to have. The workaround of local implied is a bit clunky at times.

I do think that given already has the connotation of propagating automatically to callees. If I am allowed to take some property as a given, everything I call is allowed to assume the same property. The situation is really analogous to other languages where there is one construct to require and propagate constraints (in Haskell: … => …) and another to establish base properties (in Haskell: instance).

You are right my previous saying is a little extreme.
The main idea is that in such case a library user can even do not know about such parameters. It works in background. So the syntax does not matter(at least for me).

It’s a good observation. But note that CorrelationID could not be any parameter type. It could not be an alias of Int, say, that would be a terrible thing to do. So: It needs to be a special type, and the constraint would be “there is an instance of CorrelationId in scope”. True, sometimes it is more direct to think of these things as parameters, and you can. But thinking of constraints instead gives better guidance. The statement “here is an (implicit x: Int) parameter” looks OK to beginners at first. “There is an instance of Int in scope” is immediately seen as non-sensical. So, better guidance.