Proposal To Revise Implicit Parameters

This thread is the SIP Committee’s request for comments on a proposal to revise implicit parameters in Scala. It forms part of a set of proposals that are collected under “Reference/Contextual Abstractions” in the Dotty Docs. Each of these proposals will get a separate discussion thread here. The relevant sections for the current proposal are Inferable Parameters and parts of Relationship with Scala-2 Implicits. Here’s a summary of the proposal that excerpts from these sections.

Note: To channel discussions, we are going to deal with individual “Contextual Abstractions” proposals in separate threads. But it would be good to read and absorb all proposals there together before starting to discuss.

Inferable Parameters

Functional programming tends to express most dependencies as simple function parameterization.
This is clean and powerful, but it sometimes leads to functions that take many parameters and
call trees where the same value is passed over and over again in long call chains to many
functions. Inferable parameters can help here since they enable the compiler to synthesize
repetitive arguments instead of the programmer having to write them explicitly.

For example, a maximum function that works for any arguments for which an ordering exists can be defined as follows:

def max[T](x: T, y: T) given (ord: Ord[T]): T =
  if (ord.compare(x, y) < 1) y else x

Here, ord is an inferable parameter. Inferable parameters are introduced with a given clause.
The max method can be applied as follows:

max(2, 3) given IntOrd

The given IntOrd part provides the IntOrd instance as an argument for the ord parameter. But the point of inferable parameters is that this argument can also be left out (and it usually is). So the following applications are equally valid:

max(2, 3)
max(List(1, 2, 3), Nil)

Anonymous Inferable Parameters

In many situations, the name of an inferable parameter of a method need not be
mentioned explicitly at all, since it is only used in synthesized arguments for
other inferable parameters. In that case one can avoid defining a parameter name
and just provide its type. Example:

def maximum[T](xs: List[T]) given Ord[T]: T =
  xs.reduceLeft(max)

maximum takes an inferable parameter of type Ord only to pass it on as an
inferred argument to max. The name of the parameter is left out.

Generally, inferable parameters may be given either as a parameter list
(p_1: T_1, ..., p_n: T_n) or as a sequence of types, separated by commas.

Inferring Complex Arguments

Here are two other methods that have an inferable parameter of type Ord[T]:

def descending[T] given (asc: Ord[T]): Ord[T] = new Ord[T] {
  def compare(x: T, y: T) = asc.compare(y, x)
}

def minimum[T](xs: List[T]) given Ord[T] =
  maximum(xs) given descending

The minimum method’s right hand side passes descending as an explicit argument to maximum(xs). With this setup, the following calls are all well-formed, and they all normalize to the last one:

minimum(xs)
maximum(xs) given descending
maximum(xs) given (descending given ListOrd)
maximum(xs) given (descending given (ListOrd given IntOrd))

Mixing Inferable And Normal Parameters

Inferable parameters can be freely mixed with normal parameters.
An inferable parameter may be followed by a normal parameter and vice versa.
There can be several inferable parameter lists in a definition. Example:

def f given (u: Universe) (x: u.T) given Context = ...

implied global for Universe { type T = String ... }
implied ctx for Context { ... }

Then the following calls are all valid (and normalize to the last one)

f("abc")
(f given global)("abc")
f("abc") given ctx
(f given global)("abc") given ctx

Syntax

Here is the new syntax of parameters and arguments seen as a delta from the standard context free syntax of Scala 3.

ClsParamClause    ::=  ...
                    |  ‘given’ (‘(’ ClsParams ‘)’ | GivenTypes)
DefParamClause    ::=  ...
                    |  GivenParamClause
GivenParamClause  ::=  ‘given’ (‘(’ DefParams ‘)’ | GivenTypes)
GivenTypes        ::=  AnnotType {‘,’ AnnotType}

InfixExpr         ::=  ...
                    |  InfixExpr ‘given’ (InfixExpr | ParArgumentExprs)

Relationship with current Implicits

The new inferable parameter syntax with given corresponds largely to Scala-2’s implicit parameters. E.g.

  def max[T](x: T, y: T) given (ord: Ord[T]): T

would be written

  def max[T](x: T, y: T)(implicit ord: Ord[T]): T

in Scala 2. The main difference concerns applications of such parameters. Explicit arguments to inferable parameters must be written using given , mirroring the definition syntax. E.g, max(2, 3) given IntOrd . Scala 2 uses normal applications max(2, 3)(IntOrd) instead. The Scala 2 syntax has some inherent ambiguities and restrictions which are overcome by the new syntax. For instance, multiple implicit parameter lists are not available in the old syntax. They require instead a complex emulation using auxiliary objects in the “Aux” pattern.

Context Bounds

Context bounds are the same in both language versions. They expand to the respective forms of implicit parameters.

Note: To ease migration, context bounds in Dotty map for a limited time to old-style implicit parameters for which arguments can be passed either with given or with a normal application. Once old-style implicits are deprecated, context bounds will map to given parameters instead.

Implementation Status and Timeline

The Dotty implementation implements both old-style implicit parameters and new-style given parameters. In fact, support for Scala-2’s implicits is an essential part of the common language subset between 2.13/2.14 and Dotty. Migration to the new abstractions will be supported by making automatic rewritings available.

It is planned to deprecate and phase out old-style implicit parameters in a version following Scala 3.0. The precise timeline will depend on adoption patterns.

3 Likes

Ok, so the advantage of this change is that there’s an unambiguous difference between implicit and explicit parameter lists at both the definition and the call site.
Wouldn’t it be more regular to achieve this with a different bracket style instead of with a keyword? The fact that you apply an argument to a method with a keyword seems pretty weird and unprecedented to me. The fact that (f given global)("abc") given ctx is a single method call is mind boggling.

Scala has 3 different types of parameter lists: types, explicit terms, implicits terms. We differentiate between the first 2 by using [] or () brackets. Why not use <> for the third?

Defining and calling method f becomes:

def f<u: Universe>(x: u.T)<Context> = ...

f("abc")
f<global>("abc")
f("abc")<ctx>
f<global>("abc")<ctx>
5 Likes

I think the variety of terminolohy could be frustrating
Single “implicit” tag-word now splitted to

  • given keyword
  • that means parameter should be inferred
  • implied definitions
  • context queries

Wouldn’t it be more wise to think about some single new word for all terms like

  • infer keyword
  • that means paarameter should be inferred
  • inferred definitions
  • infer functions
4 Likes

Multiple meanings of implicit were exactly the reason for splitting.

< and > are valid method/ operators names, so there would be some grammar headaches.

Something like ([ and ]) would be more easier to model in grammar, I think. Then the syntax would be: f([global])("abc")([ctx]) which is still relatively easy to grasp.

Or maybe go for: f(given global)("abc")(given ctx)??? That should be least confusing and also fit well into implicit hints in IDE: IntelliJ Scala plugin 2018.2: advanced “Implicit” support, improved patterns autocompletion, semantic highlighting, scalafmt and more | The IntelliJ Scala Plugin Blog

1 Like

Yes. I agree with that. However if we can just draw line between implicit parameters and implicit conversions, it would be enough.
implied, inferred, context queries and given all are about implicit parameters

6 Likes

IIUC implicit conversions are gone already in the new scheme. You need to replace them with typeclasses and/ or extension classes.

There was some effort to write the docs without using the term implicit at all. Just to see whether we could wean ourselves off. I expect that once the new proposals are in, we will come back to talk about implicit parameters, or implicit function types. So this is largely a matter of exposition only.

In terms of syntax, there’s one additional distinction compared to the status quo: Implicit parameters are named given and implicit instances are named implied. It would not work having one term for both because then you would have atrocities like

   given tc given X for Y { ... }

instead of

  implied tc given X for Y { ... }

Also, one of the iron rules of the design is that we want to align parameter syntax and call syntax. I believe that principle forces us to split instance and parameter syntax, otherwise things would get very confusing.

2 Likes

But can we at least call related term accordingly?
So for inferred paremeters we can use infer keyword and rename context queries to inferring functions or inferring expressions or something like that.

3 Likes

I love the capability!

I am less sold on the syntax. The main issue I see is that it sets up a clash between standard visual parsing of precedence and the actual precedence when you write

def f given (a: A)(b: B)

or, less bad but still not great,

def f(a: A) given (b: B): C

The problem is that both visually and by normal rules of precedence, paren-to-paren joins have extremely high precedence while space-separated words have very low precedence. Here it is the opposite: given has higher precedence than adjacent parens or type ascription. Consider, for contrast, if the following pairs were equivalent:

if (p) foo() else bar()()
(if (p) foo() else bar())()
if (p) foo else bar: B
(if (p) foo else bar): B

I don’t have an incredibly awesome idea of what to do about this. You can force the given to go into parens, but then without additional rules you lose the very pleasant

foo(x) given ctx

Nonetheless, I think the irregular precedence problem is sufficiently bad that we should try quite hard for an improvement. (I don’t know what erased means, but I think it has the same problem.)

The most boring possibility is just to restrict it to terminal position (leaving out the implicit syntax for simplicity):

DefParamClauses  ::= {DefParamClause} [[nl] GivenParamClause]
DefParamClause   ::= `(` (GivenParams | [DefParams]) `)`
GivenParamClause ::= 'given' ('(' DefParams ')' | GivenTypes)
GivenParams      ::= 'given' (DefParams | GivenTypes)

with other things as they are now. (Similar changes for ClsParamClauses, and ParArgumentExprs should get an extra entry for `’(’ ‘given’ ExprsInParens ‘)’.)


Again, I love the capabilities. It’s just the syntax which is weird.

4 Likes

Choosing
x.f(given global)("abc")(given ctx)
syntax over
(x.f given global)("abc") given ctx
reduces the distance between parentheses. Also, if x is a multiline expression then wrapping it in extra parentheses can lead to some formatting ugliness.

I believe for most cases given comes last and behaves syntactically as expected. For the few cases where it does not come last and is followed by normal parameters: yes that takes some getting used to. But note that these cases were not expressible at all until now, so they should be pretty rare.

Also note that in expressions given has the same precedence as alphanumeric infix operators, so no surprises there.

If having given inside the earlier parens is not implemented, maybe the existing syntax is fine but the recommended style for non-terminal given blocks should be

def foo[A] given(x: X[A]) (a: A) given(y: Y[A]): Foo[A]

to make it as visually clear as possible that the given is associated with exactly one parameter block following the keyword. The orphan (a: A) still looks kind of weird.

I assume allowing early given is desirable so partial application is a bijection, i.e. for every mix of regular and implicit functions val foo = implicit X => A => implicit Y => Foo there is a unique method representation, and for every method there is a curried form with implicit and regular functions?

Or is it just for path-dependent types (as in the example) where you want to avoid superfluous wrapper objects to get the desired API?

1 Like

The limitation of implicits in Scala 2 that is mentioned here is that you cannot have multiple implicit parameter lists.

But I can’t find how this proposal enables multiple implicit parameter lists? Any examples?

Also, why not just have Scala 2 implicit parameter syntax with multiple parameter lists? What’s wrong with:

def myMethod(x: Int)(implicit y: Int)(implicit z: Int): Int = ???

1 Like

I’m still unclear what problem this proposal is trying to solve. Is it so that you can have implicit parameter lists before regular parameter lists?

How about a place-holder for implicit/inferred parameter lists such as (…) for method calls. In method definitions, we can replace (implicit …) by (. … .)

Like:

def m(. a: A, b: B .)(c: C, d: D)(. e: E, f: F .): R

which would be equivalent to, if this was legal:

def m(implicit a: A, b: B )(c: C, d: D)(implicit e: E, f: F): R

This can be called like this:

m(a, b)(c, d)(e, f)

m(a, b)(c, d)

m(a, b)(c, d)(…)

m(…)(c, d)

m(…)(c, d)(…)

Anonymous parameters would be simply underscores, just like in pattern matching:

def m(. _ : A, _ : B .)(c: C, d: D)(. _ : E, _ : F .): R

Let’s please not. It’s visually very hard to parse. Additionally, it’s unpronounceable.

3 Likes

(Often?) you don’t need ... sections if you place . at the beginning of implicit arguments list.

Going back to my proposal we can have:

def f(given u: Universe)(x: u.T)(given Context) = ???
// callable as
f(given global)("abc")(given ctx) // all arguments explicit
f("abc") // Universe and Context are inferred
f(given global)("abc") // only Context is inferred

Presence of given at the beginning of arguments list determines whether we skip or fill implicit arguments list.

However, what if f returns a class with apply method which signature starts with implicit parameters list?

def f(a: Int) given (b: Int) = new X
class X {
  def apply given (b: Int)(c: Int) = ???
}
(f(5) given 8)(11)

Where the given 8 went? To f or to apply?

3 Likes

I like what you are suggesting.

But this approach still has problems. For instance, when you have

def f(given u: Universe)(given s: Bar)(x: u.T)(given Context) = ???

How would be

f(given a)("abc")

interpreted? Whether do we put the first given argument inferred or the second?


Of course, in practice we always can make the call more explicit, like

f(given a)(given the)("abc")

or

f(given the)(given a)("abc")

but the question is how to interpret an ambiguous case?

If you allow for consecutive skippable implicit argument lists then you’ll have such ambiguities in both my proposal and Martin’s one (I think). To disambiguate we could use solution proposed by @curoli which is:

f(given a)(given ...)("abc")
f(given ...)(given a)("abc")

... means any number of implicit arguments.

Consecutive implicit parameter lists probably won’t be popular enough to warrant such special syntax.

Probably in the simplest way - assume you always start with first implicit arguments list from the consecutive ones.

I think that for someone has no previous background in Scala, something like this would be a lot more intuitive:

def max[T](x: T, y: T, ord: Ord[T] = implicit): T =
  if (ord.compare(x, y) < 1) y else x
max(2, 3, ord = IntOrd)
max(2, 3)
max(List(1, 2, 3), Nil)

I personally do not like the given syntax, and think a default-arg-based syntax would be easier for everyone coming from programming languages with default args (i.e. all of them, these days). But by this point I’m probably sounding like a very small shell script, so I’ll leave it at that. Take from this what you will!

20 Likes

Lately I have been using definitions such as the following:

def foo[A, B, C](a: A, b: B)
                given Ctx1, Ctx2, Ctx3: Lifted[C] = ???

where to update one of the Ctx params I would use the implied for Ctx = ??? syntax. Otherwise, if you want to use given at the call site it’s best to define your inferrable parameters in a curried form, abstracted through types, or else you have to supply the entire list.

I do think it feels a bit surprising to someone unfamiliar with the desugared form that the compiler doesn’t generate the other parameters if you only supply one with given at the call site.