Proposal To Revise Implicit Parameters

@odersky in fact, it might be worth posting your entire Background section as a separate top-level post, so it can be discussed independently, both separate from this thread (where it’ll get buried) and separate from any concrete implementation/proposal (since it really is its own thing).

I like the idea. That would provide a natural place to have meta-discussions about the design approach. In the end, everything hangs together so it’s hard to have these meta discussions on individual features.

I may take you up on your offer to review an alternate proposal, if I have time to put one together in a proper amount of detail, to see how things would look if we went the other route (i.e. implicits more similar to normal params, rather than more different)

Yes, I think that would be good to work out!

I believe there is one use case for implicits as defaults that neither current implicits nor the new proposal cover well. It’s where you want to pass an argument implicitly without polluting the name-space with further implicit instances. That’s what macwire does. I asked @adamw why he needed to invent macwire instead of just using implicit parameters for dependency injection. He responded that using implicits would make every implicitly injected component an implicit instance that was visible everywhere, which is neither needed nor wanted. But if default arguments could be implicits, we can formulate it like this:

class GreenModule(red: RedModule = implicit) { ... }
class RedModule(green: GreenModule = implicit, blue: BlueModule = implicit) = { ... }
class BlueModule(red: RedModule = implicit) { ... }

// wiring components together
object System {
  private implicit object green extends GreenModule()
  private implicit object red extends RedModule()
  private implicit object blue extends BlueModule()
}

This works since the implicit objects green, red, blue are confined to the System scope. It shows a general tradeoff between how “special” an implicit type should be vs how confined its instances are. In the macwire example, the implicit types are general components, so not very special. That works if the implicit instances of these types are very confined.

Anyway, it might be worth to pursue this line of thought either in a standalone system or in the context of another proposal.

1 Like

There could be another util method for that:

def given[T,E](a:T)(f:ImplicitFunction1[T, E]) = f(a)
//---------------
val newFancy = given(new C) { fancy }

Polluting the implicit namespace is only one of the reasons. The other (maybe even more important) is that the technical details of how “dependency injection” or “create the module graph” is done (which are the same thing) leak into the implementation of the modules.

With the plain-constructors approach, you simply have a class (a module) which declares its dependencies as constructor parameters. That’s it - no @Inject annotation, no implicit keywords etc. The class knows absolutely nothing and doesn’t care on how it’s going to be wired together with other modules.

Now, macwire is only a convenience thing to avoid writing a lot of new ... invocations. But to be honest, in most smaller projects nowadays I simply write the new ... without any macros :). But in larger projects macwire is still very useful.

Adam

2 Likes

Well, considering that C was not only once the most popular language, but also fundamentally influenced the syntax of almost every popular language today except Python, they must have been doing something right.

Since C heavily relies on pointers, you need ways to reference and dereference instances, and since it is statically typed, you need types and for every type a pointer-of type. That’s a challenge to notate. As Python has neither pointers nor static types, it is so much simpler.

So another proposal can’t hurt, right? So I thought I’d drop in my own! This is actually my first time writing a proposal like this, it was pretty fun! :slight_smile:

This does involve both the syntax for taking in implicits and declaring them, but I couldn’t figure out how to meaningfully split it up across the two discussion topics. This ended up being pretty long, so I can put this as a separate topic if people think that would be a good idea.

Overall Idea

I got this idea after speaking with some friends from my high school (now a mix of first year college students and junior/senior high school students), where I led the use of Scala as the core language for our robotics team. We had used implicits quite heavily in our robotics framework, but definitely saw similar issues as those reported in this thread with implicits being hard to pick up.

The summary of our discussions was that the core issue with implicits is not the syntax or overall idea, but instead the confusion that happens when they don’t “just work”. In this situation, it becomes quite difficult to guess (at a conceptual/mental level) where implicit parameters were (or were unable to be) sourced from especially because there are multiple ways implicit parameters can be synthesized.

As I see it, implicit parameters serve two core and rarely overlapping roles: to provide context (“repetitive arguments” in the original proposal, such as the CorrelationId argument) and to derive instances of typeclasses (and similar constructs that associate a type with some behavior). So this proposal splits up implicits into two separate concepts: context and derived.

Trying to write out a detailed proposal, I tried a couple experiments, so I decided to add in all the experiments after the main proposal here with comments on what I was aiming to do with each and why I decided to drop the ones I did.

Context

The first and more straightforward part here is context, since passing it down is already a simpler concept. The goal here is simple: reduce the amount of code written when some context is repeatedly passed down through functions.

One special property of context when passed down in programs is that it is a fixed instance that does not need to be generated at each level. Functions simply take in the context and pass it down to functions in their body that need it. To match this, context values must always be defined as vals.

case class Config(appName: String)

// elsewhere...

object MyApp {
  context val config = Config("Demo App")
}

Restricting context definitions to vals cuts down on the potential sources for where a context parameter could have come from, which reduces the mental load for developers trying to understand where a context value was sourced.

To take in context, the syntax is identical to implicits right now:

def displayName(name: String)(context config: Config): String = {
  s"My name is $name -- generated by ${config.appName}"
}

Passing through context is also the same:

def displayNames(names: Seq[String])(context config: Config): String = {
  names.map(displayName).mkString("\n") // displayName eta-expanded to (name) => displayName(name)(config)
}

Manually passing in context is a bit different, it requires an explicit context keyword to clarify that this is a special type of parameter and to distinguish it from derived later:

test("displayName shows correct app name suffix") {
  assert(displayName("Test Name")(context Config("Test App")) ==
    "My name is Test Name -- generated by Test App")
}

implicitly[T] here is replaced with context[T]

Relation to Implicit Functions

Context values should work well with implicit functions, which (at least in the code I have seen/imagined) would usually take some context value to reduce the repetitiveness of taking parameters over and over. More analysis is needed to understand how these two concepts would interact with each other.

Slight Aside: Similarities to React

As I wrote down this proposal, I realized that context here aligns very closely to context in React.js, which has been a very successful feature. In React, context can be added to an application by injecting it with a context provider component, which makes the context available to components rendered inside it.

(from the React docs)

// Context lets us pass a value deep into the component tree
// without explicitly threading it through every component.
// Create a context for the current theme (with "light" as the default).
const ThemeContext = React.createContext('light');

class App extends React.Component {
  render() {
    // Use a Provider to pass the current theme to the tree below.
    // Any component can read it, no matter how deep it is.
    // In this example, we're passing "dark" as the current value.
    return (
      <ThemeContext.Provider value="dark">
        <ThemedButton />
      </ThemeContext.Provider>
    );
  }
}

class ThemedButton extends React.Component {
  // Assign a contextType to read the current theme context.
  // React will find the closest theme Provider above and use its value.
  // In this example, the current theme is "dark".
  static contextType = ThemeContext;
  render() {
    return <Button theme={this.context} />;
  }
}

The context provided is always sourced from a fixed value (passed in as the value property). As it moves down the tree, the only way context can be modified is to re-inject a replacement value with another provider. This mirrors the proposal for context here, where context can be injected from a fixed context val value and can only be replaced down the function tree by explicitly passing in an alternative context.

Derived

Now comes the second half of what implicits let us do: derive behavior for specific types. Splitting this part off into a separate construct both helps developers conceptualize where values come from (just like context) and also aligns the feature with equivalents in other languages like Rust derived traits.

When declaring a derivation for a particular typeclass, there are four pieces of information: a name for the derivation to distinguish it from others, other context and derived values it needs, what type of object is being derived, and finally the actual implementation. Method declarations have a similar format, so I decided to just adapt that format with def replaced with derived.

trait Reader[T] {
  def read(str: String): T
}

object Reader {
  // first, a simple reader for parsing ints
  derived intReader = new Reader[Int] {
    def read(str: String) = str.toInt
  }

  // equivalent to

  derived intReader: Reader[Int] = _.toInt // single abstract method style
}

Taking in derived parameters is just like implicits (and thus like context). With derived definitions and parameters combined, we can then have recursive derivations.

derived seqReader[T](derived elemReader: Reader[T]) = new Reader[Seq[T]] {
  def read(str: String) = str.split(' ').toSeq.map(elemReader.read)
}

Finally, derivations can be manually passed in with a derived keyword marking the set of parameters as belonging to that set.

// parses strings like 1,2,3 4,5,6 to Seq(Seq(1, 2, 3), Seq(4, 5, 6))
def parseSpacesThenCommas[T](str: String)(derived elemReader: Reader[T]): Seq[Seq[T]] = {
  val commasReader = new Reader[Seq[T]] {
    def read(str: String) = str.split(',').toSeq.map(elemReader.read)
  }
  
  // things are much clearer when using named parameters, but I'm not sure if that should be enforced
  seqReader(derived elemReader = commasReader).read(str)
}

implicitly[T] here is replaced with derived[T]

Migration from Scala 2

To handle migration to this new style, I would propose allowing old-style implicits to fulfill both context and derived values, since this proposal just splits the two uses of implicits but does not significantly change the semantics of either. Similarly, context and derived can fulfill implicit parameters.

Experiment 1: anonymous context

Often times a context parameter will be taken in, not used in the body, but just passed down to another function. In this case, having to name all the context parameters adds unnecessary overhead. So one idea is to just make naming context values optional (then the two syntaxes are name: Type or just Type separated by commas). For example, this would enable usage such as:

def displayNames(names: Seq[String])(context Config): String = {
  names.map(displayName).mkString("\n")
}

This doesn’t complicate the syntax too much and could clean up a lot of code, so I think this would be a helpful addition to the proposal.

Experiment 2: type-named context

Inspired by React’s API for getting context values with contextType (used in the above example), this experiment drops context parameter names completely in favor of identifying context data by the type.

def displayName(name: String)(context Config): String = {
  s"My name is $name -- generated by ${Config.appName}"
}

While a nice shorthand, I am afraid this will cause too many conflicts with companion objects and is potentially very confusing, so I don’t think this should be implemented.

3 Likes

After playing around with implied instances and reading all docs, I’m intrigued about this approach. I don’t have anything to add semantic wise at this point, I like its current principles. But, given that this change is mostly about syntax, I think it might be useful to mention I struggle to get used to the implied syntax.

Isn’t there a better keyword we could use to refer to the same concept? Possibly a keyword that bears no indirect relationship with implicit and that doesn’t read as an adjective?

The one that comes to mind is infer. Given that implicits is a way to infer terms, it seems appropriate to use that word instead and it being a verb means it reads like I’m telling the computer to do something.

Here’s how the code snippet in Implied Instances would look like with it:

trait Ord[T] {
  def compare(x: T, y: T): Int
  def (x: T) < (y: T) = compare(x, y) < 0
  def (x: T) > (y: T) = compare(x, y) > 0
}

infer IntOrd for Ord[Int] {
  def compare(x: Int, y: Int) =
    if (x < y) -1 else if (x > y) +1 else 0
}

infer ListOrd[T] given (ord: Ord[T]) for Ord[List[T]] {
  def compare(xs: List[T], ys: List[T]): Int = (xs, ys) match {
    case (Nil, Nil) => 0
    case (Nil, _) => -1
    case (_, Nil) => +1
    case (x :: xs1, y :: ys1) =>
      val fst = ord.compare(x, y)
      if (fst != 0) fst else xs1.compareTo(ys1)
  }
}

Advantages of using infer over implied:

  1. It’s a verb and it’s similar to def both syntax wise (it comes from the verb define) and semantics-wise (it defines an inferred term without looking like a straight value definition).
  2. It’s two characters shorter.
  3. it reads nicely followed by for and given.

Disadvantages

  1. It’s not clear what we should use to import an inferrable instance. I prefer either infer import or import infer but we could also go for import inferred. If you don’t like either, we can probably find another variation that fits the bill.

EDIT: I’ve just seen @Odomontois already proposed the same thing in passing but no one seems to have given feedback on the suggestion.

1 Like

Hi,
I find the new syntax unfamiliar and looking like another sublanguage.
My expectation would be to learn another concept on top what I already have learned.
So I first learn about classes and traits, them objects and companion objects.
Next, come implicits. At this point, I know all basic Scala syntax

A thing I like a lot is companion objects.
They are easy to understand, with clear syntax and benefits (them being singletons).

The thing I would not like to see is mixing companion objects with implicits/typeclass instances.
I think of this as breaking the single responsibility principle.

My proposal is to get rid of the new keywords and stick to the syntax everyone already knows.
With the addition of introducing another “companion object like” keyword.

trait Ord[T] {
    def compare(x: T, y: T): Int
    def (x: T) < (y: T) = compare(x, y) < 0
    def (x: T) > (y: T) = compare(x, y) > 0
}

object Ord {
    // standard companion object stuff
    // the below "implicit companion object" could also be here
    // but my preference would be to break these apart
}

[typeclasses | instances | implicits | infer] Ord { // <- choose your favourite keyword
    // inside all val's and def's are implicit but the syntax should be 100% normal scala syntax

   val intOrd: Ord[Int] = { // 
       def compare(x: Int, y: Int) =
          if (x < y) -1 else if (x > y) +1 else 0
   }

   def listOrd[T](ord: Ord[T]): Ord[List[T]] = { // here all the arguments and implicit
       def compare(xs: List[T], ys: List[T]): Int = (xs, ys) match {
         // ...
       }
   }
}

As far as imports are concerned
I am in favor of making imports explicit

import my.Ord // imports class and companion object without any implicits
import instances my.Ord // import implicits

Advantages:

  1. The same, easy syntax you’ve already before you reached the implicit level
  2. Separation of companion object from implicits
  3. New separate rules can be defined to work inside the new “instances” companion object.
  4. No new syntax
  5. Cleaner syntax (no repetition of implied, inferred, for, given) with instance every definition

Disadvantages:

  1. Probably plenty, I am sure You can find more than I

I too think it is not worth having it in the language. It looks confusing and messy, and in the context of trying to sell scala in a java shop maybe even scary. I can’t imagine selling that at an average-ish workplace, if adoption is a concern.

I have had the complete opposite experience in teaching current implicit to the “averagish” Java workshop. Current implicits are basically parameter passing (with some quirks) which Java people understand very well. Then when I explain to them how the implicit part works, I just draw comparisons with Guice (which does similar things as implicits but at runtime) and then they understand fine.

I think your averag’ish Java developer would have more difficulty understanding something that is being packaged closer to theorem proving rather than parameter passing.

2 Likes

Not sure how your comment relates to mine, since I was reacting to the part of the discussion concerned with limiting the usage of given “to the case where it works well”, while you are comparing it with current implicits. Unless I misunderstood something.

Slightly misread what you were saying, in which case I agree with you (my point is that I think the new proposal is actually scarier than what we currently have)

I like the recent changes concerned with contextual abstractions. I think what you are saying is right, on the other hand though, when the ease of understanding of the concept of implicits is considered together with also having to understand the patterns it is used in, the proposed changes work better. The “expressing intent vs expressing mechanism” argument works for me.

It might turn off slightly more people in the earliest stages of learning than implicits. But to me it seems like a large subset of those would end up not liking scala with implicits either, and I’m yet to see a single person from that group to change their mind. Often the issue is that scala is just not in the part of the typesystem/PL landscape that they are prepared to leave anyway.

The proposed changes might serve better than implicits those newcomers who were more likely to stay with the language just a little bit longer in the first place, and it is this latter group that has the potential to impact adoption.

1 Like

If given is an infix operator then why can’t we turn it into a dotted notation? We as Scala programmers know that generally object operator argument can be rewritten as object.operator(argument). If that applies to given as well then normalization could look like this:

f("abc")
f.given(global)("abc")
f("abc").given(ctx)
f.given(global)("abc").given(ctx)

Which looks much saner to me and has the very desirable (IMO) property that there’s less distance between opening and closing parentheses, e.g. f.given(global)("abc") as opposed to (f given global)("abc").

7 Likes

My first reaction: I like .given. That reads even better than as an infix operator.

9 Likes

Sorry, only just saw this. Apart from the “it works better” reasons for liking the implied given for syntax (modulo I’d wish to change implied to instance or deriving in the declarations) is that it feels like a much cleaner and distinct model for describing derivable values, and how to construct them from one-another. The implied def/object/val constructs in scala 2.* always felt to me as if derived values were being simulated using a source-level DSL within scala. That we were writing the plumbing of an implementation of this machinery. Using implied given for, there is a single, first-class syntax for this, which only does this, and does all of this. It is far, far clearer to see at a glance what code is introducing and manipulating supplied values, and what code is introducing and manipulation derived values. I find it a much lower cognitive overhead writing even quite complex stacks of derived instances using the new syntax, that I’ve never been able to do in practice with the old syntax.

2 Likes

That would be lovely!

(I think it would be great if match could be also; but I’ll take every dottable operator-or-method-like thing I can get. I always appreciate the regularity-of-flexibility in invocation.)

1 Like

Too bad I missed most of the discussion (since it happened on Passover).

I think one thing that hasn’t really been expressed is that there’s a huge risk involved. We don’t know what effect it will have on people open to begin using Scala or to stop using Scala. Maybe it will make more people learn Scala. Maybe it won’t. Maybe it will cause people that were on the fence to stop using Scala. Maybe it won’t. We don’t know.

Personally, my reaction to the new syntax is that perhaps it could be justified, but it’s not Scala. It feels like after a decade of using Scala, to continue using it I have to learn a new language now. :frowning:

Because of the above I feel very strongly that we should take this slowly. There is already a huge amount of changes in Scala 3.0. There is no reason that this has to be in the same release. It can wait. (Personally, I would make the release that includes this Scala 4.0, even if it were released only a week after 3.0.)

Meanwhile, all of the proposed improvements to implicits that don’t break code or involve paradigm shifts, should go into 3.0. Or 2.14. Those have the least risk for the greatest gain. Then we can decide whether it’s worth the bolder move.

2 Likes

IMO given doesn’t make Scala much more different. Overall it seems that what given changes is giving implicit arguments less priority than regular arguments. It won’t substantially change the shape of Scala programs if method notation for given arguments (i.e. f.given(a)(b).given(c)) will be used.

OTOH removing assignment operator from typeclasses instances definition could require tedious restructuring. instance abc of Type given xyz { ...class members... } is greatly limited compared to implicit abc: Type given xyz = anything.
Update: I missed aliases in implied instances proposal. Well, it changes the situation then.

I love the simplicity and targeted nature of your proposal.

What do you think of just enforcing that passing implicit arguments must be by-name instead of positional? Instead of:

Just doing:

maximum(xs)(ord = descending)

Do you think that would solve the same problem and be even less radical of a change?

The problem with your proposal is that it could also be a normal argument list, and you cannot see it from here.

It also goes against two of the principles as discussed in the other thread (leaving out of scope whether or not you agree with them):

  • It doesn’t mirror call and use sites.
  • It doesn’t distance implicit parameters from normal parameters anymore than is done now, which is one of the reasons the change it.
1 Like