Proposal To Revise Implicit Parameters

I think you’re over-interpreting this. I mean, Martin does not have the final say here, the SIP committee does. Yes, he’s extremely influential in the process, but we’ve long since moved on from “everything Martin says is final”.

He’s stating his strongly-held opinions. That isn’t the same thing as meaning that this is no longer open to discussion – it simply says what it says, that he’s spent a great deal of time hashing through this (probably more than anybody else here), and these principles seem to him to be crucial for success.

In various places, over the span of the past year or two, sure. None of this is coming straight out of the blue in this conversation…

I have clarified the guiding principles behind the design, and told you that I will be stubborn defending them. That leaves several avenues for productive feedback open:

  • you can work within the principles and make suggestions for improvements - this would be very welcome, and has the best chance to influence the design.
  • you can start a meta discussion about the principles, and suggest different ones. This would also be interesting if it is done thoughtfully.
  • you can work on an alternative proposal. That would admittedly imply a long stretch to acceptance but nothing is ruled out at this point.

Really? Martin doesn’t have the final say? Hey, I didn’t know that!

How is the SIP committee selected?

Your comments are now completely off-topic. If you’d like to discuss governance more, please open a separate thread.

It would be great to know a bit more the reasoning behind those guidelines.

Let’s see:

(1) Implicit parameters and arguments should use the same syntax

I think I’m actually cool with that one.

(2) That syntax should be very different from normal parameters and arguments

Hm, how different would qualify as very different? Does it have to be an extra keyword?

(3) The new syntax should be clear also to casual readers. No crypic brackets or symbols are allowed.

What do we mean by “cryptic” - or does this mean any brackets or symbols are forbidden, so that it does have to be an extra keyword?

Is the consequence of (1), (2) and (3) that there just has to be an extra keyword for every implied parameter list, both at declaration and call site? Or is there any other option left?

I would suggest that operationally that someone only mildly familiar with Scala should be able to see that something different is going on, and someone expert with Scala glancing quickly at the code should be unable to mistake it for anything else.

This doesn’t completely rule out non-keyword based solutions, but it severely limits them. It also has to be pleasant to use, which probably kills the remaining symbolic solutions given that Scala doesn’t rely overly much on symbols. For instance,

def foo(i: Int ;; Context): Foo = ???

foo(7)
foo(3 ;; ctx)

meets the “very different” criteria but probably not “aesthetically pleasing” except to people like me who really like symbols. (FWIW, I would be totally happy with the above syntax.)

I think something like ;; jammed into the middle of an argument list is as clear as most key words–it’s clearly demarcating something very different, and the stuff afterwards is also clearly some sort of parametery thing, which is about as much as a casual reader can get from a keyword like given.

However, the “eew, symbol soup, I hate soup with symbols in it!” reaction is probably less good than “Oh, given? I like being given things!”.

Anyway, I don’t think symbols are necessarily out, but given how much displeasure people have expressed about relatively innocuous stuff like /:, I think it’d have to be quite marvelous in order to have a chance.

I think people’s displeasure with symbols was mostly against the heavy use of symbols as identifiers, which especially one particular library was doing. Symbols as other syntax elements should be fine.

No, I don’t think so. Witness people’s relative pleasure in using Python where there symbolic syntax is used very sparingly (mostly it’s words); witness also the frequent consternation about (char*)&(*(x+7)) stuff in C/C++ (not to mention lambda notation).

Symbols typically require a great deal more expertise to use proficiently, and thus are only worth it when the thing in question is so incredibly common that even a few extra characters is an unbearable burden, or when it is so important that things be visually distinct that it warrants devoting a new glyph to it.

There are nonetheless plenty of cases where symbols are critical, but the bar is high.

Thanks Martin! Having this end-goal front and center definitely helps provide motivation for a lot of this proposal. I’m not sure I agree with the end goal, or the current details, but having this explicitly stated definitely helps put a lot of things in context. I don’t have any immediate response but it’s definitely something for me to think about

I may take you up on your offer to review an alternate proposal, if I have time to put one together in a proper amount of detail, to see how things would look if we went the other route (i.e. implicits more similar to normal params, rather than more different)

@odersky in fact, it might be worth posting your entire Background section as a separate top-level post, so it can be discussed independently, both separate from this thread (where it’ll get buried) and separate from any concrete implementation/proposal (since it really is its own thing).

Even if you yourself are 98% committed to that approach, having a separate discussion on the fundamental principles could be a good way to sell or refine the idea and try to get people on board, more so than spending time bikeshedding over the spelling of keywords.

1 Like

@odersky in fact, it might be worth posting your entire Background section as a separate top-level post, so it can be discussed independently, both separate from this thread (where it’ll get buried) and separate from any concrete implementation/proposal (since it really is its own thing).

I like the idea. That would provide a natural place to have meta-discussions about the design approach. In the end, everything hangs together so it’s hard to have these meta discussions on individual features.

I may take you up on your offer to review an alternate proposal, if I have time to put one together in a proper amount of detail, to see how things would look if we went the other route (i.e. implicits more similar to normal params, rather than more different)

Yes, I think that would be good to work out!

I believe there is one use case for implicits as defaults that neither current implicits nor the new proposal cover well. It’s where you want to pass an argument implicitly without polluting the name-space with further implicit instances. That’s what macwire does. I asked @adamw why he needed to invent macwire instead of just using implicit parameters for dependency injection. He responded that using implicits would make every implicitly injected component an implicit instance that was visible everywhere, which is neither needed nor wanted. But if default arguments could be implicits, we can formulate it like this:

class GreenModule(red: RedModule = implicit) { ... }
class RedModule(green: GreenModule = implicit, blue: BlueModule = implicit) = { ... }
class BlueModule(red: RedModule = implicit) { ... }

// wiring components together
object System {
  private implicit object green extends GreenModule()
  private implicit object red extends RedModule()
  private implicit object blue extends BlueModule()
}

This works since the implicit objects green, red, blue are confined to the System scope. It shows a general tradeoff between how “special” an implicit type should be vs how confined its instances are. In the macwire example, the implicit types are general components, so not very special. That works if the implicit instances of these types are very confined.

Anyway, it might be worth to pursue this line of thought either in a standalone system or in the context of another proposal.

1 Like

There could be another util method for that:

def given[T,E](a:T)(f:ImplicitFunction1[T, E]) = f(a)
//---------------
val newFancy = given(new C) { fancy }

Polluting the implicit namespace is only one of the reasons. The other (maybe even more important) is that the technical details of how “dependency injection” or “create the module graph” is done (which are the same thing) leak into the implementation of the modules.

With the plain-constructors approach, you simply have a class (a module) which declares its dependencies as constructor parameters. That’s it - no @Inject annotation, no implicit keywords etc. The class knows absolutely nothing and doesn’t care on how it’s going to be wired together with other modules.

Now, macwire is only a convenience thing to avoid writing a lot of new ... invocations. But to be honest, in most smaller projects nowadays I simply write the new ... without any macros :). But in larger projects macwire is still very useful.

Adam

2 Likes

Well, considering that C was not only once the most popular language, but also fundamentally influenced the syntax of almost every popular language today except Python, they must have been doing something right.

Since C heavily relies on pointers, you need ways to reference and dereference instances, and since it is statically typed, you need types and for every type a pointer-of type. That’s a challenge to notate. As Python has neither pointers nor static types, it is so much simpler.

So another proposal can’t hurt, right? So I thought I’d drop in my own! This is actually my first time writing a proposal like this, it was pretty fun! :slight_smile:

This does involve both the syntax for taking in implicits and declaring them, but I couldn’t figure out how to meaningfully split it up across the two discussion topics. This ended up being pretty long, so I can put this as a separate topic if people think that would be a good idea.

Overall Idea

I got this idea after speaking with some friends from my high school (now a mix of first year college students and junior/senior high school students), where I led the use of Scala as the core language for our robotics team. We had used implicits quite heavily in our robotics framework, but definitely saw similar issues as those reported in this thread with implicits being hard to pick up.

The summary of our discussions was that the core issue with implicits is not the syntax or overall idea, but instead the confusion that happens when they don’t “just work”. In this situation, it becomes quite difficult to guess (at a conceptual/mental level) where implicit parameters were (or were unable to be) sourced from especially because there are multiple ways implicit parameters can be synthesized.

As I see it, implicit parameters serve two core and rarely overlapping roles: to provide context (“repetitive arguments” in the original proposal, such as the CorrelationId argument) and to derive instances of typeclasses (and similar constructs that associate a type with some behavior). So this proposal splits up implicits into two separate concepts: context and derived.

Trying to write out a detailed proposal, I tried a couple experiments, so I decided to add in all the experiments after the main proposal here with comments on what I was aiming to do with each and why I decided to drop the ones I did.

Context

The first and more straightforward part here is context, since passing it down is already a simpler concept. The goal here is simple: reduce the amount of code written when some context is repeatedly passed down through functions.

One special property of context when passed down in programs is that it is a fixed instance that does not need to be generated at each level. Functions simply take in the context and pass it down to functions in their body that need it. To match this, context values must always be defined as vals.

case class Config(appName: String)

// elsewhere...

object MyApp {
  context val config = Config("Demo App")
}

Restricting context definitions to vals cuts down on the potential sources for where a context parameter could have come from, which reduces the mental load for developers trying to understand where a context value was sourced.

To take in context, the syntax is identical to implicits right now:

def displayName(name: String)(context config: Config): String = {
  s"My name is $name -- generated by ${config.appName}"
}

Passing through context is also the same:

def displayNames(names: Seq[String])(context config: Config): String = {
  names.map(displayName).mkString("\n") // displayName eta-expanded to (name) => displayName(name)(config)
}

Manually passing in context is a bit different, it requires an explicit context keyword to clarify that this is a special type of parameter and to distinguish it from derived later:

test("displayName shows correct app name suffix") {
  assert(displayName("Test Name")(context Config("Test App")) ==
    "My name is Test Name -- generated by Test App")
}

implicitly[T] here is replaced with context[T]

Relation to Implicit Functions

Context values should work well with implicit functions, which (at least in the code I have seen/imagined) would usually take some context value to reduce the repetitiveness of taking parameters over and over. More analysis is needed to understand how these two concepts would interact with each other.

Slight Aside: Similarities to React

As I wrote down this proposal, I realized that context here aligns very closely to context in React.js, which has been a very successful feature. In React, context can be added to an application by injecting it with a context provider component, which makes the context available to components rendered inside it.

(from the React docs)

// Context lets us pass a value deep into the component tree
// without explicitly threading it through every component.
// Create a context for the current theme (with "light" as the default).
const ThemeContext = React.createContext('light');

class App extends React.Component {
  render() {
    // Use a Provider to pass the current theme to the tree below.
    // Any component can read it, no matter how deep it is.
    // In this example, we're passing "dark" as the current value.
    return (
      <ThemeContext.Provider value="dark">
        <ThemedButton />
      </ThemeContext.Provider>
    );
  }
}

class ThemedButton extends React.Component {
  // Assign a contextType to read the current theme context.
  // React will find the closest theme Provider above and use its value.
  // In this example, the current theme is "dark".
  static contextType = ThemeContext;
  render() {
    return <Button theme={this.context} />;
  }
}

The context provided is always sourced from a fixed value (passed in as the value property). As it moves down the tree, the only way context can be modified is to re-inject a replacement value with another provider. This mirrors the proposal for context here, where context can be injected from a fixed context val value and can only be replaced down the function tree by explicitly passing in an alternative context.

Derived

Now comes the second half of what implicits let us do: derive behavior for specific types. Splitting this part off into a separate construct both helps developers conceptualize where values come from (just like context) and also aligns the feature with equivalents in other languages like Rust derived traits.

When declaring a derivation for a particular typeclass, there are four pieces of information: a name for the derivation to distinguish it from others, other context and derived values it needs, what type of object is being derived, and finally the actual implementation. Method declarations have a similar format, so I decided to just adapt that format with def replaced with derived.

trait Reader[T] {
  def read(str: String): T
}

object Reader {
  // first, a simple reader for parsing ints
  derived intReader = new Reader[Int] {
    def read(str: String) = str.toInt
  }

  // equivalent to

  derived intReader: Reader[Int] = _.toInt // single abstract method style
}

Taking in derived parameters is just like implicits (and thus like context). With derived definitions and parameters combined, we can then have recursive derivations.

derived seqReader[T](derived elemReader: Reader[T]) = new Reader[Seq[T]] {
  def read(str: String) = str.split(' ').toSeq.map(elemReader.read)
}

Finally, derivations can be manually passed in with a derived keyword marking the set of parameters as belonging to that set.

// parses strings like 1,2,3 4,5,6 to Seq(Seq(1, 2, 3), Seq(4, 5, 6))
def parseSpacesThenCommas[T](str: String)(derived elemReader: Reader[T]): Seq[Seq[T]] = {
  val commasReader = new Reader[Seq[T]] {
    def read(str: String) = str.split(',').toSeq.map(elemReader.read)
  }
  
  // things are much clearer when using named parameters, but I'm not sure if that should be enforced
  seqReader(derived elemReader = commasReader).read(str)
}

implicitly[T] here is replaced with derived[T]

Migration from Scala 2

To handle migration to this new style, I would propose allowing old-style implicits to fulfill both context and derived values, since this proposal just splits the two uses of implicits but does not significantly change the semantics of either. Similarly, context and derived can fulfill implicit parameters.

Experiment 1: anonymous context

Often times a context parameter will be taken in, not used in the body, but just passed down to another function. In this case, having to name all the context parameters adds unnecessary overhead. So one idea is to just make naming context values optional (then the two syntaxes are name: Type or just Type separated by commas). For example, this would enable usage such as:

def displayNames(names: Seq[String])(context Config): String = {
  names.map(displayName).mkString("\n")
}

This doesn’t complicate the syntax too much and could clean up a lot of code, so I think this would be a helpful addition to the proposal.

Experiment 2: type-named context

Inspired by React’s API for getting context values with contextType (used in the above example), this experiment drops context parameter names completely in favor of identifying context data by the type.

def displayName(name: String)(context Config): String = {
  s"My name is $name -- generated by ${Config.appName}"
}

While a nice shorthand, I am afraid this will cause too many conflicts with companion objects and is potentially very confusing, so I don’t think this should be implemented.

3 Likes

After playing around with implied instances and reading all docs, I’m intrigued about this approach. I don’t have anything to add semantic wise at this point, I like its current principles. But, given that this change is mostly about syntax, I think it might be useful to mention I struggle to get used to the implied syntax.

Isn’t there a better keyword we could use to refer to the same concept? Possibly a keyword that bears no indirect relationship with implicit and that doesn’t read as an adjective?

The one that comes to mind is infer. Given that implicits is a way to infer terms, it seems appropriate to use that word instead and it being a verb means it reads like I’m telling the computer to do something.

Here’s how the code snippet in Implied Instances would look like with it:

trait Ord[T] {
  def compare(x: T, y: T): Int
  def (x: T) < (y: T) = compare(x, y) < 0
  def (x: T) > (y: T) = compare(x, y) > 0
}

infer IntOrd for Ord[Int] {
  def compare(x: Int, y: Int) =
    if (x < y) -1 else if (x > y) +1 else 0
}

infer ListOrd[T] given (ord: Ord[T]) for Ord[List[T]] {
  def compare(xs: List[T], ys: List[T]): Int = (xs, ys) match {
    case (Nil, Nil) => 0
    case (Nil, _) => -1
    case (_, Nil) => +1
    case (x :: xs1, y :: ys1) =>
      val fst = ord.compare(x, y)
      if (fst != 0) fst else xs1.compareTo(ys1)
  }
}

Advantages of using infer over implied:

  1. It’s a verb and it’s similar to def both syntax wise (it comes from the verb define) and semantics-wise (it defines an inferred term without looking like a straight value definition).
  2. It’s two characters shorter.
  3. it reads nicely followed by for and given.

Disadvantages

  1. It’s not clear what we should use to import an inferrable instance. I prefer either infer import or import infer but we could also go for import inferred. If you don’t like either, we can probably find another variation that fits the bill.

EDIT: I’ve just seen @Odomontois already proposed the same thing in passing but no one seems to have given feedback on the suggestion.

1 Like

Hi,
I find the new syntax unfamiliar and looking like another sublanguage.
My expectation would be to learn another concept on top what I already have learned.
So I first learn about classes and traits, them objects and companion objects.
Next, come implicits. At this point, I know all basic Scala syntax

A thing I like a lot is companion objects.
They are easy to understand, with clear syntax and benefits (them being singletons).

The thing I would not like to see is mixing companion objects with implicits/typeclass instances.
I think of this as breaking the single responsibility principle.

My proposal is to get rid of the new keywords and stick to the syntax everyone already knows.
With the addition of introducing another “companion object like” keyword.

trait Ord[T] {
    def compare(x: T, y: T): Int
    def (x: T) < (y: T) = compare(x, y) < 0
    def (x: T) > (y: T) = compare(x, y) > 0
}

object Ord {
    // standard companion object stuff
    // the below "implicit companion object" could also be here
    // but my preference would be to break these apart
}

[typeclasses | instances | implicits | infer] Ord { // <- choose your favourite keyword
    // inside all val's and def's are implicit but the syntax should be 100% normal scala syntax

   val intOrd: Ord[Int] = { // 
       def compare(x: Int, y: Int) =
          if (x < y) -1 else if (x > y) +1 else 0
   }

   def listOrd[T](ord: Ord[T]): Ord[List[T]] = { // here all the arguments and implicit
       def compare(xs: List[T], ys: List[T]): Int = (xs, ys) match {
         // ...
       }
   }
}

As far as imports are concerned
I am in favor of making imports explicit

import my.Ord // imports class and companion object without any implicits
import instances my.Ord // import implicits

Advantages:

  1. The same, easy syntax you’ve already before you reached the implicit level
  2. Separation of companion object from implicits
  3. New separate rules can be defined to work inside the new “instances” companion object.
  4. No new syntax
  5. Cleaner syntax (no repetition of implied, inferred, for, given) with instance every definition

Disadvantages:

  1. Probably plenty, I am sure You can find more than I

I too think it is not worth having it in the language. It looks confusing and messy, and in the context of trying to sell scala in a java shop maybe even scary. I can’t imagine selling that at an average-ish workplace, if adoption is a concern.

I have had the complete opposite experience in teaching current implicit to the “averagish” Java workshop. Current implicits are basically parameter passing (with some quirks) which Java people understand very well. Then when I explain to them how the implicit part works, I just draw comparisons with Guice (which does similar things as implicits but at runtime) and then they understand fine.

I think your averag’ish Java developer would have more difficulty understanding something that is being packaged closer to theorem proving rather than parameter passing.

2 Likes

Not sure how your comment relates to mine, since I was reacting to the part of the discussion concerned with limiting the usage of given “to the case where it works well”, while you are comparing it with current implicits. Unless I misunderstood something.