Principles for Implicits in Scala 3

[virusdave]

@odersky

That syntax should be very different from normal parameters and arguments.
EDIT: In fact it’s better not to think of them as parameters at all, but rather see them as constraints .

I’ve been trying to catch up on all of these related threads recently while also asking myself where i have observed problems and downsides to implicits (personally when first learning scala, and more recently when reading other people’s code as they learn it).

Several of the cases have been brought up already here and elsewhere: conversions, poor compiler / tooling output, explaining them to noobs, etc. I think the poor compiler/tooling issue is HUGE. Intellij’s recent steps forward have migrated the situation from ‘untenable’ to ‘fine for non-complex use cases’. But there’s still a ways to go there.

The biggest meta-issue i’ve seen for early-to-mid scala folks, myself included, ended up being “When should i use them” and “How can i avoid horrible-to-use and horrible-to-understand code when using them”. My understanding on this has evolved quite a bit through a combination of:

1. Personal experience / banging head against the wall / hard knocks
2. Learning MUCH more about FP through years of study, conferences, and videos
3. Internalizing the mathematical foundations of FP
4. Learning Haskell

Now, after many years, i feel I can use implicits effectively and responsibly, not to the detriment of the codebase i’m working in. The biggest takeaway i’ve internalized about implicits is indeed as Martin mentions here:

• They should generally be treated as constraints rather than parameters.

YES!!!

This realization took me a long time to come to, but looking back over my last 6+ years with scala, most of the non-tooling problems i’ve seen with implicits seem to roughly boil down to treating them as “just another parameter” that one can helpfully use the syntax around to avoid typing a few characters. The complexity and readability costs of this mistake are common, cumulative, and in my opinion the biggest downside to existing implicits.

Marking this powerful, useful concept as something distinct from “just another parameter” (even if, in low level reality, that’s what it boiled down to) is likely to have a huge impact on reducing the future detrimental effects i mentioned above, as well as help push the new-scala-dev training / explanation of implicits and their responsible use into a much better (and simpler) direction.

This is why my own thoughts on the current proposal have changed. I no longer consider @lihaoyi 's “alternate proposal X is much more familiar to people” which retains more of the familiar syntax to be a better approach, whereas i would have initially. I think that the lack of familiar syntax should be considered a feature rather than a bug for a concept as powerful and easy to misuse as implicits. This feature should be “easy to use ‘the right way’” and “hard to use ‘the wrong way’” for some suitable definition thereof, and if familiar syntax doesn’t achieve that goal because it maps too easily into the familiar (and thus to misuse), then the familiarity is a disadvantageous flaw in any proposal.

My two cents…

[tarsa]

@virusdave

The biggest meta-issue i’ve seen for early-to-mid scala folks, myself included, ended up being “When should i use them” and “How can i avoid horrible-to-use and horrible-to-understand code when using them”. My understanding on this has evolved quite a bit through a combination of:

1. Personal experience / banging head against the wall / hard knocks
2. Learning MUCH more about FP through years of study, conferences, and videos
3. Internalizing the mathematical foundations of FP
4. Learning Haskell

I also had to take course in Haskell (LYAHFGG in my case) to understand monads, typeclasses and FP’s approach to application design in general (because after years of doing imperative OOP I couldn’t find deep sense in avoiding side effects using seemingly complicated mechanics). But I do not think changing the looks of a programming language as proposed here will help anything. All the new constructs are generally as flexible as previous constructs. The biggest change is removal of implicit conversions. Other than that the similarity is so high that automatic rewriting is proposed. I do not see how new syntax makes bad patterns harder to use, except implicit conversions. To remove implicit conversions you do not need to remove implicit keyword.

Splitting implicit to e.g. given and implied won’t prevent programmers of thinking of them in the same terms. In fact you must describe given and implied using very similar terminology like implicit instances. If we forget about implicit conversions then situation is as follows:

• (in current Scala) implicit marks declarations that are automatically passed to parameters marked implicit
• (in future Scala) implied and given mark declarations that are automatically passed to parameters marked given

Not a big change. The biggest change is to remember when you have to use implied and when to use given, where previously there was one keyword (implicit).

[virusdave]

@tarsa

I do not see how new syntax makes bad patterns harder to use, except implicit conversions.

I suppose my post was a longer-winded way of saying that “using syntax unfamiliar to people who would otherwise see a familiar-ish syntax and use implicits in a familiar-ish-but-leads-to-bad-code way” is a net win, even if two proposals (and existing state) all have similar expressive power.

Someone upthread (or cross-thread) made the comment that “Java folks understood implicits relatively easily by comparison to Guice”, and i think that captures exactly what i don’t want to see as a model for what this feature is used for. I’ve seen it already, and the results are awful. Hard pass, thanks.

[tarsa]

@virusdave

“Java folks understood implicits relatively easily by comparison to Guice”, and i think that captures exactly what i don’t want to see as a model

implied and given will be equally as easy to explain by comparison to Guice.

Remember that Guice has more complicated model than implicits in Scala:

• in current Scala you have just implicit
• in future Scala there could be given and implied
• Guice OTOH has several annotations, Inject, Provider, Singleton, etc plus various scopes (e.g. request), plugins, etc

Both implicit markers as well as implied and given markers can be seen as simplified versions of Guice annotations.

It’s just a matter of time until given will be given the same abuse as implicit.

Scala is not a policeman that prevents people from:

• abusing tuples instead of defining classes with proper names
• creating very deep inheritance hierarchies
• passing functions with super generic signatures like (String => Int) through many layers of methods, so their contract and purpose is very hard to track
• using mutability
• using unsafe constructs
• etc

Scala never warns about such design issues. One can add such policeman.

[curoli]

Could you give an example of the usage you hope will be discouraged?

[kevinwright]

@tarsa

But I do not think changing the looks of a programming language as proposed here will help anything. All the new constructs are generally as flexible as previous constructs. The biggest change is removal of implicit conversions. Other than that the similarity is so high that automatic rewriting is proposed. I do not see how new syntax makes bad patterns harder to use, except implicit conversions. To remove implicit conversions you do not need to remove implicit keyword.

What the new syntax offers is insight and intuition. Current syntax for implicits does not clearly establish that what you are doing is:

• term inference
• specifying constraints

Naming conventions and patterns, such as those used in shapeless, can establish this more clearly; but the mere need for such patterns just demonstrates that the syntax isn’t quite hitting the mark.

Existing syntax also has some technical failings. I can’t use a method in point-free style if it takes implicits, and I can’t have a method taking arguments with a path-dependent type based on an implicit, or allow regular and implicit params to participate in type inference in all the ways I’d like. Some of this can be fixed by creating interim objects with apply methods… but that’s inefficient and somewhat hacky.

The proposed syntax fixes all this and, coupled with the unification of type members and parameters, makes it possible to solve a great many more problems using type-level constraints in a manner that won’t scare others away as being unmaintainable. Especially with native HLists coming to the language, this is a very big deal indeed.

[nafg]

How exactly does the current syntax not clearly establish that what you are doing is term inference?

As for specifying constraints, well that’s only one use case for implicits. Besides, every parameter is a constraint. That’s what enables the reasoning power of parametricity.

And as for the technical failings, again, that can be solved without creating a whole new language, which I have zero interest in learning.

[mdedetrich]

@kevinwright

Existing syntax also has some technical failings. I can’t use a method in point-free style if it takes implicits, and I can’t have a method taking arguments with a path-dependent type based on an implicit, or allow regular and implicit params to participate in type inference in all the ways I’d like. Some of this can be fixed by creating interim objects with apply methods… but that’s inefficient and somewhat hacky.

As mentioned before, these are quality of life issues that can be resolved. The issue with point-free style/currying in current implicts is that sometimes .apply works on the explicitly parameter list, and sometimes on the implicit parameter list. We should make this consistent, this also does not require a language change.

[acruise]

@lihaoyi

All this stuff is already on the classpath, we just need to scan for it in the compiler and surface it.

Great idea! We could even use the Ancient and Venerable ServiceLoader pattern to quickly find library authors’ suggested entry points. The class files mentioned in the manifests could be some kind of ImplicitHelper type that would be given information about the calling context.

[bmeesters]

Exactly. I would also say that the biggest problem with implicits - after diagnostics - is that they are abused.

The newly proposed syntax does not prevent it, but at least it does not encourage it. Whereas the old syntax does, since it shows they are just parameters that you conveniently do not have to pass around.

I think part of the migration should be more clear guidelines and when to use implicits and when not to use them (and what to use instead).

[mdedetrich]

The new proposal doesn’t change any of this. Firstly, there is no such thing as “constraints” in Haskell, or at least I am not aware of what @virusdave means when he talks about Haskell constraints in context of implicits.

From the top of my head, the only type of constraints that Haskell/GHC has are

• Class constraints ( show as Show)
• Implicit parameters ( roughly equivalent of current Scala)
• Equality constraints (i.e. ~)

So I am not seeing how Haskell is helping or dong anything different here. Note that “constraint” is just an English word, if we want to be exact about it we need to talk about what constraint means in terms of lambda calculus and types.

The only major thing that this proposal does is it puts precedence on implicit parameters over explicit ones (in all cases) as a sidestepping way of fixing current issues.

Also I do not see how this solves any abuse issues,

[LPTK]

@mdedetrich Sala’s implicits are closer to Haskell’s type classes than to Haskell’s implicit parameters.

Otherwise, I think I agree with you. The proposal is mostly syntax and doesn’t fundamentally change anything. For it to achieve its stated purpose, I think it should come with more normative usage rules that emphasize “implicits: the good parts”. This means actually restricting and refocusing the new features, rather than making them at parity with the current ones.

[mdedetrich]

@LPTK

@mdedetrich Sala’s implicits are closer to Haskell’s type classes than to Haskell’s implicit parameters.

When Scala implicits are used as typeclasses then yes, although there are other cases as well (unless I misunderstand something). I was also talking about pure implicit parameters.

[bmeesters]

I guess for that to happen we first need to determine what exactly are the valid use cases for implicits and which ones aren’t.

Currently, I expect most people would argue that type classes are a valid use case. But what about implicit loggers, configurations, execution contexts, actor systems, materializers, etc? Can we clearly identify what the good cases have in common?

We first need to determine what should be allowed before we can provide any restrictions.

[mdedetrich]

@bmeesters

Currently, I expect most people would argue that type classes are a valid use case. But what about implicit loggers, configurations, execution contexts, actor systems, materializers, etc? Can we clearly identify what the good cases have in common?

Actually I would argue that implicits are not a good valid case for type classes due to the (well documented) collisions you get with typeclasss implicits in Scala when you combine them, although we can only solve this when you get specific coherent typeclass syntax

@bmeesters

Currently, I expect most people would argue that type classes are a valid use case. But what about implicit loggers, configurations, execution contexts, actor systems, materializers, etc? Can we clearly identify what the good cases have in common?

Well to start off with, its good to point out that there are differences between Scala and Haskell in terms of design principles. Haskell leans more to the python attitude of only doing one thing correctly, well as Scala leans more to using orthogonal tools to correctly fix the job (I am using extreme examples here to illustrate a point)

@bmeesters

Currently, I expect most people would argue that type classes are a valid use case. But what about implicit loggers, configurations, execution contexts, actor systems, materializers, etc? Can we clearly identify what the good cases have in common?

Well we use implicits for all of those (apart from maybe loggers). Pure FP people tend to use ReaderT for things like configuration, but there are a lot of disadvantages to using such methods (even the cats book highlights disadvantages of using ReaderT for such purposes).

[arashi01]

@bmeesters

Given that we have a statically typed language, and the changes proposed are not changing the semantics, it should be fairly easy to do all the refactoring with an automated tool.

You just parse Scala 2 code as an abstract syntax tree, and let it represent that tree as the newly proposed in syntax in Scala 3.

Also the old style implicits are still supported so you don’t have to do it right away.

It may well be the case that tools can be created however we don’t know that yet.

It seems to me that the whole point of this excercise is to make the semantics / syntax more approachable? Assuming one agrees that the current syntax is not, I personally can’t see that there is a fundamental improvement in the proposed changes - at least not enough in my eyes to make it worth the pain of migration.

[bmeesters]

@mdedetrich

Actually I would argue that implicits are not a good valid case for type classes due to the (well documented) collisions you get with typeclasss implicits in Scala when you combine them, although we can only solve this when you get specific coherent typeclass syntax

I think it is unrealistic to expect this to change. Given the constraints type classes have in Scala, I think most people would say that they are a valid use case for implicits.

@mdedetrich

Well we use implicits for all of those (apart from maybe loggers). Pure FP people tend to use ReaderT for things like configuration, but there are a lot of disadvantages to using such methods (even the cats book highlights disadvantages of using ReaderT for such purposes).

I know, which is why I brought it up (including loggers). I think that as a community we should get clearer guideline if all of these should indeed be used in implicits (I personally don’t think so).

Currently there is no coherent view on where implicits should be used. And as a result we also cannot agree on whether or not this proposal takes us in the right direction, and what else should be done.

[tarsa]

I wonder how you want to impose restrictions regarding implicit definitions. IMO there always will be some escape hatch, leading to frequently used anti patterns anyways. If you have any suggestions then there are threads where you can post them.

@bmeesters

I guess for that to happen we first need to determine what exactly are the valid use cases for implicits and which ones aren’t.

Currently, I expect most people would argue that type classes are a valid use case. But what about implicit loggers, configurations, execution contexts, actor systems, materializers, etc? Can we clearly identify what the good cases have in common?

We first need to determine what should be allowed before we can provide any restrictions.

You can create fake typeclasses to emulate implicit parameters. See:
https://twitter.com/viktorklang/status/841702704749637632
Lgd. Viktor Klang by viktorklang

import java.util.UUID, scala.concurrent._

final case class Customer(uuid: UUID)

abstract class CustomerComponent {
  type EC[_] = ExecutionContext

  final def byId[_ : EC](customerId: UUID): Future[Customer] =
    selectCustomer(customerId)

  protected def selectCustomer[_ : EC](uuid: UUID): Future[Customer] 
}

A lot of time this notation (using fake typeclasses) could even lead to less boilerplate! [_: EC] instead of (implicit ec: EC)

If you have many implicit parameters to pass then implicit function / context queries are here to help:
Implicit Function Types | The Scala Programming Language
http://dotty.epfl.ch/docs/reference/contextual/query-types.html
type Contextual[T] = given (Logger, Config, ExecutionContext, ActorSystem) => T

As for guiding people towards desirable implicits hierarchies, I think the best solution is to implement selective compiler suggestions after compilation errors. If compiler suggests imports only for specific locations of implicits then library authors would be very encouraged to follow that rules. Otherwise a library lacking compiler suggestions would lose users to library that gets better treatment from a compiler.

[bmeesters]

@odersky

I think this section on the principles for implicits in Scala 3 have cleared up a lot of things and has been a welcome addition. However, given the latest comments I would say the current bottleneck is not knowing/agreeing on where implicits are a success, and what should be kept.

Maybe you can expand the principles a bit more on what we want to keep / what to encourage. What are implicit patterns and anti-patterns.

Maybe this goes a bit too far and I might ask a bit too much (sorry if I do), but I think it would help get a consensus on where implicits should go.

[mdedetrich]

@bmeesters

I think it is unrealistic to expect this to change. Given the constraints type classes have in Scala, I think most people would say that they are a valid use case for implicits.

Well there is a proposal for typeclasses, so its not that unrealistic for Dotty

@bmeesters

I know, which is why I brought it up (including loggers). I think that as a community we should get clearer guideline if all of these should indeed be used in implicits (I personally don’t think so).

Currently there is no coherent view on where implicits should be used. And as a result we also cannot agree on whether or not this proposal takes us in the right direction, and what else should be done.

I am not trying to dismiss your suggestion, but I don’t think this reflects how Scala works considering its a multi paradigm language with orthogonal constructs. The best thing you can do is to establish best practices and use tools like wart-remover to remove styles you don’t like.

I don’t think this is practically achievable nor should it be really a goal of Scala (at least for the base level of the language).