Canonical "Non-Deterministic" Implicit Conversion Based on Example in Scala Documentation

Non-deterministic has to be understood as a metaphor only in this context. Technically Scala implicit conversions are deterministic. They don’t change program behaviour on each run or re-compile. Non-deterministic is sometime used in arguments against implicit conversions to describe non-obvious behaviour.

It is based directly on an example taken right from Scala Documentation Where does Scala look for implicits? in FAQ where it explains Implicit scope of an argument’s type.

class A(val n: Int) {
  def +(other: A) = new A(n + other.n)
}

object A {
  implicit def fromInt(n: Int) = new A(n)
}

1 + new A(1)

Example doesn’t specify what should be a resulting type of 1 + new A(1) expression. How anyone learning should understand how compiler assumes that it should be A? From there his train of thoughts can go out of rails and I think it can make it hard to even understand how it works. It is not a good example for this alone but it is worse than that.

Let’s try to fully understand how it can manifest as a problem. The reason is that this supposedly simple example actually shows type inference driven by implicits.

In a given example there is Int => A implicit conversion defined in a scope which drives expression to return A type and results in A(2) value.

But if we would define implicit conversion for A as A => Int instead, the result would be 2 and type Int.

class A(val n: Int)
object A {
  implicit def toInt(a: A) = a.n
}

1 + new A(1) // result is 2

It seems I can just change implicit conversion in a library and client code can potentially compile but run differently. I’m sure it doesn’t happens often but when it does I can imagine it could be quite hard to debug. For sure if experience and knowledge of implicit conversions is not top-notch.

It is driven by which implicit conversions are in the scope. Such code as 1 + new A(1) (returned from def without return type) or val x = 1 + new A(1) (which can be part of more complex computation)
can be driven by different imports or implicit conversions available from implicit scope of types withing the same package (which is even more hidden). Or changed by redefining implicit conversion(s) in A, see puzzler. The problem is that if I change implicit conversion it can still work with type inference.

Documentation is promoting it by using an example which is prone to it.

Puzzler for you
What is the result? Which rule is used and why? Can you tell without trying?
Where is the rule defined?

class A(val n: Int) {
  def +(other: A) = new A(n + other.n)
}

object A {
  // Defining both conversions in A
  implicit def fromInt(n: Int): A = new A(n)
  implicit def toInt(a: A): Int = a.n
}

1 + new A(1)

What is the result of the last expression?

  1. 2
  2. A(2)
  3. Compilation fails as ambiguous

What if anyone adds the second implicit to A sometime later? Hopefully it doesn’t compile. But it might manifest a bit far away if type inference would make it work. The worst case is that it will compile (I believe).

Conclusion
It seems that using plain implicit conversion and relaying on type inference is quite dangerous.

Would requiring that implicit conversions are tried only if there is explicit target type annotation? When implicit conversions are used for type substitution the target type is known.

// it wouldn't compile if implicit coversion is needed
val x = 1 + new A(1)
def compute() = 1 + new A(1)

// these two would compile
val x: A = 1 + new A(1)
val y: Int = 1 + new A(1)
def compute(): A = 1 + new A(1)

As a side note the example makes it needlessly harder because it uses infix syntax. If beginner tries to read it he is possibly not used to it yet. Docs should expect that reader is not fluent yet in all Scala syntax sugars.

1 Like

I seems I can just change implicit conversion in a library and client code can potentially compile but run differently.

Precisely. In fact, implicit conversion is just one of a set of “Church-typing” features that exhibit this behavior. Such features exist in many languages, including Java.

I’m sure it doesn’t happens often but when it does I can imagine it could be quite hard to debug. For sure if experience and knowledge of implicit conversions is not top-notch.

I see where your concerns are coming from. It is definitely possible to produce confusing behavior with defined implicit conversions, but in practice, these are not such issues as they might first seem.

With regard to debugging, implicit conversions are selected statically, not at run-time. Tools do, in fact, point out where implicit conversion will be used, right in your editor, using only type-checking to figure it out. So you do not even need to “debug” to spot all your implicit conversions.

With regard to requiring “top-notch” knowledge, it is best not to imagine that people will write whatever implicit conversions are possible. There are rules of good taste regarding implicit conversions, just as it would be tasteless to write your library to only take arguments and return results of type Any, even though it’s possible.

The space of “tasteful implicit conversions” is quite proscribed. For example, JavaConversions, which “merely” implicitly converts between Java and Scala collection types, was deemed too potentially confusing, and deprecated in favor of the more explicit—yet still implicit-based—JavaConverters.

The notion of taste is subjective, to be sure. But it definitely precludes deliberately confusing—“puzzling”—conversions, or conversions with “surprising” behavior, such as wanton side-effects or exception-throwing. A good choice for implicit conversion is one that looks like utter boilerplate if you have to call it explicitly. If you are defining a conversion that does not clarify, rather than obscure, downstream code, consider leaving it out.

Would requiring that implicit conversions are tried only if there is explicit target type annotation?

This would be an impractical restriction for a wide variety of uses of implicits, such as syntax extension (cf. a recent example where implicit class does not work.).

As a side note the example makes it needlessly harder because it uses infix syntax. If beginner tries to read it he is possibly not used to it yet. Docs should expect that reader is not fluent yet in all Scala syntax sugars.

I’m not sure a “deep dive” such as this document can be useful if it assumes that basic Scala knowledge is missing. My suggestion to such a beginner would be, assume at first that applied implicit conversions will “do what you mean”. As you get comfortable with the language, that is the time to dig into its foibles.

3 Likes

It is interesting point. I think I could craft an example in a dynamic language with similar properties after a while. Up to limitations of structural subtyping. Or probably if I would have deep formal understanding of “Church-typing” I would know there are more such properties.

As well I’m sure it exists in Java. But if I discount reflection hackery I believe the effect is marginal. If anybody has example or two it would be helpful.

Out of these Scala implicit conversions are the most flexible, more than structural typing I would say. It seems, what brings it to another level is combination of implicit conversion and type inference. I deem them fairly safe in isolation with good practices.

It is actually what attracted me to Scala, it’s balance between type-safe and dynamic language. I admit it doesn’t have to be significant problem in practice. I don’t know. But I have my worries and I feel I need to understand this aspect better before it manifest in practice in total despair.

My concern is that even in dynamic language I would have to use explicit conversion for many cases I can solve with Scala implicit conversions.

It is really helpful. But I’m uncertain if it actually helps in problematic cases. Whenever such a problem occurs I think the main question to answer is to “How the heck it happened that this particular implicit conversion is being used?”. IDE shows me it is used, it jumps to declaration. But without actual deep understanding how implicit rules work it is likely not to be enough.

I would love it to be exactly what happens in reality. Unfortunately I think it is a very high expectation. Going through experience with commodity developers I learnt they can come up with coding ideas I would never deem possible. One has to even learn downsides and best practices from a good resource or will get burned hard first.

I can be mistaken and I would like to be proven wrong. But unfortunately impression at the moment is that everybody talks about them, nobody has seen them. I’m really looking for a comprehensive resource which doesn’t consist from 10 unofficial blogs. Not having to read again and again through the same basics to find that nitty-gritty detail I’m after explained in a language a humble commodity enterprise developer can understand. If anybody can help I would really appreciate it. I would like to forward my developers to it and safe my time with baby-sitting them. I have bought Scala Programming Third Edition to achieve it. It’s chapter about implicit is just a good introduction. I have figured out and learnt many rules but it costed me quite a bit of time. As well we are professionals and partial information doesn’t cut it. At the same time most of developers in my team has hard time to read Language Specs which is not a good source for best practices anyway.

Is there a comprehensive breakdown why exactly they were wrong decision? What actual problems it caused in practice? It could be a great learning resource to avoid the same mistakes. I have never used them. I would be concerned because such implicit conversion can have performance impact whenever it actually converts and not just retrieve embedded collection. Performance regressions could easily leak implicit conversions during debugging. As well I can imagine seamlessly moving between similar types with very different set operations and usage semantics within a single transformation can be confusing but I haven’t try it how it goes. I would be interested in actual code samples deemed confusing. Anything more?

It would be strange anyway to sometime require target type annotation and sometime not. It would quite beat point of type inference.

This document is more “deep dive” that it has to be. I would simply see it this way. There is nothing to gain if there is a potential to confuse reader for whatever reason. It is always possible to add syntax sugar and more complexity into explanations. It is harder to get back a reader who ran away with “Scala is cryptic and confusing” because it is not his second nature yet. Regardless it is actually nicely regular. The example is needlessly complex explaining given feature. Infix syntax is completely unrelated. Implicit conversion which adds extension method through implicit conversion into the picture brings it to another level, type inference even more.

Isn’t the point just to explain that when a type is used its implicit scope is made available? Even because it is part of the same package or it is imported? Much simpler example can do the job.

First of all, generally: I agree Scala documentation could be better in principle, and it would be better for Scala and the Scala community.

And I understand that in a company, immature documentation is a cost. And those who want Scala to succeed want to reduce such costs.
Even so, don’t take this the wrong way, but a company using open source isn’t a customer who has paid a product and gets a warranty for it. Yes, people working on Scala do their best. But no, there aren’t millions of such people, and there’s tons of work.
I love Scala, and I’d love people to use it, but I can’t tell you its advanced features have the easiest learning curve, or that its documentation is as good as it could be. As Steve McConnell writes, unfortunately being “early adopters” does mean facing rough edges. I suspect most companies using Scala just try hiring enough Scala experts or buying good Scala training.

While this is a side point, and most programming documentation is guilty of much worse crimes: I agree with @vladap on this general point—this is standard advice when writing academic papers. I also agree on this specific example—yes, usually you’d learn infix syntax before implicits, but I know exceptions first-hand. You can’t discuss implicits without types, but infix syntax is an irrelevant obstacle.
I also agree with @S11001001 though: after fixing this issue, these docs would be better, but still hard for most beginners. As you yourself point out, there are likely bigger improvements possible here.

Changing the example to use .add rather than + (or some other method where infix syntax is less appropriate) would not create obstacles to making the point.
So if anybody sends a PR to fix this (without unrelated changes), feel free to ping me for positive support.

One note though: such advice works best if one is willing to contribute rather than expecting others to. I’ve offered my support, so you know that a PR has a chance.

And I don’t just mean “you should contribute because this is open source”, but “you should contribute such things because sadly few have time to write docs at all, and even fewer have time have time to make them as beginner-friendly as possible.” Also, I haven’t yet seen computer science courses teaching technical writing. I heard there was the idea of hiring (where?) a technical writer to write Scala docs, which would be excellent, but I don’t think it’s easy.

Implicit conversion which adds extension method through implicit conversion into the picture brings it to another level, type inference even more.

Well, that is a standard pattern in Scala which should be explained somewhere. But if you can explain implicit scope without introducing this pattern, and then explain it elsewhere, go ahead.

Now, talking about bigger issues:

I think I agree with @vladap also here—I’ve seen examples of implicits that work (in Odersky et al.'s Programming in Scala), tried to invent my own patterns that should have worked (and which IIRC weren’t unreasonable, but I was still a Scala padawan), failed, and then stopped experiments. I have never seen “rules” or “best practices” stated at all, much less widespread. I’m happy to be proved wrong by actual links, preferably to “important” blogs—but anything, really.

How do implicit conversions relate to Church typing? Church-style typing, as opposed to Curry-style typing, actually means that the language semantics doesn’t assign meaning to ill-typed terms (Simply typed lambda calculus - Wikipedia, which happens to be very accurate even though it’s on Wikipedia). People misuse Church-style typing to mean “lambdas have type annotations on arguments” (which is improper, as discussed there), but even that doesn’t seem to cover implicits. Would you clarify?

EDIT: answering @vladap’s original point here:

That happens also if you change the implementation of a library function foo (and keep the type in a typed language).
Or if you change the return type of foo with another type with similar methods but different behavior. More examples are possible.
However, even a reasonable developer who’s a beginner with implicits can change implicits in a library and affect clients without realizing, while the two cases above are likely to be clearer. That’s probably true. Right now, Scala’s answer is “limit your use of implicits, especially in an API, if you’re not already expert” — that’s one of the reasons why, since Scala was changed by SIP-18, you need to enable implicit conversions by writing import scala.language.implicitConversions, else Scalac warns you when you define implicits.

I like Scala the most, anytime I program in Java or Javascript I miss it.

The main problem I notice is that syntax sugar take some time to get used to it. It is highly individual. Reader can just read it section before but then he will be still decompiling it in his head. And if gets lost he doesn’t even know how it is called and where to look. Java7 developer can have quite a leap ahead to get lambdas, put some underscores here and he just go to learn elsewhere.

I might eventually try. I don’t know if I even understand it well, my English is poor, employer won’t give me time for it, wants more time than he pays for and time spent with family is the main competitor. But time is expensive resource for everybody. I appreciate time all of you spent here with some advice.

Yeah, there are various ways how we can break things, violate LSP (which itself is possibly more complex then all implicits together).

1 Like

Fair enough — just please have patience with our imperfections :slight_smile:

1 Like