Principles for Implicits in Scala 3

#2

I always thought of them as (formal/mathematical) assumptions

Will there be a separate SIP for implied imports (or where should I comment on its semantics)?

#3

There will be a separate SIP. Should be out shortly. I believe @smarter was taking the lead on this one.

#4

Would you explain what do you mean as “run-away success”?.
For example I think scala collection library is a success which is impossible without implicits.

Are there concrete tasks which can be said as success or failed?

May be, the main disappointments of implicits are wrong tasks
For example I cannot imagine success with builder pattern whether there is new syntax or there is not.

1 Like
#5

In my opinion, changing keywords from implicit to given/ implied/ whatever is just another minor tweak when it comes to beginner friendliness, albeit bringing heavy migration pain.

First I’ll link to my previous post on this subject:

Typical problem with implicits is that implicit conversion doesn’t work. Changing syntax for implicits has zero influence on that. A pretty rare and simple to fix problem is when we have to disambiguate between implicit and explicit parameter lists with explicit apply:

def method(explicit1: Int)(implicit implicit1: String): String => Int =
  x => explicit + x.length

method(5).apply("wow") // implicit1 stays implicit

Why implicit conversion doesn’t work in a particular place (but works somewhere else)?

  • because you forgot to import some implicit conversions or implicit values
  • types don’t match so implicit conversions are rejected silently
  • there are implicits ambiguities
  • you forgot to mark something implicit
  • etc

What Scala compiler reports when it can’t find a extension method coming from implicit conversion (or implicit class which is a syntactic sugar for it)?
value print is not a member of Int

What Rust compiler says?
https://play.rust-lang.org/?version=stable&mode=debug&edition=2015&gist=0dc03e79269cc6efff0ad31947811ebe

error[E0369]: binary operation `<<` cannot be applied to type `f32`
 --> src/main.rs:6:1
  |
6 | x << 2;
  | ^^^^^^
  |
  = note: an implementation of `std::ops::Shl` might be missing for `f32`

https://play.rust-lang.org/?version=stable&mode=debug&edition=2015&gist=4dd0b62770be7752e1d354028246f3c4

error[E0119]: conflicting implementations of trait `main::MyTrait` for type `main::Foo`:
  --> src/main.rs:15:1
   |
7  | impl<T> MyTrait for T {
   | --------------------- first implementation here
...
15 | impl MyTrait for Foo { // error: conflicting implementations of trait
   | ^^^^^^^^^^^^^^^^^^^^ conflicting implementation for `main::Foo`

I wrote an application using Rust. Here’s very simplified case from it.
In one file I have:

pub trait FixedPoint where Self: Sized {
  ...
}
pub trait FixI32: FixedPoint<Raw=i32> {
  ...
}
impl<T: FixedPoint<Raw=i32>> FixI32 for T {}

In second file I have:

#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)]
pub struct NoFractI32(i32);

impl FixedPoint for NoFractI32 {
  ...
}

In third file I forgot to import relevant typeclasses (signalled by commenting out):

// use demixer::fixed_point::{FixedPoint, FixI32, FixU32};
use demixer::fixed_point::types::{NoFractI32, Log2D};

#[test]
fn initial_cost_corresponds_to_one_bit_costs_series() {
  ...
  let new_tracker = old_tracker.updated(NoFractI32::ONE.to_fix_i32());
  ...
}

What Rust compiler says?

error[E0599]: no method named `to_fix_i32` found for type `demixer::fixed_point::types::NoFractI32` in the current scope
  --> tests/cost_tracking.rs:33:59
   |
33 |     let new_tracker = old_tracker.updated(NoFractI32::ONE.to_fix_i32());
   |                                                           ^^^^^^^^^^
   |
   = help: items from traits can only be used if the trait is in scope
help: the following trait is implemented but not in scope, perhaps add a `use` for it:
   |
20 | use demixer::fixed_point::FixI32;
   |

Plus similar error message about FixedPoint. Following compiler suggestions correctly solves the problem with missing imports (uses in Rust parlance).

There’s a huge discrepancy between scalac error messages and rustc error messages. Usefulness of Rust’s error messages is IMO the main reason Rust is so liked.

In Scala we don’t have the simple rules and useful error messages that Rust has. Instead:

  • Scala has very complicated prioritized implicit resolution, while Rust just rejects any ambiguities
  • typeclasses instances in Scala can be defined anywhere, while Rust rejects orphans
  • typeclasses instances in Scala can be stored in anything, while Rust allows only global definitions
  • Scala just says that a method is not a member of some type and doesn’t even try to suggest any solution while Rust gives ready to copy-and-paste code snippet that usually solves the problem

Rust also has more simplifications compared to Scala, e.g. Rust doesn’t have method overloading: https://internals.rust-lang.org/t/justification-for-rust-not-supporting-function-overloading-directly/7012

Complex implicit conversions/ classes/ whatever are done almost exclusively by library writers. Library user’s job is usually to have proper implicits imported into scope. Implicits defined by ordinary Scala programmers are usually simple, like implicit correlation ID or implicit ExecutionContext.

Providing useful suggestion in compilation errors for all existing code may be unfeasible now, but when Scala compiler starts giving suggestions to fix missing implicits in scope then library writers will reorganize their libraries to make the compiler suggestions more useful. Rust compiler was designed to provide useful error messages from the start (i.e. from the first public release, I think), so if we want to compete with Rust in this area we must make useful error messages a core feature of the compiler.

Rust’s syntax related to typeclasses wasn’t perfect either, but that didn’t give Rust bad PR. Some explanation: https://doc.rust-lang.org/edition-guide/rust-2018/trait-system/dyn-trait-for-trait-objects.html

Using just the trait name for trait objects turned out to be a bad decision. The current syntax is often ambiguous and confusing, even to veterans, and favors a feature that is not more frequently used than its alternatives, is sometimes slower, and often cannot be used at all when its alternatives can.

To summarize:
Typical problem with implicits is not the syntax, but figuring out what to import to have required extension methods available on type.

6 Likes
#6

I agree that Scala can learn a thing or two from Rust’s compiler errors. But that is completely orthogonal to the proposal. You can change the syntax and improve error messages.

I disagree that the proposal is just about changing implicit to implied/given. It is about making the language more regular, and having a more distinct syntax between various usages of implicits, like conversions and type classes. The new syntax also makes a more clear distinction between normal parameters and implicit parameters.

That said, you make a fair observation about cryptic error messages and we should maybe give it more emphasis than is currently done.

#7

I completely agree.

It does not matter what. If ide code assistant does not help me I will not like such methods.
I need that people in my company do not waist time on it.
So I would prefer ‘CollectionHelper.asScala(someVal)’ just because it is more simple to remember. If I do not use it often.

#8

Theoretically compiler errors are orthogonal to language redesign, but in this case language redesign is motivated by comparisons to other languages that have similar features (look at the section “background” in Martin’s original post). Therefore I’ve provided a more informed comparison, I think (I’ve actually programmed in Rust for many months).

I also think that Rust’s syntax isn’t any better than current Scala’s syntax for defining typeclasses. Rust typeclasses are pretty limited and it is often easier to define them in Scala. But because Scala is more flexible and powerful, people can encode much more complex typeclass derivation in Scala leading to very complex to use libraries. However, logic that can be easily expressed in both languages doesn’t look worse in Scala.

2 Likes
#9

(1) That syntax should be very different from normal parameters and arguments.
EDIT: In fact it’s better not to think of them as parameters at all, but rather see them as constraints.

I see. I always thought of implicit parameters as, well, parameters you can omit. If they are not parameters, then what are they?

Constraints or assumptions or the like, sounds like facts to be asserted or used for reasoning, as if the Scala compiler has now become a theorem prover. Like i < 5 is a constraint. But that’s not what we mean.

Implicts are really just a convenient way to stash away some instances and get them back via their type. Essentially, a key store with types as keys. Let’s use the keyword implied as if it was a key store:

def caller(): Unit = {

implied.put(“Yo”)
called(42)

called(42)(“Hello”)

}

def called(i: Int)(s: String = implied.get): Unit = ???

1 Like
#10

I wasn’t able to find documentation covering the differences (if any) between implicit scope in Scala 2 and Dotty. Can you point me in that direction?

I’d like to review it before commenting further, but generally I’m concerned by anything which increases the “import tax” of implicits, which this point seems to imply is desirable.

2 Likes
#11

I agree that this is the #1 problem. I also basically agree with your comparison to Rust.

I disagree, however, that reworking the term inference system is not part of the solution. The problem with any-random-mix-of-values-defs-and-implicit-classes as a source of inferred terms is that it gets really hard for everyone (compiler included) to know what is going on.

Whether this proposal does enough to simplify term inference so that it’s easy enough to make the compiler be more helpful, I do not know. It will, I think, be easier on users; but as the existing difficulty is mostly in making sure the one right thing is available, and wrong things are not available, I’m not sure that being easier on users is enough.

Rust does that by weakening the system of term inference so that there can only be one right thing, and not many places in which to look for it. That’s handy, but very limiting; if you have a type T then your ordered trees containing type T as a key can have only one order. More generally, you can’t use it for scope injection at all, because the scope is always the same.

Since Rust has zero-cost newtypes, and the newtypes can have alternate orders, and Into is pretty flexible, there are a kinda clunky but moderately okay workarounds. But I don’t think we can just emulate Rust in full here without giving up a lot.

I do think the question of discoverability and comprehensible errors are very important, though.

3 Likes
#12

Yes, it is presently a theorem prover about types. That’s why people worry about soundness: has the compiler correctly proven (and is it even possible to prove that) that this thing is valid given the types involved?

So there are dual functions of implicits: one is as a constraint that limits what is valid to call; the other is a source of functionality that allows you to exploit the now-known constraint.

1 Like
#13

Implementing suggestions in compilation errors will allow to validate whether redesigning implicits improve discoverability or not. Amount of correct suggestions would be an objective measure of that (i.e. discoverability).

#14

A question: Rust prohibits orphan instances, so I was under the impression that all implementions have to be defined with a type that forms part of the trait that’s implemented. In an analogous situation Scala would not require an import at all; the implicit would be available anyway. Is that correct, or did I miss something?

#15

Ok, technically, type inference is a kind of theorem proving. But most people writing code would not consider themselves in the business of proving theorems. For them, type inference is just a convenience allowing them to omit stuff. Telling them they need to worry about “proving” things sounds like a mighty deterrent.

#16

I’m talking more about availability of extension methods rather than typeclass instances per se. In Rust, typeclass instance is not available as a value (even trait objects used for having virtual methods are bound to specific struct or enum instances) but instead visibility of a typeclass adds extension methods to a type.

Example code in Rust https://www.ideone.com/SXefDQ

use struct_here::MyStruct;
//use trait_here::MyTrait; // rustc suggest that, IntelliJ inserts automatically
 
mod struct_here {
    pub struct MyStruct;
 
    impl MyStruct {}
}
 
mod trait_here {
    use struct_here::MyStruct;
 
    pub trait MyTrait {
        fn print_hello(&self);
    }
 
    impl MyTrait for MyStruct {
        fn print_hello(&self) {
            println!("Hello, World!")
        }
    }
}
 
fn main() {
    let instance = MyStruct {};
    instance.print_hello(); // <- situation: I'm writing this line of code
}

Compiler error:

error: no method named `print_hello` found for type `struct_here::MyStruct` in the current scope
  --> prog.rs:26:14
   |
26 |     instance.print_hello(); // <- situation: I'm writing this line of code
   |              ^^^^^^^^^^^
   |
   = help: items from traits can only be used if the trait is in scope; the following trait is implemented but not in scope, perhaps add a `use` for it:
   = help: candidate #1: `use trait_here::MyTrait`

Analogous code in Scala https://www.ideone.com/kY7zRG

import TypeClasses1.struct_here.MyStruct
 
object TypeClasses1 {
  object struct_here {
    class MyStruct
  }
 
  object trait_here {
    trait MyTrait[A] {
      def printHello(): Unit
    }
 
    implicit val myTraitForMyStruct: MyTrait[MyStruct] =
      new MyTrait[MyStruct] {
        override def printHello(): Unit =
          println("Hello, World!")
      }
  }
 
  def main(args: Array[String]): Unit = {
    val instance = new MyStruct
    instance.printHello()
  }
}

Compiler error:

Main.scala:22: error: value printHello is not a member of TypeClasses1.struct_here.MyStruct
    instance.printHello()

I know that typeclasses work somewhat differently in Scala and Rust (and above Scala code can’t be fixed with extra import), but Rust’s approach doesn’t look inherently less capable. Typeclasses are ultimately used for extension methods and extra constants. Here’s how Rust typeclass can add a constant to a base type:

use struct_here::MyStruct;
use trait_here::MyTrait;

mod struct_here {
    pub struct MyStruct;

    impl MyStruct {}
}

mod trait_here {
    use struct_here::MyStruct;

    pub trait MyTrait {
        const MY_CONSTANT: u8;
    }

    impl MyTrait for MyStruct {
        const MY_CONSTANT: u8 = 8;
    }
}

fn main() {
    println!("{}", MyStruct::MY_CONSTANT); // MY_CONSTANT added to MyStruct
}

If I forget to add use trait_here::MyTrait; then rustc prints the following:

error[E0599]: no associated item named `MY_CONSTANT` found for type `struct_here::MyStruct` in the current scope
  --> src/main.rs:23:30
   |
5  |     pub struct MyStruct;
   |     -------------------- associated item `MY_CONSTANT` not found for this
...
23 |     println!("{}", MyStruct::MY_CONSTANT); // MY_CONSTANT added to MyStruct
   |                    ----------^^^^^^^^^^^
   |                    |
   |                    associated item not found in `struct_here::MyStruct`
   |
   = help: items from traits can only be used if the trait is in scope
help: the following trait is implemented but not in scope, perhaps add a `use` for it:
   |
1  | use trait_here::MyTrait;
   |

Overall, in Rust I do not even need to know the names of typeclasses. Rust compiler will find them for me and suggest them. Same for IntelliJ - it will suggest methods or constants from typeclasses and import those typeclasses (traits) immediately.

Things change when I use a generic type - then we need to have generic type constraints, i.e. explicitly type typeclass names, e.g. (snippet from my project):

fn mul_wide<A: FixI32, B: FixI32, R: FixI64>(a: &A, b: &B) -> R { ... }

but even then rustc can help when I omit something, e.g. when I change A: FixI32 to just A. Suggestions aren’t always working, but it’s better than no suggestions as in Scala. In this case rustc suggested me to change A to A: FixedPoint where FixedPoint is a supertrait of FixI32. Suggestion failed, but was somewhat close to truth. I can search for subtraits of FixedPoint - there are only 4 of them (FixI32, FixI64, FixU32 and FixU64) so choice is easy.

#17

I’m a huge fan of the proposed changes to implicits as well as these principles. As a software engineer and researcher that hops around a variety of codebases in different languages, my main frustrations with Scala implicits as they current exist are:

  1. Often times when I hit errors for missing an implicit argument that I then correct by importing some module, it seems random or arbitrary. As I write code without an IDE with auto-imports, the ambiguity as to why this particular import is fixing the problem can make the code harder to understand. A specific syntax for importing instances for implicit arguments is a huge quality of life improvement.
  2. The syntax for implicit conversions in Scala is not super explicit. While it’s a small issue, it’s annoying to have to remember the difference between implicit def and the other areas where implicit appears. I think the implied instance-based syntax that is being proposed will make it much more clear that an implicit conversion is being defined.
  3. The Scala 2 syntax for implicit parameter lists is also confusing. It looks as though only the first item in that list is implicit. Fixing that with the given syntax will make things a lot more clear, while also allowing for multiple implicit parameters lists (if I’m understanding the proposal correctly). The given syntax is the proposal I’m most excited about, as I believe it much better illustrates the “contextual” nature of implicit parameters.
  4. Generally the different meaning of implicit depending on the context makes code harder to understand.

While these may seem like small quibbles for more experienced or involved Scala engineers, these issues are why I do not make use of implicits in my own Scala code. I do think, as has been mentioned, that more helpful compiler errors are necessary to make using implicits easier for newcomers.

2 Likes
#18

I agree with this argument, and it’s why the evidence proposal sits poorly with me. I know that there’s a theorem prover in there, and I sorta-vaguely understand what is going on with it, but it’s not how I think about normal Scala programming. And I think that 95%+ of the Scala programmers out there would simply scratch their heads at it.

So while the whole types-as-proofs thing is real, I’d be very cautious about shoving peoples’ faces in it too aggressively – IMO, it’s likely to turn a number of people away…

#19

Thanks for explaining! To give better error messages here, the compiler would have to know the complete set of sources in a project and its dependencies. I have the impression that’s made possible by the fact that in Rust package management and compilation are tightly integrated. The Scala and Java world is different in that respect.

#20

Are sources really needed? Lots of information can be extracted from ordinary .class files. For example if I have such class in Java:

class MyComparator extends Comparator<MyType> {
  ...
}

Then I can analyze MyComparator with reflection and know that it parametrizes Comparator by MyType. https://www.ideone.com/EeLiq4

import java.lang.reflect.ParameterizedType;
import java.lang.reflect.Type;
import java.util.Comparator;
 
public class Main {
    class MyType {
    }
 
    class MyComparator implements Comparator<MyType> {
        @Override
        public int compare(MyType o1, MyType o2) {
            return 0;
        }
 
        @Override
        public boolean equals(Object obj) {
            return false;
        }
    }
 
    public static void main(String[] args) {
        Class<?> klass = MyComparator.class;
        Type comparatorType = klass.getGenericInterfaces()[0];
        System.out.println(comparatorType.getTypeName());
        ParameterizedType preciseType = (ParameterizedType) comparatorType;
        System.out.println(preciseType.getActualTypeArguments()[0]);
    }
}

Result:

java.util.Comparator<Main$MyType>
class Main$MyType

Scala constructs are obviously more complicated than Java ones, but Scala has its own reflection mechanism based on extra information stored (“pickled”) in .class files.

#21

Sorry, it’s the class files. But there again, the first step is to realize which sources or class files form part of a system. And in the Java world, that’s not so easy…