Synthesize constructor for opaque types

I agree that pattern matching against an opaque type pattern should give at least an unchecked warning: https://github.com/lampepfl/dotty/pull/10664

7 Likes

Would it be possible to make it illegal? People that want to do a type coercion could always use asInstanceOf. Maybe wouldn’t be so bad to have some precedence for illegal type testing? People defining opaque types could always define TypeTest instances if they want type tests to be possible.

Ignoring unchecked warnings is already unsafe, so I don’t think that going further for one special case is useful, perhaps there’s a debate to be had on whether unchecked warnings should be errors by default but that’s straying off topic.

2 Likes

Yeah that’s fascinating. I’ve had plenty of experiences where avoiding boxing made a significant perf improvement (very rigorously validated by JMH), and I’ve also had experiences where I’d a bunch of work that is undoubtedly a theoretic improvement only the see the results either not change or even get worse sometimes. Part of me thinks domain vs generic is a heuristic for determining with boxing will make a difference but I wouldn’t put much faith in that, even myself. The JVM still perplexes me to this day. Anyway…

I did not close any PR or issue, or shut down any discussion, just offered my opinion. Others are free to disagree.

100% and I want to emphasise that I haven’t seen you personally close a PR or shut down a discussion. As far as I can see, no one at all has shut down any discussion on this topic yet. To clarify, the reason I mentioned that at all is that it’s a pattern I’ve seen a few times where a discussion concludes but, for right or wrong, a majority (or maybe just a very large group) are still unconvinced, and the documentation and/or the community fail to sway new users who come across the same common use case. In those situations, PRs and discussions do end up just getting shut down because in the eyes of maintainers, the ship has sailed, the debate has been had and there’s no value in repeating it. To me this issue is (was?) starting to look like it would go down that path which is why I want to highlight it now so that even if we don’t modify the implementation, we beef up the documentation to transparently best address those common use cases as best we can, even if a significant amount of people disagree that opaque types are relevant to those use cases.

4 Likes

Fair enough. I think it’s safe to say that I already derailed this topic to the limit already. :sweat_smile:

Before I get a grip on myself and stop spamming I just want to say thanks to @odersky and @smarter for taking the time to address this despite everything on your plate with the Scala 3 launch! As a newcomer on this forum, that feels great! :tada:

3 Likes

Some new info gained from experiments is here: https://github.com/lampepfl/dotty/issues/10662#issuecomment-739852480

5 Likes

This is very interesting. Am I correct in assuming this deals with edge cases where the newtype-like opaque type usage is unsafe? (I’m missing the tl;dr)

Yes, that’s right. I’ll try to summarize:

EDIT: To clarify, the solution makes it safe to emulate Haskell’s newtype using opaque type, but only if no upper type bound is exposed, like in Haskell which has no subtype polymorphism at all, only parametric polymorphism.

Summary

Problem

The semantics of opaque type is not the same as newtype in Haskell. opaque type is actually much more general. The main reason opaque type doesn’t do encapsulation in the same way newtype do is that since Scala allows you to pattern match on litterally Anything, you can expose the underlying type representation (intentionally or by mistake) using pattern matching.

Quoted example (minimized)

For this reason, users using the pattern of defining a constructor/unapply method will experience some gotcha moments when opaque types don’t behave like case classes would have.*

Solution

This is solved in two steps:

  1. Disallow opaque types in typed patterns (like the one above).
  2. Restrict Any so that only subtypes of a new trait Matchable can be used as the scrutinee in pattern matching.

(1) prevents conversions like String -> Name by warning on patterns like:

(str: String) match {case n: Name => n}

Because Name can’t appear in a typed pattern in the case clause anymore.

(2) prevents conversions like Name -> String by warning on patterns like:

(n: Name) match {case str: String => str}

Because Name isn’t a subtype of Matchable so it can’t be pattern matched on at all.

Demo: (minimized)
scala> object n :
     |   opaque type Name = String
     |   object Name :
     |     def apply(str: String): Name = str
     |     def unapply(n: Name): Some[String] = Some(n)
// defined object n

scala> import n._

scala> "hi there" match {case n: Name => n}
1 |"hi there" match {case n: Name => n}
  |                       ^^^^^^^
  |                     the type test for n.Name cannot be checked at runtime
val res0: String & n.Name = hi there

scala> Name("Franz Kafka") match {case str: String => str}
1 |Name("Franz Kafka") match {case str: String => str}
  |                                     ^^^^^^
  |                      pattern selector should be an instance of Matchable,
  |                      but it has unmatchable type n.Name instead
val res1: n.Name & String = Franz Kafka

See this for more:

Note that this isn’t in master yet, and warnings won’t be turned on in 3.0.

*This is also why case classes should still be encouraged and preferred when their semantics is needed! Some kind of sugar like case opaque type could maybe be added in the future when the semantics of opaque type are better understood in practice and the feature has matured more. Preferably these unsafe patterns should be compiler errors rather than just warnings before such sugar is added.

1 Like

Matchable isn’t enough to make all usages of opaque types safe as I mentioned in Add `Matchable` trait - #3 by smarter, the problem is that users are free to define a visible upper-bound for their opaque type which is itself a subtype of Matchable.

2 Likes

Yes, it is enough to emulate Haskells newtype though (unless I’m mistaken?) since that doesn’t support type bounds anyway.

I have the feeling that this discussion, like the previous ones on the topic, has failed to conclude one major question – What is the main motivation behind opaques?

The original SIP explains the motivation as so:

Authors often introduce type aliases to differentiate many values that share a very common type (e.g. String , Int , Double , Boolean , etc.). In some cases, these authors may believe that using type aliases such as Id and Password means that if they later mix these values up, the compiler will catch their error. However, since type aliases are replaced by their underlying type (e.g. String ), these values are considered interchangeable (i.e. type aliases are not appropriate for differentiating various String values).

The current design of opaque types does not achieve that in the slightest, IMHO, and for one simple reason – opaques do not provide access to the methods of the underlying type (like type aliases do).

If better performing value-classes/wrappers is what people want, then there’s nothing wrong with the current design, but please let’s at least have the motivation for opaques cleared up and updated.

1 Like

It’s rather boilerplate heavy, as (at least as I understand it) macros can’t currently produce definitions and you have to re-wrap the type after each call, but you can use them in a way that provides access to the methods of the underlying type:

object types {
  object Name {
    opaque type Type <: String = String
    def apply(s: String): Type = s
  }
  type Name = Name.Type
  
  object Email {
    opaque type Type <: String = String
    def apply(s: String): Type = s
  }
  type Email = Email.Type
}
object using {
  import types.{Name, Email}

  @main
  def test (): Unit = {  
    def foo(str: String): Unit = {
      println(str)
    }
    def bar(n: Name, e: Email): Unit = {
      println(s"$n @ $e")
    }

    val name = Name("JDoe")
    val email = Email("[email protected]")
    
    //foo(name)
    //  Found:    (name : types.Name.Type)
    //  Required: String

    //foo(email)
    //  Found:    (email : types.Email.Type)
    //  Required: String

    foo(name)
    foo(email)

    bar(name, email)

    println(name.toLowerCase())
    
    //bar(name.toLowerCase(), email)
    //  Found:    String
    //  Required: types.Name
    
    bar(Name(name.toLowerCase()), email)
    
    //bar(email, name)
    //  Found:    (email : types.Email.Type)
    //  Required: types.Name
    //  Found:    (name : types.Name.Type)
    //  Required: types.Email
  }
}

You can extract the boiler plate into a trait::

  trait OpaqueOf[A] {
    opaque type Type <: A = A
    def apply(s: A): Type = s
  }
  object Name extends OpaqueOf[String]
  type Name = Name.Type
  object Email extends OpaqueOf[String]
  type Email = Email.Type

And it can be extended further. One could add a validating function to go from A to the opaque type, for example. I think that isn’t the biggest issue here. The criticism, as I understand it, is that you can’t do this:

object types {
  object Duration {
    opaque type Type <: Double = Double
    def apply(n: Double): Type = n
  }
  type Duration = Duration.type
}

object using {
  import types.Duration

  @main
  def test(): Unit = {
    val d1 = Duration(5.0)
    val d2 = Duration(2.0)
    val d3: Duration = d1 + d2  // should compile but doesn't
    println(s"d1: $d1 d2: $d2 total: $d3")

    val d4 = 10.0 + d1 // should not compile but does
  }
}

My understanding is that this is less than ideal because it will force primitives to box. I’d normally use this for stuff that’s already objects, so it’s less of a problem for me.

With a minor tweak, it does work. The missing step is that the result needs to be re-wrapped because the result of _: Duration + _: Duration is a Double - which makes sense because operation on the value may make it no longer valid, like there’s no way to guarantee that (_: Email).take(5) is still a valid Email:

object types {
  object Duration {
    opaque type Type <: Double = Double
    def apply(n: Double): Type = n
  }
  type Duration = Duration.Type
}

object using {
  import types.Duration

  @main
  def test(): Unit = {
    val d1 = Duration(5.0)
    val d2 = Duration(2.0)
    val d3: Duration = Duration(d1 + d2)
    println(s"d1: $d1 d2: $d2 total: $d3")

    val d4: Double = 10.0 + d1
    val d5: Duration = Duration(10.0d + d1)
  }
}

The d4 compiles because d1 known to be a Double subclass, which is usually fine for these sort of wrappers. Usually what you’d want to avoid is mixing subclasses, or passing a Double where you want to be sure that it’s a valid Duration.

Edit: sorry for being scatter-brained, here’s the Scastie link

1 Like

This can be reduced to

object types {
  opaque type Name <: String = String
  def Name(s: String): Name = s

  opaque type Email <: String = String
  def Email(s: String): Email = s
}
1 Like

It’s boilerplate heavy if you wish for better performing value-classes; it’s irrelevant if you wish for non-interchangeable type aliases.

I can’t think of any evidence or reasoning to support that point of view. How can you objectively declare how much boilerplate is relevant for each (hair-splitting) case? There’s nothing inherent about a non-interchangeable type alias that means any amount of boilerplate is irrelevant, and nothing inherent about value-classes to mandate the opposite. Maybe you can explain your reasoning a bit more?

To me: boilerplate is boilerplate. I don’t care about the categorisation of the domain I’m applying it to. I just think “well this could be automated but isn’t” and then I think about stuff like {frequency,ease} of {creating,modification} relative to value it provides.

1 Like

Don’t you think this is a wild exaggeration? Opaque type aliases give you abstraction without overhead, which is exactly what they set out to do. You are free to define an API by defining methods that operate on the opaque type, where you also specify type signatures.

What do you mean by non-interchangeable type alias? How would they work? What is the semantics?

If I define an opaque type:

object foobar:
  opaque type Foo = Bar

Do you want Foo to unconditionally simply have all methods of Bar with all occurrences of Bar changed to Foo?

I’m genuinely trying to understand what it is you want, because I don’t see what it is you want to do that can’t be done in the scope of the current proposal.

3 Likes

What I meant is that the boilerplate of the current proposal (of opaques) has nothing to do with non-interchangeable type alias, as the current proposal does not seem to address that particular feature/use case/motivation. Obviously, if and when that feature will be addressed, boilerplate would be something to take into account.

I do not. I would not use (the current design of) opaques for the purpose described in the original SIP (non-interchaneable type aliases), for the same reason I never used value-classes for that purpose nor mere wrapper classes. There is a very basic assumption that I believe is incorrect in the original SIP:

One appropriate solution to the above problem is to create case classes which wrap String .

That is, In my opinion, a workaround. Not an appropriate solution. And if for some reason I considered that an appropriate solution, I would still use mere wrapper classes and not opaques, as I would prefer their familiarity and simpleness over opaques, which only give a performance benefit I do not care for (and believe that most should not care for it either in most cases).

Authors often introduce type aliases to differentiate many values that share a very common type. […] However, since type aliases are replaced by their underlying type, these values are considered interchangeable.

I would have hoped that this SIP would address exactly those questions – how would they work and what would be the semantics – and try to come up with a design that focuses on providing a solution to the core motivation presented. Instead, it focused on providing a solution to a minor problem (performance) to one of the workarounds for the core motivation (wrapper classes).

I think that’s what’s been done. Opaque types allow you to define a type alias which is interchangeable in the scope they are defined, and to be non-interchangeable outside of that scope. It’s a very general and versatile solution with far broader use cases than value classes, which is why so many people have been arguing against weakening them to better performing value classes a la Haskell’s newtype by synthesizing constructors for them by default. (which is the whole point of this thread)

As I just said, opaque types are much more general than better performing value classes.

After reading your reply, I’m still non the wiser as to what you think is missing. It’s very hard to figure out a solution with well defined semantics without understanding what the problem you want to solve is.

So:

What is the problem that you want to solve? Opaque types give you non-interchangeable type aliases. Do you want to somehow expose all methods from the underlying type? What is it that I can’t do in the current proposal that you want to be able to do?

1 Like