Pre-SIP: Unboxed wrapper types

[NthPortal]

@non

I agree that the enrichment is a little bit cumbersome (which is why the first version of our proposal included it) but on balance I think the added flexiblity and power of opaque types is worth a bit of verbosity for enrichment.

I am very much behind opaque types, and I don’t actually think they add too much verbosity in most situations. However, for basic wrappers, they do.

Let me motivate my concern with the following example:

case class BrittleUser(id: Long, firstName: String, lastName: String, email: String)

case class User(id: User.Id, firstName: User.FirstName, lastName: User.LastName, email: User.Email)

object User {
  case class Id(value: Long) extends AnyVal
  case class FirstName(value: String) extends AnyVal
  case class LastName(value: String) extends AnyVal
  case class Email(value: String) extends AnyVal
}

By using a few short wrapper types, you get type safety, preventing you from getting the order of the fields wrong. However, to accomplish the same with opaque types is… a little bit ridiculous.

I like the idea of opaque types, and I think they add flexibility at extremely low cost for types which are more than a simple wrapper (such as your Mode type). However, in some situations, they lose to AnyVals in source maintainability even though they win in performance.

[lihaoyi]

My thoughts on that SIP:

• I really like the parallel of "class => actual reified/allocatable thing, type => compile-type thing with no runtime representation"

• AnyVal boxing unpredictably is a huge downside. Good performance is good, bad performance is meh, but unpredictable performance is the absolute worst.

• “Badly” behaved isInstanceOf/asInstanceOf is much less of a problem than people think.

  • Those two methods are badly behaved in Scala.js, at least compared to Scala-JVM: (1: Int).isInstanceOf[Double] == true, anyThingAtAll.asInstanceOf[SomeJsTrait] never fails, etc. Of the things people get confused with about Scala.js, this doesn’t turn up that often.

  • Scala-JVM isInstanceOf/asInstanceOf are already a bit sloppy due to generic type-erasure, e.g. List[Int]().isInstanceOf[List[String]] == true. Opaque wrapper types would simply be extending the type-erasure to non-generic contexts

  • Being able to say "myStringConstant".asInstanceOf[OpaqueStringConstantType] is widely used in Scala.js by “normal” users e.g. for wrapping third-party library constant/enum/js-dictionary-like things in something more type-safe. It’s actually very, very convenient, and empirically the odd behavior of isInstanceOf/asInstanceOf when you make mistakes in such cases just doesn’t seem to cause much confusion for people.

• In a similar vein, I would be happy for Array[MyOpaqueTypeWrappingDouble]().isInstanceOf[Array[Double]] == true. I want the predictable performance more than I want the predictable isInstanceOf behavior: if I wanted the other way round, I would use AnyVals or just normal boxes instead!

• I think @NthPortal’s concern about boilerplate is valid. While it’s “ok” to provide the low-level opaque-type and then tell people to build helpers on top using implicit extensions, it would be “nice” to have a concise syntax for common cases: one of which is opaque type + manual conversions to-and-from the underlying/wrapped type.

  For example, given

  opaque type Id = Long
  
  object Id {
    def apply(value: Long): Id = value
    implicit class Ops(val self: Id) extends AnyVal {
      def value: Long = self
    }
  }
  

  Would it be possible to extract all the boilerplate def apply/implicit class into a helper trait?

  opaque type Id = Long
  object Id extends OpaqueWrapperTypeCompanion[Long, Id] // comes with `def apply` and `implicit class`
  

• If we allowed asInstanceOf to be sloppy and let you cast things between the underlying and opaque types, we could dispense the the special “underlying type can be seen as opaque type in companion object, and vice versa” rule: you want to convert between them, use a asInstanceOf, and be extra-careful just like when using asInstanceOf in any other place.

  • This asInstanceOf behavior may already be unavoidable anyway (from and implementation point of view) if you want to avoid boxing in all cases, e.g. when assigned to Anys, or when put in Array[MyOpaqueType]s.

  • And asInstanceOf already has the correct connotation in users’ heads: “be extra careful here, we’re doing a conversion that’s not normally allowed”

  • This also obliviates defining a companion object for the common case of “plain” newtypes, without any computation/validation: just use (x: Double).asInstanceOf[Wrapper] and (x: Wrapper).asInstanceOf[Double] to convert between them. If someone wants to add custom computation/validation, they can still write a companion and do that. In particular, @NthPortal’s examples could then just use casting and not define a bunch of boilerplaty companion objects.

• It might be worth exploring a custom syntax to smooth out the very-common case of “opaque type with methods defined on it”, rather than relying on implicit classes in the companion to satisfy this need.

  People could continue to use implicit class extension methods when extending the types externally, but I think having some operations “built in” is common enough to warrant special syntax. I don’t have any idea what they might look like

[non]

Yeah, this is my misunderstanding of the value classes spec.

Value classes do have to expose a public wrapper and unwrapper to the JVM, but you’re correct that they aren’t exposed to Scala. I’ll edit my post to correct it.

For the curious, here’s some Java code that constructs an invalid Mode and is able to access its internals (but this wouldn’t work in Scala):

package example;

import example.Mode;

class Test {

    public static void test() {
        Mode m = new Mode(999);
        System.out.println(m.foo$bar$Mode$$raw());
    }
}

[non]

I think casting will largely “work” in the same ways it does now (i.e. you can cast between erased and unerased types on the JVM and things won’t throw a CCE):

def forget(xs: List[Int]): List[Object] =
  xs.asInstanceOf[List[Object]]
def remember(xs: List[Object]): List[Int] =
  xs.asInstanceOf[List[Int]]

remember(forget(1 :: 2 :: 3 :: Nil))
// res2: List[Int] = List(1, 2, 3)

I don’t think it’s worth giving up on the transparency in the companion, opacity elsewhere since it makes it easy for authors to signal which operations they’d like the compiler to consider type safe. Library consumers will still have most of the same escape hatches they have elsewhere (and the same warnings about using them).

[galedric]

About the syntax boilerplate, maybe it could be possible to allow opaque as a modifier for a class to mean “an opaque type + the standard boilerplate”, similarly to how implicit class means “a class + an implicit def”.

For example:

opaque [implicit] class T ([private] val a: A) {
	{defs...}
}

Could be syntaxic sugar for:

opaque type T = A
object T {
	[implicit] def apply(a: A): T = a
	
	implicit class T$Ops ($self: T) extends AnyVal {
		inline(?) [private] def a: A = t
		{defs...}
	}
}

Requirements of value classes would apply equally to “opaque” classes:

• A single val parameter
• No fields definitions, only methods
• Cannot be nested inside another class

Ideally, this inside the opaque class body would be rewritten to $self and have type T.

Adding implicit to the opaque class definition would add implicit on the apply method, meaning that the value can be implicitly wrapped, otherwise explicit wrapping is required.

Adding private to the constructor parameter would prevent the underlying value from being accessed directly and require explicit accessors to be defined.

Maybe we could even imagine opaque class T private ( ... ) that would mean that the apply method synthesized on the companion object is also private, requiring custom wrapper to be defined on the companion object (eg. to perform validation).

While this syntax is arguably more complex that the single opaque type definition, I believe it allows many common use cases of opaque type to be expressed with a lot less boilerplate. It would also be syntactically very similar to current value classes, meaning that converting code would be as easy as replacing the extends by opaque to get the unboxed semantic.

This syntax might also scale to future JVM-level value classes by allowing more than a single parameter in the class constructor, but who knows. This may not be a goal for this feature.

A last idea: if some people are not willing to introduce opaque as a keyword, maybe the inline keyword from Dotty could be used instead (an inline class is a class that disappear at runtime), it obviously work a lot better in the inline class than in the the inline type version.

[LPTK]

@NthPortal

I am very much behind opaque types, and I don’t actually think they add too much verbosity in most situations. However, for basic wrappers, they do.

@lihaoyi

it would be “nice” to have a concise syntax for common cases

@galedric

maybe it could be possible to allow opaque as a modifier for a class

Is there really no love for macro annotations here? (ping @xeno-by)
It would be straightforward to make the syntax proposed by @galedric work with an @opaque macro to annotate class definitions (which would then expand into opaque types).

@opaque class T(val a: A) { ... }  // expands into opaque type with helpers

Why make the spec of the language more and more complicated, when the problem is simply boilerplate, a problem that is basically solved by macro annotations? IMHO the language should continue in its original goal of being scalable, providing simple but powerful tools to be used as primitives for building advanced features –– as opposed to being an ad-hoc assembly of specific features from someone’s wish-list at instant t.

[non]

@LPTK

Is there really no love for macro annotations here? (ping @xeno-by)
It would be straightforward to make the syntax proposed by @galedric work with an @opaque macro to annotate class definitions (which would then expand into opaque types).

I was about to write a similar reply, but I’ll instead just second this suggestion. I’d prefer that opaque types form a foundation that things like newtypes or other projects can build on to reduce boilerplate or provide specific semantics that folks want.

It seems like this shouldn’t be too hard. Maybe someone with a bit more experience can weigh in on this?

One reason including this kind of syntax in this SIP might be difficult is that we’ve already received feedback that the first version (which included this kind of syntax) overlapped too much with value classes, and that the feature didn’t seem sufficiently novel. By focusing on the types themselves (and introducing the opaque type companion) we can keep things more orthogonal, and leave this part of the design space to either future value class improvements (another SIP anyone?) or to macros or some other higher-level library.

[NthPortal]

@LPTK

Is there really no love for macro annotations here?

I think that’s a really good solution!

[NthPortal]

Can opaque types be parameterized?

A possible use could be for complex numbers of any numeric type

opaque type Complex[A] = (A, A) // possibly Complex[A: Numeric]?

object Complex {
  def apply[A: Numeric](real: A, imaginary: A): Complex[A] = (real, imaginary)
  def apply[A: Numeric](pair: (A, A)): Complex[A] = pair

  implicit class Ops[A](val self: Complex[A]) extends AnyVal {
    def real: A = self._1
    def imaginary: A = self._2
    def +(that: Complex[A])(implicit num: Numeric[A]): Complex[A] =
      (num.plus(self._1, that._1), num.plus(self._2, that._2))
    // More ops here...
  }
}

Alternatively, one could imagine a type for unsigned primitives

opaque type Unsigned[A] = A

object Unsigned {
  def apply(value: Int): Unsigned[Int] = value
  def apply(value: Long): Unsigned[Long] = value
  // ...

  implicit class Ops[A](val self: Unsigned[A]) extends AnyVal {
    // math ops here
  }
}

[sjrd]

Yes, they can be parameterized, like any other type alias.

[S11001001]

@lihaoyi

If we allowed asInstanceOf to be sloppy and let you cast things between the underlying and opaque types, we could dispense the the special “underlying type can be seen as opaque type in companion object, and vice versa” rule: you want to convert between them, use a asInstanceOf, and be extra-careful just like when using asInstanceOf in any other place.

This is sort of like getting visibility on the type equality with a well-placed shotgun blast. A well-typed program should not be forced to break the type system repeatedly with the “I am not a well-typed program anymore” operator to get at basic functionality.

The companion should be thought of as a generalization of member methods, functionality-wise, from “substitute exactly one in parameter position” to “substitute in any position”.

@lihaoyi

or when put in Array[MyOpaqueType]s

There are two interesting approaches to Array:

1. If you’re designing an opaque type, you get to choose whether it should be Array-able, by either supplying an implicit ClassTag[MyOpaqueType] in the companion to allow it, or not doing so to disallow it, as with type parameters. This is how it stands with the spec now, I guess.
2. Compiler ClassTag support can subst an opaque type’s RHS’s ClassTag.

[olafurpg]

Is there really no love for macro annotations here?

Macro annotations need to be included in the language spec before they’re used as an official complement for new language features.

[LPTK]

Sure. I was hoping that they would be part of the language eventually. At least, that’s what we have been led to believe.

Macro annotations are such an important feature IMO, for the reason I mentioned (an overhead-free option to defeating boilerplate), that I really hope they make it. Other languages are moving in that direction too: see Template Haskell, the code-generation mixins of D, the “meta-classes” proposal for the next C++ (which are pale in comparison!), etc.

[TomasMikula]

I would like to revive the issue of “code that knows multiple opaque types” brought up above.

@jvican

Multiple opaque types

As I explain in the meeting, this has several problems:

• APIs of different opaque types get mixed, hampering readability of the code.
• Users cannot define a method tag for two different opaque types that have the same underlying type. The same happens with implicits.
• Use sites of these opaque types do not know where these methods are defined. It’s way clearer to see Logarithm.tag than tag somewhere in your program.

These points all assume that there will be no type companions, but that need not be the case. @S11001001 only said “type companions may be eliminated.”

@jvican

I don’t like the idea of defining multiple opaque types in the same prefix. I’m personally in favor of opaque type companions, and I think companions are a natural way of thinking about Scala code.

Still, the problem of how to write a method that sees through multiple opaque types remains.

---

One (unsatisfactory) solution is to introduce artificial nesting:

object graphics {
  opaque type Image = ImageImpl
  object Image {
    opaque type Point = PointImpl
    object Point {
      opaque type Color = ColorImpl
      object Color {
        def colorAt(img: Image, pt: Point): Color = {
          // The definitions of Image, Point and Color are all visible here.
          ???
        }
      }
    }
  }
}

---

Another (still unsatisfactory IMO) solution is to provide type equalities:

object graphics {
  opaque type Image = ImageImpl
  object Image {
    private[graphics] def ev: Image =:= ImageImpl = implicitly
  }

  opaque type Point = PointImpl
  object Point {
    private[graphics] def ev: Point =:= Pointimpl = implicitly
  }

  opaque type Color = ColorImpl
  object Color {
    private[graphics] def ev: Color =:= ColorImpl = implicitly
  }

  def colorAt(img: Image, pt: Point): Color = {
    // first convert the inputs to their representations
    val rawImg = Image.ev(img)
    val rawPt  = Point.ev(pt)

    // do the work
    val rawColor = ???

    // covert the result to the public opaque type
    Color.ev.flip(rawColor)
  }

This doesn’t look too scary yet, but more generally we would also need

• calls to =:=.substitute, which are less readable than =:=.apply used above;
• higher-kinded versions of type equality (i.e. equality of type constructors), which is not even part of the standard library;
• witnesses of subtyping (<:<) and their higher-kinded versions (again not part of the standard library).

This is a lot of boilerplate and advanced concepts to achieve a no-op conversion.

---

@sjrd suggested to qualify the visibility of opaque types:

object graphics {
  opaque[graphics] type Image = ImageImpl
}

meaning that " the opaque type alias can be dealiased everywhere inside graphics , but not outside of it."

This seems like a reasonable solution to me.

[RichType]

So if I might indulge by sharing my own use case.

A lot of my data is collections of 2d points, collections of 3d points, collections of 2 and 3 dimensional distance points (2000.0 is not the same thing as 2000.0 meters), collections of latitude-longitudes and collections of 2 dimensional integer coordinates.

So taking the worst case of the integer coordinates. Say I’ve got a sequence of 6 2-dimenional integer coordinates. I think I’m right in saying that they can be represented by an integer Array of only 384 bytes. Where as if I store them as a List of coordinates, without optimisation, the run time (which in my case is both JVM and Js) would produce 18 heap objects. This surely isn’t acceptable, especially on the Js runtime where you can be far less confident of unnecessary objects being optimised away.

My instinct would have been to forgo type safety and just use type aliases for Int and Double Arrays. However I have a Transformable type class, as many things can be transformed eg buttons and flags, not just plain polygons. Using type aliases for the compiler will not be able to distinguish between the transform implicit for a 2 dimensional vector collection and a 3d vector collection which need to be different.

So I presume the current recommendation would be to use boxed Arrays. I could use value classes, but as these will box the array, I don’t see what the advantage would be over using a regular case class, or an ordinary class so I can fully control the constructors and factories.

Hopefully it should be then be relatively straight forward to convert the boxed array classes into Opaque types at a later date when they land in the standard language.

[adamw]

Hello,

first of all - thanks Erik for the presentation during the Typelevel summit, great source on what’s the current status of the proposal!

Now to the feedback - in general I like the proposal very much and I think it might prove very handy when available. However, I’ve got one concern as, if I understand things correctly, a lot of use-cases might overlap with what value classes are used today for.

The SIP-35 proposal (https://docs.scala-lang.org/sips/opaque-types.html#) does include a section which discusses differences between opaque types and value classes, however on a quite advanced level. What I think might be a problem is the beginner / first-time user coming to Scala. There’s already a number of features in Scala and this is sometimes cited as Scala’s disadvantage. If we are to add yet another one, I think it would be great to have very clear guidelines on when to use the feature and how does it interact with other Scala features. Especially that both value classes and opaque types solve the newtype problem, although in different ways.

Not sure if that’s the scope of the SIP, but maybe apart from the code, it should include documentation, including introductory documentation which would end up on http://scala-lang.org? (Maybe it does exist, but if so, I didn’t find it.)

In this case, the docs might include a guide which would clearly specify when to use opaque types, and when to use value classes (this could also preempt a number of blogs on the topic ), and maybe even more importantly, when not to use them. That way a non-expert user has would have clarity which construct to use when.

Adam

[jvican]

@TomasMikula I’m thinking about that use case. Will drop a comment soon with my thoughts.

@adamw

If we are to add yet another one, I think it would be great to have very clear guidelines on when to use the feature and how does it interact with other Scala features. Especially that both value classes and opaque types solve the newtype problem, although in different ways.

Indeed. If opaque types get merged, my personal view is that we should officially recommend Scala developers not to use value classes for newtype use cases, and only use them for extension methods.

@adamw

In this case, the docs might include a guide which would clearly specify when to use opaque types, and when to use value classes (this could also preempt a number of blogs on the topic ), and maybe even more importantly, when not to use them. That way a non-expert user has would have clarity which construct to use when.

This is a good suggestion! I also think this will need to be done. We can try to have a preliminary view of that documentation in the SIP, even though in general all that information will be added to the Scala tours and the Scala documentation after the SIP is accepted and available in a Scala version.

[adriaanm]

Indeed! The intent is that opaque types + direct support for extension methods (not SIPped yet, but actively researched in dotty) would subsume and replace value classes and implicit classes. A very nice simplification, which I’m very excited about!

[SethTisue]

At the SIP meeting I voiced some skepticism about having both opaque types and value classes. I’m feeling better about that now, based on this discussion and other discussions, since the understanding is that the overlap is temporary. Text in the SIP itself should reflect this, but once that text is in, I’ll be satisfied — even before the Dotty work on replacing value classes has fully played out.

[smarter]

@adriaanm

The intent is that opaque types + direct support for extension methods (not SIPped yet, but actively researched in dotty) would subsume and replace value classes and implicit classes.

I find this very surprising (and this goes against what I’ve heard @sjrd say, for example here: https://gitter.im/lampepfl/dotty?at=5ab02ef2e4ff28713a4f8bc7), what happens when Java gets value classes then?