Pre-SIP: Unboxed wrapper types

[non]

I think casting will largely “work” in the same ways it does now (i.e. you can cast between erased and unerased types on the JVM and things won’t throw a CCE):

def forget(xs: List[Int]): List[Object] =
  xs.asInstanceOf[List[Object]]
def remember(xs: List[Object]): List[Int] =
  xs.asInstanceOf[List[Int]]

remember(forget(1 :: 2 :: 3 :: Nil))
// res2: List[Int] = List(1, 2, 3)

I don’t think it’s worth giving up on the transparency in the companion, opacity elsewhere since it makes it easy for authors to signal which operations they’d like the compiler to consider type safe. Library consumers will still have most of the same escape hatches they have elsewhere (and the same warnings about using them).

[galedric]

About the syntax boilerplate, maybe it could be possible to allow opaque as a modifier for a class to mean “an opaque type + the standard boilerplate”, similarly to how implicit class means “a class + an implicit def”.

For example:

opaque [implicit] class T ([private] val a: A) {
	{defs...}
}

Could be syntaxic sugar for:

opaque type T = A
object T {
	[implicit] def apply(a: A): T = a
	
	implicit class T$Ops ($self: T) extends AnyVal {
		inline(?) [private] def a: A = t
		{defs...}
	}
}

Requirements of value classes would apply equally to “opaque” classes:

• A single val parameter
• No fields definitions, only methods
• Cannot be nested inside another class

Ideally, this inside the opaque class body would be rewritten to $self and have type T.

Adding implicit to the opaque class definition would add implicit on the apply method, meaning that the value can be implicitly wrapped, otherwise explicit wrapping is required.

Adding private to the constructor parameter would prevent the underlying value from being accessed directly and require explicit accessors to be defined.

Maybe we could even imagine opaque class T private ( ... ) that would mean that the apply method synthesized on the companion object is also private, requiring custom wrapper to be defined on the companion object (eg. to perform validation).

While this syntax is arguably more complex that the single opaque type definition, I believe it allows many common use cases of opaque type to be expressed with a lot less boilerplate. It would also be syntactically very similar to current value classes, meaning that converting code would be as easy as replacing the extends by opaque to get the unboxed semantic.

This syntax might also scale to future JVM-level value classes by allowing more than a single parameter in the class constructor, but who knows. This may not be a goal for this feature.

A last idea: if some people are not willing to introduce opaque as a keyword, maybe the inline keyword from Dotty could be used instead (an inline class is a class that disappear at runtime), it obviously work a lot better in the inline class than in the the inline type version.

[LPTK]

@NthPortal

I am very much behind opaque types, and I don’t actually think they add too much verbosity in most situations. However, for basic wrappers, they do.

@lihaoyi

it would be “nice” to have a concise syntax for common cases

@galedric

maybe it could be possible to allow opaque as a modifier for a class

Is there really no love for macro annotations here? (ping @xeno-by)
It would be straightforward to make the syntax proposed by @galedric work with an @opaque macro to annotate class definitions (which would then expand into opaque types).

@opaque class T(val a: A) { ... }  // expands into opaque type with helpers

Why make the spec of the language more and more complicated, when the problem is simply boilerplate, a problem that is basically solved by macro annotations? IMHO the language should continue in its original goal of being scalable, providing simple but powerful tools to be used as primitives for building advanced features –– as opposed to being an ad-hoc assembly of specific features from someone’s wish-list at instant t.

[non]

@LPTK

Is there really no love for macro annotations here? (ping @xeno-by)
It would be straightforward to make the syntax proposed by @galedric work with an @opaque macro to annotate class definitions (which would then expand into opaque types).

I was about to write a similar reply, but I’ll instead just second this suggestion. I’d prefer that opaque types form a foundation that things like newtypes or other projects can build on to reduce boilerplate or provide specific semantics that folks want.

It seems like this shouldn’t be too hard. Maybe someone with a bit more experience can weigh in on this?

One reason including this kind of syntax in this SIP might be difficult is that we’ve already received feedback that the first version (which included this kind of syntax) overlapped too much with value classes, and that the feature didn’t seem sufficiently novel. By focusing on the types themselves (and introducing the opaque type companion) we can keep things more orthogonal, and leave this part of the design space to either future value class improvements (another SIP anyone?) or to macros or some other higher-level library.

[NthPortal]

@LPTK

Is there really no love for macro annotations here?

I think that’s a really good solution!

[NthPortal]

Can opaque types be parameterized?

A possible use could be for complex numbers of any numeric type

opaque type Complex[A] = (A, A) // possibly Complex[A: Numeric]?

object Complex {
  def apply[A: Numeric](real: A, imaginary: A): Complex[A] = (real, imaginary)
  def apply[A: Numeric](pair: (A, A)): Complex[A] = pair

  implicit class Ops[A](val self: Complex[A]) extends AnyVal {
    def real: A = self._1
    def imaginary: A = self._2
    def +(that: Complex[A])(implicit num: Numeric[A]): Complex[A] =
      (num.plus(self._1, that._1), num.plus(self._2, that._2))
    // More ops here...
  }
}

Alternatively, one could imagine a type for unsigned primitives

opaque type Unsigned[A] = A

object Unsigned {
  def apply(value: Int): Unsigned[Int] = value
  def apply(value: Long): Unsigned[Long] = value
  // ...

  implicit class Ops[A](val self: Unsigned[A]) extends AnyVal {
    // math ops here
  }
}

[sjrd]

Yes, they can be parameterized, like any other type alias.

[S11001001]

@lihaoyi

If we allowed asInstanceOf to be sloppy and let you cast things between the underlying and opaque types, we could dispense the the special “underlying type can be seen as opaque type in companion object, and vice versa” rule: you want to convert between them, use a asInstanceOf, and be extra-careful just like when using asInstanceOf in any other place.

This is sort of like getting visibility on the type equality with a well-placed shotgun blast. A well-typed program should not be forced to break the type system repeatedly with the “I am not a well-typed program anymore” operator to get at basic functionality.

The companion should be thought of as a generalization of member methods, functionality-wise, from “substitute exactly one in parameter position” to “substitute in any position”.

@lihaoyi

or when put in Array[MyOpaqueType]s

There are two interesting approaches to Array:

1. If you’re designing an opaque type, you get to choose whether it should be Array-able, by either supplying an implicit ClassTag[MyOpaqueType] in the companion to allow it, or not doing so to disallow it, as with type parameters. This is how it stands with the spec now, I guess.
2. Compiler ClassTag support can subst an opaque type’s RHS’s ClassTag.

[olafurpg]

Is there really no love for macro annotations here?

Macro annotations need to be included in the language spec before they’re used as an official complement for new language features.

[LPTK]

Sure. I was hoping that they would be part of the language eventually. At least, that’s what we have been led to believe.

Macro annotations are such an important feature IMO, for the reason I mentioned (an overhead-free option to defeating boilerplate), that I really hope they make it. Other languages are moving in that direction too: see Template Haskell, the code-generation mixins of D, the “meta-classes” proposal for the next C++ (which are pale in comparison!), etc.

[TomasMikula]

I would like to revive the issue of “code that knows multiple opaque types” brought up above.

@jvican

Multiple opaque types

As I explain in the meeting, this has several problems:

• APIs of different opaque types get mixed, hampering readability of the code.
• Users cannot define a method tag for two different opaque types that have the same underlying type. The same happens with implicits.
• Use sites of these opaque types do not know where these methods are defined. It’s way clearer to see Logarithm.tag than tag somewhere in your program.

These points all assume that there will be no type companions, but that need not be the case. @S11001001 only said “type companions may be eliminated.”

@jvican

I don’t like the idea of defining multiple opaque types in the same prefix. I’m personally in favor of opaque type companions, and I think companions are a natural way of thinking about Scala code.

Still, the problem of how to write a method that sees through multiple opaque types remains.

---

One (unsatisfactory) solution is to introduce artificial nesting:

object graphics {
  opaque type Image = ImageImpl
  object Image {
    opaque type Point = PointImpl
    object Point {
      opaque type Color = ColorImpl
      object Color {
        def colorAt(img: Image, pt: Point): Color = {
          // The definitions of Image, Point and Color are all visible here.
          ???
        }
      }
    }
  }
}

---

Another (still unsatisfactory IMO) solution is to provide type equalities:

object graphics {
  opaque type Image = ImageImpl
  object Image {
    private[graphics] def ev: Image =:= ImageImpl = implicitly
  }

  opaque type Point = PointImpl
  object Point {
    private[graphics] def ev: Point =:= Pointimpl = implicitly
  }

  opaque type Color = ColorImpl
  object Color {
    private[graphics] def ev: Color =:= ColorImpl = implicitly
  }

  def colorAt(img: Image, pt: Point): Color = {
    // first convert the inputs to their representations
    val rawImg = Image.ev(img)
    val rawPt  = Point.ev(pt)

    // do the work
    val rawColor = ???

    // covert the result to the public opaque type
    Color.ev.flip(rawColor)
  }

This doesn’t look too scary yet, but more generally we would also need

• calls to =:=.substitute, which are less readable than =:=.apply used above;
• higher-kinded versions of type equality (i.e. equality of type constructors), which is not even part of the standard library;
• witnesses of subtyping (<:<) and their higher-kinded versions (again not part of the standard library).

This is a lot of boilerplate and advanced concepts to achieve a no-op conversion.

---

@sjrd suggested to qualify the visibility of opaque types:

object graphics {
  opaque[graphics] type Image = ImageImpl
}

meaning that " the opaque type alias can be dealiased everywhere inside graphics , but not outside of it."

This seems like a reasonable solution to me.

[RichType]

So if I might indulge by sharing my own use case.

A lot of my data is collections of 2d points, collections of 3d points, collections of 2 and 3 dimensional distance points (2000.0 is not the same thing as 2000.0 meters), collections of latitude-longitudes and collections of 2 dimensional integer coordinates.

So taking the worst case of the integer coordinates. Say I’ve got a sequence of 6 2-dimenional integer coordinates. I think I’m right in saying that they can be represented by an integer Array of only 384 bytes. Where as if I store them as a List of coordinates, without optimisation, the run time (which in my case is both JVM and Js) would produce 18 heap objects. This surely isn’t acceptable, especially on the Js runtime where you can be far less confident of unnecessary objects being optimised away.

My instinct would have been to forgo type safety and just use type aliases for Int and Double Arrays. However I have a Transformable type class, as many things can be transformed eg buttons and flags, not just plain polygons. Using type aliases for the compiler will not be able to distinguish between the transform implicit for a 2 dimensional vector collection and a 3d vector collection which need to be different.

So I presume the current recommendation would be to use boxed Arrays. I could use value classes, but as these will box the array, I don’t see what the advantage would be over using a regular case class, or an ordinary class so I can fully control the constructors and factories.

Hopefully it should be then be relatively straight forward to convert the boxed array classes into Opaque types at a later date when they land in the standard language.

[adamw]

Hello,

first of all - thanks Erik for the presentation during the Typelevel summit, great source on what’s the current status of the proposal!

Now to the feedback - in general I like the proposal very much and I think it might prove very handy when available. However, I’ve got one concern as, if I understand things correctly, a lot of use-cases might overlap with what value classes are used today for.

The SIP-35 proposal (https://docs.scala-lang.org/sips/opaque-types.html#) does include a section which discusses differences between opaque types and value classes, however on a quite advanced level. What I think might be a problem is the beginner / first-time user coming to Scala. There’s already a number of features in Scala and this is sometimes cited as Scala’s disadvantage. If we are to add yet another one, I think it would be great to have very clear guidelines on when to use the feature and how does it interact with other Scala features. Especially that both value classes and opaque types solve the newtype problem, although in different ways.

Not sure if that’s the scope of the SIP, but maybe apart from the code, it should include documentation, including introductory documentation which would end up on http://scala-lang.org? (Maybe it does exist, but if so, I didn’t find it.)

In this case, the docs might include a guide which would clearly specify when to use opaque types, and when to use value classes (this could also preempt a number of blogs on the topic ), and maybe even more importantly, when not to use them. That way a non-expert user has would have clarity which construct to use when.

Adam

[jvican]

@TomasMikula I’m thinking about that use case. Will drop a comment soon with my thoughts.

@adamw

If we are to add yet another one, I think it would be great to have very clear guidelines on when to use the feature and how does it interact with other Scala features. Especially that both value classes and opaque types solve the newtype problem, although in different ways.

Indeed. If opaque types get merged, my personal view is that we should officially recommend Scala developers not to use value classes for newtype use cases, and only use them for extension methods.

@adamw

In this case, the docs might include a guide which would clearly specify when to use opaque types, and when to use value classes (this could also preempt a number of blogs on the topic ), and maybe even more importantly, when not to use them. That way a non-expert user has would have clarity which construct to use when.

This is a good suggestion! I also think this will need to be done. We can try to have a preliminary view of that documentation in the SIP, even though in general all that information will be added to the Scala tours and the Scala documentation after the SIP is accepted and available in a Scala version.

[adriaanm]

Indeed! The intent is that opaque types + direct support for extension methods (not SIPped yet, but actively researched in dotty) would subsume and replace value classes and implicit classes. A very nice simplification, which I’m very excited about!

[SethTisue]

At the SIP meeting I voiced some skepticism about having both opaque types and value classes. I’m feeling better about that now, based on this discussion and other discussions, since the understanding is that the overlap is temporary. Text in the SIP itself should reflect this, but once that text is in, I’ll be satisfied — even before the Dotty work on replacing value classes has fully played out.

[smarter]

@adriaanm

The intent is that opaque types + direct support for extension methods (not SIPped yet, but actively researched in dotty) would subsume and replace value classes and implicit classes.

I find this very surprising (and this goes against what I’ve heard @sjrd say, for example here: https://gitter.im/lampepfl/dotty?at=5ab02ef2e4ff28713a4f8bc7), what happens when Java gets value classes then?

[sjrd]

It doesn’t completely go against what I said (FTR, here).

First, most current uses of value classes currently should have been extension methods from the start, had we had those. So there’s that.

Then, a bunch of other use cases of value classes can be advantageously replaced by opaque type aliases and extension methods, when they were used to avoid boxing. Opaque type aliases will not be a magical tool removing all boxing, but it will box less often (and more predictably) than value classes, which is still valuable.

I still think there are a few use cases for value classes per se: when observing a distinct class at run-time is more valuable than the perks of opaque types. But I think they will be rare, at least until we can make multi-parameter value classes work, possibly with the help of the JVM value classes.

On the other hand, implicit classes should be completely subsumed by extension methods.

Overall, I agree that opaque type aliases + extension methods will subsume almost all current use cases of value classes and implicit classes. And for the remaining parts, it will probably give us more leeway to adapt AnyVals to whatever the JVM has to offer, in the future.

[NthPortal]

And if we get rid of the current AnyVals soon, and long enough before Valhalla arrives, it leaves open the possibility of reusing the type/syntax for the more flexible and powerful JVM value types.

[adamw]

Ah, I was wondering how extension methods add to this mix in Dotty - now that’s clear, thanks.

But even more so, if value classes are soon-to-be deprecated, I suppose such an explanatory note - presumably in the guides for 2.13 - that newtypes should be handled by opaque types, and value classes, temporarily, for extension methods (in combination w/ implicit classes) would already be a valuable and concise guide for beginners. Or even not-so-beginners who don’t follow the forums