Improve opaque types

[Jasper-M]

@gvolpe

These approaches partially address my concerns. I see them as workarounds that involve quite some boilerplate. I would love to see this feature being baked in into the language, if that’s possible.

While true, a lot of this could be hidden behind a simple interface in a library.

scala> class NewType[A]:
     |   opaque type Type = A
     |   def apply(a: A): Type = a
     |   extension (a: Type) def value: A = a
     | end NewType
// defined class NewType

scala> class NewIntType extends NewType[Int]:
     |   override opaque type Type = Int
     |   override def apply(a: Int): Type = a
     |   extension (a: Type) override def value: Int = a
     | end NewIntType
// defined class NewIntType

scala> import scala.compiletime.erasedValue

scala> transparent inline def newType[A] = inline erasedValue[A] match {
     |   case _: Int => new NewIntType
     |   case _ => new NewType[A]
     | }
def newType[A] => NewType[? >: Int & A <: Int | A]

scala> val Foo = newType[Int]; type Foo = Foo.Type
val Foo: NewIntType = NewIntType@346d8002
// defined alias type Foo = Foo.Type

scala> val Bar = newType[String]; type Bar = Bar.Type
val Bar: NewType[String] = NewType@593d5f39
// defined alias type Bar = Bar.Type

Notice you have one entrypoint to make a newtype “module”, newType[A], and when A is an AnyVal you statically get the specialized instance, otherwise the generic variant for types that don’t suffer from boxing.

Taking it even further would probably require something like macro annotations…

[gvolpe]

That’s not bad! Could definitely use something like this if it’s not built into the language

[dcsobral]

@gvolpe

These approaches partially address my concerns. I see them as workarounds that involve quite some boilerplate. I would love to see this feature being baked in into the language, if that’s possible.

Why burden the language with something that can be implemented as a library? That goes against Scala’s design principles.

To avoid boxing, we need to create multiple traits for every primitive type.

Yes, so what? The user doesn’t have to deal with it, and even the code for the traits can be done mechanically as generated source – except for method redirection which needs to be curated. More on this later.

trait NewStringType:
  opaque type Type = Int
  def apply(a: Int): Type = a
  extension (a: Type) def value: Int = a
end NewIntType

Strings are not primitive. They are just used like one.

Surely this could be made into a library and we can all move on. However, I think it’s a pity we have a complete new language without this feature so the idea of sparking this conversation once again was to see whether it was possible to ship proper newtypes as part of the language.

I’ve already mentioned the design principle concern, so let’s talk about why it should not be provided as part of the standard library. First, I heard the standard library is intentionally being left alone for 3.0, so there’s that.

But more importantly, there hasn’t been enough exploration on the design of a library like this. Haskell’s newtype isn’t a good match because Haskell doesn’t have inheritance, it doesn’t have to support the Java type hierarchy, and it isn’t object oriented.

The example implementation I gave is rather useless, because it has no methods other than extracting the value again, but simply inline/forwarding methods won’t work either for various reasons I remarked on before. Some existing methods would break the opacity and need to be excluded, some need modified type signatures, and some take primitive/string parameters that should not be changed to the opaque types, or have return values be likewise.

This is something that needs to be explored at length, tried out, and revised based on real world usage. For that matter, there might be a call for different designs to satisfy different needs, in which case it might be best not to have it in the standard library at all. But even if that’s the right place for it, it would probably be better to wait on valhalla and, at any rate, now is the wrong time.

It would be great to start a discussion or a project implementing such a library for Scala 3. That would be more productive, in my opinion.

[LPTK]

@gvolpe

To avoid boxing, we need to create multiple traits for every primitive type.

Again, this is not needed, provided we can use the @specialized annotation (it already works in Scala 2, not sure about Scala 3, but it should be ported in any case).

[gvolpe]

But more importantly, there hasn’t been enough exploration on the design of a library like this. Haskell’s newtype isn’t a good match because Haskell doesn’t have inheritance, it doesn’t have to support the Java type hierarchy, and it isn’t object oriented.

Newtypes are not about inheritance, they are plain zero-cost wrappers for extra type-safety at compile time. The runtime representation is the underlying datatype, which can be treated differently if it’s a primitive (i.e. no boxing). This is basically opaque types with extra steps, which should be better handled by the language itself. It doesn’t have anything to do with inheritance, or Java type hierarchy.

Why burden the language with something that can be implemented as a library? That goes against Scala’s design principles.

The above response should also answer this question.

Yes, so what? The user doesn’t have to deal with it, and even the code for the traits can be done mechanically as generated source – except for method redirection which needs to be curated. More on this later.

I’ll say it again, I’m not opposed to have this feature as a library. After all, I use newtypes library in Scala 2 but I was hoping Scala 3 could replace this library with the opaque types feature. I only realized when I started trying to migrate some projects that this is not the case.

simply inline/forwarding methods won’t work either for various reasons I remarked on before. Some existing methods would break the opacity and need to be excluded, some need modified type signatures, and some take primitive/string parameters that should not be changed to the opaque types, or have return values be likewise.

Exactly. This is why I believe this feature would be much better if implemented in the language itself.

It would be great to start a discussion or a project implementing such a library for Scala 3. That would be more productive, in my opinion.

That’s a different discussion. One I was hoping it would not be necessary but I guess I was wrong…

[gvolpe]

Again, this is not needed , provided we can use the @specialized annotation (it already works in Scala 2, not sure about Scala 3, but it should be ported in any case).

I would love to see this working in Scala 3.

[olafurpg]

@gvolpe

workarounds that involve quite some boilerplate

The boilerplate is limited to the library and does not leak to the user (when combined with new powerful inlining features). It’s clear that people have different preferences for the newtype encoding so the current design makes a lot of sense IMO since it’s flexible and caters to different needs.

[Jasper-M]

@LPTK

Again, this is not needed , provided we can use the @specialized annotation (it already works in Scala 2, not sure about Scala 3, but it should be ported in any case).

AFAIU there’s no intention to port @specialized to Scala 3. Correct me if I’m wrong.

[LPTK]

Damn. Is there any public documentation/explanation about this decision?

[sideeffffect]

Btw, the companion objects of opaque types are part of given (aka implicit) scope, so something like this works:

trait SemiGroup[T]:
   extension (x: T) def combine (y: T): T

object A:
  opaque type EvenInt = Int
  object EvenInt:
    // EvenInt forms a SemiGroup with addition
    given SemiGroup[EvenInt] with
      extension (x: EvenInt) def combine (y: EvenInt): EvenInt = x + y

object B:
  def instance = summon[SemiGroup[A.EvenInt]] // works

Scastie - An interactive playground for Scala

That makes me really happy

[dcsobral]

@gvolpe

But more importantly, there hasn’t been enough exploration on the design of a library like this. Haskell’s newtype isn’t a good match because Haskell doesn’t have inheritance, it doesn’t have to support the Java type hierarchy, and it isn’t object oriented.

Newtypes are not about inheritance

You completely misunderstood my point, which is that they are clearly defined in Haskell because it does not have to contend with a type system with inheritance and an object oriented design.

For example, if you “newtype” String, what is the type of its toString() method? Can you answer that without any reference to the type hierarchy?

You are basically saying it can be easily done without actually going to the trouble of explaining how it should be done. Give the exact rules and I’ll give examples that break them.

This is basically opaque types with extra steps, which should be better handled by the language itself. It doesn’t have anything to do with inheritance, or Java type hierarchy.

Why is it better handled by the language? It’s funny that you mention below that “the above response should also answer this question” but you actually didn’t answer anything. You say it is better handled by the language without any arguments to support it. I gave arguments to support the position that it is not handled better by the language.

simply inline/forwarding methods won’t work either for various reasons I remarked on before. Some existing methods would break the opacity and need to be excluded, some need modified type signatures, and some take primitive/string parameters that should not be changed to the opaque types, or have return values be likewise.

Exactly. This is why I believe this feature would be much better if implemented in the language itself.

You believe it needs to be implemented in the language itself because the classes from external libraries are full of special cases? That makes no sense whatsoever.

[Jasper-M]

@LPTK

Damn. Is there any public documentation/explanation about this decision?

Don’t know of any (semi-)official reference, but I think I’ve seen it mentioned in passing in other discussions or issues. As a data point, there’s now a compiler phase specifically dedicated to the specialization of functions only (Revive function specialization by liufengyun · Pull Request #10452 · lampepfl/dotty · GitHub).

There’s no mention of this anywhere in the “dropped features” or “changed features” sections of the documentation. Would be nice to have someone from the dotty team chime in on this.

[gvolpe]

simply inline/forwarding methods won’t work either for various reasons

You mentioned this before, from which I understood there are a lot of edge cases to handle to guarantee unboxing. This kind of stuff is usually really hard to deal with at the library level and would be better done in the language itself.

Anyway, this discussion is going south and that was not the intention of the post. If there are no intentions to support newtypes in Scala 3 that’s okay. The idea was to communicate that there are actual unsolved problems and whether there’s something we could do.

[odersky]

Scala 3.0 only has function specialization. Full specialization was outside the scope of what we could do for this release. That’s not a statement that we want to get rid of specialization. It’s an acknowledgment that we lacked so far the resources to do it properly. Specialization is really hard, so it won’t be easy to port. If you want another example, just look at how long Valhalla is taking. We need the right people to do it and we need to have them funded so that they have the necessary time to devote on it.

Also, we might want to rethink the approach to specialization. The current one is not the only possibility. All of this will take time.

Specialization is an (important) implementation detail. So it might well land later in the 3 cycle.

[gvolpe]

I started something out from what I learned from this and the other post: https://github.com/gvolpe/newtypes

Any contributions more than welcome. If this proves to be useful, I can publish it to Maven Central under ProfunKtor.

[curoli]

Does this mean that in Scala 3.0, things like Option[Int] or Seq[Int] will always contain boxed Ints?

[sjrd]

@curoli

Does this mean that in Scala 3.0, things like Option[Int] or Seq[Int] will always contain boxed Int s?

Option[Int] and Seq[Int]s have always contained boxed Ints. The collections are not specialized in Scala 2 any more than in Scala 3. The only things that are specialized in the std lib are function types up to 2 parameters, and tuples up to 2 (or 3?) elements.

[nafg]

More generally, what are the ramifications of this regression?

[sjrd]

In code that only uses @specialized from (transitively) the std lib, probably none.

For libraries whose designs heavily rely on @specialized to provide decent performance … it’s probably bad.

For Scala.js code … significant code size improvement, and probably an improvement in performance as well.

[amsayk]

Whatever happened to dotty-linker ?