Improve opaque types

As many people have expressed in the other thread, the cost of writing one extra line for a wrapper function, and one extra line for a unwrapping function is lower than the cost of having to suppress these functions when they are not wanted.

I would love if the apply / unapply would be generated by default for all opaque types. This will cover most of the cases but only if we are allowed to modify their visibility (ie. private or public). E.g. sometimes people want to disallow creating invalid instances and want to run some validation first, returning Option or Either. In such cases, they could make the default apply private and create a function acting as a smart constructor.

I personally don’t do this since I believe it conflates validation with the idea of newtypes. This is the reason why I use and endorse the idea of Newtypes + Refined. The former to differentiate similar types (e.g. String) and the latter to define the validation rules, both at compile time and runtime.

I went through most of the responses in the other thread and I believe some disagree with having a default apply / unapply. If that’s the case, would it be possible to somehow tell Scala we want these wrapping / unwrapping functions to be automatically generated? Maybe a compiler flag? An import?

This would absolutely be possible! I actually proposed it here:

That would be awesome! Has there been any progress on this proposal? I like it, even if it’s just sugar, it greatly improves the user experience.

1 Like

A compiler flag would be problematic for many reasons. If possible, I think a compiler plugin would be the way to go.

My personal opinion is that the current implementation does what it does well, and fleshing it out with sugar too early might do more harm than good. The derives thing is so small that it could probably be added quite easily. But more invasive sugar should wait until opaque types have matured and at least until 3.1 when Matchable warnings are turned on.

1 Like

A compiler flag would be problematic for many reasons.

Thanks, I can understand how a compiler flag might be problematic.

My personal opinion is that the current implementation does what it does well, and fleshing it out with sugar too early might do more harm than good.

I understand. Although, at the same time, I would love for Scala 3 to provide a better alternative to what we can already do in Scala 2. Newtypes have become an essential tool both at work and in the open source projects I maintain. Whether that means further enhancing opaque types or introduce proper support for newtypes, I hope Scala 3 can deliver.

To give a bit more of context, here is an ongoing effort to cross-compile a small library I help maintaining: https://github.com/cr-org/neutron/pull/144/files

I was hoping cross-compiling would be easier but when one uses a few compiler plugins + libraries such as this project does, it becomes a tedious task.

1 Like

Note you can avoid the erasure issue by defining the extensions in the companion objects of the opaque types:

object data:
  opaque type Bar = String
  object Bar:
    def apply(v: String): Bar = v
    extension (x: Bar) def value: String = x
  end Bar

  opaque type Foo = String
  object Foo:
    def apply(v: String): Foo = v
    extension (x: Foo) def value: String = x
  end Foo
end data

An added benefit is that the extensions are then always in scope – no need to import data members.

6 Likes

Sweet, thanks!

2 Likes

You can abstract this stuff away pretty well:

scala> trait NewType[A]:
     |   opaque type Type = A
     |   def apply(a: A): Type = a
     |   extension (a: Type) def value: A = a
     | end NewType
// defined trait NewType

scala> object Bar extends NewType[String]
// defined object Bar

scala> val bar = Bar("foo")                                                
val bar: Bar.Type = foo

scala> bar.value
val res6: String = foo

Of course it’s not 100% the same API and has some drawbacks as discussed in the other thread (mainly with boxing of primitives).

2 Likes

In fact, extensions are in scope no matter whether they are defined alongside the opaque type or in a companion object.

4 Likes

Of course it’s not 100% the same API and has some drawbacks as discussed in the other thread (mainly with boxing of primitives).

Thanks, I saw this in the other thread.

The idea of newtypes / opaque types is that they should never box primitives (aka zero-cost wrappers). If we lose this guarantee, I think we would be better off using case classes instead.

Moreover, this still feels like a hack. I would expect a better solution in a completely new language such as Scala 3. Hopefully, we’re still on time to agree on what a “better API” could be :pray:

The boxing of primitives only happens if you use the generics. It can easily be avoided with, for example,

trait NewIntType:
  opaque type Type = Int
  def apply(a: Int): Type = a
  extension (a: Type) def value: Int = a
end NewIntType
1 Like

I just want to chime in and say opaque types are probably the feature I’m most looking forward to using (I really wish they could have been in 2.13). Also, I really like that the default does not make an API that builds the type from the internal representation.

If we look back at the SIP, that was quite intentional and I like that choice.

I hope opaque types stay as they are in scala 3.

9 Likes

The other thread also mentions that using @specialized should resolve the erasure problem. The thing is, I don’t know what’s the status of support for @specialized for class type parameters in Scala 3. Does anyone know if it’s supported, or will be in the future?

1 Like

I gathered by the different responses that the community is a bit split on this topic so I’ll try to reformulate what I meant with starting this post because I noticed the title might sound demanding, which was not the intention (tried to edited it but it seems I can not?).

Some people are happy with opaque types as they are. Some others like myself want something similar to newtypes. It would be great if you could make both sides happy.

I would love if Scala 3 would provide a native way to define proper newtypes, either in a future version or in the planned 3.0.0 release (obviously the sooner the better).

Why is it so important to me? Because I make use of the scala-newtype library absolutely everywhere, to the point that it has become a blocker when it comes to migrating over to Scala 3.

I’ve promoted the idea of newtypes + refinement types extensively. There’s also a section about them in my book Practical FP in Scala. From what I gathered, a lot of folks have started using both the scala-newtype and refined libraries as well, so I believe these are the users that would benefit from having newtypes support in Scala 3.

Can we have both?

This may sound silly but, is there any chance we could have both opaque types and newtypes (probably built on top of opaque types) in Scala 3? I would love to see this feature being part of the language but if that’s a no-op, I would be fine having it as a library, if that’s possible to build.

Has this been discussed before?

At the moment, cross-building my projects with Scala 3 is really painful due to the use of newtypes and some other plugins like context-applied.

Appreciate your consideration :pray:

Sincerely,
Gabriel.

4 Likes

The replies Improve opaque types and Improve opaque types address your concerns about boilerplate for declaring newtypes and avoiding boxing of primitives. Can you elaborate what is your remaining issue with the current design of opaque types?

The replies Improve opaque types and Improve opaque types address your concerns about boilerplate for declaring newtypes and avoiding boxing of primitives. Can you elaborate what is your remaining issue with the current design of opaque types?

These approaches partially address my concerns. I see them as workarounds that involve quite some boilerplate. I would love to see this feature being baked in into the language, if that’s possible.

To avoid boxing, we need to create multiple traits for every primitive type.

trait NewIntType:
  opaque type Type = Int
  def apply(a: Int): Type = a
  extension (a: Type) def value: Int = a
end NewIntType

trait NewStringType:
  opaque type Type = Int
  def apply(a: Int): Type = a
  extension (a: Type) def value: Int = a
end NewIntType

trait NewBooleanType:
  opaque type Type = Int
  def apply(a: Int): Type = a
  extension (a: Type) def value: Int = a
end NewIntType

// and so on...

object data:
  object ZeroCost extends NewIntType
  type ZeroCost = ZeroCost.Type

  object Bar extends NewBooleanType
  type Bar = Bar.Type

  object Foo extends NewStringType
  type Foo = Foo.Type
end data

Surely this could be made into a library and we can all move on. However, I think it’s a pity we have a complete new language without this feature so the idea of sparking this conversation once again was to see whether it was possible to ship proper newtypes as part of the language.

1 Like

While true, a lot of this could be hidden behind a simple interface in a library.

scala> class NewType[A]:
     |   opaque type Type = A
     |   def apply(a: A): Type = a
     |   extension (a: Type) def value: A = a
     | end NewType
// defined class NewType

scala> class NewIntType extends NewType[Int]:
     |   override opaque type Type = Int
     |   override def apply(a: Int): Type = a
     |   extension (a: Type) override def value: Int = a
     | end NewIntType
// defined class NewIntType

scala> import scala.compiletime.erasedValue

scala> transparent inline def newType[A] = inline erasedValue[A] match {
     |   case _: Int => new NewIntType
     |   case _ => new NewType[A]
     | }
def newType[A] => NewType[? >: Int & A <: Int | A]
scala> val Foo = newType[Int]; type Foo = Foo.Type
val Foo: NewIntType = NewIntType@346d8002
// defined alias type Foo = Foo.Type

scala> val Bar = newType[String]; type Bar = Bar.Type
val Bar: NewType[String] = NewType@593d5f39
// defined alias type Bar = Bar.Type

Notice you have one entrypoint to make a newtype “module”, newType[A], and when A is an AnyVal you statically get the specialized instance, otherwise the generic variant for types that don’t suffer from boxing.

Taking it even further would probably require something like macro annotations…

5 Likes

That’s not bad! Could definitely use something like this if it’s not built into the language :+1:

Why burden the language with something that can be implemented as a library? That goes against Scala’s design principles.

Yes, so what? The user doesn’t have to deal with it, and even the code for the traits can be done mechanically as generated source – except for method redirection which needs to be curated. More on this later.

Strings are not primitive. They are just used like one. :slight_smile:

I’ve already mentioned the design principle concern, so let’s talk about why it should not be provided as part of the standard library. First, I heard the standard library is intentionally being left alone for 3.0, so there’s that.

But more importantly, there hasn’t been enough exploration on the design of a library like this. Haskell’s newtype isn’t a good match because Haskell doesn’t have inheritance, it doesn’t have to support the Java type hierarchy, and it isn’t object oriented.

The example implementation I gave is rather useless, because it has no methods other than extracting the value again, but simply inline/forwarding methods won’t work either for various reasons I remarked on before. Some existing methods would break the opacity and need to be excluded, some need modified type signatures, and some take primitive/string parameters that should not be changed to the opaque types, or have return values be likewise.

This is something that needs to be explored at length, tried out, and revised based on real world usage. For that matter, there might be a call for different designs to satisfy different needs, in which case it might be best not to have it in the standard library at all. But even if that’s the right place for it, it would probably be better to wait on valhalla and, at any rate, now is the wrong time.

It would be great to start a discussion or a project implementing such a library for Scala 3. That would be more productive, in my opinion.

1 Like

Again, this is not needed, provided we can use the @specialized annotation (it already works in Scala 2, not sure about Scala 3, but it should be ported in any case).

1 Like

But more importantly, there hasn’t been enough exploration on the design of a library like this. Haskell’s newtype isn’t a good match because Haskell doesn’t have inheritance, it doesn’t have to support the Java type hierarchy, and it isn’t object oriented.

Newtypes are not about inheritance, they are plain zero-cost wrappers for extra type-safety at compile time. The runtime representation is the underlying datatype, which can be treated differently if it’s a primitive (i.e. no boxing). This is basically opaque types with extra steps, which should be better handled by the language itself. It doesn’t have anything to do with inheritance, or Java type hierarchy.

Why burden the language with something that can be implemented as a library? That goes against Scala’s design principles.

The above response should also answer this question.

Yes, so what? The user doesn’t have to deal with it, and even the code for the traits can be done mechanically as generated source – except for method redirection which needs to be curated. More on this later.

I’ll say it again, I’m not opposed to have this feature as a library. After all, I use newtypes library in Scala 2 but I was hoping Scala 3 could replace this library with the opaque types feature. I only realized when I started trying to migrate some projects that this is not the case.

simply inline/forwarding methods won’t work either for various reasons I remarked on before. Some existing methods would break the opacity and need to be excluded, some need modified type signatures, and some take primitive/string parameters that should not be changed to the opaque types, or have return values be likewise.

Exactly. This is why I believe this feature would be much better if implemented in the language itself.

It would be great to start a discussion or a project implementing such a library for Scala 3. That would be more productive, in my opinion.

That’s a different discussion. One I was hoping it would not be necessary but I guess I was wrong…

Again, this is not needed , provided we can use the @specialized annotation (it already works in Scala 2, not sure about Scala 3, but it should be ported in any case).

I would love to see this working in Scala 3.

1 Like