Improve opaque types

Hi folks,

I’m a heavy user of scala-newtype, which allows you to have zero-cost wrappers in Scala 2. For those unfamiliar with it, this is how one can define a newtype:

object data {
  @newtype case class Foo(value: String)
}

This library is built with macros so it is not available in Scala 3. However, that should not be a problem since we have opaque types.

Current status

After trying to migrate a library that uses newtype to cross-compile to Scala 3, I found that opaque types are bit cumbersome in comparison. Here’s the amount of code I need to write to get more or less something equivalent in Scala 3.

object data:
  opaque type Foo = String
  object Foo:
    def apply(v: String): Foo = v
  end Foo

  extension (x: Foo) {
    def value: String = x
  }
end data

But it does not end here. Another bigger issue pops up when we introduce more newtypes and want to access the underlying primitive - String in this case.

object data:
  opaque type Bar = String
  object Bar:
    def apply(v: String): Bar = v
  end Bar
    
  opaque type Foo = String
  object Foo:
    def apply(v: String): Foo = v
  end Foo

  extension (x: Bar) {
    def value: String = x
  }

  extension (x: Foo) {
    def value: String = x
  }
end data

This fails to compile with the following error.

Double definition:
def value(x: data.Bar): String in object data at line 13 and
def value(x: data.Foo): String in object data at line 17
have the same type after erasure.

Consider adding a @targetName annotation to one of the conflicting definitions
for disambiguation.

If we follow the compiler’s suggestion, we can get it working by adding a @targetName to each extension method.

extension (x: Bar) {
  @targetName("value_Bar")
  def value: String = x
}

extension (x: Foo) {
  @targetName("value_Foo")
  def value: String = x
}

So this works, albeit involving quite a lot of boilerplate.

Summarizing, it takes as much as 18 lines of Scala 3 code and only 4 lines of Scala 2 code (using the scala-newtype library) to define two newtypes.

Proposal

There are two main things I would like to propose, though, I’m not sure how feasible they are since I’m not very experienced on the Scala compiler side.

Auto-generate wrapping / unwrapping functions

For comparison, here’s all the necessary code to define a newtype in Haskell:

newtype Foo = Foo String

A function String -> Foo - i.e. apply in Scala - will be automatically generated for us. It is also possible to define a function that goes the other way around: Foo -> String, aka the unwrapping function, if we define it as follows.

newtype Foo = Foo { getFoo: String }

Creating both the wrapping and unwrapping functions is something that the scala-newtype does by default in Scala 2.

@newtype case class Foo(getFoo: String)

In my opinion, this greatly improves the user experience, and it would be great if we can get this feature with opaque types in Scala 3.

Generic extension methods

This is more a nice-to-have and would not probably be necessary for the main use case (unwrapping) if we get opaque types with unwrapping functions by default. So, assuming that unwrapping functions do not exist, would it be possible to define a generic extension method for any opaque type?

e.g. a generic unwrap function (please, assume this compiles, I don’t know exactly what the equivalent of asking for an implicit ev: OpaqueTypeOf[A, B] is in Scala 3).

extension[A] (x: A) {
  def value[B](given ev: OpaqueTypeOf[A, B]): B = x
}

Moreover, it would be useful to define generic typeclass instances for opaque types. Though, I get the feeling this is probably possible in Scala 3 using typeclass derivation?

In Scala 2, we can do the following with scala-newtype.

@newtype case class Foo(value: String)
object Foo {
  implicit val eq: Eq[Foo] = deriving
}

If there’s an Eq[String] in scope, the Eq[Foo] can be derived. There’s also an oficially non-recommended typeclass named Coercible, which looks similar to the OpaqueTypeOf[A, B] I made up above. By exploiting this typeclass, we can derive typeclass instances generically for any newtype. I do - and recommend - this all the time to avoid writing boilerplate. Here’s an example.

import cats.Eq
import cats.implicits._
import io.estatico.newtype.Coercible

object eq {
  implicit def coercibleEq[A: Coercible[B, *], B: Eq]: Eq[A] =
    Eq[B].contramap[A](_.asInstanceOf[B])
}

However, it would be much better if we could use typeclass derivation for opaque types. Assuming there are instances of Eq[String] and Show[String] in scope, would something like this be possible?

opaque type Foo = String derives (Eq, Show)

That’s all I’ve got for now. Please forgive my ignorance if something does not make sense, I’m a newbie to Scala 3 :slight_smile:

I would love to hear your thoughts.

23 Likes

A thousand times yes :star_struck:

3 Likes

@FelixHargreaves I’m just seeing your post: Synthesize constructor for opaque types, shame on me I didn’t see it before.

Anyway, hopefully this post sparks the discussion once again. I’ll give it a thorough read.

1 Like

Was just about to link it :slight_smile: the derives thing is my preferred route

This can be written a bit more concisely:

object data:
  opaque type Foo = String
  def Foo(v: String): Foo = v
  extension (x: Foo) def value: String = x
end data

As many people have expressed in the other thread, the cost of writing one extra line for a wrapper function, and one extra line for a unwrapping function is lower than the cost of having to suppress these functions when they are not wanted.

Opaque types is an advanced feature, and while forcing users to write wrappers and unwrappers may be perceived as boilerplate, it’s actually more user friendly since it forces users to understand what’s going on, and is generally a better fit with the current semantics.

This would absolutely be possible! I actually proposed it here:

As I already noted in that post however, this would just be convenient sugar. You can already derive instances like this:

object data:
  opaque type Foo = String
  def Foo(v: String): Foo = v
  extension (x: Foo) def value: String = x

  given (using ev: Eq[String]): Eq[Foo] = ev
  given (using ev: Show[String]): Show[Foo] = ev
5 Likes

Second the derivation, as a newbie I ran into issues like these: https://github.com/lampepfl/dotty/issues/10947

I originally tried reaching out for derives when I figured it wasn’t supported. Derivation would make it bit more simpler to use opaque types.

As many people have expressed in the other thread, the cost of writing one extra line for a wrapper function, and one extra line for a unwrapping function is lower than the cost of having to suppress these functions when they are not wanted.

I would love if the apply / unapply would be generated by default for all opaque types. This will cover most of the cases but only if we are allowed to modify their visibility (ie. private or public). E.g. sometimes people want to disallow creating invalid instances and want to run some validation first, returning Option or Either. In such cases, they could make the default apply private and create a function acting as a smart constructor.

I personally don’t do this since I believe it conflates validation with the idea of newtypes. This is the reason why I use and endorse the idea of Newtypes + Refined. The former to differentiate similar types (e.g. String) and the latter to define the validation rules, both at compile time and runtime.

I went through most of the responses in the other thread and I believe some disagree with having a default apply / unapply. If that’s the case, would it be possible to somehow tell Scala we want these wrapping / unwrapping functions to be automatically generated? Maybe a compiler flag? An import?

This would absolutely be possible! I actually proposed it here:

That would be awesome! Has there been any progress on this proposal? I like it, even if it’s just sugar, it greatly improves the user experience.

1 Like

A compiler flag would be problematic for many reasons. If possible, I think a compiler plugin would be the way to go.

My personal opinion is that the current implementation does what it does well, and fleshing it out with sugar too early might do more harm than good. The derives thing is so small that it could probably be added quite easily. But more invasive sugar should wait until opaque types have matured and at least until 3.1 when Matchable warnings are turned on.

1 Like

A compiler flag would be problematic for many reasons.

Thanks, I can understand how a compiler flag might be problematic.

My personal opinion is that the current implementation does what it does well, and fleshing it out with sugar too early might do more harm than good.

I understand. Although, at the same time, I would love for Scala 3 to provide a better alternative to what we can already do in Scala 2. Newtypes have become an essential tool both at work and in the open source projects I maintain. Whether that means further enhancing opaque types or introduce proper support for newtypes, I hope Scala 3 can deliver.

To give a bit more of context, here is an ongoing effort to cross-compile a small library I help maintaining: https://github.com/cr-org/neutron/pull/144/files

I was hoping cross-compiling would be easier but when one uses a few compiler plugins + libraries such as this project does, it becomes a tedious task.

1 Like

Note you can avoid the erasure issue by defining the extensions in the companion objects of the opaque types:

object data:
  opaque type Bar = String
  object Bar:
    def apply(v: String): Bar = v
    extension (x: Bar) def value: String = x
  end Bar

  opaque type Foo = String
  object Foo:
    def apply(v: String): Foo = v
    extension (x: Foo) def value: String = x
  end Foo
end data

An added benefit is that the extensions are then always in scope – no need to import data members.

5 Likes

Sweet, thanks!

2 Likes

You can abstract this stuff away pretty well:

scala> trait NewType[A]:
     |   opaque type Type = A
     |   def apply(a: A): Type = a
     |   extension (a: Type) def value: A = a
     | end NewType
// defined trait NewType

scala> object Bar extends NewType[String]
// defined object Bar

scala> val bar = Bar("foo")                                                
val bar: Bar.Type = foo

scala> bar.value
val res6: String = foo

Of course it’s not 100% the same API and has some drawbacks as discussed in the other thread (mainly with boxing of primitives).

2 Likes

In fact, extensions are in scope no matter whether they are defined alongside the opaque type or in a companion object.

3 Likes

Of course it’s not 100% the same API and has some drawbacks as discussed in the other thread (mainly with boxing of primitives).

Thanks, I saw this in the other thread.

The idea of newtypes / opaque types is that they should never box primitives (aka zero-cost wrappers). If we lose this guarantee, I think we would be better off using case classes instead.

Moreover, this still feels like a hack. I would expect a better solution in a completely new language such as Scala 3. Hopefully, we’re still on time to agree on what a “better API” could be :pray:

The boxing of primitives only happens if you use the generics. It can easily be avoided with, for example,

trait NewIntType:
  opaque type Type = Int
  def apply(a: Int): Type = a
  extension (a: Type) def value: Int = a
end NewIntType
1 Like

I just want to chime in and say opaque types are probably the feature I’m most looking forward to using (I really wish they could have been in 2.13). Also, I really like that the default does not make an API that builds the type from the internal representation.

If we look back at the SIP, that was quite intentional and I like that choice.

I hope opaque types stay as they are in scala 3.

9 Likes

The other thread also mentions that using @specialized should resolve the erasure problem. The thing is, I don’t know what’s the status of support for @specialized for class type parameters in Scala 3. Does anyone know if it’s supported, or will be in the future?

1 Like

I gathered by the different responses that the community is a bit split on this topic so I’ll try to reformulate what I meant with starting this post because I noticed the title might sound demanding, which was not the intention (tried to edited it but it seems I can not?).

Some people are happy with opaque types as they are. Some others like myself want something similar to newtypes. It would be great if you could make both sides happy.

I would love if Scala 3 would provide a native way to define proper newtypes, either in a future version or in the planned 3.0.0 release (obviously the sooner the better).

Why is it so important to me? Because I make use of the scala-newtype library absolutely everywhere, to the point that it has become a blocker when it comes to migrating over to Scala 3.

I’ve promoted the idea of newtypes + refinement types extensively. There’s also a section about them in my book Practical FP in Scala. From what I gathered, a lot of folks have started using both the scala-newtype and refined libraries as well, so I believe these are the users that would benefit from having newtypes support in Scala 3.

Can we have both?

This may sound silly but, is there any chance we could have both opaque types and newtypes (probably built on top of opaque types) in Scala 3? I would love to see this feature being part of the language but if that’s a no-op, I would be fine having it as a library, if that’s possible to build.

Has this been discussed before?

At the moment, cross-building my projects with Scala 3 is really painful due to the use of newtypes and some other plugins like context-applied.

Appreciate your consideration :pray:

Sincerely,
Gabriel.

4 Likes

The replies Improve opaque types and Improve opaque types address your concerns about boilerplate for declaring newtypes and avoiding boxing of primitives. Can you elaborate what is your remaining issue with the current design of opaque types?

The replies Improve opaque types and Improve opaque types address your concerns about boilerplate for declaring newtypes and avoiding boxing of primitives. Can you elaborate what is your remaining issue with the current design of opaque types?

These approaches partially address my concerns. I see them as workarounds that involve quite some boilerplate. I would love to see this feature being baked in into the language, if that’s possible.

To avoid boxing, we need to create multiple traits for every primitive type.

trait NewIntType:
  opaque type Type = Int
  def apply(a: Int): Type = a
  extension (a: Type) def value: Int = a
end NewIntType

trait NewStringType:
  opaque type Type = Int
  def apply(a: Int): Type = a
  extension (a: Type) def value: Int = a
end NewIntType

trait NewBooleanType:
  opaque type Type = Int
  def apply(a: Int): Type = a
  extension (a: Type) def value: Int = a
end NewIntType

// and so on...

object data:
  object ZeroCost extends NewIntType
  type ZeroCost = ZeroCost.Type

  object Bar extends NewBooleanType
  type Bar = Bar.Type

  object Foo extends NewStringType
  type Foo = Foo.Type
end data

Surely this could be made into a library and we can all move on. However, I think it’s a pity we have a complete new language without this feature so the idea of sparking this conversation once again was to see whether it was possible to ship proper newtypes as part of the language.

1 Like