Pre-SIP: A New Type for Optionals and Error Handling (2)

Scala Improvement Process
21

I’m still ingesting this proposal, so I don’t have thoughts on it yet.

I had been working on a different proposal for the same problem, which I have now released (Pre-SIP: Optional parameters by symetry with varargs)

22

Well, you would then have to flip T? to ?T. Not impossible.

You also have the problem that reading left-to-right makes the left-hand thing seem like the main one. So apart from people who have gotten used to Either, pretty much everyone else expects “String or Exception” to mean “I desire a String, and know that if my desires are thwarted, it will be an Exception”. “Exception or String” is weird.

“I will climb that mountain or die trying” is not a statement that the favored branch is to die trying.

1 Like
23

its not implemented as an opaque type, but as a fake class with special erasure (like Tuple), the opaque type was just a handwavey way to present the idea

24

I’m not sure if this is the best possible representation for Valid.

Consider the implementation from Kyo (called PresentAbsent): kyo/kyo-data/shared/src/main/scala/kyo/Maybe.scala at fe609be30cf5baeee04873ce5314163d6d0c7d5e · getkyo/kyo · GitHub

So, instead of a case class Valid(elem: Valid | Null), you could have case class Valid(depth: Int) and avoid a bunch of nested objects/allocations.

25

this would be parametric for just boxing nulls, but (v ? e1) ? e2 needs to support Valid(Fail("oops"))

so that prompts the following probably terrible idea: two classes - one for Valid(Fail(x)) and one for runs of nested Valid(Valid(null)) - but then expanding at a deeply nested level is some pain maybe

2 Likes
26

I really like this proposal!

If it comes to the T ? E, we should mention similar Kotlin’s rich errors proposal. They restrict the set of possible Es like @sjrd said (but for other reasons)

27

Huh. The Kotlin proposal is quite neat, they just use union types. But add an error type that is not a subtype of Any, so they can know the union is disjunct, even with generics (as the generic type cannot be an error type). It seems null works the same in Kotlin, so T | Null is also just fine.

28

Oh, right, I skimmed the error handling part and didn’t notice that Valid was redefined :person_facepalming:

so that prompts the following probably terrible idea: two classes - one for Valid(Fail(x)) and one for runs of nested Valid(Valid(null)) - but then expanding at a deeply nested level is some pain maybe

I guess the shape is always going to be “a potentially empty chain of Valid followed by a null or an error”
I think you can also just do something like:

case class Valid(depth: Int, elem: Null | Fail[E]) (yes, I know I’m missing the E, but if the plan is to make this an Any, it doesn’t matter)

Such that

None                    -> null
Left(error)             -> Fail(error)
Some(None)              -> Valid(0, null)
Some(Some(None))        -> Valid(1, null)
Some(Some(Left(error))) -> Valid(1, Fail(error))

Although I guess this sounds a bit more like case class Nested[+E](depth: Int, error: Null | Fail[E]). Calling that elem feels a bit misleading, if it’s either nothing or an error :sweat_smile:.

2 Likes
29

What’s the point of avoiding the nesting? We have to follow O(1) pointers at every call site anyway. And the depth is rarely going to exceed 2 in practice, occasionally exceed 1.

It’s not without cost either: it complicates the logic for unnesting because we have to look at the depth in addition to the type.

I don’t think Kyo did this for the nesting aspect, but for the caching aspect. It wants to reuse the same instance of Some(None)/Ok(null) every time. My own UOption did that too. It’s easy to do if you don’t have the Fail paths to deal with. With the added Fail paths, I don’t think the benefits will outweigh the costs.

3 Likes
30

Yeah, I was thinking about reducing allocations, but now that I think a bit more about it, without caching it doesn’t really help (it might actually make things worse).

I agree with you, it’s probably overkill and might even make things slower, I was just thinking out loud.

2 Likes
31

I read their proposal. It’s much stronger than that. They put their error types in a completely separate type hierarchy. If you have a FooError error type, it is not a subtype of Any?.

That means: you can’t make a List<String | FooError>! Indeed, generic types cannot range over error types, so the type parameter T of List<T> remains upper-bounded by Any?.

In contrast, with the current proposal for Scala, we will be able to make a List[String ? FooError], since all types are still well-behaved, as part of the proper type hierachy rooted in Any.

6 Likes
32

I wonder how Iterator Pattern methods like traverse, sequence, etc would work (wrt left-hand type bias for F[_]):

val xs: List[String ? FooError] = List("foo", "bar", "baz")
val ys: List[String] ? FooError = xs.sequence

From my understanding, it would require ? to be some ‘proper’ type or type alias that we can provide Applicative instance with.

2 Likes
33

? is an actual, well-behaving type in the present proposal. So yes, you can define an Applicative for it.

4 Likes
34

Some more infos on this proposal:

Implementation

There is now a prototype implementation of the proposed scheme. Here is a list of things that needed to be added or changed:

  • Maybe types are represented internally as a trait.

    package scala.compiletime

@experimental
sealed trait Maybe[+T, +E] extends Any, Matchable:
  def isEmpty: Boolean
  def get: T

The trait is a only a compiletime artifact, since the erasure of a maybe type
is either the underlying result type or Object

The trait has members isEmpty and get, which makes it eligible as a
result type of unapply methods. Their implementations are special-cased in the pattern matcher.

  • The companion object of Maybe defines extension methods on maybe and result types:

    object Maybe {
  extension [A, E](x: Maybe[A, E])
    transparent inline def ? (using maybe.CanErr[E]): A = x match
      case Ok(y) => y
      case Err(e) => break(Err(e))

    def withErr[E1](e: E1): A ? E1 = x match
      case Ok(y) => Ok(y)
      case Err(_) => Err(e)

    def mapErr[E1](f: E => E1): A ? E1 = x match
      case Ok(y) => Ok(y)
      case Err(e) => Err(f(e))

    def map[B](f: A => B): A ? E = x match
      case Ok(y) => Ok(f(y))
      case Err(e) => Err(e)

    def flatMap[B](f: A => B ? E): B ? E = x match
      case Ok(y) => f(y)
      case Err(e) => Err(e)

    ...

  • The parser now understands postfix ? for both types and terms.

  • The printer prints instances of Maybe using the source-level ? form.

  • TypeComparer handles three new cases

    • T <: T? if T is disjoint from Null and Maybe[?, ?]
    • T? <: T | Null if T disjoint from Null and Maybe[?, ?]
    • Null <: T ? Unit

  • TypeErasure erases T ? E to the erasure of T if the following
    conditions are met:

    • T is a reference type disjoint from Null
    • E is Unit or Nothing

    Otherwise T ? E is erased to Object.

Prerequisite

The proposed extensions need explicit nulls to be enabled globally.

5 Likes
35

This should be Null <: T ? Unit, no ?

1 Like
36

Yes, indeed. I fixed it in the comment.

37

On this note, I see that the prototype only implements WithFilter for Maybe[A, Unit].
Although Try also has a WithFilter implementation, so maybe there should also be one for Maybe[A, Throwable]?

(I won’t defend it that much, since I always found filtering on Try/Future a bit awkward to use, just pointing out the inconsistency).

38

What I’m worried about with this proposal is that this will lead to more ecosystem fragmentation:

There are now two different good ways to make an optional T:

  • Option[T]
  • T?

Each with different strengths and weaknesses
Even if in a vacuum T? dominates Option[T] in all cases (which it looks like it might), Option[T] still has the benefits of being “old”: more familiar, more widely taught, and present and supported in libraries.

This seems like a very big deal to me!, and I’m a bit surprised by the fact that it wasn’t brought up before(?)

The alternate proposals of SLC: `Conversion[A, Option[A]]` and Pre-SIP: Optional parameters by symetry with varargs don’t suffer from this downside, they both make Option[T] stronger, making it the best option in more cases
This leads to less choices for our users to make, which we should strive for

Furthermore, if the syntactic clarity is what we want, we can adopt one of the above and:

type T? = Option[T]
type T ? E = Either[E, T]

(meta: it’s also not quite clear if T ? E is part of this Pre-SIP, or only T?)

As for the unboxing aspect, it’s super cool (honest), but is this something we really need ?
I remember seeing people saying the performance impact of boxing Option was minimal, is it worth splitting the ecosystem for ?

In conclusion, I really believe if we were building Scala from scratch, this proposal would have been perfect: It’s clean, efficient, reduces boilerplate, and feels very Scala-y.
But Scala is already here, and it already has Option, which is already 99% good, and it even still has available ways to improve !

4 Likes
39

At the application level, that doesn’t bother me. It’s a choice to make, but that’s pretty routine for Scala-centric projects – by the nature of the ecosystem, any serious project makes choices about idiom all the time.

The really serious fragmentation concern is specifically for libraries. Option is present in a bazillion signatures across the ecosystem. I suspect that some libraries would choose to evolve towards Maybe, and some wouldn’t. That means that consuming applications are likely to wind up with a grab-bag of signatures to play with, often needing to mix Option functions with Maybe ones frequently, at least in the early years.

I think I’m prepared to live with that, but only if Option and Maybe are interoperable in a fairly seamless way, so that application code doesn’t have to perform a lot of ceremony to work with both. I think that implies baking automatic conversions in, in both directions. There would probably be some efficiency hit in that; that doesn’t bother my use cases, but I suspect it would be more problematic for some folks.

But anyway, that’s the use case I think we need to think through really carefully. I quite like the Maybe proposal, but we need to understand what a world that is half-Option, half-Maybe would look like, and what the implications are.

10 Likes
40

This is especially true given that at least for now, Maybe requires explicit nulls (and it’s hard to see how the types could work otherwise). The argument would have to be that Maybe makes explicit nulls so easy (and efficient) to use that nobody would want to use anything else.

2 Likes