Make Null a subclass of AnyVal under -Yexplicit-nulls

[wtetzner]

@Ichoran

Suppose you have Option[String | Null]. This can be inhabited by Some("eel") and Some(null) and None. All have different meanings.

But your default equivalent collapses Some(null) into Some(null).get, which is null. Same as None.

I seem to be doing a very poor job of explaining my idea. I’m trying to describe a world where Option[T] is the only type that you will see in Scala’s type system, and something like T | null would only exist as a runtime representation.

In the case where you have a runtime representation of Option[String | Null], the type system would see it as Option[Option[String]], and wouldn’t type check against String | null, and there for the conversion code I showed above wouldn’t be used. That specific code only runs when you are trying to write (either as a method argument or setting a variable) to a T | null. This is why I tried to emphasize the importance of tracking both the type-system type, and the runtime representation.

Some(None) is still distinguishable from None, because T | null is only equivalent to Option[T]. If you have a runtime representation of Option[T] | null, it still maps to Option[Option[T]] in the type system. If you try to write a runtime value represented as Option[T] | null to a runtime representation of say Option[T | null], it’s going to know that it needs an option, so it would convert null to None, and Some(t) to Some(Some(t)).

I hope that makes sense. Maybe I need some better examples. But the overall idea is that the Scala compiler uses Option[T] as its only type-level representation, but it can support different runtime representations. However, those runtime representations are meant to only exist at the edges of interop. E.g., if you subclass a Java class that has a field foo that can be a String or null, then even though it looks like foo: Option[String] to the type system, the compiler knows to keep the runtime representation as String | null. But when you read from that field, it will convert it to Option[T], and when you write to it, it will convert the Option[T] back to T | null. The conditional conversion would come into play when you don’t know the runtime representation of the value.

The only edge cases I can really think of are when you somehow just don’t know (maybe because of type erasure?) either the runtime representation of the destination or the source. But I’m not entirely sure when you could run into that problem.

[Ichoran]

@wtetzner

I’m trying to describe a world where Option[T] is the only type that you will see in Scala’s type system, and something like T | null would only exist as a runtime representation.

You’re explaining fine.

Consider. I have an Option[Option[Option[String]]].

In your encoding, how do I represent each of the following:

Some(Some(Some("eel")))
Some(Some(None))
Some(None)
None

These are exactly the types (up to the choice of string) that the type signature says must be independently inhabited.

Consider also the case where I have Option[AnyRef] and I put various things into it…including an Option[Option[String]]. Now what? I still want to be able to recover my Some(Some("eel")) and Some(None) and None from inside my Option[AnyRef].

[ragnar]

@Sporarum

To me it seems like there is broad consensus that “null should only live at the interop boundary”

Okay, I think after our private discussion, you made me consider the “null is just interop” case a bit more.

Lets start from the assumption that we only need null because there are some desirable things we cannot do without (calling Java methods, backwards compat, avoiding boxing, whatever). But we want to avoid it because it has weird edge cases.

But one can argue this for the subtypes of AnyVal as well. Short, Int, Long have limited ranges and overflow. Float and Double cannot represent decimal literals exactly and have weird Inf Nan -0 cases. Char is this super weird encoding of text characters that’s basically useless these days. Byte is this weird mix of “small number” and “just data”, its signed even though most cases don’t want that. Custom subtypes of AnyVal only exist to avoid boxing.

Maybe not an argument that null should be AnyVal, but I think a counterpoint to “null is weird so should not be a value”.

I think it‘s also maybe an argument why T <: AnyVal is not that useful/common, as it seems to just say “I want to deal with all the weird values”.

However a code search on Github told me, that its used in macro code or code that tries to serialize stuff, precisely to deal with all these special cases. I was wondering if it would be fine to add Null <: AnyVal but given that its not sealed, one cannot expect to be exhaustive anyways, so I guess its fine

@Sporarum

It even seems like there is a path we can follow to gradually shed doubts:

1. Do Proposal: Fixing null.methodOfAny() under explicit-nulls (without Null <: AnyVal): null is less of a footgun (not a footgun anymore ?) and we preserve the ability to exclude it from generics => no possible loss wrt interop

Just to say something boring: Yes! :–) And I think the other thread seems pretty void of arguments against, so I would just assume this going forward.

@Sporarum

• Make Null <: AnyVal and add NonNull, makes null even more normal, and we preserve same capability wrt interop
• Remove NonNull once we show that it is not useful anymore, either because it never was, or because we added feature(s) tailored to better suiting what it was useful for

I still don’t understand the issue with NonNull. I think sjrd said in the other thread that we don’t have NonNull today anyways, and I don’t see how the place of Null in the type hierarchy would interact with any “magic NonNull” implementation in the compiler.

@Sporarum

They make sense as steps, but note that 2 and 3 would most likely have to happen in the same release, otherwise we would have too many backward compat issues.

And to also make that clear, as I think that got lost a bit: I do agree with this conclusion! (even if 2+3 don’t really make sense to me as individual steps, but the end result would be the same: make the Any methods on null safe as step one, and move it to the values as the next step)

[wtetzner]

@Ichoran

Consider. I have an Option[Option[Option[String]]].

In your encoding, how do I represent each of the following:

Some(Some(Some("eel")))
Some(Some(None))
Some(None)
None

I’m not proposing a new encoding, I’m suggesting that for the option-like encoding that already lives in the wild (T | null), Scala maps it into a real scala.Option[T], and when writing a scala.Option[T] back out to a field or method that accepts T | null, we convert it back into T | null.

Obviously representing Option[T] as T | null doesn’t work, because it’s lossy. But representing T | null as Option[T] is lossless.

Here’s an example of what I’m trying to describe. Let’s say we have a Java class:

public class Foo {
    ...
    public String value; // Nullable field
    ...
}

To the Scala type checker, this would look like:

class Foo {
    ...
    val value: Option[String]
    ...
}

But when doing a write to that field, the code generator would emit something like this:

foo.value = switch (obj) {
    case scala.None -> null;
    case scala.Some s -> s.value;
};

And when reading the value, it would emit something like this:

String foo_value = switch foo.value {
    case null => scala.None
    default => scala.Some(foo.value)
};

So as a Scala programmer, you’re always working with a real, nestable scala.Option[T]. But T | null fields coming from the wild will still look like Option[T] to the programmer.

There are obviously performance considerations, but I think those mostly disappear with value classes. I think being able to distinguish between nullable and non-nullable at the byte code layer with Valhalla would also make that easier.

The biggest problems would probably be implementation complexity, and whether or not people actually want this behavior. Maybe it’s just not a good idea. But I wanted to make sure that the actual idea got across clearly. I’m not trying to ever represent Option[T] with a different runtime represenation, I’m trying to take things that are functionally a subset of Option[T], and make them look like Option[T] to the programmer.

[Sporarum]

@wtetzner

Here’s an example of what I’m trying to describe. Let’s say we have a Java class:

Thank you for the precise example !

Here is why this fails (sadly):

public class Foo<T> {
    ...
    public T value; // Nullable field
    ...
}

=>

class Foo[T] {
    ...
    val value: Option[T]
    ...
}

And foo = Foo[Option[T]] therefore foo.value: Option[Option[T]]

When doing a write to that field:

foo.value = switch (obj) {
    case scala.None -> null;
    case scala.Some s -> s.value;
    # Some(null) => null, same as None, so we lose information
};

And when reading the value:

String foo_value = switch foo.value {
    case null => scala.None # We can't tell if the null was a None or Some(None)
    default => scala.Some(foo.value)
};

So we get:

val a = Some(null)
foo.value = a
val b = foo.value

assert a == b // fails, a == Some(null), b == None

[wtetzner]

@Sporarum

foo.value = switch (obj) {
    case scala.None -> null;
    case scala.Some s -> s.value;
    # Some(null) => null, same as None, so we lose information
};

Ah OK, I see where you’re coming from. The idea is that Some(null) would no longer be possible. Any existing case where an Option[T] could contain a null would become Option[Option[T]] instead.

So in the foo = Foo[Option[T]] case, the type system sees foo.value: Option[Option[T]], but the compiler knows that the runtime value is actually Option[T] | null. When you do a write to that field, we have a real Option[Option[T]], where Some cannot contain null (because otherwise we’d have expanded it to yet another level of Option nesting). So Some(None) gets written as None, None gets written as null, and Some(Some("foo")) gets written as Some("foo").

Of course, this might not be possible due to backwards compatibility concerns (e.g., Some(null) is already allowed in existing code).

[Ichoran]

@wtetzner

Here’s an example of what I’m trying to describe. Let’s say we have a Java class:

public class Foo {
    ...
    public String value; // Nullable field
    ...
}

But this doesn’t extend well.

public interface Bar {
  public Object thing();
}

Now we want to create a Bar that returns a String in Scala. We have two choices:

abstract class Barrish extends Bar {
  def thing(): String
}

abstract class Barred extends Bar {
  def thing(): Option[String]
}

If the Java interface is interpreted as AnyRef | Null, then the first one makes perfect sense. We’re restricting the output type in the subclass (valid) to a non-null String. Java is happy, we’re happy, nobody has any extra boxing or other work to do. And if we want to still handle passing null, we def thing(): String | Null. What you say is what you get. Easy! Except Bar returns an awkward AnyRef | Null, which is a bit of a pain to deal with.

If the Java interface is interpreted as Option[AnyRef], then the second one makes perfect sense. We can see the type we’re actually dealing with. But now what happens if we want our subclass to return only non-null strings? String is wrong, because String <: Option[String] is wrong. Some[String] is “correct”, but it is also wrong because there is no point to carrying Some[String] around just to remind you that you meant String but had to interop with Java somewhere but it doesn’t actually matter any longer.

So I think it is better not to get tricky with our types. Even boxing of primitives only barely works–and it’s much trickier when the boxed thing can have its own type parameters. Types should straightforwardly say what they mean, and that includes that if Option is in fact a union of Some(x) and None, that it really is that, and null, which is different, is different.

Because of how complicated it makes inheritance, I think our types need to be as forthright as possible. That means Foo | Null, not Magic[Foo] (unless type Magic[Foo] = Foo | Null). We can then give ourselves the best tools we can think of to deal with these types, and that may mean that we need to tweak the compiler to express type subtractions, e.g. the type of x match { case null => ???; case y => y } clearly cannot be inhabited by null in the non-exception case but we cannot really express that based on the type of x. But I think magically papering over two different abstractions for dealing with missing values is a mistake.

Java-transparently-as-Option is simpler until it is a hair-pulling nightmare, and it does not take much to fall asleep and be in the nightmare realm.

[wtetzner]

Yeah, you make a very compelling case. Inheritance in particular makes it weird. I’m sure there are ways around these issues, but none of them are particularly pleasant. In fact, the only way I can think of to make the ergonomics not-too-bad is to somehow “fake” Some[T] <: T, and that only adds (a lot) to the complexity.

Well, thanks for humoring me and at least working through the idea to it’s ultimate conclusion.

I guess the ergonomics are already not too bad, since you can always call Option(nullableValue) and option.orNull to convert between them.

[abosshard]

I wonder if expressing Null as opaque type Null = ... could work.

This way the RHS isn’t forced to live outside AnyVal | AnyRef, and hiding the choice of representation avoids the Null <:< AnyVal problem. WDYT?

[Sporarum]

IIRC opaque types are only known to be >: Nothing <: Any, and so this would boil down to keeping Null in the same spot it is currently

[abosshard]

Exactly. The LHS would be >: Nothing <: Any while the RHS could be under AnyVal or AnyRef, whatever is most convenient for the implementation. So Null would look like it’s sitting outside AnyVal | AnyRef while the partitioning of Any into AnyVal and AnyRef still holds behind the scenes.

[Sporarum]

Yes, but I think that would also eliminate the benefits

[abosshard]

Which benefit do you have in mind?

[sjrd]

An opaque type alias does not solve anything.

For starters, you need something on the rhs of =. But let’s assume that would be “magic”.

If you keep the default <: Any bound or use any bound that does not exclude AnyRef, then it won’t be disjoint from AnyRef. That effectively means that AnyRef is nullable again. Nobody wants that.

If you use a bound such as <: AnyVal that rules out AnyRef, then it will basically behave like the current proposal where a class extends AnyVal. Except it won’t give a proper class to the null value, which is theoretically annoying. Same practical properties and worse theoretical properties means it’s strictly worse.

[abosshard]

@sjrd

If you keep the default <: Any bound or use any bound that does not exclude AnyRef, then it won’t be disjoint from AnyRef. That effectively means that AnyRef is nullable again. Nobody wants that.

I totally agree that nobody wants Null <:< AnyRef to hold. With the opaque type neither Null <:< AnyRef nor Null <:< AnyVal would hold. So I wonder if this could be enough for what “Explicit Nulls” is trying to solve.

[abosshard]

@sjrd

For starters, you need something on the rhs of =.

I would lean towards implementing the null value as object null. This would give it a distinct identity. The RHS would then be null.type.

[abosshard]

I have put a demo of the idea on Scastie.

UPDATE 1. While I was playing around with this extended version (Scastie) on my own machine using Scala-CLI and moved ExplicitNull to its own compilation unit ExplicitNull.scala, I found that the marked line in exhaustiveMatch_1 crashes the Scala 3.8.4 compiler:

//> using scala 3.8.4

import ExplicitNull.*

def exhaustiveMatch_1(x: String | Int | Null) =
  x match
    case s: String => s"Hello, $s"
    case n: Int    => s"The number is $n"
    case _: Null   => "null" // this line crashes the compiler

def exhaustiveMatch_2(x: String | Int | Null) =
  x.orUnit match
    case s: String => s"Hello, $s"
    case n: Int    => s"The number is $n"
    case ()        => "null"

def exhaustiveMatch_3(x: String | Int | Null) =
  x.toOption match
    case Some(s: String) => s"Hello, $s"
    case Some(n: Int)    => s"The number is $n"
    case None            => "null"

def brittleMatch(x: String | Int | Null) =
  x match
    case s: String => s"Hello, $s"
    case n: Int    => s"The number is $n"
    case _         => "null"
    //   ^ brittle user code: if another case is added to the type of x,
    //   the exhaustivity checker does not remind you to also cover it
    //   in the match statement

Full compiler output

opaque-null % scala compile .
Compiling project (Scala 3.8.4, JVM (25))
[warn] ./demo.scala:9:10
[warn] the type test for ExplicitNull.Null cannot be checked at runtime because it refers to an abstract type member or type parameter
[warn]     case _: Null => "null" // this line crashes the compiler
[warn]          ^
Error compiling project (Scala 3.8.4, JVM (25))
Error: Unexpected error when compiling opaque-null_26a722ea5d: java.lang.AssertionError: assertion failed: private object NullValue in object ExplicitNull in ExplicitNull.scala accessed from method exhaustiveMatch_1 in /opaque-null/demo.scala
        at scala.runtime.Scala3RunTime$.assertFailed(Scala3RunTime.scala:10)
        at dotty.tools.dotc.transform.ExpandPrivate.ensurePrivateAccessible(ExpandPrivate.scala:95)
        at dotty.tools.dotc.transform.ExpandPrivate.transformIdent(ExpandPrivate.scala:100)
        at dotty.tools.dotc.transform.ExpandPrivate.transformIdent(ExpandPrivate.scala:99)
        at dotty.tools.dotc.transform.MegaPhase.goIdent(MegaPhase.scala:621)
        at dotty.tools.dotc.transform.MegaPhase.transformNamed$1(MegaPhase.scala:240)
        at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:452)
        at dotty.tools.dotc.transform.MegaPhase.loop$3(MegaPhase.scala:486)
        at dotty.tools.dotc.transform.MegaPhase.transformTrees(MegaPhase.scala:486)
        at dotty.tools.dotc.transform.MegaPhase.transformUnnamed$1(MegaPhase.scala:296)
        at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:454)
        at dotty.tools.dotc.transform.MegaPhase.transformUnnamed$1(MegaPhase.scala:325)
        at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:454)
        at dotty.tools.dotc.transform.MegaPhase.loop$2(MegaPhase.scala:471)
        at dotty.tools.dotc.transform.MegaPhase.transformBlock(MegaPhase.scala:476)
        at dotty.tools.dotc.transform.MegaPhase.transformUnnamed$1(MegaPhase.scala:315)
        at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:454)
        at dotty.tools.dotc.transform.MegaPhase.transformNamed$1(MegaPhase.scala:278)
        at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:452)
        at dotty.tools.dotc.transform.MegaPhase.mapDefDef$1(MegaPhase.scala:265)
        at dotty.tools.dotc.transform.MegaPhase.transformNamed$1(MegaPhase.scala:268)
        at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:452)
        at dotty.tools.dotc.transform.MegaPhase.loop$1(MegaPhase.scala:465)
        at dotty.tools.dotc.transform.MegaPhase.transformStats(MegaPhase.scala:465)
        at dotty.tools.dotc.transform.MegaPhase.transformUnnamed$1(MegaPhase.scala:376)
        at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:454)
        at dotty.tools.dotc.transform.MegaPhase.transformNamed$1(MegaPhase.scala:272)
        at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:452)
        at dotty.tools.dotc.transform.MegaPhase.loop$1(MegaPhase.scala:465)
        at dotty.tools.dotc.transform.MegaPhase.transformStats(MegaPhase.scala:465)
        at dotty.tools.dotc.transform.MegaPhase.mapPackage$1(MegaPhase.scala:396)
        at dotty.tools.dotc.transform.MegaPhase.transformUnnamed$1(MegaPhase.scala:399)
        at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:454)
        at dotty.tools.dotc.transform.MegaPhase.transformUnit(MegaPhase.scala:481)
        at dotty.tools.dotc.transform.MegaPhase.run(MegaPhase.scala:493)
        at dotty.tools.dotc.core.Phases$Phase.runOn$$anonfun$1(Phases.scala:411)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
        at scala.collection.immutable.List.foreach(List.scala:327)
        at dotty.tools.dotc.core.Phases$Phase.runOn(Phases.scala:403)
        at dotty.tools.dotc.Run.runPhases$1$$anonfun$1(Run.scala:380)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
        at scala.collection.ArrayOps$.foreach$extension(ArrayOps.scala:1324)
        at dotty.tools.dotc.Run.runPhases$1(Run.scala:373)
        at dotty.tools.dotc.Run.compileUnits$$anonfun$1$$anonfun$2(Run.scala:420)
        at dotty.tools.dotc.Run.compileUnits$$anonfun$1$$anonfun$adapted$1(Run.scala:420)
        at scala.Function0.apply$mcV$sp(Function0.scala:42)
        at dotty.tools.dotc.Run.showProgress(Run.scala:482)
        at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:420)
        at dotty.tools.dotc.Run.compileUnits$$anonfun$adapted$1(Run.scala:432)
        at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:69)
        at dotty.tools.dotc.Run.compileUnits(Run.scala:432)
        at dotty.tools.dotc.Run.compileSources(Run.scala:319)
        at dotty.tools.dotc.Run.compile(Run.scala:304)
        at dotty.tools.dotc.Driver.doCompile(Driver.scala:37)
        at dotty.tools.xsbt.CompilerBridgeDriver.run(CompilerBridgeDriver.java:141)
        at dotty.tools.xsbt.CompilerBridge.run(CompilerBridge.java:22)
        at sbt.internal.inc.AnalyzingCompiler.compile(AnalyzingCompiler.scala:91)
        at sbt.internal.inc.bloop.internal.BloopHighLevelCompiler.compileSources$1(BloopHighLevelCompiler.scala:157)
        at sbt.internal.inc.bloop.internal.BloopHighLevelCompiler.$anonfun$compile$9(BloopHighLevelCompiler.scala:188)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at sbt.internal.inc.bloop.internal.BloopHighLevelCompiler.$anonfun$compile$1(BloopHighLevelCompiler.scala:74)
        at bloop.tracing.NoopTracer$.trace(BraveTracer.scala:53)
        at sbt.internal.inc.bloop.internal.BloopHighLevelCompiler.timed$1(BloopHighLevelCompiler.scala:73)
        at sbt.internal.inc.bloop.internal.BloopHighLevelCompiler.$anonfun$compile$8(BloopHighLevelCompiler.scala:188)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at monix.eval.internal.TaskRunLoop$.startFull(TaskRunLoop.scala:81)
        at monix.eval.internal.TaskRestartCallback.syncOnSuccess(TaskRestartCallback.scala:101)
        at monix.eval.internal.TaskRestartCallback.onSuccess(TaskRestartCallback.scala:74)
        at monix.eval.internal.TaskExecuteOn$AsyncRegister$$anon$1.run(TaskExecuteOn.scala:71)
        at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(ForkJoinTask.java:1753)
        at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(ForkJoinTask.java:1745)
        at java.base/java.util.concurrent.ForkJoinTask$InterruptibleTask.exec(ForkJoinTask.java:1662)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:511)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1450)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:2019)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:187)

Compilation failed

The workarounds exhaustiveMatch_2 and exhaustiveMatch_3 work fine though.

Hope this helps.

UPDATE 2. Scala’s stellar expressiveness never stops to amaze me. Adding a TypeTest[Any, Null] resolves the issue completely and avoids the crash. So here is a little more comprehensive, working demo on Scastie. I’ll leave it at that, promise.

[soronpo]

Compiler crashes should not happen. Please submit a (minimized) ticket on the Scala3 repo.

[abosshard]

Done – AssertionError due to private access violation #26754