Make Null a subclass of AnyVal under -Yexplicit-nulls

Language Design
41

We have AnyKind already so Null <: AnyKind could also be possible with not sub classing Any. But then I guess be need to allow values still to be of type Null

42

I am with @sjrd here. AnyVal is the obvious superclass of Null. null is a value, but not a reference (in the explicit-nulls model). All values that are not references are under AnyVal. The only things that are under Any bit not AnyVal nor AnyRef are universal traits that can be implemented by both value and reference classes.

1 Like
43

that’s a strong opinion.

is there a language with proper nulls (i.e. null pointers or references, not null-like stuff like empty optionals) that doesn’t define null as pointer or reference?

java calls null a reference and a value:
Chapter 4. Types, Values, and Variables .

The null reference is the only possible value of an expression of null type.

c# calls null a reference and a value:

The null keyword is a literal that represents a null reference, one that doesn’t refer to any object. null is the default value of reference-type variables. Ordinary value types can’t be null, except for nullable value types.

c# has explicit nullability mode. i haven’t dug deep if the definitions change under explicit nullability mode, but the null keyword page doesn’t mention such special cases.

there are two options: either the scala beginner should be prepared to deal with nulls or not:

  • if not, then simply redraw the diagram to make nulls less prominent and don’t elaborate on null
  • if yes, then lying that null is not a reference is counterproductive

imho the people that are most likely to run into null-related exceptions are beginners, irrespective whether nullness is explicit or not, so presenting some weird leaky abstractions to them won’t make them less affected by nulls. as i’ve written previously:

expand to see previous common type of example of unintentionally running into nulls

from my point of view, simplifications are sometimes more detrimental than helpful. good example is monads. when i was trying to grasp what’s the deal with monads, i’ve read many tutorials and most often they used some weird (sometimes even childish) analogies that didn’t help at all.

2 Likes
44

Well, certainly in the regular case, null has multiple (diamond) inheritance with every reference class; Null is the bottom type of the hierarchy that has AnyRef as its top type.

With explicit nulls, whether or not you put null under AnyVal, you cut all those inheritance links. But it is an actual instantiable value, so you have to put it somewhere. You can match on it, so it has to be Matchable. The choices are really only (1) AnyVal or (2) its own thing directly under Matchable.

Then the question is that if you choose (1), do you swap the inheritance from AnyRef to AnyVal on explicit nulls, or is null always part of AnyVal and the only question is whether it also has diamond inheritance with AnyRef. So do you add a link, or only cut them?

Those are really the only two questions left. The issues about types lying will still be there; those are orthogonal. If String might be null, it should be typed as String | Null. Will it be, while genuinely always-instantiated string values are typed as String? The problem with the non-lie Null is that it is not a helpful truth. People rarely check nullness anyway, because the happy path is the one where you have ensured that the values have no nulls. But if that is intent, why not match the intent with the type?

45

i’m not an expert but i think you might have this wrong. in java java.lang.Object seems to be the root class, not the root type. boxed types like java.lang.Integer come under this root but primitives like int (meant to be represented in scala by AnyVal) have no common root. so in fact java.lang.Object =:= AnyRef and really your statement is an argument for removing Null from the AnyRef hierarchy and not an argument against putting it under AnyVal

46

You’re right that in Java int doesn’t inherit from java.lang.Object but that’s because it doesn’t participate in any subtyping scheme. If you say “root”, you imply subtyping.

Theoretically, java.lang.Object is AnyRef, but that’s paying lip service, because in fact Any is java.lang.Object, much like how Scala’s Int is getting boxed into java.lang.Integer, from Scala’s point of view, int existing as a backend optimization.

To be clear, I don’t really care about the hierarchy, what I care about are these 2 use-cases:

  1. The ability to easily say “these are any kind of object, but not null” (concrete usecase being the queue implementations of JCTools)
  2. Compiler banning any method calls on nullable references — If you make null an AnyVal you cannot ban methods such as toString or equals on nullable references (Liskov substitution principle), so as long as Any has this definition and as long as null <: Any, the “explicit nulls” feature doesn’t make null-usage safe, being half baked.

Kotlin solves these 2 cases for me. Java’s JSpecify does as well.

1 Like
47

Theoretically, java.lang.Object is AnyRef, but that’s paying lip service, because in fact Any is java.lang.Object, much like how Scala’s Int is getting boxed into java.lang.Integer, from Scala’s point of view, int existing as a backend optimization.

i don’t think i agree that Any =:= java.lang.Object simply because primitives are sometimes autoboxed (i assume that’s what you’re getting at). what is really happening is that Int <: AnyVal | AnyRef, ie. it is sometimes a value and sometimes an object. that doesn’t make all types really object types at all. and remember this is only true internally because of inadequacies in the jvm (inadequacies that people are trying to resolve). on a different backend the primitives might be true primitives and not have the same problems (this is mostly the case for scala.js i believe). i don’t reckon it’s a good idea to entrench jvm inadequencies in the general type system except where absolutely necessary (this mentality is why scala has AnyVal and the unified type system, not untethered primitives)

Compiler banning any method calls on nullable references — If you make null an AnyVal you cannot ban methods such as toString or equals on nullable references (Liskov substitution principle), so as long as Any has this definition and as long as null <: Any, the “explicit nulls” feature doesn’t make null-usage safe, being half baked.

agreed on point #1 but why do you feel that null has to not respond to methods? it feels like you are equating Null with Nothing (ie. an absence of any value). null is a real thing so has to come under Any. it has a data representation (0), a string representation (null in most languages), and can be equated. so to me it’s far more bizarre that it doesn’t respond to most normal methods (this feels like a mistake inherited from java). having these methods would not only be honest about what null is but it’d also be helpful in some cases. and banning all methods on null also wouldn’t really solve the problems people encounter. most methods are banned right now (it only supports eq and asInstanceOf and i’m not sure what else) and yet it seems pretty uncontroversial to say that there are issues with null that need to be resolved. null responding to methods when you expect it to explode instead would merely be a symptom of a larger problem, not the problem itself. and dragging it out of from Any and untethering it will just cause issues in the same way primitives in java cause issues (surely this is the ultimate source of the boxing issue: implementing primitives as untyped and separate from true types and thus unusable in generics)

what it sounds like you want is the ultimate goal of the ‘explicit nulls’ work. once nulls are properly accounted for in the type system and it’s honest about where nulls might appear then nulls should become simple to deal with and the problems should mostly disappear. whether it responds to methods is neither here nor there

btw, looks like kotlin doesn’t give you those 2 things:

>>> null.toString()
res0: kotlin.String = null
>>> null.equals(null)
res1: kotlin.Boolean = true)
>>> null.hashCode()
res2: kotlin.Int = 0
48

That’s because it has convenient extension methods defined in the standard library. Here’s how it looks like:

public inline fun Any?.hashCode(): Int = this?.hashCode() ?: 0

Those extension methods are not virtual methods defined on Any?. If you try accessing the actual methods, Kotlin doesn’t let you, because Any? is not Any and doesn’t have hashCode() defined. Good example though. See how null.hashCode() behaves in Scala and compare.

49

well, that’s kinda interesting, but i don’t see the relevance here. it shouldn’t matter where the method is implemented. the reality is in scala null blows up on most methods and you seem unhappy with it, but in kotlin it responds to more methods and you seem happy. so responding to methods can’t be the problem surely

the key issue in how it handles null seems to be the ‘Any? is not Any’ part. in kotlin it’s truthful about the nullability but scala isn’t (yet) truthful. once it is then the same benefits should come to scala. perhaps, if you don’t already, it would be worth turning on ‘explicit nulls’ to see how it works for you and report back any problems. i don’t know myself but it seems the right path

50

to talk about the original post for a moment, if i ignore the graph confusion part and focus on the core idea of Null <: AnyVal then i’m more and more convinced it’s a good idea

null is a real value (ie. stored in memory), represented by a 0 in 4 or 8 bytes on most systems. it is seen as a pointer but it’s not inherently a pointer (indeed it points to nothing). rather it’s a marker for the absence of a pointer. it is conflated with a pointer-to-data / object type by traditionally being stored in the same word of memory (essentially a classic union type, done for simplicity and performance, not for likeness of types). it’s true nature is as a unitary value and so nulls have a common immutable value, not individual instances (akin to 0 : Int or true : Boolean and not to AnyRef). as it’s unitary it’s minimal representation would be a 0 in a half bit (ie. unary digit / unit) if such a thing existed. no more space is required. it’s easily stored on the stack directly and has no associated heap space, again like an AnyVal and unlike an AnyRef. the idea of a boxed null is a meaningless concept given it’s uniqueness within the jvm as something that is essentially typeless. but i guess you could say it’ll never be boxed on any backend and be correct, so again like any strict value type. to me it is so very like Unit that they almost feel like the same thing: a single value for when there is no other useful value but we need something (maybe just remove Null and do val s : String | Unit = () instead? save us the hassle :grin:)

so overall it feels very value-like. it could easily fit directly under Any but it feels to me to match with AnyVal more

1 Like
51

But it does. We are talking about the subtyping hierarchy. Any in Scala has the methods of java.lang.Object, Any? in Kotlin does not. Not sure what you’re trying to argue for here to be honest. Even if you introduce extension methods in Scala, which isn’t possible currently BTW, or extra linting checks, what you’d still have is this:

val x: Int | Null = null
// upcast is perfectly legal due to subtyping relatioship
val y: Any = x 
y.hashCode()

Scala is unsound here for as long as Null <: Any, whereas Kotlin is not.

If you go down that path, then all types are imaginary, it’s all zeroes and ones and we should all get back to programming in C and just cast everything to void*.

null is a value, yes, with its own type that shouldn’t inherit from java.lang.Object, and in Scala it does.

Unit is not the one billion dollar mistake, null is, and the reason for that is because it can be a subtitute for types without being able to perform their function (Liskov substitution principle is violated).

You can wax poetic about values, and Unit and so on, but facts are facts and one fact is that -Yexplicit-nulls is currently flawed and will remain flawed until Null <: Any stops being true. And in the meantime, Kotlin has a better design and Java has working solutions, such as JSpecify, and even their JEP proposal and preview feature doesn’t fall into Scala’s trap.

And the current proposal basically does nothing to address any issues in actual programming, other than … making the type hierarchy look nicer I guess? :man_shrugging:

4 Likes
52

what if Null is kept as direct subtype of Any but java.lang.Object’s methods are removed from Any? these methods can be reintroduced in AnyVal and AnyRef types, but not Null. then calling java.lang.Object’s methods on Null wouldn’t be possible, which would be an upgrade from current situation. additionally, scala’s stdlib could define type NonNull as either AnyVal | AnyRef (assumming both types guarantee no nulls) or an additional trait with only AnyVal and AnyRef as subtypes, i.e. Null wouldn’t be a subtype of NonNull trait.

null values are needed for compatibility with foreign code, i.e. java, javascript, native ffi, etc. otherwise, scala code should be maximally null-free. making Null a subtype of AnyVal removes no-nulls guarantee from code that operates on AnyVal and makes defining NonNull type harder.

rust has nulls only in the form of raw pointers, but raw pointers can’t be dereferenced outside of unsafe blocks. scala doesn’t have the notion of dereferencing a raw pointer, so instead of prohibiting explicit dereferencing, scala can just remove all java.lang.Object’s methods from Null type.

4 Likes
53

Not sure what you’re trying to argue for here to be honest

you said you care about 2 use cases. i’ve been asking you how your use case #2 makes sense / matters at all. that’s basically it. i still don’t know why you think it matters (i don’t think it truly matters either way so i am making no argument there)

Any in Scala has the methods of java.lang.Object, Any? in Kotlin does not

Any doesn’t respond to all the methods from java.lang.Object, only a subset. the subset that presumably every value could respond to, not just refs. it doesn’t have any of the thread methods for instance. that’s because Any !=:= java.lang.Object

val x: Int | Null = null
// upcast is perfectly legal due to subtyping relatioship
val y: Any = x 
y.hashCode()

Scala is unsound here for as long as Null <: Any, whereas Kotlin is not.

this could and should be perfectly fine and it only isn’t because, bizarrely, null doesn’t respond to the methods the types say it should, thus breaking type guarantees. that’s the fundamental problem there. it has nothing to do with Null <: Any. going back to your use case #2, you seem to have shown that null not responding to methods is the direct issue, not the solution

If you go down that path, then all types are imaginary

i only said that to contrast it with Nothing (that has no value) and AnyRef (where the data isn’t the value itself)

Unit is not the one billion dollar mistake, null is, and the reason for that is because it can be a subtitute for types without being able to perform their function (Liskov substitution principle is violated).

it is the billion-dollar mistake precisely because it was a mere untyped and unchecked value conflated with values of a different type. whilst scala has mistakes of its own that need to be fixed, it’s a different situation

You can wax poetic about values

i gave the real qualities of null to support it’s placement under AnyVal. there’s no poetry anywhere. feel free to actually address the points instead

facts are facts and one fact is that [it] will remain flawed until Null <: Any stops being true

that’s not a fact. that’s an opinion. there are multiple possible ways that this could work well and kotlin’s way, whilst better than scala at present, isn’t the one true way

-Yexplicit-nulls is currently flawed

you might’ve written this somewhere else, so forgive me if you have, but perhaps you could explain how it’s flawed so that your issues can be addressed

And the current proposal basically does nothing to address any issues in actual programming, other than … making the type hierarchy look nicer I guess? :man_shrugging:

agreed. if that’s the sole purpose here then there’s probably little point. the important work is in the ‘explicit nulls’ project: moving null out from under AnyRef to literally anywhere else (resolves the ‘null not responding to methods it claims it does’ issue), and ensuring a value can’t be null unless it’s part of the type

and with that i shall bow out

54

NPE as a boxed null :smiley:

55

I believe it is important to be able to describe types which are not null, otherwise your code will need to become polluted with unnecessary null checks.

56

Thank you all very much for your input. I’ve been very busy over the past few months, but I did read and consider everything happening here.

We recently discussed this feedback internally, where I presented what I write in this post, below. We decided that we would go ahead with the change.

(From a process point of view, it will eventually have to come back for more potential discussion when we actually make a SIP for the whole explicit-nulls feature.)

Below, I summarize the major concerns that have been brought up, and address them.

Before diving into it, I’d like to acknowledge @alexandru’s point here:

The fact that (x: Any).toString()—in general, (x: T).someMethodOfAny()—is unsafe is a valid point. However, making Null not even a subtype of Any is definitely not an option for Scala, so that cannot be the solution. Therefore, this concern is orthogonal to the change proposed here. I will address it in a separate post/proposal.

Now, here we go. I will use headings summarizing categories of feedback/concerns, for easy visual parsing. Because that’s a very short form, it may sound cynical sometimes. Please bear with it.

If you feel you voiced a concern that was not appropriately taken into account among the categories below (and is not about (x: T).someMethodOfAny()), feel free to point it out.

We don’t really care about the diagram

Indeed. This was certainly my mistake. I put the diagram first as a justification, and in particular the difficulty of teaching it.

The diagram is but a symptom of a deeper issue, though. The Scala value system is fundamentally divided into values with and without identity (also: unforgeable versus forgeable). AnyRef and AnyVal are a very concrete embodiment of this dichotomy in the type system. The rules applying to AnyRef don’t apply to AnyVal, and conversely. They are the materialization of two different sets of semantics.

If Null cannot be put into either of those categories, it complicates the design. It means the compiler needs to be aware of Null in many places. The spec needs to treat Null specially. It is a liability to the type system core as much as to the surface, diagrammatic view.

If we can meaningfully put Null into either of those categories, it’s a win for the type system, the compiler, and teachability overall. If that does not cause meaningful drawbacks, we should do it, in an effort to simplify the language as a whole.

We won’t be able to say “T can be anything except Null”

True. Currently, we can do this with T <: AnyRef | AnyVal. This won’t be possible anymore.

What is such a bound good for, though? There have been two use cases put forward.

I want to safely call (x: T).toString()

This is definitely valid, and comes back to the point I mentioned before about (x: Any).toString(). I will address it fully in a separate post. For now, recall that safe alternatives exist: String.valueOf(x), s"$x", "" + x, etc. They’re not nice, but they’re safe.

I want to use null internally as a sentinel

This is the Queue[T] example mentioned above in the thread.

Why do we want the Null exclusion? From an API perspective, ruling out Null is clearly a liability. If we can offer the same service while allowing Null for the users of our Queue, it is a win. So it must be for the benefit of the implementation: we don’t want user values to be confused with the null we internally use as sentinel.

The underlying problem is that null is forgeable (like other AnyVals but unlike other AnyRefs). You can’t prevent someone else from materializing a value that is indistinguishable from the null that you want to use as sentinel.

There is an alternative to null as the choice of sentinel that solves that problem: a private object Sentinel.

Instead of


if (x == null) ...

you write


if (x eq Sentinel) ...

and you’re gold. Nobody outside your Queue can forge another value that will be eq Sentinel, so you don’t have to rule out Sentinel from T. And clearly you also don’t have to rule out Null. Everyone wins.

Now some will say: but performance! Comparing to Sentinel (a constant after JIT’ing) is not as fast as comparing to null. null is 0, and processors compare faster to 0 than to other values.

That is true, but is also a very small effect. The only reason one might care is if they are micro-optimizing a particularly hot path. That can happen, of course, but when it happens, you also definitely don’t want boxing! In that scenario, you should restrict T <: AnyRef in the first place. And therefore, whether Null <: AnyVal is irrelevant.

In conclusion, IMO there is no reason for ruling out Null but not other AnyVals.

It feels wrong / null is a reference

We can also say: “It feels right”, and “null is a value”. We can’t justify any of those point of views, though, so it’s kind of moot.

Is there any other language that does not call null a reference?

As a matter of fact, yes: JavaScript.

In JavaScript, the type Null is explicitly part of the primitive types, which are distinct from Objects.

But I don’t think this is a very interesting question in the context of Scala.

A related, more useful question is: does null fit the semantics of AnyRef or AnyVal?

We have already established that it fits the semantics of AnyVal but not of AnyRef. Chief among them is the ability to forge a value that is indistinguishable from null. See also: the above point about the sentinel, and why Sentinel is better than null (because it can’t be forged).

The JVM says null is a reference

Yes, it does. It also says a bunch of other things that Scala models differently:

  • int is a primitive and can’t be used in generic signatures,

  • void has no value (we have Unit and ()),

  • return is a statement, not an expression,

  • etc.

We must not confuse our source language model and the target we compile to. They have different rules. We can and should abstract away the underlying machine if it suits our source language model better (while giving access to the raw performance bits when necessary).

throw null (technicality)

When you write throw null, it throws a NullPointerException. That is true, and is specified. That’s not because null is the same as a NullPointerException. It’s because you can’t actually throw null, just the same that you can’t call null.foo(). The type system lets you (at least Scala without explicit nulls and the JVM type system), but at run-time it’s not a valid operation. So just as when you try to call null.foo(), the JVM throws an NullPointerException when you try to throw null.

The “idiomatic” way of performing throw null in Scala 3 with explicit nulls would be to write null.nn.

10 Likes
57

Here is the follow-up: Proposal: Fixing null.methodOfAny() under explicit-nulls

1 Like
58

A bunch of very good points !

It still feels weird, but I think with those arguments I can envision that maybe it’s like the variance of functions, and that this weirdness will pass with time !

(also I’m pretty sure the “null is NullPointerException” thing was a joke, but I always have trouble telling with @som-snytt)

59

That was me riffing on som (I am not sure what som intended), and it was both joke and not at the same time.

Primitives have a canonical boxed form on the JVM–java.lang.Integer and so on. Unit does not, so Scala invented one (BoxedUnit).

But null, curiously, also has a boxed form: NullPointerException. It’s converted to the boxed form not on exposure to generic context but upon method access. So, just like 7 and java.lang.Integer(7) correspond across the AnyVal / Object divide, so too do null and java.lang.NullPointerException.

Now, we can’t use that, because any way we use it would be a logic error: we don’t want to silently swallow NPEs or carry them along as a stack trace, or worse yet as a stackless exception from who knows where. So, no–this is not business as usual where boxed primitives play happily like they’re any other Object.

It was just amusing to me (joke part) that in fact Java already does treat null like a primitive in that you can’t directly name its type as part of the Object hierarchy, and when an objecty behavior is needed, it’s boxed in a canonical way.

60

NOTE: this post kinda veered off-topic (it’s related in null alone) but I want to keep it documented instead of just letting it vanish into the aether. If there’s a more appropiate topic I find I’ll move it there instead.

The use case for “T can be anything except Null” is for interoperating with java libraries using TYPE_USE annotations (that can target type parameter usages, in particular), like jSpecify.
Currently in the compiler I don’t think these are implemented, but if they were in the future implemented there would be some form of “without null” type to correctly implement these annotations.

Here’s an example of a use case:

public interface Foo<T extends @Nullable Object> {
    // this ALWAYS excludes null, regardless of the nullability of T
   @NonNull T notNull();
   // This ALWAYS includes null, regardless of the nullability of T
   @Nullable T nullable();
   // This passes through the nullability of T.
   T passthrough();
}

Curiously, the first case can’t even be expressed in current scala (unless I’m mistaken, feel free to correct me). AFAIK there’s no type to “cast away null”, unlike how there’s a method to “cast away null”.

It’s important to have a way to cast away null at the type level specifically when wrapping java type variables, intersection types, and union types. Of course I don’t think java really ever lets you expose the intersection or union types but type variables are indeed frequently exposed.
Like the jSpecify spec states:

If tool authors prefer, they can safely produce MINUS_NULL in any case in which it is equivalent to NO_CHANGE. For example, there is no difference between Foo NO_CHANGE and Foo MINUS_NULL for any class type Foo (nor for any array type or the null type). The difference is significant for intersection types, type variables, and union types.

(emphasis added by me)
The idea is that specifically for type variables, intersection types, and union types, it’s important to keep track that null was explicitly excluded from a set.

1 Like