Proposal: Fixing null.methodOfAny() under explicit-nulls

[sjrd]

This is a follow-up to Make Null a subclass of AnyVal under -Yexplicit-nulls - #56 by sjrd . A particular concern that was brought up is that, even under explicit-nulls, the methods of Any are still null-unsafe.

To be clear: all of this thread is under the context of -Yexplicit-nulls. I do not propose to change anything in the currently standard Scala.

Null <: Any in Scala. This is unavoidable, because, by definition, Any is the supertype of all proper types. Proper types are the types of values, as opposed to higher-kinded types, which are the types of other types.

That poses a problem, because several methods available on Any are not supported by null. They throw a NullPointerException. The most obvious example is toString(). Here is a complete table of the methods we are allowed to call on x: Any:

Method of Any	Behavior for null
==, !=	Safe (null == null and to nothing else)
##	returns 0
isInstanceOf[X]	returns false for all X
asInstanceOf[X]	let’s say: not more unsafe than on other values
hashCode()	throws NPE
equals(y)	throws NPE
toString()	throws NPE
getClass()	throws NPE
nn	throws NPE, but that’s the point

x.hashCode() and x.equals(y) have had good alternatives forever: use x.## and x == y instead.

x.toString() sort of has alternatives: s"$x", "" + x, String.valueOf(x) all return the string "null" when x == null. They’re not very nice, though.

x.getClass() has no alternative. You need a type test.

Proposal

I propose the following changes:

• Deprecate Any.hashCode() and Any.equals() altogether. Override them as non-deprecated in AnyRef (aka jl.Object)
• Change the behavior of Any.toString() and Any.getClass():
  • null.toString() should return the string "null"
  • null.getClass() should return classOf[Null]

We can implement the changes using additional dispatch introduced by the compiler. For x: T with Null <: T, we will compile

Source	Rewritten
x.toString()	String.valueOf(x)
x.getClass()	if (x ne null) x.asInstanceOf[AnyRef].getClass() else classOf[Null]

This way, it will be safe to call all (non-deprecated) methods of Any on null. Problem solved.

Prior art

I am told Kotlin allows to call x.toString() when x: Any?, and that it returns "null". Scala’s Any type is equivalent to Kotlin’s Any?: they each are the supertype of all proper types in their respective type systems.

For getClass(), it’s a bit convoluted, but since we can write classOf[Null] in source and do get an actual instance of jl.Class, I believe that is the natural choice.

WDYT?

[soronpo]

@sjrd

To be clear: all of this thread is under the context of -Yexplicit-nulls. I do not propose to change anything in the currently standard Scala.

What is supposed to happen during compiletime/runtime with a code under the explicit null flag that calls a library that did not turn on the flag and vice-versa?

[sjrd]

The semantics of any particular call will depend on the options used to compile that call.

• explicit-nulls calls non-explicit-nulls calls toString() → NPE.
• non-explicit-nulls calls explicit-nulls calls toString() → "null".

[ragnar]

Why deprecate hashCode and equal and specializing toStringand getClass? Why not deprecate all 4 and add alternatives for toString and getClass? Or specialize all 4?

Maybe, because the specialization induces some overhead (if not JIT optimized), so its better to explicitly use the alternatives?
I guess deprecating toString and getClass on AnyVal would be quite inconvenient, even if an alternative exists.

Mostly asking to poke at the design, the proposed variant does not seem worse to me.

[sjrd]

Because ##, ==, toString and getClass are already the idiomatic things to use today. So every code that’s already doing the right thing won’t have to change.

[ragnar]

And the other way round? Why not also make hashcode and equals work?

[sjrd]

Because they do not respect cooperative equality. And they must be allowed to stay that way, because when we override them in user classes we must not deal with cooperative equality. That’s why ## exists in the first place.

[Sporarum]

Something which was not explicitly stated on this post but might be useful to note:

We can do this whether null <: AnyVal or not, right ?

---

@sjrd

all of this thread is under the context of -Yexplicit-nulls. I do not propose to change anything in the currently standard Scala.

What are the obstacles to doing it without -Yexplicit-nulls ?
It seems like the idea of doing

@sjrd

For x: T with Null <: T, we will compile

Source	Rewritten
x.toString()	String.valueOf(x)
x.getClass()	if (x ne null) x.asInstanceOf[AnyRef].getClass() else classOf[Null]

This way, it will be safe to call all (non-deprecated) methods of Any on null. Problem solved.

also works with implicit nulls, the difference being that now all T <: AnyRefs (except Bottom) need these adaptations, not just Any, AnyVal and T | null.

With it I only see two regressions:

1. code expecting null.{toString,getClass}() to fail
2. performance (since we now essentially add this rewrite everywhere)

I don’t have a big grasp of performance, so I don’t have a good idea of how big the hit would be
(and potentially it’s so obviously monstruous it is the reason @sjrd didn’t propose it)

---

Somewhat related, the proposed rewrite would affect code like this, right ?
(With -Yexplicit-nulls, regardless of null <: AnyVal)

def foo[T](x: T) =
  println(s"I was passed ${x.toString()} of ${x.getClass()}")

Since T <: Any, its possible to pass a T s.t. null <: T, and therefore we need to handle this case

So if there are performance concerns above, might they not also apply to the proposal as-is ?
(Depends on how prevalent things like [T](x: T) are wrt (x: SomeKnownType))

[sjrd]

@Sporarum

Something which was not explicitly stated on this post but might be useful to note:

We can do this whether null <: AnyVal or not, right ?

Absolutely. I mentioned in the other thread that this issue is orthogonal. But it’s worth reiterating here: this proposal is completely orthogonal to whether Null <: AnyVal or not.

@Sporarum

What are the obstacles to doing it without -Yexplicit-nulls ?

Without explicit-nulls, it doesn’t qualitatively solve anything. Because even though you can safely do x.toString() for any x, you still cannot safely do str.substring(1). Your str: String might be null, and it would still NPE.

Under explicit-nulls, all the non-Any call sites are already safe. This proposal is the last straw to fix them all.

Explicit-nulls is still malleable. Changing non-explicit-nulls would require a SIP on its own. It’s not worth the hassle since it won’t qualitatively solve anything.

Performance has nothing to do with why I’m not proposing this for non-explicit-nulls.

@Sporarum

Somewhat related, the proposed rewrite would affect code like this, right ?
(With -Yexplicit-nulls, regardless of null <: AnyVal)

def foo[T](x: T) =
  println(s"I was passed ${x.toString()} of ${x.getClass()}")

Since T <: Any, its possible to pass a T s.t. null <: T, and therefore we need to handle this case

Yes. This is explicitly included in the proposal:

@sjrd

For x: T with Null <: T, we will compile […]

Performance is not a concern. We are talking about a ~100%-predictable null check, in front of a) a megamorphic virtual call or b) a reflection-related call. For all intents and purposes, the changes will have zero overhead.

[Sporarum]

Thanks for your reply, I think I understand everything now !

I guess my remaining question is: Are there any reasons not to do this ?
(for everyone)

[tarsa]

@Sporarum

Are there any reasons not to do this ?
(for everyone)

i’m lurking at scala.js codebase from time to time and remember that bare something.getclass() expression (i.e. without using the result of getclass) was an idiom for null-check. this proposal will change the semantics of such trick. to reduce such unforeseen consequences, scala linter should catch such expressions and warn that semantics has changed.

[sjrd]

We use that inside the Scala.js javalib, yes. The javalib is compiled with Scala 2.12, so don’t worry about it

User code should use .nn for that, which is much more convenient and self-explanatory.

[tarsa]

but anyway, statements like whatever.getclass() will become side effect free, woudn’t they? if so, then compiler should be taught of that and then linter should catch it as side effect free expression whose value is unused.

[ragnar]

@Sporarum

Are there any reasons not to do this ?

Maybe just to state explicitly, my questions were answered and I really like this proposal.

[mberndt]

@Sporarum

I guess my remaining question is: Are there any reasons not to do this ?
(for everyone)

Well, to play the devil’s advocate here: there’s the risk of sweeping bugs under the rug when a null somehow manages to creep into a data structure where it doesn’t belong. When that happens and I call .toString on it, it’ll yell at me and I can go fix my program. With this change, I might never know.

On the other hand, arguably the real problem in that case isn’t that toString doesn’t explode when called on null but the fact that explicit-nulls still has some holes like Java interop (unless no-flexible-types is used) and object initialization (unless safe-init is used).