"MINUS_NULL" and the future of explicit nulls

Language Design
1

Hello,
I was talking in the Make Null a subclass of AnyVal under -Yexplicit-nulls and things may be getting a little bit off topic, so I want to separate out my topic.

As a recap, the primary issue I’m declaring with this proposal is java interop, specifically with this problem from the Kotlin KEEP definitely non-nullable types proposal:

public interface JBox {
    <T> void put(@NotNull T t);
}

Basically, in Kotlin implementing this type required some way to say “T but without null”. Kotlin didn’t have user facing intersection types (they were always part of the spec but never user denotable). They had to add T & Any as an option to remove null.

We in Scala DO have intersection types and can currently express this like so:
T & (AnyRef | AnyVal). Of course this is very scuffed and not really intended or officially supported by the compiler, that being obvious with the new proposal.

What this topic is about if, in the future, we do say Null <:< AnyVal in explicit nulls (and eventually, all worlds as explicit nulls becomes the only type system, with the only variance being if the compiler reports these null errors), how will Scala correctly represent the above type?

What we need is some way to “cast away null” at the type level, and as well the compiler should correctly produce these kinds of types when @NonNull is explicitly used, instead of just skipping it. This ties into my idea to support jSpecify NullMarked and NullUnmarked annotations. The jSpecify spec is quite a good guide as a baseline for any implementation of null checking. Of course we don’t need to follow it to the letter, we aren’t actually tooling for java so we come first. However, for augmented types (basically, types that also have a nullness operator on them), they have some good rules.

Its rules more or less match the current rules scala uses when parsing java code:

  • If the type usage is annotated with @Nullable and not with @NonNull, its nullness operator is UNION_NULL

    In scala, this is T | Null or NullMode.Explicit

  • If the type type usage is annotated with @NonNull and not with @Nullable, its nullness operator is MINUS_NULL

    This is different from Scala, which instead uses NO_CHANGE, or NullMode.Skip

  • If the type usage is the parameter of equals(Object) in a subclass of java.lang.Record then its nullness operator is UNION_NULL

    This is some jank to prevent sadness with autogenerated record classes in null-marked scopes, as bytecode doesn’t mark the generated equals method as generated, meaning any manually written ones are can’t be seen as actually manually written. Scala doesn’t do anything with this info, this is purely some jSpecify jank.

  • If the type usage appears in a null-marked scope, its nullness operator is NO_CHANGE

    In the above linked PR this is indeed the behavior observed. The default mode in a null-marked scope is NullMode.Skip

  • By default, its nullness operator is UNSPECIFIED

    In Scala with flexible types, this is the same as T? or NullMode.Flexible

There’s an extra important aside that jSpecify has in their spec:

If tool authors prefer, they can safely produce MINUS_NULL in any case in which it is equivalent to NO_CHANGE. For example, there is no difference between Foo NO_CHANGE and Foo MINUS_NULL for any class type Foo (nor for any array type or the null type). The difference is significant for intersection types, type variables, and union types.

There is no difference between Foo NO_CHANGE and Foo MINUS_NULL for any class type Foo (or any array type or the null type). The difference is significant for intersection types, type variables and union types.

I think that Scala’s type system can handle this equivalence already, without any special casing. So I don’t really think there’s any harm in changing the @NonNull case from NullMode.Skip to a new NullMode.MinusNull, that adds an intersection with a type that holds all types except null. In current scala that type is AnyVal | AnyRef and in the future with the AnyVal Null proposal it will need to be a new type (my strawman type for this will be NotNull, defined as Any except for Null)

I don’t think Null can ever leave the type tree, for the simple fact that it needs to extend Matchable for c match { case null => ??? } to work. I personally wouldn’t like NotNull to be a magic compiler difference type but if that’s the only magic in explicit nulls world compared to null being the bottom type in non-explicit nulls then I could tolerate it.

Of course this MINUS_NULL business only really matters in a few select cases, but I think its worth keeping in mind. Something else that’s important is jSpecify’s rules also apply to type bounds, so we have to make sure to correctly transform a type parameters bounds.

I know this is quite a large post but I have a lot to say about this feature - I work very heavily with Minecraft mods which the whole deal there is interoping with Java, so any tiny imperfection is something I run into countless times. Some are easily fixable with shim libraries, but with explicit nulls it’s more of a fundamental issue. This isn’t a dig at explicit nulls, it’s greatly improved over the years (I remember ye old days of trying to use it and every thing in Minecraft being X | Null) and I suggest anyone using pure Scala use it.

If anyone has any additional feedback, I’d love to hear it. Thanks for reading this monster of a topic.

2 Likes
2

You discarded the using erased NotGiven[Null <:< T] from the other thread but that seems to be a pretty idiomatic way to prevent that T is Null at the type level?

In general, I think Scala current behaviour of allowing put[Null](null) seems to be good design, as an unbounded generic should work with any type. It seems that the intent of @NonNull would be to prevent put[Object](null)which is prevented by default with explicit nulls in Scala.

If you don’t want to allow T to be null, then maybe def put[T <: AnyRef](t: T)would be what you want (or back to the using clause).

Or in other words, to me it seems that @NonNull T is meant to say „for any reference type that you might substitute for T, make sure that the concrete type does not allow for null“. Which is the default for explicit nulls.

Maybe I am not really understanding your usecase. Do you have any examples for behaviours you would like to prevent that are currently not expressible?

3

I like the idea of a `NullMode.MinusNull` (or maybe NullMode.NonNull) type that contains everything but null. It seems easy to understand, and unproblematic to use with existing type machinery.

4

the NullMode is a reference to compiler internals, so it’s not a literal type for Scala apps. Maybe I should have been more explicit about that. Scala would end up getting a type like NonNull

Kotlin currently uses the upper bound T: Any to do something like that. That guard is useful but it could complicate signatures coming from java (although afaik because its erased that mean it doesnt really matter)
That may be a useful way of declaring that a type can’t contain null, but right now erased is afaik experimental so maybe we don’t want to couch this bound behind an experimental gate.
Besides, why I discarded it there was because for that specific code snippet it was wrong.

public interface JBox {
    // code was slightly incorrect copied from the KEEP, so I'm fixing it here
    <T extends @Nullable Object> void put(@NonNull T t);
}

In a null marked scope, this is the canonical jSpecify representation of this code:

public interface JBox {
  <T extends Object UNION_NULL> void put(T MINUS_NULL t);
}

Again for almost every thing MINUS_NULL is the same as NO_CHANGE but for type parameters that’s not true.

The main use case is interop. That’s the reason Kotlin implemented this. Of course one may say “who cares about interop” but there are cases where a NonNull type would be useful and if that type existed then a MINUS_NULL mode could naturally fall out of that with T & NonNull.

5

an in-built NotNull seems like the easiest solution, but would proper generic type negation be a better solution for this? i feel like it would solve the problem by enhancing scala’s type expressiveness in a way that is more broadly useful, and without further entrenching the idea that Null is a unicorn

not sure of the best operator for this but something like:

type NotNull[T] = T & !Null

type FunnyAnimal = Echidna | Pangolin | ...
type Animal = FunnyAnimal | /* boring... */ Aardvark | Alpaca | ...
type AnyThatIsNotBoringAnimal = !Animal | FunnyAnimal
6

From what I remember, while type negation is totally doable in theory, it’s very hard to integrate into an existing type system, even moreso one as complex as Scala’s

7

yeah I figured so, which is why I didn’t end up proposing anything like full difference types. A magic type would probably be the best solution here. Type negation would be interesting but any change to the type system as fundamental as adding a negation type would be very painful. Seems like TypeScript has wanted to do this but the PR that did it never got finished so it’s likely a mess. TypeScript’s system is probably less complex than Scala considering our match types and the like, so I highly doubt any sane implementation will work.

A NonNull type defined as “Any without Null” should be very easy to make magically in the compiler though. If in the future our compiler writers decide to subject them to endless months of torture a full negation type could be implemented.

8

I think that;s not quite right. Type negation is totally doable in theory, but we try to keep Scala’s type system as simple as possible, So any addition has to show it carries its weight relative to the additional complexity this causes, and type negation has not yet made this case. My main activity over the last 20 years has been to fight complex additions to the type system, so that we will not arrive at a point where we say “cant add this since the type system is too complex to understand already”.

5 Likes
9

i guess the examples should be a bit more involved to avoid giving too simple solutions in the proposed new placement of null in type hierarchy.

e.g.

// assuming current type system under explicit-nulls,
//          where null is neither anyval nor anyref
type NonNull = AnyVal | AnyRef
type MinusNull[T] = T & NonNull

trait MyTrait[T] { // equivalent to MyTrait[T <: Any] ???
  def myMethod(nullableArg: T, nonNullableArg: MinusNull[T]): Unit
}

would that work, especially as an example?

10

yeah that would probably be a better example.

public interface Foo<T extends @Nullable Object> {
    void someMethod(T parametricNullableArg, @NonNull T nonNullableArg);
}

note the parametric-ness of the first arg - it’s as nullable the T of Foo.

This also reminds me of the fact of bounds in jSpecify.

Consider this interface:

@NullMarked
public interface Bar<T> {}

What’s the upper bound of T? Well, it’s @NonNull Object of course! (In Scala this would be NonNull or whatever we decide on)
Obviously this may cause issues in current code if we were to do this in null-marked scopes but its Technically Correct (the best kind of correct!)
For things extending a @Nullable Object then the Scala bound would not be specified (or just T <: Any)
by default in null unmarked scopes we’d probably just give up and let Any take the wheel

Scala bounds can already do this kind of thing but its usually suggested to not explicitly bound a type as accepting null, as you can just do T | Null in the signature instead. That usually makes type inference tolerate whatever shenanagins is going on in the input type.

I unfortunately do not see many usecases for definitely non-nullable types in the wild. The one thing I found was this post for a kind of scuffed case, so maybe we shouldn’t pay attention to that.

However, Kotlin and jSpecify make heavy use of upper bounds that aren’t nullable, which Scala won’t have with Null <: AnyVal, so that’s what a NonNull type would solve. With intersection types the non-nullable types falls out, so it’s kind of free - all that the compiler would do (if it felt like it) is do more implicit null interop by correctly making anything marked as @NonNull intersect with the NonNull type (and probably have an optimization to detect if X & NonNull =:= X and to skip if it does).

Honestly I’d like to know why it’s not more common in Scala to have this kind of bound aside from “it’s not really natively supported”. There are good examples of functions that really ought to reject null as an input, or some other constraint related to nullabilty should be upheld.
You could possibly consider 0.nn nonsensical.
Lets give nn this signature instead:

extension [T <: NonNull](x: T | Null) inline def nn: x.type & T = ???

Assuming the compilers inference is good enough to derive the T, then the compiler now rejects 0.nn. I don’t know if kotlin’s !! does this but if I had to guess it probably does.
There are probably more examples but I’m very sleepy. If there are other examples where a NonNull bound would be useful I’d like to see them.

11

No, it wouldn’t. It would infer T = Int (which is <: NonNull), and then Int <: Int | Null so you can pass 0 as a valid argument to an Int | Null.

That’s the thing, there aren’t really good examples of functions, or type signatures, that should reject null. In general, there aren’t good examples to reject specific types.

Functions want to accept certain inputs, because they want to perform certain operations on them (like call a given method). Rejecting specific types does not give the function any new capability. Rejecting means a) less freedom for the user b) without any new capability/freedom for the implementation. That is a lose-lose situation for everyone, so this is virtually never what we should do.

The only use case that people bring up that requires rejecting specific values is to use those values (typically null) as sentinels. I have argued at Make Null a subclass of AnyVal under -Yexplicit-nulls - #56 by sjrd that this is a misuse, and that there are better alternatives that don’t require rejecting specific values.

It’s unfortunate that JSpecify defaults Foo<T> to mean T extends @NonNull Object (technically Object NO_CHANGE). As I argued previously, IMO there is almost never a good reason to restrict null out of a general type parameter, so they’re choosing the IMO-wrong interpretation by default.

At the same time, because it’s probably the wrong thing to do anyway, I also think it’s not a big deal if we don’t represent it exactly. It’s probably fine to translate a JSpecify Object NO_CHANGE to our <FromJavaObject>. The latter allows any S <: <FromJavaObject>, including if Null <: S. Yes, it means Scala would allow instantiating a type parameter to a nullable type, although JSpecify said it was not allowed, oh well :man_shrugging:.

I don’t think adding NonNull will carry its weight. It would only be useful for Java interop cases, and then only for Object.

For other classes, Scala interprets them as non-nullable anyway. For Object mentioned in Scala code, it is non-nullable as well. It’s only a Java interop issue because we translate Object mentioned in Java code to <FromJavaObject>.

Perhaps a path forward for JSpecify-annotated code would be to introduce <FromJavaObjectNonNull>. Then, like we have the rule:

  • S <: <FromJavaObject> if S <: Any

we could potentially add something like

  • S <: <FromJavaObjectNonNull> if S <: Any and not Null <: S.

That would contain the “NonNull problem” to Java interop.

With that path forward, there’s no real need to solve the problem now. We can decide to add that later if experience actually shows that it is really an issue. I expect that interpreting Object NO_CHANGE as our existing <FromJavaObject> will result in a better experience than trying to shoehorn NonNull in there.

1 Like
12

saying that java interop doesn’t matter much feels weird to me as the whole scala ecosystem is tremendously dependent on java standard library, either in original or reimplemented (api-compatible) form.

initial java value types pull request is being reviewed https://github.com/openjdk/jdk/pull/31120 and the nullness markers JEP draft: Null-Restricted Value Class Types (Preview) JEP draft: Null-Restricted and Nullable Types (Preview) will be follow-ups sometime in the future. even though it says It is not a goal (at this time) to apply the language enhancements to the standard libraries, it can be assumed that in the future java standard library will get sprinkled with nullness markers, as they are designed to be backwards compatible like java generics.

additionally, JSpecify nullness design FAQ | JSpecify says:

OpenJDK has been participating in JSpecify, and we are in regular communication about the prospects for a longer-term language feature.

so we can probably expect similar challenges in interop with either jspecify or with future java nullness markers.

one thing that would be silly if nullness markers are poorly approximated is losing jvm optimizations. since there are no official nullness-marker based optimizations yet to test, it’s hard to come up with examples and benchmarks, but java in recent years gained some high performance stuff that can depend on small details (like whether a var handle is strictly static and immutable, etc). such high performance stuff is obviously under project valhalla, but also project panama (foreign memory, ffi, vector api that implemented portable simd implementation for huge speedups in certain scenarios).

4 Likes
13

Given it seems like the main counter-argument to your proposal is “it wouldn’t be useful”,
I think it would help the discussion if you could enter more into details around the kind of issues you have been experiencing

While a good argument in general, I feel like it doesn’t really apply in the case of interop
If many java devs use null as a sentinel, so much so that there is tooling around accommodating it, then we might need to add tools to make interacting with these libraries easier
(While discouraging their use away from the interop boundary)

Maybe this is too much of a tangent, but it reminds me of a lot of the wayland situation.
Some features where not implemented because “this is the wrong way to do things” so now some apps don’t run well on wayland. For example applications aren’t/weren’t allowed to place their windows as “this is the compositor’s job”, but this means multi-window applications (think drawing app with canvas on one window, brushes on another) are super frustrating on wayland.

1 Like
14

While I in general agree that from a point of view of the functions, rejecting certain values do not make sense. However, from a usability standpoint, I disagree. I have found a few cases where I want to overload a bunch of functions like map and flatmap for values which are nullable (or db nullable in this case).

However, I don’t want to map, filter and flatmap functions on all values of all types. It makes it harder to reason about these functions, if you can also use them on non nullable types.

Here’s the code defining these functions as a reference.

As a workaround for not being able to define functions on A | SqlNull, I define an opaque type and function creating this opaque type and define the rest of the functions on this opaque type, like this.

object Nullable:
  opaque type NullableSyntax[A] = A | SqlNull
  def syntax[A](v: A | SqlNull)(using NotGiven[SqlNull <:< A]): NullableSyntax[A & v.type] = v

  extension [A](v: NullableSyntax[A]) def toOption: Option[A] = v match
    case SqlNull => None
    case _       => Some(v.unsafeGet)

Usage looks like this Nullable.syntax(v).toOption.

Everything around this is ugly, and I wish I could think of a better way, or that Scala offered a better way. If Scala offered a way to reject certain types from calling a function, this would solve a lot of this.

Again, I agree that functionally, none of the code there actually cares if a non nullable value was widened to become nullable.

However, I, as the programmer writing this code do care about not being offered functions I don’t care about, that are not relevant to my current values. I don’t want to wonder why (fa: SomeMonad[A]).map(v => doSomething(v)) is returning a value of type SomeMonad[A] | SqlNull, while also failing to compile, and then figure out it’s because it widened SomeMonad[A] to SomeMonad[A] | SqlNull, and then picked the wrong extension method.

15

This sounds like you don’t want the static type to widen, rather than rejecting certain values.

The opaque type also seems like a pretty good solution to me, if you want to prevent widening? You will always need to have something like syntax, because explicitly declaring when methods are available seems to be what you want?

You could try something like this to keep the “some” case look like an A:

  object SqlNull
  opaque type SqlSome[A] <: A = A
  type Nullable[A] = SqlSome[A] | SqlNull.type
  
  def some[A](a: A): SqlSome[A] = a

Scastie

16

Yes it does–it gives the capability to .toString without worrying about nontermination.

Within Scala, your suggestion works fine…because we get to have the compiler insert itself into the story.

But we can’t insert the Scala compiler into other people’s stories.

And so it is nice to be able to say, “Within the Scala compiler, we have verified that this thing is not null, so we can hand it off to something that faceplants if it is null”.

null is an ugly wart in the type hierarchy. It promises normal operation, and yet never delivers normal operation. That’s pretty bad. This is the sort of thing that you would hope that types would help you keep track of.

But there are two types of solution. One is wholly within-Scala: “Can we make null bite us less”. There, you have an answer that looks like yes to me. I like that one.

But another concern is, “Can we make null bite others less.” And if we cannot easily express “not null”, then the answer is, “no, others get bitten.” That isn’t a very satisfying answer.

1 Like
17

I would like to echo what @tarsa has said, that “null as sentinel value” is a big enough justification for needing a way to represent non-null types in the type system. Even if it’s just to interop with Java, I don’t think we can simply ignore the degree to which Scala depends on existing Java code.

18

For all it’s worth, if the PR to add NullMarked support gets merged and I upgrade to the version with the fix, most of the issues (metals override autocomplete causing flexible type spam) would be fixed.

In the wild I do see Kotlin libraries restricting their input types. It seems Kotlin’s extension syntax rejects null by default as well (doing fun String.foo() rejects null, need to do fun String?.foo() for null) which makes sense, because most functions that take a receiver usually want the receiver to be non-null (of course this is already achievable with explicit nulls now for concrete types).

There are quite a few use cases that could benefit from a nonnull type, I think the reason we don’t see it more in Scala right now is because it’s so obscure (AnyVal | AnyRef is a mouthful). I think discounting it as “not useful” isn’t really helpful - it’s never really had an opportunity to shine in Scala.

We haven’t mentioned making Null !<: Any in this thread, but the general sentiment was that “it would nuke the type system”. My main issue I see with that is we need Null <: Matchable and we aren’t adding “actually universal traits” to Scala anytime soon considering how “exciting” the base feature was. It would arguably make more sense from a ideological standpoint (or whatever nerdy standpoint I’m actually trying to say here) but that would be such a major change to the type system (even more than explicit nulls already does) that it would likely be more effort than its worth.

19

I think this is by design !
null is just very clunky to use and dangerous, so instead the goal was to provide features which cover all its use-cases in a safer way
For example Option[T] which is a substitute for making things optional through T | null (better: Can be nested, supports nice operations, etc)

Personally, I think we could make it work, but I don’t think we would get any benefit:

NonNull Null !<: Any
Any Any | Null
NonNull Any
Int Int
Int | Null Int | Null
[T] is implicitly [T <: Any] [T] is implicitly either [T <: Any] or [T <: Any | Null]
Nothing Nothing

So the biggest difference is actually the default upper bound to type parameters, and whether it includes Null or not, something we could do more directly with “[T] now means [T <: NonNull]”
(But I don’t think we should, and it would break a lot of code, for example List(null, null))

20

yeah I think you’re right about it being by design. That makes sense to me, we don’t want people using it, especially when valhalla comes around and basically makes any performance qualms about Option moot. Should’ve thought a bit more before saying that :slight_smile: