"MINUS_NULL" and the future of explicit nulls

[Sporarum]

Actually, maybe we don’t need NonNull at all for interop*:

This is the worst case I could come up with:

public interface JBox {
    <T extends @Nullable Object> void put(
       @NonNull T notNullable,
       T maybeNullable,
       @Nullable T nullable
    );
}

And to my understanding, we can perfectly preserve which arguments are valid/invalid at the cost of a more complicated type clause:

def put[Base <: AnyRef, Aug <: Base | Null](
    notNullable: Base,
    maybeNullable: Aug,
    nullable: Base | Null
)

Example: Scastie - An interactive playground for Scala.

In the case where there is only two of the three parameters, we can instead use a single type parameter.

Otherwise we have to either under- or over-correct:

Under-correct:

def put[T <: AnyRef | Null](
    notNullable: T,
    maybeNullable: T,
    nullable: T | Null
)

jbox.put[Foo       ](Foo(), Foo(), null) // allowed, should be
jbox.put[Foo       ](Foo(), null,  null) // forbidden, should be
jbox.put[Foo | Null](Foo(), null,  null) // allowed, should be
jbox.put[Foo | Null](null,  null,  null) // allowed, should be forbidden !

(equivalent to, but more further from the source: T <: AnyRef, maybeNullable: T | Null)

Over-correct:

def put[T <: AnyRef](
    notNullable: T,
    maybeNullable: T,
    nullable: T | Null
)

jbox.put[Foo       ](Foo(), Foo(), null) // allowed, should be
jbox.put[Foo       ](Foo(), null,  null) // forbidden, should be allowed !
jbox.put[Foo | Null](Foo(), null,  null) // allowed, should be
jbox.put[Foo | Null](null,  null,  null) // forbidden, should be

Note that we can use asInstanceOf to still be able to do the forbidden case:

jbox.put[Foo](Foo(), null.asInstanceOf, null)

*Since we change the type parameter clause depending on the signature, this might cause issues when the java source is updated: Adding parameters/changing nullability

[bulbyvr]

I feel like I did realize earlier that yes this kind of thing would work but I really, really didn’t want to change the number of type parameters, as that ruins any API docs.
Under-correction is a perfectly fine compromise considering how niche this is, but really, it would be nice to perfectly represent the type. Of course that’s not much of an argument considering how much Scala already approximates.

---

As an aside, considering this whole thread is about interop, is there a particular reason that java.lang.Integer and other boxed primitives are considered separate types from scala.Int and the scala versions? I understand why the companion objects are different but Kotlin directly maps int to kotlin.Int, and maps Integer to kotlin.Int? (outside of generics, in generics its kotlin.Int!). Of course it makes some amount of sense why Scala never did this for nullability concerns but this is unironically a segment of code in one of my minecraft mods:

val INT: Codec[Int] = Codec.INT.asInstanceOf[Codec[Int]]
val LONG: Codec[Long] = Codec.LONG.asInstanceOf[Codec[Long]]
val BOOL: Codec[Boolean] = Codec.BOOL.asInstanceOf[Codec[Boolean]]
val BYTE: Codec[Byte] = Codec.BYTE.asInstanceOf[Codec[Byte]]
val SHORT: Codec[Short] = Codec.SHORT.asInstanceOf[Codec[Short]]
val FLOAT: Codec[Float] = Codec.FLOAT.asInstanceOf[Codec[Float]]
val DOUBLE: Codec[Double] = Codec.DOUBLE.asInstanceOf[Codec[Double]]

Complete insanity. Pretty sure in Kotlin you could just directly use the instances on Codec and considering the cast succeeds at runtime and nothing else breaks Scala could very easily accept this too.

[Ichoran]

@bulbyvr

is there a particular reason that java.lang.Integer and other boxed primitives are considered separate types from scala.Int

Yes, because it is incredibly difficult to express what you need to to ensure good performance when you can’t talk about the types! Int is Java int. If you only ever talk about Int, the Scala compiler handles all the boxing and unboxing for you. However, (1) you may not pretend it is actually java Object–it is NOT; that’s why Any exists; and (2) in a generic context it is transported as java.lang.Integer.

scala> def cl[A](a: A) = a.getClass.getName
def cl[A](a: A): String
                                                                                
scala> cl(2)
val res0: String = "java.lang.Integer"

I don’t know how you’ve encoded your Codec, but the code there looks…odd. I don’t know what Codec.INT would be if not a Codec[Int]. That seems at the very least like quite bad naming. If that’s some odd Java thing where it says INT but it means java.lang.Integer, well, don’t blame Scala for Java making a mess!

@bulbyvr

There are quite a few use cases that could benefit from a nonnull type

Yes, but it’s kind of more complicated than people are letting on here, due to opaque types sitting outside of basically the entire typical type hierarchy.

scala> object OpaqueExample:
         opaque type O = Int
         object O:
           def apply(i: Int): O = i
           extension (o: O)
             def value: Int = o

scala> val o = OpaqueExample.O(5)
val o: OpaqueExample.O = 5
                                                                                
scala> val x: AnyVal = o
-- [E007] Type Mismatch Error: -------------------------------------------------
1 |val x: AnyVal = o
  |                ^
  |Found:    (o : OpaqueExample.O)
  |Required: AnyVal
  |Note that implicit conversions were not tried because the result of an implicit conversion
  |must be more specific than AnyVal
  |
  | longer explanation available when compiling with `-explain`
1 error found
                                                                                
scala> val x: AnyRef = o
-- [E007] Type Mismatch Error: -------------------------------------------------
1 |val x: AnyRef = o
  |                ^
  |Found:    (o : OpaqueExample.O)
  |Required: AnyRef
  |Note that implicit conversions were not tried because the result of an implicit conversion
  |must be more specific than AnyRef
  |
  | longer explanation available when compiling with `-explain`
1 error found
                                                                                
scala> def couldbenull[N >: Null](n: N): Unit = println("Could be")
def couldbenull[N >: Null](n: N): Unit
                                                                                
scala> couldbenull(o)
Could be

You would have to be able to declare opaque type O <: NotNull = Int in order to know. But then you would have to have a NotNull that you could speak of.

[sjrd]

@tarsa

saying that java interop doesn’t matter much feels weird to me as the whole scala ecosystem is tremendously dependent on java standard library, either in original or reimplemented (api-compatible) form.

Ah, this is not the message I intended to convey. Of course Java interop is important. What I find not-obviously-very-important is to faithfully represent a bound of java.lang.Object MINUS_NULL in the Scala type system.

It’s worth remembering that java.lang.Object itself is already not faithfully represented. We cannot know whether it’s supposed to mean AnyRef or Any, so we represent it as this weird ambivalent <FromJavaObject> that tries to accommodate both use cases.

This is the only reason we have this discussion/issue. If java.lang.Object unambiguously meant AnyRef, then under explict-nulls, the translation of Object UNION_NULL would be AnyRef | Null and that of Object MINUS_NULL would be AnyRef. Done, easy.

That is what happens for every other class type: String UNION_NULL becomes String | Null while String MINUS_NULL becomes String.

Because we try to accommodate jl.Object being interpreted as Any, it becomes less clear what to do with MINUS_NULL. And this is why I think a <FromJavaObjectNonNull> is perhaps the way to go.

---

@Ichoran

@sjrd

Functions want to accept certain inputs, because they want to perform certain operations on them (like call a given method). Rejecting specific types does not give the function any new capability.

Yes it does–it gives the capability to .toString without worrying about nontermination.

This will independently be fixed by Proposal: Fixing null.methodOfAny() under explicit-nulls

@Ichoran

And so it is nice to be able to say, “Within the Scala compiler, we have verified that this thing is not null, so we can hand it off to something that faceplants if it is null”.

In this direction, we have no problem. If from Scala we know something is an AnyRef, we can tell Java it’s an Object MINUS_NULL. And if it’s an AnyRef | Null, or an Any, we can tell Java it’s an Object UNION_NULL.

The issue exposed in this thread is only about interpreting Java into Scala. Not to communicate from Scala to Java.

@Sporarum

And to my understanding, we can perfectly preserve which arguments are valid/invalid at the cost of a more complicated type clause:

Using a different number of type arguments is a non-starter. The mismatch between the Java API (and its documentation) and Scala’s interpretation and usage would be devastating.

@Sporarum

Otherwise we have to either under- or over-correct:

This is precisely what <FromJavaObject> achieves today: it both under- and over-corrects Object as AnyRef or Any depending on usage.

[sjrd]

@Ichoran

Yes, but it’s kind of more complicated than people are letting on here, due to opaque types sitting outside of basically the entire typical type hierarchy.

scala> object OpaqueExample:
         opaque type O = Int
         object O:
           def apply(i: Int): O = i
           extension (o: O)
             def value: Int = o

scala> val o = OpaqueExample.O(5)
val o: OpaqueExample.O = 5
                                                                                
scala> val x: AnyVal = o
-- [E007] Type Mismatch Error: -------------------------------------------------
1 |val x: AnyVal = o
  |                ^
  |Found:    (o : OpaqueExample.O)
  |Required: AnyVal
  |Note that implicit conversions were not tried because the result of an implicit conversion
  |must be more specific than AnyVal
  |
  | longer explanation available when compiling with `-explain`
1 error found
                                                                                
scala> val x: AnyRef = o
-- [E007] Type Mismatch Error: -------------------------------------------------
1 |val x: AnyRef = o
  |                ^
  |Found:    (o : OpaqueExample.O)
  |Required: AnyRef
  |Note that implicit conversions were not tried because the result of an implicit conversion
  |must be more specific than AnyRef
  |
  | longer explanation available when compiling with `-explain`
1 error found
                                                                                
scala> def couldbenull[N >: Null](n: N): Unit = println("Could be")
def couldbenull[N >: Null](n: N): Unit
                                                                                
scala> couldbenull(o)
Could be

You would have to be able to declare opaque type O <: NotNull = Int in order to know. But then you would have to have a NotNull that you could speak of.

Gosh this is such a good point! It means AnyRef | AnyVal is already today not at all a valid way to represent NotNull.

Therefore, making Null <: AnyVal won’t “destroy the ability to represent NonNull”: that ability already does not exist!

[abosshard]

Hold on. Declaring opaque type O <: AnyVal = Int really would express that O is non-nullable without revealing too much about the RHS type. So I would consider this another use case in favor of keeping Null separate from AnyVal.

Yes, couldbenull(o) would still print “Could be”, but so would couldbenull(5). The reason is that N is inferred as Int | Null.

[sjrd]

That would still require every opaque type alias definition to follow that scheme. Clearly, they don’t.

[abosshard]

@sjrd

That would still require every opaque type alias definition to follow that scheme.

Not really, only in cases where you need it:

@Ichoran

You would have to be able to declare opaque type O <: NotNull = Int in order to know. But then you would have to have a NotNull that you could speak of.

[tarsa]

@sjrd

That would still require every opaque type alias definition to follow that scheme. Clearly, they don’t.

only the opaque types that participate in (null-marked) java interop would need the nonnull bound to have seamless integration. pure scala code can ignore nullness.

opaque types having ambiguous nullness hurts explicit-nulls mode whether null is a direct subtype of any or direct subtype of anyval.

lack of nonnull bound could be mitigated by having following scheme:

type NonNull = AnyRef | AnyVal

extension [T](x: T)
  inline def nn: T & NonNull = { assert(x != null); x.asInstanceOf[T & NonNull] }

it could be used at interop boundaries and would solve the type conflict problem

[Ichoran]

@abosshard

Yes, couldbenull(o) would still print “Could be”, but so would couldbenull(5). The reason is that N is inferred as Int | Null.

Okay, that’s not the best example because of how type widening works. This one is better (though the other has problems too):

scala> object OpaqueExample:
         opaque type O = Int
         object O:
           def apply(i: Int): O = i
           extension (o: O)
             def value: Int = o
         opaque type P = AnyRef | Null
         object P:
           def apply(a: AnyRef | Null): P = a
           extension (p: P)
             def value: (AnyRef | Null) = p
       
// defined object OpaqueExample
                                                                                
scala> couldbenull[OpaqueExample.P](OpaqueExample.P("eel"))
-- [E057] Type Mismatch Error: -------------------------------------------------
1 |couldbenull[OpaqueExample.P](OpaqueExample.P("eel"))
  |                          ^
  |        Type argument OpaqueExample.P does not conform to lower bound Null
  |
  | longer explanation available when compiling with `-explain`
1 error found

Here we have explicitly hidden, with opaque types, that we can wrap null. And so anything we write that could tell us at compile-time will be wrong.

We in fact cannot tell the difference between O and P at all in terms of content, only that “they are somehow different at the type level”. That is the point of opaque types.

And yet, if opaque types are used widely–and they are incredibly useful for keeping things straight where ordinary type distinctions won’t cut it–this really undermines our ability to encode NotNull at the type level, for interacting with other libraries or languages that need it treated specially.

Opaque types also work to our advantage.

scala> object Example:
         opaque type NotNull[A] <: A = A
         object NotNull:
           def of[A](a: A): Option[NotNull[A]] = (a: Any) match
             case null => None
             case _ => Some(a: NotNull[A])

scala> val a: String | Null = "eel"
val a: String | Null = "eel"
                                                                                
scala> val b: String | Null = null
val b: String | Null = null

val res0: Option[Example.NotNull[String]] = Some("eel")
                                                                                
scala> Example.NotNull.of(b)
val res1: Option[Example.NotNull[String]] = None
                                                                                
scala> val p = OpaqueExample.P(null: String | Null)
val p: OpaqueExample.P = null
                                                                                
scala> Example.NotNull.of(p)
val res2: Option[Example.NotNull[OpaqueExample.P]] = None

As long as we don’t cast, we can have an opaque type wrapper that witnesses by checking at runtime (or at compile time with inline summonFrom for cases that can be known statically) that something really truly honestly genuinely is not null.

With transparent inline and summonFrom, you can make NotNull.of(NotNull.of("eel")) reduce to NotNull[String].

So I think the answer is: actually, Scala has all the machinery needed to create a highly robust, fully honest NotNull even in the face of opaque types.

We just need to do it, and maybe hook it up to Java annotations.

[sjrd]

@Ichoran

Opaque types also work to our advantage.

scala> object Example:
         opaque type NotNull[A] <: A = A
         object NotNull:
           def of[A](a: A): Option[NotNull[A]] = (a: Any) match
             case null => None
             case _ => Some(a: NotNull[A])

This faithfully encodes that NonNull[A] can only contain values of type A that are also not null. However, it is not that useful. It does not let you use that knowledge.

As a simple example, for s: NotNull[String | Null], you still can’t call s.substring(1). All you know is that NotNull[String | Null] <: String | Null. You don’t know that it’s actually <: String.

Of course, you won’t directly write that. But you will instantiate a type parameter T that was used in a NotNull[T] signature (coming from Java, hooked up from Java annotations) to T = String | Null. And then you’ll be stuck receiving a NonNull[String | Null] from a Java API and not be able to do anything with it.

Similarly, you can’t assign a NonNull[String | Null] to a String.

IMO, that opaque type NotNull is robust and honest, sure, but also useless in practice.

[Ichoran]

@sjrd

IMO, that opaque type NotNull is robust and honest, sure, but also useless in practice.

Well, it is useful if you might have covert nulls as opaque types. This makes sure they are excluded, so that if you pass them into an API that can’t handle nulls, it won’t get them. It’s an export layer, essentially, while retaining usability.

For Java interop, this would not be the import layer that goes from T | Null to T. You would need something else for that. The question then would be how the compiler presents the information that a Java or Kotlin method has a type parameter, but null is permitted. If that surfaces Kotlin T? in a way that def from[A](a: A | Null) can extract A = T, then you could write the corresponding import layer method, and it would be usable because once you pull it out of the option, NotNull[A] <: A.

But it doesn’t solve the whole problem; on the import side it needs compiler help.