we currently have two ways to represent optional values in Scala: Option[A] and A | Null. I think it’s safe to say that the former is generally preferable: it can be nested and it has a useful API with methods like getOrElse, fold etc.. Therefore I’d like to generally reserve A | Null for interoperability (Java/JS/native) use cases.
However A | Null has an advantage: it’s more convenient for methods with optional parameters. def foo(x: Int | Null = null) can be called as foo(42), while def foo(x: Option[Int] = None) needs to be called as foo(Some(42)).
But we can easily get rid of the wrapping: add given[A] => Conversion[A, Option[A]] = Some(_) to the Option companion object. I used to think that having this conversion is bad because then you’d be able to call Option methods on all objects. But that’s not the case when you place the conversion in the companion object: it’ll only be applicable in places where an Option is expected anyway. And on top of that, implicit conversions will soon require into, making any unwanted conversions even less likely. Unintended use of this conversion seems extremely unlikely under these conditions.
We could also go one step further: add given[A] => Conversion[A | Null, Option[A]] = Option(_) to the Option companion object. This would make interop a little easier, allowing you to use nullable references from Java/JS/native in places where Option is expected.
given foo: [A] => Conversion[A | Null, Option[A]] = ???
well i think for one this is expensive as it is a non-monomorphic conversion, so a new Conversion will be created at each callsite - (but perhaps we can cache an erased version).
But yeah a maybe off-topic comment from me: conversion use cases like this make me very skeptical about deprecating implicit def (or we should provide a successor in the given syntax that introduces the same use-site requirement to import language.implicitConversions, which is itself not required for into parameters)
This was something of an FAQ back in Scala 2 days — I would expect there are multiple threads about it in the forum archives (this forum, and the older ones when we were on Google Groups).
But I don’t think it’s been freshly considered in the Scala 3 era.
If the concern is performance (because of the wrapper allocation), perhaps there’s a way for the compiler to emit more efficient bytecode for this without changing the language design…?
I think another factor to consider is the mental overhead of reading the code with implicit conversions.
Probably when a programmer reads some code then they expect that a passed argument has the same logical meaning as an accepted argument in a method. Most conversions serve the purpose of making structurally different interfaces in code act interchangeably e.g. Java List and Scala Seq are practically the same thing. But T and Option[T] are not the same thing on a logical level. Implicit construction of things makes code a lot harder to understand. I am not sure that it is a reasonable price to pay for a little convenience in interop.
Also, Option is widely used in pure Scala codebases whereas Java lists are used almost exclusively for interop.
I generally think of that as the traditional argument against this sort of thing (and one that did get rehashed a hundred times in the implicit def era): it’s getting to slightly uncomfortable levels of “magic”. The code makes intuitive sense, and is nicely concise, but sometimes leaves people scratching their heads when they try to understand how it actually works.
Once we have into, though, it becomes a more interesting question. In theory, that signals “there might be a conversion involved here”. Is that going to be effective in removing confusion? I’m honestly not sure. But once that’s in mainline Scala post-3.9, this seems worth serious consideration.
As with so much, though, I’d probably recommend that folks try it out in userland and get some real experience with the ergonomics, before we start seriously talking about it in stdlib.
(All of the above about the Conversion[A, Option[A]], note – I’m significantly less comfortable about the version that adds null-wrapping simultaneously.)
One major obstacle to this proposal, even in a Scala 3 context with into, is that you will never get people to agree on the semantics. You mentioned both alternatives:
A => Option[A] = Some(_)
A | Null => Option[A] = Option(_)
The latter is not just “going further” than the former. They are incompatible. A can be instantiated to String | Null for the former and String for the latter, and then passing null would behave differently. The former gives Some(null), the latter None.
I would defend the former interpretation, as that is the only type-parametric alternative. And if you’re going to do something implicitly, it had better be type-parametric. Otherwise you’re at the mercy of type inference to know what run-time semantics you actually get. But I suspect many users will advocate for nothing else but the second alternative, for convenience’s sake.
Thank you for bringing this up. This is the best kind of comment: the one that makes you think – and so here are my thoughts.
People not being able to agree on which one is better is definitely a possibility. But as far as I’m concerned, both of them would be better than the status quo, simply because they behave identically in the common case where the convertee isn’t nullable. Nulls just aren’t that common in Scala. Not picking either of them reminds me of the story of Buridan’s donkey, who was standing between two identical haystacks, and, because there was no rational reason to choose either one over the other, ended up starving to death.
Now let’s look at possible use cases. Since both variants work the same for non-nullable types, we only need to compare how they work for nullable types.
Converting from a nullable type to Option is idiomatic Scala, it’s what you’re supposed to do to deal with nullable values that you might get from Java or native APIs. Nulls inside Options on the other hand are a code smell: by using the Option type in the first place, you’re creating the expectation on the reader’s side that absent values are represented that way – only to then have their code explode in their face due to a null value that was hiding inside a Some. Now that’s what actual malice in programming looks like. I don’t see this as a convenience issue so much (although the null-eliminating variant is more convenient) but rather as a choice between two possibilities:
one that implicitly does something nasty when confronted with a null value (hiding it inside an Option)
one that doesn’t do the nasty thing and on top of that also enables a very reasonable use case: converting nullable values to an Option.
So what exactly is the tradeoff here?
Regarding abstract principles like parametricity, I agree that they are of huge value because they do steer us towards better designs most of the time. But you have to look at where you end up when following such principles, and when it’s not a good place, then abstract principles need to give way.
I’d also like to make the point that it’s actually not this Conversion that’s breaking parametricity, it’s union types that don’t behave parametrically. But for better or worse, they’re now part of Scala, so we might as well put them to use.
Regarding your point about type inference, I’m afraid I don’t quite understand it. Depending on which variant we pick, nulls are either always converted to None or always wrapped in Some. It never depends on type inference, so how are we at its mercy? I only see one issue (though I struggle to even call it that): the null-eliminating variant can be used even in places where an Option[A | Null] type is expected, but it will never actually create a Some(null) value, even though the type would admit that. Honestly, I think that is perfectly acceptable. Not only would the user have to use an Option[A | Null] type – which is questionable code in itself – on top of that they’d have to mark it into[…] as well in order to get anywhere near surprising behaviour. This is not something I’m going to lose any sleep over. My conclusion is that we should pick the null-eliminating conversion – it gives us everything the other one does, minus the landmine.
Yes, I think the performance problem can easily be dealt with using a static Conversion[Any, Option[Any]] and a cast.
Actually I’ve had some thoughts about this before. This issue doesn’t only affect Conversion, it would actually be quite nice if Scala supported polymorphic val or object declarations (or givens). Unfortunately it’s not so easy to make that work because you’d have to be quite restrictive. For example, any kind of state might somehow refer to a mutable object, which instantly breaks soundness. But it’s certainly an interesting idea to entertain (in another thread, preferably).
Regarding the comments about magic and mental overhead: I’m not buying it. Eliminating Some(…) doesn’t increase mental overhead, it does the opposite. Mental overhead is low when the signal-to-noise ratio is high. In Some(42), 42 is signal, Some(…) is noise, because all Some tells you is that a value is present – but I already know that because I see the value.
Lean4 has this, you can pass an x where some x is expected and have the wrapping implicit. That’s a good sign that this is a reasonable feature to add.
def foo[T](x: Option[T]) = x
val x = 4
foo(x) // Some(4)
foo(Some(x)) // Some(4)
// They return the same value
val y = Some(4)
foo(y) // Some(4)
foo(Some(y)) // Some(Some(4))
// They don't return the same value
def forwarder[T](x: T) = foo(x)
forwarder(x) // Some(4) == foo(x)
forwarder(y) // Some(Some(4)) != foo(y) !!!
// Creating a forwarder changes semantics
To me this is already a deal breaker, currently I don’t need to reason about what is going on:
Either it works, or it doesn’t compile.
With this change, there might be tricky edge cases when I need to reason about “Will this parameter get wrapped ?”
Especially it looks like seemingly safe refactoring would actually change the semantics
I also 100% agree with this
And furthermore, I feel like no matter which of the two conversions we take, people will complain that we didn’t take the other one
(Either the “I want strong guarantees camp” or the “I just want it to work for obvious cases” camp)
Finally the one given use case,
can I think be better accomplished by adding some syntactic sugar:
def foo[T](x: T?): Option[T] = x // x has type Option[T] just like `T*` are `Seq[T]`
// Make the call-site both clear and predictible
foo() // None
foo(1) // Some(1)
foo(Some(1)) // Some(Some(1))
// Still allows full control when needed
foo(?None) // None
foo(?Some(1)) // Some(1)
foo(?Option.when(myCondition)(myValue)) // depends on myCondition
foo(?xs.headOption)
(I am currently writing a Pre-SIP for this, but it will still take some time, as I’m looking into all the edge-cases)
I’m not sure I agree, I used Lean a lot and wrote a lot of proofs in it; Lean is designed for mathematics, and mathematics has a ridiculously high amount of implicit knowledge and conversions that mathematicians do in their heads all the time without worrying about things like performance, runtime surprises, readability of the proof-code, etc. So Lean has so much automatic implicit conversion all over the place because of its special needs and goals. Programmers’ needs are quite different I think.
The thing is, getting rid of the wrapping is inherently the difference between tagged and untagged unions. So if you don’t want to wrap your values, use union types.
I think its rare to want the option API on this kind of default value, pattern matching is clearer. And wrapping in option is just one call away, which is not too bad within the method.
If you worry about bad interactions with null and union types, use a private object as the other value (or just use an overloaded method). You likely only need one in your library that you can reuse for all methods, if you make sure to never return it anywhere.
It’s also designed for functional programming (stdlib HTTP server landed a few months ago btw!), and I assume that their decision to make x sugar to some x was done for convenient programming, regardless of the motivations for other implicit designs in the language.
Anyway, I have never found this sugar confusing in Lean, which I use for general programming and not mathematics. This behavior is one of the niceties I miss whenever I come back to Scala from Lean. I think people in this thread are exaggerating how confusing this could be, relative to the convenience it will bring, as they always do anytime any cool feature is ever proposed.
I read through Sporarum’s callsite example, and I don’t think it’s that confusing or that code like this will be very likely, compared to how much no longer needing to write Some(x) everywhere will save in normal everyday code.