SLC: `Conversion[A, Option[A]]`

Scala Standard Library
1

Hi,

we currently have two ways to represent optional values in Scala: Option[A] and A | Null. I think it’s safe to say that the former is generally preferable: it can be nested and it has a useful API with methods like getOrElse, fold etc.. Therefore I’d like to generally reserve A | Null for interoperability (Java/JS/native) use cases.

However A | Null has an advantage: it’s more convenient for methods with optional parameters. def foo(x: Int | Null = null) can be called as foo(42), while def foo(x: Option[Int] = None) needs to be called as foo(Some(42)).

But we can easily get rid of the wrapping: add given[A] => Conversion[A, Option[A]] = Some(_) to the Option companion object. I used to think that having this conversion is bad because then you’d be able to call Option methods on all objects. But that’s not the case when you place the conversion in the companion object: it’ll only be applicable in places where an Option is expected anyway. And on top of that, implicit conversions will soon require into, making any unwanted conversions even less likely. Unintended use of this conversion seems extremely unlikely under these conditions.

We could also go one step further: add given[A] => Conversion[A | Null, Option[A]] = Option(_) to the Option companion object. This would make interop a little easier, allowing you to use nullable references from Java/JS/native in places where Option is expected.

What do you think?

2 
given foo: [A] => Conversion[A | Null, Option[A]] = ???

well i think for one this is expensive as it is a non-monomorphic conversion, so a new Conversion will be created at each callsite - (but perhaps we can cache an erased version).

But yeah a maybe off-topic comment from me: conversion use cases like this make me very skeptical about deprecating implicit def (or we should provide a successor in the given syntax that introduces the same use-site requirement to import language.implicitConversions, which is itself not required for into parameters)

3

This was something of an FAQ back in Scala 2 days — I would expect there are multiple threads about it in the forum archives (this forum, and the older ones when we were on Google Groups).

But I don’t think it’s been freshly considered in the Scala 3 era.

4

If the concern is performance (because of the wrapper allocation), perhaps there’s a way for the compiler to emit more efficient bytecode for this without changing the language design…?

1 Like
5

I think another factor to consider is the mental overhead of reading the code with implicit conversions.

Probably when a programmer reads some code then they expect that a passed argument has the same logical meaning as an accepted argument in a method. Most conversions serve the purpose of making structurally different interfaces in code act interchangeably e.g. Java List and Scala Seq are practically the same thing. But T and Option[T] are not the same thing on a logical level. Implicit construction of things makes code a lot harder to understand. I am not sure that it is a reasonable price to pay for a little convenience in interop.

Also, Option is widely used in pure Scala codebases whereas Java lists are used almost exclusively for interop.

6

I generally think of that as the traditional argument against this sort of thing (and one that did get rehashed a hundred times in the implicit def era): it’s getting to slightly uncomfortable levels of “magic”. The code makes intuitive sense, and is nicely concise, but sometimes leaves people scratching their heads when they try to understand how it actually works.

Once we have into, though, it becomes a more interesting question. In theory, that signals “there might be a conversion involved here”. Is that going to be effective in removing confusion? I’m honestly not sure. But once that’s in mainline Scala post-3.9, this seems worth serious consideration.

As with so much, though, I’d probably recommend that folks try it out in userland and get some real experience with the ergonomics, before we start seriously talking about it in stdlib.

(All of the above about the Conversion[A, Option[A]], note – I’m significantly less comfortable about the version that adds null-wrapping simultaneously.)

1 Like
7

One major obstacle to this proposal, even in a Scala 3 context with into, is that you will never get people to agree on the semantics. You mentioned both alternatives:

  • A => Option[A] = Some(_)
  • A | Null => Option[A] = Option(_)

The latter is not just “going further” than the former. They are incompatible. A can be instantiated to String | Null for the former and String for the latter, and then passing null would behave differently. The former gives Some(null), the latter None.

I would defend the former interpretation, as that is the only type-parametric alternative. And if you’re going to do something implicitly, it had better be type-parametric. Otherwise you’re at the mercy of type inference to know what run-time semantics you actually get. But I suspect many users will advocate for nothing else but the second alternative, for convenience’s sake.

2 Likes
8

Thank you for bringing this up. This is the best kind of comment: the one that makes you think – and so here are my thoughts.

People not being able to agree on which one is better is definitely a possibility. But as far as I’m concerned, both of them would be better than the status quo, simply because they behave identically in the common case where the convertee isn’t nullable. Nulls just aren’t that common in Scala. Not picking either of them reminds me of the story of Buridan’s donkey, who was standing between two identical haystacks, and, because there was no rational reason to choose either one over the other, ended up starving to death.

Now let’s look at possible use cases. Since both variants work the same for non-nullable types, we only need to compare how they work for nullable types.
Converting from a nullable type to Option is idiomatic Scala, it’s what you’re supposed to do to deal with nullable values that you might get from Java or native APIs. Nulls inside Options on the other hand are a code smell: by using the Option type in the first place, you’re creating the expectation on the reader’s side that absent values are represented that way – only to then have their code explode in their face due to a null value that was hiding inside a Some. Now that’s what actual malice in programming looks like. I don’t see this as a convenience issue so much (although the null-eliminating variant is more convenient) but rather as a choice between two possibilities:

  • one that implicitly does something nasty when confronted with a null value (hiding it inside an Option)
  • one that doesn’t do the nasty thing and on top of that also enables a very reasonable use case: converting nullable values to an Option.

So what exactly is the tradeoff here?

Regarding abstract principles like parametricity, I agree that they are of huge value because they do steer us towards better designs most of the time. But you have to look at where you end up when following such principles, and when it’s not a good place, then abstract principles need to give way.
I’d also like to make the point that it’s actually not this Conversion that’s breaking parametricity, it’s union types that don’t behave parametrically. But for better or worse, they’re now part of Scala, so we might as well put them to use.

Regarding your point about type inference, I’m afraid I don’t quite understand it. Depending on which variant we pick, nulls are either always converted to None or always wrapped in Some. It never depends on type inference, so how are we at its mercy? I only see one issue (though I struggle to even call it that): the null-eliminating variant can be used even in places where an Option[A | Null] type is expected, but it will never actually create a Some(null) value, even though the type would admit that. Honestly, I think that is perfectly acceptable. Not only would the user have to use an Option[A | Null] type – which is questionable code in itself – on top of that they’d have to mark it into[…] as well in order to get anywhere near surprising behaviour. This is not something I’m going to lose any sleep over. My conclusion is that we should pick the null-eliminating conversion – it gives us everything the other one does, minus the landmine.

9

Yes, I think the performance problem can easily be dealt with using a static Conversion[Any, Option[Any]] and a cast.

Actually I’ve had some thoughts about this before. This issue doesn’t only affect Conversion, it would actually be quite nice if Scala supported polymorphic val or object declarations (or givens). Unfortunately it’s not so easy to make that work because you’d have to be quite restrictive. For example, any kind of state might somehow refer to a mutable object, which instantly breaks soundness. But it’s certainly an interesting idea to entertain (in another thread, preferably).

Regarding the comments about magic and mental overhead: I’m not buying it. Eliminating Some(…) doesn’t increase mental overhead, it does the opposite. Mental overhead is low when the signal-to-noise ratio is high. In Some(42), 42 is signal, Some(…) is noise, because all Some tells you is that a value is present – but I already know that because I see the value.

10

Lean4 has this, you can pass an x where some x is expected and have the wrapping implicit. That’s a good sign that this is a reasonable feature to add.

1 Like
11

I am against this proposal.

It would make call-sites behave weirdly:

def foo[T](x: Option[T]) = x

val x = 4

foo(x)       // Some(4)
foo(Some(x)) // Some(4)
// They return the same value

val y = Some(4)

foo(y)       // Some(4)
foo(Some(y)) // Some(Some(4))
// They don't return the same value

def forwarder[T](x: T) = foo(x)

forwarder(x) // Some(4) == foo(x)
forwarder(y) // Some(Some(4)) != foo(y) !!!
// Creating a forwarder changes semantics

To me this is already a deal breaker, currently I don’t need to reason about what is going on:
Either it works, or it doesn’t compile.
With this change, there might be tricky edge cases when I need to reason about “Will this parameter get wrapped ?”
Especially it looks like seemingly safe refactoring would actually change the semantics

I also 100% agree with this
And furthermore, I feel like no matter which of the two conversions we take, people will complain that we didn’t take the other one
(Either the “I want strong guarantees camp” or the “I just want it to work for obvious cases” camp)

Finally the one given use case,

can I think be better accomplished by adding some syntactic sugar:

def foo[T](x: T?): Option[T] = x // x has type Option[T] just like `T*` are `Seq[T]`

// Make the call-site both clear and predictible
foo() // None
foo(1) // Some(1)
foo(Some(1)) // Some(Some(1))

// Still allows full control when needed
foo(?None) // None
foo(?Some(1)) // Some(1)
foo(?Option.when(myCondition)(myValue)) // depends on myCondition
foo(?xs.headOption)

(I am currently writing a Pre-SIP for this, but it will still take some time, as I’m looking into all the edge-cases)

2 Likes
12

(just FYI, it may resolve some of your problems without the big language changes)

if it comes to the boxing, it is possible to define lightweight Option

if it comes to the auto wrapping method parameters, AVSystem invented sth like OptArg

def meth(a: OptArg[Int] = OptArg.Empty, b: OptArg[String] = OptArg.Empty) = 
  a.foreach(println)
  b.foreach(println)

meth() // 
meth(4) // 4
meth(b = "ok") // "ok"
meth(3, "hi") // 3 \n "hi"

Implicit conversion for such a popular type as Option will have unpredictable results.

13

I’m not sure I agree, I used Lean a lot and wrote a lot of proofs in it; Lean is designed for mathematics, and mathematics has a ridiculously high amount of implicit knowledge and conversions that mathematicians do in their heads all the time without worrying about things like performance, runtime surprises, readability of the proof-code, etc. So Lean has so much automatic implicit conversion all over the place because of its special needs and goals. Programmers’ needs are quite different I think.

1 Like
14

I very much agree with this.

The thing is, getting rid of the wrapping is inherently the difference between tagged and untagged unions. So if you don’t want to wrap your values, use union types.

I think its rare to want the option API on this kind of default value, pattern matching is clearer. And wrapping in option is just one call away, which is not too bad within the method.

If you worry about bad interactions with null and union types, use a private object as the other value (or just use an overloaded method). You likely only need one in your library that you can reuse for all methods, if you make sure to never return it anywhere.

15

It’s also designed for functional programming (stdlib HTTP server landed a few months ago btw!), and I assume that their decision to make x sugar to some x was done for convenient programming, regardless of the motivations for other implicit designs in the language.

Anyway, I have never found this sugar confusing in Lean, which I use for general programming and not mathematics. This behavior is one of the niceties I miss whenever I come back to Scala from Lean. I think people in this thread are exaggerating how confusing this could be, relative to the convenience it will bring, as they always do anytime any cool feature is ever proposed.

I read through Sporarum’s callsite example, and I don’t think it’s that confusing or that code like this will be very likely, compared to how much no longer needing to write Some(x) everywhere will save in normal everyday code.

1 Like
16

i’m properly surprised by this. is this an issue with implicit conversion or am i misunderstanding something? it’s seemingly deciding whether to convert based the presence of a literal type constructor and not on the type of the value itself, but Implicit Conversions | Tour of Scala | Scala Documentation says ‘Implicit conversions are applied if an expression e is of type S, and S does not conform to the expression’s expected type T’

assuming implicit conversions are correct, what if we had a way to say given [A !<: Option] => Conversion[A, Option[A]] = Some(_)? a type negation if you will :face_with_tongue: would that solve the problem? it feels like it would but i clearly don’t know the technical details properly

17

i really like your idea generally. it would simplify using Option, making it more likeable for those used to null, and also kinda match the now-familiar optional type syntax seen in other languages

however i feel it would be better if you made some changes:

  1. the alternating the side of the ? bothers me so much :laughing: left side makes the most sense to me as it would be read as optional thing everywhere and that matches how it’s currently written:
    def foo[T](x: ?T): ?T = x // foo[T](x: Option[T]): Option[T]
foo(?val) // foo(Some(val))
  2. don’t make it some magic that seemingly allows the callee to change the behaviour of the call site (i feel this is always bad as the caller should retain full control, not be force into writing in some code mode seen nowhere else). rather just make it a short form for Option everywhere. that would allow it to be used anywhere a type is required and allow infinite nesting. like so:
     // same behaviour as now
 foo() // error
 foo(1) // error
 foo(None) // None
 foo(Some(1)) // Some(1)
 foo(Option.when(myCondition)(myValue)) // Some(myValue) | None
 foo(xs.headOption) // Some(xs.head) | None

 // new behaviour
 foo(? /* ie. Nothing */) // None
 foo(?1) // Some(1)
 foo(?Option.when(myCondition)(myValue)) // Some(Some(myValue) | None)
 foo(if myCondition then ?myValue : ?) // Some(myValue) | None

 // nesting
 def unwrapTwice[T](x: ??T): ?T = x.value.value // foo[T](x: Option[Option[T]]): Option[T]
 unwrapTwice(??2 /* Some(Some(2)) */) // 2
 unwrapTwice(??? /* Some(Some(Some(None))) */) // Some(None)

so ? is always read as ‘optional’ and, depending on context, behaves like Option[T] or Option() except without the unusual ‘null as sentinel’ behaviour of Option.apply

only problem is the None form conflicts with existing ??? and ? at the start of identifiers. not sure if these are solvable. possibly context is sufficient, or possibly change ??? to something else and deprecate ? at the start of identifiers. if not possible then try ?! or just stick with None

otherwise i feel like it works well, simplifies optional types a decent amount, and doesn’t invert the thought process so should be pretty easy for anyone to understand. thoughts?

18

I hadn’t tested it, I have now, and I confirm this is what is going on: Scastie - An interactive playground for Scala.

The reason is that methods are type-checked locally, for example:

def example[T](x: T) =
  val y: String = x
  ()

example[String]("Hi")

Doesn’t type check, since example has no information about T, val y: String = x is not valid.
But you might object that it should type-check, since there is no usage of foo where we don’t pass String.
It’s totally possible to make a type-checker that works like that, IIRC that’s what C++ does.
But there are downsides to doing it that way:

  1. It’s possible to write library functions which typecheck and look correct, but which cannot be called (no combination of type parameters satisfy all the internal requirements)
  2. The compiler has to type-check the body for every type-instantiation of the method (whereas scalac can just type check it once, and know it is valid for all instantiations)
  3. It means you need to reason (and therefore have access) to the whole body of a method to know what instantiations are valid (in Scala only the signature matters)

But we can make example compile, by moving the constraint to its signature:

def example[T <: String](x: T) = // T is guaranteed to contain only strings
  val y: String = x
  ()

example[String]("Hi")

To come back to our original example, hopefully you now have intuition on why:

def forwarder[T](x: T) = 
  // T is not <: Option[?], foo(x) doesn't compile
  foo(x)
  // Oh, but we have an implicit conversion, 
  // let's secretly replace this with `foo(Some(x))`to make it type-check

forwarder(Some(1)) // Some(Some(1))

And again, if we move the constraint to the signature, we can avoid the implicit conversion:

def forwarder[T <: Option[?]](x: T) = 
  foo(x)

forwarder(Some(1)) // Some(1)

(Scastie with the above examples)

I think this is in part why I and others in this thread have such an instinctive “this smells really bad” reaction to this proposal: We’ve been burned by things like this before

Well sadly locality would byte us again:

def forwarder[T](x: T) =
  // cannot prove that T <: Option[?] => can't call foo directly
  // cannot prove that T !<: Option[?] => can't apply the implicit conversion
  foo(x)

And this is not only a type checker issue, implicit conversions are real methods that need to be called, so we would need to either specialize the calls, or check the type at runtime (which is complex (impossible?) because of erasure)

19

I think a conversion from A to Option[A] would be useful, but we should add it only when into is more established. Which for me means that conversions either need a language import or an into target. Right now, use of a conversion without either only gives a feature warning, which is toothless.

The problem with introducing the conversion too early is that then everyone would use it immediately - it’s just too useful not to - and then would get bitten when we tighten the screws on mandatory into. So let’s do mandatory into first, and then consider this conversion.

3 Likes
20

It will continue to be the case that it either works or doesn’t compile. The reason your example compiles is because it’s incomplete: it doesn’t show what is later done with the values returned by those functions. The moment you actually do anything meaningful with those values, you’re pretty much guaranteed to get a type error. And if you don’t use it, then you must be calling the function for side effects – but then you’re not going to use a generic Option[T] because you can’t perform meaningful side effects with an unconstrained T object.

Besides, I don’t even understand which of the two is supposed to be confusing. forwarder(x) is just a weird way to write Some(x). foo is just the identity function for Options. Which one of these is meant to be confusing? I know what you’re going to say: “well, it’s just an example to demonstrate a larger point”. My response to that is: if you can’t intentionally come up with a realistic example of the larger issue, how am I supposed to believe that the larger issue even exists, and more importantly, that people might write this kind of code unintentionally?

Your foo(x) example won’t even compile any more very soon because it’s not an into[…] parameter. If you think it’s confusing – just don’t use into. There’s a reason into was added, and it’s exactly because it allows you to exercise your judgement about whether or not to admit conversions.

Besides, the assumption that forwarder(x) should always behave identically to foo(x) doesn’t generally hold once implicit conversions are in the picture – that’s true of any given Conversion, not just this one. Being opposed to implicit conversions in general is a fair position, but that’s a different discussion from whether this particular Conversion is a good idea. There’s also a simple way to avoid it: def forwarder[A](a: Option[A]) = foo(a), and then the conversion will happen at the call site if necessary.

If we’re going to give up on everything that somebody might some day complain about, we might as well pack up and go home.

That’s a false dichotomy, because the Option(_) variant does offer a strong guarantee: it’ll never put a null inside your Option. That’s much more useful than whatever guarantee the Some(_) variant offers. Which, frankly, I don’t know what it’s even supposed to be. Parametricity roughly means an operation behaves the same for all types. That is true for both variants: the Some(_) variant never eliminates null, the Option(_) one always does. Separately, to the extent there’s any parametricity wrinkle at all here, it comes from union types not being parametric, not from this Conversion. I don’t think it’s reasonable to expect the standard library to solve parametricity problems that result from the language itself.

Allow me to ask: how many use cases do you expect for a Conversion? Because you’re making it sound like one (very common) use case isn’t enough. But actually, avoiding explicit conversions is the only thing a Conversion is supposed to accomplish. What else would you expect?

I’m not a fan of adding more language features for something that can be achieved with one line of standard library code.

I agree, adding this Conversion once into is required is what I had in mind to begin with. I should have been more explicit about that.