Funnily enough I just run into this problem of the compiler inferring
List[TraitX with Serializable with Product]
I would definitely like Serializable and Product removed from case classes. What if you want some of your leaf classes to be case classes and some not to be leaf classes? However I have long thought it would bee good if traits and classes could be marked as non-inferable and require an explicit type ascription. That could then be applied to other types like AnyVal.
I disagree with this WRT Serializable. Using Product as a LUB is reasonable, as all subclasses extend it.
The biggest Serializable headaches I’ve had were when working with Spark, and they all had the same root cause: Serializable isn’t an interface which can be verified at compile time, it’s a promise made by the programmer, and automatically inferring this guarantees nothing.
It’s entirely possible, even easy, to create an ADT where all of the values are known at compile time (no generics), the compiler “knows” the classes extend Serializable, and the ADT cannot be serialized.
Backwards compatibility concerns may prevent removing this inference, but I don’t see how this could be considered “correct” behavior.
How about having each case class extend a trait CaseClassBase, which extends Product with Serializable?
There’s an open issue tracking this: 1799.
The TL;DR is:
Why would Serializable be incorrect? Are you saying case classes do not extend Serializable, or are you just saying that Serializable is not useful?
One possibility would be to use a special form of extension. For lack of a better idea, let’s use the protected keyword for now. E.g.
class C extends A, protected Serializable, protected Product { ... }
class D extends A, protected Serializable, protected Product { ... }
We could then specify that upper bounds don’t take protected superclasses into account. So we’d
infer type A for x in
val x = if (???) C() else D()
instead of A & Serializable & Product. But normal subtyping would not depend on protectedness. E.g. the following would work OK:
def f(p: Product) = ...
f(C())
The extension syntax is admittedly clunky, but that should be not a big issue since most of the problematic cases are auto-generated anyway from case classes and enum cases.
~~
A separate question is whether protected inheritance should mean anything in addition to “omit from upper bounds”. One possible refinement could be to require that a protected superclass only adds protected members to its subclass. I.e. the following would be OK
trait A { def a: Int }
trait B { def a: Int = ???; protected def b: Int = ??? }
class C extends A, protected B
Here, the members of C are a and b, and the latter is protected. But if B was defined like this
trait B { def a: Int = ???; def b: Int = ??? }
then the definition of C would give a compile time error, since the public b member comes from a protected superclass.
The advantage of this rule is that it gives us a way to state and check that a trait such as IndexedSeqOptimized is only inherited for performance.
Unlike in C++'s private inheritance, I do not propose to retroactively change the visibility of members of inherited classes. See also the discussion in Allow traits to be transparently or "invisibly" mixed in.
I’m saying it’s not useful because there’s no way to verify that it’s correct to infer Serializable unless you actually attempt serialization.
For example:
Welcome to Scala 2.12.6 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_121).
Type in expressions for evaluation. Or try :help.
scala> :paste
// Entering paste mode (ctrl-D to finish)
class NotSerializable(val value: String)
sealed abstract class Example
case class ErrorMessage(msg: NotSerializable) extends Example
case object Done extends Example
// Exiting paste mode, now interpreting.
defined class NotSerializable
defined class Example
defined class ErrorMessage
defined object Done
scala> val serializableInferred = List(ErrorMessage(new NotSerializable("oops!")), Done)
serializableInferred: List[Product with Serializable with Example] = List(ErrorMessage(NotSerializable@136ccbfe), Done)
scala> val os = new java.io.ByteArrayOutputStream()
os: java.io.ByteArrayOutputStream =
scala> val oos = new java.io.ObjectOutputStream(os)
oos: java.io.ObjectOutputStream = java.io.ObjectOutputStream@4ea1aa52
scala> oos.writeObject(serializableInferred)
java.io.NotSerializableException: NotSerializable
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at scala.collection.immutable.List$SerializationProxy.writeObject(List.scala:476)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1028)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
... 28 elided
The problem in general with Serializable is that it is a tag type – it adds no capabilities.
There is nothing you can do with a Foo & Serializable that you can’t do with a Foo. Calculating the intersection with it in a LUB is not useful in the sense of static typing and proving correctness during compilation.
I propose that tag types are not inferred unless the alternative is Any.
A tag type is one that adds no capabilities – no methods or members.
Everything you do with a tag type is a runtime check, so if you want to screen for one, use a pattern match.
I don’t think Serializable should be removed from case classes, I think Scala’s inference needs to change. The root of the problem is inference of unfriendly LUB types.
As for Product this is a truly useful trait in many cases, but just noise in others. ADTs automatically extending it could make cleaner LUBs. The ‘protected extends’ proposal or some sort of annotation to provide inference ‘hints’ are appealing to me. The question in my mind is whether that declaration goes with the definition of the type, or at the use site where it is extended from, or some combination of both.
My gut feeling is that the case for the former is strong with tag types or anywhere that the type has too little connection to static type safety.
I think that inferring Any is still preferable, because you have things like -Xlint:infer-any. The problem with Serializable and friends is often that it prevents this warning from firing.
If you suggest that the compiler automatically detects if a type adds no members and then doesn’t infer it, I’m not sure that’s a good idea since you could still have extension methods.
I would prefer the compiler not to infer Any. At least in my code Any as LUB is pretty much invariably an error. Just because the JVM offers silent boxing doesn’t mean that the Scala language has to use it.
Up until now Jvm Scala has been canonical Scala. And that makes sense with Scala.js, because to a large degree the capabilities of the Js platform are a subset of the capabilities of the Jvm. That won’t be the case with Scala Native. Scala Native is potentially not one platform, but many, as well as Linux, Mac and Windows it could eventually support WebAssembly via LLVM. So for example I would like canonical Scala to offer fully powered Structs / Compound value types, that could be just ignored and treated as normal AnyRef classes on Js and also on the Jvm. If Vallhalla arrives with limited Compound Value types then Scala would hopefully be able implement them on the Jvm with its limitations without having to limit Scala capabilities on all platforms.
Anyway I can see an argument for allowing AnyRef to be inferred, although even there I would tend to prefer it to be explicit, but not for inferring Any or AnyVal.
I think that inferring
Anyis still preferable, because you have things like-Xlint:infer-any. The problem withSerializableand friends is often that it prevents this warning from firing.
That is a good point. My thought process had been that inference of a tag type is no worse than Any, and potentially very slightly better. I’d rather be forced to ascribe Any if I’m not in a REPL and have the compiler fail rather than infer it, and its probably the same with a naked tag type that is basically a runtime distinguishable alias for Any or AnyRef.
If you suggest that the compiler automatically detects if a type adds no members and then doesn’t infer it, I’m not sure that’s a good idea since you could still have extension methods.
A tag type has nothing useful at compile time other than the type itself, no capabilities. It is mostly a runtime tag, for dynamic checks. Dotty has opaque types now, which seems to cover most of the compile time needs for a use case where one wants to create one type from another without adding capabilities.
Other JDK tag types (marker interfaces) besides Serializable are Cloneable and Remote. If these were made today, they would probably be implemented with annotations, but the JVM did not have annotations at the time they were created. A special exception could be made for only these, but I’m not convinced that is better than the general case.
Does anyone have any real world use cases of what I’m calling a tag type that are not dependent on runtime checks of the type? Its possible in Scala to do some things at compile time based on this type, but none that I can think of are actually useful given the other tools available, like opaque types.
Wouldn’t it be better to have an -Xlint:infer-serializable?
I don’t do it often, but there are useful examples, largely having to do with state. An example I sometimes teach goes something like this:
case class Book()
trait CheckedIn
trait CheckedOut
def checkout(book: Book with CheckedIn): Book with CheckedOut = ...
def checkin(book: Book with CheckedOut): Book with CheckedIn = ...
It’s different from opaque types, because I’m not trying to hide anything about Book – I’m just literally adding a compile-time tag to it, so that the compiler can enforce that only something in an appropriate state can be passed to certain functions.
A sealed trait or abstract class has the compile time property of exhaustive match checks. When unifying ADT cases, even if some are non-case classes or objects, I do want the common sealed ‘type tag’ to be inferred.
A sealed trait or abstract class has the compile time property of exhaustive match checks. When unifying ADT cases, even if some are non-case classes or objects, I do want the common sealed ‘type tag’ to be inferred.
In Dotty, one would use an enum for an ADT, which would not qualify as a tag type.
As for sealed traits in general, they should not qualify. Although they are not adding capabilities, they are adding restrictions. The same is true of an abstract class with no members or methods – it represents a restriction. What I’m after can be mixed in with anything, not much more than Any. A sealed trait is more than that.
There are probably more restrictions needed to define what I’m after – essentially what Java calls a marker interface.
Re: opaque types – yes, if you don’t want to hide anything they aren’t a good fit.
Given the features available in Dotty, would you still teach the approach above? There is nothing preventing “new Book with CheckedIn with CheckedOut” for example. I would expect an ADT here, or some other structure if one needs to check in and check out various things and can not impose extra restrictions on Book.
The api above implies that one can not check out a naked book, but only one that has the CheckedIn state on it.
Inferring List[CheckedIn | CheckedOut] is not especially useful, if it might have Movies or Books in it, for example.
sealed trait LibraryItem
trait CheckedIn extends LibraryItem
trait CheckedOut extends LibraryItem
case class Movie()
case class Book()
Obvious uses of the above would indicate we need a few more rules:
inferring a sealed type in an intersection is OK
inferring a type that extends from anything other than Any / AnyRef (AnyVal?) OK.
In summary, I think I have to redefine what I’m looking for as a tag trait:
A trait
To use the Book case above and gain inference on CheckedIn and CheckedOut, you would have to make the types a member of a sealed trait or extend another trait. Using a sealed trait would help the use case in any event.
Alternatively, there could be a way to allow such traits to be inferred in intersections by default, something like @inferrable trait CheckedIn, or the opposite @noinfer trait Serializable.
EDIT:
Summary of my position:
Serializable and I suppose Clonable and other Java marker interfaces are examples of use cases where the type is really only useful for a runtime check of some sort. Inferring these when LUBbing is not helpful, and its worse if the LUB contains part of an ADT. I suspect there is a way to categorize these in a more general way than making them special cases, but I may be wrong. I would want to simply not infer them when calculating a LUB. They can be inferred otherwise without issue, as far as I can tell.
Syntax like case trait X that would expand to trait X extends Product with Serializable would solve the problem in easy way.
case trait Base // expands to: trait Base extends Product with Serializable
case class A(x: Int) extends Base
case class B(y: String, z: Char) extends Base
case object C extends Base
val seq = Seq(A(5), B("dd", 'z'), C) // inferred to be Seq[Base]
seq would be inferred to Seq[Base] and not Seq[Base with Product with Serializable] because Product and Serializable are already in Base.
Is there any problem with such approach?
It would certainly solve the issue with inferring Product.
For Serializable, I’d really like for a case class or case object to only extend Serializable if all it’s fields explicitly extend Serializable. It’s not perfect, but it’s a reasonable middle ground between our current situation where Serializable is effectively useless, and the correct-but-infeasible solution that would require explicitly extending Serializable where applicable.
While case trait is more literate than the Scala 2 solution of reminding people to have their ADT rootextends product with serializable, it’s hardly any more effective at preventing the LUB problem. Library code could neglect it, or you could be extending classes other than the ADT root.
Also, we already have a new keyword for marking an ADT root; enum occupies that role.
I would like the opposite be practicioned - only make case classes of (immutable and thus) serializable data. Case classes are a good fit for immutable value classes not for e.g. avoiding new keyword for class with side effects.
Computing LUB has many quirks, I don’t think working around Product and Serializable will solve all LUB surprises. Sample REPL session:
Welcome to Scala 2.12.7 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_201).
Type in expressions for evaluation. Or try :help.
scala> abstract class A extends Ordered[A]
defined class A
scala> abstract class B extends Ordered[B]
defined class B
scala> List(null: A, null: B)
res2: List[scala.math.Ordered[_ >: B with A <: scala.math.Ordered[_ >: B with A <: Object]]] = List(null, null)
res2 got a crazy and useless inferred type.
It doesn’t matter how many supertypes a case class extends as long as it extends at least one common case trait (or analogously: as long as it extends one common trait <name> extends Product with Serializable).
scala> trait T extends Product with Serializable // common trait shared among case classes
defined trait T
scala> case class A() extends T with Ordered[A] with Iterator[Int] { def hasNext: Boolean = ??? ; def next(): Int = ??? ; def compare(that: A): Int = ??? }
defined class A
scala> case class B() extends T with Ordered[B] with Iterator[String] { def hasNext: Boolean = ??? ; def next(): String = ??? ; def compare(that: B): Int = ??? }
defined class B
scala> List(null: A, null: B)
res5: List[T with Iterator[Any] with scala.math.Ordered[_ >: B with A <: T with Iterator[Any] with scala.math.Ordered[_ >: B with A <: T]]] = List(null, null)
res5 got a crazy LUB, but neither Product nor Serializable is explicitly present in it, because T absorbed it.
case is not a new keyword. It’s already used for marking ADT leaves. Marking ADT root with it would make the whole syntax more regular.