I see. I imagined it would be better to implement first in a standalone library. This would make it simpler to implement and try out all serialization framework such as Kryo and upickle in one place. We would be able to implement and stabilize it faster. In a second time we could move the interface definition and the basic java serialization implementation to the standard library.
We could try adding it as experimental in the standard library, but then to cross validate with other serialization libraries we would need to wait full release cicles. I feel this would be much slower and require more work that the first approach.
I could start a repo with this code and add the basic functionality, but might need some help.
2 Likes
Thanks a lot, all make sense, will do & publish
1 Like
finished, special thanks to @nicolasstucki and @DmytroMitin
AutoLift.scala
import java.io.{ByteArrayInputStream, ByteArrayOutputStream, ObjectInputStream, ObjectOutputStream}
import java.util.Base64
import scala.quoted.*
object AutoLift {
trait SerializingAutoLift[T] extends ToExpr[T] {
def apply(x: T)(
using
Quotes
This file has been truncated. show original
this leads to some surprisingly simple macros, e.g. a TypeTag impl under 100 lines.
There is some serialization error:
AutoLiftSpec.scala:11:36: Exception occurred while executing macro expansion.
java.io.NotSerializableException: scala.quoted.runtime.impl.TypeImpl
but they are easy to fix
3 Likes
I’ve seen that attempts to fix the serialization error has already started in dotty.tools.dotc.quoted.PickledQuotes
, good news, I’ll play with it more and see if the lightweight TypeTag can be published
Partially successful! The pickler can move a Type across compilers using the following code:
trait SerialisingLowering[T] extends ToExpr[T] {
def apply(x: T)(
using
Quotes
): Expr[T] = {
val stringsExpr: Expr[Seq[String]] = Varargs(serialize(x).map(Expr(_)))
deserialize(stringsExpr)
}
protected def serialize(x: T): Seq[String]
protected def deserialize(expr: Expr[Seq[String]])(
using
Quotes
): Expr[T]
}
object SerialisingLowering {
object JVMNativeLowering {
@transient lazy val encoder: Base64.Encoder = Base64.getEncoder
@transient lazy val decoder: Base64.Decoder = Base64.getDecoder
private inline val MAX_LITERAL_LENGTH = 32768
protected def deserializeImpl[T](strings: Seq[String]): T = {
val bytes = strings.map(decoder.decode).reduce(_ ++ _)
val bIStream = new ByteArrayInputStream(bytes)
val oIStream = new ObjectInputStream(bIStream)
val v = oIStream.readObject()
v.asInstanceOf[T]
}
def deserialize[T: Type](expr: Expr[Seq[String]])(
using
Quotes
): Expr[T] = {
'{ deserializeImpl[T]($expr) }
}
object Implicits {
given only[T <: Serializable: Type]: JVMNativeLowering[T] = JVMNativeLowering[T]()
}
}
class JVMNativeLowering[T <: Serializable] extends SerialisingLowering[T] {
import JVMNativeLowering.*
override protected def serialize(x: T): Seq[String] = {
val bOStream = new ByteArrayOutputStream()
val oOStream = new ObjectOutputStream(bOStream)
oOStream.writeObject(x)
val serialized = encoder.encodeToString(bOStream.toByteArray)
serialized.sliding(MAX_LITERAL_LENGTH, MAX_LITERAL_LENGTH).toSeq
}
override protected def deserialize(expr: Expr[Seq[String]])(
using
Quotes
): Expr[T] = {
JVMNativeLowering.deserialize(expr)
}
}
}
case class TypeTag[T](
pickle: List[String]
) extends Serializable {
import TypeTag.*
def runtimeClass: Class[T] = ???
def unbox(quotes: Quotes): Type[T] = {
given Quotes = quotes
val q = quotes.asInstanceOf[QuoteUnpickler]
val cucumber: Type[Nothing] = q.unpickleTypeV2(pickle, null)
cucumber.asInstanceOf[Type[T]]
}
}
object TypeTag {
import SerialisingLowering._
def compile[T]()(
using
Type[T],
Quotes
): Expr[TypeTag[T]] = {
val tt = implicitly[Type[T]]
val qq = implicitly[Quotes]
val raw = (tt, qq) match {
case (t: TypeImpl, q: QuotesImpl) =>
given ctx: Context = q.ctx
val tree = t.typeTree
val pickle = PickledQuotes.pickleQuote(tree)
val txt = tree.showIndented(2)
println(txt)
val raw = TypeTag[T](pickle)
raw
case _ =>
???
}
import Lowering.JVMNativeLowering.Implicits.given
// given toExpr: ToExpr[TypeTag[T]] = JVMNativeSerializingToExpr.Implicits.only[TypeTag[T]]
Expr.apply[TypeTag[T]](raw)
}
inline given get[T]: TypeTag[T] = ${ compile[T]() }
}
test code:
object Fixture {
lazy val testCompiler: staging.Compiler = staging.Compiler.make(getClass.getClassLoader)
}
val tag1 = TypeTag.get[Seq[T1[Int]]]
testCompiler.run { q =>
given Quotes = q
given Context = q.asInstanceOf[QuotesImpl].ctx
val t1: Type[Seq[T1[Int]]] = tag1.unbox(q)
val str = t1.asInstanceOf[TypeImpl].typeTree.show
println(str)
'{}
}
this can display the original type: