In multi-stage compilation, should we use a standard serialisation method to ship objects through stages?

Language Design Macros
21

I see. I imagined it would be better to implement first in a standalone library. This would make it simpler to implement and try out all serialization framework such as Kryo and upickle in one place. We would be able to implement and stabilize it faster. In a second time we could move the interface definition and the basic java serialization implementation to the standard library.

We could try adding it as experimental in the standard library, but then to cross validate with other serialization libraries we would need to wait full release cicles. I feel this would be much slower and require more work that the first approach.

I could start a repo with this code and add the basic functionality, but might need some help.

2 Likes
22

Thanks a lot, all make sense, will do & publish

1 Like
23

finished, special thanks to @nicolasstucki and @DmytroMitin

this leads to some surprisingly simple macros, e.g. a TypeTag impl under 100 lines.

There is some serialization error:

AutoLiftSpec.scala:11:36: Exception occurred while executing macro expansion.
java.io.NotSerializableException: scala.quoted.runtime.impl.TypeImpl

but they are easy to fix

3 Likes
24

I’ve seen that attempts to fix the serialization error has already started in dotty.tools.dotc.quoted.PickledQuotes, good news, I’ll play with it more and see if the lightweight TypeTag can be published

25

Partially successful! The pickler can move a Type across compilers using the following code:

trait SerialisingLowering[T] extends ToExpr[T] {

  def apply(x: T)(
      using
      Quotes
  ): Expr[T] = {
    val stringsExpr: Expr[Seq[String]] = Varargs(serialize(x).map(Expr(_)))

    deserialize(stringsExpr)
  }

  protected def serialize(x: T): Seq[String]

  protected def deserialize(expr: Expr[Seq[String]])(
      using
      Quotes
  ): Expr[T]
}

object SerialisingLowering {

  object JVMNativeLowering {

    @transient lazy val encoder: Base64.Encoder = Base64.getEncoder
    @transient lazy val decoder: Base64.Decoder = Base64.getDecoder

    private inline val MAX_LITERAL_LENGTH = 32768

    protected def deserializeImpl[T](strings: Seq[String]): T = {
      val bytes = strings.map(decoder.decode).reduce(_ ++ _)
      val bIStream = new ByteArrayInputStream(bytes)
      val oIStream = new ObjectInputStream(bIStream)
      val v = oIStream.readObject()
      v.asInstanceOf[T]
    }

    def deserialize[T: Type](expr: Expr[Seq[String]])(
        using
        Quotes
    ): Expr[T] = {

      '{ deserializeImpl[T]($expr) }
    }

    object Implicits {

      given only[T <: Serializable: Type]: JVMNativeLowering[T] = JVMNativeLowering[T]()
    }
  }

  class JVMNativeLowering[T <: Serializable] extends SerialisingLowering[T] {

    import JVMNativeLowering.*

    override protected def serialize(x: T): Seq[String] = {
      val bOStream = new ByteArrayOutputStream()
      val oOStream = new ObjectOutputStream(bOStream)
      oOStream.writeObject(x)
      val serialized = encoder.encodeToString(bOStream.toByteArray)
      serialized.sliding(MAX_LITERAL_LENGTH, MAX_LITERAL_LENGTH).toSeq
    }

    override protected def deserialize(expr: Expr[Seq[String]])(
        using
        Quotes
    ): Expr[T] = {

      JVMNativeLowering.deserialize(expr)
    }
  }

}

case class TypeTag[T](
    pickle: List[String]
) extends Serializable {

  import TypeTag.*

  def runtimeClass: Class[T] = ???

  def unbox(quotes: Quotes): Type[T] = {

    given Quotes = quotes

    val q = quotes.asInstanceOf[QuoteUnpickler]

    val cucumber: Type[Nothing] = q.unpickleTypeV2(pickle, null)

    cucumber.asInstanceOf[Type[T]]
  }
}

object TypeTag {
  import SerialisingLowering._

  def compile[T]()(
      using
      Type[T],
      Quotes
  ): Expr[TypeTag[T]] = {

    val tt = implicitly[Type[T]]
    val qq = implicitly[Quotes]

    val raw = (tt, qq) match {

      case (t: TypeImpl, q: QuotesImpl) =>
        given ctx: Context = q.ctx

        val tree = t.typeTree
        val pickle = PickledQuotes.pickleQuote(tree)

        val txt = tree.showIndented(2)

        println(txt)
        val raw = TypeTag[T](pickle)

        raw
      case _ =>
        ???
    }

    import Lowering.JVMNativeLowering.Implicits.given
    //      given toExpr: ToExpr[TypeTag[T]] = JVMNativeSerializingToExpr.Implicits.only[TypeTag[T]]

    Expr.apply[TypeTag[T]](raw)
  }

  inline given get[T]: TypeTag[T] = ${ compile[T]() }
}

test code:

object Fixture {

  lazy val testCompiler: staging.Compiler = staging.Compiler.make(getClass.getClassLoader)

}


      val tag1 = TypeTag.get[Seq[T1[Int]]]

      testCompiler.run { q =>

        given Quotes = q
        given Context = q.asInstanceOf[QuotesImpl].ctx

        val t1: Type[Seq[T1[Int]]] = tag1.unbox(q)
        val str = t1.asInstanceOf[TypeImpl].typeTree.show
        
        println(str)

        '{}
      }

this can display the original type:

image
image1040×579 30 KB