Feature proposal: Lightweight TypeTag (and serialising lowering in multi-stage compilation)

[tribbloid]

Following the successful experiment in In multi-stage compilation, should we use a standard serialisation method to ship objects through stages? - #22 by tribbloid, I’d like to publish its outcome as an independent library.

This includes a new way to implement TypeTag that is far more lightweight than izumi-reflect and with none of its limitations.

(see list of these limitations at GitHub - zio/izumi-reflect: TypeTag without scala-reflect. Supports Scala 2 and Scala 3.)

I still can’t decide how to represent the underlying types in serialized form, Type on its own are not serializable in either Scala 2 or 3, I could only think of 2 ways to convert it:

• use the compiler built-in pickler (dotty.tools.dotc.quoted.PickledQuotes and scala.quoted.runtime.QuoteUnpickler) to convert it into string or BLOB.

  • Pros: can be easily reconstructed in multi-stage programming.
  • Cons: almost impossible to be used elsewhere, QuoteUnpickler is only implemented within the context of a compiler.

• use metaprogramming tools to convert it into TASTy type data.

  • Pros: can live independently without a specific compiler instance, it is even compatible with Scala2 tools!
  • Cons: I haven’t figure out how to do this yet, is there a reference library or metaprogramming tool I can look into? Not sure about the relevance of scalameta as it is quite old by now.

Once implemented it will be published regularly as an interop layer between Scala 2 (which already has TypeTag) and 3, old capabilities of TypeTag (e.g. convert to runtime class, pattern matching) can be gradually revived with little effort.

I expect this feature to marginally reduce upgrade difficulty for some frameworks that heavily rely on TypeTag and JVM runtime reflection (Apache Spark, obviously), thanks in advance if you would like to add more use cases and feature suggestions.

[bishabosha]

@tribbloid

Cons: I haven’t figure out how to do this yet, is there a reference library or metaprogramming tool I can look into? Not sure about the relevance of scalameta as it is quite old by now.

there is Tasty Query library, i.e. as long as you have access to the classpath you can recreate the type at runtime

[tribbloid]

thanks a lot @bishabosha. I can’t find any part of that library interacting with scala.quoted.Type:

was I looking into the wrong place? (I’m only looking for a minimal use case to recreate the type at runtime)

[tribbloid]

BTW, the new Anthropic claude3 gave an incorrect answer, I wonder if it is caused by old implementation contaminating the training set:

//---------

In Scala 3, you can convert a scala.quoted.Type into its TASTy (Typed Abstract Syntax Tree) data object using the TypeRepr class from the scala.quoted.runtime.Expr package. Here’s how you can do it:

1. First, make sure you have the necessary dependencies in your project. You’ll need the scala3-compiler and scala3-tasty-inspector dependencies. Add the following to your build.sbt file:

libraryDependencies ++= Seq(
  "org.scala-lang" %% "scala3-compiler" % scalaVersion.value,
  "org.scala-lang" %% "scala3-tasty-inspector" % scalaVersion.value
)

2. Import the required classes and objects:

import scala.quoted._
import scala.quoted.runtime.Expr._
import scala.tasty.inspector._

3. Define a method that takes a scala.quoted.Type and returns its TASTy data object:

def typeToTasty(tpe: Type[_])(using Quotes): TastyData = {
  val typeRepr = TypeRepr.of(using tpe)
  val tastyBytes = typeRepr.asInstanceOf[dotty.tools.dotc.core.Types.TypeRepr].toTasty
  TastyData.unpickle(tastyBytes)
}

In this method, we first convert the Type to a TypeRepr using TypeRepr.of. Then, we cast the TypeRepr to dotty.tools.dotc.core.Types.TypeRepr to access the toTasty method, which returns the TASTy bytes of the type. Finally, we use TastyData.unpickle to convert the TASTy bytes into a TastyData object.

4. You can now use the typeToTasty method to convert a Type into its TASTy data object:

def myMethod[T: Type]()(using Quotes): Unit = {
  val tpe = Type.of[T]
  val tastyData = typeToTasty(tpe)
  // Use the tastyData object as needed
}

In this example, myMethod takes a type parameter T and an implicit Quotes context. It retrieves the Type of T using Type.of[T] and then passes it to the typeToTasty method to obtain the corresponding TastyData object.

Note that the typeToTasty method requires an implicit Quotes context, which is provided by the myMethod in this case.

With the TastyData object, you can inspect and analyze the TASTy representation of the type using the methods provided by the scala.tasty.inspector package.

Remember to handle any necessary error cases and ensure that you have the appropriate dependencies and imports in your project.

[sjrd]

tasty-query has its own tastyquery.Types.Type, which is independent of the compiler and of Quotes.

[tribbloid]

Make sense, TASTy is only a communication protocol and every library is free to interpret it independently. This implies that TypeTag can serialise into it to achieve long-term maintainability.

the relevant code must be in the compiler core (in both Scala 2/3), please give me some time to figure out a roadmap

[tribbloid]

turns out that PickledQuotes is already emitting TASTy binary format:

  private def pickle(tree: Tree)(using Context): Array[Byte] = {
    quotePickling.println(i"**** pickling quote of\n$tree")
    val pickler = new TastyPickler(defn.RootClass)
    val treePkl = new TreePickler(pickler, Attributes.empty)

So there is only 1 roadmap and it is the one I’m already working on

I still have 2 problems:

• All implementations of unpickling appears to be demanding a Context, which is only available within a compiler instance. How can I get one using only current runtime classpath?

• Given a runtime Type, can a JVM class always be reconstructed from its classSymbol? We may have very little option otherwise to interop with some legacy code that rely heavily on Java reflection