At this point there is no decision nor active discussion about an alternative for scala-reflect in Scala 3.
There are opinions that:
Reflection is not important, everything may be done at compile time
All the use cases for reflection can be satisfied by a macro
At the same time many projects rely on scala-reflect at the moment. Among these projects are Spark, doobie, etc, etc.
Also not all the reflection usages can be covered in compile time. Our project, distage – a module system with a solver and garbage collector (or a DI mechanism if you prefer) – represents instances required for an application as a graph and reshapes the graph according to user input.
Across all the reflections APIs two things have the highest importance: equality check (=:=) and a subtype check (<:<).
I’ve tried to implement a macro plus a runtime to re-implement these basic operations and got it working. My model is not perfect but despite the belief that subtype check is deadly hard to implement, it’s good enough for many practical purposes. But, unfortunately, I cannot make it perfect with just a macro (I don’t have enough data to provide complete support for path-dependent types).
I’ve described my experience in an article. That experience is kinda controversial - the model works and it’s just 2K LoC. But it’s kinda complicated and relies on several unspecified behaviours in macro APIs. So I don’t think that custom macro-based approach is a right way to go.
Despite all its implementation issues scala-reflect is a very attractive and important feature for many engineers and companies.
So, I wish to try to initiate a discussion about implementing a sane alternative to TypeTag in Scala 3. Of course it shouldn’t expose internal compiler data structures and implementation details, should be type-safe and sound. But feature set may be significantly reduced also. Just equality and subtype checks would cover more than a half of reflection usecases.
This may be an issue where perfect is the enemy of good.
What you’re suggesting is that a simplified runtime type system (for 80% of runtime type checks) is something you still want for your use case. scala-reflect meant to be a 100% solution, and wound up having LOTS of issues because of it. TypeTag is actually one of those issues.
The goal in Scala 3 is what we had before scala-reflect. For use-cases where you only need an 80% solution, you should be able to accomplish that with straight up Java reflection. If you need more, TASTY can provide you the basics. However, we don’t think a 100% solution is something folks need, and it’s unclear if there should be a “core” implementation that is not 100%.
The library you aimed to provide is an example of what we’d like to see come out of dropping scala-reflect. It’d be nice to have a runtime-reflection library that does not implement the EXACT scala type system, but is good enough for the 80% case. Additionally, it should fully document where it “cheats” and what limitations it has. You may have a good start at something that the community could build on for reflection. My only comment is that you might be able to make use of the TASTY data encoded in the class (plus some Java reflection) to grab your type tags from any object, without even needing to grab them in a macro if you wanted to be more reflective (although personally I prefer a tagged approach).
Scala 3 has the quoted package, with quoted.Expr as a representation of expressions and quoted.Type as a representation of types. quoted.Type esentially replaces TypeTag. It does not have the same API but has similar functionality. It should be easier to use since it integrates well with quoted terms and pattern matching.
What you’re suggesting is that a simplified runtime type system (for 80% of runtime type checks) is something you still want for your use case.
Not just this. There may be different views on the problem (including “nothing needs to be done”), so I just wished to make a discussion and explore what people think.
scala-reflect meant to be a 100% solution, and wound up having LOTS of issues because of it. TypeTag is actually one of those issues.
Actually scala-reflect was never an 100% solution - there were issues with PDTs at least - they were just unsound (and most likely there was no way to make them completely sound).
For use-cases where you only need an 80% solution, you should be able to accomplish that with straight up Java reflection.
Unfortunately no. Java reflection can’t even handle erasure, but in distage we need things like type tags for unapplied types - which even scala-reflect cannot provide out of the box.
If you need more, TASTY can provide you the basics.
But from what I can understand there is no way to access/manipulate TASTY data at runtime for now?
It’d be nice to have a runtime-reflection library that does not implement the EXACT Scala type system, but is good enough for the 80% case.
Okay, I made one and it is good enough to be useful…
Additionally, it should fully document where it “cheats” and what limitations it has.
… and it’s not a problem to document the pitfalls - it doesn’t handle PDTs properly and it ignores type boundaries. But I’m relying on undocumented behaviours! Actually I made a pretty sane support for boundaries but a necessary undocumented behaviour was broken in 2.13
you might be able to make use of the TASTY data encoded in the class (plus some Java reflection) to grab your type tags from any object
Could you give me any hint where should I look at to explore this?
Thanks for pointing out. Though from what I can see at this point (0.17/0.18) Type does not provide any functionality. Also the following simple example works on 0.17 but doesn’t compile on 0.18, it can’t find QuoteContext for some reason:
Could you give me a hint where may I read/discuss future plans for this functionality? Also - is it going to be possible to get a Type instance for an unapplied type (List[_]) and then apply it to another type (Int)?
In general - seems like this is what we need, it just needs to be improved and equality/subtype checks needs to be implemented.
This QuoteContext context that is missing contains the reflection logic (i.e. some compiler logic). If you use it at runtime you can get a QuoteContext within a run or withQuoteContext (see http://dotty.epfl.ch/docs/reference/metaprogramming/staging.html#api). These will move to a different package soon, but no other API change will happen for now.
The code you wrote could be updated as follows
import scala.quoted._
// import scala.quoted._ // probable new package name for `run`, `withQuoteContext` and `Toolbox`
object Test {
delegate for Toolbox = Toolbox.make(getClass.getClassLoader)
def tag[F[_]: Type] given QuoteContext: String = {
the[Type[F]].show
}
def main(args: Array[String]): Unit = {
println(withQuoteContext { tag[List] })
}
}
It is possible to apply List[_] to Int. For example we could adapt the previous example to
def tag[F[_]: Type] given QuoteContext: String = {
val appliedF = '[F[Int]]
appliedF.show
}
The subtype check is currently possible trough the low level TASTy reflect API. Though it would be possible to have it as well in directly on quoted types. For example we could implement it like
import scala.quoted._
import scala.quoted.staging._
object Test {
delegate for Toolbox = Toolbox.make(getClass.getClassLoader)
def isSubtype[T1, T2] given (t1: Type[T1], t2: Type[T2]) given (qctx: QuoteContext): Boolean = {
import qctx.tasty._
val tpe1: qctx.tasty.Type = t1.unseal.tpe
val tpe2: qctx.tasty.Type = t2.unseal.tpe
tpe1 <:< tpe2
}
def main(args: Array[String]): Unit = {
println(withQuoteContext { isSubtype[List[Int], List[Any]] }) // true
println(withQuoteContext { isSubtype[List[Int], Vector[Int]] }) // false
}
}
It would require a large subset of the compiler to work on on ScalaJS. That might be difficult or even impossible. Form the staging framework side it might be possible to use withQuoteContext to analyze the types could potentially work after I remove some reflective calls. But that would never work for run.
Please consider seriously providing scala-reflect in Scala 3. ScalaJack is a good example of a deeply reflective project/use-case (provides frictionless JSON serialization). It reflects deeply on Scala classes to understand their structure, types, and lots of other info in order to best understand how to serialize to/from JSON and other formats.
This information cannot be accomplished compile-time. I honestly can’t say if it can be done w/macros or not. (I can’t do it with macros at this point, as my understanding of them is not solid.) I’m open to learn, but do harbor suspicions. ScalaJack’s real power is that it doesn’t know what you’re going to throw at it until runtime. Today ScalaJack lives 'n dies by the TypeTag, not just equality checking (we use that too).
BTW, I’m 100% ok cleaning up scala-reflect to make it a little more friendly for use case such as mine. It works great, but sometimes it does feel like drilling for oil trying to divine the structure of complex types.
Simple example:
trait Pet{ name: String }
case class Cat(name: String, isPurring: Boolean) extends Pet
case class Dog(name: String) extends Pet
case class Pets(all:List[Pet])
val pets = Pets(List(Cat("Slinky",true),Dog("Spot"))
val sj = ScalaJack()
val js = sj.render(p)
// renders: {"all":[{"_hint":"com.mypackage.Cat","name":"Slinky","isPurring":true},{"_hint":"com.mypackage.Dog","name":"Spot"}]}
val orig = sj.read[Pets](js) // builds the original object hierarchy using the hints
ScalaJack doesn’t know until runtime the concrete types of the pets in the list until runtime, and it needs this to properly handle the type, in this case rendering a type hint field into the JSON.
This is a very simple example but this sort of thing is used extensively to support ScalaJack’s functionality over arbitrarily complex input types. Don’t even get me started on the internal gymnastics required to untangle all the different collections, especially now that CanBuildFrom is deprecated! Let’s just say that’s deep into the TypeTag weeds.
==> Update: I’ve spent a few days reading through the Macros and related docs for Scala 3. I’m still unclear now these mechanisms would support my use case. The essence of the “Pet problem” is this: (at runtime) given a list of Pet, generate JSON for each concrete Pet class, along with type hints. This means knowing the concrete type of each Pet in the list at runtime, and then extracting the case class constructor fields for these types. If I’m wrong and the new mechanisms can do that, can anyone point me in a good direction/sample?
I have a use-case in Quill which is very similar to your Pets example. If this LightTypeTag can be used to summon implicits and be passed around at runtime, I would be glad to use it. For Dotty-Quill, I specifically want to separate out Quotation from db-specific Contexts but that means I need to pass the types from quoted blocks (e.g. val quotedStuff = quote { stuff }) to executions (e.g. run(quotedStuff)). When quotedStuff is inline, I can just get types from the Tasty tree, when quotedStuff is runtime (e.g. for dynamic queries) that approach doesn’t work and the only way to go is to have runtime-passable TypeTags… which Dotty currently doesn’t have.
Currently we have several blockers for complete izumi-reflect implementation (#8520, #8514, #8764). I really hope they can be fixed in foreseeable future.
Though right now our last release correctly handles =:= and <:< works for many simple cases when no type constructors with variant argument are involved.
tasty-reflect development is short-handed. If you think there’s a useful internal compiler method that should be exposed in the tasty-reflect API like baseClasses, I suggest making a PR yourself to add it instead of waiting for someone else to do it, here’s an example of how to add something to tasty-reflect: https://github.com/lampepfl/dotty/commit/7e8014ebb26d54bf3c589321e4f2c83de44339a0
