Design of -Xasync

[eyalroth]

I believe that this post from another thread is quite relevant in here as well:

@lihaoyi

The big problem with compile-time coroutine transformation is things like this: we would need to compile-time coroutine-transform the def foreach higher-order function as well! But we compiled that earlier, so what can we do?

1. Pre-compile two copies of everything in anticipation?
2. Build everything from source and make a closed world assumption?
3. Do post-compile bytecode transformations like the Quasar Fibers project?
4. Try to push it into the underlying runtime like the JDK Fibers project?
5. Define some kind of syntactic/type-driven transform for the callsites of higher-order functions that make them do “the right thing” in most cases?

None of these options are easy. I think (5) is the one promising for the Scala compiler, and I have some ideas/intuition for how it can play out, but it will take effort to spec out and make work. And it’s not just .foreach: idiomatic Scala code uses tons of higher order functions: map, filter, flatMap, fold, takeWhile, getOrElse, getOrElseUpdate, Console.withOut, the whole works.

This is less of a problem in other languages, where higher-order-functions are rarely used, and the program transform just needs to support a finite number of control flow constructs that cover the vast majority of idiomatic code. That is not the case for Scala

This isn’t just an issue for coroutines: it is also a big problem for the usability of async/await syntax in Scala as well, which is just as good as the ones in Python/C#/Javascript/etc., but hampered by Scala’s widespread use of higher-order functions. The old Scala-CPS (continuation-passing-style) compiler plugin also faced this issue, which prevented its usage in any non-trivial Scala code.

If we can figure out the story for how such transforms interact with higher order functions, that would bring us most of the way to making these kinds of syntax transforms possible. Otherwise we’d be basically limiting our transforms to only the most trivial Scala code containing if-else, while-loops, and really not much else. Basically the subset that async/await supports today.