It would be more obviously wrong in for-comprehensions:
for {
a <- fa
_ = throw new RuntimeException(a.toString)
} yield a
// or
for {
a <- fa
} yield throw new RuntimeException(a.toString)
Either one is pretty obviously strange. The point is that async/await syntax encourages people to use imperative patterns within code which is actually monadic. That’s good! That’s the point of the syntax, but it means that things like this are inherent to the problem space.
To clarify further: when I say it’s being “handled incorrectly” I mean “it’s really not going to behave well, or even consistently, even on the same types”.
What I mean is you can’t really detect pure vs impure code, so if you want to handle this properly, you have to assume that all expressions are impure. You would basically never use pure anymore unless looking at a literal. In general this turns into effect types, which are very clearly out of scope for monadless (or -Xasync).
For the record, F#'s Computation Expressions are precisely that. Just like do-notation, it’s a syntactic sugar for using certain functions that avoids nesting. But it’s a very good one, generalized (beyond what Haskell or Scala currently provide) and a very polished one.
It’s so good, that only blindly copying F# in this regard would improve Scala’s situation immensely. Imagine then what we could achieve, if we put our mind to it! And now’s the best time to do it, since we’re in the 2.x → 3 transition. As I’ve written above, Scala can improve a lot by taking inspiration from other programming languages.
Is there any way we can get some thoughts from the upstream team? Particularly @retronym would be great to hear from. Basically what I’d like to find out is what the general thoughts are on this situation. Is it something that is considered worthwhile to resolve, or is the API behind -Xasync considered relatively final?
I completely agree with all of this. The thing is that this is exactly what Cats and Cats Effect do. They have answers for this (e.g. rewrite foreach as traverse), but without making the transformation entirely dependent on Cats Effect, it’s difficult or impossible to come up with any reliable answer here. It’s the same problem as for-comprehensions, which are treated as a purely syntactic transformation, but that in turn denies them access to the pure function and the knowledge that map(id) <-> id, which in turn gives rise to most of the warts (withFilter et al).
Note that I’m not saying that for-comprehensions should become dependent on cats.Monad, just that if they were, then they would be a lot more reliable at what they do. Similarly, if -Xasync were dependent on Cats Effect, it would have a reliable answer to Li Haoyi’s concerns, but since it isn’t (and it shouldn’t be), these concerns are 100% valid and real.
As an aside, it’s worth noting for anyone considering it that building -Xasync (or Monadless) with a dependency on Cats Effect would actually rule out Future itself, since Future doesn’t form a lawful monad, and the ways in which it is unlawful are extremely visible to users and amplified by something like -Xasync, particularly if you attempt to solve the higher-order function problem.
I don’t think the fact that it’s a syntactic transformation is necessarily the problem here, it’s that the syntactic constructs and underlying operations that those constructs are translated to in the current for comprehension aren’t sufficient to avoid those warts.
I don’t think it’s as hopeless as this
Look again at the F# computation expressions. The way that it works is substantially similar to the way the for translation works in Scala. However there are more operations supported by the translation than those supported by Scala’s for comprehension, perhaps enough to support libraries similar to Cats Effect.
See for example the translation of while expressions:
{ while expr do cexpr } builder.While(fun () -> expr, builder.Delay({ cexpr }))
The requirements to use the while construct in a computation expression are that the underlying builder object implements the Delay: (unit -> M<'T>) -> M<'T> and While (unit -> bool) * M<'T> -> M<'T> methods. I think that it would be really interesting to see how far we could get by supporting a richer vocabulary of operations using a syntactic transformation like @sideeffffect suggets.
It’s not so hard to imagine translating while(cond) { a <- expr } to something implemented using whileM. Also, the fact that it’s a syntactic transformation enables all of the implicit constraints required by libraries like Cats Effect to be used.
Another approach that seems interesting is prototyping a replacement for the for translation using the new metaprogramming system in Scala 3 as this would enable the translation to make more use of type information.
What if instead of the compiler trying to provide a one-size-fits-all AST transformation based on a bunch of assumptions that it sounds like are necessarily too narrow, there was some way to plug in the AST transformation itself. IIUC the benefit of this approach is that it’s a different compiler phase. Couldn’t that somehow be opened up though?
In any case what’s the plan for Dotty? It’s supposed to be around the corner. Maybe there’s a bigger picture approach that should be taken somehow?
I guess opening up a custom compiler phase would end up being pretty similar to writing the transformation as a macro, so I’m not sure if it would be worthwhile to provide a custom transformation via a different mechanism. I could be wrong though! I guess that it’s basically an open research question as to whether there’s a bigger picture approach.
This is a way, chooses in dotty-cps-async: GitHub - rssh/dotty-cps-async: experimental CPS transformer for dotty.
For now, it supports all scala constructions, all monads, which implements appropriative typeclass with some interoperability between and even some support of high-order functions.
Here is an introduction talk from ScalaR: https://www.youtube.com/watch?v=ImlUuTQUeaQ
// (sorry for no-documentation, I have guessed that we have some time before dotty release, plan to start to write manual during next weekend slot.)
Any patches, help, feedback, experiments will be accepted with gratitude.
It’s worth noting that dotty-cps-async suffers from the same problems as -Xasync, albeit in a more elegant way: spawn/fulfill are fundamentally unimplementable operations for any non-eager, non-memoizing monad. Here again the mistake is in assuming that everything is Future-like. Though, ironically fulfill takes it a step further still because it’s blocking, meaning that anyone using dotty-cps-async is giving up their asynchrony.
Don’t get me wrong, it’s a worthwhile effort, but I honestly think that it might be better to take a step back here. Cats Effect has spent years refining abstractions for this stuff and quantifying, in general, what it means to be an asynchronous effectful computation. It’s almost certainly easier to start from it, at the very least as a point of reference and inspiration, rather than attempting to reinvent such abstractions from scratch.
I do like @yangbo’s proposal btw, though I need to read through it more to understand it better. Perhaps it’s a more promising future direction.
But spawn/fullfill is not used in cps-transform at all.
In 1-st iteration of design they was created for interoperability with Future (i.e. if you want use in one expression Future and you monad). But in latest version, currently only asyncCallback is using for this. Spawn/fulfill should be refactored to other interface. (which is needed for channels, which is not in scope of scala async, but used in tests now).
To be clear: we define 3 interfaces:
CpsMonad: map, flatMap, pure
CpsTryMonad: error/restore
CpsAsyncMonad: adoptCallbackStyle, spawn, fullfill
In transform, CpsMonad is used for all except try/catch; CpsTryMonad - for try/catch, CpsAsyncMonad not used at all.
In interoperability: CpsAsyncMonad.adoptCallbackStyle when we want mix our monad and Future
Ah if spawn/fulfill aren’t being used, then it’s basically Monadless, which I’m okay with. Just as long as you understand those functions are both a) extremely dangerous, and b) unimplementable for things that aren’t literally Future. I don’t see how you can handle rewrites over higher-order functions though without some more machinery (e.g. Traverse in Cats).
I spent a bit of time thinking about how straightforward it would be to use the approach that is used by F# computation expressions and to me it seems kind of promising. Writing the underlying builder class looks like it could be pretty straightforward, and the translation of the syntax within a block doesn’t have to be so complicated, see e.g.:
My thoughts are that we get a few advantages from copying the F# approach:
Most of the new Scala developers I have worked with are totally mystified by the idea that for can be overloaded to work on non-collection type things. The builder syntax makes it a bit more clear that something magical is happening and that it’s not just the normal for loop that they’re used to.
Doesn’t really use any new keywords as we can reuse old ones inside the block (e.g. for, while, do)
Individual method signatures can be as complicated and constrained as required by the underlying libraries
Builders can be as polymorphic or as specific as we want, e.g. we can still write an expression builder against F if we want
and a few disadvantages:
You would really need to specify the F you are interested explicitly in if there are any non-trivial constraints involved rather than letting type inference do the work
The fact it’s a syntactic transformation limits optimization opportunities a bit as you can’t do dependency analysis of the expressions, but this is probably OK because ordering is significant here anyway
We really can’t write something as generic as the original for comprehension using this approach - the builder class has to typecheck
Is reusing syntax like for a good idea? Then expressions that use the results of traverse look pretty weird due to multiple <-, i.e.
async[Task] {
deploymentIds <- for (name <- verticleNames) {
Vertx.deployVerticleL(name)
}
do StdIn.readLine("Press enter to shut down")
for (deploymentId <- deploymentIds) {
Vertx.undeployVerticleL(deploymentId)
}
}
Some interesting questions:
How to let the compiler know that this builder class triggers the desugaring? Extending a magic trait and other similar things seem kind of gross.
Could this be implemented by making the builder factory method a Scala 3 macro that accepts the block of code to desugar? Would that limit the syntax that could be used within the block to things that superficially typecheck before the macro expansion or would they just have to parse?
I prefer to say ‘applicable only in imperative scenarios’ instead ‘extremely dangerous’. Exists one case, where it’s not Future but have a high practical value: it’s a ‘something like Future,’ running inside one-threaded dispatch. When we know that we are always in the same thread, then we can avoid operations, that flush processor cache [volatile access, CAS… etc.] and, as a result - have some speedup over plain Future. [No tests for dotty-cps-async yet, but I remember experiments on 2.12 with something like 10-x difference]. Although I know that Future in 2.13.x also was optimized for a one-thread case, so maybe now the difference is not so promising. Another potential example is a Loom-monad when we will able to introduce after Loom will be available. Speed of such monad will be in-between Future and one-thread case: (Loom dispatch contains volatile access).
Naming, of course, can be better.
currently, we assume that the developer can implement a ‘shifted’ version of the high-order method and bind one with object type via typeclass. And provide such shifted versions for the classes defined in the standard library. Also, it is possible to translate lambda-expressions automatically.
We don’t provide rewrites of third-party high-order functions automatically now. We implement AsyncShifted[T] for Iterable[T], Collection[T] … etc. Monadless approach for transform can be represented as something like ‘generation of given Shifted[T] from Traverable[T]’. , which potentially can be implemented on top of dotty-cps-async, but this is a big separate direction.
Also exists two potential other approaches, each with its own set of gains and drawbacks.
The third notable difference from ‘Monadless’ is ‘ghost interfaces.’ We run after the typing phase, so we should use real interfaces. If map/ flatMap can be defined as inline, we can hope that compiler will inline one, but I yet not played with this.
Also, I have thought about passing ‘rewriters’ to macro, instead of interfaces, but faced with the impossibility to pass non-trivial argument (tree rewriter) from the context of macro application to macro.
I mean, fulfill really is dangerous, imperative or otherwise, because of the thread blockage. In fact, it’s even more dangerous in a single-threaded environment (like ScalaJS) since it will actually deadlock. Literally the only circumstances under which fulfill is not dangerous are either a) no use of Promise whatsoever in the entire application (i.e. so Future is basically Try in this scenario), or b) running under Loom.
Blocking functions are simply not viable. They are dangerous, regardless of whether the stylistic context is imperative or functional.
Ah I’ll have to look. That sounds a lot like Cats’ Traverse, which is basically what you would need to do something like this in any case (assuming the goal is to avoid blocking).
This is very nice to be sure, though this benefit disappears a bit the moment anything is abstracted. If you, for example, wrote a hypothetical layer which adapted dotty-cps-async to Cats Effect, certainly dotty-cps-async would not have any visible interfaces in the emitted bytecode, but Cats Effect would. This is exactly the same as Monadless. The only case where dotty-cps-async is able to inline things that Monadless (and similar) cannot is where the dotty-cps-async adapters (e.g. AsyncMonad) are directly implemented for a specific concrete type (e.g. Future).
Please moderators, feel free to move it to