What happens here? Where does the exception go? Do you require a MonadError[F, Throwable] ? Do you just throw it and hope for the best? Cats Effect guarantees that exceptions are caught in Sync#delay , but it makes no guarantees about flatMap / map (and it is in fact a violation of the functor laws to catch within those combinators, though in practice I think all of the practical IO s do it anyway).
That’s just syntactic sugar for:
fa.map(a => throw new RuntimeException(a.toString))
The behavior depends on the implementation, as expected.
Yeah so that’s a good example of something that’s being handled incorrectly. throw within map is straight-up undefined, and while most implementations of IO-like things will sort of handle it correctly, it isn’t at all guaranteed, and you can get very complicated consequences if you’re composing transformers and such. However, if it were wrapped as fa.flatMap(a => Sync.delay(throw new RuntimeException(a.toString))), then it would be guaranteed to work every time.
I realize that this probably isn’t solvable in general in a layer like Monadless, since really making it work properly (without giving up on pure Applicative/Monad) would require require implementing an effect type system. My suggestion would be to spin off separate syntax for anyone who wants the pure version of async. So something like:
asyncPure {
val a = await(fa)
throw new RuntimeException(a.toString) // undefined, because it's just map
}
async {
val a = await(fa)
throw new RuntimeException(a.toString) // wrapped in Sync
}
The downside is this would mean that the async syntax would require Cats Effect. An alternative solution would be to have an effectful combinator which replaces async and has the Sync wrapping (so, same idea but in reverse). I believe Monadless supports this right now.
Anyway, this discussion is relevant to scala-async because these are the kinds of cases that need to be thought out and handled correctly.
Yeah so that’s a good example of something that’s being handled incorrectly.
How so? It’s just syntactic sugar similarly to for-comprehensions. It’s as wrong as if someone throws in a map function or in an for-comprehension.
I realize that this probably isn’t solvable in general in a layer like Monadless, since really making it work properly (without giving up on pure Applicative / Monad ) would require require implementing an effect type system. My suggestion would be to spin off separate syntax for anyone who wants the pure version of async
That’s interesting, I don’t think it’s as complicated as you describe. Monadless currently doesn’t do any special handling of pure trees but it could be easily modified to wrap them with a method call.
It would be more obviously wrong in for-comprehensions:
for {
a <- fa
_ = throw new RuntimeException(a.toString)
} yield a
// or
for {
a <- fa
} yield throw new RuntimeException(a.toString)
Either one is pretty obviously strange. The point is that async/await syntax encourages people to use imperative patterns within code which is actually monadic. That’s good! That’s the point of the syntax, but it means that things like this are inherent to the problem space.
To clarify further: when I say it’s being “handled incorrectly” I mean “it’s really not going to behave well, or even consistently, even on the same types”.
What I mean is you can’t really detect pure vs impure code, so if you want to handle this properly, you have to assume that all expressions are impure. You would basically never use pure anymore unless looking at a literal. In general this turns into effect types, which are very clearly out of scope for monadless (or -Xasync).
For the record, F#'s Computation Expressions are precisely that. Just like do-notation, it’s a syntactic sugar for using certain functions that avoids nesting. But it’s a very good one, generalized (beyond what Haskell or Scala currently provide) and a very polished one.
It’s so good, that only blindly copying F# in this regard would improve Scala’s situation immensely. Imagine then what we could achieve, if we put our mind to it! And now’s the best time to do it, since we’re in the 2.x → 3 transition. As I’ve written above, Scala can improve a lot by taking inspiration from other programming languages.
Is there any way we can get some thoughts from the upstream team? Particularly @retronym would be great to hear from. Basically what I’d like to find out is what the general thoughts are on this situation. Is it something that is considered worthwhile to resolve, or is the API behind -Xasync considered relatively final?
I completely agree with all of this. The thing is that this is exactly what Cats and Cats Effect do. They have answers for this (e.g. rewrite foreach as traverse), but without making the transformation entirely dependent on Cats Effect, it’s difficult or impossible to come up with any reliable answer here. It’s the same problem as for-comprehensions, which are treated as a purely syntactic transformation, but that in turn denies them access to the pure function and the knowledge that map(id) <-> id, which in turn gives rise to most of the warts (withFilter et al).
Note that I’m not saying that for-comprehensions should become dependent on cats.Monad, just that if they were, then they would be a lot more reliable at what they do. Similarly, if -Xasync were dependent on Cats Effect, it would have a reliable answer to Li Haoyi’s concerns, but since it isn’t (and it shouldn’t be), these concerns are 100% valid and real.
As an aside, it’s worth noting for anyone considering it that building -Xasync (or Monadless) with a dependency on Cats Effect would actually rule out Future itself, since Future doesn’t form a lawful monad, and the ways in which it is unlawful are extremely visible to users and amplified by something like -Xasync, particularly if you attempt to solve the higher-order function problem.
I don’t think the fact that it’s a syntactic transformation is necessarily the problem here, it’s that the syntactic constructs and underlying operations that those constructs are translated to in the current for comprehension aren’t sufficient to avoid those warts.
I don’t think it’s as hopeless as this
Look again at the F# computation expressions. The way that it works is substantially similar to the way the for translation works in Scala. However there are more operations supported by the translation than those supported by Scala’s for comprehension, perhaps enough to support libraries similar to Cats Effect.
See for example the translation of while expressions:
{ while expr do cexpr } builder.While(fun () -> expr, builder.Delay({ cexpr }))
The requirements to use the while construct in a computation expression are that the underlying builder object implements the Delay: (unit -> M<'T>) -> M<'T> and While (unit -> bool) * M<'T> -> M<'T> methods. I think that it would be really interesting to see how far we could get by supporting a richer vocabulary of operations using a syntactic transformation like @sideeffffect suggets.
It’s not so hard to imagine translating while(cond) { a <- expr } to something implemented using whileM. Also, the fact that it’s a syntactic transformation enables all of the implicit constraints required by libraries like Cats Effect to be used.
Another approach that seems interesting is prototyping a replacement for the for translation using the new metaprogramming system in Scala 3 as this would enable the translation to make more use of type information.
What if instead of the compiler trying to provide a one-size-fits-all AST transformation based on a bunch of assumptions that it sounds like are necessarily too narrow, there was some way to plug in the AST transformation itself. IIUC the benefit of this approach is that it’s a different compiler phase. Couldn’t that somehow be opened up though?
In any case what’s the plan for Dotty? It’s supposed to be around the corner. Maybe there’s a bigger picture approach that should be taken somehow?
I guess opening up a custom compiler phase would end up being pretty similar to writing the transformation as a macro, so I’m not sure if it would be worthwhile to provide a custom transformation via a different mechanism. I could be wrong though! I guess that it’s basically an open research question as to whether there’s a bigger picture approach.
This is a way, chooses in dotty-cps-async: GitHub - rssh/dotty-cps-async: experimental CPS transformer for dotty.
For now, it supports all scala constructions, all monads, which implements appropriative typeclass with some interoperability between and even some support of high-order functions.
Here is an introduction talk from ScalaR: https://www.youtube.com/watch?v=ImlUuTQUeaQ
// (sorry for no-documentation, I have guessed that we have some time before dotty release, plan to start to write manual during next weekend slot.)
Any patches, help, feedback, experiments will be accepted with gratitude.
It’s worth noting that dotty-cps-async suffers from the same problems as -Xasync, albeit in a more elegant way: spawn/fulfill are fundamentally unimplementable operations for any non-eager, non-memoizing monad. Here again the mistake is in assuming that everything is Future-like. Though, ironically fulfill takes it a step further still because it’s blocking, meaning that anyone using dotty-cps-async is giving up their asynchrony.
Don’t get me wrong, it’s a worthwhile effort, but I honestly think that it might be better to take a step back here. Cats Effect has spent years refining abstractions for this stuff and quantifying, in general, what it means to be an asynchronous effectful computation. It’s almost certainly easier to start from it, at the very least as a point of reference and inspiration, rather than attempting to reinvent such abstractions from scratch.
I do like @yangbo’s proposal btw, though I need to read through it more to understand it better. Perhaps it’s a more promising future direction.
But spawn/fullfill is not used in cps-transform at all.
In 1-st iteration of design they was created for interoperability with Future (i.e. if you want use in one expression Future and you monad). But in latest version, currently only asyncCallback is using for this. Spawn/fulfill should be refactored to other interface. (which is needed for channels, which is not in scope of scala async, but used in tests now).
To be clear: we define 3 interfaces:
CpsMonad: map, flatMap, pure
CpsTryMonad: error/restore
CpsAsyncMonad: adoptCallbackStyle, spawn, fullfill
In transform, CpsMonad is used for all except try/catch; CpsTryMonad - for try/catch, CpsAsyncMonad not used at all.
In interoperability: CpsAsyncMonad.adoptCallbackStyle when we want mix our monad and Future
Ah if spawn/fulfill aren’t being used, then it’s basically Monadless, which I’m okay with. Just as long as you understand those functions are both a) extremely dangerous, and b) unimplementable for things that aren’t literally Future. I don’t see how you can handle rewrites over higher-order functions though without some more machinery (e.g. Traverse in Cats).
I spent a bit of time thinking about how straightforward it would be to use the approach that is used by F# computation expressions and to me it seems kind of promising. Writing the underlying builder class looks like it could be pretty straightforward, and the translation of the syntax within a block doesn’t have to be so complicated, see e.g.:
My thoughts are that we get a few advantages from copying the F# approach:
Most of the new Scala developers I have worked with are totally mystified by the idea that for can be overloaded to work on non-collection type things. The builder syntax makes it a bit more clear that something magical is happening and that it’s not just the normal for loop that they’re used to.
Doesn’t really use any new keywords as we can reuse old ones inside the block (e.g. for, while, do)
Individual method signatures can be as complicated and constrained as required by the underlying libraries
Builders can be as polymorphic or as specific as we want, e.g. we can still write an expression builder against F if we want
and a few disadvantages:
You would really need to specify the F you are interested explicitly in if there are any non-trivial constraints involved rather than letting type inference do the work
The fact it’s a syntactic transformation limits optimization opportunities a bit as you can’t do dependency analysis of the expressions, but this is probably OK because ordering is significant here anyway
We really can’t write something as generic as the original for comprehension using this approach - the builder class has to typecheck
Is reusing syntax like for a good idea? Then expressions that use the results of traverse look pretty weird due to multiple <-, i.e.
async[Task] {
deploymentIds <- for (name <- verticleNames) {
Vertx.deployVerticleL(name)
}
do StdIn.readLine("Press enter to shut down")
for (deploymentId <- deploymentIds) {
Vertx.undeployVerticleL(deploymentId)
}
}
Some interesting questions:
How to let the compiler know that this builder class triggers the desugaring? Extending a magic trait and other similar things seem kind of gross.
Could this be implemented by making the builder factory method a Scala 3 macro that accepts the block of code to desugar? Would that limit the syntax that could be used within the block to things that superficially typecheck before the macro expansion or would they just have to parse?
Please moderators, feel free to move it to