I don’t think the fact that it’s a syntactic transformation is necessarily the problem here, it’s that the syntactic constructs and underlying operations that those constructs are translated to in the current for comprehension aren’t sufficient to avoid those warts.
I don’t think it’s as hopeless as this
Look again at the F# computation expressions. The way that it works is substantially similar to the way the for translation works in Scala. However there are more operations supported by the translation than those supported by Scala’s for comprehension, perhaps enough to support libraries similar to Cats Effect.
See for example the translation of while expressions:
{ while expr do cexpr } builder.While(fun () -> expr, builder.Delay({ cexpr }))
The requirements to use the while construct in a computation expression are that the underlying builder object implements the Delay: (unit -> M<'T>) -> M<'T> and While (unit -> bool) * M<'T> -> M<'T> methods. I think that it would be really interesting to see how far we could get by supporting a richer vocabulary of operations using a syntactic transformation like @sideeffffect suggets.
It’s not so hard to imagine translating while(cond) { a <- expr } to something implemented using whileM. Also, the fact that it’s a syntactic transformation enables all of the implicit constraints required by libraries like Cats Effect to be used.
Another approach that seems interesting is prototyping a replacement for the for translation using the new metaprogramming system in Scala 3 as this would enable the translation to make more use of type information.
What if instead of the compiler trying to provide a one-size-fits-all AST transformation based on a bunch of assumptions that it sounds like are necessarily too narrow, there was some way to plug in the AST transformation itself. IIUC the benefit of this approach is that it’s a different compiler phase. Couldn’t that somehow be opened up though?
In any case what’s the plan for Dotty? It’s supposed to be around the corner. Maybe there’s a bigger picture approach that should be taken somehow?
I guess opening up a custom compiler phase would end up being pretty similar to writing the transformation as a macro, so I’m not sure if it would be worthwhile to provide a custom transformation via a different mechanism. I could be wrong though! I guess that it’s basically an open research question as to whether there’s a bigger picture approach.
This is a way, chooses in dotty-cps-async: GitHub - rssh/dotty-cps-async: experimental CPS transformer for dotty.
For now, it supports all scala constructions, all monads, which implements appropriative typeclass with some interoperability between and even some support of high-order functions.
Here is an introduction talk from ScalaR: https://www.youtube.com/watch?v=ImlUuTQUeaQ
// (sorry for no-documentation, I have guessed that we have some time before dotty release, plan to start to write manual during next weekend slot.)
Any patches, help, feedback, experiments will be accepted with gratitude.
It’s worth noting that dotty-cps-async suffers from the same problems as -Xasync, albeit in a more elegant way: spawn/fulfill are fundamentally unimplementable operations for any non-eager, non-memoizing monad. Here again the mistake is in assuming that everything is Future-like. Though, ironically fulfill takes it a step further still because it’s blocking, meaning that anyone using dotty-cps-async is giving up their asynchrony.
Don’t get me wrong, it’s a worthwhile effort, but I honestly think that it might be better to take a step back here. Cats Effect has spent years refining abstractions for this stuff and quantifying, in general, what it means to be an asynchronous effectful computation. It’s almost certainly easier to start from it, at the very least as a point of reference and inspiration, rather than attempting to reinvent such abstractions from scratch.
I do like @yangbo’s proposal btw, though I need to read through it more to understand it better. Perhaps it’s a more promising future direction.
But spawn/fullfill is not used in cps-transform at all.
In 1-st iteration of design they was created for interoperability with Future (i.e. if you want use in one expression Future and you monad). But in latest version, currently only asyncCallback is using for this. Spawn/fulfill should be refactored to other interface. (which is needed for channels, which is not in scope of scala async, but used in tests now).
To be clear: we define 3 interfaces:
CpsMonad: map, flatMap, pure
CpsTryMonad: error/restore
CpsAsyncMonad: adoptCallbackStyle, spawn, fullfill
In transform, CpsMonad is used for all except try/catch; CpsTryMonad - for try/catch, CpsAsyncMonad not used at all.
In interoperability: CpsAsyncMonad.adoptCallbackStyle when we want mix our monad and Future
Ah if spawn/fulfill aren’t being used, then it’s basically Monadless, which I’m okay with. Just as long as you understand those functions are both a) extremely dangerous, and b) unimplementable for things that aren’t literally Future. I don’t see how you can handle rewrites over higher-order functions though without some more machinery (e.g. Traverse in Cats).
I spent a bit of time thinking about how straightforward it would be to use the approach that is used by F# computation expressions and to me it seems kind of promising. Writing the underlying builder class looks like it could be pretty straightforward, and the translation of the syntax within a block doesn’t have to be so complicated, see e.g.:
My thoughts are that we get a few advantages from copying the F# approach:
Most of the new Scala developers I have worked with are totally mystified by the idea that for can be overloaded to work on non-collection type things. The builder syntax makes it a bit more clear that something magical is happening and that it’s not just the normal for loop that they’re used to.
Doesn’t really use any new keywords as we can reuse old ones inside the block (e.g. for, while, do)
Individual method signatures can be as complicated and constrained as required by the underlying libraries
Builders can be as polymorphic or as specific as we want, e.g. we can still write an expression builder against F if we want
and a few disadvantages:
You would really need to specify the F you are interested explicitly in if there are any non-trivial constraints involved rather than letting type inference do the work
The fact it’s a syntactic transformation limits optimization opportunities a bit as you can’t do dependency analysis of the expressions, but this is probably OK because ordering is significant here anyway
We really can’t write something as generic as the original for comprehension using this approach - the builder class has to typecheck
Is reusing syntax like for a good idea? Then expressions that use the results of traverse look pretty weird due to multiple <-, i.e.
async[Task] {
deploymentIds <- for (name <- verticleNames) {
Vertx.deployVerticleL(name)
}
do StdIn.readLine("Press enter to shut down")
for (deploymentId <- deploymentIds) {
Vertx.undeployVerticleL(deploymentId)
}
}
Some interesting questions:
How to let the compiler know that this builder class triggers the desugaring? Extending a magic trait and other similar things seem kind of gross.
Could this be implemented by making the builder factory method a Scala 3 macro that accepts the block of code to desugar? Would that limit the syntax that could be used within the block to things that superficially typecheck before the macro expansion or would they just have to parse?
I prefer to say ‘applicable only in imperative scenarios’ instead ‘extremely dangerous’. Exists one case, where it’s not Future but have a high practical value: it’s a ‘something like Future,’ running inside one-threaded dispatch. When we know that we are always in the same thread, then we can avoid operations, that flush processor cache [volatile access, CAS… etc.] and, as a result - have some speedup over plain Future. [No tests for dotty-cps-async yet, but I remember experiments on 2.12 with something like 10-x difference]. Although I know that Future in 2.13.x also was optimized for a one-thread case, so maybe now the difference is not so promising. Another potential example is a Loom-monad when we will able to introduce after Loom will be available. Speed of such monad will be in-between Future and one-thread case: (Loom dispatch contains volatile access).
Naming, of course, can be better.
currently, we assume that the developer can implement a ‘shifted’ version of the high-order method and bind one with object type via typeclass. And provide such shifted versions for the classes defined in the standard library. Also, it is possible to translate lambda-expressions automatically.
We don’t provide rewrites of third-party high-order functions automatically now. We implement AsyncShifted[T] for Iterable[T], Collection[T] … etc. Monadless approach for transform can be represented as something like ‘generation of given Shifted[T] from Traverable[T]’. , which potentially can be implemented on top of dotty-cps-async, but this is a big separate direction.
Also exists two potential other approaches, each with its own set of gains and drawbacks.
The third notable difference from ‘Monadless’ is ‘ghost interfaces.’ We run after the typing phase, so we should use real interfaces. If map/ flatMap can be defined as inline, we can hope that compiler will inline one, but I yet not played with this.
Also, I have thought about passing ‘rewriters’ to macro, instead of interfaces, but faced with the impossibility to pass non-trivial argument (tree rewriter) from the context of macro application to macro.
I mean, fulfill really is dangerous, imperative or otherwise, because of the thread blockage. In fact, it’s even more dangerous in a single-threaded environment (like ScalaJS) since it will actually deadlock. Literally the only circumstances under which fulfill is not dangerous are either a) no use of Promise whatsoever in the entire application (i.e. so Future is basically Try in this scenario), or b) running under Loom.
Blocking functions are simply not viable. They are dangerous, regardless of whether the stylistic context is imperative or functional.
Ah I’ll have to look. That sounds a lot like Cats’ Traverse, which is basically what you would need to do something like this in any case (assuming the goal is to avoid blocking).
This is very nice to be sure, though this benefit disappears a bit the moment anything is abstracted. If you, for example, wrote a hypothetical layer which adapted dotty-cps-async to Cats Effect, certainly dotty-cps-async would not have any visible interfaces in the emitted bytecode, but Cats Effect would. This is exactly the same as Monadless. The only case where dotty-cps-async is able to inline things that Monadless (and similar) cannot is where the dotty-cps-async adapters (e.g. AsyncMonad) are directly implemented for a specific concrete type (e.g. Future).
Yes, it should be not called inside monad. In any IO you have something like runUnsafe and to actually run program, you should call this method. It can be hidden under something like IOApp, but in this case construction and evaluation of IOApp will play role of fulfill.
I.e. if we need the only transformation and always will run received program in a cats-effect, than you don’t need export you runUnsafe. But if we want that third-party to be able actually run program, not knowing that it come form cats-effect, than I can’t imagine API which not include something like ‘Await.run’, ‘runUnsafe’, ’ IOApp.run’, etc.
Hmm, … maybe exists way to specify implicit value, which saying 'we are not inside monad` to prevent dangerous use… Will think, but not sure that this is possible.
Well, the trick with IOApp is that it doesn’t give users the ability to run the IO… and then do something else. Running the IOis the whole program, so that avoids two problems: visibility of side-effects, and blocking of threads. A thread is blocked (main), but it’s the only one and it’s irrelevant since nothing happens after. The unsafeRun functions obviously don’t have that same kind of safety. They’re basically always dangerous, though the Async variants of them are safer (since they don’t block). fulfill corresponds to unsafeRunSync, which is the most dangerous variant, particularly on ScalaJS (where it really shouldn’t exist at all, since it almost always results in deadlock).
The implementation could be more efficient if monix.Task exposed a way to query if a value is immediately available. In the analagous case for scala Futures, scala-async doesn’t occur a thread-switch.
The Task returned by this async is non-strict and can be run more than once (e.g within a retry combinator).
The Finite State Machine (FSM) transform, which effectively flattens all the behaviour and state capture in the lambdas you would get with use of map/flatMap into a single method. This transform is possible because we know that a execution of the FSM is sequential.
The Selective A-Normal Form transform (ANF), a precursor to FSM transform that ensures that the execution order of the code is not accidentally changed by subsequent rewrites. (I noted that Monadless, as currently implemented, does change the execution order in some cases)
Analysis to detect when intermediate captured state will no longer be required by the FSM which nulls this out, reducing footprint of in-progress computations.
An API for different future data types to participate.
The new implementation is more efficient, correct and IMO more maintainable because a) trees are easier to work with post-erasure b) we with full access to the compiler API. Because we generalized it enough to work with a variety of Future/Task types with only a small adapter, it seemed worthwhile to ship as a compiler phase rather than as a separate compiler plugin.
-Xasync itself may be evolvable based on what out there in Kotlin / C# / F#. But shipping the first version also doesn’t prevent these extensions – the facade API is currently purely structural (we don’t ship the StateMachine interface in scala-library) so a future version can support an extended or alternate version of this.
a mode to automatically parallelize await(startFuture1) + await(startFuture2), by calling startFuture2 before awaiting the result of startFuture1: https://github.com/scala/scala-async/issues/138
Allow await in some lambdas in code like async { None.getOrElse(await(foo) } by inlining combinators or rewriting calls to a CPS-lifted version of the combinator: https://github.com/scala/scala-async/issues/32
An effect DSL based on Applicative/Traverse, like monadless, is a different prospect. While the user experience may be quite similar, the requirements it places on the data types and the compiler implementation different. If there are opportunities to share code with the implementation -Xasync all the better! If not, the existence of one DSL does not preclude the other.
Thanks for the Monix example. If there’s no need to have an executor/scheduler in scope, then Xasync still lets us write libraries with methods that produce tasks and leave it to someone else to choose how/where/when to run the returned task.
The integration seems intense though: the integration code needs a macro, mutable vars that start out null, required methods with dollar signs, AnyRef + asInstanceOf, and calls to seemingly internal apis (c.internal). I’d be worried about asking my team to maintain this for our own integrations, especially having to ramp up on scala.reflect.macros.
Integrating a custom task type with monadless was straightforward by comparison:
Note that although the Scala 2 version of cats-effect-cps is based off -XAsync, the Scala 3 version depends on dotty-cps-async, and is arguably better.
As implementor of the Scala 2 version of cats-effect-cps, I can attest integrating cats-effect with the -XAsync machinery was very hard, partially due to lack of documentation, partially due to how the design of -XAsync really doesn’t work well with anything that doesn’t have Future-like semantics (in particular, it simply can’t work with cats.effect.Resource). It kinda works for cats.effect’s IO (and some monad transformers) due to the Dispatcher construct provided by cats-effect, which allows some form of interoperability between pure and impure code, but it’s easy to find monads that wouldn’t work well with -XAsync (Free monads, for instance)
Not saying that -XAsync has no reason to be in its current form, but I very much think it’s incorrect to advertise it as the blessed way for providing direct-style for any monad.
Please moderators, feel free to move it to