It does not conflict with the current rule, because the current rule is applied to e' only if it’s not a DSL expression. However, it is indeed problematic. In fact the following code will never compile as long as we translate them to flatMap/map/foreach member methods, because these member methods defined on collections and futures conflict with each other.
The solution would be to translate them to rebindable keywords.flatMap / keywords.map / keywords.foreach. I can write down another proposal for details. But now I just removed this part from this proposal.
Updated.
Every proposal increases complexity. This proposal actually increases less complexity than alternatives. For example, neither Scala Continuations nor Scala Async has a set of standardized translation rules. They actually increase the hidden complexity and do not guarantee their behaviors. Also note that, unlike alternatives, this proposal does not introduce new keywords.
The complexity consideration is also the reason I would like to create multiple independent proposals to improve for expressions, not a single huge proposal.
No, because existing for comprehension will not be affected. Rules introduced in this proposal is applied to DSL expressions only, which were invalid syntax before this proposal.
Yes. Originally I thought the DSL interface should be minimal, e.g. generate code with only flapMap / pure, like what we did in Each, Dsl.scala, Monadless, Scala Continuations and Scala Async. But now I realized the F# approach may be better. The rules to translate control flow to functions are crystal clear, and they also allow libraries to create runtime that is optimized for a specific DSL, for example operator fusion in GPU programming.
If I may, I would like to question whether this is a direction we want to go with.
Railway oriented programming, a model that relies on the for comprehension in Scala, is a great approach for writing code that has at most two major paths – a happy and a sad path, where handling mostly indistinguishable failures can be made simpler and clearer, either in sequential or concurrent (Future) code.
I’m not sure this model holds for data flows which are not as simple as to have one or two paths. Once the data flow of a certain part of the code starts to diverge and form branches, I find it much more readable to have explicit control flows – if, else and pattern matching.
Seeing an “if or else” function in the midst of a railway of joint functions just feels wrong. I would say the same thing about Future.transform, Try.transform, Either.fold or Option(value).filter(condition).map(happy).getOrElse(sad).
This is just my opinion though, so take it with a grain of salt
I think the railway-oriented programing analogy is a bit flawed. Every line of code in an imperative programming language can throw an exception, so in a sense, proceeding to the next line is always the “happy path”, right? Doesn’t that mean that all code is railway oriented from the perspective of the interpreter that runs it?
So to me it seems that normal code is semantically not so different from code written using a monadic style. In the functional programming style the errors are values rather than a control flow jump.
At the moment we have a nice syntax for “successfully getting to the next line” (flatMap) in the value-oriented programming style, but not for many other control structures.
Exceptions and ROP are not alternatives of each other, but rather different tools for similar yet distinguishable enough roles. I simply wouldn’t use exceptions for plain “happy sad railways”.
Instead, ROP is an alternative to imperative control flow, where instead of declaring the branches of the flow using if then return else then return, one uses monadic composition. But really, my point is not whether one is better than the other, as they are both completely fine in my opinion.
My point is that currently with Scala, the monadic composition capabilities and specifically the for comprehension, dissuade one from writing a block of code that is heavily nested, and instead encourages one to separate the branching flow to other functions / code blocks.
One can still write a large branching control flow with a heavily nested if else / pattern match code block, but I would consider it a bad practice, and I would like to eliminate the opportunity of writing such code with monadic compositions.
All of those combinators compose together really well and generally make writing code in this style a lot easier!
So when I say ROP is not a good analogy - it’s a great analogy for explaining the interaction of Either and Option with map and flatMap, and you’ll note that was the context of the talk you linked above - but it’s a very limited view of what we can do with monadic types. You have lots of great options for monadic control flow!
We can write those combinators and they work great, but they aren’t currently supported by for - so that is why you commonly see less control flow in monadic code - because the tools aren’t there in the standard library. That is what proposals like this one aim to fix.
Since @julienrf has mentioned it I’m really interested in all of your thoughts about a syntax inspired by the F# computation expression.
Please excuse my idiosyncratic notation!
For some computation builder bldr:
// Builder expression
//
// Returns a custom class implemented by the user.
//
// It contains declarations of the DSL operations supported by this
// computation expression.
//
// As a syntactic transformation only, unsupported syntax would trigger a compile error
// as the current for comprehension does now when it is used with types that don't support
// the required operations
//
// def bldr[F[_]]: BuilderType[F]
//
//
desugar('{ bldr[F] { expr } })
==>
val $ = bldr[F] // $ would be some fresh name
desugar('expr)
// Compound expressions in a builder
desugar('{ cexpr })
==>
desugar('cexpr)
// Bare expressions in a builder
desugar('{ a })
==>
a
// Unused enumerators in a builder
desugar('{ b <- a })
desugar('a)
// Composing expressions
//
// def flatMap[A, B](fa: F[A])(f: A => F[B]): F[B]
//
// bare expressions:
//
desugar('{ a; expr })
==>
$.flatMap(desugar('a))(_ => desugar('expr))
// with enumerators
desugar('{ b <- a; expr })
==>
$.flatMap((desugar('a))(b => desugar('expr))
// Yielding values
//
// def unit[A](a: A): F[A]
//
desugar('{ return a })
==>
$.unit(a)
// Possible optimizations
desugar('{ return a; b <- a })
==>
$.flatMap($.unit(a))(b => f(b))
==>
f(a) // via monad left identity
desugar('{ b <- a; return b })
==>
$.flatMap(desugar('a))(b => $.unit(b))
==>
a // via monad right identity
// def map[A, B](fa: F[A])(f: A => B): F[A]
//
desugar('{ b <- a; return f(b) })
==>
$.flatMap(desugar('a))(b => $.unit(f(b)))
==>
$.map(desugar('a))(f) // via map/flatMap coherence law
// For loops
//
// def traverse[G[_], A, B](ga: G[A])(f: A => F[G]): F[G[B]]
//
desugar('{ for (b <- a) { c } })
==>
$.traverse(a)(b => desugar('c))
// Conditionals
//
// def ifThenElse[A](cond: F[Boolean])(thenExpr: F[A])(elseExpr: F[A]): F[A]
//
// def zero[A]: F[A]
//
desugar('{ if (a) then b })
==>
$.ifThenElse(desugar('a))(desugar('b))($.zero)
desugar('{ if (a) then b else c })
==>
$.ifThenElse(desugar('a))(desugar('b))(desugar('c))
// While loops
//
// def while(cond: F[Boolean])(body: F[Unit])
//
desugar('{ while (a) { b } })
==>
$.whileLoop(desugar('a))(desugar('b))
// Lifting plain Scala code into the builder type
//
// def delay[A](a: => A): F[A]
//
desugar('{ do a })
==>
$.delay(a)
Obviously this is not really close enough to a formal specification of such a desugaring but I hope this helps illustrate the idea.
You can see what the experience of defining a new builder type would be like in the following code sample, along with some example expansions of the syntax.
I expanded a bit on the advantages and disadvantages of this in the other thread, but there are some unanswered questions about porting such a design to Scala:
How could the compiler know that this builder class triggers the desugaring if there are no enumerators present? Extending a magic trait and other similar things seem kind of gross and also don’t really fit into a syntactic expansion of the desugaring.
Could this be implemented by making the builder factory method a Scala 3 macro that accepts the block of code to desugar? Would that limit the syntax that could be used within the block to things that superficially typecheck before the macro expansion or would they just have to parse? The former is probably too limiting.
Is reusing <- a bad idea? What else could we use? It collides with the for syntax, especially when using the results of a for expression, i.e. deploymentIds <- for (vertName <- vertNames) { Vertx.deployVerticleL(vertName) }
I assume we don’t need translation if no <- is present. Do you have any use cases where the assumption is invalid?
No. Both for translation and new rules introduced from this proposal are name based, which require the ability to produce untyped AST. Unfortunately Scala 3 macros can’t produce untyped AST.
The code in your example is valid Scala 2 code, which should behave exactly the same as before even with this proposal.
Interesting take! But I think we don’t really need to copy F# verbatim here. F# does certain things the way it does, because it lacks other features that Scala possesses even today. Needing to have an explicit “builder” object is one such example. In Scala, we have implicit parameters, extension methods and higher-kinded types. We should do fine with only those. What we would need, is a new do construct.
Let me illustrate it with this short snippet:
def fetchAndDownload(url: Url): IO[Data] =
do { // marks the start of the computation expression. alternatively
val urlStripped = strip(url) // usual variable binding; it's nice you can do that even as the first thing
val! data = downloadData(urlStripped) // `val!` desugars to `.flatMap`
val processedData = processData(data) // another usual variable binding
if!(callVerificationService(processedData).map(_.isError)) { // `if!` with then branch `IO[Unit]` and without else branch desugars to calling `.whenA`
throw! CustomError("wrong data") // `throw!` desugars to calling `.raiseError`
}
use! monitor = createMonitor(processedData) // `use!` desugars to calling `.use`
do! notifyMonitor(monitor) // `do!` on `IO[Unit]` desugars to calling `.flatMap(() => ...)`
processedData.pure // `IO[Data]` is expcted here, so we produce it by calling the `.pure` extension method
}
The precise names of the extension methods this desugars to is not terribly important, I just used the names as they are in cats toady. But Scala authors could choose different names, and then, I’m sure, cats, zio & co. would accommodate.
What is important, is that the user-visible syntax of the “monadic” operations is intentionally similar to that of their imperative counterparts, so val/val!, if/if!, throw/throw! and so on…
Agreed. I would like to highlight the ability of reifying control flow in this proposal. With the help of reification, you can create type-level interpreters to hande really complex usage of imperative control flows.
My instinct here is that Scala has already far exceeded its novelty budget and that we should not be adding core features that take advantage of Scala’s most complex constructs in the surface syntax of the language. The thing that is attractive to me about the F#-like solution is that the desugaring is extremely simple, it can happen in the parser only, it does not use any new keywords, it doesn’t require extra bookkeeping in the parser to keep track of enclosing constructs. With the inline and erased keywords in Scala 3 we could also eliminate the intermediate builder object and the associated method calls. On the other hand I have a suspicion that it could be confusing that those keywords are reused.
This is actually by far my preferred approach, and for a second or two I hoped that this would be possible in Scala 3 because do {} while loops are being deprecated. Unfortunately though, it was deprecated to make way for for {} do {} so the concept of a do {} block is already in use for something different. If we can come up with an alternative keyword to do {} it doesn’t seem like an insurmountable problem though.
There are a couple of other things about the Ocaml/F# approaches that I think will cause confusion and consternation. Firstly I don’t think we would get a lot of buy in to the idea of adding a new symbol to the core syntax of Scala (the ! or + enhancing existing keywords) as a lot of the recent changes to the language seem to be aimed at addressing perceived criticisms of the language - one of which is overuse of symbolic operators. Secondly the equals = symbol is used to bind names that desugar to flatMap calls. This makes FP folk very upset because it breaks referential transparency - the resulting code does not behave the same under inlining.
I’ve seen a fair number of language-modification proposals that want to make apparently-plain Scala desugar into various other things which aren’t plain Scala. I understand the motivation for this. Just want to say:
I’ve been down that road, and IME it ends in despair. The stuff you’re trying to protect people from (like existing syntax that describes effectful programs, I guess?) are things they probably don’t want or need protection from. At least for my use cases, I’ve found it’s better to just follow reasonable idioms in APIs. Use monadic constructs when it makes sense to do so; people aren’t afraid of learning how to use that. Making it so that a piece of code which uses monadic constructs looks like it’s using monadic constructs (e.g. with for) is good, not bad – if it looks like imperative Scala code encased in an easily-missed mystery {} block, it doesn’t end up looking all that much aesthetically better, but makes it much more confusing for the reader.
If one really wanted to disguise monadic code as regular code, there’s now a built-in (!!!) compiler phase which does that kind of thing. It sort of snuck through, but there’s a -Xasync flag in the latest compiler that does a bunch of double-secret rewrites of imperative code to be… something else which is also double-secret. Maybe this could be utilized to accomplish what’s being sought here? That would obviate the need for new val! kinds of tokens, since normal vals are already rewritten to… something. If you figure it out, documenting that process would also be really useful to everyone who doesn’t understand exactly what -Xasync does or why it was merged without much in the way of SIP process.
It has been implemented in Dsl.scala. You can checkout the example here. The syntax could be better if we have an improved version of for comprehension.
You are basically saying Python, C#, ECMAScript, Kotlin, and even C++ 20 are all wrong, and Scala will never be a language for those developers who are familiar with generators and coroutines.
The purpose of this proposal is to provide a standardized rewrite rule as an alternative to Scala Async. The only reason why val! is not desirable is that mainstream languages don’t use that. Despite of the complexity of the runtime and rewrite rules (including the ability of reifiable control flow), the users should be able to create coroutines and generators just like other mainstream languages with the help of this proposal.
Nope, not saying that at all. I’m not saying anything is wrong, just offering that I’ve been down the road of “monadic things are hard, let’s make them look like they’re not hard” starting with writing Future-wrapping DSLs on top of the original CPS plugin, which seems like 100 years ago. It was a lot of effort, and it turned out that understanding the compile-failure modes of the resulting downstream code was a lot more effort than just learning about Future and “monadic”(-ish) syntax.
I do understand that, and I think that’s great. From what I understand, though (and this is what I was trying to say), there’s already a de-facto standard rewrite rule that’s been merged into the language. It claims to be flexible to various F[_]s, so it might be the place to start. Certainly if you’re not happy with it, I’d be interested to hear some noise about that, since I’ve no idea what its rewrite rules are and they don’t seem particularly documented nor were they discussed in a meaningful way on this forum. I guess what I’m trying to say is – I think we all have the same complaint, but let’s all complain about the same thing
Note, “complain about the same thing” meaning, here – there’s a flag which does some purportedly general “non-monadic to monadic” transformation which, given that it’s merged into the language, should presumably be the one way to accomplish such a thing. But it’s not general if it’s not documented – since nobody else can implement it – and there was no opportunity to discuss what the transformation should be in order to generalize most completely.
One more thing @yangbo – I get that there’s more to what you’re proposing, in that it also requests some ability to treat raw ASTs (pre-typing) in order to virtualize them. I really don’t think that’s going to happen, although I can think of a lot of great use cases myself – Scala isn’t going to become racket. There’s already some pretty powerful tools for monadic DSLs, and if there’s a general-purpose transformation of imperative to monadic – as long as it’s documented and accessible – I think that would answer a lot of such use cases (at least, when IDEs catch up with it)
Virtualizing AST commonly happens. Existing for comprehension, pattern matching, and scala.Dynamic are all AST virtualization. Even though we did not specify the translation rules in -XAsync, it is definitely translating AST to name based function calls.
In the .Net world F# was the first mover for async syntax, but then C# won with async/await. At least that’s what I see. Scala does have generalized async/await now. Furthermore, the JVM might get something like native coroutine support with Loom. So my conclusion is that adding new syntax to the language now is premature. Let’s see first how async/await and Loom play out.
My personal opinion here is that “virtualized Scala” or “lifted Scala” would be incredibly valuable, but the design space is large enough I would like to see an implementation become popular “in the wild” via a macro or compiler plugin before considering it for the core language.
Among the possible alternatives:
Purely applicative lifting, like Mill does
Mostly applicative lifting, like SBT does
“One-shot” monads, like scala-async does
Generalized monads, like Monadless does
Full-syntax virtualization, like F# or DSL.scala do
Each of these points in the design space have different tradeoffs, benefits and limitations. Exploring the viability and usability and utility of even one of the above alternatives is already a huge task, hence why I do not believe a single SIP discussion would be enough to design on a design.
We already have an implementation for DSL.scala, but it has not gotten broadly popular enough that I would be comfortable considering it for “upstreaming” in the language. Others like the Mill or SBT implementations similarly remain niche. Like it or not, these kinds of language features are complex enough that no matter how much we discuss the design, it’s only through broad experience using them that their true utility and limitations become apparent.
Scala-Async has a long history of broad discussions and usage, and is a known quantity. While I’m not totally comfortable with finalizing it without further discussion, I do think it has a good claim to being broadly accepted and useful. For other ideas, as much as I think they show potential, I would like to see some broader community usage before we have enough experience to discuss and compare them in an informed manner
