Pre SIP: `for` with Control Flow (an alternative to Scala Async, Scala Continuations and Scala Virtualized)

Motivation

Coroutines (including generators and async functions), were considered as a killer feature, but now become a usual practice, after being adopted in ECMAScript and other mainstream languages. Due to lack of runtime support in JVM, all previous implementations of this feature in Scala world, including Scala Continuations, Scala Async, Each, Monadless, Dsl.scala, are all based on compile time CPS translation.

Unfortunately, the exact translation rules have never been standardized. The former SIP-22 Async contains an Async Transform Specification section, which is too general to tell how should an implementation behave. This proposal is one of a series of proposals, aiming to standardize the exact rules that turn existing for comprehension into a full featured continuation, covering not only all the use cases of coroutines in mainstream languages, but also repeatable continuations including data-binding and reactive event observables.

This proposal enables the generator operator <- to appear not only as a direct child statement of a for loop or a for comprehension, but also in any block, as long as the block is a descendant node of a for.

Motivating Examples

Suppose you are creating a function to fetch the website of a Github repository, which is the home page of the repository if available, or the owner’s blog site otherwise. With the help of this proposal, you can create something like this:

import org.scalajs.dom.ext.Ajax
import scala.scalajs.js.JSON
import scala.concurrent._
def findWebSite(repositorySlug: String) = {
  for {
    repositoryResponse <- Ajax.get(s"https://api.github.com/repos/$repositorySlug")
    repository = JSON.parse(repositoryResponse.responseText)
    homepage = repository.homepage
    webSite = if (homepage != null) {
      homepage.toString()
    } else {
      ownerResponse <- Ajax.get(repository.owner.url.toString())
      owner = JSON.parse(ownerResponse.responseText)
      owner.blog.toString()
    }
  } yield webSite
}

According to this proposal, blocks that contain <- should be translated to a for expression, while control flow keywords that contain <- should be converted to function calls. Finally, the above code should be translated as the following code:

def findWebSite(repositorySlug: String) = {
  for {
    repositoryResponse <- Ajax.get(s"https://api.github.com/repos/$repositorySlug")
    repository = JSON.parse(repositoryResponse.responseText)
    homepage = (repository: JSON.parse).homepage
    webSite <- keywords.ifThenElse(
      keywords.pure(homepage != null),
      keywords.pure {
        homepage.toString()
      },
      for {
        ownerResponse <- Ajax.get(repository.owner.url.toString())
        owner = JSON.parse(ownerResponse.responseText).owner
      } yield owner.blog.toString()
    )
  } yield webSite
}

This proposal also includes some runtime library requirements:

package scala.concurrent

object keywords {

  def pure[A](a: => A)(using ExecutionContext) = Future(a)

  def ifThenElse[A](
    ifFuture: Future[Boolean],
    thenFuture: Future[A],
    elseFuture: Future[A],
  )(using ExecutionContext) = {
    ifFuture.flatMap { b => if (b) thenFuture else elseFuture }
  }

  // Other runtime libraries for each Scala control flow
  // ...
}

With the help of the runtime libraries, the return type of findWebSite will eventually be inferred as a Future[String].

Design

Virtualization

Scala Virtualization was a previous attempt to translate Scala control flow to function calls solved by name instead of by symbol. Unfortunately, Scala Virtualization does not take CPS translation into account, as a result, it conflicts with Scala Continuation and Scala Async.

This proposal extends the idea of virtualization to CPS translation. Therefore, by import different implementations of keywords object, for with control flow should work for various of types. Also it’s trivial to create a keywords object that delegates calls to a generic type class, like MonadError.

Reifiable control flow

The implementation of keywords can be case classes that preserve the structure of control flow, both at type level and runtime. For example:

trait Mappable[Keyword, +Value]

def[Upstream, Value, Mapped, MappedValue](
  upstream: Upstream
)flatMap(
  using Mappable[Upstream, Value]
)(
  flatMapper: Value => Mapped
)(
  using Mappable[Mapped, MappedValue]
) : keywords.FlatMap[Upstream, Value, Mapped] =
  keywords.FlatMap(upstream, flatMapper)

def[Upstream, Value, Mapped, MappedValue](
  upstream: Upstream
)map(
  using Mappable[Upstream, Value]
)(
  mapper: Value => Mapped
)(
  using Mappable[Upstream, Value]
) : keywords.Map[Upstream, Value, Mapped] =
  keywords.Map(upstream, mapper)

object keywords {
  case class IfThenElse[If, Then, Else](ifTree: If, thenTree: Then, elseTree: Else)
  export IfThenElse.{apply => ifThenElse}
  given[If, Then, Else, Value](using thenMappable: Mappable[Then, Value], elseMappable: Mappable[Else, Value]) as Mappable[IfThenElse[If, Then, Else], Value]

  case class Pure[A](a: () => A)
  def pure[A](a: A) = Pure(() => a)
  given[A] as Mappable[Pure[A], A]

  final case class FlatMap[Upstream, UpstreamValue, Mapped](
    upstream: Upstream,
    flatMapper: UpstreamValue => Mapped
  )
  given[Upstream, UpstreamValue, Mapped, Value](using Mappable[Mapped, Value]) as Mappable[FlatMap[Upstream, UpstreamValue, Mapped], Value]
  final case class Map[Upstream, UpstreamValue, Mapped](
    upstream: Upstream,
    mapper: UpstreamValue => Mapped
  )
  given[Upstream, UpstreamValue, Mapped] as Mappable[Map[Upstream, UpstreamValue, Mapped], Mapped]

  // Other control flow ASTs
  // ...
}

object Ajax {
  trait Response {
    def responseText: String
  }
  case class get[Url](url: Url)
  given[Url] as Mappable[Ajax.get[Url], Response]
}


import scala.language.dynamics
case class SelectDynamic[Parent, Field <: String & Singleton](parent: Parent, field: Field) extends Dynamic
def [Parent <: Dynamic](parent: Parent)selectDynamic(field: String) = SelectDynamic[Parent, field.type](parent, field)
object JSON {
  case class parse(json: String) extends Dynamic
}

If we import the above definitions of keywords, JSON and Ajax instead of scala.concurrent.keywords, scala.scalajs.js.JSON, and org.scalajs.dom.ext.Ajax, then the findWebSite function should return a reified tree of the following type:

keywords.FlatMap[
  keywords.Map[Ajax.get[String], Ajax.Response, (Ajax.Response, JSON.parse, 
    SelectDynamic[JSON.parse, "homepage"]
  )],
  (Ajax.Response, JSON.parse, SelectDynamic[JSON.parse, "homepage"]), 
  keywords.Map[
    keywords.IfThenElse[keywords.Pure[Boolean], keywords.Pure[String], 
      keywords.Map[
        keywords.Map[Ajax.get[String], Ajax.Response, (Ajax.Response, 
          SelectDynamic[JSON.parse, "owner"]
        )],
        (Ajax.Response, SelectDynamic[JSON.parse, "owner"]),
        String
      ]
    ],
    String,
    String
  ]
]

It’s a tree of control flow, which can then interpreted by type-level interpreters.

Tagged partial functions

When control flow is reified, each case clause in pattern matching or exception handling might produce different reified types. In case of inferring them to Any type, the list of case clauses will be translated to a partial function whose return value is wrapped in a couple of keywords.right and keywords.left calls. The number of keywords.right calls indicates the index of the case clause. For example:

def matchWebSite(repositorySlug: String) = {
  for {
    repositoryResponse <- Ajax.get(s"https://api.github.com/repos/$repositorySlug")
    repository = JSON.parse(repositoryResponse.responseText)
    homepage = repository.homepage
    webSite = homepage match {
      case null =>
        ownerResponse <- Ajax.get(repository.owner.url.toString())
        owner = JSON.parse(ownerResponse.responseText)
        owner.blog.toString()
      case nonNullHomepage =>
        nonNullHomepage.toString()
    }
  } yield webSite
}

will be translate to

def matchWebSite(repositorySlug: String) = {
  for {
    repositoryResponse <- Ajax.get(s"https://api.github.com/repos/$repositorySlug")
    repository = JSON.parse(repositoryResponse.responseText)
    homepage = (repository: JSON.parse).homepage
    webSite <- keywords.matchCase(
      keywords.pure(homepage),
      {
        case null =>
          keywords.left(
            for {
              ownerResponse <- Ajax.get(repository.owner.url.toString())
              owner = JSON.parse(ownerResponse.responseText).owner
            } yield owner.blog.toString()
          )
        case nonNullHomepage =>
          keywords.right(keywords.left(keywords.pure(nonNullHomepage.toString())))
      },
    )
  } yield webSite
}

For naive keywords implementation keywords.right and keywords.left can be simply identity function. However, for keywords implementation that reifies control flow to data structures that preserves everything at type-level, keywords.right and keywords.left can be implemented as a coproduct or scala.Either.

Specification

DSL expressions

A DSL expression is a DSL control flow expression or a DSL block.

A DSL control flow expression is a control flow expression, which could be a try, catch, finally, if, else, match, case, do or while but neither a for loop nor a for comprehension, and one or more of its direct child expressions are DSL expressions.

A DSL block is block expression, and one or more of its direct child expressions are generator operators <-, DSL blocks or DSL control flow expressions.

To create DSL expressions, the grammar of blocks should be changed to:

Block        ::=  BlockStat {semi BlockStat} [ResultExpr]
               |  Enumerators

Enumerators  ::=  Enumerator {semi Enumerator}

Enumerator   ::=  Generator
               |  Expr1

For example:

// This is a DSL block:
{
  b <- a
  c <- b
  f(c)
}
// This is a DSL control flow expression:
while (x) {
  b <- a
  c <- b
  f(c)
}

A DSL expression must eventually belongs to a for loop or a for comprehension, or it will not compile.

// Should not compile
def f(s: Seq[Int]) = {
  while (scala.util.Random.nextBoolean()) {
    a <- s
    println(a)
  }
}

Translation rules

The following translation rules should be added to existing for expression translation procedure:

  • A value definition p = e inside a for loop or for comprehension expression, where e is a DSL expression, is translated to p <- wrap(e), where wrap is an internal AST translation process (see below).
  • A generator p <- e inside a for loop or for comprehension expression, where e is a DSL expression, is translated to p' <- wrap(e); p <- p'.
  • A for comprehension for (p <- e) yield e', where e' is a DSL expression, should raise a compile error, unless the translation rule is specified in another proposal, which could be Rebindable for.
  • A for loop for (p <- e) do e', where e' is a DSL expression, should raise a compile error, unless the translation rule is specified in another proposal, which could be the Rebindable for.
  • wrap(if (e1) e2 else e3) is expanded to keywords.ifThenElse(wrap(e1), wrap(e2), wrap(e3))
  • wrap(do e1 while e2) is expanded to keywords.doWhile(wrap(e1), wrap(e2))
  • wrap(while e1 do e2) is expanded to keywords.whileDo(wrap(e1), wrap(e2))
  • wrap(e1 match e2) is expanded to keywords.matchCase(wrap(e1), wrapCaseList(e2)), where wrapCaseList is an internal AST translation process (see below).
  • wrap(try e1 finally e2) is expanded to keywords.tryFinally(wrap(e1), wrap(e2))
  • wrap(try e1 catch e2 finally e3) is expanded to keywords.tryCatchFinally(wrap(e1), wrapCaseList(e2), wrap(e3)).
  • wrap(try e1 catch e2) is expanded to keywords.tryCatch(wrap(e1), wrapCaseList(e2)).
  • wrap { ...stats; r } is expanded to for { ...stats } yield r.
  • wrap(e), where e is not a DSL expression, is expanded to keywords.pure(e).
  • wrapCaseList { case p0 => b0 ... case pn => bn } is expanded to { case p0 => keywords.left(wrap(b0)) ... case pn => keywords.right( ... keywords.right(keywords.left(wrap(bn))) ... )) }, where n is the index of each case clause, and keywords.right repeats n times.

Next steps

Single block for

Both for comprehension and for loop need two blocks of clause. It would be good if for without yield is allowed, like this:

for {
  b <- a
  c <- b
  c
}

However, the above syntax is not included in this proposal. Instead, it should be addressed in another proposal.

for with definitions

Definitions other than generator operator <- and value definition = in a DSL block are not covered in this proposal. There should be another proposal for lazy val, var, def, class, trait, object definitions. For example:

for {
  b <- a
  lazy val d = {
    c <- b
    c
  }
  e <- d
} yield e

!-notation

The generator operator <- is verbose. It would be good if we have a conciser syntax:

for {
  c = !f(!f(a))
} yield c

Which should equal to the following code.

for {
  b <- f(a)
  c <- f(b)
} yield c

The detail of the !-notation will be discussed in another proposal.

Rebindable for

It’s common to use loops or comprehension in an asynchronous context. In ECMAScript, we can create a for loop with await in an async function:

for (const element of collection) {
  await f(element);
}

However, Scala for comprehension can’t. The reason is this functionality requires a different type signature other than map/foreach, which should be instead the signature similar to traverse/traverseU in Scalaz or Cats. Unfortunately a Scala for comprehension is always translated to map on the collection. To allows for comprehension in an asynchronous context, we need a mechanism to overload the map method defined on the collection. This need will be addressed in another proposal.

5 Likes

That’s an interesting read, thank you @yangbo for writing this up!

This rule seems to conflict with the current rule that desugars to map calls.

This rule is incomplete, we should also pass wrap(e2) and wrap(e3) as parameters to ifThenElse.

Overall, the proposal seems very ambitious and introduces a lot of new things. Currently, explaining how for comprehensions are desugared is already a bit convoluted. I would be hesitant to generalize the desugaring as you propose.

Also, with these new rules is there a risk to introduce a runtime performance overhead when using the standard collections?

It would be interesting to compare your approach with what F# does.

2 Likes

It does not conflict with the current rule, because the current rule is applied to e' only if it’s not a DSL expression. However, it is indeed problematic. In fact the following code will never compile as long as we translate them to flatMap/map/foreach member methods, because these member methods defined on collections and futures conflict with each other.

def asyncLoop(futureSeq: Seq[Future[String]]): Future[Unit] = {
  for (future <- futureSeq) {
    string <- future
    println(string)
  }
}

The solution would be to translate them to rebindable keywords.flatMap / keywords.map / keywords.foreach. I can write down another proposal for details. But now I just removed this part from this proposal.

Updated.

Every proposal increases complexity. This proposal actually increases less complexity than alternatives. For example, neither Scala Continuations nor Scala Async has a set of standardized translation rules. They actually increase the hidden complexity and do not guarantee their behaviors. Also note that, unlike alternatives, this proposal does not introduce new keywords.

The complexity consideration is also the reason I would like to create multiple independent proposals to improve for expressions, not a single huge proposal.

No, because existing for comprehension will not be affected. Rules introduced in this proposal is applied to DSL expressions only, which were invalid syntax before this proposal.

Yes. Originally I thought the DSL interface should be minimal, e.g. generate code with only flapMap / pure, like what we did in Each, Dsl.scala, Monadless, Scala Continuations and Scala Async. But now I realized the F# approach may be better. The rules to translate control flow to functions are crystal clear, and they also allow libraries to create runtime that is optimized for a specific DSL, for example operator fusion in GPU programming.

3 Likes

If I may, I would like to question whether this is a direction we want to go with.

Railway oriented programming, a model that relies on the for comprehension in Scala, is a great approach for writing code that has at most two major paths – a happy and a sad path, where handling mostly indistinguishable failures can be made simpler and clearer, either in sequential or concurrent (Future) code.

I’m not sure this model holds for data flows which are not as simple as to have one or two paths. Once the data flow of a certain part of the code starts to diverge and form branches, I find it much more readable to have explicit control flows – if, else and pattern matching.

Seeing an “if or else” function in the midst of a railway of joint functions just feels wrong. I would say the same thing about Future.transform, Try.transform, Either.fold or Option(value).filter(condition).map(happy).getOrElse(sad).

This is just my opinion though, so take it with a grain of salt :slight_smile:

1 Like

I think the railway-oriented programing analogy is a bit flawed. Every line of code in an imperative programming language can throw an exception, so in a sense, proceeding to the next line is always the “happy path”, right? Doesn’t that mean that all code is railway oriented from the perspective of the interpreter that runs it?
So to me it seems that normal code is semantically not so different from code written using a monadic style. In the functional programming style the errors are values rather than a control flow jump.
At the moment we have a nice syntax for “successfully getting to the next line” (flatMap) in the value-oriented programming style, but not for many other control structures.

3 Likes

Exceptions and ROP are not alternatives of each other, but rather different tools for similar yet distinguishable enough roles. I simply wouldn’t use exceptions for plain “happy sad railways”.

Instead, ROP is an alternative to imperative control flow, where instead of declaring the branches of the flow using if then return else then return, one uses monadic composition. But really, my point is not whether one is better than the other, as they are both completely fine in my opinion.

My point is that currently with Scala, the monadic composition capabilities and specifically the for comprehension, dissuade one from writing a block of code that is heavily nested, and instead encourages one to separate the branching flow to other functions / code blocks.

One can still write a large branching control flow with a heavily nested if else / pattern match code block, but I would consider it a bad practice, and I would like to eliminate the opportunity of writing such code with monadic compositions.

2 Likes

They do this only because the tools for writing more complex control flow are missing from the standard library though.

That is not the case elsewhere in the ecosystem - we have monadic equivalents for:

All of those combinators compose together really well and generally make writing code in this style a lot easier!

So when I say ROP is not a good analogy - it’s a great analogy for explaining the interaction of Either and Option with map and flatMap, and you’ll note that was the context of the talk you linked above - but it’s a very limited view of what we can do with monadic types. You have lots of great options for monadic control flow!

We can write those combinators and they work great, but they aren’t currently supported by for - so that is why you commonly see less control flow in monadic code - because the tools aren’t there in the standard library. That is what proposals like this one aim to fix.

2 Likes

Since @julienrf has mentioned it I’m really interested in all of your thoughts about a syntax inspired by the F# computation expression.

Please excuse my idiosyncratic notation!

For some computation builder bldr:

// Builder expression
// 
// Returns a custom class implemented by the user.
// 
// It contains declarations of the DSL operations supported by this
// computation expression.
// 
// As a syntactic transformation only, unsupported syntax would trigger a compile error
// as the current for comprehension does now when it is used with types that don't support
// the required operations
//
// def bldr[F[_]]: BuilderType[F]
//
//
desugar('{ bldr[F] { expr } })
  ==>
    val $ = bldr[F] // $ would be some fresh name
    desugar('expr)

// Compound expressions in a builder
desugar('{ cexpr })
  ==>
    desugar('cexpr)

// Bare expressions in a builder
desugar('{ a })
  ==>
    a

// Unused enumerators in a builder
desugar('{ b <- a })
    desugar('a)

// Composing expressions
//
// def flatMap[A, B](fa: F[A])(f: A => F[B]): F[B]
//
// bare expressions:
//
desugar('{ a; expr })
  ==>
    $.flatMap(desugar('a))(_ => desugar('expr))

// with enumerators
desugar('{ b <- a; expr })
  ==>
    $.flatMap((desugar('a))(b => desugar('expr))

// Yielding values
//
// def unit[A](a: A): F[A]
//
desugar('{ return a })
  ==>
    $.unit(a)

// Possible optimizations
desugar('{ return a; b <- a })
  ==>
    $.flatMap($.unit(a))(b => f(b))
  ==>
    f(a) // via monad left identity

desugar('{ b <- a; return b })
  ==>
    $.flatMap(desugar('a))(b => $.unit(b))
  ==>
    a // via monad right identity

// def map[A, B](fa: F[A])(f: A => B): F[A]
//
desugar('{ b <- a; return f(b) })
  ==>
    $.flatMap(desugar('a))(b => $.unit(f(b)))
  ==>
    $.map(desugar('a))(f) // via map/flatMap coherence law
    
// For loops
//
// def traverse[G[_], A, B](ga: G[A])(f: A => F[G]): F[G[B]]
//
desugar('{ for (b <- a) { c } })
  ==>
    $.traverse(a)(b => desugar('c))
    
// Conditionals
//
// def ifThenElse[A](cond: F[Boolean])(thenExpr: F[A])(elseExpr: F[A]): F[A]
// 
// def zero[A]: F[A]
//
desugar('{ if (a) then b })
  ==>
    $.ifThenElse(desugar('a))(desugar('b))($.zero)

desugar('{ if (a) then b else c })
  ==>
    $.ifThenElse(desugar('a))(desugar('b))(desugar('c))

// While loops
// 
// def while(cond: F[Boolean])(body: F[Unit])
//
desugar('{ while (a) { b } })
  ==>
    $.whileLoop(desugar('a))(desugar('b))

// Lifting plain Scala code into the builder type
// 
// def delay[A](a: => A): F[A]
//
desugar('{ do a })
  ==>
    $.delay(a)

Obviously this is not really close enough to a formal specification of such a desugaring but I hope this helps illustrate the idea.

You can see what the experience of defining a new builder type would be like in the following code sample, along with some example expansions of the syntax.

I expanded a bit on the advantages and disadvantages of this in the other thread, but there are some unanswered questions about porting such a design to Scala:

  • How could the compiler know that this builder class triggers the desugaring if there are no enumerators present? Extending a magic trait and other similar things seem kind of gross and also don’t really fit into a syntactic expansion of the desugaring.
  • Could this be implemented by making the builder factory method a Scala 3 macro that accepts the block of code to desugar? Would that limit the syntax that could be used within the block to things that superficially typecheck before the macro expansion or would they just have to parse? The former is probably too limiting.
  • Is reusing <- a bad idea? What else could we use? It collides with the for syntax, especially when using the results of a for expression, i.e. deploymentIds <- for (vertName <- vertNames) { Vertx.deployVerticleL(vertName) }

I assume we don’t need translation if no <- is present. Do you have any use cases where the assumption is invalid?

No. Both for translation and new rules introduced from this proposal are name based, which require the ability to produce untyped AST. Unfortunately Scala 3 macros can’t produce untyped AST.

The code in your example is valid Scala 2 code, which should behave exactly the same as before even with this proposal.

Interesting take! But I think we don’t really need to copy F# verbatim here. F# does certain things the way it does, because it lacks other features that Scala possesses even today. Needing to have an explicit “builder” object is one such example. In Scala, we have implicit parameters, extension methods and higher-kinded types. We should do fine with only those. What we would need, is a new do construct.

Let me illustrate it with this short snippet:

def fetchAndDownload(url: Url): IO[Data] =
  do {                                          // marks the start of the computation expression. alternatively
    val urlStripped = strip(url)                // usual variable binding; it's nice you can do that even as the first thing
    val! data = downloadData(urlStripped)       // `val!` desugars to `.flatMap`
    val processedData = processData(data)       // another usual variable binding
    if!(callVerificationService(processedData).map(_.isError)) {  // `if!` with then branch `IO[Unit]` and without else branch desugars to calling `.whenA`
      throw! CustomError("wrong data")          // `throw!` desugars to calling `.raiseError`
    }
    use! monitor = createMonitor(processedData) // `use!` desugars to calling `.use`
    do! notifyMonitor(monitor)                  // `do!` on `IO[Unit]` desugars to calling `.flatMap(() => ...)`
    processedData.pure                          // `IO[Data]` is expcted here, so we produce it by calling the `.pure` extension method
  }

The precise names of the extension methods this desugars to is not terribly important, I just used the names as they are in cats toady. But Scala authors could choose different names, and then, I’m sure, cats, zio & co. would accommodate.
What is important, is that the user-visible syntax of the “monadic” operations is intentionally similar to that of their imperative counterparts, so val/val!, if/if!, throw/throw! and so on…

6 Likes

Agreed. I would like to highlight the ability of reifying control flow in this proposal. With the help of reification, you can create type-level interpreters to hande really complex usage of imperative control flows.

My instinct here is that Scala has already far exceeded its novelty budget and that we should not be adding core features that take advantage of Scala’s most complex constructs in the surface syntax of the language. The thing that is attractive to me about the F#-like solution is that the desugaring is extremely simple, it can happen in the parser only, it does not use any new keywords, it doesn’t require extra bookkeeping in the parser to keep track of enclosing constructs. With the inline and erased keywords in Scala 3 we could also eliminate the intermediate builder object and the associated method calls. On the other hand I have a suspicion that it could be confusing that those keywords are reused.

This is actually by far my preferred approach, and for a second or two I hoped that this would be possible in Scala 3 because do {} while loops are being deprecated. Unfortunately though, it was deprecated to make way for for {} do {} so the concept of a do {} block is already in use for something different. If we can come up with an alternative keyword to do {} it doesn’t seem like an insurmountable problem though.
There are a couple of other things about the Ocaml/F# approaches that I think will cause confusion and consternation. Firstly I don’t think we would get a lot of buy in to the idea of adding a new symbol to the core syntax of Scala (the ! or + enhancing existing keywords) as a lot of the recent changes to the language seem to be aimed at addressing perceived criticisms of the language - one of which is overuse of symbolic operators. Secondly the equals = symbol is used to bind names that desugar to flatMap calls. This makes FP folk very upset because it breaks referential transparency - the resulting code does not behave the same under inlining.

Yes, this aspect is really interesting. :+1:

I’ve seen a fair number of language-modification proposals that want to make apparently-plain Scala desugar into various other things which aren’t plain Scala. I understand the motivation for this. Just want to say:

  1. I’ve been down that road, and IME it ends in despair. The stuff you’re trying to protect people from (like existing syntax that describes effectful programs, I guess?) are things they probably don’t want or need protection from. At least for my use cases, I’ve found it’s better to just follow reasonable idioms in APIs. Use monadic constructs when it makes sense to do so; people aren’t afraid of learning how to use that. Making it so that a piece of code which uses monadic constructs looks like it’s using monadic constructs (e.g. with for) is good, not bad – if it looks like imperative Scala code encased in an easily-missed mystery {} block, it doesn’t end up looking all that much aesthetically better, but makes it much more confusing for the reader.
  1. If one really wanted to disguise monadic code as regular code, there’s now a built-in (!!!) compiler phase which does that kind of thing. It sort of snuck through, but there’s a -Xasync flag in the latest compiler that does a bunch of double-secret rewrites of imperative code to be… something else which is also double-secret. Maybe this could be utilized to accomplish what’s being sought here? That would obviate the need for new val! kinds of tokens, since normal vals are already rewritten to… something. If you figure it out, documenting that process would also be really useful to everyone who doesn’t understand exactly what -Xasync does or why it was merged without much in the way of SIP process.

It has been implemented in Dsl.scala. You can checkout the example here. The syntax could be better if we have an improved version of for comprehension.

1 Like

You are basically saying Python, C#, ECMAScript, Kotlin, and even C++ 20 are all wrong, and Scala will never be a language for those developers who are familiar with generators and coroutines.

The purpose of this proposal is to provide a standardized rewrite rule as an alternative to Scala Async. The only reason why val! is not desirable is that mainstream languages don’t use that. Despite of the complexity of the runtime and rewrite rules (including the ability of reifiable control flow), the users should be able to create coroutines and generators just like other mainstream languages with the help of this proposal.

Nope, not saying that at all. I’m not saying anything is wrong, just offering that I’ve been down the road of “monadic things are hard, let’s make them look like they’re not hard” starting with writing Future-wrapping DSLs on top of the original CPS plugin, which seems like 100 years ago. It was a lot of effort, and it turned out that understanding the compile-failure modes of the resulting downstream code was a lot more effort than just learning about Future and “monadic”(-ish) syntax.

I do understand that, and I think that’s great. From what I understand, though (and this is what I was trying to say), there’s already a de-facto standard rewrite rule that’s been merged into the language. It claims to be flexible to various F[_]s, so it might be the place to start. Certainly if you’re not happy with it, I’d be interested to hear some noise about that, since I’ve no idea what its rewrite rules are and they don’t seem particularly documented nor were they discussed in a meaningful way on this forum. I guess what I’m trying to say is – I think we all have the same complaint, but let’s all complain about the same thing :slight_smile:

4 Likes

Note, “complain about the same thing” meaning, here – there’s a flag which does some purportedly general “non-monadic to monadic” transformation which, given that it’s merged into the language, should presumably be the one way to accomplish such a thing. But it’s not general if it’s not documented – since nobody else can implement it – and there was no opportunity to discuss what the transformation should be in order to generalize most completely.

1 Like

One more thing @yangbo – I get that there’s more to what you’re proposing, in that it also requests some ability to treat raw ASTs (pre-typing) in order to virtualize them. I really don’t think that’s going to happen, although I can think of a lot of great use cases myself – Scala isn’t going to become racket. There’s already some pretty powerful tools for monadic DSLs, and if there’s a general-purpose transformation of imperative to monadic – as long as it’s documented and accessible – I think that would answer a lot of such use cases (at least, when IDEs catch up with it)

Virtualizing AST commonly happens. Existing for comprehension, pattern matching, and scala.Dynamic are all AST virtualization. Even though we did not specify the translation rules in -XAsync, it is definitely translating AST to name based function calls.