Quiet for comprehensions

This is far from a serious proposal yet, and it’s entirely possible that what I’m suggesting here is infeasible – I’m not a compiler engineer, just a longtime hardcore Scala programmer. And this is more an initial concept, not deeply thought out. So consider this food for thought.

I’ve been watching the experiments towards adopting async/await with a fair amount of trepidation. Honestly, none of it feels like Scala to me - it feels like an ill-advised attempt to be trendy, with machinery that doesn’t have the generality of “good Scala”.

So I stepped back and asked, “What is the problem here? If I was king of the forest, how would I solve it?” And I find myself thinking that the problem isn’t that our concepts are all that hard – after all, async tends to spread virally through the code the same way monads do, so folks should be used to that. The problem is that for comprehensions just plain look weird and intimidating.

So in the spirit of the recent moves towards “quiet syntax”, I suggest “quiet for comprehensions”. To wit:

We elevate the <- operator, making it legal inside of any val. When detected, the compiler interprets that as the opening of what is effectively a for comprehension, covering the rest of the block. The last expression of the block is interpreted as a yield.

The result is that you can write code along these lines:

def userLogin(username: String, password: String): Future[UserInfo] = {
  val dbReq = CheckUserLoginRequest(username, password)
  val userId <- userDb.login(dbReq)
  val profile <- profileDb.fetch(userId)
  val avatar <- avatarDb.fetch(userId)
  UserInfo(profile, avatar)
}

That’s approximately “direct syntax” in a nice clean way – quieter than current comprehensions but fully in line with how they work. I think it’s orthogonal to async/await, and less noisy. It’s limited in the same ways as for, but good enough for the large majority of functions that I write, and I suspect would be a little more comfortable to people coming newly to Scala.

Anyway, something to consider. If it was doable, it would make a lot of modern Scala code less boilerplatey without changing anything other than a bit of surface syntax.

3 Likes

To confirm, this is how the userLogin example would desugar (or at least it’s for-comprehension equivalent)?

def userLogin(username: String, password: String): Future[UserInfo] = {
  val dbReq = CheckUserLoginRequest(username, password)
  for {
    userId <- userDb.login(dbReq)
    profile <- profileDb.fetch(userId)
    avatar <- avatarDb.fetch(userId)
  } yield UserInfo(profile, avatar)
}
1 Like

That’s the intent, yes. At least, more or less – since for is itself just syntax sugar, what I’m proposing is effectively an alternate way to get to the same underlying nested flatMap+map calls.

This is very similar to the “backpassing” syntax in Roc, although it doesn’t assume flatMap and map: Roc Tutorial

I think this is the right general direction, but I think using <- is a mistake because it doesn’t work with expressions (e.g. if isUserActive then ... else ...).

Instead, I’d suggest using a keyword like do, which is gone in Scala 3 anyway.

Then your example would become:

def userLogin(username: String, password: String): Future[UserInfo] = {
  val dbReq = CheckUserLoginRequest(username, password)
  val userId = do userDb.login(dbReq)
  val profile = do profileDb.fetch(userId)
  val avatar = do avatarDb.fetch(userId)
  UserInfo(profile, avatar)
}

One more character to type, for the added advantage that you can use the syntax in expressions (e.g. if do isUserActive then ... else ...).

In any case, though I suspect any suggestion along these lines will be discarded (because it allows, indeed, even encourages use of the dreaded M-word), it’s a very conservative change that would make functional programmers such as myself very happy, while not shutting the door to more experimentation down the road via context functions, capability types, etc.

In my view, Scala should not unfairly discriminate against the functional programmers who have and continue to invest so heavily in the Scala OSS ecosystem. Yes, pure functional programming is not for everyone, but it is for some people, and a little work showing appreciation for the ones crazy enough to be unafraid of (and even admire) the ‘M’ thingies would go a long way toward realizing the ‘Big Tent’ vision of Scala that Odersky laid out so long ago.

The fact that monadic effect systems might become easier-to-use under a proposal like the one laid out in this thread doesn’t preclude other types of effect systems from being brought into Scala, including those not based on pure functional programming.

2 Likes

Interesting idea; one potential downside is that with traditional do notation it is very clear what is the scope of the effectful code:

// hypothetical syntax
def userLogin(username: String, password: String): Future[UserInfo] = do
  val dbReq = CheckUserLoginRequest(username, password)
  val userId  <- userDb.login(dbReq)
  val profile <- profileDb.fetch(userId)
  val avatar  <- avatarDb.fetch(userId)
  UserInfo(profile, avatar)

In the proposal you’d need to scan the function for do occurrences, no?

2 Likes

What would be the proposed desugaring for this?

if do isUserActive then foo else bar

It would be awesome if Scala improved upon the current state of for-comprehensions.
We don’t have to look far for inspiration, we can do something similar to what F# is doing with so called Computation expressions

For Scala the code then could look something like this

def userLogin(username: String, password: String): Future[UserInfo] = for
  val dbReq = CheckUserLoginRequest(username, password)
  val! userId = userDb.login(dbReq)
  val! profile = profileDb.fetch(userId)
  val! avatar = avatarDb.fetch(userId)
  yield UserInfo(profile, avatar)

A general de-sugaring table could be like this

ordinary code analogue for-comprehension sugar for
val ... = ... val! ... = ... (legacy ... <- ...) flatMap
if ... then ... else ... if! ... then ... else ... ifM
if ... then ... if! ... then ... whenM
throw ... throw! ... raiseError
try ... catch ... try! ... catch ... handleErrorWith
while ... do ... while! ... do ... whileM
use ... = ... (hypothetically) use! ... = ... use

The point here is to

  • enable as many analogues to syntactic constructs from ordinary code (val, try, if, …) as possible
  • make them look as similar to ordinary code constructs as possible. Appending ! is one possible way to do that. val! is better analogue to val than <- can ever be.

Previously discussed on this forum:

On Reddit:
https://old.reddit.com/r/scala/comments/y6zyx9/the_case_against_effect_systems_eg_the_io_data/#ist9r0f

5 Likes

A poorly-fleshed out thought: this seems like it could be a generalization of Rust’s ? for early return to the Future monad. If do or whatever syntax was rewritten by the compiler a local return for general monads, that would be a huge win for the language in general by making Result much easier to use and would get this behavior “for free” for Futures too.

I’m imagining a method called def orElseReturn: Either[T, U] or something that gets specially desugared by the compiler to a local return just like in Rust. If the function returns a Right then the U is the result of the expression, otherwise the compiler returns out of the surrounding function with a the Left T.

I’m sure this has been talked about before elsewhere.

I recommend watching Odersky’s full presentation on Direct Style Scala.

As discussed,

  1. This effort brings continuations to Scala, not which solves many, many more problems than async-await. Examples include error handling, effect tracking, concurrency, and more.
  2. Direct-style programming is simpler than the monadic style and composes effortlessly. There’s no need for monad transformers or higher order methods like traverse. You will be able to set breakpoints and debug your code easily (at least on the JVM with project Loom). It dramatically simplifies your code.

Direct Style is not just a replacement for monadic for-comprehensions but is a whole new paradigm that requires time to fully appreciate.

Your example might look like this in direct-style:

def userLogin(username: String, password: String): Future[UserInfo] =
  Future:
    val dbReq = CheckUserLoginRequest(username, password)
    val userId = userDb.login(dbReq).value
    val profile = profileDb.fetch(userId).value
    val avatar = avatarDb.fetch(userId).value
    UserInfo(profile, avatar)

Which is similar to the suggested for-comprehension change, but more general.

If this code is executing on the JVM, we can do even better. Assuming we are using a Virtual-Thread-based web framework like Helidon Nima, your userLogin method is already being called on a Virtual Thread, so we can remove Future entirely (since blocking is free on Virtual Threads). userLogin then becomes

def userLogin(username: String, password: String): UserInfo =
    val dbReq = CheckUserLoginRequest(username, password)
    val userId = userDb.login(dbReq)
    val profile = profileDb.fetch(userId)
    val avatar = avatarDb.fetch(userId)
    UserInfo(profile, avatar)
4 Likes

The fact that monadic effect systems might become easier-to-use under a proposal like the one laid out in this thread doesn’t preclude other types of effect systems from being brought into Scala, including those not based on pure functional programming.

In what way are continuations not pure functional programming?

I think ZIO and CE can adapt to direct-style and greatly benefit their users in the process. The syntax will be easier to write and debug, but retaining the same laziness and concurrency operators. Scala-fix can help to migrate for-comprehensions to direct style.

1 Like

In both your examples profileDb.fetch & avatarDb.fetch will execute sequentially when most likely they should be executed concurrently. Executing them concurrently will require additional bells and whistles (like zip operator) when by all means it should be done automatically like compiler/processor instructions reordering.

Also, no examples of cancellation and RAII where given in the mentioned presentation. And structural concurrency, cancellation and resource allocation/release are the most interesting things in effects system like CE or ZIO (at least for me).

1 Like

To have them execute concurrently, you could rewrite the example as:

def userLogin(username: String, password: String): Future[UserInfo] =
  Future:
    val dbReq = CheckUserLoginRequest(username, password)
    val userId = userDb.login(dbReq).value
    val profile = profileDb.fetch(userId)
    val avatar = avatarDb.fetch(userId)
    UserInfo(profile.value, avatar.value)

Yes, but it is the very same pitfall we have now with Futures: subtle differences between val a: Future[?], def b: Future[?], etc.

A lot of people do not know the difference between

for
  a <- foo
  b <- bar
yield (a, b)

val fa = foo
val fb = bar

for
  a <- fa
  b <- fb
yield (a, b)

And now we gonna shoot the same leg with different bullet?

3 Likes

Yes, but it is the very same pitfall we have now with Futures: subtle differences between val a: Future[?], def b: Future[?], etc.

This is because Future is eager; it’s tangential to the direct-style discussion.

You could easily write a lazy Future (like ZIO or CE) that supports direct-style.

Hopefully, Virtual Threaded web frameworks become popular, and the majority of users get async performance without having to directly use Future or any concurrency monad again. Those that do need to write concurrent code can do so much easier with direct-style, whether it is with an eager or lazy concurrency primitive.

That’s something we do not know. Presentation says that “direct style continuations” are very much WIP and who knows if there would be support for .? and .value in user land?

In Rust, which was used as an example, ? is possible to use only with Rust own Option and Result and async/await – only with Rust own Future. No support for a third party types.

Any adoption of VTs are depended on JVM 21 LTS. Which is at least September 23. And looking at numbers of Java 11/17 adoption we are talking about 2030. Scala itself supports Java 8.

So any direct or monadic style needs to work with what we have now and (maybe) switch to VTs for runtime if they are available. But the interface must be universal.

And I will say it again: cancelation and RAII. Project Loom does nothing about it, so constructs like try-with-resource/Using should be at least compatible with direct style.

3 Likes

I want to throw out one word of caution here. One of the common complaints about Scala, especially from the standpoint of educators, is that there are too many ways to do the same thing. Every new special syntax for things provides another argument for some not to adopt Scala as a teaching language. I understand that many of these options improve lives for professional developers, but there are also costs associated with having special syntax that I want to make sure are being considered.

5 Likes

Yep, agreed. That’s one of the reasons my suggestion is as conservative as it is: it’s just a modest simplification of the existing syntax, that I think is a bit easier to teach. It’s aimed at beginners at least as much as experienced engineers. (If it worked, I suspect that for would become the “advanced” syntax.)

I agree, but I think it goes beyond teaching. You want code that’s readable. Yes, we want to avoid boilerplate and we want powerful abstractions. But by hiding too much, you run the risk of making code harder to read for outsiders. I’m not judging this particular proposal, but, as a general principle, let’s not be too quiet and obscure the code’s intent for the sake of typing fewer characters.

3 Likes

The thing that I don’t understand is how this is an improvement over a purely library-level solution that is slightly more explicit but in a Li Haoyi-like style where simplicitly of the most-used path is an extremely strong goal.

For example, if you write

def userLogin(username: String, password: String): Future[UserInfo] =
  val dbReq = CheckUserLoginRequest(username, password)
  val userId = future:
    userDb.login(dbReq.?)
  val profile = future:
    profileDb.fetch(userId.?)
  val avatar = future:
    avatarDb.fetch(userId.?)
  future:
    UserInfo(profile.?, avatar.?)

even if I don’t tell you what anything means or how it is implemented, isn’t it clear what is going on?

If you forgot a .? and the compiler complained about needing Foo instead of Future[Foo], then it would be even clearer. No?

4 Likes