Hello,
I’m the author of better-monadic-for compiler plugin (bm4 for short). It’s been brought to my attention that dotty would only allow pre-typer plugins in nightly builds, which means there cannot be a version of bm4 for production Scala 3.
Originally, the goal was to allow writing monadic code with destructuring. I.e. the code like
for {
(a, b) <- computeThings
} yield a + b
could fail to compile depending on whether a value returned by computeThings has a withFilter method, and a common workaround is to use a temporary variable and destructure with a subsequent assignment:
for {
temp <- computeThings
(a, b) = temp
} yield a + b
Luckily, #2578 was fixed in dotty, so for-comprehensions don’t filter until asked to with a case keyword, greatly reducing the need for bm4 in the first place.
Two other features of the plugin that seems to be quite popular are ability to introduce values into implicit scope in the middle of a for-comprehension and the removal of map call as the last action. These are oriented towards users of effect monads (cats-effect IO/monix Task/ZIO) and related utilities (e.g. Resource and fs2.Stream).
These are things I propose Scala 3 includes directly. I’ll list my thoughs on that below.
The plugin also has removal of extraneous generation of TupleN from value bindings (reported as #2573), but unlike the two above it doesn’t affect how you write code with for-comprehensions, only its runtime characteristics, so it’s of lower priority.
Eliminating map from for-comprehension
Motivation
Scala for-comprehensions desugar to nested flatMap+map calls, for instance:
for {
a <- f1
b <- f2(a)
c <- f3(a, b)
} yield c
becomes:
f1.flatMap { a =>
f2(a).flatMap { b =>
f3(a, b).map { c => c }
}
}
Notice that there is .map { c => c } call at the end. With any valid Functor from cats or scalaz it is guaranteed to not change anything in the result, and this is true without these libraries for most types from standard library (Option/Try/Either, immutable collections and Future). So, all it does here is allocate more objects for no good reason.
With effect types like IO, it gets worse, as this property makes for-comprehension unfit for describing infinite loops. I.e. code like such:
def loop: IO[Unit] =
for {
_ <- IO(println("Doing periodic things"))
_ <- IO.sleep(1.second)
_ <- loop
} yield ()
will slowly but surely generate garbage elements with the .map((_: Unit) => ()) construct that desugaring generates, eventually crashing the application with OOME. better-monadic-for can detect such constructs and eliminate them, but without it the only option user has is to rewrite the definition entirely, not using for-comprehension.
Proposals
I see two possible solutions:
- Do this elimination in scalac
- Extend for-comprehension syntax to support use cases where you do NOT want any
map.
Option #1 would provide quite an intuitive behavior, but would make desugaring itself a bit more complex. To handle cases such as loop above, it’s also required that it happens after typer (this is how bm4 plugin does it). It is also a double-edged sword for migration. Nothing would change for people who use bm4 already, and for immutable types it would bring performance improvements, but will break code that relies on that .map. I haven’t seen any code like this in the wild, but it could exist, and it will silently break if migrated as-is.
Option #2 would require people to know about it’s existence and also we would have more options for something we already have. For example, we would allow loops to be written as:
def loop: IO[Unit] =
for {
_ <- IO(println("Doing periodic things"))
_ <- IO.sleep(1.second)
} yield <- loop
that desugar into
IO(println("Doing periodic things")).flatMap { _ =>
IO.sleep(1.second).flatMap { _ =>
loop
}
}
This doesn’t conflict with Scala 2.x, as <- is a reserved character sequence and could not be used as an identifier. All existing code (that is not affected by #2578 fix already) would retain current behavior.
Declaring implicits in the middle of for-comprehension
Implicits (givens? I’m not sure what’s the current name) are the defining feature of Scala, but limited syntax of for-comprehension makes it impossible to define them within one. Every binding is effectively a val, with no other options. defs can be simulated with function values, vars - with AtomicReference or something alike, but nothing in Scala of today would let you define an equivalent of implicit val. For effect types in particular, there are things that have to be created in IO/Resource context, such as clients, thread pools and tagless final algebras, but using them in 2.x requires either passing implicit parameters explicitly:
def createIOPool: IO[ExecutionContext]
class HttpClient[F[_]]
object HttpClient {
def create[F[_]: Async](implicit pool: ExecutionContext): F[HttpClient]
}
def doPolling[F[_]: Monad: HttpClient](url: String): F[Unit] = ???
for {
blockingPool <- createIOPool
httpClient <- HttpClient.create[IO](implicitly, blockingPool)
_ <- doPolling("http://example.com")(implicitly, httpClient)
_ <- doPolling("http://scala-lang.org")(implicitly, httpClient)
} yield ()
or ditching for-comprehensions so that you can use a lambda with implicit:
createIOPool.flatMap { implicit blockingPool =>
HttpClient.create[IO].flatMap { implicit client =>
doPolling("http://example.com").flatMap { _ =>
doPolling("http://scala-lang.org")
}
}
}
better-monadic-for adds implicit0 construct which can be used as a pattern inside for-comprehension directly:
for {
implicit0(blockingPool: ExecutionContext) <- createIOPool
implicit0(httpClient: HttpClient[IO]) <- HttpClient.create[IO]
_ <- doPolling("http://example.com")
_ <- doPolling("http://scala-lang.org")
} yield ()
I’ve found this to be very helpful occasionally. Particularly at the application entry point, where everything is initialized and dependencies are wired together, rewriting to version without for-comprehensions is extremely tedious and creates quite a bit of nesting. It’s also supported by Intellij IDEA since 2019.1.
Proposal
Allow alias given definitions at the left hand side of a generator or a = binding:
for {
given blockingPool as ExecutionContext <- createIOPool
given as HttpClient[IO] <- HttpClient.create[IO]
_ <- doPolling("http://example.com")
_ <- doPolling("http://scala-lang.org")
} yield ()
This can actually compile a specific case (given as Type where Type has no type parameters) with 2.12 syntax if somebody were to write e.g.
object as {
def unappply(a: (Any, Any)): Option[(Any, Any)] = Some(a)
}
but due to given being a keyword, it would require rewrite to update already.
P.S. While we’re talking about for-comprehensions, maybe we should also address scala bug #907 too? I can’t do it in the plugin without ridiculous parser hacking.
