Adding try-with-resource to Scala

I recently opened up a PR (https://github.com/scala/scala/pull/6347) to add scala.util.Loan, an implementation of try-with-resource, to the Scala library. @lrytz suggested that it might be better to extend the native try construction. I just want something that works like Java’s try-with-resource in Scala (language or library). Is it time to have a simple try-with-resource construction in Scala? WDYT?

FYI: There appears to be a tangentially related discussion on the topic of automatic resource management in https://contributors.scala-lang.org/t/automatic-resource-management-library/ .

2 Likes

I agree this feature is missed in Scala.

Approximating the feature with combinators, like in your PR, does have some drawbacks (generated code is messier, harder to debug, less efficient etc; refactoring a try to a Loan (or back) has a cost higher than benefit it gives you.

Before I discuss a compiler change, it’s worth sketching out what sort of API would be possible with a macro:

closing {
  val a = newFileStream
  val b = newFileStream
  try {
    f(a, b)
  }
}

The macro closing could checks that its argument is a block containing a group of ValDefs followed by a concluding Try, and then rewrite the finally accordingly. A variation on this might make the user-written try optional.

The nice thing about this macro is that it all typechecks before the macro expansion, so IDEs don’t need to special case it, and can be used on existing Scala versions. I’m happy to help out with the implementation if there is interest in this idea.

How about native support for try? The Java syntax looks okay to me, and generally we try not to needlessly differentiate from control flow syntax familiar to Java programmers.

try (val a = newFileStream; val b = newFileStream) {
  f(a, b)
}

It looks like a clear small win to me.

The costs to consider:

  • modifications to the compiler and spec (low)
  • modification to tools (IDEs, syntax highlighting) (many * small)
  • updates to documentation, books, training etc (many * small, but not urgent)
  • AST change likely to break compiler plugins, meta-programming (medium)

An open question is when to desugar the try-with-resources to a vanilla try. The sooner we do it (parser or typer), the less toolchain breakage.

2 Likes

The downside of these Java-compatible proposals is that they are all unsafe by default. It’s on the user to use the more verbose syntax to make them safe. Even though interoperability with the unsafe Java APIs is desirable it would be nice to make Scala’s native version safer, just like we encourage Option even though we do support null.

2 Likes

Scope-requiring staff like suggested by @densh (e.g. here) in the standard library looks exactly like you are saying: it is imposingly safe and each time you try to go unsafe, compiler catches you.

How about reusing the for syntax (making the resources monads)?

for (a <- newFileStream; b <- newFileStream) {
  f(a, b)
}
6 Likes

Monads do not seem to fit the idea because they represent just dependent calculations, so there will be nothing to call at the end of such for to perform cleanup automatically.

And, as Martin mentioned in some of his lectures about Dotty (when introducing automatic implicit parameters when return is a function with implicit parameter), monads are rather about sequencing than scoping and actually monads is a too much powerful construct than it is actually needed for organization of the scoping.

And again, what with this type of syntax would prevent a coder from forgetting to fee resources? Scoping approach I mentioned above does such prevention. Monadic-like approach seems to not.

Also think about a situation when you have only s single resource to control. It looks strange to put a single resource acquiring in a monadic-like for. Scopes I’m mentioning looks the same whether you have a single resource or twenty of them.

I like @ctongfei’s suggestion. Using for comprehensions seems to be the more idiomatic solution here (but without necessarily making resources monads). There is no need for introducing any extra features to Scala and making the language more complex as a result.

What’s wrong with this? It seems to do what’s required:

  trait RscHandler[A] {
    def map[R](f: A => R): R
    @inline final def flatMap[R](f: A => R): R = map(f)
    @inline final def foreach(f: A => Unit): Unit = map(f)
  }
  case class FileHandler(filePath: String) extends RscHandler[scala.io.Source] {
    @inline def map[R](f: scala.io.Source => R): R = {
      val src = scala.io.Source.fromFile(filePath)
      try f(src)
      finally src.close()
    }
  }
  
  for {
    orders <- FileHandler("data/orders.tbl.1")
    lineitem <- FileHandler("data/lineitem.tbl.1")
  } {
    println(orders.getLines().next() + lineitem.getLines().next())
    println("Done.")
  }

No, it doesn’t look strange to me to put a single resource in a for comprehension, and yes it also looks the same whether you have a single resource or twenty of them.

3 Likes

Personally I think that usage of for-comprehensions for lots of unrelated stuff is not so good.

Moreover, @densh’s proposal of scoping it implementable in the current Scala, so it also does not impose any change in the language. Comparing yours

and @densh’s

Scope { (implicit sc) => // this parameter can be omitted in Dotty
  val orders = FileHandler("data/orders.tbl.1")
  val lineitem = FileHandler("data/lineitem.tbl.1")

  println(orders.getLines().next() + lineitem.getLines().next())
  println("Done.")
}
// at this point all FileHandlers are closed.

the second looks much better to me. Moreover, in this case

  val orders = FileHandler("data/orders.tbl.1")

won’t compile without the Scope. But using for-comprehensions style, you can always forget to put your FileHandler creation into a for.

Moreover, using Scope-style, it’s easy to do conditional and nested stuff, like

Scope { (implicit sc1) => // parameter can be omitted in Dotty
  val foo = FileHandler("whatever1")

  if ( decision(read(foo)) ) {
    Scope { (implicit sc2) => // parameter can be omitted in Dotty
      val bar = FileHandler("another/file")

      // Do whatever with both foo and bar
    }
    // Here bar is closed
  }
}
// Here foo is closed

It doesn’t matter if FileHandler("data/orders.tbl.1") compiles without the for, because the result cannot be directly used as a file. You’ll have to call map manually, which will work out as expected.

I disagree, but that’s subjective. Anyway, I think we agree that this is a library design problem, not a language extension one (as opposed to what was floated earlier in this thread).

2 Likes

Yes, I agree with this part: it doesn’t look like a language extension need.

1 Like

“Just dependent calculations” describes the identity monad. More generally monads allow effectful dependent calculations. Managing resources like file handles can be modeled as an effect.

You cannot forget to free resources because you cannot allocate them without running the interpreter for the monadic computation. Leaking effectful values (like raw file handles) out of a computation as part of its result can be prevented by using abstract types for them. If such a value leaks, it is in an invalid state, but this doesn’t matter because you can’t do anything with it.

2 Likes

I don’t see how this is unrelated. The whole point of for is that it provides a consistent syntax for “flatMappy” sorts of operations – stuff that nests and flattens naturally – and this seems like a great fit.

IMO, @LPTK’s suggested implementation is spot-on: it’s fundamentally safe (the resource only exists inside the safe zone), it’s consistent with standard Scala syntax, and it lets you nest multiple resources in a natural way. Sounds right to me.

Not that any of this is mutually exclusive: since it’s all perfectly fine userland code, there’s no reason not to try both and let the market decide. But the monadic version looks more natural to me, personally…

3 Likes

This is similar to how it’s done in better-files (Lightweight ARM)

for {
  in <- file1.newInputStream.autoClosed
  out <- file2.newOutputStream.autoClosed
} in.pipeTo(out)
// The input and output streams are auto-closed once out of scope

See relevant code: ManagedResource.scala.

3 Likes

The problem with try-finally for resources is that if f(src) throws exception e0 and src.close() throws exception e1, then the resulting exception is e1 and the e0 exception quietly disappears. Java’s try-with-resource would throw e0 with e1 suppressed.

@densh 's Scopes didn’t suppress an exception from the finally block on the exception from the main try block. This can of course be fixed.

The biggest problem with resource management is not how you want to syntactically represent it (i.e. do you want to use monads or implicit effect’s or lifetime loan or RAII) but rather the fact that exceptions make it really hard to make correct resource management that also composes nicely, and these problems come up again and again (i.e. you can see https://github.com/typelevel/cats-effect/issues/88 for a recent discussion on this topic).

From what I have seen, the only languages that really claim to correctly implement resource management are languages where either exceptions don’t exist or exceptions do exist but you can’t (or almost never) should catch them (C++/Rust fall into this territory).

Because Scala is based on JVM which interopts with Java, we have to deal with the fact that in Scala/Java people use exceptions as another control flow for error management (rather then just using exceptions only for really exceptional situations, i.e. exceptions cause your app to crash because it can’t behave correctly otherwise and you never should catch exceptions).

I think we also need to focus on the above point, because there is little reason to try and come up with correct resource management if its not really correct (or putting it more accurately, we need to determine what is correct in this context).

1 Like

I think it’s time to revisit this topic. Scala has for some time had scala.util.Using for resource management but my main issue with it is that it’s not type-safe when using more than one resource. Here’s an example:

import java.io.{BufferedReader,FileReader}
import scala.util.{Try,Using}

def readFirstLineFromFile(path: String): Try[String] = Using.Manager { use =>
  val fr = use(new FileReader(path))
  val br = use(new BufferedReader(fr))

  br.readLine()
}

If you forget to wrap either or both of the resource acquisitions in the use function, then you have a resource leak that compiler doesn’t catch. I’d like to propose using a for-comprehension:

def readFirstLineFromFile(path: String): String =
  for
    fr <- AutoClose(new FileReader(path))
    br <- AutoClose(new BufferedReader(fr))
  do
    br.readLine()

I know that a for-comprehension was tried before and found to have issues especially with the .map method: 2.13 scala.util.Using: surprising behavior in for-comprehensions, `.map` method is non-compositional · Issue #11225 · scala/bug · GitHub

However this time I’d like to suggest the less ambitious goal of only supporting types that extend java.lang.AutoCloseable. This also solves the problem shown in the GH issue. Well, to be accurate, it ‘solves’ it by catching it at compile time:

      resource <- AutoClose(new Res())
                           ^
<pastie>:32: error: inferred type arguments [(Res, Long)] do not conform to method map's type parameter bounds [R2 <: AutoCloseable]

      resource <- AutoClose(new Res())
               ^
<pastie>:32: error: type mismatch;
 found   : Res => (Res, Long)
 required: Res => R2

Finally, after reading the thread it seems the question was asked, would the functionality suggested here be ‘correct’? And what is ‘correct’ in this context?

I think Scala’s standard library should be rather unopinionated (as much as possible), but specially supporting java.lang.AutoCloseable as resources which can be auto-closed is a very reasonable step to take. There is already some support for it with Using but it’s not obvious. For advanced use-cases there are already third-party libraries like Cats Effect and fs2 which offer quite sophisticated effect and resource management. But for the standard library, offering for-comprehension support with AutoClose would be a quick win and showcase Scala’s power and simplicity compared to Java.

Full implementation and details of the example above:

class AutoClose[R <: AutoCloseable] private (val acquire: () => R) extends AnyVal:
  import AutoClose.apply

  def foreach[U](f: R => U): U =
    val resource = acquire()
    try f(resource) finally Option(resource).foreach(_.close())

  def flatMap[R2 <: AutoCloseable](f: R => AutoClose[R2]): AutoClose[R2] =
    foreach(f)

  def map[R2 <: AutoCloseable](f: R => R2): AutoClose[R2] = apply(foreach(f))

object AutoClose:
  def apply[R <: AutoCloseable](resource: => R): AutoClose[R] =
    new AutoClose(() => resource)

class Res extends AutoCloseable:
  private var open = true

  override def close() = open = false
  def use() = if (!open) throw new Exception("Already closed!")

object App:
  def main =
    val startTime = System.nanoTime

    for
      resource <- AutoClose(new Res)
      endTime = System.nanoTime
    do
      println(s"initialization took: ${endTime-startTime} ns")
      resource.use()

Working version of the above (no compile error):

object App:
  def main =
    val startTime = System.nanoTime

    for
      resource <- AutoClose(new Res)
    do
      val endTime = System.nanoTime
      println(s"initialization took: ${endTime-startTime} ns")
      resource.use()
1 Like

It’s a bit odd to complain about an opt-in feature that it doesn’t work when you don’t opt in.

However, if in practice the use statement does get forgotten–I guess these things can happen–then use Using.resource instead. Then the difference between a managed resource and an unmanaged one is bigger, so you don’t have to worry about forgetting the use keyword:

Using.resource(new FileReader(path)){ fr =>
  Using.resource(new BufferedReader(fr)){ br =>
    ...
  }
}

Isn’t this an adequate workaround to the forgotten-use problem? It’s a little clunkier than for statements, but it’s not that bad.

isn’t the whole point of having for-comprehension syntax so we can avoid the clunky callback style? Otherwise we could say the same thing about flatmapping over lists, futures, etc., no?

It’s true, but we already have a solution that avoids the clunky callback style. Given that we already have that, and we already have a clunkier style that makes it much harder to forget to do the wrapping (callback style is much harder to forget even than using the for-style…you have a pretty optimistic case where you don’t need to do a bunch of work between grabbing the first resource and the second), I am less sure of the need for a standard for-compatible solution. Your argument seems to boil down to “I can remember to use <- but I can’t remember to use use”, which…is plausible, actually, given the amount of training we get to use <-, but isn’t as clear-cut as a case without a brief alternative.

As a third-party library, sure, great!