PRE-SIP: limiting value discards:

History

Date Version
Sep 27 2019 Pre-SIP
Oct 15 2019 added description of Try.discard and Scalafix rule
Oct 22, 2019, added variant B (same motivation, but more simple solution) [still in progress]
Oct 27, 2019 added handling of builder case for B variant;

Summary:

Limit implicit value discarding.

Variant A: Allows to configure value discarding by allowing to provide appropriative implicit conversions for the given value types.

Variant B: Disable discarding values by default and using typing ascription to explicit discarding. [TODO: think about moving to separate thread]. [TODO: reference to pull request, track authoring section]

Overview:

In current scala version exists two cases, when the value is discarded:

  • When this is a value of the non-final expression in the block

  • When this is a value that should be cast to Unit.
    In such cases, the evaluated value is discarded without error.

    We propose to limit the value discarding:

  • variant A). Allow discarding only those values, which implements a conversion to Unit (or a special marked class DiscardedUnit). If some type not supports discarded value conversion, then dropping values of this type is prohibited.

  • variant B), disallow value discarding unless explicit unit type ascription is provided if this value
    – is not an instance of ‘this.type’ (to support builder pattern)
    – TODO: java interoperability: StringBuilder.append("sss") returns ```StringBuilder``
    Ie in any case, we need some mechanism, for specifying such types.

Motivation:

Example 1. Limit the number of potentials defects when effects, encoded in values are not handled properly, as in the next example:

Let’s look on the following code block:

def doSomething(input:X): Try[Y]  = ....

It is a common error to forgot to handle Try failure and write something like:

{
   doSomething(“A”)
   doSomething(“B”)
   “C”
}

Instead correct, but verbose

for {  _ <- doSomething(“A”)
       _  <- doSomething(“B”)
} yield  C

With limited value discard:

  • Variant A: Try will define an implicit discarded conversion which will check exception, simple sequencing of sentences in the block will work as expected. So, it will become possible to use API written in effects wrapping style from programs written in direct effect style.
    The desugared block will look like:
{
    implicitly[Try[String] => Unit].apply(doSomething(“A”))
    implicitly[Try[String]=> Unit].apply(doSomething(“B”))
   “C”
}
  • Variant B: This code will cause a compile-time error. Explicit unit type ascription will be needed for forcing value discarding. I.e., if we want to discard values of doSomething(A) and doSomething(B), we will need to write:
{
   doSomething(“A”):Unit
   doSomething(“B”):Unit
   “C”
}

Example 2. Prevent error-prone situations during the integration of async monadic constructs into control flow concurrency libraries.
Ie. Imagine, that we work with some framework, which allows developers to avoid the accidental complexity of async operations. We can do this in many ways. (for example, via macroses or project Loom[ https://wiki.openjdk.java.net/display/loom/Main ]),

I.e., if we have an API, which looks like:

def doSomething(input:X): Future[Unit] 

And some concurrency library, which maps monadic expressions to control flow, when we can write something like:

 (given ctx: ContinuationContext) => 
{
   doSomething(“A”)
  If (someCondition) {
       doSomething(“B”)
  }
}

Instead of verbose

val spawnA  =  doSomething(“A”)
var spawnB = if (someCondition) doSomethong(“B”) else Future success ()
for {  _ <-  spawnA
       _  <- spawnB
} yiled  ()

Then, with current value discarding, is impossible to lift effects from doSomething("A") into control flow, without transforming full AST of enclosing block. We have no point to insert the adapter, because the value will be discarded. So, value discarding makes the task of building such a framework quite complicated.

With limited value discarding, we will be able to handle this situation locally:

*Variant A:
Defining implicit conversion:

implicit def discardFuture[T](f: Future[T]):Unit  given (ctx: ContinuationContext)  = {
   ctx.adopt(f)
}

and then use doSomething("A") in control flow.

  • Variant B
    Direct using ‘doSomething(“A”)’ in control flow will be impossible. The programmer will write something like:
  await doSomething(“A”)
  If (someCondition) {
       await doSomething(“B”)
  }

And it will be impossible to forget put await before ``doSomething``` because the compiler will produce an error about value discard.

Needed changes:

  • variant A: Allow value discarding only in case when exists an implicit conversion from type of discarded value to the unit (variant – special DiscardedUnit marker) in the given context.

The value in position when discarded (i.e., with a non-final location in a block which is not a call of the super or primary constructor) should be evaluated as follows:

  • If exists conversion to the discarded unit – apply one before discarding.
  • If not – generate a compile-time error.

Add to the standard library:

  • empty default convertors for primitive values;
  • effect convertors for Try;
  • add to Try discard() method, which does nothing and return void.
  • add to scalafix rule which will save the current semantics of discarded values for effects

*variant B: Disallow value discarding without type ascription.

  • when the value in position should be discarded and this is not a type ascription: generate an error.
  • add to scalafix rule, which transforms discarding values into explicit unit ascription.

Example 3:

Implementation:

Variant A: The proof of concept implementation (not excessive tested and should be viewed only as a starting point) is available at https://github.com/rssh/dotty/tree/no-discard (diff: https://github.com/rssh/dotty/pull/1/commits ) as a patch to the dotty master branch.
The feature is enabled by ‘-Yno-discard-values’ flag. (Scalafix part yet not implemented).

1 Like

How does your idea relate to: Allow suppressing value-discard warning via type ascription to Unit? In Scala 2.13 there are already options like -Ywarn-value-discard -Xfatal-warnings and a way to silence particular value discarding warning by explicit ascription to Unit, e.g. (someExpression: Unit).

– It allows a user to disable the warning about value discarding, but not change semantics. (btw, it is also possible to write : val _ = someExpression in current syntax )

This pre-sip about other: we allow a user to configure a translation from the value-based effect to direct effect, and, if such translation is not available for a given type, generate a compile-time error. I.e. if we have an someExpression, which returns Try[Unit], then using this expression from direct style code when discarding value, will throw an appropriate exception; someExpression:Unit will silently ignore one.

// and in dotty we have a check: ‘pure expression does nothing’.

Totally disallowing value discarding without special syntax, partially solve a problem of ‘forgotten effects’ (i.e. if a developer wrote expr:Unit then he/she probably know something). But problems still leave:

  • we can’t automatically translate effects (i.e. ‘exceptions’ and ‘suspending’)
  • we can still bypass throwing away effects via special support syntax because a person can mechanically copy code, etc… Ie. this is delegating part of work, which compiler can do, to a human.

// Historical note: this pre-sip was written, when playing with the adoption of project Loom and discovered, that value discarding make difficult to wrap continuations with structured concurrency into non-verbose Scala API: we need some point, where we can add computation to concurrency scope implicitly, when a value just discarded we have no way to do this. This can be fixed or by providing implicit conversion to Unit, as in this proposal, or by totally disallowing discarding values. //

Looks like unnecessary complexity. Compiler can only check so much for programmers.

This essentially is un-monading without users awareness. Think of the situation, doSomethingA is allowed to fail, and doSomethingB should be done after A but does not depend on A. This way it will cause unexpected results, and the user won’t even know why.

1 Like

What if I purposefully ignore (some of) the side-effects? E.g. I write:

Try(some side-effecting code).foreach(println) // this is an expression of type Unit
// notice how I conveniently silenced NonFatal exceptions here

Does position of that println change anything? I could have written something like this:

Try(println(some side-effecting code))
// notice how I discarded value of type Try[Unit]

and it would be semantically equivalent to previous example.

Output of

Try(some side-effecting code).foreach(println)

and

Try(println(some side-effecting codу))

is different now (catch/print vs print/catch), and the difference will be the same (i.e. absence of printing in case of failure for 2)

Try(println(some side-effect code)) 

will produce the same output but will throw exception.

What was missed in original pre-sip – that we should add a rule to scalafix, to transform such expressions into

Try(println(some-side-effect-code)).discard

and add .discard to Try. (or use :Unit syntax).
to have exactly the same meaning as in scala 2.

I prefer to see in this case explicit allowance of doSomethingA in the code.

Note, than in this case user know why and what failed: exception is generated.

In current scala we have a reverse situation: silently ignoring exception inside Try of Future, is an actual common pitfall for scala novices. (look at https://stackoverflow.com/questions/24453080/scala-futures-default-error-handler-for-every-new-created-or-mapped-exception – something like LoggedFuture is a common workarround)

No, the output for normal Scala is the same. If side-effecting code throws exception then println part will not be invoked (because computation of value to print failed). Try(println(throw new Exception)) silences exception and outputs nothing (i.e. doesn’t output anything) just like Try(throw new Exception).foreach(println).

1 Like

Ahh, foreach is biased… yes, you are right. (in 1-st case foreach throws out value, in 2-nd: try value is discarded, so behaviour is semantically different, when output is the same)

Could you elaborate on that? I know very little about continuations in context of Project Loom. How would Java programmers solve the problems you want to solve? Java doesn’t have any form of user provided implicit conversions.

That’s the opposite how Scala programmers write code imo. Wrapping an exception in a monad instead of throwing a runtime exception. Plus the behavior in my post is a compile time problem and shouldn’t be discovered only when it fails at runtime.

// During last few months I want to write a separate blog-post about ‘ideal concurrency fir scala’ but fail to allocate time. Maybe next weekend.

Java Loom provide low-level api, which is quite verbose on ‘library side’ but perfect on ‘usage side’. It is something like CPS [continuation passing style, remember shift/reset] plugin on JVM level: in any time you can suspend execution of program and save current stack into a continuations. In short, to main entities in Loom are java.lang.Continuation and java.lang.Fiber. Continuation keep stack and provide two main methods run and yield. Run - running associated runnable when called first, and resume execution when called next; yield - suspend current execution, which will be resumed during next call of run.

Fiber - is a more high-level construct, which is a Continuation + Scheduler. The main new constructs – ability to return self as future and ‘yield’ for interruption: Ie. run other fiber and when one is finished – continue themself. So, next code in fiber:

a = localOperation1()
x = doRemoteCall(a)
localOperation2()

Looks like blocking invocation of doRemoteCall(), but in fact it parking fiber until remote call will return. So, inside it’s more like asynchronous code.
Also they patches standard API, to make synchronous blocking call be asynchronous when calling from fibers.

This effectively make reactive programming as something special, obsolete. For example, Oracle dropped async database access API. (https://mail.openjdk.java.net/pipermail/jdbc-spec-discuss/2019-September/000529.html )

(Note, that description is simplified, yet one theme [structured concurrency] is not described at all, but I guess main ideas are shown).

Also note, that ‘by default’ ‘await’ is implicit, i.e. not written, and ‘spawn’ is explicit. (Kotlin coroutines follows the same convention).

Now, imagine we want to make a Scala API. It’s obvious to want interpolation and it’s obvious to follow the same conventions (i.e. represent async operation in fiber as virtually blocking call without additional operators). For this we need a place to insert our fiber await. So, if we want to have some type, which can be ‘by default’ for blocking async operation, we want or something like conversion for discarded value (as proposed here), or we will need to transform each ast node in block inside some bounds, which is complicated [scala is relative big language].

This is not very exact, maybe I have thinking too out of square…

  • Sorry, I can’t get what you refer to.

OK. Thanks for some rough explanations. I still don’t have any idea why would anyone want to unpack all Throwables kept in scala.util.Failure when using Fibers. It seems very arbitrary. You can represent failures using other constructs, e.g. Either (i.e. Left) or Option (i.e. None) - would you want to unpack them and throw exceptions?

I guess only Try have general direct meaning of failure and can be converted.
For Either and Option better not to provide any conversion, so discarding such value will be compile error.

For instances of MonadError and other effects (Future systems, where conversion can be await, or runUnsafe … etc …) conversion can be provided on library level by library authors.

Because I want to insert Futures (maybe special FiberFuture, but let think Future) into direct control flow.
i.e.

localOperation1()
doRemoteCall(a)
localOperation2()

instead

localOperation1()
doRemoteCall().flatMap(_ => localOperation2())

(Sorry, it was not answer, but explanation, how Fiber relates)

With Try situation is much simpler:
Exists common novice error: Discard value of Try without error handling. Disabling automatic discarding (and provide a conversion to effect when this conversion is clean and well-defined) prevent this novice error. That’s all.

What if the _ <- and () weren’t necessary?

I’m strongly in favour of the motivation behind this SIP (I authored the initial version of that "Allow suppressing value-discard warning via type ascription to Unit" PR) but I wonder: how much of this proposal would be necessary if instead for notation supported:

for {
  doSomething("A")
  doSomething("B")
} yield "C"

// and

for {
  spawnA
  spawnB
}

Can better monadic comprehension use instead? We have two sides here:

  1. Disallow discarding values until the effect of discarding (or absence of one) nod defined. (Ie. make it harder to shoot themself).
    This part is orthogonal to better monadic comprehension. (I.e. PRE-SIP is steel needed)
  2. Allow a style of programming when monad composition used instead of control flow. Better monads can help here. From the other side - this is part of the long-long discussion when people testes are different.
    I think that wrapping all in yet one layer is add a small cognitive load for the developer when writing each operation (as brackets). And if when we can work without an extra layer, better not use one.
    Here ( http://blog.paralleluniverse.co/2015/08/07/scoped-continuations/ ) is a detailed description of the CPS/Monad dilemma. In the functional land, effect algebras can achieve the same: we can reinterpret control flow syntax in monads and vice-versa. So, better monads ( ‘ideal monadic syntax’) will be the same as control flow. Which brings to the table yet one question - why we need to have two languages (monadic and control-flow)? With two different camps behind, of course.

Since we (SIP people) were specifically called out about this topic, I’ll give my opinion.

TBH I don’t think the proposal is a good fit for Scala. Currently Scala has statements and expressions, and they are what they are. It also has for comprehensions that specially call out as being rewritten into maps and flatMaps. Having a feature where normal statements are rewritten into method chains introduces another, more hidden way of doing things that can already be done, safely, with for comprehensions.

The concerns about some values being discarded unsafely can already be addressed with -Ywarn-value-discard, as was mentioned earlier.

5 Likes

– problem, that it is possible to lost effect, writing discarded value without for comprehension. I.e. the main point is a disallow errors,

How you think about just forbidding discarding values, and propose some syntax for explicit discard, instead? (i.e. make something like -Yerror-value-discard enabled by default)