Cannot overload flatMap

Scala Platform
1

this is a copy from https://github.com/scala/bug/issues/11232 where it was suggested to raise the issue here. I choose Scala Platform due to lack of better fitting categories. This issue describes unexpected behaviour, i.e. I regard it as a bug, however scala/bug wants more precise bug descriptions. Maybe adding a “pre-bug” category to scala contributors?

Dear Scala developers,

I just stumbled upon the standard case that Option and Iterable might be hard to intertwine in a scala for loop. See for instance this stackoverflow question

I tried to fix this by just overloading flatMap with a version which should work, however it doesn’t.

When I run the following in scala 2.11.12 REPL

implicit class OptionFlatMapIterable[A](o: Option[A]){
  def flatMap[B](f: A => Iterable[B]): Iterable[B] = if (o.isEmpty) Seq.empty else f(o.get)

  def flatMap2[B](f: A => Option[B]): Option[B] = o.flatMap(f)
  def flatMap2[B](f: A => Iterable[B]): Iterable[B] = if (o.isEmpty) Seq.empty else f(o.get)
}

Some(42).flatMap2{ (o: Int) =>
  Seq(o, o + 1, o + 3).map{ (i: Int) =>
    (i,i)
  }
}

Some(42).flatMap{ (o: Int) =>
  Seq(o, o + 1, o + 3).map{ (i: Int) =>
    (i,i)
  }
}

for{
  o <- Some(42)
  i <- Seq(o, o + 1, o + 3)
} yield (o, i)

I get the following outputs

res0: Iterable[(Int, Int)] = List((42,42), (43,43), (45,45))

<console>:14: error: type mismatch;
 found   : Seq[(Int, Int)]
 required: Option[?]
         Seq(o, o + 1, o + 3).map{ (i: Int) =>
                                 ^
<console>:15: error: type mismatch;
 found   : Seq[(Int, Int)]
 required: Option[?]
         i <- Seq(o, o + 1, o + 3)
           ^

As you can see it perfectly works when using my dummy flatMap2, but fails for flatMap itself.

If this would work, it could massively simplify the support of complex flatMap chainings and hence complex for loops with different Monads intermixing a base Iterable Monad.

please, anyone who can help?

2

Your use case isn’t valid. A map procedure should return the same type of collection (in this case, Option.map returns Option, not Iterable).

Your code should work this way (since you want an Iterable, just convert Option into Iterable before map):

Scala> Some(42).toList.flatMap{ (o: Int) =>
  Seq(o, o + 1, o + 3).map{ (i: Int) =>
    (i,i)
  }
}

res1: List[(Int, Int)] = List((42,42), (43,43), (45,45))
3

Also, you say you overloaded flatMap, but unfortunately it doesn’t work that way. You cannot really retroactively overload a method through an implicit conversion. This works as designed as far as I know. So that would be a feature request which would probably require an SIP.

Apparently I’m getting old and my memory isn’t what it used to be.

1 Like
4

I think the right name for this operation is traverse?

5

relevant: https://twitter.com/headinthebox/status/1046521593524965376, https://github.com/scala/bug/issues/3674 . the example in 3674 works, but it’s monomorphic. Erik’s example and schlichtanders’s example fail because the original method is polymorphic. Reduced:

scala 2.12.7> class E; class C { def foo[T <: E](x: T) = "C#foo" }
defined class E
defined class C

scala 2.12.7> implicit class D (val c: C) { def foo(x: AnyRef) = "D#foo" }
defined class D

scala 2.12.7> (new C).foo(new E)
res0: String = C#foo

scala 2.12.7> (new C).foo(new AnyRef)
<console>:14: error: inferred type arguments [Object] do not conform to method foo's type parameter bounds [T <: E]
              (new C).foo(new AnyRef)
                      ^

it doesn’t work in Dotty either.

1 Like
7

@texasbruce nice one. While this works for Option, because it is just the same as a List with either 0 or 1 element, this does not work for Monads with other SideEffects which are not isomorphic to List/Iterable

Of course I disagree that map procedures should return the same type of collections ;-). That may make sense for most usecases, but as I showed you there is a usecase for having a different return type.

8

@Jasper-M @SethTisue thanks for pointing out that this is in general not possible right now in Scala.

Very surprising indeed. Does anyone know whether there is already a ticket/issue which tracks this generic feature request?

9

I have looked in scala/bug and not found one, but it’s a hard thing to search for, I can’t be certain there isn’t an existing report. Perhaps it will ring a bell for someone reading this thread, I also asked in scala/contributors Gitter if it rings a bell for anyone.

10

Just tried a workaround by manually casting to something which has overloaded flatMap, which indeed works

class OptionFlatMapIterable[A](o: Option[A]){ self =>
  def map[B](f: A => B): Option[B] = o.map(f)
  def foreach[B](f: A => B): Unit = o.foreach(f)
  def flatMap[B](f: A => Option[B]): Option[B] = o.flatMap(f)
  def flatMap[B](f: A => Iterable[B]): Iterable[B] = if (o.isEmpty) Seq.empty else f(o.get)

  def filter(p: A => Boolean): Option[A] = o.filter(p)

  /** Necessary to keep $option from being implicitly converted to
    *  [[scala.collection.Iterable]] in `for` comprehensions.
    */
  def withFilter(p: A => Boolean): WithFilter = new WithFilter(p)

  /** We need a whole WithFilter class to honor the "doesn't create a new
    *  collection" contract even though it seems unlikely to matter much in a
    *  collection with max size 1.
    */
  class WithFilter(p: A => Boolean) {
    def map[B](f: A => B): Option[B] = self filter p map f
    def flatMap[B](f: A => Option[B]): Option[B] = self filter p flatMap f
    def flatMap[B](f: A => Iterable[B]): Iterable[B] = if (o.isEmpty) Seq.empty else f(o.get)
    def foreach[U](f: A => U): Unit = self filter p foreach f
    def withFilter(q: A => Boolean): WithFilter = new WithFilter(x => p(x) && q(x))
  }
}


new OptionFlatMapIterable(Some(42)).flatMap{ o: Int =>
  Seq(o, o + 1, o + 3).map{ i =>
    (i,i)
  }
}
// res0: Iterable[(Int, Int)] = List((42,42), (43,43), (45,45))

for{
  o: Int <- new OptionFlatMapIterable(Some(42))
  i <- Seq(o, o + 1, o + 3)
} yield (o, i)
// res1: Iterable[(Int, Int)] = List((42,42), (42,43), (42,45))

however the type inference is not working satisfactorily… the o: Int type annotation is indeed necessary for this to compile…

11

for those interested in the compiler/spec angle here, https://github.com/scala/scala/pull/7396 (which Adriaan and Stefan worked so hard on today, veins bulged alarmingly on their foreheads and the temperature in the room rose five degrees)

3 Likes