Spark and Scala 2.12

In short, there are two pending issues:

  • Java 8 lambdas in Scala 2.12 breaks the Dataset API: more details
    • they need to remove a method = requires major number bump (ex: 2.2 to 2.3)
    • this is from March 2016
    • this is solved / trivial
  • Closure cleaning for Scala 2.12: 1% of tests still fails (2 tests) more details
    • PR fixing 99% in November 2017
    • remaining 1% is a hard problem, they might cut a release anyway

My guess is that it might be available in 2.3. The version 2.2 was released the 1st of December 2017. I would say this will happen in 2018 :smiley:.

4 Likes