Scala Center project updates: mid-Oct to mid-Nov

Scala Center Updates
1

Dear Contributors,

The newest updates on Scala Center projects are here!

Enjoy reading (o:
Darja

At a glance

Scalameta

Ólafur Páll Geirsson @olafurpg

Released Scalameta v2.1.0, see changelog.
Highlights:

  • #1143 Fix incompatibility with the semanticdb-scalac compiler plugin
    and scalamacros/paradise macro annotation.
  • #1144 Upgrade semanticdb-scalac to 2.12.4. Due to large internal refactorings
    in scalac typer, this upgrade turned out to be unusually complicated.
  • #1145 Fix bug in handling of by-name and repeated parameters.
  • #1148 Add Denotation.members. This opens the possibility to reproduce
    lexical scoping at a position enabling rules like 1) ExplicitResultTypes
    to safely insert unqualified named with accompanying imports or 2) an
    OrganizeImports rule to safely.
  • #1154 Fix bug where Foo in new Foo {} resolved to an anonymous class
    instead of the Foo symbol.
  • #1158 Amongst many things, significantly reduces memory pressure while compiling
    sources with the semanticdb-scalac compiler plugin, which is required by Scalafix.

Scalafix

Ólafur Páll Geirsson @olafurpg and Guillaume Massé @MasseGuillaume

Guillaume joins Ă“lafur on the Scalafix project. He will work on a wide range of topics, including adding new rules and polish the general user experience, which is quite rough at the moment.

Released Scalafix v0.5.4, see changelog.

Highlights:

  • #412 Make it easy to enable sbt-scalafix for custom configurations like
    IntegrationTest. (@olafurpg)
  • #415 Suppress false linter errors with comments. This PR makes
    it practical to use Scalafix for linting, since previously there was no way
    to silence Scalafix for exceptional cases where the linter reports a bogus
    diagnostic. (@olafurpg, @MasseGuillaume)
  • #418 Added detailed documentation comparing scalafix with alternative tools
    like Scala Refactoring, WartRemover, ScalaStyle and IntelliJ Scala plugin. (@olafurpg
  • #419 Simplify installation of sbt-scalafix. Based on feedback from users,
    this PR adds a scalafixEnable command that can automatically setup all the
    correct scala compiler settings from an sbt shell session, regardless of
    existing settings in build.sbt. Previously, it was necessary to manually
    update build.sbt with fairly tricky corner cases. (@olafurpg)
  • #420 Reviewed PR adding documentation for how to use scalafix for Maven builds. (@olafurpg)
  • #431 Implement a Syntaxtic rule to disable Scala keyword, xml, tabs, carriage return and semicolons (@MasseGuillaume)

Macros

Ólafur Páll Geirsson @olafurpg

  • A lot of failed experiments trying to automatically expanded trees from
    scala-reflect macros that that mix untyped and typed trees.
  • Studying @odersky’s ideas about a new design for macros that unifies LMS
    and macros, see discussion
  • Orienting Ryan Culpepper, the author of Racket macros, on the design challenges in this new macro rework. Ryan has expressed interest in contributing to this project.
  • Most recently, reading up on Squid, a
    metaprogramming framework based on type-safe and hygienic quasiquotes,
    developed at the DATA lab at EPFL.
    Squid seems to address a lot of the main issues in scala-reflect.
    Squid also builds on top of scala-reflect, which means it could maybe provide
    a smooth migration story for the existing ecosystem of macros.
    I only recently found out about Squid, and I am very excited to look into
    it deeper. Tomorrow, I am meeting the author of Squid to get a hands-on
    tour.

Compiler Profiling

Jorge Vicente Cantero @jvican

scalac-profiling has been significantly improved to be battle-tested in big,
well-known open source libraries of the Scala community. This work started as
part of my writing of a technical guide to let people in the Scala community
use the plugin to profile their projects. This technical guide will be
published towards the end of November.

Improvements to the project:

  • Add flamegraph generation for implicit searches (!).
  • Add dot graph generation for implicit searches.
  • Add more measurements of the compiler to get more data for
    users    .
  • Profile the cost of macros and implicit search in all the integration builds.
  • Add shapeless, scalatest, better-files and scalac as new integration
    builds     of the
    scalac-profiling build (complementing our previous integration builds
    circe and monocle). Every merge in the project is tested against these
    projects in the CI to ensure there’s bad interaction.

For a sneak preview, head to this page that explains
the current status of the visualizations of implicit search. If you’re too
busy, check the Flamegraph and dot graph for the example in
the Circe website.

The plan for the technical guide is the following:

  • Explain the overall idea of scalac-profiling and why it allows people to
    fix their compile times.
  • Write the guide by showing how to profile compile times for several use
    cases. Every use case will be for some of the integration builds that we
    have: shapeless, scalatest, and circe.

Zinc

Jorge Vicente Cantero @jvican

(Read the bolg article)

My work on Zinc has focused on several areas:

  1. Improving classpath hashing for incremental
    compilations    . The classpath hashing
    was added before the Scala Center got involved in the project and created
    a severe regression in incremental compiles for users of Zinc 1.0 –
    incremental compiles were much slower because it had to hash all the
    classpath entries. This regression was measured to be in the range of 15 to
    40 seconds in medium to big projects.
  2. Fix #127: Use unexpanded names.
  3. Reviewed around 8 PRs on our queue; some have been merged, some are still
    under work.
  4. Fix #436: Remove annoying log4j scripted exception.
  5. Work on a better build and test workflow for Zinc. The scripted integration
    was slow, and that put off contributions and made changes to Zinc slow to
    happen. The rework of the build is splited into several PRs (#438,
    #428, #429 and #440) that add:
    1. A Zinc scripted mechanism that is reproducible (isolated from users’
      cache) and fast (parallel execution of tests). Note that Zinc uses an
      independent scripted than sbt.
    2. On-merge releases (so that users of Zinc can depend on versions of pull
      requests that have been merged right away instead of being blocked on
      maintainers to release them).
    3. A publishLocal implementation that is cached and avoids publishing when
      the same version is already in the cache.
    4. Better UI for both the build and the test execution (colors,
      highlighting, log collection in an independent tmp folder, etc).
    5. Documentation for all the steps of the build and removal of dead code.

Scala Platform

Jorge Vicente Cantero @jvican

  1. Installed the new Drone CLI version (0.8) in
    platform.scala-lang.org for the benefit of the Scala Platform maintainers
    and the Scala Center engineers.
  2. Coordinated two pull requests for Scala JSON and Better files that
    use the new Scala Platform infrastructure.
  3. Made several significant improvements to
    the Scala Platform plugin, which now fully integrates with
    the sbt-release-early and has a detailed test infrastructure.

Scala Improvement Process

Jorge Vicente Cantero @jvican

  1. Coordination with the Committee on several topics.
  2. Organisation of a voting in the next SIP meeting to vote for the admission
    of a Typelevel representative: Miles Sabin.

SBT 1.x Plugin Migration

As part of SCP-013, we migrated two of the most starred open source sbt plugins that had not been
ported to sbt 1.0 yet:

Guillaume Massé @MasseGuillaume and Martin Duhem @Duhemm

Based on the sbt 1.x migration wiki. We
migrated non-commercial plugins that have an impact on the community:

  • scala-native is a new target for the Scala programming
    language. It uses llvm to link an run Scala code to any platform. It has a seamless integration with sbt. You
    can run and test just like you do it on the JVM. PR #1039 (@MasseGuillaume)

  • Dotty is a research platform for new language concepts and compiler technologies for Scala. It’s tightly
    coupled with sbt and Zinc and the migration to 1.X is still in progress. There is more detail on the Dotty
    section (@Duhemm)

  • sbt-dependency-graph is a tool to understand how your
    external dependencies are wired togetter. It’s essential to debug anomalies on the classpath. PR #140 PR #139 (@MasseGuillaume)

  • g8 in an sbt plugin that powers the sbt new command. It allows
    the creation of template to quickly create new projects. PR #350 (@MasseGuillaume)

  • sbt-aws-lambda is an sbt plugin that helps developer in
    deploying code to AWS lambda. PR #39 (@Duhemm)

  • sbt-spark-submit is an sbt plugin that makes it
    easy to submit jobs to spark, using spark-submit under the hood.
    PR #9. (@Duhemm)

Thanks to the authors of these plugins for the quick reviews!

As far as we can tell, most of the open source, non-commercially owned sbt plugin now support sbt
1.0. Please let us know if we missed something important!

Dotty

Martin Duhem @Duhemm

Specializing functions

The work I started last month (from
an initial PR by Felix Mulder) to specialize
functions in the dotc compiler is working. Our tests show that the code that is produced by dotty
allocates fewer boxes to hold primitive values. However, it turns out that it has a negative impact
on the performance of the compiler and increases compilation
time.

I am working on reducing the overhead of this optimization so that compilation times are not
affected.

The work is happening on this pull request.

Embedding documentation in TASTY

I started working on embedding documentation in TASTY, so that it is always available with the
classfiles of a compiled project. The goal is to be able to easily query the documentation from the
REPL, or an IDE:

scala> object Foo { /** hello! */ def bar = 123 }
scala> object Random {
     |   /** Returns a random number
     |    *  @return A number chosen by fair dice roll,
     |    *          guaranteed to be random.
     |    */
     |   def getRandomNumber(): Int = 4
     | }
defined object Random

scala> :? Random.getRandomNumber
/** Returns a random number
 *  @return A number chosen by fair dice roll,
 *          guaranteed to be random.
 */
def getRandomNumber(): Int

At the moment, my work in progress can store the documentation strings in a new section of
the TASTY information and the REPL starts to support queries but needs more work.

Improving the Dotty Community Build

The Dotty Community Build is a corpus of open
source Scala projects that we use to detect regression in Dotty. At the time of writing, eight projects
are part of the Dotty Community Build.

Every night, we use the latest nightly release of Dotty to compile those eight projects, and can
therefore quickly detect any regression. However, we didn’t have a simple way to test the Dotty
community build against a pull request, a tag or a specific commit.

I took some time to improve on that and added a new script that allows us, given a reference inside
the Dotty repository, to locally build, publish and test Dotty
against the community build. This way, we can merge with confidence pull requests that may cause a
regression that would be hard to catch with smaller tests.

Here is the pull request.

We do not have integration with our bot at the moment, be I hope to add that soon.

Porting Dotty to sbt 1

The dotty sbt plugin can currently only be used with sbt 0.13. However, now that sbt 1.0 is out, it
makes sense to upgrade to the latest versions of sbt. Future releases of Dotty will drop support for
sbt 0.13.

I started upgrading our build, plugin and compiler
interface to sbt 1.0 and the latest release of
Zinc. Upgrading to the latest version of sbt lets us benefit from the recent improvements that were
made in sbt, and most notably it allows us to enjoy faster incremental builds, thanks to the new
class based name hashing featured in Zinc 1.0.

Because of the many changes that happened in Zinc, especially regarding how a Scala compiler must
communicate API and dependency analyses with the incremental compiler, upgrading our compiler
interface was a large task. We also discovered several bugs in Zinc in that process.

After I got our build to work and all our tests to pass, I needed to upgrade all the projects that
appear in our community build to use sbt 1.0. The changes that were made to upgrade the build
definitions to sbt 1.0 have been proposed for inclusion the upstream projects, and I am waiting to
hear back from them.

The work is happening on this pull request.

Fixing a bug in generic java signatures

My first task with Dotty was to add support for generic java
signatures. See my description in our previous Scala
Center
update
to know what this work is all about.

We discovered a bug with the generic signatures,
where additional parameter lists may be missing.

I fixed this issue in a subsequent pull request.

MOOCs

Julien Richard-Foy @julienrf

Added XQueue support to our grading infrastructure.

Scala.js Bundler

Julien Richard-Foy @julienrf

  • Made it possible to release scalajs-bundler by pushing a Git tag #184, so that any contributor that has write access to the
    scalajs-bundler GitHub repository can cut a release (they don’t need to deal with Sonatype
    access rights)
  • Released scalajs-bundler 0.9.0

Collections

Julien Richard-Foy @julienrf

  • Fixed undefined references in API documentation #274
  • Ported scalacheck tests of the current collections to the strawman #276
  • Reviewed @marcelocenerine’s work on lazyZip. The goal of this operation is to provide
    a more discoverable alternative to (xs, ys).zipped. The new syntax is xs lazyZip ys.
    This operation lazily zips collections together (the zipped collection is not evaluated
    unless it is iterated on) and provides transformation operations (such as map and
    filter) that don’t create intermediate tuples. #223
  • Tried an alternative approach to implement MultiMap: create its branch in the
    hierarchy. #269
  • Fixed performance regressions #270,
    #271.
  • Added an updateWith/updatedWith operation to Map collection types (in collections-contrib)
    #268.

Zinc standalone

Martin Duhem and Jorge Vicente Cantero are
collaborating together in a Zinc standalone tool that allows users to consume the incremental compiler via the CLI

Presentations and Workshops

Scala.io

Julien Richard-Foy @julienrf

Lambda World

Martin Duhem @Duhemm and Jorge Vicente Cantero @jvican
Organised the Spree at Lambda World in which
around 20 people attended the event. More than 15 pull request was merged in
well-known open source projects, and we had special help from Daniela Sfregola @DanielaSfregola!

14 Likes
2

Thank you for this summation. I find it extremely useful.

5 Likes
3

Very happy to hear that! Thank you!

4

Yes this is awesome, thank you for your discipline on this.

-Dan

2 Likes
5

Huge, huge thank you, both for the write up and all the awesome work being done!

2 Likes
6

:heart_eyes: will keep it coming, and YES, great job Scala Center!!!