Scala Center project updates 2018: mid-Jan to mid-Feb

Scala Center Updates
1

Dear Scala Community, dear Contributors,

After the winter break in December and a bit in January, we are back to full speed here at Scala Center! Happy to restart the monthly project updates, keep on scrolling (o:

At a glance

  • Scalameta
  • Build Server Protocol
  • MOOCs
  • Collections
  • Bloop
  • Scala Platform
  • Scala Compiler
  • Scalac profiling
  • scalajs-bundler
  • Scalafix

Scalameta

Ólafur Páll Geirsson @olafurpg

Collaborated with Eugene Burmako on specifying SemanticDB and bringing the implementation in line with the spec https://github.com/scalameta/scalameta/blob/master/semanticdb/semanticdb3/semanticdb3.md

  • #1224 Print type ref prefix with type arguments, for scalafix ExplicitResultTypes.
    Thanks to this PR I was able to run ExplicitResultTypes on all signatures in the Akka project and
    it still compiled afterwards.
  • #1216 Print this type for abstract prefix, for scalafix ExplicitResultTypes.
  • #1185 Guard against NoPosition in InputOps, bugfix.
  • #1172 Add InteractiveSemanticdb utility, a simple API to go from String to SemanticDB.
  • Presented Metals at the LSP Tooling meeting

Build Server Protocol

Ólafur Páll Geirsson @olafurpg

Wrote thoughts on how a potential “Build Server Protocol” might look like https://github.com/scalacenter/bsp.
The document is inspired by the Language Server Protocol but addresses the communication between
a language server and build tool instead of editor and language server.
Currently, every language server must implement custom integrations for the most popular Scala Build
tools (sbt, Maven, pants, …) in order to extract compilation information such as classpaths and source directories.
The Build Server Protocol is an attempt to solve that problem.

MOOCs

Julien Richard-Foy @julienrf

I finished porting our four first MOOCs (Functional Programming Principles in Scala,
Functional Program Design, Parallel Programming and Big Data Analysis with Scala and
Spark) to the EdX platform.

I setup a horizontally scalable grading infrastructure that communicates with an EdX
instance. We performed some end-to-end tests and load tests (500 users submit their
work at 1 second of interval).

Last, I compiled the old Akka assignments with the last Akka version (I had almost
nothing to change!), and collected all the material that we used to have for the
reactive programming course.

Collections

(SCP-007)
Julien Richard-Foy @julienrf

I implemented some macro-benchmarks (#344).
Our current micro-benchmarks take a week to run because they test N operations × M types of
collections × O number of elements. The idea of having macro-benchmarks is that they should
take less time to complete while still allowing us to detect performance regressions.
I’ve also implemented a few optimizations (
#340,
#348).

Speaking of performance, I’ve written a blog article
(#832) about the design of the new collections
and their performance. I’ve explained that the transformation operations of the new collections
are, by design, correct by default (which was not the case in the old collections), but should
be overridden for performance in strict collections.

I spent several days improving the design of Views so that:

  • they don’t prematurely evaluate elements #343,
  • they don’t forget too much about the underyling viewed collection #436.

I spent several days reducing incompatibilities with 2.12’s collections. I tried to make
existing projects compile with both 2.12 and a special distribution of 2.13 that has
the new collections (thanks to @szeiger). In particular, I worked on the scala-parser-combinators
module (#134),
and added the following compatibility improvements to the new collections:
#355,
#358,
#364,
#427.

I happily collaborated with Lukas to fix a few issues:
#369,
#370,
#371.
Luckily, @marcelocenerine took over most of the other bugs!
Lukas also created an FAQ page, which I contributed to.

I also reviewed exciting contributions from the community.

scalajs-bundler

Julien Richard-Foy @julienrf

Released v0.10.0,
with nice contributions from the community!

Bloop

Jorge Vicente Cantero @jvican

There has been a lot of progress in bloop since our past blog
post.

My work has focused on the following areas:

  1. Bloop has now a robust community build for both sbt 0.13 and 1.0 projects.
    We plan to add more OSS projects to the future, but to this date we support:
    apache/spark, scala/scala, guardian/frontend, lihaoyi/utest,
    sbt/sbt, and a trimmed down version of pathikrit/better-files.
  2. Bloop has now a hugo-based
    website    . The website will contain the
    docs of how to use bloop in a wide array of scenarios. Still work in progress.
  3. Bloop has now a Maven integration that integrates with
    scala-maven-plugin, reading up on its values and allowing to set some
    bloop-specific configuration options.
  4. Bloop supports now BSP (the Build Server Protocol). BSP is Scala
    Center’s attempt to scale IDE integrations with build tools supporting the
    Scala programming language. The protocol is work in progress and we are
    working together with the tooling team at Lightbend, the Intellij team
    (@jastice) and the Dotty team to curate it.
    It is extensible to other programming languages by design and highly
    inspired by LSP.
  5. Bloop has a better UX than its previous version, with improvements on the
    CLI interface and the nailgun communication.
  6. Bloop has a better internal design: several of the internal APIs and external
    integrations have been significantly improved.

You can see my recent (closed) pull requests
here.

Aside from these tasks, I have been researching on how to create integrations
for build tools like Bazel (worker servers) and Gradle. Bloop will support
these build tools in the close future.

Martin Duhem @Duhemm

I have been working on Bloop together with Jorge since November 2017. In this time frame, we have
taken Bloop from an early prototype to a solid tool that is able to compile very large open source
projects, such as the Scala Compiler or Apache Spark with better performance than sbt.

Here’s an highlight of the tasks I’ve been working on:

Rewrite how we do logging

Our first prototype didn’t really care about logging and would write to the standard output. We
improved it to use Log4J instead.

Set up the initial integration test infrastructure

Testing in Bloop is complicated, because we need to generate the configuration for our tool from an
existing build definition that can be found in an open source project. This means that we need to
run sbt to generate the configuration, and finally use it for our integration tests.

Making it simple to add more projects to our integration tests infrastructure and having it produce
reliable results took some effort, but was important in making sure that we could actually compile
large projects and produce correct results.

Add support for running tests in Bloop

Test support is the second feature that we wanted to support in Bloop. Our implementation works
using the same mechanism as sbt and is therefore compatible with all the test frameworks that work
with sbt.

The integration works out of the box. No extra step is needed when importing the project to have the
tests run.

Set up a benchmarking infrastructure

We developed a benchmarking infrastructure that is a mix from Dotty’s and Scalac’s infrastructures:
We use Dotty’s system to be able to schedule benchmarks easily from Github, and Scalac’s integration
to perform comparative benchmarks of the compilation time of sbt, scalac and bloop on many several
projects, including Apache Spark and the Scala Library for instance.

The results are then displayed as graphs to follow the evolution of the compilation times.

We also produce the same results for the time required to load a project, etc.

Add support for starting a REPL (sbt console)

The third feature that we wanted to support was starting a REPL with a project on the classpath
(this is what is known in sbt as the console and consoleQuick tasks).

Add support for running code in Bloop

The last feature that we wanted to support was the run task. It allows to run a class that has a
main method. Support for that has been merged in Bloop.

Add support for forking

sbt lets us define whether tests and runs should happen in the same JVM, or if a new JVM should be
started to perform the action. We wanted Bloop to also support that, so we added support for forking
(previously, everything would happen in the same JVM).

Set up parts of our release process

We worked on automating some parts of our release process, so that we can automatically generate
installation scripts along with a release.

For Mac OS users, we also push a Homebrew formula that let’s them use the Homebrew package manager
to install Bloop and keep it up to date.

Porting to Windows

We worked on making sure that Bloop works on Windows, and set up a Jenkins CI instance running on
Windows to check that there are no regression. The Windows CI is not yet enabled, it is still a work
in progress.

Documenting Bloop

We set up a documentation website for Bloop and wrote some documentation. It is still a work in
progress.

Talking about Bloop

Jorge and I submitted talks about Bloop, and we’ll be talking about Bloop at:

  • ScalaSphere
  • ScalaDays Berlin
  • ScalaDays New York

Scala Platform

Jorge Vicente Cantero @jvican

Video of our meeting in February.

I have organized the first Scala Platform meeting of the year, together with
Darja Jovanovic. Our plan is to have a first release of the Scala Platform in
the first week of March. In our next meeting, we’d like to synchronize with
the Scala Platform Committee and the Scala Community to shed more light on
which other modules we’re missing in the Scala Platform.

In our meeting, we discussed the following topics:

  1. “Make Scala Platform API independent of Java
    namespaces”    .

    • Guests: Sébastien Doeraene (Scala.js) and Denys Shabalin (Scala Native).
    • Proposition: Scala Platform modules should be all cross-platformed.
    • Discussion revolves around motivation, resourcing, technical feasibility.

  2. Status update of the Scala JSON AST; technical discussion about external
    dependencies.

    • Blocker: what should be the immutable collection interface for users?
    • General discussion about external dependencies; when are Platform modules
      allowed to depend on external modules? Tooling solutions to the problems?

Scala Compiler

Jorge Vicente Cantero @jvican

I have continued @retronym’s investigation on why compiler plugins that are
dynamically classloaded destabilize and slow down warmed up compilers. This
issue was initially discovered in a Scalameta
benchmark.

The result of this work has been the following:

The speedup of this change seems to be in the range between 16% and 35% of
warmed up compilation.

For more context and details, check the
SD-458 ticket and the
aforementioned pull request.

Scalac profiling

Jorge Vicente Cantero @jvican

  1. Make the sbt plugin cross-compile to both sbt 0.13.x and 1.x.
  2. Add Magnolia to the community build of scalac-profiling.
  3. Modify build to use a custom compiler version of Scala 2.12.5 with:

I am still writing the detailed profiling guide on how to use the plugin, with a tour on how to identify compilation bottlenecks and speed up implicit search. This is still work in progress because it is a careful process that requires:

  1. Knowledge about the codebase that is being tested and how things interact
    with each other.
  2. Iterations to assess the impact of changes to a current codebase.
  3. An faithful interpretation of the results.

This work will be available before the next Advisory Board meeting in March,
and will include examples for both Circe and Scalatest.

Scalafix

Guillaume Massé @MasseGuillaume
New Rule: Convert Single Abstract Method

Add Rule configurations: DisableSyntax.regex

New feature: –diff --diff-base

  • scalafix --diff --diff-base v0.5.6 will apply scalafix only on the code
    that has changed since v0.5.6.

New feature: escape rules with comments

  • // scalafix: ok [RuleName] disable RuleName for an expression
  • // scalafix: off [RuleName], // scalafix: on [RuleName] disable RuleName between off and on

New feature: sbt task scalafixCli

  • Expose all cli arguments to the sbt plugin
  • Autocompletion for cli argument
  • Autocompletion on the git history for the --diff-base argument

Ólafur Páll Geirsson @olafurpg

Released Scalafix v0.5.10 in close collaboration with @MasseGuillaume and
@vovapolu (ENSIME sponsored developer).

Full release notes https://github.com/scalacenter/scalafix/releases/tag/v0.5.10

Been working on “scope aware pretty printing” for past two weeks.
This feature will enable features like organize imports and allow ExplicitResultTypes
to insert short readable names along with imports instead of _root_ qualified names.
I’ve made good progress, but still more time to make sure the feature works correctly.

10 Likes