The Scala Center's roadmap for a Unified Scala.js Ecosystem

Scala Center Updates
1

The Scala Center team is dedicated to providing regular and transparent community updates about project plans & progress. In this forum we created a new topic category to allow quicker orientation going forward: “Scala Center Updates”. Even though all feedback is welcome, we keep the right to make executive decisions about projects we lead. Overview of all our activities can always be found at https://scala.epfl.ch/records.html .

The Scala.js ecosystem contains individual pieces of great quality, from the compiler to the UI libraries. However, it is a challenge for every newcomer to find the pieces that are relevant, to connect them, and to build a good development experience. Good practices are only built in-house and sporadically shared through ephemeral channels like Gitter and Discord: incremental save-reload setups, integration with npm modules and their facade types, deployment together with a web server, etc. This means that only the most enthusiastic developers (and that means almost exclusively existing Scala developers) actually invest in it and build professional projects on top of it.

We believe that we can dramatically improve on this situation, and hopefully attract new people to Scala(.js), by providing a clear, “unified Scala.js ecosystem”. As a result, we should be able to advertise Scala as a great language to write front-and-back-end systems.

Here is our roadmap for this endeavor, which we are starting now (early February 2022).

List existing libraries, tools, tutorials and templates

Our first phase is to build up more-or-less comprehensive lists of all the Scala.js-related resources out there: libraries, tools, tutorials, getting-started templates, best practices, etc. Those lists will be used as the basis for the following tasks.

In addition, identify the major actors of the current ecosystem: library authors, users active in the Discord/Gitter channels, etc.

Expected time span: 1-2 weeks

Brainstorming meeting with Scala.js ecosystem actors

With the people who would like to participate, build up consensus on what are the best libraries for Scala.js, what are the best “alliances” between libraries, what are the best tools, and best practices. In addition, identify what are the biggest “holes” in our ecosystem: what tools are missing, etc. Going out of this brainstorming session, we would like to create a “task force” for the rest of this project with willing actors.

Expected time span: 2-3 weeks

Map out the ecosystem

Starting from the pieces identified earlier, map out the Scala.js ecosystem, its use cases, the libraries and tools that fit those use cases, etc. Build a sketch of what we would want the experience of a new user to be: what path would they follow and how, depending on their use case. We would need to reach consensus within the task force to proceed further.

Expected time span: 3-4 weeks

Implementation

Up to here was the “research” phase. Next come three areas of “implementation”:

  • New design of the website to follow and introduce the map of the ecosystem, bringing newcomers from zero to where they want to be.
  • Bring together the relevant templates and tutorials, and write the ones that are missing.
  • Develop missing tools.

Expected time span for a first full iteration: 2 months (the different areas above would happen in parallel). Refinements will continue afterwards.

This is a rather long-term project, and it will require the participation of a number of people. We hope that the Scala.js community, and the larger Scala community, will join in on this effort.

Feedback welcome

35 Likes
2

A concern I have about “the scalajs ecosystem” is that I would like to be able to write libraries that just automatically work for scalajs (unless I’m using JDK features that aren’t supported, obviously) without having to do anything special in my build.

As a library author, it’s a pain for me to have to change my workflow to accommodate the plugin, so I choose not to opt in to the sbt plugin and the changes to my build that it demands. But it doesn’t mean that I want to exclude scalajs devs from using my software.

Are there any plans to address the burden that scalajs demands on library authors in my situation?

4 Likes
3

I think that this is an excellent idea. I am fairly familiar with Scala and wanted to build a project using Scala.js on the client side and Scala on the server side (which I did), but it took a while to start, find a good starting template, find which UI libs are best, work out the best way to plumb the client and server together, decide whether to use mill or sbt, find out which libraries work with Scala 3, etc.

So, personally, I think that what you suggest here is likely to be a great help to those in the community who are interested in trying scala.js, and lowering the barrier to entry.

8 Likes
4

This probably deserves a separate discussion. This project here is about having a better, clearer picture of what exists, and how to effectively use what exists as you’re starting a Scala.js project. Improving the tooling situation for non-Scala.js-specific libraries does not fall in the goals of this project.

I don’t deny the existence of these issues. I even mention it when users ask for a Scala.js version of a library, that not every maintainer wants the burden of maintaining cross builds. And I most certainly remember you bringing this up regularly over the years.

That said, it doesn’t seem to be a blocker for the Scala.js ecosystem. Some libraries are not available, and won’t be available because their maintainers don’t want to support it. That’s OK. There are plenty of them that are available.

So no, I’m afraid there are currently no plans to address that.

3 Likes
5

I think the most important thing for the ecosystem would be:

  • Extract the npm dependency logic from scalajs-bundler into its own plugin and publish it
  • Standardize it. Publishing libraries that depend on JS libraries should not be build-tool dependent.
  • Up to date support for Webpack, Vite, Snowpack, and whatever else is standard these days

I just found https://github.com/iterative-works/mill-webpack. But I think tools like that, scalajs-bundler, and ScalablyTyped need to be more than just individuals’ projects. I think they are critical to developing in JS and should be recognized as basic pillars. I’m not getting into GitHub organizations or maintainer ship or other politics, just that these need to be (a) known as the standard and (b) contributed to so that they are up to par

6 Likes
6

I am not sure if you have the concept of a tools API like Scala Native or not. It seems like it would be helpful to put forward and publish as much of this as possible. Then different plugins like sbt, Mill, scala-cli, or other tools can have access to what is needed to make the ecosystem flourish.

This might open up alternative ways to interact with the compiler etc. Maybe this is part of what Sam was suggesting because then one of the normal build tools should not be needed.

4 Likes
7

@nafg The tooling wrt. the JS ecosystem(s) is definitely one of the top things I have in mind. I hope most of the required logic already exists, scattered in people’s builds, and that we can consolidate it with meaningful tooling.

@ekrich Scala Native took its tools API concept from Scala.js. :wink: Yes, we have it, although it’s now decomposed in two independent modules: the linker API, which deals with production of .js files from .sjsir files; and the JS envs API, which deals with executing .js files using various JS engines. This is definitely published, and already used by sbt, Mill, scala-cli, and I think every other tool that handles Scala.js.

For interacting with the compiler itself, it’s no different than the Scala/JVM compiler. The only difference is one more library on the classpath (scalajs-library_x.jar) and one compiler flag (-scalajs for Scala 3, -Xplugin:.../scalajs-compiler_x.jar for Scala 2).

Now, if we build more tooling that needs to interact with the npm ecosystem (which is quite likely), I agree that we should always put the core functionality in build-tool-independent libraries.

4 Likes
8

I think your message got truncated?

9

Hum, I think the pending “This was” at the end of my message was from an early draft. I removed it.

2 Likes
10

Very cool. I figured so much but I am not all that familiar with the internals or artifacts published.

11

I’ve used scalafiddle a lot few years ago while stackoverflowing :slight_smile: )… my experience was much better than with scastie.scala-lang.org but it is no longer there… at least most of the time.

2 Likes
12

Sounds great! The ecosystem is actually fantastic but I’ve noticed many people struggle to learn what’s available, and how to get started with it all.

Full-stack Scala has been my passion for around 10 years, and I’ve put a lot of work into creating missing tools and improving the experience. Would love to be involved in this.

I don’t put much effort into evangelising my stuff but here are a few quick resources of mine that I find really useful, that I think would be quite valuable to share more proactively with the community:

  • Scala Test State - IMO this is the best webapp testing experience in the Scala world. Great for both unit testing your UIs as normal Scala.js tests (you can even property-test your UIs), and doing external webapp testing on the JVM via Selenium. In fact, it’s possible to share 99% of your test code so that and reuse it in both internal unit tests and external Selenium tests.

  • scala-graal - Makes it very safe and performant to run Scala.js output (i.e. your webapp JS) on a Graal JVM. Personally, my main use case for this is React SSR (server-side rendering) which is very important for getting frontends to load extremely fast despite the massive Scala.js blobs of JS taking a while to download, parse, and run. (tutorial)

  • webapp-util - A little treasure trove of useful utilities when making full-stack Scala apps. Unfortunately it’s pretty time-consuming extracting this stuff from closed-source webapps and making it OSS; there’s much more I’d love to extract and use in all future webapps I make, and improved documentation would be great too. But I think what’s in there already is pretty useful.

There’s also other stuff like scalajs-benchmark and ScalaCSS but they come up pretty quickly in Google searches when you’re looking for a tool in that space, so don’t need to be highlighted as much. Would still be nice to be included in whatever map-of-the-ecosystem Scala Centre comes up with though.

Moving on, I’ve also noticed some common problems come up quite often with people wanting to get started with full-stack webapps:

  • lack of sbt new / giter8 templates
  • how to serve their Scala.js. Two flavours to this:
    • they just want a blank html served that loads the JS, server just serves static files
    • they want a proper Scala backend with their own logic and endpoints, but don’t know how to integrate their Scala.js output
  • having a decent dev experience in a full-stack Scala app, inevitably requires some coding on the sbt side. A bit tedious for experts and people like me who work on multiple full-stack projects, very daunting if not a showstopper for many less experienced developers. I’ve noticed that many devs don’t know how to do much with sbt beyond the very basics of module and library management. As long as this necessity exists and the onus is on devs to customise their builds so intimately, this is always going to be a significant issue for adoption. It’s not always the same code either, what you do really depends on the type of webapp and the tech involved. It might be solvable via a very flexible sbt plugin, or maybe a collection of well-documented snippets could be maintained?
  • scalajs-bunder initially seems appealing and lots of people try it out and really do try to make it work (myself including, back in the day) but I never see anyone stick with. The consensus seems to be that it doesn’t work properly and causes much more trouble (and pain, sadly) than it’s worth. I’d recommend not suggesting it to newcomers but if you’d like to plough ahead and do so, I think some time and effort is required to really understand it’s problems and improve the tool first; this includes considering how to keep it working, not just get it into a seemingly-working state once.

That’s enough of a brain-dump for now. Hope it helps

12 Likes
13

Thank you for the detailed comment, @japgolly. This is exactly the kind of information I’m looking forward to gather. :slight_smile:

3 Likes
14

Hi

This is a great initiative. We have been using scala.js for years and are now building fairly large frontend apps with scala.js and scala jvm on the backend.

I have personally contributed to scalajs-bundler and a few libraries for the ecosystem and in general I’m quite satisfied with happy with the experience. I’m always amazed that we can run very complex multithread algorithms (including cancellation) developed on the jvm just as well on js.

There are a few pain points of course where there could improvements with tools and code/documentation.

  • Integration on the js world. Writing facades is not very fun and in general you can’t rely on the ones published as they are often not maintained and expose only what the original user needed. Scalably Typed is a life saver here and it maybe promoted more.
  • JS tooling. The JS world moves at a different speed with lots of development time invested there. scalajs-bundler has been a good start but it is essentially impossible to keep up with the js world. Moving from major version of webpack has been really hard and there are many other options. It maybe better to move to a model where you only have a way to declare npm dependencies but the bundling process is left to js tools evolving at their own pace.
  • Development experience. The scala.js team has done a wonderful work with the tools and the sbt integration. It would be great too to have some improvements on this area though. In general debugging on browser is harder and t for very large apps we are getting into some limits. as reported on the Intermediate Splitting mode issues.
  • node/browser balance. I guess most scala.js apps will be either browser focused but there is space for apps running on the backend on node. It seems now most of the documentation is actually neutral but newcomers may need better introductions to either style

This is just some ideas, I’d love to keep discussion and bring some of our experiences on big apps in production

7 Likes
15
  • node/browser balance. I guess most scala.js apps will be either browser focused but there is space for apps running on the backend on node. It seems now most of the documentation is actually neutral but newcomers may need better introductions to either style

Disclaimer: I don’t use scala-js directly so my understanding is very shallow.

Running scala-js programs on AWS Lambda is definitely a thing, and we’re now seeing libraries (Skunk for example) cross-built specifically for this purpose. Such libraries might not work at all in the browser, and certainly there are libraries that make no sense running on node. It would be nice if there were some kind of standard strategy for identifying the target platform so you don’t inadvertently pull in something that’s going to barf at runtime.

I will also note that @cquiroz and I are on the same team so +1 for his comments.

4 Likes
16

It would be great if tooling would support the generation of node modules including TypeScript type definition files. This would ease the usage of libraries written in Scala.js by applications that are mainly developed in TypeScript.

2 Likes
17

Ah indeed! I do want to clarify one of my comments about scalajs-bundler: the JS world moves fast and oh do they break things. Keeping a tool that uses webpack or similar, up-to-date so that it always works is no trivial task, and will require regular upkeep. It’s not to do with the quality of code or whatever on the Scala side, it’s that it genuinely is quite a moving target that’s not easy to keep up with without regular attention. One on project I worked on, we used webpack outside of sbt just to generate blobs of 3rd-party JS dependencies and even without any Scala involved, it needed more maintenance than you’d expect.

Speaking of which, this reminds me of two other points for this initiative

  • external dependencies. Nearly all webapps end up pulling in 3rd-party deps at some point, usually JS but also WASM, webfonts, styles/themes, etc. There are many ways of managing these, as well as many needs depending on your project and environment. Examples:

    • add CDN links to static html
    • bundle them with Scala.js output
    • create separate bundles that your Scala.js will expect to be loaded by the time it runs

    For very large projects, I find externally-managed bundles to be the best but that literally requires a separate JS build for the frontend assets. I have (yet another, sigh) tool called webtamp that’s for JS builds and does a bunch of very useful things in this space, including Scala integration. Dunno if you want to spruik it but it might be an idea to read through the README to see the kind of needs that I had in a very large Scala.js project.

  • dev vs release builds. It’s not uncommon for JS libs have dev builds and release builds. For example, React will perform more checks and provide more dev info when you’re using the dev builds, where as you’d want to use the faster, smaller production build of React in your prod releases. This impacts external dependency management (mentioned above) and usually needs some custom sbt love to support in your build. On smaller projects we’ve just manually maintained different html files with cdn links + some custom sbt code. On my largest project, we use webpack and webtamp to generate different sets of modules:

    • dev - frontend assets when sbt is started normally
    • prod - frontend assets when sbt is started with a custom flag that puts it into release-mode
    • test - for inclusion in standard Scala.js unit-tests which still run on phantomjs, both for legacy reasons (migration to node is actually very hard), and because it’s still like 10x faster than testing on node
    • test-node - for inclusion in Scala.js unit-tests that run on node
    • ssr - for the JVM to load for React SSR
1 Like
18

i found myself investing some weeks just to have a glimpse of the Scala.js User Interface ecosystem, however it does not end by knowing the available options but you have a choosing dilemma, one may believe there are just few Scala.js libraries out there but found that there are more than one would like, the problem is that making choices consumes way more time, especially when you have several high quality libraries, do you want react? then you have two options @japgolly’s ScalaJs-React or @shadaj Slinky ES6-like DSL, if you are a purist and want to use a native scala.js library you have plenty of more options, to mention some: Elm inspired Tyrian by @davesmith00000, observable based Laminar and reactive and vdom based on snabbdom Outwatch, the only good news is that for games we only have one Indigo also by @davesmith00000, and that’s just for the scala.js libraries, for the plethora of typescript libraries you can use them by generating automated scala.js facades using scalablytyped by @oyvindberg.
Having many options to choose indicates the ecosystem is growing but is time consuming and leads to fragmentation, nothing we can do but help users to discover options and choose faster and save their time.

2 Likes
19

For a Scala/Scala.js website i would like to have a language principles page like Flix lang principles, unlike superficial marketing/advertising, principles talk by themselves and are self-evident, they enable a higher form of evangelism, anyone that understand the design principles of Scala does not have to appeal to comparisons with other languages to convince itself of using Scala.js. It may not cause effect in all people, but it may be amazingly effective for some that care.

1 Like
20

Hi,

My biggest problem was (and still is to a certain extent), which framework should I choose for frontend work. There seem to be a lot of rather small frameworks (small in number of contributors, and/or activity c.q. documentation). There is no single framework that really stands out. The only thing you could go by would be to use a react frontend (I tried Slinky for example), but if react is not your cup of tea, there is not really an alternative (community size wise).

Also a lot of the discussions seem to take place on platforms such as discord, or gitter, but finding information is not easy on those platforms. Quality of documentation varies a lot from framework to framework and finding examples in the wild is difficult. For example, for outwatch, a very interesting framework, it’s very hard to find examples.

It does take a certain investment of time in order to choose one and get fluent in it, so you don’t want to choose a framework that will be abandoned one or two years down the road.

All this leads to quite a bit of anxiety over which one to choose, at least it did (does) for me.

3 Likes