The Scala Center's roadmap for a Unified Scala.js Ecosystem

Scala Center Updates
21

Thank you for your comments, @saulpalv and @marcvk. If I understand correctly what you’re asking for regarding the choice of front-end library, it would definitely be a part of the “map of the ecosystem” that tells users where to go given their use cases.

I think Laminar is the React-less alternative with the biggest community size.

While I think this is something interesting to have, I don’t think it falls within the scope of this project. Definitely not for Scala in general (not Scala.js-related). For Scala.js-specific language principles, the canonical reference is Chapter 2 of my PhD thesis, Cross-Platform Language Design.

3 Likes
22

Actually I feel there’s some good Scala.js libraries for front-end. I’m not into frontend but I had the opportunity to test Laminar (a reactive-oriented lib mixing logic and UI description) and, younger, Tyrian (Elm-inspired, separating UI description and events. More functional-oriented than Laminar IMO).

Note Scala has less “big framework-culture” compared to other languages like Java, PHP, JS for front etc… You’ll tend to find small libraries focusing on a specific feature/theme than a big framework covering your entire application.

3 Likes
23

Thank you for starting this initiative, it’s very important to bring the ecosystem forward!

I’m working with ScalaJs for about 5 years now and built a few bigger projects with it. I’m also one of the authors of Outwatch. Here are some things I’d like to see being improved:

  • Using external JS dependencies is always tricky. Earlier I was writing facades myself, which were annoying to maintain. Over time, the automatic facade generation (ScalablyTyped) got much better, which makes it possible use many JS libraries out of the box, which is wonderful. But one is still encountering many bugs for specific libraries, which puts you back to writing facades on your own. Another annoying issue is this one: https://github.com/ScalablyTyped/Converter/issues/293

  • Upgrading external JS dependencies is difficult. Using scalajs-bundler, it’s not possible to use js tooling, like dependabot to automatically update JS dependencies. And if you’re updating, your facades might be out of date and throw runtime errors. A stable scalablytyped converter which reads its JS dependencies from a package.json file might be a direction to look into.

  • Scalajs-bundler As already mentioned, scalajs bundler is difficult to manage for larger projects, because you need to write your custom webpack-config anyways. Others already mentioned this, so I’ll skip the details.

  • A working Scalajs-fiddle is needed, where users can add their own dependencies (scalajs AND JS using ScalablyTyped), similar to scastie. This way, it’s easier for a) library authors to provide interactive examples that can be embedded inside the documentation - and b) for users to play with new libraries quickly instead of setting up example projects.

  • Connecting to postgres from scalajs running in node turned out to be not so easy. The JVM database libraries like doobie don’t work with ScalaJS. So I tried to turn to JS libraries like pg-node, Prisma, Knex, etc and hit a few bugs in scalablytyped (https://github.com/ScalablyTyped/Converter/issues/created_by/fdietze). I just learned about Skunk from this thread and will look into it soon :slight_smile:

  • Sane source map paths for scalajs builds are not a default. That’s why most source maps point to local project folders of the library authors. Here I found a nice plugin to use: https://github.com/ThoughtWorksInc/sbt-scala-js-map (but a plugin shouldn’t be necessary here in the first place)

  • Scala-dom-types (https://github.com/raquo/scala-dom-types) Is a wonderful project that should be used more among frontend-libraries.

  • Scalajs-dom should ideally be generated from typescript facades for maximum compatibility with libraries.

With everything we’ve learned over the years, @cornerman and me are working on a full-stack serverless scala framework called fun-stack. It is a work in progress and the documentation is missing many things, but we’re already using it in production for some projects. And the template is working well and a great way to play with it: https://github.com/fun-stack/fun-stack-example
Some quick facts:

  • Full-stack Scalajs (frontend in (but not limited to) Outwatch, backend with AWS lambda)
  • Serverless. Costs almost nothing when deployed for projects with low traffic (CockroachDb serverless (https://www.cockroachlabs.com) turned out to be a good database choice here)
  • Good local development experience. Hot reloading of frontend and lambda. Lambda is running locally with emulation of APIGateway. Lambda can still access AWS resources, even when running locally.
  • Webpack sane defaults for dev and prod
  • Frontend communicates with backend via websocket (boopickle/sloth) and/or http (tapir)
  • Auth integration with hosted UI (cognito)
  • Deployment via Terraform (static page hosting on Cloundfront, Cognito, ApiGateway etc…)
  • In the future: send events from backend to frontend over websocket, exposed as observables in the frontend
6 Likes
24

Thanks, this is a much needed initiative.

Personally I was never quite able to make sense of Scala’s tooling after switching from JS world to Scala.js a few years ago, but because I have to use Scala tooling now, I’ve lost my grip on JS tooling too, so now I’m not really qualified to give technical advice on these matters at all. In terms of desired outcomes, I agree with a lot of points raised here, especially as it relates to decoupling scalajs-bundler and working with npm dependencies.

I will say that for me it is a lot easier to find libraries, resources and support for issues with JS tooling like webpack than to figure out anything specific to Scala.js tools like scalajs-bundler. Not because JS tooling is better or simpler, but simply because the JS community is much larger than that of Scala.js, and JS tooling is the standard for millions of devs. This will never change, so we should embrace it and make better use of those people’s work.

Additionally, I would like to double down on the ScalaFiddle issue since it’s been relatively underrepresented.

ScalaFiddle was very useful to try code snippets, and to share them with users when providing help. Scastie doesn’t work well for trying DOM-manipulating code (it’s slower than ScalaFiddle, uses old versions of Scala.js and scalajs-dom, doesn’t have an obvious DOM output window, and can even manipulate DOM itself from under you). I tried to get a Laminar example to run on it right now, but couldn’t.

Also, I don’t remember if ScalaFiddle had this feature, but it would be even more awesome to have ScalaFiddle output embeddable along with an “Edit on ScalaFiddle” button. That way I could make my live examples editable by users (currently they’re read-only, implemented with mdoc). But maybe that would invite too high of a server cost for ScalaFiddle, I don’t know.

4 Likes
25

As pointed out by many, working with scala-js-bundler is frustrating for a newcomer. One may want to use Scala.js for writing code/tests and let a modern Javascript build tool (instead of sbt) do all the wiring.

But I could not find any example or template which shows how to do this and also achieve: a) use of Javascript libs defined in package.json from Scala.js b) HMR (Hot Module Reloading) during dev flow and c) successfully run unit tests written ScalaTests.

After a lot of trial and error, I finally managed to make a Scala.js sbt project work in harmony with Vite.js. See the code here, there is also a README if you want to try it out.

The most tricky part was wiring up scala-js-env-selenium for tests. I am still not sure if this is the idiomatic way of doing it. Would like to see some guidance for this use case on official Scala.js website. Curious to hear how others are addressing this problem.

Edit: I wanted for use ES-Modules instead of CommonJS because browser was the main target environment and also because Vite.js works really well with ESM. Hence scala-js-env-selenium was required which allowed me to run unit tests in headless Chrome.

3 Likes
26

Thank you for the recent replies. I am falling a bit behind on answering to individual comments, but I am definitely adding the points raised to my list of things to take into account. :wink:

It seems that gripes with scalajs-bundler is a recurring theme already. Clearly we will need to do something about this. Directly using JS bundlers such as Vite is probably the right path forward.

7 Likes
27

Hi, I was working enthusiastically with ScalaJS when it came out in 2014, seeing its amazing potential. I then left for academia for a few years but came back to Scala in the past year, writing a web server and client libraries for Tim Berners Lee’s Solid (Social Linked Data) project Solid-Control. Some projects

  • I have added asymmetric cryptography to the bobcats cryptography library from Typelevel, so that it can work in Java and in the Browser. Todo: is adding support for nodeJS. (I found that one has to learn about the differences between these ecosystems, which I fear is going to be quite some work)
  • Use that library to implement the IETFs “Signing HTTP Messages” RFC in httpSig which compiles to Java and JS (in the browser. Todo extend it for Node cryptography).
  • upgraded to Scala3 the RDF library banana-rdf so that one can write semantic web code and switch in one line to have it compile to various Java or JS RDF frameworks.

In this process I have been stumbling getting sbt to work, especially when working with tools like ScalaJSBundlerPlugin or webpack (eg here). I get the feeling I am reduced to magic incantations when trying things out somewhat randomly here.

I have found using sbt-typelevel to be very helpful, though even there I am getting stuck on the ScalaJSBundlerPlugin.

2 Likes
28

I would suggest that Scala.js should embrace JavaScript/Node ecosystem as much as possible, rather than try to insulate Scala/JVM devs from it. This might be an unpopular suggestion among the Scala developers though.

Concretely I mean:

  • if Scala.js could be installed as an NPM package;
  • all its binaries compiled to JS and available via npx on command line;
  • Scala.js loaders for Webpack and esbuild;
  • could build and publish CJS and ESM modules;

Integration story should be close to TypeScript. Ideally, Scala.js should not require even the JRE installed. That would make the language appeal to JS developers that would otherwise be reluctant to get into JVM. If you are coming from Scala/JVM, just imagine you could get started with F#/Fable without installing .NET framework. And if you target AWS Lambda NodeJS runtime, why would you?

7 Likes
29

That would likely grow the scalajs community tremendously. But it’s very challenging technically.

5 Likes
30

I’m unsure about this. I’m not saying I have any better ideas though, and I’m totally open to suggestions. :smiley:

You cited TypeScript as a role model which is a good call. For comparison: At the other end of the spectrum we have, for example, Elm (and ReasonML, and PureScript - but I’m picking on Elm because I’m familiar with it).

I believe Elm meets all your bullet points but this has not lead to mass adoption, and their stated goal was to convert the hoards of JS developers rather than the relatively few Haskell’ers. The elephant in the room being that most JS devs don’t seem to want to do Haskell-like super-constrained functional programming. Strangely enough.

I’d suggest that Scala.js is arguably somewhere in the middle of Elm and TypeScript, admittedly leaning towards TypeScript.

The conundrum is, I think:

Scala devs aren’t brought up as frontend devs (we’re trained to do microservices and big data), and have a hard time moving over conceptually. It’s all a bit scary and different and fiddly and confusing and let’s face it: Not the JVM.

JavaScript devs know the frontend inside-out and back-to-front, but aren’t necessarily interested in deviating away from JavaScript much further than TypeScript has (at least at the moment). However, there are A LOT more JS devs than Scala devs, so you only need a relatively small percentage to adopt (as Elm tried to do).

So which audience do you go for?

If you went for the Scala crowd, actually you might be better off avoiding as much of the NPM ecosystem as possible, since the commonly held view is that it’s all a messy nightmare and our ivy/maven world is much better and familiar and less fraught with weird dangers.

If you want the JS devs then yes, your suggested direction is probably sensible. But ecosystem integration is technically hard and really, its only table-stakes. After you’ve got that baseline tooling sorted, then the work begins to convince people - JS Devs - to actually use it.

There’s tooling work to do either way to be sure, and currently we’re neither one thing nor the other. Is that a bad thing? Is there an interesting middle-ground?

Speaking purely selfishly: I want to see if we can teach the Scala community to be creative. To see if they can get as excited about generative art, and games, and slick interface animations, as they do about effectual monads and reactive streams. If we could get the Scala community to build GUI’s that are comparable to what the JavaScript crowd can, but with fewer bugs, better security, a faster dev cycle, and an enviable process/experience …maybe we’d win anyway?

…but then again, maybe I’m dreaming. :grinning_face_with_smiling_eyes:

7 Likes
31

Yes, I totally get it and that’s why I said this suggestion might be unpopular. And yes, Elm is a great example of great integration too.

Your arguments got me to think that maybe converting even the smallest percentage of JS devs to try Scala.js would be a huge influx for Scala ecosystem.

I would also argue, that my suggestion may also benefit Scala devs. Currently, insulating JS-tooling behind sbt and plugins often results in leaky and limited abstractions. And when abstractions are limiting, there is no way around something that would be easy in JS-world. JS ecosystem moves much faster than Scala.js and it’s a losing battle trying to catch up building abstraction layers over it. Maybe I’m just lucky, but I run into unsupported edge-cases all the time :smile:

If JS-tooling was first-class for Scala.js, developers would be able to workaround things by pulling off the abstraction layer and going directly to NPM, Yarn, Webpack or any other JS-tool. Insulation layer of sbt on top would still be limiting, but would likely make most Scala devs happy, while still allow more advanced use cases.

I’m also just dreaming, but maybe someone more close to Scala.js tooling could comment on details of what would it involve to support native JS-tooling?

1 Like
32

Assuming I’m following you correctly, I think we agree about that. I’m not a fan of hiding things behind sbt either - for one thing - I like using Mill just as much as sbt! :grinning_face_with_smiling_eyes:

Most of Tyrian’s examples use Parcel.js rather than scalajs-bundler for that very reason. Sooner or later you’re going to need to include CSS and images and so on, so you might as well roll up your sleeves.

Having said that, there might be an alternate state we could reach.

It’s true that the world of JavaScript itself moves quickly, and the understandable knee-jerk reaction is to get on-board the latest trendy bandwagon …but you know: We don’t have to keep up, we could just do something else.

Back to Elm: Elm’s killer app should have been Elm-UI (oh and careful! There are two elm-ui’s! :man_facepalming:), which abstracts away HTML and CSS. Think of it! No more cross browser compatibility issues! In theory at least, you just say “put a red box in the middle of the screen” and all the gnarly HTML+CSS+polyfills+kitchen-sink is generated for you like magic.

So if ‘HTML+CSS+JavaScript’ are the ‘assembly language’ of the web, and we’ve replaced the JavaScript bit with Scala.js, what would it mean if we made all that other stuff into a build artefact too?

Well for one thing, the normal JavaScript tooling and ecosystem would be redundant. Suddenly it’s perfectly reasonable that everything works just like normal Scala projects do, and it’s not a fudge. I love that idea.

In the meantime, while we wait for some brave soul to go on that particular adventure, my feeling is that both worlds could co-exist quite comfortably next to one another. Scala.js as a better JavaScript, largely strapped into the usual JS/web toolchain, and later, maybe as something else too.

I digress: Yes, baking all the things into directly into sbt definitely has its uses, but I personally don’t feel like it should be the default encouraged solution.

…I tend to be in the minority when I say things like that though. :sweat_smile:

(We may be going off topic here, so I’ll try and bow out at this juncture.)

3 Likes
33

@shishkin I think I’ve been exploring similar ideas to you. For the reasons you mentioned I also felt that we have it backwards and rather than try and wrap JS ecosystem in sbt we should try to make Scala.js easily accessible from standard JS tooling.

Towards that, I have two experiments:

  1. http4s-cli, which uses npm as the build tool to publish a CLI app to npm. scala-cli does the heavy-lifting on the Scala front. In fact, I created the project by following a Typescript tutorial and replacing tsc with scala-cli since they are fundamentally doing the same thing.
  2. A PR to an existing template for a fullstack Scala.js frontend + Scala JVM backend app, that completely replaces sbt with npm+scala-cli. It’s not perfect, but you can run tests, start a preview server, and package it all up into a Docker image, all from npm. Even better, the scala-cli team has been very responsive to the issues I encountered while trying this out.

These are still exploratory ideas — for example, I haven’t had the chops to involve a bundler in this yet :slight_smile: — but I hope that this approach might be interesting to some folks and shows where we might want to invest our resources.

2 Likes
34

@armanbilge Thanks for sharing! I didn’t know about scala-cli and it does look like a great tool. Especially prebuilt as native binary for the CLI part. I think packaging and installing it via NPM should be easy.

I wonder if it could ever be made to work without the JVM :thinking: Coursier provides native binaries, but I don’t know about the whole Bloop thing.

35

The compiler cannot work without the JVM. It needs its class loading and reflection capabilities for macros and compiler plugins.

2 Likes
36

This might be a bit out of scope, but I’d like to share some experiments I did many years ago.

I created a sbt plugin extending the ScalaJS plugin ( I think), I hooked into the fastOptJS file output stream and used websockets to push it directly into the client without going via the filesystem. The client then swapped the script dom node with the new compiled Javascript.

I think this was inspired by a blog post or presentation by @lihaoyi ?

Without any kind of optimizations, (I just pushed a huge string using fs2-http client and fs2-streams), this is the fastest devcycle I have experienced in any Javascript / ScalaJS project.

I think it could have been even faster, if it was possible to create a Javascript file containing the full Scala API, adding this to the webpage as any other Javascript library (for development purposes only). Then when compiling your Scala project, you only push your own code to the browser (maybe even only the diffs?), it would then use the Scala API lib which already is in your webpage.

As I remember this was not possible to achieve at the time due to how the compiler worked, but maybe this is possible to achieve now using the module system somehow?

Unfortunately I never had time to follow this up and the code is gone it seems.

I think this would make it possible to have a complete standalone ScalaJS dev setup with sbt out of the box. That would be valuable if a backend Scala developer would like to do some web development, without having to learn the full node stack as well.

2 Likes
37

A project generator on scala-js.org would be a great help to drive adoption/lower the entry bar I think. Few g8 templates exist and they quickly get outdated or are too opinionated. Links on scala-js unfortunately in many cases link to outdated/unmaintained libraries and example projects which leads to a frustrating experience if trying to get a picture of scala-js land from there.

A project generator could help developers up and running in a much more pleasant way. I imagine making a series of choices online and have the generator create a zip with all project files generated and set up (or some other delivery mechanism). Making choices on a website allows for much richer and helpful help/information during choosing than g8 and a cli. And choices at each step would determine possible/reasonable choices in next steps.

We could start off simple with for example the following sequence of choices:

Scala version (checkbox choices)

  • 3
  • 2.13
  • 2.12

Platforms (checkbox choices)

  • jvm
  • js
  • native

Project type (radio choices based on platform choices)

  • jvm/js/shared
  • .jvm/.js/shared
  • …

Just getting a project scaffolded with these simple choices woult be a great start, specially for scalajs newbies which we want to attract.

Then the generator could be expanded gradually with more choices, like for instance (in no particular order):

Client framework

  • Vite
  • Laminar
  • React-Js
  • Tyrian
    …

Server framework

  • Play
  • Akka Http
  • Lagom
    …

Example code

  • Hello world
  • Todo list
  • Real world implementation
    …

Authentication?
other useful generated code…

Idially integration element choices/source generation should be supplied by library authors in some standardized way and be guaranteed to be up-to-date to avoid blind alleys.

I recall having seen an online project generator elsewhere but haven’t been able to find it again - anybody know of one?

1 Like
38

Having better project generation features is a good idea.

But why on some web site? This is something that belongs into your build tool of choice, imho.

Extending on the features of sbt new¹ or bleep build new² would be imho the right way.

I guess building a kind of “Wizard” into the build tools wouldn’t be too difficult.

On the other hands side building a web-based Wizard that can generate at the same time projects for all the build tools out there would be more complex.

The other thing is of course: Websites and templates need to be kept up to date. Otherwise this looks really bad, indeed!

¹ sbt Reference Manual — sbt new and Templates
² Create new native build | Bleep

2 Likes