Scala documentation website roadmap

[julienrf]

Dear Scala contributors,

The Scala documentation website is where both newcomers and experienced Scala developers expect to find answers to any Scala question they might have. Over the years, and especially with the recent release of Scala 3, a lot of documentation content has been scattered at various places. This thread presents our Scala Center project for updating and improving the current state of the documentation.

Problem

Since the beginning of the development of Dotty (which just became Scala 3), its documentation has been hosted at https://dotty.epfl.ch. The Scala documentation website progressively incorporated content related to Scala 3, and we are now in a situation where there is a high level of overlap between the content of the Scala 3 documentation on https://docs.scala-lang.org, and https://dotty.epfl.ch.

In parallel, unrelated to Scala 3, the content on https://docs.scala-lang.org has grown organically over the years. As a result, the website contains a lot of important information (like how to publish a library), but that information is a hard to find out when browsing the site. Furthermore, as recently raised in Next steps in further improving community experience after the awesome Scala 3 release?, the instructions for getting started are still too cumbersome (this point is also supported by SCP-26).

The goal of the Scala Center is to improve the current state of the Scala documentation website, by centralizing the content at a single place, by simplifying the organization of the documentation to make things easy to find, and by double-checking that the content is up to date.

Solution

• (done) Review the content of the Scala 3 book to make sure its content is up to date with the recent developments of the language. Related pull requests: #2062 #2073 #2076 #2079 #2088.
• (done) Merge the content of https://dotty.epfl.ch into https://docs.scala-lang.org, so that the content is at one place only.
  This will include removing duplicated set up instructions, and creating new pages on docs.scala-lang.org to host the Scala 3 reference documentation. After that, https://dotty.epfl.ch will be archived (read-only, with a banner pointing to https://docs.scala-lang.org). Related pull requests: #2112, #13094, #2137, #2136.
• (in progress) Overhaul the structure of the documentation. This work requires some brainstorming, and we are happy to hear your suggestions in this thread. We will post an update with a plan, open to discussion, as soon as it is ready. Delivery estimate: November 2021.

[amitport]

Any chance we could try to avoid the type of document separation that exists in scala<3:

IMHO, there is really no need to have: ‘language spec’, ‘overviews’, ‘scala book’, and ‘style guide’.
The first 3 should be merged into one comprehensive scala book (overviews are just an introductory sections to bigger chapters, language spec is just an addendum to a chapter) this way there is nothing hiding in the spec (which currently hides a lot of surprising details, and is generally written in a less friendly manner. Maybe because it is only meant to be read by “professionals”). ‘style guide’, is just another chapter.

Another thing is that ‘tour of scala’, should be added to ‘getting started’ IMO, but that’s less critical.

This is a very real issue with scala in my opinion and although there is an amazing amount of quality content, I really hope it could be better organized (scala 3 opens the opportunity to approach this early).

Thank you

[martijnhoekstra]

The spec really shouldn’t be relegated to an addendum to a chapter of a book I think.

[julienrf]

I agree that there are too many types of documents, with a significant amount of overlap between them (e.g., tour, “book”, and overviews are very similar).

On the other hand, I believe that some pages (e.g. Implementing Custom Collections) are quite specific, and I do think they should be aside from the “main” documentation.

So, finding the right balance is not easy!

[jducoeur]

While the documentation definitely needs a lot of reorganization and rationalization, I disagree with the notion that it can or should be squished down to just one or two documents.

There are different use cases, different needs, and different audiences. The spec is a technical document – extremely precise, extremely detailed, but inappropriate for casual use. That’s entirely correct: good languages need precise specs, but those specs aren’t something that most folks should be using. I think I’ve used the spec fewer than ten times in the dozen years I’ve been using the language – I’m not the audience.

Similarly, a good tutorial is not the same thing as a reference manual. We have a good reference manual, but I usually steer people away from it when they are starting out. Reference manuals are rarely good pedagogy, and pedagogy is what matters when someone is learning a language.

There’s a lot of room for improvement. But IMO there are at least three different purposes here, and those are unlikely to be well-served by a single document.

[morgen-peschke]

I agree that having three different documents makes sense from the standpoint of style and consistency of detail.

As fewer of these documents are existing in dead-tree format, one way to reduce the lack of visibility and confusion may be to make extensive use of references & hyperlinks between these layers of documentation.

For example, in addition to a [Next](...) link at the bottom of a page in the tutorial, there could be a link to the relevant section in the reference manual and some thing like this:

For additional details, see The Big Reference Book, Chapter 3, Section 2, which can be found [here](...)

The sections of the reference book could likewise have a link at the end of each pointing to the appropriate sections of the spec for the bits referenced in that section.

Presumably when these are being written, the authors are referencing at least the next most detailed layer of documentation, so the links may already be close to hand in the authors’ notes (even if they don’t currently make it into the final text).

[amitport]

I really disagree regarding the reference (scala book) vs spec, the part about having some quick separate tutorial makes sense, I didn’t mean that all tutorials will be in the reference.

When someone actually writes some code, even very basic scala programs, they very quickly they need to understand parts that are only in the spec, they may not realize it, because of… stackoverflow, google, random github issues search, copy-paste methods. Those do get people moving forward, but we can do better. (Just to give some context, I am teaching principles of programming languages courses to undergraduates).

The ‘quick guide with missing details’ approach is very problematic, details matter. I’m not saying that everything should be written as a tutorial, but every topic should have a guide/examples and accurate full details in the same place, if someone doesn’t want to look at the syntax tree or the precise inference algorithm he/she will skip but it should be easy to find and written in a manner that expects that people will be reading it and not just compiler experts (and I don’t believe the ‘compiler experts’ or the ‘casual users’ are going to find any of this harder to read, even experts need the occasional friendly example).

Also note, I’ve learned, forgot, and relearned scala a few times over the years, really, the experience is fresh to me, you can’t write code in scala without using stackoverflow answers that (inaccurately) dumb down parts that are only in the language spec. I hope this could be improved, scala is complicated, I don’t see the point of hiding details in a separate spec.

[chaotic3quilibrium]

Given everything is public and you have a strong position on how this ought work, why don’t you give your idea a quick stab and flesh out a mockup. Don’t do the entire thing. Just demonstrate your particular perspective with enough fleshed out it conveys what you are attempting to describe.

It makes it much easier to be open to a “new idea” when the provider/purveyor of the new idea has demonstrated skin-in-the-game by investing time in a partial prototype implementation.

[alvin]

I’m sorry that I haven’t had time to reply to this, we’ve been pushing hard to get the Cookbook out the door. But I’ve been jotting down notes from time to time, and I think these are the most useful.

First, I’d start with the desired use cases, meaning, who is going to read these docs, and what do they need?

Second, I’d make “search” available from every page. Currently it’s available from https://docs.scala-lang.org, but not from a page like Collections Types | Scala 3 — Book | Scala Documentation.

Third, one thing I’ve thought about writing for the Scala collections methods is a page on, “I have this problem, what method should I use?” For instance, you might know that you have a List and want to traverse it to end up with a resulting scalar value, but you don’t know what method (or methods) will get you there. So this would be a problem->solution page dedicated to collections methods. (In terms of use cases and actors this is something that might help A1, A2, and L1 devs, though I haven’t thought about its overall importance.)

Fourth, personally, I would put all the documentation navigation on the left side. At first I thought this might be a “preference thing,” but the more I thought about it, I believe it’s a usability issue. For example, when I work with Flutter and Dart, I click a hundred times a day on their ToCs/indexes on the left, and view the results on the right. As an important part of this, I think that ToC should scroll independently of the content on the right. I think this is a very natural way of working, and I prefer it so much that I use it on my https://fpsimplified.com site.

Fifth, anything to make the docs as interactive as possible, such as being like Jupyter notebooks, is good (i.e., use tools like mdoc/tut wherever possible).

Sixth — and I haven’t thought through whether this next idea is any good — but I think a page like The Architecture of Scala 2.13’s Collections | Scala Documentation would be good in an ongoing blog. But, a blog is a commitment, and I don’t know if that’s desirable (though I personally would enjoy a “tip of the week”, which would drive traffic). There are often good nuggets on the Gitter channel that could be made into a short blog post.

Seventh, as an edit to my original post, I think a “Related” section might also be helpful on many pages, so if you happen to be reading a post about LazyList, that section would shown other articles related to LazyList. (Like the Related block on StackOverflow and other sites.)

Also (from previous discussions), I thought the https://documentation.divio.com website had some good ideas for what docs are needed.

I’m sorry that this a little rushed, but I wanted to take a few moments to share these notes that I’ve been jotting down.

[morgen-peschke]

@alvin

Third, one thing I’ve thought about writing for the Scala collections methods is a page on, “I have this problem, what method should I use?”

This would be really helpful because one of the things that I’ve noticed seems to characterize the subset of our developers who are juuuust reaching a level of comfort with Scala, is a tendency to overuse pattern matching to re-implement existing methods.

Don’t get me wrong, pattern matching is awesome, but .getOrElse exists and it’s probably better to use that

A place to link to that has a bunch of common “if you want to do this, there’s a method for that”, along the lines of Your Options Do Not Match would be a very beneficial resource for onboarding new devs.

[alvin]

Cool, that looks like a good resource! I wrote something similar in a How to use HOFs with Option (instead of match), but I haven’t had time to build on it yet.

I know that when I first started with Scala I used for expressions for quite some time before I even started to use map, so anything that can help new developers get down that learning trail sounds good to me.

[julienrf]

Hi all!

After some brainstorming sessions, we built the following plan regarding the documentation:

1. Simplify the path to get started

Problem: the landing page https://scala-lang.org does not link to a “getting started” tutorial (it links to “download” pages that don’t really bring the readers to having a working “hello world” program).

Solution: the landing page should show links to “getting started” tutorials.

Note that there is already work in progress (see scala/scala-lang#1209 and scala-scala-lang#1254), but these PRs are blocked by coursier#1957, which the Scala Center is working on resolving to unblock the situation.

In parallel with this work, we should also consider having a single “getting started” page (that works for both Scala 2 and Scala 3, highlighting how to define the Scala version to use).

PR: https://github.com/scala/docs.scala-lang/pull/2197

2. Add the search bar to every page

Problem: the search bar is only available on the landing page (https://docs.scala-lang.org).
Solution: show it on all pages.
PR: https://github.com/scala/docs.scala-lang/pull/2201

3. Consolidate the navigation menu

Problem: some pages of the documentation are not directly reachable from within the landing page, but only from the navigation menu. However, the navigation menu is not shown on the landing page…

This is the current landing page:

We see below that in the navigation menu there are other entries that are not shown on the landing page (e.g., “Scala for Java Programmers”, or “Scala on Android”):

That problem was also recently reported at scala/docs.scala-lang#2169.

Solution: consider using the same data source to generate the navigation menu as the landing page so that anything on the landing page is also displayed on the navigation menu and vice versa. Or, do not hide the navigation menu on the landing page…

PR: https://github.com/scala/docs.scala-lang/pull/2206

4. Consolidate the structure of the Scala 3 documentation

Problem: the current structure mixes guides, language reference, and news at the same level:

Furthermore, the language reference is sometimes too discursive (which should go into the guides, or the Scala Book). It also contains compatibility notes with Scala 2 (which should go to a specific, dedicated section like the migration guide).

Solution: organize the content into categories according to their orientation (learning, problem-based, discursive, or dry description). Extract the parts of the language reference that are too descriptive, and put them into the Book. Extract the parts of the language reference that compare Scala 3 to Scala 2, and put them into the migration guide.

[AMatveev]

Furthermore, the language reference is sometimes too discursive (which should go into the guides, or the Scala Book)

I use many languages in my work, so a language reference is important for me. I need topics of language features which can be sorted alphabetically.
A topic should contains

• syntax summary
• short description
• reference to guide.

It can be in any place of course. But when it is absent it is really annoying to dug a book.

[martijnhoekstra]

Wonderful! These are all great ideas that I think will significantly improve on current situation.

[julienrf]

Hey all,

This is November, so here is a quick update

We implemented the first three items listed above:

1. docs.scala-lang.org/getting-started now covers both Scala 2 and Scala 3 on the same page
2. The search bar is now available on every page of the Scala documentation website
3. The navigation menu now allows quick access to more documentation pages (including tutorials, which were previously not listed).

I believe this has already significantly improved the state of the documentation website!

From my previous post, there is one remaining item, “consolidate the structure of the Scala 3 documentation”, which aims to organize better the content of the Scala 3 documentation. I realized that before I can start working on this item, we need to settle on the way the Scala 3 documentation website is produced. We are currently considering alternatives to Jekyll, with the benefits of a simpler setup (no need to install Ruby), the ability to produce documentation with interactive code examples, and the ability to maintain the source of the documentation in the same Git repository as the compiler implementation.

Last but not least, another very important point is to simplify the user experience from landing to https://scala-lang.org to actually having all the necessary tools set up to start programming in Scala. This task is addressed by the project Simplify the Getting Started experience with Coursier, roadmap.

[hkt]

It’s not just pattern matching. Developers often miss the subtlety of the placement say maps with getOrElse, which would affect the result.

Know Thy Option

[hkt]

Add the search bar to every page will solve most of the needs … especially if the search bar shows in a drop down the first few top hits with one or two lines of content for context.