Scala3doc - doctool for Scala 3

[AleksanderBG]

(The Scala Org aims to release Scala 3 by the end of fall 2020. We are about 15 employees (some of whom work part-time), spread in 4 organisations (+ active community members), focusing on finalising 52 essential projects in 6 months. Project leads publish the road-maps under the category “Scala 3 release projects” to share with you what is to be expected and hopefully get your advice & contributions as well. All the projects’ road-maps come after an extensive feedback gathering, rounds of discussion, and involvement of major stakeholders, we now need the community to help push this effort over the line. Your collaboration is highly appreciated, thank you in advance!)

Documenting Scala 3

All programming languages need a doctool, and Scala 3 is no exception.

Our plan (temporary codename : Scala3doc) is to write a tool that leverages APIs Scala 3 already exposes (namely, Tasty Reflect) to cleanly hook into compiler knowledge. In order to actually generate documentation, we will hook into Dokka, which already can generate documentation for Java and Kotlin.

We have two primary reasons for depending on Dokka. One of them is division of labour - Dokka already has a team of maintainers, and it supports an excellent API which already allowed us to quite easily generate documentation. By depending on Dokka, we will be able to share a large portion of the maintenance burden. The second reason is very pragmatic - on our own, it’d be difficult for us to reach even feature parity with Scaladoc, simply because of workforce constraints. Meanwhile, Dokka maintainers from VirtusLab reached out to us with an offer of help, which we were happy to take.

New features

With Scala3doc, we will be taking the opportunity to add some new and exciting features to Scaladoc.

First off: we will be adding support for writing docs in Markdown. Since Scaladoc was written, the Wiki syntax started to look positively esoteric compared to how ubiquitous Markdown became, so an update here is sorely needed. Of course, we won’t be forcing everyone to change their comments before they can see any documentation output - old Wiki syntax will be supported as well.

This brings us to the second addition: Scala3doc will also support adding project-level documentation in the form of Markdown files. This is a surprisingly common use-case that suprisingly few documentation tools support. As an example from our own backyard, take a look at the documentation of Future: it links to a separate document explaining the high-level concepts behind futures and promises! Another example: Cats’ documentation features many pages explaining high-level concepts behind FP in Scala. With Scala3doc, everyone will be able to define Markdown documentation alongside their project, link from this documentation to API documentation and back again, and easily display both together, as many projects want to and already do.

Finally, Dokka has an early-stage support for plugins and by depending on Dokka, we should be able to have such a support as well. This would enable the Scala community to write plugins that would extend functionality of Scala3doc. While this is Scala3doc 2.0 material, we could potentially support plugins that extend our capabilities for simple static site generation, adjust the appearance of the output or even add new features such as Latex snippets.

Current status

Quick summary is: status is good!

We already have a PoC-level implementation at https://github.com/lampepfl/scala3doc.

We can generate a reasonable version (especially reasonable given that the work started around a month ago!) of Dotty’s website, which you can see here. We also can generate the project docs. Finally, you can also see our own docs.

We already can document many kinds of definitions that Scala supports, such as classes, traits, objects, vals, vars, defs as well as top-level definitions. This includes generating signatures for each of the above with links to appropriate pages.

We support both Wiki and Markdown syntaxes, as well as basic link resolution.

Finally, since we depend on Dokka to generate our documentation, we also get a variety of utility features such as search and tree-based documentation explorer.

Future milestones

We still have some work ahead of us! Here are milestones we need to reach:

1. Publish an initial version of the tool together with an SBT plugin

2. Replace Dottydoc as the dedicated tool for documenting Dotty code

   This includes:

   • fully supporting Dotty’s current documentation pages, including the blog
   • releasing together with Dotty as the dedicated documentation tool

3. Support all kinds of Dotty definition and generate documentation for the standard library

4. Reach feature parity with Scaladoc

Can I help?

First off: we’re happy that you want to! You can:

1. Write code!

   We have two issue labels you should take a look at: good first issue and self-contained. First is easy pickings: you’ll be able to contribute without needing to dive too deep into the project. Second is reverse: it’s an issue that’s probably interesting, complex and self-contained enough that you can continue chipping away at it without needing to worry too much about merge conflicts.

2. Run the tool on your codebase!

   Our first milestone is to publish an SBT plugin that’d allow the community to easily generate documentation for their projects. Once we’re there, we’ll be very happy to hear your feedback and also receive the inevitable bug reports!

People involved

So far, we’ve been working on Scala3doc in a team of 3 people: Aleksander Boruch-Gruszecki (me), Krzysztof Romanowski and Filip Zybała. Krzysztof and Filip are affiliated with VirtusLab. I’d like here to thank Krzysztof and Filip for their work, and Krzysztof specifically for involving VirtusLab with the project - if it wasn’t for him, the project could not possibly be where it is right now.

I’d also like to give a shoutout to Guillaume Raffin / TheElectronWill, who has offered to help us with the frontend appearance of the documentation.

Finally, I’d like to thank Bryan Abate, Felix Mulder and Martin Duhem, authors of two previous iterations of Scala3doc - Tastydoc and Dottydoc. Experience and ideas from those two projects have helped us in this latest attempt at a documentation tool

[AMatveev]

The documentaion says:

We also support -d <documentation> argument to provide static documentation. 

Would you explain what are the main advantages to use it instead of an external static site generator.

IIUC: You have implemented such feature themselves in this file:

[AleksanderBG]

Well, the advantage is that it’s a simple static site generator built into the tool. This means that it’s easy to generate project documentation to be displayed alongside API documentation, and that we can process project documentation similarly to how we already process API docs - for instance, we can resolve links to Scala definitions. Something like the Cat’s documentation is a good example - it doesn’t really take much to generate documentation that looks like that, and these docs could really benefit from being able to easily link to API docs.

[robstoll]

Some notes on dokka:

<< rant on

I had a very bad experience in the past. At least until 0.10.0 the maintenance of dokka was horrible. Lots of bugs were reported, on github and on slack… basically no reaction from the maintainers only after one or two months or so, some had even fixes in form of PRs… nothing happened. I fixed a bug, created a PR on Jun 22, 2018, first review was given on the same day, (perfect nothing to complain about, more than I would expect), fixed the necessary things on the same day - took until Mar 11, 2019 until it was merged into master, another two, three months until it was released.

I really hope this is better now because seriously, if it is still the same then Scala will not win anything by using dokka but a burden. Of course, scala3doc maintainers can always branch away and continue from there but then the question is, was it really the right fit? I can only speak for v.0.10.0 maybe the new version which soon (as they always say) will be released is better.
The performance of dokka is very bad, it outputs a ton of false positives (type Xy is unknown, things like that) and its memory consumption is mad; I had to give 8GB to dokka for a fairly small project in order that I was able to generate the documentation, otherwise it would run into a OutOfMemoryException.

rant off >>

I like that scala will get a Markdown supporting documentation tool and don’t get me wrong, it’s good to see that documentation tooling is revised

Some questions:

• will it also be possible to point to external documentations (externalDocumentationLink)?
• dokka suffers from the problem that one cannot distinguish between type parameters and parameters of a constructor (both are defined with @param), will scala3doc have the same flaw?
• will it support union types (as Kotlin does not have them)? => the answer is yes https://scala3doc.s3.eu-central-1.amazonaws.com/pr-master/self/main/tests/genericMethods/-types/index.html
• how about lower bounds?
• how about context functions?
• etc.
• I cannot find any documentation in the above link for methods, do you have an link at hand?

Some details about the generated documentation:

• Edit this page on GitHub points to https://scala3doc.s3.eu-central-1.amazonaws.com/edit/master//
• it is unusable on a mobile phone but I guess you (nor the dokka team) have focused on a responsive design yet.

[TheElectronWill]

I’ve never used dokka before, but I can testify that it’s no longer that buggy nor slow. I haven’t done a proper benchmark but scala3doc runs on itself and on Dotty fairly quickly (you can try it ).

Some answers to your questions:

• IIUC, linking to external documentation is on our TODO list (https://github.com/lampepfl/scala3doc/issues/68).
• We plan to support the documentation @tags of Scaladoc, so @tparam will be available for type parameters. We extend dokka to parse doc comments the way we want to, so that’s not a problem at all.
• As Aleksander said we plan to support all dotty definitions, with givens, bounds, and everything else. This is WIP.

Making the doc mobile-friendly will be part of my contribution to the front-end (WIP too). Stay tuned!

Regarding the bug of the “edit” button, I think that’s worth an issue

[nafg]

What were the issues with dottydoc?

[gabro]

It’s great to see this being worked on, thanks to all the people involved!

One question: the current scaladoc is very accessible from mobile (see for example https://typelevel.org/cats/api/cats/index.html) while the linked examples are barely usable on mobile (the layout is broken in several ways).

Is this a limitation of dokka? I understand it’s not the most compelling feature, but it would be nice not to break mobile accessibility

[romanowski]

dokka 0.10.x and current dokka has pretty much only a name in common. Whole tool was basically rewritten from scratch. In terms of support situation also changed since now there is a team of 5 in Kraków working on it and both Filip and me are faimilar with dokka internal as well.

Dokka 1.4 (that scala3doc is based on) is released and I suggest to try it out since it is much, much better then older ones.

[AleksanderBG]

@robstoll It’s a shame that you had a bad experience with Dokka in the past. The way Scala3doc is written, we mostly depend on Dokka as a “rendering” layer for our documentation. We intend to support all Scala 3 language features and all documentation features of Scaladoc, and we’re not really coupled to how Dokka interprets Kotlin documentation. If we see a need to decouple ourselves from Dokka in the future, it should be possible with a reasonable amount of effort. So far however, our experience with Dokka has been very positive - it has given us many features for free (such as: search, navigation, identifier linking).

@nafg the issue with Dottydoc was quite prosaic - we judged finishing it to be roughly the same amount of effort as reimplementing the tool and delegating large part of the work to Dokka.

[RichType]

@AleksanderBG

Scala3doc will also support adding project-level documentation in the form of Markdown files.

Will html files be supported? Its just personally I decided that, while it might be good for forums and Gitter channels, for project documentation Markdown didn’t really carry its weight. The complexity of another text format and the impedance of conversion to HTML wasn’t justified by the time-savings of the syntax. Complex HTML and CSS annotations can always be added programmatically later.

[romanowski]

Will html files be supported?

Index page of dotty documentation is a static html site and scala3doc is able to render. For now it is a special case however In future we will have support for both .html pages and .html layouts so documentation can be written in simple markdown and more addvance things added using HTML/Javascript/CSS .

Custom .css and .js files are supported and can be added to .md pages even now.

[eyalroth]

A big upvote for markdown in Scaladoc!

Will the tool be able to integrate with IntelliJ’s quick doc? I’m not familiar with Dokka, but a quick search suggests that it does integrate with quick doc.

It might be worth noting that there has been some recent activity regarding problems with Scaladoc rendering in IntelliJ quick doc (SCL-18007).

[AleksanderBG]

Will the tool be able to integrate with IntelliJ’s quick doc?

I think this one is up to IntelliJ’s team. For what it’s worth: if IntelliJ decides to rely on TASTy then I imagine they won’t have too much trouble just copying our code, which would make it easier for them to maintain feature parity with the doctool.

[rcano]

Is this going to become the scala 3 documentation tool without any community approval?

The official documentation tool is really important as it molds all the scala documentation that is going to be online for the next decade or so, and in that regard I have a million concerns. The minimum usability bar should be at scaladoc level, nothing less is acceptable; in that regard, dokka, current dottydoc, javadoc, they are all terrible.

Can there be a discussion/brainstorming on what this UI should look/be like?

[Katrix]

Have to agree that I’m pretty suspicious of the current design. How much of that is just a side effect of this being a prototype, and how much will stay for the final design?

If people are interested I can supply a list of things I’m missing/confused about in the current design.

[TheElectronWill]

To reassure @rcano: the goal is definitely to be as good as scaladoc, if not better! We’re not there yet, but after all we’ve not even released the first version of scala3doc. It will get better and better.

The current UI will get reworked/replaced. I’m interested in hearing what you don’t like and what you would like to see.

[AleksanderBG]

Yes, our goal is definitely to both an improvement and a continuation of Scaladoc, not a completely new tool. This is why our final milestone is to reach feature parity with Scaladoc. From our perspective, it’s useful to think of Dokka as nothing more than a (very well-made and useful) engine for rendering documentation.

Currently our output looks very close to Kotlin docs, but with @TheElectronWill’s help, this is going to change and become closer to how Scaladoc looks. If you think that there is some specific feature that we’re missing and which is important to include, we can always discuss it in a Github issue. Of course, you are even more welcome to contribute.

[rcano]

@TheElectronWill

I’m interested in hearing what you don’t like and what you would like to see.

I’m sure a lot more people in the community have a thing to say in this regard. Is this getting enough visibility? Some days ago I was about to make a post regarding how I didn’t find dottydoc to be ready at all for the imminent release and if whether someone was working on this or not, reading that work only started a month ago makes me want to suggest delaying scala 3 until next year.
The documentation tool is as important to the language as its collections api, and for the redesign of the later we went through a long redesign process with many prototypes. Rushing the doc tool in the last 4 months of the year is very worrisome.

I can definitely share what I’d want the tool to do but is this the place?

[AleksanderBG]

@rcano I’d appreciate it if you could open GH issue(s) for what you want to discuss, and (maybe) link them here. I’d prefer this topic to be more announcement-centered.

[benhutchison]

In Scala 3, Type Classes (TCs) finally get full recognition as a supported language feature [https://dotty.epfl.ch/docs/reference/contextual/type-classes.html].

In Type Class programming, many/most methods available to operate on a data item are not available because they were defined or inherited from the class/trait of the data. Rather they are extension methods associated with a type class instance.

Scaladoc has never catered for type class -originated methods. This makes documenting type class centric code more difficult since the Scaladocs have greatly reduced value. I end up using mdoc or tut tutorials, or reading the source code, or experimenting in Ammonite, as a necessary alternative. But normally I’d prefer just to read API docs if they were accurate.

It would be great to see this rectified in Scala 3 with the deeper embrace of TCs in the core language.

However, documenting methods available from TCs is ambiguous, since the set of available methods depends upon the imports in scope. A solution would be to allow the doc generator to generate API docs given a specified set of imports determined by the tool user (for example List("scala._", "cats._") ). Each TC extension method could indicate where it came from.