Scala3doc - doctool for Scala 3

If if understood correctly then it would be a very nice feature to have.

The data could be published by the libraries and downloaded by the doctool, which would only need to know about, for example, the library’s git repo url, or its name on Maven Central. :open_mouth: It could also provide a library search based on Scaladex.

@benhutchison I think even a simpler version where a project can specify imports based on which extension methods should be added to its classes would be useful; it’s also more within reach. It should be feasible to implement such a feature with a reasonable amount of effort, just based on TASTy APIs. This is also a good opportunity to get ahead-of-time experience with TASTy, which is going to be the macro API when Scala 3 rolls around. I created an issue for this: https://github.com/lampepfl/scala3doc/issues/151, contributions are welcome!

1 Like

This is exciting! Thank you for sharing this update.

Has it been discussed whether the markdown documentation in Scala3doc should support executable examples? Similar to https://github.com/tpolecat/tut and https://scalameta.org/mdoc/

```scala mdoc
val x = 1 + 3
// x: Int = 4
```

It would be nice if the compiler could support this feature out of the box instead of relying on 3rdparty tooling.

15 Likes

We’d like to support executable code blocks both in doc comments and in Markdown files, yes. There’s some nice possibilities - we could, for instance, hyperlink the types in code snippets (again, Cats docs would really benefit from this). There are also benefits from a tighter integration with the compiler - we could, for instance, make incremental compilation compile both the actual source files of a project and the executable blocks. Ideally, we’d also support extensions that alter what compiled code blocks render to - for instance, an extension could render them to interactive online REPLs. I see mdoc has a similar idea and supports extensions along these lines.

This is all in a very early stage, though - for now we really need to focus on Scaladoc parity.

2 Likes

Actually @olafurpg: if you’d like to, it’d help if you could open an issue on Github about adding support for executable blocks and describe your experiences with mdoc - what features were actually important in your opinion, what made the tool work well, etc. It’d provide a nice starting point for the discussion on how we should support executable (compiled?) code blocks in Scala3doc.

3 Likes

Possibly a stupid question but… why not use mdoc? Its basically a pre-processing step of Markdown files, no?

1 Like

I didn’t think of that! If mdoc outputs Markdown files then you’re right that we could stack the tools like that. I’m not sure if we’d be easily able to hyperlink types in the code snippets this way though, and I don’t think we’d be easily able to support compiling code blocks in doc comments.

One observation here - Dokka actually can output Markdown files as an alternative to HTML files. If one wanted to, one could output the entire documentation as Markdown files and afterwards run mdoc on those files, uniformly transforming code blocks in standalone Markdown files and in doc comments. Just food-for-thought.

Maybe it make more sense to run an mdoc-like tool om the doc comments before the documentation engine even runs. Orchestration could then be done by the build tool. That may make it easier to provide the user with location information for the errors without having to create some sort of sourcemap.

2 Likes

Just to make it clear: I think that ideally, mdoc-like functionality should be integrated into Scala3doc. This’d mean that we can provide additional features (such as hyperlinking types in code blocks) and we are guaranteed to integrate well, i.e. we expand code blocks in doc comments and Markdown files in exactly the same way.

If we won’t be able to do that (and given the time constraints, we probably won’t implement it on Scala 3 release), someone from the community will still be able to integrate the two tools together, either by running mdoc on MD files and feeding them as input to Scala3doc, or by outputting Markdown from Scala3doc and running mdoc on the result - whichever one finds more preferable.

2 Likes

I’m not very concerned with the use of a third party renderer for the doc personally.

But I am somewhat concerned in that the doctool consists of a significant part of kotlin code and requires orchestrating a gradle build of the kotlin code as an sbt task. I’m worried that once the tool is more or less complete and the original authors will have moved on, finding people willing and able to maintain a tool that leverages scala, sbt, kotlin and gradle is going to be hard.

Piecing it all together is also hard: I haven’t been able to find API documentation for dokka and its plugin infrastructure that’s being leveraged either. I’m sure you could just read the source, but that that source is not scala also presents an extra hurdle.

4 Likes

I agree that depending on a large Kotlin codebase is a problem for core Scala tooling.

What I can tell you is that:

  1. Plugin support in Dokka is still in early stages and because of that, there’s little documentation. However, as far as I know, there are plans to fix this.
  2. The gradle project is necessary because some Dokka classes are only extensible with Kotlin syntax. Again, there are plans (from people who are working on Scala3doc) to fix this by making said classes extensible from Java.
  3. In the absolute worst case, we should be able to rewrite Dokka to Scala. The codebase is written in a style which isn’t far off from Scala, so it should “just” be a mostly mechanical translation. It’s not going to be effortless, but the option is there.

In any case, I’d like to emphasize that we benefit a lot from integrating with Dokka. We do pay some costs, yes, but we benefit from ongoing maintenance of Dokka and from the work which already went into making it a good documentation rendering engine. Dokka already supports features such as alternative output formats (HTML, Javadoc HTML, Markdown), documenting Java sources, and support for multiplatform code. Adding those features to Dottydoc on our own would be very much a non-trivial task, but with Scala3doc depending on Dokka, it looks perfectly within reach for post-release improvements.

1 Like

If there are guarantees that the people who wrote it so far will keep it maintained up to the point where those things have changed, then my worries are relieved. Are there though?

I genuinely do not know what to tell you other than: yes, it’s in the interest of every person working on Scala3doc right now to have it succeed, which includes making it as easily maintainable as possible. In particular, it’s in the interest of Dokka maintainers (some of whom are working on Scala3doc) to have Scala3doc succeed - both from a personal perspective and from the perspective that sees Scala3doc as a success story for Dokka plugin architecture. Seeing as said maintainers have written the Gradle sub-project and told me that they’d want to integrate it into Dokka, I think that it will actually be integrated into Dokka.

5 Likes

Hope it can make use of mdoc and scalafiddle


https://scalafiddle.io

and I like the elixir style too
https://hexdocs.pm/elixir/String.html#content

1 Like

I have remembered that a Scala tool already exists for transforming Markdown to formatted documentation. I have used it in conjunction with MDoc. It is called Laika. Documentation can be found here.

My question is, wouldn’t using Laika solve most of the issues. I think it would then be a question of automatically generating Markdown files for the API and passing that through Laika. Maybe this could be the first phase in the pipeline. Additional text can be added manually (Markdown and/or HTML). And code snippets are taken care of in a second phase via MDoc.

Of course this means an external dependency, but its Scala.

3 Likes

On the topic of documenting Scala, I think Metabrowse is a tool with huge potential and is a bit under-appreciated at present.

It’s a source code browser based on scalameta that properly understands the structure of Scala code and supports IDE-style click-navigation. Demo showing the standard lib.

Due to the limitations of Scaladoc around typeclasses and implicits (which may well remain in Scaladoc3), I rarely find Scaladocs useful for functional scala (eg Cats and friends).

Rather I typically read sourcecode to figure out APIs. I imagine this is a common experience. I envisage I would get a lot value from making Scala sources easier to navigate.

Eventual github integration is nirvana, but source navigation via doc-sites would also be valuable, especially if various ecosystem sites cross-linked, which I believe metabrowse is capable of.

7 Likes

Hello everyone,

I think everyone’s due an update on the status of the doctool.

I’m happy to announce that we have actually already hit all three major milestones mentioned in the initial message!

That is, Scala3doc:

  1. was released together with Scala 3!
  2. has replaced Dottydoc as the engine for generating Scala 3 blog and documentation
  3. can document all Scala 3 definitions/types/modifiers, both old and new
  4. can document Scala 3’s standard library

If you want to try Scala3doc out, you can do that simply by adding useScala3doc := true to your SBT build, as long as you use at least Scala3 M1.

You can also see Scala3 website generated by Scala3doc on https://dotty.epfl.ch.

In addition, we also nearly on par with Scaladoc in terms of features! Which is to say that we:

  1. list implicitly-available definitions
  2. support Scaladoc-like member filtering (by name or by modifiers)
  3. render an inheritance graph for classes
  4. display known children, as well as class ancestors in initialization order
  5. have (nearly) Scaladoc-compatible definition linking

The only major Scaladoc features which we don’t yet have are @define variables (and we have a plan for those) and member grouping.

While there are still bugs to be fixed and visual improvements to be made, we will be able to release Scala3doc’s RC1 together with Scala3’s own RC1 with some time still to spare. This means that we’ll be able to spend some of our effort on planning what role exactly should Scala3doc plan in the wider Scala3 ecosystem. Look forward to a blog post on the matter!

6 Likes

Since we’ve announced Scala3doc many people have raised various concerns, asked questions and suggested ideas. We are incredibly grateful for it.

Scala3doc is much more mature than a few months ago and we are finally in a position to address raised points.

Dokka has a bad reputation, is slow and may not cover all Scala features

So far, we are able to document all features in Scala 3 using Scala3doc (that includes match types, givens and other features that don’t have any equivalents in Kotlin). In order to do that, we do not try to fit Scala constructs into the Kotlin model but rather create our own model. We still use some parts of the Kotlin model (function parameters, generics etc.), but we plan to migrate that away into our model as well (by properly implementing Signature and adding API there).

The latest releases of Dokka (1.4.x) are stable and well-received. Major Kotlin libraries are starting to adopt Dokka as well (e.g. kotlinx.serializaion ).

In terms of performance, it takes under 20 seconds to generate dotty.epfl.ch once code is compiled. It is also worth mentioning that we haven’t spent any time optimizing Scala3doc so far.

Dokka is written in Kotlin and it is outside our control

In the current version, we do not need any Kotlin code within Scala3doc. Previously static documentation was handled by dokka-site that was also written in Kotlin, but it was migrated into Scala3doc (and rewritten to Scala).

We still depend on Dokka and we do not plan to remove that dependency so far. However, we adopt a policy to isolate areas where we use APIs and constructs that are Dokka specific to have as little coupling as possible. In case something really bad happens on the Dokka side, these measures should make migration away from Dokka a much easier task.

Using existing tools in Scala3doc

Many great tools were suggested that Scala3doc can use (mdoc or Laika to name a few). We would love to have the freedom to use it within scala3doc. However, we are a part of the compiler ecosystem and we should avoid any Scala dependency. Depending on another Scala library is troublesome when releasing a new version of the language (to release Scala3doc for the new language version, we’d first need to migrate our dependency. Because of that, we were forced to give up all our Scala dependencies as well. This means that we needed to e.g. create a really trivial clone of scalatags.

We are aware that there is a need in the community to expand or consume documentation. We plan to introduce the ability to provide extensions for Scala3doc. However Scala3doc is not mature enough to provide any more details now.

We would love to hear from people interested in writing extensions what kind of extension point they would want to consume.

We are considering building in some features known from mdoc, scasite/scalafiddle, tut, Laika, metabrowse and other tools. Please let us know if there are other tools that we should take a look at.

Also, please let us know if there is an interest in consuming documentation as javadoc or .md files.

Documenting implicitly provided members

Scala3doc is currently able to document members implicitly provided by:

For now, we document mostly trivial cases, however, in some cases it may not be accurate. The current implementation is mostly a proof of concept, mainly to find a proper way to display such members from a UX/UI perspective.

We are actively exploring how we can provide full and precise information, including entries provided by dependencies as well. We hope that this mechanism combined with abilities similar to unidoc sbt plugin will make documentation more usable in the context of libraries like cats.

Documenting implicitly provided members is not a trivial problem so we can’t promise that we will be able to come up with a satisfying solution.

IDE integration

We are in touch with Scala Intellij Plugin and Metals developers to provide support for new syntax of documentation comments as well as we are exploring ways how Scala3doc can enhance the developer experience.

Review and community approval of the design

Initially, we decided to come out with our design of how documentation generated by Scala3doc looks and behaves. We wanted to create a usable version of the documentation that covers all features quickly. We also wanted to stabilize the way we create and render pages.

We are quite close to that goal and we have tooling in place that will allow us to quickly prototype new designs. We are really close to inviting everyone to review and discuss how future documentation for Scala 3 should look and behave (remaining issues are here).

8 Likes

Great work :clap:, looks promising.
I am still concerned about performance though. Someone tried to migrate Atrium (Kotlin library) during hacktoberfest to the new version and failed (see also https://kotlinlang.slack.com/archives/C0F4UNJET/p1603915431040000?thread_ts=1603915431.040000&cid=C0F4UNJET) if he included more than 27 submodules. Dokka always ran into a GC overhead (or similar). I didn’t not look into the issue myself but that sounds like the old dokka :pensive:

1 Like

I know that It took us quite a long to answer but a lot happened and I wrote 3 different versions of this reply but before each version was ready something changed and I needed to write it from the beginning.

So we’ve just refactored scala3doc so it does no longer depend on Dokka/Kotlin. In meantime, we’ve started to building docs for projects in community build and applies few simple optimizations in doctool itself (there is quite a lot work there still to minimize both time and size of generated docs).

Today it takes around 2 GB of RAM and around 60 seconds on my machine to generate documentation for scala 3 (all you can see here: https://dotty.epfl.ch/docs/index.html, compilation time is excluded). I think those are reasonable numbers at this stage of the project.

9 Likes