Snippet validation in Scaladoc for Scala 3

Snippet validation in Scaladoc

This feature is still under development and it does not target 3.0.0 but rather 3.0.1 or later.

The main functionality of project documentation is to help people understand and use the project properly. Sometimes a part of a project needs few words to show its usage, but every developer knows that there are moments where description is not enough and nothing is better than a good ol’ example. The convenient way of providing examples in documentation is to create code snippets presenting usage of given functionality. The problem with code snippets is that simultaneously with project development, they need to be updated. Sometimes changes in one part of a project may break examples in other parts. The number of snippets and the amount of time passed since they’ve been written makes it impossible to remember every place where you need to fix them. After some time you realize that your documentation is a complete mess and you need to go through all examples and rewrite them. Many Scala 2 projects use typechecked markdown documentation with tut or mdoc. Almost everyone at least heard about these tools. As they turned out to be very useful and the Scala community successfully adopted them, we’re planning to incorporate the features of tut and mdoc into the compiler so that it’s included out of the box with Scaladoc.

Context

Our goal is to make snippets behave as much as possible as it is defined within a given member. For us it brings a natural feel to snippets. To achieve this, we implemented a wrapping mechanism that provides a context for each snippet.

Let’s assume that for some method foo inside class Bar in package baz, we’ve got a snippet:

2+2

Result of wrapping a snippet using our mechanism looks as follows:

package baz
trait Snippet { self: Bar =>
2+2
}

Besides our main goal, it reduces the boilerplate of a snippet, because you don’t need to import members of the same package and instantiate documented class.

As there might be a situation when your snippet doesn’t work as expected, we’ve exposed an extra flag that prints a wrapped snippet to console in order to debug it.

Hiding code

Sometimes it’s inevitable to import some symbols from different parts of projects or prepare enviroment for a snippets that we want to show to the reader. A big block of imports and initializations of necessary classes can result in loss of reablity. On the other hand, we’ve read a lot of opinions that people would like to be able to see the whole code. For that case we introduced special syntax for snippets that hides certain fragments of code, but you can always expand the snippet in documentation.

Example:

// {{
import scala.collections.immutable.List
// }}
val intList: List[Int] = List(1, 2, 3)

Assert compilation errors

Scala with its rich type system allows us to ensure almost complete correctness of code in compile-time. Creators use this mechanism to detect mistakes at compile-time instead of runtime.

E.g.
```scala sc:compile
val iAmWrong: Int = 2.3
for(i ← 1.to(10))
yield i*2
```

pasted image 0

Examples presenting code that fails at compile-time are sometimes very important. For example you can show how the library is secured against incorrect code. The other use case is to present common mistakes and how to solve them. Taking that into account, we decided to provide a functionality in our snippet validation tool to make it ensure that marked code snippets don’t compile.

For snippet that compiles without error:

```scala sc:failing
val iAmGood: Int = 2
for(i ← 1.to(10))
yield i*2
```
pasted image 0 (1)

For snippet that fails to compile:

```scala sc:failing
val iAmWrong: Int = 2.3
for(i ← 1.to(10))
yield i*2
```
Passes the snippet validation and shows expected compilation errors in documentation.

We also have an idea to expand this feature to not only let the user mark a snippet as failing to compile, but also to let the user point out the place in code and error with which the snippet should fail.

Example:

```scala sc:failing
doesNotExist()
// ^ Not found
```

For this example, the compiler produces an error: Not found: doesNotExist at line 1 and column 0, which satisfies the provided condition.

How would you name a flag that marks a snippet which should not compile?

Configuration

Snippet compiler affects only scala snippets (it recognizes them by ```scala part). By default snippet validation is turned off for all snippets. Snippet validation configuration is very straightforward and is similar to configuration of other options in Scaladoc. There are two ways to provide configuration: using CLI arguments or by specifying a flag directly in the snippet.

CLI arguments:

  1. -snippet-compiler:arg(,arg)+
    arg := (path=)?flag

Where:

path - optional parameter that defines prefix of paths of source files (relative to sources root) in which snippets should be checked. Argument without path defines default behavior. By setting path parameter, it’s possible to disable the snippet compiler on a per-directory basis.

flag - one of available flags:

  • compile - turns on snippet checking
  • nocompile - turns off snippet checking
  • failing - turns on snippet checking, snippets should not compile
  1. snippet-compiler-debug - Setting this option causes all compiled snippets in documentation to be replaced with their wrapped version (to be able to debug problems)

Directly in the snippet:

```scala sc:flag
//snippet
```

CLI flags are overridden by flags provided directly in the snippet.

And that’s all!

Edit: We edited the section about mdoc and tut, since it was not sounding like we properly recognized these great tools!

23 Likes

One more good feature would be not only compiling, but actually running each snippet as a test in a test suite for documented module, like rustdoc does.

This way snippets can guarantee not only compiling but working examples and will increase code test coverage:

```scala sc:test
assert(2 + 2 == 4)
```
8 Likes

I’d like to echo this! The aforementioned mdoc does already allow for this (for them it is actually the default I believe) and at least personally I would perceive not having this functionality as somewhat of a regression.

1 Like

This looks good! Here’s one idea: the Scala extension for VSCode could (maybe?) add code lenses to snippets in .md files that would allow checking whether the snippet compiles, and previewing the “expanded” version of the snippet.

That seems like a great idea! We should be able to do that already in Metals with mdoc and maybe extend to Scala 3 doc? Though, I don’t think we would be able to work on it soon.

Hey, thank you for the feedback! The first version of snippet validation will be focused on validation and reporting described above. In future, we will consider implementing snippet evaluation/tests.

Could it be used with markdown files as well? For example those used by the scaladoc static site feature

Yes, that also works

2 Likes