Coordinate compiler and tooling releases?

[julienrf]

@fthomas

I think it is easier to implement time delays for new PRs (Time-based delay configuration for new dependencies · Issue #1581 · scala-steward-org/scala-steward · GitHub) and set default delays for Scala, sbt, and Scala.js

That is indeed a more general and easier solution

[gabro]

Another idea, probably floated in the issue, is to add some metadata in the POM to decide the release strategy once the jar has been discovered:

• immediately (default)
• GitHub release (waits for a matching GitHub release which contains the version in the title or something like that)
• delayed

The configurations for delayed and GitHub release can live as metadata in the POM as well.

[lrytz]

@morgen-peschke

Does this principle still hold if the repository is explicitly a pre-release repo

Yeah, imagine a library maintainer has a jar in his cache (or in his CI cache) that’s different (maybe binary incompatible) with the actual release.

[fthomas]

@gabro

Another idea, probably floated in the issue, is to add some metadata in the POM to decide the release strategy once the jar has been discovered

I think this is the best option we have and I’m willing to modify Scala Steward in this direction. Would this also work for the Scala 2, Scala 3, sbt, and Scala.js maintainers?

[sjrd]

I somehow don’t think POM files are the right place for this information, as POMs are frozen forever, and we’re talking about information that is intrinsically about a transient process. That said, if that is what it takes for Scala Steward to delay its PRs, I will do it. Also I don’t really have a better idea.

---

Edit: something that would make a lot of sense to me would be to store a link to the public release notes URL. As long as that URL returns 404, delay the PRs. Once it returns 200, then go ahead. This idea was floated before as well. It applies to @gabro’s use case of the GitHub release as well, since you can predict the URL of the GitHub release page (e.g., Scala.js 1.8.0 will be https://github.com/scala-js/scala-js/releases/tag/v1.8.0). The release notes URL makes sense to me because it is also a frozen piece of information; at any point in the future, it is still relevant.

[SethTisue]

(confining my remarks to Scala 2, since that’s the part I’m responsible for)

For the steward part of this, I like the approach of delaying the creation of PRs until release notes are available. (Whereas POM-based stuff sounds dodgy.) We could easily modify the Scala 2 release process so that we don’t create the release (as opposed to merely a tag) on GitHub until full-announcement time. This would have a side benefit of lessening the probability of community folks jumping the gun on publicizing the release, since GitHub won’t notify anyone about the release until it’s done.

The other change I suggest we make to the Scala 2 release steps is to not announce until:

• at least 48 non-weekend hours have passed since the JARs hit Maven Central, and
• we have signoff from a Scala Center rep (Seb) and a VirtusLab rep (Krzysztof) that everything downstream is good to go.

(I can also notify the maintainers of popular plugins such as kind-projector and scoverage, but I don’t think we need to actually block on that.)

These changes are easy to make and together they would solve the problem, as far as I can see.

The only downside I see is that we lose the early-warning aspect when the PRs go out early, since those PRs might turn up problems. But I think that’s acceptable given that the Scala 2 community build is so robust.

[SethTisue]

P.S. the Scala 2 release steps are at https://github.com/scala/scala-dev/blob/scala-dev/.github/ISSUE_TEMPLATE/release.md and are PR-able

[fthomas]

@SethTisue

For the steward part of this, I like the approach of delaying the creation of PRs until release notes are available. (Whereas POM-based stuff sounds dodgy.) We could easily modify the Scala 2 release process so that we don’t create the release (as opposed to merely a tag) on GitHub until full-announcement time.

How would Scala Steward know that it should delay Scala 2 PRs until the GitHub release notes are created? It sounds like this would be a special rule for Scala 2 in Scala Steward. Having a URL of the release notes / announcement in the POM that can be checked for 404 / 200 would not require any special rules and could be used by other tools too.

[SethTisue]

I wasn’t imagining hardcoding anything for particular repos.

I was imagining adding a “wait for release notes” flag in the Steward config file that any repo could opt-in to. When enabled, then the creation of PRs would be gated by the availability of a release on GitHub (as opposed to just a tag).

[fthomas]

@SethTisue

I was imagining adding a “wait for release notes” flag in the Steward config file that any repo could opt-in to.

I’m confused. Would this config file be in the repo that receives updates or in the repo with the releases on which Scala Steward should wait?

I would also like to note that there is some guesswork involved when going from the POM to the release notes on GitHub. If a project does not use $version or v$version for tags, you have to call a GitHub API to find the release notes URL.

[som-snytt]

So basically Seth gets a thumb drive with the release, and he flies around the world, updating “where is Seth sleeping?” of course, hand-delivering the jars to privileged consumers, and then when he returns home, days or weeks later, he hits a button to confirm the release.

This is how Christmas works, where you’re not allowed to open any presents until Santa has returned to the north pole.

[SethTisue]

@fthomas

Would this config file be in the repo that receives updates or in the repo with the releases on which Scala Steward should wait?

Neither. I’m imagining introducing flags in repos/repos-github.md at main · scala-steward-org/repos · GitHub to configure the steward’s behavior when considering when to send out upgrades referencing particular repos.

So for example we’d change scala/scala to scala/scala: wait-for-release-notes or something like that, and then no Scala version upgrades would go out to any downstream repos until release notes were available.

Sorry, I should have been clearer about this to begin with.

I would also like to note that there is some guesswork involved when going from the POM to the release notes on GitHub. If a project does not use $version or v$version for tags, you have to call a GitHub API to find the release notes URL.

Yeah. But it seems reasonable to me to require that a repo that wants customized behavior to adhere to that standard, and the actual repos that we care most about (Scala 2, Scala 3, Scala.js, Scalameta, and so on) already do follow that standard.

[fthomas]

@SethTisue

So for example we’d change scala/scala to scala/scala: wait-for-release-notes or something like that, and then no Scala version upgrades would go out to any downstream repos until release notes were available.

Sounds good to me and I think it is basically the same idea as putting the URL of the release notes into the POM but instead of being in the POM it would be in a Scala Steward config file and instead of being an explicit URL it would be implicit.

[sirocchj]

Couldn’t the pattern of the release version be part of the configuration in Scala steward? So as per Seth’s proposal above one could provide there the pattern for the release note url (so steward wouldn’t need to do any guessing, I believe)?

[fthomas]

Good idea! The URL pattern could then be something like https://www.scala-lang.org/news/$version.

[sirocchj]

@julienrf

Hey @fthomas,
First, let me say that I believe Scala Steward is one of the most helpful tools in the Scala ecosystem, so thank you so much for that!

By the way - I wholeheartedly agree! I’m an avid user of Scala Steward and I am immensely grateful to you and all contributors for it (alongside all your other contributions and libraries like refined)

[SethTisue]

For the record, I don’t think the POM idea is fatally flawed. But I should be more specific about why I’m also interested in discussing other options:

• As a library maintainer, I can’t change my POM without publishing a new version
• Publishing to Sonatype is a notoriously tricky and fragile process, and Sonatype is notoriously picky about the contents of POMs
• It would be cool if there were a cultural norm in Maven land that POMs link to release notes, and if the Maven powers-that-be propagated some kind of standard for how to do it. But in practice, we’re not going to succeed in establishing that as a norm. In practice, only a relative handful of repos are going to actually do this, and the only reason they’re going to do it is to tell the steward what to do. It seems like unnecessary levels of indirection to me when I we could just be telling the steward directly.

[fthomas]

@SethTisue

We could easily modify the Scala 2 release process so that we don’t create the release (as opposed to merely a tag) on GitHub until full-announcement time.

Just checked if this would actually work. I pushed two tags to Tags · scala-steward-org/test-repo-2 · GitHub and created a release for v0.1 but not for v0.2. Unfortunately this does not make a difference for the status code of the release notes URLs:

$ curl --head -s -o /dev/null -w "%{http_code}\n" https://github.com/scala-steward-org/test-repo-2/releases/v0.1
200
$ curl --head -s -o /dev/null -w "%{http_code}\n" https://github.com/scala-steward-org/test-repo-2/releases/v0.2
200

So just tagging and not creating the release on GitHub does not make a difference with regards to the status code. It is only 404 if the tag does not exist:

$ curl --head -s -o /dev/null -w "%{http_code}\n" https://github.com/scala-steward-org/test-repo-2/releases/v0.3
404

[gabro]

You can probably use the GitHub API v3 (non graphql) for that

[fthomas]

@fthomas

@sjrd

Well, at least back then I wasn’t pressured into releasing Scala.js within an hour of Scala compiler artifacts being on Maven Central. And that, irrespective of the day of the week, or the hour of the day or night.

I feel bad about that Scala Steward has this effect. Its purpose is to ease the maintenance burden and not to increase it. I’m sorry.

Btw, I just configured my public Scala Steward instance to ignore all future Scala 2.x updates until it is able to wait for release announcements. @sjrd I hope this helps reducing the pressure to release matching Scala.js artifacts.