Release Manager Runbook

Release process overview

Here’s a quick review of what the release process looks like from the community perspective:

This is the expected release cadence .
The release calendar is awesome. It gives you an agenda with the expected duties.
Here’s what the project expects all contributors to do for backports/patches .

If the builds break, you can take a look at some common issues to see if we’ve encountered them before.

Verify you have access

If you don’t have access to any of the following, contact a member of the TOC or SC.

You’re a member of the release-managers@spinnaker.io group and a manager of the spinnaker-announce@googlegroups.com group. (You’ll get a permissions error on those pages if you don’t have access.
You’re a member of the release-managers GitHub team .
You’re able to view our GCP spinnaker-community cloudbuilds . You should see a lot of builds.

One week before the branches are cut (Monday)

Ping #dev reminding everyone to merge outstanding changes by Monday:

The release manager will be cutting the $VERSION release branches next Tuesday, so if there are any outstanding PRs that you’d like to get into $VERSION, please make sure they are merged by EOD next Monday. Once the branch is cut, only fixes will be accepted into the release branches.

The day the branches are cut (Tuesday)

If there are any outstanding autobump PRs , make the required fixes to allow them to merge. (You can ignore keel and swabbie; those repositories aren’t part of a Spinnaker release.)
Tag repositories with their respective next semVer minor version:
1. Tag HEAD of master in kork if required, merge autobump PRs.
2. Tag HEAD of master in fiat with v{major}.{minor}.0, merge autobump PRs.
3. Tag HEAD of master in orca with v{major}.{minor}.0, merge autobump PR in kayenta.
4. Tag HEAD of master in remaining repos with {major}.{minor}.0.
5. Don’t tag spin repository. We’ll tag after branch cut.
The BOM will become these tags.
Create the release branches by running <code>buildtool new_release_branch</code>
1. Create the spin release branch at HEAD of master. Later we will tag our new branch with the Spinnaker Release version which creates our artifacts.
Ping #dev with some version of this message.
The release branches for Spinnaker $VERSION have been cut from master! Those branches are only accepting fixes for existing features. Please post in this channel and tag me @$YOUR_NAME if you would like a fix cherry-picked into the release. If you would like to highlight a specific fix or feature in the release’s changelog, please make a pull request against the curated changelog by Friday.

One week after branches are cut (Monday)

Audit backport candidates .
FIXME: Run buildtool integration tests in Google Cloud Build to validate artifacts
TODO: We may need a BOM or at least a list of artifacts to run integration tests with.
Iterate until integration tests pass; fixing issues, tagging repositories and running the integration tests.
Build the BOM by running buildtool build_bom
Build the Changelog by running buildtool build_changelog
Push the Changelog to a gist by running buildtool push_changelog_to_gist
Raise a PR for Changelog at spinnaker.io by running buildtool publish_changelog
Add tag’s to both slim and ubuntu containers - see buildtool’s release_helper.sh
1. at {tag} WITHOUT unvalidated, For example: tag orca:1.2.3-unvalidated with orca:1.2.3. Halyard expects container tags and debian packages to match the BOM.
2. at release-{major}-{minor}-{patch} - friendly tag for use by Spinnaker users Kubernetes and other manifests. For example: spinnaker-1.27.0
Upload the BOM file as 1.{minor}.{patch} to the GCS - https://console.cloud.google.com/storage/browser/halconfig/bom
Update versions.yml in place with the new version.
1. version is 1.{minor}.{patch}
2. alias is v1.{minor}.{patch}
3. changelog is the URL to the gist you created earlier
4. minimumHalyardVersion should remain unchanged unless you know of a reason to change it
5. lastUpdate is the current Linux epoch plus 000 due to use of millisecond resolution for timestamps in Halyard. For example:
```
$ date +%s
1651632868

# concatenate above with '000' becomes:
lastUpdate=1651632868000
```
Deprecate the n-3 release (i.e. when releasing 1.18, deprecate 1.15).
1. Make a PR against the deprecated changelog here , adding deprecated to the list of tags.

Ping the #spinnaker-releases channel to let them know that a new release is available.

> Hot Tip! You can use giphy to tell everyone it's released!
>
> `/giphy #caption "Spinnaker {VERSION} has been released!" gif search query`

Publish a Spin CLI minor version.
1. Each Spin CLI release is tied to a version of Gate. To ensure compatibility, regenerate the Gate Client API.
2. Follow the instructions to update the gate client.
3. Ensure that the GitHub Action’s for the above PR merge were successful and then push a git tag for the release version 1.{minor}.{patch} to the spin repository. This will kick off binary and container build & push to GCS and GAR.

Every subsequent Monday: Patch a previous Spinnaker version

Repeat weeklyish for each supported version.

Audit backport candidates . To view what’s been merged into each release branch since the last release, see the changelog gist on Github.
FIXME: Rerun the Flow_BuildAndValidate_${RELEASE} job and get a blue build.
FIXME: Run Publish_SpinnakerPatchRelease:
1. Enter the major and minor version of the release you’re patching (ex: 1.18) in MAJOR_MINOR_VERSION.
2. All other fields can be left as defaults/blank.
This looks for a currently active release with this major and minor version. It copies all parameters from that release (name, changelog gist, minimum Halyard version), increments the patch version, and triggers Publish_SpinnakerRelease with these parameters. In general, this is exactly the behavior we want, but if you need to override this behavior (such as to increment the minimum Halyard version in a patch release), you can call Publish_SpinnakerRelease directly and pass the exact parameters that you’d like the new release to have.
After the job has completed, run hal version list and verify that the version you just released is listed, and the prior patch release for the minor version is no longer listed.
Go to to Versions page and verify the following (leaving time for the site to rebuild):
1. Verify the version you just released is listed.
2. Verify the prior patch release for the minor version has been moved to the “Deprecated Versions” section.
3. Verify the changelog for the new version looks correct. It should start with the changelog for the specific patch release, then list the changelog for each patch release of the minor version in reverse order.

Ping the #spinnaker-releases channel to let them know that the new patch is available.

> Hot Tip! You can use giphy to tell everyone it's released!
>
> `/giphy #caption "Spinnaker {VERSION} has been released!" gif search query`

Approve the spinnaker-announce email (link will come in email). You can approve the message in the spinnaker-announce group .

Release minor-version Halyard

Repeat as needed.

Check for outstanding PRs.
Tag master with next {minor} tag. For Example v1.45.0 next minor is v1.46.0. This will result in a new debian package at 1.46.0 and a new containers tagged 1.46.0-unvalidated(-ubuntu).
Create release-{major}-{minor}-x branch at the new tag. This enables backports and {patch} releases to be made.
TODO: Any validation?
Re-tag Halyard container without -unvalidated. halyard repository
Post in #halyard that a new version of Halyard has been released.
Hot Tip! You can use giphy to tell everyone it’s released!
/giphy #caption "Halyard {VERSION} has been released!" gif search query

Release patch-version Halyard

Repeat as needed.

Ensure you have audited all Halyard backport candidates .
Tag release-{major}-{minor}-x branch with next {patch} tag. For example v1.46.0 next tag is v1.46.1
Post in #halyard that a new version of Halyard has been released.
Hot Tip! You can use giphy to tell everyone it’s released!
/giphy #caption "Halyard {VERSION} has been released!" gif search query

Publish a new version of deck-kayenta

Repeat as needed.

Follow the instructions in deck-kayenta’s README .

Audit backport candidates

Repeat weekly.

Audit each PR that has been labelled a backport candidate .

If a candidate meets the [release branch patch criteria]({{> ref “releasing#release-branch-patch-criteria” >}}):

1. Remove the `backport-candidate` label from the PR.

1. Determine which versions the PR needs to be backported to. If it gets backported to an older version, all new versions should get the backport as well. Go only as far back as the supported [stable versions](/docs/releases/versions/#latest-stable).

1. Add a comment instructing
   [Mergify](https://doc.mergify.io/commands.html#backport) to create
   backport PRs against one or more release branches. For example, to
   create backport PRs against the 1.19, 1.20 and 1.21 release branches, comment:

   > @Mergifyio backport release-1.19.x release-1.20.x release-1.21.x

1. Approve and merge the backport PRs.

1. If Mergify cannot create a backport because there are merge conflicts,
   ask the contributor to open a PR against the target release branches with
   their commits manually
   [cherry-picked](https://git-scm.com/docs/git-cherry-pick).

If a candidate does not meet the release branch patch criteria , add an explanation to the contributor as a comment.

1. If it's impossible for the candidate to meet the criteria (for example, it doesn't
   fix a regression), remove the `backport-candidate` label.

1. If the contributor can amend the candidate to meet the criteria (for example,
   add test coverage), don't remove the `backport-candidate` label.

Last modified May 5, 2022: docs(runbook): Update Release Manager Runbook (#193) (9e9aaa8)