Release Manager Runbook
Release process overview
Here’s a quick review of what the release process looks like from the community perspective:
- This is the expected release cadence .
- The release calendar is awesome. It gives you an agenda with the expected duties.
- Here’s what the project expects all contributors to do for backports/patches .
If the builds break, you can take a look at some common issues to see if we’ve encountered them before.
Verify you have access
If you don’t have access to any of the following, contact a member of the TOC or SC.
- You’re a member of the release-managers@spinnaker.io group and a manager of the spinnaker-announce@googlegroups.com group. (You’ll get a permissions error on those pages if you don’t have access.
- You’re a member of the release-managers GitHub team .
- You’re able to view our GCP spinnaker-community cloudbuilds . You should see a lot of builds.
One week before the branches are cut (Monday)
Ping #dev reminding everyone to merge outstanding changes by Monday:
The release manager will be cutting the $VERSION release branches next Tuesday, so if there are any outstanding PRs that you’d like to get into $VERSION, please make sure they are merged by EOD next Monday. Once the branch is cut, only fixes will be accepted into the release branches.
The day the branches are cut (Tuesday)
If there are any outstanding autobump PRs , make the required fixes to allow them to merge. (You can ignore
keel
andswabbie
; those repositories aren’t part of a Spinnaker release.)Tag repositories with their respective next semVer minor version:
- Tag
HEAD
ofmaster
inkork
if required, merge autobump PRs. - Tag
HEAD
ofmaster
infiat
withv{major}.{minor}.0
, merge autobump PRs. - Tag
HEAD
ofmaster
inorca
withv{major}.{minor}.0
, merge autobump PR inkayenta
. - Tag
HEAD
ofmaster
in remaining repos with{major}.{minor}.0
. - Don’t tag
spin
repository. We’ll tag after branch cut.
The BOM will become these tags.
- Tag
Create the release branches by running <code>buildtool new_release_branch</code>
- Create the
spin
release branch at
HEAD of
master
. Later we will tag our new branch with the Spinnaker Release version which creates our artifacts.
- Create the
spin
release branch at
HEAD of
Ping #dev with some version of this message.
The release branches for Spinnaker $VERSION have been cut from master! Those branches are only accepting fixes for existing features. Please post in this channel and tag me @$YOUR_NAME if you would like a fix cherry-picked into the release. If you would like to highlight a specific fix or feature in the release’s changelog, please make a pull request against the curated changelog by Friday.
One week after branches are cut (Monday)
Audit backport candidates .
FIXME: Run buildtool integration tests in Google Cloud Build to validate artifacts
TODO: We may need a BOM or at least a list of artifacts to run integration tests with.
Iterate until integration tests pass; fixing issues, tagging repositories and running the integration tests.
Build the BOM by running buildtool build_bom
Build the Changelog by running buildtool build_changelog
Push the Changelog to a gist by running buildtool push_changelog_to_gist
Raise a PR for Changelog at spinnaker.io by running buildtool publish_changelog
Add tag’s to both
slim
andubuntu
containers - see buildtool’s release_helper.shat
{tag}
WITHOUTunvalidated
, For example: tagorca:1.2.3-unvalidated
withorca:1.2.3
. Halyard expects container tags and debian packages to match the BOM.at
release-{major}-{minor}-{patch}
- friendly tag for use by Spinnaker users Kubernetes and other manifests. For example:spinnaker-1.27.0
Upload the BOM file as
1.{minor}.{patch}
to the GCS - https://console.cloud.google.com/storage/browser/halconfig/bomUpdate versions.yml in place with the new version.
version is
1.{minor}.{patch}
alias is
v1.{minor}.{patch}
changelog is the URL to the gist you created earlier
minimumHalyardVersion should remain unchanged unless you know of a reason to change it
lastUpdate is the current Linux epoch plus
000
due to use of millisecond resolution for timestamps in Halyard. For example:$ date +%s 1651632868 # concatenate above with '000' becomes: lastUpdate=1651632868000
Deprecate the n-3 release (i.e. when releasing 1.18, deprecate 1.15).
- Make a PR against the deprecated changelog
here
,
adding
deprecated
to the list of tags.
- Make a PR against the deprecated changelog
here
,
adding
Ping the #spinnaker-releases channel to let them know that a new release is available.
> Hot Tip! You can use giphy to tell everyone it's released! > > `/giphy #caption "Spinnaker {VERSION} has been released!" gif search query`
Publish a Spin CLI minor version.
Each Spin CLI release is tied to a version of Gate. To ensure compatibility, regenerate the Gate Client API.
Follow the instructions to update the gate client.
Ensure that the GitHub Action’s for the above PR merge were successful and then push a git tag for the release version
1.{minor}.{patch}
to thespin
repository. This will kick off binary and container build & push to GCS and GAR.
Every subsequent Monday: Patch a previous Spinnaker version
Repeat weeklyish for each supported version.
Audit backport candidates . To view what’s been merged into each release branch since the last release, see the changelog gist on Github.
FIXME: Rerun the
Flow_BuildAndValidate_${RELEASE}
job and get a blue build.FIXME: Run Publish_SpinnakerPatchRelease:
Enter the major and minor version of the release you’re patching (ex: 1.18) in MAJOR_MINOR_VERSION.
All other fields can be left as defaults/blank.
This looks for a currently active release with this major and minor version. It copies all parameters from that release (name, changelog gist, minimum Halyard version), increments the patch version, and triggers Publish_SpinnakerRelease with these parameters. In general, this is exactly the behavior we want, but if you need to override this behavior (such as to increment the minimum Halyard version in a patch release), you can call Publish_SpinnakerRelease directly and pass the exact parameters that you’d like the new release to have.
After the job has completed, run
hal version list
and verify that the version you just released is listed, and the prior patch release for the minor version is no longer listed.Go to to Versions page and verify the following (leaving time for the site to rebuild):
Verify the version you just released is listed.
Verify the prior patch release for the minor version has been moved to the “Deprecated Versions” section.
Verify the changelog for the new version looks correct. It should start with the changelog for the specific patch release, then list the changelog for each patch release of the minor version in reverse order.
Ping the #spinnaker-releases channel to let them know that the new patch is available.
> Hot Tip! You can use giphy to tell everyone it's released! > > `/giphy #caption "Spinnaker {VERSION} has been released!" gif search query`
Approve the spinnaker-announce email (link will come in email). You can approve the message in the spinnaker-announce group .
Release minor-version Halyard
Repeat as needed.
Check for outstanding PRs.
Tag
master
with next{minor}
tag. For Examplev1.45.0
next minor isv1.46.0
. This will result in a new debian package at1.46.0
and a new containers tagged1.46.0-unvalidated(-ubuntu)
.Create
release-{major}-{minor}-x
branch at the new tag. This enables backports and{patch}
releases to be made.TODO: Any validation?
Re-tag Halyard container without
-unvalidated
. halyard repositoryPost in #halyard that a new version of Halyard has been released.
Hot Tip! You can use giphy to tell everyone it’s released!
/giphy #caption "Halyard {VERSION} has been released!" gif search query
Release patch-version Halyard
Repeat as needed.
Ensure you have audited all Halyard backport candidates .
Tag
release-{major}-{minor}-x
branch with next{patch}
tag. For examplev1.46.0
next tag isv1.46.1
Post in #halyard that a new version of Halyard has been released.
Hot Tip! You can use giphy to tell everyone it’s released!
/giphy #caption "Halyard {VERSION} has been released!" gif search query
Publish a new version of deck-kayenta
Repeat as needed.
Follow the instructions in deck-kayenta’s README .
Audit backport candidates
Repeat weekly.
Audit each PR that has been labelled a backport candidate .
If a candidate meets the [release branch patch criteria]({{> ref “releasing#release-branch-patch-criteria” >}}):
1. Remove the `backport-candidate` label from the PR. 1. Determine which versions the PR needs to be backported to. If it gets backported to an older version, all new versions should get the backport as well. Go only as far back as the supported [stable versions](/docs/releases/versions/#latest-stable). 1. Add a comment instructing [Mergify](https://doc.mergify.io/commands.html#backport) to create backport PRs against one or more release branches. For example, to create backport PRs against the 1.19, 1.20 and 1.21 release branches, comment: > @Mergifyio backport release-1.19.x release-1.20.x release-1.21.x 1. Approve and merge the backport PRs. 1. If Mergify cannot create a backport because there are merge conflicts, ask the contributor to open a PR against the target release branches with their commits manually [cherry-picked](https://git-scm.com/docs/git-cherry-pick).
If a candidate does not meet the release branch patch criteria , add an explanation to the contributor as a comment.
1. If it's impossible for the candidate to meet the criteria (for example, it doesn't fix a regression), remove the `backport-candidate` label. 1. If the contributor can amend the candidate to meet the criteria (for example, add test coverage), don't remove the `backport-candidate` label.