Spinnaker – Kubernetes

Docs: Annotation-Driven UI

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to surface information about your Kubernetes resources in Deck’s details panel using Kubernetes annotations. These annotations can be text or HTML and can include templated values that are populated when the annotation is rendered by the browser.

Example Usage

Here’s a quick example of usage that will add a “Pod Info” section to the details panel for a pod:

kubectl annotate pod my-prod-pod-v000 \
  pod-info.details.html.spinnaker.io="<a href='https://internal-elk.net/{{"{{ name "}}}}'>Internal Logs Service</a>"

Here’s how this annotation will render in Spinnaker’s UI:

Dissecting the annotation, here’s how the UI is constructed:

The section title, “Pod Info” comes from the annotation key: pod-info.details.html.spinnaker.io. Notice that hyphens are replaced with spaces and the section title is rendered using Title Case.
The entry is to be rendered as HTML. This also comes from the annotation’s key: pod-info.details.html.spinnaker.io. Excluding “html” here would have rendered the link as plain text.
The HTML content is taken from the annotation’s value: <a href='https://internal-elk.net/{{"{{ name "}}}}'>Internal Logs Service</a>
The pod’s name will be interpolated into the link. Notice the {{"{{ name "}}}} template value in the href attribute. The full set of available values are listed at the end of this document.

Rendering Text Annotations

To render the annotation as plain text, use an annotation key following this pattern:

(section-title).details.spinnaker.io/(key-name)

The key-name portion can be included or omitted. If included the text will be rendered as a key/value pair with key-name in bold and hyphens replaced with spaces.

Rendering HTML

To render the annotation as HTML, use an annotation key following this pattern:

(section-title).details.html.spinnaker.io/(key-name)

key-name will not be rendered but is available to allow multiple HTML entries under a single section title.

Using Templates

Template values can be included in the content of the annotation and will be replaced when they are rendered by Deck. A templated value has the following appearance in an annotation: {{ "{{ templateKey "}} }} where templateKey will vary depending on the available set of keys for the resource that is annotated. The complete set of available keys is documented below.

Instances

account - the spinnaker account for this resource
apiVersion - the kubernetes apiVersion of this resource
cloudProvider - this will always be kubernetes
displayName - the name of the resource prepared for UI display
hasHealthStatus - a boolean indicating whether the instance has health status
healthState - the instance’s health status, if available
id - the instance’s id
kind - the kubernetes kind of this resource
manifest - the kubernetes manifest as JSON object
name - the resource’s name
namespace - the kubernetes namespace in which this resource resides

Load Balancers

account - the spinnaker account for this resource
apiVersion - the kubernetes apiVersion of this resource
cloudProvider - this will always be kubernetes
detail - the spinnaker detail, if any, for this resource
displayName - the name of the resource prepared for UI display
kind - the kubernetes kind of this resource
manifest - the kubernetes manifest as JSON object
name - the resource’s name
namespace - the kubernetes namespace in which this resource resides
stack - the spinnaker stack, if any, for this resource
type - this resource’s spinnaker type

Security Groups

account - the spinnaker account for this resource
apiVersion - the kubernetes apiVersion of this resource
application - the spinnaker application name, if any, for this resource
cloudProvider - this will always be kubernetes
detail - the spinnaker detail, if any, for this resource
displayName - the name of the resource prepared for UI display
id - the security group’s id
kind - the kubernetes kind of this resource
manifest - the kubernetes manifest as JSON object
name - the resource’s name
namespace - the kubernetes namespace in which this resource resides
stack - the spinnaker stack, if any, for this resource
type - this resource’s spinnaker type

Server Groups

account - the spinnaker account for this resource
apiVersion - the kubernetes apiVersion of this resource
app - the spinnaker application name, if any, for this resource
category - the spinnaker category, if any, for this resource
cloudProvider - this will always be kubernetes
cluster - the spinnaker cluster for this resource
createdTime - the time this resource was created, if available
detail - the spinnaker detail, if any, for this resource
disabled - a boolean that is true if this server group is disabled
disabledDate - a number representing the date this server group was disabled
displayName - the name of the resource prepared for UI display
kind - the kubernetes kind of this resource
manifest - the kubernetes manifest as JSON object
name - the resource’s name
namespace - the kubernetes namespace in which this resource resides
region - the region this server group is in
stack - the spinnaker stack, if any, for this resource
type - this resource’s spinnaker type

Server Group Managers

account - the spinnaker account for this resource
apiVersion - the kubernetes apiVersion of this resource
cloudProvider - this will always be kubernetes
displayName - the name of the resource prepared for UI display
kind - the kubernetes kind of this resource
manifest - the kubernetes manifest as JSON object
name - the resource’s name
namespace - the kubernetes namespace in which this resource resides

Docs: Configure Automated Rollbacks in the Kubernetes Provider

Mon, 01 Jan 0001 00:00:00 +0000

There are clear benefits to having your stored manifests (in Git, GCS, etc…) match as closely as possible to what’s running in your cluster: you have auditability, versioned changes, and a high confidence in what code you have running. However, in some cases, pushing changes from someplace like GitHub through your build system all the way to production can simply take too long, and an escape hatch is needed when something has gone seriously wrong. Automated rollback is that escape hatch.

When using a Deployment object, you can see the history of rollouts in the “Clusters” tab:

Version 4 is active, and version 3 has no pods running.

Spinnaker exposes “Undo Rollout” functionality in two places, in the Clusters tab , and as a pipeline stage .

Ad-hoc rollbacks

In cases where you see something is immediately wrong and isolated to a resource in the “Clusters” tab, you can select “Undo Rollout” from the “Actions” dropdown:

And select the healthy revision to make active again:

Notice that the old configuration (version 3) will be rolled forward into a new version (version 5):

Automated rollbacks

You can also configure automated rollbacks inside of Spinnaker pipelines. These stages are best configured to run when other stages or branches fail, indicating that something has gone wrong in your rollout.

One parameter to watch out for is Revisions Back, which counts how many revisions the current active revision should be rolled back by. If you have the following state:

nginx-deployment-2d8178b77 (Revision 5) # active
nginx-deployment-7bdd110f7 (Revision 4)
nginx-deployment-0b13cc8c1 (Revision 1)

And roll back by “1” revision, (Revision 4) will be active again. Roll back by “2” revisions and (Revision 1) will be active again.

Keep in mind that Kubernetes will implicitly rollforward the old configuration, creating (Revision 6) in both cases.

Parameterized rollbacks

It’s worth mentioning that you can parameterize the target resource to roll back. It can point to something specified using pipeline parameters, and upstream deploy stage, or another stage’s outputs. See more details in the pipeline expressions guide .

Pitfalls

If the artifacts deployed in your manifest (Docker image, ConfigMap, Secret, …) are not versioned, rolling back your manifest will likely not roll back your code or config changes. See more details here .

Docs: Best Practices for the Kubernetes Provider

Mon, 01 Jan 0001 00:00:00 +0000

The Kubernetes provider enables a wide variety of ways to deploy your manifests into Kubernetes clusters. This page provides best-practices for doing so.

Deploy Docker images by digest

If your registry exposes image changes by digest , we recommend deploying images by their digest rather than tag, e.g. gcr.io/my-image@sha256:95ff090... rather than gcr.io/my-image:v1.

The digest is a content-addressable way to reference your image, because it’s derived from the hash of the image contents. If your manifest points to an image by its digest, you know that each time it’s deployed it points to the same contents (binary, dependencies, config, etc…). If you rely on the tag, deploying the same Manifest twice can have different results.

There are two ways to achieve this in Spinnaker:

Templatize your manifests outside of Spinnaker to point to the digest you want to deploy. This means using some templating language ( jsonnet , jinja ) to express & render your manifests before sending them to Spinnaker to deploy.
Rely on Spinnaker’s artifact substitution and a trigger that supplies your image by digest.

Emit your deployed manifests to an audit log

To keep track of what’s changing in your cluster, and under what circumstances it is changing, we highly recommend configuring Spinnaker to emit events to an external audit log .

Spinnaker already generates events for any events running through its orchestration engine, all that’s needed is an endpoint to send them to. When set up, any pipeline that deploys or updates a manifest will have your fully hydrated manifest recorded.

Version your ConfigMaps and Secrets

Anytime Spinnaker deploys a ConfigMap or Secret, it appends a version to its name. If that exact ConfigMap or secret is already running in the cluster, it’s not redeployed. Downstream stages that reference the ConfigMap or secret will deploy the version chosen by Spinnaker. Unless your application requires hot-reloading of configuration, this is essential for practicing safe delivery.

Let’s go through two examples:

You want to slowly roll out a configuration change

Say you are rolling out a new feature, hidden behind a feature-flag. All that’s needed is an update to a ConfigMap referenced by a Deployment.

If your Deployment has mounted this ConfigMap, pushing a change to it immediately allows all Pods to read the update, rendering any sort of rollout useless. This happens even if the Deployment rolls out another configuration change using the typical Rolling Update policy.

If instead you push a new ConfigMap, and edit the Deployment to reflect this change, only the newly deployed Pods will have your feature flag enabled.

You need to roll back a broken configuration change

With an unversioned ConfigMap, the “Rollback” feature on Deployments, StatefulSets, and DaemonSets does not have the desired effect, because a material change to a ConfigMap referenced by a Pod in these controllers does not show up in your PodSpec, only the ConfigMap’s data section. In this case, the only option is to “Roll forward” by pushing an update to your ConfigMap, and any (potentially related) changes to controllers depending on your ConfigMap. However, this approach can have a few problems:

Every Pod reading that ConfigMap is rolled forward to the old configuration at once, even ones belonging to controllers that aren’t necessarily broken.
If the broken controller was rolled out with a dependent binary change, rolling forward the config at the same time can exacerbate the problem in the current broken controller until the rollout completes.

Avoid using the ad-hoc “edit” features when possible

Spinnaker provides quick ways to edit your deployed Manifests in the infrastructure screen. This is done to provide you a quick fallback when mitigating a broken rollout, or to increase the number of pods serving traffic.

However, as long as your manifests are stored either in an external store, or in Spinnaker pipelines, these edits are overwritten the next time you deploy your manifests. For this reason, any edits that can be made in your stored manifests should be made there and redeployed.

Docs: Deploy Helm Charts

Mon, 01 Jan 0001 00:00:00 +0000

Spinnaker surfaces a Bake (Manifest) stage to turn templates into manifests with the help of a templating engine. Helm relies on the helm template command. For more details, see helm template --help.

This stage is intended to help you package and deploy applications that you own, one that are actively developing and redeploying frequently. It is not intended to serve as a one-time installation method for third-party packages. If that is your goal, it’s better to call helm install once when bootstrapping your Kubernetes cluster.

Make sure that you have configured artifact support in Spinnaker first. All Helm charts are fetched/stored as artifacts in Spinnaker. Read more in the reference pages .

Configure the Bake (Manifest) stage

When configuring the Bake (Manifest) stage using a Helm (Helm 2 or Helm 3) render engine, you can specify the following:

The release name (required)

The Helm release name for this chart. This determines the name of the artifact produced by this stage.

Note: this name overrides any changes you make to the name in the Produces Artifacts section.
The template artifact (required)

The Helm chart that you deploy, stored remotely as a .tar.gz archive. You can produce this by running helm package /path/to/chart. For more details, helm package --help.

Configure API versions and a Kubernetes version

To begin, you must set API_VERSIONS_ENABLED to true. This is an essential Deck environment variable that enables the functionality necessary to work with API versions and a Kubernetes version in your CD pipeline.

The capabilities apiVersions (optional)

The apiVersions field in the Capabilities object represents a set of API versions that are dependent on the Kubernetes version. You can pass these API versions as an argument to the --api-versions parameter in the helm template command. This enables you to specify which Kubernetes API versions should be used when rendering your Helm templates.
The release kubeVersion (optional)

The kubeVersion field in the Capabilities object signifies the Kubernetes version itself. You can pass this Kubernetes version as argument to the --kube-version parameter in the helm template command. It specifies the exact Kubernetes version you want to use when rendering your Helm templates.

Note: Not all Helm charts contain apiVersions and kubeVersion definitions in their manifests. Make sure that your manifests contain the following code:
```
data: 
  apiVersions: {{ .Capabilities.ApiVersions }}
  kubeVersion: {{ .Capabilities.KubeVersion }}
```
In this example, you have a fully configured Bake (Manifest) stage, including the ApiVersions and KubeVersion fields:
The release namespace (optional)

The Kubernetes namespace to install release into. If parameter is not specified, Spinnaker uses the default namespace.

Note: Not all Helm charts contain namespace definitions in their manifests. Make sure that your manifests contain the following code:
```
metadata:
  namespace: {{ .Release.Namespace }}
```
Helm chart file path (optional)

Helm chart file path is only relevant (and visible) when the template artifact is a git/repo artifact. It specifies the directory path to Chart.yaml in the git repo. If absent, Spinnaker looks for Chart.yaml in the root directory of the git repo.

Given: A git repo where your Chart.yaml is in: sub/folder/Chart.yml
Then: helmChartFilePath: "sub/folder/"

Note: Leading slashes do not work in helmChartFilePath.
Zero or more override artifacts (optional)

The files passed to --values parameter in the helm template command. Each is a remotely stored artifact representing a Helm Value File .
Statically specified overrides

The set of static key/value pairs that are passed as --set parameters to the helm template command.

Configuration example

In this example, you have a fully configured Bake (Manifest) stage:

Notice that in the Produces Artifacts section, Spinnaker has automatically created an embedded/base64 artifact that is bound when the stage completes, representing the fully baked manifest set to be deployed downstream.

If you are programatically generating stages, here is the JSON representation of the same stage:

{
  "type": "bakeManifest",
  "templateRenderer": "HELM2",
  "name": "Bake nginx helm template",
  "outputName": "nginx",
  "inputArtifacts": [
    {
      "account": "gcs",
      "id": "template-id"
    },
    {
      "account": "gcs",
      "id": "value-id"
    }
  ],
  "overrides": {
    "replicas": "3"
  },
  "expectedArtifacts": [
    {
      "defaultArtifact": {},
      "id": "baked-template",
      "matchArtifact": {
        "kind": "base64",
        "name": "nginx",
        "type": "embedded/base64"
      },
      "useDefaultArtifact": false
    }
  ]
}

Helmfile

In addition to standard Helm deployments, Spinnaker supports Helmfile as a templating engine. You can find the supported version in Rosco’s Dockerfile .

Configuring a Helmfile deployment follows the specification for standard helm chart deployments (defined above) with one exception - helmChartFilePath is substituted for helmfileFilePath and is specified as follows:

helmfile file path (optional)

helmfile file path is only relevant (and visible) when the template artifact is a git/repo artifact. It specifies the directory path to helmfile.yaml in the git repo. If absent, spinnaker looks for helmfile.yaml in the root directory of the git repo.

Given: A git repo where your helmfile.yaml is in: sub/folder/helmfile.yml
Then: helmfileFilePath: "sub/folder/"

Note: Leading slashes do not work in helmfileFilePath.

Configure a downstream deployment

Now that your manifest set has been baked by Helm, configure a downstream stage (in the same pipeline or in one triggered by this pipeline) your Deploy (Manifest) stage to deploy the artifact produced by the Bake (Manifest) stage:

Note: Make sure to select “embedded-artifact” as the artifact account for your base64 manifest set. This is required to translate the manifest set into the format required by the deploy stage.

When this stage runs, you can see every resource in your Helm chart get deployed at once:

Other Templating Engines

In addition to Helm, Spinnaker also supports Kustomize as a templating engine. For more information, see Using Kustomize for Manifests .

Docs: Deploy Kubernetes Manifests

Mon, 01 Jan 0001 00:00:00 +0000

This guide shows the basics of how to deploy a Kubernetes manifest using the Kubernetes provider .

There are two main steps:

Specify which manifest to deploy

This is required.
Override artifacts in the manifest

Optionally, you can override some artifacts (as fields) at run time (for example, which Docker image to use.)

Specify your manifest

Depending on your needs, there is more than one way to specify the manifest that you want to deploy:

In either case, start by selecting the Deploy (Manifest) stage from the stage selector:

Don’t select the regular Deploy stage; it deploys more opinionated “Server Groups” using other providers.

Specify manifests statically

If you know ahead of time what you expect to deploy using a certain manifest (even if you don’t know what version of your Docker image it will run), you can declare it directly in the pipeline by providing the manifest specification:

Notice that by selecting Text as the Manifest Source, we get to enter the manifest YAML by hand.

If you are generating your pipeline definitions rather than entering them into the UI, the stage definition would look more like this:

{
  "name": "Deploy my manifest",   // human-readable name
  "type": "deployManifest",       // tells orchestration engine what to run
  "account": "nudge",             // account (k8s cluster) to deploy to
  "cloudProvider": "kubernetes",
  "source": "text",
  "manifest": {
                                  // manifest contents go here
  }
}

Specify manifests dynamically

If you are storing your manifests outside of Spinnaker’s pipeline repository , or want a single deploy stage to deploy a variety of manifests, you can specify your manifest using an Artifact .

The idea is: artifacts in Spinnaker allow you to reference remote, deployable resources. When referencing an artifact from a Deploy Manifest stage , that artifact must be a text file containing the Manifest specification. This can be stored in GitHub or an object store (like GCS).

Changes to manifests can trigger pipelines. For more information:

Assuming you have declared an expected artifact upstream to your Deploy manifest stage, you can reference it in the Deploy configuration:

Notice that by selecting Artifact as the Manifest Source, we get to pick which upstream artifact to deploy.

Note: Make sure that the Artifact Account field matches an account with permission to download the manifest.

Keep in mind that the artifact bound in the upstream stage can match multiple incoming artifacts. If instead we had configured it to listen to changes using a regex matching .*\.yml, it would bind any YAML file that changes in your artifact source, and deploy it when it reaches your Deploy stage.

Override artifacts

In general, when we deploy changes to our infrastructure, the majority of changes come in the form of a new Docker image, or perhaps a feature-flag change in a ConfigMap. For this reason, we have first-class mechanisms for easily overriding the version of…

Docker image
Kubernetes ConfigMap
Kubernetes Secret

When one of these objects exists in the pipeline context from an upstream stage, Spinnaker automatically tries to inject it into the manifest you’re deploying.

For example, say you trigger your pipeline using a webhook coming from a Docker registry. At a high level, the event says “Image gcr.io/my-project/my-image has a new digest sha256:c81e41ef5e...”. In the pipeline that gets triggered, you’ve configured a Deploy Manifest stage with the following spec:

# ... rest of manifest
  containers:
  - name: my-container
    image: gcr.io/my-project/my-image
# rest of manifest ...

Because the pipeline was triggered by a change to the Docker image, the orchestration engine send that artifact along with the manifest to Spinnaker’s cloud provider integration service, which, based on the name of the Docker image, deploys the following:

# ... rest of manifest
  containers:
  - name: my-container
    image: gcr.io/my-project/my-image@:sha256:c81e41ef5e...
# rest of manifest ...

To ensure that the correct artifacts get bound, you can force the stage to either bind all required artifacts, or fail before deploying. Here’s an example where we specify that the docker image gcr.io/my-project/my-image must be bound in the manifest, otherwise the stage fails:

Keep in mind that even if you don’t specify an artifact as required, it can still be bound in the manifest. This is just to ensure that all artifacts you expect will be bound.

Docs: Manage Traffic Using Kubernetes Manifests

Mon, 01 Jan 0001 00:00:00 +0000

This guide shows the basics of how to manage traffic during deployments using the Kubernetes provider . This includes automatically attaching a Service to a workload during deployment, and defining pipelines to perform blue/green deployments.

Attach a Service to a Workload

Kubernetes manages the relationship between Pods and Services using Labels and Selectors . Spinnaker acknowledges this relationship with the blue load balancer icon on a workload as shown here:

You can also tell Spinnaker which Service objects a workload should be associated with for its lifetime using the <code>traffic.spinnaker.io/load-balancers</code> annotation. To try this out, do the following:

Deploy the following Service using the “Create Load Balancer” button:

kind: Service
apiVersion: v1
metadata:
  name: my-service
spec:
  selector:
    app: myapp
  ports:
  - protocol: TCP
    port: 80

And then deploy the following Replica Set using the “Create Server Group” button:

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: frontend
  labels:
    tier: frontend
  annotations:
    traffic.spinnaker.io/load-balancers: '["service my-service"]' # <-- annotation
spec:
  replicas: 3
  selector:
    matchLabels:
      tier: frontend
  template:
    metadata:
      labels:
        tier: frontend
    spec:
      containers:
      - name: frontend
        image: us-docker.pkg.dev/google-samples/containers/gke/gb-frontend:v5

Notice that the selector app: myapp appearing on the Service did not have to be specified on the Replica Set’s PodSpec; Spinnaker adds this automatically.

Caveats

There are a few caveats when using this annotation:

The list of Services supplied in this annotation does not need to specify all Services a workload may be attached to. If your workload’s labels already attach it Service A, and you list Service B in the annotation, both Service A and B will route requests to this workload.

You can think of this annotation as only supplying the list of additional services to attach to, not as an exhaustive list.
If you use this annotation, Spinnaker will edit the workload’s labels, either in the metadata.labels, or spec.template.metadata.labels sections. As a result, if the labels added to or removed from the spec.template.metadata.labels conflict with the label requirements in the object’s spec.selector.match[Labels/Expressions] section, the deployment will fail.
This annotation tells Spinnaker which service or services to make traffic changes to. For example, if you specify service “A” in the annotation of a manifest and subsequently disable it, only traffic from service “A” will be disabled, even if you receive traffic from both services “A” and “B”.

Route Traffic During a Deployment (Blue/Green)

Spinnaker supports dynamically enabling and disabling traffic to a particular Kubernetes resource. This can be used in a pipeline to configure Blue/Green rollouts, as shown below.

You Must Use Replica Sets

Spinnaker manages traffic using labels, and has to edit them on running resources. Given that edits to Deployment objects trigger rollouts, the only way to carry out Blue/Green deployments without editing the Service object directly is using Replica Sets, which are safely edited in-place by Spinnaker. To be clear, these label edits are abstracted into higher-level “Enable” and “Disable” stages to manage traffic, and are described below.

In turn, when a user submits a change to a Replica Set (either ad-hoc, or using a pipeline) to update an image, configuration, etc… Spinnaker makes sure to deploy a new Replica Set with a unique -vNNN suffix. This is in contrast with the edits to labels Spinnaker performs in place to manage traffic. The interaction between edits to Replica Sets made by users and those made by Spinnaker will be made clear in the below sample pipeline walkthrough.

Sample Blue/Green Pipeline

We will create a pipeline with two stages: Deploy, and Disable. The goal is to first deploy a new version of your application, have the stage wait until all Pods’ health checks pass, and then disable the prior version of your application. When completed, an execution of this pipeline will look like:

Prerequisites

You need the following:

Spinnaker with the Kubernetes provider configured
One service called my-service running in the cluster and namespace your pipeline will deploy to. For the purpose of this example we will assume it has the following specification:

kind: Service
apiVersion: v1
metadata:
  name: my-service
spec:
  selector:
    frontedBy: my-service # will be applied to backends by Spinnaker
  ports:
  - protocol: TCP
    port: 80

Configure the Deploy Stage

Create a pipeline and add a “Deploy (Manifest)” stage, and configure the following (text) manifest to deploy:

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  annotations:
    strategy.spinnaker.io/max-version-history: '2'
    traffic.spinnaker.io/load-balancers: '["service my-service"]'
  labels:
    app: myapp
  name: myapp-frontend
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
        - image: 'us-docker.pkg.dev/google-samples/containers/gke/gb-frontend:v5' 
          name: frontend

The stage configuration should look like this:

There are two important annotations to pay attention to here:

traffic.spinnaker.io/load-balancers

This annotation is explained above , and is used here to both indicate to Spinnaker which Service to attach during deployment time, and which Service to detach during the disable stage.
strategy.spinnaker.io/max-version-history

This annotation is explained in the reference section , and is used to ensure that no more than a certain number of versions of this Replica Set are running for garbage collection purposes.

Configure the Disable Stage

Add a “Disable (Manifest)” stage after the Deploy stage, and set the following values:

Field	Value
Namespace	`default`
Kind	`replicaSet`
Selector	`Choose a target dynamically`
Cluster	`replicaSet frontend`
Target	`Second Newest`

The stage configuration should look like this:

There are two important fields here:

Selector: Choose a target dynamically

With every run of the pipeline a new Replica Set is created, and therefore we cannot know ahead of time which version of the Replica Set to disable. This field allows us to pick a Replica Set based on the Target value.
Target: Second Newest

By picking the Second Newest target, we select the second newest Replica Set when this stage starts, or in other words, what was just previously serving traffic.

Run the Pipeline

After a few runs of the pipeline, you should see the following in your cluster screen:

Docs: Parameterize Kubernetes Manifests

Mon, 01 Jan 0001 00:00:00 +0000

Spinnaker can inject context from the currently executing pipeline into your manifests as they are deployed, whether they are deployed statically or dynamically .

This can be applied to a wide range of use-cases, but we will focus on using a pipeline parameter to specify the target namespace.

Configure your pipeline parameters

First, register a pipeline parameter in the “configuration” tab of the pipeline editor (only the Name is required):

See more details on how to provide parameters to pipelines programmatically in the webhooks page.

Warning: there are several reserved parameter keys (names) that cause unexpected behavior and failures if overwritten by a pipeline parameter definition. See the list of reserved parameter and evaluate variable key names .

Configure your manifest

In this scenario, we’re using a parameter to specify the manifest’s namespace. Edit your manifest so the metadata section contains:

# ... other keys
metadata:
  namespace: '${ parameters.namespace }'
# other keys ...

Keep in mind this manifest can be stored in the pipeline, or in an external store such as GitHub .

When you go to run the pipeline by hand, you will see the following:

Parameterizing non-string keys

When parameterizing a YAML value that’s not a string (such as the replica count), you will need to explicitly convert the evaluated expression to the correct type .

If you were expecting the replica count to arrive in parameter replicas, you would write:

# ... other keys
spec:
  replicas: '${ #toInt( parameters.replicas ) }'
# other keys ...

More advanced parameterization

Please read the pipeline expressions guide .

Docs: Patch Kubernetes Manifests

Mon, 01 Jan 0001 00:00:00 +0000

This guide shows the basics of how to update a Kubernetes resource in place using the patch manifest stage for the Kubernetes provider provider.

There are a few steps:

Specify which manifest to patch
Specify the patch content
(Optional) Override artifacts in the patch content

Optionally, you can override some artifacts (as fields) at run time (for example, which Docker image to use.)
(Optional) Override patch specific options

Specify the resource to patch

Start by selecting the Patch (Manifest) stage from the stage selector:

To identify the Kubernetes resource to patch, specify the following required fields:

Field	Value
Account	The Spinnaker account that manages the Kubernetes cluster
Namespace	The Kubernetes namespace that your resource is located in
Kind	The Kubernetes Kind of your resource e.g. deployment, service etc
Name	The name of your Kubernetes resource

Specify your patch content

The patch content is similar to a manifest in the Deploy (Manifest) stage. However, unlike the deploy manifest, this does not have to be the full resource manifest but only the portion you want to patch.

Depending on your needs, there is more than one way to specify the patch content:

Specify patch content statically

You can enter the patch content YAML by hand. For instance, if you want to patch your manifest to add a new label, you will specify the following:

metadata:
  labels:
    foo: bar

Notice that by selecting Text as the Manifest Source, we get to enter the manifest YAML by hand.

Specify patch content dynamically

Like the Deploy (Manifest) stage , you can also reference an artifact as the source if you are storing your patch content externally. The artifact must be a text file containing the patch content.

You can also set up the pipeline to trigger based on changes to the patch content:

Assuming you have declared an expected artifact upstream to your Patch (Manifest) stage, you can reference it in the Patch configuration:

Notice that by selecting Artifact as the Manifest Source, we get to pick which upstream artifact to deploy.

Note: Make sure that the Artifact Account field matches an account with permission to download the manifest.

Override artifacts

When patching with a strategic or merge strategy, the Patch (Manifest) stage also allows you to override artifacts like in the deploy manifest stage.

For instance, say you have a pipeline with a Patch (Manifest) stage with the following patch content:

spec:
  template:
    spec:
      containers:
        - name: my-container
          image: gcr.io/my-project/my-image

If your pipeline was triggered due to a new Docker image tag being pushed to your Docker registry (say my-image:2.0), Spinnaker will override the version of the container image with the new version:

#...rest of manifest
containers:
  - name: my-container
    image: gcr.io/my-project/my-image:2.0

For more information on how this works, check out the binding artifacts docs .

Specify Patch Options

You can also specify the following options:

Record Patch Annotation

Defaults to true. When selected, the patch operation including the patch content will be recorded on the patched resource as the kubernetes.io/change-cause annotation. If the annotation already exists, the contents are replaced.
Merge Strategy
- strategic: This is the default. It is a customized version of JSON merge patch specific to Kubernetes that allows Kubernetes objects to be either replaced or merged based on the object struct tags. It is particularly useful when you want to add a new item to a list (e.g. a new annotation, label, or even a new container to a pod spec) instead of replacing the list.
- json: This will patch the manifest using a standard RFC 6902 JSON patch .
- merge: This will patch the manifest using RFC 7386 JSON Merge Patch .

Docs: Rollout Strategies

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to take advantage of the Kubernetes provider’s first-class support for common rollout strategies, including dark, highlander, and blue/green rollouts.

The implementation of these rollout strategies currently leverages Spinnaker’s existing traffic management strategy and is valid for ReplicaSets only. See traffic management for more information.

Rollout Strategy Options

As of version 1.14, you will notice a Rollout Strategy Options section in the Deploy (Manifest) stage. When enabled, these configuration options allow you to associate a workload with one or more services, decide whether the workload should receive traffic, and determine how Spinnaker should handle any previous versions of the workload in the same cluster and namespace.

Configuration options:

Service namespace

Select the namespace containing the service(s) you would like to associate with the workload.
Service(s)

Select one or more services you would like to associate with the workload. Spinnaker will add a traffic.spinnaker.io/load-balancers annotation listing the selected services as described here .
Traffic

Check this box if you would like the workload to begin receiving traffic from the selected services as soon as it is ready. If you do not check this box, you can add a subsequent Enable (Manifest) stage to begin sending traffic to the workload.
Strategy

Select a strategy if you would like Spinnaker to handle previous versions of the workload currently deployed to the same cluster and namespace. Select None if you do not want Spinnaker to take any action regarding existing workloads.

Dark Rollouts

Use a dark rollout to deploy a new version of your application alongside the existing version(s), but do not immediately route any traffic to the new version.

Optionally, add subsequent Enable (Manifest) and Disable (Manifest) stage to begin sending traffic to the new version and stop sending traffic to the old version(s).

Example configuration:

Highlander Rollouts

Use a highlander rollout to deploy a new version of your application alongside the existing version(s), send client traffic to the new version, and then disable and destroy existing versions in the cluster.

Example configuration:

Red/Black (Blue/Green) Rollouts

Use a red/black (blue/green) rollout to deploy a new version of your application alongside the existing version(s), send client traffic to the new version, and then disable existing versions in the cluster.

Optionally, add subsequent Destroy (Manifest) stages to clean up any unwanted workloads in the cluster.

Alternately, easily roll back to a previous version by configuring an Enable (Manifest) stage or using an ad-hoc Enable operation from the Clusters tab.

Example configuration:

Blue/Green replaces Red/Black in Spinnaker v1.30+.

Docs: Run Job (Manifest) Stage

Mon, 01 Jan 0001 00:00:00 +0000

The Run Job (Manifest) stage can be used to execute a Kubernetes Job as part of your pipeline. This stage will deploy a Job manifest and wait until it completes, allowing you to gate your pipeline’s continuation on the job’s success or failure.

For example use cases, check out this post on the Spinnaker blog!

Viewing execution logs

As with any job runner, logs are the primary form of feedback and it’s important that viewing them is a first-class experience within Spinnaker. Since users of Spinnaker are all at various stages of Kubernetes adoption we’ve provided 2 ways to view logs within Spinnaker.

Link to external system

Most Kubernetes deployments will have some utility for forwarding logs from containers and into external systems for log analysis. If you’re using one of these platforms, you can configure your Job with the annotation job.spinnaker.io/logs and a templated URL to your logging system. This value of this annotation will be used to render a link in the UI.

To make it easier to pinpoint the specific job, the annotation value can be templated with values from the deployed Job manifest. To use templates, use {{ "{{ templateKey "}} }} where templateKey is the path to the value you wish to use. The deployed manifest will be passed into the template as JSON. This functionality mirrors that of the Annotation Driven UI .

For example, if your Job is deployed with name myjob-12345, the annotation…

job.spinnaker.io/logs: 'https://internal-logging/jobs/{{ "{{ manifest.metadata.name "}}}}'

…will result in the value

https://internal-logging/jobs/myjob-12345

Within Spinnaker

You can still view the logs of your Job within Spinnaker even if your Kubernetes deployment doesn’t forward them to an external system. In the absense of the job.spinnaker.io/logs annotation, Spinnaker fetches the deployed manifest and displays logs directly in the UI. These logs are only be available for a short period following the job’s completion because Kubernetes only stores events about an object for a short amount of time. If you’d like to view your logs for longer than this timeframe, it’s recommended that you use a tool like Fluentd to forward logs to a more persistent platform like Elasticsearch or Datadog.

Capturing output

In most cases, jobs are meant to do some work and provide feedback in the form of an output, such as structured JSON. This output can then be used to influence downstream stages via SpEL . There are 2 ways of doing this with the Run Job (Manifest) stage: using logs and using artifacts .

Logs

Note: Some Jobs output a massive amount of data which can be expensive to process. If your job outputs megabytes or gigabytes of log data it may be best to utilize the artifact based approach outlined in the Artifacts section.

The easiest and most accessible way of capturing output from your jobs is to output data to your logs by writing to stdout. When a job completes, the log data is captured and analyzed for 2 different markers. These markers are as follows:

`SPINNAKER_PROPERTY_*`

SPINNAKER_PROPERTY_*=* can be used to provide single key/value pairs, similar to Jenkins build.properties files. Everything after SPINNAKER_PROPERTY_ is captured as a key/value. For example, given the following log output

Checkout spinnaker/spinnaker source code...
Latest tag is: 1.1.0.
Tag spinnaker/spinnaker with next minor version...
Uploading release...
Release uploaded.

SPINNAKER_PROPERTY_RELEASE=1.1.1
SPINNAKER_PROPERTY_URL=https://github.com/spinnaker/spinnaker/releases/tag/1.1.1

the captured output would be in the following format

{
    "RELEASE": "1.1.1",
    "URL": "https://github.com/spinnaker/spinnaker/releases/tag/1.1.1"
}

`SPINNAKER_CONFIG_JSON`

SPINNAKER_CONFIG_JSON can be used to provide complex or structured data. In this case, anything following SPINNAKER_CONFIG_JSON= on a single line of stdout is directly consumed and parsed as JSON. Using the previous example, notice how the output has changed.

Checkout spinnaker/spinnaker source code...
Latest tag is: 1.1.0.
Tag spinnaker/spinnaker with next minor version...
Uploading release...
Release uploaded.

SPINNAKER_CONFIG_JSON={"RELEASE": "1.1.1", "URL": "https://github.com/spinnaker/spinnaker/releases/tag/1.1.1"}

Again, using this solution would result in the following format:

{
    "RELEASE": "1.1.1",
    "URL": "https://github.com/spinnaker/spinnaker/releases/tag/1.1.1"
}

If SPINNAKER_CONFIG_JSON is found multiple times within the log then each is parsed and added to the resulting output. If multiple keys are the same, for example the key foo is contained in multiple instances of SPINNAKER_CONFIG_JSON, then the last one wins. The results of multiple occurrences are not merged.

If you source JSON from a file (SPINNAKER_CONFIG_JSON=$(cat file.json)), be sure that the file content does not introduce newline characters since each SPINNAKER_CONFIG_JSON is only parsed from one line in the log file.

Artifacts

Artifacts are a mechanism within Spinnaker that enables pipelines to reference resources stored in external systems. Artifacts can be anything — Docker images, Kubernetes manifests, and in this case, data produced by the Run Job stage. As mentioned above, if your job is producing so many logs that parsing them for output data would be an expensive operation, utilizing artifacts would be the best solution.

In order to use artifacts to capture output data, your job needs to push output to one of Spinnaker’s supported artifact stores . This output is captured at the end of the job execution and injected into the pipeline for use in downstream stages via SpEL. Artifacts captured as output must be in JSON format.

As an example, let’s imagine you are running a job which pushes output into an S3 bucket at the end of its execution.

...
Operation completed successfully.
Publishing output to S3.
Output published to s3://my-bucket/my-job/output.json

In order to capture this output, you’ll want to configure the Run Job (Manifest) stage to capture the output from an Artifact.

Cleaning up old Jobs

Spinnaker doesn’t clean up any Jobs deployed via the Run Job (Manifest) stage. In order to clean up old Jobs it’s recommended that you implement some type of garbage collection so as not to overrun your Kubernetes deployment with a massive amount of old Jobs.

As of Kubernetes 1.12, automatic <code>Job</code> cleanup is an alpha feature, available behind the TTLAfterFinished feature flag. This flag must be enabled on the API server when it’s started. To verify if your deployment has this feature enabled, check with your Kubernetes administrator.

Docs: Use Kustomize for Manifests

Mon, 01 Jan 0001 00:00:00 +0000

Kustomize is a tool that lets you create customized Kubernetes deployments without modifying underlying YAML configuration files. Since the files remain unchanged, others are able to reuse the same files to build their own customizations. Your customizations are stored in a file called kustomization.yaml. If configuration changes are needed, the underlying YAML files and kustomization.yaml can be updated independently of each other.

To learn more about Kustomize and how to define a kustomization.yaml file, see the following links:

In the context of Spinnaker, Kustomize lets you generate a custom manifest, which can be deployed in a downstream Deploy (Manifest) stage. This manifest is tailored to your requirements and built on existing configurations.

Overview

Kustomize works by running kustomize build against a kustomization.yaml file located in a Git repository. This file defines all of the other files needed by Kustomize to render a fully hydrated manifest.

Configure the “Bake (Manifest)” stage

Note: Kustomize requires a git/repo artifact type.**

Select Kustomize as the Render Engine and define the artifact for your kustomization.yaml.

You can specify the following:

Account (required)

The git/repo account to use.
URL (required)

The location of the Git repository.
Branch (optional)

The branch of the repository you want to use. [Defaults to master]
Subpath (optional)

By clicking Checkout subpath, you can optionally pass in a relative subpath within the repository. This provides the option to checkout only a portion of the repository, thereby reducing the size of the generated artifact.
Kustomize File (required)

The relative path to the kustomization.yaml file residing in the Git repository.

Using Spinnaker 1.16

Select Kustomize as the Render Engine and define the artifact for your kustomization.yaml:

Configuring the Produced Artifact

With the Bake (Manifest) Configuration completed, configure a Produced Artifact to use the result in a stage downstream. Add an artifact:

Define the artifact:

You can now run your pipeline and get a Kustomize rendered manifest!

Updating 1.16 “Bake (Manifest)” stage for 1.17

As of 1.17, git/repo is the only supported artifact type when configuring the Bake (Manifest) stage in the UI with Kustomize.

Pipelines configured to use Kustomize in 1.16 will continue to work in 1.17. However, editing a Bake (Manifest) stage in 1.17, which was originally created in 1.16, requires you to update the Bake (Manifest) Configuration to use the git/repo artifact type. To do so, use the following instructions:

Click on the Account dropdown and select a configured git/repo account. If none appear, make sure you have configured a git/repo account
Note: You should click and select a git/repo account even if one already appears in the UI prior to your doing so. This will force the underlying JSON to be updated to use the new artifact.
Update the URL. This should be the location of the git repository.
example: https://github.com/kubernetes-sigs/kustomize
Provide the Kustomize file path. This should be the relative path to the kustomization.yaml within the repository.
example: examples/wordpress/mysql/kustomization.yaml

Before updating. The fields highlighed in red should be updated as described above.

After updating.

Other Templating Engines

In addition to Kustomize, Spinnaker also supports Helm (and Helmfile) as templating engines. For more information, see Deploy Helm Charts .