Spinnaker – What is a Spinnaker Artifact?

Docs: Artifacts from Build Triggers

Mon, 01 Jan 0001 00:00:00 +0000

Overview

When an external CI system triggers a pipeline, Spinnaker can use the CI build information to inject relevant artifacts into the pipeline. The instructions here assume that you have set up a CI system and are familiar with using property files to pass variables from CI builds to Spinnaker pipelines.

The extraction of artifacts from the build information is done via a Jinja template; the template uses the trigger as context and outputs a list of artifacts to inject into the pipeline. Spinnaker provides a set of standard templates to use for artifact extraction, which users can augment with custom templates.

Requirements

The instructions here assume that you have set up a CI system and are familiar with using property files to pass variables from CI builds to Spinnaker pipelines.

Select a template

To configure Spinnaker to use a Jinja template for artifact extraction, export the following properties from your CI build:

messageFormat: the name of the Jinja template to use
customFormat: true if messageFormat refers to a user-configured template; false or omitted if it refers to a Spinnaker-supplied template

For example, to use the Spinnaker-provided JAR template, you would export the following properties from your CI job:

messageFormat=JAR

The recommended way to configure artifact templates is by adding the following blocks to igor-local.yml:

artifacts:
  templates:
  - name: <name of template>
    templatePath: <path to the template>

You can then use the configured custom template by exporting the following as properties from your CI build:

messageFormat=<name of template>
customFormat=true

Bind variables into templates

In general, artifact-extracting templates will read other properties that are exported by the CI job. The general pattern is to export any build-specific information in the property file and to have the Jinja template construct the artifact by looking in trigger.properties.

For example, consider a Jenkins job uploads a .jar file to a maven repository. We might define a custom Jinja template custom-jar.jinja as follows:

{% raw %}

  {
    "reference": "{{ properties.group }}-{{ properties.artifact }}-{{ properties.version }}",
    "name": "{{ properties.artifact }}-{{ properties.version }}",
    "version": "{{ properties.version }}",
    "type": "maven/file"
  }

{% endraw %}

We could then generate a useful artifact by having the CI job export the following:

group=test.group
artifact=test-artifact
version=123
messageFormat=custom-jar
customFormat=true

Supplied templates

The templates that are supplied with Spinnaker can be found in the following folder .

JAR

The JAR template creates an artifact representing a JAR archive in a Maven or Ivy repository. This template expects the following properties to be exported:

group
artifact
version
(optional) classifier

By default, the artifact represents an archive in a Maven repository; to create an artifact for an archive in an Ivy repository, export the property repotype=ivy.

Docs: Artifacts In App Engine

Mon, 01 Jan 0001 00:00:00 +0000

Artifacts can be used for the following in the Google App Engine provider:

They can be used to reference a GCS bucket that contains source code to be deployed on Google App Engine.
They can be used to reference a docker image stored in Google Container Registry to be deployed on App Engine Flex.
They can be used to reference config files (such as app.yaml) to be used during deployment to App Engine.

Artifacts referencing source code from GCS

To use an artifact to reference source code that is stored in a GCS bucket, use the “GCS” Source Type when configuring a Server Group.

In the image below, a GCS Artifact has been configured in the Create Server Group modal as the source code to deploy to App Engine:

Artifacts referencing Docker images from GCR

To use an artifact to reference docker images stored in Google Container Registry, set the Server Group’s Source Type to “Container Image” and set the “Resolve URL” field to “via pipeline artifact”.

In the below image, a Docker Artifact has been configured in the Create Server Group modal as the container image URL to use for deployment to App Engine Flex:

Artifacts referencing config files

To use an artifact to reference config files used during deployment to App Engine, click the “Add Config Artifact” button in the Create Server Group modal.

In the below image a GCS Artifact has been added to the Config Files section of the Create Server Group modal:

Docs: Artifacts In Cloud Foundry

Mon, 01 Jan 0001 00:00:00 +0000

Artifacts referencing app manifest

To use an artifact to reference a manifest file (such as a manifest stored in a GitHub repository), choose “Artifact from execution context” in the Application section of the “Configure Deployment Cluster” screen, and configure the artifact details.

In a Deploy stage, choose a GitHub file artifact as the manifest source for a server group.

Artifacts referencing app archive

To use an artifact to reference an application archive (such as a JAR file from a Maven repository), choose “Artifact from execution context” in the Manifest section of the “Configure Deployment Cluster” screen, and configure the artifact details.

In a Deploy stage, choose a Maven artifact as the application source for a server group.

Docs: Artifacts In Kubernetes

Mon, 01 Jan 0001 00:00:00 +0000

Manifests as artifacts

There are two ways to deploy a manifest:

The manifest is supplied statically to a pipeline as text
The manifest is supplied as an artifact

The image below shows a deploy stage that deploys a manifest stored in a GCS bucket:

Depending on how this pipeline is configured, it will only run when the referenced file in GCS is modified.

Kubernetes objects as artifacts

Once a manifest is successfully deployed using a pipeline (either from text or an artifact containing text), it is injected back into the pipeline’s context as an output of the deploy stage. Why this is useful is explained below , but for now focus on the distinction between…

An artifact representing a manifest stored as text in github:

{
     "type": "github/file",
     "name": "manifests/frontend-configs.yml",
     "reference": "https://api.github.com/repos/your-application/..."
}

An artifact representing a deployed kubernetes object:

{
     "type": "kubernetes/configMap",
     "name": "frontend-configs",
     "location": "prod",
     "version": "v001"
}

As described in the manifests as artifacts section, a deploy stage would consume artifact 1, but produce artifact 2 as an output.

When running pipelines, you can always check the produced outputs for any stage by examining the execution’s “source” directly:

Versioned Kubernetes objects

According to the Kubernetes reference documentation , certain resources are “versioned,” meaning anytime a change is made to an object’s manifest and deployed using Spinnaker, it is redeployed with a new version suffix (-vNNN). This is critical to supporting immutable deployments, as rolling out new ConfigMaps, secrets, or other versioned resources should require any manifests that reference them to be updated as well. Luckily, Spinnaker makes handling these updates easy, as explained below .

Binding artifacts in manifests

Generally, artifacts represent resources that you update as a part of your deployment/delivery pipelines. Given that Docker images and ConfigMaps are what will likely be updated within a manifest, we provide easy, first-class ways of injecting them into your manifests. If you’re familiar with Pipeline Expressions and are curious why we don’t just rely on those, read why not pipeline expressions below.

Spinnaker binds artifacts in your manifest based on a simple heuristic:

When a field’s referenced type and value match an incoming artifact’s type and name, the field’s value is replaced with the artifact’s reference

A “field’s referenced type” sounds ambiguous, but in practice it’s straightforward. The field spec.template.spec.containers.*.image always refers to a Docker image, so clearly it matches the artifact type docker/image. The field spec.template.spec.volumes.*.configMap.name always refers to a ConfigMap, so it clearly matches the artifact type kubernetes/configMap. The same logic applies throughout.

Let’s go through an example to make this clear:

We have the following manifest to be deployed in a Spinnaker pipeline:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - image: gcr.io/my-images/nginx # possible artifact
          name: nginx
          ports:
            - containerPort: 80
          volumeMounts:
            - mountPath: /opt/config
              name: my-config-map
      volumes:
        - configMap:
            name: configmap             # possible artifact
          name: my-config-map

And when the deploy stage executes, we have the following artifacts in our execution context (likely populated from a trigger event, or prior deployments):

[
  {
    "type": "docker/image",
    "name": "gcr.io/my-images/nginx",
    "reference": "gcr.io/my-images/nginx@sha256:0cce25b9a55"
  },
  {
    "type": "kubernetes/configMap",
    "name": "configmap",
    "version": "v001",
    "location": "default",
    "reference": "configmap-v001"
  }
]

The ConfigMap and Docker image are replaced by the artifacts in the context, and the resulting manifest is deployed into your cluster:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - image: gcr.io/my-images/nginx@sha256:0cce25b9a55    # bound by spinnaker
          name: nginx
          ports:
            - containerPort: 80
          volumeMounts:
            - mountPath: /opt/config
              name: my-config-map
      volumes:
        - configMap:
            name: configmap-v001                              # bound by spinnaker
          name: my-config-map

Why not pipeline expressions?

Pipeline Expressions offer a great way to reference pipeline context programmatically using short snippets of code. Of course, it’s possible to construct expressions that allow you to reference the Docker image’s reference that triggered a pipeline, or the name of the ConfigMap you deployed in a prior stage. But we want an easier way to express updates to these resources, leaving your Kubernetes Manifests natively deployable without making Spinnaker’s expression engine a hard dependency.

Docs: Artifacts In Pipelines

Mon, 01 Jan 0001 00:00:00 +0000

Much of the behavior described here depends on looking up execution history in Redis. Deleting recent executions from Redis can cause unexpected behavior.

Now that you have an idea of what an artifact is in Spinnaker, you need to understand how it’s used within pipelines.

Spinnaker uses an “expected artifact” to enable a stage to bind an artifact from another pipeline execution, stage output, or running environment.

Expected artifacts

An “expected artifact” is a specification of what properties (found in the URI decoration) against which to match when searching for the desired artifact, plus optional fallback behavior.

Expected artifacts exist for two reasons:

To declare what artifacts need to be present at a certain point during execution
To provide easy ways to reference those artifacts

An expected artifact consists of an artifact to match (by name, type, etc…) plus, optionally, what to do if no artifact is matched. When an artifact is matched, we say it’s bound to that expected artifact.

The matching behavior is as follows: The artifact to match declared by the expected artifact has the same format as a regular artifact; however, its values are interpreted as regular expressions to match the corresponding values in the incoming artifact:

Normal strings must match exactly for the artifact to match because Spinnaker effectively treats them as regular expressions. For example, Spinnaker interprets inputstring as ^inputstring$.
Explicit regular expressions are also valid. For example, Spinnaker matches .*inputstring.* to any string including the substring inputstring.

Match Artifact	Incoming Artifact	Matches?
`{"type": "docker/image"}`	`{"type": "docker/image", "reference: "gcr.io/image"}`	✔
`{"type": "docker/image"}`	`{"type": "gce/image", "reference: "www.googleapis.com/compute/v1/projects..."}`	✘
`{"type": "docker/image", "version": "v1"}`	`{"type": "docker/image", "reference: "gcr.io/image:test", "version": "v1"}`	✔
`{"type": "docker/image", "version": "v1"}`	`{"type": "docker/image", "reference: "gcr.io/image:test", "version": "v1.2"}`	✘
`{"type": "docker/image", "version": "v1\..*"}`	`{"type": "docker/image", "reference: "gcr.io/image:v1.2", "version": "v1.2"}`	✔
`{"type": "docker/image", "version": "v1\..*"}`	`{"type": "docker/image", "reference: "gcr.io/image:test", "version": "test"}`	✘

If an expected artifact matches anything other than a single artifact (and no fallback behavior is defined), the execution fails. Therefore it’s always safe to say “use the artifact bound by this upstream expected artifact” because if no artifact was bound, the pipeline wouldn’t be running this downstream stage.

Triggers

In Pipeline Configuration, you can now declare which artifacts a pipeline expects to have present before the pipeline starts running.

The UI provides short-hand for defining some types of artifacts—in this example a docker image. This is optional, but helps quickly define common types of artifacts.

You can define fallback behavior for when the artifact isn’t bound at the start of the Pipeline. The two options are evaluated in order:

Use Prior Execution

Use the artifact bound by the pipeline’s previous execution. This is useful in cases where multiple sources can trigger the pipeline, but each has incomplete information about what artifacts to supply.
Use Default Artifact

Specify exactly which artifact to bind.

Once you have declared which artifacts are expected by this pipeline, you can assign expected artifacts to individual triggers.

A pubsub subscription configured to listen to changes in one GCR registry. Since this registry can contain many repositories, we’ve assigned it an expected artifact to ensure only changes in one repository can run this pipeline.

When a trigger has one or more expected artifacts, it only runs when each expected artifact can bind to one of the artifacts in the trigger’s payload.

Artifacts in trigger payloads

Artifacts are supplied by payload as a list of artifacts in a top-level artifacts key—the value is automatically injected into any triggered pipeline’s execution context. However, it’s possible that you’re not the author of the incoming message, and are instead depending on a third-party system to provide an event to Spinnaker to trigger a pipeline. If this event contains information such as “these images were created,” or “these files were modified,” it’s useful to be able to extract artifacts from that payload.

For this reason, we allow you to supply Jinja templates that transform the incoming payload into a list of artifacts to be injected into your pipeline.

Take for example the notification format for GCR . While it captures the information we need, it’s not in the format we want. So, we can register the following Jinja template for our GCR subscription:

[
  {
    {% raw %}
    {% set split = digest.split("@") %}
    "reference": "{{ digest }}",
    "name": "{{ split[0] }}",
    "version": "{{ split[1] }}",
    "type": "docker/image"
    {% endraw %}
  }
]

Find artifact from execution

To allow you to promote artifacts between executions, you can make use of the “Find Artifact from Execution” stage. All that’s required is the pipeline ID whose execution history to search, and an expected artifact to bind.

A common use case would be to “promote” the image deployed to staging to a pipeline that’s deploying to production.

Passing artifacts between pipelines

Artifacts can be passed between pipelines. If a pipeline triggers the execution of a second pipeline, this second pipeline will have access to any artifacts that were available to the first pipeline; this includes both artifacts in the first pipeline’s trigger as well as artifacts emitted by the first pipeline.

Two concrete cases where artifacts can be passed are as follows:

Pipeline triggered by the completion of another pipeline

To set up this configuration, go to the Configuration screen for a pipeline (Pipeline B) and add an automated trigger of type Pipeline pointing to another pipeline (Pipeline A). Whenever Pipeline A completes, it will trigger a run of Pipeline B, and Pipeline B will have access to all artifacts from Pipeline A.
Pipeline that is a stage of a parent pipeline

To set up this configuration, go to the parent pipeline (Pipeline A), choose Add Stage, and add a stage of type Pipeline pointing to another pipeline (Pipeline B). In this case, Pipeline B will have access to any artifacts from Pipeline A that are upstream from where it was triggered.

Stages that produce artifacts

Stages can be configured to ‘Produce’ artifacts if they expose the following Stage configuration:

If you are configuring your stages using JSON, the expected artifacts are placed in a top-level expectedArtifacts: [] list.

There are two ways to use this:

To bind artifacts injected into the stage context

If your stage emits artifacts (such as a “Deploy (Manifest)” stage) into the pipeline context, you can match against these artifacts for downstream stages to consume.
To artificially inject artifacts into the stage context

If you are running a stage that does not natively emit artifacts (such as the “Run Job” stage), you can use the default artifact, which always binds to the expected artifact, to be injected into the pipeline each time it is run. Keep in mind: If the matching artifact is empty, it will bind any artifact, and your default artifact will not be used.

A visual explanation

To help explain how artifacts & expected artifacts work, let’s walk through a demo pipeline. To begin, here is the key:

Say we’ve configured the following pipeline:

The pipeline declares that it expects an artifact matching 1 (perhaps a docker image) when the pipeline starts. This is done in the pipeline configuration tab. It also expects an artifact matching 2 in pipeline stage B (perhaps a “Find Artifact from Execution” stage).

The pipeline is triggered by some source (maybe a Webhook) supplying two artifacts:

Artifact 1 is bound, but both incoming artifacts are placed into the trigger, so any downstream stages (in this case all of them) can consume them. The advantage of the expected artifact is that stages can explicitly reference whatever artifact is bound downstream, rather than have to check for existance of the artifact at runtime.

It’s important to keep in mind that artifact 1 was bound when the pipeline started. If we reference the expected artifact downstream (such as in the “Deploy Manifest”) stage shown below, it is using the artifact that was bound when the pipeline first executed, not when the stage shown runs.

When stage B starts executing, it needs to bind expected artifact 2. If, for example, it was a “Find Artifact from Execution” stage, it would do so by looking up the artifact from a another pipeline’s execution, and binding it here.

If stages C or D needed to reference an upstream artifact, they would have different artifacts accessible to them, since they have different upstream stages. For examples, stage D does not have access to artifact 2.

Docs: Types of Artifacts

Mon, 01 Jan 0001 00:00:00 +0000

While an artifact can reference any remote, deployable resource, we have first-class support for in the form of:

Parsing events from other services. For example, reading pub/sub messages from GCR images.
Credentials for downloading artifacts. For example, a GitHub access token to read repository contents.
Integrations with stages that require certain types of artifacts. For example, App Engine can only deploy a Docker image as a custom runtime, never an AMI.

The pages in this section serve to document the format Spinnaker expects these artifacts in, to make custom integrations with them easier. We recommend reading the artifact format first.

For information about how to use artifacts in pipelines, see About Spinnaker Artifacts .