Google Cloud Pub/Sub
Spinnaker pipelines can be triggered using a Google Pub/Sub subscription .
Prerequisites
You need a
Google Cloud Platform
(GCP) project to with the
Pub/Sub API
enabled
,
and
<code>gcloud</code>
installed locally. You can check that gcloud
is installed and authenticated
by running:
gcloud info
A Pub/Sub subscription
The Pub/Sub integration can be used to either:
Start pipelines by sending Pub/Sub messages from a system you own.
Start pipelines using Pub/Sub messages originating from a Google Cloud system, such as Google Cloud Storage , or Google Container Registry .
Configuration and setup for all cases are described below.
Sending your own Pub/Sub messages
First, record the fact that your $MESSAGE_FORMAT
is CUSTOM
, this will be
needed later.
MESSAGE_FORMAT=CUSTOM
You need a topic with name $TOPIC
to publish messages to:
gcloud beta pubsub topics create $TOPIC
This topic needs a a pull subscription named $SUBSCRIPTION
to let Spinnaker
read messages from. It is important that Spinnaker is the only system reading
from this single subscription. You can always create more subscriptions for
this topic if you want multiple systems to recieve the same messages.
gcloud beta pubsub subscriptions create $SUBSCRIPTION --topic $TOPIC
At this point, you can use the publisher guide to learn how to publish messages programmatically to the topic you have created.
Note that your topic messages need to be valid JSON, otherwise an exception
will be raised in the echo
service and your pipeline will not be triggered.
Receiving messages from Google Cloud Storage (GCS)
First, record the fact that your $MESSAGE_FORMAT
is GCS
, this will be
needed later.
MESSAGE_FORMAT=GCS
Given that you’ll be listening to changes in a GCS bucket ($BUCKET
), the
following command will create (or use an existing) topic with name $TOPIC
to
publish messages to:
gsutil notification create -t $TOPIC -f json gs://${BUCKET}
Finally, create a pull subscription named $SUBSCRIPTION
to listen to changes
to this topic:
gcloud beta pubsub subscriptions create $SUBSCRIPTION --topic $TOPIC
To understand the format that GCS will publish messages in, please read the reference material .
Receiving messages from Google Container Registry (GCR)
Warning
The GCR message type extracts the image digest by default, but Google App Engine doesn’t support deploying from a digest. So if you’re using artifacts from GCR messages to trigger a GAE deployment, you need to use a custom message template. The template must copy the image’s tag into the artifact’s reference and version fields.First, record the fact that your $MESSAGE_FORMAT
is GCR
, this will be
needed later.
MESSAGE_FORMAT=GCR
Given a project name $PROJECT
, GCR will always try to publish messages to a
topic named projects/${PROJECT}/topics/gcr
for any repositories in
$PROJECT
. To ensure that GCR has a valid topic to publish to, try to create
the following topic:
gcloud beta pubsub topics create projects/${PROJECT}/topics/gcr
If the command fails with “Resource already exists in the project”, then the topic already exists (which is OK).
Finally, create a pull subscription named $SUBSCRIPTION
to listen to changes
to this topic:
gcloud beta pubsub subscriptions create $SUBSCRIPTION \
--topic projects/${PROJECT}/topics/gcr
To understand the format that GCR will publish messages in, please read the reference material .
Credentials
Spinnaker needs a
service
account
to
authenticate as against GCP, with the roles/pubsub.subscriber
role enabled. If
you don’t already have such a service account with the corresponding JSON key
downloaded, you can run the following commands to do so:
SERVICE_ACCOUNT_NAME=spinnaker-pubsub-account
SERVICE_ACCOUNT_DEST=~/.gcp/pubsub-account.json
gcloud iam service-accounts create \
$SERVICE_ACCOUNT_NAME \
--display-name $SERVICE_ACCOUNT_NAME
SA_EMAIL=$(gcloud iam service-accounts list \
--filter="displayName:$SERVICE_ACCOUNT_NAME" \
--format='value(email)')
PROJECT=$(gcloud config get-value project)
gcloud projects add-iam-policy-binding $PROJECT \
--role roles/pubsub.subscriber --member serviceAccount:$SA_EMAIL
mkdir -p $(dirname $SERVICE_ACCOUNT_DEST)
gcloud iam service-accounts keys create $SERVICE_ACCOUNT_DEST \
--iam-account $SA_EMAIL
Warning
It’s possible to restrict access to a subscription by service
account in GCP. If this is how your subscription is configured, you may need
to grant $SA_EMAIL
additional permissions following the instructions on the
Once you have run these commands, your GCS JSON key is sitting in a file
called $SERVICE_ACCOUNT_DEST
.
Editing your Pub/Sub settings
All that’s required are the following values:
# See 'Credentials' section above
SERVICE_ACCOUNT_DEST=
# See 'A Pub/Sub Subscription' section above
MESSAGE_FORMAT=
PROJECT=
SUBSCRIPTION=
# You can pick this name, it's meant to be human-readable
PUBSUB_NAME=my-google-pubsub
First, make sure that Google Pub/Sub support is enabled:
hal config pubsub google enable
Next, add your subscription
hal config pubsub google subscription add $PUBSUB_NAME \
--subscription-name $SUBSCRIPTION \
--json-path $SERVICE_ACCOUNT_DEST \
--project $PROJECT \
--message-format $MESSAGE_FORMAT
There are more options described here if you need more control over your configuration.