Deploy GCS Pub/Sub Artifacts to Google App Engine
The goal of this codelab is to trigger a Spinnaker pipeline with a Pub/Sub message from GCS upon upload of a tarball.
In this codelab, you will deploy artifacts to Google App Engine (GAE) via a Spinnaker pipeline. The pipeline is configured to trigger on Pub/Sub messages from a GCS bucket serving as an artifact repository. The Pub/Sub messages contain context that allows Spinnaker to parse and deploy the stored artifacts.
This codelab takes about an hour.
This codelab assumes you have a billing-enabled GCP project. Also, install gcloud if you haven’t already.
Set up your environment
Create a GCS bucket to store artifacts
In the environment where you have
gcloud installed, run the following commands:
$ BUCKET_NAME=gs://<some_name> && PROJECT_ID=<project_id> $ gcloud auth login $ gsutil mb -p $PROJECT_ID $BUCKET_NAME
Set up Google Cloud Pub/Sub to listen to bucket object changes
GCP documentation includes the steps for configuring GCS to publish Pub/Sub messages, but here is a summary.
Enable the Cloud Pub/Sub API .
Name your topic and subscription:
Create the GCS Pub/Sub notification:
gsutil notification create -t $TOPIC_NAME -f json $BUCKET_NAME.
gsutil notification list $BUCKET_NAME.
Create a pull subscription:
gcloud pubsub subscriptions create $SUBSCRIPTION_NAME --topic $TOPIC_NAME.
Configure and deploy your Spinnaker instance
Configure Spinnaker to deploy to GAE
Note that while configuring the GAE cloud provider, you will create a service account with
enabled and set two environment variables:
In addition to the
roles/storage.admin, the service account also needs
Add the role to the service account:
gcloud projects add-iam-policy-binding $PROJECT \ --role roles/pubsub.subscriber \ --member serviceAccount:$SA_EMAIL # service account email from GAE setup.
Optionally generate a new service account key:
rm $SERVICE_ACCOUNT_DEST && gcloud iam service-accounts keys create $SERVICE_ACCOUNT_DEST \ --iam-account $SA_EMAIL
We’ll need this service account later, so keep these environment variables handy.
Configure Spinnaker to listen to the Google Cloud Pub/Sub subscription
First, configure your GCS artifact provider.
Enable artifact support .
Enable the GCS artifact provider:
hal config artifact gcs enable
Add Spinnaker configuration for the GCS artifact provider’s account:
hal config artifact gcs account add --json-path $SERVICE_ACCOUNT_DEST my-artifact-account
Note that we’re using the environment variable set in configuring the GAE provider here.
Now configure Spinnaker to receive messages from your Google Cloud Pub/Sub subscription.
Enable Google Pub/Sub:
hal config pubsub google enable
Add your subscription to Google Pub/Sub:
hal config pubsub google subscription add --project $PROJECT_ID --json-path $SERVICE_ACCOUNT_DEST --subscription-name $SUBSCRIPTION_NAME --message-format GCS my-gcs-subscription
Deploy Spinnaker with Halyard
Select a Spinnaker version:
hal config version edit --version <version>. List the available versions with
hal version list.
sudo hal deploy apply
Wait a few minutes for the deploy to complete.
Configure your pipeline
Create an application:
a. Navigate to the Spinnaker UI by going to
localhost:9000in a browser.
b. Create a new Spinnaker application by selecting
c. Give the application a name and an admin email. If you have more than one cloud provider configured, add ‘appengine’ to the
d. Click the check box beside ‘Consider only cloud provider health when executing tasks’ and create the application.
Create a new pipeline and add an expected artifact.
a. Select the
Pipelinestab in the Spinnaker UI and click
Createto create a new pipeline.
b. After naming the pipeline, you will be brought to the pipeline configuration screen.
c. Add an artifact under the
Expected Artifactssection of the pipeline configuration.
GCSas the artifact type.
e. In the
Object pathtext box, put
gs://$BUCKET_NAME/app.tarwith the explicit value for the bucket name (Spinnaker won’t know about your environment variables).
Configure a pipeline trigger.
a. In the
Automated Triggersconfiguration section, add a new trigger.
b. Select ‘Pub/Sub’ as the trigger type.
c. Select ‘google’ as the
Pub/Sub System Typeand select ‘my-gcs-subscription’ as the
d. Add an attribute constraint with key ‘eventType’ and value ‘OBJECT_FINALIZE’. This prevents your pipeline from triggering twice from one GCS event.
e. Select the ‘…/app.tar’ expected artifact in the
Expected Artifactsdrop down.
f. Save the pipeline with the
Save Changesbutton in the bottom right corner of the pipeline configuration screen.
Create and configure a deploy stage in your pipeline.
Add stagenear the top of the configuration screen to add a new stage.
b. Select ‘Deploy’ as the stage
Add server groupto open the server group configuration modal.
Basic Settings, select ‘GCS’ as the
d. Select ‘my-artifact-account’ as the
e. Select ‘via pipeline artifact’ in
f. Select the ‘…/app.tar’ artifact in the
Expected Artifactdrop down.
Config Files, add ‘app.yaml’ in the
Package and upload your application to GCS
Acquire a “Hello World” application and upload to GCS
Clone a sample GAE application:
git clone https://github.com/GoogleCloudPlatform/python-docs-samples.git
Package your application as a tarball:
cd python-docs-samples/appengine/standard/hello_world; tar -cvf app.tar *
Upload the tarball to the GCS bucket:
gsutil cp app.tar $BUCKET_NAME