CI/CD with Kubernetes

Sandeep Parikh@crcsmnkyHead of Solutions, US EastGoogle Cloud Platform

Why Is This ImportantBulletproof process to go from code to production

Centralized, repeatable building/tagging/pushing

Iterate quickly and reliably

Hands off my kubectl

Continuous Integration

TestCode Commit Push TestBuild

The ideal development flow

Continuous Integration

TestCode Commit Push TestBuild

The ideal development flow

TestD’oh!Code Commit Push Test

Crap!Build

Whoops!

The real development flow

Continuous Delivery

TestBuild Deploy

Testing

Staging

Continuous Deployment

DeployApproveDeliver $$$

Why Continuous?Fail fast

The build broke!

Triage fast

Why did the build break?

Deliver faster

Better software in customers hands!

Tools

Jenkins

Kubernetes Cluster (GKE)

Kubernetes Cluster

Node 1 Node 3Node 2

Kubernetes Cluster (GKE)

Jenkins Leader

Node 1 Node 3Node 2

Jenkins Leader

Kubernetes Cluster (GKE)

Jenkins Service

Node 1 Node 3Node 2

Jenkins Leader

Jenkins Service

Kubernetes Cluster (GKE)

Jenkins Ingress

Node 1 Node 3Node 2

Jenkins Leader

Jenkins Service

Google Cloud Load Balancer

Kubernetes Cluster (GKE)

Jenkins Builders

Node 1 Node 3Node 2

Jenkins Leader

Jenkins Builder

Jenkins Builder

Jenkins Builder

Jenkins Builder

Jenkins Service

Jenkins Builder

Google Cloud Load Balancer

Microservices

Backend

Frontend

Kubernetes Cluster (GKE)

Node 1

Microservices

Node 2

Kubernetes Cluster (GKE)

Node 1

Microservices

Node 2

BackendBackend Backend

Kubernetes Cluster (GKE)

Microservices

Node 2Node 1

Backend

Backend

Service

Backend Backend

Google Cloud Load Balancer

Kubernetes Cluster (GKE)

Microservices

Node 2Node 1

Backend

Frontend Frontend

Backend

Service

Frontend

Backend Backend

Google Cloud Load Balancer

Kubernetes Cluster (GKE)

Microservices

Node 2Node 1

Backend

Frontend Frontend

Backend

Service

Frontend

Frontend

Service

Backend Backend

Google Cloud Load Balancer

Microservices & NamespacesVirtual clusters backed by the same physical cluster

Divide cluster resources by different use cases

Supports quotas for managing resources

Test, Staging, Production, etc.

Pods

Time

Microservices & Rolling Update DeploymentsReplace Replication Controller Deployment A with Replication Controller Deployment B by updating one Pod at a time.

A

B

A

B

AB A

B

A

B

A

B

Rolling Update vs. DeploymentsImperative vs Declarative

Rollback

kubectl edit and kubectl apply

Server-side

...

Workflow

Kubernetes Workflow

CreatePushPackage Expose

Mapping To JenkinsWorkflow plugin, Freestyle doesn’t fit

Define a flexible, extensible, script-based CD pipeline

Groovy for scripting actions

Supports human input/approval

node('docker') { checkout scm // Kubernetes cluster info def cluster = 'gtc' def zone = 'us-central1-f' def project = 'REPLACE_WITH_YOUR_PROJECT_NAME' // Run tests stage 'Go tests' docker.image('golang:1.5.1').inside { sh('go get -d -v') sh('go test') } // Build image with Go binary stage 'Build Docker image' def img = docker.build("gcr.io/${project}/gceme:${env.BUILD_TAG}") sh('gcloud docker -a') img.push()

Jenkinsfile

// Deploy image to cluster in dev namespace stage 'Deploy to QA cluster' docker.image('buildpack-deps:jessie-scm').inside { sh('apt-get update -y ; apt-get install jq') sh('export CLOUDSDK_CORE_DISABLE_PROMPTS=1 ; curl https://sdk.cloud.google.com | bash') sh("/root/google-cloud-sdk/bin/gcloud container clusters get-credentials ${cluster} --zone ${zone}") sh('curl -o /usr/bin/kubectl https://storage.googleapis.com/.../release/v1.0.1/bin/linux/amd64/kubectl ; chmod +x /usr/bin/kubectl') sh("kubectl --namespace=staging rollingupdate gceme-frontend --image=${img.id}") sh("kubectl --namespace=staging rollingupdate gceme-backend --image=${img.id}") sh("echo http://`kubectl --namespace=staging get service/gceme --output=json | jq -r '.status.loadBalancer.ingress[0].ip'`> staging") }

Jenkinsfile

// Deploy to prod if approved stage 'Approve, deploy to prod' def url = readFile('staging').trim() input message: "Does staging at $url look good? ", ok: "Deploy to production" sh('gcloud docker -a') img.push('latest') docker.image('buildpack-deps:jessie-scm').inside { sh('apt-get update -y ; apt-get install jq') sh('export CLOUDSDK_CORE_DISABLE_PROMPTS=1 ; curl https://sdk.cloud.google.com | bash') sh("/root/google-cloud-sdk/bin/gcloud container clusters get-credentials ${cluster} --zone ${zone}") sh('curl -o /usr/bin/kubectl https://storage.googleapis.com/.../release/v1.0.1/bin/linux/amd64/kubectl ; chmod +x /usr/bin/kubectl') sh("kubectl --namespace=production rollingupdate gceme-frontend --image=${img.id}") sh("kubectl --namespace=production rollingupdate gceme-backend --image=${img.id}") sh("echo http://`kubectl --namespace=production get service/gceme --output=json | jq -r '.status.loadBalancer.ingress[0].ip'`") }}

Jenkinsfile

Workflow

Repo

Kubernetes Cluster (GKE)Github

Jenkins

µservice

µservice

Default Staging

Production

Workflow

Repo

Kubernetes Cluster (GKE)Github

Jenkins

µservice

µservice

Default Staging

Production1

Push1

Workflow

Repo

Kubernetes Cluster (GKE)Github

Jenkins

µservice

µservice

Default Staging

Production1 2

Push

Build

1

2

Workflow

Repo

Kubernetes Cluster (GKE)Github

Jenkins

µservice

µservice

Default Staging

Production1 2

3

Push

Build

Clone

1

2

3

Workflow

Repo

Kubernetes Cluster (GKE)Github

Jenkins

µservice

µservice

Default Staging

Production1 2

3

4

Push

Build

Clone

Stage

1

2

3

4

Workflow

Repo

Kubernetes Cluster (GKE)Github

Jenkins

µservice

µservice

Default Staging

Production1 2

3

5

4

Push

Build

Clone

Stage

Approve

1

2

3

4

5

Workflow

Repo

Kubernetes Cluster (GKE)Github

Jenkins

µservice

µservice

Default Staging

Production1 2

3

5

4

Push

Build

Clone

Stage

Approve

Deploy

1

2

3

4

5

6

6

Questions, Comments@crcsmnkyRepo

https://github.com/GoogleCloudPlatform/continuous-deployment-on-kubernetesUpdates coming (built pre-Jenkins 2.0, which includes Workflow)Changes include Ingress, Deployments, Jenkins+K8S Plugin (auto-spawn builders)Docs

Automated Image Builds with Jenkins, Packer, and KubernetesDistributed Load Testing using KubernetesReal-time data analysis with Kubernetes, Google Cloud Pub/Sub, and BigQuery

Resources

Join the Google Cloud Platform User Group