Kubernetes automation in production

@pbakker#Kubernetes

Kubernetes Automation

Paul Bakker @pbakker paulbakker.io

@pbakker

Paul Bakker Software architect at Luminis Technologies

Why Kubernetes

• Run Docker in clusters

• scheduling containers on machines

• networking

• storage

• automation

The basics

Docker container

Docker container

Docker containerDocker

container

Node

Docker container

Docker container


container

Pods

Master

Node

Pods

API

etcdetcdetcd

Docker container

Docker container


container

Node

Docker container

Docker container


container

Pods

Docker container

Docker container

Docker containerReplication Controller

Master

schedules

schedules

Node

Pods

nginx

web files

Pod

• May contain multiple containers

• Lifecycle of these containers bound together

• Containers in pod see each other on localhost

• Env vars for services

pod

REDIS_SERVICE_HOST=10.201.159.165 REDIS_PORT_6379_TCP_PORT=6379

Container

Container

Networking

• We run many pods on a single machine

• Pods may expose the same ports

• How to avoid conflicts!?

Dynamic IP addresses

• Each pod gets a virtual IP

• Ports not shared with other pods

pod

pod

Docker container

Docker container

Docker containerService

Services

Fixed, virtual IP address

Dynamic IP address

Dynamic IP address

Multi component deployments

• Each component deployed as a pod

• Individually update and scale pods

• Use services for component communication


frontendbackend service 1

backend service 2

Redis

pod pod

pod

pod

backend service 1backend service 1backend service 1


serv i c e

serv i c e

serv i c e



backend service 2

Redis

pod pod

pod

pod



serv i c e

serv i c e

serv i c e

application



backend service 2

Redis

pod pod

pod

pod



serv i c e

serv i c e

serv i c e

component / service

Namespaces

pod

service

rcrcrcpodpod

serviceservice

pod

service

rcrcrcpodpod

serviceservicepod

service

rcrcrcpodpod

serviceservice

Namespace A

Namespace B

Namespace C

kubectl

kubectl create -f my-rc.yml

kubectl create -f my-service.yml

apiVersion: v1 kind: ReplicationController metadata: name: nginx spec: replicas: 3 selector: app: nginx template: metadata: name: nginx labels: app: nginx spec: containers: - name: nginx image: nginx ports: - containerPort: 80


On how many nodes should this run?



Describes our Docker container Ports, storage needs, etc.



Labels, this (loosely) couples controllers,

pods and services together

Describes our Docker container Ports, storage needs, etc.

HTTP Load balancing

HTTP load balancing

• Expose Kubernetes services to the outside world

• SSL offloading

• Gzip

• Redirects

Kubernetes ingress

• Built in support for GCE load balancers

• Future support for extensions (not quite there yet)

• What about your own environment!?

Using a custom load balancer

• Use Ha-proxy in front of Kubernetes

• Configure Ha-proxy dynamically

• The same works for nginx, apache…

pod

pod

Docker container

Docker container


Dynamic IP address

Dynamic IP address

Load balancer node

ha-proxyHTTPS

SSL offloading

pod

pod

Docker container

Docker container


Dynamic IP address

Dynamic IP address

Load balancer node

ha-proxyHTTPS

AWS

E LB

SSL offloading

pod

pod

Docker container

Docker container


Dynamic IP address

Dynamic IP address

Load balancer node

ha-proxyHTTPS

AWS

E LB

Virtual private network

How does ha-proxy know about our services?

• Ha-proxy uses a static config file

• Auto-generate it based on data in etcd

• Confd

Automation

Using the API

• /v1/namespaces/mynamespace/pods

• /v1/namespaces/mynamespace/services

• /v1/namespaces/mynamespace/replicationcontrollers

REST API that gives access to everything

Client libraries

• Amdatu Kubernetes OSGi

• Amdatu Kubernetes Go

• Clojure, Node, Python etc…

kubernetes.listNodes().subscribe(nodes -> { nodes.getItems() .forEach(System.out::println); });

pods, err := kubernetes.ListPods(TEST_NAMESPACE) if err != nil { panic(err) }

for _,pod := range pods.Items { log.Println(pod.Name) }

Java

Go

Blue-green deployment

• Deployment without downtime

• Only one version is active at a time

• Rolls back on failed deployment

Docker container

Docker container

Docker container

pod v1

ha-proxyHTTPS

Docker container

Docker container

Docker container

pod v1

ha-proxyHTTPS

deploy new versionv2v2v2pod v2

deployer

Docker container

Docker container

Docker containerv1

ha-proxyHTTPS

health check…v2v2v2v2

deployer

Docker container

Docker container

Docker containerv1

ha-proxyHTTPS

v2v2v2v2

confd

Update configdeployer

v1

ha-proxyHTTPS

v2v2v2v2

v1v1v1

ha-proxyHTTPS

v2v2v2v2

Deployer

The Deployer

Kubernetes API

Deployer

Create RC

The Deployer

Kubernetes API

Deployer

pod pod pod pod

Create RC

service

Creates

The Deployer

Kubernetes API

Deployer

pod pod pod pod

GET /health Create RC

service

Creates

The Deployer

Kubernetes API etcd

Deployer

pod pod pod pod


confdWatch

Switch Load Balancer Backend

service

Creates

The Deployer

Kubernetes API

HAProxy

etcd

Deployer

pod pod pod pod


generate config

confdWatch

Switch Load Balancer Backend

service

Creates

The Deployer

Deployer

Kubernetes API

Deployer

1- Create RC

Kubernetes API

Deployer

pod pod pod pod

1- Create RC

service

2- Creates

Kubernetes API

Deployer

pod pod pod pod

3- GET /health 1- Create RC

service

2- Creates

Kubernetes API etcd

Deployer

pod pod pod pod


confd5- Watch

4- Switch Load Balancer Backend

service

2- Creates

Kubernetes API

HAProxy

etcd

Deployer

pod pod pod pod


6- generate config

confd5- Watch

4- Switch Load Balancer Backend

service

2- Creates

Amdatu Kubernetes Deployer

• Kubernetes deployment orchestration

• Load balancer configuration

• Blue-green deployment

• Apache licensed

• Go

{ "deploymentType": "blue-green", "namespace": "default", "useHealthCheck": true, "newVersion": "#", "appName": "cloudrti-demo", "replicas": 2, "frontend": "cloud-rti-demo.amdatu.com", "podspec": {} }

Amdatu Deploymentctl

• UI for setting up deployments

• Deployment history

• Webhooks for triggering from external events

• OSGi / Vertx / Angular 2

Build / deploy pipelinesBuild Server

Docker Hub

builds image

alpha

Deployer

webhook

deploys

Scaling

Kubernetes node

How to scale a Kubernetes cluster?

Kubernetes node

pod pod pod

pod pod pod



Kubernetes node

pod pod pod

pod pod pod

pod pod pod

pod pod pod


Kubernetes nodeKubernetes nodeKubernetes node

pod pod pod

pod pod pod

pod pod pod

pod pod pod

pod pod pod

pod pod pod

pod pod pod

pod pod pod



pod pod pod

pod pod pod

pod pod pod

pod pod pod

pod pod pod

pod pod pod

pod pod pod pod pod pod


Kubernetes nodeKubernetes node

pod pod pod

pod pod pod

pod pod pod

pod pod pod

pod pod pod

pod pod pod

pod pod pod pod pod pod

Scaling up

1. Use AWS API to start new nodes (ScalingGroup)

2. Cloud-init to register node to Kubernetes cluster

Scaling down

1. Set node to “unschedulable”

2. Drain node (relocate pods to other machines)

3. Remove node from Kubernetes

4. Use AWS API to terminate nodes (ScalingGroup)

Amdatu scalerd

• CLI to add/remove nodes to a cluster

• Node draining to prevent downtime

• Scheduled automated scaling

{ "name": "night", "cron": "0 0 21 * * *", "description": "Switch to half capacity at night", "desiredCapacity": 2, "appScaleTemplates": [ { "app": "demo", "replicationControllerScaleTemplates": [ { "replicationController": "*", "replicas": 1 } ] } ] }

scalerctl create nighttime.json

How and where to run these tools?

• In Kubernetes of course!

• Bootstrap using kubectl scripts

MasterAPI

etcdetcdetcd

Kubernetes Node

Kubernetes Node

Kubernetes Node

Kubernetes Node

HA-Proxy

VPN

MasterAPI

etcdetcdetcd

Kubernetes Node

Kubernetes Node

Kubernetes Node

Kubernetes Node

HA-Proxy

VPN

What about my

database!?

Datastores in Kubernetes

• Kubernetes does have persistent volumes

• Most data stores require lots of tuning

• … don’t auto scale

• … require manual steps to configure cluster

MasterAPI

etcdetcdetcd

Kubernetes Node

Kubernetes Node

Kubernetes Node

Kubernetes Node

HA-Proxy

VPN

etcdetcdmongo

etcdetcdKafka

• Fully managed Kubernetes

• Centralised logging

• Application / cluster monitoring

@YourTwitterHandle#DVXFR14{session hashtag} @pbakker#Kubernetes

Q & A

https://bitbucket.org/amdatulabs

Open source projects:

https://bitbucket.org/amdatulabs

Kubernetes automation in production

Software

Transcript of Kubernetes automation in production