LEVERAGINGLEVERAGING

EPHEMERAL NAMESPACESEPHEMERAL NAMESPACES

IN A CI/CD PIPELINEIN A CI/CD PIPELINE

Can Yücel (@canthefason)

Senior Software EngineerKubeCon EU 2016

HighlightsHighlights

Fundamentals of namespaces

Breaking the idea of having separate clusters

Ephemeral namespaces

Talk about some Kubernetes early stage features

Running every single piece as Kubernetes components

NamespacesNamespaces

“ A namespace is a mechanism to partition resourcescreated by users into a logically named group.

~ Kubernetes Docs

Isolation on Different LevelsIsolation on Different Levels

Network level isolation

Access policies

Resource control

Network Level IsolationNetwork Level IsolationLeveraging subdomainsLeveraging subdomains

Access PoliciesAccess Policies{"user":"admin"}{"user":"scheduler", "readonly": true, "resource": "pods"}{"user":"scheduler", "resource": "bindings"}{"user":"proxy", "resource": "services"}{"user":"proxy", "resource": "endpoints"}{"user":"kubelet", "resource": "pods"}{"user":"kubelet", "resource": "nodes"}{"user":"kubelet", "readonly": true, "resource": "services"}{"user":"kubelet", "readonly": true, "resource": "endpoints"}{"user":"kubelet", "resource": "events"}{"user":"bob", "readonly": true, "namespace": "prod"}{"user":"alice", "namespace": "prod"}

policy.jsonlABAC provides much more granularity on policy

management

Resource ControlResource Control

apiVersion: v1kind: ResourceQuotametadata: name: quotaspec: - hard: memory: "1Gi" cpu: 20 pods: 15 services: 5 replicationcontrollers: 10 resourcequotas: 1

Cluster:32 GB RAM, and 16 cores

Team A:20 GB RAM, and 10 cores

Team B:10 GB RAM, and 4 cores

How to Partition?How to Partition?

Environment based partitioningEnvironment based partitioning

qa, stage, production...

System / team based partitioning System / team based partitioning

kube-system, devops, bots

Project based partitioningProject based partitioning

example.com, better-example.com

A Day of a CI/CD PipelineA Day of a CI/CD Pipeline

Provision separate machines for every build

Run your tests on isolated clusters

When all tests are successful tear down the cluster

If it fails keep the cluster up for a while for debugging

Ephemeral Namespaces!Ephemeral Namespaces!

namespaces

namespaces

namespace

Ephemeral Namespaces AreEphemeral Namespaces Are

Isolated environments that are running different versionsof services on top of it

The environments where we run our integrations/e2e tests,and gets dumped when we get the end results

Namespaces with Benefits!Namespaces with Benefits!

Time effective provisioning

Efficient resource utilization

In a CI/CD pipeline, namespaces provide:In a CI/CD pipeline, namespaces provide:

Time Effective ProvisioningTime Effective Provisioning

It takes only a couple of seconds to create all

services

Efficient Resource UtilizationEfficient Resource Utilization

Let your scheduler decide on whichhost you will run your test

instances

Deployment ProcessDeployment Process

1. Run your unit tests

2. Build Docker Image

3. Deploy to sandbox

4. Provision services that you will run your

tests against

5. Run your integration/e2e tests

6. Delete namespace

7. Deploy updated services to staging/prod

Happy Path!

Provisioning Test EnvironmentsProvisioning Test Environments

Identical environments with different versions!

Pods From Different NamespacesPods From Different Namespaces

➜ kubectl get po --namespace=e2e-1NAME READY STATUS RESTARTS AGEmongo-oij3f 1/1 Running 0 10mnginx-44k6p 1/1 Running 0 10mselenium-9bcfc 1/1 Running 0 10mtodo-service-phgrb 1/1 Running 0 10mtodo-service-rbrjl 1/1 Running 0 10m

➜ kubectl get po --namespace=e2e-2NAME READY STATUS RESTARTS AGEmongo-p6g8c 1/1 Running 0 5mnginx-mgdzz 1/1 Running 0 5mselenium-9l81p 1/1 Running 0 5mtodo-service-mt9gh 1/1 Running 0 5mtodo-service-yxo9v 1/1 Running 0 5m

➜ kubectl get po --namespace=e2e-3NAME READY STATUS RESTARTS AGEmongo-llm3x 1/1 Running 0 1mnginx-vvov6 1/1 Running 0 1mnightwatch 1/1 Running 0 34sselenium-g2g1i 1/1 Running 0 1mtodo-service-1k8vc 1/1 Running 0 1mtodo-service-ddfjw 1/1 Running 0 1m

Adding E2E Components as PodsAdding E2E Components as Pods

Selenium serverNightwatch.js scripts

All Tests Passed All Tests Passed ✓✓$ kubectl delete namespace e2e-10

It will dump every Kubernetes componentwithin that namespace!

Test Gets Failed Test Gets Failed 😞

Find a way to connect to the Selenium Server fordebugging

Expose VNC Port 5900

kubectl port-forward selenium :5900 --namespace=e2e-1

Live In ActionLive In Action

How We Use GoCDHow We Use GoCD

Idempotent pipeline stages

Dependency management is handled with fan-inresolution

Evaluating DependenciesEvaluating Dependencies

Text

GO_DEPENDENCY_LABEL_E2E=5.2eedd92GO_DEPENDENCY_LOCATOR_E2E=e2e-tests/5/buildImage/1GO_DEPENDENCY_LABEL_TODO=35.86ca86cGO_DEPENDENCY_LOCATOR_TODO=todo-service/35/deployK8s/1GO_DEPENDENCY_LABEL_NGINX=12.4288a7cGO_DEPENDENCY_LOCATOR_NGINX=nginx/12/deployK8s/1

Each GO_DEPENDENCY variable hasdependant pipeline information

For Provisioning Test Environments: Create all dependencies

For Deployment: Compare versions and call create/rollingupdate

Running Every Piece in PodsRunning Every Piece in PodsNightwatch scripts

kubectl run -i -tty nightwatch --image=canthefason/e2e-tests:$E2E_IMAGE_TAG \--restart=Never --namespace=e2e-$GO_PIPELINE_LABEL

state=$(kubectl get -o template po nightwatch $kubeargs \--template={{.status.phase}})

while [ "$state" == "Running" ]; do sleep 5 echo "waiting for the state" state=$(kubectl get -o template po nightwatch $kubeargs \ --template={{.status.phase}})done

echo "State: $state"if [ "$state" == "Failed" ]; then exit 1fi

Running Every Piece in PodsRunning Every Piece in PodsSelenium manifest

apiVersion: v1kind: ReplicationControllermetadata: name: seleniumspec: replicas: 1 selector: app: selenium template: metadata: name: selenium labels: app: selenium spec: volumes: - name: shm hostPath: path: /dev/shm containers: - name: selenium image: selenium/standalone-chrome-debug:2.52.0 ports: - containerPort: 4444 - containerPort: 5900 imagePullPolicy: Always volumeMounts: - name: shm mountPath: /dev/shm

Health CheckersHealth Checkers

Text

curl -k --retry 10 --retry-delay 5 -v \ https://$KUBE_HOST/api/v1/proxy/namespaces/sandbox/services/todo/ping

curl -k --silent --output /dev/stderr --write-out "%{http_code}" -v \https://$KUBE_HOST/api/v1/proxy/namespaces/sandbox/services/todo/ping

if [ "$STATUSCODE" -ne "200" ]; then if [ "$rcExist" != "ReplicationController" ]; then kubectl delete -f scripts/rc.yml $kubeargs fi exit 1fi

Future WorkFuture Work

Scale down the pods when the namespace is idle

Automatically delete namespaces that are olderthan certain age

Build a Selenium Grid infrastructure and utilizeSelenium Agents among the namespaces

TakeawaysTakeaways

Never ever expose your Apiserver 8080 port!

Think twice before defining your ssh keys assecrets!

Make sure that you properly setup kubeletgarbage collectors

--maximum-dead-containers=100 --maximum-dead-containers-per-container=2 --minimum-container-ttl-duration=1m0s

LinksLinks

http://github.com/canthefason/kubecon

https://github.com/kubernetes/contrib

Thanks ToThanks To

Kubernetes Team

LaunchPad Central

Quest Henkart

UK Consulate in NY...

Q & AQ & A

Twitter: @canthefason

GitHub: /canthefason