Continuous Delivery with Containers:The Good, the Bad, and the Ugly

Daniel Bryant @danielbryantuk

Containers: Expectations versus reality

21/05/2018 @danielbryantuk

“DevOps”

@danielbryantuk

• Independent Technical Consultant, Product Architect at Datawire• Architecture, DevOps, Java, microservices, cloud, containers

• Continuous Delivery (CI/CD) advocate

• Leading change through technology and teams

21/05/2018 @danielbryantuk

Setting the scene…

• Continuous delivery is a large topic• No business focus today (value stream etc)• PaaS and Serverless are super interesting…• But I’m assuming you’re all-in on containers

• Focusing today on the process and tooling• No live coding today• Mini-book contains more details (thanks nginx!)

21/05/2018 @danielbryantuk

bit.ly/2jWDSF7

TL;DR – Containers and CD

• Container image becomes the build pipeline ‘single binary’

• Adding metadata to containers images is vital, but challenging

• Must validate container constraints on system quality attributes (NFRs)

21/05/2018 @danielbryantuk

Continuous Delivery 101

21/05/2018 @danielbryantuk

Continuous Delivery

• Produce valuable and robust software in short cycles

• Optimising for feedback and learning

• Not (necessarily) Continuous Deployment

21/05/2018 @danielbryantuk

Velocity (with stability) is key to business success

“Continuous delivery is achieved when stability and speed can satisfy business demand.

Discontinuous delivery occurs when stability and speed are insufficient.”

- Steve Smith (@SteveSmithCD)

21/05/2018 @danielbryantuk

Creation of a build pipeline is mandatory for continuous delivery

21/05/2018 @danielbryantuk

21/05/2018 @danielbryantuk

Feedback: - Was our initial

hypothesis proven?- How can we improve

business, architecture and ops?

Stability

21/05/2018 @danielbryantuk

21/05/2018 @danielbryantuk

Speed

21/05/2018 @danielbryantuk

21/05/2018 @danielbryantuk

Feedback: - Was our initial

hypothesis proven?- How can we improve

business, architecture and ops?

The impact of containers on CD

21/05/2018 @danielbryantuk

Container technology (and CD)

• OS-level virtualisation• cgroups, namespaces, rootfs

• Share the OS kernel

• Package and execute software

• Container image == ‘single binary’

21/05/2018 @danielbryantuk

21/05/2018 @danielbryantuk

21/05/2018 @danielbryantuk

Should I build my own container platform?

Probably not(Unless you are Google, AWS or IBM)

Whatever you decide…push it through a pipeline ASAP!

21/05/2018 @danielbryantuk

But what about microservices?

21/05/2018 @danielbryantuk

21/05/2018 @danielbryantuk

Independent service deployment

PATTERN

https://www.slideshare.net/dbryant_uk/deliveragile-2018-continuous-

delivery-patterns-for-modern-architectures

Working Locally

21/05/2018 @danielbryantuk

21/05/2018 @danielbryantuk

Make your dev environment like production

• Develop locally or copy/code in container

• Must build/test containers locally• Perform (at least) happy path tests

• Use identical base images from production• With same configuration

21/05/2018 @danielbryantuk

Quick digression: Working remotely, locally…

21/05/2018 @danielbryantuk

https://opencredo.com/working-locally-with-microservices/

https://www.telepresence.io/

Make your dev environment like production

• Develop locally or copy/code in container

• Must build/test containers locally• Perform (at least) happy path tests

• Use identical base images from production• With same configuration

21/05/2018 @danielbryantuk

Lesson learned: Dockerfile content is super important

• OS choice (distroless?)

• Configuration

• Build artifacts

• Exposing ports, user

• Java• JDK vs JRE and Oracle vs OpenJDK?

• Golang• Statically compiled binary in scratch?

• Python• Virtualenv?

21/05/2018 @danielbryantuk

Please talk to the sysadmin people:Their operational knowledge is invaluable

21/05/2018 @danielbryantuk

Different test and prod containers?

• Create “test” version of container

• Full OS (e.g. Ubuntu)

• Test tools and data

• Easy to see app/configuration drift

• Use test sidecar containers instead

• ONTEST proposal by Alexi Ledenev

21/05/2018 @danielbryantuk

http://blog.terranillius.com/post/docker_testing/

Docker multi-stage builds

21/05/2018 @danielbryantuk

https://docs.docker.com/develop/develop-images/multistage-build/https://github.com/GoogleContainerTools/distroless

Building Artifacts

21/05/2018 @danielbryantuk

21/05/2018 @danielbryantuk

Building images with Jenkins

• My report covers this

• Build as usual…

• Build Docker Image• Cloudbees Docker Build and Publish Plugin

• Push image to registry

21/05/2018 @danielbryantuk

Building in the Cluster?

21/05/2018 @danielbryantuk

https://github.com/GoogleContainerTools/kanikohttps://www.infoq.com/news/2018/03/jenkins-x-kubernetes

Deployment

21/05/2018 @danielbryantuk

https://blog.hasura.io/draft-vs-gitkube-vs-helm-vs-ksonnet-vs-metaparticle-vs-skaffold-f5aa9561f948

Building Artifacts: Metadata

21/05/2018 @danielbryantuk

Lesson learned: Metadata is valuable

• Application metadata• Version / GIT SHA

• Build metadata• Build date• Image name• Vendor

• Quality metadata• QA control, signed binaries, ephemeral support• Security profiles (AppArmor), Security audited etc

21/05/2018 @danielbryantuk

Metadata – Beware of “latest” Docker Tag

• Beware of the ‘latest’ Docker tag

• “Latest” simply means • the last build/tag that ran without

a specific tag/version specified

• Ignore “latest” tag• Version your tags, every time• danielbryantuk/test:2.4.1

21/05/2018 @danielbryantuk

Metadata - Adding Labels at build time

• Docker Labels

• Add key/value data to image

21/05/2018 @danielbryantuk

Metadata - Adding Labels at build time

• Microscaling Systems’ Makefile

• Labelling automated builds on DockerHub (h/t Ross Fairbanks)• Create file ‘/hooks/build’

• label-schema.org • microbadger.com

21/05/2018 @danielbryantuk

Metadata - Adding Labels at runtime

21/05/2018 @danielbryantuk

$ docker run -d --labeluk.co.danielbryant.lbname=frontdoor nginx

• Can ’docker commit’, but creates new image

• Not possible to update running container

• Docker Proposal: Update labels #21721

External registry with metadata support

21/05/2018 @danielbryantuk

New Solution: Grafeas + Kritis

21/05/2018 @danielbryantuk

https://github.com/grafeas/grafeashttps://www.infoq.com/news/2018/05/grafeas-kritis-security

Grafeas Metadata “kinds” (Schema)

21/05/2018 @danielbryantuk

21/05/2018 @danielbryantuk

Grafeas Metadata “kinds” (Schema)

21/05/2018 @danielbryantuk

21/05/2018 @danielbryantuk

Quality Assurance

21/05/2018 @danielbryantuk

21/05/2018 @danielbryantuk

Validating Container Structure

21/05/2018 @danielbryantuk

https://github.com/GoogleContainerTools/container-structure-testhttps://github.com/GoogleContainerTools/container-diff

Quality Assurance: Functional

21/05/2018 @danielbryantuk

Testing (in Production??)

21/05/2018 @danielbryantuk

martinfowler.com/bliki/TestPyramid.htmlhttps://medium.com/@copyconstruct/testing-microservices-the-sane-way-9bb31d158c16

Component testing

21/05/2018 @danielbryantuk

Testing: Jenkins Pipeline (as code)

21/05/2018 @danielbryantuk

21/05/2018 @danielbryantuk

Testing individual containers

21/05/2018 @danielbryantuk

Integration testing

21/05/2018 @danielbryantuk

Introducing Docker Compose

21/05/2018 @danielbryantuk

Quality Assurance: Nonfunctional

21/05/2018 @danielbryantuk

Testing NFRs in the build pipeline

• Architecture

• Performance and Load testing • Gatling / jmeter / Flood.io

• Security testing • Findsecbugs / OWASP Dependency check• Bdd-security (OWASP ZAP) / Arachni• Gauntlt / Serverspec• Docker Bench for Security / CoreOS Clair

21/05/2018 @danielbryantuk

Architectural Visibility

21/05/2018 @danielbryantuk

Quick digression: Testing Architecture

21/05/2018 @danielbryantuk

https://www.archunit.org/

Performance/soak testing

21/05/2018 @danielbryantuk

Mechanical sympathy: Docker and Java

• Watch for JVM cgroup/taskset awareness (with JDK <= 8) • getAvailableProcessors() may incorrectly report the number of cpus in Docker (JDK-8140793)

• Runtime.availableProcessors() ignores Linux taskset command (JDK-6515172)

• Default fork/join thread pool sizes (and others) is based from host CPU count

• Set container memory appropriately • JVM requirements = Heap size (Xmx) + Metaspace + JVM overhead• Account for native thread requirements e.g. thread stack size (Xss)

• Entropy • Host entropy can soon be exhausted by crypto operations and /dev/random blocks• -Djava.security.egd=file:/dev/./urandom (notes on this)

21/05/2018 @danielbryantuk 64

Security Visibility: Basic Code Scanning

21/05/2018 @danielbryantuk

Dependency Scanning

21/05/2018 @danielbryantuk

www.owasp.org/index.php/OWASP_Dependency_Check

Static Image Scanning

21/05/2018 @danielbryantuk

github.com/arminc/clair-scanner

Verifying Container/Platform Security

21/05/2018 @danielbryantuk

https://github.com/docker/docker-bench-securityhttps://github.com/aquasecurity/kube-benchhttps://github.com/Shopify/kubeaudit

Delaying NFRs to the ‘Last Responsible Moment’

Newsflash!Sometimes the

last responsible moment is up-front

Modern platforms/architectures don’t necessarily make this easier

21/05/2018 @danielbryantuk

Quality Assurance: Post-Deploy

21/05/2018 @danielbryantuk

Observability is core to continuous delivery

21/05/2018 @danielbryantuk

www.infoq.com/articles/monitoring-containers-at-scale

Canarying and Synthetic Txns

21/05/2018 @danielbryantuk

https://blog.getambassador.io/canary-deployments-a-b-testing-and-microservices-with-ambassador-f104d0458736https://martinfowler.com/bliki/SyntheticMonitoring.html

Choas Engineering/Testing

21/05/2018 @danielbryantuk

https://github.com/asobti/kube-monkey https://www.infoq.com/news/2018/01/powerfulseal-chaos-kubernetes

When bad things happen, people are always involved…

21/05/2018 @danielbryantuk | @oakinger

HealthCare.gov

21/05/2018 @danielbryantuk

Mikey Dickerson's Hierarchy of Reliability

21/05/2018 @danielbryantuk

www.infoq.com/news/2015/06/too-big-to-fail

Containers are not a silver bullet

21/05/2018 @danielbryantuk

Moving to containers: Going all-in?

21/05/2018 @danielbryantuk

OR

Containerise an existing (monolithic) app?

• For

• We know the monolith well

• Allows homogenization of the pipeline and deployment platform

• Can be a demonstrable win for tech and the business

• Against

• Can be difficult (100+ line scripts)

• Often not designed for operation within containers, nor cloud native

• Putting lipstick on a pig?

21/05/2018 @danielbryantuk

Key lessons learned

• Conduct an architectural review • Architecture for Developers, by Simon Brown• Architecture Interview, by Susan Fowler

• Look for data ingress/egress• File system access

• Support resource constraints/transience• Optimise for quick startup and shutdown • Evaluate approach to concurrency• Store configuration (secrets) remotely

21/05/2018 @danielbryantuk

Using containers does not get rid of the need for good architectural practices

21/05/2018 @danielbryantuk

21/05/2018 @danielbryantuk

https://speakerdeck.com/caseywest/containercon-north-america-cloud-anti-patterns

Summary

21/05/2018 @danielbryantuk

In summary

• Continuous delivery is vitally important in modern architectures/ops

• Container images must be the (single) source of truth within pipeline• And metadata added as appropriate…

• Mechanical sympathy is important (assert properties in the pipeline)• Not all developers are operationally aware

• The tooling is now becoming stable/mature• We need to re-apply existing CD practices with new technologies/tooling

21/05/2018 @danielbryantuk

Thanks for listening…

Twitter: @danielbryantukEmail: daniel.bryant@tai-dev.co.uk

Writing: https://www.infoq.com/profile/Daniel-Bryant

Talks: https://www.youtube.com/playlist?list=PLoVYf_0qOYNeBmrpjuBOOAqJnQb3QAEtM

21/05/2018 @danielbryantuk

bit.ly/2jWDSF7

Coming soon!

Bedtime reading

21/05/2018 @danielbryantuk