The Tale of a Docker-based Continuous Delivery Pipeline by Rafe Colton (ModCloth)

doing the old thing the new wayby @rafecolton

brief prologue

“rafe” (rafecolton on the internets)

software engineer, platform @ modcloth

using docker in prod since v0.7.0

*todo: explain “doing the old thing the new way”

obligatory slide with a bunch of logos

the stack that was

the stack that would be

motivations• simplify application architecture

• support a variety of application languages

• make provisioning and deployment more accessible




goals• push-button provisioning and deployment

• consolidated, pluggable platform

• move to linux




goals• push-button provisioning and deployment


• move to linux

bonus points• chatops

• actual button for provisioning and deployment

motivations

goals

bonus points• chatops

• actual button for provisioning and deployment

does docker

facilitate such a

solution?

• simplify application architecture



• push-button provisioning and deployment


• move to linux

the stack that would be

Clear and Simple Statement.

case study: modcloth.com/style-gallery

http://modcloth.com/style-gallery


smartos

nginx ruby

rails

webserver



smartos

nginx ruby

rails

smartos

cron ruby

rails

sidekiq workerswebserver



smartos

nginx ruby

rails

webserver

smartos

cron ruby

rails

sidekiq workers

how complex could

it be?



nginxruby

rails

webserver

ubuntu

docker



nginxruby

rails

webserver

ubuntu

docker

cron



nginxruby

rails

webserver

ubuntu

docker

cron

supervisord



nginxruby

rails

webserver

ubuntu

docker

cron

supervisord

rails

sidekiq

workers



nginxruby

rails

webserver

ubuntu

docker

cron

supervisord

rails

sidekiq

workers

nodejs rsyslogd

nad

sshd


• overall complexity

• maintainability

• image consistency

• container reliability

• log aggregation

• monitoring

challenges

• overall complexity

• maintainability

• image consistency

• container reliability

• log aggregation

• monitoring

challenges

• don’t do the new thing the old way

• consider division of responsibility

lessons


webserver

ruby

rails

ubuntu

docker cron

nginx

sidekiq workers

ruby

rails

nodejs rsyslogd

nad


the stack

observations

docker is an excellent packaging and distribution system

observations

docker is an excellent packaging and distribution system

containers are the canonical building block for a continuous delivery pipeline

• docker/fig

• deis/deis

• flynn/flynn

• coreos/fleet

• ansible/ansible

• opscode/chef

• progrium/dokku

• newrelic/centurion

begin github search…projects for orchestrating containers:

• docker/fig

• deis/deis

• flynn/flynn

• coreos/fleet

• ansible/ansible

• opscode/chef

• progrium/dokku


• mesosphere/marathon

• airbnb/chronos

• GoogleCloudPlatform/kubernetes

• openshift/geard

• VoltFramework/volt

projects for orchestrating containers:

begin github search…

• docker/fig

• deis/deis

• flynn/flynn

• coreos/fleet

• ansible/ansible

• opscode/chef

• progrium/dokku


• mesosphere/marathon

• airbnb/chronos

• GoogleCloudPlatform/kubernetes

• openshift/geard

• VoltFramework/volt

• octohost/octohost

• makeusabrew/decking

• signalfuse/maestro-ng

• shipyard/shipyard

• DevTable/gantryd

• mcuadros/dockership

• longshoreman/longshoreman

• marmelab/gaudi

• etc.

begin github search…projects for orchestrating containers:

• rafecolton/docker-builder

• mitchellh/packer

• swipely/dockly

• ???

begin github search…projects for building containers:

observation

everybody is building containers differently.

observation

everybody is building containers differently.

how do we build production-ready containers?

writing a good Dockerfilelesson 0: getting started

writing a good Dockerfilelesson 0: getting started use a docker hub base

writing a good Dockerfilelesson 0: getting started

set your env

writing a good Dockerfilelesson 1: order matters

deps before bundling


ADD only Gemfile* first


`ADD .` as late as possible

writing a good Dockerfilelesson 2: optimize for size, repeatability

combine RUN commands

whenever possible

writing a good Dockerfilelesson 2: optimize for size, repeatability

RUN dependent

steps together

writing a good Dockerfilelesson 3: use a standard entrypoint

use a *simple*

entrypoint script


operate on docker-specific

environment variables


wrap verbose

CMD options


exec "$@" # give yourself a shell

the image development lifecycle

build

tag

push*type things*

the image development lifecycle

pushbuild

tag

*type things*

> docker build -t myapp:latest .

> export latest="$(docker images | grep myapp:latest | head -n 1 | awk '{print $3}’)"

> docker tag $latest "$(git rev-parse -q HEAD)" # sha

> docker tag $latest "$(git describe --always --dirty --tags)" # tag

> docker tag $latest "$(git rev-parse -q --abbrev-ref HEAD)" # branch

> for image in $(docker images | grep myapp | awk '{print $1 ":" $2}' | head -n 4) ; \

do docker push $image ; done

the image development lifecycle: docker-builder

push

build

tag

*type things*

> docker-builder build .

teh pipeline

docker build

server*

appapp

app

app

*https://github.com/rafecolton/docker-builder

https://github.com/rafecolton/docker-builder

conclusion

building containers is like writing ruby code:

conclusion

building containers is like writing ruby code:

it’s easy to do

it’s hard to do correctly

so what did we learn? (or, through what did you sleep?)

stuff:

• the complexity will come naturally (so don’t force it)

• be intentional about your Dockerfile


stuff:



• docker is an excellent packaging and distribution system

• containers are the canonical building blocks


stuff:



• docker is an excellent packaging and distribution system

• containers are the canonical building blocks

• consider division of responsibility between the host and the container

• don’t do the new thing the old way (do the old thing the new way!)

brief epilogue

goal: move to linux

=> all apps (less one) now employing docker/ansible/linux

brief epilogue

goal: move to linux


goal: consolidated, pluggable platform

=> shared monitoring, log aggregation, & load balancing services

brief epilogue

goal: move to linux




goal: push-button provisioning and deployment

=> it works, minimal magic… and it’s well documented

brief epilogue

goal: move to linux




goal: push-button provisioning and deployment

=> it works, minimal magic… and it’s well documented

could easily be maintained by only two people… dun dun dun

brief epilogue

I’m job hunting…

brief epilogue

I’m job hunting…

…and I haven’t shot anything yet.

so if you’re hiring, come talk to me after the show.

twitter: @rafecolton

github: rafecolton

rafecolton.com

http://rafecolton.com

thank you

The Tale of a Docker-based Continuous Delivery Pipeline by Rafe Colton (ModCloth)

Technology

Transcript of The Tale of a Docker-based Continuous Delivery Pipeline by Rafe Colton (ModCloth)