THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY

MICHIEL ROOK

@michieltcs

THE SYSTEM - SAN DIEGO

▸ ... or the Big Ball Of Mud

▸ Large legacy monolith

▸ Generates significant income

▸ Slow & complex

▸ Technical debt

@michieltcs

SAN DIEGO FRONTEND

MYSQL DB

SAN DIEGO BACKEND

LOAD BALANCERS / VARNISH

ITBANEN INTERMEDIAIR NATIONALEVACATUREBANK

SAN DIEGO FRONTEND

SAN DIEGO FRONTEND

SAN DIEGO FRONTEND

SAN DIEGO BACKEND

SAN DIEGO BACKEND

SAN DIEGO BACKEND

MEMCACHE FTP EXT. SERVICES

SOLR

@michieltcs

THE SYSTEM - SAN DIEGO

▸ Infrequent, manual releases

▸ Fragile tests

▸ Frequent outages & issues

▸ Frustrated team

▸ Low confidence modifying existing code

@michieltcs

GOALS

▸ Reduce issues

▸ Reduce cycle time

▸ Increase productivity

▸ Increase motivation

REFACTOR? REBUILD?

APPROACH

▸ Strangler pattern

▸ API first

▸ Services per domain object (job, jobseeker, ...)

▸ Migrate individual pages

@michieltcs

ORIGINAL MONOLITH

PROXY

SERVICEORIGINAL MONOLITH

ORIGINAL MONOLITH

SERVICE SERVICE

SERVICE

PROXY

DB

DBDB

DB

DB DB

@michieltcs

APPROACH

▸ Services behind load balancers

▸ Access legacy db’s

▸ Continuous deployment

▸ Docker containers

▸ Frontends are services

@michieltcs

CONTINUOUS EVERYTHING

DEV BUILD / TEST

CONTINUOUS INTEGRATION

@michieltcs

DEV BUILD / TEST ACCEPTANCE PRODUCTION

CONTINUOUS DELIVERY

@michieltcs

DEV BUILD / TEST STAGING PRODUCTION

CONTINUOUS DEPLOYMENT

@michieltcs

WHY CONTINUOUS DEPLOYMENT

▸ Small steps

▸ Early feedback

▸ Reduce cycle time

▸ Reduce risk

▸ Experiments!

@michieltcs

SUCCESSFUL TEAMS HAVE

@michieltcs

ONLY COMMIT TO MASTER

EVERY COMMIT GOES TO PRODUCTION

PAIR PROGRAMMING

QUALITY GATES

FEATURE TOGGLES, A/B TESTS

@michieltcs

DASHBOARDS

DEVOPS

AUTOMATE REPEATABLE THINGS

CONTINUOUS TESTING

CONTINUOUS TESTING

UNIT TESTS

INTEGRATION TESTS

ACCEPTANCE TESTS

UI TESTS

SMOKETESTS

Cost Speed

Exploratorytesting Monitoring

@michieltcs

PIPELINE AS CODE

node { stage 'Run tests' sh "phpunit" sh "behat" stage 'Build docker image' sh "phing build" sh "docker build -t jobservice:${env.BUILD_NUMBER} ." sh "docker push jobservice:${env.BUILD_NUMBER}" stage 'Deploy acceptance' sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER} -i acc deploy.yml" stage 'Deploy production' sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER} -i prod deploy.yml" }

@michieltcs

DEPLOYING

PULL IMAGE

START NEW CONTAINER

WAIT FOR PORT

SMOKE TESTS / HEALTH CHECKS

ADD NEW CONTAINER TO LB

REMOVE OLD CONTAINER FROM LB

STOP OLD CONTAINER

docker pull

@michieltcs

DEPLOYING

PULL IMAGE

START NEW CONTAINER

WAIT FOR PORT

SMOKE TESTS / HEALTH CHECKS

ADD NEW CONTAINER TO LB

REMOVE OLD CONTAINER FROM LB

STOP OLD CONTAINER

docker run

@michieltcs

DEPLOYING

PULL IMAGE

START NEW CONTAINER

WAIT FOR PORT

SMOKE TESTS / HEALTH CHECKS

ADD NEW CONTAINER TO LB

REMOVE OLD CONTAINER FROM LB

STOP OLD CONTAINER

wait_for: port=8080 delay=5 timeout=15

@michieltcs

DEPLOYING

PULL IMAGE

START NEW CONTAINER

WAIT FOR PORT

SMOKE TESTS / HEALTH CHECKS

ADD NEW CONTAINER TO LB

REMOVE OLD CONTAINER FROM LB

STOP OLD CONTAINER

uri: url: http://localhost:8080/_health status_code: 200 timeout: 30

@michieltcs

DEPLOYING

PULL IMAGE

START NEW CONTAINER

WAIT FOR PORT

SMOKE TESTS / HEALTH CHECKS

ADD NEW CONTAINER TO LB

REMOVE OLD CONTAINER FROM LB

STOP OLD CONTAINER

template: src=haproxy.cfg.j2 dest=/etc/haproxy/haproxy.cfg

service: name=haproxy state=reloaded

@michieltcs

DEPLOYING

PULL IMAGE

START NEW CONTAINER

WAIT FOR PORT

SMOKE TESTS / HEALTH CHECKS

ADD NEW CONTAINER TO LB

REMOVE OLD CONTAINER FROM LB

STOP OLD CONTAINER

template: src=haproxy.cfg.j2 dest=/etc/haproxy/haproxy.cfg

service: name=haproxy state=reloaded

@michieltcs

DEPLOYING

PULL IMAGE

START NEW CONTAINER

WAIT FOR PORT

SMOKE TESTS / HEALTH CHECKS

ADD NEW CONTAINER TO LB

REMOVE OLD CONTAINER FROM LB

STOP OLD CONTAINER

docker stop

docker rm

@michieltcs

BUILD PIPELINE

@michieltcs

FEEDBACK!

RESULTS

RESULTS

▸ Total build time per service < 10 minutes

▸ Significantly improved page load times

▸ Improved audience stats (time on page, pages per session, session duration, traffic, seo ranking, etc)

▸ Increased confidence and velocity

▸ Experimented with new tech/stacks (angular, jvm, event sourcing)

▸ More fun

@michieltcs

LESSONS LEARNED

▸ Team acceptance

▸ Change is hard

▸ Mentality / discipline

▸ Docker stability/orchestration

▸ Issues with traffic between Amazon <-> on-premise datacenter

@michieltcs

LESSONS LEARNED

▸ Experience with new tech

▸ Stability of build pipelines

▸ Business alignment

▸ Limit feature toggles

▸ Keep focus on replacing legacy application

@michieltcs

Turn-key Continuous Deployment

Zero downtime deployments

Modern, autoscaling infrastructure with built-in monitoring

Pipeline in five minutes

THANK YOU!

@michieltcs / michiel@make.io

Turn-key Continuous Deployment