1. 1. #mesoscon 2015 Steve Hoffman Senior Principal Engineer @bacoboy Enabling Hybrid Workows with Docker/Mesos @Orbitz
2. 2. #mesoscon 2015
3. 3. #mesoscon 2015 Multiple Brands Websites Webservices Multiple Backends 500+ apps / thousands of instances Deployments Daily (sometimes more) If you havent heard of us
4. 4. #mesoscon 2015 Mesos as a Microservices Platform using Docker a Jenkins Build Farm a Jenkins Deployment Farm HA Notes CentOS Notes Questions
5. 5. #mesoscon 2015 Case 1: Docker Microservices Platform Launch Docker apps in Multiple environments (dev -> qa -> staging -> production) Multiple datacenters Update Docker apps in rolling fashion Restart anything that needs it
6. 6. #mesoscon 2015 Build Unit Test Deploy Dev Deploy Prod Deploy Staging Acceptance Test Code Review & Push Production Pre-Production Open RFC Close RFC
7. 7. #mesoscon 2015 App App App
8. 8. #mesoscon 2015 App App App
9. 9. #mesoscon 2015 - tasks: marathon: 1.2.16 1.2.16 1.2.16 Deploy
10. 10. #mesoscon 2015 - tasks: marathon: 1.2.16 1.2.16 1.2.16 Deploy
11. 11. #mesoscon 2015 - tasks: marathon: PUT /apps/editorial-module { image: orbitz/editorial-module:1.2.17 } 1.2.16 1.2.16 1.2.16 Deploy
12. 12. #mesoscon 2015 Deploy PUT /apps/editorial-module { image: orbitz/editorial-module:1.2.17 } - tasks: marathon: 1.2.16 1.2.16 1.2.16 app = GET /v2/apps/editorial-module if not app then deploy_id = POST /v2/apps { image: orbitz/editorial-module:1.2.17, id: editorial-module } else deploy_id = PUT /v2/apps/editorial-module { image: orbitz/editorial-module:1.2.17 } end if while GET /v2/deployments contains deploy_id // still deploying end // deploy complete
13. 13. #mesoscon 2015 - tasks: marathon: PUT /apps/editorial-module { image: orbitz/editorial-module:1.2.17 } 1.2.16 1.2.16 1.2.16 Deploy
14. 14. #mesoscon 2015 - tasks: marathon: 1.2.16 1.2.16 1.2.16 Deploy
15. 15. #mesoscon 2015 - tasks: marathon: 1.2.16 1.2.16 1.2.16 1.2.17 1.2.17 1.2.17 Deploy
16. 16. #mesoscon 2015 - tasks: marathon: /health 1.2.16 1.2.16 1.2.16 1.2.17 1.2.17 1.2.17 Deploy
17. 17. #mesoscon 2015 - tasks: marathon: /health 200 OK 200 OK 200 OK 1.2.16 1.2.16 1.2.16 1.2.17 1.2.17 1.2.17 Deploy
18. 18. #mesoscon 2015 - tasks: marathon: 1.2.17 1.2.17 1.2.17 Deploy
19. 19. #mesoscon 2015 And off to the next environment 1.2.17 1.2.17 1.2.17 Deploy
20. 20. #mesoscon 2015 1.2.17 1.2.17 1.2.17 What if?
21. 21. #mesoscon 2015 1.2.17 1.2.17
22. 22. #mesoscon 2015 1.2.17 1.2.17
23. 23. #mesoscon 2015 1.2.17 1.2.17 1.2.17
24. 24. #mesoscon 2015 /health 200 OK 1.2.17 1.2.17 1.2.17
25. 25. #mesoscon 2015 1.2.17 1.2.17 1.2.17
26. 26. #mesoscon 2015 Video: http://bit.ly/oww-dockercon2015-video Slides: http://bit.ly/oww-microservices-dockercon2015
27. 27. #mesoscon 2015 Case 2: The Build Farm Existing Solution Dedicated Jenkins Slaves Hand created Snapshotted & Rolled Back to Clean state after each Job Hard to Manage Build Environment for 300+ apps across many OSes, Java versions, Ruby versions, perl versions, python versions, protocol buffer compiler versions, etc
28. 28. #mesoscon 2015 Master Slave Commit/Push or Pull Request/Merge Before
29. 29. #mesoscon 2015 Master Slave Poll or Push Trigger Commit/Push or Pull Request/Merge Before
30. 30. #mesoscon 2015 Master Slave Poll or Push Trigger Commit/Push or Pull Request/Merge Before
31. 31. #mesoscon 2015 Master Slave Poll or Push Trigger Commit/Push or Pull Request/Merge Clone & Build Push Artifacts Before
32. 32. #mesoscon 2015 Master Slave Before
33. 33. #mesoscon 2015 Thanks EBay! http://bit.ly/ebay-jenkins-docker
34. 34. #mesoscon 2015 Master Master Slave Commit/Push or Pull Request/Merge After
35. 35. #mesoscon 2015 Master Master Slave Poll or Push Trigger Commit/Push or Pull Request/Merge After
36. 36. #mesoscon 2015 Master Master Slave Slave Poll or Push Trigger Commit/Push or Pull Request/Merge After
37. 37. #mesoscon 2015 Master Master Slave Slave Poll or Push Trigger Commit/Push or Pull Request/Merge After
38. 38. #mesoscon 2015 Master Master Slave Slave Poll or Push Trigger Commit/Push or Pull Request/Merge Clone & Build Push Artifacts After
39. 39. #mesoscon 2015 Master Master SlaveAfter
40. 40. #mesoscon 2015 Jenkins mesos-plugin https://github.com/jenkinsci/mesos-plugin
41. 41. #mesoscon 2015 Plugin Conguration Dont forget to install libmesos on Jenkins master! Point at ZK to nd active master Leave framework running
42. 42. #mesoscon 2015 In Mesos Console Framework Registered
43. 43. #mesoscon 2015 Plugin Conguration Per Slave Type Jenkins Slave Tag Which Mesos Servers this job can use Jenkins Slave Image 1 job per docker slave short timeout for single use RAM/CPU needed
44. 44. #mesoscon 2015 Job Conguration Run on specic Docker slave Mark ofine immediately
45. 45. #mesoscon 2015 Running Docker Slaves
46. 46. #mesoscon 2015 Success! Ephemeral Docker Slave in Mesos its tag Docker IP
47. 47. #mesoscon 2015 Case 2+: The Docker Build Farm I ALSO need to build my jenkins/docker slaves Need a jenkins docker slave to build docker images Can I do Docker-in-Docker-on-Mesos? Will need to build manually the rst time http://kb.sparknearby.com/wp-content/uploads/2015/05/chicken-or-egg-cropped1.jpg
48. 48. #mesoscon 2015 The docker-builder Jenkins Slave (Java - jre) Docker daemon (running & in supervisor mode!) Registry credentials provided to slave via Credentials Binding Plugin from Jenkins managed security https://wiki.jenkins-ci.org/display/JENKINS/Credentials+Binding +Plugin Reuse docker layers (aka share /var/lib/docker)
49. 49. #mesoscon 2015 wrapdocker script Start Docker daemon and then start jenkins slave https://github.com/jpetazzo/dind/blob/master/wrapdocker I had to change (variable substitution wasnt working): [[ $1 ]] && exec "$@" to: [[ $1 ]] && eval exec $@ https://blog.docker.com/2013/09/docker-can-now-run-within-docker/
50. 50. #mesoscon 2015 docker-builder Dockerle FROM docker.orbitz.net/centos:7 MAINTAINER Steve Hoffman # Need to override default YUM repos # and DNS resolution RUN rm /etc/yum.repos.d/* ADD src/repos/*.repo /etc/yum.repos.d/ ADD src/dns/resolv.conf /etc/ RUN # load repo metadata from above yum clean all && yum makecache & install packages (jenkins needs at least # java and git) yum install -y jre1.8.0_51 git docker-engine & update everything not already newer yum update -y # For git to work in bridged mode, we need to setup user identity ADD src/git/gitconfig /root/.gitconfig # Include helper script ADD src/wrapdocker /usr/local/bin/wrapdocker # Mount docker daemon storage VOLUME /var/lib/docker
51. 51. #mesoscon 2015 Plugin Conguration Additions Privileged Mode Docker Builder Image Wrapper script to start Docker daemon then run Jenkins slave Shared Docker layers for reuse Additional Docker daemon options
52. 52. #mesoscon 2015 Job Conguration Bind Docker registry credentials to ENV variable Copy to users ~/.dockercfg Cleanup! Run on docker-builder Build
53. 53. #mesoscon 2015 For Example: The go-builder Jenkins Slave Compile GO program Package as Docker app
54. 54. #mesoscon 2015 go-builder Dockerle FROM docker.orbitz.net/docker-builder:latest MAINTAINER Steve Hoffman RUN mkdir -p go/{bin,pkg,src} ENV GOPATH /go RUN # install packages yum install -y golang & update everything not already newer yum update -y & remove local caching repos yum clean all
55. 55. #mesoscon 2015 Case 3: The Deploy Farm Create single purpose client images with tools baked in Run corresponding Jenkins work against that slave Not just talking to Marathon Talk to anything
56. 56. #mesoscon 2015 The marathon-deployer Launch/Upgrade/Downgrade Docker apps via marathon use marathon python module library https://github.com/thefactory/marathon-python inside Ansible playbook http://www.ansible.com
57. 57. #mesoscon 2015 marathon-deployer Dockerle (template) FROM docker.orbitz.net/centos:7 MAINTAINER Steve Hoffman # Need to override default YUM repos and DNS resolution RUN rm /etc/yum.repos.d/* ADD src/repos/*.repo /etc/yum.repos.d/ ADD src/dns/resolv.conf /etc/ RUN # Need java to run a jenkins slave and git yum install -y jre1.8.0_31 git # For git to work in bridged mode, we need to setup user identity ADD src/git/gitconfig /root/.gitconfig # extra RPMs stored in git RUN mkdir /tmp/packages ADD src/packages/*.rpm /tmp/packages/ RUN yum clean all && yum makecache && yum install -y ansible pythons && rpm -Uvh /tmp/packages/*.rpm RUN pip install{{ with .HTTPS_PROXY }} -- proxy={{ . }}{{ end }} -v marathon &&yum install -y python-boto python-requests python-crypto & update everything not already newer & clean yum update -y && yum clean all && rm -rf /tmp/packages
58. 58. #mesoscon 2015 marathon-deployer Jenkins Job Run Template Engine To Apply ENV to Dockerle
59. 59. #mesoscon 2015 Case 2++/3+: AWS Builder/Deployer Build AMIs using Packer in AWS using amazon-ebs provider Build with Jenkins from source in Git with Packer shell provisioner Also perform scaling & rolling upgrades via aws-cli Needed AWS capable Jenkins Slave
60. 60. #mesoscon 2015 The aws-monkey Jenkins Slave (jenkins user + java) AWS CLI, Ansible & Packer (pre-installed) http://aws.amazon.com/cli/ http://www.ansible.com/ http://packer.io AWS credentials provided to slave via Credentials binding plugin from Jenkins managed