Enabling Hybrid Workflows with Docker/Mesos @Orbitz

Click here to load reader

  • date post

    22-Jan-2018
  • Category

    Technology

  • view

    2.956
  • download

    0

Embed Size (px)

Transcript of Enabling Hybrid Workflows with Docker/Mesos @Orbitz

  1. 1. #mesoscon 2015 Steve Hoffman Senior Principal Engineer @bacoboy Enabling Hybrid Workows with Docker/Mesos @Orbitz
  2. 2. #mesoscon 2015
  3. 3. #mesoscon 2015 Multiple Brands Websites Webservices Multiple Backends 500+ apps / thousands of instances Deployments Daily (sometimes more) If you havent heard of us
  4. 4. #mesoscon 2015 Mesos as a Microservices Platform using Docker a Jenkins Build Farm a Jenkins Deployment Farm HA Notes CentOS Notes Questions
  5. 5. #mesoscon 2015 Case 1: Docker Microservices Platform Launch Docker apps in Multiple environments (dev -> qa -> staging -> production) Multiple datacenters Update Docker apps in rolling fashion Restart anything that needs it
  6. 6. #mesoscon 2015 Build Unit Test Deploy Dev Deploy Prod Deploy Staging Acceptance Test Code Review & Push Production Pre-Production Open RFC Close RFC
  7. 7. #mesoscon 2015 App App App
  8. 8. #mesoscon 2015 App App App
  9. 9. #mesoscon 2015 - tasks: marathon: 1.2.16 1.2.16 1.2.16 Deploy
  10. 10. #mesoscon 2015 - tasks: marathon: 1.2.16 1.2.16 1.2.16 Deploy
  11. 11. #mesoscon 2015 - tasks: marathon: PUT /apps/editorial-module { image: orbitz/editorial-module:1.2.17 } 1.2.16 1.2.16 1.2.16 Deploy
  12. 12. #mesoscon 2015 Deploy PUT /apps/editorial-module { image: orbitz/editorial-module:1.2.17 } - tasks: marathon: 1.2.16 1.2.16 1.2.16 app = GET /v2/apps/editorial-module if not app then deploy_id = POST /v2/apps { image: orbitz/editorial-module:1.2.17, id: editorial-module } else deploy_id = PUT /v2/apps/editorial-module { image: orbitz/editorial-module:1.2.17 } end if while GET /v2/deployments contains deploy_id // still deploying end // deploy complete
  13. 13. #mesoscon 2015 - tasks: marathon: PUT /apps/editorial-module { image: orbitz/editorial-module:1.2.17 } 1.2.16 1.2.16 1.2.16 Deploy
  14. 14. #mesoscon 2015 - tasks: marathon: 1.2.16 1.2.16 1.2.16 Deploy
  15. 15. #mesoscon 2015 - tasks: marathon: 1.2.16 1.2.16 1.2.16 1.2.17 1.2.17 1.2.17 Deploy
  16. 16. #mesoscon 2015 - tasks: marathon: /health 1.2.16 1.2.16 1.2.16 1.2.17 1.2.17 1.2.17 Deploy
  17. 17. #mesoscon 2015 - tasks: marathon: /health 200 OK 200 OK 200 OK 1.2.16 1.2.16 1.2.16 1.2.17 1.2.17 1.2.17 Deploy
  18. 18. #mesoscon 2015 - tasks: marathon: 1.2.17 1.2.17 1.2.17 Deploy
  19. 19. #mesoscon 2015 And off to the next environment 1.2.17 1.2.17 1.2.17 Deploy
  20. 20. #mesoscon 2015 1.2.17 1.2.17 1.2.17 What if?
  21. 21. #mesoscon 2015 1.2.17 1.2.17
  22. 22. #mesoscon 2015 1.2.17 1.2.17
  23. 23. #mesoscon 2015 1.2.17 1.2.17 1.2.17
  24. 24. #mesoscon 2015 /health 200 OK 1.2.17 1.2.17 1.2.17
  25. 25. #mesoscon 2015 1.2.17 1.2.17 1.2.17
  26. 26. #mesoscon 2015 Video: http://bit.ly/oww-dockercon2015-video Slides: http://bit.ly/oww-microservices-dockercon2015
  27. 27. #mesoscon 2015 Case 2: The Build Farm Existing Solution Dedicated Jenkins Slaves Hand created Snapshotted & Rolled Back to Clean state after each Job Hard to Manage Build Environment for 300+ apps across many OSes, Java versions, Ruby versions, perl versions, python versions, protocol buffer compiler versions, etc
  28. 28. #mesoscon 2015 Master Slave Commit/Push or Pull Request/Merge Before
  29. 29. #mesoscon 2015 Master Slave Poll or Push Trigger Commit/Push or Pull Request/Merge Before
  30. 30. #mesoscon 2015 Master Slave Poll or Push Trigger Commit/Push or Pull Request/Merge Before
  31. 31. #mesoscon 2015 Master Slave Poll or Push Trigger Commit/Push or Pull Request/Merge Clone & Build Push Artifacts Before
  32. 32. #mesoscon 2015 Master Slave Before
  33. 33. #mesoscon 2015 Thanks EBay! http://bit.ly/ebay-jenkins-docker
  34. 34. #mesoscon 2015 Master Master Slave Commit/Push or Pull Request/Merge After
  35. 35. #mesoscon 2015 Master Master Slave Poll or Push Trigger Commit/Push or Pull Request/Merge After
  36. 36. #mesoscon 2015 Master Master Slave Slave Poll or Push Trigger Commit/Push or Pull Request/Merge After
  37. 37. #mesoscon 2015 Master Master Slave Slave Poll or Push Trigger Commit/Push or Pull Request/Merge After
  38. 38. #mesoscon 2015 Master Master Slave Slave Poll or Push Trigger Commit/Push or Pull Request/Merge Clone & Build Push Artifacts After
  39. 39. #mesoscon 2015 Master Master SlaveAfter
  40. 40. #mesoscon 2015 Jenkins mesos-plugin https://github.com/jenkinsci/mesos-plugin
  41. 41. #mesoscon 2015 Plugin Conguration Dont forget to install libmesos on Jenkins master! Point at ZK to nd active master Leave framework running
  42. 42. #mesoscon 2015 In Mesos Console Framework Registered
  43. 43. #mesoscon 2015 Plugin Conguration Per Slave Type Jenkins Slave Tag Which Mesos Servers this job can use Jenkins Slave Image 1 job per docker slave short timeout for single use RAM/CPU needed
  44. 44. #mesoscon 2015 Job Conguration Run on specic Docker slave Mark ofine immediately
  45. 45. #mesoscon 2015 Running Docker Slaves
  46. 46. #mesoscon 2015 Success! Ephemeral Docker Slave in Mesos its tag Docker IP
  47. 47. #mesoscon 2015 Case 2+: The Docker Build Farm I ALSO need to build my jenkins/docker slaves Need a jenkins docker slave to build docker images Can I do Docker-in-Docker-on-Mesos? Will need to build manually the rst time http://kb.sparknearby.com/wp-content/uploads/2015/05/chicken-or-egg-cropped1.jpg
  48. 48. #mesoscon 2015 The docker-builder Jenkins Slave (Java - jre) Docker daemon (running & in supervisor mode!) Registry credentials provided to slave via Credentials Binding Plugin from Jenkins managed security https://wiki.jenkins-ci.org/display/JENKINS/Credentials+Binding +Plugin Reuse docker layers (aka share /var/lib/docker)
  49. 49. #mesoscon 2015 wrapdocker script Start Docker daemon and then start jenkins slave https://github.com/jpetazzo/dind/blob/master/wrapdocker I had to change (variable substitution wasnt working): [[ $1 ]] && exec "$@" to: [[ $1 ]] && eval exec $@ https://blog.docker.com/2013/09/docker-can-now-run-within-docker/
  50. 50. #mesoscon 2015 docker-builder Dockerle FROM docker.orbitz.net/centos:7 MAINTAINER Steve Hoffman # Need to override default YUM repos # and DNS resolution RUN rm /etc/yum.repos.d/* ADD src/repos/*.repo /etc/yum.repos.d/ ADD src/dns/resolv.conf /etc/ RUN # load repo metadata from above yum clean all && yum makecache & install packages (jenkins needs at least # java and git) yum install -y jre1.8.0_51 git docker-engine & update everything not already newer yum update -y # For git to work in bridged mode, we need to setup user identity ADD src/git/gitconfig /root/.gitconfig # Include helper script ADD src/wrapdocker /usr/local/bin/wrapdocker # Mount docker daemon storage VOLUME /var/lib/docker
  51. 51. #mesoscon 2015 Plugin Conguration Additions Privileged Mode Docker Builder Image Wrapper script to start Docker daemon then run Jenkins slave Shared Docker layers for reuse Additional Docker daemon options
  52. 52. #mesoscon 2015 Job Conguration Bind Docker registry credentials to ENV variable Copy to users ~/.dockercfg Cleanup! Run on docker-builder Build
  53. 53. #mesoscon 2015 For Example: The go-builder Jenkins Slave Compile GO program Package as Docker app
  54. 54. #mesoscon 2015 go-builder Dockerle FROM docker.orbitz.net/docker-builder:latest MAINTAINER Steve Hoffman RUN mkdir -p go/{bin,pkg,src} ENV GOPATH /go RUN # install packages yum install -y golang & update everything not already newer yum update -y & remove local caching repos yum clean all
  55. 55. #mesoscon 2015 Case 3: The Deploy Farm Create single purpose client images with tools baked in Run corresponding Jenkins work against that slave Not just talking to Marathon Talk to anything
  56. 56. #mesoscon 2015 The marathon-deployer Launch/Upgrade/Downgrade Docker apps via marathon use marathon python module library https://github.com/thefactory/marathon-python inside Ansible playbook http://www.ansible.com
  57. 57. #mesoscon 2015 marathon-deployer Dockerle (template) FROM docker.orbitz.net/centos:7 MAINTAINER Steve Hoffman # Need to override default YUM repos and DNS resolution RUN rm /etc/yum.repos.d/* ADD src/repos/*.repo /etc/yum.repos.d/ ADD src/dns/resolv.conf /etc/ RUN # Need java to run a jenkins slave and git yum install -y jre1.8.0_31 git # For git to work in bridged mode, we need to setup user identity ADD src/git/gitconfig /root/.gitconfig # extra RPMs stored in git RUN mkdir /tmp/packages ADD src/packages/*.rpm /tmp/packages/ RUN yum clean all && yum makecache && yum install -y ansible pythons && rpm -Uvh /tmp/packages/*.rpm RUN pip install{{ with .HTTPS_PROXY }} -- proxy={{ . }}{{ end }} -v marathon &&yum install -y python-boto python-requests python-crypto & update everything not already newer & clean yum update -y && yum clean all && rm -rf /tmp/packages
  58. 58. #mesoscon 2015 marathon-deployer Jenkins Job Run Template Engine To Apply ENV to Dockerle
  59. 59. #mesoscon 2015 Case 2++/3+: AWS Builder/Deployer Build AMIs using Packer in AWS using amazon-ebs provider Build with Jenkins from source in Git with Packer shell provisioner Also perform scaling & rolling upgrades via aws-cli Needed AWS capable Jenkins Slave
  60. 60. #mesoscon 2015 The aws-monkey Jenkins Slave (jenkins user + java) AWS CLI, Ansible & Packer (pre-installed) http://aws.amazon.com/cli/ http://www.ansible.com/ http://packer.io AWS credentials provided to slave via Credentials binding plugin from Jenkins managed