Overview of NPS/Voice of Customer approach across eBay, Inc.

Delivering eBay's CI Solution with Apache Mesos & DockerMohit SoniSoftware EngineerAshish HunnargikarSoftware Engineer

Why we did it.... Thousands of VMs dedicated to run Jenkins! Utilization is less than 5% VMs idle most of time (no one runs CI builds frequently)

VM sprawl and poor resource utilization

Polyglot application stacks (Java, C++, Node.js, Python, Scala) Different OS flavors (Ubuntu & RHEL) Software version conflicts Special hardware requirements

Build slave management and maintenance is a nightmare 11And heres what we did....Mesos for cluster managementMarathon to launch Jenkins mastersJenkins Mesos-plugin to launch build slavesBuild slaves Dockerized as well as regular

2

3

4Why use Docker containers? Managing varied slave flavors in a large cluster is hard! Docker effectively isolates dependencies inside the container Docker allows host OS to remain independent of workload dependencies

Why build Docker-in-Docker? Run Docker build/push/pull inside a Docker container Eliminates redundant CI build dependency downloads Helps to enforce a heterogeneous Mesos slave cluster5Allow different OS flavorsPrevent software version conflictsApplication dependenciesHardware independence

5Basic Use Case

Build C++ & Java applications on both RHEL & Ubuntu

Mesos slave host server is running Ubuntu

Jenkins job is running builds in a RHEL or Ubuntu container

Application dependencies can be preinstalled in base images OR can be downloaded during build time

6Advanced Use Case Build RHEL & Ubuntu Docker imagesMesos slave host server is running Ubuntu

Jenkins job is running inside a Docker container

Jenkins job is building a RHEL/Ubuntu Docker image from an application Dockerfile

7Enabling the Docker CI workflow using Jenkins & Mesos

8Initial Build Strategy Individual Docker-in-Docker Setup

Complete isolation

Docker daemon running inside the Docker container requires privileged mode

I/O overhead of downloading and duplicating all the AUFS layers9

Optimized Build Strategy

Shared Docker-in-Docker Setup

Docker daemon not required inside the container so no privileged mode

Single reusable AUFS image layer cache

Redundant network and disk I/O eliminated for layer downloads

Lack of isolation for layers 10

For more information visit:Apache Mesos: http://mesos.apache.orgJenkins Mesos Plugin: https://github.com/jenkinsci/mesos-pluginDocker: https://www.docker.ioMarathon Framework: https://github.com/mesosphere/marathonQ&Ahttp://www.ebaytechblog.com/2014/04/04/delivering-ebays-ci-solution-with-apache-mesos-part-i/

http://www.ebaytechblog.com/2014/05/12/delivering-ebays-ci-solution-with-apache-mesos-part-ii/www.ebaytechblog.com11