DockerCon SF 2015: Enabling Microservices @Orbitz

Enabling Microservices @OrbitzSteve Hoffman @bacoboyRick Fast @tortiepointSenior Principal Engineers

Agenda

• Brief Architecture Overview/History

• From Monolithic to Services to Microservices with Docker

• Automated Pipelines

• Questions

#dockercon

2000

#dockercon

2000

Yes, we are STILL hiring!

@OrbitzTalent

#dockercon

2000

Web Layer

Business Layer

#dockercon

2000

#dockercon

2000

Business Layer

#dockercon

Business Layer

20032000

#dockercon

Business Layer

2003

#dockercon

2003

Business Layer

#dockercon

20042003

#dockercon

2004

#dockercon

20042012

#dockercon

2012

• Multiple Brands

• Websites

• Webservices

• Multiple Backends

• 500+ apps / thousands of instances

• Deployments Daily (sometimes more)

2015

#dockercon

Process Overkill

#dockercon

Different Provisioning Tools

DEV OPS

Application Platform!=

#dockercon

A New Experiment

• Microservices?

• Decompose single “service” into the actual 40+ sub-services - Any change to sub-service was a deployment of many

• Could it be a simple Spring Boot App in Docker (12 factor?)

• Backward compatible with existing service infrastructure.

• Code to Production w/o help from other Humans

#dockercon

The Docker Slave aka “The Rickbot”

App

App

App

#dockercon

10.10.10.1010.10.10.10:31002

App

App

App Register & Lookup

#dockercon

App

App

App

#dockercon

#dockercon

http://consul.io

#dockercon

10.10.10.10

10.10.10.10:51515

App

App

App

#dockercon

10.10.10.1010.10.10.10:1337

App

App

App

#dockercon

Almost done…

App

App

App

#dockercon

App

App

App

#dockercon

App

App

App

VIP

https://github.com/QubitProducts/bamboo

#dockercon

App

App

App

#dockercon

AppApp

App

A Little More Background

#dockercon

#dockercon

Module

Module

Module Module

#dockercon

Editorial Module

• A Continuously Deployed Microservice

• www.orbitz.com - (scroll down)

#dockercon

OrbitzContent

Orchestration Service

Content (Solr)

Editorial Module

Search Module

Hotel Module

#dockercon

OrbitzContent


Content (Solr)

Editorial Module

Search Module

Hotel Module

Hotel Team Search Team Content Team

#dockercon

OrbitzContent


Content (Solr)

Editorial Module

Search Module

Hotel Module

#dockercon

OrbitzContent


Content (Solr)

Editorial Module

Search Module

Hotel Module

Hotel Team

Search Team

Content Team

#dockercon

Pre-Continuous Delivery

1.2-BETA-20150401-113002

Test Discard

Test Discard1.2-BETA-20150402-093002

Build

Build

And so on…

#dockercon

Pre-Continuous Delivery

1.2-BETA-20150401-113002

Test Discard

Test Discard1.2-BETA-20150402-093002

Build

Build

Test Deploy1.2

Build

Continuous Delivery with Jenkins, Docker, and Ansible (And then Marathon)

#dockercon

Yorfast-mbp:git rfast$ yo microservice

_-----_ | | .--------------------------. |--(o)--| | Welcome to the kickass | `---------´ | Microservice | ( _´U`_ ) | generator! | /___A___\ '--------------------------' | ~ | __'.___.'__ ´ ` |° ´ Y `

? Enter the name of your service. (E.G. "my-service")

#dockercon

Committer

Pull Request

#dockercon

Committer

Reviewer

Pull Request

Jenkins Pipeline

#dockercon

if(…) { x ++; }

Merge

#dockercon

Pipeline (Simplified)

build unit test publish

deploy dev

acceptance tests

deploy qa

deploy staging open RFC deploy

prod close RFC

#dockercon

Build

if(…) { x ++; }

1.2

Merge

#dockercon

Build

if(…) { x ++; }

1.2

Merge

1.2.

editorial-module.jar

./gradlew build

{BUILD_NUMBER}

#dockercon

Build

1.2

Merge


if(…) { x ++; }

171.2.

#dockercon

Build

1.2

Merge


FROM orbitz/java-8 ADD build/editorial-module.jar /opt/orbitz CWD /opt/orbitz CMD java -jar editorial-module.jar

if(…) { x ++; }

171.2.

#dockercon

Build

1.2

Merge

editorial-module.jar orbitz/editorial-module:1.2.17

if(…) { x ++; }

171.2.

#dockercon

Build

orbitz/editorial-module:1.2.17

#dockercon

Deploy

- hosts: dev - tasks: - name: find previous images docker_facts: image=orbitz/{{application}} register: previous - name: deploy new image docker: image=orbitz/{{application}}:{{version}} … - name: wait for service wait_for: port={{port}} … - name: check health endpoint uri: url="http://{{fqdn}}:{{port}}/health" … - name: kill old image docker: {{previous}}

git pull

playbook.yml (abridged)

#dockercon

host-001 host-002 host-003 host-004 host-005

- hosts: dev

#dockercon

Deploy

host-001

1.2.16

#dockercon

Deploy

1.2.16 1.2.17

host-001

#dockercon

1.2.16 1.2.17

host-001

Deploy

DATABASE_URL=jdbc://whatever/db/stuff CONSUL_HOST={{ansible_fqdn}} LOGSTASH_HOST={{ansible_fqdn}} GRAPHITE_HOST={{ansible_fqdn}}

jdbc://whatever/db/stuff

#dockercon

Deploy

1.2.16 1.2.17

/health

host-001

#dockercon

Deploy

1.2.16 1.2.17

/health 200 OK

host-001

#dockercon

Deploy

1.2.16 1.2.17

host-001

#dockercon

Deploy

1.2.17

host-001

#dockercon

Deploy

host-002

1.2.16

And so on…host-001

1.2.17

#dockercon

What’s next?

#dockercon

How to handle failure?

#dockercon

What happens when a VM is moved?

#dockercon

What if I need to add capacity?

#dockercon

Deploy (Second Attempt)

- hosts: dev - tasks: - name: find previous images docker_facts: image=orbitz/{{application}} register: previous - name: deploy new image docker: image=orbitz/{{application}}:{{version}} … - name: wait for service wait_for: port={{port}} … - name: check health endpoint uri: url="http://{{fqdn}}:{{port}}/health" … - name: kill old image docker: {{previous}}

git pull


#dockercon

Deploy (Second Attempt)- hosts: dev - tasks: - name: find previous images docker_facts: image=orbitz/{{application}} register: previous - name: deploy new image docker: image=orbitz/{{application}}:{{version}} … - name: wait for service wait_for: port={{port}} … - name: check health endpoint uri: url="http://{{fqdn}}:{{port}}/health" … - name: kill old image

git pull


- hosts: localhost - tasks: - name: marathon deploy marathon: image=orbitz/{{application}}:{{version}} instances=3

#dockercon


New host setup

#dockercon


Mesos Agent

New host setup

1.2.16

#dockercon


- tasks: marathon: …

playbook.yml

1.2.16 1.2.16

1.2.16

#dockercon



1.2.16 1.2.16

1.2.16

#dockercon



PUT /apps/editorial-module { “image”: “orbitz/editorial-module:1.2.17” … }

1.2.16 1.2.16

1.2.16

#dockercon




1.2.16 1.2.16

1.2.16

app = GET /v2/apps/editorial-module

if not app then deploy_id = POST /v2/apps { “image”: “orbitz/editorial-module:1.2.17”, “id”: “editorial-module” } else deploy_id = PUT /v2/apps/editorial-module { “image”: “orbitz/editorial-module:1.2.17” } end if

while GET /v2/deployments contains deploy_id // still deploying end

// deploy complete

#dockercon




1.2.16 1.2.16

1.2.16

#dockercon



1.2.16 1.2.16

1.2.16

#dockercon



1.2.16 1.2.16

1.2.16

1.2.17 1.2.17

1.2.17

#dockercon



/health

1.2.16 1.2.16

1.2.16

1.2.17 1.2.17

1.2.17

#dockercon



/health

200 OK 200 OK

200 OK

1.2.16 1.2.16

1.2.16

1.2.17 1.2.17

1.2.17

#dockercon


1.2.17 1.2.17

1.2.17

#dockercon

And off to the next environment…

1.2.17 1.2.17

1.2.17

#dockercon

1.2.17 1.2.17

1.2.17

What if?

#dockercon

1.2.17 1.2.17

#dockercon

1.2.17 1.2.17 1.2.17

#dockercon

/health

200 OK

1.2.17 1.2.17 1.2.17

#dockercon

1.2.17 1.2.17 1.2.17

#dockercon

Smoke/Acceptance Testing

1.2.17

1.2.17

1.2.17

#dockercon

Paper Trail

create

tickets

commit(s)

#dockercon

Paper Trail

fail!

close/fail

#dockercon

Paper Trail

ok

close

Build Unit Test

Deploy Dev

Deploy Prod

Deploy Staging

Acceptance TestCode

Review & Push

Production

Pre-Production

Open RFC

Close RFC

#dockercon

What’s inside the containers?

• Spring Boot

• Dropwizard Metrics (formerly Coda Hale, Yammer)

• Consul Registration/Discovery

• Logstash + Logback

• Swagger

• Hystrix

• Retrofit + Consul

OrbitzWorldwide/consul-client

#dockercon

–You

“Why didn’t you use {{.x}}?”

#dockercon

Keeping an eye on…

Kubernetes Amazon Elastic Container

Service

Docker Swarm

#dockercon

Summary

• Create a shared platform for docker deployments using shared and app-specific “localhost” helpers — this was ours, yours SHOULD look different — adapt to change, don’t fight it.

• Take people out of the release process - Docker - repeatable applications

- Chef - repeatable infrastructure — Environment aware

- Jenkins - repeatable releases

• Delineate configuration concerns: - Known at Compile time — Bake into Docker image

- Known at Boot time — Bake into Playbook/Launcher - parameter to Docker run

- Changes Anytime — Externalize (consul K/V, etcd, zookeeper)

Thanks!Steve Hoffman

@bacoboy

Rick Fast

@tortiepoint

DockerCon SF 2015: Enabling Microservices @Orbitz

Technology

Transcript of DockerCon SF 2015: Enabling Microservices @Orbitz