Priming Your Teams For Microservice Deployment to the Cloud

Priming Your Teams for Microservice Deployment to the Cloud

Matt Callananlinkedin.com/in/matthewcallanan@mcallana

© 2015 Expedia, Inc. All rights reserved.

How Do You Create A Microservice For The Cloud?

Git Repo

Working Code Base

Basic Test Suite

Immutable Servers

Infrastructure As Code

Centralised Logging

CentralisedMonitoring

Chat Notifications

Load Balancer

Blue-‐Green Deploys

Continuous Delivery Pipeline

Opportunity Cost: Time Value of Information

“Primer”

DevOps Toolchain

Based on devops toolchain by dto solutions: http://www.slideshare.net/AnthonyShortland/dto-chefconf2012

Application

Infrastructure

Build Deploy

Source Code Repo

Artifact Repo

Build Console

Deployment Console

Resource Model

InfrastructureManager

Code

Config

Automation

Environment Data

Automation

Config

Packages Images

DevOps Toolchain

Developer Toolchain

Operations Toolchain

Dev – Starting from Scratch

Single page app• Secured with LDAP• Choose:

– App Name– Team Name– Notifications Email– HipChat Room– Technology

Build triggers remote Jenkins job• Benefits

– Easy to configure– Simple API– Versioned scripts– Centralised logs / results

Dev – App Types

• dwsvc -‐ Java Dropwizard service• emr-‐datapipeline -‐ An Elastic MapReduce -‐ Data Pipeline

service• spark-‐streaming -‐ A Spark Streaming App• expressjs -‐ A Node application that uses expressjs• php -‐ A PHP application• python -‐ A Python application• scalatra– Lightweight Scala app based off Sinatra• sinatra – Lightweight Ruby web service • springboot -‐ A SpringBoot service• storm-‐topology -‐ A storm topology service• ticker -‐ A Ticker (trend service) application• webmvc-‐ A customer facing Brand Expedia Spring MVC

Web application• grails -‐ A Groovy-‐based web application framework for JVM• AWS Lambda -‐ Java based AWS Lamda implementation

*each app type has it’s own GitHub repo for source, issues, etc

Dev – Generating App and Pipeline

Jenkins Job – create-‐new-‐application• Creates new GitHub repo


Jenkins Job – create-‐new-‐application• Creates new GitHub repo

GitHub• Promote open source model for teams• Use pull requests as mechanism for ensuring

reviews / pairing• WebHooks for devs to tweak workflows


Jenkins Job – create-‐new-‐application• Creates new GitHub repo• Copies chosen technology template and

pushes back to repo

Tech Templates• Uses latest version of template• Replaces app names, package names, etc

with your app name• Simple interface – each template codebase

has clone.sh using cp and sed• Java-‐based templates use Maven

Archetypes to generate codebase• Easy to extend and fork



pushes back to repo• Generates Continuous Delivery pipeline in

Jenkins

Jenkins• Generates Jenkins build that is triggered

by commits to master branch• Jenkins config via templated XML• Uses standard commands for target

template (mvn, npm, etc)• Uploads zip to S3• Triggers downstream deploy job passing

Git SHA



pushes back to repo• Generates Continuous Delivery pipeline in

Jenkins• Sends email to user with details / links

DevOps Toolchain

Developer Toolchain

Operations Toolchain

Operations Pipeline

• Immutable Servers• Chef Cookbooks• CloudFormation Templates• Deploy to immutable servers using infrastructure as code that dev and ops can maintain

• Blue-‐green deploys• HipChat notifications • Centralised logging -‐ Splunk• Centralisedmonitoring -‐ Hubble

Chef Cookbooks and Config


roles/primer-‐demo-‐service.rb_______________________________________________________________________name "primer-demo-service"description "Webapp for primer-demo-service"run_list "recipe[s3curl]","recipe[primer-demo-service]"

override_attributes("java" => {

"jdk_name" => "jdk8_0_05",}

)


cookbooks/primer-‐demo-‐service/recipes/default.rb_______________________________________________________________________

node.default[:springboot][:application_name] = 'primer-demo'node.default[:springboot][:role_name] = 'primer-demo-service'

include_recipe "expedia-platform::springboot"


deployment_config/<env>/primer-‐demo-‐service/config.json{"instances": {"min_count": 1,"max_count": 2,"auto_scale": true,"zones": {"us-‐east-‐1": ["us-‐east-‐1d"],"us-‐west-‐2": ["us-‐west-‐2c"]

},"destroy_old": true,"removeOldFromElb": true

},"app_type": "springboot","app_port": 8080,"isactive_support": {"enabled": true,"file": "/opt/primer-‐demo-‐service/active.txt"

},"source_repo": ".../primer-‐demo-‐service.git","notify": {"hipchat": {"room": "Demo Room"},"email": {"addresses": ["[email protected]"]},"s3": {"bucket": "..."}

},"team": "Exp-‐Ops-‐Wotif-‐Tech-‐Australia","prime_role": "role-‐prime-‐web"

}

Deploy Build

Jenkins Chef knife

AMI

Route53

ELBCloudFormation

ASG EC2 InstanceEC2 Instance

Deployment – Email Notification

Blue-‐Green DeploysSafely Releasing Updates

Separating Deploy & Release

Deploy Test Release

Blue Green Deploys – Deploy

Deploy

Blue Green Deploys

Release

Blue Green Deploys – Release

Release – Email Notification

HipChatNotifications

Centralised Logging – Splunk

CentralisedMonitoring & Alerting

Hubble Dashboard

Hubble Dashboard – Graphene

Hopscotch – Transparent Traffic Router

• Proxy built on Nginx (similar to Netflix Zuul)• expedia.com/xyz -‐> xyz.expedia.com

• Faster to update rules than Akamai 45min turnaround• E.g. Register microservice with hopscotch in Singapore (20% of traffic) route 10% of

Singapore traffic to cloud app, when happy, dial up to 100%, then roll out to Europe and USA• Nice way to move traffic out of data centre into AWS with no customer impact

Primer Cleanup

Cleanup

• Deleting project– Delete Codebase (Git repo), Build

• Delete Cloudformation stack• Delete unused cloudformation stacks -‐ ELB request count 1hr

Cleanup – Delete the Microservice

Cleanup – Email Confirmation

Cleanup – Delete Cloud Formation Stack

Cleanup – Auto Deletion of Unused Stack in Test Environment

Primer 2.0Docker on Amazon ECS

DevOps Toolchain – Primer 2.0: Docker on Amazon ECS

Inspired by devops toolchain by dto solutions: http://www.slideshare.net/AnthonyShortland/dto-chefconf2012

Application

Infrastructure

Build Deploy

Source Code Repo

Artifact Repo

Build Console

Deployment Console

Resource Model

InfrastructureManager

Code

Config

Automation

Environment Data

Automation

Config

Packages Images

gzip AMI

git Jenkins JenkinsAmazon S3 Chef CloudFormationDocker Registry

Docker image

JSON Amazon ECS

Primer 2.0 – Docker on Amazon ECS

• Using Amazon EC2 Container Service (ECS)• Deploy Primer applications as Docker containers to an ECS cluster• Deploys take 7mins (instead of 30mins)• No AMI creation, no CloudFormation• Cost Optimisation• Using AWS SDK:• Register ECS Container Definition -‐> Task Definition -‐> Service• Create ELB• Create CNAME

• Blue/Green Deploys with Canary service

Primer 2.0 – Docker on ECS

Test Environment

Primer 2.0 – Docker on ECS

Production Environment

c3vis – Open Source

• c3vis: “Cloud Container Cluster Visualiser”• Visualises container deploys and resource utilization on Amazon ECS clusters

• Open Source Software from Expedia• github.com/ExpediaDotCom/c3vis

Results

Results

• 800 Primer apps and growing• Culture of Experimentation– Freedom of choice– Keys to prod – trust

• Enabling Innovation– Open Source model for tool• visible• accessible• easy to contribute• “I want to add Grails to this tool!”, “Help

yourself!”

• Lowering Cost– Counters the cost of learning legacy

deployment processes– Generate microservice without thinking

about it

Tips & TricksBuilding Your Own “Primer”

Tips & Tricks

• Don’t “Lift & Shift” your monolith• Rethink • Start with non-‐critical workflow apps

• Don’t need to start heterogeneous (i.e. multiple app types)• Doesn't need to be in the cloud• Cloud has benefits like immutable infrastructure, scaling, automation• But can do the same thing in the data centre

• Small interface• /isActive endpoint for LB healthcheck• /buildInfo endpoint for version

• "Just do it" attitude• Open source model helped spread it across the organisation

Q & AThanks!

Any Questions?

Matt Callananlinkedin.com/in/matthewcallanan@mcallana

Image Attribution

Image

“Clock Time Lapse Video Download“ (https://youtu.be/NSuNpiW-‐LwI) by LifeBloodMarketing is licensed under “CC Free to Use”

“Pipelines descending to Inveruglas Power Station” (http://www.geograph.org.uk/photo/2214366) is licensed under CC BY SA 2.0 (http://creativecommons.org/licenses/by-‐sa/2.0/) / Desaturated and cropped from original

“The Future” (https://flic.kr/p/26YCn1) by Kristian Bjornard is licensed under CC BY SA 2.0 (https://creativecommons.org/licenses/by-‐sa/2.0/)

“CTA Loop Junction” (https://commons.wikimedia.org/wiki/File:CTA_loop_junction.jpg) by Daniel Schwen is licensed under CC BY SA 3.0 (https://creativecommons.org/licenses/by-‐sa/3.0/)

“Logging operations at Millmoor Rig” (http://bit.ly/1Nb20LS) by Walter Baxter is licensed under CC BY SA 2.0 (https://creativecommons.org/licenses/by-‐sa/2.0/)

“Traffic Monitoring” (https://commons.wikimedia.org/wiki/File:Traffic_Monitoring.JPG) by Suryasuharman is licensed under CC BY SA 3.0 (https://creativecommons.org/licenses/by-‐sa/3.0/)

“DNS logo” (https://commons.wikimedia.org/wiki/File:DNS_logo.jpg) by I laramide I is licensed under CC BY SA 3.0 (https://creativecommons.org/licenses/by-‐sa/3.0/)

“Light bulb icon tips” (https://commons.wikimedia.org/wiki/File:Light_bulb_icon_tips.svg) by YassineMrabet is licensed under CC BY SA 4.0 (https://creativecommons.org/licenses/by-‐sa/4.0/)

“Matrix-‐code-‐computer-‐pc-‐data” (https://pixabay.com/en/matrix-‐code-‐computer-‐pc-‐data-‐356024/) by Comfreak is licensed under CC ZERO (https://creativecommons.org/publicdomain/zero/1.0/)

“Sample-‐color-‐blue-‐green” (https://pixabay.com/en/sample-‐color-‐blue-‐green-‐rubber-‐815141/ ) by LyraBelacqua-‐Sally is licensed under CC ZERO (https://creativecommons.org/publicdomain/zero/1.0/)

“Fashion-‐wristwatch-‐time” (https://www.pexels.com/photo/fashion-‐wristwatch-‐time-‐watch-‐1252/) by SplitShire.com is licensed under CC ZERO (https://creativecommons.org/publicdomain/zero/1.0/)

“Chat” (https://openclipart.org/detail/129049/chat) by Merlin2525 is licensed under unlimited-‐commercial-‐use (https://openclipart.org/unlimited-‐commercial-‐use-‐clipart)

“scales” (https://openclipart.org/detail/24101/scales) by scott_kirkwood is licensed under unlimited-‐commercial-‐use (https://openclipart.org/unlimited-‐commercial-‐use-‐clipart)

“Wooden Bookcase” (https://openclipart.org/detail/191463/wooden-‐bookcase) by sirgazil is licensed under unlimited-‐commercial-‐use (https://openclipart.org/unlimited-‐commercial-‐use-‐clipart)

“Compiz GIT Repository” (https://flic.kr/p/Ssras) by -‐= Treviño =-‐ is licensed under BY NC SA 2.0 (https://creativecommons.org/licenses/by-‐nc-‐sa/2.0)

“logs” (https://flic.kr/p/9F8tjX) by Rick Payette is licensed under CC BY NC ND 2.0 (https://creativecommons.org/licenses/by-‐nc-‐nd/2.0)

Docker logo used according to https://www.docker.com/brand-‐guidelines

Priming Your Teams For Microservice Deployment to the Cloud

Software

Transcript of Priming Your Teams For Microservice Deployment to the Cloud