Continuous Integration and Deployment Best Practices on AWS
-
Upload
danilo-poccia -
Category
Technology
-
view
875 -
download
1
description
Transcript of Continuous Integration and Deployment Best Practices on AWS
CONTINUOUS INTEGRATION AND DEPLOYMENT BEST PRACTICES ON AWS
Danilo Poccia – Solutions Architect
Compute & Networking
Storage & Content Delivery
AWS Global Infrastucture
Database
App Services
Anal
ytic
s
Deployment & Management
API
API
API
API
API
API
Regions Availability Zones Edge Locations
EC2 VPC Route 53 Direct Connect
S3 Glacier CloudFront Storage Gateway
RDS DynamoDB ElastiCache Redshift
SQS SNS SWF SES AppStream CloudSearch Elastic Transcoder
IAM CloudTrail CloudWatch CloudFormation Elastic Beanstalk OpsWorks
EMR
Dat
a Pi
pelin
eK
ines
is
WorkSpaces
Regions Availability Zones Edge Locations
S3 Glacier CloudFront Storage Gateway
RDS DynamoDB ElastiCache Redshift
SQS SNS SWF SES AppStream CloudSearch Elastic Transcoder
IAM CloudTrail CloudWatch CloudFormation Elastic Beanstalk OpsWorks
EMR
Dat
a Pi
pelin
eK
ines
is
EC2 VPC Route 53 Direct Connect WorkSpaces
Regions
Availability ZonesEdge Locations
EC2
VPC Route 53 Direct Connect
S3Glacier
CloudFront
Storage Gateway
RDSDynamoDB
ElastiCache
RedshiftSQS
SNS
SWF
SES
AppStreamCloudSearch
Elastic Transcoder
IAM
CloudTrail
CloudWatch CloudFormation
Elastic
Beanstalk
OpsWorks
EMR
Data Pipeline
Kinesis
WorkSpaces
http://commons.wikimedia.org/wiki/File:Lego_bricks.jpg
BUILDING BLOCKS TO IMPLEMENT YOUR IDEA
FILE SYNC
DATA ANALYSIS
DATA ANALYSIS COSTS < VALUE
AMAZON EC2 SPOT INSTANCES
2013
280
2011
82
2010
612009
48
2008
24
2007
9 Amazon FPS Red Hat EC2
SimpleDB CloudFront EBS Availability Zones Elastic IPs
Relational Database Service Virtual Private Cloud Elastic Map Reduce
Auto Scaling Reserved Instances
Elastic Load BalancerSimple Notification Service
Route 53 RDS Multi-‐AZ
Singapore Region Identity Access Management
Cluster Instances
Elastic Beanstalk Simple Email Service CloudFormation RDS for Oracle ElastiCache
Redshift Kinesis
WorkSpaces Route 53 Health Check
OpsWorks CloudTrail
number of released features, sample services described
2012
159Glacier
DynamoDB Simple Workflow
CloudSearch Storage Gateway
Route 53 Latency Based Routing
NEW SERVICES + NEW FEATURES = NEW IDEAS
AMAZON KINESIS REAL-TIME PROCESSING
OF STREAMING DATA AT MASSIVE SCALE
!Data!Sources!
App.4%!
[Machine!Learning]!
!!!!!!!!!!!!!!!
! ! !AW
S!En
dpoint!
App.1%%
[Aggregate!&!De:Duplicate]!
!Data!Sources!
Data!Sources!
!Data!Sources!
App.2%%
[Metric!Extrac=on]!
S3
DynamoDB
Redshift
App.3%[Sliding!Window!Analysis]!
!Data!Sources!
Availability Zone
Shard 1
Shard 2 Shard N
Availability Zone
Availability Zone
CLICKSTREAM ANALYTICS INTERNET OF THINGS METERING & BILLING EVENT PROCESSING SOCIAL NETWORKS LOG COLLECTION …
AMAZON SQS + AUTO SCALING + SPOT INSTANCES !
= FREEMIUM
AMAZON SNS MOBILE PUSH
NOTIFICATIONS
EVE
Online
ANY NEW
IDEA TO BUILD?
THE JOURNEY OF CODE TO
PRODUCTION
WRITTEN TESTED
CONSUMED DEPLOYED
~ MANUFACTURING ~
LEAN MANUFACTURING
Transport Inventory Motion Waiting
Over Production
Over Processing Defects Talent!
& Skills
INFORMATION TECHNOLOGY
Transport Inventory Motion Waiting
Over Production
Over Processing Defects Talent!
& Skills
OPTIMIZE WITH AWS
Transport Inventory Motion Waiting
Over Production
Over Processing Defects Talent!
& Skills
OPTIMIZE WITH AWS
OPPORTUNITY STRATEGY
USE ADJUST
= PROGRAMMABLE
PLATFORM
!
IF YOU CAN DO ITYOU CAN AUTOMATE IT
“Improving daily work is even more important than doing daily work.”
!
!
Gene Kim, Kevin Behr, George Spafford, The Phoenix Project: A Novel About IT, DevOps,
and Helping Your Business Win
CONTINUOUS INTEGRATION
DEVELOPER
SOURCE CODE REPOSITORY
SOURCE CODE REPOSITORY
PROJECT MANAGEMENT SERVER
SOURCE CODE REPOSITORY
PROJECT MANAGEMENT SERVER
CONTINUOUS INTEGRATION SERVER
SOURCE CODE REPOSITORY
PROJECT MANAGEMENT SERVER
CONTINUOUS INTEGRATION SERVER
PICK TASKS
SOURCE CODE REPOSITORY
PROJECT MANAGEMENT SERVER
CONTINUOUS INTEGRATION SERVER
SUBMITCODE
SOURCE CODE REPOSITORY
PROJECT MANAGEMENT SERVER
CONTINUOUS INTEGRATION SERVER
SCHEDULEBUILD
SOURCE CODE REPOSITORY
PROJECT MANAGEMENT SERVER
CONTINUOUS INTEGRATION SERVER
AUTOMATIC BUILDS
SOURCE CODE REPOSITORY
PROJECT MANAGEMENT SERVER
CONTINUOUS INTEGRATION SERVER
CODEFETCH
SOURCE CODE REPOSITORY
PROJECT MANAGEMENT SERVER
CONTINUOUS INTEGRATION SERVER
CODE QUALITY TESTS
TESTRESULTS
SOURCE CODE REPOSITORY
PROJECT MANAGEMENT SERVER
CONTINUOUS INTEGRATION SERVER
BUILD OUTPUT
SOURCE CODE REPOSITORY
PROJECT MANAGEMENT SERVER
CONTINUOUS INTEGRATION SERVER
DOCS
BINARIES& PACKAGES
TDD TEST-DRIVEN
DEVELOPMENT
BDD BEHAVIOR-DRIVEN
DEVELOPMENT
PAIN POINTS • UNIT TESTS INCOMPLETE • MOCK TESTS MAINTENANCE • EXPENSIVE TEST ENVIRONMENT • TEST ENVIRONMENT ≠ PRODUCTION • DEPLOYMENT CYCLES
“Complexity arises when the dependencies among the elements
become important.” !
!
Scott E. Page, John H. Miller,Complex Adaptive Systems:
An Introduction to Computational Models of Social Life
REAL USERS =
REAL FEEDBACK
CLOSE THE FEEDBACK
LOOP AS QUICKLY
AS POSSIBLE
CONTINUOUS INTEGRATION
CONTINUOUS DEPLOYMENT
SOFTWARE DEPLOY
≠PRODUCT LAUNCH
CONTINUOUS DEPLOYMENT
SMALL, FREQUENT CHANGES CONSTANTLY INTEGRATING INTO
PRODUCTION
KEY = ITERATION
ITERATION =
MODIFY THE SYSTEM TO BETTER MEET THE EXPECTATIONS OF
YOUR USERS
FEATURE SWITCH FEATURE FLIPPER FEATURE TOGGLE
FEATURE FLAG CONDITIONAL FEATURE
// To totally enabled a feature !$server_config[‘foo'] = ‘on’;!!
// To totally disabled a feature!$server_config['foo'] = ‘off’;!!
// Feature with variant turned on for everyone!$server_config['foo'] = 'blue_background';!!
// Enabled for 10% of regular users and all admins!$server_config[‘foo'] = array(! 'enabled' => 10,! 'admin' => 'on',!!);
https://github.com/etsy/feature
!if (Feature::isEnabled('my_feature')) {!! // do stuff!!}!! !!if (Feature::isEnabled('my_feature')) {!! !! switch (Feature::variant('my_feature')) {!! case 'foo':!! // do stuff appropriate for the foo variant!! break;!! case 'bar':!! // do stuff appropriate for the bar variant!! break;!! }!!}
https://github.com/etsy/feature
A / B TESTING CONTENT
// Single-variant feature in 50/50 A/B test!!
$server_config['foo'] = array('enabled' => 50);!!
// Multi-variant feature in A/B test!// with 20% of users seeing each variant!// (and 40% left in control group)!!
$server_config['foo'] = array(!! 'enabled' => array(!! 'blue_background' => 20,!! 'orange_background' => 20,!! 'pink_background' => 20,!! ),!!);
https://github.com/etsy/feature
AWS OPSWORKS MODEL AND MANAGE
THE ENTIRE APPLICATION
14 BILLION REQUESTS/MONTH50 000 DATABASE UPDATES / SEC
NO CACHE
CASE STUDY
A / B TESTING NOT JUST FOR CONTENT
USER INTERFACE ALGORITHMS
ARCHITECTURES COST OPTIMIZATION
AWS CLOUDFORMATION INFRASTRUCTURE IS CODE
CLOUDFORMATION TEMPLATE
APPLICATION VERSIONS
+INFRASTRUCTURE
VERSIONS
CLOUDFORMATION TEMPLATE
AWS ELASTIC BEANSTALK
QUICKLY DEPLOY AND MANAGE APPLICATIONS
Production Environment
Test Environment
master branch
test branch
Git Repository
git commit
git aws.push
git aws.push
Swap URLs
AMAZON EC2 API / SDK / CLI
for ip in $(!!
aws ec2 describe-instances! --filter Name=instance-state-name,Values=running,! Name=tag:aws:autoscaling:groupName,Values=XXX! --query Reservations[].Instances[].PublicDnsName! --output text!!
); do ssh -t ec2-user@$ip "/app/update.sh"; done!!
ALL TOGETHER NOW! !
!
!
!
!
DEPLOYMENTS AT AMAZON.COM
!
!
!
11.6s
Mean time between deployments (weekday)
1,079
Max number of deployments in a
single hour
10,000
Mean number of hosts
simultaneously receiving a deployment
30,000
Max number of hosts
simultaneously receiving a deployment
CONTINUOUS DEPLOYMENT
= CONTINUOUS
EXPERIMENTATION
CONTINUOUS DEPLOYMENT
= CONTINUOUS INNOVATION
http://commons.wikimedia.org/wiki/File:Lego_bricks.jpg
« Want to increase innovation? Lower the cost of failure »
!
Joi Ito
CONTINUOUS INTEGRATION AND DEPLOYMENT BEST PRACTICES ON AWS
Danilo Poccia – Solutions Architect