Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013
-
Upload
amazon-web-services -
Category
Technology
-
view
8.226 -
download
2
description
Transcript of Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013
© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
DMG202 - Zero to Sixty:
AWS OpsWorks
Thomas Metschke, Amazon
November 13th, 2013
AWS OpsWorks
Helps you to model your entire application from
load balancers to databases starting from
templates for common technologies or building
your own.
You have full control of deployments, scaling,
monitoring, and automation of each component.
AWS Application Management Services
Elastic Beanstalk OpsWorks CloudFormation EC2
Convenience Control
Higher-level services Do it yourself
The Heart of AWS OpsWorks
Agent on each
EC2 instance
OpsWorks talks with
The Heart of AWS OpsWorks
Agent on each
EC2 instance Understands a set of commands
that are triggered by OpsWorks.
The agent then runs Chef solo.
The Heart of AWS OpsWorks
Agent on each
EC2 instance Understands a set of commands
that are triggered by OpsWorks.
The agent then runs Chef solo.
The Heart of AWS OpsWorks
Agent on each
EC2 instance Understands a set of commands
that are triggered by OpsWorks.
The agent then runs Chef solo.
AWS OpsWorks Agent Events
setup configure deploy undeploy shutdown
A Stack
A Stack with Layers
A Stack with Layers and Instances
Enough Talking
DEMO TIME
• Created a new Stack from scratch
• Created an app and database layer
• Started instances in different Availability Zones
• Configured Auto Scaling
• Load balanced by Elastic Load Balancing
• Deployed an application
Recap of Demo
Continuous Integration
John Gaa VP of Engineering at BeachMint
• Founded in 2011
• Next generation social commerce company
• Parent to: JewelMint, ShoeMint, StyleMint, & IntiMint
• Over 6 million registered customers
Problem Set …
Our Store Verticals are Powered By
• Amazon RDS
• Amazon ElastiCache
• Amazon SQS
• Amazon SNS
• Amazon CloudFront
• Amazon Redshift
RDS
ElastiCache
SQS
SNS
CloudFront
Redshift
• Multiple releases a day
• 1 sysadmin
• 100+ instances between production
and test environments
Jenkins & AWS OpsWorks to the Rescue
Confidence to Release Every Day
• Unit tests
• Integration tests
• Load testing
• Manual testing
What it Looks Like in AWS OpsWorks
Environments are represented as
stacks
Our different verticals are
represented as separate layers within
each stack
Architecture Repeated in Each Environment
JewelMint IntiMint ShoeMint StyleMint
JavaScript widget
API
Checkout Registration Product Social
RDS Solr ElastiCache Drupal Drools
Layers
Sta
ck
The Details
AWS OpsWorks in Our CI Processes
• Get instances associated to a layer
• Update code based on gittag version stored in
custom JSON
• Target the instances and run Chef scripts
The Glue: Custom JSON
{
"environment":{
"env":"production"
},
"jewelmint":{
"gittag":"20131008d_release" },
"intimint":{
"gittag":"v3.7.0_release" },
"stylemint":{
"gittag":"v3.7.0_release" },
"shoemint":{
"gittag":"v3.7.0_release" },
"cdn":{
"media":{
"origin":"beachmint-origin-domain.com",
"domain":"beachmint-cloudfront.cloudfront.net"
}
},
"solr":{
"gittag":"master",
"server_name":"beachmint-solr-domain.com",
"nodes":{ "1":"node-ip-1",
"2":"node-ip-2",
"3":"node-ip-3" }
}
}
"environment":{
"env":"production”
},
Determines what
environment to be built
The Glue: Custom JSON
"jewelmint"{
"gittag":"20131008d_release" },
"intimint":{
"gittag":"v3.7.0_release" },
"stylemint":{
"gittag":"v3.7.0_release" },
"shoemint":{
"gittag":"v3.7.0_release" },
Code version to be
deployed to a layer
{
"environment":{
"env":"production"
},
"jewelmint":{
"gittag":"20131008d_release" },
"intimint":{
"gittag":"v3.7.0_release" },
"stylemint":{
"gittag":"v3.7.0_release" },
"shoemint":{
"gittag":"v3.7.0_release" },
"cdn":{
"media":{
"origin":"beachmint-origin-domain.com",
"domain":"beachmint-cloudfront.cloudfront.net"
}
},
"solr":{
"gittag":"master",
"server_name":"beachmint-solr-domain.com",
"nodes":{ "1":"node-ip-1",
"2":"node-ip-2",
"3":"node-ip-3" }
}
}
The Glue: Custom JSON
"cdn":{
"media":{
"origin":"beachmint-origin-domain.com",
"domain":"beachmint-cloudfront.cloudfront.net" }},
"solr":{
"gittag":"master",
"server_name":"beachmint-solr-domain.com",
"nodes":{
"1":"node-ip-1",
"2":"node-ip-2",
"3":"node-ip-3" }} Configuration and
attributes of
supporting systems
{
"environment":{
"env":"production"
},
"jewelmint":{
"gittag":"20131008d_release" },
"intimint":{
"gittag":"v3.7.0_release" },
"stylemint":{
"gittag":"v3.7.0_release" },
"shoemint":{
"gittag":"v3.7.0_release" },
"cdn":{
"media":{
"origin":"beachmint-origin-domain.com",
"domain":"beachmint-cloudfront.cloudfront.net"
}
},
"solr":{
"gittag":"master",
"server_name":"beachmint-solr-domain.com",
"nodes":{ "1":"node-ip-1",
"2":"node-ip-2",
"3":"node-ip-3" }
}
}
Commit
Jenkins post commit hook
Get instances
Update code on instances
Run selenium or unit tests
Stop
pass Ready for
QA Create new
tag Update git tag on custom json
Update code on instances
Run load test
Build
Sta
ck
Load Stack
Y
N
use Aws\Common\Aws;
$aws = Aws::factory('./config.php');
$opsWorks = $aws->get('OpsWorks');
//Get the Stack we are updating
$onlineInstanceIds = array();
$allInstances = $opsWorks->describeInstances(array('LayerId' => $layerId))-
>get("Instances");
foreach($allInstances as $instance) {
if ($instance['Status'] == 'online') {
$onlineInstanceIds[] = $instance['InstanceId'];
}
}
Commit
Jenkins post commit hook
Get instances
Update code on instances
Run selenium or unit tests
Stop
pass Ready for
QA Create new
tag Update git tag on custom json
Update code on instances
Run load test
Build
Sta
ck
Load Stack
Y
N
use Aws\Common\Aws;
$aws = Aws::factory('./config.php');
$opsWorks = $aws->get('OpsWorks');
$deployment = $opsWorks->createDeployment(array(
'StackId' => $stackId,
'Command' => array(
'Name' => 'execute_recipes',
'Args' => array(
'recipes' => $recipe
)
),
'InstanceIds' => $onlineInstanceIds
));
Load Testing
• Code
• Configuration
• EC2 instance type
• System architecture configuration
• Capacity planning
Commit
Jenkins post commit hook
Get instances
Update code on instances
Run selenium or unit tests
Stop
pass Ready for
QA Create new
tag Update git tag on custom json
Update code on instances
Run load test
Build
Sta
ck
Load Stack
Y
N
//Get the Stack’s custom JSON from OpsWorks
$customJsonArray = json_decode($theBuildStack[0]["CustomJson"]);
//Replace the gittag with the new tag
$customJsonArray->{"jewelmint"}->{"gittag"} = "thenewgittag";
//Convert back to string
$modifiedJson = json_encode($customJsonArray);
//Update Stack settings in OpsWorks
$updateRes = $opsWorks->updateStack(array("StackId" => "the-build-
stackid","CustomJson" => $modifiedJson));
Commit
Jenkins post commit hook
Get instances
Update code on instances
Run selenium or unit tests
Stop
pass Ready for
QA Create new
tag Update git tag on custom json
Update code on instances
Run load test
Build
Sta
ck
Load Stack
Y
N
use Aws\Common\Aws;
$aws = Aws::factory('./config.php');
$opsWorks = $aws->get('OpsWorks');
$deployment = $opsWorks->createDeployment(array(
'StackId' => $stackId,
'Command' => array(
'Name' => 'execute_recipes',
'Args' => array(
'recipes' => $recipe
)
),
'InstanceIds' => $onlineInstanceIds
));
Path to Production
• QA and Product group determines what goes to
production
• Each release candidate has a tag associated
with it
• QA updates QA and Stage environments using a
tool based on the AWS SDK
Thanks!
www.jewelmint.com
www.stylemint.com
www.intimint.com
www.shoemint.com
Automate Standard Operating Procedures
3 AM – Alarm goes off
Wouldn’t it be nice to have help?
The idea: gather and ship logs to Amazon S3
as soon as the CPU load is to high
We will use the
following setup
AWS OpsWorks stores 1-minute
metrics in CloudWatch
Every instance creates an
alarm for high CPU load
CloudWatch alarm action:
write to SNS topic
SNS publishes to
queue in SQS
Watcher instance polls
SQS for notifications
On alarm notification,
call OpsWorks API to …
Execute a script on the
affected server
Logs are gathered and
written to Amazon S3
And Again
DEMO TIME
Demo Recap
• Instances set up alarms on CloudWatch
• Alarms create notification in SNS
• SNS publishes to SQS queue
• Watcher instance polls queue and calls AWS
OpsWorks to execute recipe on instance
• The recipe runs a script to upload logs to S3
More Information about AWS OpsWorks
• If not done yet, do the AWS OpsWorks lab!
• Find us in the AWS Booth
• Follow us on Twitter @AWSOpsWorks
• Find us on YouTube
• AWS OpsWorks survey
http://tinyurl.com/OpsWorksSurvey2013
Other Talks During re:Invent
DMG304 - AWS OpsWorks Under the Hood
• Jonathan Weiss & Reza Spagnolo
• Thursday, Nov 14, 3:00 PM - 4:00 PM – Murano 3206
DMG305 - How Intuit Leveraged AWS OpsWorks as the Engine of Our PaaS
• Capen Brinkley & Rick Mendes of Intuit, Inc.
• Thursday, Nov 14, 4:15 PM - 5:15 PM – Murano 3206
Please give us your feedback on this
presentation
As a thank you, we will select prize
winners daily for completed surveys!
DMG202
Backup Slides
Recipes
• Default Writes Ruby script to the instance and runs it
• create_alarm Writes a Ruby script to create an alarm and ties it to the right SNS
topic
• send_logs Writes a Ruby script to the instance that can pack and ship the logs
to S3
Watcher Code
queue = AWS::SQS.new.queues["<%= node[:watcher][:sqs][:url] %>"]
queue.poll(:initial_timeout => false) do |msg|
begin
alarm = JSON.parse(msg.body)['Subject'].start_with?('ALARM')
instance_id = JSON.parse((JSON.parse(msg.body)['Message']))['Trigger']
['Dimensions'].first['value']
if alarm
opsworks = AWS::OpsWorks.new.client
deployment = opsworks.create_deployment(
:stack_id => "<%= node[:opsworks][:stack][:id] %>",
:instance_ids => [instance_id],
:command => {
:name => 'execute_recipes',
:args => {'recipes' => <%= node[:watcher][:execute_recipes] %>}
},) end end end
Alarm Creation
id = "<%= node[:opsworks][:instance][:id] %>"
action ="<%= node[:watcher][:sns][:arn] %>"
alarm = AWS::CloudWatch.new.alarms.create("#{id}_cpu_idle", {
:namespace => 'AWS/OpsWorks',
:metric_name => 'cpu_idle',
:dimensions => [{:name => 'InstanceId', :value => id}],
:comparison_operator => 'LessThanOrEqualToThreshold',
:evaluation_periods => 1,
:period => 60,
:statistic => 'Average',
:threshold => 20,
:actions_enabled => true,
:alarm_actions => ["#{action}"],
:alarm_description => 'watching the available CPU on the instance'
})
Pack and Send Logs – Generated Ruby Script
timestamp = Time.now.utc.to_i
archive = "/tmp/#{timestamp}.tgz"
source = "/var/log"
s3bucket = "<%= node[:watcher][:s3][:bucket] %>"
object = "<%= node[:opsworks][:stack][:name] %>/<%= node[:opsworks][:instance][:hostname]
%>/#{timestamp}.tgz"
`cd #{File.dirname(source)} && tar czf #{archive} #{File.basename(source)}`
s3 = AWS::S3.new.buckets[s3bucket].objects[object].write(:file => archive)