© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

DMG202 - Zero to Sixty:

AWS OpsWorks

Thomas Metschke, Amazon

November 13th, 2013

AWS OpsWorks

Helps you to model your entire application from

load balancers to databases starting from

templates for common technologies or building

your own.

You have full control of deployments, scaling,

monitoring, and automation of each component.

AWS Application Management Services

Elastic Beanstalk OpsWorks CloudFormation EC2

Convenience Control

Higher-level services Do it yourself

The Heart of AWS OpsWorks

Agent on each

EC2 instance

OpsWorks talks with

The Heart of AWS OpsWorks

Agent on each

EC2 instance Understands a set of commands

that are triggered by OpsWorks.

The agent then runs Chef solo.

The Heart of AWS OpsWorks

Agent on each

EC2 instance Understands a set of commands

that are triggered by OpsWorks.

The agent then runs Chef solo.

The Heart of AWS OpsWorks

Agent on each

EC2 instance Understands a set of commands

that are triggered by OpsWorks.

The agent then runs Chef solo.

AWS OpsWorks Agent Events

setup configure deploy undeploy shutdown

A Stack

A Stack with Layers

A Stack with Layers and Instances

Enough Talking

DEMO TIME

• Created a new Stack from scratch

• Created an app and database layer

• Started instances in different Availability Zones

• Configured Auto Scaling

• Load balanced by Elastic Load Balancing

• Deployed an application

Recap of Demo

Continuous Integration

John Gaa VP of Engineering at BeachMint

• Founded in 2011

• Next generation social commerce company

• Parent to: JewelMint, ShoeMint, StyleMint, & IntiMint

• Over 6 million registered customers

Problem Set …

Our Store Verticals are Powered By

• Amazon RDS

• Amazon ElastiCache

• Amazon SQS

• Amazon SNS

• Amazon CloudFront

• Amazon Redshift

RDS

ElastiCache

SQS

SNS

CloudFront

Redshift

• Multiple releases a day

• 1 sysadmin

• 100+ instances between production

and test environments

Jenkins & AWS OpsWorks to the Rescue

Confidence to Release Every Day

• Unit tests

• Integration tests

• Load testing

• Manual testing

What it Looks Like in AWS OpsWorks

Environments are represented as

stacks

Our different verticals are

represented as separate layers within

each stack

Architecture Repeated in Each Environment

JewelMint IntiMint ShoeMint StyleMint

JavaScript widget

API

Checkout Registration Product Social

RDS Solr ElastiCache Drupal Drools

Layers

Sta

ck

The Details

AWS OpsWorks in Our CI Processes

• Get instances associated to a layer

• Update code based on gittag version stored in

custom JSON

• Target the instances and run Chef scripts

The Glue: Custom JSON

{

"environment":{

"env":"production"

},

"jewelmint":{

"gittag":"20131008d_release" },

"intimint":{

"gittag":"v3.7.0_release" },

"stylemint":{

"gittag":"v3.7.0_release" },

"shoemint":{

"gittag":"v3.7.0_release" },

"cdn":{

"media":{

"origin":"beachmint-origin-domain.com",

"domain":"beachmint-cloudfront.cloudfront.net"

}

},

"solr":{

"gittag":"master",

"server_name":"beachmint-solr-domain.com",

"nodes":{ "1":"node-ip-1",

"2":"node-ip-2",

"3":"node-ip-3" }

}

}

"environment":{

"env":"production”

},

Determines what

environment to be built

The Glue: Custom JSON

"jewelmint"{

"gittag":"20131008d_release" },

"intimint":{

"gittag":"v3.7.0_release" },

"stylemint":{

"gittag":"v3.7.0_release" },

"shoemint":{

"gittag":"v3.7.0_release" },

Code version to be

deployed to a layer

{

"environment":{

"env":"production"

},

"jewelmint":{

"gittag":"20131008d_release" },

"intimint":{

"gittag":"v3.7.0_release" },

"stylemint":{

"gittag":"v3.7.0_release" },

"shoemint":{

"gittag":"v3.7.0_release" },

"cdn":{

"media":{

"origin":"beachmint-origin-domain.com",

"domain":"beachmint-cloudfront.cloudfront.net"

}

},

"solr":{

"gittag":"master",

"server_name":"beachmint-solr-domain.com",

"nodes":{ "1":"node-ip-1",

"2":"node-ip-2",

"3":"node-ip-3" }

}

}

The Glue: Custom JSON

"cdn":{

"media":{

"origin":"beachmint-origin-domain.com",

"domain":"beachmint-cloudfront.cloudfront.net" }},

"solr":{

"gittag":"master",

"server_name":"beachmint-solr-domain.com",

"nodes":{

"1":"node-ip-1",

"2":"node-ip-2",

"3":"node-ip-3" }} Configuration and

attributes of

supporting systems

{

"environment":{

"env":"production"

},

"jewelmint":{

"gittag":"20131008d_release" },

"intimint":{

"gittag":"v3.7.0_release" },

"stylemint":{

"gittag":"v3.7.0_release" },

"shoemint":{

"gittag":"v3.7.0_release" },

"cdn":{

"media":{

"origin":"beachmint-origin-domain.com",

"domain":"beachmint-cloudfront.cloudfront.net"

}

},

"solr":{

"gittag":"master",

"server_name":"beachmint-solr-domain.com",

"nodes":{ "1":"node-ip-1",

"2":"node-ip-2",

"3":"node-ip-3" }

}

}

Commit

Jenkins post commit hook

Get instances

Update code on instances

Run selenium or unit tests

Stop

pass Ready for

QA Create new

tag Update git tag on custom json

Update code on instances

Run load test

Build

Sta

ck

Load Stack

Y

N

use Aws\Common\Aws;

$aws = Aws::factory('./config.php');

$opsWorks = $aws->get('OpsWorks');

//Get the Stack we are updating

$onlineInstanceIds = array();

$allInstances = $opsWorks->describeInstances(array('LayerId' => $layerId))-

>get("Instances");

foreach($allInstances as $instance) {

if ($instance['Status'] == 'online') {

$onlineInstanceIds[] = $instance['InstanceId'];

}

}

Commit

Jenkins post commit hook

Get instances

Update code on instances

Run selenium or unit tests

Stop

pass Ready for

QA Create new

tag Update git tag on custom json

Update code on instances

Run load test

Build

Sta

ck

Load Stack

Y

N

use Aws\Common\Aws;

$aws = Aws::factory('./config.php');

$opsWorks = $aws->get('OpsWorks');

$deployment = $opsWorks->createDeployment(array(

'StackId' => $stackId,

'Command' => array(

'Name' => 'execute_recipes',

'Args' => array(

'recipes' => $recipe

)

),

'InstanceIds' => $onlineInstanceIds

));

Load Testing

• Code

• Configuration

• EC2 instance type

• System architecture configuration

• Capacity planning

Commit

Jenkins post commit hook

Get instances

Update code on instances

Run selenium or unit tests

Stop

pass Ready for

QA Create new

tag Update git tag on custom json

Update code on instances

Run load test

Build

Sta

ck

Load Stack

Y

N

//Get the Stack’s custom JSON from OpsWorks

$customJsonArray = json_decode($theBuildStack[0]["CustomJson"]);

//Replace the gittag with the new tag

$customJsonArray->{"jewelmint"}->{"gittag"} = "thenewgittag";

//Convert back to string

$modifiedJson = json_encode($customJsonArray);

//Update Stack settings in OpsWorks

$updateRes = $opsWorks->updateStack(array("StackId" => "the-build-

stackid","CustomJson" => $modifiedJson));

Commit

Jenkins post commit hook

Get instances

Update code on instances

Run selenium or unit tests

Stop

pass Ready for

QA Create new

tag Update git tag on custom json

Update code on instances

Run load test

Build

Sta

ck

Load Stack

Y

N

use Aws\Common\Aws;

$aws = Aws::factory('./config.php');

$opsWorks = $aws->get('OpsWorks');

$deployment = $opsWorks->createDeployment(array(

'StackId' => $stackId,

'Command' => array(

'Name' => 'execute_recipes',

'Args' => array(

'recipes' => $recipe

)

),

'InstanceIds' => $onlineInstanceIds

));

Path to Production

• QA and Product group determines what goes to

production

• Each release candidate has a tag associated

with it

• QA updates QA and Stage environments using a

tool based on the AWS SDK

Thanks!

www.jewelmint.com

www.stylemint.com

www.intimint.com

www.shoemint.com

Automate Standard Operating Procedures

3 AM – Alarm goes off

Wouldn’t it be nice to have help?

The idea: gather and ship logs to Amazon S3

as soon as the CPU load is to high

We will use the

following setup

AWS OpsWorks stores 1-minute

metrics in CloudWatch

Every instance creates an

alarm for high CPU load

CloudWatch alarm action:

write to SNS topic

SNS publishes to

queue in SQS

Watcher instance polls

SQS for notifications

On alarm notification,

call OpsWorks API to …

Execute a script on the

affected server

Logs are gathered and

written to Amazon S3

And Again

DEMO TIME

Demo Recap

• Instances set up alarms on CloudWatch

• Alarms create notification in SNS

• SNS publishes to SQS queue

• Watcher instance polls queue and calls AWS

OpsWorks to execute recipe on instance

• The recipe runs a script to upload logs to S3

More Information about AWS OpsWorks

• If not done yet, do the AWS OpsWorks lab!

• Find us in the AWS Booth

• Follow us on Twitter @AWSOpsWorks

• Find us on YouTube

• AWS OpsWorks survey

http://tinyurl.com/OpsWorksSurvey2013

Other Talks During re:Invent

DMG304 - AWS OpsWorks Under the Hood

• Jonathan Weiss & Reza Spagnolo

• Thursday, Nov 14, 3:00 PM - 4:00 PM – Murano 3206

DMG305 - How Intuit Leveraged AWS OpsWorks as the Engine of Our PaaS

• Capen Brinkley & Rick Mendes of Intuit, Inc.

• Thursday, Nov 14, 4:15 PM - 5:15 PM – Murano 3206

Please give us your feedback on this

presentation

As a thank you, we will select prize

winners daily for completed surveys!

DMG202

Backup Slides

Recipes

• Default Writes Ruby script to the instance and runs it

• create_alarm Writes a Ruby script to create an alarm and ties it to the right SNS

topic

• send_logs Writes a Ruby script to the instance that can pack and ship the logs

to S3

Watcher Code

queue = AWS::SQS.new.queues["<%= node[:watcher][:sqs][:url] %>"]

queue.poll(:initial_timeout => false) do |msg|

begin

alarm = JSON.parse(msg.body)['Subject'].start_with?('ALARM')

instance_id = JSON.parse((JSON.parse(msg.body)['Message']))['Trigger']

['Dimensions'].first['value']

if alarm

opsworks = AWS::OpsWorks.new.client

deployment = opsworks.create_deployment(

:stack_id => "<%= node[:opsworks][:stack][:id] %>",

:instance_ids => [instance_id],

:command => {

:name => 'execute_recipes',

:args => {'recipes' => <%= node[:watcher][:execute_recipes] %>}

},) end end end

Alarm Creation

id = "<%= node[:opsworks][:instance][:id] %>"

action ="<%= node[:watcher][:sns][:arn] %>"

alarm = AWS::CloudWatch.new.alarms.create("#{id}_cpu_idle", {

:namespace => 'AWS/OpsWorks',

:metric_name => 'cpu_idle',

:dimensions => [{:name => 'InstanceId', :value => id}],

:comparison_operator => 'LessThanOrEqualToThreshold',

:evaluation_periods => 1,

:period => 60,

:statistic => 'Average',

:threshold => 20,

:actions_enabled => true,

:alarm_actions => ["#{action}"],

:alarm_description => 'watching the available CPU on the instance'

})

Pack and Send Logs – Generated Ruby Script

timestamp = Time.now.utc.to_i

archive = "/tmp/#{timestamp}.tgz"

source = "/var/log"

s3bucket = "<%= node[:watcher][:s3][:bucket] %>"

object = "<%= node[:opsworks][:stack][:name] %>/<%= node[:opsworks][:instance][:hostname]

%>/#{timestamp}.tgz"

`cd #{File.dirname(source)} && tar czf #{archive} #{File.basename(source)}`

s3 = AWS::S3.new.buckets[s3bucket].objects[object].write(:file => archive)