Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

58
© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc. DMG202 - Zero to Sixty: AWS OpsWorks Thomas Metschke, Amazon November 13 th , 2013

description

AWS OpsWorks is a solution for managing applications of any scale or complexity on the AWS cloud. Accelerate your use of OpsWorks by learning how to use several of its operational features in this Zero to Sixty session. It starts with a demo of the OpsWorks main workflows—manage and configure instances, create and deploy apps, monitoring, and security. BeachMint will explain how they set up OpsWorks as part of their continuous deployment pipeline. The session finishes off by explaining how to use the OpsWorks API and Chef recipes to automate standard operating procedures. Demos and code samples are available to all session attendees. Are you new to AWS OpsWorks? Get up to speed for this session by first completing the 60-minute Introduction to AWS OpsWorks lab in the Self-Paced Hands-On Lab Lounge. It will lead you through all major functions of the service with a fun example.

Transcript of Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

Page 1: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

DMG202 - Zero to Sixty:

AWS OpsWorks

Thomas Metschke, Amazon

November 13th, 2013

Page 2: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

AWS OpsWorks

Helps you to model your entire application from

load balancers to databases starting from

templates for common technologies or building

your own.

You have full control of deployments, scaling,

monitoring, and automation of each component.

Page 3: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

AWS Application Management Services

Elastic Beanstalk OpsWorks CloudFormation EC2

Convenience Control

Higher-level services Do it yourself

Page 4: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

The Heart of AWS OpsWorks

Agent on each

EC2 instance

OpsWorks talks with

Page 5: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

The Heart of AWS OpsWorks

Agent on each

EC2 instance Understands a set of commands

that are triggered by OpsWorks.

The agent then runs Chef solo.

Page 6: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

The Heart of AWS OpsWorks

Agent on each

EC2 instance Understands a set of commands

that are triggered by OpsWorks.

The agent then runs Chef solo.

Page 7: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

The Heart of AWS OpsWorks

Agent on each

EC2 instance Understands a set of commands

that are triggered by OpsWorks.

The agent then runs Chef solo.

Page 8: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

AWS OpsWorks Agent Events

setup configure deploy undeploy shutdown

Page 9: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

A Stack

Page 10: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

A Stack with Layers

Page 11: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

A Stack with Layers and Instances

Page 12: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

Enough Talking

DEMO TIME

Page 13: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

• Created a new Stack from scratch

• Created an app and database layer

• Started instances in different Availability Zones

• Configured Auto Scaling

• Load balanced by Elastic Load Balancing

• Deployed an application

Recap of Demo

Page 14: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

Continuous Integration

Page 15: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

John Gaa VP of Engineering at BeachMint

Page 16: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

• Founded in 2011

• Next generation social commerce company

• Parent to: JewelMint, ShoeMint, StyleMint, & IntiMint

• Over 6 million registered customers

Page 17: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

Problem Set …

Page 18: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

Our Store Verticals are Powered By

• Amazon RDS

• Amazon ElastiCache

• Amazon SQS

• Amazon SNS

• Amazon CloudFront

• Amazon Redshift

RDS

ElastiCache

SQS

SNS

CloudFront

Redshift

Page 19: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

• Multiple releases a day

• 1 sysadmin

• 100+ instances between production

and test environments

Page 20: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

Jenkins & AWS OpsWorks to the Rescue

Page 21: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

Confidence to Release Every Day

• Unit tests

• Integration tests

• Load testing

• Manual testing

Page 22: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

What it Looks Like in AWS OpsWorks

Environments are represented as

stacks

Our different verticals are

represented as separate layers within

each stack

Page 23: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

Architecture Repeated in Each Environment

JewelMint IntiMint ShoeMint StyleMint

JavaScript widget

API

Checkout Registration Product Social

RDS Solr ElastiCache Drupal Drools

Layers

Sta

ck

Page 24: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

The Details

Page 25: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

AWS OpsWorks in Our CI Processes

• Get instances associated to a layer

• Update code based on gittag version stored in

custom JSON

• Target the instances and run Chef scripts

Page 26: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

The Glue: Custom JSON

{

"environment":{

"env":"production"

},

"jewelmint":{

"gittag":"20131008d_release" },

"intimint":{

"gittag":"v3.7.0_release" },

"stylemint":{

"gittag":"v3.7.0_release" },

"shoemint":{

"gittag":"v3.7.0_release" },

"cdn":{

"media":{

"origin":"beachmint-origin-domain.com",

"domain":"beachmint-cloudfront.cloudfront.net"

}

},

"solr":{

"gittag":"master",

"server_name":"beachmint-solr-domain.com",

"nodes":{ "1":"node-ip-1",

"2":"node-ip-2",

"3":"node-ip-3" }

}

}

"environment":{

"env":"production”

},

Determines what

environment to be built

Page 27: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

The Glue: Custom JSON

"jewelmint"{

"gittag":"20131008d_release" },

"intimint":{

"gittag":"v3.7.0_release" },

"stylemint":{

"gittag":"v3.7.0_release" },

"shoemint":{

"gittag":"v3.7.0_release" },

Code version to be

deployed to a layer

{

"environment":{

"env":"production"

},

"jewelmint":{

"gittag":"20131008d_release" },

"intimint":{

"gittag":"v3.7.0_release" },

"stylemint":{

"gittag":"v3.7.0_release" },

"shoemint":{

"gittag":"v3.7.0_release" },

"cdn":{

"media":{

"origin":"beachmint-origin-domain.com",

"domain":"beachmint-cloudfront.cloudfront.net"

}

},

"solr":{

"gittag":"master",

"server_name":"beachmint-solr-domain.com",

"nodes":{ "1":"node-ip-1",

"2":"node-ip-2",

"3":"node-ip-3" }

}

}

Page 28: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

The Glue: Custom JSON

"cdn":{

"media":{

"origin":"beachmint-origin-domain.com",

"domain":"beachmint-cloudfront.cloudfront.net" }},

"solr":{

"gittag":"master",

"server_name":"beachmint-solr-domain.com",

"nodes":{

"1":"node-ip-1",

"2":"node-ip-2",

"3":"node-ip-3" }} Configuration and

attributes of

supporting systems

{

"environment":{

"env":"production"

},

"jewelmint":{

"gittag":"20131008d_release" },

"intimint":{

"gittag":"v3.7.0_release" },

"stylemint":{

"gittag":"v3.7.0_release" },

"shoemint":{

"gittag":"v3.7.0_release" },

"cdn":{

"media":{

"origin":"beachmint-origin-domain.com",

"domain":"beachmint-cloudfront.cloudfront.net"

}

},

"solr":{

"gittag":"master",

"server_name":"beachmint-solr-domain.com",

"nodes":{ "1":"node-ip-1",

"2":"node-ip-2",

"3":"node-ip-3" }

}

}

Page 29: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

Commit

Jenkins post commit hook

Get instances

Update code on instances

Run selenium or unit tests

Stop

pass Ready for

QA Create new

tag Update git tag on custom json

Update code on instances

Run load test

Build

Sta

ck

Load Stack

Y

N

use Aws\Common\Aws;

$aws = Aws::factory('./config.php');

$opsWorks = $aws->get('OpsWorks');

//Get the Stack we are updating

$onlineInstanceIds = array();

$allInstances = $opsWorks->describeInstances(array('LayerId' => $layerId))-

>get("Instances");

foreach($allInstances as $instance) {

if ($instance['Status'] == 'online') {

$onlineInstanceIds[] = $instance['InstanceId'];

}

}

Page 30: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

Commit

Jenkins post commit hook

Get instances

Update code on instances

Run selenium or unit tests

Stop

pass Ready for

QA Create new

tag Update git tag on custom json

Update code on instances

Run load test

Build

Sta

ck

Load Stack

Y

N

use Aws\Common\Aws;

$aws = Aws::factory('./config.php');

$opsWorks = $aws->get('OpsWorks');

$deployment = $opsWorks->createDeployment(array(

'StackId' => $stackId,

'Command' => array(

'Name' => 'execute_recipes',

'Args' => array(

'recipes' => $recipe

)

),

'InstanceIds' => $onlineInstanceIds

));

Page 31: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

Load Testing

• Code

• Configuration

• EC2 instance type

• System architecture configuration

• Capacity planning

Page 32: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

Commit

Jenkins post commit hook

Get instances

Update code on instances

Run selenium or unit tests

Stop

pass Ready for

QA Create new

tag Update git tag on custom json

Update code on instances

Run load test

Build

Sta

ck

Load Stack

Y

N

//Get the Stack’s custom JSON from OpsWorks

$customJsonArray = json_decode($theBuildStack[0]["CustomJson"]);

//Replace the gittag with the new tag

$customJsonArray->{"jewelmint"}->{"gittag"} = "thenewgittag";

//Convert back to string

$modifiedJson = json_encode($customJsonArray);

//Update Stack settings in OpsWorks

$updateRes = $opsWorks->updateStack(array("StackId" => "the-build-

stackid","CustomJson" => $modifiedJson));

Page 33: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

Commit

Jenkins post commit hook

Get instances

Update code on instances

Run selenium or unit tests

Stop

pass Ready for

QA Create new

tag Update git tag on custom json

Update code on instances

Run load test

Build

Sta

ck

Load Stack

Y

N

use Aws\Common\Aws;

$aws = Aws::factory('./config.php');

$opsWorks = $aws->get('OpsWorks');

$deployment = $opsWorks->createDeployment(array(

'StackId' => $stackId,

'Command' => array(

'Name' => 'execute_recipes',

'Args' => array(

'recipes' => $recipe

)

),

'InstanceIds' => $onlineInstanceIds

));

Page 34: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

Path to Production

• QA and Product group determines what goes to

production

• Each release candidate has a tag associated

with it

• QA updates QA and Stage environments using a

tool based on the AWS SDK

Page 35: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

Thanks!

www.jewelmint.com

www.stylemint.com

www.intimint.com

www.shoemint.com

Page 36: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

Automate Standard Operating Procedures

Page 37: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

3 AM – Alarm goes off

Page 38: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

Wouldn’t it be nice to have help?

Page 39: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

The idea: gather and ship logs to Amazon S3

as soon as the CPU load is to high

Page 40: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

We will use the

following setup

Page 41: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

AWS OpsWorks stores 1-minute

metrics in CloudWatch

Page 42: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

Every instance creates an

alarm for high CPU load

Page 43: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

CloudWatch alarm action:

write to SNS topic

Page 44: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

SNS publishes to

queue in SQS

Page 45: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

Watcher instance polls

SQS for notifications

Page 46: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

On alarm notification,

call OpsWorks API to …

Page 47: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

Execute a script on the

affected server

Page 48: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

Logs are gathered and

written to Amazon S3

Page 49: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

And Again

DEMO TIME

Page 50: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

Demo Recap

• Instances set up alarms on CloudWatch

• Alarms create notification in SNS

• SNS publishes to SQS queue

• Watcher instance polls queue and calls AWS

OpsWorks to execute recipe on instance

• The recipe runs a script to upload logs to S3

Page 51: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

More Information about AWS OpsWorks

• If not done yet, do the AWS OpsWorks lab!

• Find us in the AWS Booth

• Follow us on Twitter @AWSOpsWorks

• Find us on YouTube

• AWS OpsWorks survey

http://tinyurl.com/OpsWorksSurvey2013

Page 52: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

Other Talks During re:Invent

DMG304 - AWS OpsWorks Under the Hood

• Jonathan Weiss & Reza Spagnolo

• Thursday, Nov 14, 3:00 PM - 4:00 PM – Murano 3206

DMG305 - How Intuit Leveraged AWS OpsWorks as the Engine of Our PaaS

• Capen Brinkley & Rick Mendes of Intuit, Inc.

• Thursday, Nov 14, 4:15 PM - 5:15 PM – Murano 3206

Page 53: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

Please give us your feedback on this

presentation

As a thank you, we will select prize

winners daily for completed surveys!

DMG202

Page 54: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

Backup Slides

Page 55: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

Recipes

• Default Writes Ruby script to the instance and runs it

• create_alarm Writes a Ruby script to create an alarm and ties it to the right SNS

topic

• send_logs Writes a Ruby script to the instance that can pack and ship the logs

to S3

Page 56: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

Watcher Code

queue = AWS::SQS.new.queues["<%= node[:watcher][:sqs][:url] %>"]

queue.poll(:initial_timeout => false) do |msg|

begin

alarm = JSON.parse(msg.body)['Subject'].start_with?('ALARM')

instance_id = JSON.parse((JSON.parse(msg.body)['Message']))['Trigger']

['Dimensions'].first['value']

if alarm

opsworks = AWS::OpsWorks.new.client

deployment = opsworks.create_deployment(

:stack_id => "<%= node[:opsworks][:stack][:id] %>",

:instance_ids => [instance_id],

:command => {

:name => 'execute_recipes',

:args => {'recipes' => <%= node[:watcher][:execute_recipes] %>}

},) end end end

Page 57: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

Alarm Creation

id = "<%= node[:opsworks][:instance][:id] %>"

action ="<%= node[:watcher][:sns][:arn] %>"

alarm = AWS::CloudWatch.new.alarms.create("#{id}_cpu_idle", {

:namespace => 'AWS/OpsWorks',

:metric_name => 'cpu_idle',

:dimensions => [{:name => 'InstanceId', :value => id}],

:comparison_operator => 'LessThanOrEqualToThreshold',

:evaluation_periods => 1,

:period => 60,

:statistic => 'Average',

:threshold => 20,

:actions_enabled => true,

:alarm_actions => ["#{action}"],

:alarm_description => 'watching the available CPU on the instance'

})

Page 58: Zero to Sixty: AWS OpsWorks (DMG202) | AWS re:Invent 2013

Pack and Send Logs – Generated Ruby Script

timestamp = Time.now.utc.to_i

archive = "/tmp/#{timestamp}.tgz"

source = "/var/log"

s3bucket = "<%= node[:watcher][:s3][:bucket] %>"

object = "<%= node[:opsworks][:stack][:name] %>/<%= node[:opsworks][:instance][:hostname]

%>/#{timestamp}.tgz"

`cd #{File.dirname(source)} && tar czf #{archive} #{File.basename(source)}`

s3 = AWS::S3.new.buckets[s3bucket].objects[object].write(:file => archive)