Post on 13-Jan-2015
description
amazonweb servicesamazonweb services
AWS 201T WorkshopJames Saull
Solution Architect jsaull@amazon.com
Agenda• House Keeping & Setup
• Introduction to Amazon Web Services
• Lab: Building a Web Property on AWS– Storage: S3, EBS
– Compute
– Content Delivery
– Relational Database
– Dynamo DB
• Lab: Scalability and Availability– Snapshots
– Load Balancing
– Auto Scaling
– Security
• Log Processing Scenario– Logging to AWS
– Elastic Map Reduce
Questions ?
House Keeping• Breaks
Setup
• aws.amazon.com
• Be sure you have permissions and are active
• Firefox or Chrome
• Windows download Putty: www.putty.org
Watch out for unexpected Costs
When the Technical Workshop comes to an end, to avoid unwanted costs:
• Delete your S3 objects
• Destroy your CloudFront distributions
• Stop or Shut Down your EC2 and RDS instances
The customer is responsible for the resources he’s using. AWS declines any responsibility if the customer forgets to shut down resources.
Agenda• House Keeping & Setup
• Introduction to Amazon Web Services
• Lab: Building a Web Property on AWS– Storage: S3, EBS
– Compute
– Content Delivery
– Relational Database
• Lab: Scalability and
Availability– Snapshots
– Load Balancing
– Auto Scaling
– Security
• Log Processing Scenario– Logging to AWS
– Elastic Map Reduce
– Dynamo DB
Who is Amazon.com?Who is Amazon.com?
8
Amazon.com’s Three BusinessesAmazon.com’s Three Businesses[[ ]]
Consumer business
Seller business
IT Infrastructure business
Consumer business
Seller business
IT Infrastructure business
}}
9
Deep experience in
building and operating
global web scale
systems
About Amazon
Web Services
?…get into cloud computing?
How did Amazon…
What is Amazon Web Services?What is Amazon Web Services?
– Reliable, scalable, low-cost infrastructure
– Every service offers APIs
– You can use only the components you need
– All the important “building blocks” are provided
– Reliable, scalable, low-cost infrastructure
– Every service offers APIs
– You can use only the components you need
– All the important “building blocks” are provided
]][
RegionRegion GovCloud RegionGovCloud Region
Global AWS InfrastructureGlobal AWS Infrastructure ]][[
Availability ZoneAvailability Zone
Global AWS InfrastructureGlobal AWS Infrastructure ]][[
Edge LocationsEdge Locations
Global AWS InfrastructureGlobal AWS Infrastructure ]][[
Dallas (2) Dallas (2)
St.LouisSt.Louis
MiamiMiami
JacksonvilleJacksonvilleLos Angeles (2)Los Angeles (2)
Palo AltoPalo Alto
SeattleSeattle
Ashburn
(2)
Ashburn
(2)
NewarkNewarkNew York (3)New York (3)
DublinDublin
London (2)London (2)Amsterdam (2)Amsterdam (2)StockholmStockholm
Frankfurt (2)Frankfurt (2)Paris
(2)
Paris
(2)
Singapore
(2)
Singapore
(2)
Hong Kong
(2)
Hong Kong
(2)
Tokyo (2)Tokyo (2)
Sao PauloSao Paulo
South BendSouth Bend
San JoseSan Jose
OsakaOsakaMilanMilan
Sydney
Hayward
Madrid
The Amazon Web Services universeThe Amazon Web Services universe[[ ]]
amazonweb servicesamazonweb services
The Amazon Web Services universeThe Amazon Web Services universe[[ ]]
SDKSDK
APIAPI
Web ConsoleWeb Console
Command LineCommand Line
[[ ]]
Management +
Interface
Management +
Interface
Cross
Service
Features
Cross
Service
Features
CloudFormationCloudFormationSimple WorkFlowSimple WorkFlow
CloudWatchCloudWatch
Elastic BeanStalkElastic BeanStalk
IAMIAM
The Amazon Web Services universeThe Amazon Web Services universe
(some services are omitted here)(some services are omitted here)
Simple Notification Service
[[ ]]
Management +
Interface
Management +
Interface
Platform
Building
blocks
Platform
Building
blocks
RDSRDS
EMREMR
DynamoDBDynamoDB
CloudFrontCloudFrontSQSSQS
CloudSearchCloudSearchSESSES
Cross
Service
Features
Cross
Service
Features
The Amazon Web Services universeThe Amazon Web Services universe
(some services are omitted here)(some services are omitted here)
RedshiftRedshift
Infrastructure
Building
Blocks
Infrastructure
Building
Blocks
[[ ]]
Management +
Interface
Management +
Interface
Platform
Building
blocks
Platform
Building
blocks
Cross
Service
Features
Cross
Service
Features
GlacierGlacier EBSEBS S3S3Virtual Private CloudVirtual Private Cloud
The Amazon Web Services universeThe Amazon Web Services universe
(some services are omitted here)(some services are omitted here)
EC2
[[ ]]
Management +
Interface
Management +
Interface
RDS CloudSearch
SES CloudFront SQS
EMR DynamoDB
RDS CloudSearch
SES CloudFront SQS
EMR DynamoDB
Elastic Beanstalk
Simple WorkFlow
CloudFormation
CloudWatch
IAM
Elastic Beanstalk
Simple WorkFlow
CloudFormation
CloudWatch
IAM
VPC EC2
EBS S3
VPC EC2
EBS S3
(some services are omitted here)(some services are omitted here)
Infrastructure
Building
Blocks
Infrastructure
Building
Blocks
Platform
Building
blocks
Platform
Building
blocks
Cross
Service
Features
Cross
Service
Features Command Line
Web Console
APIs
SDK
Command Line
Web Console
APIs
SDK
The Amazon Web Services universeThe Amazon Web Services universe
Agenda• House Keeping & Setup
• Introduction to Amazon Web Services
• Lab: Building a Web Property on AWS– Storage: S3, EBS
– Compute
– Content Delivery
– Relational Database
– Dynamo DB
• Lab: Scalability and Availability– Snapshots
– Load Balancing
– Auto Scaling
– Security
• Log Processing Scenario– Logging to AWS
– Elastic Map Reduce
Labs
During this workshop, we will build from scratch a highly available, redundant,
scalable web property on AWS.
Storage
Amazon Simple Storage Service (S3)
Amazon S3 Bucket Volume SnapshotAmazon Elastic
Block Storage
(EBS)
Amazon Elastic Block Storage (Amazon EBS)
Our Building Blocks
Compute
Our Building Blocks
Amazon Elastic Compute Cloud (EC2)
Amazon EC2 Instance Instances AMI DB on
Instance
Instance with
CloudWatch
Elastic IP
Content Delivery & Database
Our Building Blocks
Amazon Database Services RDS/DDB
Amazon RDS MySQL DB Instance DynamoDB
Amazon Cloudfront
Amazon
Cloudfront
Download
Distribution
Edge LocationStreaming
Distribution
console.aws.amazon.com
• Object-based storage
(no Filesystem)
• Easily store/retrieve data
• Durability of 99.999999999% or
99.99%
• Integrated with other AWS
Services
• Scalable
• Redundancy is managed
transparently
• File (Object): up to 5 TB each
• HTTP, HTTPS, BitTorrent
protocols
28
Amazon S3Simple Storage Service
Data
Any Amazon S3 Region
Your Data
Data
Data
DataData
Amazon S3 Redundancy
Data is replicated
multiple times
In case of failure,
data is replicated again,
transparently
• Backup
• Archive (Glacier)
• Disaster Recovery (DR)
• Content Storage
• Website
30
Amazon S3 Use cases
Lab Exercise
• Create an S3 Bucket
• Upload and Download a File
• Check out properties and permissions
Questions ?
• Delivery of content
worldwide
• Static: HTTP / HTTPS
• Streaming: RTMP
• Dynamic Content
• Origin server: S3, EC2,
non-AWS
• Public/Private
Distributions
33
Amazon CloudFrontContent Delivery Network
Regions, Availability Zones, Edge Locations
Dallas
St.Louis Miami
Jacksonville
Los Angeles (2)
Palo Alto
Seattle
Ashburn
Newark
New York (2)
Dublin
London
Amsterdam
Stockholm
FrankfurtParis
Singapore
Hong Kong
Tokyo
Sao Paulo
South Bend
San Jose
38 Edge Locations in total (as of Dec 2012)
Sydney
Let’s simplify a bit:we consider only a few of them
Stockholm
Hong Kong
Sao Paulo
San Jose
Your web servers
in Singapore
Stockholm
Hong Kong
Sao Paulo
San Jose
Content Delivery Network:How it works
Dynamic pages (PHP, Java) (from web servers)
Static content or streaming (with CloudFront)
Your web servers
in Singapore
• Accelerated web content delivery
• Off-load traffic from web servers
• Big spikes in traffic
• Event streaming
• Marketing campaigns
37
Amazon CloudFront Use cases
Lab Exercise
• Create a New Distribution
• Test the Content
Questions ?
• Linux / Windows virtual
machines
• Elastic (scale in minutes)
• Many Instance types
(micro, small, large...)
• AMI: Amazon Machine
Image
• VPC: Virtual Private Cloud
• Security (Groups, Keys...)
• VM Import
• Auto Scaling, Elastic Load
Balancing
40
Amazon EC2Elastic Compute Cloud
Availability ZoneAvailability Zone
I see 20 Availability Zones in total (as of Feb 2013)I see 20 Availability Zones in total (as of Feb 2013)
• Media
• Hosting
• High Performance
Computing
• Dev & Test
• Internal Applications
• Gaming
• ... Everything that
needs computing!
42
EC2 Use cases
Lab Exercise• Create a new Security Group
• Launch an Amazon EC2 instance (Linux)
• Log in with SSH as ec2-user@
• Install a web server
• Create a simple web page
• Test it on a browser: it works!
• Create and attach an Elastic IP
• Create an AMI from an EC2 Instance
Demo Windows• Create a new Security Group for Windows
• Launch a new EC2 instance (Windows)
• Log in with RDP
From Windows: log in with RDP
Questions ?
• Block Level Storage for use
with EC2
• Volume: 1 GB to 1 TB
• Raw unformatted block
device
• Local to an Availability Zone
• Redundant
• Persistent
• Point-in-time snapshots to
Amazon S3
• Integration with CloudWatch
46
EBSElastic Block Storage
Demo EBS• Create a new EBS volume
• Attach it to an EC2 Instance
• Partition it and create a File System
• Mount it; create a file in it
• Create an EBS Snapshot
• Create a new volume from the Snapshot
Questions ?
• Relational Database “as a
Service”
• Simple to Deploy
• Managed by the AWS team
• MySQL, SQLServer or
Oracle (as of 2012)
• Scalable
• Optional: automatic Standby
Replica
• Optional: multiple Read-Only
copies
• Easy DB Snapshots and
automated backup
49
RDSRelational Database Service
Demo RDS• Create a DB Instance on RDS (MySQL)
• Enable Multi-AZ Deployment
• Enable one Read Replica
• Optional: connect to the DB Instance
Questions ?
Agenda• House Keeping & Setup
• Introduction to Amazon Web Services
• Lab: Building a Web Property on AWS– Storage: S3, EBS
– Compute
– Content Delivery
– Relational Database
– Dynamo DB
• Lab: Scalability and Availability– Snapshots
– Load Balancing
– Auto Scaling
– Security
• Log Processing Scenario– Logging to AWS
– Elastic Map Reduce
Let’s take our Web Application to
the next level
EBS
Elastic Load Balancer• Automatically distribute incoming traffic to
multiple Amazon EC2 instances (in the same
Region).
• Automatic Health check
• IPv6 support
• Can be integrated with AutoScaling
Questions ?
Snapshots & AMIs
• Copies of EBS Volumes
• Essential to Reusability
• Copy between Regions
• Durability in S3
Lab Exercise• Duplicate your entire architecture by
making an AMI
• Increase your availability by spreading your application across availability zones
• Bring up an ELB in front of your website
• Optional – Create a CNAME to the ELB
Questions ?
CloudWatch
• Integrated across AWS platform
• Nothing to install
• Custom Metrics
• Set Alarms
• View graphs and statistics
• Feeds into Autoscaling
Lab Exercise• Check out the metrics of your web
application
• Set up an alarm to monitor your site
Questions ?
AutoScaling
• Auto Up and Auto Down
• Runs on CloudWatch metrics
• Notifications via SNS
• Spot or On-demand
• No additional Fees
AutoScaling
• Launch config: AMI to be used
• Autoscaling group: where/how to launch
• Autoscaling policy: what should AS do
• Autoscaling trigger: what will activate AS
Demo Autoscaling
JMETER
Questions ?
Security
• Security Groups
• Granular tiered secure architecture
• Roles for services
• Best Practices - Bastions
Before we get started on the controls…
• AWS Reports, Certifications & Accreditations• SOC 1, Type 2 report
• SOC 2 report
• ISO 27001
• PCI DSS Level 1 service provider
• FISMA Moderate
• MPAA
• Look at http://aws.amazon.com/security
A Protected
Infrastructure:
IAM
VPC
ACL
ELB
AZ
SG
TMG
RDP
WSUS
AV
Lab Exercise• Secure your website by creating a DMZ
between the ELB and your application
• Create a bastion host
Questions ?
Agenda• House Keeping & Setup
• Introduction to Amazon Web Services
• Lab: Building a Web Property on AWS– Storage: S3, EBS
– Compute
– Content Delivery
– Relational Database
– Dynamo DB
• Lab: Scalability and Availability– Snapshots
– Load Balancing
– Auto Scaling
– Security
• Log Processing Scenario– Logging to AWS
– Elastic Map Reduce
Getting your Data into S3
S3Console Upload
FTP
S3 API
AWS Import / Export
Direct Connect
Tsunami UDP
Storage Gateway
3rd Party Commercial
Applications
CloudFrontFlume
AWS Data Pipeline
S3 and Big Data
• Why S3?
• Hadoop Overview
• Hadoop on the Cloud
• Hadoop File System
Questions ?
Introducing Apache Hadoop
• Apache Hadoop• Software for distributed data analysis
• Map/Reduce framework
• Focus on data
• But• Complex
• Hard to setup
• Cap-ex intensive
• Difficult to manage
EMR is Hadoop in the Cloud
Hadoop is an open-source framework for
a cluster of machines
Hadoop is an open-source framework for
parallel processing huge amounts of data on
a cluster of machines
What is Amazon Elastic MapReduce (EMR)?
How does it work?
EMR
EMR ClusterS3
Put the data
into S3
Choose: Hadoop distribution, #
of nodes, types of nodes, custom
configs, Hive/Pig/etc.
Get the output
from S3
Launch the cluster using
the EMR console, CLI, SDK,
or APIs
You can also store
everything in HDFS
011001101
DynamoDB and Big Data
• What is Dynamo?
• Dynamo in Big Data – Volume & Velocity
Questions ?
Watch out for unexpected Costs
When the Technical Workshop comes to an end, to avoid unwanted costs:
• Delete your S3 objects
• Destroy your CloudFront distributions
• Stop or Shut Down your EC2 and RDS instances
The customer is responsible for the resources he’s using. AWS declines any responsibility if the customer forgets to shut down resources.
amazonweb servicesamazonweb services
Thank YouJames Saull
Solution Architect jsaull@amazon.com