Improving Infrastructure Governance on AWS - AWS June 2016 Webinar Series
Aws for the Retail Industry, Webinar, September 2012
-
Upload
amazon-web-services -
Category
Technology
-
view
1.790 -
download
3
description
Transcript of Aws for the Retail Industry, Webinar, September 2012
![Page 1: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/1.jpg)
AWS for the Retail Industry
Ryan Shuttleworth – Technical Evangelist @ryanAWS
Adam Bidwell – eCommerce Manager, Kurt Geiger
![Page 2: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/2.jpg)
Agenda
Amazon Web Services Background
Utility computing & Elasticity
AWS & Retail
Security & compliance
Highly available customer facing systems
Core platforms
Customer analytics
Kurt Geiger – Customer Story
![Page 3: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/3.jpg)
Your feedback is important
Tell us:
What’s good, what’s not
What you want to see at these events
What you want AWS to deliver for
you
![Page 4: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/4.jpg)
background
![Page 5: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/5.jpg)
Consumer Business
Tens of millions of active customer
accounts
Eight countries: US, UK, Germany,
Japan, France, Canada, China, Italy
Seller Business
Sell on Amazon websites
Use Amazon technology for your own retail website
Leverage Amazon’s massive fulfillment
center network
IT Infrastructure Business
Cloud computing infrastructure for hosting web-scale
solutions
Hundreds of thousands of
registered customers in over 190 countries
![Page 6: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/6.jpg)
Deep experience in building
and operating global web
scale systems
About Amazon Web Services
?
…get into cloud computing?
How did Amazon…
![Page 7: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/7.jpg)
Over 10 years in the making
Enablement of sellers on Amazon
Internal need for scalable deployment environment
Early forays proved developers were hungry for more
![Page 8: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/8.jpg)
AWS Mission
Enable businesses and developers to use web services* to build scalable,
sophisticated applications.
*What people now call “the cloud”
![Page 9: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/9.jpg)
![Page 10: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/10.jpg)
Each day AWS adds the equivalent server capacity to power Amazon when it was a global, $2.76B
enterprise
(circa 2000)
![Page 11: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/11.jpg)
0.000
250.000
500.000
750.000
1000.000
1 Trillion
750k+ peak transactions per second
Objects in S3
![Page 12: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/12.jpg)
Utility computing
![Page 13: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/13.jpg)
Utility computing
On demand Pay as you go
Uniform Available
![Page 14: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/14.jpg)
On demand Pay as you go
Uniform Available
Utility computing
![Page 15: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/15.jpg)
Utility computing
![Page 16: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/16.jpg)
Compute
Storage
Security Scaling
Database
Networking Monitoring
Messaging
Workflow
DNS
Load Balancing
Backup CDN
On demand Pay as you go
Uniform Available
Utility computing
![Page 17: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/17.jpg)
On a global footprint
Region
US-WEST (N. California) EU-WEST (Ireland)
ASIA PAC (Tokyo)
ASIA PAC (Singapore)
US-WEST (Oregon)
SOUTH AMERICA (Sao Paulo)
US-EAST (Virginia)
GOV CLOUD
![Page 18: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/18.jpg)
Availability Zone
On a global footprint
![Page 19: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/19.jpg)
Edge Locations
Dallas(2)
St.Louis
Miami
Jacksonville Los Angeles (2)
Palo Alto
Seattle
Ashburn(2)
Newark
New York (2)
Dublin
London(2)
Amsterdam
Stockholm
Frankfurt(2)
Paris(2)
Singapore(2)
Hong Kong
Tokyo
Sao Paulo
South Bend
San Jose
Osaka Milan
Sydney
On a global footprint
![Page 20: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/20.jpg)
Elasticity
![Page 21: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/21.jpg)
Traditional IT
capacity
Elastic capacity
Capacity
Time Your IT needs
![Page 22: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/22.jpg)
On and Off Fast Growth
Variable peaks Predictable peaks
Elastic capacity
![Page 23: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/23.jpg)
On and Off Fast Growth
Predictable peaks Variable peaks
WASTE
CUSTOMER DISSATISFACTION
Elastic capacity
![Page 24: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/24.jpg)
Elastic cloud capacity
Traditional
IT capacity
Your IT needs
Time
Capacity
Elastic capacity
![Page 25: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/25.jpg)
Fast Growth On and Off
Predictable peaks Variable peaks
Elastic capacity
![Page 26: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/26.jpg)
503 Service Temporarily Unavailable
The server is temporarily unable to service
your request due to maintenance downtime or
capacity problems. Please try again later.
![Page 27: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/27.jpg)
503 Service Temporarily Unavailable
The server is temporarily unable to service
your request due to maintenance downtime or
capacity problems. Please try again later.
![Page 28: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/28.jpg)
From one instance…
![Page 29: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/29.jpg)
…to thousands
![Page 30: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/30.jpg)
And back again…
![Page 31: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/31.jpg)
Num
ber
of E
C2 I
nsta
nces
4/12/2008 4/14/2008 4/15/2008 4/16/2008 4/18/2008 4/19/2008 4/20/2008 4/17/2008 4/13/2008
40 servers to 5000 in 3 days
EC2 scaled to peak of 5000 instances
“Techcrunched”
Launch of Facebook modification
Steady state of ~40 instances
![Page 32: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/32.jpg)
Security you can rely upon
![Page 33: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/33.jpg)
Foundation Services
Compute Storage Database Networking
AWS Global Infrastructure Regions
Availability Zones
Edge Locations Am
azo
n
Shared responsibility
![Page 34: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/34.jpg)
Foundation Services
Compute Storage Database Networking
AWS Global Infrastructure Regions
Availability Zones
Edge Locations Am
azo
n
Shared responsibility
Sarbanes-Oxley (SOX)
ISO 27001 Certification
Payment Card Industry Data Security
Standard (PCI DSS) Level 1 Compliant
SAS70(SOC 1) Type II Audit
FISMA A&As Multiple NIST Low Approvals to Operate (ATO) NIST Moderate, GSA issued ATO FedRAMP
DIACAP MAC III Sensitive IATO Customers have deployed various compliant applications such as HIPAA (healthcare)
![Page 35: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/35.jpg)
Foundation Services
Compute Storage Database Networking
AWS Global Infrastructure Regions
Availability Zones
Edge Locations
Client-side Data Encryption & Data Integrity Authentication
Server-side Encryption (File System and/or Data)
Network Traffic Protection (Encryption/Integrity/Identity)
Platform, Applications, Identity & Access Management
Operating System, Network & Firewall Configuration
Customer Data
Am
azo
n
You
Shared responsibility
![Page 36: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/36.jpg)
AWS and Retail
![Page 37: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/37.jpg)
Customer facing infrastructure
1
![Page 38: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/38.jpg)
DNS Application Data
Rule 1: Service all web requests
a) Make sure requests get to your ‘front door’
![Page 39: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/39.jpg)
DNS Application Data Request
Rule 1: Service all web requests
a) Make sure requests get to your ‘front door’
![Page 40: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/40.jpg)
DNS Application Data Request
a) Make sure requests get to your ‘front door’
Rule 1: Service all web requests
![Page 41: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/41.jpg)
DNS Application Data Request
…then this is irrelevant
Clients can’t resolve you?
Rule 1: Service all web requests
a) Make sure requests get to your ‘front door’
![Page 42: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/42.jpg)
DNS Application Data Request
“100% Available”
SLA
Rule 1: Service all web requests
Route53
Feature Details
Global Supported from AWS global edge locations for fast and reliable domain name resolution
Scalable Automatically scales based upon query volumes
Latency based routing Supports resolution of endpoints based upon latency, enabling multi-region application delivery
Integrated Integrates with other AWS services allowing Route 53 to front load balancers, S3 and EC2
Secure Integrates with IAM giving fine grained control over DNS record access
http://aws.amazon.com/route53/sla
a) Make sure requests get to your ‘front door’
![Page 43: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/43.jpg)
DNS Application Data Request
Rule 1: Service all web requests
a) Make sure requests get to your ‘front door’ b) Make sure you open the door when they arrive
Route53
![Page 44: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/44.jpg)
Region
DNS Application Data Request
Rule 1: Service all web requests
Elastic Load
Balancer Region
Availability Zone
Availability Zone
Availability Zone
Availability Zone
Route53
a) Make sure requests get to your ‘front door’ b) Make sure you open the door when they arrive
Elastic load balancing Multi-availability zone Multi-region
![Page 45: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/45.jpg)
Region
Rule 1: Service all web requests
DNS Application Data Request
Region
a) Make sure requests get to your ‘front door’ b) Make sure you open the door when they arrive c) Have the data to form a response
Elastic Load
Balancer Region
Availability Zone
Availability Zone
Availability Zone
Availability Zone
Route53
![Page 46: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/46.jpg)
Region
Rule 1: Service all web requests
DNS Application Data Request
Region
Elastic Load
Balancer
Route53
Region
Availability Zone
Availability Zone
Availability Zone
Availability Zone
a) Make sure requests get to your ‘front door’ b) Make sure you open the door when they arrive c) Have the data to form a response
Multi-AZ RDS
(Master-slave)
Inter-region replication
Read-replicas
![Page 47: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/47.jpg)
Rule 2: Service requests as fast as possible
![Page 48: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/48.jpg)
Rule 2: Service requests as fast as possible
a) Choose the fastest route
Region A
Route53
Region B
Request
![Page 49: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/49.jpg)
Rule 2: Service requests as fast as possible
a) Choose the fastest route
Region A
Route53
Region B
16ms 92ms
Request
![Page 50: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/50.jpg)
Rule 2: Service requests as fast as possible
a) Choose the fastest route
Region A
Route53
Region B
16ms 92ms
Request
![Page 51: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/51.jpg)
Rule 2: Service requests as fast as possible
Region A
Route53
Region B
16ms
Request
Region A DNS entry
a) Choose the fastest route
![Page 52: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/52.jpg)
Rule 2: Service requests as fast as possible
a) Choose the fastest route b) Offload your application servers
London
Paris
NY
Served from S3
/images/*
3
Served from EC2
*.php
2
Single CNAME
www.mysite.com
1
CloudFront World-wide content distribution network
Easily distribute content to end users with low
latency, high data transfer speeds, and no
commitments.
![Page 53: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/53.jpg)
Without CloudFront
EC2 webservers/app servers loaded by user
requests
Rule 2: Service requests as fast as possible
a) Choose the fastest route b) Offload your application servers
![Page 54: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/54.jpg)
With CloudFront
Load of user requests pushed into
CloudFront, EC2 cluster can scale
down
Offload Scale Down
Rule 2: Service requests as fast as possible
a) Choose the fastest route b) Offload your application servers
![Page 55: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/55.jpg)
Rule 2: Service requests as fast as possible
Res
po
nse
Tim
e
Serv
er L
oad
Res
po
nse
Tim
e
Serv
er
Load
Res
po
nse
Tim
e
Serv
er
Load
No CDN CDN for
Static
Content
CDN for
Static &
Dynamic
Content
Offload Scale Down
a) Choose the fastest route b) Offload your application servers
![Page 56: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/56.jpg)
Rule 3: Handle requests at any scale
a) Scale up
Vertical Scaling
From $0.02/hr
Basic unit of compute capacity
Range of CPU, memory & local disk options
14 Instance types available, from micro through cluster
compute to SSD backed
Scale up with Elastic Compute Cloud (EC2)
![Page 57: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/57.jpg)
Rule 3: Handle requests at any scale
a) Scale up b) Scale out
Trigger
auto-scaling
policy
as-create-auto-scaling-group MyGroup
--launch-configuration MyConfig
--availability-zones eu-west-1a
--min-size 4
--max-size 200
Auto-scaling Automatic re-sizing of compute clusters based upon demand
![Page 58: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/58.jpg)
Manually
Send an API call or use CLI to launch/terminate instances – Only need
to specify capacity change (+/-)
By Schedule
Scale up/down based on date and time
a) Scale up b) Scale out
By Policy
Scale in response to changing conditions, based on user configured real-time
monitoring and alerts
Auto-Rebalance
Instances are automatically launched/terminated to ensure the
application is balanced across multiple Azs
Rule 3: Handle requests at any scale
![Page 59: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/59.jpg)
Manually
Send an API call or use CLI to launch/terminate instances – Only need
to specify capacity change (+/-)
By Schedule
Scale up/down based on date and time Preemptive manual scaling of capacity
e.g. before a marketing event add 10 more instances
Regular scaling up and down of instances
e.g. scale from 0 to 2 to process SQS messages every night or double capacity
on a Friday night
a) Scale up b) Scale out
By Policy
Scale in response to changing conditions, based on user configured real-time
monitoring and alerts
Auto-Rebalance
Instances are automatically launched/terminated to ensure the
application is balanced across multiple Azs
Rule 3: Handle requests at any scale
Dynamic scale based upon custom metrics
e.g. SQS queue depth, Average CPU load, ELB latency
Maintain capacity across availability zones
e.g. Instance availability maintained in event of AZ becoming unavailable
![Page 60: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/60.jpg)
Rule 3: Handle requests at any scale
a) Scale up b) Scale out c) Dial it up
Elastic Block Store Provisioned IOPS up to 1000 per EBS
volume
Predictable performance for
demanding workloads such as
databases
DynamoDB Provisioned read/write performance per
table
Predictable high performance scaled via
console or API
![Page 61: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/61.jpg)
Relational Database Service Database-as-a-Service
No need to install or manage database instances
Scalable and fault tolerant configurations
DynamoDB Provisioned throughput NoSQL database
Fast, predictable performance
Fully distributed, fault tolerant architecture
Use RDS for databases
Use DynamoDB for high performance key-
value DB
Rule 4: Simplify architecture with services
![Page 62: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/62.jpg)
Amazon SQS
Processing
task/processing
trigger
Processing results
Amazon SQS Reliable, highly scalable, queue service
for storing messages as they travel
between instances
Task A
Task B
(Auto-scaling)
Task C
2
3
1
Simple Workflow Reliably coordinate processing steps
across applications
Integrate AWS and non-AWS resources
Manage distributed state in complex
systems
Push inter-process workflows into the cloud with SWF
Reliable message queuing without
additional software
Rule 4: Simplify architecture with services
![Page 63: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/63.jpg)
Cloud Search Elastic search engine based upon
Amazon A9 search engine
Fully managed service with
sophisticated feature set
Scales automatically
Document Server
Results
Search Server
Don’t install search software, use CloudSearch
Process large volumes of data cost effectively
with EMR
Elastic MapReduce Elastic Hadoop cluster
Integrates with S3 & DynamoDB
Leverage Hive & Pig analytics scripts
Integrates with instance types such as
spot
Rule 4: Simplify architecture with services
![Page 64: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/64.jpg)
![Page 65: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/65.jpg)
“Amazon CloudSearch is a game-changing product that has allowed us to deliver powerful
new search capabilities. Our customers can now find what they are looking for faster and more
easily than ever before…
….We saved many months of re-architecture and development time by going with Amazon
CloudSearch”
Don MacAskill CEO & Chief Geek
SmugMug
![Page 66: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/66.jpg)
![Page 67: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/67.jpg)
![Page 68: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/68.jpg)
10 Million records 44 GB collection more than 2,000 operations per second Order volumes increase substantially during the holidays necessitating elasticity
![Page 69: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/69.jpg)
Core platforms
2
![Page 70: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/70.jpg)
Certification of SAP BusinessObjects business intelligence solutions and SAP Rapid Deployment Solutions (RDS) on
Linux & Windows Server 2008 R2
Certification of SAP Business All-in-One on Linux & Windows Server 2008 R2
Certified database engines for production SAP deployments: MaxDB, DB2, MS SQL Server 2008 R2
![Page 71: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/71.jpg)
Non production systems
(dev, test, staging)
Backup, archive and recovery
(databases, AMIs)
http://aws.amazon.com/sap/
Production systems
(Analytics, branch etc)
![Page 72: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/72.jpg)
![Page 73: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/73.jpg)
Relational Database Service Database-as-a-Service
No need to install or manage database instances
Scalable and fault tolerant configurations
Feature Details
Platform support Create MySQL, SQL Server and Oracle RDBMS
Preconfigured Get started instantly with sensible default settings
Automated patching Keep your database platform up to date automatically
Backups Automatic backups and point in time recovery and full DB backups
Backups Volumes can be snapshotted for point in time restore
Failover Automated failover to slave hosts in event of a failure
Replication Easily create read-replicas of your data and seamlessly replicate data across availability zones
![Page 74: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/74.jpg)
Disaster recovery in AWS
Pilot light architecture
Build resources around
replicated dataset
Keep ‘pilot light’ on by replicating core
databases
Build AWS resources around dataset and
leave in stopped state
![Page 75: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/75.jpg)
Pilot light architecture
Build resources around
replicated dataset
Keep ‘pilot light’ on by replicating core
databases
Build AWS resources around dataset and
leave in stopped state
Scale resources in AWS in
response to a DR event
Start up pool of resources in AWS when
events dictate
Match current production capacity through
auto-scaling polcies
Disaster recovery in AWS
![Page 76: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/76.jpg)
Pilot light architecture
Build resources around
replicated dataset
Keep ‘pilot light’ on by replicating core
databases
Build AWS resources around dataset and
leave in stopped state
Scale resources in AWS in
response to a DR event
Start up pool of resources in AWS when
events dictate
Match current production capacity through
auto-scaling policies
Switch-over to system in AWS
Disaster recovery in AWS
![Page 77: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/77.jpg)
Customer analytics
3
![Page 78: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/78.jpg)
We can collect more
Big Data
![Page 79: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/79.jpg)
There is more
Big Data
![Page 80: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/80.jpg)
And data has gravity…
Big Data
![Page 81: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/81.jpg)
Data App App
http://blog.mccrory.me/2010/12/07/data-gravity-in-the-clouds/
Data has gravity
Compute Storage Big Data
![Page 82: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/82.jpg)
Data
http://blog.mccrory.me/2010/12/07/data-gravity-in-the-clouds/
…and inertia at volume…
Compute Storage Big Data
![Page 83: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/83.jpg)
Data
http://blog.mccrory.me/2010/12/07/data-gravity-in-the-clouds/
…easier to move applications to the data
Compute Storage Big Data
![Page 84: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/84.jpg)
Lorem ipsum dolor sit
amet, consectetur
adipiscing elit. Etiam
quis ligula neque, eget
venenatis sem.
Suspendisse non eros
nulla, at placerat nibh.
Very large dataset seeks strong &
consistent compute for
short term relationship,
possibly longer. GSOH a
plus aws.amazon.com
Personal
Lorem ipsum dolor sit
amet, consectetur
adipiscing elit. Etiam
quis ligula neque, eget
venenatis sem.
Suspendisse non eros
nulla, at placerat nibh.
Cras id lectus mattis est
ullamcorper blandit.
Proin ut nisi vitae enim
vulputate tempor.
Phasellus id commodo
eros. Mauris nec
dignissim turpis. Nunc
Cras id lectus mattis
est ullamcorper
blandit. Proin ut nisi
vitae enim vulputate
tempor. Phasellus id
commodo eros.
Mauris nec dignissim
turpis. Nunc
Bring compute capacity to the data
Compute Storage Big Data
Cras id lectus mattis
est ullamcorper
blandit. Proin ut nisi
vitae enim vulputate
tempor. Phasellus id
commodo eros.
Mauris nec dignissim
turpis. Nunc
![Page 85: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/85.jpg)
Cloud has the power to process
![Page 86: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/86.jpg)
From one instance…
Compute Storage Big Data
![Page 87: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/87.jpg)
…to thousands
Compute Storage Big Data
![Page 88: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/88.jpg)
and back again…
Compute Storage Big Data
![Page 89: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/89.jpg)
The revolution
![Page 90: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/90.jpg)
have data
![Page 91: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/91.jpg)
can store
have data
![Page 92: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/92.jpg)
can store can analyse
have data
![Page 93: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/93.jpg)
economically
![Page 94: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/94.jpg)
fast
![Page 95: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/95.jpg)
Who is your customer really?
What do people really like?
What is happening socially with your products?
How do people really use your products?
![Page 96: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/96.jpg)
96
![Page 97: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/97.jpg)
![Page 98: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/98.jpg)
Lesson 1: don’t leave your Amazon account logged in at home
Lesson 2: use the data you have to
drive proactive processes
![Page 99: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/99.jpg)
1 instance for 100 hours =
100 instances for 1 hour
![Page 100: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/100.jpg)
Small instance = $8
![Page 101: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/101.jpg)
![Page 102: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/102.jpg)
Amazon Elastic MapReduce
![Page 103: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/103.jpg)
Elastic MapReduce Managed, elastic Hadoop cluster
Integrates with S3 & DynamoDB
Leverage Hive & Pig analytics scripts
Integrates with instance types such as spot
Feature Details
Scalable Use as many or as few compute instances running Hadoop as you want. Modify the number of instances while your job flow is running
Integrated with other services Works seamlessly with S3 as origin and output. Integrates with DynamoDB
Comprehensive Supports languages such as Hive and Pig for defining analytics, and allows complex definitions in Cascading, Java, Ruby, Perl, Python, PHP, R, or C++
Cost effective Works with Spot instance types
Monitoring Monitor job flows from with the management console
![Page 104: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/104.jpg)
But what is it?
![Page 105: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/105.jpg)
A framework Splits data into pieces Lets processing occur
Gathers the results
![Page 106: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/106.jpg)
Elastic MapReduce
Code Name node
Output S3 + SimpleDB
S3 + DynamoDB
Elastic cluster
HDFS Queries
+ BI Via JDBC, Pig, Hive
Input data
![Page 107: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/107.jpg)
Very large click log (e.g TBs)
![Page 108: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/108.jpg)
Very large click log (e.g TBs)
Lots of actions by John Smith
![Page 109: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/109.jpg)
Very large click log (e.g TBs)
Lots of actions by John Smith
Split the log into
many small pieces
![Page 110: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/110.jpg)
Very large click log (e.g TBs)
Lots of actions by John Smith
Split the log into
many small pieces
Process in an EMR cluster
![Page 111: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/111.jpg)
Very large click log (e.g TBs)
Lots of actions by John Smith
Split the log into
many small pieces
Process in an EMR cluster
Aggregate the results
from all the nodes
![Page 112: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/112.jpg)
Very large click log (e.g TBs)
What John Smith
did
Lots of actions by John Smith
Split the log into
many small pieces
Process in an EMR cluster
Aggregate the results
from all the nodes
![Page 113: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/113.jpg)
What John Smith
did
Very large click log (e.g TBs) Insight in a fraction of the time
![Page 114: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/114.jpg)
1 instance for 100 hours =
100 instances for 1 hour
![Page 115: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/115.jpg)
Small instance = $8
![Page 116: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/116.jpg)
1 instance for 1,000 hours =
1,000 instances for 1 hour
![Page 117: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/117.jpg)
Small instance = $80
![Page 118: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/118.jpg)
Features powered by Amazon Elastic MapReduce:
People Who Viewed this Also Viewed
Review highlights Auto complete as you type on search
Search spelling suggestions Top searches
Ads
200 Elastic MapReduce jobs per day Processing 3TB of data
![Page 119: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/119.jpg)
“With AWS, our developers can now do things they couldn’t before…
…Our systems team can focus their energies on other
challenges.”
Dave Marin Search and data-mining engineer
![Page 120: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/120.jpg)
Elastic MapReduce Web log analysis and recommendation algorithms
![Page 121: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/121.jpg)
Adam Bidwell eCommerce Manager
![Page 122: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/122.jpg)
Overview of Kurt Geiger
Kurt Geiger are responsible for the operation of three retail websites: • Kurtgeiger.com • Shoeaholics.com • Ninewest.co.uk In total serving upwards of a half-million page views a day.
![Page 123: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/123.jpg)
Our interest in Amazon, is to host:
• Frontend systems - three Magento installations which the stores are built on.
• Administration systems – backend tasks, such
as product enrichment and reporting. • Testing – load-testing systems, and other
‘sandpit’ tasks • Research/Development – one-off installations for
investigation purposes.
![Page 124: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/124.jpg)
Challenges faced by Kurt Geiger:
• Rapidly changing business needs – fast pace
makes it difficult to predict long-term requirements
• Marketing activity – drives large traffic spikes
![Page 125: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/125.jpg)
Why Amazon?
• Unique model – we’ve used several cloud providers
Amazon offer a wide range of network/server infrastructure and services.
• Self-service – 24/7 help yourself approach, suits us to take what we need when we need it
![Page 126: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/126.jpg)
Future
• Larger capacity architectures
• More API based “pop-up” systems on demand
• Reserved instances - further cost savings
![Page 127: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/127.jpg)
Benefits
• Hourly billing – the cost adapts with our current set
up, no tie-in
• Large capacity – Whether capacity will be there is not a consideration, it just is
• Trusted provider – architecture still requires planning for good reliability, but AWS has robust infrastructure to build on
![Page 128: Aws for the Retail Industry, Webinar, September 2012](https://reader030.fdocuments.net/reader030/viewer/2022020306/54b869b54a795970478b491b/html5/thumbnails/128.jpg)
aws.amazon.com/free