Amazon Web Services (slides) - Scott...

© 2010 R. Scott GrannemanLast updated 20101013

You are free to use this work, with certain restrictions.For full licensing information, please see the last slide/page.

SLUUGR. Scott Granneman

[email protected]

Jans [email protected]

A Short GuideAmazon Web Services

1Thursday, October 14, 2010

History


Launched July 2002Why?

Take advantage of underutilizedinfrastructure,technologies,& operations

that power Amazon’s$10 billion online store


60,000 systems320,000 CPUs

400 Gbps of bandwidth


June 2007330,000 developers using AWS

March 2010102,000,000,000 objects in S3


Common AWS characteristicsAimed at developers,

not end usersBilled on usage;

no contracts & no minimums99.95% reliability

(some o!er 99.999999999%!)


Current Services


ComputeContent Delivery

DatabaseE-CommerceMessagingMonitoringNetworking

Payments& BillingStorageSupport

Web Tra"cWorkforce

AWS Current Serviceshttp://aws.amazon.com/products/


ComputeElastic Compute Cloud (EC2)

“scalable, pay-as-you-gocompute capacity in the cloud”

Elastic MapReduce“easily & cost-e!ectively process

vast amounts of data”Auto Scaling

“automatically scale … EC2 capacityup or down

according to conditions you define”10Thursday, October 14, 2010

Content DeliveryCloudFront

“distribute contentwith low latency

via a global networkof edge locations”


DatabaseSimpleDB

“run queries on structured datain real time …

in conjunction with S3 and EC2”Relational Database Service (RDS)

“set up, operate, and scalea relational database in the cloud”


E-CommerceFulfillment Web Service (FWS)

“allows merchantsto deliver products

using Amazon.com’sworldwide fulfillment capabilities”


MessagingSimple Queue Service (SQS)

“hosted queue for storing messagesas they travel between computers,

making it easy to buildautomated workflow

between Web services”Simple Notification Service (SNS)

“set up, operate, and sendnotifications from the cloud”


MonitoringCloudWatch“monitoring

for AWS cloud resources,starting with Amazon EC2”


NetworkingVirtual Private Cloud (VPC)

“connectexisting enterprise infrastructure

to isolated AWS compute resourcesvia a VPN”

Elastic Load Balancing“automatically distributes

incoming application tra"cacross multiple EC2 instances”


Payments & BillingFlexible Payments Service (FPS)

“facilitates digital transfer of moneybetween any two entities,

humans or computers”DevPay

“billing & account management servicewhich enables developers

to collect paymentfor their AWS applications”


StorageSimple Storage Service (S3)

“fully redundantdata storage infrastructure”Elastic Block Storage (EBS)“block level storage volumes

for use with Amazon EC2 instances”AWS Import/Export“accelerates moving

large amounts of datainto and out of AWS”


SupportAWS Premium Support

“one-on-one, fast-responsesupport channel”


Web Tra"cAlexa Web Information Service

“huge repository of dataabout structure & tra"c patterns

on the Webavailable to developers”

Alexa Top Sites“exposes global website tra"c data

as it iscontinuously collected & updated”


WorkforceAmazon Mechanical Turk

“access thousands of global workerson demand

& programmatically integratetheir work

into various business processes”


S3


0

22,000,000,000

44,000,000,000

66,000,000,000

88,000,000,000

110,000,000,000

Oct. 2007 Jan. 2008 Oct. 2008 Mar. 2009 Mar. 2010

Number of Objects Stored at S3


Who uses S3?SmugMugDropbox

Ubuntu OneSlideshare

TwitterWoot


S3: Simple Storage ServiceWrite, read, & delete files

Each file can be 1 byte to 5 GBUnlimited number of files

Files can be public or privateor require authentication

Automatic torrents:append ?torrent to file’s URL

Versioning25Thursday, October 14, 2010

Objects are stored in bucketsLike folders, but not really

Bucket names must be uniqueacross all of S3,

not just your account


Domain mappingName a bucket test.foo.com

In DNS,create a CNAME recordpointing test.foo.com

totest.foo.com.s3.amazonaws.com.Note: no https access to that bucket

without a security alert


Pricing


S3 pricing1st 50 TB/month

Storage $0.15/GB

Data Transfer In Free ‘til 11/1/10Data Transfer In $0.10/GB

Data Transfer Out $0 for 1st GBData Transfer Out $0.11–0.15/GBPUT, COPY, POST, or LIST requests $0.01/1000 requests

GET & all other requests

$0.01/10,000 requests


Full pricing list ataws.amazon.com/s3/#pricing


Amazon Web ServicesSimple Monthly Calculator

calculator.s3.amazonaws.com/calc5.html


Using S3


Amazon provides a service,not software


Many of Amazon’s other servicesuse S3


GUI tools


Amazon Management Consoleconsole.aws.amazon.com/s3/home

Web-basedFree


S3Foxwww.s3fox.net

Firefox extension


DragonDiskwww.dragondisk.com

FreeWindows & Linux


Déjà Duplive.gnome.org/DejaDup/

FreeGNOME


s3fscode.google.com/p/s3fs/wiki/

FuseOverAmazonFUSEFree


Cyberduckcyberduck.ch

FreeMac OS X


Transmitwww.panic.com/transmit/

$35Mac OS X


S3fmwww.s3fm.com

Web-basedFree or license for integration


CLI tools


awstimkay.com/aws/

PerlFree


JetS3tjets3t.s3.amazonaws.com/index.html

JavaFree


s3cmds3tools.org/s3cmd

PythonFree

Read “s3cmd sync HowTo”s3tools.org/s3cmd-sync


S3syncwww.s3sync.net

RubyFree


Create~/.s3conf/s3config.yml


# Your AWS Access Keyaws_access_key_id: 0G60EWS0AZ09JTFCB2P2# Your AWS Secret Keyaws_secret_access_key: b9w5ytreBH8jfGTyplZaa0X2lnZx/uOHWgrymU+# Path to Certificate Authority keysssl_cert_dir: /Users/[username]/bin/s3sync/s3certs# Number of HTTP errors before s3sync exitss3sync_retries: 5# Character sets3sync_native_charset: UTF-8


$ ~/bin/s3sync/s3sync.rb--ssl

--delete--verbose--progress--recursive

/Users/rsgranne/Documents/Clientele/granneclientele:

Others:--dryrun--debug


EBS


EBS: Elastic Block StorageHard drive storage

mounted to EC2 instance1GB to 1TB

Snapshots saved to S3


EBS pricing$0.10/GB/month

$0.10/1 million IO requests


Public Data Sets on AWSaws.amazon.com/publicdatasets/

“repository of public data setsthat can be seamlessly integrated

into AWS cloud-based applications”


Sample Public Data SetsAnnotated Human Genome Data

Various US Census DatabasesUniGene

Freebase Data DumpSloan Digital Sky Survey

Wikipedia Extraction


Managing


Amazon EC2 API Toolsdeveloper.amazonwebservices.com/connect/entry.jspa?externalID=351

CLI for managing EBSGet info, attach storage, & more


EC2


EC2: Elastic Compute CloudServers on the fly,

running Linux or WindowsIncrease or decrease capacity

as needed, in minutesStatic IP addresses

associated with your account,not an instance

Elastic Load BalancingSnapshots saved to S3


Pricing


EC2 operating systems

Red Hat EL Server 2003OpenSolaris Server 2008

FedoraSUSE

GentooAmazon Linux

UbuntuDebian


EC2 instancesRAM CPU Storage Bits

Small 1.7 GB 1 160 GB 32

Large 7.5 GB 4 850 GB 64

Extra Large 15 GB 8 1690 GB 64

High-MemoryHigh-MemoryHigh-MemoryHigh-MemoryHigh-Memory

Quad Extra Large 68 GB 26 850 GB 64

High-CPUHigh-CPUHigh-CPUHigh-CPUHigh-CPU

Extra Large 7 GB 20 1690 GB 6467Thursday, October 14, 2010

EC2 instancesRAM CPU Storage Bits

Micro 613 MB 2 EBS 32/64


EC2 pricing

Small $0.085/hr $0.12/hr

Large $0.34/hr $0.48/hr

Extra Large $0.68/hr $0.96/hr

High-MemoryHigh-MemoryHigh-Memory

Quad Extra Large $2.00/hr $2.48/hr

High-CPUHigh-CPUHigh-CPU

Extra Large $0.68/hr $1.16/hr69Thursday, October 14, 2010

EC2 pricing

Micro $0.02/hr $0.03/hr


EC2 pricing

Data Transfer

InIn

1st 1 GB Out

Up to 10 TB Out

Next 40 TB Out

To & from S3

Free ‘til 11/1/10$0.10/GB

$0

$0.15/GB

$0.11/GB

$071Thursday, October 14, 2010

Small EC2 instance$0.085/hr

#24 hours

=$2.04/day

$14.28/week$63.24/month


Instances & AMIs


Uses Xen virtualizationA virtual machine is an instance


You could bootstrap everythingMuch easier to start with an AMI

(Amazon Machine Image)An AMI instantiates an instance


Amazon AMIsdeveloper.amazonwebservices.com/

connect/kbcategory.jspa?categoryID=171


SUSE Linux Enterprise Server 10Amazon Linux AMI (Cent OS)

Gentoo Linux 2010Ubuntu 10.04


BitNami LAMPStackBitNami Joomla! Stack

BitNami WordPress StackBitNami Trac Stack

BitNami MediaWiki StackBitNami phpBB Stack

BitNami DokuWiki Stack


IBM WebSphere Application ServerIBM Lotus Domino

IBM InfoSphereIBM Informix Dynamic Server

IBM DB2 Express


O"cial Ubuntu support for EC2


O"cial Ubuntu AMIsuec-images.ubuntu.com/releases/10.04/release/


Ubuntu Server on Cloud 10 ProgramAnyone can try out

Ubuntu 10.10 Server Editionon Amazon EC2

for free for one hour10.cloud.ubuntu.com


Alesticalestic.com

Ubuntu and Debian AMIsfor Amazon EC2


EC2 has no persistent storageTemporary storagesurvives reboots

but not terminationOriginally relied on S3 for booting

Now you can instead useEBS-backed volumes for booting


If you’re starting withan instance-store root device type

“Create EBS Boot AMI”www.capsunlock.net/

2009/12/create-ebs-boot-ami.html


If you’re starting withan EBS-backed instance

“Using Amazon EBS-BackedAMIs and Instances”

docs.amazonwebservices.com/AWSEC2/latest/UserGuide/

index.html?Using_BootFromEBS.html


The basic workflow is1. Create snapshots

of all volumesattached to currently running

EBS-backed instance2. Register an AMI

that references those snapshots


Full set of instructions atfiles.granneman.com/presentations/cloud/

HOWTO_Create_EBS-backed_AMI_for_AWS.txt


Managing


Amazon EC2 API Toolsdeveloper.amazonwebservices.com/connect/entry.jspa?externalID=351

CLI for managing EC2Get info, start, terminate,

attach storage, associate IP,& more


VPC


VPC: Virtual Private CloudConnect corporate infrastructure

to isolated EC2 instancesvia VPN


VPC pricing1st 10 TB/month

Connection $0.05/hour

Data Transfer In $0.10/GB

Data Transfer Out $0.15/GB


Reading


“Je! Bezos’ Risky Bet”Business Week (13 November 2006)www.businessweek.com/magazine/

content/06_46/b4009001.htm


AWS Documentationaws.amazon.com/documentation/


Thank you!

Email: [email protected]: www.granneman.com

Publications: www.granneman.com/pubsBlog: blog.granneman.comTwitter: scottgranneman


© 2010 R. Scott GrannemanLast updated 20101013

You are free to use this work, with certain restrictions.For full licensing information, please see the last slide/page.

SLUUGR. Scott Granneman

[email protected]

Jans [email protected]

A Short GuideAmazon Web Services


Amazon Web Services (slides) - Scott...

Documents

Transcript of Amazon Web Services (slides) - Scott...