Utilizing Amazo n S3 and EC2in Rails

Jonathan Weiss, Peritor Wissensmanagement GmbHRailsConf Europe Berlin, 2007

2

Starting Po int

QuickTime and aﾪTIFF (LZW) decompressor

are needed to see this picture.

One machine:

• Apache

• Ruby / Rails

• MySQL

3

Wo rst Case Po pulariy

4

A Difficult Path

QuickTimeﾪ and aTIFF (LZW) decompressor

are needed to see this picture.

5

Pro ble m: Backup

• High availability

• Redundancy

• Very big data sets

6

Pro ble m: File Syste m

• Important files have to be accessed by many servers

• NFS / Samba not practical

7

Pro ble m: Spo ntane o us Traffic

8

Pro ble m: Lo ad Fluctuatio n

9

Do n‘t re inve nt the w he e l!

10

Amazo n We b Se rvice s

Simple Storage Service - S3

Redundant Data Store

$0,15 per GB data per month

$0,10 - $0,20 per GB traffic

Virtual server per hour

$0,10 per CPU hour

$0,10 - $0,20 per GB traffic

Elastic Compute Cloud - EC2

…….

E-Commerce S..

11

S3 - Simple Sto rage Se rvice

• Redundant storage - as much as you like

• max. 5 GB per object

• Organized in „Buckets“

• Web Service API for uploads

• Downloads via• Web Service• HTTP / HTTPS• BitTorrent

12

S3 - Bucke ts

• Unique over all S3

• Contains many key-value-metadata tupel

• Cannot contain other buckets!

• Key can contain „/“

MyBucket_name

S3

foto_5.jpg

backups/januar.zip

fotos/2007/001.png

www.peritor.com

site/screen.css

site/script.js

FreeBSD-6.2.iso

13

S3 w ith AWS::S3

Upload

Download

14

EC2 - Elastic Co mpute Clo ud

• Based on XEN virtualization

• On demand virtual servers - controlled with Web Service API

• 1,7 GHz

• 1,75 GB RAM

• 160 GB local file system (not persistent!)

• Use your favorite Linux distro (Linux 2.6.16),Amazon Machine Images (AMI) are stored on S3

• ACLs for hosts/ports access control

15

EC2 To o ls

List availabe images

Start a new instance

Login with SSH

Shutdown instance

16

amazo n-e c2 ge mSetup

Usage

17

And no w ?

Ho w do e s this so lve my pro ble ms?

18

S3 - Backup

• s3sync.rb

• Brackup

• Jungle Disk

• S3 FUSE

• s3DAV

• Duplicity

• S3Browser

• Firefox S3 Organizer extension

• …

19

s3sync.rb

Backup

Restore

20

S3 Asse t Ho st

21

S3 Asse t Ho st

Setup DNS

Rails configuration

22

S3 Asse t Ho st

welcome.rhtml template

Output

23

S3 - Authe nticate d Use r Data

24

attachme nt_fu Rails plugin

Setup

25

attachme nt_fu Rails plugin

Upload

Download

26

On-De mand Co mputing w ith EC2

Time based, e.g. with cron

27

On-De mand Co mputing w ith EC2

Load based, e.g. with Monit

28

On-De mand Co mputing w ith EC2

29

EC2 fo r e xtra capacity

30

Lo ad Balance r / Pro xy

Example mod_proxy_blancer:

• Talks to multiple backend servers (Mongrel)

• Central Proxy/Load-Balancer configuration that has knowledge about nodes

• Typically proxy restart on config change

31

Sw iftiply

Swiftiply Proxy:

• Multiple backend clients have a persistent connection to the backend port

• Proxy listens on cluster port for requests and forwards them

No re-configuration

32

Sw iftiply Pro xy

Configuration

Start

Installation

33

Sw iftiplie d Mo ngre l

• Gem plugin that patches Mongrel

• Transforms Mongrel in Swiftiply client

• Experimental

Start

or

34

EC2 o n De mand be fo re Sw iftiply

35

EC2 o n De mand w ith Sw iftiply

36

EC2 o n De mand w ith Sw iftiply

37

Re sso urce s

• Amazon Web Serviceshttp://aws.amazon.com

• Swiftiplyhttp://swiftiply.swiftcore.org

• Attachment_fuhttp://svn.techno-weenie.net/projects/plugins/attachment_fu/

• Elastic Railshttp://www.elasticrails.com

• Capazonhttp://capazon.rubyforge.org

• RightScalehttp://www.rightscale.com

38

Peritor Wissensmanagement GmbH

Lenbachstraße 212157 Berlin

Internet: www.peritor.comEmail: info@peritor.com

Peritor Wissensmanagement GmbH - All Rights Reserved

Telefon: +49 (0)30 69 40 11 94Telefax: +49 (0)30 69 40 11 95