AWS Summit Berlin 2013 - "Angrybirds fly in the cloud" - Scaling and Market needs for the Angrybirds...

Freitag, 3. Mai 13

Angrybirds fly in the cloud

Freitag, 3. Mai 13

Angrybirds fly in the cloud

Presented byDaniel Pötzinger

AWS Summit 2013

Freitag, 3. Mai 13

Freitag, 3. Mai 13

danpoetz

Freitag, 3. Mai 13

Freitag, 3. Mai 13

152,2 Million €

68,5 Million € Consumer Products

Freitag, 3. Mai 13

Freitag, 3. Mai 13

PC games

Plush toys, shirts, ...

Freitag, 3. Mai 13

> 24.000.000 likes > 500.000 followers

Social MediaImpact

Freitag, 3. Mai 13

Irregular Traffic Behaviour

Freitag, 3. Mai 13

Challenges

Freitag, 3. Mai 13

Challenges

Frequent development and releasing of new

features

Freitag, 3. Mai 13

Challenges

Frequent development and releasing of new

features

Scalable and stable Operation

Freitag, 3. Mai 13

Freitag, 3. Mai 13

The solution?

Freitag, 3. Mai 13

Freitag, 3. Mai 13

Continuous Deployment

CloudDevOps

Freitag, 3. Mai 13

Why Amazon Web Services ?

Freitag, 3. Mai 13

✓Scales quickly on demand✓Pay only what for what

you use

Freitag, 3. Mai 13

8 regions with 2-4 availability zonesA lot of „edge servers“ for content delivery

Freitag, 3. Mai 13

Think cleanly about your operation infrastructureFreitag, 3. Mai 13

Setting up a new environment should always be faster than fixing an existing one

Freitag, 3. Mai 13

What you should do

Freitag, 3. Mai 13

What you should do

Automate the setup of new servers

Freitag, 3. Mai 13

What you should do


Automate failover

Freitag, 3. Mai 13

What you should do


Automate failover

Automate scaling

Freitag, 3. Mai 13

What you should do


Automate failover

Automate scaling

Don‘t do work that others did better

Freitag, 3. Mai 13

Powerfull AWS Services for Angrybirds

Freitag, 3. Mai 13


EC2

Freitag, 3. Mai 13


S3EC2

Freitag, 3. Mai 13


S3EC2 RDS

Freitag, 3. Mai 13


CloudFrontS3EC2 RDS

Freitag, 3. Mai 13


CloudFront ElastiCacheS3EC2 RDS

Freitag, 3. Mai 13


CloudFront ElastiCacheS3EC2 RDS ELB

Freitag, 3. Mai 13


CloudFront ElastiCacheS3EC2 RDS ELB R53

Freitag, 3. Mai 13


Amazon Web Services & Amazon Console


Freitag, 3. Mai 13


Rightscale Services

Amazon Web Services & Amazon Console


Freitag, 3. Mai 13

Freitag, 3. Mai 13

Cloud Architecture

Freitag, 3. Mai 13

Freitag, 3. Mai 13

Varnish Array

Freitag, 3. Mai 13

Varnish ArrayEC2 EC2

Freitag, 3. Mai 13

Varnish Array

Frontend ArrayEC2 EC2 EC2

EC2 EC2

Freitag, 3. Mai 13

Varnish Array

Frontend Array Backend ArrayEC2 EC2EC2 EC2 EC2

EC2 EC2

Freitag, 3. Mai 13

Varnish Array

Frontend Array Backend Array

RDS

EC2 EC2EC2 EC2 EC2

EC2 EC2

Freitag, 3. Mai 13

RDS

Varnish Array


RDS

EC2 EC2EC2 EC2 EC2

EC2 EC2

Freitag, 3. Mai 13

RDS

Varnish Array


RDS

S3Assets Storage

EC2 EC2EC2 EC2 EC2

EC2 EC2

Freitag, 3. Mai 13

RDS

Varnish Array


RDS

CloudFrontCDN for Assets

S3Assets Storage

EC2 EC2EC2 EC2 EC2

EC2 EC2

Freitag, 3. Mai 13

RDS

Varnish Array

Frontend Array Backend Array Worker Array

RDS


S3Assets Storage

EC2 EC2 EC2EC2 EC2 EC2

EC2 EC2

Freitag, 3. Mai 13

RDS

Varnish Array


ELB

RDS


S3Assets Storage


EC2 EC2

Freitag, 3. Mai 13

RDS

„Deployment Definition“ + Autoscaling

Varnish Array


ELB

RDS


S3Assets Storage


EC2 EC2

Freitag, 3. Mai 13

RDS


Varnish Array


ELB

RDS


S3Assets Storage


EC2 EC2Monitoring

Freitag, 3. Mai 13

RDS


Varnish Array


ELB

RDS


S3Assets Storage


EC2 EC2MonitoringAPI

Freitag, 3. Mai 13

RDS


Varnish Array


CloudFrontCDN for Skin

ELB

RDS


S3Assets Storage



Freitag, 3. Mai 13

RDS


Varnish Array



ELBR53DNS

RDS


S3Assets Storage



Freitag, 3. Mai 13

RDS


Varnish Array



ELBR53DNS

RDSElastiCache

Cache Backend


S3Assets Storage



Freitag, 3. Mai 13

RDS


Varnish Array



ELBR53DNS

RDSS3

DeploymentElastiCache

Cache Backend


S3Assets Storage



Freitag, 3. Mai 13

Scaling is about finding the bottleneck

Freitag, 3. Mai 13

CPU

Network

Storage (DB)Traffic

Possible BottlenecksOffloading, Optimisation, Caching, add more CPU (scale)...

Offloading, Optimisation, Scale Storage UP (H/V), Replace Storage...

Offloading (CDN, Proxy)Optimize Size of the SiteScale Network IS...

Freitag, 3. Mai 13

HTTP Caching

Traffic

Freitag, 3. Mai 13

HTTP Caching

Appl

icat

ion

Traffic

Freitag, 3. Mai 13

HTTP Caching

Appl

icat

ion

Offload expensive requests

Traffic

Freitag, 3. Mai 13

HTTP Caching

Appl

icat

ion


Traffic

Brow

ser

Traffic

Freitag, 3. Mai 13

HTTP Caching

Appl

icat

ion


Traffic

CDN

Brow

ser

Traffic

Freitag, 3. Mai 13

HTTP Caching

Appl

icat

ion

Varnish (Reverse Proxy) Traffic


Traffic

CDN

Brow

ser

Traffic

Freitag, 3. Mai 13

HTTP Caching

Appl

icat

ion

Varnish (Reverse Proxy) Traffic


Traffic

CDN

Brow

ser

TrafficPurging

Freitag, 3. Mai 13

10.000+Possible Pages / Seconde

from ELB & Varnish

Freitag, 3. Mai 13

Autoscaling

Traffic

Freitag, 3. Mai 13

Autoscaling

Traffic

Fron

tend

Arr

ay

EC2

EC2

EC2

Freitag, 3. Mai 13

Autoscaling

Traffic

Fron

tend

Arr

ay

EC2

EC2

EC2

EC2

EC2

EC2

Freitag, 3. Mai 13

Autoscaling

•Groups with policys•Min, Max•Votingrules for scaling up & down•Replace broken instances

Traffic

Fron

tend

Arr

ay

EC2

EC2

EC2

EC2

EC2

EC2

Freitag, 3. Mai 13

10.000possible Orders / Hour

Freitag, 3. Mai 13

Be prepared for changes

Freitag, 3. Mai 13

Freitag, 3. Mai 13

Don‘t be scared of releasing to production

Freitag, 3. Mai 13

Late night bugfixing?Freitag, 3. Mai 13

That’s how a deployment should look like

Freitag, 3. Mai 13

And you?

Freitag, 3. Mai 13

And you?

• Do you use continuous integration?

Freitag, 3. Mai 13

And you?

• Do you use continuous integration?• Do you use automated acceptance tests?

Freitag, 3. Mai 13

And you?

• Do you use continuous integration?• Do you use automated acceptance tests?• Do you keep them green?

Freitag, 3. Mai 13

And you?

• Do you use continuous integration?• Do you use automated acceptance tests?• Do you keep them green?• Do you use continuous deployment?

Freitag, 3. Mai 13


Freitag, 3. Mai 13


We want on-demand deployment

Freitag, 3. Mai 13


We want on-demand deploymentof tested features and fixes

Freitag, 3. Mai 13


We want on-demand deploymentof tested features and fixesin a quick and reliable way

Freitag, 3. Mai 13

We reach this by

Freitag, 3. Mai 13

Automating everything

We reach this by

Freitag, 3. Mai 13


We reach this by

Build

Freitag, 3. Mai 13


We reach this by

BuildTesting

Freitag, 3. Mai 13


We reach this by

BuildTesting

Deployment

Freitag, 3. Mai 13

..and

Freitag, 3. Mai 13

doing it frequently

..and

Freitag, 3. Mai 13

doing it frequently

..and

flickr deploys 3 times a day...

Freitag, 3. Mai 13

Freitag, 3. Mai 13

The Deployment Pipeline

Freitag, 3. Mai 13

Freitag, 3. Mai 13

Commitand Build

Build Downstream and Test Strategy

SVN Backup Storage

Angry Birds Deployment Pipeline

Feedback to Developers

Freitag, 3. Mai 13

Commitand Build


SVN Backup Storage

Unit Tests



Freitag, 3. Mai 13

Commitand Build


SVN Backup Storage

Unit Tests Install on “latest”



Freitag, 3. Mai 13

Selenium Acceptance Tests

Commitand Build


SVN Backup Storage




Freitag, 3. Mai 13

Selenium Acceptance Tests

Commitand Build


SVN Backup Storage


Performance Tests



Freitag, 3. Mai 13


Install on Deployment

SystemCopy to S3

Cloud Deployment

Staging

Integration Tests

AOE + Rovio

Cloud deployment production

Freitag, 3. Mai 13

Varnish Array

AutoscalingFrontend Array

Backend Array

Worker Array

Cloud Deployment

ELB

Deployment „Version 88“

Start new release by✓ cloning old deployment✓ adjusting release number

RDSmySQL S3

V V V

WFE FE FE BE BE BE

R53DNS

Freitag, 3. Mai 13

Varnish Array


Backend Array

Worker Array

Cloud Deployment

ELB


RDSmySQL S3

V V V

WFE FE FE BE BE BE

R53DNS

Freitag, 3. Mai 13


Varnish Array


Backend Array

Worker Array

Cloud Deployment

ELB


RDSmySQL S3

Varnish Array


Backend Array

Worker Array

V V V

WFE FE FE BE BE BE

R53DNS

Freitag, 3. Mai 13


Varnish Array


Backend Array

Worker Array

Cloud Deployment

ELB


RDSmySQL S3

Varnish Array


Backend Array

Worker Array

ELB

W

V V V

WFE FE FE BE BE BE

R53DNS

Freitag, 3. Mai 13


Varnish Array


Backend Array

Worker Array

Cloud Deployment

ELB


RDSmySQL S3

Varnish Array


Backend Array

Worker Array

ELB

WFE FE FE BE BE BE

V V VV V V

WFE FE FE BE BE BE

R53DNS

Freitag, 3. Mai 13

0Seconds Downtime

Freitag, 3. Mai 13

Freitag, 3. Mai 13

The real life

Freitag, 3. Mai 13

Freitag, 3. Mai 13

Day 1 - Release 1

Freitag, 3. Mai 13

Day 2 - Release 2 - “Get ready for Space”

Freitag, 3. Mai 13

Day 3 - „People want it so badly“

Freitag, 3. Mai 13

New release deployed


Freitag, 3. Mai 13


Space game launchedArray scales up


Freitag, 3. Mai 13


Space game launchedArray scales up

USA wakes up


Freitag, 3. Mai 13

Day 4 - Release 3 „Performance Improvement“

Freitag, 3. Mai 13

40% average CPU usage.

1-2 FE Servers are enough for normal operations

Day 4 - Release 3 „Performance Improvement“

Freitag, 3. Mai 13

Freitag, 3. Mai 13

Thanks!Follow us on Twitter:

AOE: @aoemediaDaniel P: @danpoetz

Flickr photo contributions in the slide:http://www.flickr.com/photos/the-wanderers-eye/4494147652/sizes/l/http://www.flickr.com/photos/51293088@N05/8195327886/sizes/l/http://www.flickr.com/photos/k1rsch/3818553037/sizes/l/

Freitag, 3. Mai 13

http://www.flickr.com/photos/the-wanderers-eye/4494147652/sizes/l/

http://www.flickr.com/photos/the-wanderers-eye/4494147652/sizes/l/

http://www.flickr.com/photos/51293088@N05/8195327886/sizes/l/

http://www.flickr.com/photos/51293088@N05/8195327886/sizes/l/

http://www.flickr.com/photos/k1rsch/3818553037/sizes/l/

http://www.flickr.com/photos/k1rsch/3818553037/sizes/l/

AWS Summit Berlin 2013 - "Angrybirds fly in the cloud" - Scaling and Market needs for the Angrybirds...

Technology

Transcript of AWS Summit Berlin 2013 - "Angrybirds fly in the cloud" - Scaling and Market needs for the Angrybirds...