Design for Scale / Surge 2010

‣ cb@opscode.com‣ @skeptomai‣ www.opscode.com

Christopher Brown VP, Engineering

Design for Scale

Who am I?

•Amazon EC2

Who am I?

•Amazon EC2

•Microsoft Edge Computing Network

Who am I?

•Amazon EC2

•Microsoft Edge Computing Network

•Opscode

Google, Amazon, Microsoftbuilt their own tools

almost everyone else is here...

... inexperienced or poorly equipped for the world in which we now operate.

The Method

http://www.flickr.com/photos/wonderlane/2090966628/sizes/l/

The Method

Bootstrapping

The Method

Bootstrapping

The Method

Bootstrapping

Configuration

The Method

Bootstrapping

Configuration

The Method

Bootstrapping

Configuration

Command & Control

The Method

Bootstrapping

Configuration

Command & ControlNanite!

Got it?

Got it?Defining the cloud is like this...

Origin Myth of EC2

Dynamism

Dynamism...not about excess capacity...

Dynamism

Dynamism• Disintermediation• Developers can freely experiment

• Isolation• Applications safely co-exist

• Utilization• Best use of expensive resources

This is what you are paying for

• Utilization• Best use of expensive resources

ScaleYou are not this BIG

You are not that BIG

• LAMP can scale on generic architecture

• 2008 - Facebook has over 800 memcached servers, with 28 terabytes of RAM

• 2010 - Github has 16 physical machines, 128 cores, 288 GB RAM

• Don’t design for A Million Users

• Ship early, Ship ugly, Ship often!

You are not that BIG

• LAMP can scale on generic architecture

• 2008 - Facebook has over 800 memcached servers, with 28 terabytes of RAM

• 2010 - Github has 16 physical machines, 128 cores, 288 GB RAM

• Don’t design for A Million Users

• Ship early, Ship ugly, Ship often!

EC2 Design Principles• Minimize management footprint

• Run in VMs just like customers.

• Forced to analyze what must run in privileged space

• “Harden everything” means separate network traffic inside the datacenter – customers and management run there

• True multi-tenancy - Customers run side-by-side

• Design by Fight Club

• "You are not a beautiful and unique snowflake“

• “On a large enough time line, the survival rate for everyone will drop to zero.”

http://www.flickr.com/photos/europedistrict/4058066840/

• Simple API, single unit of work

• think of early Unix tools (MH)

• Can compose with other APIs

• Does not define policy / coupling

• Customers will surprise youPrimitives

APIs, Mashups

http://www.flickr.com/photos/jfseesthings/4293062294/sizes/l/

Simplify

• Move complexity “up the stack”

• Easier to debug

• “Simple and Open” wins

• OAuth, OpenID

• ATOM, REST

• Example: EC2 Metadata - HTTP

Cost• CapEx versus OpEx

• The Cloud is not “Cheaper”

Cost• CapEx versus OpEx

• Do you have money, time, or experience?

What are you willing to pay for?

• CapEx versus OpEx

• Do you have money, time, or experience?

Nobody ever imagined a band of Orcs would steal a database table

Charles Stross - Halting State

MTTF & MTTRUnderstanding how, when and why things fail is great ... but

http://www.flickr.com/photos/dierken/948171048/sizes/z/

MTTF & MTTRUnderstanding how, when and why things fail is great ... but

If your Mean Time to Recover exceeds the time value of your data, your business is

http://www.flickr.com/photos/dierken/948171048/sizes/z/

Testing

• Test with production-like dataset and performance

• Don’t do “Design by Laptop”

• A/B Testing

• API versioning

Pull the Plug

•Create test environment

•Pull the plug

•Document

•Pull the plug again!

http://www.flickr.com/photos/rosipaw/5033284534/sizes/m/in/photostream/

Pull the Plug

•Create test environment

•Pull the plug

•Document

•Pull the plug again!

http://www.flickr.com/photos/rosipaw/5033284534/sizes/m/in/photostream/

Theo Morpheus

• Vertical vs Horizontal Scale

• Availability

• Reliability

• 99% vs 99.x% per unit?

Theo Morpheus

Free your mind...

• Availability

• Reliability

Theo Morpheus

Free your mind...

• Availability

• Reliability

Theo Morpheus

You are not Theo

Free your mind...

• Availability

• Reliability

Theo Morpheus

You are not Theo You’re probably not Morpheus either

Free your mind...

• Availability

• Reliability

Theo Morpheus

You are not Theo You’re probably not Morpheus either

Availability• For a distributed system to be continuously

available, every request received by a non-failing node in the system must result in a response.

• “Read globally, Write locally" with inconsistent cache

• Service Level Agreements, even (especially?) internally

Think Globally, Act Locally

• Global but inconsistent aggregate view

• Local action where data is authoritative

• Autonomy

• “Rightsizing” your failure domain

http://www.flickr.com/photos/28634332@N05/3872137437/sizes/m/in/photostream/

Distributed Systems Design• Avoid execution caching

• “Don’t lie, don’t retry”

• Embrace failure

• Don’t block the client

• Avoid internal policy

• Ensure the system makes forward progress

• It’s OK to apologize

• It’s better to completely fail for some users than penalize all of them

• The Web is all about “Hit Refresh”

Embrace Failure

Apologize...to Pat Helland

• Distributed Throttling

• Staged / Pipeline with back pressure

• Measure scalability at each stage

• Degraded performance

• Make progress for admitted requests

• At odds with “stateless” / session-less

Admission Control

http://www.flickr.com/photos/jayneandd/4450623309/sizes/m/in/photostream/

• Distributed Throttling

• Staged / Pipeline with back pressure

• Measure scalability at each stage

• Degraded performance

• Make progress for admitted requests

• At odds with “stateless” / session-less

Admission Control

http://www.flickr.com/photos/jayneandd/4450623309/sizes/m/in/photostream/

Make Forward Progress• MVCC, vector clocks, & reconciliation

• Don’t resurrect objects

• always go forward, never go back

• "name" is a property of an object, not its unique key

• Break the link, garbage collect later

• Model “degraded service” performance

Request Signing

• Stateless - no session tracking to lose or to purge later

• X509 - only public information on front-end boxes. More secure against exploit

• Shared secret - faster, smaller signature but requires secret info close to request front-end

Measure Monitor

Respond• Save *everything* *forever*

• Histograms / Pareto Chart

• tp99.9, tp99, and tp90

• ignore tp50, “average”

• http://en.wikipedia.org/wiki/Control_chart

• http://www.newrelic.com/

• http://www.splunk.com/

• skewness, kurtosis

Control Chart

• Day over Day

• Same Day, Year over Year

• Confidence Intervals

“Shewhart stressed that bringing a production process into a state of statistical control, where there is only common-cause variation, and keeping it in control, is necessary to predict future output and to manage a process economically.”

• http://en.wikipedia.org/wiki/Control_chart

Characteristic Curves

Periodicity

SLA, Variance, Troubleshooting

Data Taxonomy

• Precious

• Cachable

• Expensive

• Cheap

Consistency

• Authoritative vs. Consultative

• is_authorized? vs list group

Performance

• Call length

• Cyclomatic Complexity

• Request ID flow

• tension between unit performance and scalability

Failure Domains

• EC2 “droplets”

• EC2 DNS

• Coordinator zones

Still with me?

Successes

•Sharable “AMI”s•Metadata (Simple and open again)•Open API ( think Eucalyptus)•No API throttling•Primitives•Pay-as you go•Free traffic between S3 and EC2•Data and Compute together

Failures• SOAP makes little girls cry

• Amazon Web Services, circa 2006 was > 75% REST or Query

• SOAP well supported by commercial vendors, with their libraries

• Still *Way* too hard to use.

• Commodity business. Driving the bottom out of cost causes quality to suffer.

• API vs UI?, User Experience in general

• IaaS (Infrastructure as a Service) is insufficient by itself

• a hangman's noose. EC2, and the other offerings,

Where are we going?

Design for Scale / Surge 2010

Technology

Transcript of Design for Scale / Surge 2010

Large Scale Learning Design

FINAL REPORT DESIGN STORM SURGE HYDROGRAPHS FOR …€¦ · FINAL REPORT DESIGN STORM SURGE HYDROGRAPHS FOR THE ... and Florida Department of Environmental Protection, ... Surge Hydrographs

High-voltage surge arresters - Siemens Energy Sector · 3EQ4 Surge arrester with composite hollow core design 93 3EQ3 Surge arrester with ... outstanding seismic ... high-voltage

Lighting Surge Protection Modules Design and Installation ...

Is.7396.1.1985 Surge Shaft Design

Design of an Oscillating Wave Surge Converter on the ...

LED Lighting Surge Protection Modules Design and Installation Guide

Design for Pressure Surge in Flowmaster

Technical surge Session · 2020. 1. 28. · surge tank’s… Volume Design pressure ... Surge tank not properly welded. CHARLATTE - Not Charlatte. Tank Leg Design. COATING AND TESTING

Design note on surge shaft

Catalogo scale design

Large Scale Urban Design

IBM Design: Design at Scale

A Fiscally Based Scale for Tropical Cyclone Storm Surge...A Fiscally Based Scale for Tropical Cyclone Storm Surge AMANDA M. WALKER AND DAVID W. TITLEY Department of Meteorology and

ENGINEERING DESIGN GUIDELINE Fluid Flow Hydraulic Surge Rev01web

DESIGN AND ANALYSIS FOR A FLOATING OSCILLATING SURGE …

Queensland Storm Surge Forecasting Model Design Using … · 2015. 12. 14. · Faivre et. Al, Queensland Storm Surge Forecasting Model Design Using Sensitivity Analysis sensitivity

Surge Tank Design Considerations for Controlling Water ... · ــــــــــــــــــــــــــــــــــــSurge Tank Design Considerations for Controlling

Weighing and receiving hopper - Buhler Group · lower surge hopper. 3 Lower surge hopper scale The lower surge hopper scale is applied underneath storage bins and fed by screw conveyors.

Design At Scale Workshop