Managing Virtual Sprawl

42
Managing Virtual Sprawl (How to not let this happen to you) Jeremy Hitchcock, [email protected]

description

http://twitter.com/jhitchco

Transcript of Managing Virtual Sprawl

Page 1: Managing Virtual Sprawl

Managing Virtual Sprawl(How to not let this happen to you)

Jeremy Hitchcock, [email protected]

Page 2: Managing Virtual Sprawl

Why care?

What you have

What you want What you got

Page 3: Managing Virtual Sprawl

Managing clouds like managing single systems increases "system" management by 10x-20x

Page 4: Managing Virtual Sprawl

Clouds Promise

• Greater efficiency

• Faster deploys/less management

• Little/no capital costs and no step functions

Page 5: Managing Virtual Sprawl

Sprawl Eats Potential

• Greater efficiency

• Faster deploys/less management

• Little/no capital costs and no step functions

Page 6: Managing Virtual Sprawl

15 years 3 years

Just not good, yet

Page 7: Managing Virtual Sprawl

Don’t just change broken light bulbs

Page 8: Managing Virtual Sprawl

Wait until it gets dark, then change them all

Page 9: Managing Virtual Sprawl

Let’s get started

1. Architectures

2. Pain points

3. Best practices

4. What do we get?

Page 10: Managing Virtual Sprawl

1: Architectures

• Architecture changes

• Decoupling

• Geography/load balancing

• Disaster recovery

Page 11: Managing Virtual Sprawl

2004

Page 12: Managing Virtual Sprawl

2007-2008

Page 13: Managing Virtual Sprawl

Opera dynamic resource pricing model

Page 14: Managing Virtual Sprawl

Decoupling

• Apps and infrastructure mirror each other

• Years of coupled development

• Hard to retrofit, easier to do from start

Page 15: Managing Virtual Sprawl

Decoupling

New:

Old:Web App DB

Dispatcher

Processing

Storage

Page 16: Managing Virtual Sprawl

Decoupling is Hard

• Logging/debugging

• Common scratch

• Images and provisioning

• Configuration data (run/boot)

• Job dispatch (async/sync)

Page 17: Managing Virtual Sprawl

Images and provisioning

Publish New Code

Add __ new front ends

Even better is that is automatic

Page 18: Managing Virtual Sprawl

Configuration Data

• Most config data is on each image

• Instead, auto populate into source control

• Config, image, controller re-architected

Page 19: Managing Virtual Sprawl

Job Dispatch (sync)

Read photo off disc

Request for photo

Log

Resize/reformat

Return photo to user

Page 20: Managing Virtual Sprawl

Job Dispatch (async)

Read photo off disc

Request for photo Log

Resize/reformat

Return photo to user

1

2

3

4

5

Page 21: Managing Virtual Sprawl

Geography/load balancing

• Data centers do not house eyeballs

• Intra/inter-site load balancing

• Names to numbers (users think names)

• Between clouds/interoperability?

Page 22: Managing Virtual Sprawl

Disaster Recovery

• Practice them

• Failovers should be automatic

• DNS (Quick DNS nit: use short TTLs)

• Contingency plans

Page 23: Managing Virtual Sprawl

Case Study: Authorize.net

Page 24: Managing Virtual Sprawl

Case Study: Authorize.net

Page 25: Managing Virtual Sprawl

Case Study: Authorize.net

; QUESTION SECTION:;secure.authorize.net. IN A

;; ANSWER SECTION:secure.authorize.net. 86400 IN A 64.94.118.32secure.authorize.net. 86400 IN A 64.94.118.33

Page 26: Managing Virtual Sprawl

Case Study: Authorize.net

Page 27: Managing Virtual Sprawl

Case Study: Authorize.net

; QUESTION SECTION:;secure.authorize.net. IN A

;; ANSWER SECTION:secure.authorize.net. 86400 IN A 64.94.118.32secure.authorize.net. 86400 IN A 64.94.118.33

GAH!

Page 28: Managing Virtual Sprawl

2: Pain Points

• Inventory

• Delivery speed

• Supply/demand

• Configuration

• Points of failure

Page 29: Managing Virtual Sprawl

“I can ping it but I don’t know where it is!”

Page 30: Managing Virtual Sprawl

Inventory

• Does it matter?

• Not an asset tag but provisioning scripts

• Audit bills (operational costs)

Page 31: Managing Virtual Sprawl

Delivery Speed

• May actually suffer (more pieces, not iron)

• Be analytical about what can be slow

• Limiting factor of what’s virtualized

• Were you looking before?

Page 32: Managing Virtual Sprawl

•Where is the testing from?

•Is this load dependent?

•Do users notice/care?

•Does it matter?

•Cost to make it faster?

•Savings to make it slower?

Graph from Gomez

Delivery Speed

Page 33: Managing Virtual Sprawl

Supply/demand

• Capital investments versus operating costs

• Big architecture changes to constant tuning

• Sampling time

Page 34: Managing Virtual Sprawl

Configuration

• Configuration in source control

• Has to move to a centralized location

• Patches, updates, revision images

• Lot of hard work here (no return)

Page 35: Managing Virtual Sprawl

Points of Failure

• It’s about risk

• All in the name, DNS

• 99.9% is different from 99.99%

• Any page is better than nothing

Page 36: Managing Virtual Sprawl

3: Best Practices

• App rewrite

• Controller (code, monitoring)

• Configuration (chef, puppet, etc)

• Dev/staging/production (Django/Rails)

• Security

• Monitoring and verification

Page 37: Managing Virtual Sprawl

Dev/Staging/Production

• This stuff works, use it

• Clouds make this possible

• ONLY exception is load testing (big exception)

• Nothing going to work out of the box

Page 38: Managing Virtual Sprawl

Security

• No “behind the firewall”

• Not an after thought, core feature

• Something to test

• Two hash encryption (private data)

• Centralized management makes security easier (At least double or nothing)

Page 39: Managing Virtual Sprawl

Monitoring and Verification

What you monitorWhat your user sees

Are they the same? Test transactions

Page 40: Managing Virtual Sprawl

4: What do we get?

• More choice on availability

• Less step functions (capacity, cost)

• Reduce computational marginal cost

Page 41: Managing Virtual Sprawl

Final Remarks

• Sprawl eats away from the promised good

• Never truly decoupled, apps dictate arch

• Management tools still lacking, more homegrown

• Make it all automatic, not easy

Page 42: Managing Virtual Sprawl

Questions?Jeremy Hitchcock, [email protected]

DynDNS.com offers a suite of DNS, email, domain registration and virtual servers for the home and small business user.

The Dynect Platform provides the enterprise with external managed DNS and traffic management services.