LEARNING TO BUILDDISTRIBUTED SYSTEMS

THE HARD WAY

@iconara

speakerdeck.com/u/iconara(real time!)

Theo / @iconara

Chief Architect at

let’s make online advertising a great experience

MAKING THIS

INTO THIS

HOW HARD CAN IT BE?

TRACKING AD IMPRESSIONS

track page views and all their adstrack visibility and send updates on changes

track events, track activity, sync cookies,and track visits

track page views and all their adstrack visibility and send updates on changes

track events, track activity, sync cookies,and track visits

LOADED

VISIBLE

HIDDEN

VISIBLE

LOADED

ASSEMBLING SESSIONS

assemble ad impressions, page views and visits,to be able to calculate things like total visible durationmix in demographics, revenue, and third-party data

assemble ad impressions, page views and visits,to be able to calculate things like total visible durationmix in demographics, revenue, and third-party data

WASLOADED

BECAMEACTIVE

BECAMEVISIBLE

WAS HIDDEN

BECAMEVISIBLEAGAIN

A CLICK! { "user_id": "M9L6R5TD0YXK", "session_id": "MAI3QAGNAIYT", "timestamp": 1347896675038, "placement_name": "example", "category": "frontpage", "embed_url": "http://example.com/", "visible_duration": 1340 "browser": "Chrome", "device_type": "computer", "click": true, "ad_dimensions":"980x300"}

3rd PARTY DATA & OTHER GOODIES

ANALYTICSprecompute metrics, count uniques, build visitor histories for attribution

precompute metrics, count uniques, build visitor histories for attribution

HOW HARD CAN IT BE?

25K REQUESTSPER SECOND

~1 billion requests per day, 1 TB raw data

ONE VISIT CAN CHANGE UP TO

100K COUNTERShundreds of millions of individual counters per day,

plus counting uniques and visitor histories

IN REAL TIMEor near real time, if you want to be pedantic

START WITH TWOOF EVERYTHINGgoing from one to two is the hardest

GIVE A LOT OF THOUGHT TO YOUR

KEYS AND IDSit will save you lots of pain

MANLO0 JME57Zmonotonically increasing,

sorts nicely

a timestampsomething random

JME57Z MANLO0uniformly distributed,

works nicely with sharding

something randoma timestamp

PUT BUFFERSBETWEEN LAYERS

queues can even out peaks, let you scale layers independently,

and let you restart services without loosing data

SEPARATE PROCESSING

FROM STORAGEthat way you can scale each independently

PLAN HOW TO GET RID OF YOUR DATAdeleting stuff is harder than you might think

××××

××

×

NoDBkeep things streaming

×

STREAM PARTITIONING

RANDOMLYwhen you have no interdependenciesbetween things it’s easy to scale out

(or round robin, it’s basically the same)

CONSISTENTLYwhen there are interdependencies you needto route using some property of the objects,but make sure you get a uniform distribution

NUMEROLOGY

12

2 | 123 | 124 | 126 | 12

8 | 245 | 60

12, 60, 120, 360superior highly composite numbers

12, 60, 120, 360superior highly composite numbers

12, 60, 120, 360superior highly composite numbers

12, 60, 120, 360superior highly composite numbers

12, 60, 120, 360superior highly composite numbers

12, 60, 120, 360superior highly composite numbers

12, 60, 120, 360superior highly composite numbers

12, 60, 120, 360superior highly composite numbers

for maximal flexibility partition with multiples of 12

for maximal flexibility partition with multiples of 12

A SHORT DIVERSION ABOUT COUNTING TO 60

the reason why there’s 60 seconds to a minute,and 360 degrees to a circle

3 SEGMENTSON EACH FINGER

= 12

3 SEGMENTSON EACH FINGER

= 12

FIVE FINGERSON OTHER HAND

= 60

log2(366) ≈ 31

$-$(ASCII code 36)-----

log2(366) ≈ 31

log2(366) ≈ 31six characters 0-9, A-Z can represent 31 bits,which is kind of almost very close to four bytes

MANLO0

MANLO0a timestamp

Time.now.to_i.to_s(36).upcase

DO YOU REALLYNEED A BACKUP?

if you got 3x replication over multipleavailability zones, is that backup really worth it?

PRODUCTION IS THE ONLY REAL TEST ENVIRONMENT

when thousands of things happen every second, new, weird and unforeseen things happen all the time,

no test can anticipate everything(but testing is good anyway, just don’t think you got everything covered)

KTHXBAI@iconara

github.com/iconaraarchitecturalatrocities.com

burtcorp.com

COME TO SWEDEN IN MARCH AND

TALK ABOUT BIG DATAscandevconf.se/2013/call-for-proposals

IDEMPOTENCE

f(f(x)) = f(x)doing something again doesn’t change the outcome

IDEMPOTENCEif you don’t have to worry about things accidentally happening twice, everything becomes much simpler

COUNTING UNIQUESwhen adding to a set it doesn’t matter how many

times you do it, the end result is the same

INC X VS SET Xincrements are not idempotent, and very scary,if you can avoid non-idempotent operations, try

KTHXBAI@iconara

github.com/iconaraarchitecturalatrocities.com

burtcorp.com