Holistic Systems Understanding

7/30/2019 Holistic Systems Understanding

1/57

Hi!

Tuesday, December 18, 2012


2/57



3/57

Simon Wistow



4/57


This isme


5/57


This is the company I helped found.

More on that later


6/57


These are some of the places Ive worked.

Some of them were even cool at the time.


7/57


Ive worked at a whole bunch of other startups.

Many of them are dead.


8/57


I also used to work doing VFX for films including this cheeky chappy.


9/57


This handsome devil (as me after for the story involving the chocolate river)


10/57


This won anOscar


11/57


12/57


Im sorry about that lastone.


13/57


I also used to work as aCowboy.


14/57


But in Australia.

And yes, I did have to wear chaps.

Although I am at pains to point out that ALL chaps are assless. Otherwise theyre just leathertrousers.

I also had to shave my legs. True Story!


15/57


How many of you read HackerNews?


16/57


How about /r/programming?


17/57


Out of curiosity -Slashdot?


18/57


Anyone even rememberKuro5hin?


19/57


The problem is that they all tend to slavishly follow the Gartner Hype Cycle


20/57

\



21/57


Node.js is magic - it means you scale without even thinking about it


22/57


Then comes the inevitable backlash.

A quick straw poll - I want to store my users passwords - what should I use?


23/57


Of course, bcrypt (or equivalent).

But lets go back to node.js ... what might be the problem?


24/57


10 users trying to log in simultaneously when you have a work factor of 12 will mean thatperson 10 will take about 3-5 seconds to log in. Thats only 10 people.


25/57


See, also, for example the wholeMongoDB is webscale

The problem is that people dont actually think about their problems.


26/57


Who remembers thistalk?


27/57


Or this one (or any of their similar ones).

The problem is that people didnt seem to understand them - they thought of them asrecipes not biographies and instead enshrined them as a set of rules Thou shalt shard,

Thou shalt use memcached


28/57


So, what has this got to do with Fastly?

A little about us - were a next generation content delivery network. We can cache dynamicas well as static content. Were real time and we give you insane amounts of control over a

request life cycle.


29/57

1. caching

2. naming3. Off BY ONE ERRORS


I am contractually obliged to include this joke. Its part of the Secret Guild of CDN providersrules.


30/57


We currently do around 5-6 Billion requests a day.


31/57

http://www.fastly.com/demo


This is a demo of our real time stats showing a subset of our customers.

We get around 90-95% hit rate depending on what sort of day some customers are having.

Out time to first byte is


32/57


Were pathologically lazy.

Out of interest - anybody seen the Halloween Garfield cartoon? Its insane.


33/57


This is Artur Bergman, our CEO.

[ INSERT origin story ]


34/57


As an aside, his name literally means Mountain Man


35/57


We famously use SSDs everywhere.

But SSDs are expensive? Not if you count IOps/$


36/57


We try and do less with more and we really try and understand the full stack.

Were willing to put the efort in where necessary.

We wrote what amounts to our own Filesystem.

And because we dont trust Linux we have a custom Kernel build which makes fsync a no-op.


37/57

gethostbyname(3)


Insert /etc/hostsanecdote.


38/57

LEFT OUTER JOIN


As an aside any thoughts on how this and SSDs might be related?


39/57


Another example of stufwere willing to do ...

How many of you have done any assembler?

[ INSERT assembler dispatch anecdote ]


40/57


However theres some stufwe just dont care about.

The speed of our API just isnt an issue.

We happen to use Unicorn. When people start doing microbenchmarks of Unicorn vs$Whatever it makes me sigh ...


41/57


Far better ofwith subjective wins, like making sure your site *feels* more responsive.


42/57


[ INSERT anecdote aboutYahoo! ]


43/57


We use MySQL cluster.

This is not a picture of MySQL cluster but its far more visually interesting.

[ INSERT Pros and Cons of MySQL Cluster ]

[ INSERT Why not Riak etc etc ]


44/57


Weve tried various forms of Messaging.

[ INSERT anecdote about Node.js ]


45/57


Which led to Artur tweeting this

[ INSERT anecdote about ZeroMQ ]

[ INSERT anecdote about switching to HTTP ]


46/57


Were not special and uniquesnowflakes.


47/57


The games industry generates stuf like this 60 times a second.

We concatenate a bunch of strings together twice a second and call it good.


48/57


Another example: lets go back to messaging again.

These fine fellows are one of the top 5 websites in the world. They are primarily a messagingwebsite.


49/57


By far their number 1 messaging moment came when Castle In The Sky, a 12 year old animeby Miyazaki was aired on Dec 9th, 2011.

They peaked 25,088 messages a second (by contrast the next biggest, the Euros final gotabout 15,358)

Lets assume that thats inbound tweets not total tweets and assume say a 10x fan out.


50/57


In contrast take this story from Forbes magazine in 2009 about NASDAQ.

http://www.forbes.com/forbes/2009/0112/056.html

Obviously thats not exactly an apples to apples comparison but ...


51/57


Lets talk about big data.

Every time people on Hacker News talk about Big Data an somewhat at CERN keels overwith sadness.

Its true. I read it on Digg.

Facebook has 40 billion photos.

Walmart does over 1 million transactions an hour. They have a database with over 2.5Petabytes.

The LHC collected 13 Pbs of data.


52/57


I dont mean to sound relentlessly negative though.

As developers were dealing with ever increasing amounts of data - when I was building thesearch engine for LiveJournal in 2006 it was designed to deal with at least 6 billion objects.Thats only half of what Yahoo! and Google had in 2001.


53/57


So playing around with new stuf is a good way to stop your brain atrophying and in thefuture these kind of techniques are going to be vital.


54/57


But lets get some perspective, dont get mired in the X SUX, Y IS TEH WIN! mentality andremember that over focussing on one particular detail is not nearly as good as having anoverall understanding of how everything fits in together.


55/57


Dont slavishly copy something just because someone else is doing it. The best technique isto analyze your problems and solve them instead.


56/57

Be language agnosticBe technology agnostic

Join the cult of DevOpsLook to the past

Be open minded about the future



57/57

QUESTIONS?REBUTTALS?

INSULTS?COMMENTARY?


Holistic Systems Understanding

Documents

Transcript of Holistic Systems Understanding