Post on 05-Mar-2021
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 1
0-1 Million in 46 DaysScaling a Facebook Application in Rails
Ikai Lan
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 2
Hi! I’m Ikai Lan
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 3
What this talk is not
One-size-fits all solution to scalingproblems
Example source code
An in-depth discussion of Ruby onRails
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 4
What this talk is
A look into the challenges we facedwith scaling a rapid growth webapplication
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 5
This is a detective story.
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 6
Rather than telling you bestpractices
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 7
We’ll talk about how we came todefine them.
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 8
Bumper Sticker?
Investigation of the Facebookplatform
Investigation of Ruby on Rails as aserious development platform
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 9
Reid Hoffman’s wager
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 10
1 million users in 45 days!
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 11
Personal Badging
Create and display badges
Share badges
Find badges
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 12
= Bumper Sticker!
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 13
Success!
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 14
… 1 million in 46 days
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 15
Trouble in Paradise
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 16
Throw hardware at it!
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 17
More hardware!
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 18
This is not working.
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 19
Current state of affairs
7 million adds
20 million pageviews
3 million timeouts a day
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 20
Strategy
Analyze MySQL slow queries log
Application log
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 21
D’oh! No logging!
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 22
Logging -> Syslog
Rails does not buffer writes tofilesystem
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 23
Clue 1: Slaves ran several hoursbehind Master
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 24
Clue 2: Updates to add count
One by one updates
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 25
Counter Cache
SQL Updates not batched
Very inefficient!
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 26
Result: less slow queries
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 27
Lesson 1: Buffereverything!
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 28
Still had lots of timeouts!
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 29
Clue 3: 80% of traffic to staticcontent
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 30
Next attempt: edge caching!
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 31
Edge caching using fb:ref
Facebook mechanism for storingdata
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 32
Fb:ref on the load balancer!
F5 BigIP set to conditionally respondto static URIs with fb:ref
Background process to constantlyupdate fb:ref in Facebook cache
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 33
Result: Static pages =perfect
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 34
Lesson 2: Edge cachewhen possible
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 35
But we were still servingmillions of timeouts!
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 36
Clue 4: Most CPU time acceptingstickers
Request taking up to 10 secondsduring peak
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 37
Slow database writes
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 38
1st attempt: Background processwrites
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 39
Result: Timeouts droppedsignificantly!
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 40
Success … ?
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 41
No.
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 42
Application would notseem responsive to useractions and seem out ofsync.
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 43
2nd attempt: Two stepwrite
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 44
Step 1: Synchronouswrite to MemCache
Step 2: Asynchronous write to
database (queue)
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 45
Result: Success!
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 46
Lesson 3: Useasynchronous processingwhen possible
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 47
Lesson 3.1: Keeping datain sync is not trivial!
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 48
Lesson 3.2: Lack ofconcurrency in Railspresents problem earlier
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 49
Lesson 3.3: Very fewproblems can be reducedto a linear equation
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 50
QED: Rails can scale.
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 51
Latest stats
16 million installs
30 million pageviews a day
Average response time of <80ms
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 52
Scaling has a price!
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 53
Complexity
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 54
No new functionality!
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 55
Key takeaways
Make long running tasksasynchronous
Use edge caching
Buffer long operations when possible
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 56
The bigger picture
Language choice is less importantthan architecture
Be creative – brute force doesn’twork
Be ready to be wrong
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 57
Questions?
Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 58
Thanks for coming!
ilan@linkedin.com
http://www.linkedin.com/in/ikailan