Three Years of Microservices at SoundCloud - Distributed Matters Berlin 2015
No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado
-
Upload
distributed-matters -
Category
Data & Analytics
-
view
569 -
download
2
Transcript of No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado
![Page 1: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/1.jpg)
No free lunch, indeed:Three years
Phil CalçadoSoundCloud
of microservicesat SoundCloud
![Page 2: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/2.jpg)
No free lunch, indeed:Three years
Phil CalçadoSoundCloud
of microservicesat SoundCloud
![Page 3: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/3.jpg)
![Page 4: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/4.jpg)
![Page 5: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/5.jpg)
> 11 hours of audio uploaded every minute
~ 300 million people every month
![Page 6: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/6.jpg)
heaps have been written about microservices in
the past year-ish
![Page 7: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/7.jpg)
![Page 8: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/8.jpg)
tl;dr
• Rapid provisioning• Basic Monitoring• Rapid application deployment
![Page 9: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/9.jpg)
These make sense
• Rapid provisioning• Basic Monitoring• Rapid application deployment
this makes sense
why does it make me so nervous?
![Page 10: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/10.jpg)
the SoundCloud story you might know
![Page 11: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/11.jpg)
we moved to microservices
because $reasons
http://bit.ly/how_we_ended_up_with_microservices
![Page 12: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/12.jpg)
![Page 13: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/13.jpg)
the pre-history
![Page 14: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/14.jpg)
SoundCloud, circa 2011
![Page 15: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/15.jpg)
let’s prepare for the “microservices
explosion"
![Page 16: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/16.jpg)
#1 provisioning
![Page 17: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/17.jpg)
what was cool in 2010-11
![Page 19: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/19.jpg)
much better than anything else at the
time
![Page 20: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/20.jpg)
a problem
no resource limits (i.e. cgroups)+
naïve scheduling=
loud neighbour in your own datacentre
![Page 21: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/21.jpg)
a problemmade for
most of our services migrated to
![Page 22: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/22.jpg)
the problem
time
start workon v1
v1 100% deployed
start work on v2
![Page 23: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/23.jpg)
attempt #1 - before we go sophisticated, let’s simplify what we have
![Page 24: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/24.jpg)
warmed up pool
machineintake
![Page 25: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/25.jpg)
just a bit of glue code
![Page 26: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/26.jpg)
just a bit of glue codea LOT
![Page 27: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/27.jpg)
machineintake
![Page 28: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/28.jpg)
#2 telemetry
![Page 29: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/29.jpg)
state of telemetry tools circa 2011-12 wasn’t
great
![Page 30: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/30.jpg)
StatsD
![Page 31: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/31.jpg)
let’s build our own!
![Page 32: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/32.jpg)
![Page 33: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/33.jpg)
but that’s not what broke…
![Page 34: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/34.jpg)
obvious with a monolith
monolith
![Page 35: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/35.jpg)
not so much now
?
?
?
![Page 36: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/36.jpg)
standardise dashboards
![Page 37: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/37.jpg)
standardise operations
https://twitter.github.io/twitter-server/Features.html#http-admin-interface
![Page 38: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/38.jpg)
visualise
![Page 39: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/39.jpg)
#3 deployment
![Page 40: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/40.jpg)
![Page 41: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/41.jpg)
> git
SquashFS
> make
unit tests integration tests
acceptance tests perf tests
> make
/dev/null
![Page 42: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/42.jpg)
we ended up with 7 different deployment
scripts
![Page 43: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/43.jpg)
> make > gitunit tests integration tests
unit tests integration tests
acceptance tests perf tests
![Page 44: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/44.jpg)
containers let you spawn your mini-
SoundCloud
![Page 45: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/45.jpg)
but why was I so nervous?
![Page 46: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/46.jpg)
because wemessed up
![Page 47: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/47.jpg)
there are simple and incremental ways to address these
• Rapid provisioning• Basic Monitoring• Rapid application deployment
![Page 48: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/48.jpg)
“wtf? do you think Netflix got it right the first time?”
— @adrianco
![Page 49: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/49.jpg)
some of the good things
![Page 50: No Free Lunch, Indeed: Three Years of Microservices at SoundCloud - Phil Calcado](https://reader031.fdocuments.net/reader031/viewer/2022022203/587882c01a28ab466c8b64f7/html5/thumbnails/50.jpg)
Q&A