Riak at Posterous

Click here to load reader

  • date post

  • Category


  • view

  • download


Embed Size (px)


Posterous recently deployed Riak to serve as their content cache. In this talk, Julio Capote will cover why the engineering team chose Riak for the use case. He'll also share some details on the old post cache and its problems, what solutions they evaluated, and how they settled on Riak.

Transcript of Riak at Posterous

  • 1. Riak at PosterousJulio CapoteSan Francisco Riak Meetup1/18/2012

2. A/S/L? Julio Capote Backend Developer at Posterous @capotej 3. Allows anyone to create multiple private or public spaces (blogs) Around since 2008 Millions of posts and users Tons of long tail trafcSome of the rst posts are still being accessed today due to search engines 4. How we store posts Original post body goes into MySQL Multiple variants are generated (nojs,mobile, etc) Expensive to generate (sanitizers,expanders) 5. Enter Variant Cache A generic read/write-through cache library Started with Memcache Moved to RedisAt the time disk store looked promising, so we moved from memcache to redis 6. Redis is awesome, but Requires both the key and value go into memory Terrible disk store performance Even with 3 machines with 64gb ram, couldnt t entire working set Forced to set a TTLredis wasnt really designed to ever hit the disk 7. The Dream 8. What we wanted Key/Value store Disk backed Built in distribution Use less boxes to serve more users Consistent performance over rawperformance 9. Percona MySQL / HandlerSocket 10. MySQL / HandlerSocketThe Good Great performance Can handle a huge number of rows Mature / Safe (at least the mysql part) 11. MySQL / HandlerSocket The Bad Sharding denitely not built in HandlerSocket is pretty much abandonedNo support going forward 12. MongoDB The Good Crazy fast Built in sharding support ...did I mention it was fast? 13. MongoDBThe Bad 30% standard deviation on fetch times (!) Would falsely acknowledge a writeThis is probably tunable, but still 14. Riak + Bitcask The Good Distributed by default Consistent and predictable performance Highly concurrent, no perf degradation Ops guy loves it! 15. Riak + BitcaskThe Bad Not crazy fast Stuck it behind memcache Still way faster than generating No multi get supportwrite and read through memcache 16. Riak in production Started using our 3 node cluster for theglobal production cache Accidentally turned off a node Keys rebalanced, site didnt skip a beat No one even noticed till hours later 17. Stats 3 nodes 2600+ requests/second 300+ GB ~200 million keys 10 GB memcache/host 18. #Protips All nodes can serve all requests, so... Use a vip, or... Pass all cluster nodes to client driver(thanks @aphyr!) Use curb instead of net/http Use Keep Alive 19. Any Questions? 20. Thanks for listening!Special thanks to@twoism@vincentchu@kangchen@argv0@pharkmillups@seancribbs@aphyr@jrecursive