Riak at ideeli

Click here to load reader

  • date post

    18-May-2015
  • Category

    Technology

  • view

    190
  • download

    0

Embed Size (px)

Transcript of Riak at ideeli

  • 1. riak@ideeli Aaron Brown, Lead Systems Engineer abrown@ideeli.com @aaronbbrown777 Thursday, February 14, 13

2. What is an ideeli? Thursday, February 14, 13 3. Flash Sales http://www.ickr.com/photos/moonlightbulb/4064060773/ Thursday, February 14, 13 4. Flash Sales Thursday, February 14, 13 5. Flash Sales Thursday, February 14, 13 6. Flash Sales Thursday, February 14, 13 7. Flash Sales Thursday, February 14, 13 8. Obligatory Stats 2,000,000 visitors/week 80,000 front end requests/minute 120 production servers 300 deploys a year 40ms front end response time Thursday, February 14, 13 9. Obligatory Riak Stats 1.2MM keys 4 virtualized nodes 4ms response 40,000 GETs/min 2,000 PUTs/min Thursday, February 14, 13 10. The Spike Thursday, February 14, 13 11. The Stack Thursday, February 14, 13 12. New Toys Are Fun (9AM - 5PM) Thursday, February 14, 13 13. What Counts 1AM - 7AM Thursday, February 14, 13 14. K.I.S.S. Keep It Simple Stupid Thursday, February 14, 13 15. K.I.S.S. Keep It Simple Stupid Kan It Sell Shoes? Thursday, February 14, 13 16. 3 Uses for Riak Cache Warming Thursday, February 14, 13 17. 3 Uses for Riak Cache Warming Feature Testing (LABS) Thursday, February 14, 13 18. 3 Uses for Riak Cache Warming Feature Testing (LABS) User Favorites Thursday, February 14, 13 19. Cache Warming Thursday, February 14, 13 20. The Spike Thursday, February 14, 13 21. Cache Warming Pseudo-code: every.30.minutes do all_urls.each do |url| curl "#{url}?force_cache=true" end end Thursday, February 14, 13 22. Cache Warming Thursday, February 14, 13 23. Response times Thursday, February 14, 13 24. History Thursday, February 14, 13 25. memcached Thursday, February 14, 13 26. memcached ~40ms Thursday, February 14, 13 27. memcached 5/6 requests ~ 40ms 1/6 requests ~ 3000ms = ~500ms average Thursday, February 14, 13 28. Domino Effect http://www.ickr.com/photos/soham_pablo/309379628/ Thursday, February 14, 13 29. Manual Intervention Thursday, February 14, 13 30. membase (couchbase) Thursday, February 14, 13 31. Riak All requests ~40ms Thursday, February 14, 13 32. Riak All requests ~40ms Thursday, February 14, 13 33. LABS Feature Testing Framework Key based on user id ValueYAML string w/ list of experiments Thursday, February 14, 13 34. Favorites Thursday, February 14, 13 35. Favorites Thursday, February 14, 13 36. Favorites Partially normalized user_#{id} array of products product_#{id} metadata about product Thursday, February 14, 13 37. Rails Integration Modied riak-client gem for Rails 2.3 Load balancing built in Multiple primary and secondary servers Custom ActiveSupport::Cache store Thursday, February 14, 13 38. Rails Integration module ActiveSupport module Cache class RailsStore < Store def read end def write end def delete end end end config.after_initialize do ActionController::Base.cache_store = Rails.rails_store.fragments end Thursday, February 14, 13 39. Monitoring Thursday, February 14, 13 40. Graphite Thursday, February 14, 13 41. New Relic Thursday, February 14, 13 42. Alerting Compute random string PUT value into Riak GET value out of Riak alert if value_in != value_out alert if operation took too long Thursday, February 14, 13 43. Tuning Thursday, February 14, 13 44. Bitcask Merges Thursday, February 14, 13 45. Disk Usage Thursday, February 14, 13 46. Heavy I/O Thursday, February 14, 13 47. expiry_grace_time Thursday, February 14, 13 48. Offset Merge Windows Thursday, February 14, 13 49. Puppet class ideeli_riak ( $merge_window_start, $merge_window_end ) { ... file { '/etc/riak/app.config' : content => template("ideeli_riak/app.config.erb"), } ... } class riak-node { $merge_window_start = inline_template( ' ) class { 'ideeli_riak' : merge_window_start => $merge_window_start, merge_window_end => 6 } } Thursday, February 14, 13 50. Riak Loves Disk Thursday, February 14, 13 51. New Disk Array Thursday, February 14, 13 52. New Disk Array Thursday, February 14, 13 53. Improvement! Thursday, February 14, 13 54. Were Hiring http://ideelicareers.com/ Thursday, February 14, 13 55. Questions? Thursday, February 14, 13