Download - Riak at ideeli

riak@ideeli

Aaron Brown, Lead Systems [email protected]@aaronbbrown777

Thursday, February 14, 13

mailto:[email protected]

mailto:[email protected]

What is an ideeli?


Flash Saleshttp://www.flickr.com/photos/moonlightbulb/4064060773/


http://www.flickr.com/photos/moonlightbulb/4064060773/

http://www.flickr.com/photos/moonlightbulb/4064060773/

Flash SalesThursday, February 14, 13

Obligatory Stats

• 2,000,000 visitors/week

• 80,000 front end requests/minute

• 120 production servers

• 300 deploys a year

• 40ms front end response time


Obligatory Riak Stats

• 1.2MM keys

• 4 virtualized nodes

• 4ms response

• 40,000 GETs/min

• 2,000 PUTs/min


The SpikeThursday, February 14, 13

The StackThursday, February 14, 13

New Toys Are Fun(9AM - 5PM)


What Counts1AM - 7AM


K.I.S.S.Keep It Simple Stupid


K.I.S.S.Keep It Simple Stupid

Kan It Sell Shoes?


3 Uses for Riak

• Cache Warming


3 Uses for Riak

• Cache Warming

• Feature Testing (LABS)


3 Uses for Riak

• Cache Warming

• Feature Testing (LABS)

• User Favorites


Cache Warming


The SpikeThursday, February 14, 13

Cache WarmingPseudo-code:

every.30.minutes do all_urls.each do |url| curl "#{url}?force_cache=true" endend


Cache WarmingThursday, February 14, 13

Response timesThursday, February 14, 13

History


memcached


memcached~40ms


memcached5/6 requests ~ 40ms

1/6 requests ~ 3000ms=

~500ms average


Domino Effecthttp://www.flickr.com/photos/soham_pablo/309379628/


http://www.flickr.com/photos/soham_pablo/309379628/

http://www.flickr.com/photos/soham_pablo/309379628/

Manual InterventionThursday, February 14, 13

membase(couchbase)


RiakAll requests ~40ms


LABS

• Feature Testing Framework

• Key based on user id

• Value YAML string w/ list of experiments


FavoritesThursday, February 14, 13

Favorites

• Partially normalized

• user_#{id} → array of productsproduct_#{id} → metadata about product


Rails Integration

• Modified riak-client gem for Rails 2.3

• Load balancing built in

• Multiple primary and secondary servers

• Custom ActiveSupport::Cache store


Rails Integrationmodule ActiveSupport module Cache class RailsStore < Store

def read end

def write end

def delete end endend

config.after_initialize do ActionController::Base.cache_store = Rails.rails_store.fragmentsend


MonitoringThursday, February 14, 13

GraphiteThursday, February 14, 13

New RelicThursday, February 14, 13

Alerting

• Compute random string

• PUT value into Riak

• GET value out of Riak

• alert if value_in != value_out

• alert if operation took too long


Tuning


Bitcask Merges


Disk UsageThursday, February 14, 13

Heavy I/OThursday, February 14, 13

expiry_grace_timeThursday, February 14, 13

Offset Merge WindowsThursday, February 14, 13

Puppetclass ideeli_riak ( $merge_window_start, $merge_window_end ) {... file { '/etc/riak/app.config' : content => template("ideeli_riak/app.config.erb"), }...}

class riak-node { $merge_window_start = inline_template( ‘<%= (scope.function_fqdn_rand([6]).to_i + 20) % 23 %>' )

class { 'ideeli_riak' : merge_window_start => $merge_window_start, merge_window_end => 6 }}


Riak Loves Disk


New Disk ArrayThursday, February 14, 13

Improvement!Thursday, February 14, 13

We’re Hiringhttp://ideelicareers.com/


http://ideelicareers.com/

http://ideelicareers.com/

Questions?
