Mongo à la Resque
-
Upload
nicolas-fouche -
Category
Software
-
view
318 -
download
0
Transcript of Mongo à la Resque
![Page 1: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/1.jpg)
Mongo à la ResqueNicolas Fouché
http:// .com http://about.me/nfo
![Page 2: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/2.jpg)
Say “rescue”
or “resquioux” in French
![Page 3: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/3.jpg)
Resque
One man:
![Page 4: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/4.jpg)
Resque
One language:
![Page 5: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/5.jpg)
Resque
One dependency:
![Page 6: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/6.jpg)
Redis
Remote Dictionary Server
“memcached on steroids”
Initial release was March 2009
![Page 7: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/7.jpg)
Redis
Redis is an open source, advanced key-value store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets.
http://redis.io
![Page 8: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/8.jpg)
Redis
In RAM
Latency: 10ns VS 1ms = 1,000,000ms
≃ 5000 Gets/s
≃ 5000 Sets/s
![Page 9: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/9.jpg)
Redis
Virtual memory
keys in memory
values as needed in memory
replication
![Page 10: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/10.jpg)
Redis
well written ANSI C
and clients in all languages
![Page 11: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/11.jpg)
Redis
keep in mind the complexity of each operation
![Page 12: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/12.jpg)
Resque
parent / child fork process
![Page 13: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/13.jpg)
Resque
handles Unix signals
=> god / monit / bluepill friendly
![Page 14: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/14.jpg)
Resque
Sinatra web UI, known as resque-web
![Page 15: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/15.jpg)
Queues
Priority:
$ QUEUES=high,low rake resque:work
![Page 16: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/16.jpg)
Queues
Consume all queues:
$ QUEUES=* rake resque:work
![Page 17: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/17.jpg)
Enqueue jobs
Resque.enqueue(Archive, @repo.id, branch)
![Page 18: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/18.jpg)
Workers
class Archive @queue = :file_serve def self.perform(repo_id, branch = nil) repo = Repository.find(repo_id) repo.create_archive(branch || "master") end end
![Page 19: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/19.jpg)
Persistence
Jobs are stored as JSON
{ "class": "Archive", "args": [ 44, "masterbrew" ] }
![Page 20: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/20.jpg)
Failure backends
redis (for resque-web)
hoptoad
and others (like exceptional)
![Page 21: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/21.jpg)
Don’t like Ruby ?
![Page 22: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/22.jpg)
Alternate implementations
https://github.com/defunkt/resque/wiki/alternate-implementations
![Page 23: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/23.jpg)
resque-mongo
![Page 24: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/24.jpg)
Translating queriesatomic operations
$set - set a particular value
$unset - set a particular value
$inc - increment a particular value by a certain amount
$push - append a value to an array
$pushAll - append several values to an array
$pull - remove a value(s) from an existing array
$pullAll - remove several value(s) from an existing array
$bit - bitwis
![Page 25: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/25.jpg)
Translating queriesPush a job
redis.sadd(:queues, queue)redis.rpush("queue:#{queue}", encode(item))
mongo << {:queue => queue, :item => item, :date => Time.now}mongo_queues.update( {:queue => queue }, {'$inc' => {:count => 1}})
![Page 26: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/26.jpg)
Translating queries
Pop a job
redis.lpop("queue:#{queue}")
doc = mongo.find_and_modify( :query => {:queue => queue}, :sort => [[:date, 1]], :remove => true) mongo_queues.update( {:queue => queue }, {'$inc' => {:count => -1}})
![Page 27: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/27.jpg)
Translating queries
queue size
redis.llen("queue:#{queue}").to_i mongo_queues.find_one(:queue => queue)['count']
![Page 28: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/28.jpg)
Translating queriesPeek a job
list_range("queue:#{queue}", start, count)
start, count = [start, count].map { |n| Integer(n) } res = mongo.find(:queue => queue).sort([:date, 1]).skip(start).limit(count).to_a res.collect! { |doc| doc['item'] } count == 1 ? res.first : res
![Page 29: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/29.jpg)
Translating queries
Remove queue
redis.srem(:queues, queue) redis.del("queue:#{queue}")
mongo.remove({:queue => queue}) mongo_queues.remove({:queue => queue})
![Page 30: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/30.jpg)
Mongo to the rescue ?
Sometimes you just know MongoDB better than Redis
Most of query features of Redis can be translated in Mongo queries
But for Redis addicts, MongoDB has some secrets
![Page 31: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/31.jpg)
Used to MongoDB ?
![Page 32: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/32.jpg)
Used to MongoDB ?[lol@cats ~]# bin/mongo MongoDB shell version: 1.6.4 connecting to: test > show dbs admin local monque > use monque switched to db monque > db.getCollectionNames(); [ "delayed_queue", "failures", "job_groups", "monque", "queues", "schedules", "schedules_changed", "stats", "system.indexes", "workers" ] > db.workers.count(); 40 > db.workers.findOne(); { "_id" : ObjectId("4c863c6c89cb535954000001"), "started" : "Tue Sep 07 2010 13:21:48 GMT+0000 (UTC)", "worker" : "myserver:22868:facebook*" }
![Page 33: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/33.jpg)
Used to MongoDB ?require 'rubygems' require 'mongo' include Mongo db = Connection.new.db('sample-db') coll = db.collection('test') coll.remove 3.times do |i| coll.insert({'a' => i+1}) end puts "There are #{coll.count()} records. Here they are:"coll.find().each { |doc| puts doc.inspect }
![Page 34: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/34.jpg)
Really used to MongoDB ?
![Page 35: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/35.jpg)
New features
Go take a look in this JSON !
![Page 36: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/36.jpg)
Search failed jobs
![Page 37: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/37.jpg)
What’s happening right now ?
Use the mongo console !
![Page 38: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/38.jpg)
Listen on queues by prefix
Queues prefix:
$ QUEUES=email*,analytics* rake resque:work
![Page 39: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/39.jpg)
Plugins
resque-mongo-groups
resque-mongo-scheduler
![Page 40: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/40.jpg)
resque-mongo-groups
by Catalin Bordianu, aka @omikronn
![Page 41: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/41.jpg)
resque-mongo-groups
Resque.enqueue( EatBreakfast, :with => ['eggs', 'bacon'], :group_id => 'meals:' + Time.now.utc.strftime('%Y%m%d') )
![Page 42: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/42.jpg)
resque-mongo-groups
class EatBreakfast extend Resque::Plugins::Groups::TrackedJob end
![Page 43: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/43.jpg)
resque-mongo-groups
def self.perform # ... self.class.atomic_op_on_complete = { '$inc' => { 'eggs' => 1, 'bacon' => 1 } } # ... end
![Page 44: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/44.jpg)
resque-mongo-groups
Resque.group_stats("meals:20110316") { '_id' => 'meals:20110316', 'total' => 4, 'delayed' => 2, 'completed' => 3, 'failed' => 1, 'exceptions' => ["#<TooMuchEggs: beware !>"] 'custom' => {:eggs => 12, :bacon => 1, :potatoes => 3} }
![Page 45: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/45.jpg)
resque-mongo-groups
gem "resque-mongo-groups"
![Page 46: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/46.jpg)
resque-mongo-scheduler
Resque.enqueue_at( 1.hour.from_now, Eat, :what => 'leftovers' )
![Page 47: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/47.jpg)
resque-mongo-scheduler
breakfast: cron: "0 8 * * *" class: Eat args: "bacon" description: Breakfast
![Page 48: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/48.jpg)
gem "resque-mongo-scheduler", :require => "resque-scheduler"
resque-mongo-scheduler
![Page 49: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/49.jpg)
resque-mongo ⚡ resque
You cannot use resque and resque-mongo in the same project
![Page 50: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/50.jpg)
resque-mongo ♥ resque
migration to resque-mongo => free
![Page 51: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/51.jpg)
resque-mongo ♥ resqueStill have to migrate from
to
![Page 52: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/52.jpg)
resque-mongo ♥ resque
easy to merge the last changes from resque to resque-mongo
easy to port existing plugins
![Page 53: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/53.jpg)
Get the gem !
gem install nfo-resque-mongo
The maintainer still did not answer to my ownership requests :(
![Page 54: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/54.jpg)
and use it with Bundler
gem "nfo-resque-mongo", :require => "resque"
![Page 56: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/56.jpg)
Help needed !Add exclusive features to resque-web
Migrate existing plugins
Create new plugins
Promote the gem
Or simply use the gem, you ‘ll surely send pull requests
![Page 57: Mongo à la Resque](https://reader034.fdocuments.net/reader034/viewer/2022051710/58eee2c71a28abe76f8b4653/html5/thumbnails/57.jpg)
That’s all folks
Credits for Flickr images: acezebragirl4j, kiwitime, fulbert05, josh_exell, benheine, tudacee