Background Processing in Ruby on Rails

Background Processing

Rob MackAustin on Rails - April 2009

Thursday, April 30, 2009

Why Background Processing?


Simple Background Processing


script/runner

• script/runner lib/do_work.rb

• script/runner ‘Worker.do_work’

From RAILS_ROOT:


Rake

• Episode #127 Railscasts

def call_rake(task,options={}) options[:rails_env] = RAILS_ENV args = options.map{|n,v| "#{n.to_s.upcase}='#{v}'"} system "rake #{task} #{args.join(' ')} &" end


Daemons

http://daemons.rubyforge.org/

http://github.com/dougal/daemon_generator/


http://daemons.rubyforge.org

http://daemons.rubyforge.org



Generate A Daemon

• script/generate daemon my_worker

create lib/daemons create script/daemons create lib/daemons/my_worker.rb create lib/daemons/my_worker_ctl create config/daemons.yml


Do Some Work

$running = trueSignal.trap("TERM") do $running = falseend

while($running) do # Do some work sleep 10end

lib/daemons/my_worker.rb


Kick It Off

lib/daemons/my_worker_ctl startlib/daemons/my_worker_ctl stop


Spawn

http://github/tra/spawn




Spawn - Do Some Work

def my_action flash[:notice] = "Processing ... " spawn do # do some processing end end


Simple Queues


ar_mailer

http://seattlerb.rubyforge.org/ar_mailer/

setup:

gem install ar_mailerar_sendmail --create-migrationar_sendmail --create-model




ar_mailer Schema

create_table "emails" do |t| t.string "from" t.string "to" t.integer "last_send_attempt", :default => 0 t.text "mail", :limit => 16777215 t.datetime "created_on" end


ar_mailer Setup

class MyMailer < ActionMailer::ARMailer

Mail Model:

environment.rbconfig.gem 'ar_mailer', :version => '1.3.1',

:lib => 'action_mailer/ar_mailer'

config.action_mailer.delivery_method = :activerecord


Mail Something

MyMailer.deliver_newsletter @recipients


The Worker Process

ar_sendmail --daemonize --max-age 0


BackgrounDRb

http://backgroundrb.rubyforge.orghttp://github.com/gnufied/backgroundrb


http://backgroundrb.rubyforge.org

http://backgroundrb.rubyforge.org

http://github.com/gnufied/backgroundrb

http://github.com/gnufied/backgroundrb

BackgrounDRb Setup

• sudo gem install chronic packet

• script/plugin install git://github.com/gnufied/backgroundrb.git

• rake backgroundrb:setup

• rake db:migrate

• edit config/backgroundrb.yml

• script/generate worker my_worker


BackgrounDRb Schema

create_table "bdrb_job_queues" do |t| t.text "args" t.string "worker_name" t.string "worker_method" t.string "job_key" t.integer "taken" t.integer "finished" t.integer "timeout" t.integer "priority" t.datetime "submitted_at" t.datetime "started_at" t.datetime "finished_at" t.datetime "archived_at" t.string "tag" t.string "submitter_info" t.string "runner_info" t.string "worker_key" t.datetime "scheduled_at" end


Do Some Work

class MyWorker < BackgrounDRb::MetaWorker set_worker_name :my_worker def create(args = nil) # initialization goes here end

def do_work(user_id = nil) # do some work endend

def my_action flash[:notice] = "Processing ..." MiddleMan.worker(:my_worker).async_do_work(:arg => @user.id)end


Persistent Work

def my_action MiddleMan(:my_worker).enq_do_work(:job_key => "unique_key")end


Poll Status

class MyWorker < BackgrounDRb::MetaWorker set_worker_name :my_worker def create(args = nil) # initialization goes here end

def do_work(user_id = nil) # do some work register_status "almost done" endend


Poll Status

def my_action @job_key = MiddleMan.new_worker( :worker => :my_worker, :job_key => "unique_key" ) MiddleMan.worker(:my_worker, @job_key). do_work(@user.id)end

def check_status @status = MiddleMan.worker(:my_worker, @job_key). ask_statusend


The Worker Process

script/backgroundrb start


BJ - Background Job

http://codeforpeople.rubyforge.org/svn/bj




Bj - Setup

script/plugin install http://codeforpeople.rubyforge.org/svn/rails/plugins/bj

script/bj setup

gem install bjrequire ‘bj’

bj setup


http://codeforpeople.rubyforge.org/svn/bj/trunk






create_table "bj_job", :primary_key => "bj_job_id" do |t| t.text "command" t.text "state" t.integer "priority" t.text "tag" t.integer "is_restartable" t.text "submitter" t.text "runner" t.integer "pid" t.datetime "submitted_at" t.datetime "started_at" t.datetime "finished_at" t.text "env" t.text "stdin" t.text "stdout" t.text "stderr" t.integer "exit_status" end

Bj Schema


create_table "bj_job_archive", :primary_key => "bj_job_archive_id" do |t| t.text "command" t.text "state" t.integer "priority" t.text "tag" t.integer "is_restartable" t.text "submitter" t.text "runner" t.integer "pid" t.datetime "submitted_at" t.datetime "started_at" t.datetime "finished_at" t.datetime "archived_at" t.text "env" t.text "stdin" t.text "stdout" t.text "stderr" t.integer "exit_status" end

Bj Schema


Do Some Work

def my_action flash[:notice] = "Processing ..." Bj.submit './script/runner /lib/do_some_work.rb'end

def my_other_action flash[:notice] = "Processing ..." Bj.submit 'echo foobar', :tag => 'sample job'end


The Worker Process

bj run --forever --rails_env=development


Delayed Job

http://github.com/tobi/delayed_job/




Delayed Job Setup

• script/plugin install git://github.com/tobi/delayed_job.git

• create a delayed jobs table


create_table "delayed_jobs" do |t| t.integer "priority", :default => 0 t.integer "attempts", :default => 0 t.text "handler" t.string "last_error" t.datetime "run_at" t.datetime "locked_at" t.datetime "failed_at" t.string "locked_by" t.datetime "created_at" t.datetime "updated_at" end

Delayed Job Schema


Do Some Work

class MyWorker attr_accessor :user_id

def initialize(user) self.user_id = user_id end

def perform User.find user_id # do some work endend


Do Some Work

def my_action flash[:notice] = "Processing ..." Delayed::Job.enqueue MyWorker.new(@user.id)end

def my_action flash[:notice] = "Processing ..." MyMailer.new(@user.id).send_later(:deliver_newsletter)end


The Worker Process

rake jobs:work


Workling

http://github.com/purzelrakete/workling/




Workling Install

script/plugin install git://github.com/purzelrakete/workling.git


Do Some Work

class MyWorker < Workling::Base def do_something(options) # do some work endend

def my_action flash[:notice] = "Processing ..." MyWorker.async_do_something :user_id => @user.idend


The Worker Process

with spawn:script/workling_client start

with starling:sudo starling -dscript/workling_client start


Starling

http://github.com/starling/starling/




Even More Options

• Background Fu - http://github.com/ncr/background-fu

• Roofus Scheduler - http://github.com/jmettraux/rufus-scheduler

• AP4R - http://rubyforge.org/projects/ap4r/


http://github.com/ncr/background-fu




http://github.com/jmettraux/rufus-scheduler




http://rubyforge.org/projects/ap4r/




Advanced Queues


Kestrel

http://github.com/robey/kestrel/




Rabbit MQ

http://www.rabbitmq.com/


http://www.rabbitmq.com

http://www.rabbitmq.com

Active MQ

http://activemq.apache.org/


http://activemq.apache.org

http://activemq.apache.org

Amazon SQS

http://aws.amazon.com/sqs/




Monitoring

• god - http://github.com/mojombo/god/

• launchd - OS X

• monit - http://mmonit.com/monit/


http://mmonit.com/monit/

http://mmonit.com/monit/

Persistant?

Scaling concerns or fear of commitment?

Yes

Delayed JobBackground Job

No

Workling

Yes

Are you the next Twitter?

StarlingKestrel

Rabbit MQProbably Not Yes

Spawn

No

Daemons

I enjoy managing a

process?

NoYes


Thanks

[email protected]

twitter.com/robmack


mailto:[email protected]






Background Processing in Ruby on Rails

Technology

Transcript of Background Processing in Ruby on Rails