Nasos Psarrakospsarrakos@gmail.com github.com/nasospsalinkedin.com/in/nasos-psarrakos

Debugging Memory Problems in Rails

The Problem > A migration needs more than 1GB memory > Memory @ Heroku: 512MB

def start_script puts "starting script..." # Remove the X to enable the parameters for tuning. # These are the default values as of Ruby 2.2.0. @child = spawn(<<-EOC.split.join(" ")) XRUBY_GC_HEAP_FREE_SLOTS=4096 XRUBY_GC_HEAP_INIT_SLOTS=10000 XRUBY_GC_HEAP_GROWTH_FACTOR=1.8 XRUBY_GC_HEAP_GROWTH_MAX_SLOTS=0 XRUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR=2.0 XRUBY_GC_MALLOC_LIMIT=16777216 XRUBY_GC_MALLOC_LIMIT_MAX=33554432 XRUBY_GC_MALLOC_LIMIT_GROWTH_FACTOR=1.4 XRUBY_GC_OLDMALLOC_LIMIT=16777216 XRUBY_GC_OLDMALLOC_LIMIT_MAX=134217728 XRUBY_GC_OLDMALLOC_LIMIT_GROWTH_FACTOR=1.2 rails runner scripts/rbup.rb & echo $! >> tmp/pids/rbup.pid EOC sleep 0.1 until alive?end

The problem 1/6Script to check memory usage

def alive? lines = `ps -p #{server_pid} | wc -l`.to_i lines == 2end

def stop_script puts "stopping script..." Process.kill :KILL, server_pid if server_pid && alive? Process.wait @child delete_pidend

def delete_pid `if test -f tmp/pids/rbup.pid; then rm tmp/pids/rbup.pid; fi`end

def server_pid `cat tmp/pids/rbup.pid`.to_iend

def memory_size_mb (`ps -o rss= -p #{server_pid}`.to_i * 1024).to_f / 2**20end

https://gist.github.com/nasospsa/f7fadd6c2f3510b420525f960dc12ddd

The problem 2/6Script to check memory usage

start_scriptseconds = 0used_mb = 0max_mem = 0

while alive? used_mb = memory_size_mb max_mem = [max_mem, used_mb].max puts "#{seconds}sec - #{'%.2f' % used_mb.round} MB" seconds += 1 sleep 1end

stop_script

puts "Total Time: #{seconds}sec"puts "Max Memory: #{'%.2f' % max_mem} MB"

https://gist.github.com/nasospsa/f7fadd6c2f3510b420525f960dc12ddd

The problem 3/6

mapped = {}

fields = [:weights].concat(FoodsController::EAGER_LOADED_FIELDS)

Food.includes(*fields).find_each do |f|

mapped[f.id] = { json_data: f.as_json(methods: [:nutrient_data_basic]) }

end

Food.update(mapped.keys, mapped.values)

First iteration of the migration:

Result: Max Memory 1,1GB | 300 seconds

fields = [:weights].concat(FoodsController::EAGER_LOADED_FIELDS)

Food.includes(*fields).find_each do |f|

f.update json_data: f.as_json(methods: [:nutrient_data_basic])

end

The problem 4/6 Second iteration of the migration:

Result: Max Memory 1GB | 273 seconds

fields = [:weights].concat(FoodsController::EAGER_LOADED_FIELDS)

Food.includes(*fields).find_in_batches(batch_size: 200) do |group|

ActiveRecord::Base.transaction do

group.each do |f|

f.update json_data: f.as_json(methods: [:nutrient_data_basic])

end

end

end

The problem 5/6Third iteration of the migration:

Result: Max Memory 380MB | 160 seconds

The problem 6/6Fourth successful iteration of the migration:

Result: Max Memory 265MB | 82 seconds

fields = [:weights].concat(FoodsController::EAGER_LOADED_FIELDS)

Food.includes(*fields).find_in_batches(batch_size: 50) do |group|

group.each do |f|

f.update_columns json_data: f.as_json(methods: [:nutrient_data_basic])

end

end

update_columns(attributes)Updates the attributes directly in the database issuing an UPDATE SQL statement and sets them in the receiver.

This is the fastest way to update attributes because it goes straight to the database, but take into account that in consequence the regular update procedures are totally bypassed.

3 Aha! Moments > a moment of sudden insight or discovery.

1. Talk to other people.

2. Sleep on it.

3. Make a presentation.

psarrakos@gmail.comgithub.com/nasospsalinkedin.com/in/nasos-psarrakos

Thank You