Memory Issues in Ruby on Rails Applications
-
Upload
simeon-simeonov -
Category
Technology
-
view
602 -
download
3
description
Transcript of Memory Issues in Ruby on Rails Applications
![Page 1: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/1.jpg)
Memory Issuesin Rails Applications
![Page 2: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/2.jpg)
I am @simeons
![Page 3: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/3.jpg)
recruit amazing people
solve hard problems !
ship !
make users happy !
repeat
![Page 4: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/4.jpg)
![Page 5: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/5.jpg)
Problems of Success (good problems)
Too many users Too much traffic Too much data
![Page 6: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/6.jpg)
Memory Issuesin Rails Applications
Common Problem of Success
![Page 7: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/7.jpg)
![Page 8: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/8.jpg)
Display AdvertisingMakes the Web Suck
User-focused optimization Tens of millions of users
1000+% better than average 200+% better than Google
Swoop Fixes That
![Page 9: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/9.jpg)
Mobile SDKs iOS & Android
Web SDK RequireJS & jQuery
Components AngularJS
NLP, etc. Python
Targe<ng High-‐Perf Java
Analy<cs Ruby 2.0
Internal Apps Ruby 2.0 / Rails 3
Pub Portal Ruby 2.0 / Rails 3
Ad Portal Ruby 2.0 / Rails 4
![Page 10: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/10.jpg)
Before 1hr @ 4Gb
![Page 11: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/11.jpg)
Before 1hr @ 4Gb
When problems grow faster than the rate at which you can throw HW at them, you actually have to solve them
![Page 12: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/12.jpg)
Before 1hr @ 4Gb
After 5min @ 230Mb
![Page 13: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/13.jpg)
Resolving Memory Issuesin Rails ApplicationsUsing Streams
![Page 14: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/14.jpg)
CSV
![Page 15: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/15.jpg)
0
125
250
375
500
0 25,000 50,000 75,000 100,000
Rows
Mem
ory
(Mb)
![Page 16: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/16.jpg)
0
125
250
375
500
0 25,000 50,000 75,000 100,000
Rows
Mem
ory
(Mb)
You are here
![Page 17: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/17.jpg)
0
125
250
375
500
0 25,000 50,000 75,000 100,000
Rows
Mem
ory
(Mb)
You are here
This sucks
![Page 18: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/18.jpg)
![Page 19: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/19.jpg)
0
125
250
375
500
0 25,000 50,000 75,000 100,000
Rows
Mem
ory
(Mb)
You are here
This sucks
Start thinking here
![Page 20: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/20.jpg)
Memory Leaks
![Page 21: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/21.jpg)
class AddDomainsStep def call(hashes) hashes.map do |hash| transform_and_return(hash) end end end
![Page 22: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/22.jpg)
1 class AddDomainsStep 2 def initialize 3 @domain_config = DomainConfig.instance 4 end 5 6 def call(hashes) 7 hashes.each do |hash| 8 hash['domain'] = 9 @domain_config. 10 domain_for(hash['domain_id']) 11 end 12 end 13 end
![Page 23: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/23.jpg)
1 class DomainConfig 2 include Singleton 3 4 def initialize 5 @domains = {} 6 end 7 8 def domain_for(id) 9 @domains[id] ||= Domain.name_for(id) || '' 10 end 11 end
![Page 24: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/24.jpg)
@domains[id] ||= Domain.name_for(id) || ''
![Page 25: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/25.jpg)
![Page 26: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/26.jpg)
Memory Leak
•Memory that will never be released by the garbage collector.
•Memory usage grows the longer the process runs.
![Page 27: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/27.jpg)
Avoid Global State
•Global variables
•Class variables
•Singletons
•Per-process instance state
![Page 28: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/28.jpg)
Memory Churn
![Page 29: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/29.jpg)
hashes.map do |hash| hash['domain'].downcase.strip end
hashes.each do |hash| hash['domain'].downcase! hash['domain'].strip! end
vs
![Page 30: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/30.jpg)
Memory Churn
•Allocating and deallocating tons of objects slows down processing
•Mutation limits allocations, but makes it easier to introduce bugs
![Page 31: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/31.jpg)
1 hashes.each do |hash| 2 hash['domain'].downcase! 3 hash['domain'].strip! 4 end
Spot the Bug!
![Page 32: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/32.jpg)
# In shared state: @domains[id] ||= Domain.name_for(id) || '' !
# Much later: hash['domain'].downcase! hash['domain'].strip!
![Page 33: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/33.jpg)
Good News!•Allocating and freeing objects is
fairly fast in Ruby •Keeping your stack frame light
will limit the effects of memory churn
![Page 34: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/34.jpg)
Memory Bloat
![Page 35: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/35.jpg)
def to_csv csv = [CSV.generate_line(headers)] !
rows.each do |row| values = headers.map do |header| row[header] || defaults[header] end !
csv << CSV.generate_line(values) end !
csv.join('') end
![Page 36: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/36.jpg)
def to_csv csv = [CSV.generate_line(headers)] !
rows.each do |row| values = headers.map do |header| row[header] || defaults[header] end !
csv << CSV.generate_line(values) end !
csv.join('') end
![Page 37: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/37.jpg)
def to_csv csv = [CSV.generate_line(headers)] !
rows.each do |row| values = headers.map do |header| row[header] || defaults[header] end !
csv << CSV.generate_line(values) end !
csv.join('') end
![Page 38: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/38.jpg)
Memory Bloat
•Memory usage grows with data set
•Loading too much data at once
![Page 39: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/39.jpg)
Laziness
![Page 40: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/40.jpg)
rename_report_fields( squash( add_domains( add_properties( unwind_variations( rows ) ) ) ) )
![Page 41: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/41.jpg)
def duplicate(number, count) if count > 0 [number] + repeat(number, count - 1) else [] end end !
def sum(list) list.inject(0) do |result, number| result + number end end
![Page 42: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/42.jpg)
sum(repeat(5,10)) # => 50
![Page 43: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/43.jpg)
duplicate :: Int -> Int -> [Int] duplicate number count | count <= 0 = [] | otherwise = number:duplicate number (count - 1) !sum :: [Int] -> Int sum [x] = x sum (x:remaining) = x + sum remaining
![Page 44: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/44.jpg)
> sum $ duplicate 5 10 50
![Page 45: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/45.jpg)
Be ProactiveAbout Being Lazy
![Page 46: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/46.jpg)
Enumerable
![Page 47: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/47.jpg)
class AddDomainsStep def initialize(source) @source = source end !
def each @source.each do |hash| hash['domain'] = DomainConfig. instance. domain_for(hash['domain_id']) yield hash end end end
![Page 48: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/48.jpg)
RenameReportFieldsStep.new( SquashStep.new( AddDomainsStep.new( AddPropertiesStep.new( UnwindVariationsStep.new( rows ) ) ) ) )
![Page 49: Memory Issues in Ruby on Rails Applications](https://reader031.fdocuments.net/reader031/viewer/2022020115/5453d98cb1af9f95228b47b4/html5/thumbnails/49.jpg)
Buffering