Modeling Denormalization - dweebd Denormalization.pdfModeling Denormalization Duncan Beevers 2008...
Transcript of Modeling Denormalization - dweebd Denormalization.pdfModeling Denormalization Duncan Beevers 2008...
![Page 1: Modeling Denormalization - dweebd Denormalization.pdfModeling Denormalization Duncan Beevers 2008 What is denormalization? Duplicating facts Duplicating facts Modeling Piles of things](https://reader034.fdocuments.net/reader034/viewer/2022042922/5f6ddb458f1ca0089c4b9d97/html5/thumbnails/1.jpg)
Modeling Denormalization
Duncan Beevers 2008
![Page 2: Modeling Denormalization - dweebd Denormalization.pdfModeling Denormalization Duncan Beevers 2008 What is denormalization? Duplicating facts Duplicating facts Modeling Piles of things](https://reader034.fdocuments.net/reader034/viewer/2022042922/5f6ddb458f1ca0089c4b9d97/html5/thumbnails/2.jpg)
What is denormalization?
Duplicating facts
Duplicating facts
![Page 3: Modeling Denormalization - dweebd Denormalization.pdfModeling Denormalization Duncan Beevers 2008 What is denormalization? Duplicating facts Duplicating facts Modeling Piles of things](https://reader034.fdocuments.net/reader034/viewer/2022042922/5f6ddb458f1ca0089c4b9d97/html5/thumbnails/3.jpg)
Modeling
Piles of things are things
class Game < ActiveRecord::Base has_many :plays has_one :lifetime_plays_countend
![Page 4: Modeling Denormalization - dweebd Denormalization.pdfModeling Denormalization Duncan Beevers 2008 What is denormalization? Duplicating facts Duplicating facts Modeling Piles of things](https://reader034.fdocuments.net/reader034/viewer/2022042922/5f6ddb458f1ca0089c4b9d97/html5/thumbnails/4.jpg)
Where we started
Fully normalized data
![Page 5: Modeling Denormalization - dweebd Denormalization.pdfModeling Denormalization Duncan Beevers 2008 What is denormalization? Duplicating facts Duplicating facts Modeling Piles of things](https://reader034.fdocuments.net/reader034/viewer/2022042922/5f6ddb458f1ca0089c4b9d97/html5/thumbnails/5.jpg)
First denormalization
![Page 6: Modeling Denormalization - dweebd Denormalization.pdfModeling Denormalization Duncan Beevers 2008 What is denormalization? Duplicating facts Duplicating facts Modeling Piles of things](https://reader034.fdocuments.net/reader034/viewer/2022042922/5f6ddb458f1ca0089c4b9d97/html5/thumbnails/6.jpg)
Why denormalize?
Large number of rows
Frequent expensive queries
Complex business rules about data
![Page 7: Modeling Denormalization - dweebd Denormalization.pdfModeling Denormalization Duncan Beevers 2008 What is denormalization? Duplicating facts Duplicating facts Modeling Piles of things](https://reader034.fdocuments.net/reader034/viewer/2022042922/5f6ddb458f1ca0089c4b9d97/html5/thumbnails/7.jpg)
Complex Business Rules
They make simple Business Rules?
![Page 8: Modeling Denormalization - dweebd Denormalization.pdfModeling Denormalization Duncan Beevers 2008 What is denormalization? Duplicating facts Duplicating facts Modeling Piles of things](https://reader034.fdocuments.net/reader034/viewer/2022042922/5f6ddb458f1ca0089c4b9d97/html5/thumbnails/8.jpg)
Model the requirement
Just get it working first
![Page 9: Modeling Denormalization - dweebd Denormalization.pdfModeling Denormalization Duncan Beevers 2008 What is denormalization? Duplicating facts Duplicating facts Modeling Piles of things](https://reader034.fdocuments.net/reader034/viewer/2022042922/5f6ddb458f1ca0089c4b9d97/html5/thumbnails/9.jpg)
class DailyHighScore < ActiveRecord::Base set_table_name 'scores'
default_scope :select => "#{quoted_table_name}.*, MAX(value) AS value" named_scope :for_date, lambda { |date| { :conditions => { :created_at => date.beginning_of_day .. date.end_of_day } } }end
class User < ActiveRecord::Base has_many :daily_high_scores end
user.daily_high_scores. for_date(Date.today).first
![Page 10: Modeling Denormalization - dweebd Denormalization.pdfModeling Denormalization Duncan Beevers 2008 What is denormalization? Duplicating facts Duplicating facts Modeling Piles of things](https://reader034.fdocuments.net/reader034/viewer/2022042922/5f6ddb458f1ca0089c4b9d97/html5/thumbnails/10.jpg)
class ActiveRecord::Base def self.default_scope options self.scoped_methods << { :find => options } endend
![Page 11: Modeling Denormalization - dweebd Denormalization.pdfModeling Denormalization Duncan Beevers 2008 What is denormalization? Duplicating facts Duplicating facts Modeling Piles of things](https://reader034.fdocuments.net/reader034/viewer/2022042922/5f6ddb458f1ca0089c4b9d97/html5/thumbnails/11.jpg)
Haphazard denormalization
![Page 12: Modeling Denormalization - dweebd Denormalization.pdfModeling Denormalization Duncan Beevers 2008 What is denormalization? Duplicating facts Duplicating facts Modeling Piles of things](https://reader034.fdocuments.net/reader034/viewer/2022042922/5f6ddb458f1ca0089c4b9d97/html5/thumbnails/12.jpg)
Minimize Indices-per-tablemore indices = slower writes
![Page 13: Modeling Denormalization - dweebd Denormalization.pdfModeling Denormalization Duncan Beevers 2008 What is denormalization? Duplicating facts Duplicating facts Modeling Piles of things](https://reader034.fdocuments.net/reader034/viewer/2022042922/5f6ddb458f1ca0089c4b9d97/html5/thumbnails/13.jpg)
Minimize complex queriesMake sure you're using your indices!
![Page 14: Modeling Denormalization - dweebd Denormalization.pdfModeling Denormalization Duncan Beevers 2008 What is denormalization? Duplicating facts Duplicating facts Modeling Piles of things](https://reader034.fdocuments.net/reader034/viewer/2022042922/5f6ddb458f1ca0089c4b9d97/html5/thumbnails/14.jpg)
Eventual Consistency
Why use real tables instead of triggers or views?
![Page 15: Modeling Denormalization - dweebd Denormalization.pdfModeling Denormalization Duncan Beevers 2008 What is denormalization? Duplicating facts Duplicating facts Modeling Piles of things](https://reader034.fdocuments.net/reader034/viewer/2022042922/5f6ddb458f1ca0089c4b9d97/html5/thumbnails/15.jpg)
Do it later
Batch processing can save you writes and lighten replication load
![Page 16: Modeling Denormalization - dweebd Denormalization.pdfModeling Denormalization Duncan Beevers 2008 What is denormalization? Duplicating facts Duplicating facts Modeling Piles of things](https://reader034.fdocuments.net/reader034/viewer/2022042922/5f6ddb458f1ca0089c4b9d97/html5/thumbnails/16.jpg)
Take a picture, it'll last longer
Denormalized models capture history
![Page 17: Modeling Denormalization - dweebd Denormalization.pdfModeling Denormalization Duncan Beevers 2008 What is denormalization? Duplicating facts Duplicating facts Modeling Piles of things](https://reader034.fdocuments.net/reader034/viewer/2022042922/5f6ddb458f1ca0089c4b9d97/html5/thumbnails/17.jpg)
Like Observers, but different
class DailyHighScore uniqueness :user_id, :statistic_id, :date
takes :user_id, :statistic_id, :from => Score
takes :value, :from => Score, :if => :new_score_better?
def new_score_better? new_score new_score.better_than? self.value endend
![Page 18: Modeling Denormalization - dweebd Denormalization.pdfModeling Denormalization Duncan Beevers 2008 What is denormalization? Duplicating facts Duplicating facts Modeling Piles of things](https://reader034.fdocuments.net/reader034/viewer/2022042922/5f6ddb458f1ca0089c4b9d97/html5/thumbnails/18.jpg)
Use Descriptive NamesVeryLongDescriptiveNamesThatProgrammingPairsThinkProvideGoodDescriptions
class LifetimeHighScoreend
class DailyGameplaysCountend
class MonthlyPaymentRunend
![Page 19: Modeling Denormalization - dweebd Denormalization.pdfModeling Denormalization Duncan Beevers 2008 What is denormalization? Duplicating facts Duplicating facts Modeling Piles of things](https://reader034.fdocuments.net/reader034/viewer/2022042922/5f6ddb458f1ca0089c4b9d97/html5/thumbnails/19.jpg)
Avoiding our mistakes
Updates to models triggered external API calls
Single Table Inheritance
![Page 20: Modeling Denormalization - dweebd Denormalization.pdfModeling Denormalization Duncan Beevers 2008 What is denormalization? Duplicating facts Duplicating facts Modeling Piles of things](https://reader034.fdocuments.net/reader034/viewer/2022042922/5f6ddb458f1ca0089c4b9d97/html5/thumbnails/20.jpg)
Use a queue
Lots of choices out there, or just roll your own
![Page 21: Modeling Denormalization - dweebd Denormalization.pdfModeling Denormalization Duncan Beevers 2008 What is denormalization? Duplicating facts Duplicating facts Modeling Piles of things](https://reader034.fdocuments.net/reader034/viewer/2022042922/5f6ddb458f1ca0089c4b9d97/html5/thumbnails/21.jpg)
Ye Olde Cleaver
http://github.com/stopdropandrew/single-table-
inheritance-cleaver
![Page 22: Modeling Denormalization - dweebd Denormalization.pdfModeling Denormalization Duncan Beevers 2008 What is denormalization? Duplicating facts Duplicating facts Modeling Piles of things](https://reader034.fdocuments.net/reader034/viewer/2022042922/5f6ddb458f1ca0089c4b9d97/html5/thumbnails/22.jpg)
Moving to the new scheme
Pulling up stakes
![Page 23: Modeling Denormalization - dweebd Denormalization.pdfModeling Denormalization Duncan Beevers 2008 What is denormalization? Duplicating facts Duplicating facts Modeling Piles of things](https://reader034.fdocuments.net/reader034/viewer/2022042922/5f6ddb458f1ca0089c4b9d97/html5/thumbnails/23.jpg)
Interim strategies
Disabling features and turning tables into placeholders
CREATE TABLE dummy_#{table_name}LIKE #{table_name}ENGINE=BLACKHOLE
ewww SQL
![Page 24: Modeling Denormalization - dweebd Denormalization.pdfModeling Denormalization Duncan Beevers 2008 What is denormalization? Duplicating facts Duplicating facts Modeling Piles of things](https://reader034.fdocuments.net/reader034/viewer/2022042922/5f6ddb458f1ca0089c4b9d97/html5/thumbnails/24.jpg)
Toolbox
Bulk insert http://gist.github.com/7681
Hash rewrite http://www.dweebd.com/ruby/hash-key-rewrite/
{ :username => 'johnnyfive' }. rewrite(:username => :login)
=> { :login => 'johnnyfive' }
Gameplay.insert( [ { :user_id => 1, :game_id => 1 }, { :user_id => 2, :game_id => 1 }] )
![Page 25: Modeling Denormalization - dweebd Denormalization.pdfModeling Denormalization Duncan Beevers 2008 What is denormalization? Duplicating facts Duplicating facts Modeling Piles of things](https://reader034.fdocuments.net/reader034/viewer/2022042922/5f6ddb458f1ca0089c4b9d97/html5/thumbnails/25.jpg)
Catch-up
Creating denormalized data should be idempotent
![Page 26: Modeling Denormalization - dweebd Denormalization.pdfModeling Denormalization Duncan Beevers 2008 What is denormalization? Duplicating facts Duplicating facts Modeling Piles of things](https://reader034.fdocuments.net/reader034/viewer/2022042922/5f6ddb458f1ca0089c4b9d97/html5/thumbnails/26.jpg)
Pick the easy ones first
There are lots of ways to speed up an application
![Page 27: Modeling Denormalization - dweebd Denormalization.pdfModeling Denormalization Duncan Beevers 2008 What is denormalization? Duplicating facts Duplicating facts Modeling Piles of things](https://reader034.fdocuments.net/reader034/viewer/2022042922/5f6ddb458f1ca0089c4b9d97/html5/thumbnails/27.jpg)
Make models judiciously
![Page 28: Modeling Denormalization - dweebd Denormalization.pdfModeling Denormalization Duncan Beevers 2008 What is denormalization? Duplicating facts Duplicating facts Modeling Piles of things](https://reader034.fdocuments.net/reader034/viewer/2022042922/5f6ddb458f1ca0089c4b9d97/html5/thumbnails/28.jpg)
Over 3 million badges awarded in August
~1.5 badges per second