MongoSV Schema Workshop

Schema Design Workshop

Sridhar Nanjundeswaran

Software Engineer, [email protected]

@snanjund

Wednesday, December 5, 12

Agenda

• Part One - Basic Schema & Patterns• Part Two - Schema Design• Part Three - Sharding• Part Four: - Replication


Why is schema design different?• RDBMS design you ask "what answers do I have"

• MongoDB you ask "what questions will I have"


Goals

• Learn Data Modeling with MongoDB• Labs to try to solve problems• Understand implications of• Replication • Sharding

Please, ask many, many questions!


Part OneBasic Schema & Patterns


So why model data?

http://bit.ly/SSs7QB




Normalization• 1970 E.F.Codd introduces 1st Normal Form (1NF)• 1971 E.F.Codd introduces 2nd and 3rd Normal Form (2NF, 3NF)• 1974 Codd & Boyce define Boyce/Codd Normal Form (BCNF)• 2002 Date, Darween, Lorentzos define 6th Normal Form (6NF)

Goals:• Avoid anomalies when inserting, updating or deleting• Minimize redesign when extending the schema• Make the model informative to users• Avoid bias towards a particular style of query

* source : wikipediaWednesday, December 5, 12

So today’s example will use...

http://bit.ly/RyIOvO




TerminologyRDBMS MongoDB

Table Collection

Row(s) JSON Document

Index Index

Join Embedding & Linking

Partition Shard

Partition Key Shard Key


Schema DesignRelational Database


Schema DesignMongoDB



linking



embedding

linking


Basic schema

Design documents that simply map to your application

> post = { author: "Hergé", date: ISODate("2011-09-18T09:56:06.298Z"), text: "Destination Moon", tags: ["comic", "movie"] }

> db.blogs.save(post)


> db.blogs.find()

{ _id: ObjectId("4c4ba5c0672c685e5e8aabf3"), author: "Hergé", date: ISODate("2011-09-18T09:56:06.298Z"), text: "Destination Moon", tags: [ "comic", "movie" ] } Notes:• ID must be unique, but can be anything you’d like• MongoDB will generate a default ID if one is not supplied

Find the document


Secondary index for “author”

// 1 means ascending, -1 means descending> db.blogs.ensureIndex( { author: 1 } )

> db.blogs.find( { author: 'Hergé' } ) { _id: ObjectId("4c4ba5c0672c685e5e8aabf3"), date: ISODate("2011-09-18T09:56:06.298Z"), author: "Hergé", ... }

Add an index, find via Index


Examine the query plan

> db.blogs.find( { author: "Hergé" } ).explain(){! "cursor" : "BtreeCursor author_1",! "nscanned" : 1,! "nscannedObjects" : 1,! "n" : 1,! "millis" : 5,! "indexBounds" : {! ! "author" : [! ! ! [! ! ! ! "Hergé",! ! ! ! "Hergé"! ! ! ]! ! ]! }}


Examine the query plan

> db.blogs.find( { author: "Hergé" } ).explain(){! "cursor" : "BtreeCursor author_1",! "nscanned" : 1,! "nscannedObjects" : 1,! "n" : 1,! "millis" : 5,! "indexBounds" : {! ! "author" : [! ! ! [! ! ! ! "Hergé",! ! ! ! "Hergé"! ! ! ]! ! ]! }}

How long it took

Number of objects returned


Query operatorsConditional operators: $ne, $in, $nin, $mod, $all, $size, $exists, $type, .. $lt, $lte, $gt, $gte, $ne...

// find posts with any tags> db.blogs.find( { tags: { $exists: true } } )

Regular expressions:// posts where author starts with h> db.blogs.find( { author: /^h/i } )

Counting: // number of posts written by Hergé> db.blogs.find( { author: "Hergé" } ).count()


Extending the Schema

http://bit.ly/PpjT1l


http://nysi.org.uk/kids_stuff/rocket/rocket.htm

http://nysi.org.uk/kids_stuff/rocket/rocket.htm

Extending the Schema> new_comment = { author: "Kyle", date: new Date(), text: "great book" }

> db.blogs.update( { text: "Destination Moon" }, { "$push": { comments: new_comment }, "$inc": { comments_count: 1 } } )


Extending the Schema> new_comment = { author: "Kyle", date: new Date(), text: "great book" }

> db.blogs.update( { text: "Destination Moon" }, { "$push": { comments: new_comment }, "$inc": { comments_count: 1 } } )

Increment counterAdd element to

array


> db.blogs.find( { author: "Hergé"} )

{ _id : ObjectId("4c4ba5c0672c685e5e8aabf3"), author : "Hergé", date : ISODate("2011-09-18T09:56:06.298Z"), text : "Destination Moon", tags : [ "comic", "movie" ], comments : [! {! ! author : "Kyle",! ! date : ISODate("2011-09-19T09:56:06.298Z"),! ! text : "great book"! } ], comments_count: 1 }



// create index on nested documents:> db.blogs.ensureIndex( { "comments.author": 1 } )

> db.blogs.find( { "comments.author": "Kyle" } )

// find last 5 posts:> db.blogs.find().sort( { date: -1 } ).limit(5)

// most commented post:> db.blogs.find().sort( { comments_count: -1 } ).limit(1)

When sorting, check if you need an index



Common Patterns

http://bit.ly/SNnt4z




Inheritance

http://bit.ly/T7MqUz




Inheritance


select * from shapes;

id type area radius length width

1 circle 3.14 1

2 square 4 2

3 rect 10 5 2

Single Table Inheritance - RDBMS


Single Table Inheritance - MongoDB> db.shapes.find() { _id: "1", type: "c", area: 3.14, radius: 1} { _id: "2", type: "s", area: 4, length: 2} { _id: "3", type: "r", area: 10, length: 5, width: 2}

missing values not stored!



// find shapes where radius > 0 > db.shapes.find( { radius: { $gt: 0 } } )



// find shapes where radius > 0 > db.shapes.find( { radius: { $gt: 0 } } )

// create index> db.shapes.ensureIndex( { radius: 1 }, { sparse:true } )

index only values present!


One to Many

http://bit.ly/Oqbt8z




One to Many

One to Many relationships can specify• degree of association between objects• containment• life-cycle


One to ManyEmbedded Array

•$slice operator to return subset of comments•some queries harder

•e.g find latest comments across all blogs

blogs: { author : "Hergé", date : ISODate("2011-09-18T09:56:06.298Z"), comments : [! { author : "Kyle",! ! date : ISODate("2011-09-19T09:56:06.298Z"),! ! text : "great book" } ] }

> db.blogs.find( { author: "Hergé" }, { comment: { $slice : 10 } } )


One to ManyNormalized (2 collections)• most flexible• more queries

blogs: { _id: 1000, author: "Hergé", date: ISODate("2011-09-18T09:56:06.298Z"), comments: [! {comment : 1)} ]}

comments : { _id : 1, blog: 1000, author : "Kyle",! ! date : ISODate("2011-09-19T09:56:06.298Z")}

> blog = db.blogs.find( { text: "Destination Moon" } );> db.comments.find( { blog: blog._id } ).limit(5);


Many to Many

http://bit.ly/QTzhBF




Many - Many

Example: • Blog can have many Tags• Tag can be used by many Blogs


// Each Tag lists the "_id" of the Blogtags: { _id: 20, name: "comic", // Unique blog_ids: [ 10, 11, 12 ] }

{ _id: 30, name: "movie", // Unique blog_ids: [ 10 ] }

Many - Many




// Each Blog lists the "tag" of the Tagsblogs: { _id: 10, name: "Destination Moon", tags: [ "comic", "movie" ] }

Many - Many




// Each Blog lists the "tag" of the Tagsblogs: { _id: 10, name: "Destination Moon", tags: [ "comic", "movie" ] }

Many - Many

links via unique key, in this case "tags", could be "_id"




// Each Blog lists the "tag" of the Tagsblogs: { _id: 10, name: "Destination Moon", tags: [ "comic", "movie" ] } // All Tags for a given Blog> db.tags.find( { blog_ids: 10 } )

Many - Many


Use _id or not?

blogs: { _id: 10, name: "..." tags: [ "comic", "movie" ] }

Pros:• Single query

Cons:• Cascade any changes

blogs: { _id: 10, name: "..." tags: [ 10, 20 ] }

Pros:• Single update

Cons:• Second query required


// Each Blog lists the _id of the Tagblogs: { _id: 10, name: "Destination Moon", tag_ids: [ 20, 30 ] } // Association not stored on the Tagtags: { _id: 20, name: "comic" }

Alternative



// All Blogs for a given Tag> db.blogs.find( { tag_ids: 20 } )

Alternative



// All Blogs for a given Tag> db.blogs.find( { tag_ids: 20 } )

// All Tags for a given Blog> blog = db.blogs.findOne( { _id: 10 } )> db.tags.find({_id: {$in : blog.tag_ids}})

Alternative


Many - Many Intersection AttributesExample: • Blog can have many Tags• Tag can be used my many Blogs• When a Tag is used, record the usage date


// Each Blog lists the _id of the Tagblogs: { _id: 10, name: "...", tag_ids: [ 20, 30 ] } // Association not stored on the Tagtags: { _id: 20, name: "comic" }

// Store the interaction and usage dateusages: { blog_id: 10, // Blog _id tag_id : 20, // Tag _id usage: ISODate("2012-10-12...") }

// Find the Tags for a Blogfor(var c = db.usages.find({ blog_id: 10 }); c.hasNext(); ){ u = c.next(); t = db.tags.findOne( { _id: c.tag_id } ) printjson( u.usage );

Many - Many Normalized


// Each Blog lists the Blog Usage Objectblogs: { _id: 10, name: "Destination Moon", tags: [ { tag: "comic", usage: ISODate("2012-10-12...") } { tag: "movie", usage: ISODate("2012-09-11...") } ] }

// Find the Tags for a Blog> db.blogs.find( { _id: 10 }, { tags: 1} ) Pros:• Usage object encapsulated where used

Cons:• If updates allowed, changes will have to be cascaded

Many - Many Intersection Attributes


Summary

• Single biggest performance factor

• More choices than in an RDBMS

• Embedding, index design, shard keys


Part TwoSchema Design


Lab #1Design Schema for Twitter

• Model each users activity stream• Users

• Name, email address, display name• Tweets

• Text• Who• Timestamp


Lab #1 - Solution ATwo Collections// users - one doc per user{ _id: "alvin", email: "[email protected]", display: "jonnyeight"}

// tweets - one doc per user per tweet{ user: "bob", for: "alvin", tweet: "20111209-1231", text: "Best Tweet Ever!", ts: ISODate("2011-09-18T09:56:06.298Z")}


mailto:[email protected]


Lab #1 - Solution BEmbedded Tweets// users - one doc per user with all tweets{ _id: "alvin", email: "[email protected]", display; "jonnyeight", tweets: [! {! ! user: "bob",! ! tweet: "20111209-1231",! ! text: "Best Tweet Ever!", ts: ISODate("2011-09-18T09:56:06.298Z")! } ]}




Embedding

• Great for read performance

• One seek to load entire object

• One roundtrip to database

• Writes can be slow if adding to objects all the time


Linking or Embedding?

Linking can make some queries easy

// Find latest 50 tweets for "alvin"> db.tweets.find( { _id:"alvin"} ) .sort( {ts:-1} ) .limit(50)

But what effect does this have on the systems?


Collection 1

Index 1


Virtual Address Space 1

Collection 1

Index 1 This is your virtual memory size

(mapped)



Physical RAM

Collection 1

Index 1

This is your resident

memory size



Physical RAM

DiskCollection 1

Index 1



Physical RAM

DiskCollection 1

Index 1

100 ns

10,000,000 ns

=

=



Physical RAM

DiskCollection 1

Index 1

> db.tweets.find( { _id: "alvin" } ) .sort( { ts: -1 } ) .limit(10)

1

2

3

Linking = Many seeks + random reads



Physical RAM

DiskCollection 1

Index 1

1

Embedding = Large Sequential Read

> db.tweets.find( { _id: "alvin" } )


Lab #2Alternative Schema

• Display last 10 tweets from today• Efficiently use memory and Disk seeks / IOPs


Lab #2 - SolutionBuckets// tweets : one doc per user per day> db.tweets.findOne()

{ _id: "alvin-2011/12/09", email: "[email protected]", tweets: [ { user: "Bob",! tweet: "20111209-1231",! text: "Best Tweet Ever!" } , ! { author: "Joe",! tweet: "20111210-9025",! date: "May 27 2011",! text: "Stuck in traffic (again)" } ]}




Lab #2 - SolutionLast 10 Tweets

> db.tweets.find( { _id: "alvin-2011/12/09" }, { tweets: { $slice : 10 } } ) .sort( { _id: -1 } ) .limit(1)


Lab #2 - SolutionAdding a Tweet> tweet = { user: "Bob",! tweet: "20111209-1231",! text: "Best Tweet Ever!" }

> db.tweets.update( { _id : "alvin-2011/12/09" }, { $push : { tweets : tweet } );


Lab #2 - SolutionGetting All Tweets> cursor = db.tweets.find ( { _id : /^alvin/ } ).sort( { _id : -1 } )

> while ( cursor.hasNext() ) { doc = cursor.next(); for ( var i=0; i<doc.tweets.length; i++ ) printjson( doc.tweets[i] )}


Lab #2 - SolutionDeleting a Tweet> db.tweets.update( { _id: "alvin-20111209" }, { $pull: { tweets: { tweet: "20111209-1231" } })



Physical RAM

DiskCollection 1

Index 1

> db.tweets.find( { _id: "alvin-2011/12/09" }, { tweets: { $slice : 10 } } ) .sort( { _id: -1 } ) .limit(1)

Bucket = 1 seek + 1 sequential read

1


Trees

http://bit.ly/Oqc8Xs




Trees

Hierarchical information


Trees

Full Tree in Document

{ retweet: [ { who: “Kyle”, text: “...”, retweet: [ {who: “James”, text: “...”, retweet: []} ]} ]}

Pros: Single Document, Performance, Intuitive

Cons: Hard to search, Partial Results, 16MB limit


Array of Ancestors// Store all Ancestors of a node { _id: "a" } { _id: "b", tree: [ "a" ], retweet: "a" } { _id: "c", tree: [ "a", "b" ], retweet: "b" } { _id: "d", tree: [ "a", "b" ], retweet: "b" } { _id: "e", tree: [ "a" ], retweet: "a" } { _id: "f", tree: [ "a", "e" ], retweet: "e" }

A B C

DE

F


Array of Ancestors// Store all Ancestors of a node { _id: "a" } { _id: "b", tree: [ "a" ], retweet: "a" } { _id: "c", tree: [ "a", "b" ], retweet: "b" } { _id: "d", tree: [ "a", "b" ], retweet: "b" } { _id: "e", tree: [ "a" ], retweet: "a" } { _id: "f", tree: [ "a", "e" ], retweet: "e" }

// find all direct retweets of "b"> db.tweets.find( { retweet: "b" } )

// find all retweets of "e" anywhere in tree> db.tweets.find( { tree: "e" } )

// find tweet history of f:> tweets = db.tweets.findOne( { _id: "f" } ).tree> db.tweets.find( { _id: { $in : tweets } } )

A B C

DE

F


Trees as Paths

Store hierarchy as a path expression• Separate each node by a delimiter, e.g. “/”• Use text search for find parts of a tree

{ retweets: [ { _id: "a", text: "initial tweet", path: "a" }, { _id: "b", text: "reweet with comment", path: "a/b" }, { _id: "c", text: "reply to retweet", path : "a/b/c"} ] }

// Find the conversations "a" started > db.tweets.find( { path: /^a/i } )

A B C

DE

F


http://bit.ly/QeNsPX

Queues & Workflows




Lab #3Following Requests• Users are allowed to "follow" another user

• User send a "follow" request• Follower approves or not• Requests are timed out after 7 days

• The approval is an async process


Lab #3 - SolutionQueues & Workflows• Need to maintain order and state• Ensure that updates are atomic

> db.approvals.insert( { inprogress: false, approved: false, priority: 1, text: "Hey Jim, want to follow you!" } );// find highest priority approval and mark as in-progressjob = db.approvals.findAndModify({ query: { inprogress: false }, sort: { priority: -1 }, update: { $set: { inprogress: true, started: new Date() } }, new: true})


Lab #3 - SolutionQueues & Workflows

{ inprogress: true, priority: 1, approved: False, started: ISODate("2011-09-18T09:56:06.298Z") ... }

updated

added


Lab #3 - SolutionQueues & Workflows• Follower approves request

// update approval after receiving approval> job = db.approvals.update( { _id: "1234" }, { $set: { approved: true } } )

• System times out request after 7 days

var limit=new Date();limit.setDate(limit.getDate()-7);

> job = db.approvals.update( { inprogress: true, started: { $gt: limit} }, { $set: { approved: false } } )


Lab #4Voting

Twitter meets Stack Overflow

• Users can "vote" for a tweet• A user can "vote" once and only once• Need to display current votes


Lab #4 - SolutionVotes// One document per voter per tweet> db.votes.insert( { tweet: "20111209-1231", voter: "alvin" } );

// Unique index guarantees the user can't vote twice> db.votes.ensureIndex( { tweet: 1, voter: 1 }, { unique: true } );

// Count will return the number of votes cast> db.votes.find({ tweet: "20111209-1231" }).count()


Count or Not?

• Indexes in MongoDB are not counting• The count has to be computed via a index scan

// One summary document per tweet, no "voter" key> db.votes.update( { tweet: "20111209-1231", voter: { $exists: false } }, { "$inc": { count: 1 } }, true, false );

// Return the count for the no "voter" document> db.votes.find( { tweet: "20111209-1231", voter: { $exists: false } }, { count: 1, _id: 0} )


Lab #5Time Series• Records votes by

• Day, Hour, Minute• Show time series of votes cast


Lab #5 - Solution ATime Series// Time series buckets, hour and minute sub-docs{ _id: "20111209-1231", ts: ISODate("2011-12-09T00:00:00.000Z") daily: 67, hourly: { 0: 23, 1: 14, 2: 19 ... 23: 72 }, minute: { 0: 0, 1: 4, 2: 6 ... 1439: 0 }}


Lab #5 - Solution ATime Series// Add one to the last minute before midnight> db.votes.update( { _id: "20111209-1231", ts: ISODate("2011-12-09T00:00:00.037Z") }, { $inc: { daily: 1 }, $inc: { "hourly.23": 1 }, $inc: { "minute.1439": 1 } )

What is the cost of updating the minute before midnight?


• Sequence of key/value pairs• NOT a hash map• Optimized to scan quickly

• 1439 skips

BSON Storage

...0 1 2 3 1439


• Can skip sub-documents

• 23 skips (hours) + 59 skips (minutes) = 82 skips

BSON Storage

1

0 ...

... ...59

1 23

1380 143960 ... 119


Lab #5 - Solution BTime Series// Time series buckets, each hour a sub-document{ _id: "20111209-1231", ts: ISODate("2011-12-09T00:00:00.000Z") daily: 67, minute: { 0: { 0: 0, 1: 7, ... 59: 2 }, ... 23: { 0: 15, ... 59: 6 } }}

// Add one to the last second before midnight> db.votes.update( { _id: "20111209-1231" }, ts: ISODate("2011-12-09T00:00:00.000Z") }, { $inc: { daily: 1 }, $inc: { "minute.23.59": 1 } })


Lab #6Inventory

• User has a number of "votes" they can use


Lab #6 - SolutionInventory // Number of votes and who voted for { _id: "alvin", votes: 42, voted_for: [] }

// Subtract a vote and add the voted for tweet // "20111209-1231" > db.user.update( { _id: "alvin", votes : { $gt : 0}, voted_for: { $ne: "20111209-1231" }}, { "$push": { voted_for: "20111209-1231"}, "$inc": { votes: -1} } )


Lab #6 - SolutionInventory // After vote > db.votes.findOne() { _id: "alvin", votes: 41, voted_for: ["20111209-1231"] }

decremented

added


Lab #7Statistic Buckets• Record referring web sites on customer sign up• Independent counter for each web site


Lab #7 - Solution AStatistic Buckets{ _id: "alvin", referrers: [ { domain: "www.google.co.uk", count: 4 }, { domain: "www.yahoo.com", count: 1 }, ] }


http://www.google.com/url?sa=D&q=www.google.co.uk&usg=AFQjCNEbx2-zqx8SDaRaqWs6Qck3L4FgzA


http://www.google.com/url?sa=D&q=www.yahoo.com&usg=AFQjCNFaa5y9zjP3x6aZXMueg5TI5mr2rw



> db.referers.update( { "referrers.domain": "www.google.co.uk" }, { $inc: { "referrers.$.count": 1 } } )






http://www.google.com/url?sa=D&q=www.google.co.uk%27&usg=AFQjCNE0kOod3NR5zjalj6kY2RL_fYr8LA



> db.referers.update( { "referrers.domain": "www.google.co.uk" }, { $inc: { "referrers.$.count": 1 } } )

{ _id: "alvin", referrers: [ { domain: "www.google.co.uk", count: 5 }, { domain: "www.yahoo.com", count: 1 }, ] }












Lab #7 - Solution AStatistic Buckets

> db.referers.update( { "referrers.domain": "www.bing.com" }, { $inc: {"referrers.$.count": 1 } }, false, true ) What happens if a new referring site is used?




Lab #7 - Solution BStatistic Buckets// Need to replace dots with underscores{ _id: "alvin", referrers: { "www_google_co_uk": 4, "www_yahoo_com": 1 }, }

// simple $inc will add www_bing_com if not present> db.referers.update( { _id: "alvin" }, { $inc: { "referrers.www_bing_com": 1 } }, true, false);






Part ThreeSharding


What is Sharding

• Ad-hoc partitioning

• Consistent hashing• Amazon Dynamo

• Range based partitioning• Google BigTable• Yahoo! PNUTS• MongoDB


MongoDB Sharding

• Automatic partitioning and management

• Range based

• Convert to sharded system with no downtime

• Fully consistent

• No code changes required


Sharding - Range distribution

shard01 shard02 shard03

sh.shardCollection("mydb.tweets", {_id: 1} , false)


Sharding - Range distribution


a-i j-r s-z


Sharding - Splits


a-i ja-jz s-z

k-r


Sharding - Splits


a-i ja-ji s-z

ji-js

js-jw

jz-r


Sharding - Auto Balancing


a-i ja-ji s-z

ji-js

js-jw

jz-r

js-jw

jz-r


Sharding - Auto Balancing


a-i ja-ji s-z

ji-js

js-jw

jz-r


Sharding for caching


Sharding for caching

shard01

a-i

j-r

s-z

300

GB

Dat

a

300 GB

96 GB Mem3:1 Data/Mem


Aggregate Horizontal Resources


a-i j-r s-z


100 GB 100 GB 100 GB

300

GB

Dat

a



j-r

s-z


Sharding Features• Shard data without no downtime • Automatic balancing as data is written• Commands routed (switched) to correct node

• Inserts - must have the Shard Key• Updates - can have the Shard Key• Queries

• With Shard Key - routed to nodes• Without Shard Key - scatter gather

• Indexed / Sorted Queries• With Shard Key - routed in order• Without Shard Key - distributed sort merge


Lab #8Sharding Twitter Pictures

User can upload pictures to Twitter feed

{ photo_id : ???? , data : <binary> }

What should photo_id be?How will photo_id be sharded?


Lab #8Sharding Key

{ photo_id : ???? , data : <binary> }

What’s the right key?• auto increment• MD5( data )• month() + MD5( data )


• Only have to keep small portion in ram• Right shard "hot" • Time Based

• ObjectId• Auto Increment

Right balanced access


• Have to keep entire index in ram• All shards "warm"

• Hash

Random access


• Have to keep some index in ram• Some shards "warm"

•Month + Hash

Segmented access


Lab #9Single Identities// Shard by _idids:{ _id : "alvin", email: "[email protected]", addresses: [ { state : "CA", country: "USA" }, { country: "UK" } ] }

How would the following queries be executed?

> db.ids.find( { _id: "alvin"} )> db.ids.find( { email: "[email protected]" } )






Sharding - Routed Query


a-i ja-ji s-z

ji-js

js-jw

jz-r

find( { _id: "alvin"} )


Sharding - Routed Query


a-i ja-ji s-z

ji-js

find( { _id: "alvin"} )

js-jw

jz-r


Sharding - Scatter Gather


a-i ja-ji s-z

ji-js

js-jw

jz-r

find( { email: "[email protected]" } )




Lab #9Multiple Identities

User can have multiple identities• twitter name• email address• facebook name• etc.

What is the best sharding key & schema design?


Lab #9 - Solution AMultiple Identities

// Shard by _id{ _id: "alvin", email: "[email protected]", fb: "alvin.richards", // facebook li: "alvin.j.richards", // linkedin tweets: [ ... ] }

Lookup by _id hits 1 node Lookup by email, li or fb is scatter gather Cannot create a unique index on email, li or fb


Lab #9 - Solution BMultiple Identitiesidentities{ _id: { _id: "alvin"}, info: "1200-42"}{ _id: { em: "[email protected]"}, info: "1200-42"}{ _id: { li: "alvin.j.richards"}, info: "1200-42"}

tweets{ _id: "1200-42", tweets: [ ... ]}

• Shard identities on { _id: 1}• Can create unique index on _id• Shard info on { _id: 1 }




Sharding - Multiple Identities


idscollection

tweetscollection

em: a-q em: r-z _id: a-z

li: s-z

li: a-c

li: d-r_id: "Min"-"1100"

_id: "1100"-"1200"

_id: "1200"-"Max"



shard01 shard02 shard03em: a-q em: r-z _id: a-z

li: s-z

li: a-c

li: d-r_id: "Min"-"1100"

_id: "1100"-"1200"

_id: "1200"-"Max"

ids.find({ _id: {"em","[email protected] })

idscollection

tweetscollection






ids.find({ _id: {"em","[email protected] })

tweets.find({ _id: "1200-‐42" })

idscollection

tweetscollection

em: a-q em: r-z _id: a-z

li: s-z

li: a-c

li: d-r_id: "Min"-"1100"

_id: "1100"-"1200"

_id: "1200"-"Max"




Part FourReplication


Types of outage• Planned

• Hardware upgrade• O/S or file-system tuning• Relocation of data to new file-system / storage• Software upgrade

• Unplanned• Hardware failure• Data center failure• Region outage• Human error• Application corruption


Replica Sets

• Data Protection• Multiple copies of the data• Spread across Data Centers, AZs

• High Availability• Automated Failover• Automated Recovery


Replica Sets

Primary

Secondary

Secondary

Read

Write

Read

Read

App

Asynchronous Replication


Replica Sets

Primary

Secondary

Secondary

Read

Write

Read

Read

App


Replica Sets

Primary

Primary

Secondary

Read

Write

Read

Automatic Election of new Primary

App


Replica Sets

Recovering

Primary

Secondary

Read

Write

Read

New primary serves data

App


Replica Sets

Secondary

Primary

Secondary

Read

Write

Read

Read

App


Elections

During an election• Most up to date• Highest priority• Less than 10s behind failed Primary


Types of Durability with MongoDB• Fire and forget• Wait for error • Wait for fsync• Wait for journal sync • Wait for replication


Network Ack- Old Default

Driver Primary

apply in memory

write


Get last error - New default

Driver Primary

getLastError apply in memory

write


Wait for Journal Sync

Driver Primary

apply in memory

write

j:trueWrite to journal

getLastError


Wait for replication

Driver Primary

apply in memory

write

w:2

Secondary

replicate

getLastError


Tunable Data DurabilityMemory Journal Secondary Other Data Center

RDBMS

networkACK

w=1

w=1j=true

w="majority"w=n

w="myTag"

Less More

async

sync


Eventual ConsistencyUsing Replicas for ReadsRead preference• primary (only)• primaryPreferred• secondary (only)• secondaryPreferred• nearest


Immediate Consistency

PrimaryThread #1

Insert

Update

Read

Read

v1

✔

✔

v2


Eventual Consistency

Primary SecondaryThread #1

Insert

Update

Read

Read

v1

Thread #2

✔

✔

v1

✖

✖v2

v2

reads v1

v1 does not exist

✔ reads v2

✔reads v1


Lab #10Replication

Primary, Secondary or both?

• Show the latest "votes" for a tweet and/or user• Changing your profile picture• Showing your thumbnail with a tweet


Summary

• Schema design is different in MongoDB

• Basic data design principals stay the same

• Focus on how the application manipulates data

• Rapidly evolve schema to meet your requirements

• Consider sharding early

• Understand the impact of eventual consistency


@mongodb

conferences, appearances, and meetupshttp://www.10gen.com/events

http://bit.ly/mongo> Facebook | Twitter | LinkedIn

http://linkd.in/joinmongo

download at mongodb.org


MongoSV Schema Workshop

Documents

Transcript of MongoSV Schema Workshop