Cost effective scalinCost effective scalinng for the …...Infobright Evolution MS e, QL, QL NoSQL...

33
Cost effective scalin Cost effective scalin ng for the enterprise ng for the enterprise

Transcript of Cost effective scalinCost effective scalinng for the …...Infobright Evolution MS e, QL, QL NoSQL...

Page 1: Cost effective scalinCost effective scalinng for the …...Infobright Evolution MS e, QL, QL NoSQL MongoDB, Couch, HBase MongoDB asy to start Open source Drivers in every major language

Cost effective scalinCost effective scalinng for the enterpriseng for the enterprise

Page 2: Cost effective scalinCost effective scalinng for the …...Infobright Evolution MS e, QL, QL NoSQL MongoDB, Couch, HBase MongoDB asy to start Open source Drivers in every major language

ou just eleased youeleased youew productp

urur

Page 3: Cost effective scalinCost effective scalinng for the …...Infobright Evolution MS e, QL, QL NoSQL MongoDB, Couch, HBase MongoDB asy to start Open source Drivers in every major language

But there arsigsig

nfrastructure costs rows faster than evenue

UsersUsers

re troubling gnsgns

• Development timeline slows dow

Page 4: Cost effective scalinCost effective scalinng for the …...Infobright Evolution MS e, QL, QL NoSQL MongoDB, Couch, HBase MongoDB asy to start Open source Drivers in every major language

osts go up

Page 5: Cost effective scalinCost effective scalinng for the …...Infobright Evolution MS e, QL, QL NoSQL MongoDB, Couch, HBase MongoDB asy to start Open source Drivers in every major language

roductivity goes down

Page 6: Cost effective scalinCost effective scalinng for the …...Infobright Evolution MS e, QL, QL NoSQL MongoDB, Couch, HBase MongoDB asy to start Open source Drivers in every major language

SummWe use relational databa

ur developers productiv

ut relational databases

Our hardware costs go uOur hardware costs go uevenue growth is linear

nd we start taking awamade our developers pr

maryases because they mave

s don’t scale up well

up exponentially whileup exponentially while r

y those features that oductive

Page 7: Cost effective scalinCost effective scalinng for the …...Infobright Evolution MS e, QL, QL NoSQL MongoDB, Couch, HBase MongoDB asy to start Open source Drivers in every major language

Database RDBMRDBM

OraclMySQMySQ

PostgreS

OLAPNetezza,

Aster, Hadoop, InfobrightInfobright

EvolutionMSMSe,

QL,QL, SQL

NoSQLMongoDB,

Couch, HBase

Page 8: Cost effective scalinCost effective scalinng for the …...Infobright Evolution MS e, QL, QL NoSQL MongoDB, Couch, HBase MongoDB asy to start Open source Drivers in every major language
Page 9: Cost effective scalinCost effective scalinng for the …...Infobright Evolution MS e, QL, QL NoSQL MongoDB, Couch, HBase MongoDB asy to start Open source Drivers in every major language

MongoDBasy to start Open sourceDrivers in every major language

asy to developSchemaless document modelFlexible query languageSecondary indexes

asy to scaleBuilt in sharding

Downloads per Month

80 000

100,000

120,000Downloads per Month

40,000

60,000

80,000

0

20,000

9-02

9-03

9-04

9-05

9-06

9-07

9-08

9-09

9-10

9-11

9-12

0-01

0-02

0-03

0-04

0-05

0-06

0-07

0-08

0-09

0-10

0-11

Page 10: Cost effective scalinCost effective scalinng for the …...Infobright Evolution MS e, QL, QL NoSQL MongoDB, Couch, HBase MongoDB asy to start Open source Drivers in every major language

Tables to DDocuments{ title: ‘MongoDB’title: MongoDB ,contributors: [ { name: ‘Eliot Horowitz’{ name: Eliot Horowitz ,email: ‘[email protected]

{ name: ‘Dwight Merrima{ gemail: ‘[email protected]

}]],model: {

relational: falserelational: false, awesome: true

}}}

Page 11: Cost effective scalinCost effective scalinng for the …...Infobright Evolution MS e, QL, QL NoSQL MongoDB, Couch, HBase MongoDB asy to start Open source Drivers in every major language

ATA AS DOCUMENTSS

Page 12: Cost effective scalinCost effective scalinng for the …...Infobright Evolution MS e, QL, QL NoSQL MongoDB, Couch, HBase MongoDB asy to start Open source Drivers in every major language

TerminDBMSble, View

ow(s)dexiin

rtitionrtition Key

nologyMongogCollectionJSON DocumentIndexE b dd dEmbedded DocumentDocumentShardShard Key

Page 13: Cost effective scalinCost effective scalinng for the …...Infobright Evolution MS e, QL, QL NoSQL MongoDB, Couch, HBase MongoDB asy to start Open source Drivers in every major language

Documog Post Document

= { author: “roger”= { author: roger ,date: new Date(),text: “Spirited Away”tags: [“Tezuka”, “Matags: [ Tezuka , Ma

b posts save(p)b.posts.save(p)

ments

”,anga”]}anga ]}

Page 14: Cost effective scalinCost effective scalinng for the …...Infobright Evolution MS e, QL, QL NoSQL MongoDB, Couch, HBase MongoDB asy to start Open source Drivers in every major language

Querb.posts.find()

{ id : ObjectId("4c4ba5{ _id : ObjectId( 4c4ba5author : "roger", date : "Sat Jul 24 2010

DT)",DT) , text : "Spirited Away", tags : [ "Tezuka" "Matags : [ "Tezuka", "Ma

otes:- id is unique but can

rying

5c0672c685e5e8aabf35c0672c685e5e8aabf3

0 19:47:11 GMT-0700

nga" ] }nga" ] }

n be anything you’d like

Page 15: Cost effective scalinCost effective scalinng for the …...Infobright Evolution MS e, QL, QL NoSQL MongoDB, Couch, HBase MongoDB asy to start Open source Drivers in every major language

Secondaryeate index on any Field

/ 1 means ascending/ 1 means ascending,

>db.posts.ensureIndex(

>db.posts.find({author: '

_id : jectId("4c4ba5c0672c6author : "roger"

y Indexesd in Document

-1 means descending-1 means descending

({author: 1})

'roger'})

685e5e8aabf3"),

Page 16: Cost effective scalinCost effective scalinng for the …...Infobright Evolution MS e, QL, QL NoSQL MongoDB, Couch, HBase MongoDB asy to start Open source Drivers in every major language

Query O• Conditiona

–$all, $exists, $mod, $ne$$

–$lt, $lte

// find posts> db posts find( {ta> db.posts.find( {ta

// f// find posts matchin> db.posts.find(

// count po

peratorsal Operators e, $in, $nin, $nor, $or, $siz$type$ ype, $gt, $gte

s with any tags ags: {$exists: true }} )ags: {$exists: true }} )

g a regular expression{author: /^rog*/i } )

sts by author

Page 17: Cost effective scalinCost effective scalinng for the …...Infobright Evolution MS e, QL, QL NoSQL MongoDB, Couch, HBase MongoDB asy to start Open source Drivers in every major language

Atomic Oset, $unset, $inc, $push, $

omment = { author: “fred”{date: new Dattext: “Best Motext: Best Mo

db posts update( { id: “db.posts.update( { _id: ...{$push: {

perations $pushAll, $pull, $pullAll, $

, ,te(),

ovie Ever”}ovie Ever }

” } }, {comments: comment}} );

Page 18: Cost effective scalinCost effective scalinng for the …...Infobright Evolution MS e, QL, QL NoSQL MongoDB, Couch, HBase MongoDB asy to start Open source Drivers in every major language

Nested Do_id : ObjectId("4c4ba5cauthor : "roger",date : "Sat Jul 24 2010date : Sat Jul 24 2010 DT)", text : "Spirited Away",tags : [ "Tezuka", "Mantags : [ Tezuka , Mancomments : [

{{author : "Fred",date : "Sat Jul 24

DT)"

ocumentsc0672c685e5e8aabf3")

19:47:11 GMT-070019:47:11 GMT-0700

ga" ],ga ],

2010 20:51:03 GMT-0

Page 19: Cost effective scalinCost effective scalinng for the …...Infobright Evolution MS e, QL, QL NoSQL MongoDB, Couch, HBase MongoDB asy to start Open source Drivers in every major language

g> map = function() {

this tags forEathis.tags.forEaemit( tag,

});};};

> reduce = function( keyl 0var total = 0;

values.forEactotal += v

});});return { count

};> out = db posts mapRe

Map Reduach( function(tag) {ach( function(tag) { {count:1} )

y, values ) {

ch( function(value) { ( ( ) {alue.count;

t: total };

educe(m r);

Page 20: Cost effective scalinCost effective scalinng for the …...Infobright Evolution MS e, QL, QL NoSQL MongoDB, Couch, HBase MongoDB asy to start Open source Drivers in every major language

Map R> db[out.r

{ _id: ‘Tezuka’, v{ id: ‘Manga’ v{ _id: Manga , v

Reduceresult].find() value : { count: 3 } }value : { count: 4 } }value : { count: 4 } }

Page 21: Cost effective scalinCost effective scalinng for the …...Infobright Evolution MS e, QL, QL NoSQL MongoDB, Couch, HBase MongoDB asy to start Open source Drivers in every major language

Inde/ Index nested documents

> db.posts.ensureIndex( “cdb.posts.find({‘commentsp ({

/ Index on tags/ Index on tags> db.posts.ensureIndex( ta> db posts find( { tags: ’Ma> db.posts.find( { tags: Ma

// geospatial index> db.posts.ensureIndex( “a> db.posts.find( “author.loc

exesscomments.author”:1 )s.author’:’Fred’})})

ags: 1)anga’ } )anga } )

author.location”: “2d” )cation” : { $near : [22,42]

Page 22: Cost effective scalinCost effective scalinng for the …...Infobright Evolution MS e, QL, QL NoSQL MongoDB, Couch, HBase MongoDB asy to start Open source Drivers in every major language

EPLOYMENT & SCALLING

Page 23: Cost effective scalinCost effective scalinng for the …...Infobright Evolution MS e, QL, QL NoSQL MongoDB, Couch, HBase MongoDB asy to start Open source Drivers in every major language

Replica

PrimaWrite

PrimaRead

SeconReadryry

Read

Seconry

Readryry

a Sets

aryary

ndaAsynchronous

Replication

nda

Page 24: Cost effective scalinCost effective scalinng for the …...Infobright Evolution MS e, QL, QL NoSQL MongoDB, Couch, HBase MongoDB asy to start Open source Drivers in every major language

Replica

PrimaWrite

PrimaRead

SeconReadryry

Read

Seconry

Readryry

a Sets

aryary

nda

nda

Page 25: Cost effective scalinCost effective scalinng for the …...Infobright Evolution MS e, QL, QL NoSQL MongoDB, Couch, HBase MongoDB asy to start Open source Drivers in every major language

Replica

PrimaPrima

PrimaWrite

Read

Seconry

Readryry

a Sets

aryary

aryAutomatic

Leader Electio

nda

Page 26: Cost effective scalinCost effective scalinng for the …...Infobright Evolution MS e, QL, QL NoSQL MongoDB, Couch, HBase MongoDB asy to start Open source Drivers in every major language

Replica

Seconry

Seconry

Read

PrimaWrite

Read

Seconry

Readryry

a Sets

ndanda

ary

nda

Page 27: Cost effective scalinCost effective scalinng for the …...Infobright Evolution MS e, QL, QL NoSQL MongoDB, Couch, HBase MongoDB asy to start Open source Drivers in every major language

Shar

K R K RKey Range0..10

Key Range31..60, 91..95

Primary Primary

Secondary

Secondaryyy

Secondar

yy

Secondaryy yy

rding

K R K RKey Range20..30

Key Range95..100

Primary Primary

Secondary

Secondaryyy

Secondar

yy

Secondaryy yy

Page 28: Cost effective scalinCost effective scalinng for the …...Infobright Evolution MS e, QL, QL NoSQL MongoDB, Couch, HBase MongoDB asy to start Open source Drivers in every major language

WriteWrite

MongoS MongoS

Key Range0..30

Key Range31..60

Primary Primary

0..30 31..60

Primary

S d

Primary

S dy

Secondary y

Secondary

ReadRead

MongoS MongoS

Key Range61..90

Key Range91.. 100

Primary Primary

61..90 91.. 100

Primary

S d

Primary

S dy

Secondary y

Secondary

Page 29: Cost effective scalinCost effective scalinng for the …...Infobright Evolution MS e, QL, QL NoSQL MongoDB, Couch, HBase MongoDB asy to start Open source Drivers in every major language

Photblem:blem:siness needed more flexibility

tion:ed MongoDBMongoDB instead of Orac

ultsults::veloped application in one spveloped application in one sp0% cost reduction compared 0% performance improvemen0% performance improvemen

to Meta-Da

y than Oracle could deliver

cle

print cycleprint cycleto Oracle

nt compared to Oraclent compared to Oracle

Page 30: Cost effective scalinCost effective scalinng for the …...Infobright Evolution MS e, QL, QL NoSQL MongoDB, Couch, HBase MongoDB asy to start Open source Drivers in every major language

CAA

blem:blem:al with massive data volume

tion:ed MongoDBMongoDB to replace Goo

ultsults::ss than1week to build prototyss than1week to build prototypid deployment of new featur

Customer AnalyticsAnalytics

across all customer sites

ogle Analytics / Omniture opti

ype and prove business caseype and prove business caseres (1/day, 1/week)

Page 31: Cost effective scalinCost effective scalinng for the …...Infobright Evolution MS e, QL, QL NoSQL MongoDB, Couch, HBase MongoDB asy to start Open source Drivers in every major language

Onlinblem:blem:SQL could not scale to hand

tion:itched from MySQL to MongMong

ultsults::ssive simplification of code bssive simplification of code bminated need for external cacx performance improvement ox performance improvement o

ne Dictiona

le their 5B+ documents

oDBoDB

basebase ching systemover MySQLover MySQL

Page 32: Cost effective scalinCost effective scalinng for the …...Infobright Evolution MS e, QL, QL NoSQL MongoDB, Couch, HBase MongoDB asy to start Open source Drivers in every major language

SummC

ost

Data

maryRDBMSRDBMS

MongoDB

a

Page 33: Cost effective scalinCost effective scalinng for the …...Infobright Evolution MS e, QL, QL NoSQL MongoDB, Couch, HBase MongoDB asy to start Open source Drivers in every major language

•Easy to s•Easy to d•Easy to s

startdevelopscale