Introducing Riak and RippleSean Cribbs
Wednesday, April 14, 2010
system “whoami”
Wednesday, April 14, 2010
system “whoami”
BCS, three years as a musician
Wednesday, April 14, 2010
system “whoami”
BCS, three years as a musician
Started with Ruby in early 2006
Wednesday, April 14, 2010
system “whoami”
BCS, three years as a musician
Started with Ruby in early 2006
Rails freelancer 2007-2010
Wednesday, April 14, 2010
system “whoami”
BCS, three years as a musician
Started with Ruby in early 2006
Rails freelancer 2007-2010
Radiant CMS
Wednesday, April 14, 2010
system “whoami”
BCS, three years as a musician
Started with Ruby in early 2006
Rails freelancer 2007-2010
Radiant CMS
March 2010 - Developer Advocate
basho
Wednesday, April 14, 2010
Wednesday, April 14, 2010
Wednesday, April 14, 2010
Rails Can’t Scale
Wednesday, April 14, 2010
Rails Can’t Scalewhat does it mean?
Wednesday, April 14, 2010
Rails Can’t Scalewhat does it mean?
Ruby
Wednesday, April 14, 2010
Rails Can’t Scalewhat does it mean?
MySQLRuby
Wednesday, April 14, 2010
Rails Can’t Scalewhat does it mean?
MySQL
insert straw man here
Ruby
Wednesday, April 14, 2010
Mythbusting “scalability”
Wednesday, April 14, 2010
Mythbusting “scalability”
Scalability is not a yes/no question
Wednesday, April 14, 2010
Mythbusting “scalability”
Scalability is not a yes/no question
It’s a ratio of benefit to cost
Wednesday, April 14, 2010
Mythbusting “scalability”
Scalability is not a yes/no question
It’s a ratio of benefit to cost
Benefits: low latency, throughput, uptime, concurrency, reliability
Wednesday, April 14, 2010
Mythbusting “scalability”
Scalability is not a yes/no question
It’s a ratio of benefit to cost
Benefits: low latency, throughput, uptime, concurrency, reliability
Costs: CPU, RAM, disk, bandwidth, power, hw/sw
Wednesday, April 14, 2010
Mythbusting “scalability”
Scalability is not a yes/no question
It’s a ratio of benefit to cost
Benefits: low latency, throughput, uptime, concurrency, reliability
Costs: CPU, RAM, disk, bandwidth, power, hw/sw
scalability = bang for your buck (ROI)
Wednesday, April 14, 2010
Scaling goes both ways
Wednesday, April 14, 2010
Scaling goes both ways
Up - high traffic, “Big Data”, multi-datacenter
Wednesday, April 14, 2010
Scaling goes both ways
Up - high traffic, “Big Data”, multi-datacenter
Down - laptop, set-top, mobile
Wednesday, April 14, 2010
How do you scale?
Wednesday, April 14, 2010
How do you scale?
Vertically - get bigger, expensive machines
Wednesday, April 14, 2010
How do you scale?
Vertically - get bigger, expensive machines
Horizontally - get more commodity machines
Wednesday, April 14, 2010
What is Riak?(the horizontal scaling bits)
Wednesday, April 14, 2010
What is Riak?
Based on Amazon’s Dynamo (2007)
(the horizontal scaling bits)
Wednesday, April 14, 2010
What is Riak?
Based on Amazon’s Dynamo (2007)
key-value storage
(the horizontal scaling bits)
Wednesday, April 14, 2010
What is Riak?
Based on Amazon’s Dynamo (2007)
key-value storage
masterless, peer-to-peer replication
(the horizontal scaling bits)
Wednesday, April 14, 2010
What is Riak?
Based on Amazon’s Dynamo (2007)
key-value storage
masterless, peer-to-peer replication
consistent hashing
(the horizontal scaling bits)
Wednesday, April 14, 2010
What is Riak?
Based on Amazon’s Dynamo (2007)
key-value storage
masterless, peer-to-peer replication
consistent hashing
eventual consistency
(the horizontal scaling bits)
Wednesday, April 14, 2010
What is Riak?
Based on Amazon’s Dynamo (2007)
key-value storage
masterless, peer-to-peer replication
consistent hashing
eventual consistency
failover - quorums, hinted handoff
(the horizontal scaling bits)
Wednesday, April 14, 2010
“O/RM is the Vietnam of Computer Science.”-- Ted Neward, 2004/6
Wednesday, April 14, 2010
“O/RM is the Vietnam of Computer Science.”-- Ted Neward, 2004/6
slippery slope
Wednesday, April 14, 2010
“O/RM is the Vietnam of Computer Science.”-- Ted Neward, 2004/6
slippery slope diminishing returns
Wednesday, April 14, 2010
Impedance mismatch
Wednesday, April 14, 2010
Impedance mismatch
Relational = predicate logic, truth statements
Wednesday, April 14, 2010
Impedance mismatch
Relational = predicate logic, truth statements
Object = identity, state, behavior, encapsulation
Wednesday, April 14, 2010
Pick your poison
Wednesday, April 14, 2010
Pick your poison
Make the objects fit the database
Wednesday, April 14, 2010
Pick your poison
Make the objects fit the database
Active Record
Wednesday, April 14, 2010
Pick your poison
Make the objects fit the database
Active Record
Make the database fit the objects
Wednesday, April 14, 2010
Pick your poison
Make the objects fit the database
Active Record
Make the database fit the objects
OODBMS, Graph DBs
Wednesday, April 14, 2010
Middle ground
Wednesday, April 14, 2010
Middle ground
Document databases (identity and state)
Wednesday, April 14, 2010
Middle ground
Document databases (identity and state)
No JOINs - denormalized / composed
Wednesday, April 14, 2010
Middle ground
Document databases (identity and state)
No JOINs - denormalized / composed
Loosely structured
Wednesday, April 14, 2010
Middle ground
Document databases (identity and state)
No JOINs - denormalized / composed
Loosely structured
Friendly transport formats, e.g. JSON
Wednesday, April 14, 2010
What is Riak?(the document database bits)
Wednesday, April 14, 2010
What is Riak?
Store your objects as JSON (or any format)
(the document database bits)
Wednesday, April 14, 2010
What is Riak?
Store your objects as JSON (or any format)
Link between objects (like hypertext)
(the document database bits)
Wednesday, April 14, 2010
What is Riak?
Store your objects as JSON (or any format)
Link between objects (like hypertext)
No SQL - query with Javascript Map-Reduce
(the document database bits)
Wednesday, April 14, 2010
“DevOps” - agile infrastructure
Wednesday, April 14, 2010
“DevOps” - agile infrastructureprogramming your infrastructure
Wednesday, April 14, 2010
“DevOps” - agile infrastructureprogramming your infrastructure
rapid “cloud” deployments
Wednesday, April 14, 2010
Befriend your sysadmin
Wednesday, April 14, 2010
Befriend your sysadmin
Integrate with existing infrastructure
Wednesday, April 14, 2010
Befriend your sysadmin
Integrate with existing infrastructure
Reduce the amount of hands-on work
Wednesday, April 14, 2010
Befriend your sysadmin
Integrate with existing infrastructure
Reduce the amount of hands-on work
Have a growth plan
Wednesday, April 14, 2010
What is Riak?(the ops-friendly bits)
Wednesday, April 14, 2010
What is Riak?
Web-shaped storage
(the ops-friendly bits)
Wednesday, April 14, 2010
What is Riak?
Web-shaped storage
Store data in its original format
(the ops-friendly bits)
Wednesday, April 14, 2010
What is Riak?
Web-shaped storage
Store data in its original format
It’s just HTTP - same techniques apply
(the ops-friendly bits)
Wednesday, April 14, 2010
What is Riak?
Web-shaped storage
Store data in its original format
It’s just HTTP - same techniques apply
load balancing, proxy caches, round-robin DNS
(the ops-friendly bits)
Wednesday, April 14, 2010
What is Riak?
Web-shaped storage
Store data in its original format
It’s just HTTP - same techniques apply
load balancing, proxy caches, round-robin DNS
No node is special - grow horizontally
(the ops-friendly bits)
Wednesday, April 14, 2010
What is Riak?
Web-shaped storage
Store data in its original format
It’s just HTTP - same techniques apply
load balancing, proxy caches, round-robin DNS
No node is special - grow horizontally
Get some sleep, fix it in the morning
(the ops-friendly bits)
Wednesday, April 14, 2010
To review, Riak is...
Wednesday, April 14, 2010
To review, Riak is...
Magical Horizontally-Scaling Unicorns
Wednesday, April 14, 2010
To review, Riak is...
Magical Horizontally-Scaling Unicorns
Application-Friendly Rainbows in the Cloud
Wednesday, April 14, 2010
To review, Riak is...
Magical Horizontally-Scaling Unicorns
Application-Friendly Rainbows in the Cloud
Data-Shredding MapReduce Ninjas
Wednesday, April 14, 2010
http://downloads.basho.com/riak/hg clone http://hg.basho.com/riak/
Wednesday, April 14, 2010
gem install ripplegit clone git://github.com/seancribbs/ripple.git
Wednesday, April 14, 2010
CRUD in Riak
Wednesday, April 14, 2010
CRUD in Riak
PUT /riak/bucket/key (C,U) (also POST)
Wednesday, April 14, 2010
CRUD in Riak
PUT /riak/bucket/key (C,U) (also POST)
GET /riak/bucket/key (R)
Wednesday, April 14, 2010
CRUD in Riak
PUT /riak/bucket/key (C,U) (also POST)
GET /riak/bucket/key (R)
DELETE /riak/bucket/key (D)
Wednesday, April 14, 2010
CRUD Demo
Wednesday, April 14, 2010
Links
Wednesday, April 14, 2010
Links
One-way, qualified “pointers” to other objects
Wednesday, April 14, 2010
Links
One-way, qualified “pointers” to other objects
“Link” header in HTTP
Wednesday, April 14, 2010
Links
One-way, qualified “pointers” to other objects
“Link” header in HTTP
</riak/bucket/key>; riaktag=”tag”
Wednesday, April 14, 2010
Link-walking
Wednesday, April 14, 2010
Link-walking
GET /riak/bucket/key/[bucket,tag,keep]*
Wednesday, April 14, 2010
Link-walking
GET /riak/bucket/key/[bucket,tag,keep]*
bucket = scope to a bucket
Wednesday, April 14, 2010
Link-walking
GET /riak/bucket/key/[bucket,tag,keep]*
bucket = scope to a bucket
tag = scope to a tag
Wednesday, April 14, 2010
Link-walking
GET /riak/bucket/key/[bucket,tag,keep]*
bucket = scope to a bucket
tag = scope to a tag
keep = return results (last one always)
Wednesday, April 14, 2010
Link-walking
GET /riak/bucket/key/[bucket,tag,keep]*
bucket = scope to a bucket
tag = scope to a tag
keep = return results (last one always)
“_” means match anything
Wednesday, April 14, 2010
Link-walking
GET /riak/bucket/key/[bucket,tag,keep]*
bucket = scope to a bucket
tag = scope to a tag
keep = return results (last one always)
“_” means match anything
Example:
Wednesday, April 14, 2010
Link-walking
GET /riak/bucket/key/[bucket,tag,keep]*
bucket = scope to a bucket
tag = scope to a tag
keep = return results (last one always)
“_” means match anything
Example:
GET /riak/artists/TheBeatles/albums,_,1/tracks,_,1
Wednesday, April 14, 2010
Links Demo
Wednesday, April 14, 2010
Riak Map-ReduceI believe I did, Bob.
Wednesday, April 14, 2010
Riak Map-Reduce
Wednesday, April 14, 2010
Riak Map-Reduce
Based on Google’s Map-Reduce idea
Wednesday, April 14, 2010
Riak Map-Reduce
Based on Google’s Map-Reduce idea
Some similarities to Hadoop and CouchDB
Wednesday, April 14, 2010
Riak Map-Reduce
Based on Google’s Map-Reduce idea
Some similarities to Hadoop and CouchDB
High data-locality
Wednesday, April 14, 2010
Riak Map-Reduce
Based on Google’s Map-Reduce idea
Some similarities to Hadoop and CouchDB
High data-locality
Phases can be Javascript or Erlang
Wednesday, April 14, 2010
Map-Reduce Terminology
Wednesday, April 14, 2010
Map-Reduce Terminology
Job: a query, composed of inputs and phases
Wednesday, April 14, 2010
Map-Reduce Terminology
Job: a query, composed of inputs and phases
Phase: a single map or reduce function application
Wednesday, April 14, 2010
Map-Reduce Terminology
Job: a query, composed of inputs and phases
Phase: a single map or reduce function application
Map: phase applied to each item - filter, transform
Wednesday, April 14, 2010
Map-Reduce Terminology
Job: a query, composed of inputs and phases
Phase: a single map or reduce function application
Map: phase applied to each item - filter, transform
Reduce: phase applied to the collection - collate, aggregate, compute
Wednesday, April 14, 2010
Map Phases
Wednesday, April 14, 2010
Map Phases
Inputs are bucket-key pairs (with optional key-specific data)
Wednesday, April 14, 2010
Map Phases
Inputs are bucket-key pairs (with optional key-specific data)
Must return a list
Wednesday, April 14, 2010
Map Phases
Inputs are bucket-key pairs (with optional key-specific data)
Must return a list
Parallel results are aggregated
Wednesday, April 14, 2010
Map Built-ins
Wednesday, April 14, 2010
Map Built-ins
Riak.mapValues
Wednesday, April 14, 2010
Map Built-ins
Riak.mapValues
Riak.mapValuesJson
Wednesday, April 14, 2010
Reduce Phases
Wednesday, April 14, 2010
Reduce Phases
Performed on a single node
Wednesday, April 14, 2010
Reduce Phases
Performed on a single node
Two processes per reduce phase increase parallelism
Wednesday, April 14, 2010
Reduce Phases
Performed on a single node
Two processes per reduce phase increase parallelism
Must return a list
Wednesday, April 14, 2010
Reduce Built-ins
Wednesday, April 14, 2010
Reduce Built-ins
Riak.reduceMin
Wednesday, April 14, 2010
Reduce Built-ins
Riak.reduceMin
Riak.reduceMax
Wednesday, April 14, 2010
Reduce Built-ins
Riak.reduceMin
Riak.reduceMax
Riak.reduceSort
Wednesday, April 14, 2010
Build a M/R Job
Wednesday, April 14, 2010
Build a M/R Job
Specify inputs
Wednesday, April 14, 2010
Build a M/R Job
Specify inputs
bucket/key, bucket/key/key-specific-arg, bucket
Wednesday, April 14, 2010
Build a M/R Job
Specify inputs
bucket/key, bucket/key/key-specific-arg, bucket
Specify phases
Wednesday, April 14, 2010
Build a M/R Job
Specify inputs
bucket/key, bucket/key/key-specific-arg, bucket
Specify phases
Each can receive a static argument
Wednesday, April 14, 2010
Build a M/R Job
Specify inputs
bucket/key, bucket/key/key-specific-arg, bucket
Specify phases
Each can receive a static argument
Each can return intermediate results
Wednesday, April 14, 2010
Map-Reduce on HTTP
Wednesday, April 14, 2010
Map-Reduce on HTTP
Job is a JSON object
Wednesday, April 14, 2010
Map-Reduce on HTTP
Job is a JSON object
POST /mapred
Wednesday, April 14, 2010
A simple M/R job
{“inputs”: “goog”,
“query”: [{“map”:
{“language”:”javascript”,
“name”: “Riak.mapValuesJson”,
“keep”: true}]}
Wednesday, April 14, 2010
A simple M/R job
{“inputs”: “goog”,
“query”: [{“map”:
{“language”:”javascript”,
“name”: “Riak.mapValuesJson”,
“keep”: true}]}
Riak::MapReduce.new(c).add(‘goog’).map(‘Riak.mapValuesJson’, :keep => true).run
Wednesday, April 14, 2010
Another M/R Job{“inputs”: “stocks”,
“query”: [{“map”:{“language”:”javascript”,
“name”: “App.extractTickers”,
“arg”: “GOOG”,
“keep”: false},
{“reduce”:{“language”:”javascript,
“name”: “Riak.reduceMin”,
“keep”: true}]}
Wednesday, April 14, 2010
Another M/R Job{“inputs”: “stocks”,
“query”: [{“map”:{“language”:”javascript”,
“name”: “App.extractTickers”,
“arg”: “GOOG”,
“keep”: false},
{“reduce”:{“language”:”javascript,
“name”: “Riak.reduceMin”,
“keep”: true}]}
Riak::MapReduce.new(c).add(‘stocks’).map(‘App.extractTickers’, :arg =>”GOOG”).reduce(“Riak.reduceMin”, :keep => true).run
Wednesday, April 14, 2010
Map-Reduce Demo
Wednesday, April 14, 2010
Ripple
Wednesday, April 14, 2010
Ripple
Document-style persistence (like MongoMapper)
Wednesday, April 14, 2010
Ripple
Document-style persistence (like MongoMapper)
ActiveModel niceties
Wednesday, April 14, 2010
Ripple
Document-style persistence (like MongoMapper)
ActiveModel niceties
Callbacks
Wednesday, April 14, 2010
Ripple
Document-style persistence (like MongoMapper)
ActiveModel niceties
Callbacks
Validations
Wednesday, April 14, 2010
Ripple
Document-style persistence (like MongoMapper)
ActiveModel niceties
Callbacks
Validations
ActionPack compatibility
Wednesday, April 14, 2010
Ripple TODO
Wednesday, April 14, 2010
Ripple TODO
Associations
Wednesday, April 14, 2010
Ripple TODO
Associations
Indexes
Wednesday, April 14, 2010
Ripple TODO
Associations
Indexes
Use a property/method as the key
Wednesday, April 14, 2010
Ripple TODO
Associations
Indexes
Use a property/method as the key
Dynamic finders
Wednesday, April 14, 2010
Ripple TODO
Associations
Indexes
Use a property/method as the key
Dynamic finders
Session/cache stores
Wednesday, April 14, 2010
Ripple Demo
Wednesday, April 14, 2010
Questions?
@seancribbs
“seancribbs” on Freenode IRC #riak
http://wiki.basho.com
Wednesday, April 14, 2010
Top Related