Riak at Engine Yard Cloud
51
Riak in the Cloud Getting Started with Big Nerd Ranch
-
Upload
ines-sombra -
Category
Technology
-
view
2.204 -
download
0
description
Given at Big Nerd Ranch in Atlanta.
Transcript of Riak at Engine Yard Cloud
Riak in the Cloud
Getting Started with
Big Nerd Ranch
Ines Sombra
@RandomMood
About Me
About Us
Our Agenda
The Problem
The Solution
Our Customers
Engine Yard
About US100% Cloud Based
Critical services MUST be up 24/7/365
CloudsRegions
StacksSupport
HA
DR
http://www.flickr.com/photos/meetrajesh/6972883401
Our Customers
An APPLICATION
App
App
App
DB Master
DB Replica
us-east-1a
us-east-1b
http://www.flickr.com/photos/brandongrasley/8227882239/
Then AWS Crashed
Our App AFTER
App
App
App
DB Master
DB Replica
us-east-1a
us-east-1b
X
!
This sucked, A LOTTime to strengthen our HA/DR Story
Customers and us need to be up all the time!
$
Relational
Document
Key/Value
Column
Data Models Consistency Partitions
Our Research
Are ACID properties really needed?
Consistency
Availability
Relational Model
Impedance Mismatch
Memory data-structure
Impedance Mismatch
We needed a fault tolerant and highly available (clustered)
datastore
Linearly scalable
Fault-tolerant
Finally provide HA/DR solution
OSS, Dynamo-based, distributed K/V store Hello RIAK!
Riak’s Use CasesApplication Type Key Value
Session User/Session id Data
Advertising Campaign id Data
Logs Date Log file
Content title, integer text, json, xml, media..
When is Riak a good fit
When availability is more important than consistency
When data can be modeled as keys/values
When the problem fits
Riak is OPS FRIENDLY
node1.2.1
node1.2.1
node1.2.1
node1.2.1
node1.2.1
Growth
Reduction
Upgrades
Failuresnode1.3.0
X
World’s most famous slide
Riak is SO MUCH MORE
Consistent HashingHinted Handoff
Active Anti EntropyTuneable consistencyN | W | R
OUR APP AFTER
App
App
node1.3.0
node1.3.0
node1.3.0
node1.3.0
node1.3.0
App
We need a Riak Cluster but we have
no hardware
RIAK In the Wild
18th and Harrison, San Francisco, CA
Let’s build A PRODUCT
http://www.flickr.com/photos/worldbank/8261699153/
1
2
3
4
Number of Nodes
Your CHOICESYou are trying
to create a Riak cluster. Do you need any help?
Instance TypeData LocationBackend Type
What we GIVE YOU
Our SUPPORT
X2
What you OPS to know
HOWTO
Prepare Yourself
More instances to compensate for the
performance variability of virtualized resources
^ m1.large & m1.xlarge
EBS-optimized or PIOPs
High I/O Quadruple XL (hi1.4xlarge) - SSDs!
Choosing INSTANCES
5 nodes or moreBest performance & growth
Scales linearly
Don’t be stingy
Cluster Sizes
Change DefaultCANNOT BE RESET
(8-64 vnodes per node)
Ring SIZE
Ring Size at 256
DONT GO OVER 512
Our PROVISIONING
App
haproxy
MyRiakCluster: node-0
MyRiakCluster: node-1
MyRiakCluster: node-2
MyRiakCluster: node-3
MyRiakCluster: node-4
Riak Cluster
HTTPor
ProtobuffApp
haproxy
App
haproxy
Riak Cluster
^ Riak process pages should not hit swap
Swap usage can result in unresponsive server
Let the kernel kill it
Disable SWAP
^ Mount with noatime flag
Use deadline scheduler on EBS
Mount & SCHEDULER
cat /sys/block/xvdj1/queue/schedulernoop [deadline] cfq
echo deadline > /sys/block/xvdj1/queue/scheduler
^ xfs, ext3
ext4 barrier = 0 data = writeback
ZFS not recommended
Linux FILESYSTEMS
^ Backup both the ring and data directories
scp or rsync data dir
EBS snapshots
Backup STRATEGY
Monitoring STRATEGY
Stats Interface
riak-admin status
Collectd
Nagios
^ Vertical
Scaling RIAK
riak-admin cluster replace <old> <new>
^ Horizontalriak-admin cluster add <new>
How we SCALE RIAK
+ PostgreSQLRIAK
Hybrid Solutions FTW
+ FTS Engine*RIAK
+ RedisRIAK
Try OUR RIAK
How difficult IS IT?
A few thoughts
Finally!
Pick the right tool for the job
Apps & Services can have multiple databases
Polyglot PERSISTENCE
On SOAMy super nice e-commerce site
Shopping Cart & Session Data
Completed Orders
Inventory and Item Pricing
Recommendations Engine
Session Storage serviceK/V Store
Order Persistence
serviceDocument
Store
Inventory & Price service
RDBMS
Nodes and relations service
Graph Store
NoSQL Distilled: Fowler & Sadalage
Don’t optimize too quickly
Take time to understand your problem
Complexity increases with more databases
We RECOMMEND
Do KEEP IN MIND
Relational Document Store!=
Different paradigms are different
Give me MOAR!
Whatever
Little Riak Book (https://github.com/coderoshi/little_riak_book/)
docs.basho.comblog.engineyard.comnosql.mypopescu.com
Great RESOURCES
Questions?
