Rusty Klophaus (@rklophaus)Basho Technologies

Riak from Small to Large

Strange Loop · October 2010

Riak is a Dynamo-inspired,open-sourced, key/value datastorebuilt to scale predictably and easily.

2

Riak is a Dynamo-inspired,open-sourced, key/value datastorebuilt to scale predictably and easily.

3

Your ops guy: calm, relaxed, and wearing a party hat.

NoSQL

4

5http://www.flickr.com/photos/ghewgill/129950124

NoSQL has changed like Keanu hasn’t.

6http://www.youtube.com/watch?v=nEubt6HpGhs

Companies, Funding, Customers, Awareness...

7http://www.flickr.com/photos/free-stock/4791385567

“NoSQL? Yeah, it’s this really cool technology, you probably haven’t heard of it yet.”

8http://www.flickr.com/photos/slava/285199203

“Johnson! Get me a memo on this ‘No sequal’ thing!”

9http://www.flickr.com/photos/proimos/4045973322

Why?

10

Because NoSQL is driven by narcissism, voyeurism, and

materialism.

The catalyst for NoSQL is the data

generated or consumedby humans.

12

13

Photos, music, and video...

14

Constantly connected devices...

15http://www.flickr.com/photos/hanspoldoja/5001818922

Social Networks...

16http://www.flickr.com/photos/mega/7358278

Your Grandmama...

Requirements That Confound an RDBMS

17

Large Data ObjectsBLOBs?

High-Availability SystemsBi-Directional Replication? Log shipping?

High-Capacity SystemsShards?

Vertical Partitioning?

Changing SchemasAuto-generated DDL?

Generic Columns?

Non-Standard AccessRecursive SQL?

The roots of NoSQL are practical, not academic.

18

Scratch an Itch Build a Company

Turning Point

19

➮

Four Important Things

20

1. Broad KnowledgeLearn from the past, both DB *and* non-DB worlds.

• VP Technical Services, Akamai • Senior Architect, Akamai• Chief of Operations, Snapfish• Release Manager, Engine Yard• Bob Ippolito (CTO/CoFounder MochiMedia)• Dr. Eric Brewer (CAP Theorum)

Four Important Things

21

2. Strong CommunityStay in touch with real problems.

Mark PhillipsBasho Community Manager

Four Important Things

22

3. AgilityEvolve to solve new problems.

Four Important Things

23

4. RestraintNo solution can do it all well.

Back to Riak

24

What characteristics of Riak become important at different cluster sizes?

25

Single Box Riak

26

Simplicity

Single Box RiakSimple Key/Value Store, Flexible Schema

27

client = RiakClient(host, port)bucket = client.bucket('mybucket')

obj = bucket.new('foo', 'bar')obj.store()

obj2 = bucket.get(‘foo’)

Clients in Many Languages

Single Box Riak

28

Predictable Development Interface

Single Box Riak

29

Riak

Riak Riak

RiakRiak

Riak

===

Development Production

Configurable Buckets

Single Box Riak

30

!=

Profile

Bucket

Audio

Bucket

Links (Lightweight Data Relations)

Single Box Riak

31

Profile Projects Comments

Small Riak Cluster

32

Parallelism and Power

Small Riak ClusterExpanding the Cluster

33

bin/riak-admin join riak@hostname

Small Riak ClusterExpanding the Cluster

34

bin/riak-admin join riak@hostname

Small Riak ClusterExpanding the Cluster

34

bin/riak-admin join riak@hostname

Hello ladies, can your datastore grow like mine?

*monocle smile*

Small Riak ClusterDistributed Queries

35

Small Riak ClusterDistributed Queries

36

Small Riak ClusterJavascript-Based Map/Reduce

37

Keys

Map

Map

Map

Map

Map

Reduce

Reduce

Reduce

...

Output

Small Riak ClusterWell-Behaved HTTP

38

Riak Riak

RiakRiak

Riak

nginx

Squid

ClientClient

Client

Browser

nginx proxy config - http://gist.github.com/323048

Small Riak ClusterRiak Full-Text Search

39nginx proxy config - http://gist.github.com/323048

bin/search-cmd install mybucket

Riak Riak

RiakRiak

Riak

"converse AND category:shoes"

Large Riak Cluster

40

Operations

No Special Nodes

Large Riak Cluster

41

SpecialNodes

Laugh at Machine Failure

Large Riak Cluster

42

Riak Riak

RiakRiak

RiakX

Scale by Adding Nodes

Large Riak Cluster

43

Riak Riak

RiakRiak

Riak

Riak

Riak

Riak

Riak

Riak

Riak

Riak

Packages or Self-Contained Installation

Large Riak Cluster

44

sudo dpkg -i riak_0.13.0-2_amd64.deb

-or-

unzip riak.zipriak/bin/riak start

$0.00

$0.00Order now, and we’ll include an

extra Riak for free!*

*Offer limited to open-source editions of Riak.

• On-Call Support 24x7x365 • Management Tools• SNMP Monitoring• Multi-site Replication

Enterprise ($$$ / ~10’s of boxes)

47

• On-Call Support 24x7x365• Management Tools • SNMP Monitoring• Multi-site Replication

Enterprise ($$$ / ~10’s of boxes)

48

Thanks! Questions?Next Steps

Read more at http://wiki.basho.com

Download binaries from http://downloads.basho.com

Get source code from http://hg.basho.com

Join the public mailing list at riak-users@lists.basho.com

Mailing list archives at http://riak.markmail.org

Thanks!

Rusty Klophaus (@rklophaus)

Mark Phillips (@pharkmillups)

49

Additional Slides

TutorialGet Riak

Connect with a Client

Store Data

Store an Object with Links

Linkwalking

Map/Reduce

51

Get RiakDownloadhttp://downloads.basho.com/riak

Start Riakcd riakbin/riak start

52

Connect with Python# Code is at http://hg.basho.comimport riak

# Connectclient = riak.RiakClient('127.0.0.1', 8098)

53

Store Datamybucket = client.bucket('mybucket')

# Create an object...obj = mybucket.new('myobject')obj.set_data({ 'foo' : 1, 'bar' : 2 })obj.store()

# Read the object...obj = mybucket.get('myobject')print obj.get_data()

# Or, open a web browser...http://127.0.0.1:8098/riak/mybucket/myobject

54

Store an Object with Linksbands = client.bucket('bands')albums = client.bucket('albums')members = client.bucket('members')

# Store a band, link to album and members...obj = bands.new('Winger') \ .add_link(albums.new('Pull', 1275922).store()) \ .add_link(albums.new('IV', 542731).store()) \ .add_link(albums.new('Karma', 200170).store()) \ .add_link(members.new('Kip Winger').store()) \ .add_link(members.new('Reb Beach').store()) \ .add_link(members.new('John Roth').store()) \ .add_link(members.new('Rod M.').store()) \ .store()

55

Linkwalking# Get the albums...albums = obj.link('albums').run()

# Get the songs (assumes data is present)...songs = obj.link('albums').link('songs').run()

# Get the members...members = riak.MapReduce(client) \ .add('bands', 'Winger') \ .link('members') \ .run()

56

Map/Reduce# Count the number of sales...result = obj \ .link('albums') \ .map("function(v) { return [v.values[0].data]; }") \ .reduce("Riak.reduceSum") \ .run()

57