CouchDBrelax

CouchDBrelax

Sander van de Graaf@svdgraaf

http://joind.in/talk/view/3009

RELAX

CONTENTS

• Introduction

• PHP Usage

• Replication/Scalability

• Backend usage

• Couchapps

•Other stuff (GeoCouch!)

CouchDBrelax

NOSQL

IT’S A MOVEMENT

1998

Carlo Strozzi

NOSQL == Not Only SQL

“[The NoSQL movement] departs from the relational model altogether, it should therefore have been called more appropriately ‘NoREL’, or something to that effect.”

- Carlo Strozzi

CouchDBrelax

NUTSHELL

SPEED

APPEND ONLY

NO REPAIR NEEDED

COMPACTING

HTTP SERVER

CAP

CAP

CouchDB

EVENTUALLY CONSISTENT

CouchDB

FULL REST API

HTTP METHODS

• GET

• PUT

• POST

•DELETE

• COPY

• SELECT

• UPDATE

• INSERT

•DELETE

• ...

{ "total_rows": 2, "offset": 0, "rows": [ { "id": "abc089b91fadfcf67a438e73f426f71d", "key": [ "2011-05-07 13:52:15", "abc089b91fadfcf67a438e73f426f71d" ], "value": { "_id": "abc089b91fadfcf67a438e73f426f71d", "_rev": "2-a38e08996586cd774b0b5c3006659de0", "message": "A2 HUISARTSENPOST BARNEVELD NIJKERKERWEG 119 3771LA 119" } }, { "id": "abc089b91fadfcf67a438e73f4286fd5", "key": [ "2011-05-07 14:00:16", "abc089b91fadfcf67a438e73f4286fd5" ], "value": { "_id": "abc089b91fadfcf67a438e73f4286fd5", "_rev": "2-691929b5f3d6cde58f27d62681c7b599", "message": "A1 LEEUWERIKSWEIDE 183 WAGENINGEN 6708LH 183" } } ] }

REPLICATION

CONTENTS

• Introduction

• PHP Usage

• Replication/Scalability

• Backend usage

• Couchapps

•Other stuff

RELAX

PHP USAGE

PHP LIBRARIES

• PHPillow (LGPL)

• PHP Object Freezer (BSD)

• PHP On Couch (GPL 2 / 3)

• PHP CouchDB Extension (PHP license)

• SAG for CouchDB (apache)

•Doctrine 2 CouchDB ODM

<?PHP// setup connection for couchdb$client = new Couchdb_Client('http://ponies.couchone.com:5984','rainbows');

// fetch a document$doc = $client->getDoc('awesome_pony');

// updating document$doc->type = "awesome";

try { $client->storeDoc($doc);}catch (Exception $e) { echo "Document storage failed : " . $e->getMessage();}

PHP ON COUCH

CONTENTS

• Introduction

• PHP Usage

• Replication/Scalability

• Backend usage

• Couchapps

•Other stuff

REPLICATION

DEFINITION

“Replication is the process of sharing information so as to ensure consistency between redundant resources, such as software or hardware components, to improve reliability, fault-tolerance, or accessibility.”

Source: wikipedia

CouchDBrelax

CouchDBrelax

CouchDBrelax

CouchDBrelax

CouchDBrelax

CouchDBrelax

CouchDBrelax

CouchDBrelax

CouchDBrelax

CouchDBrelax

CouchDBrelax

CouchDBrelax

CouchDBrelax

CouchDBrelax

CouchDBrelax

US NL

BE

USE CASEIDG Publishers

IDG NL IDG DE

IDG US IDG AUSTRALIA

CouchDBrelax

CouchDBrelax

CouchDBrelax

CouchDBrelax

CouchDBrelax

CouchDBrelax

CouchDBrelax

IDG NL IDG US

IDG DE IDG Italy

IDG Brazil

IDG Romania

IDG Poland

IDG SpaniaIDG UK

IDG Ghana

IDG Hungary

IDG Sweden

IDG Portugal

IDG Japan IDG Thailand IDG Vietnam

IMPLICATIONEvery single bit of IDG content published

worldwide, is available anywhere, anytime in msec.

P2P WEB

“World Domination”

HOW DO I DO THAT?

CLUSTERING“The fun stuff ”

CouchDBrelax

CouchDBrelax

loadbalancer

...n

CHALLENGES

• Large amounts of data

• Large views (with big/long map/reduce queries)

• LOTS of traffic

• Location based partitions

• For fun and profit

MAP/REDUCE

INPUT

IP Bytes

212.122.174.13 18271

212.122.174.13 191726

212.122.174.13 198

74.119.8.111 91272

74.119.8.111 8371

212.122.174.13 43

MAPPER => REDUCER

IP Bytes

212.122.174.13

18271

212.122.174.13191726

212.122.174.13198

212.122.174.13

43

74.119.8.11191272

74.119.8.1118371

AFTER REDUCE

IP Bytes

212.122.174.13 210238

74.119.8.111 99643

PARTITION INPUT

Partition IP Bytes

0 212.122.174.13 18271

0 212.122.174.13 191726

0 212.122.174.13 198

1 74.119.8.111 91272

1 74.119.8.111 8371

0 212.122.174.13 43

MAPPER => REDUCER

Partition IP Bytes

0 212.122.174.13

18271

0 212.122.174.13191726

0 212.122.174.13198

0 212.122.174.13

43

1 74.119.8.11191272

1 74.119.8.1118371

AFTER REDUCE

IP Bytes

212.122.174.13 210238

74.119.8.111 99643

• CouchDB Lounge

• Pillow

• BigCouch

CLUSTERING OPTIONS

LOUNGE

•partitioning/clustering

•Nginx module

•meebo.com

• ‘easy’

•http://tilgovi.github.com/couchdb-lounge/

LOUNGE

• dumb_proxy => proxy for simple PUT/GET’s

• smart_proxy => proxy for map/reduce over shards

• replicator => updates all copies, redundantly

CouchDBrelax

CouchDBrelax

nginx

...n

dumb_proxy

CouchDBrelax

CouchDBrelax

nginx

...n

smart_proxy

Bonus:

other nginx modules work too

PILLOW

•Erlang based

• router/rereducer (map/reduce over multiple systems)

• In development (but promising!)

•https://github.com/khellan/Pillow

BIGCOUCH

•Fork

•100% api compatible

•Open Source/Commercial

•https://cloudant.com/#!/solutions/bigcouch

CONTENTS

• Introduction

• PHP Usage

• Replication/Scalability

• Backend usage

• Couchapps

•Other stuff

BACKEND USAGE

PROXIED

CouchDBrelax

DIRECT

CouchDBrelax

NOSQL && SQL HYBRID

• onSave, onCommit hooks available in every major framework

• onSave -> make a JSON representation of your object, and PUT it to couchdb (#protip: only ‘public’ data)

• sql db is leading, you don’t care about versioning in couchdb

• you can use your data directly from couchdb within your frontend javascript

<?php

class Pony extends Application_models{ public function toArray() { $data = $this->_getData();

// we don't want any private data pushed unset($data['created_on']); unset($data['created_by']); unset($data['access_level']); unset($data['private_data']);

$data['tags'] = $this->getTags(); $data['categories'] = $this->getCategories(); $data['rainbows'] = 'double'; return $data; }}

MODEL

AFTER_SAVE<?php

class article_module extends admin_module{ public function after_save() { parent::after_save(); $data = $this->toJson();

// store the document in couchdb $res = CouchDB::put($data); // set the local id and revision, so we can // use it the next time $this->_id = $res->_id; $this->_rev = $res->_rev; }}

RewriteEngine OnRewriteRule /data/(.*) http://127.0.0.1:5984/db/$1 [P,L]

PROXY

?callback=foobar

JSONP

foobar(...)

JAVASCRIPT

<script type="text/javascript">$.getJSON("/db/ponies/_design/ponies/_view/best-ponies?include_docs=true",function(res){ for(i in res.rows) { doc = res.rows[i].doc; // do stuff }});</script>

CONTENTS

• Introduction

• PHP Usage

• Replication/Scalability

• Backend usage

• Couchapps

•Other stuff

COUCHAPP

“Distributed, scalable, web applications you say?

omgwtfbbq!?!1!!!one1!1!eleven”

_attachments

CouchDBrelax

CouchDBrelax

CouchDBrelax

INSTALLATION

Couchapp 0.7.5

$ couchapp init

LAYOUT

$ couchapp push http://ponies.couchone.com:5984/rainbows

get your own copy:

$ couchapp clone http://svdgraaf.couchone.com/couchappdemo/_design/↩ phpday phpday

https://github.com/brandon-beacher/couchapp-tmbundle

CONTENTS

• Introduction

• PHP Usage

• Replication/Scalability

• Backend usage

• Couchapps

•Other stuff

OTHER STUFF

REWRITES

_REWRITE

$ curl "http://ponies.couchone.com/rainbows/_design/ponies/_view/best-ponies?descending=true&limit=5&key=”foobar”"

STEP 1

{ .... "rewrites": [{ "from": "/best-5-ponies", "to": "ponies/_view/best-ponies", "method": "GET", "query": { "descending": true, "limit": 5, "key": "foobar" } }]}

STEP 1

$ curl "http://ponies.couchone.com/rainbows/_design/ponies/_view/best-ponies?descending=true&limit=5&key=”foobar”"

STEP 1

$ curl "http://ponies.couchone.com/rainbows/_design/ponies/_rewrite/best-5-ponies"

STEP 1

[vhosts]awesomeponies.com = /rainbows/_design/ponies/_rewrite

STEP 2

$ curl "http://ponies.couchone.com/rainbows/_design/ponies/_rewrite/best-5-ponies"

STEP 2

$ curl "http://awesomeponies.com/best-5-ponies"

STEP 2

_CHANGES

RELAX

$ curl -X GET "http://ponies.couchone.com/rainbows/_changes"

{"results":[

], "last_seq":0}

curl -X PUT http://ponies.couchone.come/rainbows/foobar -d '{"type":"awesome"}'

{"results":[ { "seq":1, "id":"foobar", "changes":[{"rev":"1-aaa8e2a031bca334f50b48b6682fb486"}] }], "last_seq":1}

{"results":[ { "seq":1, "id":"foobar", "changes":[{"rev":"1-aaa8e2a031bca334f50b48b6682fb486"}] }, { "seq":2, "id":"foobar2", "changes":[{"rev":"1-e18422e6a82d0f2157d74b5dcf457997"}] }], "last_seq":2}

_CHANGES OPTIONS

• ?since

• Longpolling

• Continuous

$ curl -X GET "http://ponies.couchone.com/rainbows/_changes?since=20"

SINCE

curl -X GET "http://ponies.couchone.com/rainbows/_changes?feed=longpoll&since=2"

LONGPOLLING

curl -X GET "http://ponies.couchone.com/rainbows/_changes?feed=continuous&since=2"

CONTINOUS

FILTERS

function(doc, req) { if(doc.priority == 'high') { return true; } return false;}

FILTERS/IMPORTANT.JS

$ curl -X GET"http://ponies.couchone.com/rainbows/_changes?feed=continuous&filter=app/important

function(doc, req) { if(doc.city.name == req.query.name) { return true; } return false;}

FILTERS/CITY.JS

curl -X GET"http://ponies.couchone.com/rainbows/_changes?feed=continuous&filter=app/name&name=verona

SHOWS

function(doc, req) { return { body: "Hello World" }}

curl -X"http://db.example.com/db/_design/foobar/_show/showfunction/docid"

function(doc) { return { "code": 302, "body": "See other", "headers": { "Location": doc.target } };}

USE CASECDN

From public hash to private file

/m/m1gz4l4a1clb.jpg

a2d85de382c8672925174e731a1d2a89

Varnish middleware

storage(http based)

proprietarywebserver

mysql

1 2 3

4

5

middleware

storage(http based)

nginx + mod_cacheVarnish

1 2 3

4

function(doc) { return { "code": 302, "body": "See other", "headers": { "Location": doc.target } };}

{ "_id": "m1gz4l4a1clb", "_rev": "2-d7855bba9445f446c0b71f044e796c28", "content-type": "image/jpeg", "target": "http://10.16.10.1/a2d85de382c8672925174e731a1d2a89",}

LUCENE

[external]fti=/path/to/python /path/to/couchdb-lucene/tools/couchdb-external-hook.py

[httpd_db_handlers]_fti = {couch_httpd_external, handle_external_req, <<"fti">>}

function(doc) { var ret=new Document(); ret.add(doc.message); ret.add(new Date(doc.datetime)); return ret;}

FULLTEXT/BY_MESSAGE/BY_QUERY.JS

curl -X GET"http://db.example.com/db/_fti/_design/app/by-query?q=foobar"

GEOCOUCHhttps://github.com/vmx/couchdb

GEOCOUCH

• Supports bbox, polygon, lines, etc.

• fork (for now)

• outputs via lists, georss possible

• directly useable by google maps

• can read GIS data

• combined with _changes makes interesting use cases possible

SPATIAL INDEXin spatial/points.js

function(doc) { if (doc.geo && doc.geo.latitude != '' && doc.geo.longitude != '') { emit( { type: "Point", coordinates: [ parseFloat(doc.geo.latitude), parseFloat(doc.geo.longitude) ] }, [ doc._id, doc ] ); } }

SPATIAL INDEXin spatial/points.js

function(doc) { if (doc.geo && doc.geo.latitude != '' && doc.geo.longitude != '') { emit( { type: "Point", coordinates: [ parseFloat(doc.geo.latitude), parseFloat(doc.geo.longitude) ] }, [ doc._id, doc ] ); } }

SPATIAL INDEXin spatial/points.js

function(doc) { if (doc.geo && doc.geo.latitude != '' && doc.geo.longitude != '') { emit( { type: "Point", coordinates: [ parseFloat(doc.geo.latitude), parseFloat(doc.geo.longitude) ] }, [ doc._id, doc ] ); } }

http://db.example.com/couchappdemo/_design/phpday/_spatial/points?bbox=0,0,180,90

Worldwide search

{"update_seq":3,"rows":[ { "id":"Verona", "bbox":[10.991748,45.438367,10.991748,45.438367], "value":[ "Verona", [10.991748,45.438367] ] }]}

bbox=west,south,east,north

https://github.com/couchbaselabs/iOS-Couchbase

COUCHDB FOR IOS

http://www.iriscouch.com/

http://www.couchbase.com/downloads

Q?

http://joind.in/talk/view/3009