CouchConf Israel Couchbase in Production 24x7

1

Couchbase Server 2.0 in Production

2

Couchbase Server 2.0: Overview

• Membase + CouchDB• Managed memory caching layer = super high performance• Clustering and online data redistribution (Rebalancing)• Indexing and Querying via JSON Map-Reduce• New SDK’s and client libraries

• Developer Preview available at:• http://www.couchbase.com/downloads

=Simple. Fast. Elastic.

3

www.facebook.com/farm_town_wars

Web Servers

Load Balancer

Couchbase Servers

Let’s build a social game…

4

COUCHBASE SERVER 2.0 IN PRODUCTION: DEV/OPS LIFECYCLE

Initial Setup

View Development Sizing

Client setup

Monitor Grow

Backup/Restore

Upgrade

Deploy

MaintainFailures

Test

5

Couchbase Server 2.0 in production: Initial Setup

Extremely easy to get up and running:

• RPM/deb/OSX/exe installation

• Simple Web UI and setup wizard

6

Data and Indexes

• Data goes “in and out” via the Memcached protocol

• Queries/indexes are created and accessed via an HTTP protocol

• Both are available separately, Couchbase-provided SDK’s will expose a single API to the developer and abstract the actual traffic

7

Data and Indexes

Example JSON Document:

{

"_id": "Keith4540",

"jsonType": "player",

"name": "Keith4540",

"level": 4,

…

…

}

Example view:

function (doc) {

if (doc.jsonType == "player"){

emit(["Level", doc.level], doc._id);}

}

+

=Secondary index of players, by level

(Demo)

8

Data and Indexes

• Indexes/views are based on incremental map-reduce:• Indexes are updated with incremental changes (not

batch)

• View processing is done per-vbucket:• Parallel processing on subset of data• Couchbase provides scatter-gather aggregation

9

Data and Indexes

• Views are “developed” off of a random (or specific) sampling of the overall dataset and then deployed• Faster• Less load on system

• Updated views can be applied without rescanning entire dataset

10

Couchbase Client SDKs

Java Client SDK

.Net SDK

PHP SDK

Ruby SDK

Python SDK

Spy memcachedConnection

HTTP couchDBconnection

Java client API

User Code

Couchbase Server

CouchbaseClient cb = new CouchbaseClient(listURIs,"aBucket", "letmein");// this is all the same as beforecb.set("hello", 0, "world");cb.get("hello");Map<String, Object> manyThings =cb.getBulk(Collection<String> keys);/* accessing a view View view = cb.getView("design_document", "my_view");Query query = new Query();query.getRange("abegin", "theend");

11


Sizing

Client setup

Deploy

12

Couchbase Server 2.0 in production: Sizing

Sizing Question: How many nodes do I need?

Considerations:RAMDiskNetworkData Safety

13

Sizing


Considerations:RAM• Metadata• Working set• Buffer/overhead

Active+Replica

14

Sizing

500,000 Documents to begin with:-20 bytes in length-average Document size of 2k-metadata of about 150 bytes per key

~1Gb to store active data, an extra 1Gb to store replica data

Adding in some headroom: Give 3GB RAM to Couchbase to start and grow with

=

15

Sizing


Considerations:Disk:• Sustained write rate• Index generation (space and IO)• Append-only B-Tree• Compaction• Rebalance capacity

16

Sizing


Considerations:Network:• Client traffic• Replication• Rebalancing

17

Sizing


Considerations:Data Safety

1 node = single point of failure (bad)2 nodes = 1 replica copy (better)3+ nodes = 1 replica copy AND data/load distribution

18

Sizing


Considerations:RAMDisk NetworkData Safety

19

Farm Town Wars App Code

Ap

plic

atio

n s

erv

er

Co

uch

ba

se

Se

rve

r

Couchbase JavaClient library

Query API

Couchbase

11210

(“smart”) library

Farm Town Wars App Code

Ap

plic

atio

n s

erv

er

Co

uch

ba

se

Se

rve

r

Couchbase PHP Client Library

Moxi (Couchbase proxy)

Query API

Couchbase

11210

Client-side Moxi

OR80928092

Client-side Deployment

20


Monitor Grow

Backup/Restore

UpgradeMaintainFailures

21

Monitoring

IMMENSE amount of information available

-Real-time traffic graphs

-REST API accessible

-Per bucket, per node and aggregate statistics

-Application and inter-node traffic

-RAM <-> Disk

-Inter-system timing

23

Growth

Going from 1 hundred users to 2 million users…

– RAM usage is growing:• Ejecting data to and fetching data from disk

• Resident item ratios decreasing – might impact failover

• Cache miss ratio increases

– Disk write queue grows higher than usual

Need to add a few more nodes...

Now we have more RAM and more disk throughput without any downtime

(Demo)

24

General Maintenance

• Persistence is using CouchDB technology:– Append-only B-tree

– Reliability and data integrity

– Constantly growing on-disk files

• Compaction is the answer:– Automatic

– “Fragmentation” data

– Scheduled compaction

– Partial compaction via per-vbucket database (saves disk space and time)

25

• Backup/Restore:– Backup is as simple as a file-level copy (thanks CouchDB!)

– Server will automatically “warmup” from disk files upon reboot

• Upgrade:– Add nodes of new version, rebalance…

– Remove nodes of old version, rebalance…

– Done!

– No disruption

– Upgrade from existing Membase 1.7 installations to

Couchbase 2.0

General Maintenance

26

Failures

• Failures happen:

– Hardware

– Network– Bugs

• Failover to replica data for immediate access

• Remove and rebalance “malfunctioning” node

(Demo)

27


Initial Setup

View Development Sizing

Client setup

Monitor Grow

Backup/Restore

Upgrade

Deploy

MaintainFailures

Test

28

THANK YOU!

Q&A

CouchConf Israel Couchbase in Production 24x7

Technology

Transcript of CouchConf Israel Couchbase in Production 24x7