Couchbase - The Engagement DatabaseAbbas Ahmed - Principal Database Engineer @ Rentalcars.com

Today’s Agenda

• Introduction • What is an engagement database? • Couchbase Architecture Overview & essential theory

• Couchbase Lab setup • Lab Exercises

2

Who am I

3

Who am I• One of six DB Engineers @ Rentalcars.com

3

Who am I• One of six DB Engineers @ Rentalcars.com• Have been a MySQL user/admin since 1999 (version 3.1)

3

Who am I• One of six DB Engineers @ Rentalcars.com• Have been a MySQL user/admin since 1999 (version 3.1)• Have worked with Oracle, SQL Server & Foxpro :)

3

Who am I• One of six DB Engineers @ Rentalcars.com• Have been a MySQL user/admin since 1999 (version 3.1)• Have worked with Oracle, SQL Server & Foxpro :)• Like doing solution designs

3

Who am I• One of six DB Engineers @ Rentalcars.com• Have been a MySQL user/admin since 1999 (version 3.1)• Have worked with Oracle, SQL Server & Foxpro :)• Like doing solution designs• Started using Couchbase in 2013

3

Who am I• One of six DB Engineers @ Rentalcars.com• Have been a MySQL user/admin since 1999 (version 3.1)• Have worked with Oracle, SQL Server & Foxpro :)• Like doing solution designs• Started using Couchbase in 2013• Was honoured as a Couchbase Champion in early 2017

3

More Introductions

4

More Introductions• Let’s introduce ourselves one at a time.

4

More Introductions• Let’s introduce ourselves one at a time.• Please highlight your experience with NoSQL or SQL

4

A Useful Quote

5

• “Different databases are designed to solve different problems. Using a single database engine for all of the requirements usually leads to non- performant solutions; storing transactional data, caching session information, traversing graph of customers and the products their friends bought are essentially different problems.”

― Pramod J. Sadalage

― Pramod J. Sadalage

To which general NoSQL category does Couchbase belong?

6

To which general NoSQL category does Couchbase belong?

6

• Key Value Store (Binary Values)

To which general NoSQL category does Couchbase belong?

6

• Key Value Store (Binary Values)• Yes

To which general NoSQL category does Couchbase belong?

6

• Key Value Store (Binary Values)• Yes

• Document Store (JSON Values)

To which general NoSQL category does Couchbase belong?

6

• Key Value Store (Binary Values)• Yes

• Document Store (JSON Values)• Yes

To which general NoSQL category does Couchbase belong?

6

• Key Value Store (Binary Values)• Yes

• Document Store (JSON Values)• Yes

• Column Family

To which general NoSQL category does Couchbase belong?

6

• Key Value Store (Binary Values)• Yes

• Document Store (JSON Values)• Yes

• Column Family• No

To which general NoSQL category does Couchbase belong?

6

• Key Value Store (Binary Values)• Yes

• Document Store (JSON Values)• Yes

• Column Family• No

• Graph

To which general NoSQL category does Couchbase belong?

6

• Key Value Store (Binary Values)• Yes

• Document Store (JSON Values)• Yes

• Column Family• No

• Graph• No

Core Couchbase PrinciplesThe core database engine provides the fundamental capabilities for document management and indexing. Based on a memory-first, async everywhere architecture, this includes core capabilities of the database engine, like caching, data persistence, and inter-node replication.

7

Where did Couchbase come from?

8

Where did Couchbase come from?

8

• Memcached -> Membase -> Couchbase

Where did Couchbase come from?

8

• Memcached -> Membase -> Couchbase• Storage Engine written in C++

Where did Couchbase come from?

8

• Memcached -> Membase -> Couchbase• Storage Engine written in C++

• ForestDB, Plasma

Where did Couchbase come from?

8

• Memcached -> Membase -> Couchbase• Storage Engine written in C++

• ForestDB, Plasma• Cluster Manager & XDCR Originally written in

Erlang

Where did Couchbase come from?

8

• Memcached -> Membase -> Couchbase• Storage Engine written in C++

• ForestDB, Plasma• Cluster Manager & XDCR Originally written in

Erlang• Being ported to Go lang

Where did Couchbase come from?

8

• Memcached -> Membase -> Couchbase• Storage Engine written in C++

• ForestDB, Plasma• Cluster Manager & XDCR Originally written in

Erlang• Being ported to Go lang

• Lots of performance and reliability improvements seen

Where did Couchbase come from?

8

• Memcached -> Membase -> Couchbase• Storage Engine written in C++

• ForestDB, Plasma• Cluster Manager & XDCR Originally written in

Erlang• Being ported to Go lang

• Lots of performance and reliability improvements seen

••

The Couchbase Data Platform

9

• An end to end data management & delivery system • Supports the emerging “Engagement” use cases very well • What is Engagement?

Couchbase Server architecture

Mobile & IoT Analytics

Preview

KV Query Search

Elastic Scale Architecture Memory-first Architecture

Unified Programming Interface

Core Database Engine

Any hardware | Any Cloud | Any Container

Cro

ss S

tack

Sec

urity

SQL & Big D

ata Integrations

How does the CAP theorem see it?

11

How does the CAP theorem see it?• A single Couchbase cluster is a ‘CP’ system

11

How does the CAP theorem see it?• A single Couchbase cluster is a ‘CP’ system

• All changes to any documents are immediately visible to other nodes in the same cluster, hence providing a consistent view.

11

How does the CAP theorem see it?• A single Couchbase cluster is a ‘CP’ system

• All changes to any documents are immediately visible to other nodes in the same cluster, hence providing a consistent view.

• It tolerates network partitions, but never at the cost of consistency

11

How does the CAP theorem see it?• A single Couchbase cluster is a ‘CP’ system

• All changes to any documents are immediately visible to other nodes in the same cluster, hence providing a consistent view.

• It tolerates network partitions, but never at the cost of consistency• Intra-cluster replication is from Memory to Memory using DCP protocol.

11

How does the CAP theorem see it?• A single Couchbase cluster is a ‘CP’ system

• All changes to any documents are immediately visible to other nodes in the same cluster, hence providing a consistent view.

• It tolerates network partitions, but never at the cost of consistency• Intra-cluster replication is from Memory to Memory using DCP protocol.

• When you think in terms of XDCR replicated clusters, however it is an AP system across clusters.

11

How does the CAP theorem see it?• A single Couchbase cluster is a ‘CP’ system

• All changes to any documents are immediately visible to other nodes in the same cluster, hence providing a consistent view.

• It tolerates network partitions, but never at the cost of consistency• Intra-cluster replication is from Memory to Memory using DCP protocol.

• When you think in terms of XDCR replicated clusters, however it is an AP system across clusters.• Because XDCR is eventually consistent

11

How does the CAP theorem see it?• A single Couchbase cluster is a ‘CP’ system

• All changes to any documents are immediately visible to other nodes in the same cluster, hence providing a consistent view.

• It tolerates network partitions, but never at the cost of consistency• Intra-cluster replication is from Memory to Memory using DCP protocol.

• When you think in terms of XDCR replicated clusters, however it is an AP system across clusters.• Because XDCR is eventually consistent• XDCR has to hit the disk before before data is pushed to remote. (Not

true for upcoming Ephemeral Buckets)

11

What services does a node run?

12

What services does a node run?

12

• memcached (The main storage)

What services does a node run?

12

• memcached (The main storage)• goxdcr (XDCR process)

What services does a node run?

12

• memcached (The main storage)• goxdcr (XDCR process)• indexer (The Indexer)

What services does a node run?

12

• memcached (The main storage)• goxdcr (XDCR process)• indexer (The Indexer)• cbq-engine (N1QL Query Engine)

What services does a node run?

12

• memcached (The main storage)• goxdcr (XDCR process)• indexer (The Indexer)• cbq-engine (N1QL Query Engine)• And other auxiliary ones

Scaling Choices for Couchbase

13

Multi-dimensional Scaling (MDS)

14

Data Organization in Couchbase

15

Data Organization in Couchbase

• What do you call a database or schema?

15

Data Organization in Couchbase

• What do you call a database or schema?• Bucket

15

Data Organization in Couchbase

• What do you call a database or schema?• Bucket

• How does a Bucket get distributed over multiple nodes?

15

Data Organization in Couchbase

• What do you call a database or schema?• Bucket

• How does a Bucket get distributed over multiple nodes?• Automatic Shards called vBuckets (1-1024)

15

Data Organization in Couchbase

• What do you call a database or schema?• Bucket

• How does a Bucket get distributed over multiple nodes?• Automatic Shards called vBuckets (1-1024)

• Clients maintain two connections to the cluster

15

Data Organization in Couchbase

• What do you call a database or schema?• Bucket

• How does a Bucket get distributed over multiple nodes?• Automatic Shards called vBuckets (1-1024)

• Clients maintain two connections to the cluster• One for Data

15

Data Organization in Couchbase

• What do you call a database or schema?• Bucket

• How does a Bucket get distributed over multiple nodes?• Automatic Shards called vBuckets (1-1024)

• Clients maintain two connections to the cluster• One for Data• One for Management (Cluster Map Updates, Topology

change updates)15

The Couchbase cluster map

16

Inter-node replication

17

Tunable Durability for Clients

18

Sizing a Couchbase cluster

19

Sizing a Couchbase cluster

19

• Outcomes: Number and Size of nodes (RAM, CPU & Disk Space)

Sizing a Couchbase cluster

19

• Outcomes: Number and Size of nodes (RAM, CPU & Disk Space)

• Inputs: Number of items, average size of items and number of replicas required.

Sizing a Couchbase cluster

19

• Outcomes: Number and Size of nodes (RAM, CPU & Disk Space)

• Inputs: Number of items, average size of items and number of replicas required.

• Max document size = 20MBytes

Time for Lab Exercises

20

Time for Lab Exercises

20

• How to plan a cluster

Time for Lab Exercises

20

• How to plan a cluster• How to start a new cluster

Time for Lab Exercises

20

• How to plan a cluster• How to start a new cluster• How to expand an existing cluster

Time for Lab Exercises

20

• How to plan a cluster• How to start a new cluster• How to expand an existing cluster• How to perform node failover

Time for Lab Exercises

20

• How to plan a cluster• How to start a new cluster• How to expand an existing cluster• How to perform node failover• Learn how to monitor a Couchbase cluster

Time for Lab Exercises

20

• How to plan a cluster• How to start a new cluster• How to expand an existing cluster• How to perform node failover• Learn how to monitor a Couchbase cluster

Time for Lab Exercises

20

• How to plan a cluster• How to start a new cluster• How to expand an existing cluster• How to perform node failover• Learn how to monitor a Couchbase cluster

• Come back before XDCR Lab

XDCR in Couchbase

21

XDCR in Couchbase

21

• Uses data Push instead of Pull (unlike MySQL)

XDCR in Couchbase

21

• Uses data Push instead of Pull (unlike MySQL)• Supports many topologies, linear, star or mesh, even you

can replicate to the same cluster in a different bucket

XDCR in Couchbase

21

• Uses data Push instead of Pull (unlike MySQL)• Supports many topologies, linear, star or mesh, even you

can replicate to the same cluster in a different bucket• Sends data over an encrypted channel, keys created

when you setup XDCR

XDCR in Couchbase

21

• Uses data Push instead of Pull (unlike MySQL)• Supports many topologies, linear, star or mesh, even you

can replicate to the same cluster in a different bucket• Sends data over an encrypted channel, keys created

when you setup XDCR• Big Data & Elasticsearch connectivity uses XDCR as well.

XDCR in Couchbase

21

• Uses data Push instead of Pull (unlike MySQL)• Supports many topologies, linear, star or mesh, even you

can replicate to the same cluster in a different bucket• Sends data over an encrypted channel, keys created

when you setup XDCR• Big Data & Elasticsearch connectivity uses XDCR as well.

XDCR in Couchbase

22

Backup & Recovery

23

Backup & Recovery

23

• Two methods available.

Backup & Recovery

23

• Two methods available.• cbbackup with/without cbbackupwrapper

Backup & Recovery

23

• Two methods available.• cbbackup with/without cbbackupwrapper• cbbackupmgr (recommended)

Backup & Recovery

23

• Two methods available.• cbbackup with/without cbbackupwrapper• cbbackupmgr (recommended)

• Both full and Incremental backups available.

Backup & Recovery

23

• Two methods available.• cbbackup with/without cbbackupwrapper• cbbackupmgr (recommended)

• Both full and Incremental backups available.• cbbackupmgr offers an enterprise grade solution.

Backup & Recovery

23

• Two methods available.• cbbackup with/without cbbackupwrapper• cbbackupmgr (recommended)

• Both full and Incremental backups available.• cbbackupmgr offers an enterprise grade solution.