Couchbase 104:Views and Indexing

Todd GreensteinSolution Architect

[Agenda]

[1] What are Views[2] Architecture in Couchbase[3] Uses[4] View Anatomy (Map/Reduce)[5] Examples

Indexing Subsystem

• Storing data and Indexing data are separate systems in all databases

• In explicit schema scenarios (RDBMS), Indexes are optimized based on the data type(s)

• In flexible schema scenarios Map-Reduce is used to create indexes

Map Reduce

• Map-Reduce is a technique designed for dealing with Big Data and processing in parallel in distributed systems

• Map-Reduce is also specifically designed for dealing with unstructured or semi-structured data

• Map functions identify data with collections, process them, and output transformed values

• Reduce functions take the output of Map functions and perform numeric aggregate calculations on them

Couchbase Map Reduce• In Couchbase, Map-Reduce is specifically used

to create Indexes• Map functions are applied to JSON

documents and they output or "emit" data that is organized in an Index form

[Architecture]

Storage to Index

Views: Eventual Consistency

Time 1

Time 2

Buckets >> Design Documents >> Views

Couchbase Bucket

Design Document 1 Design Document 2

View ViewViewViewView

Indexers Are Allocated Per Design Doc

All Updated at Same TimeAll Updated at Same TimeAll Updated at Same Time

Can Only Access Data in the Bucket Namespace

Can Only Access Data in the Bucket Namespace

33 2

Eventually indexed Views – Data flow2

Managed Cache

Dis

k Q

ueue

Disk

Replication Queue

App Server

Couchbase Server Node

Doc 1Doc 1

Doc 1

To other node

View engine

Doc 1

Index building details

•All the views within a design document are incrementally updated when the view is accessed or auto-indexing kicks in

•The entire view is recreated if the view definition has changed

•The index information stored on disk consists of the combination of both the key and value information defined within your view.

Automatic Index Updates

• Views can be updated automatically according to:– Update Interval (time period); default 5000 milliseconds– Update Documents (number of changed documents);

default 5000 changes– Replica Update Interval for index replicas

• Can be set globally• Can be set on individual design documents– Configured using REST API

• Operate independently of manual updates• Only updates with eligible documents

Manual Index Updates

• All queries support ‘stale’ parameter to view• Allows application to control updates– Has performance considerations

• Update request for one view updates all views within corresponding design document

• Can be triggered through UI or REST API as well

• Only updates with eligible documents

Understanding Stale

stale = UPDATE_AFTER (default if nothing is specified)always get fastest responsecan take two queries to read your own writesstale = OKauto update will trigger eventuallymight not see your own writes for a few minutesleast frequent updates -> least resource impactstale = FALSEUse with Persistence observe if data needs to be included in view resultsBUT aware of delay it adds, only use when really required

Replica Indexes• In addition to data replicas, optionally create replica for indexes

– Build an index using the data in replica vBuckets• Enabled per bucket (see bucket config) or per design document

(REST API)• Implies additional CPU and I/O overhead

– Each node must update index for active data and replica data

• In the event of failover:– Without replica indexes, queries may be out of date for longer while

rebuilt– Replica indexes enabled if present and queries remain consistent– Replicas indexes are rebuilt on replica nodes

[Uses]

Use Cases

• Iteration• Retrieval (UUID/GUID)• Ranges (Parameters)• Aggregating (Built in Reduces)• Statistical Analysis

[View Anatomy]

Map() Function => Index

function(doc, meta) {emit(doc.username, doc.email)

} indexed key output value(s)create row

json doc doc metadata

Every Document passes through View Map() functions

Map

Single Element Keys (Text Key)

function(doc, meta) {emit(doc.email, doc.points)

} text key

Map

meta.id

doc.emaildoc.po

ints

u::1abba@couchbase.co

m1000

u::35

jasdeep@couchbase.com

1200

u::20

zorro@couchbase.com

900

Compound Keys (Array)

function(doc, meta) {emit(dateToArray(doc.timestamp), 1)

} array key

Array Based Index Keys get sorted as Strings, but can be grouped by array elements

Map

meta.id

dateToArray(doc.timestamp)

value

u::20 [2012,10,9,18,45] 1

u::1 [2012,9,26,11,15] 1

u::35 [2012,8,13,2,12] 1

Reduce Functions• Optional• Operate on View Value emitted by map function

• Built-in reduce– _count – provides a count of unique keys– _sum – provides a sum total of values– _stats – provides statistics (max, min, avg, etc.) of values

• Automatically re-reduces up B-tree• Results stored pre-computed for fast access

• Custom reductions possible if needed but less performant

View Query Parameters

[Examples]

Sample Inventory Bucket

{"item":"KS-00114629 Dark Grey Glasses","vendor":"kate spade","category":"glasses"}

{"item":"IH-00884501 Triple ring Shoulder bag","vendor":"iron heart","category":"handbag",”discount":"15"}