Deep Dive into N1QL with Global Secondary Indexes – Couchbase Live New York 2015
Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0:...
-
Upload
couchbase -
Category
Technology
-
view
221 -
download
0
Transcript of Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0:...
![Page 1: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/1.jpg)
GLOBAL SECONDARY INDEXESNEW HIGH PERFORMANCE INDEXER
Cihan BiyikogluDir. Product Management
![Page 2: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/2.jpg)
©2015 Couchbase Inc. 2
Goals
Get to know Global Secondary Indexes (GSI) – the new high performance indexer for N1QL
Look at Indexing lifecycle & management with GSI
Cover top best practices and tips with GSI
![Page 3: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/3.jpg)
©2015 Couchbase Inc. 3
Agenda Overview
Indexing in Couchbase Server 4.0 Couchbase Server 4.0 Architecture Indexers in Couchbase Server 4.0 Indexing today and Indexing with GSI
Working with Global Secondary Indexes GSI Architecture GSI Lifecycle - Creation & Maintenance Index Availability & Rebalance Index Placement and Load Balancing Monitoring GSI Best Practices for GSI
Q&A
![Page 4: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/4.jpg)
Overview
![Page 5: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/5.jpg)
©2015 Couchbase Inc. 5
Couchbase Server Cluster Architecture
STORAGE
Couchbase Server 1
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Managed CacheStorage
Data Service
Index Service
Query Service
STORAGE
Couchbase Server 2
Managed Cache
Cluster ManagerCluster Manager
Data Service
Index Service
Query Service
STORAGE
Couchbase Server 3
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Data Service
Index Service
Query Service
STORAGE
Couchbase Server 4
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Data Service
Index Service
Query Service
STORAGE
Couchbase Server 5
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Data Service
Index Service
Query Service
STORAGE
Couchbase Server 6
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Data Service
Index Service
Query Service
Managed CacheStorage
Managed CacheStorage
Managed CacheStorage
Managed CacheStorage
Managed CacheStorage
![Page 6: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/6.jpg)
©2014 Couchbase Inc.
Couchbase Server Cluster Architecture
STORAGE
Couchbase Server 1
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Managed CacheStorage
Data Service
Index Service
Query Service
STORAGE
Couchbase Server 2
Managed Cache
Cluster ManagerCluster Manager
Data Service
Index Service
Query Service
STORAGE
Couchbase Server 3
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Data Service
Index Service
Query Service
STORAGE
Couchbase Server 4
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Data Service
Index Service
Query Service
STORAGE
Couchbase Server 5
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Data Service
Index Service
Query Service
STORAGE
Couchbase Server 6
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Data Service
Index Service
Query Service
Managed CacheStorage
Managed CacheStorage
Managed CacheStorage
Managed CacheStorage
Managed CacheStorage
![Page 7: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/7.jpg)
©2015 Couchbase Inc. 7
Indexing in Couchbase Server 4.0 Multiple Indexers
GSI – Index ServiceNew indexing for N1QL for low latency queries without compromising on mutation performance (insert/update/delete)Independently partitioned and independently scalable indexes in Indexing Service
Map/Reduce Views – Data ServicePowerful programmable indexer for complex reporting and indexing logic. Full partition alignment and paired scalability with Data Service.
Spatial View – Data ServiceIncremental R-tree indexing for powerful bounding-box queriesFull partition alignment and paired scalability with Data Service
New
![Page 8: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/8.jpg)
©2015 Couchbase Inc. 8
Query and Index Today
Once upon a time in a User Profile System…. Q1: Find the top 10 most “active” customer by
#logins in Jan 2015
{…“customer_name” : ”Cihan”,“total_logins”: {…
“aug_2015”:100,…}
“type” : “customer_profile”…}
…
Q1Active @ Jan
2015
![Page 9: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/9.jpg)
©2015 Couchbase Inc. 9
Query and Index TodayINDEX ON Customer_bucket(customer_name, total_logins.jan_2015)WHERE type=“customer_profile”;
SELECT customer_name, total_logins.jan_2015 FROM customer_bucketWHERE type=“customer_profile”ORDER BY total_logins.jan_2015 DESC LIMIT 10;
…
Q1Active @ Jan
2015
Q1: Execution Plan on N nodes• Scatter: Execute Q1 on N nodes• Gather: gather N results• Finalize: Execute Q1 on
governor node
1
2 2 2 2 2
3
123
![Page 10: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/10.jpg)
©2015 Couchbase Inc. 10
Query and Index with GSIINDEX ON Customer_bucket(customer_name, total_logins.jan_2015)WHERE type=“customer_profile”;
SELECT customer_name, total_logins.jan_2015 FROM customer_bucketWHERE type=“customer_profile”ORDER BY total_logins.jan_2015 DESC LIMIT 10;
…
Q1Active @ Jan
2015
Q1: Execution Plan on N nodes• Execute Q1 on N1QL Service
node• Scan index on Index Service node
12
13
![Page 11: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/11.jpg)
©2015 Couchbase Inc. 11
Introducing Global Secondary Indexes
What are Global Secondary Indexes? High performance indexes for low latency queries with powerful caching, storage and independent placement.
Power of GSI Fully integrated into N1QL Query Optimization and
Execution Independent Index Distribution for Limiting scatter-
gather Independent Scalability with Index Service – more on
this later Powerful caching and storage with ForestDB
![Page 12: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/12.jpg)
©2015 Couchbase Inc. 12
Which to choose – GSI vs Views
Workloads New GSI in v4.0
Map/Reduce Views
Complex Reporting
Just In Time Aggregation Pre-aggregated
Workload Optimization
Optimized for Scan Latency & Throughput
Optimized for Insertion
Flexible Index Logic
N1QL Functions Javascript
Secondary Lookups
Single Node Lookup Scatter-Gather
Tunable Consistency
Staleness false or ok or everything in between
Staleness false or ok
![Page 13: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/13.jpg)
©2015 Couchbase Inc. 13
Which to choose – GSI vs Views
Capabilities New GSI in v4.0
Map/Reduce Views
Partitioning Model Independent – Indexing Service
Aligned to Data – Data Service
Scale Model Independently Scale Index Service
Scale with Data Service
Fetch with Index Key Single Node Scatter-Gather
Range Scan Single Node Scatter-Gather
Grouping, Aggregates With N1QL Built-in with Views API
Caching Managed Not Managed
Storage ForestDB Couchstore
Availability Multiple Identical Indexes load balanced
Replica Based
![Page 14: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/14.jpg)
Deep Dive
![Page 15: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/15.jpg)
GSI Architecture
![Page 16: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/16.jpg)
©2015 Couchbase Inc. 16
Data Service
Projector & Router
Indexing Service
Query ServiceIndex Service
SupervisorIndex maintenance &
Scan coordinator
Index#2
Index#1
Query Processorcbq-engine
Bucket#1
Bucket#2
DCP StreamIndex#4Index#3
...
Bucket#2
Bucket#1
Projector and Router: 1 Projector and Router per node1 stream of changes per buckets per supervisor
ForestDBStorage Engine
Supervisor1 Supervisor per nodeMany indexes per Supervisor
![Page 17: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/17.jpg)
©2015 Couchbase Inc. 17
Deeper Dive into Architecture
@3.45 - Architecture Track
Deep Dive into Global Secondary Indexing Architecture in Couchbase
Server 4.0
John Liang, Architect, Couchbase
![Page 18: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/18.jpg)
GSI Lifecycle
![Page 19: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/19.jpg)
©2015 Couchbase Inc. 19
Indexing Lifecycle
Primary vs Secondary Primary Index is a full list of document keys within a given bucket
CREATE PRIMARY INDEX index_nameON bucket_name USING GSI|VIEWWITH `{"nodes”: [“node_name”], “defer_build”:true}`; //GSI-ONLY
Secondary Index is an index on a field/expression on a subset of documents for lookups
CREATE INDEX index_nameON bucket_name (field/expression, …)USING GSI|VIEWWHERE filter_expressionsWITH `{"nodes”: [“node_name”], “defer_build”:true}`; //GSI-ONLY
![Page 20: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/20.jpg)
©2015 Couchbase Inc. 20
Deferred Index Building
Index building can be deferred to build multiple indexes all at once with greater scan efficiency.
CREATE INDEX … WITH {…“defer_build”:true};CREATE INDEX … WITH {…“defer_build”:true};…BUILD INDEX ON bucket_name(index_name1, …) USING GSI;
![Page 21: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/21.jpg)
DEMOQuick tour of GSI
![Page 22: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/22.jpg)
GSI Partitioning and Placement
![Page 23: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/23.jpg)
©2015 Couchbase Inc. 23
GSI Placement and Partitioning
Place GSI Indexes using NODES clause Each GSI reside on 1 node
You can specify the node using nodes clause
You can scale out the index by creating identical indexes (load balanced)
CREATE INDEX i1 … WITH {“nodes”:”node1”};
CREATE INDEX i1 … WITH {“nodes”:”node1”};CREATE INDEX i2 … WITH {“nodes”:”node2”};
…
![Page 24: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/24.jpg)
©2015 Couchbase Inc. 24
GSI Placement and Partitioning
Partition Indexes Manually with WHERE clause
You can partition with the WHERE clause and place on various nodes for scaling outCREATE INDEX i1 … WHERE zipcode between “94000” and
“94999” …CREATE INDEX i2 … WHERE zipcode between “95000” and
“96000” …
![Page 25: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/25.jpg)
GSI Availability and Rebalance
![Page 26: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/26.jpg)
©2015 Couchbase Inc. 26
GSI Availability and Rebalance
Use multiple identical indexes for availability
GSI Availability Create multiple identical indexes on separate nodes for
availability GSI will auto divert traffic if any copy goes down
GSI & Rebalance Removing/Failing a node with index service, remove the
GSI indexes on that node Adding a node with index service, won’t automatically
move some indexes to the node.
![Page 27: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/27.jpg)
Monitoring GSI
![Page 28: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/28.jpg)
©2015 Couchbase Inc. 28
Monitoring GSI Indexes
Index Size and Maintenance StatsIndex Scan Stats
![Page 29: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/29.jpg)
Best Practices with GSI
![Page 30: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/30.jpg)
©2015 Couchbase Inc. 30
New Consistency Settings!
View Stale-ness Ok: unbounded – query what’s available in the
index/view now False: query after all changes up to the request
timestamp (and maybe more) has been indexed for a given index or view.
New Indexes with Couchbase Server 4.0 Improves granularity of the consistency logical-
timestamp. New: Scan Consistency can be set to any logical
timestamp
Indicate stale=false to stale=ok and everything in between
![Page 31: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/31.jpg)
©2015 Couchbase Inc. 31
Flexible Consistency Settings Time
t1 insert (k1, v1)…
t2 do other business logic computation…
t3 issue query/read on (k1,v1) with t3 vs t1
Catch up all the indexes to t3 and then issue query
Identical to “stale=false”
Catch up all the indexes to t1 and then issue query
Improved efficiency over “stale=false”
![Page 32: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/32.jpg)
©2015 Couchbase Inc. 32
Complex Types and GSI
Indexing Complex Types Sub-documents attributes
CREATE INDEX ifriend_id ON default(friends.id) USING GSI;
SELECT * FROM default WHERE friends.id= "002819”;
{ "id": "00000000000001", "desc": ”---", "type": "friends", "tags": [0,1,2,3,4,5,6,7,8,9], "friends": { "id": "002819", "class": "005" } }
![Page 33: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/33.jpg)
©2015 Couchbase Inc. 33
Complex Types and GSI
Indexing Complex Types Compound Keys
CREATE INDEX ifriends_id_class ON default(friends.id, friends.class) USING GSI;
SELECT * FROM default WHERE friends.id="002819" and friends.class="005”;
{ "id": "00000000000001", "desc": ”---", "type": "friends", "tags": [0,1,2,3,4,5,6,7,8,9], "friends": { "id": "002819", "class": "005" } }
![Page 34: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/34.jpg)
©2015 Couchbase Inc. 34
Complex Types and GSI
Indexing Complex Types Sub-documents
CREATE INDEX ifriend ON default(friends) USING GSI;
SELECT * FROM default WHERE friends= {"class": "005","id":"002819"};
{ "id": "00000000000001", "desc": ”---", "type": "friends", "tags": [0,1,2,3,4,5,6,7,8,9], "friends": { "id": "002819", "class": "005" } }
![Page 35: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/35.jpg)
©2015 Couchbase Inc. 35
Complex Types and GSI
Indexing Complex Types Arrays
CREATE INDEX itags_sorted ON default(ARRAY_SORT(tags)) USING GSI;
SELECT * FROM default WHERE tags= ARRAY_SORT([0,1,2,3,4,5,6,7,8,9]);
{ "id": "00000000000001", "desc": ”---", "type": "friends", "tags": [0,1,2,3,4,5,6,7,8,9], "friends": { "id": "002819", "class": "005" } }
![Page 37: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/37.jpg)
Get Started Today Couchbase Server 4.0 & N1QL
Couchbase.com/beta
![Page 38: Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015](https://reader031.fdocuments.net/reader031/viewer/2022032620/55c9dda1bb61eb1d4d8b47ce/html5/thumbnails/38.jpg)
Thank you.