What's new in couchbase 4.0: couchbase connect 2015

WHAT’S NEW IN COUCHBASE SERVER 4.0?

Chin HongVP Product Management, Couchbase

©2015 Couchbase Inc. 2

Agenda

What’s New in Couchbase Server 4.0 Querying with N1QL Indexing Multi-Dimensional Scaling ForestDB: Next-Generation Storage Engine Security


And There’s More …

Feature Description

XDCR Filtering Key-based filtering on XDCR streams addressing reference data & geo-based replication use cases.

Bloom Filters Significantly improved latency for low % working set (DGM) situations

Memory Defragmentation

Use memory according to one’s needs

Improved Cluster Robustness

Rogue view queries no longer cause cluster manager outage

Breakpad Compact and readable crash logs for memcached

SDK First class SDK support across Couchbase Server 4.0 features

SUSE, Oracle Enterprise Linux, CentOS 7, RHEL 7, Ubuntu 14.04

Expand the supported production platforms

Querying with N1QL


What is N1QL?

5

Flexibility of JSON

Power of SQL


How Does One Evaluate a Query Language?

Expressive

Declarative

Ecosystem


N1QL is Expressive: E-Commerce Examples

SELECT product.name, SUM(items.count) AS unitsSold FROM purchases UNNEST purchases.lineItems AS items JOIN product ON KEYS items.product GROUP BY product ORDER BY unitsSold DESC LIMIT 10

SELECT SUBSTR(purchases.purchasedAt, 0, 7) AS month,

ROUND(SUM(product.unitPrice * items.count)/1000000, 3) revenueMillion

FROM purchases UNNEST purchases.lineItems AS items

JOIN product ON KEYS items.product

GROUP BY SUBSTR(purchases.purchasedAt, 0, 7)

ORDER BY month


N1QL is Expressive: Image Processing Example

SELECT

m.image AS gallery_image,

s.image AS source_image, s.x, s.y,

m.r AS gallery_r, m.g AS gallery_g, m.b AS gallery_b,

s.r AS source_r, s.g AS source_g, s.b AS source_b,

FROM SourcePixels s

JOIN GalleryMeans m ON KEY s.rgb

Source: http://www.metanautix.com/tr01

Learn more about N1QL @ query.couchbase.com/tutorial


Sessions on N1QL Introducing N1QL: Query Without Compromise

May 3 @ 1:00 pm – 1:45 pm, Architecture TrackMay 4 @ 11:15 am – 12:00 pm, Architecture Track

N1QL and SDK Support for Java, .NET, and Node.jsMay 3 @ 4:30 pm - 5:15 pm, Developer Track

Migrating MySQL Queries to N1QLMay 4 @ 1:00 am – 1:45 pm, Developer Track

Big Data Query Landscape: N1QL and MoreMay 4 @ 4:30 pm – 5:15 pm, Architecture Track

Interactive Data Analytics with Couchbase N1QL – NielsenMay 3 @ 1:45 pm - 2:30 pm, Developer Track

Agility with Data Modeling at DIRECTV Using N1QL in Couchbase with Node.js - DIRECTVMay 4 @ 3:45 pm - 4:30 pm, Developer Track


N1QL is Declarative: What vs. How

You specify WHATCouchbase Server figures out HOW


N1QL is Declarative: Top 10 Selling Products

SELECT product.name, SUM(items.count) AS unitsSold FROM purchases UNNEST purchases.lineItems AS items JOIN product ON KEYS items.product GROUP BY product ORDER BY unitsSold DESC LIMIT 10

WHAT

HOW


Sessions on N1QL – Deep Dive Under the Hood: Couchbase Server 4.0 Architecture

May 3 @ 11:30 pm - 12:15 pm, Architecture Track Deep Dive into N1QL: Power Features and Internals in Couchbase 4.0

May 3 @ 2:30 pm - 3:15 pm, Architecture Track Tuning Query Performance with N1QL in Couchbase Server 4.0

May 4 @ 1:45 pm – 2:30 pm, Architecture Track Enterprise Architect’s Perspectives of Couchbase with N1QL

May 4 @ 5:15 pm – 6:00 pm, Architecture Track


N1QL Connects To Your Enterprise Ecosystem

Standards-based drivers

Integrations, partnerships

ODBC / JDBC

App

CB Node

ODBC / JDBC

ETL

ODBC / JDBC

BI

ODBC / JDBC

Visualization

CB Node CB Node


Sessions on N1QL – Partners Operational Analytics with Looker and N1QL- Looker

May 3 @ 1:00 pm – 1:45 pm, Big Data Track Enterprise Data Integration: Using Informatica ETL with Couchbase –

InformaticaMay 3 @ 3:45 pm – 4:30 pm, Big Data Track

Visual Analytics with Tableau & Couchbase – Tableau SoftwareMay 3 @ 4:30 pm - 5:15 pm, Architecture Track

Open Connectivity: BI, Integration, and Applications on Couchbase using ODBC and JDBC - SimbaMay 4 @ 1:45 pm – 2:30 pm, Big Data Track

Metanautix Quest and Couchbase: Scalable Analytics Across NoSQL, RBDMS, Hadoop – MetanautixMay 4 @ 3:45 pm – 4:30 pm, Big Data Track

Indexing


Why Do We Need Indexes?

Indexes are used to quickly look up objects

meeting user-specified criteria without having to search over every

object in the database.


What Are the Different Types of Indexes?

Primary IndexesSecondary Indexes

Spatial IndexesText Indexes


Index Types … Let’s Use the Library Analogy

Source: http://www.montel.com/en/markets/library/


Secondary Indexes

Local secondary indexes – Map/Reduce views in 3.0 Co-located with data Higher write performance Lower read performance: scatter-gather Scaling bottleneck with a high number of indexes or data

nodes

Global secondary indexes (GSI) – New in 4.0 Higher query performance Async writes to a large number of global indexes Independently scaled and partitioned Isolated from Key-Value operations


Sessions on Global Secondary Indexes Introducing Global Secondary Indexes: The New High Performance

Indexer with Couchbase 4.0May 3 @ 1:45 pm - 2:30 pm, Architecture Track

Deep Dive into Global Secondary Indexing Architecture in Couchbase Server 4.0May 3 @ 3:45 pm - 4:30 pm, Architecture Track


Box on Points Box Intersects Multi Dimensions Geo + Others

Spatial Indexes

Experimental in 3.0, now GA in 4.0 Performance and stability improvements

GeoJSON output Bounding-box and range queries on multiple

dimensions


Session on Spatial Indexes Introducing Spatial Views for Location Aware Applications

with Couchbase Server 4.0May 4 @ 4:30 pm – 5:15 pm, Architecture Track


Text Indexes (Developer Preview)

…

wise

…

engineer…

…

…

…, akay1980, …

…, akay1980, …

engineers

engineer Exact

Match

Inverted Index

Apply the sametext analysisat search timethat we usedat index time.


Session on Text Indexes A Sneak Peek of CBFT, Full Text Search for Couchbase

May 3 @ 5:15 pm –6:00 pm, Architecture Track

Multi-Dimensional Scaling


Workload and Scaling

Horizontal scaling Partitions a dataset onto one or more homogenous nodes Each node runs the same mixed workloads Re-partition dataset with additional hardware capacity

Node 8

Index Service

Couchbase Cluster

Query Service

Data Service

Node 1



Isolated Service for minimized interference Independent “zones” for query, index, and data

services

Minimize indexing and query overhead on core key-value operations.

Index Service

Couchbase Cluster

Query Service Data Service

Node 1 Node 8



Independent scalability for the best computational capacity per service

Couchbase Cluster

node1 Node 8 Node 9

Data Service

Index ServiceQuery Service

More storage for indexing: scale up index service nodes.

More cores for query processing:scale up query service nodes.

Node 1


Session on Multi-Dimensional Scaling Multi-Dimensional Scaling: A New Architecture for Scaling

Big Data ApplicationsMay 4 @ 10:30 am – 11:15 am, Architecture Track

ForestDB: Next-Generation Storage Engine


Common Storage Structures – B+-Tree

31

Reads Writes Engines

B+-Tree Good read performance if the fan-out is high (“short” tree) for small fixed-length keys. Read performance degrades for variable-length keys.

Update-In-Place results in random writes and bad write latency.

Append-Only file improves write performance but requires periodic compaction.

BerkeleyDB, InnoDB, Couchstore (append-only), WiredTiger B+

04/26

…

…

…Key

Value (or Pointer)

longer keys


Common Storage Structures – LSM-Tree

32

Reads Writes EnginesLSM-Tree

Reads may have to traverse multiple trees – typically worse than B+ tree.

WAL(Write-Ahead Log) improves writes within-memory trees that are appended to the end of the log.

LevelDB, RocksDB, Cassandra, WiredTiger LSM

…

In-memory

Sequential log

flush/merge merge

C1 tree C2 tree

merge

Capacity increases exponentially


Fast Storage – ForestDB

Fundamentally rethinking the requirements for fast storage Compact and efficient storage for a variety of data Unified engine that performs well for various workloads Unified engine that scales from small devices to large

servers Optimized for new SSD storage technology

33

ForestDB provides a compact index structure

that significantly reduces the disk I/Oand write amplification.


ForestDB

ForestDB Evaluations: LevelDB, RocksDB

34

Throughput

UP TO

FASTER6X

Future Proof

FITTED

FORSSD

Efficiency

UP TO

5X

COMPACT


Trie (prefix tree) for which the nodes are B+-Trees HB+-Trie was originally presented at ACM SIGMOD 2011 Programming

Contest, by Jung-Sang Ahn who works at Couchbase(http://db.csail.mit.edu/sigmod11contest/sigmod_2011_contest_poster_jungsang_ahn.pdf)

HB+-Trie (Hierarchical B+-Trie)


Sessions on ForestDB Next Generation Storage Engine: ForestDB

May 3 @ 4:30 pm – 5:15 pm, Architecture Track

Under the hood ForestDB: Performance on SSDs and File SystemsMay 4 @ 2:30 pm – 3:15 pm, Architecture Track

Intel: Next Generation SSDs and ForestDB: Next Generation Storage EngineMay 4 @ 2:30 pm – 3:15 pm, Architecture Track

Security


Previously… In 2.2 In 2.5 In 3.0 New in 4.0

SASL AuthN with Bucket Passwords

Admin User

Secure Build Platform

Read-Only User

Easy Admin Password

Reset

Non-Root User

Deployments

Secure Communication for XDCR

Encrypted Client- Server

Communication

Encrypted Admin Access

Access Log

Data-at-Rest Encryption

• Simplified compliance with admin auditing

• External identity management for admins using LDAP

Couchbase Security Features

In a fewslides ..

38


External Identity Management Using LDAP

39

Centralized identity management Define multiple read-only admins and full-admins Centralized security policy management for

admin accounts for stronger passwords, password rotation, and auto lockouts

Individual accountability; simplified compliance Define UIDs in LDAP, and map UIDs to the read-

only/full admin role in Couchbase Comprehensive audit trails with LDAP UIDs in

audit records


Admin Auditing in Couchbase

40

Rich audit events Over 25+ different, detailed admin audit events Auditing for tools including backup

Configurable auditing Configurable file target Support for time-based log rotation and audit

filtering

Easy integration JSON format allows for easy integration with

downstream systems using flume, logstash, and syslogd


Auditing a successful login

41

{ "timestamp":"2015-02-20T08:48:49.408-08:00", "id":8192, "name":"login success", "description":"Successful login to couchbase cluster", "role":"admin", "real_userid": { "source":"ns_server", "user":"bjones” }, "sessionid":"0fd0b5305d1561ca2b10f9d795819b2e", "remote":{"ip":"172.23.107.165", "port":59383}}

WHEN

WHO

WHAT

HOW


Sessions on Security Security Features in Couchbase Server

May 3 @ 3:45 pm – 4:30 pm, Operations Track

Securing Your Couchbase Server Deployment Using VormetricMay 4 @ 10:30 pm –11:15 pm, Operations Track

Querying with N1QLIndexing

Multi-Dimensional ScalingForestDB: Next-Generation

Storage EngineSecurity

Get Started with Couchbase Server 4.0: www.couchbase.com/beta

Get Trained on Couchbase: training.couchbase.com

What's new in couchbase 4.0: couchbase connect 2015

Technology

Transcript of What's new in couchbase 4.0: couchbase connect 2015