ELASTIC AND COUCHBASE: CONFIGURATION TO CAPABILITIES TO REAL-TIME BIBala VenkatraoSenior DirectorProduct Management @ Elastic
David MaierPrincipal Solutions Engineer
©2015 Couchbase Inc. 2
Agenda
Welcome Introduction to Elastic Demo Behind the scenes What’s new in the 2.0 Plug-in
©2015 Couchbase Inc. 3
Welcome!
Bala VenkatraoSenior Director, Product Management
Introduction to Elastic Unlocking the value of your data through search, analytics, and more . . .
www.elastic.co
www.elastic.co Copyright Elastic 2015 Copying, publishing and/ordistributing without written permission is strictly prohibited
Problem: Complex Data + Diverse Requirements
Social
Location
User-Activit
y Machine
Documents
5
COMPLEX & DIVERSE DATA
MISSION CRITICALBUSINESS REQUIREMENTS
Developer requirements
Many users / use cases
Fast data processing
Large data volumes
Data quality & integrity
How do you build
scalable, Search/Anal
ytics platform?
Cross-source insights
www.elastic.co Copyright Elastic 2015 Copying, publishing and/ordistributing without written permission is strictly prohibited
6
New Systems to Address New Requirements
Schemaless
Horizontal Scale
Real Time Data Availability
Fast Search & Query Execution
Structured and unstructured
Flexibility
Sophisticated Query Language
www.elastic.co Copyright Elastic 2015 Copying, publishing and/ordistributing without written permission is strictly prohibited
7
Elasticsearch: Search, Store, and Analyze
Distributed, scalable, and resilient Designed for scale-out; high availability
Developer friendlyAPI-first; schemaless, native JSON, client libraries for any language
Real-time Search & AnalyticsReal-time aggregations, geospatial, full-text search; query structured and unstructured data
www.elastic.co Copyright Elastic 2015 Copying, publishing and/ordistributing without written permission is strictly prohibited
8
Introducing the Elastic Product Portfolio
KibanaVisualize and explore data
ElasticsearchStore, search, analyze
LogstashHadoop
ConnectorCollect, parse and enrich data
MarvelMonitor and manage
ShieldSecure and protect
FoundElasticsearch as a Service
Open Source Products
Commercial Products
Training Professional Services Support Subscriptions
WatcherNotify and Alert
Powering diverse Use Cases from Search, Analytics, Log Analysis and more…
9www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
• Analytics/Search Engine based on Apache Lucene
• Schema-free JSON document store• Every field is
indexed/searchable• Scalable sharded distributed
architecture• Simple REST API• (Near) Real-time Search• Aggregations/Analytics support• Geospatial support• Extensibility via plugins
Why use Elasticsearch and Couchbase together?
Elasticsearch• Key-Value document store• Supports JSON, Binary, XML …• N1QL Query Language =>
SQL + JSON• Scalable, distributed architecture
• Active memory management (built-in cache)
• SDK for language integration in Java, .NET, Node.js, Python…• Uses binary wire protocol for data
access
• Extends to iOS, Android and Xamarin with Couchbase Mobile
Couchbase
Couchbase & ElasticsearchA brief Introduction
©2015 Couchbase Inc. 11
Couchbase Server & Elasticsearch
Key Reference
tim #2
bob #3
jim #1
Secondary Index
Key Reference
#1 { name : ‘jim’ }
#2 { name : ‘tim’ }
#3 { name : ‘bob’ }
Key-Value AccessTerm Count Referenc
e
tim 1 #1
is 2 #1, #2
sitting 2 #1, #2
next 2 #1, #2
to 2 #1, #2
bob 2 #1, #2
jim 1 #2
Inverted Index
©2015 Couchbase Inc. 12
What is it all about?
+ +
= awesome
Demohttps://github.com/dmaier-couchbase/cb-twitter-
demo
©2015 Couchbase Inc. 14
Demo Overview
0
1 2
3 4
©2015 Couchbase Inc. 15
Twitter Dashboard
Behind the Scenes
©2015 Couchbase Inc. 17
Reference Architecture
©2015 Couchbase Inc. 18
XDCR
Bucket A
Bucket B
Bucket C
Cluster 1
Bucket A
Bucket B
Bucket C
Cluster 2
Continuous Replication
Topology Aware Recovery from
Network Failure
©2015 Couchbase Inc. 19
The Couchbase Transport Plug-in
Needs to be installed on side of Elasticsearch
REST Service Per default on port 9091 XDCR protocol v1
bin/plugin -install transport-couchbase -url \http://packages.couchbase.com.s3.amazonaws.com/releases/elastic-search-adapter/2.0.0/elasticsearch-transport-couchbase-2.0.0.zip
I speak ‘Couchbase’!
©2015 Couchbase Inc. 20
Connecting Couchbase Server to Elasticsearch
©2015 Couchbase Inc. 21
Couchbase specific Types
©2015 Couchbase Inc. 22
Default Mapping Applies to every index couchbaseCheckpoint
The _source field contains the content of the document not the metadata
Don’t store or index Don’t make it accessible via the _all field
couchbaseDocument Handled by the overridden _default_ mapping The _source field contains the content of the
metadata (document id, ..) Don’t include the metadata in the _all field Index the content by using Elasticsearch’s
_default_ mapping Can be overridden by using Elasticsearch’s REST
interface
E = M(c)
©2015 Couchbase Inc. 23
Plug-in Settings - Document Type Selector
couchbase.typeSelector Fully qualified class name of a type selector class DefaultTypeSelector maps to ‘couchbaseDocument’ e.g. RegExp on the document id which matches
‘review-123’
couchbase.typeSelector: org.elasticsearch.transport.couchbase.capi.RegexTypeSelector couchbase.documentTypes.review: ^review-(\d)+$
©2015 Couchbase Inc. 24
Plug-in Settings - Advanced couchbase.port couchbase.username couchbase.password couchbase.num_vbuckets couchbase.defaultDocumentType couchbase.checkpointDocumentType couchbase.ignoreDeletes couchbase.wrapCounters …
What’s new in the 2.0 Plug-in
©2015 Couchbase Inc. 26
What’s new in the 2.0 Plug-in
Type Selectors Support of more than one document
type Support for filtering documents
Possibility to Ignore deletes Ignore Elasticsearch errors To automatically wrap Couchbase
counter documents with JSON for indexing purposes
Bug fixes
Did you know … ?
Thank you!Q&A
Top Related