Integrating Elasticsearch for Real-Time Analytics with Kibana

Post on 14-Jun-2015

2.598 views 5 download

Tags:

description

Speaker: David Ostrovsky, Couchbase The Cross Datacenter Replication (XDCR) feature in Couchbase Server lets us synchronize data between different database clusters. One of the most interesting uses of XDCR is replicating data from Couchbase to ElasticSearch in near real-time. On its own, Couchbase is awesome; combining it with the search and analytics capabilities of ElasticSearch turns the awesomeness up to eleven! In this session, we’ll learn how to tie Couchbase and ElasticSearch together into a seamless data storage and analysis platform. We’ll examine the benefits that each brings to the table and how we can use the former for efficient storage and retrieval, and the latter for fast indexing, full-text searches, and geo-spatial querying. But mainly, we'll explore the most interesting use case of combining ElasticSearch with Couchbase: using Kibana to create dynamic, real-time data analytics dashboards on top of the data stored in Couchbase Server.

Transcript of Integrating Elasticsearch for Real-Time Analytics with Kibana

@DavidOstrovsky

Senior Architect @ SelaCo-author of:

“Pro Couchbase Server” – Apress Media 2014“Essentials of Developing Windows Store Apps Using C#”“Advanced Windows Store App Development Using C#”

Big data and NoSQL nerd

CodeHardBlog.azurewebsites.net

What’s up?

Why Couchbase and ElasticSearch?How do I connect the two?What can I do with this?Does it scale?

Why use Couchbase Server?

If you don’t know this, you might be in the wrong room.

Why use ElasticSearch?

Fast full-text searchesSupports over a hundred languages

Geo-querying, scoring, facetingEasy to deploy, use and scaleHuge number of plug-insHigh buzzword count

Why these two in particular?

Both use JSON documentsComplimenting strengths:

ElasticSearch for flexible search, geolocation, analytics tools, and moreCouchbase for efficient data storage and retrieval, extremely fast response times

Couchbase uses XDCR to push data to ElasticSearchLower latency than polling, e.g. with River

Demo

Demo time!

Common use cases

Ad-hoc querying an existing Couchbase databaseGeospatial searches on dataReal-time analytics of Couchbase data

Demo

More demo time!

Putting it all together

A typical Couchbase-ElasticSearch deployment uses a 2:1 or 3:1 ratio of nodes

ElasticSearch only stores indexed dataSource data stored in CouchbaseAll CRUD operations go to Couchbase

Kibana is a dashboard for real-time visualization of streaming dataTalks directly to ElasticSearchDrag-and-drop dashboard construction

No coding required

Live analytics with Kibana

CouchbaseElasticSearch KibanaServer ClientCouchbase

Couchbase

Demo

So many demos!

Summary

Connecting Couchbase to ElasticSearch and Kibana is awesome!Lets developers and analysts focus on what each does bestProvides a resilient, scalable data storage and BI solution

Questions