Post on 26-Jan-2015
description
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Clinton Gormley @clintongormley
Scaling real time search and analytics with
elasticsearch
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
elasticsearch.org/guide
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
elasticsearch
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
elasticsearch• real-time
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
elasticsearch• real-time • distributed
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
elasticsearch• real-time • distributed • search
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
elasticsearch• real-time • distributed • search • analytics
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
how to use it?
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
how to use it?
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
how does it work?
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 1:making text searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
where content like “%darling%buds%”
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
slow & inflexible
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Term Doc 1 Doc 2 Doc 3breathebringsbudsbutbycan…damaskeddarlingdatedaydeafdeathdeclinesdelight
sorted list of unique terms
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Term Doc 1 Doc 2 Doc 3breathebringsbudsbutbycan…damaskeddarlingdatedaydeafdeathdeclinesdelight
where they occur
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Term Doc 1 Doc 2 Doc 3breathebringsbudsbutbycan…damaskeddarlingdatedaydeafdeathdeclinesdelight
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Term Doc 1 Doc 2 Doc 3breathebringsbudsbutbycan…damaskeddarlingdatedaydeafdeathdeclinesdelight
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index• term frequencies
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index• term frequencies » relevance
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index• term frequencies • text length
» relevance
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index• term frequencies • text length
» relevance » doc weight
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index• term frequencies • text length • term positions
» relevance » doc weight
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index• term frequencies • text length • term positions
» relevance » doc weight » word proximity
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index• term frequencies • text length • term positions • char offsets
» relevance » doc weight » word proximity
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index• term frequencies • text length • term positions • char offsets
» relevance » doc weight » word proximity » highlighting
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted indexnot just for text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted indexnumbers, dates, bools, enums
geopoints, geoshapes, etc
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 2:analytics
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
for searchmap values → doc_ids
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
for searchmap values → doc_ids
for analyticsmap doc_ids → values
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
uninvert the index
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
uninvert the indexcache values in memory
called “fielddata”
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
uninvert the indexdata access from RAM
very fast
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
on-the-fly analyticsin the context of
a user’s query
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
on-the-fly analyticsrelevant analytics
for each user
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
calculate metricscount, min, max, sum, avg,
percentiles, cardinality, stddev, variance, sum of squares
!
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
grouped bypopular terms, significant terms, ranges, dates, geolocation, etc
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
grouped bygroups can
… contain subgroups … which contain subgroups
etc
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 3:building the inverted index
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
immutable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
immutable• cache friendly
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
immutable• cache friendly • reads from RAM
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
immutable• cache friendly • reads from RAM • fielddata never changes
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
immutable• cache friendly • reads from RAM • fielddata never changes • compressible
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
immutable• cache friendly • reads from RAM • fielddata never changes • compressible • no locking
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
but, immutable…
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 4:dynamic inverted index
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
in-memory buffer
commit
segment
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
commit point
segment
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
commit point
segment
searchable
commit
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
commit point
segment
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
commit point
segment
searchable
commit
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
commit point
segment
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene commit
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene commit• write new segment
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene commit• write new segment • write new commit point
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene commit• write new segment • write new commit point • fsync
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene commit• write new segment • write new commit point • fsync • clear buffer
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene commit• write new segment • write new commit point • fsync • clear buffer • reopen index
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene commit• write new segment • write new commit point • fsync ← expensive! • clear buffer • reopen index
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 5:near real-time search
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
in-memory buffer
flush
segment
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
flush
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
flush
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchablecommit
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
commit point
segment
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene flush
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene flush• write new segment • clear buffer • reopen index !
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene flush• write new segment • clear buffer • reopen index • no fsync
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene flush• write new segment • clear buffer • reopen index • no fsync → lightweight
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
but…data not safe until fsync’ed!
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 6:don’t lose data
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 6:don’t lose data → transaction log
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
in-memory buffer
flush
segment
translog
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
translog
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
flush
translog
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
translog
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
translog
commit
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
translog
commit point
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
elasticsearch “refresh”• lucene “flush” • makes changes searchable • lightweight !
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
• lucene “commit” • clears transaction log • persists changes • heavy !
elasticsearch “flush”
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
refresh every second
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
near real-time search!
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
near real-time search!near real-time analytics!
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
but…
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
• slow searches • poor term frequencies • poor compression !
!
too many segments
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 7:reduce segments
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
merge process• many small → one big • removes deleted docs • runs in background • throttled
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
but…
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.“Any wonder it broke down” by Brian Snelson is licensed under CC BY 2.0
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
sometimes you need another truck
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 8:scale out, not up
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
shard your data
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
shard your datatransparent in elasticsearch
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
many segments
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
one shard
ss
many segments →
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
one shard
ss
many segments
ssssssss
many shards
ss
→
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
one shard
ss
many segments
one index
IIssssssss
many shards
ss
→
→
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
“node”running instance of elasticsearch
≈ one server
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
“shard”bucket of data
lives on one nodephysical worker unit
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
“index”logical namespace
points to one or more shards
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
“index”logical namespace
points to one or more shards
shard = hash(_id) % no_of_shards
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
PUT doc _id:1
hash(1) % 3 ⇒ shard_2
node_A
shard_0
node_B
shard_1
node_C
shard_2
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
GET doc _id:2
hash(2) % 3 ⇒ shard_0
node_A
shard_0
node_B
shard_1
node_C
shard_2
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Search all docs
shard = hash(_id) % no_of_shards
node_A
shard_0
node_B
shard_1
node_C
shard_2
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 9:scaling elastically
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
start smallnode_A
shard_0
shard_1
shard_2
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
add more nodesnode_A
shard_0
shard_1
shard_2
node_B node_C
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
shards migratenode_A
shard_0
shard_1
shard_2
node_B
shard_1
node_C
shard_2
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
rebalancednode_A
shard_0
node_B
shard_1
node_C
shard_2
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
add new indexnode_A
shard_0
shard_1
node_B
shard_1
shard_2
node_C
shard_0
shard_2
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
but…
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
but…more hardware?
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
but…more hardware?
more hardware failure
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
at 3am on sunday…node_A
shard_0
shard_1
node_B
shard_1
shard_2
node_C
shard_0
shard_2
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
boom!node_A
shard_0
shard_1
node_B
shard_1
shard_2
node_C
shard_0
shard_2
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 10:add redundancy
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
for every shard…make a copy
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
“primary shard”main shard
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
“replica shard(s)”copy of primary shard
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
one nodenode_A
P0
P1
P2
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
add a nodenode_A
P0
P1
P2
node_B
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
add a nodenode_A
P0
P1
P2
node_B
R0
R1
R2
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
redundancynode_A
P0
P1
P2
node_B
R0
R1
R2
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
add a nodenode_A
P0
P1
P2
node_B
R0
R1
R2
node_C
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
add a nodenode_A
P0
P1
P2
node_B
R0
R1
R2
R1
node_C
P0
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
rebalancednode_A
P0
P1
P2
node_B
R0
R1
R2
node_C
P0
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lose a nodenode_A
P0
P1
P2
node_B
R0
R2
R1
node_C
P0
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
replica ⇒ primarynode_A
P0
P1
P2
node_B
R0
R2
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
replica ⇒ primarynode_A
P0
P1
P2
node_B
P0
R2
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
allocate replicasnode_A
P0
P1
P2
node_B
P0
R2
R0
R1
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
rebalancednode_A
P0
P1
P2
node_B
P0
R2
R0
R1
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
primary shard• just a role • receives doc changes first • forwards new doc to replicas in parallel • number of primaries fixed
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
replica shard• copy of primary shard • serves read/search requests • number of replicas can be changed • more replicas → more read throughput
*if you have more hardware*
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
but…
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
but…who controls all this?
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 11:the master node
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.“Master Yoda” by Gonzalo Martín is licensed under CC BY-SA 2.0
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
“node”running instance of elastic search
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
“node”running instance of elastic search
node_A
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
“cluster”one or more nodes
with same cluster name working together
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
“cluster”
node_A node_B
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A node_B node_C
discover a clusterwith multicast/unicast
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A node_B node_C
discover a clusterwith multicast/unicast
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A node_B node_C
request routingsend request to any node
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A node_B node_C
request routingforwards to correct node
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
how?
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
how?every node knows where
every document is
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
cluster stateevery node knows where
every document is
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
cluster statecluster level information
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
cluster statecluster level information
indices ⇔ shards ⇔ nodes
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
cluster statecan only be updated by
the master node
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A
master nodeelected when cluster forms
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A node_B
master node
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A node_B
master node
node_C
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A node_B node_C
master node
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A node_B node_C
master nodejust a role
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A node_B node_C
master nodere-elected if master fails
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_B node_C
master node
node_A
re-elected if master fails
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_B node_C
master nodere-elected if master fails
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
master nodeonly manages
cluster level changes
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
master nodenot doc-level
get/put/search
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
the result?
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
distributed real-time
search & analytics
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
which works in the same way on your laptop…
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
…as on your 1,000 node cluster
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
who is using it?
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
• full text search • highlighted search snippets • search-as-you-type • did-you-mean suggestions
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
• combine visitor logs with social network data
• real-time feedback to editors
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
• combines full text search with geolocation
• uses more-like-this to find related questions and answers
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
• search repositories, users, issues, pull requests
• search 130 billion lines of code • track all alerts, events, logs
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
• index and analyse 5TB of log data every day
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
thank you@clintongormley
elasticsearch.org/downloadselasticsearch.com/support
elasticsearch.com/jobs