Productionalizing Elasticsearch at Scale

About me

Myself Jeeva, enthusiastic engineer, working as technical lead in Paypal.

I have been a backbone in designing and implementing a highly scalable system that powers the elastic search engine to process billions of records every day.

Having intense hands on experience on multiple cutting-edge technologies like Elasticsearch, Kafka, Akka, Scala and Reactive programming, I provide excellent expertise in building resilient distributed systems.

Contact me at subramaniajeeva [gmail, twitter, fb, linkedin]

Agenda

• Elasticsearch - How it works

• Scaling Elasticsearch in production

• Resilient by design

• Do’s and Don’ts

• Monitoring and Alerting

• Summary

Elasticsearch – Data Organization

Shard 0

Shard 1 Replica

Shard 1

Shard 0 Replica

Node 2 Node 1

cluster Index

Disk xx GB Disk xx GB

Shard 0 Primary

Elasticsearch – Routing

Shard 1 Replica

Shard 1 Primary

Shard 0 Replica

Node 2 Node 1

Indexing request Document A

Index document A to shard1

Replicate document A

{"text": "brown fox jumped into the well"}

Dissecting Indexing

The <em>Quick brown</em> fox jumped over the lazy dog

The Quick brown fox jumped over the lazy dog

The two <em>lazy</em> dogs were slower than the less lazy <em>dog</em>, Rover

The two lazy dogs were slower than the less lazy dog, Rover

Source Text

html_strip filter

Standard tokenizer

Stop words filter

Lower case filter

The Quick jumped

over

fox brown

the lazy dog

the quick jumped

over

fox brown

the lazy dog

than

slower were two The lazy dogs

dog the less lazy Rover

than

slower were two the lazy dogs

dog the less lazy rover

quick jumped

over

fox brown

lazy dog

slower two lazy dogs

dog less lazy rover

Term Document

quick 1

brown 1

fox 1

jumped 1

over 1

lazy 1, 2

dog 1, 2

two 2

dogs 2

slower 2

less 2

rover 2

Document 1 Document 2 Inverted Index

Dissecting Search

The lazy Fox

The lazy Fox html_strip

filter

Standard tokenizer

Stop words filter

Lower case filter

Fox lazy The

Term Document

quick 1

brown 1

fox 1

jumped 1

over 1

lazy 1, 2

dog 1, 2

two 2

dogs 2

slower 2

less 2

rover 2

Search for “The lazy Fox”

Inverted Index

fox lazy the

fox lazy

lazy

fox

matches document 1 and 2

matches document 1

Scaling Elasticsearch in production

•

•

•

•

# Transactions Latency

< 10 ms

Resiliency

•

•

•

•

•

Back-pressure

Kafka to ES stream

Kafka Source Transformer Bulkizer ES client Sink

pull pull pull pull

Back-pressure

Kafka to ES stream

Kafka Source Transformer Bulkizer ES client Sink

push push push push

pull pull pull

Elasticsearch

5.6 6.2

•

•

Do

•

•

•

•

•

Monitoring app

ES metrics

C1 C2 C3 C4

Monitoring ES

Monitoring and Alerting

Summary

•

•

•

•

•

Use cases

•

•

•

•

•

•

References

•

•

•

•

•