Elasticsearch for Logs & Metrics - a deep dive

48
Elasticsearch for logs and metrics (a deep dive) Rafał Kuć and Radu Gheorghe Sematext Group, Inc.

Transcript of Elasticsearch for Logs & Metrics - a deep dive

Page 1: Elasticsearch for Logs & Metrics - a deep dive

Elasticsearch for logs and metrics (a deep dive)

Rafał Kuć and Radu GheorgheSematext Group, Inc.

Page 2: Elasticsearch for Logs & Metrics - a deep dive

About us

LogseneSPM

ES API

metrics

...

Products Services

Page 3: Elasticsearch for Logs & Metrics - a deep dive

Agenda

Index layout

Cluster layout

Per-index tuning of settings and mappings

Hardware+OS options

Pipeline patterns

Page 4: Elasticsearch for Logs & Metrics - a deep dive

Daily indices are a good start

...indexing, most searches

Indexing is faster in smaller indices

Cheap deletes

Search only needed indices

“Static” indices can be cached

Page 5: Elasticsearch for Logs & Metrics - a deep dive

The Black Friday problem*

* for logs. Metrics usually don’t suffer from this

Page 6: Elasticsearch for Logs & Metrics - a deep dive

Typical indexing performance graph for one shard*

* throttled so search performance remains decent

At this point it’s better to index in a new shard

Typically 5-10GB, YMMV

Page 7: Elasticsearch for Logs & Metrics - a deep dive

INDEX

Y U NO AS FAST

Page 8: Elasticsearch for Logs & Metrics - a deep dive

INDEX

Y U NO AS FAST

more merges

more expensive

(+uncached) searches

Mostly because

Page 9: Elasticsearch for Logs & Metrics - a deep dive

Rotate by size*

* use Field Stats for queries or rely on query cache:https://github.com/elastic/kibana/issues/6644

Page 10: Elasticsearch for Logs & Metrics - a deep dive

Aliases; Rollover Index API*

* 5.0 feature

Page 11: Elasticsearch for Logs & Metrics - a deep dive

Slicing data by time

For spiky ingestion, use size-based indices

Make sure you rotate before the performance drop(test on one node to get that limit)

Page 12: Elasticsearch for Logs & Metrics - a deep dive

Multi tier architecture (aka hot/cold)

Client

Client

Client

Data

Data

Data

...

Data

Data

Data

Master

Master

Master

We can optimize data nodes layer

Ingest

Ingest

Ingest

Page 13: Elasticsearch for Logs & Metrics - a deep dive

Multi tier architecture (aka hot/cold)

logs_2016.11.07

indexing

es_hot_1 es_cold_1 es_cold_2

Page 14: Elasticsearch for Logs & Metrics - a deep dive

Multi tier architecture (aka hot/cold)

logs_2016.11.07logs_2016.11.08

indexing

move

es_hot_1 es_cold_1 es_cold_2

curl -XPUT localhost:9200/logs_2016.11.07/_settings -d '{

"index.routing.allocation.exclude.tag" : "hot",

"index.routing.allocation.include.tag": "cold"

}'

Page 15: Elasticsearch for Logs & Metrics - a deep dive

Multi tier architecture (aka hot/cold)

logs_2016.11.08 logs_2016.11.07

indexing

es_hot_1 es_cold_1 es_cold_2

Page 16: Elasticsearch for Logs & Metrics - a deep dive

Multi tier architecture (aka hot/cold)

logs_2016.11.11 logs_2016.11.07logs_2016.11.09

logs_2016.11.08logs_2016.11.10

indexing, most searches long running searches

good CPU, best possible IO heap, IO for backup/replication and stats

es_hot_1 es_cold_1 es_cold_2

SSD or RAID0 for spinning

Page 17: Elasticsearch for Logs & Metrics - a deep dive

Hot - cold architecture summary

Costs optimization - different hardware for different tier

Performance - above + fewer shards, less overhead

Isolation - long running searches don't affect indexing

Page 18: Elasticsearch for Logs & Metrics - a deep dive

Elasticsearch high availability & fault tolerance

Dedicated masters is a mustdiscovery.zen.minimum_master_nodes = N/2 + 1

Keep your indices balancednot balanced cluster can lead to instability

Balanced primaries are also goodhelps with backups, moving to cold tier, etc

total_shards_per_node is your friend

Page 19: Elasticsearch for Logs & Metrics - a deep dive

Elasticsearch high availability & fault tolerance

When in AWS - spread between availability zonesbin/elasticsearch -Enode.attr.zone=zoneAcluster.routing.allocation.awareness.attributes: zone

We need headroom for spikesleave at least 20 - 30% for indexing & search spikes

Large machines with many shards?look out for GC - many clusters died because of thatconsider running smaller ES instances but more

Page 20: Elasticsearch for Logs & Metrics - a deep dive

Which settings to tune

Merges → most indexing time

Refreshes → check refresh_interval

Flushes → normally OK with ES defaults

Page 21: Elasticsearch for Logs & Metrics - a deep dive

Relaxing the merge policyLess merges ⇒ faster indexing/lower CPU while indexingSlower searches, but:

- there’s more spare CPU- aggregations aren’t as affected, and they are typically the bottleneck

especially for metricsMore open files (keep an eye on them!)

Increase index.merge.policy.segments_per_tier ⇒ more segments, less mergesIncrease max_merge_at_once, too, but not as much ⇒ reduced spikesReduce max_merged_segment ⇒ no more huge merges, but more small ones

Page 22: Elasticsearch for Logs & Metrics - a deep dive

And even more settingsRefresh interval (index.refresh_interval)*

- 1s -> baseline indexing throughput- 5s -> +25% to baseline throughput- 30s -> +75% to baseline throughput

Higher indices.memory.index_buffer_size higher throughput

Lower indices.queries.cache.size for high velocity data to free up heap

Omit norms (frequencies and positions, too?)

Don't store fields if _source is used

Don't store catch-all (i.e. _all) field - data copied from other fields

* https://sematext.com/blog/2013/07/08/elasticsearch-refresh-interval-vs-indexing-performance/

Page 23: Elasticsearch for Logs & Metrics - a deep dive

Let’s dive deeper into storage

Not searches on a field, just aggregations ⇒ index=false

Not sorting/aggregating on a field ⇒ doc_values=false

Doc values can be used for retrieving (see docvalue_fields), so:

● Logs: use doc values for retrieving, exclude them from _source*

● Metrics: short fields normally ⇒ disable _source, rely on doc values

Long retention for logs? For “old” indices:

● set index.codec=best_compression

● force merge to few segments

* though you’ll lose highlighting, update API, reindex API...

Page 24: Elasticsearch for Logs & Metrics - a deep dive

Metrics: working around sparse dataIdeally, you’d have one index per metric type (what you can fetch with one call)

Combining them into one (sparse) index will impact performance (see LUCENE-7253)

One doc per metric: you’ll pay with space

Nested documents: you’ll pay with heap (bitset used for joins) and query latency

Page 25: Elasticsearch for Logs & Metrics - a deep dive

What about the OS?

Say no to swap

Disk scheduler: CFQ for HDD, deadline for SSD

Mount options: noatime, nodiratime, data=writeback, nobarrier

because strict ordering is for the weak

Page 26: Elasticsearch for Logs & Metrics - a deep dive

And hardware?

Hot tier. Typical bottlenecks: CPU and IO throughputindexing is CPU-intensiveflushes and merges write (and read) lots of data

Cold tier: Memory (heap) and IO latencymore data here ⇒ more indices&shards ⇒ more heap

⇒ searches hit more filesmany stats calls are per shard ⇒ potentially choke IO when cluster is idle

Generally:network storage needs to be really good (esp. for cold tier)network needs to be low latency (pings, cluster state replication)network throughput is needed for replication/backup

Page 27: Elasticsearch for Logs & Metrics - a deep dive

AWS specifics

c3 instances work, but there’s not enough local SSD ⇒ EBS gp2 SSD*c4 + EBS give similar performance, but cheaper

i2s are good, but expensived2s are better value, but can’t deal with many shards (spinning disk latency)m4 + gp2 EBS are a good balance

gp2 → PIOPS is expensive, spinning is slow3 IOPS/GB, but caps at 160MB/s or 10K IOPS (of up to 256kb) per driveperformance isn’t guaranteed (for gp2) ⇒ one slow drive slows RAID0

Enhanced Networking (and EBS Optimized if applicable) are a must

* And used local SSD as cache. With --cachemode writeback for async writing: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Logical_Volume_Manager_Administration/lvm_cache_volume_creation.html

block size?

Page 28: Elasticsearch for Logs & Metrics - a deep dive

The pipeline

read buffer deliver

Page 29: Elasticsearch for Logs & Metrics - a deep dive

The pipeline

read buffer deliver

Log shipper reason #1

Page 30: Elasticsearch for Logs & Metrics - a deep dive

The pipeline

read buffer deliver

Log shipper reason #1

Files? Sockets? Network?What if buffer fills up?

Processing before/after buffer?

How?

Others besides Elasticsearch?How to buffer if $destination is down?

Overview of 6 log shippers: sematext.com/blog/2016/09/13/logstash-alternatives/

Page 31: Elasticsearch for Logs & Metrics - a deep dive

Types of buffers

buffer

application.log Log file can act as a buffer

Memory and/or disk of the log shipperor a dedicated tool for buffering

Page 32: Elasticsearch for Logs & Metrics - a deep dive

Where to do processing

Logstash(or Filebeat or…)

Buffer(Kafka/Redis)

here

Logstash Elasticsearch

Page 33: Elasticsearch for Logs & Metrics - a deep dive

Where to do processing

Logstash Buffer(Kafka/Redis)

here

Logstash Elasticsearch

something else

Page 34: Elasticsearch for Logs & Metrics - a deep dive

Where to do processing

Logstash Buffer(Kafka/Redis)

here

Logstash Elasticsearch

something elseOutputs

need to be in sync

Page 35: Elasticsearch for Logs & Metrics - a deep dive

Where to do processing

Logstash Kafka Logstash Elasticsearch

something elseLogstashElasticsearch

offsetotheroffset

here

here,too

Page 36: Elasticsearch for Logs & Metrics - a deep dive

Where to do processing (syslog-ng, fluentd…)

input

hereElasticsearch

something else

here

Page 37: Elasticsearch for Logs & Metrics - a deep dive

Where to do processing (rsyslogd…)

input

herehere

here

Page 38: Elasticsearch for Logs & Metrics - a deep dive

Zoom into processing

Ideally, log in JSON

Otherwise, parse

For performance and maintenance(i.e. no need to update parsing rules)

Regex-based (e.g. grok)Easy to build rules

Rules are flexible

Slow & O(n) on # of rules

Tricks:

Move matching patterns to the top of the list

Move broad patterns to the bottom

Skip patterns including others that didn’t match

Grammar-based(e.g. liblognorm, PatternDB)

Faster. O(1) on # of rules. References:

Logagent

Logstash

rsyslog syslog-ng

sematext.com/blog/2015/05/18/tuning-elasticsearch-indexing-pipeline-for-logs/www.fernuni-hagen.de/imperia/md/content/rechnerarchitektur/rainer_gerhards.pdf

Page 39: Elasticsearch for Logs & Metrics - a deep dive

Back to buffers: check what happens if when they fill up

Local files: when are they rotated/archived/deleted?

TCP: what happens when connection breaks/times out?

UNIX sockets: what happens when socket blocks writes?

UDP: network buffers should handle spiky load

Check/increasenet.core.rmem_max net.core.rmem_default

Unlike UDP&TCP,both DGRAM and STREAM

local socketsare reliable/blocking

Page 40: Elasticsearch for Logs & Metrics - a deep dive

Let’s talk protocols now

UDP: cool for the app, but not reliable

TCP: more reliable, but not completely

Application-level ACKs may be needed:

No failure/backpressure handling needed

App gets ACK when OS buffer gets it ⇒ no retransmit if buffer is lost*

* more at blog.gerhards.net/2008/05/why-you-cant-build-reliable-tcp.html

sender receiverACKs

Protocol Example shippers

HTTP Logstash, rsyslog, syslog-ng, Fluentd, Logagent

RELP rsyslog, Logstash

Beats Filebeat, Logstash

Kafka Fluentd, Filebeat, rsyslog, syslog-ng, Logstash

Page 41: Elasticsearch for Logs & Metrics - a deep dive

Wrapping up: where to log?

critical?

UDP. Increase network buffers on destination, so it can handle spiky

traffic

Paying with RAM or IO?

UNIX socket. Local shipper with memory

buffers, that can drop data if needed

Local files. Make sure rotation is in place or you’ll run out of disk!

no

IO RAM

yes

Page 42: Elasticsearch for Logs & Metrics - a deep dive

Flow patterns (1 of 5)

application.log

application.log

Logstash

Logstash

Elasticsearch

Easy&flexible

Overhead

Page 43: Elasticsearch for Logs & Metrics - a deep dive

Flow patterns (2 of 5)

application.log

application.log

Filebeat

Filebeat

Elasticsearch(with Ingest)

Light&simple

Harder to scale processing

sematext.com/blog/2016/04/25/elasticsearch-ingest-node-vs-logstash-performance/

Page 44: Elasticsearch for Logs & Metrics - a deep dive

Flow patterns (3 of 5)

Elasticsearch

files,sockets (syslog?),localhost TCP/UDP

LogagentFluentdrsyslog

syslog-ng

Light, scales

No central control

sematext.com/blog/2016/09/13/logstash-alternatives/

Page 45: Elasticsearch for Logs & Metrics - a deep dive

Flow patterns (4 of 5)

ElasticsearchKafka

FilebeatLogagentFluentdrsyslog

syslog-ng

Good for multiple destinations

More complex

somethingelse

Logstash,custom consumer

Page 46: Elasticsearch for Logs & Metrics - a deep dive

Flow patterns (5 of 5)

Page 47: Elasticsearch for Logs & Metrics - a deep dive

Thank you!

Rafał Kuć[email protected]@kucrafal

Radu [email protected]@radu0gheorghe

[email protected]://sematext.com@sematext

Join Us! We are hiring!

http://sematext.com/jobs