Effective monitoring with StatsD

30
Effective Monitoring with

description

Best practices for monitoring your IT infrastructure using StatsD. Find dashboard examples here: https://p.datadoghq.com/sb/9b246c4ade Monitor StatsD easily with Datadog. Learn more at https://www.datadoghq.com

Transcript of Effective monitoring with StatsD

Page 1: Effective monitoring with StatsD

EffectiveMonitoring

with

Page 2: Effective monitoring with StatsD

@alqCTO at

Datadog

Page 3: Effective monitoring with StatsD

An applicationthrough the naked eye

Page 4: Effective monitoring with StatsD

An applicationthrough a monitoring

tool

Page 5: Effective monitoring with StatsD

OODA Loop (simplified)

Observe Orient

DecideAct

Page 6: Effective monitoring with StatsD

OODA Loop (simplified)

Observe Orient

DecideAct

Page 7: Effective monitoring with StatsD

OODA Loop (simplified)

Observe Orient

DecideAct

Monito

ring

Tool

Page 8: Effective monitoring with StatsD

OODA Loop (simplified)

Observe Orient

DecideAct

Monito

ring

Tool Yo

u

Page 9: Effective monitoring with StatsD

OODA Loop (simplified)

Observe Orient

DecideAct

Monito

ring

Tool Yo

u

You

Page 10: Effective monitoring with StatsD

OODA Loop (simplified)

Observe Orient

DecideAct

Monito

ring

Tool Yo

u

You

You

Page 11: Effective monitoring with StatsD

Observations need to be...

1.Timely2.Correct3.Comprehensive

Page 12: Effective monitoring with StatsD

Observations need to be...

1.Timely2.Correct3.Comprehensive

Page 13: Effective monitoring with StatsD

Observations need to be...

1.Timely2.Correct3.Comprehensive

Else

Page 14: Effective monitoring with StatsD

Observations need to be...

1.Timely2.Correct3.Comprehensive

Garbage In, Garbage Out

Else

Page 15: Effective monitoring with StatsD

Timely

Initial set of metrics

Initialassumptions

Revised set of metrics

Contact with reality

Revisedassumptions

Page 16: Effective monitoring with StatsD

Timely

Initial set of metrics

Initialassumptions

Revised set of metrics

Contact with reality

Revisedassumptions

Minutes

Not w

eeks

Page 17: Effective monitoring with StatsD

Comprehensive

WorkResources ValueResourcesResourcesResourcesResources

Page 18: Effective monitoring with StatsD

Comprehensive

WorkResources ValueResourcesResourcesResourcesResources

Easy to collectgeneric

but not actionable

Page 19: Effective monitoring with StatsD

Comprehensive

WorkResources ValueResourcesResourcesResourcesResources

Easy to collectgeneric

but not actionable

Harder to collect,custom

but most actionable

Page 20: Effective monitoring with StatsD

statsD

Easy

Page 21: Effective monitoring with StatsD

statsD

Easy

Timely

Page 22: Effective monitoring with StatsD

statsD

Easy

Timely Comprehensive

Page 23: Effective monitoring with StatsD

How statsD works

Client libraries talk to asimple UDP server...

pageviews:100|[email protected]:320|msbacklog:333|guniques:765|s

...using a simple text protocol

Page 24: Effective monitoring with StatsD

statsD typesType Definition Example

Gauges Absolute values Queue size

Counters Per-second rates Page views

Histograms Gauge summary Page Latency

Timers Gauge distribution Page Latency

Sets Counters of unique things Unique visitors

Page 25: Effective monitoring with StatsD

statsD problemsType Definition Problem

Gauges Absolute valuesLatest value wins.Gauge deltas???

Counters Per-second ratesRates, not counts (!

= rrdtool)

Histograms Gauge summaryAssumes normal

distribution

Timers Gauge distributionCan measure much

more than time

Sets Counters of unique things :-)

Page 26: Effective monitoring with StatsD

#1 pitfall: “Counters”

http://dtdg.co/tokyo-counters

Page 27: Effective monitoring with StatsD

How we use statsD

http://dtdg.co/tokyo-dog

Page 28: Effective monitoring with StatsD

Essential: Tagging

http://dtdg.co/tokyo-tags

Page 29: Effective monitoring with StatsD

How to get started

• statsD https://github.com/etsy/statsd• client libraries https://github.com/etsy/statsd/wiki

(my company) 1-stop shop http://www.datadoghq.com

Page 30: Effective monitoring with StatsD

ありがとうございました。質問?@alq

Thank you very much!Questions? @alq