Logs Management

infrastructure

logs management

logflow

logstash

Logstash tool for managing events and logs

Logstash written in JRuby

logstash inputs -> filters -> outputs

logstash “all in one” jar file

logstash plugins

Inputseventlog, file, s3, syslog, tcp, udp, websocket, wmi

Filterscsv, dns, geoip, grok, mutate

Outputselasticsearch, email, exec, mongodb, rabbitmq, redis

example

http://logstash.net/docs/1.2.2/

logstash example

input {stdin{

type => “example“ }}

with puppet

=

logstash example

output {redis {

data_type => "list" host => [’10.0.22.26'] key => "logstash”

}}

with puppet

=

logflow

redis

Redis open source, advanced key-value store

Redis buffer new logs from any type of slower parsing

logflow

logflow

logstash example

input {syslog {

port => 5544 type => "hapr" }}

with puppet

=

log4net example

logflow

logstash example

=output {

elasticsearch {cluster => "logs”embedded => false

index => "%{type}-%{+YYYY.MM.dd}”

node_name => "logstash_output" }}

with puppet

logflow

elasticsearch

ES distributed restful search and analytics engine

ES build on top of apache lucene

ES distributed, highly available

ES document oriented, schema free

ES restfull api

elasticsearchData Structure

http://stackoverflow.com/questions/15694724/shards-and-replicas-in-elasticsearch

Index

shards

replica

elasticsearchData Structure

http://stackoverflow.com/questions/15694724/shards-and-replicas-in-elasticsearch

Index

shards

replica

elasticsearchData Structure

http://stackoverflow.com/questions/15694724/shards-and-replicas-in-elasticsearch

Index

shards

replica

elasticsearchDemo

http://logs.adform.com:9200/_plugin/bigdeskhttp://logs.adform.com:9200/_plugin/paramedic/

logflow

kibana 3

kibana: HTML + JavaScript  kibana: analytics and search interface to timestamped data sets stored in ElasticSearch

kibana: browser connects directly to ElasticSearch

kibana 3http://logs.adform.com/kibana

logflow

logflow

logflow

statsDstatsD: NodeJS daemon

statsD: extarcts metrics data and flushes to backend

statsD: counters, timers, gouges

450120553994334844675496

=>

mean_90 496upper_90 844

sum_90 3472upper 994lower 120count 8sum 4466mean 558.2

timers example:

http://blog.pkhamre.com/2012/07/24/understanding-statsd-and-graphite/

statsD

.NET 4.0 : https://github.com/robbihun/NStatsD.Client

C#: https://github.com/goncalopereira/statsd-csharp-client

.NET: https://github.com/peschuster/graphite-client

.NET 3.5 – 4.5: http://www.nuget.org/packages/StatsdCsharpClient/

https://github.com/etsy/statsd/wiki

clients

statsD

NStatsD.Client.Current.Increment("testing.increment");NStatsD.Client.Current.Decrement("testing.decrement");NStatsD.Client.Current.Timing("testing.timing", 2345);NStatsD.Client.Current.Gauge("testing.gauge", 45);

https://github.com/robbihun/NStatsD.Client

example

logflow

graphite

graphite: highly scalable real-time graphing system graphite: good when you don’t know the names metrics

graphite

DEMO

web

graphite

.NET: https://github.com/peschuster/graphite-client

• WCF• MSBuild• ELMAH• SQL Server• PerfCounterMonitor.exe (Graphite.System)

• Performance counters• Event log• IIS Application Pools (with)

clients

what's next

what's next

LEARN!!!

what's nextdashboards

http://techblog.netflix.com/2012/12/hystrix-dashboard-and-turbine.htmlhttp://shopify.github.io/dashing/#widgetshttp://fdietz.github.io/team_dashboard/https://github.com/ripienaar/gdash