Logstash - CeBIT 2014 - Open Source Forum

www.netways.de // blog.netways.de // @netways

We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310

CEBIT 2014 – 12.03.2014

LOG- UND EVENTMANAGEMENT

MIT LOGSTASH

BERND ERK | NETWAYS

GMBH



AGENDA

￭ Kurzvorstellung

￭ Einführung

￭ Architektur

￭ Installation

￭ Routing und Filterung von Events

￭ Interfaces & API

￭ Integration in Nagios und Icinga

￭ Fragen & Antworten



KURZVORSTELLUNG



KURZVORSTELLUNG NETWAYS

• Firmengründung 1995

• Open Source seit 1997

• 40 Mitarbeiter

• Spezialisierung in den

Bereichen Open Source

Systems Management und

Open Source Datacenter

Infrastructure

http://jobs.netways.de



NETWAYS KOMPETENZEN

• Monitoring & Reporting

• Configuration Management

• Service Management

• Knowledge Management

• Backup & Recovery

• High Availability & Clustering

• Cloud Computing

• Load Balancing

• Virtualization

• Database Management

OPEN SOURCESYSTEMS MANAGEMENT

OPEN SOURCEDATA CENTER

MANAGED SERVICES

MONITORING HARDWARE

KONFERENZEN



Open Source Datacenter

Conference

• 08. – 10. April 2014

• Datacenter | Automation | DevOps

PuppetCamp 2014

• 11. April Berlin

Open Source Monitoring

Conference

• 11. April Berlin

NETWAYS KONFERENZEN



EINFÜHRUNG



LOGS

Logs -> Fluss an unstrukturierten Daten

Oct 4 16:57:24 web sshd[25828]: Received disconnect from 10.10.0.31: 11:

disconnected by user

bestehend aus Timestamp und Message



EVENTS

Event -> Fluss an strukturierten Daten

Event {

Time: Oct 4 16:57:24

Process: sshd

State: Received disconnect from 10.10.0.31

Client: 10.10.0.31

}

bestehend aus konkreten Attributen



LOG & EVENTMANAGEMENT

Logs > Event > Analyse (Korrelation) > Aktion



TOOLS

￭ Nagios & Icinga Addons• check_logfiles• NagTrap• EventDB• EDBC

￭ Logmanagement-Tools• Graylog• Fluentd• Logstash



LOGSTASH

Logstash



ARCHITEKTUR & INSTALLATION



LOGSTASH

￭ Logmanagement auf Basis von JRuby

￭ Konfigurierbare “Pipe”

￭ Flexible Plugin-Architektur für• Input• Filter• Output

￭ Standardplugins für alle gängige Protokolle

￭ Webinterface

￭ Single File Deployment



Outputs• amqp

• boundary

• circonus

• cloudwatch

• datadog

• datadog_metrics

• elasticsearch

• elasticsearch_http

• elasticsearch_river

• email

• exec

• file

• ganglia

• gelf

• gemfire

• google_cloud_storage

• graphite

• graphtastic

• hipchat

LOGSTASH - IO

Inputs• amqp

• drupal_dblog

• elasticsearch

• eventlog

• exec

• file

• ganglia

• gelf

• gemfire

• generator

• graphite

• heroku

• imap

• irc

• log4j

• lumberjack

• pipe

• rabbitmq

• redis

• relp

• s3

• snmptrap

• sqlite

• sqs

• stdin

• stomp

• syslog

• tcp

• twitter

• udp

• unix

• varnishlog

• websocket

• wmi

• xmpp

• zenoss

• zeromq

• http

• irc

• jira

• juggernaut

• librato

• loggly

• lumberjack

• metriccatcher

• mongodb

• nagios

• nagios_nsca

• null

• opentsdb

• pagerduty

• pipe

• rabbitmq

• redis

• riak

• riemann

• s3

• sns

• sqs

• statsd

• stdout

• stomp

• syslog

• tcp

• udp

• websocket

• xmpp

• zabbix

• zeromq



INSTALLATION - LOGSTASH

￭ Download - http://logstash.net

￭ java -jar logstash-x.x.x-flatjar.jar agent -f <config-

file>



ARCHITEKTUR

Shipper

Shipper

Shipper

Broker Search & Storage WebinterfaceIndexer



REDIS

￭ NoSQL in memory auf Basis von C

￭ Unterstützung verschiedener “Datentypen”• strings• hashes• lists• sets and sorted sets

￭ Support für verschiedene Replikationsszenarien

￭ SAUSCHNELL$ ./redis-benchmark -r 1000000 -n 2000000 -t get,set,lpush,lpop -qSET: 122556.53 requests per secondGET: 123601.76 requests per secondLPUSH: 136752.14 requests per secondLPOP: 132424.03 requests per second



INSTALLATION - REDIS

￭ Download - http://redis.io/download

￭ make

￭ make test

￭ make install

￭ /usr/local/bin/redis-server



ELASTICSEARCH

￭ Schemafreier RESTful Suchserver auf Basis von

Java

￭ Basierend auf Lucene Core

￭ “Vergleichbar” mit Apache Solr

￭ Verteilte Architektur durch• Shards• Replicas• Gateways

￭ Realtime-Suche als Basis für Kibana



INSTALLATION - ELASTICSEARCH

￭ Download – http://elasticsearch.org/download/

￭ Entpacken des Archives

￭ Ausführung von bin/elasticsearch



ROUTING UND FILTERUNG VON EVENTS



ÜBERSICHT

Shipper

Shipper

Shipper




KONFIGURATION - LOGSTASH - SHIPPER

￭ Übermittlung von Logs an Logstash• Logstash• Lumberjack• Syslog• Log4J• Gelf• File-Read• u.v.a.m.



KONFIGURATION - LOGSTASH - SHIPPER

￭ Konfigurationinput { file { path => "/root/osmc/demodata/access.log.1” type => "apache-access" }}output { stdout {

debug => true } redis { host => "127.0.0.1" data_type => "list" key => "logstash.apache" }}

￭ java -jar logstash-current.jar agent -f

logstash_shipper.conf

Shipper

Shipper

Shipper




KONFIGURATION - LOGSTASH - INDEXER

￭ Konfigurationinput { redis { host => "127.0.0.1" type => "redis-input" # these settings should match the output of the agent data_type => "list" key => "logstash.apache” }}output { stdout { debug => true } elasticsearch_http { host => "127.0.0.1" }}

Shipper

Shipper

Shipper




KONFIGURATION - LOGSTASH – INDEXER - APACHE

￭ Konfiguration für Apache-Logsinput { redis { host => "127.0.0.1" type => "apache-access” data_type => "list" key => "logstash.apache” format => "json_event" }}filter { if [type] == "apache-access" { grok { match => [ "message", "%{COMBINEDAPACHELOG}" ] } }}output { elasticsearch_http { host => "127.0.0.1” }}

Shipper

Shipper

Shipper




KONFIGURATION - LOGSTASH – INDEXER - GEOIP

￭ Konfiguration für Geo-Dateninput { redis { host => "127.0.0.1" type => "apache-access” data_type => "list" key => "logstash.apache” }}filter { grok { type => "apache-access" pattern => "%{COMBINEDAPACHELOG}" } geoip { source => "clientip" add_tag => ["geotag"] }}output { elasticsearch_http {host => "127.0.0.1”}}

Shipper

Shipper

Shipper




INTERFACES & API



KIBANA

Kibana



KIBANA



ELASTICHQ



KIBANA - DEMO

DEMO



INTEGRATION NAGIOS UND ICINGA



REALTIME LOGANALYSE

￭ Analyse verschiedener Quellen in Realtime

￭ Prüfung auf Patterns und States• Facilitites• Regex• Programs

￭ Übermittlung als Passiver Event



ÜBERSICHT LOGSTASH UND ICINGA

Search & Storage WebinterfaceIndexer

Icinga –WebIcinga - Commandpipe



KONFIGURATION - LOGSTASH – INDEXER - ICINGA

￭ Konfiguration für Icinga-Alertinput { …}

filter { if [type] == "syslog" { grok {match => [ "message", "%{SYSLOGBASE}" ] } grep { match => [ "message", "Error" ] drop => false add_tag => "nagios-update" add_field => [ # "nagios_host", "%{@source_host}", "nagios_host", "localhost", "nagios_service", "Logstash", "nagios_level", "2”] }}}

output { elasticsearch {host => "127.0.0.1”} nagios { commandfile => "/var/lib/icinga/rw/icinga.cmd" }}

Shipper

Shipper

Shipper




LOGSTASH – ICINGA - DEMO

DEMO



ZUGABE



REALTIME GRAPHING



STATSD & GRAPHITE

￭ StatsD• Netzwerkdaemon auf Basis von UDP• Bucket -> Value -> Flush• Entkoppelte Zwischenaggretion für Statisik

￭ Graphite• Graphing-Framework bestehend aus• Whisper (Datenbank)• Carbon (Engine)• Graphite-Web (Interface)



INSTALLATION – STATSD - NODEJS

￭ apt-get install make python g++ checkinstall

￭ mkdir nodejs && cd nodejs

￭ wget -N http://nodejs.org/dist/node-latest.tar.gz

￭ tar xzvf node-latest.tar.gz && cd `ls -rd node-v*`

￭ checkinstall



INSTALLATION – STATSD

￭ wget

https://github.com/etsy/statsd/archive/master.zip

￭ unzip master.zip

￭ node stats.js config.js



MONITORING - STATSD

￭ Status Informationen• echo stats | nc 127.0.0.1 8126• echo health | nc 127.0.0.1 8126

￭ Timer- und Counterinfo• echo counters | nc 127.0.0.1 8126• echo timers| nc 127.0.0.1 8126



INSTALLATION – GRAPHITE

￭ Download der Sources• git clone https://github.com/graphite-

project/graphite-web.git• git clone https://github.com/graphite-

project/carbon.git• git clone https://github.com/graphite-

project/whisper.git



INSTALLATION – GRAPHITE

￭ Installation Whisperpushd whispersudo python setup.py installpopd

￭ Installation Carbonpushd carbonsudo python setup.py installpopd

￭ Konfiguration Carbonpushd /opt/graphite/confcp carbon.conf.example carbon.confcp storage-schemas.conf.example storage-schemas.conf



INSTALLATION – GRAPHITE - WEBAPP

￭ Check Dependencies Graphite webapppushd graphite-webpython check-dependencies.pypopd

￭ Installation Graphite webapppushd graphite-webpython setup.py installpopd

￭ Konfiguration Apacheexample-graphite-vhost.conf



ÜBERSICHT STATSD UND GRAPHITE

Search & Storage WebinterfaceIndexer

GraphiteStatsd



KONFIGURATION - LOGSTASH – INDEXER - STATSD

￭ Konfiguration für Statsdinput { redis { host => "127.0.0.1" type => "apache-access” data_type => "list" key => "logstash.apache” format => "json_event” add_field=> ["sitename","www.icinga.org"] }}filter { if [type] == "apache-access" { grok {match => [ "message", "%{COMBINEDAPACHELOG}" ] } }}output { stdout { debug => true } if [type] == "apache-access" { statsd { host => "localhost" port => 8125 namespace => "logstash" debug => false increment => "apache.%{sitename}.response.%{response}” count => ["apache.%{sitename}.bytes", "%{bytes}"] } } elasticsearch_http {host => "127.0.0.1”}}

Shipper

Shipper

Shipper


StatsD



GRAPHITE - DEMO

DEMO



FRAGEN & ANTWORTEN



NETWAYS GmbH

Deutschherrnstrasse 15-19

90429 Nürnberg

Tel: +49 911 92885-0

Fax: +49 911 92885-77

Email: [email protected]

Website: www.netways.de

Twitter: twitter.com/netways

Facebook:

facebook.com/netways

Blog: blog.netways.de

FRAGEN & ANTWORTEN

DANKE

mailto:[email protected]

mailto:[email protected]

Logstash - CeBIT 2014 - Open Source Forum

Technology

Transcript of Logstash - CeBIT 2014 - Open Source Forum