Download - Don’t turn your logs into cuneiform

Don’t turn your logs into cuneiform

Andrei Rebrov

Cuneiform

What’s the difference?

Let’s dig into

• Too much logs• Too much information inside• They are distributed across several

machines• We are not supposed to read blogs

Time for tools

• Open source

• Collects and parses

Logstash

http://logstash.net

Key feature

Too many sources?•syslog•nginx access log•application logs•database logs

How about their format?

З parts of Logstash• Inputs

•Filters

•Outputs

http://logstash.net/docs/1.1.12/

Inputs• file• eventlog• ganglia• heroku• syslog• tcp

Filters


• anonymize• date• mutate• grok• grep



Outputs• file• graphite• http• irc• email• zabbix

input { stdin { type => "stdin-type"} }output { stdout { debug => true debug_format =>"json"} }

java -jar logstash-1.1.9-monolithic.jar agent -flogstash-simple.conf

Easy to adopt

Example

input { stdin { type => "stdin-type"} }filter { grok { type => "stdin-type" pattern =>"Hello %{DATA:message}!" } }output { stdout { debug => true debug_format =>"json"} }

java -jar logstash-1.1.9-monolithic.jar agent -flogstash-simple.conf

Time to parse

Example

input { stdin { type => "stdin-type" } }output {

stdout { debug => true debug_format => "json" }elasticsearch { embedded => true }

}

java -jar logstash-1.1.9-monolithic.jar agent -flogstash-search.conf

How to output

Example

Where to store

ElasticSearch

http://www.elasticsearch.org

What is Elasticsearch?

Distibuted RESTful search server«Real-time» searchRESTful APIFulltext searchYAML/JSON configuration

Beautiful UI

http://kibana.org

User-friendly UI

Test Node

Test Node

Logstash ElasticSearch

Kibana

???

How to compose them

We need shippers!

Logstash shippers

• beaver - python, multiple outputs• woodchuck - ruby, multiple outputs• awesant - perl, multiple outputs supported• lumberjack - C, encrypted+compressed transport• syslog-shipper - ruby, syslog tcp• remote_syslog - ruby, syslog tcp/tls• Message::Passing - perl, multiple inputs and

outputs• nxlog - C, multi platform including windows,

tcp/udp/ssl/tls• logtail - perl, from flat files to redis

Build time

Lumberjack installationapt-get install rubygemsgem install fpmexport PATH=$PATH:/var/lib/gems/1.8/bingit clone https://github.com/jordansissel/lumberjack.gitcd lumberjackmakemake debdpkg -i lumberjack_0.0.8_amd64.deb

https://github.com/jordansissel/

Logstash installation

mkdir /opt/logstashwget https://logstash.objects.dreamhost.com/

release/logstash-1.1.9-monolithic.jar -O /opt/logstash/logstash.jar

https://logstash.objects.dreamhost.com/

https://logstash.objects.dreamhost.com/

Elasticsearch installation

wget http://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.20.2.tar.gztar -zxf elasticsearch-0.20.2.tar.gz

http://download.elasticsearch.org/

Kibana installation

git clone --branch=kibana-ruby https://github.com/rashidkpc/Kibana.git /opt/kibanaapt-get install rubygems libcurl4-openssl-devexport PATH=$PATH:/var/lib/gems/1.8/bincd /opt/kibanabundle install

https://github.com/

Lumberjack startup

/opt/lumberjack/bin/lumberjack --host your.logstash.host --port port-for-these-logs --ssl-ca-path /etc/ssl/logstash.pub

Для генерации ключей на logstash сервере:openssl req -x509 -newkey rsa:2048 -keyout /etc/ ssl/logstash.key -out /etc/ssl/logstash.pub - nodes -days 365

Configuring Logstash #1

input { lumberjack {

type => "apache-access" port => 3338

ssl_certificate => "/etc/ssl/logstash.pub" ssl_key => "/etc/ssl/logstash.key"

} }


filter { date {

type => "apache-access" timestamp =>

"dd/MMM/yyyy:HH:mm:ss Z" }

}


output { elasticsearch {

embedded => false cluster => logs host => "172.28.2.2" index => "apache-%{+YYYY.MM}" type => "apache-access"

}}

Logstash startup

/usr/bin/java -jar /opt/logstash/logstash.jar agent -f <path-to-your.conf> -l <path-to-where-you-want-the.log>

Configuring Elasticsearch

cluster.name: logsindex.number_of_replicas: 0path.data: /elasticsearch/datapath.work: /elasticsearch/workpath.logs: /elasticsearch/logsbootstrap.mlockall: truediscovery.zen.ping.multicast.enabled: false

Elasticsearch startup

./bin/elasticsearch –f

or as a daemon

./bin/elasticsearch

Kibana startup

ruby kibana.rb

Profit!

What to read http://www.logstashbook.com/code/ https://github.com/logstash/logstash/blob/v1.1.12/patterns/

grok-patterns http://grokdebug.herokuapp.com/ http://www.infoq.com/articles/review-the-logstash-booл http://www.elasticsearch.org/tutorials/using-elasticsearch-

for-logs/ https://lucene.apache.org/core/old_versioned_docs/

versions/3_5_0/queryparsersyntax.html http://www.elasticsearch.org/tutorials/elasticsearch-on-ec2/ http://blog.lusis.org/blog/2012/01/31/load-balancing-

logstash-with-amqp/

Twitter@andrebrov

[email protected]

Skyperebrov.andrey

Q&A