Don’t turn your logs into cuneiform

Andrei Rebrov

Cuneiform

Logs

What’s the difference?

Let’s dig into

• Too much logs• Too much information inside• They are distributed across several

machines• We are not supposed to read blogs

Time for tools

• Open source

• Collects and parses

Logstash

http://logstash.net

Key feature

Too many sources?•syslog•nginx access log•application logs•database logs

How about their format?

З parts of Logstash• Inputs

•Filters

•Outputs

http://logstash.net/docs/1.1.12/

Inputs• file• eventlog• ganglia• heroku• syslog• tcp

Filters

http://logstash.net/docs/1.1.12/

• anonymize• date• mutate• grok• grep

http://logstash.net/docs/1.1.12/

Outputs• file• graphite• http• irc• email• zabbix

input { stdin { type => "stdin-type"} }output { stdout { debug => true debug_format =>"json"} }

java -jar logstash-1.1.9-monolithic.jar agent -flogstash-simple.conf

Easy to adopt

Example

input { stdin { type => "stdin-type"} }filter { grok { type => "stdin-type" pattern =>"Hello %{DATA:message}!" } }output { stdout { debug => true debug_format =>"json"} }

java -jar logstash-1.1.9-monolithic.jar agent -flogstash-simple.conf

Time to parse

Example

input { stdin { type => "stdin-type" } }output {

stdout { debug => true debug_format => "json" }elasticsearch { embedded => true }

}

java -jar logstash-1.1.9-monolithic.jar agent -flogstash-search.conf

How to output

Example

Where to store

ElasticSearch

http://www.elasticsearch.org

What is Elasticsearch?

Distibuted RESTful search server«Real-time» searchRESTful APIFulltext searchYAML/JSON configuration

Beautiful UI

http://kibana.org

User-friendly UI

Test Node

Test Node

Logstash ElasticSearch

Kibana

???

How to compose them

We need shippers!

Logstash shippers

• beaver - python, multiple outputs• woodchuck - ruby, multiple outputs• awesant - perl, multiple outputs supported• lumberjack - C, encrypted+compressed transport• syslog-shipper - ruby, syslog tcp• remote_syslog - ruby, syslog tcp/tls• Message::Passing - perl, multiple inputs and

outputs• nxlog - C, multi platform including windows,

tcp/udp/ssl/tls• logtail - perl, from flat files to redis

Build time

Lumberjack installationapt-get install rubygemsgem install fpmexport PATH=$PATH:/var/lib/gems/1.8/bingit clone https://github.com/jordansissel/lumberjack.gitcd lumberjackmakemake debdpkg -i lumberjack_0.0.8_amd64.deb

Logstash installation

mkdir /opt/logstashwget https://logstash.objects.dreamhost.com/

release/logstash-1.1.9-monolithic.jar -O /opt/logstash/logstash.jar

Elasticsearch installation

wget http://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.20.2.tar.gztar -zxf elasticsearch-0.20.2.tar.gz

Kibana installation

git clone --branch=kibana-ruby https://github.com/rashidkpc/Kibana.git /opt/kibanaapt-get install rubygems libcurl4-openssl-devexport PATH=$PATH:/var/lib/gems/1.8/bincd /opt/kibanabundle install

Lumberjack startup

/opt/lumberjack/bin/lumberjack --host your.logstash.host --port port-for-these-logs --ssl-ca-path /etc/ssl/logstash.pub

Для генерации ключей на logstash сервере:openssl req -x509 -newkey rsa:2048 -keyout /etc/ ssl/logstash.key -out /etc/ssl/logstash.pub - nodes -days 365

Configuring Logstash #1

input {   lumberjack {    

type => "apache-access"   port => 3338

        ssl_certificate => "/etc/ssl/logstash.pub"      ssl_key => "/etc/ssl/logstash.key" 

} }

Configuring Logstash #2

filter {  date {    

type => "apache-access"   timestamp =>

"dd/MMM/yyyy:HH:mm:ss Z"  }

}

Configuring Logstash #3

output {   elasticsearch {    

embedded => false    cluster => logs     host => "172.28.2.2"    index => "apache-%{+YYYY.MM}"   type => "apache-access"   

}}

Logstash startup

/usr/bin/java -jar /opt/logstash/logstash.jar agent -f <path-to-your.conf> -l <path-to-where-you-want-the.log>

Configuring Elasticsearch

cluster.name: logsindex.number_of_replicas: 0path.data: /elasticsearch/datapath.work: /elasticsearch/workpath.logs: /elasticsearch/logsbootstrap.mlockall: truediscovery.zen.ping.multicast.enabled: false

Elasticsearch startup

./bin/elasticsearch –f

or as a daemon

./bin/elasticsearch

Kibana startup

ruby kibana.rb

Profit!

What to read http://www.logstashbook.com/code/ https://github.com/logstash/logstash/blob/v1.1.12/patterns/

grok-patterns http://grokdebug.herokuapp.com/ http://www.infoq.com/articles/review-the-logstash-booл http://www.elasticsearch.org/tutorials/using-elasticsearch-

for-logs/ https://lucene.apache.org/core/old_versioned_docs/

versions/3_5_0/queryparsersyntax.html http://www.elasticsearch.org/tutorials/elasticsearch-on-ec2/ http://blog.lusis.org/blog/2012/01/31/load-balancing-

logstash-with-amqp/

Twitter@andrebrov

E-mailarebrov@scrumtrek.ru

Skyperebrov.andrey

Q&A