Are you visualizing your logfiles? -...
-
Upload
trinhnguyet -
Category
Documents
-
view
216 -
download
0
Transcript of Are you visualizing your logfiles? -...
![Page 1: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/1.jpg)
Bastian Widmer / @dasrecht
Are you visualizing your logfiles?
![Page 2: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/2.jpg)
Bastian Widmer / @dasrecht
Visualizing Logfiles with ELK Stack
![Page 3: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/3.jpg)
Hola ¿Com estàs?
Bastian Widmer
@dasrecht / bastianwidmer.ch
DrupalCI: Modernizing Testbot Initiative
Chief YoloOps Evangelist
![Page 4: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/4.jpg)
Agenda 1 Introduction2
3
4
ArchitectureELK Stack
Tools!AutomationP22N - Performance Optim…
5
6
![Page 5: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/5.jpg)
Visualizing Logfiles, why?
![Page 6: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/6.jpg)
„Can you check the errors from yesterday between 15.02 and 15.07“
![Page 7: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/7.jpg)
![Page 8: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/8.jpg)
Visualization > Plaintext
![Page 9: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/9.jpg)
![Page 10: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/10.jpg)
Patch deployed, instant feedback!
![Page 11: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/11.jpg)
Visualization > Plaintext
![Page 12: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/12.jpg)
VISUALIZATION > Plaintext
![Page 13: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/13.jpg)
Do you log to database? dblog?
![Page 14: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/14.jpg)
Okay for one site, but what if you have 70+ sites logging into your
database?
![Page 15: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/15.jpg)
Use Cases• Audit Trail - Who changed what?
• Content
• Modules
• Errors - Fixing errors and getting instant feedback by easy readable graphs
• Billing
• Application Speed
• Deep Inspection (TOR Nodes)
![Page 16: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/16.jpg)
![Page 17: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/17.jpg)
![Page 18: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/18.jpg)
ELK Stack!ELK Stack!
![Page 19: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/19.jpg)
ELK Stack!ELK Stack!ElasticsearchLogstashKibana
![Page 20: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/20.jpg)
Sidenote : Things move fast! Even with minor releases
![Page 21: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/21.jpg)
Elasticsearch
![Page 22: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/22.jpg)
Elasticsearch®• Java
• Search and Index
• Distributed — Copies & Shards
• Clustering (Zen Discovery - Multi/Unicast)
• API — JSON / RESTful
• Apache Lucene
• Disk-Based Shard Allocation
![Page 23: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/23.jpg)
Elasticsearch• Index
like a Database
• Replica Copies for Fault Tolerance
• ShardLucene Instance which indexes the Data see : http://blog.liip.ch/archive/2013/07/19/on-elasticsearch-performance.html
![Page 24: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/24.jpg)
Elasticsearch{ "status" : 200, "name" : "es-03", "cluster_name" : "cluster01", "version" : { "number" : "1.7.1", "build_hash" : "b88f43fc40b0bcd7f173a1f9ee2e97816de80b19", "build_timestamp" : "2015-07-29T09:54:16Z", "build_snapshot" : false, "lucene_version" : "4.10.4" }, "tagline" : "You Know, for Search" }
![Page 25: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/25.jpg)
Elasticsearch
![Page 26: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/26.jpg)
ElasticSearch Plugins
• New Integrated Plugin System
• ‚Bundles‘ Plugins with Elasticsearch
• „bin/plugin -install YOURPLUGIN"
![Page 27: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/27.jpg)
ElasticSearch Security
• Speak with me:
• „I will hereby solemnly swear not to expose my Elasticsearch Server to public, never-ever!“
• Elastic Shield - Provides Security (Subscription Feature)
![Page 28: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/28.jpg)
ElasticSearch Security - cheap
• Run Elasticsearch bound to localhost
• use an internal network • ssh [email protected] -N -L
9200:127.0.0.1:9200'
![Page 29: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/29.jpg)
Thankmelater™
• Security can be an issue
• curl -XDELETE ‚http://localhost:9200/*/’
• curl -XDELETE ‚http://localhost:9200/_all/’
• action.destructive_requires_name: true
https://www.elastic.co/guide/en/elasticsearch/reference/1.7/_parameters.html
![Page 30: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/30.jpg)
Marvel
• Shows Cluster Health and Real-Time Analysis
• Free during development product
• Deep insights into index creation across cluster, routing decisions and much more
![Page 31: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/31.jpg)
Logstash
![Page 32: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/32.jpg)
Did the Catalan Citizens invent Logstash?
![Page 33: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/33.jpg)
![Page 34: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/34.jpg)
Logstash
• Multiple Input / Multiple Output
• Centralize and Process Log Data
• Collect
• Parse
• Store / Forward
![Page 35: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/35.jpg)
The life of an event
• Input
• Filters
• Codecs
• Output
![Page 36: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/36.jpg)
Logstash• JRuby*
• >1.4.0 - FlatJAR Release is gone
• Instead of running „java -jar logstash.jar“ — „bin/logstash“
• Contrib Plugins
• Daily Indices !
* see https://gist.github.com/jordansissel/978956
![Page 37: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/37.jpg)
Input
• File
• Syslog
• Redis
• logstash-forwarder (former Lumberjack)
![Page 38: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/38.jpg)
Filters
• Grok
• Mutate
• Drop
• Clone
• GeoIP (!!!)
![Page 39: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/39.jpg)
Outputs
• Elasticsearch
• File / S3
• Graphite
• StatsD
![Page 40: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/40.jpg)
Logstash 1 input {! 2 stdin { }! 3 }! 4 ! 5 output {! 6 stdout {! 7 codec => rubydebug! 8 }! 9 }!!
![Page 41: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/41.jpg)
Logstash
1 vagrant@precise64$ ./logstash agent -f 1_simpleconfig.cfg! 2 very important log message!! 3 {! 4 "message" => "very important log message!",! 5 "@version" => "1",! 6 "@timestamp" => "2014-04-21T16:18:02.952Z",! 7 "host" => "precise64"! 8 }
![Page 42: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/42.jpg)
Logstash 1 input {! 2 stdin { }! 3 }! 4 output {! 5 elasticsearch{! 6 host => "127.0.0.1"! 7 }! 8 stdout {! 9 codec => rubydebug! 10 }! 11 }
![Page 43: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/43.jpg)
Logstash 1 input {! 2 file {! 3 path => "/var/log/syslog"! 4 start_position => beginning! 5 }! 6 }! 7 ! 8 output {! 9 stdout {! 10 codec => rubydebug! 11 }! 12 elasticsearch{! 13 host => "127.0.0.1"! 14 }! 15 }
![Page 44: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/44.jpg)
Kibana
![Page 45: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/45.jpg)
![Page 46: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/46.jpg)
Some history
• Ruby
• PHP
• Just Javascript (the crowd applauds)
• Node Webserver and Javascript (Kibana 4)
![Page 47: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/47.jpg)
Kibana 4
• D3.js - more fancyness
• More complex backend
• Much better flexibility
• Analytics and Aggregations
![Page 48: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/48.jpg)
Architecture
![Page 49: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/49.jpg)
Architecture
Shipper
Shipper
Shipper Broker IndexerSearch
and Storage
![Page 50: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/50.jpg)
Architecture
Shipper
Shipper
Shipper Broker IndexerSearch
and Storage
Syslog
![Page 51: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/51.jpg)
Architecture
Shipper
Shipper
Shipper Broker IndexerSearch
and Storage
Syslog Logstash
![Page 52: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/52.jpg)
Architecture
Shipper
Shipper
Shipper Broker IndexerSearch
and Storage
Syslog Logstash Elasticsearch
![Page 53: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/53.jpg)
But, Bastian…
![Page 54: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/54.jpg)
ArchitectureThe real deal!
![Page 55: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/55.jpg)
Logstash-Forwarder
• Written in Go
• Lightweight utility to forward logs to logstash
• Low resource usage
• TLS/SSL Encrypted Transfer
![Page 56: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/56.jpg)
Indexer
Architecture
Shipper
BrokerBroker IndexerSearch
and Storage
Logstash Redis Logstash Elasticsearch
nginx.log
drupal.log
auth.log
Shipper
Shipper Shipper
Logstash-Forwarder
![Page 57: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/57.jpg)
Indexer
Architecture
Shipper
BrokerBroker IndexerSearch
and Storage
Logstash Redis Logstash Elasticsearch
nginx.log
drupal.log
auth.log
Shipper
Shipper Shipper
Logstash-Forwarder
And from here you can go crazy!
![Page 58: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/58.jpg)
Indexer
Architecture High-Available
Shipper
Broker
Broker Indexer
Search and
Storage
Search and
Storage
Logstash Redis Logstash Elasticsearch
nginx.log
drupal.log
auth.log
Shipper
Shipper
Logstash-Forwarder
![Page 59: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/59.jpg)
But, Bastian!!!
![Page 60: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/60.jpg)
No!
![Page 61: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/61.jpg)
Indexer
High Available Setup with Rocketfuel!
Shipper
Broker
Broker Indexer
Search and
Storage
Search and
Storage
Logstash Redis Logstash Elasticsearch
nginx.log
drupal.log
auth.log
Shipper
Forwarder
Logstash Forwarder
HAProxy
HAProxy
KeepaliveD
![Page 62: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/62.jpg)
Tools!(because anyone needs a bit help)
![Page 63: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/63.jpg)
Elasticsearch Head
http://mobz.github.io/elasticsearch-head/ ./plugin -install mobz/elasticsearch-head
![Page 64: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/64.jpg)
Elasticsearch Kopf
./plugin -install lmenezes/elasticsearch-kopf
![Page 65: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/65.jpg)
Curator
• Time Series Indices? THIS IS THE TOOL!
• Close Indexes
• Delete (by space or time)
• Disable Bloom Filter
• Optimize / ForceMerge
• https://github.com/elasticsearch/curator
![Page 66: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/66.jpg)
Curator
• Time Series Indices? THIS IS THE TOOL!
• Close Indexes
• Delete (by space or time)
• Disable Bloom Filter
• Optimize / ForceMerge
• https://github.com/elasticsearch/curator
Curator Perfect for Time Series Indexes
![Page 67: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/67.jpg)
Curator
• Close indices older than 14 days, delete indices older than 30 days curator --host my-elasticsearch -d 30 -c 14
• Disable bloom filter for indices older than 2 days, close indices older than 14 days, delete indices older than 30 days:curator --host my-elasticsearch -b 2 -c 14 -d 30
![Page 68: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/68.jpg)
Curator 1 root@precise64:/home/vagrant# curator -c 7 -b 2 -d 10! 2 2014-04-21T17:57:19.419 INFO main:333 Job starting...! 3 2014-04-21T17:57:19.420 INFO _new_conn:180 Starting new HTTP connection (1): localhost! 4 2014-04-21T17:57:19.422 INFO log_request_success:49 GET http://localhost:9200/ [status:200 request:0.002s]! 5 2014-04-21T17:57:19.423 INFO main:359 Deleting indices older than 10 days...! 6 2014-04-21T17:57:19.430 INFO log_request_success:49 GET http://localhost:9200/logstash-*/_settings?expand_wildcards=closed [status:200 request:0.007s]! 7 2014-04-21T17:57:19.433 INFO find_expired_indices:209 logstash-2014.04.21 is 10 days, 0:00:00 above the cutoff.! 8 2014-04-21T17:57:19.433 INFO index_loop:309 DELETE index operations completed.! 9 2014-04-21T17:57:19.433 INFO main:364 Closing indices older than 7 days...! 10 2014-04-21T17:57:19.434 INFO log_request_success:49 GET http://localhost:9200/logstash-*/_settings?expand_wildcards=closed [status:200 request:0.001s]! 11 2014-04-21T17:57:19.435 INFO find_expired_indices:209 logstash-2014.04.21 is 7 days, 0:00:00 above the cutoff.! 12 2014-04-21T17:57:19.435 INFO index_loop:309 CLOSE index operations completed.! 13 2014-04-21T17:57:19.435 INFO main:369 Disabling bloom filter on indices older than 2 days...! 14 2014-04-21T17:57:19.437 INFO log_request_success:49 GET http://localhost:9200/logstash-*/_settings?expand_wildcards=closed [status:200 request:0.002s]! 15 2014-04-21T17:57:19.438 INFO find_expired_indices:209 logstash-2014.04.21 is 2 days, 0:00:00 above the cutoff.! 16 2014-04-21T17:57:19.438 INFO index_loop:309 DISABLE BLOOM FILTER FOR index operations completed.! 17 2014-04-21T17:57:19.438 INFO main:379 Done in 0:00:00.020348.!
![Page 70: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/70.jpg)
Grok Filters?!
1 grok {! 2 match => { "message" => "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }! 3 add_field => [ "received_at", "%{@timestamp}" ]! 4 }!
![Page 73: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/73.jpg)
Elasticsearch : The Definitive Guide
http://www.elastic.co/guide /en/elasticsearch/guide/current/index.html
![Page 74: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/74.jpg)
Performance Optimisationor short P22N
![Page 75: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/75.jpg)
Performance
• Remember: It’s just Java
• File Descriptors >32k
• Give enough Memory (-Xms -Xmx Values)
• Leverage File System Cache
https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html#_give_half_your_memory_to_lucene
![Page 76: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/76.jpg)
Automation!
![Page 77: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/77.jpg)
Puppet Modules
• elasticsearch/elasticsearch (PuppetLabs Approved)
• elasticsearch/logstashforwarder
• elasticsearch/logstash
![Page 78: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/78.jpg)
Puppetclass { 'elasticsearch': repo_version => '1.7', manage_repo => true, java_install => true, config => { 'cluster.name' => 'cluster01' }, datadir => '/var/lib/elasticsearch/' } !
elasticsearch::instance { 'es-01': config => { 'node.name' => 'es-01' } }
![Page 79: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/79.jpg)
Take Home• Centralized Logging saves time
• Is fun with the ELK Stack
• Gives you Graphs to Interpret
• „can you check the errors from yesterday between 15.02 and 15.07“ get’s A LOT easier
• Start here tomorrow: http://logstash.net/docs/1.4.2/tutorials/getting-started-with-logstash
![Page 80: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/80.jpg)
Thank you for having me here! Slides: http://s.nrdy.ch/drupalcon-bcn
Feedback: http://s.nrdy.ch/rateme
![Page 82: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/82.jpg)
Legal (because Legal…)
• Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries.
• Kibana is a trademark of Elasticsearch BV, registered in the U.S. and in other countries.
• Elastic, Logstash and Marvel are trademarks of Elasticsesarch BV
![Page 83: Are you visualizing your logfiles? - bastianwidmer.chbastianwidmer.ch/slides/2015-09-22_VisualizingLogfilesWithElkStack... · Okay for one site, but what if you have 70+ sites logging](https://reader030.fdocuments.net/reader030/viewer/2022041208/5d66ea7b88c99332038ba3a9/html5/thumbnails/83.jpg)
Images Used
• Elk : https://www.flickr.com/photos/ucumari/353839518/
• Architecture : https://www.flickr.com/photos/dasrecht/6743411525/
• VideoWall : https://twitter.com/webtuesday/status/433296964055470080/photo/1
• Tió de Nadal http://en.wikipedia.org/wiki/Image:Cagatio.jpg (CC-BY-SA 3.0)