Log analysis using Logstash,ElasticSearch and Kibana - Desert Code Camp 2014
-
Upload
clairvoyantllc -
Category
Technology
-
view
91 -
download
4
Transcript of Log analysis using Logstash,ElasticSearch and Kibana - Desert Code Camp 2014
![Page 1: Log analysis using Logstash,ElasticSearch and Kibana - Desert Code Camp 2014](https://reader035.fdocuments.net/reader035/viewer/2022062219/587338cd1a28abf21b8b4f87/html5/thumbnails/1.jpg)
Log Analysis – Logstash, Elastic Search, Kibana Avinash Ramineni Shantanu Mirajkar
![Page 2: Log analysis using Logstash,ElasticSearch and Kibana - Desert Code Camp 2014](https://reader035.fdocuments.net/reader035/viewer/2022062219/587338cd1a28abf21b8b4f87/html5/thumbnails/2.jpg)
• Logging• Pains of Log Management• Introducing Logstash• Elasticsearch• Kibana• Demo• Installing Logstash, Elasticsearch Kibana• Questions
Agenda
![Page 3: Log analysis using Logstash,ElasticSearch and Kibana - Desert Code Camp 2014](https://reader035.fdocuments.net/reader035/viewer/2022062219/587338cd1a28abf21b8b4f87/html5/thumbnails/3.jpg)
• Why do we need Logging ?– Troubleshoot Issues– Security
• Analyze logs to detect patterns• Detect Malware Activity - Intrusion Detection, Denial of Service• Unauthorized Resource Usage
– Monitoring• Monitor Resource Usage
• Developers and Logging– Logging Aids in Development ?– Forget about Production !!!!!
Logging
![Page 4: Log analysis using Logstash,ElasticSearch and Kibana - Desert Code Camp 2014](https://reader035.fdocuments.net/reader035/viewer/2022062219/587338cd1a28abf21b8b4f87/html5/thumbnails/4.jpg)
• “Capture-it-all” Approach• What to Log? Everything • DevOps Movement• Logs are archived for years• Big Data • Application Usage Statistics
Logging
![Page 5: Log analysis using Logstash,ElasticSearch and Kibana - Desert Code Camp 2014](https://reader035.fdocuments.net/reader035/viewer/2022062219/587338cd1a28abf21b8b4f87/html5/thumbnails/5.jpg)
• Searching the logs– Command line, cat, tail, sed, grep, awk– Regular Expressions
• Multiple Servers behind the load balancer• Multi-Tier Architecture
– Web Application– Service Layer– Correlation between various components in a System
• Geographically distributed– Timestamps
Log management
![Page 6: Log analysis using Logstash,ElasticSearch and Kibana - Desert Code Camp 2014](https://reader035.fdocuments.net/reader035/viewer/2022062219/587338cd1a28abf21b8b4f87/html5/thumbnails/6.jpg)
• Centralize all the Logs– Too much information to go through– Increasingly hard to correlate the contextual Data
• Add Searching and Indexing Technology– grep– Custom logging frameworks , custom integration of logging, searching
technologies• Monitor the Logs
Log management
![Page 7: Log analysis using Logstash,ElasticSearch and Kibana - Desert Code Camp 2014](https://reader035.fdocuments.net/reader035/viewer/2022062219/587338cd1a28abf21b8b4f87/html5/thumbnails/7.jpg)
• Logstash to the Rescue– Integration Framework• Log Collection• Centralization• Parsing • Storage and Search
Logstash
![Page 8: Log analysis using Logstash,ElasticSearch and Kibana - Desert Code Camp 2014](https://reader035.fdocuments.net/reader035/viewer/2022062219/587338cd1a28abf21b8b4f87/html5/thumbnails/8.jpg)
• JRuby– Run on Java Virtual Machine (JVM)– Simple Message Based Architecture– Single Agent that can be configured for multiple things– OPEN SOURCE
• Four Components– Shipper– Broker and Indexer– Search and Storage– Web Interface
Logstash
![Page 9: Log analysis using Logstash,ElasticSearch and Kibana - Desert Code Camp 2014](https://reader035.fdocuments.net/reader035/viewer/2022062219/587338cd1a28abf21b8b4f87/html5/thumbnails/9.jpg)
Architecture
Image courtesy of Logstashbook
![Page 10: Log analysis using Logstash,ElasticSearch and Kibana - Desert Code Camp 2014](https://reader035.fdocuments.net/reader035/viewer/2022062219/587338cd1a28abf21b8b4f87/html5/thumbnails/10.jpg)
Architecture - Broker
• Acts as Temp Buffer between Logstash Agents and the Central server– Enhance Performance by providing caching buffer
for log events– Adds Resiliency • Incase the Indexing fails, the events are held in a queue
instead of getting lost
• AMQP,0MQ, Redis
![Page 11: Log analysis using Logstash,ElasticSearch and Kibana - Desert Code Camp 2014](https://reader035.fdocuments.net/reader035/viewer/2022062219/587338cd1a28abf21b8b4f87/html5/thumbnails/11.jpg)
• Indexing and Searching Tool– Built on Lucene
• Search and Index data available Restfully as JSON over HTTP• Comes bundled with Logstash – embedded• Text indexing Search Engine
– Searches on the Index rather than on the content• Creates Indexes of the incoming content
– Uses Apache Lucene to create Indexes• ElasticSearch can have a schema – Fields on which Indexes are
created
ElasticSearch
![Page 12: Log analysis using Logstash,ElasticSearch and Kibana - Desert Code Camp 2014](https://reader035.fdocuments.net/reader035/viewer/2022062219/587338cd1a28abf21b8b4f87/html5/thumbnails/12.jpg)
• Indexes are stored in Lucene Instances called “Shards”
• ElasticSearch can have multiple nodes• Two Types of Shards– Primary– Replica
• Replicas of Primary Shards– Protect the data– Make Searches Faster
ElasticSearch
![Page 13: Log analysis using Logstash,ElasticSearch and Kibana - Desert Code Camp 2014](https://reader035.fdocuments.net/reader035/viewer/2022062219/587338cd1a28abf21b8b4f87/html5/thumbnails/13.jpg)
• Wouldn’t it be good to have a webpage to do search on ElasticSearch instead of searching it through a Service
• Kibana provides a Simple but Powerful web Interface– Customizable Dashboards– Search the log events
• Support Lucene Query Syntax
– Creation of tables, graphs and sophisticated visualizations
Kibana
![Page 14: Log analysis using Logstash,ElasticSearch and Kibana - Desert Code Camp 2014](https://reader035.fdocuments.net/reader035/viewer/2022062219/587338cd1a28abf21b8b4f87/html5/thumbnails/14.jpg)
Kibana
![Page 15: Log analysis using Logstash,ElasticSearch and Kibana - Desert Code Camp 2014](https://reader035.fdocuments.net/reader035/viewer/2022062219/587338cd1a28abf21b8b4f87/html5/thumbnails/15.jpg)
Kibana
![Page 16: Log analysis using Logstash,ElasticSearch and Kibana - Desert Code Camp 2014](https://reader035.fdocuments.net/reader035/viewer/2022062219/587338cd1a28abf21b8b4f87/html5/thumbnails/16.jpg)
Demo
![Page 17: Log analysis using Logstash,ElasticSearch and Kibana - Desert Code Camp 2014](https://reader035.fdocuments.net/reader035/viewer/2022062219/587338cd1a28abf21b8b4f87/html5/thumbnails/17.jpg)
• Send Alerts– Emails– Instant Messaging– Other Monitoring System
• Collect and Deliver Metrics to metric engine
Alerts / Monitoring Support
![Page 18: Log analysis using Logstash,ElasticSearch and Kibana - Desert Code Camp 2014](https://reader035.fdocuments.net/reader035/viewer/2022062219/587338cd1a28abf21b8b4f87/html5/thumbnails/18.jpg)
• Small VMs with limited memory• Outsourced managed servers• Java not installed• Alternatives
– Syslog• Rsyslog• Syslogd• Syslog-NG
– Logstash Forwarder (Lumber Jack)
Shipping Logs with Logstash Agent
![Page 19: Log analysis using Logstash,ElasticSearch and Kibana - Desert Code Camp 2014](https://reader035.fdocuments.net/reader035/viewer/2022062219/587338cd1a28abf21b8b4f87/html5/thumbnails/19.jpg)
• Scale each component as needed• Can be built into using chef and puppet scripts
Scaling / Deployment