MySQL Slow Query log Monitoring using Beats & ELK
-
Upload
youngheon-roy-kim -
Category
Data & Analytics
-
view
2.964 -
download
1
Transcript of MySQL Slow Query log Monitoring using Beats & ELK
![Page 1: MySQL Slow Query log Monitoring using Beats & ELK](https://reader036.fdocuments.net/reader036/viewer/2022081422/58ed96021a28ab35238b46d9/html5/thumbnails/1.jpg)
using Beats & ELKMySQL Slow Query log Monitoring
![Page 3: MySQL Slow Query log Monitoring using Beats & ELK](https://reader036.fdocuments.net/reader036/viewer/2022081422/58ed96021a28ab35238b46d9/html5/thumbnails/3.jpg)
Architecture
MySQL Slow Log
DB Servers ELK Server
Logstash Elasticsearch
FileBeat Kibana
3
![Page 4: MySQL Slow Query log Monitoring using Beats & ELK](https://reader036.fdocuments.net/reader036/viewer/2022081422/58ed96021a28ab35238b46d9/html5/thumbnails/4.jpg)
Install & Config FileBeat4
# rpm -ivh filebeat-1.0.1-x86_64.rpm
$vi /etc/filebeat/filebeat.ymlfilebeat: prospectors: paths: - /db/data01/mysql-slow.log //slow query pathoutput: #elasticsearch: //comment #hosts: ["localhost:9200"] //comment logstash: //uncomment # The Logstash hosts hosts: ["10.xx.xx.xx:5044"] //logstash server ip
1. Install FileBeat – on DB servers
2. Parameter configuration
![Page 5: MySQL Slow Query log Monitoring using Beats & ELK](https://reader036.fdocuments.net/reader036/viewer/2022081422/58ed96021a28ab35238b46d9/html5/thumbnails/5.jpg)
5
# /etc/init.d/filebeat startStarting filebeat: [ OK ]
3. Start FileBeat – on DB servers
Install & Config FileBeat
![Page 6: MySQL Slow Query log Monitoring using Beats & ELK](https://reader036.fdocuments.net/reader036/viewer/2022081422/58ed96021a28ab35238b46d9/html5/thumbnails/6.jpg)
Install & Config Elasticsearch6
# tar –xzvf elasticsearch-2.1.1.tar.gz1. Install Elasticsearch – on ELK servers
2. configuration
$ vi ./elasticsearch-2.1.1/config/elasticsearch.ymlcluster.name : log_cluster # cluster namenode.name : slow_log # node namepath.data: /DATA/data # index data pathpath.logs: /DATA/logs # log pathnetwork.host : 10.xxx.xxx.xxx # server’s ip
![Page 7: MySQL Slow Query log Monitoring using Beats & ELK](https://reader036.fdocuments.net/reader036/viewer/2022081422/58ed96021a28ab35238b46d9/html5/thumbnails/7.jpg)
7
3. start elasticsearch
$./bin/elasticsearch
You can’t run elastisearch as root.
Install & Config Elasticsearch
![Page 8: MySQL Slow Query log Monitoring using Beats & ELK](https://reader036.fdocuments.net/reader036/viewer/2022081422/58ed96021a28ab35238b46d9/html5/thumbnails/8.jpg)
Install & Config Logstash8
1. Install Logstash – on ELK server
$ rpm –ivh logstash-2.1.1-1.noarch.rpm
2-1. Configure(input plugin)# vi /etc/logstash/conf.d/10-slow-log.confinput { beats { port => 5044 codec => multiline{ pattern => "^# Time:" negate => true what => previous } }}
![Page 9: MySQL Slow Query log Monitoring using Beats & ELK](https://reader036.fdocuments.net/reader036/viewer/2022081422/58ed96021a28ab35238b46d9/html5/thumbnails/9.jpg)
9
2-2. Configure(filter plugin)filter { grok { match => [ "message", "^# User@Host: %{USER:query_user}(?:\[[^\]]+\])?\s+@\s+%{HOSTNAME:query_host}?\s+\[%{IP:query_ip}?\]" ] } grok { match => [ "message", "^# Thread_id: %{NUMBER:thread_id:int}\s+Schema: %{USER:schema}\s+Last_errno: %{NUMBER:last_errno:int}\s+Killed: %{NUMBER:killed:int}"] } grok { match => [ "message", "^# Query_time: %{NUMBER:query_time:float}\s+Lock_time: %{NUMBER:lock_time}\s+ Rows_sent: %{NUM-BER:rows_sent:int} \s+Rows_examined: %{NUMBER:rows_examined:int}\s+Rows_affected: %{NUMBER:rows_affected:int}\s+Rows_read: %{NUMBER:rows_read:int}"] } grok { match => [ "message", "^# Bytes_sent: %{NUMBER:bytes_sent:float}"] } grok { match => [ "message", "^SET timestamp=%{NUMBER:timestamp}" ] } grok { match => [ "message", "^SET timestamp=%{NUMBER};\s+%{GREEDYDATA:query}" ] } date { match => [ "timestamp", "UNIX" ] } mutate { remove_field => "timestamp" }
}
Install & Config Logstash
![Page 10: MySQL Slow Query log Monitoring using Beats & ELK](https://reader036.fdocuments.net/reader036/viewer/2022081422/58ed96021a28ab35238b46d9/html5/thumbnails/10.jpg)
10
2-3. Configure(output plugin)
output { elasticsearch { hosts => "10.xx.xx.xx" }}
3. Beats plugin install
# cd /opt/logstash/bin# ./plugin install logstash-input-beats
4. Start logstash
# /etc/init.d/logstash start
Install & Config Logstash
![Page 11: MySQL Slow Query log Monitoring using Beats & ELK](https://reader036.fdocuments.net/reader036/viewer/2022081422/58ed96021a28ab35238b46d9/html5/thumbnails/11.jpg)
Install & Config Kibana11
1. Install Kibana – on ELK server
$ tar –xvf kibana-4.3.1-linux-x64.tar.gz
2. Configure
$ vi ./kibana-4.3.1-linux-x64/config/kibana.ymlserver.host: "10.xx.xx.xx“ # kibana server ipelasticsearch.url: "http://10.xx.xx.xx:9200" # elasticsearch server ip
3. Start Kibana
$ ./bin/kibana
![Page 12: MySQL Slow Query log Monitoring using Beats & ELK](https://reader036.fdocuments.net/reader036/viewer/2022081422/58ed96021a28ab35238b46d9/html5/thumbnails/12.jpg)
Visualize – Slow query graph 12
mouse over
Install & Config Kibana
![Page 13: MySQL Slow Query log Monitoring using Beats & ELK](https://reader036.fdocuments.net/reader036/viewer/2022081422/58ed96021a28ab35238b46d9/html5/thumbnails/13.jpg)
13
New Visualization > Line chart
Install & Config KibanaVisualize – Slow query graph
![Page 14: MySQL Slow Query log Monitoring using Beats & ELK](https://reader036.fdocuments.net/reader036/viewer/2022081422/58ed96021a28ab35238b46d9/html5/thumbnails/14.jpg)
Dashboard - sample 14
http://10.xxx.xxx.xxx:5601
Install & Config Kibana
![Page 15: MySQL Slow Query log Monitoring using Beats & ELK](https://reader036.fdocuments.net/reader036/viewer/2022081422/58ed96021a28ab35238b46d9/html5/thumbnails/15.jpg)
Thank You