Prometheus Monitoring MySQL with - Percona · Prometheus Prometheus Ben Kochie - Prometheus Lead -...
Transcript of Prometheus Monitoring MySQL with - Percona · Prometheus Prometheus Ben Kochie - Prometheus Lead -...
![Page 1: Prometheus Monitoring MySQL with - Percona · Prometheus Prometheus Ben Kochie - Prometheus Lead - GitLab. Prometheus. About Prometheus Metrics collection Time-series database Graphing](https://reader034.fdocuments.net/reader034/viewer/2022042506/5f7dae4777ca4a106a5c4d7e/html5/thumbnails/1.jpg)
Monitoring MySQL with Prometheus
Prometheus
Ben Kochie - Prometheus Lead - GitLab
![Page 2: Prometheus Monitoring MySQL with - Percona · Prometheus Prometheus Ben Kochie - Prometheus Lead - GitLab. Prometheus. About Prometheus Metrics collection Time-series database Graphing](https://reader034.fdocuments.net/reader034/viewer/2022042506/5f7dae4777ca4a106a5c4d7e/html5/thumbnails/2.jpg)
Prometheus
![Page 3: Prometheus Monitoring MySQL with - Percona · Prometheus Prometheus Ben Kochie - Prometheus Lead - GitLab. Prometheus. About Prometheus Metrics collection Time-series database Graphing](https://reader034.fdocuments.net/reader034/viewer/2022042506/5f7dae4777ca4a106a5c4d7e/html5/thumbnails/3.jpg)
About Prometheus● Metrics collection● Time-series database● Graphing● Alerting
Prometheus
![Page 4: Prometheus Monitoring MySQL with - Percona · Prometheus Prometheus Ben Kochie - Prometheus Lead - GitLab. Prometheus. About Prometheus Metrics collection Time-series database Graphing](https://reader034.fdocuments.net/reader034/viewer/2022042506/5f7dae4777ca4a106a5c4d7e/html5/thumbnails/4.jpg)
Performance
Prometheus
![Page 5: Prometheus Monitoring MySQL with - Percona · Prometheus Prometheus Ben Kochie - Prometheus Lead - GitLab. Prometheus. About Prometheus Metrics collection Time-series database Graphing](https://reader034.fdocuments.net/reader034/viewer/2022042506/5f7dae4777ca4a106a5c4d7e/html5/thumbnails/5.jpg)
Performance● Millions of Timeseries● 800k samples per second● 1.3 bytes per sample (varbit)● Thousands of targets
Prometheus
![Page 6: Prometheus Monitoring MySQL with - Percona · Prometheus Prometheus Ben Kochie - Prometheus Lead - GitLab. Prometheus. About Prometheus Metrics collection Time-series database Graphing](https://reader034.fdocuments.net/reader034/viewer/2022042506/5f7dae4777ca4a106a5c4d7e/html5/thumbnails/6.jpg)
In Production● 1500+ Tables● 160+ Databases● 30+ Clusters● 250+ Servers● 6000+ Events Digests
Prometheus
![Page 7: Prometheus Monitoring MySQL with - Percona · Prometheus Prometheus Ben Kochie - Prometheus Lead - GitLab. Prometheus. About Prometheus Metrics collection Time-series database Graphing](https://reader034.fdocuments.net/reader034/viewer/2022042506/5f7dae4777ca4a106a5c4d7e/html5/thumbnails/7.jpg)
Real-World Load● 1.5 million timeseries● 80k samples per second● 400GB (28 days, old encoding)● 10 cores (2.0ghz Ivy-Bridge)
Prometheus
![Page 8: Prometheus Monitoring MySQL with - Percona · Prometheus Prometheus Ben Kochie - Prometheus Lead - GitLab. Prometheus. About Prometheus Metrics collection Time-series database Graphing](https://reader034.fdocuments.net/reader034/viewer/2022042506/5f7dae4777ca4a106a5c4d7e/html5/thumbnails/8.jpg)
Prometheus Basics
Prometheus
![Page 9: Prometheus Monitoring MySQL with - Percona · Prometheus Prometheus Ben Kochie - Prometheus Lead - GitLab. Prometheus. About Prometheus Metrics collection Time-series database Graphing](https://reader034.fdocuments.net/reader034/viewer/2022042506/5f7dae4777ca4a106a5c4d7e/html5/thumbnails/9.jpg)
Prometheus
![Page 10: Prometheus Monitoring MySQL with - Percona · Prometheus Prometheus Ben Kochie - Prometheus Lead - GitLab. Prometheus. About Prometheus Metrics collection Time-series database Graphing](https://reader034.fdocuments.net/reader034/viewer/2022042506/5f7dae4777ca4a106a5c4d7e/html5/thumbnails/10.jpg)
mysqld_exporter Basics
Prometheus
![Page 11: Prometheus Monitoring MySQL with - Percona · Prometheus Prometheus Ben Kochie - Prometheus Lead - GitLab. Prometheus. About Prometheus Metrics collection Time-series database Graphing](https://reader034.fdocuments.net/reader034/viewer/2022042506/5f7dae4777ca4a106a5c4d7e/html5/thumbnails/11.jpg)
https://github.com/prometheus/mysqld_exporter
CREATE USER 'exporter'@'localhost' IDENTIFIED BY 'pass';GRANT PROCESS, REPLICATION CLIENT ON *.* TO 'exporter'@'localhost';GRANT SELECT ON performance_schema.* TO 'exporter'@'localhost';
DATA_SOURCE_NAME=’exporter:pass@(localhost:3306)/’
Prometheus
![Page 12: Prometheus Monitoring MySQL with - Percona · Prometheus Prometheus Ben Kochie - Prometheus Lead - GitLab. Prometheus. About Prometheus Metrics collection Time-series database Graphing](https://reader034.fdocuments.net/reader034/viewer/2022042506/5f7dae4777ca4a106a5c4d7e/html5/thumbnails/12.jpg)
Start mysqld_exporter./mysqld_exporter \ -collect.info_schema.innodb_metrics \ -collect.info_schema.userstats \ -collect.perf_schema.eventsstatements \ -collect.perf_schema.indexiowaits \ -collect.perf_schema.tableiowaits
Prometheus
![Page 13: Prometheus Monitoring MySQL with - Percona · Prometheus Prometheus Ben Kochie - Prometheus Lead - GitLab. Prometheus. About Prometheus Metrics collection Time-series database Graphing](https://reader034.fdocuments.net/reader034/viewer/2022042506/5f7dae4777ca4a106a5c4d7e/html5/thumbnails/13.jpg)
Exporter Output (http://localhost:9104/metrics)
# HELP mysql_global_status_commands_total Total number of executed MySQL commands.# TYPE mysql_global_status_commands_total countermysql_global_status_commands_total{command="admin_commands"} 795205mysql_global_status_commands_total{command="alter_db"} 0
# HELP mysql_global_variables_read_only Generic gauge metric from SHOW GLOBAL VARIABLES.# TYPE mysql_global_variables_read_only gaugemysql_global_variables_read_only 1
# HELP mysql_slave_status_seconds_behind_master Generic metric from SHOW SLAVE STATUS.# TYPE mysql_slave_status_seconds_behind_master untypedmysql_slave_status_seconds_behind_master 0
Prometheus
![Page 14: Prometheus Monitoring MySQL with - Percona · Prometheus Prometheus Ben Kochie - Prometheus Lead - GitLab. Prometheus. About Prometheus Metrics collection Time-series database Graphing](https://reader034.fdocuments.net/reader034/viewer/2022042506/5f7dae4777ca4a106a5c4d7e/html5/thumbnails/14.jpg)
Configuration
Prometheus
![Page 15: Prometheus Monitoring MySQL with - Percona · Prometheus Prometheus Ben Kochie - Prometheus Lead - GitLab. Prometheus. About Prometheus Metrics collection Time-series database Graphing](https://reader034.fdocuments.net/reader034/viewer/2022042506/5f7dae4777ca4a106a5c4d7e/html5/thumbnails/15.jpg)
global: scrape_interval: 15s evaluation_interval: 15s
scrape_configs: - job_name: 'mysql' target_groups: - targets: - localhost:9104
Prometheus
![Page 16: Prometheus Monitoring MySQL with - Percona · Prometheus Prometheus Ben Kochie - Prometheus Lead - GitLab. Prometheus. About Prometheus Metrics collection Time-series database Graphing](https://reader034.fdocuments.net/reader034/viewer/2022042506/5f7dae4777ca4a106a5c4d7e/html5/thumbnails/16.jpg)
PromQL
Prometheus
![Page 17: Prometheus Monitoring MySQL with - Percona · Prometheus Prometheus Ben Kochie - Prometheus Lead - GitLab. Prometheus. About Prometheus Metrics collection Time-series database Graphing](https://reader034.fdocuments.net/reader034/viewer/2022042506/5f7dae4777ca4a106a5c4d7e/html5/thumbnails/17.jpg)
rate(mysql_global_status_connections[2m])
sum( rate(mysql_global_status_commands_total[2m]))
Prometheus
![Page 18: Prometheus Monitoring MySQL with - Percona · Prometheus Prometheus Ben Kochie - Prometheus Lead - GitLab. Prometheus. About Prometheus Metrics collection Time-series database Graphing](https://reader034.fdocuments.net/reader034/viewer/2022042506/5f7dae4777ca4a106a5c4d7e/html5/thumbnails/18.jpg)
sum( rate(mysql_global_status_commands_total[2m]))
Prometheus
![Page 19: Prometheus Monitoring MySQL with - Percona · Prometheus Prometheus Ben Kochie - Prometheus Lead - GitLab. Prometheus. About Prometheus Metrics collection Time-series database Graphing](https://reader034.fdocuments.net/reader034/viewer/2022042506/5f7dae4777ca4a106a5c4d7e/html5/thumbnails/19.jpg)
sum by (instance) ( rate(mysql_global_status_commands_total[2m]))
Prometheus
![Page 20: Prometheus Monitoring MySQL with - Percona · Prometheus Prometheus Ben Kochie - Prometheus Lead - GitLab. Prometheus. About Prometheus Metrics collection Time-series database Graphing](https://reader034.fdocuments.net/reader034/viewer/2022042506/5f7dae4777ca4a106a5c4d7e/html5/thumbnails/20.jpg)
sum by (command) ( rate(mysql_global_status_commands_total[2m]))
Prometheus
![Page 21: Prometheus Monitoring MySQL with - Percona · Prometheus Prometheus Ben Kochie - Prometheus Lead - GitLab. Prometheus. About Prometheus Metrics collection Time-series database Graphing](https://reader034.fdocuments.net/reader034/viewer/2022042506/5f7dae4777ca4a106a5c4d7e/html5/thumbnails/21.jpg)
sum by (command) ( rate(mysql_global_status_commands_total{ command=~"(insert|update|delete)" }[2m]))
Prometheus
![Page 22: Prometheus Monitoring MySQL with - Percona · Prometheus Prometheus Ben Kochie - Prometheus Lead - GitLab. Prometheus. About Prometheus Metrics collection Time-series database Graphing](https://reader034.fdocuments.net/reader034/viewer/2022042506/5f7dae4777ca4a106a5c4d7e/html5/thumbnails/22.jpg)
mysql_slave_status_seconds_behind_master
Prometheus
![Page 23: Prometheus Monitoring MySQL with - Percona · Prometheus Prometheus Ben Kochie - Prometheus Lead - GitLab. Prometheus. About Prometheus Metrics collection Time-series database Graphing](https://reader034.fdocuments.net/reader034/viewer/2022042506/5f7dae4777ca4a106a5c4d7e/html5/thumbnails/23.jpg)
mysql_slave_status_seconds_behind_master - mysql_slave_status_sql_delay
Prometheus
![Page 24: Prometheus Monitoring MySQL with - Percona · Prometheus Prometheus Ben Kochie - Prometheus Lead - GitLab. Prometheus. About Prometheus Metrics collection Time-series database Graphing](https://reader034.fdocuments.net/reader034/viewer/2022042506/5f7dae4777ca4a106a5c4d7e/html5/thumbnails/24.jpg)
Rules Files
Prometheus
![Page 25: Prometheus Monitoring MySQL with - Percona · Prometheus Prometheus Ben Kochie - Prometheus Lead - GitLab. Prometheus. About Prometheus Metrics collection Time-series database Graphing](https://reader034.fdocuments.net/reader034/viewer/2022042506/5f7dae4777ca4a106a5c4d7e/html5/thumbnails/25.jpg)
mysql_slave_status_seconds_behind_master - mysql_slave_status_sql_delay
Prometheus
![Page 26: Prometheus Monitoring MySQL with - Percona · Prometheus Prometheus Ben Kochie - Prometheus Lead - GitLab. Prometheus. About Prometheus Metrics collection Time-series database Graphing](https://reader034.fdocuments.net/reader034/viewer/2022042506/5f7dae4777ca4a106a5c4d7e/html5/thumbnails/26.jpg)
mysql_slave_lag_seconds = mysql_slave_status_seconds_behind_master - mysql_slave_status_sql_delay
Prometheus
![Page 27: Prometheus Monitoring MySQL with - Percona · Prometheus Prometheus Ben Kochie - Prometheus Lead - GitLab. Prometheus. About Prometheus Metrics collection Time-series database Graphing](https://reader034.fdocuments.net/reader034/viewer/2022042506/5f7dae4777ca4a106a5c4d7e/html5/thumbnails/27.jpg)
mysql_slave_lag_seconds = mysql_slave_status_seconds_behind_master - mysql_slave_status_sql_delay
ALERT MySQLSlaveLag IF mysql_slave_lag_seconds > 300 FOR 1m LABELS { severity = "warning" } ANNOTATIONS { summary = "Slave lag is too damn high." }
Prometheus
![Page 28: Prometheus Monitoring MySQL with - Percona · Prometheus Prometheus Ben Kochie - Prometheus Lead - GitLab. Prometheus. About Prometheus Metrics collection Time-series database Graphing](https://reader034.fdocuments.net/reader034/viewer/2022042506/5f7dae4777ca4a106a5c4d7e/html5/thumbnails/28.jpg)
mysql_slave_lag_seconds = mysql_slave_status_seconds_behind_master - mysql_slave_status_sql_delay
ALERT MySQLSlaveLag IF (mysql_slave_lag_seconds > 60 AND predict_linear(mysql_slave_lag_seconds[2m],60*60)) > 0 FOR 2m LABELS { severity = "warning" } ANNOTATIONS { summary = "Slave lag is too damn high." }
Prometheus
![Page 29: Prometheus Monitoring MySQL with - Percona · Prometheus Prometheus Ben Kochie - Prometheus Lead - GitLab. Prometheus. About Prometheus Metrics collection Time-series database Graphing](https://reader034.fdocuments.net/reader034/viewer/2022042506/5f7dae4777ca4a106a5c4d7e/html5/thumbnails/29.jpg)
Examples/Graphs
Prometheus
![Page 30: Prometheus Monitoring MySQL with - Percona · Prometheus Prometheus Ben Kochie - Prometheus Lead - GitLab. Prometheus. About Prometheus Metrics collection Time-series database Graphing](https://reader034.fdocuments.net/reader034/viewer/2022042506/5f7dae4777ca4a106a5c4d7e/html5/thumbnails/30.jpg)
Basic Traffic Overview
Prometheus
![Page 31: Prometheus Monitoring MySQL with - Percona · Prometheus Prometheus Ben Kochie - Prometheus Lead - GitLab. Prometheus. About Prometheus Metrics collection Time-series database Graphing](https://reader034.fdocuments.net/reader034/viewer/2022042506/5f7dae4777ca4a106a5c4d7e/html5/thumbnails/31.jpg)
sum by (digest_text) (rate( mysql_perf_schema_events_statements_tmp_disk_tables_total[2m]))
Prometheus
![Page 32: Prometheus Monitoring MySQL with - Percona · Prometheus Prometheus Ben Kochie - Prometheus Lead - GitLab. Prometheus. About Prometheus Metrics collection Time-series database Graphing](https://reader034.fdocuments.net/reader034/viewer/2022042506/5f7dae4777ca4a106a5c4d7e/html5/thumbnails/32.jpg)
topk(5, sum by (schema,digest_text) (rate(mysql_perf_schema_events_statements_total[5m])))
Prometheus
![Page 33: Prometheus Monitoring MySQL with - Percona · Prometheus Prometheus Ben Kochie - Prometheus Lead - GitLab. Prometheus. About Prometheus Metrics collection Time-series database Graphing](https://reader034.fdocuments.net/reader034/viewer/2022042506/5f7dae4777ca4a106a5c4d7e/html5/thumbnails/33.jpg)
The Future
Prometheus
![Page 34: Prometheus Monitoring MySQL with - Percona · Prometheus Prometheus Ben Kochie - Prometheus Lead - GitLab. Prometheus. About Prometheus Metrics collection Time-series database Graphing](https://reader034.fdocuments.net/reader034/viewer/2022042506/5f7dae4777ca4a106a5c4d7e/html5/thumbnails/34.jpg)
Demo Time
Prometheus