Performance with Percona Monitoring and Monitoring MySQL ... · Analyze MySQL performance using...
Transcript of Performance with Percona Monitoring and Monitoring MySQL ... · Analyze MySQL performance using...
Santa Clara, California | April 23th – 25th, 2018
MIchael Coburn, Product Manager
Monitoring MySQL Performance with Percona
Monitoring and Management
2
Your Presenter
• Product Manager for PMM (also Percona Toolkit and Percona XtraBackup)
• With Percona for 6 years through 6 different roles• Consultant, Managing Consultant, Principal Architect, Technical Account
Manager, Principal Support Engineer
• http://bit.ly/JoinPerconaLiveSlack #monitoring-mysql-perf
Percona Live App
3
Agenda
• Introductions - 1:30pm• Part 1 - Installation and Configuration - 1:45pm• Break ~3:00pm• Part 2 - Query Analytics - 3:15pm• Part 3 - Metrics Monitor - 3:45pm• Questions - 4:15pm
4
Goals of Today's Tutorial
1. Understand the components of PMM○ pmm-client - Client tools & agents you install on each server○ PMM Serveri. Prometheus, Grafana, Consul, QAN
2. Able to install PMM Server at your site○ OVF - Open Virtualization Format (VMware, Microsoft Systems Center, Virtualbox)○ docker○ AWS Marketplace
3. Review MySQL queries using Query Analytics4. Analyze MySQL performance using Metrics Monitor
5
What is PMM
● Free, Open Source database troubleshooting and performance optimization platform for MySQL and MongoDB○ We also support:
■ ProxySQL■ Amazon RDS MySQL■ Amazon Aurora MySQL
● Runs in your secure environment (this is not a SaaS product!), on your equipment
● Secured with SSL between client and server
6
PMM Distribution Methods
● docker○ docker pull percona/pmm-server:latest
● Virtual Appliance○ Supports VMware, RedHat Virtualization, Microsoft Systems Center○ … and VirtualBox!
● AWS Marketplace○ Production-ready AMI running in EC2○ Available since November 2017
7
AWS Marketplace
● Deploy directly to EC2● Running CentOS 7
Search for "pmm" or "Percona Monitoring and Management"
https://aws.amazon.com/marketplace/pp/B077J7FYGX
8
PMM Architecture
● pmm-client○ mysqld_exporter○ node_exporter○ qan-agent
● PMM Server○ Query Analytics
■ QAN API■ QAN App
○ Metrics Monitor■ Prometheus■ Grafana■ Consul
9
PMM Server Components
● Metrics Monitor○ Prometheus
■ Timeseries database■ Powerful PromQL query language
○ Grafana■ Visualization platform
○ Consul■ Tracks which services are available to be scraped by Prometheus
● Query Analytics○ View query performance in real-time○ Aggregated for queries consuming most amount of time in MySQL○ Query drill-down for individual query performance
■ Rows read, Rows scanned, Query time, Query count■ InnoDB statistics (Percona Server for MySQL only
10
pmm-client Components
● pmm-admin○ Command-line tool for client management
● node_exporter○ Agent that exports Linux metrics
● mysqld_exporter○ Agent that exports MySQL server metrics
● qan-agent○ Agent that collects query metrics from MySQL Slow Log or
PERFORMANCE_SCHEMA
11
Prometheus Data Collection
● Prometheus server asks Consul for which services & instances to query○ by IP address and port○ Example: curl https://192.168.56.3:42000/metrics
● Prometheus exporter performs data collection upon curl request● Exporter generates text exposed via web server at :42002/metrics[root@ps57r ~]# curl -s -k https://10.91.136.33:42002/metrics-hr |grep mysql | head -8# HELP mysql_exporter_collector_duration_seconds Collector time duration.# TYPE mysql_exporter_collector_duration_seconds gaugemysql_exporter_collector_duration_seconds{collector="collect.global_status"} 0.019977679mysql_exporter_collector_duration_seconds{collector="collect.info_schema.innodb_metrics"} 0.006224816mysql_exporter_collector_duration_seconds{collector="connection"} 2.1584e-05# HELP mysql_exporter_hr_last_scrape_error Whether the last scrape of metrics from MySQL resulted in an error (1 for error, 0 for success).# TYPE mysql_exporter_hr_last_scrape_error gaugemysql_exporter_hr_last_scrape_error 0
Part 1Installation and configuration
13
Environment Notes
● Authentication:○ percona / percona
● client 192.168.56.3○ ssh [email protected]
● server 192.168.56.2○ ssh [email protected]
~/.ssh/config
Host client
User percona
HostName 192.168.56.3
IdentityFile ~/.ssh/pmm
14
PMM Installation Steps
● High level steps:○ Import Appliance into VirtualBox○ Configure VirtualBox Host-Only Networking○ Validate client and server can communicate○ Start up sysbench test (optional)○ Configure pmm-admin to start:
■ linux:metrics■ mysql:metrics■ mysql:queries
15
Configuring VirtualBox
● Import Appliance○ File > Import Appliance…
● Configure Network○ VirtualBox > Preferences > Network
> Host-only Networks■ Create new host-only network via
button on right○ Adapter
■ Configure IPv4 Address: 192.168.56.1
● DHCP Server○ Enable Server
■ Configure Server Address: 192.168.56.1
■ Configure Lower Address Bound: 192.168.56.2
■ Configure Upper Address Bound: 192.168.56.10
● Consider setting Execution Cap at 50%, 2 cores○ Per instance
16
VirtualBox Host Only Network < v5 - OSX
17
VirtualBox > v5 - Windows
18
VirtualBox Network Interfaces - OSX
19
VirtualBox Network Interfaces - Windows
20
VirtualBox Network Interfaces - Windows
● Start the two instances via VirtualBox console
● Connect using ssh○ Optional:
■ ssh-agent bash■ ssh-add ~/.ssh/pmm
○ ssh -l percona 192.168.56.3○ [client $] ssh [email protected]
● Ensure the nodes can see each other○ [percona@client ~]$ ping pmm-server○ [admin@pmm-server ~]$ ping client
● Configure pmm-server for correct time○ [pmm-server $] date --set='TimeFromClient'
~/.ssh/config
Host client
User percona
HostName 192.168.56.3
IdentityFile ~/.ssh/pmm
21
Starting Sysbench (optional)
● We use sysbench to generate load in MySQL● Start the script:
○ client$ ./sysbench.sh● Configuration is for 1 trx/sec of 20 events on a 10k row InnoDB table
22
Configuring PMM
● FINALLY!● Connect client to pmm-server
○ sudo pmm-admin config --server 192.168.56.2 --server-user percona --server-password percona
● Start data collection○ sudo pmm-admin add mysql --create-user --socket
/var/lib/mysql/mysql.sock --user percona --password percona○ Generates a MySQL user account with bare minimum privileges
● Creates three services○ pmm-linux-metrics○ pmm-mysql-metrics○ pmm-mysql-queries
23
Confirming it all Works
● PMM Server: http://192.168.56.2/● Prometheus http://192.168.56.2/prometheus● Do they work? ● Great - take a break!● No? Let's Troubleshoot (next slide…)
24
Troubleshooting PMM
● Check for any red fields:○ sudo pmm-admin list○ sudo pmm-admin check-network
■ Time drift ?● pmm-server$ sudo date --set='<TimeFromClient>'
● Restarting one or all components○ sudo pmm-admin restart linux:metrics pmm-client○ sudo pmm-admin restart --all
● Logs are in /var/log/pmm-*.log● Check targets status in Prometheus
○ http://192.168.56.2/prometheus/targets
25
Confirming IP address of PMM Server
● Confirm the IP address the server obtained from DHCP
26
If all else fails...
● I have a few PMM Server instances available in EC2:○ http://13.58.238.204○ http://52.14.41.0○ http://18.221.240.156○ http://18.221.171.225○ http://13.58.244.142
● To obtain a shell:○ ssh -i ~/.ssh/pmm ec2-user@<IP>
Break!We'll resume in 15 minutes
Using Query Analytics (QAN)Examining queries in depth
29
Query Analytics Dashboard
30
Query Analytics Overview
● Query Abstract○ Query pattern with placeholders
● ID○ Unique fingerprint, used for query group by
● Load○ Grand Total Time - percentage of time that MySQL server spent executing the
query● Count
○ QPS, total count during window, % of total● Latency
○ Min, Med, Avg, P95, Max
31
PERFORMANCE_SCHEMA
32
Slow Log - Percona Server Enhanced
33
EXPLAIN - Table and JSON
34
CREATE TABLE, TABLE STATUS, and INDEXES
35
Server Summary Information
● Collects and displays per Server:○ pt-summary○ pt-mysql-summary
● _PMM System Summary● Summary can be downloaded from the UI
Part 3 - Using Metrics MonitorEye candy
37
Grafana in a Nutshell
● Open Source data visualisation tool● Popular datasources
○ Prometheus○ CloudWatch○ Graphite○ Elasticsearch
● Templated Variables○ Define your graph metrics, and let the hosts get filled in automatically○ GREAT for large, dynamic environments where hosts are considered ephemeral
38
Prometheus revisited
● Timeseries database - metric name + key/value pairs○ mysql_global_variables_innodb_buffer_pool_instances{instance=
"ps57",job="mysql"} = 8○ mysql_slave_status_slave_io_running{instance="ps57r",job="mys
ql",master_host="10.91.136.32",master_uuid="9809315d-4d97-11e6-b85e-0007cb03dc86"} = 1
● Flexible query language - PromQL● Collection of metrics based on HTTP pull● Targets identified via service discovery or static configuration files
○ We're using consul in PMM for service discovery
39
How can I...
● Compare servers to each other○ Cross Server graphs
● Show behaviour now() compared to past period○ Trends Overview dashboard
● At a glance MySQL + indepth○ MySQL Overview, InnoDB, InnoDB Advanced
● Table statistics*○ Largest tables by rows and size, total DB size, tables by rows read and changed,
auto_increment usage (about to hit the limit?)● User statistics*
○ Top users by connection count, network usage, rows read/changed
40
Annotations
● Visualize Application Events in PMM○ pmm-admin annotate "Application deployment v1.3"
41
Alerting
● Alerting○ Cannot use Templated Variables○ Instead, replace with string constants for instance name
Almost the end
Parting thoughts
43
Advice
● PMM Metrics retention is 30 days○ We are looking at options to present a longer history
● mysql:metrics are polled at 1s, 5s, and 60s resolutions, and linux:metrics is every 1s○ On high-latency links you might need to tune scrape_interval up
● Don't skimp on resources○ Prometheus in particular needs a lot of CPU cores and fast disks, in order to
sequence scrape data before writing chunks to disk● Consider disabling some mysqld_exporter features to minimise
performance impact○ --disable-tablestats, --disable-processlist
● Keep queries in the database (security)○ --disable-queryexamples
44
PMM Roadmap
● Prometheus 2.x - faster, more instances per PMM Server● PostgreSQL Support● MySQL -> ClickHouse for QAN datastore
○ faster, aggregation across all servers, new filtering and sorting options● Long term metrics storage (past 30 days)● One-click ticket submission*● Standardised data collection for tickets*● Any feedback of what you'd like to see in PMM?
* for Percona Subscribers (Customers) only
45
Thank You Sponsors!!
46
Questions?
● Michael Coburn [email protected]
● Percona is looking for MongoDB and MySQL rockstars! Be sure to stop by Percona’s booth.
● Do you have any areas or benchmarks you want Percona to talk about in blogs together? Any features or tools you think we should focus on to meet the community's needs?
47
Rate My Session
Thank You!