MySQL Performance Optimization and Troubleshooting with PMM
Transcript of MySQL Performance Optimization and Troubleshooting with PMM
![Page 1: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/1.jpg)
MySQL Performance Optimization and Troubleshooting with PMM
Peter Zaitsev, CEO, PerconaPercona Technical Webinars
9 May 2018
![Page 2: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/2.jpg)
2
Few words about Percona Monitoring and Management (PMM)
100% Free, Open Source database troubleshooting and performance optimization platform for MySQL and MongoDB
Based on Industry Leading Technology
Roll your own in and out of the Cloud
![Page 3: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/3.jpg)
3
Exploring Percona Monitoring and Management
• http://bit.ly/InstallPMMYou should be able to install PMM in
15 minutes or less
• https://pmmdemo.percona.comWould like to
follow along in the demo ?
![Page 4: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/4.jpg)
4
In the Presentation
Practical approach to deal with some of the
common MySQL Issues
![Page 5: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/5.jpg)
5
PMM is not just for MySQL
Supports MongoDB as well
Other databases can be added via External Exporters
This Presentation is MySQL Focused
![Page 6: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/6.jpg)
6
Assumptions
You’re looking to Have your MySQL Queries Run Faster
You want to troubleshoot sudden MySQL Performance Problem
You want to find way to run more efficiently (use less Resources)
![Page 7: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/7.jpg)
7
How to Look at MySQL Performance
Query Based Approach
• All the users (developers) care is how quickly their queries perform
Resource Based Approach
• Queries use resources. Slow Performance often caused by resource constraints
![Page 8: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/8.jpg)
8
Primary Resources
CPU
Disk IO
Memory
Network
![Page 9: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/9.jpg)
9
Low Resource Usage + Poor Performance
Contention
• Table Locks/Row Level Locks • Locking/Latching in MySQL
and Kernel
Mixed Resource Usage
• Single worker spending 33% on CPU
• 33% Waiting on Disk• 33% on Network • Will not be seen as directly
constrained by any resource
![Page 10: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/10.jpg)
10
Load Average
• What can you tell me about server load ?
![Page 11: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/11.jpg)
11
Problems with Load Average
Mixes CPU and IO resource usage (on Linux)
Is not normalized for number of CPU cores available
Does not keep into account Queue Depth Needed for optimal storage performance
![Page 12: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/12.jpg)
12
CPU Usage
• Can observe overall or per core• Matching Load Average in the previous screen
![Page 13: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/13.jpg)
13
Saturation Metrics • Good to understand where waits are happening• IO Load is not normalized
![Page 14: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/14.jpg)
14
Looking at CPU Saturation Separately
• Can normalize CPU Saturation based on number of threads
![Page 15: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/15.jpg)
15
Row Locks – Logical Contention
• Row Locks are often declared by transaction semantics • But more transactions underway also mean more locks
![Page 16: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/16.jpg)
16
Zooming in on Row Locks Wait Load
• How many MySQL Connections are Blocked because or Row Level Lock Waits
![Page 17: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/17.jpg)
17
“Load at MySQL Side”
• “threads_running” - MySQL is busy handling query
• CPU ? Disk ? Row Level Locks ? Need to dig deeper
![Page 18: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/18.jpg)
18
MySQL Questions – Inflow of Queries
• Are we serving more queries or less queries ?• Any spikes or dips ?
![Page 19: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/19.jpg)
19
Innodb Rows – Actual Work Being Done
• Better number to think re system capacity • Not all rows are created equal, but more equal than queries
![Page 20: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/20.jpg)
20
Commands – What kind of operations
• Note if prepared statements are used MySQL is “double counting”
![Page 21: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/21.jpg)
21
MySQL “Handlers” low lever row access
• Works for all storage engines• Gives more details on access type • Mixes Temporary Tables and Non-Temporary tables together
![Page 22: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/22.jpg)
22
Memory usage by MySQL
Leave some memory available for OS Cache and other needs
![Page 23: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/23.jpg)
Innodb in Depth
![Page 24: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/24.jpg)
24
Innodb Checkpointing
• The log file size is good enough as Uncheckpointed bytes are fraction of log file size
![Page 25: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/25.jpg)
25
Innodb Checkpointing
• Very Close – Innodb Log File Size too small for optimal performance
![Page 26: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/26.jpg)
26
Innodb Transaction History - not yet Purged Transactions• Short term spikes are normal if some longer transactions are ran on the
system
![Page 27: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/27.jpg)
27
Innodb Transaction History
• Growth over long period of time without long queries in the processlist• Often identifies orphaned transactions (left open)
![Page 28: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/28.jpg)
28
Transaction History Recovery
• If Backlog is resolved quickly it is great • If not you may be close to the limit of purge subsystem
![Page 29: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/29.jpg)
29
Is your Innodb Log Buffer Large Enough?
• You will be surprised to see how little log buffer space Innodb needs
![Page 30: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/30.jpg)
30
Another way to look at Logging Performance
![Page 31: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/31.jpg)
31
Innodb IO
• Will often roughly match disk IO • Allows to see the writes vs fsyncs
![Page 32: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/32.jpg)
32
Hot Tables
• It is often helpful to know what tables are getting most Reads• And Writes
![Page 33: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/33.jpg)
33
Hot Tables through Performance Schema
• Even more details available in Performance Schema• Load is a better measure of actual cost than number of events
![Page 34: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/34.jpg)
34
Most Active Indexes
• See through which index queries access tables
![Page 35: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/35.jpg)
35
What about Queries causing the most load?• Can examine through Query Analytics application
![Page 36: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/36.jpg)
36
Latency Details Explored
• Not enough to look at Average Latency
![Page 37: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/37.jpg)
37
What are Top Queries ?
Queries Sorted by their “Load”
Query ran 10 times over second each time taking 0.2 sec will be load 2
Not making a difference between queries “causing” the load or just impacted by it
![Page 38: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/38.jpg)
38
Whole Server Summary #1
• Server Summary Gives a good idea what is going on query wise
![Page 39: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/39.jpg)
39
Whole Server Summary #2
![Page 40: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/40.jpg)
40
Specific Query – Update Query
• Significant part of response time comes from row level lock waits
![Page 41: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/41.jpg)
41
Expensive SELECT Query
• Examining lots of rows per each row sent
![Page 42: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/42.jpg)
42
Check Query Example
• Expensive Query not poorly optimized one
![Page 43: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/43.jpg)
43
Explain and JSON Explain
![Page 44: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/44.jpg)
44
Explore Any Captured Metrics
• Standard Dashboards are only tip of the iceberg• You can also use Prometheus directly
![Page 45: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/45.jpg)
Lets Look at Couple of Case Studies
![Page 46: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/46.jpg)
46
Impact Of Durability ?
Running sysbench with rate=1000 to inject 1000 transactions every second
System can handle workloads with both settings
System previously running with sync_binlog=0 and innodb_flush_log_at_trx_commit=0
Set them to sync_binlog=1 and innodb_flush_log_at_trx_commit=1
![Page 47: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/47.jpg)
47
IO Bandwith
• IO Bandwidth is not significantly impacted
![Page 48: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/48.jpg)
48
IO Saturation Jumps a Lot
![Page 49: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/49.jpg)
49
Read and Write Latencies are Impacted
• This SSD (Samsung 960 Pro) Does not like fsync() calls
![Page 50: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/50.jpg)
50
More Disk IO Operations
• Frequent Fsync() causes more writes of smaller size to storage
![Page 51: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/51.jpg)
51
Increase In Disk IO Load
• IO Avg Latency Increase + More IOPs = Load Increase
![Page 52: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/52.jpg)
52
Disk IO Utilization jumps to 100%
• There is at least one disk IO Operation in flight all the time
![Page 53: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/53.jpg)
53
Average IO Size is down
• Large block writes to binlog and innodb transaction logs do not happen any more
![Page 54: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/54.jpg)
54
Number of Running Threads Impacted
• Need higher concurrency to be able to drive same number of queries/sec
![Page 55: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/55.jpg)
55
MySQL Questions
• Why does it increase with same inflow of transactions ?
![Page 56: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/56.jpg)
56
Because of Deadlocks
• Some transactions have to be retried due to deadlocks• Your well designed system should behave the same
![Page 57: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/57.jpg)
57
Higher Row Lock Time
• Rows Locks can be only released after successful transaction commit• Which now takes longer time due to number of fsync() calls
![Page 58: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/58.jpg)
58
And Load Caused by Row Locks
![Page 59: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/59.jpg)
59
Log Buffer Used even less with durability on
![Page 60: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/60.jpg)
60
Is Group Commit Working ?
• Do we relay on Group Commit for our workload
![Page 61: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/61.jpg)
61
Top Queries Impacted
• Commit is now the highest load contributor
![Page 62: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/62.jpg)
Changing Buffer Pool Size
![Page 63: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/63.jpg)
63
MySQL 5.7 Allows to change BP Online
• Changing buffer pool from 48GB to 4GB online
mysql> set global innodb_buffer_pool_size=4096*1024*1024;Query OK, 0 rows affected (0.00 sec)
![Page 64: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/64.jpg)
64
QPS Impact
• While resizing is ongoing capacity is limited – Queueing happens• After resize completed backlog has to be worked off having higher
number of queries
![Page 65: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/65.jpg)
65
Saturation spike and when stabilizing on higher level • Guess why the spike with lower QPS Level ?
![Page 66: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/66.jpg)
66
Two IO Spikes
• First to Flush Dirty Pages • Second to work off higher query rate
![Page 67: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/67.jpg)
67
What is about Disk IO Latency ?
• Higher Number of IOPS does not always mean much higher latency
![Page 68: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/68.jpg)
68
Longer Transactions = More Deadlocks
![Page 69: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/69.jpg)
69
More IO Load Less Contention ?
• Unsure why this is the case • Note not ALL contention is shown in those graphs
![Page 70: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/70.jpg)
70
Now we see query 80% IO Bound
![Page 71: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/71.jpg)
71
Summary
Can get a lot of Insights in MySQL Performance with PMM
Great tool to have when you’re challenged troubleshoot MySQL
A lot of insights during benchmarking and evaluation
![Page 72: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/72.jpg)
72
![Page 73: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/73.jpg)
73
Percona to Support PostgreSQL
![Page 74: MySQL Performance Optimization and Troubleshooting with PMM](https://reader030.fdocuments.net/reader030/viewer/2022012812/61c3bc4257fcf83ca100c206/html5/thumbnails/74.jpg)
Thank You!