Mysql tech day_paris_ps_and_sys

167
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 12 1

Transcript of Mysql tech day_paris_ps_and_sys

Page 1: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 12!1

Page 2: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 12!2

MySQL Tech Day, Paris Performance Schema & MySQL sys

Mark Leith Senior Software Development Manager MySQL @ Oracle

Page 3: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!3

Program Agenda

▪What is Performance Schema anyway? ▪ Performance Schema Configuration ▪ Profiling General Instance Activity ▪ Profiling Statement Activity ▪ Improvements made to date in MySQL 5.7 ▪ The MySQL SYS Schema ▪ Things still to come

Page 4: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!4

Know Your Audience

▪ Using 5.1 (yes, still)? ▪ Using 5.5 (but I’ll upgrade soon, honest)? ▪ Using 5.6? ▪ Using … 5.7? ▪ Used Performance Schema? ▪ #monitoringlove ▪ #monitoringhate

Page 5: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!5

What is Performance Schema anyway?

Page 6: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!6

What is Performance Schema anyway?

▪ Introduced in MySQL 5.5 ▪ A storage engine, built for recording instrumentation

– PERFORMANCE_SCHEMA – Manages fixed buffers in memory with lock free hashes – Real engine, unlike INFORMATION_SCHEMA

▪ A database schema to expose the instrumentation – performance_schema

▪ A set of interfaces in the MySQL Server wrapping calls to trace – i.e. pthread_mutex_lock() -> mysql_mutex_lock()

Page 7: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!7

What is Performance Schema anyway?

▪ Records latency of events that happen within the server ▪ All latency exposed to picosecond (a trillionth of a second) precision

▪ Though may be tracked at different precisions internally ▪ Also tracks other information as appropriate

▪ Bytes, source position, object metadata, etc.

Page 8: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!8

Performance Schema in MySQL 5.5

▪ 17 Tables ▪ 222 Instruments

Instrument Event ClassFile IO wait/io/file/%Mutexes wait/synch/mutex/%Read/Write Locks

wait/synch/rwlock/%Conditions wait/synch/cond/%

Page 9: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!9

Performance Schema in MySQL 5.5

Page 10: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!10

Performance Schema in MySQL 5.5

▪ Laid out the framework for instrumentation ▪ Focused early on low level instruments

– Prove it could handle load, build from ground up ▪ Not turned on by default, had some overhead issues

Page 11: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!11

Wait Eventsmysql> select * from events_waits_history_long where event_name like 'wait/io/table/%'\G!*************************** 1. row ***************************! THREAD_ID: 137746! EVENT_ID: 3604! END_EVENT_ID: 3604! EVENT_NAME: wait/io/table/sql/handler! SOURCE: handler.cc:2568! TIMER_START: 17101711846046490! TIMER_END: 17101711857643010! TIMER_WAIT: 11596520! SPINS: NULL! OBJECT_SCHEMA: mem__events! OBJECT_NAME: events! INDEX_NAME: PRIMARY! OBJECT_TYPE: TABLE!OBJECT_INSTANCE_BEGIN: 140246283168288! NESTING_EVENT_ID: 3603! NESTING_EVENT_TYPE: STAGE! OPERATION: fetch! NUMBER_OF_BYTES: NULL! FLAGS: NULL

Event Context

Event type and origin

Timing (~11.5 microseconds)

Database object info

Object type

Whether the event was nested

Further info as appropriate

Page 12: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!12

Performance Schema in MySQL 5.6

▪ Fixed lots of performance issues ▪ Focused on bringing the instrumentation to the DBA / Developer

▪ Statements / Statement Digests ▪ Execution Stages ▪ Object tracking (table / index IO, table locks) ▪ Network IO

▪ Turned on by default, with a subset of instrumentation disabled

Page 13: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!13

Performance Schema in MySQL 5.6

▪ 52 Tables (+35) ▪ 545 Instruments (+323)

Instrument Type

Event ClassStatements statement/%Stages stage/%Table IO wait/io/table/%Table Locks wait/lock/table/%Network IO wait/io/socket/%Idle Timing idle

Page 14: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!14

Performance Schema in MySQL 5.6

Page 15: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 12!15

“I really think Performance Schema overhead is reasonable for most workloads. … On my old server I got some 20.2K QPS with Performance Schema Disabled and 19.4 QPS with Performance Schema enabled which is overhead of less than 5%. For most workloads paying 5% to have insight about what is happening with the system is a very fair trade.”

▪ Peter Zaitsev, CEO, Percona

http://www.mysqlperformanceblog.com/2014/02/11/performance_schema-vs-slow-query-log/

Page 16: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!16

Statement & Stage Events (5.6)

!▪ Statements have two types of event

▪ SQL Statements statement/sql/% ▪ Protocol Commands statement/com/%

▪ Stages are the thread states. Like SHOW PROFILE output, but available across connections ▪ Mostly one form stage/sql/% ▪ Apart from .. stage/mysys/Waiting for table level lock

Page 17: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!17

Stage Eventsmysql> select * from events_stages_history limit 2\G!*************************** 1. row ***************************! THREAD_ID: 23! EVENT_ID: 18898! END_EVENT_ID: 18898! EVENT_NAME: stage/sql/Master has sent all binlog to slave; waiting for more updates! SOURCE: rpl_binlog_sender.cc:420! TIMER_START: 87465041640516000! TIMER_END: 87465045562629000! TIMER_WAIT: 3922113000! NESTING_EVENT_ID: 10!NESTING_EVENT_TYPE: STATEMENT!*************************** 2. row ***************************! THREAD_ID: 23! EVENT_ID: 18899! END_EVENT_ID: 18900! EVENT_NAME: stage/sql/Sending binlog event to slave! SOURCE: rpl_binlog_sender.cc:432! TIMER_START: 87465045562629000! TIMER_END: 87465045593812000! TIMER_WAIT: 31183000! NESTING_EVENT_ID: 10!NESTING_EVENT_TYPE: STATEMENT

Page 18: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!18

Statement Eventsmysql> select * from events_statements_history_long limit 1\G!*************************** 1. row ***************************! THREAD_ID: 138683! EVENT_ID: 11268! END_EVENT_ID: 11295! EVENT_NAME: statement/sql/update! SOURCE: socket_connection.cc:94! TIMER_START: 87644080265364000! TIMER_END: 87644080504810000! TIMER_WAIT: 239446000! LOCK_TIME: 63000000! SQL_TEXT: /* mem dbpool.default */ update `mem__inventory`.`Agent` set `hasHostname`=1400665852927, `hasReachable`=1400665852927, `timestamp`=1400665852927, `hasVersion`=1400665852927 where hid=x'FA8FDC4C1BC344A0899DAB320757CDF2'! DIGEST: cc389abfcb093ae95cacfe42ed085191! DIGEST_TEXT: UPDATE `mem__inventory` . `Agent` SET `hasHostname` = ? , `hasReachable` = ? , `timestamp` = ? , `hasVersion` = ? WHERE `hid` = ?! CURRENT_SCHEMA: mem! OBJECT_TYPE: NULL! OBJECT_SCHEMA: NULL! OBJECT_NAME: NULL! OBJECT_INSTANCE_BEGIN: NULL!…………

Page 19: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!19

Statement Events (cont.)mysql> select * from events_statements_history_long limit 1\G!*************************** 1. row ***************************!……! MYSQL_ERRNO: 0! RETURNED_SQLSTATE: 00000! MESSAGE_TEXT: Rows matched: 1 Changed: 1 Warnings: 0! ERRORS: 0! WARNINGS: 0! ROWS_AFFECTED: 1! ROWS_SENT: 0! ROWS_EXAMINED: 1!CREATED_TMP_DISK_TABLES: 0! CREATED_TMP_TABLES: 0! SELECT_FULL_JOIN: 0! SELECT_FULL_RANGE_JOIN: 0! SELECT_RANGE: 0! SELECT_RANGE_CHECK: 0! SELECT_SCAN: 0! SORT_MERGE_PASSES: 0! SORT_RANGE: 0! SORT_ROWS: 0! SORT_SCAN: 0

NO_INDEX_USED: 0! NO_GOOD_INDEX_USED: 0! NESTING_EVENT_ID: 11248! NESTING_EVENT_TYPE: TRANSACTION! NESTING_EVENT_LEVEL: 0

Page 20: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!20

Performance Schema Table Types

Setup Tables

▪ Used to define certain configuration dynamically

▪ Can perform DML against these tables

mysql> select table_name! -> from information_schema.tables! -> where table_schema like 'perf%'! -> and table_name like 'setup%';!+-------------------+!| table_name |!+-------------------+!| setup_actors |!| setup_consumers |!| setup_instruments |!| setup_objects |!| setup_timers |!+-------------------+

Page 21: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!21

Performance Schema Table Types

Raw Data Tables

▪ Expose events, objects, or instances of instruments in a raw manner

▪ Allow seeing a (brief) history of raw event metrics as well

+-------------------------------------------+!| table_name |!+-------------------------------------------+!| accounts |!| cond_instances |!| events_stages_current |!| events_stages_history |!| events_stages_history_long |!| events_statements_current |!| events_statements_history |!| events_statements_history_long |!| events_waits_current |!| events_waits_history |!| events_waits_history_long |!| file_instances |!| host_cache |!| hosts |!| mutex_instances |!| performance_timers |!| rwlock_instances |!| session_account_connect_attrs |!| session_connect_attrs |!| socket_instances |!| threads |!| users |!+-------------------------------------------+

Page 22: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!22

Performance Schema Table Types

Summary Tables

▪ Summarise event information over multiple dimensions

▪ Useful for longer term monitoring of activity

+------------------------------------------------------+!| table_name |!+------------------------------------------------------+!| events_stages_summary_by_account_by_event_name |!| events_stages_summary_by_host_by_event_name |!| events_stages_summary_by_thread_by_event_name |!| events_stages_summary_by_user_by_event_name |!| events_stages_summary_global_by_event_name |!| events_statements_summary_by_account_by_event_name |!| events_statements_summary_by_digest |!| events_statements_summary_by_host_by_event_name |!| events_statements_summary_by_program |!| events_statements_summary_by_thread_by_event_name |!| events_statements_summary_by_user_by_event_name |!| events_statements_summary_global_by_event_name |!| events_waits_summary_by_account_by_event_name |!| events_waits_summary_by_host_by_event_name |!| events_waits_summary_by_instance |!| events_waits_summary_by_thread_by_event_name |!| events_waits_summary_by_user_by_event_name |!| events_waits_summary_global_by_event_name |!| file_summary_by_event_name |!| file_summary_by_instance |!| objects_summary_global_by_type |!| socket_summary_by_event_name |!| socket_summary_by_instance |!| table_io_waits_summary_by_index_usage |!| table_io_waits_summary_by_table |!| table_lock_waits_summary_by_table |!+------------------------------------------------------+

Page 23: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!23

Performance Schema Configuration

Page 24: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!24

Performance Schema Configuration

▪ There are three distinct types of configuration ▪ A way to configure how much memory to allocate for instruments

▪ Set with various system variables in options files, not dynamic ▪ A way to enable/disable instrumentation at startup

▪ Again set with options files, only available as of 5.6 ▪ A way to enable/disable instrumentation dynamically

▪ Set by updating the various setup_% tables dynamically

Page 25: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!25

Performance Configuration

History / Summary Size

▪ Define total rows per type ▪ The %_history_size are per-thread ▪ The %_history_long_size are total

rows ▪ Only settable in my.[cnf|ini]

mysql> select variable_name,! -> variable_value! -> from information_schema.global_variables! -> where variable_name like 'perf%'! -> and variable_name not like '%instances'! -> and variable_name not like '%classes'! -> order by variable_name;!+--------------------------------------------------------+----------------+!| variable_name | variable_value |!+--------------------------------------------------------+----------------+!| PERFORMANCE_SCHEMA | ON |!| PERFORMANCE_SCHEMA_ACCOUNTS_SIZE | 100 |!| PERFORMANCE_SCHEMA_DIGESTS_SIZE | 5000 |!| PERFORMANCE_SCHEMA_EVENTS_STAGES_HISTORY_LONG_SIZE | 1000 |!| PERFORMANCE_SCHEMA_EVENTS_STAGES_HISTORY_SIZE | 10 |!| PERFORMANCE_SCHEMA_EVENTS_STATEMENTS_HISTORY_LONG_SIZE | 1000 |!| PERFORMANCE_SCHEMA_EVENTS_STATEMENTS_HISTORY_SIZE | 10 |!| PERFORMANCE_SCHEMA_EVENTS_WAITS_HISTORY_LONG_SIZE | 1000 |!| PERFORMANCE_SCHEMA_EVENTS_WAITS_HISTORY_SIZE | 10 |!| PERFORMANCE_SCHEMA_HOSTS_SIZE | 100 |!| PERFORMANCE_SCHEMA_MAX_FILE_HANDLES | 32768 |!| PERFORMANCE_SCHEMA_MAX_TABLE_HANDLES | 732 |!| PERFORMANCE_SCHEMA_SESSION_CONNECT_ATTRS_SIZE | 512 |!| PERFORMANCE_SCHEMA_SETUP_ACTORS_SIZE | 100 |!| PERFORMANCE_SCHEMA_SETUP_OBJECTS_SIZE | 100 |!| PERFORMANCE_SCHEMA_USERS_SIZE | 100 |!+--------------------------------------------------------+----------------+

Page 26: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!26

Performance Configuration

Max Classes / Instances

▪ Classes count instrument implementations, i.e: ▪ “wait/io/file/sql/binlog”

▪ Instances are the different runtime instances of those things, i.e: ▪ “/data/logs/binlog.000123”

mysql> select variable_name, variable_value! -> from information_schema.global_variables! -> where variable_name like 'perf%classes'! -> or variable_name like 'perf%instances'! -> order by variable_name;!+------------------------------------------------------+----------------+!| variable_name | variable_value |!+------------------------------------------------------+----------------+!| PERFORMANCE_SCHEMA_MAX_COND_CLASSES | 80 |!| PERFORMANCE_SCHEMA_MAX_COND_INSTANCES | 946 |!| PERFORMANCE_SCHEMA_MAX_FILE_CLASSES | 50 |!| PERFORMANCE_SCHEMA_MAX_FILE_INSTANCES | 2170 |!| PERFORMANCE_SCHEMA_MAX_MUTEX_CLASSES | 200 |!| PERFORMANCE_SCHEMA_MAX_MUTEX_INSTANCES | 4586 |!| PERFORMANCE_SCHEMA_MAX_RWLOCK_CLASSES | 40 |!| PERFORMANCE_SCHEMA_MAX_RWLOCK_INSTANCES | 3283 |!| PERFORMANCE_SCHEMA_MAX_SOCKET_CLASSES | 10 |!| PERFORMANCE_SCHEMA_MAX_SOCKET_INSTANCES | 86 |!| PERFORMANCE_SCHEMA_MAX_STAGE_CLASSES | 150 |!| PERFORMANCE_SCHEMA_MAX_STATEMENT_CLASSES | 189 |!| PERFORMANCE_SCHEMA_MAX_TABLE_INSTANCES | 569 |!| PERFORMANCE_SCHEMA_MAX_THREAD_CLASSES | 50 |!| PERFORMANCE_SCHEMA_MAX_THREAD_INSTANCES | 143 |!+------------------------------------------------------+----------------+

Page 27: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!27

Performance Configuration

Max Classes / Instances

▪ Monitor the %_lost status variables opposite to see whether these need tweaking

▪ You will generally not need to modify classes, only when loading new plugins

mysql> select variable_name, variable_value! -> from information_schema.global_status! -> where variable_name like 'perf%classes_lost'! -> or variable_name like 'perf%instances_lost'! -> order by variable_name;!+-------------------------------------------+----------------+!| variable_name | variable_value |!+-------------------------------------------+----------------+!| PERFORMANCE_SCHEMA_COND_CLASSES_LOST | 0 |!| PERFORMANCE_SCHEMA_COND_INSTANCES_LOST | 0 |!| PERFORMANCE_SCHEMA_FILE_CLASSES_LOST | 0 |!| PERFORMANCE_SCHEMA_FILE_INSTANCES_LOST | 0 |!| PERFORMANCE_SCHEMA_MUTEX_CLASSES_LOST | 0 |!| PERFORMANCE_SCHEMA_MUTEX_INSTANCES_LOST | 0 |!| PERFORMANCE_SCHEMA_RWLOCK_CLASSES_LOST | 0 |!| PERFORMANCE_SCHEMA_RWLOCK_INSTANCES_LOST | 0 |!| PERFORMANCE_SCHEMA_SOCKET_CLASSES_LOST | 0 |!| PERFORMANCE_SCHEMA_SOCKET_INSTANCES_LOST | 0 |!| PERFORMANCE_SCHEMA_STAGE_CLASSES_LOST | 0 |!| PERFORMANCE_SCHEMA_STATEMENT_CLASSES_LOST | 0 |!| PERFORMANCE_SCHEMA_TABLE_INSTANCES_LOST | 0 |!| PERFORMANCE_SCHEMA_THREAD_CLASSES_LOST | 0 |!| PERFORMANCE_SCHEMA_THREAD_INSTANCES_LOST | 0 |!+-------------------------------------------+----------------+

Page 28: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!28

Performance Configuration

setup_instruments

▪ Turn on/off individual instruments, or whether to just count and not time events

▪ Use UPDATE to modify dynamically

mysql> select * from setup_instruments limit 5;!+-------------------------------------------------------+---------+-------+!| NAME | ENABLED | TIMED |!+-------------------------------------------------------+---------+-------+!| wait/synch/mutex/sql/TC_LOG_MMAP::LOCK_tc | NO | NO |!| wait/synch/mutex/sql/LOCK_des_key_file | NO | NO |!| wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_commit | NO | NO |!| wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_commit_queue | NO | NO |!| wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_done | NO | NO |!+-------------------------------------------------------+---------+-------+!5 rows in set (0.00 sec)!!mysql> update setup_instruments! -> set enabled = 'yes', timed = 'yes'! -> where name like '%MYSQL_BIN_LOG%';!Query OK, 16 rows affected (0.00 sec)!Rows matched: 16 Changed: 16 Warnings: 0!!mysql> select * from setup_instruments limit 5;!+-------------------------------------------------------+---------+-------+!| NAME | ENABLED | TIMED |!+-------------------------------------------------------+---------+-------+!| wait/synch/mutex/sql/TC_LOG_MMAP::LOCK_tc | NO | NO |!| wait/synch/mutex/sql/LOCK_des_key_file | NO | NO |!| wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_commit | YES | YES |!| wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_commit_queue | YES | YES |!| wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_done | YES | YES |!+-------------------------------------------------------+---------+-------+

Page 29: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!29

Performance Configuration

setup_consumers

▪ Define how much to record, either in history, or summaries (by enabling the %_current for the class of event)

mysql> select * from setup_consumers;!+----------------------------------+---------+!| NAME | ENABLED |!+----------------------------------+---------+!| events_stages_current | NO |!| events_stages_history | NO |!| events_stages_history_long | NO |!| events_statements_current | YES |!| events_statements_history | NO |!| events_statements_history_long | NO |!| events_transactions_current | NO |!| events_transactions_history | NO |!| events_transactions_history_long | NO |!| events_waits_current | NO |!| events_waits_history | NO |!| events_waits_history_long | NO |!| global_instrumentation | YES |!| thread_instrumentation | YES |!| statements_digest | YES |!+----------------------------------+---------+

Page 30: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!30

Performance Schema Consumer Hierarchy

Page 31: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!31

Performance Configuration

setup_objects (5.6)

▪ Define exactly which database objects to monitor

▪ Filters standard databases by default

▪ Allows explicit enable/disable, matches on most specific

mysql> select * from setup_objects;!+-------------+--------------------+-------------+---------+-------+!| OBJECT_TYPE | OBJECT_SCHEMA | OBJECT_NAME | ENABLED | TIMED |!+-------------+--------------------+-------------+---------+-------+!| TABLE | mysql | % | NO | NO |!| TABLE | performance_schema | % | NO | NO |!| TABLE | information_schema | % | NO | NO |!| TABLE | % | % | YES | YES |!+-------------+--------------------+-------------+---------+-------+!4 rows in set (0.00 sec)!!mysql> insert into setup_objects! -> values ('TABLE', 'foo', 'bar', 'no', 'no');!Query OK, 1 row affected (0.01 sec)!!mysql> select * from setup_objects;!+-------------+--------------------+-------------+---------+-------+!| OBJECT_TYPE | OBJECT_SCHEMA | OBJECT_NAME | ENABLED | TIMED |!+-------------+--------------------+-------------+---------+-------+!| TABLE | mysql | % | NO | NO |!| TABLE | performance_schema | % | NO | NO |!| TABLE | information_schema | % | NO | NO |!| TABLE | % | % | YES | YES |!| TABLE | foo | bar | NO | NO |!+-------------+--------------------+-------------+---------+-------+

Page 32: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!32

Performance Configuration

setup_actors (5.6)

▪ Define exactly which users to monitor

▪ By default monitors all connections (% is wildcard)

▪ Modifications only apply to new connections

mysql> select * from setup_actors;!+------+------+------+!| HOST | USER | ROLE |!+------+------+------+!| % | % | % |!+------+------+------+!1 row in set (0.00 sec)!!mysql> delete from setup_actors;!Query OK, 1 row affected (0.00 sec)!!mysql> insert into setup_actors! -> values ('%', 'mark', '%');!Query OK, 1 row affected (0.00 sec)!!mysql> select * from setup_actors;!+------+------+------+!| HOST | USER | ROLE |!+------+------+------+!| % | mark | % |!+------+------+------+!1 row in set (0.00 sec)

Page 33: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!33

Performance Schema Options Files Config

▪ As of 5.6, allows altering configuration at start up ▪ Enabling / disabling instrumentation

▪ performance_schema_instrument = ‘instrument_name=value’ ▪ Value = [on|1|true] | [off|0|false] | counted ▪ instrument_name can have wildcards (‘wait/synch/mutex/%=off’)

▪ Enabling / disabling consumers ▪ performance_schema_consumer_consumer_name=value

▪ Value = [on|1|true] | [off|0|false]

Page 34: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!34

Profiling General Instance Activity

Page 35: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!35

Profiling Considerations

▪ It is easy to just enable everything, but there are overhead concerns ▪ On busy systems mutexes can be locked millions of times per second

▪ It is best to just pick higher latency event types ▪ Statements, Stages, Table IO, File IO, maybe Network IO ▪ It is good to enable all %_current consumers, and statement history

▪ For concurrency/profiling debugging toggle other instruments on an as needed basis

Page 36: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!36

Profiling Types

▪ Once you’ve narrowed down what you’re interested in, there are two ways to start monitoring

▪ View raw data in the summary views ▪ Gives you an overall picture of usage on the instance

▪ Snapshot data, and compute deltas over time ▪ Gives you an idea of the rates of change for the events

Page 37: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!37

Analyzing Global Waits

▪ Some waits can include other waits ▪ Table IO latency may also include some mutex and file IO latency

▪ Wait times include concurrency across all threads ▪ Do not assume you can sum all events in global tables and compare

to wall clock times

Page 38: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!38

Top Waits By Latency

mysql> select event_name,! -> count_star as count,! -> sys.format_time(sum_timer_wait) as total_latency,! -> sys.format_time(avg_timer_wait) as avg_latency,! -> sys.format_time(max_timer_wait) as max_latency! -> from events_waits_summary_global_by_event_name! -> where event_name != 'idle'! -> order by sum_timer_wait desc limit 5;!+--------------------------------------+----------+---------------+-------------+-------------+!| event_name | count | total_latency | avg_latency | max_latency |!+--------------------------------------+----------+---------------+-------------+-------------+!| wait/io/table/sql/handler | 21888502 | 1.27h | 208.06 us | 2.21 s |!| wait/io/file/innodb/innodb_data_file | 4276800 | 00:48:12.49 | 676.32 us | 1.49 s |!| wait/io/file/innodb/innodb_log_file | 1948199 | 00:25:24.36 | 782.45 us | 1.30 s |!| wait/io/file/myisam/kfile | 4566406 | 00:13:45.92 | 180.87 us | 1.17 s |!| wait/io/file/myisam/dfile | 1277589 | 00:05:46.23 | 271.01 us | 1.18 s |!+--------------------------------------+----------+---------------+-------------+-------------+

Page 39: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!39

Analyzing Global Waits

▪ Some mutex events that can affect global concurrency (if high in list): ▪ wait/synch/mutex/innodb/buf_pool_mutex

▪ Increase innodb_buffer_pool_instances ▪ wait/synch/mutex/sql/Query_cache::structure_guard_mutex

▪ Look in to disabling the Query Cache ▪ wait/synch/mutex/myisam/MYISAM_SHARE::intern_lock

▪ Use Innodb …

Page 40: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!40

Analyzing Global Waits

▪ Some File IO events to watch for (if high in list): ▪ wait/io/file/sql/FRM

▪ Tune table_open_cache / table_definition_cache ▪ wait/io/file/sql/file_parser (view definition parsing)

▪ If high on 5.5, upgrade to 5.6, (which can cache these like tables) ▪ wait/io/file/sql/query_log and wait/io/file/sql/slow_log

▪ Disable the general ▪ Disable or tune what is logged to the slow log (decent long_query_time)

Page 41: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!41

Top Files By Total IO mysql> select sys.format_path(file_name) as file,! -> count_read,! -> sys.format_bytes(sum_number_of_bytes_read) as total_read,! -> sys.format_bytes(IFNULL(sum_number_of_bytes_read / count_read, 0)) as avg_read,! -> count_write,! -> sys.format_bytes(sum_number_of_bytes_write) as total_written,! -> sys.format_bytes(IFNULL(sum_number_of_bytes_write / count_write, 0)) as avg_write,! -> sys.format_bytes(sum_number_of_bytes_read + sum_number_of_bytes_write) as total,! -> IFNULL(ROUND(100-((sum_number_of_bytes_read/(sum_number_of_bytes_read + sum_number_of_bytes_write))*100), 2), 0.00) as write_pct! -> from file_summary_by_instance! -> order by (sum_number_of_bytes_read + sum_number_of_bytes_write) desc limit 5;!

+----------------------------------+------------+------------+-----------+-------------+---------------+-----------+-------------+-----------+!| file | count_read | total_read | avg_read | count_write | total_written | avg_write | total | write_pct |!+----------------------------------+------------+------------+-----------+-------------+---------------+-----------+-------------+-----------+!| @@datadir/ibdata1 | 888 | 15.84 MiB | 18.27 KiB | 1089824 | 61.99 GiB | 59.64 KiB | 62.00 GiB | 99.98 |!| @@datadir/mem__events/events.ibd | 114 | 1.80 MiB | 16.14 KiB | 117370 | 2.14 GiB | 19.14 KiB | 2.14 GiB | 99.92 |!| @@datadir/cerberus-bin.000010 | 296103 | 1.01 GiB | 3.59 KiB | 362852 | 1.00 GiB | 2.89 KiB | 2.01 GiB | 49.63 |!| @@datadir/ib_logfile0 | 6 | 68.00 KiB | 11.33 KiB | 506837 | 1.46 GiB | 3.03 KiB | 1.46 GiB | 100.00 |!| @@datadir/ib_logfile1 | 0 | 0 bytes | 0 bytes | 476961 | 1.45 GiB | 3.19 KiB | 1.45 GiB | 100.00 |!+----------------------------------+------------+------------+-----------+-------------+---------------+-----------+-------------+-----------+

High IO on InnoDB per-tablespace tables can show candidates for a separate mountpoint/disk using “DATA DIRECTORY” in CREATE TABLE

Page 42: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!42

Analyzing User Activity

▪ All event_% summaries are exposed with a number of dimensions ▪ To analyze connection activity you can do this in 4 ways

▪ By User ▪ By Host ▪ By Account (User@Host) ▪ By Thread

▪ The follow examples are by user, but could be replaced with the host, account or thread summary views

Page 43: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!43

Top Users By Statement Latency

mysql> select user,! -> sum(count_star) as statements,! -> sys.format_time(sum(sum_timer_wait)) as total_latency,! -> sys.format_time(sum(sum_timer_wait) / sum(count_star)) as avg_latency! -> from events_statements_summary_by_user_by_event_name! -> where user is not null! -> group by user! -> order by sum(sum_timer_wait) desc;!+------+------------+---------------+-------------+!| user | statements | total_latency | avg_latency |!+------+------------+---------------+-------------+!| root | 7229032 | 15.17h | 7.55 ms |!| mark | 3072 | 1.77 s | 575.29 us |!+------+------------+---------------+-------------+!

Page 44: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!44

Top Users By IO Latency

mysql> select user, sum(count_star) as count,! -> sys.format_time(sum(sum_timer_wait)) as total_latency! -> from events_waits_summary_by_user_by_event_name! -> where event_name like 'wait/io/file/%'! -> and user is not null! -> group by user! -> order by sum(sum_timer_wait) desc;!+------+----------+---------------+!| user | count | total_latency |!+------+----------+---------------+!| root | 10892980 | 00:46:19.67 |!| mark | 20043 | 346.79 ms |!+------+----------+---------------+!

Replace with some other pattern here for “Top user by

wait class”

Page 45: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!45

Top Users By Connectionsmysql> select * from users where user is not null order by current_connections desc;!+------+---------------------+-------------------+!| USER | CURRENT_CONNECTIONS | TOTAL_CONNECTIONS |!+------+---------------------+-------------------+!| root | 8 | 151655 |!| mark | 1 | 1 |!+------+---------------------+-------------------+!2 rows in set (0.00 sec)!!mysql> select * from accounts where user is not null order by current_connections desc;!+------+-----------+---------------------+-------------------+!| USER | HOST | CURRENT_CONNECTIONS | TOTAL_CONNECTIONS |!+------+-----------+---------------------+-------------------+!| root | localhost | 39 | 151698 |!| mark | localhost | 1 | 1 |!+------+-----------+---------------------+-------------------+!2 rows in set (0.00 sec)

Page 46: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!46

User Statement Activitymysql> select *! -> from events_statements_summary_by_user_by_event_name! -> where user is not null! -> order by user, sum_timer_wait desc limit 20\G!*************************** 1. row ***************************! USER: mark! EVENT_NAME: statement/sql/select! COUNT_STAR: 4! SUM_TIMER_WAIT: 1692883853000! MIN_TIMER_WAIT: 88330000! AVG_TIMER_WAIT: 423220963000! MAX_TIMER_WAIT: 1692583043000! SUM_LOCK_TIME: 408000000! SUM_ERRORS: 0! SUM_WARNINGS: 0! SUM_ROWS_AFFECTED: 0! SUM_ROWS_SENT: 212! SUM_ROWS_EXAMINED: 15067!SUM_CREATED_TMP_DISK_TABLES: 3! SUM_CREATED_TMP_TABLES: 4

SUM_SELECT_FULL_JOIN: 0! SUM_SELECT_FULL_RANGE_JOIN: 0! SUM_SELECT_RANGE: 0! SUM_SELECT_RANGE_CHECK: 0! SUM_SELECT_SCAN: 2! SUM_SORT_MERGE_PASSES: 6! SUM_SORT_RANGE: 0! SUM_SORT_ROWS: 6458! SUM_SORT_SCAN: 2! SUM_NO_INDEX_USED: 1! SUM_NO_GOOD_INDEX_USED: 0

Summarized by statement type, e.g:

!statement/sql/select statement/sql/update

Page 47: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!47

User Top Stagesmysql> select user, event_name as stage,! -> count_star as total,! -> sys.format_time(sum_timer_wait) as total_latency,! -> sys.format_time(avg_timer_wait) as avg_latency,! -> sys.format_time(max_timer_wait) as max_latency! -> from events_stages_summary_by_user_by_event_name! -> where sum_timer_wait > 0! -> order by user, sum_timer_wait desc;!+------+--------------------------------+--------+---------------+-------------+-------------+!| user | stage | total | total_latency | avg_latency | max_latency |!+------+--------------------------------+--------+---------------+-------------+-------------+!| root | stage/sql/starting | 591184 | 00:47:52.47 | 4.86 ms | 3.96 s |!| root | stage/sql/update | 193940 | 00:15:00.84 | 4.64 ms | 4.02 s |!| root | stage/sql/updating | 50334 | 00:01:41.38 | 2.01 ms | 2.35 s |!| root | stage/sql/statistics | 73869 | 00:01:41.38 | 1.37 ms | 4.54 s |!| root | stage/sql/Sending data | 73211 | 00:01:38.31 | 1.34 ms | 1.04 s |!| root | stage/sql/Opening tables | 404230 | 00:01:23.48 | 206.51 us | 473.72 ms |!| root | stage/sql/closing tables | 567351 | 00:01:13.40 | 129.38 us | 79.52 ms |!| root | stage/sql/removing tmp table | 11603 | 56.87 s | 4.90 ms | 964.17 ms |!| root | stage/sql/freeing items | 567520 | 19.75 s | 34.80 us | 219.19 ms |!| root | stage/sql/init | 316805 | 8.26 s | 26.06 us | 217.54 ms |!………

Page 48: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!48

Analyzing Table Activity

▪ 3 summary views have been added to 5.6 on top of Table IO ▪ objects_summary_global_by_type

▪ A high level aggregate view of database object latency ▪ table_io_waits_summary_by_table

▪ A detailed aggregate by table, breaking down reads, writes, etc. ▪ table_io_waits_summary_by_index_usage

▪ Further breaking down usage per index

Page 49: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!49

Top Tables By Latencymysql> select object_schema,! -> object_name,! -> count_star,! -> sys.format_time(sum_timer_wait) as total_latency,! -> sys.format_time(sum_timer_wait / count_star) as avg_latency,! -> sys.format_time(max_timer_wait) as max_latency! -> from objects_summary_global_by_type! -> order by sum_timer_wait desc limit 5;!+------------------+-------------------+------------+---------------+-------------+-------------+!| object_schema | object_name | count_star | total_latency | avg_latency | max_latency |!+------------------+-------------------+------------+---------------+-------------+-------------+!| mem__inventory | mysqlserver | 1049496 | 00:04:48.32 | 274.72 us | 1.50 s |!| mem__events | events | 227939 | 00:01:00.50 | 265.41 us | 2.15 s |!| mem__events | event_chain_heads | 328664 | 50.80 s | 154.57 us | 2.21 s |!| mem__inventory | agent | 442751 | 44.82 s | 101.23 us | 1.69 s |!| mem__instruments | fsstatistics | 2233484 | 16.53 s | 7.40 us | 269.27 ms |!+------------------+-------------------+------------+---------------+-------------+-------------+

Less events, yet higher latency, is a sign that there could be

concurrency issues

Page 50: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!50

Table Usage Detailed Breakdown!mysql> select object_schema, object_name,! -> count_fetch as selects, sys.format_time(sum_timer_fetch) as select_latency,! -> count_insert as inserts, sys.format_time(sum_timer_insert) as insert_latency,! -> count_update as updates, sys.format_time(sum_timer_update) as update_latency,! -> count_delete as deletes, sys.format_time(sum_timer_delete) as delete_latency! -> from table_io_waits_summary_by_table! -> order by sum_timer_wait desc limit 10;!+------------------+-------------------+---------+----------------+---------+----------------+---------+----------------+---------+----------------+!| object_schema | object_name | selects | select_latency | inserts | insert_latency | updates | update_latency | deletes | delete_latency |!+------------------+-------------------+---------+----------------+---------+----------------+---------+----------------+---------+----------------+!| mem__inventory | mysqlserver | 418518 | 00:04:32.21 | 0 | 0 ps | 48534 | 15.51 s | 0 | 0 ps |!| mem__events | events | 58512 | 5.54 s | 2701 | 4.01 s | 52414 | 50.83 s | 0 | 0 ps |!| mem__events | event_chain_heads | 156194 | 6.93 s | 78 | 59.75 ms | 52366 | 43.60 s | 0 | 0 ps |!| mem__inventory | agent | 188113 | 40.06 s | 0 | 0 ps | 6998 | 4.54 s | 0 | 0 ps |!| mem__instruments | fsstatistics | 2222058 | 15.12 s | 1517 | 1.37 s | 1 | 72.69 us | 0 | 0 ps |!| mem__instruments | qrtidata | 2771546 | 2.57 s | 3030 | 12.52 s | 0 | 0 ps | 0 | 0 ps |!| mem__inventory | network | 91187 | 2.24 s | 0 | 0 ps | 11864 | 8.42 s | 0 | 0 ps |!| mem__inventory | os | 79736 | 6.08 s | 0 | 0 ps | 8827 | 4.07 s | 0 | 0 ps |!| mem__inventory | networking | 99114 | 4.24 s | 0 | 0 ps | 5928 | 5.54 s | 0 | 0 ps |!| mem__inventory | environment | 100445 | 5.29 s | 0 | 0 ps | 6540 | 2.53 s | 0 | 0 ps |!+------------------+-------------------+---------+----------------+---------+----------------+---------+----------------+---------+----------------+

Page 51: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!51

Analyzing Table Activity

Tables with full scans

▪ Search in table_io_waits_summary_by_index_usage where index_name is null

▪ This is a catch all row for rows that are read without indexes

mysql> select concat(object_schema, '.', object_name) as object,! -> count_read as rows_scanned,! -> sys.format_time(sum_timer_wait) as latency! -> from table_io_waits_summary_by_index_usage! -> where index_name is null! -> and count_read > 0! -> order by sum_timer_wait desc limit 20;!+---------------------------------------+--------------+-----------+!| object | rows_scanned | latency |!+---------------------------------------+--------------+-----------+!| mem__instruments.qrtidata | 2735434 | 14.56 s |!| mem__instruments.databaseactivitydata | 2735633 | 4.68 s |!| mem__instruments.diskiototaldata | 31813 | 3.26 s |!| mem__instruments.connectionsdata | 2530816 | 3.17 s |!| mem__inventory.agent | 117544 | 2.89 s |!| mem__instruments.fsstatistics | 738626 | 2.69 s |!| mem__instruments.diskioopstotaldata | 25955 | 2.43 s |!| mem__inventory.environment | 42549 | 1.55 s |!| mem__enterprise.whats_new_entries | 1453 | 1.16 s |!| mem__inventory.networking | 42549 | 559.14 ms |!+---------------------------------------+--------------+-----------+

Page 52: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!52

Analyzing Table Activity

Unused Indexes

▪ Search in table_io_waits_summary_by_index_usage where count_star = 0

▪ You should ensure you have representative time frame before taking any actions!

mysql> select object_schema,! -> object_name,! -> index_name! -> from table_io_waits_summary_by_index_usage! -> where index_name is not null! -> and count_star = 0! -> order by object_schema, object_name limit 10;!+-------------------+---------------------+------------+!| object_schema | object_name | index_name |!+-------------------+---------------------+------------+!| mem__advisors | advisor_initialized | PRIMARY |!| mem__advisors | advisor_schedules | PRIMARY |!| mem__advisors | app_identity_path | PRIMARY |!| mem__advisor_text | template_meta | PRIMARY |!| mem__bean_config | plists | path |!| mem__bean_config | plists | PRIMARY |!| mem__bean_config | plist_name_values | PRIMARY |!| mem__config | asset_notes | PRIMARY |!| mem__config | group_selections | PRIMARY |!| mem__config | group_selections | name |!+-------------------+---------------------+------------+

Page 53: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!53

Host Cache Usage and Errors

▪ New host_cache view within 5.6, exposing hosts in the host cache ▪ Counts errors, by the type of error that can occur ▪ Can show when errors started happening ▪ Compare the sum_connect_errors counter to the

max_connect_errors system variable

Page 54: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!54

host_cachemysql> desc host_cache;!+--------------------------------------------+------------------+------+-----+---------------------+-------+!| Field | Type | Null | Key | Default | Extra |!+--------------------------------------------+------------------+------+-----+---------------------+-------+!| IP | varchar(64) | NO | | NULL | |!| HOST | varchar(255) | YES | | NULL | |!| HOST_VALIDATED | enum('YES','NO') | NO | | NULL | |!| SUM_CONNECT_ERRORS | bigint(20) | NO | | NULL | |!| COUNT_HOST_BLOCKED_ERRORS | bigint(20) | NO | | NULL | |!| COUNT_NAMEINFO_TRANSIENT_ERRORS | bigint(20) | NO | | NULL | |!| COUNT_NAMEINFO_PERMANENT_ERRORS | bigint(20) | NO | | NULL | |!| COUNT_FORMAT_ERRORS | bigint(20) | NO | | NULL | |!| COUNT_ADDRINFO_TRANSIENT_ERRORS | bigint(20) | NO | | NULL | |!| COUNT_ADDRINFO_PERMANENT_ERRORS | bigint(20) | NO | | NULL | |!| COUNT_FCRDNS_ERRORS | bigint(20) | NO | | NULL | |!| COUNT_HOST_ACL_ERRORS | bigint(20) | NO | | NULL | |!| COUNT_NO_AUTH_PLUGIN_ERRORS | bigint(20) | NO | | NULL | |!| COUNT_AUTH_PLUGIN_ERRORS | bigint(20) | NO | | NULL | |!| COUNT_HANDSHAKE_ERRORS | bigint(20) | NO | | NULL | |!| COUNT_PROXY_USER_ERRORS | bigint(20) | NO | | NULL | |!| COUNT_PROXY_USER_ACL_ERRORS | bigint(20) | NO | | NULL | |!| COUNT_AUTHENTICATION_ERRORS | bigint(20) | NO | | NULL | |!| COUNT_SSL_ERRORS | bigint(20) | NO | | NULL | |!| COUNT_MAX_USER_CONNECTIONS_ERRORS | bigint(20) | NO | | NULL | |!| COUNT_MAX_USER_CONNECTIONS_PER_HOUR_ERRORS | bigint(20) | NO | | NULL | |!| COUNT_DEFAULT_DATABASE_ERRORS | bigint(20) | NO | | NULL | |!| COUNT_INIT_CONNECT_ERRORS | bigint(20) | NO | | NULL | |!| COUNT_LOCAL_ERRORS | bigint(20) | NO | | NULL | |!| COUNT_UNKNOWN_ERRORS | bigint(20) | NO | | NULL | |!| FIRST_SEEN | timestamp | NO | | 0000-00-00 00:00:00 | |!| LAST_SEEN | timestamp | NO | | 0000-00-00 00:00:00 | |!| FIRST_ERROR_SEEN | timestamp | YES | | 0000-00-00 00:00:00 | |!| LAST_ERROR_SEEN | timestamp | YES | | 0000-00-00 00:00:00 | |

Page 55: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!55

Computing Rates Of Change

▪ To look at how things change over time, you have a couple of options ▪ Record the events you are interested in within persistent history tables

▪ Get current stats from performance_schema, the last rows of history, compute the time delta and event count deltas, store new deltas along with the raw values

▪ Or use some tool like MySQL Enterprise Monitor ▪ Read the tables regularly, and run TRUNCATE TABLE on them

immediately afterwards ▪ Some users have done this by dumping local traces to files on

disk, and resetting stats every 15 minutes, for example

Page 56: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!56

Computing Rates Of Change

▪ For an example, the following event tracks the time spent waiting in the replication SQL thread for new events to be written to the relay log by the IO thread ▪ wait/synch/cond/sql/MYSQL_RELAY_LOG::update_cond

▪ This is essentially the SQL thread idle time, for a single SQL thread, if we snapshot this over time, we can find overall busy time

Page 57: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!57

Slave SQL Load Average

▪ An example of monitoring this event over time !

http://www.markleith.co.uk/2012/07/24/a-mysql-replication-load-average-with-performance-schema/

Page 58: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!58

Slave SQL Load Average

http://www.markleith.co.uk/2012/07/24/a-mysql-replication-load-average-with-performance-schema/

Page 59: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!59

Computing Rates Of Change

▪ If interested in calculating percentages, copy the approach shown in the replication load average – you have to take in to account waiting 100% on other events, or just this event, etc.

▪ Note, this method can not be used generally, you have to be able to correlate it to specific thread(s)

▪ As latency is per thread, and you have to compare to a known time delta, summary views do not give this to you (you don’t get thread counts per interval)

▪ Should still compute the deltas for global events however, for event throughput / latency statistics over time !

Page 60: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!60

Profiling Statement Activity

Page 61: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!61

Statement Profiling Options

▪ The new instrumentation in 5.6 gives many options ▪ See statements running currently with events_statements_current ▪ Summary views by user, host, account ▪ Statement histories within events_statements_history% tables ▪ A normalized view within events_statements_summary_by_digest

Page 62: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!62

Currently Executing Statementsmysql> select * from events_statements_current where timer_end is null\G!*************************** 8. row ***************************! THREAD_ID: 156945! EVENT_ID: 15312! END_EVENT_ID: NULL! EVENT_NAME: statement/sql/select! SOURCE: socket_connection.cc:94! TIMER_START: 99343994725205000! TIMER_END: NULL! TIMER_WAIT: NULL! LOCK_TIME: 145000000! SQL_TEXT: /* mem dbpool.ui */ select normalized0_.round_robin_bin as round1_1256_, /* ..snip .. */! DIGEST: NULL! DIGEST_TEXT: NULL! CURRENT_SCHEMA: mem! OBJECT_TYPE: NULL! OBJECT_SCHEMA: NULL! OBJECT_NAME: NULL! OBJECT_INSTANCE_BEGIN: NULL! MYSQL_ERRNO: 0! RETURNED_SQLSTATE: NULL! MESSAGE_TEXT: NULL! ERRORS: 0! WARNINGS: 0! ROWS_AFFECTED: 0! ROWS_SENT: 27! ROWS_EXAMINED: 0!CREATED_TMP_DISK_TABLES: 0! CREATED_TMP_TABLES: 0

SELECT_FULL_JOIN: 0! SELECT_FULL_RANGE_JOIN: 0! SELECT_RANGE: 1! SELECT_RANGE_CHECK: 0! SELECT_SCAN: 0! SORT_MERGE_PASSES: 0! SORT_RANGE: 0! SORT_ROWS: 0! SORT_SCAN: 0! NO_INDEX_USED: 0! NO_GOOD_INDEX_USED: 0! NESTING_EVENT_ID: NULL! NESTING_EVENT_TYPE: NULL! NESTING_EVENT_LEVEL: 0

All counters are live, and increment whilst the statements

execute

Page 63: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!63

Per Thread Statement Historymysql> select thread_id,! -> sys.format_statement(sql_text) as stmt,! -> sys.format_time(timer_wait) as latency! -> from events_statements_history! -> order by thread_id, event_id;!+-----------+-------------------------------------------------------------------+-----------+!| thread_id | stmt | latency |!+-----------+-------------------------------------------------------------------+-----------+!...!| 158162 | /* mem dbpool.default */ commit | 5.39 ms |!| 158162 | /* mem dbpool.default */ inser ... currentlyOpen), currentlyOpen) | 1.25 ms |!| 158162 | /* mem dbpool.default */ commit | 9.69 ms |!| 158162 | /* mem dbpool.default */ inser ... , VALUES(lastSeen)), lastSeen) | 259.75 us |!| 158162 | /* mem dbpool.default */ inser ... (bytesTotal), bytesTotal), col | 383.05 us |!| 158162 | /* mem dbpool.default */ commit | 51.41 ms |!| 158162 | /* mem dbpool.default */ inser ... ullScan), rowsReadviaFullScan) | 147.04 ms |!| 158162 | /* mem dbpool.default */ commit | 33.87 ms |!| 158162 | /* mem dbpool.default */ inser ... ALUES(indexUsage), indexUsage) | 1.10 ms |!| 158162 | /* mem dbpool.default */ commit | 14.78 ms |!...!+-----------+-------------------------------------------------------------------+-----------+!370 rows in set (0.17 sec)

Also has same columns as events_statements_current

Page 64: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!64

Statement Digests

▪ A new summary view introduced in 5.6 ▪ Aggregates statistics based on normalised statements ▪ Can be used to drill in to your problem statements instead of using the

slow query log (with less contention, and no need for post-processing) ▪ Each statement gets an MD5 ‘DIGEST’, that is also available to link in

to events_statements_history% etc. for raw per-query statistics

Page 65: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!65

events_statements_summary_by_digestmysql> desc events_statements_summary_by_digest;!+-----------------------------+---------------------+------+-----+---------------------+-------+!| Field | Type | Null | Key | Default | Extra |!+-----------------------------+---------------------+------+-----+---------------------+-------+!| SCHEMA_NAME | varchar(64) | YES | | NULL | |!| DIGEST | varchar(32) | YES | | NULL | |!| DIGEST_TEXT | longtext | YES | | NULL | |!| COUNT_STAR | bigint(20) unsigned | NO | | NULL | |!| SUM_TIMER_WAIT | bigint(20) unsigned | NO | | NULL | |!| MIN_TIMER_WAIT | bigint(20) unsigned | NO | | NULL | |!| AVG_TIMER_WAIT | bigint(20) unsigned | NO | | NULL | |!| MAX_TIMER_WAIT | bigint(20) unsigned | NO | | NULL | |!| SUM_LOCK_TIME | bigint(20) unsigned | NO | | NULL | |!| SUM_ERRORS | bigint(20) unsigned | NO | | NULL | |!| SUM_WARNINGS | bigint(20) unsigned | NO | | NULL | |!| SUM_ROWS_AFFECTED | bigint(20) unsigned | NO | | NULL | |!| SUM_ROWS_SENT | bigint(20) unsigned | NO | | NULL | |!| SUM_ROWS_EXAMINED | bigint(20) unsigned | NO | | NULL | |!| SUM_CREATED_TMP_DISK_TABLES | bigint(20) unsigned | NO | | NULL | |!| SUM_CREATED_TMP_TABLES | bigint(20) unsigned | NO | | NULL | |!| SUM_SELECT_FULL_JOIN | bigint(20) unsigned | NO | | NULL | |!| SUM_SELECT_FULL_RANGE_JOIN | bigint(20) unsigned | NO | | NULL | |!| SUM_SELECT_RANGE | bigint(20) unsigned | NO | | NULL | |!| SUM_SELECT_RANGE_CHECK | bigint(20) unsigned | NO | | NULL | |!| SUM_SELECT_SCAN | bigint(20) unsigned | NO | | NULL | |!| SUM_SORT_MERGE_PASSES | bigint(20) unsigned | NO | | NULL | |!| SUM_SORT_RANGE | bigint(20) unsigned | NO | | NULL | |!| SUM_SORT_ROWS | bigint(20) unsigned | NO | | NULL | |!| SUM_SORT_SCAN | bigint(20) unsigned | NO | | NULL | |!| SUM_NO_INDEX_USED | bigint(20) unsigned | NO | | NULL | |!| SUM_NO_GOOD_INDEX_USED | bigint(20) unsigned | NO | | NULL | |!| FIRST_SEEN | timestamp | NO | | 0000-00-00 00:00:00 | |!| LAST_SEEN | timestamp | NO | | 0000-00-00 00:00:00 | |

Page 66: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!66

Statement Normalization

▪ Normalization folds certain constructs of statements ▪ Strip whitespace / comments ▪ Replace literals with ?

▪ “WHERE foo = 1” becomes “WHERE foo = ?” ▪ Fold lists of things

▪ “IN (1,2,3)” becomes “IN (…)” ▪ Fold multi-row inserts

▪ “VALUES (1), (2), (3)” becomes “VALUES (?) /*, … */” !

Page 67: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!67

Statement Digest Output

mysql> select * from events_statements_summary_by_digest order by sum_timer_wait desc limit 5\G!...!*************************** 4. row ***************************! SCHEMA_NAME: mem! DIGEST: 5f41a0036bae4fa1c79339c1c7da3c9e! DIGEST_TEXT: SELECT DISTINCTROW `agent0_` . `hid` AS `hid1239_` ...! COUNT_STAR: 64136! SUM_TIMER_WAIT: 1835031591954000! MIN_TIMER_WAIT: 209280000! AVG_TIMER_WAIT: 28611569000! MAX_TIMER_WAIT: 4050621865000! SUM_LOCK_TIME: 31388434000000! SUM_ERRORS: 0! SUM_WARNINGS: 0! SUM_ROWS_AFFECTED: 0! SUM_ROWS_SENT: 64133! SUM_ROWS_EXAMINED: 187363!SUM_CREATED_TMP_DISK_TABLES: 59624! SUM_CREATED_TMP_TABLES: 59624!

SUM_SELECT_FULL_JOIN: 0! SUM_SELECT_FULL_RANGE_JOIN: 0! SUM_SELECT_RANGE: 0! SUM_SELECT_RANGE_CHECK: 0! SUM_SELECT_SCAN: 59625! SUM_SORT_MERGE_PASSES: 0! SUM_SORT_RANGE: 0! SUM_SORT_ROWS: 0! SUM_SORT_SCAN: 0! SUM_NO_INDEX_USED: 59625! SUM_NO_GOOD_INDEX_USED: 0! FIRST_SEEN: 2014-05-20 11:26:54! LAST_SEEN: 2014-05-21 14:50:54

Page 68: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!68

Statements With Temporary Tables

mysql> select sys.format_statement(digest_text) as stmt,! -> count_star as total,! -> sum_created_tmp_tables as in_memory,! -> sum_created_tmp_disk_tables as on_disk,! -> round(sum_created_tmp_tables / count_star) as avg_per_stmt,! -> round((sum_created_tmp_disk_tables/sum_created_tmp_tables) * 100) as to_disk_pct! -> from events_statements_summary_by_digest! -> where sum_created_tmp_tables > 0! -> order by sum_created_tmp_disk_tables desc, sum_created_tmp_tables desc limit 5;!+-------------------------------------------------------------------+-------+-----------+---------+--------------+-------------+!| stmt | total | in_memory | on_disk | avg_per_stmt | to_disk_pct |!+-------------------------------------------------------------------+-------+-----------+---------+--------------+-------------+!| SELECT DISTINCTROW `agent0_` . ... gent` `agent0_` INNER JOIN ... | 64668 | 60116 | 60116 | 1 | 100 |!| SELECT DISTINCTROW `mysqlconne ... conne0_` . `socketPath` AS ... | 5605 | 5604 | 5604 | 1 | 100 |!| SELECT * FROM ( SELECT digest ... ed AS `noIndexUsedCount` , ... | 1431 | 10017 | 4293 | 7 | 43 |!| SELECT `s` . `identityId` , `s ... `subject_id` = `s` . `id` ... | 1406 | 4218 | 2812 | 3 | 67 |!| SELECT plugin_name FROM inform ... atus = ? ORDER BY plugin_name | 1726 | 1726 | 1726 | 1 | 100 |!+-------------------------------------------------------------------+-------+-----------+---------+--------------+-------------+!5 rows in set (0.01 sec)

Page 69: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!69

Statements With Full Table Scans

mysql> select sys.format_statement(digest_text) as stmt,! -> count_star as total,! -> sum_no_index_used as scan_count,! -> round((sum_no_index_used/count_star) * 100) as scan_pct,! -> sum_rows_examined as rows_scanned! -> from events_statements_summary_by_digest! -> where digest_text like 'select%'! -> and (sum_no_index_used > 0 OR sum_no_good_index_used > 0)! -> order by rows_scanned desc limit 5;!+-------------------------------------------------------------------+-------+------------+----------+--------------+!| stmt | total | scan_count | scan_pct | rows_scanned |!+-------------------------------------------------------------------+-------+------------+----------+--------------+!| SELECT `hid` , TIMESTAMP , `en ... DIV ? AS `slice` , `hid` , ... | 4258 | 4257 | 100 | 31644768 |!| SELECT TIMESTAMP , SUM ( `sele ... by_bucket` GROUP BY TIMESTAMP | 807 | 807 | 100 | 4657959 |!| SELECT TIMESTAMP , SUM ( `opti ... by_bucket` GROUP BY TIMESTAMP | 807 | 807 | 100 | 4656726 |!| SELECT TIMESTAMP , SUM ( `tota ... by_bucket` GROUP BY TIMESTAMP | 807 | 807 | 100 | 4454821 |!| SELECT * FROM ( SELECT digest ... ed AS `noIndexUsedCount` , ... | 1434 | 1434 | 100 | 3874018 |!+-------------------------------------------------------------------+-------+------------+----------+--------------+

Page 70: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!70

Tracing Individual Statement Activity

▪ By taking all available data within the %_history_long tables, we can build a complete picture of where latency lines within statements

▪ We can link the histories of Statements, Stages and Waits using the event_id and nesting_event_id columns, which define hierarchy !

!

Page 71: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!71

Tracing Individual Statement Activity

Page 72: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!72

This means we can graph their relationships!

Page 73: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!73

Tracing Individual Statement Activity

Page 74: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!74

Tracing Individual Statement Activity

▪ This is only really effective on development / test systems, production systems with high concurrency / throughput age history very quickly, and turning the %_history_long tables on with all instrumentation on is not recommend generally

▪ Good to disable all other threads, or all actors, and just enable a single thread in development / test environments, to get a full trace

▪ Procedure to dump the graph data is available within the MySQL SYS schema !

!

Page 75: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!75

Improvements made to date in MySQL 5.7

Page 76: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 12

Insert Picture Here

!76

▪Memory Usage ▪Metadata Locking ▪ Replication Configuration & Status ▪ Prepared Statements ▪ Transactions ▪ Stored Programs

Page 77: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!77

Improvements made to date in MySQL 5.7

▪ 75 Tables (+23) ▪ 784 Instruments (+239)

Instrument Type Event Class

Transactions transaction

Memory memory/%

Metadata Locks wait/lock/metadata/%

Page 78: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!78

Improvements made to date in MySQL 5.7

Page 79: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!79

WL#3249!PERFORMANCE SCHEMA, Instrument memory usage

Page 80: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 12!80

“Understanding where MySQL can allocate memory can help us to find the cause in most cases. It is not as straightforward as it should be and I’m very hopeful future releases of MySQL, MariaDB or Drizzle bring improvements in this space allowing us to see directly for what purpose memory is allocated and so detect all kinds of memory usage problems easier.”

▪ Peter Zaitsev, CEO, Percona

http://www.mysqlperformanceblog.com/2012/03/21/troubleshooting-mysql-memory-usage/

Page 81: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!81

Instrument memory usage

▪ Added 212 different memory instrumentation types so far ▪ Records current, high and low water marks of allocations ▪ Does not track latency of memory allocation ▪ 5 new tables

▪ memory_summary_by_account_by_event_name ▪ memory_summary_by_host_by_event_name ▪ memory_summary_by_thread_by_event_name ▪ memory_summary_by_user_by_event_name ▪ memory_summary_global_by_event_name

Page 82: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!82

Instrument memory usage - global summarymysql> SELECT * FROM sys.memory_global_by_current_allocated\G!*************************** 1. row ***************************! event_name: memory/performance_schema/internal_buffers! current_count: 60! current_alloc: 497.00 MiB!current_avg_alloc: 8.28 MiB! high_count: 60! high_alloc: 497.00 MiB! high_avg_alloc: 8.28 MiB!*************************** 2. row ***************************! event_name: memory/mysys/KEY_CACHE! current_count: 3! current_alloc: 8.00 MiB!current_avg_alloc: 2.67 MiB! high_count: 3! high_alloc: 8.00 MiB! high_avg_alloc: 2.67 MiB

Page 83: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!83

Instrument memory usage - thread summariesmysql> select * from memory_by_thread_by_current_allocated\G!*************************** 1. row ***************************! user: sql/main! current_count: 2407! current_alloc: 10.89 MiB!current_avg_alloc: 4.63 KiB!current_max_alloc: 8.00 MiB! total_allocated: 30.55 MiB! thread_id: 1!*************************** 2. row ***************************! user: mem@localhost! current_count: 1914! current_alloc: 1.50 MiB!current_avg_alloc: 824 bytes!current_max_alloc: 816.67 KiB! total_allocated: 9.25 GiB! thread_id: 4336

Page 84: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!84

Instrument memory usage - thread detailsmysql> SELECT event_name,! -> sys.format_bytes(current_number_of_bytes_used) AS current_used! -> FROM performance_schema.memory_summary_by_thread_by_event_name! -> WHERE thread_id = 24! -> ORDER BY current_number_of_bytes_used DESC;!+-----------------------------------------------------+--------------+!| event_name | current_used |!+-----------------------------------------------------+--------------+!| memory/sql/Filesort_buffer::sort_keys | 255.94 KiB |!| memory/sql/sp_head::main_mem_root | 103.64 KiB |!| memory/mysys/IO_CACHE | 64.05 KiB |!| memory/mysys/lf_dynarray | 46.17 KiB |!| memory/mysys/array_buffer | 24.20 KiB |!| memory/sql/thd::main_mem_root | 23.95 KiB |!| memory/sql/String::value | 16.13 KiB |!| memory/sql/TABLE | 9.44 KiB |!| memory/sql/TABLE_SHARE::mem_root | 8.70 KiB |!| memory/myisam/MI_INFO | 7.07 KiB |!| memory/sql/THD::transactions::mem_root | 4.02 KiB |!| memory/myisam/MYISAM_SHARE | 3.29 KiB |!…

Page 85: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!85

Instrument memory usage - other details

!▪ Disabled by default !!!!

▪ Still requires InnoDB instrumentation

UPDATE setup_instruments SET enabled = ‘YES’ WHERE name LIKE ‘memory/%’;

Page 86: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!86

WL#5879!PERFORMANCE SCHEMA, MDL lock instrumentation

Page 87: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 12!87

“Unfortunately, it’s unlikely that I’ll be able to create a reproducible test case, because there’s no way to actually see what is happening. I hope that a future version of MySQL will include a more comprehensive set of tables for inspecting locks, requests, and waits at all layers of the server.”

▪ Baron Schwartz, CEO, VividCortex

http://www.xaprb.com/blog/2012/08/28/debugging-metadata-locking-in-mysql-5-5/

Page 88: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!88

MDL Lock Instrumentation

▪ Added the wait/lock/metadata/sql/mdl instrument ▪ 2 new tables

▪ metadata_locks ▪ table_handles

Page 89: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!89

Metadata Locks Table Structure

+-----------------------+---------------------+------+-----+---------+-------+!| Field | Type | Null | Key | Default | Extra |!+-----------------------+---------------------+------+-----+---------+-------+!| OBJECT_TYPE | varchar(64) | NO | | NULL | |!| OBJECT_SCHEMA | varchar(64) | YES | | NULL | |!| OBJECT_NAME | varchar(64) | YES | | NULL | |!| OBJECT_INSTANCE_BEGIN | bigint(20) unsigned | NO | | NULL | |!| LOCK_TYPE | varchar(32) | NO | | NULL | |!| LOCK_DURATION | varchar(32) | NO | | NULL | |!| LOCK_STATUS | varchar(32) | NO | | NULL | |!| SOURCE | varchar(64) | YES | | NULL | |!| OWNER_THREAD_ID | bigint(20) unsigned | YES | | NULL | |!| OWNER_EVENT_ID | bigint(20) unsigned | YES | | NULL | |!+-----------------------+---------------------+------+-----+---------+-------+

Page 90: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!90

MDL Lock Instrumentation▪ OBJECT_TYPE

▪ GLOBAL, SCHEMA, TABLE, FUNCTION, PROCEDURE, TRIGGER, EVENT, COMMIT

▪ LOCK_TYPE ▪ INTENTION_EXCLUSIVE, SHARED, SHARED_HIGH_PRIO,

SHARED_READ, SHARED_WRITE, SHARED_UPGRADABLE, SHARED_NO_WRITE, SHARED_NO_READ_WRITE, EXCLUSIVE

▪ LOCK_DURATION ▪ STATEMENT, TRANSACTION, EXPLICIT

▪ LOCK_STATUS ▪ PENDING, GRANTED, VICTIM, TIMEOUT, KILLED

Page 91: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!91

Metadata Locks Table Structuremysql> select object_type as scope, object_schema, object_name, lock_type, lock_duration, lock_status! -> from metadata_locks! -> order by object_type = 'global' desc, object_type = 'schema' desc,! -> object_type = 'table' desc, object_type = 'commit' desc;!+--------+--------------------+----------------+---------------------+---------------+-------------+!| scope | object_schema | object_name | lock_type | lock_duration | lock_status |!+--------+--------------------+----------------+---------------------+---------------+-------------+!| GLOBAL | NULL | NULL | SHARED | EXPLICIT | GRANTED |!| GLOBAL | NULL | NULL | INTENTION_EXCLUSIVE | STATEMENT | PENDING |!| GLOBAL | NULL | NULL | INTENTION_EXCLUSIVE | STATEMENT | PENDING |!| GLOBAL | NULL | NULL | INTENTION_EXCLUSIVE | STATEMENT | PENDING |!| GLOBAL | NULL | NULL | INTENTION_EXCLUSIVE | STATEMENT | PENDING |!| GLOBAL | NULL | NULL | INTENTION_EXCLUSIVE | STATEMENT | PENDING |!| GLOBAL | NULL | NULL | INTENTION_EXCLUSIVE | STATEMENT | PENDING |!| GLOBAL | NULL | NULL | INTENTION_EXCLUSIVE | STATEMENT | PENDING |!|!| TABLE | mem__events | action_logs | SHARED_READ | TRANSACTION | GRANTED |!| TABLE | mem__events | events | SHARED_READ | TRANSACTION | GRANTED |!| TABLE | performance_schema | metadata_locks | SHARED_READ | TRANSACTION | GRANTED |!| COMMIT | NULL | NULL | SHARED | EXPLICIT | GRANTED |!+--------+--------------------+----------------+---------------------+---------------+-------------+

Page 92: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!92

WL#3656!PERFORMANCE SCHEMA table for!SHOW SLAVE STATUS

Page 93: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!93

Replication Instrumentation

▪ 6 new tables ▪ replication_connection_configuration ▪ replication_connection_status ▪ replication_execute_configuration ▪ replication_execute_status ▪ replication_execute_status_by_coordinator ▪ replication_execute_status_by_worker

Page 94: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!94

Replication Instrumentation

Page 95: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!95

Replication Connection Configurationmysql> select * from performance_schema.replication_connection_configuration\G *************************** 1. row ***************************! HOST: 127.0.0.1! PORT: 3306! USER: rpl! NETWORK_INTERFACE:! AUTO_POSITION: 1! SSL_ALLOWED: YES! SSL_CA_FILE:! SSL_CA_PATH:! SSL_CERTIFICATE:! SSL_CIPHER:! SSL_KEY:!SSL_VERIFY_SERVER_CERTIFICATE: YES! SSL_CRL_FILE:! SSL_CRL_PATH:! CONNECTION_RETRY_INTERVAL: 40! CONNECTION_RETRY_COUNT: 10

Page 96: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!96

Replication Connection Status

mysql> select * from replication_connection_status\G!*************************** 1. row ***************************! SOURCE_UUID:! THREAD_ID: NULL! SERVICE_STATE: CONNECTING!RECEIVED_TRANSACTION_SET:! LAST_ERROR_NUMBER: 1045! LAST_ERROR_MESSAGE: error connecting to master 'repl@localhost:5613' - retry-time: 60 retries: 1! LAST_ERROR_TIMESTAMP: 2014-04-02 05:41:20

Page 97: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!97

Replication Execute Status

mysql> select * from replication_execute_status;!+---------------+-----------------+!| SERVICE_STATE | REMAINING_DELAY |!+---------------+-----------------+!| ON | NULL |!+---------------+-----------------+

▪ Very high level overview of status

Page 98: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!98

Replication Coordinator Status

mysql> select * from replication_execute_status_by_coordinator\G!*************************** 1. row ***************************! THREAD_ID: 1281! SERVICE_STATE: ON! LAST_ERROR_NUMBER: 0! LAST_ERROR_MESSAGE:!LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00

▪ Status of Coordinator thread in multi-threaded replication

Page 99: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!99

Replication Worker Status

mysql> select * from replication_execute_status_by_worker;!+-----------+-----------+---------------+-----------------------------------------+-------------------+--------------------+----------------------+!| WORKER_ID | THREAD_ID | SERVICE_STATE | LAST_SEEN_TRANSACTION | LAST_ERROR_NUMBER | LAST_ERROR_MESSAGE | LAST_ERROR_TIMESTAMP |!+-----------+-----------+---------------+-----------------------------------------+-------------------+--------------------+----------------------+!| 1 | 1282 | ON | | 0 | | 0000-00-00 00:00:00 |!| 2 | 1283 | ON | | 0 | | 0000-00-00 00:00:00 |!| 3 | 1284 | ON | | 0 | | 0000-00-00 00:00:00 |!| 4 | 1285 | ON | | 0 | | 0000-00-00 00:00:00 |!| 5 | 1286 | ON | | 0 | | 0000-00-00 00:00:00 |!| 6 | 1287 | ON | | 0 | | 0000-00-00 00:00:00 |!| 7 | 1288 | ON | | 0 | | 0000-00-00 00:00:00 |!| 8 | 1289 | ON | 181f2b36-a0b4-11e3-9ac7-1025863574a7:16 | 0 | | 0000-00-00 00:00:00 |!+-----------+-----------+---------------+-----------------------------------------+-------------------+--------------------+----------------------+

▪ Status of worker threads within multi-threaded replication

Page 100: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!100

WL#5768!PERFORMANCE SCHEMA, prepared statements instrumentation

Page 101: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!101

Prepared Statement Instrumentation

▪ Instrumentation depends on already existing statement instruments ▪ statement/com/prepare, statement/com/execute ▪ statement/sql/prepare_sql, statement/sql/execute_sql

▪ 1 new table ▪ prepared_statements_instances

Page 102: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!102

Prepared Statement Instancesmysql> select * from performance_schema.prepared_statements_instances\G!*************************** 1. row ***************************! OBJECT_INSTANCE_BEGIN: 140198306602144! STATEMENT_ID: 1! STATEMENT_NAME: stmt1! SQL_TEXT: select * from test.t1! OWNER_THREAD_ID: 54003! OWNER_EVENT_ID: 935! OWNER_OBJECT_TYPE: NULL! OWNER_OBJECT_SCHEMA: NULL! OWNER_OBJECT_NAME: NULL! TIMER_PREPARE: 15386274000! COUNT_REPREPARE: 0! COUNT_EXECUTE: 1! SUM_TIMER_EXECUTE: 217699000! MIN_TIMER_EXECUTE: 217699000! AVG_TIMER_EXECUTE: 217699000! MAX_TIMER_EXECUTE: 217699000

SUM_LOCK_TIME: 0! SUM_ERRORS: 0! SUM_WARNINGS: 0! SUM_ROWS_AFFECTED: 0! SUM_ROWS_SENT: 0! SUM_ROWS_EXAMINED: 0!SUM_CREATED_TMP_DISK_TABLES: 0! SUM_CREATED_TMP_TABLES: 0! SUM_SELECT_FULL_JOIN: 0! SUM_SELECT_FULL_RANGE_JOIN: 0! SUM_SELECT_RANGE: 0! SUM_SELECT_RANGE_CHECK: 0! SUM_SELECT_SCAN: 0! SUM_SORT_MERGE_PASSES: 0! SUM_SORT_RANGE: 0! SUM_SORT_ROWS: 0! SUM_SORT_SCAN: 0! SUM_NO_INDEX_USED: 0! SUM_NO_GOOD_INDEX_USED: 0

Page 103: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!103

WL#5864!PERFORMANCE SCHEMA, instrument TRANSACTIONS

Page 104: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!104

Transaction Instrumentation▪ Added the transaction instrument ▪ 8 new tables

▪ events_transactions_current ▪ events_transactions_history ▪ events_transactions_history_long ▪ events_transactions_summary_by_account_by_event_name ▪ events_transactions_summary_by_host_by_event_name ▪ events_transactions_summary_by_thread_by_event_name ▪ events_transactions_summary_by_user_by_event_name ▪ events_transactions_summary_global_by_event_name

Page 105: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!105

Transaction Instrumentation

▪ Exposes details of transactions in raw or summary views ▪ Show details such as

▪ Transaction latency ▪ Isolation levels ▪ Auto commit ▪ Savepoint info ▪ GTID or transaction IDs

▪ Link transactions to the statements within the transactions

Page 106: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!106

Current Transaction Detailsmysql> select * from events_transactions_current\G!*************************** 1. row ***************************! THREAD_ID: 1! EVENT_ID: 23733! END_EVENT_ID: 23742! EVENT_NAME: transaction! STATE: COMMITTED! TRX_ID: 281479898269256! GTID: NULL! XID: NULL! XA_STATE: NULL! SOURCE: handler.cc:1246! TIMER_START: 31140612726000! TIMER_END: 31140647445000! TIMER_WAIT: 34719000

ACCESS_MODE: READ WRITE! ISOLATION_LEVEL: REPEATABLE READ! AUTOCOMMIT: YES! NUMBER_OF_SAVEPOINTS: 0!NUMBER_OF_ROLLBACK_TO_SAVEPOINT: 0! NUMBER_OF_RELEASE_SAVEPOINT: 0! OBJECT_INSTANCE_BEGIN: NULL! NESTING_EVENT_ID: NULL! NESTING_EVENT_TYPE: NULL

Page 107: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!107

Transaction Summary - per usermysql> select * from events_transactions_summary_by_user_by_event_name\G!*************************** 1. row ***************************! USER: mem! EVENT_NAME: transaction! COUNT_STAR: 400044! SUM_TIMER_WAIT: 21208048458267000! MIN_TIMER_WAIT: 160744000! AVG_TIMER_WAIT: 53014289000! MAX_TIMER_WAIT: 24028520397000! COUNT_READ_WRITE: 400044!SUM_TIMER_READ_WRITE: 21208048458267000!MIN_TIMER_READ_WRITE: 160744000!AVG_TIMER_READ_WRITE: 53014289000!MAX_TIMER_READ_WRITE: 24028520397000! COUNT_READ_ONLY: 0! SUM_TIMER_READ_ONLY: 0! MIN_TIMER_READ_ONLY: 0! AVG_TIMER_READ_ONLY: 0! MAX_TIMER_READ_ONLY: 0

Page 108: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!108

WL#5766!PERFORMANCE SCHEMA, stored programs instrumentation

Page 109: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!109

Stored Program Instrumentation▪ Tracks Stored Procedures, Stored Functions, Triggers and Events ▪ Added 16 new statement/sp/% instruments

▪ Expose the different work flows that stored programs use, such as cursor operations, workflow controls etc.

▪ Integrated in to the normal statement instrumentation ▪ 1 new tables

▪ events_statements_summary_by_program

Page 110: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!110

Stored Program Summarymysql> select * from events_statements_summary_by_program\G!*************************** 1. row ***************************! OBJECT_TYPE: PROCEDURE! OBJECT_SCHEMA: ps_demo! OBJECT_NAME: ps_demo_proc! COUNT_STAR: 1! SUM_TIMER_WAIT: 6970931000! MIN_TIMER_WAIT: 6970931000! AVG_TIMER_WAIT: 6970931000! MAX_TIMER_WAIT: 6970931000! COUNT_STATEMENTS: 5! SUM_STATEMENTS_WAIT: 6802181000! MIN_STATEMENTS_WAIT: 16372000! AVG_STATEMENTS_WAIT: 1360436000! MAX_STATEMENTS_WAIT: 6484366000! SUM_LOCK_TIME: 176401000000! SUM_ERRORS: 0! SUM_WARNINGS: 0! SUM_ROWS_AFFECTED: 1! SUM_ROWS_SENT: 0

SUM_ROWS_EXAMINED: 1!SUM_CREATED_TMP_DISK_TABLES: 0! SUM_CREATED_TMP_TABLES: 0! SUM_SELECT_FULL_JOIN: 0! SUM_SELECT_FULL_RANGE_JOIN: 0! SUM_SELECT_RANGE: 0! SUM_SELECT_RANGE_CHECK: 0! SUM_SELECT_SCAN: 0! SUM_SORT_MERGE_PASSES: 0! SUM_SORT_RANGE: 0! SUM_SORT_ROWS: 0! SUM_SORT_SCAN: 0! SUM_NO_INDEX_USED: 0! SUM_NO_GOOD_INDEX_USED: 0

Page 111: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!111

Stored Program Instrumentation

▪ Like Transaction instrumentation, stored programs link to the statements within them, or even link to transactions within them, which in turn link to their statements

Page 112: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!112

Full Hierarchy of New Instrumentation

Page 113: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 12!113

Page 114: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 12!114

Page 115: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 12!115

Page 116: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 12!116

Page 117: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 12!117

Page 118: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 12!118

Page 119: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 12!119

The MySQL SYS Schema

Page 120: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!120

MySQL SYS Schema Overview

▪ Originally called “ps_helper” ▪ Started as a collection of views, procedures and functions, designed

to make reading raw Performance Schema data easier ▪ Implements many of the common DBA and Developer use cases,

including many of those shown already ▪ Now bundled within MySQL Workbench 6.1 ▪ Available on GitHub

▪ https://github.com/MarkLeith/mysql-sys

Page 121: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!121

Loading MySQL SYS

$ git clone https://github.com/MarkLeith/mysql-sys.git /tmp/mysql-sys $ cd /tmp/mysql-sys $ mysql -u user -p < sys_<version>.sql !<version> can be 56 or 57, for 5.6 and 5.7 respectively

Page 122: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!122

Loading MySQL SYS

▪ Once loaded creates a new “sys” schema containing all objects

mysql> select object_type, count! -> from sys.schema_object_overview! -> where db = 'sys';!+-------------+-------+!| object_type | count |!+-------------+-------+!| PROCEDURE | 16 |!| FUNCTION | 8 |!| VIEW | 71 |!+-------------+-------+

Page 123: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!123

MySQL SYS Functions

▪ Make the raw data more readable to a human ▪ format_time() / format_bytes()

▪ Compress data for CLI output ▪ format_statement() / format_path()

▪ Extract some data to assist with JOIN in certain cases ▪ extract_[schema|table]_from_file_name()

▪ Other functions ▪ ps_is_account_enabled() / ps_thread_stack()

Page 124: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!124

MySQL SYS Views

▪ Reference set of views solving various admin use cases ▪ Build upon both Performance Schema and INFORMATION_SCHEMA ▪ Both formatted and raw views are available

▪ All raw views are prefixed with x$ ▪ Allows tooling to poll raw views, but humans to use normal ones

Page 125: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!125

User Summary Views

▪ High level user overview ▪ Breakdowns of IO usage ▪ Drill in to stages per user ▪ Drill in to statement details per

user

mysql> show tables like 'user%';!+-----------------------------------+!| Tables_in_sys (user%) |!+-----------------------------------+!| user_summary |!| user_summary_by_file_io |!| user_summary_by_file_io_type |!| user_summary_by_stages |!| user_summary_by_statement_latency |!| user_summary_by_statement_type |!+-----------------------------------+

Page 126: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!126

User Summary (5.7)mysql> select * from user_summary\G!*************************** 1. row ***************************! user: mark! statements: 3072! statement_latency: 1.77 s! statement_avg_latency: 575.29 us! table_scans: 7! file_ios: 20043! file_io_latency: 346.79 ms! current_connections: 1! total_connections: 1! unique_hosts: 1! current_memory: 515.81 KiB!total_memory_allocated: 30.69 MiB

Page 127: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!127

User File IO Summarymysql> select * from user_summary_by_file_io_type where user != 'background';!+------+--------------------------------------+---------+-------------+-------------+!| user | event_name | total | latency | max_latency |!+------+--------------------------------------+---------+-------------+-------------+!| mark | wait/io/file/myisam/dfile | 19540 | 215.20 ms | 120.79 ms |!| mark | wait/io/file/myisam/kfile | 501 | 131.58 ms | 57.74 ms |!| mark | wait/io/file/sql/dbopt | 2 | 17.29 us | 9.17 us |!| root | wait/io/file/innodb/innodb_log_file | 2079432 | 00:28:27.62 | 1.30 s |!| root | wait/io/file/myisam/kfile | 5195927 | 00:17:07.33 | 1.17 s |!| root | wait/io/file/myisam/dfile | 1533813 | 00:07:27.59 | 1.18 s |!| root | wait/io/file/sql/binlog | 3517008 | 00:03:07.34 | 1.42 s |!| root | wait/io/file/innodb/innodb_data_file | 20720 | 00:02:37.81 | 492.23 ms |!| root | wait/io/file/sql/FRM | 41412 | 3.17 s | 80.43 ms |!| root | wait/io/file/sql/dbopt | 110993 | 1.23 s | 63.27 ms |!| root | wait/io/file/sql/binlog_index | 80 | 1.16 s | 439.79 ms |!| root | wait/io/file/csv/metadata | 16 | 175.97 ms | 79.53 ms |!| root | wait/io/file/archive/data | 3345 | 18.35 ms | 5.05 ms |!| root | wait/io/file/sql/file_parser | 142 | 5.80 ms | 331.82 us |!| root | wait/io/file/sql/misc | 99 | 760.67 us | 25.19 us |!| root | wait/io/file/csv/data | 6 | 596.99 us | 348.11 us |!+------+--------------------------------------+---------+-------------+-------------+

Page 128: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!128

User Statement Latency Summary

mysql> select * from user_summary_by_statement_latency where user != 'background';!+------+---------+---------------+-------------+--------------+-----------+---------------+---------------+------------+!| user | total | total_latency | max_latency | lock_latency | rows_sent | rows_examined | rows_affected | full_scans |!+------+---------+---------------+-------------+--------------+-----------+---------------+---------------+------------+!| root | 8322401 | 19.37h | 00:07:08.11 | 00:40:26.25 | 10110692 | 68391445 | 4198113 | 189531 |!| mark | 3072 | 1.77 s | 1.73 s | 1.96 ms | 534 | 15389 | 0 | 7 |!+------+---------+---------------+-------------+--------------+-----------+---------------+---------------+------------+!

Page 129: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!129

User Statement Type Summary

mysql> select * from user_summary_by_statement_type where user != 'background';!+------+-----------------------+---------+---------------+-------------+--------------+-----------+---------------+---------------+------------+!| user | statement | total | total_latency | max_latency | lock_latency | rows_sent | rows_examined | rows_affected | full_scans |!+------+-----------------------+---------+---------------+-------------+--------------+-----------+---------------+---------------+------------+!| mark | select | 4 | 1.69 s | 1.69 s | 408.00 us | 212 | 15067 | 0 | 1 |!| mark | Field List | 146 | 67.91 ms | 40.02 ms | 1.11 ms | 0 | 0 | 0 | 0 |!| mark | jump_if_not | 2496 | 2.68 ms | 7.07 us | 0 ps | 0 | 0 | 0 | 0 |!| mark | show_tables | 4 | 1.70 ms | 545.41 us | 296.00 us | 292 | 292 | 0 | 4 |!| mark | freturn | 418 | 1.15 ms | 9.29 us | 0 ps | 0 | 0 | 0 | 0 |!| mark | show_databases | 2 | 849.40 us | 431.05 us | 146.00 us | 30 | 30 | 0 | 2 |!| mark | Init DB | 2 | 114.95 us | 59.58 us | 0 ps | 0 | 0 | 0 | 0 |!| root | commit | 2463190 | 13.33h | 12.18 s | 0 ps | 0 | 0 | 0 | 0 |!| root | insert | 2501839 | 3.78h | 6.64 s | 00:31:18.63 | 192 | 0 | 3674716 | 0 |!| root | select | 1119999 | 1.74h | 16.34 s | 00:07:42.99 | 6187336 | 63933992 | 0 | 163913 |!| root | update | 494217 | 00:15:23.09 | 4.83 s | 00:01:05.05 | 126 | 496604 | 495334 | 0 |!| root | Binlog Dump GTID | 1 | 00:06:12.69 | 00:06:12.69 | 0 ps | 0 | 0 | 0 | 0 |!| root | delete | 206176 | 00:01:59.54 | 2.22 s | 6.69 s | 12 | 20201 | 20203 | 0 |!| root | set_option | 1068360 | 00:01:10.73 | 45.37 ms | 0 ps | 0 | 0 | 0 | 0 |!| root | show_engine_status | 2222 | 46.62 s | 4.96 s | 0 ps | 0 | 0 | 0 | 0 |!| root | rollback | 64527 | 26.21 s | 407.15 ms | 0 ps | 0 | 0 | 0 | 0 |!| root | show_variables | 11219 | 12.28 s | 255.00 ms | 1.19 s | 2542172 | 2542172 | 0 | 11219 |!| root | show_binlogs | 1850 | 8.96 s | 1.64 s | 0 ps | 0 | 0 | 0 | 0 |!| root | show_tables | 4250 | 6.86 s | 123.34 ms | 182.80 ms | 5277 | 5277 | 0 | 4250 |!| root | show_status | 5485 | 6.50 s | 242.60 ms | 581.02 ms | 1331264 | 1331264 | 0 | 5485 |!

Page 130: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!130

IO Summary Views

▪ IO breakdown by thread ▪ Global summaries by file and wait

class, by both bytes and latency ▪ Stream of last raw file IO stats

mysql> show tables like 'io_%';!+------------------------------+!| Tables_in_sys (io_%) |!+------------------------------+!| io_by_thread_by_latency |!| io_global_by_file_by_bytes |!| io_global_by_file_by_latency |!| io_global_by_wait_by_bytes |!| io_global_by_wait_by_latency |!| latest_file_io |!+------------------------------+

Page 131: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!131

IO Per Thread Summarymysql> select * from io_by_thread_by_latency;!+---------------------+---------+---------------+-------------+-------------+-------------+-----------+----------------+!| user | total | total_latency | min_latency | avg_latency | max_latency | thread_id | processlist_id |!+---------------------+---------+---------------+-------------+-------------+-------------+-----------+----------------+!| io_write_thread | 1270599 | 00:28:51.42 | 1.46 us | 1.36 ms | 1.39 s | 9 | NULL |!| page_cleaner_thread | 3356577 | 00:21:01.53 | 410.93 ns | 2.08 ms | 1.49 s | 19 | NULL |!| io_write_thread | 338216 | 00:14:31.16 | 1.67 us | 2.58 ms | 1.17 s | 12 | NULL |!| io_log_thread | 47023 | 00:04:06.10 | 3.65 us | 5.23 ms | 538.77 ms | 4 | NULL |!| io_write_thread | 290066 | 00:02:20.99 | 2.78 us | 486.07 us | 703.20 ms | 10 | NULL |!| io_write_thread | 221417 | 00:02:12.64 | 1.85 us | 599.05 us | 773.64 ms | 11 | NULL |!| root@localhost | 587531 | 00:02:03.29 | 444.86 ns | 200.24 us | 372.09 ms | 2515 | 2495 |!| io_read_thread | 5102 | 00:01:09.66 | 12.24 us | 13.65 ms | 281.13 ms | 5 | NULL |!| srv_master_thread | 55199 | 58.11 s | 693.68 ns | 1.40 ms | 504.03 ms | 17 | NULL |!| root@localhost | 1622373 | 41.20 s | 486.33 ns | 25.40 us | 359.49 ms | 23 | 3 |!| io_read_thread | 2401 | 37.27 s | 11.86 us | 15.52 ms | 258.97 ms | 6 | NULL |!| io_read_thread | 2156 | 35.73 s | 12.02 us | 16.57 ms | 284.81 ms | 8 | NULL |!| io_read_thread | 1719 | 29.19 s | 11.37 us | 16.98 ms | 269.93 ms | 7 | NULL |!| srv_purge_thread | 3874 | 12.16 s | 433.55 ns | 3.14 ms | 246.79 ms | 18 | NULL |!| main | 11083 | 3.12 s | 588.12 ns | 547.21 us | 234.02 ms | 1 | NULL |!| root@localhost | 5985 | 1.48 s | 625.82 ns | 1.06 ms | 239.17 ms | 137759 | 137739 |!| root@localhost | 157 | 434.38 ms | 497.64 ns | 8.09 ms | 205.84 ms | 178521 | 178501 |!| mark@localhost | 20043 | 346.79 ms | 418.47 ns | 94.10 us | 120.79 ms | 150606 | 150586 |!| root@localhost | 138 | 204.72 ms | 486.33 ns | 2.68 ms | 86.54 ms | 178524 | 178504 |!| root@localhost | 415 | 126.43 ms | 471.25 ns | 195.15 us | 37.21 ms | 178269 | 178249 |!| root@localhost | 9 | 106.21 ms | 6.40 us | 8.85 ms | 103.88 ms | 178522 | 178502 |!+---------------------+---------+---------------+-------------+-------------+-------------+-----------+----------------+!21 rows in set (0.02 sec)

Page 132: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!132

IO Per File Summariesmysql> select * from io_global_by_file_by_latency limit 1\G!*************************** 1. row ***************************! file: @@datadir/ibdata1! total: 1395714!total_latency: 00:28:53.06! count_read: 1006! read_latency: 4.51 s! count_write: 1326529!write_latency: 33.39 s! count_misc: 68179! misc_latency: 00:28:15.16!!mysql> select * from io_global_by_file_by_bytes limit 1\G!*************************** 1. row ***************************! file: @@datadir/ibdata1! count_read: 1006! total_read: 17.69 MiB! avg_read: 18.00 KiB! count_write: 1327221!total_written: 77.82 GiB! avg_write: 61.48 KiB! total: 77.84 GiB! write_pct: 99.98

Page 133: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!133

Latest File IOmysql> select * from latest_file_io;!+-----------------------------+----------------------------------+-----------+-----------+-----------+!| thread | file | latency | operation | requested |!+-----------------------------+----------------------------------+-----------+-----------+-----------+!| root@localhost:63153:179371 | @@datadir/ib_logfile1 | 7.16 us | lock | NULL |!| root@localhost:63153:179371 | @@datadir/ib_logfile1 | 6.16 us | write | 1.50 KiB |!| root@localhost:63153:179371 | @@datadir/ib_logfile1 | 14.36 ms | sync | NULL |!| root@localhost:63153:179371 | @@datadir/cerberus-bin.000012 | 28.25 us | write | 905 bytes |!| root@localhost:63177:179394 | @@datadir/ib_logfile1 | 15.25 us | write | 1.00 KiB |!| root@localhost:57487:3 | @@datadir/cerberus-bin.000012 | 32.12 us | read | 905 bytes |!| root@localhost:63177:179394 | @@datadir/ib_logfile1 | 6.65 us | write | 1.00 KiB |!| root@localhost:63177:179394 | @@datadir/ib_logfile1 | 213.32 us | sync | NULL |!| root@localhost:63177:179394 | @@datadir/ib_logfile1 | 17.27 us | write | 1.00 KiB |!| root@localhost:63177:179394 | @@datadir/ib_logfile1 | 286.01 us | sync | NULL |!| root@localhost:63177:179394 | @@datadir/cerberus-bin.000012 | 18.77 us | write | 601 bytes |!| root@localhost:57487:3 | @@datadir/cerberus-bin.000012 | 7.55 us | read | 601 bytes |!| root@localhost:63177:179394 | @@datadir/ib_logfile1 | 10.67 us | write | 1.00 KiB |!| root@localhost:63177:179394 | @@datadir/ib_logfile1 | 200.28 us | sync | NULL |!| root@localhost:63177:179394 | @@datadir/cerberus-bin.000012 | 12.91 us | write | 601 bytes |!| root@localhost:57487:3 | @@datadir/cerberus-bin.000012 | 6.47 us | read | 601 bytes |!| root@localhost:63177:179394 | @@datadir/ib_logfile1 | 9.22 us | write | 1.50 KiB |!……

Page 134: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!134

Schema Analysis Views

▪ Object overview ▪ Table usage stats ▪ Index usage stats

mysql> show tables like 'schema%';!+-------------------------------------+!| Tables_in_sys (schema%) |!+-------------------------------------+!| schema_index_statistics |!| schema_object_overview |!| schema_table_statistics |!| schema_table_statistics_with_buffer |!| schema_tables_with_full_table_scans |!| schema_unused_indexes |!+-------------------------------------+

Page 135: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!135

Schema Object Overviewmysql> select * from schema_object_overview;!+-------------------------+---------------+-------+!| db | object_type | count |!+-------------------------+---------------+-------+!| information_schema | SYSTEM VIEW | 60 |!| mem__advisors | BASE TABLE | 3 |!| mem__advisors | INDEX (BTREE) | 5 |!| mem__advisor_text | BASE TABLE | 2 |!| mem__advisor_text | INDEX (BTREE) | 5 |!| mem__bean_config | BASE TABLE | 4 |!| mem__bean_config | INDEX (BTREE) | 6 |!| mem__config | BASE TABLE | 12 |!| mem__config | INDEX (BTREE) | 21 |!| mem__enterprise | BASE TABLE | 2 |!| mem__enterprise | INDEX (BTREE) | 3 |!| mem__events | BASE TABLE | 32 |!| mem__events | INDEX (BTREE) | 69 |!| mem__instruments | BASE TABLE | 117 |!| mem__instruments | INDEX (BTREE) | 582 |

Page 136: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!136

Table statisticsmysql> select * from schema_table_statistics limit 1\G!*************************** 1. row ***************************! table_schema: mem__quan! table_name: normalized_statements_by_server_by_schema_data! total_latency: 00:55:07.80! rows_fetched: 239685! fetch_latency: 17.10 s! rows_inserted: 1102895! insert_latency: 00:52:18.10! rows_updated: 55910! update_latency: 00:02:28.62! rows_deleted: 17899! delete_latency: 3.98 s! io_read_requests: 20639! io_read: 324.94 MiB! io_read_latency: 00:03:04.10!io_write_requests: 1965973! io_write: 40.36 GiB! io_write_latency: 25.46 s! io_misc_requests: 18217! io_misc_latency: 00:14:06.02

Page 137: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!137

Index statistics

mysql> select * from schema_index_statistics limit 1\G!*************************** 1. row ***************************! table_schema: mem__inventory! table_name: mysqlconnectionconfiguration! index_name: PRIMARY! rows_selected: 131399!select_latency: 00:07:05.03! rows_inserted: 0!insert_latency: 0 ps! rows_updated: 41608!update_latency: 21.96 s! rows_deleted: 0!delete_latency: 0 ps

Page 138: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!138

InnoDB Buffer Usage Views

▪ Buffer usage by schema ▪ Buffer usage by table

mysql> show tables like 'innodb%';!+-------------------------------+!| Tables_in_sys (innodb%) |!+-------------------------------+!| innodb_buffer_stats_by_schema |!| innodb_buffer_stats_by_table |!+-------------------------------+

Page 139: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!139

Buffer Usage By Schema

mysql> select * from innodb_buffer_stats_by_schema;!+------------------+------------+------------+-------+--------------+-----------+-------------+!| object_schema | allocated | data | pages | pages_hashed | pages_old | rows_cached |!+------------------+------------+------------+-------+--------------+-----------+-------------+!| mem__quan | 69.94 MiB | 45.39 MiB | 4476 | 4476 | 4476 | 123226 |!| mem__instruments | 20.70 MiB | 10.75 MiB | 1325 | 1325 | 1325 | 82812 |!| mem__events | 3.08 MiB | 2.09 MiB | 197 | 197 | 197 | 3411 |!| mem__inventory | 1.34 MiB | 52.49 KiB | 86 | 86 | 86 | 18 |!| mysql | 432.00 KiB | 251.34 KiB | 27 | 27 | 27 | 2252 |!| mem__config | 144.00 KiB | 3.86 KiB | 9 | 9 | 9 | 29 |!+------------------+------------+------------+-------+--------------+-----------+-------------+

Page 140: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!140

Buffer Usage By Tablemysql> select * from innodb_buffer_stats_by_table limit 20;!+------------------+------------------------------------------------+------------+------------+-------+--------------+-----------+-------------+!| object_schema | object_name | allocated | data | pages | pages_hashed | pages_old | rows_cached |!+------------------+------------------------------------------------+------------+------------+-------+--------------+-----------+-------------+!| mem__quan | normalized_statements_by_server_by_schema_data | 49.97 MiB | 33.12 MiB | 3198 | 3198 | 3198 | 238845 |!| mem__quan | example_statements | 15.95 MiB | 9.71 MiB | 1021 | 1021 | 1021 | 32435 |!| mem__events | events | 2.75 MiB | 1.97 MiB | 176 | 176 | 176 | 5044 |!| mem__quan | normalized_statements | 2.19 MiB | 1.42 MiB | 140 | 140 | 140 | 2297 |!| mem__quan | normalized_statements_by_server_by_schema | 1.06 MiB | 693.58 KiB | 68 | 68 | 68 | 3840 |!| mem__instruments | innodbundologsdata | 1.05 MiB | 754.07 KiB | 67 | 67 | 67 | 5529 |!| mem__instruments | fsstatistics | 736.00 KiB | 603.64 KiB | 46 | 46 | 46 | 2092 |!| mem__instruments | networktrafficadvisor_networktraffic | 496.00 KiB | 291.38 KiB | 31 | 31 | 31 | 1160 |!| mysql | innodb_index_stats | 384.00 KiB | 230.37 KiB | 24 | 24 | 24 | 1986 |!| mem__quan | explains | 368.00 KiB | 274.42 KiB | 23 | 23 | 23 | 333 |!| mem__instruments | innodbchangebufferusagedata | 272.00 KiB | 126.94 KiB | 17 | 17 | 17 | 1104 |!| mem__instruments | rowscanratiodata | 272.00 KiB | 141.18 KiB | 17 | 17 | 17 | 1207 |!| mem__instruments | connectionsmaxdata | 256.00 KiB | 150.98 KiB | 16 | 16 | 16 | 1262 |!| mem__instruments | innodbopenfilesdata | 256.00 KiB | 145.30 KiB | 16 | 16 | 16 | 1170 |!| mem__instruments | myisamkeybufferusagedata | 256.00 KiB | 155.60 KiB | 16 | 16 | 16 | 1192 |!| mem__instruments | connectionsdata | 240.00 KiB | 129.12 KiB | 15 | 15 | 15 | 1106 |!| mem__instruments | diskioopstotaldata | 240.00 KiB | 143.94 KiB | 15 | 15 | 15 | 533 |!| mem__instruments | diskiototaldata | 240.00 KiB | 143.94 KiB | 15 | 15 | 15 | 533 |!| mem__instruments | innodbchangebufferactivitydata | 240.00 KiB | 124.43 KiB | 15 | 15 | 15 | 1003 |!| mem__instruments | innodbcheckpointagedata | 240.00 KiB | 122.65 KiB | 15 | 15 | 15 | 1052 |!+------------------+------------------------------------------------+------------+------------+-------+--------------+-----------+-------------+

Page 141: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!141

Statement Analysis Views

▪ Statement overview ▪ Find statements by errors, full

scans, sorting, temporary tables ▪ Find the statements with the

longest runtimes !

mysql> show tables like 'statement%';!+---------------------------------------------+!| Tables_in_sys (statement%) |!+---------------------------------------------+!| statement_analysis |!| statements_with_errors_or_warnings |!| statements_with_full_table_scans |!| statements_with_runtimes_in_95th_percentile |!| statements_with_sorting |!| statements_with_temp_tables |!+---------------------------------------------+

Page 142: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!142

Statement Analysismysql> select * from statement_analysis limit 11\G!*************************** 1. row ***************************! query: INSERT INTO `mem__quan` . `nor ... nDuration` = IF ( VALUES ( ...! db: mem! full_scan:! exec_count: 1110067! err_count: 0! warn_count: 0! total_latency: 1.93h! max_latency: 5.03 s! avg_latency: 6.27 ms! lock_latency: 00:18:29.18! rows_sent: 0! rows_sent_avg: 0! rows_examined: 0!rows_examined_avg: 0! tmp_tables: 0! tmp_disk_tables: 0! rows_sorted: 0!sort_merge_passes: 0! digest: d48316a218e95b1b8b72db5e6b177788! first_seen: 2014-05-20 10:42:17

Page 143: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!143

Statement With Errors And Warnings

mysql> select * from statements_with_errors_or_warnings limit 1\G!*************************** 1. row ***************************! query: CREATE TEMPORARY TABLE IF NOT ... _logs` ( `id` INT8 NOT NULL )! db: mem! exec_count: 1725! errors: 1725! error_pct: 100.0000! warnings: 0!warning_pct: 0.0000! first_seen: 2014-05-20 10:42:32! last_seen: 2014-05-21 18:39:22! digest: 51fb979dbc3910a6294c7cdabd7839c3

Page 144: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!144

Statement With Sorting

mysql> select * from statements_with_sorting limit 1\G!*************************** 1. row ***************************! query: SELECT `s` . `identityId` , `s ... `subject_id` = `s` . `id` ...! db: mem! exec_count: 1406! total_latency: 00:03:01.22!sort_merge_passes: 0! avg_sort_merges: 0!sorts_using_scans: 1406! sort_using_range: 1406! rows_sorted: 39865! avg_rows_sorted: 28! first_seen: 2014-05-20 10:42:08! last_seen: 2014-05-21 14:04:02! digest: 9972d1d2cc12b92fe938ec5bd74d2f1d

Page 145: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!145

Wait Analysis Views

▪ Wait summaries by class and per instrument globally

▪ Wait details per user !

mysql> show tables like 'wait%';!+------------------------------------+!| Tables_in_sys (wait%) |!+------------------------------------+!| wait_classes_global_by_avg_latency |!| wait_classes_global_by_latency |!| waits_by_user_by_latency |!| waits_global_by_latency |!+------------------------------------+

Page 146: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!146

Global Wait Summarymysql> select * from waits_global_by_latency;!+------------------------------------------------------+----------+---------------+-------------+-------------+!| events | total | total_latency | avg_latency | max_latency |!+------------------------------------------------------+----------+---------------+-------------+-------------+!| wait/io/table/sql/handler | 24196659 | 1.75h | 259.72 us | 2.29 s |!| wait/io/file/innodb/innodb_data_file | 5964255 | 1.36h | 822.01 us | 1.49 s |!| wait/io/file/innodb/innodb_log_file | 2272647 | 00:36:41.37 | 968.64 us | 1.30 s |!| wait/io/file/myisam/kfile | 5463587 | 00:18:14.16 | 200.26 us | 1.17 s |!| wait/io/file/myisam/dfile | 1715187 | 00:08:00.18 | 279.96 us | 1.18 s |!| wait/io/file/sql/binlog | 3722079 | 00:03:23.71 | 54.73 us | 1.42 s |!| wait/lock/table/sql/handler | 10074274 | 00:01:56.63 | 11.58 us | 186.90 ms |!| wait/io/socket/sql/client_connection | 3879827 | 25.36 s | 6.54 us | 219.25 ms |!| wait/io/file/sql/FRM | 43994 | 3.27 s | 74.35 us | 80.43 ms |!| wait/io/file/sql/dbopt | 116765 | 1.29 s | 11.03 us | 63.27 ms |!| wait/io/file/sql/binlog_index | 95 | 1.20 s | 12.67 ms | 439.79 ms |!| wait/io/file/csv/metadata | 32 | 577.39 ms | 18.04 ms | 245.23 ms |!| wait/synch/mutex/sql/THD::LOCK_thd_data | 10300455 | 221.84 ms | 21.49 ns | 12.62 ms |!| wait/synch/mutex/sql/THD::LOCK_query_plan | 5656929 | 168.50 ms | 29.78 ns | 11.89 ms |!| wait/io/file/archive/data | 3533 | 19.11 ms | 5.41 us | 5.05 ms |!| wait/io/file/mysys/cnf | 5 | 3.75 ms | 750.59 us | 3.60 ms |!| wait/synch/mutex/myisam/MYISAM_SHARE::intern_lock | 88257 | 2.20 ms | 24.88 ns | 361.49 us |!| wait/io/file/sql/ERRMSG | 5 | 1.14 ms | 227.99 us | 502.64 us |!| wait/io/file/csv/data | 12 | 1.03 ms | 85.56 us | 348.11 us |!……

Page 147: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!147

User Wait Summarymysql> select * from waits_by_user_by_latency;!+------+------------------------------------------------------+----------+---------------+-------------+-------------+!| user | event | total | total_latency | avg_latency | max_latency |!+------+------------------------------------------------------+----------+---------------+-------------+-------------+!| mark | wait/io/file/myisam/dfile | 19540 | 215.20 ms | 11.01 us | 120.79 ms |!| mark | wait/io/file/myisam/kfile | 501 | 131.58 ms | 262.63 us | 57.74 ms |!| mark | wait/io/file/sql/dbopt | 2 | 17.29 us | 8.64 us | 9.17 us |!| root | wait/io/table/sql/handler | 24189319 | 1.74h | 258.77 us | 2.29 s |!| root | wait/io/file/innodb/innodb_log_file | 2162650 | 00:30:52.87 | 856.76 us | 1.30 s |!| root | wait/io/file/myisam/kfile | 5460992 | 00:18:12.83 | 200.11 us | 1.17 s |!| root | wait/io/file/myisam/dfile | 1694624 | 00:07:59.50 | 282.95 us | 1.18 s |!| root | wait/io/file/sql/binlog | 3711637 | 00:03:23.16 | 54.74 us | 1.42 s |!| root | wait/io/file/innodb/innodb_data_file | 22577 | 00:03:05.73 | 8.23 ms | 492.23 ms |!| root | wait/lock/table/sql/handler | 10069000 | 00:01:56.59 | 11.58 us | 186.90 ms |!| root | wait/io/socket/sql/client_connection | 3858476 | 25.36 s | 6.57 us | 219.25 ms |!| root | wait/io/file/sql/FRM | 42701 | 3.18 s | 74.52 us | 80.43 ms |!| root | wait/io/file/sql/dbopt | 116693 | 1.29 s | 11.03 us | 63.27 ms |!| root | wait/io/file/sql/binlog_index | 80 | 1.16 s | 14.47 ms | 439.79 ms |!| root | wait/io/file/csv/metadata | 32 | 577.39 ms | 18.04 ms | 245.23 ms |!| root | wait/synch/mutex/sql/THD::LOCK_thd_data | 10280287 | 221.84 ms | 21.49 ns | 12.62 ms |!| root | wait/synch/mutex/sql/THD::LOCK_query_plan | 5642052 | 168.50 ms | 29.78 ns | 11.89 ms |!| root | wait/io/file/archive/data | 3529 | 19.09 ms | 5.41 us | 5.05 ms |!……

Page 148: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!148

MySQL SYS Procedures - P_S Analysis Helpers

▪ Dump the data to create a graph dot file for a thread trace ▪ ps_trace_thread()

▪ Look in the statement history table trying to capture more info ▪ ps_trace_statement_digest()

▪ Reset all summary data ▪ ps_truncate_all_tables()

Page 149: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!149

Tracing Statement Digests

▪ ps_trace_statement_digest() analyzes live traffic looking for certain statement digest

▪ Captures statistics on each matching statement it finds ▪ Returns a report of the statistics

▪ An overall summary ▪ A break down for the longest running example ▪ An EXPLAIN (if the statement is not truncated)

Page 150: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!150

Tracing Statement Digests

▪ ps_trace_statement_digest() parameters ▪ in_digest The statement digest to analyze ▪ in_runtime How long to run analysis for ▪ in_interval How often to snapshot for data ▪ in_start_fresh Whether to truncate p_s tables first ▪ in_auto_enable Whether to auto enable required config

Page 151: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!151

Tracing Statement Digests Examplemysql> call analyze_statement_digest('6134e9d6f25eb8e6cddf11f6938f202a', 60, 1, true, true); +--------------------+ | SUMMARY STATISTICS | +--------------------+ | SUMMARY STATISTICS | +--------------------+ 1 row in set (59.93 sec) !+------------+-----------+-----------+-----------+---------------+------------+------------+ | executions | exec_time | lock_time | rows_sent | rows_examined | tmp_tables | full_scans | +------------+-----------+-----------+-----------+---------------+------------+------------+ | 360 | 1.41 s | 138.39 ms | 720 | 0 | 0 | 0 | +------------+-----------+-----------+-----------+---------------+------------+------------+ 1 row in set (59.93 sec) !+--------------------------------+-------+-----------+ | event_name | count | latency | +--------------------------------+-------+-----------+ | stage/sql/Sending data | 14 | 1.10 s | | stage/sql/update | 198 | 667.34 ms | | stage/sql/removing tmp table | 9 | 240.48 ms | ... !

Page 152: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!152

Tracing Statement Digests Example+---------------------------+ | LONGEST RUNNING STATEMENT | +---------------------------+ | LONGEST RUNNING STATEMENT | +---------------------------+ 1 row in set (59.95 sec) !+-----------+-----------+-----------+-----------+---------------+------------+-----------+ | thread_id | exec_time | lock_time | rows_sent | rows_examined | tmp_tables | full_scan | +-----------+-----------+-----------+-----------+---------------+------------+-----------+ | 23277 | 77.64 ms | 97.00 us | 2 | 0 | 0 | 0 | +-----------+-----------+-----------+-----------+---------------+------------+-----------+ 1 row in set (59.95 sec) !+-------------------------------------------------------------------------------------------+ | sql_text | +-------------------------------------------------------------------------------------------+ | /* mem dbpool.default */ insert into `mem__quan`.`normalized_statements_by_server_by_schema` (firstSeen, lastSeen, normalized_statement_id, `schema`, server, id) values (1378983745000, 1378998600000, x'808ACEB88FC4B45BC47C4FE9D86C7F26', 'mem', '753c939e-1b99-11e3-b6d4-bc761a1f4f2f', x'7CFE81F98B1E3FE0895D1AA2C39B326D') ON DUPLICATE KEY UPDATE ... +-------------------------------------------------------------------------------------------+ ... !

Page 153: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!153

Tracing Statement Digests Example+--------------------------------+----------+ | event_name | latency | +--------------------------------+----------+ | stage/sql/init | 10.41 ms | | stage/sql/checking permissions | 2.35 us | | stage/sql/Opening tables | 10.32 us | | stage/sql/init | 13.53 us | | stage/sql/System lock | 3.66 us | | stage/sql/update | 61.74 ms | | stage/sql/end | 1.11 us | | stage/sql/query end | 1.31 us | | stage/sql/closing tables | 5.44 ms | | stage/sql/freeing items | 23.31 us | +--------------------------------+----------+ 10 rows in set (59.99 sec) !+----+-------------+-------+------+---------------+------+---------+------+------+----------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+------+---------------+------+---------+------+------+----------------+ | 1 | SIMPLE | NULL | NULL | NULL | NULL | NULL | NULL | NULL | No tables used | +----+-------------+-------+------+---------------+------+---------+------+------+----------------+ ... !

Page 154: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!154

Tracing Threads

▪ ps_trace_thread() monitors a specific thread for a period ▪ Captures as much information on the thread activity as possible ▪ Returns a “dot” formatted file, that can graph the event hierarchy as

previously seen ▪ http://en.wikipedia.org/wiki/DOT_(graph_description_language)

Page 155: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!155

Tracing Threads

▪ ps_trace_thread() parameters ▪ in_thread_id The thread to analyze ▪ in_outfile The file to dump the data to (INTO OUTFILE) ▪ in_max_runtime How long to run analysis for ▪ in_interval How often to snapshot for data ▪ in_start_fresh Whether to truncate p_s tables first ▪ in_auto_setup Whether to auto enable required config ▪ in_debug Whether to also print debug info (source info)

Page 156: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!156

Tracing Threads Examplemysql> call dump_thread_stack(27768, '/tmp/stack_27768.dot', 60, 0.1, true, true, true); +------------------------------------------------+ | Info | +------------------------------------------------+ | Data collection starting for THREAD_ID = 27768 | +------------------------------------------------+ 1 row in set (4.82 sec) !+---------------------------------------------+ | Info | +---------------------------------------------+ | Stack trace written to /tmp/stack_27768.dot | +---------------------------------------------+ 1 row in set (60.90 sec) !+--------------------------------------------------------+ | Convert to PDF | +--------------------------------------------------------+ | dot -Tpdf -o /tmp/stack_27768.pdf /tmp/stack_27768.dot | +--------------------------------------------------------+ 1 row in set (60.90 sec) !+--------------------------------------------------------+ | Convert to PNG | +--------------------------------------------------------+ | dot -Tpng -o /tmp/stack_27768.png /tmp/stack_27768.dot | +--------------------------------------------------------+ 1 row in set (60.90 sec)+ !!

Page 157: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!157

Tracing Threads Dot Format

cerberus:~ mark$ more Documents/statement_graphs/deletes.dot!digraph events {! graph [rankdir=LR, nodesep="0.10"];! node [label="\N"];! graph [bb="0,0,1860,17476"];! 1 [label="(0.20μ) mysys/THR_LOCK_threads\n", style=filled, color=lightskyblue, pos="209,18", width="3.19", height="0.50"];! 2 [label="(0.18μ) sql/THD::LOCK_thd_data\n", style=filled, color=lightskyblue, pos="209,61", width="3.06", height="0.50"];! 3 [label="(0.24μ) sql/hash_filo::lock\n", style=filled, color=lightskyblue, pos="209,104", width="2.42", height=“0.50"];! 4 [label="(0.31μ) sql/LOGGER::LOCK_logger - read_lock\n", style=filled, color=orchid, pos="209,147", width="4.22", height="0.50"];! 5 [label="(0.08μ) sql/LOG::LOCK_log\n", style=filled, color=lightskyblue, pos="209,190", width="2.64", height="0.50"];! 6 [label="(12.97μ) wait/io/file/sql/query_log - write 49 bytes\n/Users/mark/mysql/lp-mysql-trunk/mysql-test/var/mysqld.1/mysqld.log\n", style=filled, color=red, shape=box, pos="209,235", width="5.78", height="0.56"];! 7 [label="(176.14μ) sql/select\nselect @@version_comment limit 1\nerrors: 0\nwarnings: 0\nlock time: 0.00μ\nrows affected: 0\nrows sent:\!209,18", width="3.19", height="0.50"];mp disk tables: 0\nselect scan: 0\nselect full join: :! 2 [label="(0.18μ) sql/THD::LOCK_thd_data\n", style=filled, color=lightskyblue, pos="209,61", width="3.06", height="0.50"];! 3 [label="(0.24μ) sql/hash_filo::lock\n", style=filled, color=lightskyblue, pos="209,104", width="2.42", height="0.50"];! 4 [label="(0.31μ) sql/LOGGER::LOCK_logger - read_lock\n", style=filled, color=orchid, pos="209,147", width="4.22", height="0.50"];! 5 [label="(0.08μ) sql/LOG::LOCK_log\n", style=filled, color=lightskyblue, pos="209,190", width="2.64", height="0.50"];!……!

Page 158: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!158

Tracing Statement Output

Page 159: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!159

MySQL SYS Procedures - P_S Setup Helpers

▪ ps_setup_show_disabled() / ps_setup_show_enabled() ▪ ps_setup_disable_thread() / ps_setup_enable_thread() ▪ ps_setup_disable_background_threads() /

ps_setup_enable_background_threads() ▪ ps_setup_disable_instrument() / ps_setup_enable_instrument() ▪ ps_setup_disable_consumer() / ps_setup_enable_consumer() ▪ ps_setup_save() / ps_setup_reload_saved() ▪ ps_setup_reset_to_default() ▪ ps_truncate_all_tables()

Page 160: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!160

Things still to come

Page 161: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!161

Status / Variable Instrumentation

▪ Deprecate I_S.[GLOBAL|SESSION]_[STATUS|VARIABLES] ▪ Replace with tables that show true state / aggregates

▪ Per thread, user, host, account ▪ See session statistics or configuration individually

▪ Improve SHOW commands to only show appropriate variables ▪ SHOW SESSION STATUS to only show session specific items ▪ Backwards compatibility switch available

▪ Less internal mutex contention for polling status

Page 162: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!162

Further Replication Statistics

▪ Redesigned lag monitoring ▪ Consistently check lag across a replication chain

▪ Further statistics on load of the replication threads ▪ Queue lengths ▪ Event counting ▪ Busy times

Page 163: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!163

Stage Progress Tracking

▪ View the progress of longer stages of execution ▪ Extends stage tables with “estimate” and “completed” columns ▪ Bounded (“estimated”) or unbounded (only “completed” filled in) ▪ Units depend on stage

▪ stage/sql/copy to tmp table - 130 (rows) completed

Page 164: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!164

Continued focus …

▪ On performance! ▪ On filling missing instrumentation points! ▪ On consolidating instrumentation within Performance schema!

Page 165: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!165

The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

Page 166: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 12!166

Page 167: Mysql tech day_paris_ps_and_sys

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 12!167