The MySQL SYS Schema

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

The MySQL SYS Schema

Mark Leith Senior Software Development Manager MySQL Enterprise Tools, Oracle



Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.


Program Agenda

Introduction to the MySQL sys schema

Installation

Summary Views

Procedures and Functions

1

2

3

4



• Originally called “ps_helper”

• A collection of views, procedures and functions, designed to help DBA, Developer, and Ops teams • Implements many of the common day to day DBA and Developer

debugging and tuning use cases


Why the sys schema?

• Performance Schema exposes a huge amount of data, that can be sliced and diced in many ways • Many of its tables can implement many different use cases when

looking at the raw data • The sys schema layers on top the (sometimes complex) queries in

to a packaged set of easy to use views • The sys schema also layers on a number of functions and

procedures that help format, and record the raw data


Program Agenda


Installation

Summary Views


1

2

3

4


Installation from the command line

• $ git clone https://github.com/MarkLeith/mysql-sys.git /tmp/sys

• $ cd /tmp/sys • $ mysql -u user -p < sys_<version>.sql !! <version> can be 56 or 57, for 5.6 and 5.7 respectively

https://github.com/MarkLeith/mysql-sys.git


Installation from MySQL Workbench 6.1+


!!select * from sys.version;!

+-------------+---------------+!

| sys_version | mysql_version |!

+-------------+---------------+!

| 1.1.0 | 5.6.17 |!

+-------------+---------------+

11

Checking sys is loadedselect *!

from sys.schema_object_overview!

where db = 'sys';!

+-----+-------------+-------+!

| db | object_type | count |!

+-----+-------------+-------+!

| sys | VIEW | 78 |!

| sys | PROCEDURE | 22 |!

| sys | FUNCTION | 11 |!

+-----+-------------+-------+


Program Agenda


Installation

Summary Views


1

3

2

4


MySQL sys views

• Reference set of views solving various administrator use cases • Simple views, create/copy your own, sys is not “locked down”

• Build upon both performance_schema and INFORMATION_SCHEMA • Both formatted and raw views are available • All raw views are prefixed with x$ • Raw views are there for tools to poll

• Formatted views are for humans and the command line


!• High level user/host overviews • Breakdowns by: • IO usage • Stages • Statement details !

• Following show users only

14

User / Host Summary Views

user_summary user_summary_by_file_io user_summary_by_file_io_type user_summary_by_stages user_summary_by_statement_latency user_summary_by_statement_type host_summary host_summary_by_file_io host_summary_by_file_io_type host_summary_by_stages host_summary_by_statement_latency host_summary_by_statement_type


user_summary viewmysql> select * from user_summary\G!*************************** 1. row ***************************! user: mark! statements: 3072! statement_latency: 1.77 s! statement_avg_latency: 575.29 us! table_scans: 7! file_ios: 20043! file_io_latency: 346.79 ms! current_connections: 1! total_connections: 1! unique_hosts: 1! current_memory: 515.81 KiB!total_memory_allocated: 30.69 MiB

MySQL 5.7.4+


user_summary_by_file_io_type view

mysql> select * from user_summary_by_file_io_type;!+------+--------------------------------------+---------+-------------+-------------+!| user | event_name | total | latency | max_latency |!+------+--------------------------------------+---------+-------------+-------------+!| mark | wait/io/file/myisam/dfile | 19540 | 215.20 ms | 120.79 ms |!| mark | wait/io/file/myisam/kfile | 501 | 131.58 ms | 57.74 ms |!| mark | wait/io/file/sql/dbopt | 2 | 17.29 us | 9.17 us |!| root | wait/io/file/innodb/innodb_log_file | 2079432 | 00:28:27.62 | 1.30 s |!| root | wait/io/file/myisam/kfile | 5195927 | 00:17:07.33 | 1.17 s |!| root | wait/io/file/myisam/dfile | 1533813 | 00:07:27.59 | 1.18 s |!| root | wait/io/file/sql/binlog | 3517008 | 00:03:07.34 | 1.42 s |!| root | wait/io/file/innodb/innodb_data_file | 20720 | 00:02:37.81 | 492.23 ms |!| root | wait/io/file/sql/FRM | 41412 | 3.17 s | 80.43 ms |!| root | wait/io/file/sql/dbopt | 110993 | 1.23 s | 63.27 ms |!!


user_summary_by_statement_latency view

mysql> select * from user_summary_by_statement_latency\G!*************************** 1. row ***************************! user: mark! total: 176!total_latency: 1.26 s! max_latency: 583.33 ms! lock_latency: 216.95 ms! rows_sent: 311!rows_examined: 307!rows_affected: 0! full_scans: 7!


user_summary_by_statement_type view

mysql> select * from user_summary_by_statement_type\G!*************************** 1. row ***************************! user: mark! statement: commit! total: 2463190!total_latency: 13.33h! max_latency: 12.18 s! lock_latency: 0 ps! rows_sent: 0!rows_examined: 0!rows_affected: 0! full_scans: 0!


!• IO by current threads • Global summaries by file and

class, aggregating by bytes or latency • Stream of last file IO events in

raw form

19

IO Summary Views

!io_by_thread_by_latency io_global_by_file_by_bytes io_global_by_file_by_latency io_global_by_wait_by_bytes io_global_by_wait_by_latency


io_by_thread_by_latency viewmysql> select * from io_by_thread_by_latency\G!*************************** 1. row ***************************! user: main! total: 1193! total_latency: 136.45 ms! min_latency: 731.38 ns! avg_latency: 269.99 us! max_latency: 20.72 ms! thread_id: 1!processlist_id: NULL!*************************** 2. row ***************************! user: root@localhost! total: 533! total_latency: 53.81 ms! min_latency: 780.39 ns! avg_latency: 8.41 ms! max_latency: 25.06 ms! thread_id: 23!processlist_id: 4


io_global_by_file_by_latency view

mysql> select * from io_global_by_file_by_latency limit 1\G!*************************** 1. row ***************************! file: @@datadir/ibdata1! total: 1395714!total_latency: 00:28:53.06! count_read: 1006! read_latency: 4.51 s! count_write: 1326529!write_latency: 33.39 s! count_misc: 68179! misc_latency: 00:28:15.16


!• Object overview • Table usage statistics • Index usage statistics • Following show users only

23

Schema Analysis Views

schema_index_statistics schema_object_overview schema_table_statistics schema_table_statistics_with_buffer schema_tables_with_full_table_scans schema_unused_indexes


schema_table_statistics viewmysql> select * from schema_table_statistics limit 1\G!*************************** 1. row ***************************! table_schema: mem__quan! table_name: normalized_statements_by_server_by_schema_data! total_latency: 00:55:07.80! rows_fetched: 239685! fetch_latency: 17.10 s! rows_inserted: 1102895! insert_latency: 00:52:18.10! rows_updated: 55910! update_latency: 00:02:28.62! rows_deleted: 17899! delete_latency: 3.98 s

io_read_requests: 20639! io_read: 324.94 MiB! io_read_latency: 00:03:04.10!io_write_requests: 1965973! io_write: 40.36 GiB! io_write_latency: 25.46 s! io_misc_requests: 18217! io_misc_latency: 00:14:06.02


schema_index_statistics view

mysql> select * from schema_index_statistics limit 1\G!*************************** 1. row ***************************! table_schema: mem__inventory! table_name: mysqlconnectionconfiguration! index_name: PRIMARY! rows_selected: 131399!select_latency: 00:07:05.03! rows_inserted: 0!insert_latency: 0 ps! rows_updated: 41608!update_latency: 21.96 s! rows_deleted: 0!delete_latency: 0 ps


schema_tables_with_full_table_scans view

mysql> select * from schema_tables_with_full_table_scans limit 10;!+--------------------+--------------------------------+-------------------+-----------+!| object_schema | object_name | rows_full_scanned | latency |!+--------------------+--------------------------------+-------------------+-----------+!| mem30__instruments | fsstatistics | 10207042 | 13.10 s |!| mem30__instruments | preparedstatementapidata | 436428 | 973.27 ms |!| mem30__instruments | mysqlprocessactivity | 411702 | 282.07 ms |!| mem30__instruments | querycachequeriesincachedata | 374011 | 767.15 ms |!| mem30__instruments | rowaccessesdata | 322321 | 1.55 s |!| mem30__instruments | connectionsmaxdata | 301242 | 857.63 ms |!| mem30__instruments | innodbdatadictionarymemorydata | 292564 | 495.38 ms |!| mem30__instruments | innodbbufferpooldata | 277112 | 1.95 s |!| mem30__instruments | avgrowaccessesdata | 269236 | 1.25 s |!| mem30__instruments | innodbrowdetailsdata | 248208 | 653.25 ms |!+--------------------+--------------------------------+-------------------+-----------+


!• Wait summaries by class • Wait details per: • User • Host • Globally

28

Wait Analysis Views

wait_classes_global_by_avg_latency wait_classes_global_by_latency waits_by_host_by_latency waits_by_user_by_latency waits_global_by_latency !


waits_global_by_latency view

mysql> select * from waits_global_by_latency limit 10;!+--------------------------------------+----------+---------------+-------------+-------------+!| events | total | total_latency | avg_latency | max_latency |!+--------------------------------------+----------+---------------+-------------+-------------+!| wait/io/table/sql/handler | 24196659 | 1.75h | 259.72 us | 2.29 s |!| wait/io/file/innodb/innodb_data_file | 5964255 | 1.36h | 822.01 us | 1.49 s |!| wait/io/file/innodb/innodb_log_file | 2272647 | 00:36:41.37 | 968.64 us | 1.30 s |!| wait/io/file/myisam/kfile | 5463587 | 00:18:14.16 | 200.26 us | 1.17 s |!| wait/io/file/myisam/dfile | 1715187 | 00:08:00.18 | 279.96 us | 1.18 s |!| wait/io/file/sql/binlog | 3722079 | 00:03:23.71 | 54.73 us | 1.42 s |!| wait/lock/table/sql/handler | 10074274 | 00:01:56.63 | 11.58 us | 186.90 ms |!| wait/io/socket/sql/client_connection | 3879827 | 25.36 s | 6.54 us | 219.25 ms |!| wait/io/file/sql/FRM | 43994 | 3.27 s | 74.35 us | 80.43 ms |!| wait/io/file/sql/dbopt | 116765 | 1.29 s | 11.03 us | 63.27 ms |!+--------------------------------------+----------+---------------+-------------+-------------+

InnoDB only instance, this MyISAM IO is all temporary tables, focus tuning there!


waits_by_user_by_latency view

mysql> select * from waits_by_user_by_latency;!+------+--------------------------------------+----------+---------------+-------------+-------------+!| user | event | total | total_latency | avg_latency | max_latency |!+------+--------------------------------------+----------+---------------+-------------+-------------+!| mark | wait/io/file/myisam/dfile | 19540 | 215.20 ms | 11.01 us | 120.79 ms |!| mark | wait/io/file/myisam/kfile | 501 | 131.58 ms | 262.63 us | 57.74 ms |!| mark | wait/io/file/sql/dbopt | 2 | 17.29 us | 8.64 us | 9.17 us |!| root | wait/io/table/sql/handler | 24189319 | 1.74h | 258.77 us | 2.29 s |!| root | wait/io/file/innodb/innodb_log_file | 2162650 | 00:30:52.87 | 856.76 us | 1.30 s |!| root | wait/io/file/myisam/kfile | 5460992 | 00:18:12.83 | 200.11 us | 1.17 s |!| root | wait/io/file/myisam/dfile | 1694624 | 00:07:59.50 | 282.95 us | 1.18 s |!| root | wait/io/file/sql/binlog | 3711637 | 00:03:23.16 | 54.74 us | 1.42 s |!| root | wait/io/file/innodb/innodb_data_file | 22577 | 00:03:05.73 | 8.23 ms | 492.23 ms |!| root | wait/lock/table/sql/handler | 10069000 | 00:01:56.59 | 11.58 us | 186.90 ms |!……

root is doing the most temporary table IO


!• Statement overview • Find statements by: • Those with errors • Those with full table scans • Those creating temp tables • Those that cause sorting • With latency in 95th %ile

31

Statement Analysis Views

statement_analysis statements_with_errors_or_warnings statements_with_full_table_scans statements_with_runtimes_in_95th_percentile statements_with_sorting statements_with_temp_tables


statement_analysis viewmysql> select * from statement_analysis limit 2\G!*************************** 1. row ***************************! query: INSERT INTO `mem30__quan` . `n ... nDuration` = IF ( VALUES ( ...! db: mem! full_scan:! exec_count: 4657! err_count: 0! warn_count: 0! total_latency: 00:03:57.40! max_latency: 10.54 s! avg_latency: 50.98 ms! lock_latency: 00:03:00.93! rows_sent: 0! rows_sent_avg: 0! rows_examined: 0!rows_examined_avg: 0! rows_affected: 4794!rows_affected_avg: 1

tmp_tables: 0! tmp_disk_tables: 0! rows_sorted: 0!sort_merge_passes: 0! digest: b98bfe333642e8e4e72c73515810d865! first_seen: 2014-09-24 11:27:01! last_seen: 2014-09-24 11:53:00


statements_with_errors_or_warnings view

mysql> select * from statements_with_errors_or_warnings limit 1\G!*************************** 1. row ***************************! query: CREATE TEMPORARY TABLE IF NOT ... _logs` ( `id` INT8 NOT NULL )! db: mem! exec_count: 1725! errors: 1725! error_pct: 100.0000! warnings: 0!warning_pct: 0.0000! first_seen: 2014-05-20 10:42:32! last_seen: 2014-05-21 18:39:22! digest: 51fb979dbc3910a6294c7cdabd7839c3


statements_with_temp_tables view

mysql> select * from statements_with_temp_tables limit 1\G!*************************** 1. row ***************************! query: SELECT * FROM ( SELECT `digest ... ` , `sum_no_index_used` AS ...! db: mysql! exec_count: 68! total_latency: 2.31 s! memory_tmp_tables: 476! disk_tmp_tables: 204!avg_tmp_tables_per_query: 7! tmp_tables_to_disk_pct: 43! first_seen: 2014-09-24 11:26:18! last_seen: 2014-09-24 12:33:00! digest: 50761c6a1818824328745d8a136b9ed6


statements_with_full_table_scans viewmysql> select * from statements_with_full_table_scans limit 1\G!*************************** 1. row ***************************! query: SELECT `pojotimese0_` . `id` A ... ty_timeseries` `pojotimese0_`! db: mem! exec_count: 1! total_latency: 940.89 us! no_index_used_count: 1!no_good_index_used_count: 0! no_index_used_pct: 100! rows_sent: 116! rows_examined: 116! rows_sent_avg: 116! rows_examined_avg: 116! first_seen: 2014-09-24 11:25:03! last_seen: 2014-09-24 11:25:03! digest: f81b5ce10baee9be754cf35fd245a516


!• Better SHOW PROCESSLIST • No mutex contention • Live statistics for query

36

Miscellaneous Views

processlist !


processlist viewmysql> select * from processlist\G!*************************** 1. row ***************************! thd_id: 7890! conn_id: 7870! user: root@localhost! db: mem! command: Query! state: update! time: 0! current_statement: /* mem dbpool.default */ inser ...<truncated>! lock_latency: 0 ps! rows_examined: 0! rows_sent: 0! rows_affected: 0! tmp_tables: 0! !!

tmp_disk_tables: 0! full_scan: NO! last_statement: NULL!last_statement_latency: NULL! last_wait: wait/io/table/sql/handler! last_wait_latency: Still Waiting! source: handler.cc:7274


Program Agenda


Installation

Summary Views


1

4

2

3


!• Make output human readable • Format time appropriately • Format bytes appropriately • Truncate output for large

width values for CLI • Extract object names • Dump a thread stack • Check instrumented actors

39

Functions

format_time format_bytes format_path format_statement extract_table_from_file_name extract_schema_from_file_name ps_thread_stack ps_is_account_enabled !


format_time() function

mysql> select format_time(23849723429) as time! -> union select format_time(8327423749233)! -> union select format_time(83274237492335);!+-------------+!| time |!+-------------+!| 23.85 ms |!| 8.33 s |!| 00:01:23.27 |!+-------------+! !!


format_bytes() function

mysql> select format_bytes(23423) as bytes! -> union select format_bytes(23432423)! -> union select format_bytes(42839479283)! -> union select format_bytes(2293848203489);!+-----------+!| bytes |!+-----------+!| 22.87 KiB |!| 22.35 MiB |!| 39.90 GiB |!| 2.09 TiB |!+-----------+!


Performance Schema Config Helper Procedures

• ps_setup_show_disabled() / ps_setup_show_enabled() • ps_setup_disable_thread() / ps_setup_enable_thread() • ps_setup_disable_background_threads() /

ps_setup_enable_background_threads() • ps_setup_disable_instrument() / ps_setup_enable_instrument() • ps_setup_disable_consumer() / ps_setup_enable_consumer() • ps_setup_save() / ps_setup_reload_saved() • ps_setup_reset_to_default() / ps_truncate_all_tables()


!• Dump all P_S data to a dot

graph file to trace a thread • Look in the statement history

table to capture more info on a statement digest

43

Statement Tracing Procedures

ps_trace_thread ps_trace_statement_digest ps_statement_avg_latency_histogram !


ps_trace_statement_digest procedure

• ps_trace_statement_digest() analyses live traffic looking for certain statement digest for a period of time • Captures statistics on each matching statement it finds • Returns a report of the captured stats • An overall summary • A break down for the longest running example • An EXPLAIN (if the statement is not truncated)



• ps_trace_statement_digest() parameters • in_digest The statement digest to analyse • in_runtime How long to run analysis for • in_interval How often to snapshot data • in_start_fresh Whether to truncate P_S tables first • in_auto_enable Whether to auto enable required config


ps_trace_statement_digest procedure examplemysql> call analyze_statement_digest('6134e9d6f25eb8e6cddf11f6938f202a', 60, 1, true, true); +--------------------+ | SUMMARY STATISTICS | +--------------------+ | SUMMARY STATISTICS | +--------------------+ 1 row in set (59.93 sec) !+------------+-----------+-----------+-----------+---------------+------------+------------+ | executions | exec_time | lock_time | rows_sent | rows_examined | tmp_tables | full_scans | +------------+-----------+-----------+-----------+---------------+------------+------------+ | 360 | 1.41 s | 138.39 ms | 720 | 0 | 0 | 0 | +------------+-----------+-----------+-----------+---------------+------------+------------+ 1 row in set (59.93 sec) !+--------------------------------+-------+-----------+ | event_name | count | latency | +--------------------------------+-------+-----------+ | stage/sql/Sending data | 14 | 1.10 s | | stage/sql/update | 198 | 667.34 ms | | stage/sql/removing tmp table | 9 | 240.48 ms | ... !!!


ps_trace_statement_digest procedure example+---------------------------+ | LONGEST RUNNING STATEMENT | +---------------------------+ | LONGEST RUNNING STATEMENT | +---------------------------+ 1 row in set (59.95 sec) !+-----------+-----------+-----------+-----------+---------------+------------+-----------+ | thread_id | exec_time | lock_time | rows_sent | rows_examined | tmp_tables | full_scan | +-----------+-----------+-----------+-----------+---------------+------------+-----------+ | 23277 | 77.64 ms | 97.00 us | 2 | 0 | 0 | 0 | +-----------+-----------+-----------+-----------+---------------+------------+-----------+ 1 row in set (59.95 sec) !+-------------------------------------------------------------------------------------------+ | sql_text | +-------------------------------------------------------------------------------------------+ | /* mem dbpool.default */ insert into `mem__quan`.`normalized_statements_by_server_by_schema` (firstSeen, lastSeen, normalized_statement_id, `schema`, server, id) values (1378983745000, 1378998600000, x'808ACEB88FC4B45BC47C4FE9D86C7F26', 'mem', '753c939e-1b99-11e3-b6d4-bc761a1f4f2f', x'7CFE81F98B1E3FE0895D1AA2C39B326D') ON DUPLICATE KEY UPDATE ... +-------------------------------------------------------------------------------------------+ ... !!!


ps_trace_thread procedure

• ps_trace_thread() monitors a specific thread for a period • Captures as much information on the thread activity as possible • Returns a “dot” formatted file, that can graph the event hierarchy • http://en.wikipedia.org/wiki/DOT_(graph_description_language)

http://en.wikipedia.org/wiki/DOT_(graph_description_language)



• ps_trace_statement_digest() parameters • in_thread_id The thread to analyse • in_outfile The file to dump the data to (INTO OUTFILE) • in_max_runtime How long to run analysis for • in_interval How often to snapshot for data • in_start_fresh Whether to truncate P_S tables first • in_auto_setup Whether to auto enable required config • in_debug Whether to also print debug info (source info)


ps_trace_thread procedure examplemysql> call ps_trace_thread(27768, '/tmp/stack_27768.dot', 60, 0.1, true, true, true); +------------------------------------------------+ | Info | +------------------------------------------------+ | Data collection starting for THREAD_ID = 27768 | +------------------------------------------------+ 1 row in set (4.82 sec) !+---------------------------------------------+ | Info | +---------------------------------------------+ | Stack trace written to /tmp/stack_27768.dot | +---------------------------------------------+ 1 row in set (60.90 sec) !+--------------------------------------------------------+ | Convert to PDF | +--------------------------------------------------------+ | dot -Tpdf -o /tmp/stack_27768.pdf /tmp/stack_27768.dot | +--------------------------------------------------------+ 1 row in set (60.90 sec) !+--------------------------------------------------------+ | Convert to PNG | +--------------------------------------------------------+ | dot -Tpng -o /tmp/stack_27768.png /tmp/stack_27768.dot | +--------------------------------------------------------+ 1 row in set (60.90 sec)+

!


ps_trace_thread procedure example

cerberus:~ mark$ more Documents/statement_graphs/deletes.dot!digraph events {! graph [rankdir=LR, nodesep="0.10"];! node [label="\N"];! graph [bb="0,0,1860,17476"];! 1 [label="(0.20μ) mysys/THR_LOCK_threads\n", style=filled, color=lightskyblue, pos="209,18", width="3.19", height="0.50"];! 2 [label="(0.18μ) sql/THD::LOCK_thd_data\n", style=filled, color=lightskyblue, pos="209,61", width="3.06", height="0.50"];! 3 [label="(0.24μ) sql/hash_filo::lock\n", style=filled, color=lightskyblue, pos="209,104", width="2.42", height=“0.50"];! 4 [label="(0.31μ) sql/LOGGER::LOCK_logger - read_lock\n", style=filled, color=orchid, pos="209,147", width="4.22", height="0.50"];! 5 [label="(0.08μ) sql/LOG::LOCK_log\n", style=filled, color=lightskyblue, pos="209,190", width="2.64", height="0.50"];! 6 [label="(12.97μ) wait/io/file/sql/query_log - write 49 bytes\n/Users/mark/mysql/lp-mysql-trunk/mysql-test/var/mysqld.1/mysqld.log\n", style=filled, color=red, shape=box, pos="209,235", width="5.78", height="0.56"];! 7 [label="(176.14μ) sql/select\nselect @@version_comment limit 1\nerrors: 0\nwarnings: 0\nlock time: 0.00μ\nrows affected: 0\nrows sent:\!209,18", width="3.19", height="0.50"];mp disk tables: 0\nselect scan: 0\nselect full join: :! 2 [label="(0.18μ) sql/THD::LOCK_thd_data\n", style=filled, color=lightskyblue, pos="209,61", width="3.06", height="0.50"];! 3 [label="(0.24μ) sql/hash_filo::lock\n", style=filled, color=lightskyblue, pos="209,104", width="2.42", height="0.50"];! 4 [label="(0.31μ) sql/LOGGER::LOCK_logger - read_lock\n", style=filled, color=orchid, pos="209,147", width="4.22", height="0.50"];! 5 [label="(0.08μ) sql/LOG::LOCK_log\n", style=filled, color=lightskyblue, pos="209,190", width="2.64", height="0.50"];!……!!


ps_trace_thread procedure example

MySQL 5.7 !Procedures Transactions Statements Stages Waits

The MySQL SYS Schema

Technology

Transcript of The MySQL SYS Schema