MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema...
Transcript of MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema...
![Page 1: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/1.jpg)
MySQL Performance Schemain 1 hour
March, 21, 2019
Sveta Smirnova
![Page 2: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/2.jpg)
•Why Performance Schema?
•How it Works?
•How to Tune?
•How to Ease?
Table of Contents
2
![Page 3: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/3.jpg)
Why Performance Schema?
![Page 4: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/4.jpg)
MetadataSchemaSupported charsetsetc.
RuntimeResource usageAcquired locksRunning queries
Difference
Metadata vs Runtime
4
![Page 5: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/5.jpg)
MetadataAlways existed
• SHOW commands• Information Schema
RuntimeWas not perfect
• Status variables• Extensions for
Information Schema
History
Metadata vs Runtime
4
![Page 6: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/6.jpg)
MetadataInformation Schema
RuntimePerformance Schema
Now
Metadata vs Runtime
4
![Page 7: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/7.jpg)
• No design issues like in Information Schema• No locking reads
Almost
• Optimized• It took one and a half major versions
• Consistently extending• New features in every release
Easy to add your own• Being mature
Performance Schema
5
![Page 8: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/8.jpg)
• No design issues like in Information Schema• No locking reads
• Optimized• It took one and a half major versions
• Consistently extending• New features in every release
Easy to add your own• Being mature
Performance Schema
5
![Page 9: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/9.jpg)
• No design issues like in Information Schema• No locking reads
• Optimized• It took one and a half major versions
• Consistently extending• New features in every release
Easy to add your own
• Being mature
Performance Schema
5
![Page 10: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/10.jpg)
• No design issues like in Information Schema• No locking reads
• Optimized• It took one and a half major versions
• Consistently extending• New features in every release
Easy to add your own• Being mature
Performance Schema
5
![Page 11: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/11.jpg)
Instruments
Tables
Variables
102
1,229
44
87
1,067
42
52
589
31
17
237
16Percona Server
8.05.75.65.5
Performance Schema Over Years
6
![Page 12: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/12.jpg)
5.5: Server insights$ iostat -p nvme0n1 5
...
avg-cpu: %user %nice %system %iowait %steal %idle
73.02 0.00 12.03 6.37 0.00 8.59
Device tps kB_read/s kB_wrtn/s kB_read kB_wrtn
nvme0n1 1235.80 5.60 16662.40 28 83312
nvme0n1p3 1158.00 5.60 16662.40 28 83312
Features
7
![Page 13: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/13.jpg)
5.5: Server insightsmysql> select thread_id, EVENT_NAME, sum(NUMBER_OF_BYTES/1024) sum_kb from events_waits_history
-> where NUMBER_OF_BYTES > 0 group by thread_id, EVENT_NAME order by sum_kb desc;
+-----------+---------------------------------------------------+-------------+
| thread_id | EVENT_NAME | sum_kb |
+-----------+---------------------------------------------------+-------------+
| 10 | wait/io/file/innodb/innodb_parallel_dblwrite_file | 1312.0000 |
| 117 | wait/io/file/sql/binlog | 3.4023 |
| 113 | wait/io/file/sql/binlog | 1.7012 |
| 58 | wait/io/file/sql/query_log | 0.2031 |
+-----------+---------------------------------------------------+-------------+
4 rows in set (0.00 sec)
Features
7
![Page 14: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/14.jpg)
5.5: Server insights5.6: Statements
mysql> SELECT THREAD_ID TID, SUBSTR(SQL_TEXT, 1, 50) SQL_TEXT, ROWS_SENT RS,
-> ROWS_EXAMINED RE,CREATED_TMP_TABLES,NO_INDEX_USED,NO_GOOD_INDEX_USED
-> FROM performance_schema.events_statements_history
-> WHERE NO_INDEX_USED=1 OR NO_GOOD_INDEX_USED=1\G
********************** 1. row **********************
TID: 10124
SQL_TEXT: select emp_no, first_name, last_name from employee
RS: 97750
RE: 397774
CREATED_TMP_TABLES: 0
NO_INDEX_USED: 1
NO_GOOD_INDEX_USED: 0
...
Features
7
![Page 15: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/15.jpg)
5.5: Server insights5.6: Statements5.6: Stages
mysql> SELECT eshl.event_name, sql_text, eshl.timer_wait/1000000000000 w_s
-> FROM performance_schema.events_stages_history_long eshl
-> JOIN performance_schema.events_statements_history_long esthl
-> ON (eshl.nesting_event_id = esthl.event_id)
-> WHERE eshl.timer_wait > 1*10000000000\G
*************************** 1. row ***************************
event_name: stage/sql/Sending data
sql_text: SELECT COUNT(emp_no) FROM employees JOIN salaries USING(emp_no)
WHERE hire_date=from_date
w_s: 0.8170
1 row in set (0.00 sec)
Features
7
![Page 16: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/16.jpg)
5.7: Prepared statementsmysql> select SQL_TEXT, sum(COUNT_EXECUTE) from prepared_statements_instances group by SQL_TEXT;
+---------------------------------------------------------+--------------------+
| SQL_TEXT | sum(COUNT_EXECUTE) |
+---------------------------------------------------------+--------------------+
| COMMIT | 11035 |
| UPDATE sbtest1 SET k=k+1 WHERE id=? | 11872 |
| UPDATE sbtest1 SET c=? WHERE id=? | 11606 |
| DELETE FROM sbtest1 WHERE id=? | 11336 |
| INSERT INTO sbtest1 (id, k, c, pad) VALUES (?, ?, ?, ?) | 11076 |
+---------------------------------------------------------+--------------------+
5 rows in set (0.22 sec)
Features
7
![Page 17: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/17.jpg)
5.7: Stored routinesmysql> call sp_test(1);
Query OK, 1 row affected (0.07 sec)
mysql> select thread_id, event_name, sql_text from events_statements_history
-> where event_name like ’statement/sp%’;
+-----------+-------------------------+-------------------------------------------+
| thread_id | event_name | sql_text |
+-----------+-------------------------+-------------------------------------------+
| 24 | statement/sp/hpush_jump | NULL |
| 24 | statement/sp/stmt | INSERT INTO t1 VALUES(val) |
| 24 | statement/sp/hpop | NULL |
+-----------+-------------------------+-------------------------------------------+
7 rows in set (0.00 sec)
Features
7
![Page 18: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/18.jpg)
5.7: Stored routinesmysql> call sp_test(NULL);
Query OK, 1 row affected (0.07 sec)
mysql> select thread_id, event_name, sql_text from events_statements_history
-> where event_name like ’statement/sp%’;
+-----------+-------------------------+-------------------------------------------+
| thread_id | event_name | sql_text |
+-----------+-------------------------+-------------------------------------------+
| 24 | statement/sp/hpush_jump | NULL |
| 24 | statement/sp/stmt | INSERT INTO t1 VALUES(val) |
| 24 | statement/sp/stmt | INSERT IGNORE INTO t1 VALUES(’Som... |
| 24 | statement/sp/stmt | GET STACKED DIAGNOSTICS CONDITION... |
| 24 | statement/sp/stmt | GET STACKED DIAGNOSTICS CONDITION... |
| 24 | statement/sp/hreturn | NULL |
| 24 | statement/sp/hpop | NULL |
+-----------+-------------------------+
7 rows in set (0.00 sec)
Features
7
![Page 19: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/19.jpg)
5.7: Metadata Locksmysql> select id, db, state, info, time from information_schema.processlist;
+----+--------------------+---------------------------------+----------------------------+------+
| id | db | state | info | time |
+----+--------------------+---------------------------------+----------------------------+------+
| 96 | sbtest | | NULL | 65 |
| 4 | NULL | Waiting on empty queue | NULL | 5451 |
| 94 | performance_schema | executing | select id, db, state, i... | 0 |
| 95 | sbtest | Waiting for table metadata lock | alter table sbtest1 add... | 53 |
+----+--------------------+---------------------------------+----------------------------+------+
4 rows in set (0.00 sec)
Features
7
![Page 20: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/20.jpg)
5.7: Metadata Locksmysql> select processlist_id, object_type, lock_type, lock_status, source
-> from metadata_locks join threads on (owner_thread_id=thread_id)
-> where object_schema=’sbtest’ and object_name=’sbtest1’;
+----------------+-------------+-------------------+-------------+-------------------+
| processlist_id | object_type | lock_type | lock_status | source |
+----------------+-------------+-------------------+-------------+-------------------+
| 96 | TABLE | SHARED_READ | GRANTED | sql_parse.cc:5850 |
| 95 | TABLE | SHARED_UPGRADABLE | GRANTED | sql_parse.cc:5850 |
| 95 | TABLE | EXCLUSIVE | PENDING | mdl.cc:3685 |
+----------------+-------------+-------------------+-------------+-------------------+
3 rows in set (0.00 sec)
Features
7
![Page 21: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/21.jpg)
5.7: Memory usage$ pidstat -r -p 13103
Linux 4.15.0-43-generic (delly) 01/20/2019 _x86_64_ (4 CPU)
09:36:55 PM UID PID minflt/s majflt/s VSZ RSS %MEM Command
09:36:55 PM 1001 13103 3.09 0.02 12133668 8228772 50.97 mysqld
Features
7
![Page 22: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/22.jpg)
5.7: Memory usagemysql> select event_name, count_alloc, count_free,
-> CURRENT_NUMBER_OF_BYTES_USED/1024/1024 as used_mb
-> from memory_summary_global_by_event_name where CURRENT_NUMBER_OF_BYTES_USED > 0
-> order by used_mb desc limit 8;
+-------------------------------+-------------+------------+---------------+
| event_name | count_alloc | count_free | used_mb |
+-------------------------------+-------------+------------+---------------+
| memory/sql/JOIN_CACHE | 15 | 6 | 9216.00000000 |
| memory/innodb/ut0link_buf | 2 | 0 | 24.00004578 |
| memory/innodb/ut0new | 6 | 0 | 16.07953835 |
| memory/innodb/lock0lock | 6241 | 0 | 10.92063904 |
| memory/mysys/KEY_CACHE | 3 | 0 | 8.00205994 |
| memory/innodb/buf0dblwr | 378 | 370 | 5.80024719 |
| memory/innodb/memory | 15686935 | 15682985 | 4.90249634 |
| memory/innodb/ut0pool | 1 | 0 | 4.000022
+-------------------------------+-------------+------------+-----------
8 rows in set (0.04 sec)
Features
7
![Page 23: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/23.jpg)
5.7: Replicationmysql> show slave status \G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 127.0.0.1
Master_User: root
Master_Port: 13000
Connect_Retry: 60
Master_Log_File: master-bin.000002
Read_Master_Log_Pos: 63810611
Relay_Log_File: [email protected]
Relay_Log_Pos: 1156
Relay_Master_Log_File: master-bin.000001
Slave_IO_Running: Yes
Slave_SQL_Running: No
Replicate_Do_DB:
Replicate_Ignore_DB:
...
Features
7
![Page 24: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/24.jpg)
5.7: Replication• IO Thread
replication connection status
replication connection status
• SQL Threadreplication applier configuration
replication applier filters
replication applier global filters
replication applier status
replication applier status by coordinator
replication applier status by worker
• Group replicationreplication group member stats
replication group members
Features
7
![Page 25: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/25.jpg)
5.7: Variablesmysql> select * from variables_by_thread where variable_name=’join_buffer_size’;
+-----------+------------------+----------------+
| THREAD_ID | VARIABLE_NAME | VARIABLE_VALUE |
+-----------+------------------+----------------+
| 129 | join_buffer_size | 262144 |
| 132 | join_buffer_size | 262144 |
| 133 | join_buffer_size | 262144 |
| 134 | join_buffer_size | 262144 |
| 135 | join_buffer_size | 262144 |
...
| 144 | join_buffer_size | 262144 |
| 145 | join_buffer_size | 262144 |
| 146 | join_buffer_size | 262144 |
| 147 | join_buffer_size | 262144 |
| 148 | join_buffer_size | 1073741824 |
+-----------+------------------+----------------+
18 rows in set (0.06 sec)
Features
7
![Page 26: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/26.jpg)
5.7: Variablesmysql> select * from status_by_thread
-> where variable_name=’Handler_write’;
+-----------+---------------+----------------+
| THREAD_ID | VARIABLE_NAME | VARIABLE_VALUE |
+-----------+---------------+----------------+
| 71 | Handler_write | 94 |
| 83 | Handler_write | 477 | -- Most writes
| 84 | Handler_write | 101 |
+-----------+---------------+----------------+
3 rows in set (0.00 sec)
Features
7
![Page 27: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/27.jpg)
5.7: Variablesmysql> select * from user_variables_by_thread;
+-----------+---------------+----------------+
| THREAD_ID | VARIABLE_NAME | VARIABLE_VALUE |
+-----------+---------------+----------------+
| 71 | baz | boo |
| 84 | foo | bar |
+-----------+---------------+----------------+
2 rows in set (0.00 sec)
Features
7
![Page 28: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/28.jpg)
8.0: Errorsmysql> select ERROR_NUMBER NUM, ERROR_NAME, SUM_ERROR_RAISED SUM, FIRST_SEEN, LAST_SEEN
-> from events_errors_summary_global_by_error where SUM_ERROR_RAISED > 0
-> order by SUM_ERROR_RAISED desc;
+------+---------------------------+-------+---------------------+---------------------+
| NUM | ERROR_NAME | SUM | FIRST_SEEN | LAST_SEEN |
+------+---------------------------+-------+---------------------+---------------------+
| 1213 | ER_LOCK_DEADLOCK | 17986 | 2019-01-20 20:51:34 | 2019-01-20 23:05:43 |
| 1287 | ER_WARN_DEPRECATED_SYNTAX | 207 | 2019-01-20 20:51:41 | 2019-01-20 23:03:29 |
| 1158 | ER_NET_READ_ERROR | 96 | 2019-01-20 20:57:36 | 2019-01-20 23:05:44 |
| 1205 | ER_LOCK_WAIT_TIMEOUT | 72 | 2019-01-20 20:52:19 | 2019-01-20 23:04:57 |
| 1295 | ER_UNSUPPORTED_PS | 64 | 2019-01-20 20:51:33 | 2019-01-20 23:03:26 |
| 3719 | ER_DEPRECATED_UTF8_ALIAS | 33 | 2019-01-20 20:50:33 | 2019-01-20 20:50:34 |
| 1064 | ER_PARSE_ERROR | 2 | 2019-01-20 21:11:33 | 2019-01-20 21:11:33 |
| 1317 | ER_QUERY_INTERRUPTED | 2 | 2019-01-20 21:24:23 | 2019-01-20 21:29:37 |
| 1054 | ER_BAD_FIELD_ERROR | 1 | 2019-01-20 22:14:57 | 2019-
| 1066 | ER_NONUNIQ_TABLE | 1 | 2019-01-20 21:23:45 | 2019
| 1160 | ER_NET_ERROR_ON_WRITE | 1 | 2019-01-20 21:51:36 | 2019-
...
Features
7
![Page 29: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/29.jpg)
5.7: Prepared statements5.7: Stored routines5.7: Metadata Locks5.7: Memory usage5.7: Replication5.7: Variables8.0: Errors
• More
Features
7
![Page 30: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/30.jpg)
How it Works?
![Page 31: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/31.jpg)
• Wraps the diagnosed codelocker = PSI_RWLOCK_CALL(start_rwlock_wrwait)(
&state, lock->pfs_psi, PSI_RWLOCK_TRYEXCLUSIVELOCK,
file_name, static_cast<uint>(line));
ret = rw_lock_x_lock_func_nowait(lock, file_name, line);
if (locker != NULL) {
PSI_RWLOCK_CALL(end_rwlock_wrwait)(locker, static_cast<int>(ret));
}
Gets the Data
9
![Page 32: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/32.jpg)
CREATE TABLE ‘events_NAME_current‘ (
‘[OWNER_]THREAD_ID‘ bigint(20) unsigned NOT NULL,
‘[OWNER_]EVENT_ID‘ bigint(20) unsigned NOT NULL,
‘END_EVENT_ID‘ bigint(20) unsigned DEFAULT NULL,
‘EVENT_NAME‘ varchar(128) NOT NULL,
‘SOURCE‘ varchar(64) DEFAULT NULL,
‘TIMER_START‘ bigint(20) unsigned DEFAULT NULL,
‘TIMER_END‘ bigint(20) unsigned DEFAULT NULL,
‘TIMER_WAIT‘ bigint(20) unsigned DEFAULT NULL,
‘OBJECT_SCHEMA‘ varchar(64) DEFAULT NULL,
‘OBJECT_NAME‘ varchar(512) DEFAULT NULL,
‘OBJECT_TYPE‘ varchar(64) DEFAULT NULL,
‘OBJECT_INSTANCE_BEGIN‘ bigint(20) unsigned NOT NULL,
‘NESTING_EVENT_ID‘ bigint(20) unsigned DEFAULT NULL,
‘NESTING_EVENT_TYPE‘ enum(’TRANSACTION’,’STATEMENT’,’STAGE’,’WAIT’) DEFAULT NULL,
EVENT-SPECIFIC-FIELDS
) ENGINE=PERFORMANCE_SCHEMA
Stores the Data
10
![Page 33: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/33.jpg)
• SQL• Same basic structure for events * tables• Others are unified whenever possible
Returns the Data
11
![Page 34: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/34.jpg)
How Does Performance Schema Works?
12
![Page 35: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/35.jpg)
Instruments: Get the Data
13
![Page 36: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/36.jpg)
• Every instrumented call is wrapped intotwo Performance Schema calls
• For rarely used calls - no performance impact• Memory allocation• Statement-related data
• Significant impact for often used
Performance Note
14
![Page 37: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/37.jpg)
• Every instrumented call is wrapped intotwo Performance Schema calls
• For rarely used calls - no performance impact• Memory allocation• Statement-related data
• Significant impact for often used
Performance Note
14
![Page 38: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/38.jpg)
• Every instrumented call is wrapped intotwo Performance Schema calls
• For rarely used calls - no performance impact• Memory allocation• Statement-related data
• Significant impact for often used
Performance Note
14
![Page 39: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/39.jpg)
• Table of 2048 rows• select * from joinit;• Test preparation
mysql> truncate performance_schema.events_waits_history_long;
Query OK, 0 rows affected (0.02 sec)
-- OR
mysql> truncate performance_schema.events_statements_history_long;
Query OK, 0 rows affected (0.02 sec)
Performance Example
15
![Page 40: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/40.jpg)
• Table of 2048 rows• select * from joinit;• The test
mysql> \P sha1sum
PAGER set to ’sha1sum’
mysql> select * from joinit;
7f188e9c9fd628613eab85366f0729604b8b5e3e -
2048 rows in set (0.09 sec)
Performance Example
15
![Page 41: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/41.jpg)
• Table of 2048 rows• select * from joinit;• events waits*
mysql> select event_name, count(*) from performance_schema.events_waits_history_long
-> where thread_id=42 group by EVENT_NAME order by ‘count(*)‘ desc;
+---------------------------------------------+----------+
| event_name | count(*) |
+---------------------------------------------+----------+
| wait/synch/mutex/innodb/rw_lock_debug_mutex | 322 |
| wait/synch/sxlock/innodb/hash_table_locks | 12 |
| wait/synch/mutex/innodb/trx_mutex | 12 |
| wait/synch/mutex/innodb/fil_system_mutex | 12 |
| wait/synch/mutex/sql/THD::LOCK_query_plan | 4 |
| wait/synch/mutex/sql/THD::LOCK_thd_data | 3 |
| wait/synch/mutex/sql/LOCK_table_cache | 2 |
...
Performance Example
15
![Page 42: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/42.jpg)
• Table of 2048 rows• select * from joinit;• events statements*
mysql> select event_name, count(*) from performance_schema.events_statements_history_long
-> where thread_id=42 group by EVENT_NAME;
+----------------------+----------+
| event_name | count(*) |
+----------------------+----------+
| statement/sql/select | 1 |
+----------------------+----------+
1 row in set (0.02 sec)
Performance Example
15
![Page 43: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/43.jpg)
• Table of 2048 rows• select * from joinit;• Depends on the instrument!
Performance Example
15
![Page 44: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/44.jpg)
Consumers: Store the Data
16
![Page 45: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/45.jpg)
• Must be supported by the component• Collects data only after they are enabled• Never frees allocated memory
Performance Schema Limitations
17
![Page 46: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/46.jpg)
How to Tune?
![Page 47: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/47.jpg)
• Defaults• ON
5.7: Only global, thread, statements and transactionsinstrumentation
8.0: Memory and MDL
• All other instruments/consumers disabled
Performance Schema Defaults
19
![Page 48: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/48.jpg)
• Defaults• ON
5.7: Only global, thread, statements and transactionsinstrumentation
8.0: Memory and MDL• All other instruments/consumers disabled
Performance Schema Defaults
19
![Page 49: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/49.jpg)
• Configuration options• setup * tables
Two Places
20
![Page 50: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/50.jpg)
• performance schema = ON
• Instruments and consumers, enabled at startup• Pattern
--performance-schema-consumer-consumer name=value
--performance-schema-instrument=’instrument name=value’
• Examples--performance-schema-consumer-events transactions history long=ON
--performance-schema-instrument=’%=OFF’
--performance-schema-instrument=’memory/innodb/%=ON’
--performance-schema-instrument=’wait/lock/metadata/sql/mdl=ON’
Startup Configuration
21
![Page 51: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/51.jpg)
• performance schema = ON• Limits of table size and number of
instrumented instances• Not dynamic!• Examples
performance schema accounts size
performance schema max thread instances
performance schema max thread classes
• Instruments and consumers, enabled at startup• Pattern
--performance-schema-consumer-consumer name=value
--performance-schema-instrument=’instrument name=value’
• Examples--performance-schema-consumer-events transactions history long=ON
--performance-schema-instrument=’%=OFF’
--performance-schema-instrument=’memory/innodb/%=ON’
--performance-schema-instrument=’wait/lock/metadata/sql/mdl=ON’
Startup Configuration
21
![Page 52: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/52.jpg)
• performance schema = ON• Instruments and consumers, enabled at startup
• Pattern--performance-schema-consumer-consumer name=value
--performance-schema-instrument=’instrument name=value’
• Examples--performance-schema-consumer-events transactions history long=ON
--performance-schema-instrument=’%=OFF’
--performance-schema-instrument=’memory/innodb/%=ON’
--performance-schema-instrument=’wait/lock/metadata/sql/mdl=ON’
Startup Configuration
21
![Page 53: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/53.jpg)
• setup * tables
Runtime Configuration
22
![Page 54: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/54.jpg)
• setup * tables• Actors: user connections
mysql> select * from setup_actors;
+------+------+------+---------+---------+
| HOST | USER | ROLE | ENABLED | HISTORY |
+------+------+------+---------+---------+
| % | pfs | % | YES | YES |
| % | root | % | NO | NO |
| % | pfs2 | % | YES | NO |
+------+------+------+---------+---------+
3 rows in set (0.00 sec)
Runtime Configuration
22
![Page 55: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/55.jpg)
• setup * tables• Threads: thread classes
mysql> select * from setup_threads\G
...
*************************** 5. row ***************************
NAME: thread/sql/one_connection
ENABLED: YES
HISTORY: YES
PROPERTIES: user
VOLATILITY: 0
DOCUMENTATION: NULL
...
Runtime Configuration
22
![Page 56: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/56.jpg)
• setup * tables• Threads: thread classes
...
*************************** 42. row ***************************
NAME: thread/sql/slave_io
ENABLED: YES
HISTORY: YES
PROPERTIES: singleton
VOLATILITY: 0
DOCUMENTATION: NULL
...
Runtime Configuration
22
![Page 57: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/57.jpg)
• setup * tables• Objects
• Types of events, their names and schemamysql> select * from setup_objects;
+-------------+---------------+-------------+---------+-------+
| OBJECT_TYPE | OBJECT_SCHEMA | OBJECT_NAME | ENABLED | TIMED |
+-------------+---------------+-------------+---------+-------+
| EVENT | % | % | NO | NO |
| FUNCTION | % | % | NO | NO |
| PROCEDURE | % | % | NO | NO |
| TABLE | % | % | NO | NO |
| TRIGGER | % | % | NO | NO |
| EVENT | my_schema | % | YES | YES |
| TABLE | my_schema | % | YES | YES |
+-------------+---------------+-------------+---------+-------+
7 rows in set (0.00 sec)
Runtime Configuration
22
![Page 58: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/58.jpg)
• setup * tables• Consumers: what to store
mysql> select * from setup_consumers;
+----------------------------------+---------+
| NAME | ENABLED |
+----------------------------------+---------+
| events_stages_current | YES |
| events_stages_history | YES |
| events_stages_history_long | NO |
| events_statements_current | YES |
| events_statements_history | YES |
| events_statements_history_long | NO |
| events_transactions_current | YES |
| events_transactions_history | YES |
| events_transactions_history_long | NO |
...
Runtime Configuration
22
![Page 59: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/59.jpg)
• setup * tables• Instruments: what to instrument
mysql> select NAME, ENABLED, TIMED, PROPERTIES, VOLATILITY from setup_instruments
-> order by rand() limit 3;
+-------------------------------------------+---------+-------+------------+------------+
| NAME | ENABLED | TIMED | PROPERTIES | VOLATILITY |
+-------------------------------------------+---------+-------+------------+------------+
| wait/synch/mutex/pfs/LOCK_pfs_share_list | NO | NO | singleton | 1 |
| memory/csv/Transparent_file | YES | NULL | | 0 |
| wait/synch/mutex/sql/THD::LOCK_thd_data | YES | YES | | 5 |
+-------------------------------------------+---------+-------+------------+------------+
3 rows in set (0.02 sec)
Runtime Configuration
22
![Page 60: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/60.jpg)
How to Ease?
![Page 61: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/61.jpg)
• Views on Performance Schema tables• Stored routines
• Easier configuration• Shortcuts to typical use cases• 5.7+: Installed by default• Before: github.com/mysql/mysql-sys
sys Schema
24
![Page 62: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/62.jpg)
• Views on Performance Schema tables• Stored routines• Easier configuration• Shortcuts to typical use cases
• 5.7+: Installed by default• Before: github.com/mysql/mysql-sys
sys Schema
24
![Page 63: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/63.jpg)
• Views on Performance Schema tables• Stored routines• Easier configuration• Shortcuts to typical use cases• 5.7+: Installed by default• Before: github.com/mysql/mysql-sys
sys Schema
24
![Page 64: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/64.jpg)
• statement analysis• statements with full table scans• statements with runtimes in 95th percentile• statements with sorting• statements with temp tables• statements with errors or warnings• memory by thread by current bytes• More
Nicer Views
25
![Page 65: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/65.jpg)
• Performance Schema• SELECTTHREAD ID, DIGEST, ROWS SENT RS,ROWS EXAMINED, CREATED TMP TABLES,NO INDEX USED, NO GOOD INDEX USEDFROMperformance schema.events statements historyWHERE NO INDEX USED=1 ORNO GOOD INDEX USED=1
• sys schema• select * fromsys.statements with full table scans
Compare!
26
![Page 66: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/66.jpg)
• sys schema• select * fromsys.statements with full table scans
Compare!
26
![Page 67: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/67.jpg)
• Enablecall sys.ps_setup_enable_consumer(YOUR_CONSUMER);
call sys.ps_setup_enable_instrument(YOUR_INSTRUMENT);
Easier configuration
27
![Page 68: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/68.jpg)
• Disablecall sys.ps_setup_disable_consumer(YOUR_CONSUMER);
call sys.ps_setup_disable_instrument(YOUR_INSTRUMENT);
Easier configuration
27
![Page 69: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/69.jpg)
• Examinecall sys.ps_setup_show_disabled_consumers();
call sys.ps_setup_show_disabled_instruments();
call sys.ps_setup_show_enabled_consumers();
call sys.ps_setup_show_enabled_instruments();
Easier configuration
27
![Page 70: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/70.jpg)
Performance Schema Dashboard
23:05:47
PMM
28
![Page 71: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/71.jpg)
Performance Schema Dashboard
23:32:15
PMM
28
![Page 72: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/72.jpg)
Performance Schema Dashboard
22:27:15
PMM
28
![Page 73: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/73.jpg)
Performance Schema Dashboard
18:08:15
PMM
28
![Page 74: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/74.jpg)
Blog of MySQL developers team
Mark Leith: author of sys schema
Official reference manual
Webinar ”Performance Schema for MySQLTroubleshooting”PMM Demo
More information
29
![Page 75: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/75.jpg)
www.slideshare.net/SvetaSmirnova
twitter.com/svetsmirnova
github.com/svetasmirnova
Thank you!
30
![Page 76: MySQL Performance Schema - percona.com · Metadata Always existed SHOW commands Information Schema Runtime Was not perfect Status variables Extensions for Information Schema History](https://reader030.fdocuments.net/reader030/viewer/2022020109/5dd0c0a9d6be591ccb628677/html5/thumbnails/76.jpg)
DATABASE PERFORMANCEMATTERS