Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic...
Transcript of Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic...
![Page 1: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/1.jpg)
Introduction to troubleshootingBasic techniques
Sveta SmirnovaPrincipal Support Engineer
March, 10, 2016
![Page 2: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/2.jpg)
Table of Contents
•Introduction•How to find problematic query•Solving issues
Syntax errorsLogic errorsHidden updatesLost connection
•Next webinars
2 www.percona.com
![Page 3: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/3.jpg)
Introduction
3 www.percona.com
![Page 4: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/4.jpg)
What is basic troubleshooting issue?
• You run a query• Receive results which you don’t expect
• Wrong dataset• No answer from server• Anything else
• These results are consistent• Repeatable each time when you run the same query
4 www.percona.com
![Page 5: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/5.jpg)
Typical workflow with the basic issue
• Find the query• Identify class of the problem
• Syntax error• Logical error• Data, inserted earlier• No answer from the server
• Fix
5 www.percona.com
![Page 6: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/6.jpg)
MySQL architecture
Connectors: C, JDBC, ODBC, Python, ...
Connection Pool: Authentication, Caches
SQL interface
Parser
Optimizer
Caches and Buffers:Global
Engine-specific
Storage engines: InnoDB, TokuDB, ...
File system: Data, Index, logs, other files
• Base• Installation layout• Log files
• Connectors• Clients, APIs
• Optimizer• Cache and buffers• Storage engines• Management
6 www.percona.com
![Page 7: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/7.jpg)
Typical installation layout
• datadir• Schema directory
• Table and trigger files: *.frm, *.ibd, *.MYD,*.par, etc.
• Schema• ...• InnoDB shared tablespace• Log files
• InnoDB redo log files• Binary, relay log files• Error log• Slow query log• General query log
• Configurable• You can setup
custom path foreach component
• Including custompaths for tables
7 www.percona.com
![Page 8: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/8.jpg)
Plugins
• Storage engines
• Authentication
• Audit
• Query rewrite
• More
8 www.percona.com
![Page 9: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/9.jpg)
Storage engines
• Own data
• Own index format
• Own locking model
• Own diagnostic
• Own log files
• CHECK TABLE
9 www.percona.com
![Page 10: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/10.jpg)
How to find problematic query
10 www.percona.com
![Page 11: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/11.jpg)
Where is the query?
• Log files of your application
• General query log
• performance schema.events statements * tables
• Scriptable Proxies
• Audit Plugin
• User complains
11 www.percona.com
![Page 12: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/12.jpg)
General Query Log
mysql> set global log_output=’table’, global general_log=1;
Query OK, 0 rows affected (0,00 sec)
mysql> select * from mysql.general_log where argument like ’%Gender%’
-> and argument not like ’%general_log%’\G
****************** 1. row ******************
event_time: 2016-02-25 03:35:39.903867
user_host: root[root] @ localhost[127.0.0.1]
thread_id: 1
server_id: 1
command_type: Query
argument: select * from employees where first_name = ’Gender’
1 row in set (0.00 sec)
12 www.percona.com
![Page 13: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/13.jpg)
performance schema.events statements *
mysql> select sql_text from performance_schema.events_statements_history_long
-> where sql_text like ’%gender%’ and current_schema=’employees’;
+-----------------------------------------------------------------+
| sql_text |
+-----------------------------------------------------------------+
| select * from employees where first_name = ’Gender’ |
| select * from performance_schema.events_statements_history_long
where sql_text like ’%gender%’ and current_schema=’employees’ |
+-----------------------------------------------------------------+
2 rows in set (0.00 sec)
13 www.percona.com
![Page 14: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/14.jpg)
MySQL Proxy: all queries, finished with error
function read_query( packet )
proxy.queries:append(1, packet )
return proxy.PROXY_SEND_QUERY
end
function read_query_result(inj)
if inj.resultset.query_status == proxy.MYSQLD_PACKET_ERR then
print(os.date("%d%m%y %H:%M:%S") .. "\t"
.. "Query \"" .. inj.query:sub(2) .. "\" finished with error")
end
end
14 www.percona.com
![Page 15: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/15.jpg)
Audit Logmysql> INSTALL PLUGIN audit_log SONAME ’audit_log.so’;
Query OK, 0 rows affected (0.09 sec)
mysql> \! cat /home/sveta/build/ps-5.6/mysql-test/var/mysqld.1/data/audit.log
<AUDIT_RECORD
NAME="Query"
RECORD="3_2016-02-26T11:03:39"
TIMESTAMP="2016-02-26T11:03:57 UTC"
COMMAND_CLASS="select"
CONNECTION_ID="1"
STATUS="0" - Check this field for non-0 status!
SQLTEXT="select * from mysql.user where user like ’%sveta%’"
USER="root[root] @ localhost [127.0.0.1]"
HOST="localhost" OS_USER="" IP="127.0.0.1" />
15 www.percona.com
![Page 16: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/16.jpg)
Solving issues
16 www.percona.com
![Page 17: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/17.jpg)
Query found: what to do next?• Check kind of the error
• Syntax• Can happen even to experienced users if complicated query created by an application
• Logic: stored data is correct• SELECT returns wrong result• DML updates wrong set of rows
• Hidden update: wrong data was already in the DB• First noted after SELECT returned not expected result• Source of the problem was an update which happened few steps ago
• Connection lost• Wrong option usage• Server crash
• Check what affects the query - Future webinars• Fix!
17 www.percona.com
![Page 18: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/18.jpg)
Query found: what to do next?
• Check kind of the error• Check what affects the query - Future webinars
• Storage engine• Configuration options• Privileges• Replication
• Fix!
17 www.percona.com
![Page 19: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/19.jpg)
Syntax errors
18 www.percona.com
![Page 20: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/20.jpg)
Syntax error
• select * fro t1 where f1 in (1,2,1);
• select max (f1) from t1;
• select * from t1 where "f1"=1;
• return "’" . str_pad(md5(rand()),
rand(1, $column[’character_maximum_length’]),
md5(rand()), STR_PAD_BOTH) . "’";
19 www.percona.com
![Page 21: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/21.jpg)
Syntax error
• select * fro t1 where f1 in (1,2,1);
• select max (f1) from t1;
• Correct syntax: max(f1) – no space• Works with SQL Mode IGNORE SPACE only
• select * from t1 where "f1"=1;
• return "’" . str_pad(md5(rand()),
rand(1, $column[’character_maximum_length’]),
md5(rand()), STR_PAD_BOTH) . "’";
19 www.percona.com
![Page 22: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/22.jpg)
Syntax error
• select * fro t1 where f1 in (1,2,1);
• select max (f1) from t1;
• select * from t1 where "f1"=1;
• – Works with SQL Mode ANSI QUOTES only
• return "’" . str_pad(md5(rand()),
rand(1, $column[’character_maximum_length’]),
md5(rand()), STR_PAD_BOTH) . "’";
19 www.percona.com
![Page 23: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/23.jpg)
Syntax error
• select * fro t1 where f1 in (1,2,1);
• select max (f1) from t1;
• select * from t1 where "f1"=1;
• return "’" . str_pad(md5(rand()),
rand(1, $column[’character_maximum_length’]),
md5(rand()), STR_PAD_BOTH) . "’";
• In the application
19 www.percona.com
![Page 24: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/24.jpg)
Error message: always check!
mysql> select * fro t1 where f1 in (1,2,1);
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual
that corresponds to your MySQL server version for the right syntax to use
near ’fro t1 where f1 in (1,2,1)’ at line 1
mysql> select max (f1) from t1;
ERROR 1630 (42000): FUNCTION test.max does not exist.
Check the Function Name Parsing and Resolution’ section
in the Reference Manual
20 www.percona.com
![Page 25: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/25.jpg)
Warnings are important
mysql> select * from t1 where "f1"=1;
Empty set, 1 warning (0.05 sec)
mysql> show warnings;
+---------+------+--------------------------------------+
| Level | Code | Message |
+---------+------+--------------------------------------+
| Warning | 1292 |Truncated incorrect DOUBLE value: ’f1’|
+---------+------+--------------------------------------+
1 row in set (0.00 sec)
21 www.percona.com
![Page 26: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/26.jpg)
Errors information in the application
• Error information• C API, Connector/C, PHP
• mysql errno• mysql error
• Warnings• C API, Connector/C, PHP
• mysql info• mysql sqlstate• mysql warning count
22 www.percona.com
![Page 27: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/27.jpg)
Errors information in the application
• Error information• JDBC, Connector/C++
• SQLException
• Warnings• JDBC, Connector/C++
• SQLWarning
22 www.percona.com
![Page 28: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/28.jpg)
Errors information in the application
• Error information• ODBC
• SQLGetDiagField
• SQLGetDiagRec
• Warnings• ODBC
• SQLGetDiagField
• SQLGetDiagRec
22 www.percona.com
![Page 29: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/29.jpg)
Errors information in the application
• Error information• Connector/Python
• errors.*Error
• Warnings• Connector/Python
• raise on warnings option
• errors.Warning is not used!
22 www.percona.com
![Page 30: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/30.jpg)
Perror: what does the error mean?
sveta@thinkie> perror 1292 - SQL error
MySQL error code 1292 (ER_TRUNCATED_WRONG_VALUE):
Truncated incorrect %-.32s value: ’%-.128s’
sveta@thinkie> perror 2 - OS error
OS error code 2: No such file or directory
sveta@thinkie> perror 150 - Engine error
MySQL error code 150: Foreign key constraint is incorrectly formed
23 www.percona.com
![Page 31: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/31.jpg)
Error handling in stored routines
• GET DIAGNOSTICS
• GET DIAGNOSTICS rows = ROW COUNT,conditions = NUMBER;
• GET DIAGNOSTICS CONDITION 1 code =RETURNED SQLSTATE, msg = MESSAGE TEXT;
• SIGNAL/RESIGNAL
• Can be used for custom errors
24 www.percona.com
![Page 32: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/32.jpg)
Logic errors
25 www.percona.com
![Page 33: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/33.jpg)
Logic errors• You expect query to work correctly
• But it does not• This can be caused by
• Error in how you interpret the query should work• MySQL bug
• How to start fixing such a query?• Simplyfy it• Consider each of parts which complicate the query• Find out what it does• Fix
• Hardest to teach• Requires practice
26 www.percona.com
![Page 34: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/34.jpg)
Example 1: movies• The issuemysql> select title from movies where title=8;
+----------------+
| title |
+----------------+
| 8 |
| 8 1/2 |
| 8 Simple Rules |
| 8 Seconds |
+----------------+
4 rows in set, 3 warnings (0.00 sec)
• Investigation• Test• Fix
27 www.percona.com
![Page 35: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/35.jpg)
Example 1: movies• The issue• Investigationmysql> show warnings;
+---------+------+---------------------------------------------------+
| Level | Code | Message |
| Warning | 1292 | Truncated incorrect DOUBLE value: ’8’ |
| Warning | 1292 | Truncated incorrect DOUBLE value: ’8 Simple Rules’|
| Warning | 1292 | Truncated incorrect DOUBLE value: ’8 Seconds’ |
+---------+------+---------------------------------------------------+
3 rows in set (0.00 sec)
• Test• Fix
27 www.percona.com
![Page 36: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/36.jpg)
Example 1: movies• The issue• Investigation• Testmysql> select 8 = ’8 Simple Rules’, ’8’ = ’8 Simple Rules’;
+----------------------+------------------------+
| 8 = ’8 Simple Rules’ | ’8’ = ’8 Simple Rules’ |
+----------------------+------------------------+
| 1 | 0 |
+----------------------+------------------------+
1 row in set, 1 warning (0.00 sec)
• Fix
27 www.percona.com
![Page 37: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/37.jpg)
Example 1: movies• The issue• Investigation• Test• Fixmysql> select title from movies where title=’8’;
+-------+
| title |
+-------+
| 8 |
+-------+
1 row in set (0.00 sec)
27 www.percona.com
![Page 38: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/38.jpg)
Example 2: too many leaders• The issue
mysql> select count(emp_no) from employees where hire_date in
-> (select hire_date from titles where title=’Technique Leader’)
-> union select count(*) from employees;
+---------------+
| count(emp_no) |
+---------------+
| 300024 |
| 300024 |
+---------------+
1 row in set (3 min 7.20 sec)
• More stranges• Investigation• Why so?
28 www.percona.com
![Page 39: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/39.jpg)
Example 2: too many leaders• The issue• More strangesmysql> select count(emp_no) from employees where hire_date in
-> (select hire_date from titles where title=’Senior engineer’);
+---------------+
| count(emp_no) |
+---------------+
| 300024 |
+---------------+
1 row in set (17 min 32.49 sec)
• Investigation• Why so?
28 www.percona.com
![Page 40: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/40.jpg)
Example 2: too many leaders• The issue• More stranges• Investigationmysql> explain extended select count(emp_no) from employees
-> where hire_date in
-> (select hire_date from titles where title=’Technique Leader’)\G
...
mysql> show warnings\G
************************ 1. row ************************
Level: Note
Code: 1276
Message: Field or reference ’employees.employees.hire_date’ of SELECT #2
was resolved in SELECT #1
...
• Why so?
28 www.percona.com
![Page 41: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/41.jpg)
Example 2: too many leaders• The issue• More stranges• Investigation• Why so?mysql> desc titles;
+-----------+-------------+------+-----+-------+-------+
| Field | Type | Null | Key |Default| Extra |
+-----------+-------------+------+-----+-------+-------+
| emp_no | int(11) | NO | PRI | NULL | |
| title | varchar(50) | NO | PRI | NULL | |
| from_date | date | NO | PRI | NULL | |
| to_date | date | YES | | NULL | |
+-----------+-------------+------+-----+-------+-------+
4 rows in set (0.11 sec)28 www.percona.com
![Page 42: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/42.jpg)
Example 3: MySQL bug
• Preparationmysql> alter table titles add index(title, emp_no);
Query OK, 0 rows affected (8.39 sec)
Records: 0 Duplicates: 0 Warnings: 0
• The issue• Investigation• Test and Fix• Based onhttps://bugs.mysql.com/bug.php?id=70359
29 www.percona.com
![Page 43: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/43.jpg)
Example 3: MySQL bug• Preparation• The issuemysql> select max(emp_no), title from titles
-> where title = ’Technique Leader’
-> or emp_no = 10005
-> group by title;
+-------------+------------------+
| max(emp_no) | title |
+-------------+------------------+
| 499942 | Technique Leader |
+-------------+------------------+
1 row in set (0.00 sec)
• Investigation• Test and Fix• Based onhttps://bugs.mysql.com/bug.php?id=70359
29 www.percona.com
![Page 44: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/44.jpg)
Example 3: MySQL bug• Preparation• The issue
But!mysql> select title from titles where emp_no = 10005;
+--------------+
| title |
+--------------+
| Senior Staff |
| Staff |
+--------------+
2 rows in set (0.00 sec)
• Investigation• Test and Fix• Based onhttps://bugs.mysql.com/bug.php?id=70359
29 www.percona.com
![Page 45: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/45.jpg)
Example 3: MySQL bug• Preparation• The issue• Investigationmysql> explain select max(emp_no), title from titles where
-> title = ’Technique Leader’ or emp_no = 10005 group by title\G
*************************** 1. row ***************************
...
possible_keys: PRIMARY,emp_no,title
key: title
...
Extra: Using where; Using index for group-by
1 row in set, 1 warning (0.00 sec)
• Test and Fix• Based onhttps://bugs.mysql.com/bug.php?id=70359
29 www.percona.com
![Page 46: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/46.jpg)
Example 3: MySQL bug• Preparation• The issue• Investigation• Test and Fixmysql> select max(emp_no), title from titles ignore index(title) where
-> title = ’Technique Leader’ or emp_no = 10005 group by title;
+-------------+------------------+
| max(emp_no) | title |
+-------------+------------------+
| 10005 | Senior Staff |
| 10005 | Staff |
| 499942 | Technique Leader |
+-------------+------------------+
3 rows in set (0.54 sec)
• Based onhttps://bugs.mysql.com/bug.php?id=70359
29 www.percona.com
![Page 47: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/47.jpg)
Example 3: MySQL bug
• Preparation
• The issue
• Investigation
• Test and Fix
• Based onhttps://bugs.mysql.com/bug.php?id=70359
29 www.percona.com
![Page 48: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/48.jpg)
Hidden updates
30 www.percona.com
![Page 49: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/49.jpg)
What do I call ”hidden update”?• Query behaves correctly, but returns wrong data• Reason can be
• Wrong update, happened earlier• No change when update expected
• How to find what really happened?• Logs of your application• Binary log - May not have variables affecting resut!• General query log, p s.events statements * tables• Scriptable Proxies
• Looks similar to wrong syntax issue?• We need to watch change time now
31 www.percona.com
![Page 50: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/50.jpg)
Hidden update exmaple• The issuemysql> select count(*) from titles where title = ’Senior Engineer’;
+----------+
| count(*) |
+----------+
| 0 |
+----------+
1 row in set (0.00 sec)
• Investigation• Possible resolutions
• In our case we can simply runupdate titles set title = ’Senior Engineer’
where title = ’Junior Engineer’;
• In more general case• Restore table from backup• Apply all binary logs, created after backup was made
32 www.percona.com
![Page 51: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/51.jpg)
Hidden update exmaple• The issue• Investigation$mysqlbinlog -v master-bin.000001
# at 308
#160305 23:25:16 server id 1 end_log_pos 467 CRC32 0xeef31919 Query
thread_id=3 exec_time=69 error_code=0
use ‘employees‘/*!*/;
SET TIMESTAMP=1457213116/*!*/;
update titles set title = ’Junior Engineer’
where title = ’Senior Engineer’
/*!*/;
# at 467
...
• Possible resolutions• In our case we can simply runupdate titles set title = ’Senior Engineer’
where title = ’Junior Engineer’;
• In more general case• Restore table from backup• Apply all binary logs, created after backup was made
32 www.percona.com
![Page 52: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/52.jpg)
Hidden update exmaple
• The issue• Investigation• Possible resolutions
• In our case we can simply runupdate titles set title = ’Senior Engineer’
where title = ’Junior Engineer’;
• In more general case• Restore table from backup• Apply all binary logs, created after backup was made
32 www.percona.com
![Page 53: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/53.jpg)
Lost connection
33 www.percona.com
![Page 54: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/54.jpg)
Two main reasons
• Server crash: new connection cannot be established• Check error log file• Will be discussed in future webinar
• You can reconnect• Network issues and timeouts• Effect of max allowed packet
• You could see ”ERROR 1064 (42000) at line XXX: You have an error in your SQLsyntax;” instead of proper ”ERROR 1153 (08S01) at line XXX: Got a packet bigger than’max allowed packet’ bytes”
• Fixed in latest versions
34 www.percona.com
![Page 55: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/55.jpg)
Timeout symptoms
• connect timeout
• Controls initial communication• Check status variable Aborted connects
• interactive timeout• wait timeout• net read timeout• net write timeout
35 www.percona.com
![Page 56: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/56.jpg)
Timeout symptoms
• connect timeout• interactive timeout• wait timeout
• How much time to wait activity from the client beforeaborting connection
• How much time can a thread be in ”Sleep” state• Check status variable Aborted clients
• net read timeout• net write timeout
35 www.percona.com
![Page 57: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/57.jpg)
Timeout symptoms
• connect timeout• interactive timeout• wait timeout• net read timeout• net write timeout
• How long to wait communication from a client whichreads/writes result
35 www.percona.com
![Page 58: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/58.jpg)
Timeout issues: how to fix
• Most obvious and dangerous way: increase timeout• Can just hide the issue, but not fix it
• Better way• Temporarily increase timeout• Check if it solves the issue or make it happen more
seldom• Find out the reason why the issue happened in the
first time• Check and fix network
36 www.percona.com
![Page 59: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/59.jpg)
Next webinars
37 www.percona.com
![Page 60: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/60.jpg)
Performance
• Introduction to troubleshooting performance• end of March
• Slow queries• everything about EXPLAIN, query execution, etc.
• Locking issues• Storage engines• Options• Hardware
38 www.percona.com
![Page 61: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/61.jpg)
Production
• Privileges
• Replication
• Crashes
39 www.percona.com
![Page 62: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/62.jpg)
Effective testing
• Tools
• Workflow
• Advanced testing techniques
• Troubleshooting and monitoring best practices
• Last webinar, end of the year
40 www.percona.com
![Page 63: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/63.jpg)
Summary
41 www.percona.com
![Page 64: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/64.jpg)
Summary
• Answers from the server are important
• Always check error messages and warnings
• Be ready to use log files
42 www.percona.com
![Page 65: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/65.jpg)
More Informaiton
• MySQL Server Logs• Percona Audit Plugin• Writing Audit Plugins• MySQL Proxy
43 www.percona.com
![Page 66: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/66.jpg)
Place for your questions
???
44 www.percona.com
![Page 67: Introduction to troubleshooting - Basic techniques · Introduction to troubleshooting Basic techniques Sveta Smirnova Principal Support Engineer March, 10, 2016](https://reader033.fdocuments.net/reader033/viewer/2022051607/6035bd19f95a8b2d2a40ada9/html5/thumbnails/67.jpg)
Thank you!
http://www.slideshare.net/SvetaSmirnova
https://twitter.com/svetsmirnova
45 www.percona.com