Undelete (and more) rows from the binary log
-
Upload
frederic-descamps -
Category
Engineering
-
view
132 -
download
0
Transcript of Undelete (and more) rows from the binary log
Hacking Session: undelete (and more) rows from the binary log
Percona Live Europe Amsterdam 2015 Frédéric -lefred- Descamps
Who am I ?● Frédéric Descamps● @lefred - follow me on twitter if you want ;-)
● http://about.me/lefred● Working for Percona since 2011● Managing MySQL since 3.23● devops believer
Why ?
Because of Scott Noyes' blog post : http://thenoyes.com/littlenoise/?p=307
● Because it's faster than point-in-time recovery
● Because I didn't remember the awk script in Scott's blog post
Requirements● Have the binary logs in ROW format● Have binlog_row_image set to FULL● Have access to the binary logs● Have access to MySQL● Have access to the MySQL source code [not mandatory]● Have a brain and being able to use it ;-)
Percona Live Europe Amsterdam 2015
Point-in-Time Recovery
Usually when we need to «undo» even one single statement, we need to perform a point-in-time recovery● restore the last backup (full or several incrementals)● then replay some binary logs (and it can be a bunch of them!)● this is a very slow operation and while the binary logs are
replayed, usually everything needs to be stopped.
Percona Live Europe Amsterdam 2015
Let's hack together a quicker process
Ready ?
Let's delete a row
mysql> select @@version, @@version_comment;+------------+------------------------------+| @@version | @@version_comment |+------------+------------------------------+| 5.6.22-log | MySQL Community Server (GPL) |+------------+------------------------------+
mysql> select * from fosdem.community_dinner;+----+--------+--------+-------+| id | name | pizzas | beers |+----+--------+--------+-------+| 1 | dim0 | 99 | 99 || 2 | Liz | 2 | 24 || 3 | Kenny | 12 | 32 || 4 | lefred | 10 | 10 |+----+--------+--------+-------+4 rows in set (0.00 sec)
Percona Live Europe Amsterdam 2015
Let's delete a row (2)
mysql> delete from community_dinner where id=2;Query OK, 1 row affected (0.05 sec)
mysql> select * from fosdem.community_dinner;+----+--------+--------+-------+| id | name | pizzas | beers |+----+--------+--------+-------+| 1 | dim0 | 99 | 99 || 3 | Kenny | 12 | 32 || 4 | lefred | 10 | 10 |+----+--------+--------+-------+3 rows in set (0.00 sec)
Percona Live Europe Amsterdam 2015
Let's find the event in the binary logmysql> show master status\G*************************** 1. row *************************** File: mysql-bin.000018 Position: 907 Binlog_Do_DB: Binlog_Ignore_DB: Executed_Gtid_Set: 1 row in set (0.00 sec)
mysql> show binlog events in 'mysql-bin.000018';+------------------+-----+-------------+-----------+-------------+---------------------------+| Log_name | Pos | Event_type | Server_id | End_log_pos | Info |+------------------+-----+-------------+-----------+------------+----------------------------+| mysql-bin.000018 | 4 | Format_desc | 1 | 120 | Server ... || mysql-bin.000018 | 120 | Query | 1 | 220 | create ... || mysql-bin.000018 | 220 | Query | 1 | 404 | use `fosdem`; || mysql-bin.000018 | 404 | Query | 1 | 478 | BEGIN || mysql-bin.000018 | 478 | Table_map | 1 | 544 | table_id: 76 (fosdem.commu|| mysql-bin.000018 | 544 | Write_rows | 1 | 653 | table_id: 76 flags: STMT_E|| mysql-bin.000018 | 653 | Xid | 1 | 684 | COMMIT /* xid=32 */ || mysql-bin.000018 | 684 | Query | 1 | 758 | BEGIN | | mysql-bin.000018 | 758 | Table_map | 1 | 824 | table_id: 76 (fosdem.commu|| mysql-bin.000018 | 824 | Delete_rows | 1 | 876 | table_id: 76 flags: STMT_E|| mysql-bin.000018 | 876 | Xid | 1 | 907 | COMMIT /* xid=34 */ |+------------------+-----+-------------+-----------+------------+====------------------------+
Percona Live Europe Amsterdam 2015
Let's find the event in the binary logmysql> show master status\G*************************** 1. row *************************** File: mysql-bin.000018 Position: 907 Binlog_Do_DB: Binlog_Ignore_DB: Executed_Gtid_Set: 1 row in set (0.00 sec)
mysql> show binlog events in 'mysql-bin.000018';+------------------+-----+-------------+-----------+-------------+---------------------------+| Log_name | Pos | Event_type | Server_id | End_log_pos | Info |+------------------+-----+-------------+-----------+------------+----------------------------+| mysql-bin.000018 | 4 | Format_desc | 1 | 120 | Server ... || mysql-bin.000018 | 120 | Query | 1 | 220 | create ... || mysql-bin.000018 | 220 | Query | 1 | 404 | use `fosdem`; || mysql-bin.000018 | 404 | Query | 1 | 478 | BEGIN || mysql-bin.000018 | 478 | Table_map | 1 | 544 | table_id: 76 (fosdem.commu|| mysql-bin.000018 | 544 | Write_rows | 1 | 653 | table_id: 76 flags: STMT_E|| mysql-bin.000018 | 653 | Xid | 1 | 684 | COMMIT /* xid=32 */ || mysql-bin.000018 | 684 | Query | 1 | 758 | BEGIN | | mysql-bin.000018 | 758 | Table_map | 1 | 824 | table_id: 76 (fosdem.commu|| mysql-bin.000018 | 824 | Delete_rows | 1 | 876 | table_id: 76 flags: STMT_E|| mysql-bin.000018 | 876 | Xid | 1 | 907 | COMMIT /* xid=34 */ |+------------------+-----+-------------+-----------+------------+====------------------------+
The event we are looking for
Percona Live Europe Amsterdam 2015
binary log's event content # mysqlbinlog mysql-bin.000018 -j 824 --stop-position=876/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/;/*!40019 SET @@session.max_insert_delayed_threads=0*/;/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;DELIMITER /*!*/;# at 4#141221 21:20:55 server id 1 end_log_pos 120 CRC32 0xff8913a3 Start: binlog v 4, server v 5.6.22-log created 141221 21:20:55 at startup# Warning: this binlog is either in use or was not closed properly.ROLLBACK/*!*/;BINLOG 'pyuXVA8BAAAAdAAAAHgAAAABAAQANS42LjIyLWxvZwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACnK5dUEzgNAAgAEgAEBAQEEgAAXAAEGggAAAAICAgCAAAACgoKGRkAAaMTif8='/*!*/;# at 824#150103 16:55:49 server id 1 end_log_pos 876 CRC32 0xaac329ee Delete_rows: table id 76 flags: STMT_END_F
BINLOG 'BRGoVCABAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAAAA7inDqg=='/*!*/;DELIMITER ;# End of log fileROLLBACK /* added by mysqlbinlog */;/*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/;
Percona Live Europe Amsterdam 2015
binary log's event content # mysqlbinlog mysql-bin.000018 -j 824 --stop-position=876/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/;/*!40019 SET @@session.max_insert_delayed_threads=0*/;/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;DELIMITER /*!*/;# at 4#141221 21:20:55 server id 1 end_log_pos 120 CRC32 0xff8913a3 Start: binlog v 4, server v 5.6.22-log created 141221 21:20:55 at startup# Warning: this binlog is either in use or was not closed properly.ROLLBACK/*!*/;BINLOG 'pyuXVA8BAAAAdAAAAHgAAAABAAQANS42LjIyLWxvZwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACnK5dUEzgNAAgAEgAEBAQEEgAAXAAEGggAAAAICAgCAAAACgoKGRkAAaMTif8='/*!*/;# at 824#150103 16:55:49 server id 1 end_log_pos 876 CRC32 0xaac329ee Delete_rows: table id 76 flags: STMT_END_F
BINLOG 'BRGoVCABAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAAAA7inDqg=='/*!*/;DELIMITER ;# End of log fileROLLBACK /* added by mysqlbinlog */;/*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/;
Percona Live Europe Amsterdam 2015
binary log's event content decoded...BINLOG 'BRGoVCABAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAAAA7inDqg=='/*!*/;...
● Let's check from the source the type code: https://github.com/mysql/mysql-server/blob/5.6/sql/log_event.h
WRITE_ROWS_EVENT = 30,
UPDATE_ROWS_EVENT = 31,
DELETE_ROWS_EVENT = 32,
Percona Live Europe Amsterdam 2015
binary log's event content decoded...BINLOG 'BRGoVCABAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAAAA7inDqg=='/*!*/;...
● Let's check from the source the type code: https://github.com/mysql/mysql-server/blob/5.6/sql/log_event.h
WRITE_ROWS_EVENT = 30,
UPDATE_ROWS_EVENT = 31,
DELETE_ROWS_EVENT = 32,
MySQL 5.5 and MariaDB use V1: WRITE_ROWS_EVENT_V1 = 23,UPDATE_ROWS_EVENT_V1 = 24,DELETE_ROWS_EVENT_V1 = 25,
Percona Live Europe Amsterdam 2015
binary log's event content decoded# python -c "print hex(32)"0x20
● Again from the source we can find the event type's offset
#define EVENT_TYPE_OFFSET 4
# python -c "import base64; print ord(base64.b64decode(b'BRGoVCABAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAAAA7inDqg==')[4])"32
● This is exactly the event we were looking for !!
Percona Live Europe Amsterdam 2015
rebuild a new binlog event● Now we can rebuild a new event by replacing \x32 by \x30# python>>> import base64>>> data=base64.b64decode(b'BRGoVCABAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAAAA7inDqg==')>>> data'\x05\x11\xa8T \x01\x00\x00\x004\x00\x00\x00l\x03\x00\x00\x00\x00L\x00\x00\x00\x00\x00\x01\x00\x02\x00\x04\xff\xf0\x02\x00\x00\x00\x03Liz\x02\x00\x00\x00\x18\x00\x00\x00\xee)\xc3\xaa'>>> hex(30)'0x1e'>>> data2=base64.b64encode("\x05\x11\xa8T\x1e\x01\x00\x00\x004\x00\x00\x00l\x03\x00\x00\x00\x00L\x00\x00\x00\x00\x00\x01\x00\x02\x00\x04\xff\xf0\x02\x00\x00\x00\x03Liz\x02\x00\x00\x00\x18\x00\x00\x00\xee)\xc3\xaa")>>> data2'BRGoVDABAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAAAA7inDqg=='
● Let's compare the two lines:BRGoVCABAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAAAA7inDqg==BRGoVB4BAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAAAA7inDqg==
Percona Live Europe Amsterdam 2015
rebuild a new binlog event & replay it● ready to replay ?●# mysqlbinlog mysql-bin.000018 -j 824 --stop-position=876 | \ > sed s/BRGoVCAB/BRGoVB4B/ | mysql
mysql> select * from fosdem.community_dinner;+----+--------+--------+-------+| id | name | pizzas | beers |+----+--------+--------+-------+| 1 | dim0 | 99 | 99 || 3 | Kenny | 12 | 32 || 4 | lefred | 10 | 10 |+----+--------+--------+-------+3 rows in set (0.00 sec)
Percona Live Europe Amsterdam 2015
rebuild a new binlog event & replay it● ready to replay ?●# mysqlbinlog mysql-bin.000018 -j 824 --stop-position=876 | \ sed s/BRGoVCAB/BRGoVB4B/ | mysql
mysql> select * from fosdem.community_dinner;+----+--------+--------+-------+| id | name | pizzas | beers |+----+--------+--------+-------+| 1 | dim0 | 99 | 99 || 3 | Kenny | 12 | 32 || 4 | lefred | 10 | 10 |+----+--------+--------+-------+3 rows in set (0.00 sec)
uuh ? did it fail ?
Percona Live Europe Amsterdam 2015
find what to replay
mysql> show binlog events in 'mysql-bin.000018';+------------------+-----+-------------+-----------+-------------+---------------------------+| Log_name | Pos | Event_type | Server_id | End_log_pos | Info |+------------------+-----+-------------+-----------+------------+----------------------------+| mysql-bin.000018 | 4 | Format_desc | 1 | 120 | Server ... || mysql-bin.000018 | 120 | Query | 1 | 220 | create ... || mysql-bin.000018 | 220 | Query | 1 | 404 | use `fosdem`; || mysql-bin.000018 | 404 | Query | 1 | 478 | BEGIN || mysql-bin.000018 | 478 | Table_map | 1 | 544 | table_id: 76 (fosdem.commu|| mysql-bin.000018 | 544 | Write_rows | 1 | 653 | table_id: 76 flags: STMT_E|| mysql-bin.000018 | 653 | Xid | 1 | 684 | COMMIT /* xid=32 */ || mysql-bin.000018 | 684 | Query | 1 | 758 | BEGIN | | mysql-bin.000018 | 758 | Table_map | 1 | 824 | table_id: 76 (fosdem.commu|| mysql-bin.000018 | 824 | Delete_rows | 1 | 876 | table_id: 76 flags: STMT_E|| mysql-bin.000018 | 876 | Xid | 1 | 907 | COMMIT /* xid=34 */ |+------------------+-----+-------------+-----------+------------+====------------------------+
● We need to replay more, from the BEGIN to the COMMIT
Percona Live Europe Amsterdam 2015
replay it, 2nd try● ready to replay, again ?●# mysqlbinlog mysql-bin.000018 -j 684 --stop-position=907 | \ > sed s/BRGoVCAB/BRGoVB4B/ | mysql
mysql> select * from fosdem.community_dinner;+----+--------+--------+-------+| id | name | pizzas | beers |+----+--------+--------+-------+| 1 | dim0 | 99 | 99 || 2 | Liz | 2 | 24 || 3 | Kenny | 12 | 32 || 4 | lefred | 10 | 10 |+----+--------+--------+-------+4 rows in set (0.00 sec)
Percona Live Europe Amsterdam 2015
replay it, 2nd try● ready to replay, again ?●# mysqlbinlog mysql-bin.000018 -j 684 --stop-position=907 | \ > sed s/BRGoVCAB/BRGoVB4B/ | mysql
mysql> select * from fosdem.community_dinner;+----+--------+--------+-------+| id | name | pizzas | beers |+----+--------+--------+-------+| 1 | dim0 | 99 | 99 || 2 | Liz | 2 | 24 || 3 | Kenny | 12 | 32 || 4 | lefred | 10 | 10 |+----+--------+--------+-------+4 rows in set (0.00 sec)
Percona Live Europe Amsterdam 2015
More ?● With the same kind of technique, we can also:
– delete rows that were inserted (sql injection?)
– un-update modified rows (more complicated)
● I've created a script that automates all that: MyUndelete
Percona Live Europe Amsterdam 2015
MyUndelete : «un-insert»● We can delete an INSERT
Percona Live Europe Amsterdam 2015
# ./MyUndelete.py -s 41989 -e 42207 -i -b mysql-bin.000004
*** WARNING *** USE WITH CARE ****
Binlog file is /var/lib/mysql/mysqld-bin.000004Start Position file is 41989End Postision file is 42207We also look to undo INSERTsEvent type ('\x1e') is an insert v2Ready to revert the statement ? [y/n]yDone... I hope it worked ;)
MyUndelete : «un-update»● This is a bit more difficult as we need to find the lenght of the
record to be able to split both records and rebuild a working binary event
Percona Live Europe Amsterdam 2015
mysql> select * from community_dinner;+----+-------+--------+-------+| id | name | pizzas | beers |+----+-------+--------+-------+| 1 | fred | 3 | 3 || 2 | lizz | 2 | 10 || 3 | dimi | 99 | 99 || 4 | kenny | 20 | 20 |+----+-------+--------+-------+
MyUndelete : «un-update» (2)
Percona Live Europe Amsterdam 2015
mysql> update community_dinner set beers=0 where id=3;Query OK, 1 row affected (0.01 sec)Rows matched: 1 Changed: 1 Warnings: 0
mysql> select * from community_dinner;+----+-------+--------+-------+| id | name | pizzas | beers |+----+-------+--------+-------+| 1 | fred | 3 | 3 || 2 | lizz | 2 | 10 || 3 | dimi | 99 | 0 || 4 | kenny | 20 | 20 |+----+-------+--------+-------+4 rows in set (0.00 sec)
MyUndelete : «un-update» (3)
Percona Live Europe Amsterdam 2015
mysql> show binlog events in 'mysqld-bin.000018';...| mysqld-bin.000018 | 1145 | Query | 1 | 1219 | BEGIN || mysqld-bin.000018 | 1219 | Table_map | 1 | 1285 | table_id: 70 (fosdem.community_dinner) || mysqld-bin.000018 | 1285 | Update_rows | 1 | 1357 | table_id: 70 flags: STMT_END_F | mysqld-bin.000018 | 1357 | Xid | 1 | 1388 | COMMIT /* xid=44 */ ...
# ./MyUndelete.py -s 1145 -e 1388 -u -b /var/lib/mysql/mysqld-bin.000018
*** WARNING *** USE WITH CARE ****
Binlog file is /var/lib/mysql/mysqld-bin.000018Start Position file is 1145End Postision file is 1357Event type ('\x1f') is an update v2We got an update!!Ready to revert the statement ? [y/n]ySending to mysql...Done... I hope it worked ;)
MyUndelete : «un-update» (4)
Percona Live Europe Amsterdam 2015
mysql> select * from community_dinner;+----+-------+--------+-------+| id | name | pizzas | beers |+----+-------+--------+-------+| 1 | fred | 3 | 3 || 2 | lizz | 2 | 10 || 3 | dimi | 99 | 99 || 4 | kenny | 20 | 20 |+----+-------+--------+-------+
Resources● Oracle MySQL Source code — https://github.com/mysql/mysql-server
● MariaDB Source Code — https://github.com/MariaDB/server
● Scott Noyes Blog Post - http://thenoyes.com/littlenoise/?p=307
● MyUndelete on github — https://github.com/lefred/MyUndelete
Percona Live Europe Amsterdam 2015
Related topic● MySQL released «Binary Log API» that could be use to decode
in a better way the binary logs– http://mysqlhighavailability.com/mysql-binlog-events-reading-and-handling-
information-from-your-binary-log/
– http://mysqlhighavailability.com/mysql-binlog-events-use-case-and-examples/
● Jeremy Cole wrote a Ruby Library to also parse binary logs– https://github.com/jeremycole/mysql_binlog
Percona Live Europe Amsterdam 2015
Thank you
Questions ?
Percona Live Europe Amsterdam 2015