Replication With MySQL• Asynchronous Replication (3.23+) • Semi-Synchronous Replication (5.5) 17...

PRIVILEGED AND CONFIDENTIAL: FOR INTERNAL KAPLAN USE ONLY

Replication With MySQL

Wednesday, October 24, 2012


Replication With MySQLLigaya Turmelle (@lig)

[email protected]://joind.in/7043



Little About Me

• Currently MySQL DBA for Kaplan Professional

• Formally of MySQL Support

• ~8 years



TODO List

• Fundamentals• MySQL Replication Overview• Asynchronous and Semi-Synchronous• Setting Things Up• Monitoring

4


Lightly touch monitoring


FUNDAMENTALS

5



Very Basics

6

Network

Wednesday, October 24, 2012It is the taking of data from one machine and putting on another.


Why would you want to do that?

7




8

• Scale Out

Wednesday, October 24, 20121) Very common - scale out for high read use case. - multiple slaves allow reading to be spread out over them - allows the master to focus just on writing - admittedly only works up to a point



9

• Scale Out• Data Redundancy

Wednesday, October 24, 20122) Very common - Data redundancy - Backups - Failover - testing system - use another storage engine for a table (Ex: need transactions and FK = InnoDB on the master, but want Full-Text Searching = MyISAM)



10

• Scale Out• Data Redundancy• Analytics

Wednesday, October 24, 20123) Analytics - Reporting server can have very different query patterns + different queries + long running transactions + Lots of locks + caching different data then what production is



11

• Scale Out• Data Redundancy• Analytics• Long Distance Data Distribution

Wednesday, October 24, 20124) Long distance data distribution (Async only) - disaster recovery - local copy for an office to work with


Definitions

12



Definitions

13

• Method of Replication

Wednesday, October 24, 2012This is how the master records its changes

The slave does not care what method of replication is used by the master. The slave can handle all the types.


Definitions

14

• Method of Replication– SBR - Statement Based Replication

Wednesday, October 24, 2012* SBR - entire SQL statement is recorded and transferred between the master and the slave* Pros: - this is proven technology and is what the original replication used - less data is written to the log file - can be used for audit purposes to see what occurred in what order* Cons: - Some statements are unsafe/nondeterministic (can’t guarantee same output for given input) + Examples: 1) DELETE and UPDATE statements that use a LIMIT clause without an ORDER BY 2) using any of the following functions: UUID(), UUID_SHORT(), USER(), FOUND_ROWS(), LOAD_FILE(), MASTER_POS_WAIT(), SLEEP(), VERSION(), etc. - More locking may be needed then Row Based - Complex statements will have to be evaluated and executed - Deterministic UDFs must be applied on the slaves


Definitions

15

• Method of Replication– SBR - Statement Based Replication– RBR - Row Based Replication

Wednesday, October 24, 2012* RBR - All/Only the changed rows are recorded - known as “events”* Pros: - All changes can be replicated - Safest form - Same as most other RDBMS - Fewer locks are usually required* Cons: - Generally more data to be logged + Ex: 1 UPDATE statement can alter N rows + so it may take longer to use the binary logs to recover the server + the binary log will be locked for the writing of the data to it


Definitions

16

• Method of Replication– SBR - Statement Based Replication– RBR - Row Based Replication– Mixed Mode

• http://dev.mysql.com/doc/refman/5.5/en/binary-log-mixed.html

Wednesday, October 24, 2012Does exactly what it says - it uses both SBR and RBR - uses SBR by default but switches to RBR in particular cases + Ex: 1) specific storage engines can dictate the method ... NDB - RBR only, InnoDB - only supports SBR in REPEATABLE READ or SERIALIZABLE isolation levels. 2) UUID() is used 3) one or more tables with AUTO_INCREMENT columns are updated and a trigger or stored function is invoked 4) any INSERT DELAYED is executed. 5) call to a UDF is involved


Definitions

• Asynchronous Replication (3.23+)• Semi-Synchronous Replication (5.5)

17

Wednesday, October 24, 2012* Asynchronous - data is written to the master binary log without any guarantees it will make it to the slave. In MySQL the slave “pulls” the data from the master whenever it connects.

* Semi-Synchronous - data is written to the master log and the master then blocks (for a while) the transaction waiting for at least 1 slave to acknowledge they have the change and have logged it. So it tries to have data on the master - and at least 1 slave increasing data integrity if anything goes wrong.

We will get into this more a little later.


Definitions

• Binary log - Master• Relay log - Slave

18

Wednesday, October 24, 2012* Binary log - where the master server keeps track of all its changes. This is written in SBR/RBR/Mixed

* Relay log - relay log is a log kept on the slave and consists of the data from the binary log of the master.


Definitions

• Master– Binlog Dump Thread

• Slave– IO Thread– SQL Thread

19

Wednesday, October 24, 2012On the Master:* Binlog Dump Thread (Waiter that serves the slave) - used to interact with the masters binary log and give the slave’s IO thread its contents. - locks the master’s binary log for reading each event in the binary log to be sent to the slave. + The lock is released when the the event is read and before the event is sent to the slave. - created for *each* connected slave.

On the Slave:* IO Thread - thread from the slave that connects to the master as a standard client connection. It is what transfers over the contents of the masters binary log into the slaves relay log.* SQL Thread - reads and executes the changes recorded in the slaves relay log on the slaves data.* (separated so they work independent of each other)


MYSQL REPLICATION OVERVIEW

20

Wednesday, October 24, 2012Ok - time to see how all that works together and give things a bit of context.


MySQL Replication Overview

21

1Master

Wednesday, October 24, 2012Step 1: Client makes a change in the data or structure on the master



22

Binarylog

1

2

Master

Wednesday, October 24, 2012Step 2: The master then writes that change to the binary log (SBR/RBR, Mixed) - right before the commit of the transaction it is written to the binary log (remember this). - transactions are written serially in the order they are to be committed - Once the change is written into the binary log - the storage engine commits the transaction.

GOTCHA - Potentially you could have a binary log entry written but never run on the master... How?(Answer: server crash between the writing to the binary log and the commit of the transaction. When the server comes back up the transaction will be rolled back, even though it is already in the binary log. Potential to get master/slave out of sync.)



23

RelayLog

Binarylog

1

3

2

Master

Wednesday, October 24, 2012Step 3: The slave’s IO thread copies the change from the master binary log and records it in the slave relay log (Async/Semi-sync)

We will go over how this is done Asynchronously and Semi-Synchronously a little later.



24

Slave

RelayLog

Binarylog

1

4

3

2

Master

Wednesday, October 24, 2012Step 4: The slave’s SQL thread then reads the relay log and makes the changes to its own data

It should be understood that the SQL thread has all ALL privileges and can read any of the formats used (SBR/RBR), so it can do anything that makes its way into the master binary log.

Note: A slave should be more powerful then a master since it is doing everything the master is - and handling its own workload/reads.


ASYNCHRONOUS AND SEMI-SYNCHRONOUS

25



Asynchronous

26



Asynchronous

27

Master Slave

Wednesday, October 24, 2012OK - you have the master and the slave


Asynchronous

28

Master SlaveIO Thread (3)

Binlog DumpThread created

Wednesday, October 24, 2012The slave creates an IO thread that connects to the master just like any other client.The master creates the Binlog Dump Thread for the connected slave.


Asynchronous

29

Master Slave

Binarylog

IO Thread (3)

Binlog DumpProcess

Wednesday, October 24, 2012The slave issues a command that asks the master to send the updates in its binary logs.The master has the Binlog Dump thread fetches the updates from the binary log and gives them to the slave IO thread.


Asynchronous

30

Master Slave

RelayLog

Binarylog

IO Thread (3)

Binlog DumpProcess

IO Thread (3)

Wednesday, October 24, 2012The slave’s IO thread then writes the binary log events into the slave’s relay log.

If there is nothing else to retrieve from the master, the IO thread can go to sleep and waits for the master to signal that there is a new event or completely disconnect from the master.


Asynchronous

31

• Pros– fast– slave not required to be continually connected

• Cons– potential loss of transactions and data integrity

Wednesday, October 24, 2012The slave pulls any changes when it is connected


Semi-Synchronous

32

Wednesday, October 24, 2012Ok - time to go over Semi-Synchronous replication


Semi-Synchronous

33

Master Slave

semi-synccapable

semi-synccapable

Wednesday, October 24, 2012OK - again you have the master and the slave.

But this time - the master and slave have the semi-synchronous replication plugins installed and enabled (only on 1 side means you will be async)


Semi-Synchronous

34

Master SlaveIO Thread (3)

Binlog DumpThread created

semi-sync capablesemi-synccapable

semi-synccapable

Wednesday, October 24, 2012This slave will connect to the master just like last time, but it will also tell the master it is semi-synchronous capable and stay connected.The master will again create the Binlog Dump thread.


Semi-Synchronous(optimal)

35

Master SlaveIO Thread

Binlog DumpThreadCreated

semi-synccapable

semi-synccapable

1

Wednesday, October 24, 2012Now when client connects to the semi-sync capable Master (that has at least 1 semi-sync capable slave connected) and makes a change...



36

Master Slave

RelayLog

Binarylog

IO Thread

Binlog DumpProcess

IO Thread (save)

semi-synccapable

semi-synccapable

1

x

Wednesday, October 24, 2012The thread that does the transaction commit on the master will block after the commit as it waits for at least 1 semi-sync slave to receive and acknowledge that it has all events for the transaction, written them to its relay log and flushed it to disk *before* the timeout (rpl_semi_sync_master_timeout = milliseconds - default 10000/AKA 10 seconds).

Now remember that the master will write to the binary log right before the transaction is committed. Which in turn triggers the slave IO thread to pick up that a change has been made... and so on.



37

Master SlaveIO Thread semi-synccapable

semi-synccapable

1

RelayLog

Binarylog

Binlog DumpProcess

IO Thread (ack)


When the master gets the acknowledgement, the reply is sent back to the client.

Can you now see how semi-synchronous replication can slow down connections?


Semi-Synchronous(timeout)

38

Wednesday, October 24, 2012Now it is time to go over what happens in semi-sync replication if there is a timeout.



39

Master SlaveIO Thread

Binlog DumpThreadCreated

semi-synccapable

semi-synccapable

1

Wednesday, October 24, 2012OK - we are back to the client connecting to the semi-sync capable Master (that has at least 1 semi-sync capable slave connected) and makes a change...



40

Master Slave

Binarylog

IO Thread

Binlog DumpProcess

semi-synccapable IO Thread (save)

semi-synccapable

1

xNetwork

Wednesday, October 24, 2012Again the thread that does the transaction commit on the master will block after the commit as it waits for at least 1 semi-sync slave to receive and acknowledge that it has all events for the transaction, written them to its relay log and flushed it to disk.



41

Master Slave

Binarylog

IO Thread

Binlog DumpProcess

semi-synccapable IO Thread (save)

semi-synccapable

1

xNetwork

TIMEOUT

Wednesday, October 24, 2012This time the master hits the timeout!



42

Master Slave

Binarylog

IO Thread

Binlog DumpProcess

async semi-synccapable

1

Wednesday, October 24, 2012The master reverts to asynchronous replication and returns to the client. The master will stay in asynchronous replication until...

This limits any performance problems caused by the master waiting on the acknowledgements from the slaves.



43

Master Slave

RelayLog

Binarylog

IO Thread

Binlog DumpProcess

IO Thread (save)





44

Master Slave

RelayLog

Binarylog

IO Thread

Binlog DumpProcess

IO Thread (save)


resync

Wednesday, October 24, 2012When at least 1 semi-sync slave catches back up to the master



45

Master Slave

RelayLog

Binarylog

IO Thread (synced)

Binlog DumpProcess


resync

Wednesday, October 24, 2012When at least 1 semi-sync slave catches back up to the master



46

Master Slave

RelayLog

Binarylog

Binlog DumpProcess

semi-synccapable

IO Thread semi-synccapable

Wednesday, October 24, 2012The master returns to semi-synchronous replication


Semi-Synchronous

47

• Pros– tries to improve data integrity

• Cons– performance implications

Wednesday, October 24, 2012If commit returns successfully, the data is known to be in at least 2 locations.


SETTING IT UP

48



Asynchronous - Master

49



Config/Option Filemy.cnf / my.ini


50

log-bin=mysql_binserver_id=1

Wednesday, October 24, 2012restart master after change


Create the replication userrequires the REPLICATION SLAVE privilege


51

mysql> CREATE USER 'repl'@'%.mydomain.com' IDENTIFIED BY 'slavepass';

mysql> GRANT REPLICATION SLAVE ON *.* TO 'repl'@'%.mydomain.com';



Get master Server binary log coordinates

requires the REPLICATION SLAVE privilege


52

mysql> FLUSH TABLES WITH READ LOCK;

mysql> SHOW MASTER STATUS;




• While you have the tables locked– if you need to make a copy of the master this

is the time to make it• Once you have the copy, release the locks

53

mysql> UNLOCK TABLES;


mysqldump or binary copy


Asynchronous - Slave

54





55

server_id=2read-only

Wednesday, October 24, 2012restart server after change



56

• Load in the copy/backup taken from the master– you should verify that any backup works



Issue the CHANGE MASTER command

use the information you got from the master status


57

mysql> CHANGE MASTER TOMASTER_HOST = 'master.mycompany.com', MASTER_USER = 'repl', MASTER_PASSWORD='slavepass', MASTER_PORT=3306, MASTER_LOG_FILE='master-bin.001', MASTER_LOG_POS=4;

mysql> START SLAVE;

Wednesday, October 24, 2012STOP SLAVE;When all that is done - don’t forget to start up the slave!


Semisynchronous - Master

58

Wednesday, October 24, 2012* To set up Semisynchronous replication, standard asynchronous replication must already be up and running*


INSTALL PLUGINMySQL Server must support dynamic loading (Check have_dynamic_loading)


59

mysql> INSTALL PLUGIN rpl_semi_sync_master SONAME 'semisync_master.so';

mysql> SHOW PLUGINS;

Wednesday, October 24, 2012Require SUPER priv

The semisynchronous replication plugins are included with MySQL distributions. Make sure they are in the plugin directory.

SHOW PLUGINS to make sure plugins are now installed.


Enable PLUGINTurned off by default


60

mysql> SET GLOBAL rpl_semi_sync_master_enabled = 1;

mysql> SET GLOBAL rpl_semi_sync_master_timeout = N;

Wednesday, October 24, 2012timeout value N is given in milliseconds. Default is 10000 (10 seconds).




61

log-bin=mysql_binserver_id=1

# semi-sync replrpl_semi_sync_master_enabled = 1rpl_semi_sync_master_timeout = 1000 # 1 sec

Wednesday, October 24, 2012Since you ran the commands for the semi-sync repl during runtime - the values placed into the option file are there for when the server restarts.


Semisynchronous - Slave

62



INSTALL PLUGINMySQL Server must support dynamic loading (Check have_dynamic_loading)


63

mysql> INSTALL PLUGIN rpl_semi_sync_slave SONAME 'semisync_slave.so';

mysql> SHOW PLUGINS;

Wednesday, October 24, 2012Require SUPER priv

The semisynchronous replication plugins are included with MySQL distributions. Make sure they are in the plugin directory.


Enable PLUGINTurned off by default


64

mysql> SET GLOBAL rpl_semi_sync_slave_enabled = 1;



Runtime


65

mysql> STOP SLAVE IO_THREAD;

mysql> START SLAVE IO THREAD;

Wednesday, October 24, 2012Bounce the slave IO thread to pick up the changes.




66

server_id=2read-only

# semi-sync replrpl_semi_sync_slave_enabled=1

Wednesday, October 24, 2012Since you ran the commands for the semi-sync repl during runtime - the values placed into the option file are there for when the server restarts.


MONITORING

67



Monitoring the Master

68



Binary Logs

70

mysql> SHOW BINARY LOGS;+---------------+-----------+| Log_name | File_size |+---------------+-----------+| binlog.000015 | 724935 || binlog.000016 | 733481 |+---------------+-----------+

• Lists the binary logs on the server

Wednesday, October 24, 2012Master really doesn’t give us a lot of information.


Monitoring the Slave

71

Wednesday, October 24, 2012brace yourself for information overload... again


Slave Status

72

mysql> SHOW SLAVE STATUS\G*************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: localhost Master_User: root Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.003 Read_Master_Log_Pos: 73 Relay_Log_File: mysql-relay-bin.005 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.003 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table:

Wednesday, October 24, 2012Slave_IO_State: A copy of the State field of the SHOW PROCESSLIST output for the slave I/O thread.Master_Log_File: master binlog file from which the I/O thread is currently reading.Read_Master_Log_Pos: position in the current master bin log file that I/O thread has read to.Relay_Log_File:relay log file from which the SQL thread is currently *reading* and executing.Relay_Log_Pos: position in relay log file up to which the SQL thread has read and executed.Relay_Master_Log_File: name of the master binlog containing the most recent event executed by the SQL thread.


Slave Status

73

Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 73 Relay_Log_Space: 552 Until_Condition: None Until_Log_File: Last_SQL_Error: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error:

Wednesday, October 24, 2012Exec_Master_Log_Pos: position in the binlog up to which the SQL thread has read and executed. - The coordinates given by (Relay_Master_Log_File, Exec_Master_Log_Pos) in the master's binary log correspond to the coordinates given by (Relay_Log_File, Relay_Log_Pos) in the relay log.Relay_Log_Space: total combined size of all existing relay log files.Seconds_Behind_Master: Not very good/accurate. In essence, this field measures the time difference in seconds between the slave SQL thread and the slave I/O thread. Last_IO_Errno/Last_IO_Error: error number and error message of the last error that caused the I/O thread to stop.Last_SQL_Errno/Last_SQL_Error: error number and error message of the last error that caused the SQL thread to stop.

** Additional notes on Seconds behind master:This field is an indication of how “late” the slave is: - When the slave SQL thread is actively processing updates, this field is the number of seconds that have elapsed since the timestamp of the most recent event on the master executed by that thread. - When the SQL thread has caught up to the slave I/O thread and is idle waiting for more events from the I/O thread, this field is zero.

Gotcha: If the network is slow, this is not a good approximation; the slave SQL thread may quite often be caught up with the slow-reading slave I/O thread, so Seconds_Behind_Master often shows a value of 0, even if the I/O thread is late compared to the master. In other words, this column is useful only for fast networks.


Questions?

74



Thank You

75

Ligaya Turmelle (@lig)[email protected]

https://joind.in/7043


Replication With MySQL• Asynchronous Replication (3.23+) • Semi-Synchronous Replication (5.5) 17...

Documents

Transcript of Replication With MySQL• Asynchronous Replication (3.23+) • Semi-Synchronous Replication (5.5) 17...