Percona Live Europe 2016 Use ProxySQL to Improve …Percona Live Europe 2016 Use ProxySQL to Improve...
Transcript of Percona Live Europe 2016 Use ProxySQL to Improve …Percona Live Europe 2016 Use ProxySQL to Improve...
Percona Live Europe 2016Use ProxySQL to Improve
Your MySQL High Availability Solution
Marco Tusa Manager ConsultingAmsterdam, Netherlands | October 3 – 5, 2016
Percona Live 2017Use ProxySQL to Improve
Your MySQL High Availability Solution
Marco tusaSanta Clara, California | April 24th – 27th, 2017
3
� Marco “The Grinch”• Open source enthusiast
About me
4
•Brief ProxySQL introduction•How it interact Master-slave •How interact with PXC•Extending with scheduler•ProxySQL HA•Time cost
Agenda
5
What is HA9 0. 0 0 0 % (36 days) MySQL Replication9 9. 9 0 0 % (8 hours) Linux Heartbeat with DRBD9 9. 9 0 0 % (8 hours) RHCS with Shared Storage (Active/Passive)9 9. 9 9 0 % (52 minutes) MHA with at least 3 nodes9 9. 9 9 0 % (52 minutes) Linux Heartbeat with DRBD and Replication9 9. 9 9 0 % (52 minutes) Linux Heartbeat with Shared Storage and Replication9 9 .9 9 5 % (26 minutes) Multi Master (galera - percona cluster) 3 node minimum9 9. 9 9 9 % (5 minutes) MySQL Cluster
6
What is HA
7
What is HA
8
Who will save the Day?
9
Who will save the Day?
10
Objectively What we have is:
MySQL Group Replication
11
The telephone game
12
All talking
13
Black box:
14
Use the Right Tool for the Job…
15
What we need
16
MANAGEBLEFAST
SCALABLEFREE
Solution should be
17
… and do the right thing!My Motto
18
ProxySQL
19
ProxySQL
20
Understand the basics - what it does ...
Write / Read (Master/Slave)
Multiple WritersSharding &
Geographic Distribution
Traffic Load dispatch (aka Mirroring)
Query re-write
21
Define UsersServers group by function
Host GroupsRules define what, where and how
Query RulesSpecial Grouping for Replication
Replication Host Group
Understand the basics – how this is done?
22
HostGroup (HG) & Replication HG
Understand the basics
23
Query rules
Understand the basics
24
Scheduler
Understand the basics
25
Basic scenario
HG: 10U: app_RW
HG: 11U: app_R
HG: 12U: ETL_R
26
insert into mysql_users(username,password,active,default_hostgroup,default_schema) values ('app_RW','test',1,10,'test_schema');
insert into mysql_users(username,password,active,default_hostgroup,default_schema) values ('app_R','test',1,11,'test_schema');
insert into mysql_users
(username,password,active,default_hostgroup,default_schema) values ('etl_R','test',1,12,’etl_schema');LOAD MYSQL USERS TO RUNTIME;SAVE MYSQL USERS TO DISK;
Basic scenario - Users
27
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES ('192.168.1.5',10,3306,1000);INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight, max_replication_lag) VALUES ('192.168.1.6',11,3306,10000,2);INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight, max_replication_lag) VALUES ('192.168.1.7',11,3306,10000,2);INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight, max_replication_lag) VALUES ('192.168.1.5',11,3306,1,2);INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES ('192.168.1.8',12,3306,10);INSERT INTO mysql_replication_hostgroups VALUES (10,11);LOAD MYSQL SERVERS TO RUNTIME; SAVE MYSQL SERVERS TO DISK;
Basic scenario – Servers / HG
28
insert into mysql_query_rules(username,destination_hostgroup,active,retries,match_digest,apply) values(’app_RW',10,1,3,'^SELECT.*FOR UPDATE',1); insert into mysql_query_rules(username,destination_hostgroup,active,retries,match_digest,apply) values(’app_RW',11,1,3,'^SELECT ',1); insert into mysql_query_rules (username,destination_hostgroup,active) values(’app_R',11,1);insert into mysql_query_rules (username,destination_hostgroup,active) values('etl_R',12,1);
LOAD MYSQL QUERY RULES TO RUNTIME;SAVE MYSQL QUERY RULES TO DISK;
Basic scenario – Query rules
29
Basic scenario – with ProxySQL
Writes
ReadsReads
30
READ_ONLY = 0 WRITE HGREAD_ONLY = 1 READ HG
ProxySQL – Replication HG – danger!
READ_ONLY=0 READ_ONLY=1 READ_ONLY=1
HG 10 & HG:11HG 11 HG 11
READ_ONLY=0
HG 10 & HG 11
31
ProxySQL + MHA (or any controller)
Writes
ReadsReads
32
No movable part like:
VIPDNS (Route 53 or similar)
No Crazy scripts to manage other proxy (ie HAProxy)
ProxySQL + MHA (or any controller)
33
MHA discovery time (ping x 3 times) -> ~9 secMHA recovery (binlog) variable time
MHA Slave shift + Master READ-ONLY < 2 sec
ProxySQL will react as soon as READ_ONLY=0
What timing with MHA?
34
Nothing all defaults
master_ip_failover_script=master_ip_failovermaster_ip_online_change_script=master_ip_online_change
Test it:masterha_master_switch --master_state=alive --conf=/etc/mha.cnf --orig_master_is_new_slave --interactive=0 --running_updates_limit=0
What is needed with MHA?
35
Given no VIP or centralize element.
Consistent distributed behavior for HA?
36
Integration with PXC\Galera
37
Integration with PXC\Galera
38
Integration with PXC\Galera
39
Integration with PXC\Galera read_onlywsrep_statuswsrep_rejectquerieswsrep_donorrejectquerieswsrep_connectedwsrep_desinccountwsrep_readywsrep_providerwsrep_segmentNumber of nodes in by segmentRetry loop
Online Offline_softOffline_hardShunned
But we also have HostgroupsReplication HG
40
Integration with PXC\Galera
41
Integration with PXC\Galera
HG: 10
HG: 10
HG: 12HG: 11
HG: 11HG: 12
42
Multiple writer vs Single writerMulti writer based on weight and Hostgroups
Need Replication HG and READ_ONLYFailover is done without the need of performing actions on the nodes
Failover is done modifying the READ_ONLY valueSingle WRITER node isolation not guarantee
Single WRITER node isolation is guarantee
Integration with PXC\Galera
43
insert into mysql_users(username,password,active,default_hostgroup,default_schema) values ('app_RW','test',1,10,'test_schema');
insert into mysql_users(username,password,active,default_hostgroup,default_schema) values ('app_R','test',1,11,'test_schema');
insert into mysql_users
(username,password,active,default_hostgroup,default_schema) values (‘appDR_R','test',1,12,’etl_schema');LOAD MYSQL USERS TO RUNTIME;SAVE MYSQL USERS TO DISK;
Integration with PXC\Galera - Users
44
insert into mysql_query_rules(username,destination_hostgroup,active,retries,match_digest,apply) values(’app_RW',10,1,3,'^SELECT.*FOR UPDATE',1);
insert into mysql_query_rules(username,destination_hostgroup,active,retries,match_digest,apply) values(’app_RW',11,1,3,'^SELECT ',1);
insert into mysql_query_rules (username,destination_hostgroup,active) values(’app_R',11,1);
insert into mysql_query_rules (username,destination_hostgroup,active) values(‘appDR_R',12,1);LOAD MYSQL QUERY RULES TO RUNTIME;SAVE MYSQL QUERY RULES TO DISK;
Integration with PXC\Galera – QR
45
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight)
VALUES ('192.168.1.5',10,3306,50000);192.168.1.6 … 10000; 192.168.1.7 … 10
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight)
VALUES ('192.168.1.5',11,3306,1);192.168.1.6 … 30000; 192.168.1.7 … 30000
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES ('192.168.1.8',12,3306,100);192.168.1.9 … 100;LOAD MYSQL SERVERS TO RUNTIME; SAVE MYSQL SERVERS TO DISK;
Integration with PXC\Galera – HG MW
46
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES
('192.168.1.5',10,3306,100);
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES
('192.168.1.5',11,3306,1);INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES
('192.168.1.6',11,3306,100);INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES ('192.168.1.7',11,3306,100);
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES ('192.168.1.8',12,3306,100);192.168.1.9 … 100;INSERT INTO mysql_replication_hostgroups VALUES (10,11);LOAD MYSQL SERVERS TO RUNTIME; SAVE MYSQL SERVERS TO DISK;
Integration with PXC\Galera HG SW
47
ProxySQL is configured Now what?
Example scripthttps://github.com/Tusamarco/proxy_sql_toolsFull guide: https://goo.gl/vcDFZx
Integration with PXC
48
Set the script:INSERT INTO scheduler (id,active,interval_ms,filename,arg1) values (10,0,2000,"/var/lib/proxysql/galera_check.pl","-u=admin -p=admin -h=192.168.1.50 -H=500:W,501:R -P=3310 --execution_time=1 --retry_down=2 --retry_up=1 --main_segment=1 --debug=0 --log=/var/lib/proxysql/galeraLog");LOAD SCHEDULER TO RUNTIME;SAVE SCHEDULER TO DISK;
update scheduler set active=1 where id=10;LOAD SCHEDULER TO RUNTIME;
Integration with PXC
49
+--------------+-------------+--------+------------+---------+
| hostgroup_id | hostname | status | weight | comment |+--------------+-------------+--------+------------+---------+
| 10 | 192.168.1.5 | ONLINE | 50000 | || 10 | 192.168.1.6 | ONLINE | 10000 | |
| 10 | 192.168.1.7 | ONLINE | 10 | |
| 11 | 192.168.1.5 | ONLINE | 1 | || 11 | 192.168.1.6 | ONLINE | 10000 | |
| 11 | 192.168.1.7 | ONLINE | 10000 | || 12 | 192.168.1.8 | ONLINE | 100 | |
| 12 | 192.168.1.9 | ONLINE | 100 | |+--------------+-------------+--------+------------+---------+
Integration with PXC MW
50
+--------------+-------------+--------+------------+---------+
| hostgroup_id | hostname | status | weight | comment |+--------------+-------------+--------+------------+---------+
| 10 | 192.168.1.5 |OFFLINE_SOFT | 50000 | 10_W_11_R_retry_up=0;10_W_11_R_retry_down=0; | 10 | 192.168.1.6 | ONLINE | 10000 | |
| 10 | 192.168.1.7 | ONLINE | 10 | || 11 | 192.168.1.5 | OFFLINE_SOFT | 1 | ... |
| 11 | 192.168.1.6 | ONLINE | 10000 | || 11 | 192.168.1.7 | ONLINE | 10000 | |
| 12 | 192.168.1.8 | ONLINE | 100 | |
| 12 | 192.168.1.9 | ONLINE | 100 | |+--------------+-------------+--------+------------+---------+
Integration with PXC MW
51
+---------------------------------+-------+| Variable_name | Value |+---------------------------------+-------+| wsrep_sst_donor_rejects_queries | ON |+---------------------------------+-------+
ProxySQL Think all is ok
Application writes will fail
Integration with PXC MW
52
+--------------+-------------+--------+------------+---------+
| hostgroup_id | hostname | status | weight | comment |+--------------+-------------+--------+------------+---------+
| 10 | 192.168.1.6 | ONLINE | 10000 | || 10 | 192.168.1.7 | ONLINE | 10 | |
| 11 | 192.168.1.6 | ONLINE | 100000 | |
| 11 | 192.168.1.7 | ONLINE | 100000 | || 12 | 192.168.1.8 | ONLINE | 100 | |
| 12 | 192.168.1.9 | ONLINE | 100 | || 9010 | 192.168.1.5 | ONLINE | 50000 | || 9011 | 192.168.1.5 | ONLINE | 1 | |+--------------+-------------+--------+------------+---------+
Integration with PXC MW
53
WRITE Fail-over is automatically perform by ProxySQL
following the weightUntil an active node exists the HG
Integration with PXC MW
54
WRITE Fail-over will not happen
Until READ_ONLY variableis modified on a reader HG node.
WRITER if DONOR will never be OFFLINE_SOFT
Integration with PXC SW
55
Integration with PXC\Galera SW manual
56
Integration with PXC\Galera SW Auto
57
Change state to:OFFLINE_SOFT = ~2 secHostGroup 9000 = ~2 secProxySQL Shunned = ~1.2 sec (5 fails in 1sec)Failover (auto) = ~3 sec
Integration with PXC - Timing
58
Avoid SPOF
59
Avoid SPOF using Keepalived
60
Avoid SPOF using Keepalived
61
Cost in time
62
Cost in time
63
So what.. 9 0. 0 0 0 % (36 days) MySQL Replication9 9. 9 0 0 % (8 hours) Linux Heartbeat with DRBD9 9. 9 0 0 % (8 hours) RHCS with Shared Storage (Active/Passive)9 9. 9 9 0 % (52 minutes) MHA with at least 3 nodes9 9. 9 9 0 % (52 minutes) Linux Heartbeat with DRBD and Replication9 9. 9 9 0 % (52 minutes) Linux Heartbeat with Shared Storage and Replication9 9 .9 9 5 % (26 minutes) Multi Master (galera - perconacluster) 3 node minimum9 9. 9 9 9 % (5 minutes) MySQL Cluster
64
Cost in $$$
65
Improve service flexibility with r/w split;Allow simple way to manage access to specific servers by
usage (like DW-ETL);Implement sharding;
Allow Query revrite/query fix/blocking without need to modify code;
Improve failover time;Reduce master downtime impact on reads;
Not SPOF (as vip)
Brief summary
66
Integration with MHA
67
68
69
ContactsTo contact Me
To follow me
http://www.tusacentral.net/
http://www.percona.com/blog/
https://www.facebook.com/marco.tusa.94
@marcotusa
http://it.linkedin.com/in/marcotusa/
“Consulting = No mission refused!”