Tungsten University: Zero-Downtime Maintenance And Schema Operations
25
©Continuent 2013 Tungsten University: Zero Downtime Maintenance and Schema Upgrade Robert Hodges, CEO Giuseppe Maxia, Director of QA
-
Upload
continuent -
Category
Technology
-
view
482 -
download
2
description
Do you know how to do rolling maintenance on your database hosts so you can make changes without stopping applications? How about upgrading schema and applications themselves? Tungsten clusters have a host of features that can help you with everything from basic administration to complex application upgrades. This webinar shows you the different types of administration you need to perform and introduce the procedures to perform them without application downtime. Course Topics • What is rolling maintenance • How to perform simple maintenance on hosts • How to plan, test, and perform a MySQL version upgrade • Standard SQL schema upgrade patterns and how to do them with zero downtime • Combining SQL schema and application code upgrades • Structuring applications to make zero-downtime upgrade work better • Backing out of failed upgrade or maintenance
Transcript of Tungsten University: Zero-Downtime Maintenance And Schema Operations
©Continuent 2013
Tungsten University: Zero Downtime Maintenance
and Schema UpgradeRobert Hodges, CEO
Giuseppe Maxia, Director of QA
©Continuent 2013
Introducing Continuent
2
• The leading provider of clustering and replication for open source DBMS
• Our Product: Continuent Tungsten
• Clustering - Commercial-grade HA, performance scaling and data management for MySQL
• Replication - Flexible, high-performance data movement
©Continuent 2013
High Availability and System Maintenance
3
©Continuent 2013
Maintenance and Upgrade Math
4
Uptime Weekly Maintenance Window
95.2% 8 hours
97.6% 4 hours
98.8% 2 hours
99.4% 1 hour
99.8% 20 minutes
99.99% 1 minute (!!)
True 4 9s availability: 53 minutes downtime per year
©Continuent 2013
Tungsten Cluster Architecturein 3 Minutes or Less
5
©Continuent 2013
Overview of a Tungsten Cluster
6
Application
Tungsten Connector
Application
Tungsten Connector
Replicator
Db2
Replicator
Db3Db1
Replicator
Slave SlaveMaster
Manager Manager Manager
Monitoring and Control
Monitoring and Control
Data Service: sjc
©Continuent 2013
Introduction to MySQL Maintenance:
Change a my.cnf variable and restart
7
©Continuent 2013
Useful cctrl maintenance commands
• set policy automatic | manual | maintenance
• datasource <name> o!ine
• datasource <name> online
• datasource <name> shun
• datasource <name> welcome
• cluster online
8
©Continuent 2013
Step 2Perform maintenance task(s)
Step 3Welcome slave back to cluster
Step 1Shun node and take services o"ine
3
1
2
Online Maintenance of a Slave
9
©Continuent 2013
Step 2Shun node and take services o"ine
Step 3Perform maintenance task(s)
Step 1Switch master to slave
3
12
4
Phase 5Switch slave back to master
5
Step 4Welcome slave back
Online Maintenance of a Master
10
©Continuent 2013
Rolling Maintenance
11
MasterSwitchSlave 1 Slave 2-N
• Shun slave 1, maintain, welcome back to cluster
• Shun old master, maintain, welcome back to cluster
• Repeat for additional slave(s)
• Switch master
©Continuent 2013
Extended MySQL Maintenance:
Upgrading to a new MySQL version
12
©Continuent 2013
Rolling MySQL Version Upgrade
13
Final UpgradeSwitchSlave Upgrade Slave Upgrade
1. Shun slave2. Dump data3. Upgrade
MySQL4. Restore data5. Welcome slave6. Backup
1. Shun slave2. Upgrade
MySQL3. Restore
1.Shun slave2. Upgrade
MySQL3. Restore
1.Switch master
©Continuent 2013
Backups and MySQL Version Upgrade
14
• Backup before upgrade to protect against failure
/cluster > datasource DB2 backup
• Backup after 1st upgrade to simplify further upgrades
/cluster > datasource DB2 backup
• Restore after subsequent node upgrades
/cluster > datasource DB3 restore
©Continuent 2013
Schema Upgrade:
Performing ALTER TABLE commands
15
©Continuent 2013
Tips and Tricks for Schema Upgrade
16
• Use replication to do simple, non-blocking changes
• Don’t log schema changes on slaves
mysql> set session sql_log_bin=0;
• Add schema rather than taking away
©Continuent 2013
Rolling Schema Upgrade
17
Final UpgradeSwitchSlave Upgrade Slave Upgrade
1. Run backup2. Shun slave3. Run upgrade4. Discard and
restore on failure
5. Welcome slave
1. Shun/upgrade/welcome
1.Shun/upgrade/welcome
1.Switch master
©Continuent 2013
Ways to Make Upgrade and Maintenance Easy or at
Least Less Painful :)
18
©Continuent 2013
Secrets of Success #1: Testing
• Develop a procedure and follow it
• Automate as much as possible
• Run tests before going production
• Use production data and lots of it
19
©Continuent 2013
Secrets of Success #2: Minimize Risk
• You can do zero-downtime upgrade without stopping any apps
• But why take chances?
• Run upgrades at periods of low activity
• Shut down non-essential services
• Upgrade low-value applications "rst
• Upgrade slaves "rst
20
©Continuent 2013
Secrets of Success #3: Application Design
• Avoid “generic” SQL statement that do not specify columns
• Build schema upgrade into application installation
21
Install base schemaRun upgrade script #1Run upgrade script #2...Application ready...
©Continuent 2013
Secrets of Success #4: Be Prepared for Trouble
22
©Continuent 2013
Wrapping Up
23
©Continuent 2013
Tungsten University Sessions
24
• Zero-Downtime Maintenance and Schema Operations (March 21 and 26)
• Geographically Distributed Tungsten Clusters (April 11 and 16)
• Replicate between MySQL and Oracle(May 2 and 7)
Send any feedback to: [email protected]
©Continuent 2012.
Continuent Web Page:http://www.continuent.com
Tungsten Replicator 2.0:http://code.google.com/p/tungsten-replicator
Our Blogs:http://scale-out-blog.blogspot.comhttp://datacharmer.org/bloghttp://www.continuent.com/news/blogs
560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel +1 (866) 998-3642 Fax +1 (408) 668-1009e-mail: [email protected]
