Upgrade and Migrate on Multi Terabyte Mission Critical RAC with Near Zero Downtime
-
Upload
dell-enterprise -
Category
Technology
-
view
2.242 -
download
11
Transcript of Upgrade and Migrate on Multi Terabyte Mission Critical RAC with Near Zero Downtime
Upgrade & Migrate on Multi-Terabyte Mission Critical RAC with Near Zero Downtime
Jane Brown Senior Database Administrator, Oracle Certified Professional (OCP)
Database Management Practice
Purpose
• Minimize planned down time
• Meet the 7x24 availability requirement for Dell Mission Critical Applications
• Streamline Dell Mission Critical databases upgrades to 11g
2
Database Management Practice
Current Versions
• DB Version: 10.2.0.4
• OS Version: Suze Linux 64-bit
3
Target Upgrade Versions• Database Version: 11.1.0.7
• OS Version: OEL 64-bit
Database Management Practice
Other Key Data:Data Files: ASMStorage: EMC SymmetricDatabase Flashback: on
High Availability Data Guard Architecture
4
Redo Apply
drdb01-06LTST_DRPhysical Standby
prddb01-06LTSTPrimary
lsdb01-06 LTST_LS Logical Standby
SQL
Ap
ply
Data Center 1 Data Center 2
Database Management Practice
I. Reimage Logical Standby
5
Other Key Data:Data Files: ASMStorage: EMC SymmetricDatabase Flashback: on
Redo Apply
drdb01-06LTST_DRPhysical Standby
prddb01-06LTSTPrimary
lsdb01-06 LTST_LS Logical Standby
SQL
Ap
ply
Data Center 1 Data Center 2
Database Management Practice
Backup Essential Files
1) Backup spfile to pfile› lsts1> create pfile='/u01/app/oracle/product/10.1.0/db_1/dbs/initltst.ora' from spfile;
2) Backup Oracle Home db_1 and save it on a different host› cd $ORACLE_HOME/..
› /u01/app/oracle/product/10.2.0> tar cvf db_1.tar db_1
› /u01/app/oracle/product/10.2.0> scp db_1.tar prdb01:/dbscratch/preReImange/lsdb01/.
3) Capture admin directories› /u01/app/oracle/admin/ltst> ls
adump bdump cdump udump
4) Capture cluster name› grep -i name $ORA_CRS_HOME/install/cluster.ini
cluster_name= sdr2ppramercrs
5) Capture LUN Mapping – by Linux System Engineer
6
Database Management Practice
Reimage
• Shutdown database
• Re-image the host cluster servers to OEL 11g image by Linux System Engineer
› Deploy Oracle Enterprise Linux 64-bit OS
› Restore EMC LUN mapping
› Deploy Oracle 11g CRS RPM
› Deploy Oracle 11g Software RPM
7
Database Management Practice
Start 11g ASM
1) Verify that CRS and Nodeapps are all up› crs_stat.sh
2) Create 11g ASM Initialization file› cp /u01/app/oracle/product/10.2.0/db_1/dbs/init+ASM1.ora /u01/app/oracle/product/11.1.0/asm_1/dbs/.
› remove *_dump_dest from the file
3) Restore password file› cp /u01/app/oracle/product/10.2.0/db_1/dbs/orapw+ASM1 /u01/app/oracle/product/11.1.0/db_1/dbs/.
4) Source ASM environment› export ORACLE_HOME=/u01/app/oracle/product/11.1.0/asm_1
› export PATH=$ORACLE_HOME/bin:/usr/bin:.:/bin:/usr/kerberos/bin:/usr/local/bin:/sbin:/usr/sbin:./
› export LD_LIBRARY_PATH=$ORACLE_HOME/lib
› export ORACLE_SID=+ASM1
5) Startup 11g ASM› sqlplus / as sysasm
› SQL> startup
8
Database Management Practice
Register ASM to CRS
1. Add ASM to CRS› srvctl add asm -n sdr2prddb01 -i +ASM1 -o /u01/app/oracle/product/11.1.0/asm_1
› # Repeat to add all 6 nodes
2. Copy Initialization and Password files to the rest of RAC nodes › init+ASM1.ora
› orapw+ASM1
3. Start ASM on nodes via srvctl› +ASM1 SQL>shutdown
› srvctl start asm –n lsprddb01 #repeat for node 2-6
9
Database Management Practice
Start 10g DB
1) Restore 10g Oracle Home to a temporary directory on node1 only› /dbscratch/> tar xvf db_1.tar
2) Create a symbolic link for the original 10g Oracle Home› mkdir –p /u01/app/oracle/product/10.2.0/
› cd /u01/app/oracle/product/10.2.0/
› ln –s /dbscratch/db_1 db_1
3) Create admin directories › mkdir -p /u01/app/oracle/admin/ltst/
› cd /u01/app/oracle/admin/ltst/
› mkdir adump bdump cdump udump
4) Source 10g environment
› . 10gR2.env
5) Startup 10g Database› sqlplus / as sysdba
› SQL> startup
10
Database Management Practice
Verify Logical Standby logs1. From Logical Standby: Startup 10g Listener
› lsnrctl start LISTENER_LSPRDDB01
2. From Primary: Query the latest online logs › SQL> select THREAD#, SEQUENCE# from v$LOG where status='CURRENT';
THREAD# SEQUENCE#---------- ----------
1 55922 40763 40894 41495 40916 4092
3. From Primary: Force log archive› alter system archive log current;
4. From Logical Standby: Verify that the latest logs are applied› SQL> select THREAD#, MAX(SEQUENCE#), APPLIED from dba_logstdby_log where applied!='YES';
THREAD# MAX(SEQUENCE#) APPLIED---------- -------------- --------
1 5592 YES2 4076 YES3 4089 YES4 4149 YES5 4091 YES6 4092 YES
11
II. Upgrade Logical Standby Database
Database Management Practice
Pre-Upgrade
1) Run pre-upgrade information tools › SQL> @/u01/app/oracle/product/11.1.0/db_1/utlu111i.sql
2) Resolve any outstanding issue
3) Create a guaranteed restore point› SQL> create restore point pre_upgrade guarantee flashback database;
› SQL> select * from v$restore_point;
4) Stop 10g listener › .~ 10gR2.env
› lsnrctl stop
5) Stop SQL Apply and shutdown 10g database› SQL> alter database stop logical standby apply;
› SQL> shutdown
13
Database Management Practice
Start 11g Database
1) Create 11g DB Initialization file› cp /u01/app/oracle/product/10.2.0/db_1/dbs/initltst.ora /u01/app/oracle/product/11.1.0/db_1/dbs/.
› Edit initialization file to
– remove *_dump_dest
– set cluster_database=false
– set dg_broker_start=false
2) Restore password file› cp /u01/app/oracle/product/10.2.0/db_1/dbs/orapwltst1 /u01/app/oracle/product/11.1.0/db_1/dbs/.
3) Restore tnsnames.ora› cp /u01/app/oracle/product/10.2.0/db_1/network/admin/tnsnames.ora
/u01/app/oracle/product/11.1.0/asm_1/network/admin/.
4) Startup 11g database › . .bash_profile
› cd $ORACLE_HOME/rdbms/admin
› SQL> startup upgrade
14
Database Management Practice
Upgrade Database
1. Upgrade database› SQL> spool /tmp/upgrade.log
› SQL> @catupgrd.sql #40 min, db shutdown at then end of upgrade
2. Run post update scripts› ltst1 SQL> startup
› ltst1 SQL> @utlu111s.sql
› ltst1 SQL> @catuppst.sql
› ltst1 SQL> @utlrp.sql
3. Enable cluster database› ltst1 SQL> alter syste set cluster_database=true
4. Restart Instance 1
15
Database Management Practice
Register Database to CRS
1. Add Database and Instances to CRS › srvctl add database -d ltst_ls -o /u01/app/oracle/product/11.1.0/db_1
› srvctl add instance -d ltst_ls -i ltst1 -n sdrseprddb01 # repeat to add all 6 instance
2. Copy following files to the rest of RAC nodes › tnsnames.ora
› initltst1.ora
› orapwltst1
3. Restart DB› ltst1 SQL> shutdown
› srvctl start database –d ltst_ls
16
Database Management Practice
III. Synchronize Archive log (10g->11g)
17
Redo Apply
drdb01-06LTST_DRPhysical Standby
prddb01-06LTSTPrimary
lsdb01-06 LTST_LS Logical Standby
SQL
Ap
ply
Data Center 1 Data Center 2
11g
Database Management Practice
Start Apply
1. Query outstanding log in Logical Standby› col file_name for a70
› select thread#, sequence#, file_name, applied from dba_logstdby_log where applied!= ‘YES'
2. Start log Apply › alter database start logical standby apply immediate;
3. Repeat step 1 until all logs are applied
18
Database Management Practice
IV. Switchover to Logical Standby
19
Redo Apply
drdb01-06LTST_DRBystand Physical Standby
prddb01-06LTSTLogical Standby
lsdb01-06 LTST_LS Primary
SQL
Ap
ply
Data Center 1 Data Center 2
11g
Database Management Practice
Transition Primary -> Logical Standby
1. Verify Switchover status› SQL> select switchover_status from v$database;
SWITCHOVER_STATUS --------------------TO STANDBY
If the query returns “SESSIONS ACTIVE’, then ensure it’s ok to shut them down
› SQL > SELECT SID, PROCESS, PROGRAM FROM V$SESSION WHERE TYPE = 'USER' AND SID <> (SELECT DISTINCT SID FROM V$MYSTAT);
2. Disable all RAC instances except instance 1› srvctl stop instance –d ltst –i ltst2; #repeat for lsts2 - 6
› alter database disable thread 2; ; #repeat for thread 2 - 6
3. Switchover› SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO LOGICAL STANDBY;;
20
Database Management Practice
Transition Logical Standby -> Primary
1. Defer Redo log transfer in Logical Standby› SQL> alter system set
log_archive_dest_3='service="(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=sdrseprddb01-vip.us.dell.com)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=ltst_XPT.prd.amer.dell.com)(INSTANCE_NAME=ltst1)(SERVER=dedicated)))"',' LGWR ASYNC NOAFFIRM delay=0 OPTIONAL max_failure=0 max_connections=1 reopen=30 db_unique_name="ltst" register net_timeout=15 valid_for=(online_logfile,primary_role)' scope=both
› SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_3=DEFER;
2. Disable all RAC instances except instance 1› srvctl stop instance –d ltst_ls –i ltst2; #repeat for instnace 2-6
› alter database disable thread 2; #repeat for thread 2-6
3. Commit Switchover› SQL> select switchover_status from v$database;
SWITCHOVER_STATUS --------------------TO PRIMARY
› SQL> alter database commit to switchover to primary;
21
Database Management Practice
Start Rest of RAC Instances
1. New Primary LTST_LS: Enable threads and start RAC instances 2-6› alter database enable thread 2
› srvctl start instance –d ltst_ls –i ltst2;
2. New Logical Standby LTST: Enable threads and start RAC instances 2-6› alter database enable thread 2;
› srvctl start instance –d ltst –i ltst2;
22
Database Management Practice
V. Reimage Original Primary Hosts
23
Redo Apply
drdb01-06LTST_DRBystand Physical Standby
prddb01-06LTSTLogical Standby
lsdb01-06 LTST_LS Primary
SQL
Ap
ply
Data Center 1 Data Center 2
11g
10g
Database Management Practice
Reimage Original Primary Host
• Follow the process used in reimaging logical standby hosts (slice 7 – 10)
• Do not sync archived log from 11g –> 10g (skip slice 11)
24
Database Management Practice
VI. Upgrade Original Primary to 11g
• Follow the process used in upgrading Logical Standby Database (slice 13–16)
25
Database Management Practice
VII.Synchronize Archive log (11g->11g)
1. Query outstanding log in Logical Standby› col file_name for a70
› select thread#, sequence#, file_name, applied from dba_logstdby_log where applied!= ‘YES'
2. Start log Apply› alter database start logical standby apply immediate;
3. Repeat step 1 until all logs are applied
26
Database Management Practice
VIII.Reimage Physical Standby Hosts
27
Redo Apply
drdb01-06LTST_DRBystand Physical Standby
prddb01-06LTSTLogical Standby
lsdb01-06 LTST_LS Primary
SQL
Ap
ply
Data Center 1 Data Center 2
11g
11g 10g
Database Management Practice
Reimage Physical Standby Hosts
• Follow the process used in reimaging Logical Standby Database (slice 7 –10, skip step 5 in slice 10)
• Mount 10g Database as standby › sqlplus / as sysdba
› SQL> startup monut
› SQL> alter database mount standby database
28
Database Management Practice
Start 11g Physical Standby Database
1) Create 11g DB initialization file› cp /u01/app/oracle/product/10.2.0/db_1/dbs/initltst.ora /u01/app/oracle/product/11.1.0/db_1/dbs/.
› Edit initialization file to remove *_dump_dest
2) Create spfile from pfile
3) Restore password file› cp /u01/app/oracle/product/10.2.0/db_1/dbs/orapwltst1 /u01/app/oracle/product/11.1.0/db_1/dbs/.
4) Restore tnsnames.ora› cp /u01/app/oracle/product/10.2.0/db_1/network/admin/tnsnames.ora
c\/u01/app/oracle/product/11.1.0/asm_1/network/admin/.
5) Mount 11g standby database› . .bash_profile
› SQL> startup nomount
› SQL> alter database mount standby database
Note: wait for dg broker configuration to apply log
29
Database Management Practice
Register Physical Standby Database to CRS
1. Add Database and Instances to CRS › srvctl add database -d ltst_dr -o /u01/app/oracle/product/11.1.0/db_1 –s mount –s physical_standby -m
dr.amer.dell.com
› srvctl add instance -d ltst_dr -i ltst1 -n sdrseprddb01 # repeat to add all 6 instance
2. Copy following files to the rest of RAC nodes › tnsnames.ora
› initltst1.ora
› orapwltst1
3. Restart DB› ltst1 SQL> shutdown
› srvctl start database –d ltst_dr
30
Database Management Practice
IX. Switchover to Original Primary
31
Redo Apply
drdb01-06LTST_DRBystand Physical Standby
prddb01-06LTSTPrimary
lsdb01-06 LTST_LS Logical Standby
SQL
Ap
ply
Data Center 1 Data Center 2
11g
11g 11g
Database Management Practice
Switchover via Data Guard Broker
I. Remove old Data Guard Broker configuration› dgmgrl > remove configuration
II. Create new Data Guard Broker configuration› dgmgrl > create configuration dg_ltst as primary database is ltst_ls connect identifier is ltst_ls;
› dgmgrl > add database ltst as connect identifier is ltst_prd maintained as logical ;
III. Switchover to Logical Standby› dgmgrl > switchover to ltst
IV. Add Physical Standby to Data Guard Broker
32
Database Management Practice
X. Verify Data Guard Status
1) Verify Database Status › dgmgrl > show configuration
2) Verify Log Applying Status in Physical Standby› Run from primary
› @dg_standby_log.sql
3) Verify Log Applying Status in Logical Standby› select thread#, sequence#, file_name, applied from dba_logstdby_log where applied!= 'YES'
33
Database Management Practice
Summary I. Reimage Logical Standby Hosts to 11g – Linux System Engineer
II. Upgrade Logical Standby DB to 11g –Database Administrator
III. Sync Archive log from 10g Primary to 11g Logical Standby - Database Administrator
IV. Switchover to Logical Standby Manually –Database Administrator
V. Reimage original Primary Hosts to 11g – Linux System Engineer
VI. Upgrade the original Primary DB to 11g – Database Administrator
VII. Sync Archive log from 11g Primary to 11g Logical Standby - Database Administrator
VIII. Reimage Physical Standby Hosts to 11g – Linux System Engineer
IX. Switchover to original Primary via DG Broker- Database Administrator
X. Verify Data Guard Status
34
Database Management Practice
Reference
• Oracle® Database Upgrade Guide 11g R1 (11.1) B28300-02
• Oracle® Data Guard Concepts and Administration 11g R1 (11.1) B28294-03
• Oracle Data Guard Broker 11g Release 1 (11.1) B28295)
• Database Rolling Upgrade Using Data Guard SQL Apply - Oracle Database 11g and 10gR2
35
Global Marketing36
For more information:
• Visit Dell at Oracle OpenWorld in Moscone South booth 521
• Visit www.dell.com/oracle for more information
• Dell Services Oracle Practice information at http://content.dell.com/us/en/enterprise/oracle-solutions-services.aspx
• Dell presentations at OOW 2010 at http://www.slideshare.net/dellenterprise
• Dell | Oracle white papers at http://content.dell.com/us/en/enterprise/oracle-solutions.aspx#facets=whitepaper&p=1
• Dell | Oracle Tested and Validated documentation at http://content.dell.com/us/en/enterprise/d/solutions/product-configurations.aspx
Database Management Practice
Questions ?
37
Database Management Practice
Versions - Production
38