Upgrade 10204-to-10205 on-2-node_rac_linux_x86_64_detail-steps_v0.1

10.2.0.5 Patch Set Apply on

Oracle Clusterware,

ASM and RDBMS on

2-Node RAC Linux x86_64

Version: V0.1

By

Raheel H Syed10g OCP CertifiedBlog: http://raheeldba.wordpress.comTwitter: https://twitter.com/raheelhsyed LinkedIn: http://in.linkedin.com/pub/raheel-hashmi-syed/33/34a/226

http://in.linkedin.com/pub/raheel-hashmi-syed/33/34a/226

https://twitter.com/raheelhsyed

http://raheeldba.wordpress.com/

10.2.0.5 PATCH APPLY ON_2-NODE_RAC_STEPS July 10, 2012

Pre-installation Tasks

RACNODE1 and RACNODE2 have separate homes for ASM and Oracle Home and CRS Home. All homes will be patched on both the nodes

Existing CRS_HOME, ASM_HOME & ORACLE_HOME version is 10.2.0.4

+ASM1:/app/asm

PROD1:/app/oracle/10.2.0/db_1

CRS home /app/crs/home_1

1. Backup CRS, ASM, DB Home and inventory for both the nodes

- Use tar to backup the homes as ‘ tar –cvf ’

2. Check opatch lsinventory on CRS, ASM and Oracle Home. Save the output.

3. Check cluster health, crs activeversion and software version on both the nodes and save the output.

$ crsctl check crs

$ crsctl query css activeversionCRS active version on the cluster is [10.2.0.4.0]

$ crsctl query css software versionCRS software version on node [racnode1] is [10.2.0.4.0]

4. Upgrading Installations that use Java Virtual Machine or Oracle interMedia

- If the database uses JVM or Oracle interMedia, then install the oracle database 10g Products installation type from 10g Companion CD before installing 10.2.0.5 Patchset. MOS ID: 293658.1

5. Update Oracle Time Zone Definitions

(A) When patching from a lower 10.2 release or upgrading from 10.1.0.x directly to 10.2.0.5, check the current version of the Oracle time zone definitions with the following command:

SELECT version FROM v$timezone_file;

>>> If this query reports version 4, no action is required.

Raheel H Syed 2


>>> If this reports a version lower or higher than 4, see My Oracle Support document 1086400.1 Actions for the DSTv4 update in the Release 10.2.0.5 patchset.

(No Action required in our case as version is 4)

6. chmod -R +w CRS_HOME/inventory/Templates/*

7. Save all db links - Incase downgrade is required you can drop the db links and recreate after if downgrade is required

spool dblinks_pre_upgrade.lstSELECT'create '||DECODE(U.NAME,'PUBLIC','public ')||'database link '||CHR(10)||DECODE(U.NAME,'PUBLIC',Null, U.NAME||'.')|| L.NAME||chr(10)||'connect to ' || L.USERID || ' identified by '''||L.PASSWORD||''' using ''' || L.host || ''''||chr(10)||';' TEXTFROM sys.link$ L,sys.user$ UWHERE L.OWNER# = U.USER#;spool off

8. Check invalid objects

spool invalid_pre_upgrade.lstselect substr(owner,1,12) owner,substr(object_name,1,30) object,substr(object_type,1,30) type, status fromdba_objects where status <> 'VALID';spool off

Run utlrp.sql and re-query invalid objects

The last queries (invalid objects check) will lists all objects that cannot be compiled before upgrade in file 'invalid_pre.lst'

9. Create pfile from spfile AND backup tnsnames.ora and listener.ora

10. Check AUD$ table, if auditing is enabled then upgrade performance will be affected

Raheel H Syed 3


SELECT count (*) FROM SYS.AUD$;

count (*)--------- 0

MOS Id: 979942.1

(No action required in our case as NO auditing is enabled)

11. Verify database components, status and version from dba_registry. Save the output.

set pages 500set lines 120col COMP_NAME for a40select comp_name, status, version from dba_registry;

12. Backup the database. Using RMAN, backup the full database and archive log in compressed format to disk. Though we had tape backups configured, we opted for disk backups as backup completion time to tapes varied to 4-6 hours. Disk backup completed in 2 hours.

#!/usr/bashexport NLS_DATE_FORMAT="DD-MON-YY HH24:MI:SS"rman target / log /tmp/before_upgrade_full_db.log<<EOFrun{allocate channel ch001 device type disk;allocate channel ch002 device type disk;allocate channel ch003 device type disk;allocate channel ch004 device type disk;BACKUP AS COMPRESSED BACKUPSET DATAFILE 1,2,3,4,5,22,23,24 format '/app/prod1/before_upgrade/%d_%s_%p_%c_%t.rmn';BACKUP AS COMPRESSED BACKUPSET DATAFILE 25,26,28,29,31,32,33,34 format '/app/prod1/before_upgrade/%d_%s_%p_%c_%t.rmn';BACKUP AS COMPRESSED BACKUPSET DATAFILE 35,36,37,39,40,42,43,44 format '/app/prod1/before_upgrade/%d_%s_%p_%c_%t.rmn';BACKUP AS COMPRESSED BACKUPSET DATAFILE 45,46,47,48,49,50,51,52 format '/app/prod1/before_upgrade/%d_%s_%p_%c_%t.rmn';BACKUP AS COMPRESSED BACKUPSET DATAFILE 53,54,55,56,58,59,60,61 format '/app/prod1/before_upgrade/%d_%s_%p_%c_%t.rmn';BACKUP AS COMPRESSED BACKUPSET DATAFILE 62,63,64,65,66,67,68,69 format '/app/prod1/before_upgrade/%d_%s_%p_%c_%t.rmn';

Raheel H Syed 4


BACKUP AS COMPRESSED BACKUPSET DATAFILE 70,71,72,73,74,75,76,77 format '/app/prod1/before_upgrade/%d_%s_%p_%c_%t.rmn';BACKUP AS COMPRESSED BACKUPSET DATAFILE 78,79,80,81,82,83,84,85 format '/app/prod1/before_upgrade/%d_%s_%p_%c_%t.rmn';BACKUP AS COMPRESSED BACKUPSET DATAFILE 6,7,8,9,10,11,12,13 format '/app1/prod1/before_upgrade/%d_%s_%p_%c_%t.rmn';BACKUP AS COMPRESSED BACKUPSET DATAFILE 14,15,16,17,18,19,20,21 format '/app1/prod1/before_upgrade/%d_%s_%p_%c_%t.rmn';BACKUP AS COMPRESSED BACKUPSET DATAFILE 27,30,38 format '/app1/prod1/before_upgrade/%d_%s_%p_%c_%t.rmn';backup as backupset format '/app1/prod1/before_upgrade/%d_%s_%p_%c_%t.ctl' current controlfile tag='Before_upgrade_CTL';sql 'alter system archive log current';sql 'alter system archive log current';sql 'alter system archive log current';BACKUP AS COMPRESSED BACKUPSET format '/app1/prod1/before_upgrade/%d_%s_%p_%c_%t.arc' archivelog all tag='Before_upgrade_ARCH';backup as backupset format '/app1/prod1/before_upgrade/%d_before_upgrade_AFTER_ARCH_%s_%p_%c_%t.ctl' current controlfile tag='Before_upgrade_CTL';release channel ch001;release channel ch002;release channel ch003;release channel ch004;}quitEOF

Note: I have explicitly specified datafiles for each compressed backupset, this is because there was not enough space at /app mount point.

Database size 1230 GB and compressed backup size on disk 139 GB.

Upgrade Clusterware1. Resolve any outstanding unresolved distributed transactions:

SQL> select * from dba_2pc_pending;

If this returns rows then run the following

SQL> select local_tran_id from dba_2pc_pending;SQL> execute dbms_transaction.purge_lost_db_entry ('<input_id>'); SQL> commit;

Raheel H Syed 5


2. Disable the jobs that run from scheduler before upgrading the database.

Downtime starts here.

Note: In our case all the production jobs were scheduled from DBMS_SCHEDULER hence we have disabled the jobs.

3. Stop database, ASM and nodeapps on both the nodes.

$ srvctl stop database –d <> $ srvctl stop asm –n node1$ srvctl stop asm –n node2$ srvctl stop nodeapps –n node1$ srvctl stop nodeapps –n node2

4. Download and unzip patch p8202632_10205_Linux-x86-64.zip

5. We have used VNC viewer to install this patch set, most of the Linux system comes with vnc service running on the system. Please check with your SA if it’s not running. To check if vnc service is running on the system or not, issue the following command

$ ps –ef | grep vnc

If the output shows …… oracle ….. <your hostname>:<number> then it means you have vnc session running under oracle user, eg: racnode1:1 or racnode1:2

If VNC service is not running then you can do this, it worked for me

As ‘Oracle’ user:

$ cd /usr/bin

$ ls –lrt vnc*

$ ./vncserver

It will ask for password, provide the password and verify.

6. CD to Disk1 and Invoke runInstaller and select CRS HOME from the path. Follow the instructions as displayed on the screen.

OR

Execute the following to ignore pre-requisite checks of parameters (If required)

$ ./runinstaller –ignoreSysPrereqs

Raheel H Syed 6


(A)

(B) Select CRS_HOME

Raheel H Syed 7


(C) Click next by default all the nodes in cluster are selected as below

(D) Warning: Backup /etc/sysctl.conf and modify fs.file_max=101365 (see below)

Raheel H Syed 8


(E)

(F) stop crs as ROOT user; As ROOT user run CRS_HOME/install/root102.sh

Raheel H Syed 9


7. Issues faced during upgrade:-

- Prerequisite check failed as fs.file-max = 655367, required value 101365.

Asked SA Admin to backup the /etc/sysctl.conf and modify the parameter.

- Error in writing to CRS_Home:/app/oracle/product/10.2.0/crs/OPatch

Change permission as $ chmod +w opatch

8. The script will automatically start crs daemon on the patched node upon completion

9. crs_stat -t (or) crs_status -t on completed node

10. Proceed to node2 and repeat.

This completes CRS upgrade to 10.2.0.5

11. Check crs softwareversion and activeversion on both the nodes. On successful upgrade crs softwareversion and activeversion should be at 10.2.0.5

$ crsctl check crs softwareversion

Raheel H Syed 10


$ crsctl check crs activeversion

Upgrade ASM and Oracle Home

1. Stop database, ASM and nodeapps on both the nodes.

$ srvctl stop database –d <>$ srvctl stop asm –n node1 $ srvctl stop asm –n node2$ srvctl stop nodeapps –n node1$ srvctl stop nodeapps –n node2

2. Invoke runInstaller and select ASM Home (ASM is separate home so we will have to upgrade ASM home first and then ORACLE_HOME.

(B) Select ASM home

Raheel H Syed 11


Click Next

Raheel H Syed 12


Click Next

Click OK and

Raheel H Syed 13


Run the script root.sh on node1 and then on node 2 as root user click OK

Click Exit

Raheel H Syed 14


3. Error during upgrade:-

- When OUI was trying to copy files in /app/oracle/product/10.2.0/db_1/jdk/bin/java (Text file busy)

I have closed and started the vncviwer, that resolved the issue

- Oracle Configuration Manager Configuration failed with oracle.sysman.ccr.configCCR.ConfigCCRPlugIn. Skip this step.

4. Execute root.sh as root on first node and then on second node.

5. This completes ASM Home upgrade to 10.2.0.5

6. $ASM_HOME/OPatch/opatch lsinventory. Save the output.

7. Once the ASM Home is upgraded, invoke runInstaller and select ORACLE HOME to upgrade ORACLE HOME.

8. $ORACLE_HOME/OPatch/opatch lsinventory. Save the output

Upgrade Oracle Database

1. Start ASM on both the nodes.

$ srvctl start asm -n node1

$ srvctl start asm -n node2

2. srvctl start instance -d DB -i DB1 -o nomount

3. SQL> ALTER SYSTEM SET CLUSTER_DATABASE=FALSE SCOPE=spfile;

4. Create pfile from spfile.

Edit pfile and comment all obsolete parameters. (Eg: sql_trace)

5. srvctl stop instance -d DB -i DB1

My advice – Run the below steps 6 & 7 in vnc, even in worst case if connectivity is lost your vnc session is still running in background

Raheel H Syed 15


6. As ‘Oracle’ User, check or set $ORACLE_HOME, $ORACLE_SID, $PATH

$ sqlplus “/as sysdba”

7. SQL> startup upgradeSQL> spool upgrade_10205.logSQL> @?/rdbms/admin/catupgrd.sqlSQL> spool offSQL> shut immediateSQL> startup

SQL> @?/rdbms/admin/utlrp.sql

8. SQL> SELECT COMP_NAME, VERSION, STATUS FROM SYS.DBA_REGISTRY;

9. SQL> shut immediate;

10. $ srvctl start instance -d DB -i DB1 -o nomount

11. SQL> ALTER SYSTEM SET CLUSTER_DATABASE=TRUE SCOPE=spfile;

12. $ srvctl stop instance -d DB -i DB1

13. $ srvctl start database -d DB $ srvctl start nodeapps -n node1$ srvctl start nodeapps -n node2

14. Health check on crs. $ sh crs_status.sh

Apply PSU patch 13343471 - 10.2.0.5.6 to Oracle Home

1. PRODUCTION nodes RACNODE1 and RACNODE2

2. Have 2 separate homes for ASM and Oracle Home.

3. Apply OPatch patch 6880880 to apply 10.2.0.5.6.

Raheel H Syed 16


- Backup the existing OPatch directory and then unzip p6880880_102000_Linux-x86-64_Opatch_4_PSUJan2012.zip in the Oracle Home

- $opatch version

- Output should be like this: 10.2.0.5.1

4. Stop database, asm, listener and nodeapps on all the nodes

$ srvctl stop database –d <>$ srvctl stop asm –n node1$ srvctl stop asm –n node2$ srvctl stop nodeapps –n node1$ srvctl stop nodeapps –n node2

Before applying patch please read “Patch 13343471 - 10.2.0.5.6 Patch Set Update” readme.html

5. unzip p13343471_10205_Linux-x86-64_PSU_JAN2012.zip

$ cd 13343471$ opatch apply

6. start database and listener.

7. cd $ORACLE_HOME/rdbms/admin

8. @catbundle.sql psu apply

9. @utlrp.sql

10. set pages 500select * from dba_registry_history;

Check the following log files in $ORACLE_HOME/cfgtoollogs/catbundle for any errors:

catbundle_PSU_<database SID>_APPLY_<TIMESTAMP>.log

catbundle_PSU_<database SID>_GENERATE_<TIMESTAMP>.log

11. Quick CRS check with custom scirpt: crs_status.sh

Raheel H Syed 17


12. Enable all the jobs that were disabled in earlier step.

13. Running changePerm.sh Script on an Oracle Database Server Home

14. $ cd $ORACLE_HOME/install$ sh changePerm.sh and specify the patched server Oracle home location

References

Oracle® Database Patch Set Notes 10g Release 2 (10.2.0.5) Patch Set 4 for Linux x86-64 Part Number E15229-01 - April 2010

Complete Checklist for Manual Upgrades to 10gR2 [ID 316889.1]

10.2.0.5 Patch Set - List of Bug Fixes by Problem Type [ID 1088172.1]

10.2.0.5 Patch Set - Availability and Known Issues [ID 1087991.1]

How to Log Good Service Requests for Database Upgrade/Migration Issues [ID 732897.1]

How to Check Oracle JVM is Installed in the Database [ID 397770.1]

Raheel H Syed 18

Upgrade 10204-to-10205 on-2-node_rac_linux_x86_64_detail-steps_v0.1

Technology

Transcript of Upgrade 10204-to-10205 on-2-node_rac_linux_x86_64_detail-steps_v0.1