FAQ OPatch Patch Questions Issues for Oracle Clusterware (Grid Infrastructure or CRS) and RAC...

10/14/13 Document Display

https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrl-state=1duz70cfo6_4 1/22

FAQ: OPatch/Patch Questions/Issues for Oracle Clusterware (Grid Infrastructure or CRS) andRAC Environments (Doc ID 1339140.1)

Modified: 28-Sep-2013 Type: FAQ

In this Document

Purpose

Questions and Answers

Oracle Home Type

What type of home do I have?

Types of Patches

What Types of Patches are available for Oracle Clusterware, Grid Infrastructure and/or theDatabase?

Patchsets Q&A

What's a Patchset?

Patch Set Updates (PSUs) Q&A

What's a Patch Set Update (PSU)?

What's the PSU release schedule?

Will the 5th digit of the version be changed after PSU is applied?

What's included in a GI PSU ?

Can a Database PSU be applied to a clusterware home?

Critical Patch Updates (CPUs) Q&A

What are Critical Patch Updates (CPUs)?

Bundle Patches Q&A

What's the difference between a Clusterware/Grid Infrastructure bundle patch and a PSU?

Interim (one-off) Patch Q&A

What's an interim patch (one-off patch)?

General Patch Q&A

What's the difference between Clusterware/Grid Infrastructure patches and Database patches?

What does a Clusterware/Grid Infrastructure patch contain?

Which Patch Applies to Which Home

What's Oracle's patching recommendation?

Which oracle home does a patch apply to?

How do I tell from the patch readme which home the patch applies to?

Do Exadata patches apply to GI or RAC homes?

Can I upgrade the Clusterware or Grid Infrastructure to a higher version while leaving database at a lowerversion?

Do I need downtime to apply a Clusterware or Grid Infrastructure patch?

Which PSU patch applies to what home in mixed environments (clusterware and database at differentversion)?

OPatch Q&A

How to find out the opatch version?

How do I install the latest OPatch release?



Why is "opatch auto" not patching my RAC database home?

What's the difference between manual opatch apply and opatch auto?

How do I apply a Grid Infrastructure patch before the root script (root.sh or rootupgrade.sh) is executed?

How to apply a patch after the root script (root.sh or rootupgrade.sh) has failed?

How to apply a Clusterware or Grid Infrastructure patch manually?

Examples

OPatch Auto Example to Apply a GI PSU (includes Database PSU)

EXAMPLE: Apply a CRS patch manually

EXAMPLE: Applying a GI PSU patch manually

Common OPatch Failures and Troubleshooting

What files to review if opatch auto fails?

"opatch apply" or "opatch napply" failure if clusterware home is not unlocked

OPatch reports: "Prerequisite check CheckSystemSpace failed"

Common causes of "The opatch version check failed"

OPatch reports: "Patch nnn requires component(s) that are not installed in OracleHome"

Common causes of "The opatch Component check failed"

Common causes of "The opatch Conflict check failed"

Common causes of "The opatch Applicable check failed"

Common causes of "patch <patch-loc> apply failed for home <ORACLE_HOME>"

When applying online patch in RAC: Syntax Error... Unrecognized Option for Apply .. OPatch failed with errorcode 14

opatch Fails to Rollback Online(Hot) Patch in RAC With oracle/ops/mgmt/cluster/NoSuchNodeException anderror code 30

opatch Fails to Rollback Online(Hot) Patch With Prerequisite check "CheckRollbackSid" and error code 30

Common causes of "defined(@array) is deprecated at crs/crsconfig_lib.pm"

opatch auto Reports: The path "<GRID_HOME>/bin/acfsdriverstate" does not exist

Applying [PSU] patch takes very long time (hours) after "verifying the update"

New OPatch Features

References

APPLIES TO:

Oracle Database - Enterprise Edition - Version 10.1.0.2 and laterInformation in this document applies to any platform.@

PURPOSE

This note lists the top opatch or patch related questions or problems in Oracle Clusterware (11gR2 Grid Infrastructureor pre-11.2 CRS) and RAC environments.

It does not intend to replace the readme that comes with a patch, rather it's recommended to go through the patch



readme thoroughly and follow it. Occasionally a patch readme may contain incorrect information, in that case, pleaseengage Oracle Support to seek clarification.

For general opatch questions or problems, refer to Document 293369.1

QUESTIONS AND ANSWERS

Oracle Home Type

What type of home do I have?

In a pre-11.2 Clusterware environment you will have:

A single Clusterware (CRS) home One or more Database (RDBMS) homes at the same or at a lower release level than the Clusterware home (tothe 4th dot in the release) Optionally a separate dedicated RDBMS home to run ASM (Automatic Storage Management) at the same or ata lower release level than the Clusterware home (to the 4th dot in the release)

In pre-11.2 installations, ASM was implemented using the RDBMS binary installation so this should betreated as a typical RDBMS home

In a 11.2 Grid Infrastructure environment you will have:

A single Grid Infrastructure (GI) Home that implements both ASM and the Clusterware

Note: During upgrade from pre-11gR2 to 11gR2 GI, the ASM upgrade can optionally be performedindependently of the Clusterware (not recommended). In this case the pre-11gR2 ASM Home would still beactive after the GI Upgrade. This is NOT the recommended method of upgrading to Grid Infrastructure andwill therefore not be discussed in this note.

Note: Starting with 11gR2 all upgrades of and to Grid Infrastructure MUST be performed "out-of-place"(into a new software home) for this reason there may be more than one clusterware home, however onlyone will be the active. This "inactive" Clusterware or GI home should be removed once satisfied that theupgrade was successful.

One or more database (RDBMS) homes at the same or at a lower release level than the GI home (to the 4thdot in the release)

Types of Patches

What Types of Patches are available for Oracle Clusterware, Grid Infrastructure and/or the Database?

Generally patches for Clusterware, Grid Infrastructure and/or the Database are catagorized into the following:

PatchsetsPatchset Updates (PSUs)Critical Patch Updates (CPUs)Bundle PatchesInterim (one-off) Patches

Patchsets Q&A

https://support.oracle.com/epmos/faces/DocumentDisplay?id=293369.1



What's a Patchset?

Compared to all other patch types, a Patchset is released the least frequently. It contains fixes for most known issuesfor the release and potentially also introduces new features. A patchset is cumulative and when applied it changes thefourth digit of the product release banner - for example, 10.2.0.5 is 4th patch set for 10.2, and 11.2.0.2 is the 1st patchset for 11.2.

A patchset must be installed via the Oracle Universal Installer (OUI) and is generally considered an "upgrade".

Prior to 11gR2, a base release had to be installed before a patchset could be applied. For example, to install 10.2.0.5on Linux, the 10.2.0.1 base release has to be installed first and then upgraded to 10.2.0.5.

Prior to 11gR2, the same patchset download is used to patch the Clusterware, ASM and Database homes. Forexample, Patch 8202632 is the 10.2.0.5 patchset, this same patch (Patch 8202632) will be used to patch the 10.2.0.xClusterware, 10.2.0.x ASM and 10.2.0.x Database to 10.2.0.5.

Starting with 11gR2, patchset releases are now full releases and no longer require a "base" release, e.g. 11.2.0.2 canbe installed directly without having to install 11.2.0.1 first.

Prior to 11gR2 - even though the CRS and RDBMS base releases were provided on separate media (downloadable zipfile or separate DVD/CD) - the patchsets for both products were delivered as one i.e. the same patchset could beapplied to the CRS as well as the RDBMS home.

Starting with 11gR2 the patchsets for Grid Infrastructure and RDBMS are delivered separately (as they are fullreleases).

Clusterware patchsets can be applied in a rolling fashion, while database patchsets cannot. For example, you canrolling upgrade clusterware to 11.2.0.2, but you have to shutdown database on all nodes to upgrade to database11.2.0.2.

Patch Set Updates (PSUs) Q&A

What's a Patch Set Update (PSU)?

As the name implies PSUs are patches that are applied on top of a given patchset release. They are released on aquarterly basis and contain fixes for known critical issues for the patchset. PSUs are subject to thorough testing and donot include changes that would alter the functionality of the software. With this in mind, PSUs are designed to provide"low risk" and "high value" allowing for customers to more easily adopt proactive patching strategies. Consider thefollowing PSU facts:

All PSUs are installed via "opatch" and are not considered an "upgrade".Database PSUs always contain the CPU for the respective quarter that the PSU is released in. PSUs and CPUsare NOT compatible meaning if you apply the 11.2.0.2.2 Database PSU and then want to apply the 11.2.0.2 JulyCPU this would result in the rollback of the 11.2.0.2.2 Database PSU. That said, once a PSU patching strategy isadopted it must be maintained.

Independent PSUs are released for both the Database and Clusterware or Grid Infrastructure installations.Clusterware PSUs (pre-11.2) are refered to as CRS PSUsGrid Infrastructure PSUs are refered to as GI PSUs

GI PSUs do contain the Database PSU for the corresponding release, e.g. 11.2.0.2.3 GI PSUcontains the 11.2.0.2.3 Database PSU

Database PSUs hold true to their name

Both Clusterware/Grid Infrastructure and Database PSU patches are cumulative. Clusterware PSU refers to CRSPSU for pre-11gR2 and GI PSU for 11gR2.



GI PSUs are always cumulative meaning that you can apply higher version GI PSU directly without having toapply a lower version one first. For example, the 11.2.0.2.2 GI PSU can be applied to a 11.2.0.2 home withouthaving to apply GI PSU 11.2.0.2.1 first.Database PSUs can be subject to overlay PSU packaging. In these cases, the PSUs are still cumulative, but ahigher PSU may require a lower PSU to be applied first; for example, to apply database PSU 10.2.0.4.7, youmust apply database PSU 10.2.0.4.4 first. If a previous PSU is a prerequisite to a later PSU the requirement willbe clearly documented in the PSU readme.For more information on PSUs please review Document 854428.1.

What's the PSU release schedule?

Generally speaking PSU are released on quarterly basis for both Clusterware/Grid Infrastructure and Database. There's cases where a Clusterware PSUs is not released for corresponding Database PSU. For example, there'sdatabase PSU 10.2.0.5.4 but no CRS PSU 10.2.0.5.4.

Will the 5th digit of the version be changed after PSU is applied?

A PSU will not physically update the 5th-digit of the release information, the updates to the 5th digit in the release arefor Documentation purposes only. So the third GI PSU that was released for 11.2.0.2 will have a documentation versionof 11.2.0.2.3. You will NOT see this change reflected in the actual software version if you query it from the inventory,clusterware or database.

What's included in a GI PSU ?

Unlike other Grid Infrastructure patches (discussed later), 11gR2 GI PSUs contains both GI PSU and Database PSU(YES, both GI and DB PSU) for a particular quarter. For example, 11.2.0.2.2 GI PSU contains both the 11.2.0.2.2 GI PSUand the 11.2.0.2.2 Database PSU.

You are able to take note of this when you extract a GI PSU, you will see 2 directories (named with the Patch number)one for the GI PSU and one for the RDBMS PSU.

How do I find out whether a bug is fixed in a Clusterware or Grid Infrastructure PSU ? To find out, check the patchreadme and the following notes:

Document 405820.1 - 10.2 CRS PSU Known issuesDocument 810663.1 - 11.1 CRS PSU Known issuesDocument 1082394.1 - 11.2.0.1 GI PSU Known issuesDocument 1272288.1 - 11.2.0.2 GI PSU Known Issues

Once the GI PSU is applied, "opatch lsinventory" will show that both GI PSU and DB PSU are applied, i.e.:

Interim patches (2) :

Patch 9654983 : applied on Thu Feb 02 20:36:47 PST 2012Patch 9655006 : applied on Thu Feb 02 20:35:53 PST 2012

And "opatch lsinventory -bugs_fixed" will list each individual bugs that have been fixed by all installed patches, i.e.:








List of Bugs fixed by Installed Patches:

Bug Fixed by Installed at Description Patch--- -------- ------------ -----------

7519406 9654983 Thu Feb 02 20:36:47 PST 2012 'J000' TRACE FILE REGARDING GATHER_STATS_JOB INTER..9783609 9655006 Thu Feb 02 20:35:53 PST 2012 CRS 11.2.0.1.0 Bundle

Can a Database PSU be applied to a clusterware home?

No, only CRS PSUs, GI PSUs or other Clusterware/GI patches can be applied to a Clusterware/GI home.

Critical Patch Updates (CPUs) Q&A

What are Critical Patch Updates (CPUs)?

CPU Patches are collection of high-priority fixes for security related issues and are only applicable to the DatabaseHome (and pre-11.2 ASM Home(s)) . CPUs are released quarterly on the same cycle as PSUs and are cumulative withrespect to prior security fixes but may contain other fixes in order to address patch conflicts with non-security patches. PSUs always contain the CPU for that respective quarter. PSUs and CPUs are NOT compatible meaning if you apply the11.2.0.2.2 Database PSU and then want to apply the 11.2.0.2 July CPU this would result in the rollback of the 11.2.0.2.2Database PSU. That said, once a PSU patching strategy is adopted it must be maintained. PSU patching is preferredover CPU Patching.

Bundle Patches Q&A

What's the difference between a Clusterware/Grid Infrastructure bundle patch and a PSU?

A Clusterware or Grid Infrastructure (GI) patch can be in the form of bundle or Patchset Update (PSU). The biggestdifference between a GI/Clusterware bundle and a PSU is that PSUs are bound to a quarterly release schedule while abundle may be released at any time throughout the course of a given quarter. If a GI/Clusterware bundle is releasedin a given quarter, the fixes in that bundle will be included in the PSU that will be released for that quarter. Thisconcept allows for development to provide critical bug fixes in a more granular timeline if necessary.

Interim (one-off) Patch Q&A

What's an interim patch (one-off patch)?

An interim patch contains fixes for one or in some cases several bugs (merge patch).

Clusterware interim patches are rare, they usually are build on top of the latest PSU (at the time) and include theentire PSU they were built on.

The same does not hold true for database interim patches, they usually do not include a PSU.



An interim clusterware patch on top of a PSU includes the PSU, for example, 11.2.0.2.2 patch 12647618 includes11.2.0.2 GI PSU2 (11.2.0.2.2). But the same is not true for database interim patch, for example, 11.2.0.2.2 databasepatch 11890804 can only be applied on top of a 11.2.0.2.2 database home.

General Patch Q&A

What's the difference between Clusterware/Grid Infrastructure patches and Database patches?

Generally speaking Clusterware/Grid Infrastructure patches modify files that belong to the Clusterware or GridInfrastructure product, while Database patches change files that belong to the database product. As they apply todifferent sets of files, they do not conflict with each other.

Please note:

"files" in this context can refer to binaries/executables, scripts, libraries etc Clusterware files can reside in all types of Oracle software homes like clusterware home, database home andASM home Prior to 11gR2, RDBMS files reside in DB/ASM homes only, while with 11gR2 RDBMS files will also reside in theGI home (as ASM is now part of GI) A GI PSU is a special type of clusterware patch as it also includes a database PSU and modifies databasebinaries.

What does a Clusterware/Grid Infrastructure patch contain?

Clusterware and Grid Infrastructure patches have at least two portions, one for the clusterware home, and the otherfor the Database home(s). Those two portions have same version number. The clusterware home portion is in the toplevel directory of the unzipped location, while the other portion is in custom/server/<bug#>. For example, if11.2.0.2.2 GI PSU is unzipped to /op/db/11.2.0.2/GIPSU2, Grid Infrastructure home portion will be in/op/db/11.2.0.2/GIPSU2/12311357 and database home specific portion will be in/op/db/11.2.0.2/GIPSU2/12311357/custom/server/12311357. This is just an example, full details will be in theREADME for the patch that is being applied. ALWAYS consult the patch README prior to applying any patch.

Which Patch Applies to Which Home

What's Oracle's patching recommendation?

Oracle recommends to apply the latest patchset and latest PSU as a general best practice.

Which oracle home does a patch apply to?

A home must meet the patch version specification to be applicable.

In CRS environments, a Clusterware patch (interim, MLR, bundle or PSU) applies to all homes (CRS, ASM anddatabase), while a Database patch applies to the ASM and Database home only

In Grid Infrastructure environments, a GI patch (interim, bundle or PSU) applies to all homes (GI and Database), whilea Database patch could be applicable to GI/ASM home if the fix applies to ASM (in this case patch README will tellclearly that it applies to GI/ASM home).

How do I tell from the patch readme which home the patch applies to?

The patch README usually tells which home it applies to.

https://support.oracle.com/epmos/faces/ui/patch/PatchDetail.jspx?patchId=12647618




Common identifier in patch README:

# Product Patched : ORCL_PF ==> Applies to database and pre-11.2 ASM home# Product Patched : RDBMS ==> Applies to database and pre-11.2 ASM home# Product Patched : CRS/RDBMS ==> Applies to both clusterware and database homes

Oracle Database 11g Release 11.2.0.2.3 ==> Applies to database home.

Oracle Clusterware 11g Release 2 (11.2.0.2.2) ==> Applies to both clusterware and databasehomes

Do Exadata patches apply to GI or RAC homes?

General speaking, Exadata patches should only be applied to Exadata environments.

Can I upgrade the Clusterware or Grid Infrastructure to a higher version while leaving database at a lowerversion?

Yes, as long as Clusterware/Grid Infrastructure is at higher version than the RAC Database homes on the cluster this isperfectly acceptable, refer to Document 337737.1 for details

Do I need downtime to apply a Clusterware or Grid Infrastructure patch?

If the Clusterware/Grid Infrastructure home is not shared (common), Clusterware/Grid Infrastructure patches can beapplied in rolling fashion so no downtime (of the entire cluster) is needed.

Which PSU patch applies to what home in mixed environments (clusterware and database at differentversion)?

Example 1:

11.2.0.2 GI + 11.2.0.2 DB, 11.1.0.7 DB and 10.2.0.5 DB

In this environment, 11.2.0.2 GI PSU applies to both 11.2.0.2 GI and 11.2.0.2 DB home(DB PSUdoes not apply to any home), 11.1.0.7 CRS PSU and 11.1.0.7 database PSU applies to 11.1.0.7DB home, and 10.2.0.5 CRS PSU and 10.2.0.5 database PSU applies to 10.2.0.5 DB home.

Example 2:

11.1.0.7 CRS + 11.1.0.7 CRS, 10.2.0.5 DB

In this environment, 11.1.0.7 CRS PSU applies to 11.1.0.7 CRS home, 11.1.0.7 CRS PSU and11.1.0.7 database PSU applies to 11.1.0.7 database home, 10.2.0.5 CRS PSU and 10.2.0.5 DBPSU applies to 10.2.0.5 DB home.

OPatch Q&A

How to find out the opatch version?

To find out the opatch version perform the following as the Oracle software owner:




% export ORACLE_HOME=<home-path>

% $ORACLE_HOME/OPatch/opatch version

How do I install the latest OPatch release?

Prior to applying a patch to a system it is HIGHLY recommended (often required) to download and install the latestversion of the OPatch utility into every ORACLE_HOME on every cluster node that is to be patched. The latest versionof OPatch can be found on MOS under Patch 6880880. Be sure to download the release of OPatch that is applicable foryour platform and major release (e.g. Linux-x86-64 11.2.0.0.0). Once OPatch has been downloaded and staged onyour target system(s) it can be installed by executing the following as the Oracle Software installation owner:

% export ORACLE_HOME=<home-path>

% unzip -d $ORACLE_HOME p6880880_112000_Linux-x86-64.zip

Why is "opatch auto" not patching my RAC database home?

Why is "opatch auto" not patching my RAC database home, refer to note 1479651.1

What's the difference between manual opatch apply and opatch auto?

In a single instance environment to apply database patch it is very simple. Shutdown all processes running from ahome to be patched and invoke "opatch apply" or "opatch napply". Very simple.

Now, to apply a Clusterware patch to a Clusterware Home or a GI patch to Grid Infrastructure Home, the clusterwarestack needs to be down, the clusterware home needs to be unlocked, the configuration needs to be saved, then thepatch can be applied; once the patch is applied, the configuration needs to be restored, the clusterware home needs tobe re-locked again, then the clusterware stack can start. Once the Clusterware patch or GI patch was applied to theClusterware/GI home, you then have to go and apply the database components of that patch to the Database Homewith a whole other list of steps. This is a complex process that had proven to be error prone if instructions were notcarefully followed.

This is where OPatch Auto comes in. The whole goal of OPatch auto is for an administrator to only have to execute asingle command to apply a patch to a RAC system, thus simplifying the process.

How do I apply a Grid Infrastructure patch before the root script (root.sh or rootupgrade.sh) is executed?

Refer to note 1410202.1 for details.

How to apply a patch after the root script (root.sh or rootupgrade.sh) has failed?

In some cases root script fails because of missing patch (e.g. Document 1212703.1 Oracle Grid Infrastructure 11.2.0.2Installation or Upgrade may fail due to Multicasting Requirement).

When this happens we must first determine whether the root script had made it to the point in which theownership/permissions of the GI software has changed. This will allow us to make the determination if unlocking theGI software is necessary prior to installing the patch. To determine if unlocking is required, go to the GI home and







execute:

# find $GRID_HOME -user root

If the above command returns no files, unlocking the software is not necessary. Should the command return rootowned files you must unlock the GI Home by executing the following as the root user:

# $GI_HOME/perl/bin/perl $GI_HOME/crs/install/rootcrs.pl -unlock

Once the GI home is unlocked (or it was determined that unlocking was not necessary, refer to answer for the question"How do I apply a Grid Infrastructure patch before the root script (root.sh or rootupgrade.sh) is executed?" to installthe required patch.

How to apply a Clusterware or Grid Infrastructure patch manually?

It's recommended to use OPatch auto when applicable; but in the case that OPatch auto does not apply or fails to applythe patch, refer to patch README and the following notes for manual patch instructions. The following notes may alsobe of use when in this situation:

Document 1089476.1 - Patch 11gR2 Grid Infrastructure Standalone (Oracle Restart)

Document 1064804.1 - Apply Grid Infrastructure/CRS Patch in Mixed environment

Examples

OPatch Auto Example to Apply a GI PSU (includes Database PSU)

Platform is Linux 64-bitAll Homes (GI and Database) are not sharedAll Homes are 11.2.0.2The Grid Infrastructure owner is gridThe Database owner is oracleThe Grid Infrastructure Home is /ocw/gridDatabase Home1 is /db/11.2/db1Database Home2 is /db/11.2/db2The 11.2.0.2.3 GI PSU will be applied to in this exampleACFS is NOT in use on this cluster

Note: This example only covers the application of the patch iteslf. It does NOT cover the additional database,ACFS or any other component specific steps required for the PSU installation. That said you must ALWAYS consultthe patch README prior to attempting to install any patch.

1. Create an EMPTY directory to stage the GI PSU as the GI software owner (our example uses a directory namedgipsu3):

% mkdir /u01/stage/gipsu3





Note: The directory must be readable, writable by root, grid and all database users.

2. Extract the GI PSU into the empty stage directory as the GI software owner:

% unzip -d /u01/stage/gipsu3 p12419353_112020_Linux-x86-64.zip

3. Verify that opatch in ALL 11.2.0.2 homes that will be patched meet minimum version requirement documented in thePSU README (see "How to find out the opatch version?"). If the version of OPatch in any one (or all) of the homesdoes not meet the minimum version required for the patch, OPatch must be upgraded in this/these homes prior tocontinuing (see "How do I install the latest OPatch release?").

4. As grid user repeat the following to validate inventory for ALL applicable homes on ALL nodes:

% $GI_HOME/OPatch/opatch lsinventory -detail -oh <home-path>

Note: If any errors or inconsistencies are returned corrective action MUST be taken prior to applying the patch.

5. As the user root, execute OPatch auto to apply GI PSU 11.2.0.2.3 to all 11.2.0.2 Grid Infrastructure and Databasehomes:

# cd /u01/stage/gipsu3# export GI_HOME=/ocw/grid

# $GI_HOME/OPatch/opatch auto /u01/stage/gipsu3

Note: You can optionally apply the patch to an individual 11.2.0.2 home by specifying the -oh <home path> to theopatch auto command:

# $GI_HOME/OPatch/opatch auto /u01/stage/gipsu3 -oh /u01/stage/gipsu3

This would apply the 11.2.0.2.3 GI PSU to ONLY the 11.2.0.2 GI Home.

6. As the grid user repeat validate inventory for all patched homes:


7. Repeat steps 1-6 on each of the remaining cluster nodes, 1 node at a time.

8. If you applied the Databse PSU to the Database Homes (as shown in this example), you must now load the ModifiedSQL Files into the Database(s) as follows:

Note: The patch README should be consulted for additional instructions!



% cd $ORACLE_HOME/rdbms/admin% sqlplus /nologSQL> CONNECT / AS SYSDBASQL> STARTUPSQL> @catbundle.sql psu apply

SQL> QUIT

EXAMPLE: Apply a CRS patch manually

The example assumes the following:

Platform is Solaris SPARC 64-bitAll homes (CRS, ASM and DB) are non-sharedAll homes are version 11.1.0.7The Clusterware Home is /ocw/crsThe Clusterware, ASM and Database owner is oracleThe ASM Home is /db/11.2/asmDatabase Home 1 is /db/11.1/db1Database Home 2 is /db/11.1/db2

Note: This example only covers the application of the patch iteslf. It does NOT cover the additional databaseor anyother component specific steps required for the patch installation. That said you must ALWAYS consult the patchREADME prior to attempting to install any patch.

1. Create an EMPTY directory to stage the 11.1.0.7.7 CRS PSU as the CRS software owner (our example uses adirectory named crspsu7):

% mkdir /u01/stage/crspsu7

2. Extract the CRS PSU into the empty stage directory as the CRS software owner:

% unzip -d /u01/stage/crspsu7 p11724953_11107_Solaris-64.zip

3. Verify that opatch in ALL 11.1.0.7 homes (for the Database homes there is a Database compnent to CRS patches)meet minimum version requirement documented in the PSU README (see "How to find out the opatch version?"). If theversion of OPatch in any one (or all) of the homes does not meet the minimum version required for the patch, OPatchmust be upgraded in this/these homes prior to continuing (see "How do I install the latest OPatch release?").

4. As oracle repeat the following to validate inventory for ALL applicable homes:

% $CRS_HOME/OPatch/opatch lsinventory -detail -oh <home-path>

5. On the local node, stop all instances, listeners, ASM and CRS:



As oracle from the ORACLE_HOME in which the resource is running:% $ORACLE_HOME/bin/srvctl stop instance -i <local instance name> -d <db name>% $ASM_HOME/bin/srvctl stop asm -n <local node>% $CRS_HOME/bin/srvctl stop nodeapps -n <local node>

As root:

# $CRS_HOME/bin/crsctl stop crs

6 As root execute the preroot patch script:

# /u01/stage/crspsu7/11724953/custom/scripts/prerootpatch.sh -crshome /ocw/crs -crsuseroracle

Note: the crsuser is not "root", it is the software install owner, this is a commonly made mistake

7. As the oracle user execute the prepatch script for the CRS Home:

% /u01/stage/crspsu7/11724953/custom/scripts/prepatch.sh -crshome /ocw/crs

8. As the oracle user execute the prepatch script for the Database/ASM Homes:

% /u01/stage/crspsu7/11724953/custom/server/11724953/custom/scripts/prepatch.sh -dbhome/db/11.1/asm

% /u01/stage/crspsu7/11724953/custom/server/11724953/custom/scripts/prepatch.sh -dbhome/db/11.1/db1

% /u01/stage/crspsu7/11724953/custom/server/11724953/custom/scripts/prepatch.sh -dbhome/db/11.1/db2

9. As oracle apply the CRS PSU to the CRS Home using opatch napply:

% export ORACLE_HOME=/ocw/crs

% $ORACLE_HOME/OPatch/opatch napply -oh $ORACLE_HOME -local /u01/stage/crspsu7/11724953

10. As the oracle user apply the database compnent of CRS PSU to the Database/ASM Homes using opatch napply:

% export ORACLE_HOME=/db/11.1/asm

% $ORACLE_HOME/OPatch/opatch napply -oh $ORACLE_HOME -local/u01/stage/crspsu7/11724953/11724953/custom/server/11724953/% export ORACLE_HOME=/db/11.1/db1

% $ORACLE_HOME/OPatch/opatch napply -oh $ORACLE_HOME -local/u01/stage/crspsu7/11724953/11724953/custom/server/11724953/% export ORACLE_HOME=/db/11.1/db2

% $ORACLE_HOME/OPatch/opatch napply -oh $ORACLE_HOME -local/u01/stage/crspsu7/11724953/11724953/custom/server/11724953/

11. As the oracle user execute the postpatch script for the CRS Home:



% /u01/stage/crspsu7/11724953/custom/scripts/postpatch.sh -crshome /ocw/crs

12. As the oracle user execute the postpatch script for the Database/ASM Homes:

% u01/stage/crspsu7/11724953/custom/server/11724953/custom/scripts/postpatch.sh -dbhome/db/11.1/asm

% u01/stage/crspsu7/11724953/custom/server/11724953/custom/scripts/postpatch.sh -dbhome/db/11.1/db1

% u01/stage/crspsu7/11724953/custom/server/11724953/custom/scripts/postpatch.sh -dbhome/db/11.1/db2

13. As root execute the postrootpatch script (this script will start the CRS stack):

# u01/stage/crspsu7/11724953/custom/scripts/postrootpatch.sh -crshome /ocw/crs

14. As the oracle user repeat the following to validate inventory for ALL applicable homes:

% $CRS_HOME/OPatch/opatch lsinventory -detail -oh <home-path>

15. Start all instances, listeners and ASM on local node (CRS was started with the postrootpatch script):

% $ORACLE_HOME/bin/srvctl start instance -i <local instance name> -d <db name>% $ASM_HOME/bin/srvctl start asm -n <local node>

% $CRS_HOME/bin/srvctl start nodeapps -n <local node>

16. Repeat steps 1-15 on each node in the cluster, one node at a time.

EXAMPLE: Applying a GI PSU patch manually

The example assumes the following:

Platform is Linux 64-bitAll Homes (GI and Database) are not sharedThe Grid Infrastructure owner is gridThe Database owner is oracleThe Grid Infrastructure Home is /ocw/gridDatabase Home1 is /db/11.2/db1Database Home2 is /db/11.2/db2All Homes are 11.2.0.2ACFS is NOT in use on this cluster

Note: This example only covers the application of the patch iteslf. It does NOT cover the additional databaseor anyother component specific steps required for the patch installation. That said you must ALWAYS consult the patchREADME prior to attempting to install any patch.

1. Create an EMPTY directory to stage the GI PSU as the GI software owner (our example uses a directory namedgipsu3):



% mkdir /u01/stage/gipsu3

Note: The directory must be readable, writable by root, grid and all database users.

2. Extract the GI PSU into the empty stage directory as the GI software owner:

% unzip -d /u01/stage/gipsu3 p12419353_112020_Linux-x86-64.zip

3. Verify that opatch in ALL 11.2.0.2 homes that will be patched meet minimum version requirement documented in thePSU README (see "How to find out the opatch version?"). If the version of OPatch in any one (or all) of the homes doesnot meet the minimum version required for the patch, OPatch must be upgraded in this/these homes prior tocontinuing (see "How do I install the latest OPatch release?").

4. As grid user repeat the following to validate inventory for ALL applicable homes on ALL nodes:


Note: If any errors or inconsistencies are returned corrective action MUST be taken prior to applying the patch.

5. On the local node, use the "srvctl stop home" command to stop the database resources:

% $GI_HOME/bin/srvctl stop home -o /db/11.2/db1 -s /tmp/statefile_db1.out -n <local node>

% $GI_HOME/bin/srvctl stop home -o /db/11.2/db2 -s /tmp/statefile_db2.out -n <local node>

6. As root unlock the Grid Infrastructure Home as follows:

# export ORACLE_HOME=/ocw/grid

# $ORACLE_HOME/perl/bin/perl $ORACLE_HOME/crs/install/rootcrs.pl -unlock ## execute this inGrid Infrastructure cluster, for Oracle Restart see the note below.

Note: If you are in a Oracle Restart environment, you will need to use the roothas.pl script instead of therootcrs.pl script as follows:# $ORACLE_HOME/perl/bin/perl $ORACLE_HOME/crs/install/roothas.pl -unlock

7. Execute the prepatch script for the Database Homes as the oracle user:

% /u01/stage/gipsu3/12419353/custom/server/12419353/custom/scripts/prepatch.sh -dbhome/db/11.2/db1

% /u01/stage/gipsu3/12419353/custom/server/12419353/custom/scripts/prepatch.sh -dbhome/db/11.2/db2



8. As the grid user apply the patch to the local GI Home using opatch napply:

% export ORACLE_HOME=/ocw/grid% $ORACLE_HOME/OPatch/opatch napply -oh $ORACLE_HOME -local /u01/stage/gipsu3/12419353

% $ORACLE_HOME/OPatch/opatch napply -oh $ORACLE_HOME -local /u01/stage/gipsu3/12419331

9. As the oracle user apply the patch to the local Database Homes using opatch napply:

% export ORACLE_HOME=/db/11.2/db1

% $ORACLE_HOME/OPatch/opatch napply -oh $ORACLE_HOME -local/u01/stage/gipsu3/12419353/custom/server/12419353% $ORACLE_HOME/OPatch/opatch napply -oh $ORACLE_HOME -local /u01/stage/gipsu3/12419331

% export ORACLE_HOME=/db/11.2/db2

% $ORACLE_HOME/OPatch/opatch napply -oh $ORACLE_HOME -local/u01/stage/gipsu3/12419353/custom/server/12419353

% $ORACLE_HOME/OPatch/opatch napply -oh $ORACLE_HOME -local /u01/stage/gipsu3/12419331

10. Execute the postpatch script for the Database Homes as the oracle user:

% /u01/stage/gipsu3/12419353/custom/server/12419353/custom/scripts/postpatch.sh -dbhome/db/11.2/db1

% /u01/stage/gipsu3/12419353/custom/server/12419353/custom/scripts/postpatch.sh -dbhome/db/11.2/db2

11. At the root user execute the rootadd_rdbms script from the GI Home:


# $ORACLE_HOME/rdbms/install/rootadd_rdbms.sh

12. As root relock the Grid Infrastructure Home as follows (this will also start the GI stack):


# $ORACLE_HOME/perl/bin/perl $ORACLE_HOME/crs/install/rootcrs.pl -patch ## execute this inGrid Infrastructure cluster, for Oracle Restart see the note below.

Note: If you are in a Oracle Restart environment, you will need to use the roothas.pl script instead of the rootcrs.plscript as follows:# $ORACLE_HOME/perl/bin/perl $ORACLE_HOME/crs/install/roothas.pl -patch

13. Restart the database resources on the local node using the "srvctl start home" command:

# $GI_HOME/bin/srvctl start home -o /db/11.2/db1 -s /tmp/statefile_db1.out -n <local node>

# $GI_HOME/bin/srvctl start home -o /db/11.2/db2 -s /tmp/statefile_db2.out -n <local node>



14. As grid user repeat the following to validate inventory for ALL patched homes:


15. Repeat steps 1-14 on each node in the cluster, one node at a time.

16. If you applied the Databse PSU to the Database Homes (as shown in this example), you must now load the ModifiedSQL Files into the Database(s) as follows:

Note: The patch README should be consulted for additional instructions!

% cd $ORACLE_HOME/rdbms/admin% sqlplus /nologSQL> CONNECT / AS SYSDBASQL> STARTUPSQL> @catbundle.sql psu apply

SQL> QUIT

Common OPatch Failures and Troubleshooting

What files to review if opatch auto fails?

The key to OPatch Auto troubleshooting is understanding where and how the OPatch Auto logs are generated. WhenOPatch auto is invoked, it will generate a opatchauto<timestamp>.log log in the $ORACLE_HOME/cfgtoollogs directoryof the ORACLE_HOME that opatch auto was invoked. This log will contain information around the execution of OPatchauto itself and will be the starting point for troubleshooting issues. Now, OPatch auto will also invoke additional OPatchsessions using the OPatch executable from the home that is being patch to perform the pre-req checks and actuallyapply the patch. Each of these subsequent invokations OPatch will generate a new opatch<timestamp>.log log under$ORACLE_HOME/cfgtoollogs/opatch within the ORACLE_HOME that is being patched. These logs are often times wherethe root cause of an OPatch auto failure will be found.

Should OPatch auto fail to restart the stack (noted in the error message) one should consult the rootcrs or roothas log(11.2 only) in $GRID_HOME/cfgtoollogs/crsconfig, as OPatch auto actually makes calls "rootcrs.pl" or "roothas.pl". Should an SR need to be opened up in the event of of a OPatch auto failure it is recommended to collect the OPatchlogs (discussed above) as well as the output of the diagcollection script as described Document 330358.1> and providethis diagnostic data at the time of SR creation.

"opatch apply" or "opatch napply" failure if clusterware home is not unlocked

If "opatch napply" or "opatch apply" is executed without unlocking clusterware home first, the following will bereported:

ApplySession failed: ApplySession failed to prepare the system. ApplySession was not able tocreate the patch_storage area: /ocw/grid/.patch_storage/<patch#_date>..OPatch failed with error code 73

OR



chmod: changing permissions of ̀/ocw/grid/bin':Operation not permitted..mv: cannot move ̀/ocw/grid/bin/oracle' to ̀/ocw/grid/bin/oracleO': Permission denied..

The solution is to carefully follow the instructions in the README packaged with the patch which will provideinstructions use "opatch auto" if applicable or the manual opatch process.

OPatch reports: "Prerequisite check CheckSystemSpace failed"

On AIX, due to unpublished bug 9780505 opatch needs about 22GB of free disk space in $GRID_HOME to apply a GIPSU/Bundle compared to about 4.4GB on other platforms. "opatch util cleanup -oh <home-path>" can be used toreclaim space in .patch_storage, refer to Document 550522.1 and Document 1088455.1 for a full explanation on thisissue as well as and other tips to work around this issue.

Common causes of "The opatch version check failed"

ZOP-49: Not able to execute the prereq. OPatch cannot inform if the patch satisfies minimum version requirement.

Incorrect patch location specifiedOPatch version does not meet the requirements for the patch (specified in the README), you must upgradeOPatch.The grid or database user can not read unzipped patch directory/files, check directory permissions

OPatch reports: "Patch nnn requires component(s) that are not installed in OracleHome"

These not-installed components are oracle.crs:<version>, oracle.usm:<version>

The patch is not applicable to the target home, i.e. if apply 11.2.0.3 GI PSU1 to 11.2.0.2 GI home, the error will bereported, refer to Document 763680.1 for troubleshooting instructions.

Common causes of "The opatch Component check failed"

Incorrect patch location specifiedThe patch was unzipped in non-empty directoryThe proper version of the OPatch executable was is not installed into the ORACLE_HOME being patched andOPatch was not launched from that ORACLE_HOMEThe current working directory is $ORACLE_HOME/OPatch of the ORACLE_HOME that is being patchedThe $ORACLE_HOME/.patch_storage cannot be created, the solution is to create the directory manually withcorresponding grid or database user with permission of 750The listener is running from database home, the solution is to shutdown listeners from database homeThe grid and database users are different, the workaround is to apply manuallyThe patch is intended for other version, i.e. when applying 11.2.0.2 GI PSU5 to 11.2.0.3 GI home

Refer to Document 1169036.1 for troubleshooting instructions.

Common causes of "The opatch Conflict check failed"

A true patch conflict, the solution is to open an SR and request merge patch

https://support.oracle.com/epmos/faces/BugDisplay?id=9780505







A Clusterware patch may report a conflict even it is super set of the currently installed patch. Normally OPatchshould rollback the installed installed patch in favor of the super set, but in some rare case, OPatch errors out;the solution is to manually rollback and apply the new patch.

Common causes of "The opatch Applicable check failed"

ZOP-46: The patch(es) are not applicable on the Oracle Home because some patch actions are not applicable. Allrequired components, however, are installed.

Prereq "checkApplicable" for patch 13653086 failed.

ZOP-46: The patch(es) are not applicable on the Oracle Home because some patch actions are not applicable. Allrequired components, however, are installed.

Prereq "checkApplicable" for patch 13653086 failed...Copy Action: Desctination File "/u02/grid/11.2.0/bin/crsd.bin" is not writeable.'oracle.crs, 11.2.0.2.0': Cannot copy file from 'crsd.bin' to '/u02/grid/11.2.0/bin/crsd.bin'

Some of the files in target home are still being locked - some processes still running, or "rootcrs.pl -unlock" /"roothas.pl -unlock" has not been executedUnzipped patch files in stage area are unreadable by grid or database user: "Copy Action: Source File"/patches/B202GIPSU4/12827731/files/bin/appvipcfg.pl" does not exists or is not readable". The solution is tounzip the patch by any OS user that has primary group oinstall, for example, grid user.Database control (Enterprise Manager)/EM agent is not stopped: "Copy Action: Desctination File"/ocw/grid/11.2/bin/emcrsp.bin" is not writeable.". To stop, as the ORACLE_HOME owner execute: $<ORACLE_HOME>/bin/emctl stop dbconsoleA directory is provides when prompted "OPatch is bundled with OCM, Enter the absolute OCM response filepath", the solution is to enter absolute OCM response file directory and filename

For other potential causes, review opatch logfile. The OPatch log file can be found in$ORACLE_HOME/cfgtoollogs/opatch.

Common causes of "patch <patch-loc> apply failed for home <ORACLE_HOME>"

OUI-67124:Copy failed from '/ocw/grid/.patch_storage/13540563_Jan_16_2012_03_31_27/files/bin/crsctl.bin' to'/ocw/grid/bin/crsctl.bin'...OUI-67124:ApplySession failed in system modification phase... 'Copy failed from'/ocw/grid/.patch_storage/13540563_Jan_16_2012_03_31_27/files/bin/crsctl.bin' to '/ocw/grid/bin/crsctl.bin'...

OUI-67124:NApply restored the home. Please check your ORACLE_HOME to make sure:

dbconsole is not down - it needs to be stopped prior to patching, as database home owner, execute "<ORACLE_HOME>/bin/emctl stop dbconsole" to stop it



opatch bug 12732495, refer to note 1472242.1 for more details.opatch bug 14308442 which is closed as duplicate of bug 14358407, affects AIX only, refer to note 1518022.1for more details.other opatch bug - use latest opatch to avoid known issues.

For other potential causes, review opatch logfile. The OPatch log file can be found in$ORACLE_HOME/cfgtoollogs/opatch.

When applying online patch in RAC: Syntax Error... Unrecognized Option for Apply .. OPatch failed with errorcode 14

Basically the earlier patch readme template is wrong for online patch for RAC, refer to note 1356093.1 for details.

opatch Fails to Rollback Online(Hot) Patch in RAC With oracle/ops/mgmt/cluster/NoSuchNodeException anderror code 30


opatch Fails to Rollback Online(Hot) Patch With Prerequisite check "CheckRollbackSid" and error code 30

Prerequisite check "CheckRollbackSid" failed.

Patch ID: 13413533

The details are:

There are no SIDs defined to check with the input SIDs.


Common causes of "defined(@array) is deprecated at crs/crsconfig_lib.pm"

"opatch auto" reports the following warning message:

defined(@array) is deprecated at crs/crsconfig_lib.pm line 2149

The issue happens when applying 11.2.0.3.1 (patch 13348650) with latest opatch, it's being tracked via internal bug13556626 bug 13555176, the message can be safely ignored

opatch auto Reports: The path "<GRID_HOME>/bin/acfsdriverstate" does not exist

"opatch auto" reports the following message:

The path "/ocw/grid/bin/acfsdriverstate" does not exist

The message can be ignored, refer to note 1469969.1 for details















Applying [PSU] patch takes very long time (hours) after "verifying the update"

opatch or "opatch auto" takes hours to finish after the following message is printed:

[Feb 7, 2013 9:27:18 PM] verifying 1009 copy files.<< no more display after this, it could stop here for hours, eventually complete

This is due to bug 13963741 which at the time of this writing is still being worked by Development, the workaround isto unset environment variable "JAVA_COMPILER=NONE", refer to note 1541186.1 for more details.

New OPatch Features

To find out new features in opatch 11.2.0.3.2, refer to note 1497639.1

For searchability:database home; RAC home; RDBMS home; GI home; DB home; ASM homeRDBMS_HOME; ASM_HOME; DB_HOME; ASM_HOMEGI user; ASM user; DB user; database user

REFERENCES

NOTE:1089476.1 - Patch 11gR2 Grid Infrastructure Standalone (Oracle Restart)NOTE:337737.1 - Oracle Clusterware - ASM - Database Version CompatibilityNOTE:1272288.1 - 11.2.0.2.X Grid Infrastructure Bundle/PSU Known IssuesNOTE:1169036.1 - Applying GI PSU using "opatch auto" fails with "The opatch Component check failed"NOTE:1356093.1 - Syntax error while applying and rolling back an online patch In RAC enivronment

NOTE:1064804.1 - Apply Grid Infrastructure/CRS Patch in Mixed Version RAC Database EnvironmentNOTE:1082394.1 - 11.2.0.1.X Grid Infrastructure PSU Known IssuesNOTE:405820.1 - 10.2.0.X CRS Bundle Patch InformationNOTE:1416823.1 - opatch Fails to Rollback Online(Hot) Patch in RAC Withoracle/ops/mgmt/cluster/NoSuchNodeException and error code 30NOTE:1410202.1 - How to Apply a Grid Infrastructure Patch Before root script (root.sh or rootupgrade.sh) is Executed?NOTE:1417268.1 - "opatch prereq CheckApplicable" Fails Agaist GRID_HOME With ZOP-46NOTE:1422190.1 - opatch Fails to Rollback Online(Hot) Patch With Prerequisite check "CheckRollbackSid" failed anderror code 30NOTE:293369.1 - Master Note For OPatch

NOTE:763680.1 - Opatch Error "UtilSession failed: Patch nnn requires component(s) that are not installed" WhenApplying CRS or Grid Infrastructure Merge, Bundle or PSU PatchesNOTE:810663.1 - 11.1.0.X CRS Bundle Patch InformationNOTE:854428.1 - Patch Set Updates for Oracle Products






















NOTE:330358.1 - CRS 10gR2/ 11gR1/ 11gR2 Diagnostic Collection Guide

NOTE:1212703.1 - Grid Infrastructure Startup During Patching, Install or Upgrade May Fail Due to MulticastingRequirementNOTE:1088455.1 - opatch CheckSystemSpace Fails With Error Code 73 While Applying GI PSU




FAQ OPatch Patch Questions Issues for Oracle Clusterware (Grid Infrastructure or CRS) and RAC...

Documents

Transcript of FAQ OPatch Patch Questions Issues for Oracle Clusterware (Grid Infrastructure or CRS) and RAC...