Oracle Database 11gR1 RAC Build Standards

1.Introduction

1.1. 64-Bit Required RPM's

1.2. Schematic 2. Prepare the cluster nodes for Oracle RAC 2.1. User Accounts 2.2. Networking 2.3. Configuring Kernel Parameters 2.4. Configuration of the Hangcheck-timer Module 2.5. Stage the Oracle Software 3. Prepare the shared storage for Oracle RAC 3.1. Example of Configuring Block Device Storage for OracleClusterware

4. Oracle Clusterware Installation and Configuration

4.1. CVU Pre Oracle Clusterware install check

4.2. Oracle Clusterware Install

5. Oracle Clusterware patching

6. Oracle ASM Home Software Install

6.1. CVU check before creating ASM instance

6.2. Oracle ASM Home Software Install

7. Oracle ASM Software Home Patching

8. Oracle RAC Database Listener Creation

9. Oracle ASM Instance and diskgroup Creation

1

CHAPTER 1INTRODUCTION

This Document is meant for Installation of Real Application Cluster (RAC) on RHEL 5.

RAC -11.1.0.7OS- Linux x86-64 (RHEL 5)

The Installation involves follow steps

PreparationPre-reqs. to make sure the cluster is setup OK.Stage all the software on one node, typically Node1

Establish Oracle Clusterware

Install the Oracle Clusterware (using the push mechanism to install on the other nodes in the

cluster)

Patch the Clusterware to the latest patchset

Establish ASM Install an Oracle Software Home for ASM Patch the ASM Home to the latest patchset Create the Listeners

1.1. 64-Bit Required RPM's

The following packages (or later versions) must be installed:

Red Hat Enterprise Linux 4 Red Hat Enterprise Linux 5 Packagesbinutils-2.15.92.0.2 binutils-2.17.50.0.6 binutils-2.16.91.0.5compat-libstdc++-33-3.2.3 compat-libstdc++-33-3.2.3 compat-libstdc++-5.0.7-22.2compat-libstdc++-33-3.2.3 (32 bit) compat-libstdc++-33-3.2.3 (32 bit)elfutils-libelf-0.97 elfutils-libelf-0.125 libelf-0.8.5Rac11gR1OnLinux < RACGuides < TWiki1.1.1. Software required for install 2elfutils-libelf-devel-0.97 elfutils-libelf-devel-0.125gcc-3.4.5 gcc-4.1.1 gcc-4.1.0gcc-c++-3.4.5 gcc-c++-4.1.1 gcc-c++-4.1.0

2

glibc-2.3.4-2.19 glibc-2.5-12 glibc-2.4-31.2glibc-2.3.4-2.19 (32 bit) glibc-2.5-12 (32 bit) glibc-32bit-2.4-31.2 (32 bit)glibc-common-2.3.4 glibc-common-2.5glibc-devel-2.3.4 glibc-devel-2.5 glibc-devel-2.4glibc-devel-2.3.4 (32-bit) glibc-devel-2.5-12 (32 bit) glibc-devel-32bit-2.4 (32 bit)libaio-0.3.105 libaio-0.3.106 libaio-0.3.104libaio-0.3.105 (32 bit) libaio-0.3.106 (32 bit) libaio-32bit-0.3.104 (32 bit)libaio-devel-0.3.105 libaio-devel-0.3.106libgcc-3.4.5 libgcc-4.1.1 libgcc-4.1.0libgcc-3.4.5 (32-bit) libgcc-4.1.1 (32 bit)libstdc++-3.4.5 libstdc++-4.1.1libstdc++-3.4.5 (32 bit) libstdc++-4.1.1 (32 bit) libstdc++-4.1.0libstdc++-devel 3.4.5 libstdc++-devel 4.1.1 libstdc++-devel-4.1.0make-3.80 make-3.81 make-3.80sysstat-5.0.5 sysstat-7.0.0 sysstat-6.0.2

To determine whether the required packages are installed, enter commands similar to the following:# rpm -q package_name

If a package is not installed, then get it installed.

1.2. SchematicThe following is a schematic of the software & hardware layout of a 2node RAC cluster. As explained in thisdocument the actual number of LUN’s required will vary depending on your mirroring requirements.

3

4

CHAPTER 2Prepare the cluster nodes for Oracle RAC

This installation routine presumes that you have a 2-node Linux cluster. There are a number of items thatrequire checking before the install commences. Getting this bit right will enhance your install experience.1. User Accounts2. Networking3. Time Sync4. Stage the Oracle Software5. Run CVUIt is essential that these items are checked and are configured correctly before the install commences.

2.1. User Accounts2.1.1. Creating the OSDBA (DBA) GroupTo determine whether the OSDBA group exists, enter the following command:# grep OSDBA_group_name /etc/groupIf the OSDBA group does not exist or if you require a new OSDBA group, then create it as follows. In thefollowing command, use the group name dba unless a group with that name already exists.# /usr/sbin/groupadd dba

2.1.2. Creating an OSOPER Group (Optional)If you require a new OSOPER group, then create it as follows. In the following command, use the group nameoper unless a group with that name already exists.# /usr/sbin/groupadd oper

2.1.3. Creating an OSASM GroupTo determine whether the OSASM group exists, enter the following command:# grep OSASM_group_name /etc/groupIf the OSASM group does not exist or if you require a new OSASM group, then create it as follows. In thefollowing command, use the group name asadmin unless a group with that name already exists.# /usr/sbin/groupadd asmadmin

2.1.4. Creating an OINSTALL GroupTo determine whether the OINSTALL group exists, enter the following command:# grep OINSTALL_group_name /etc/groupIf the OINSTALL group does not exist or if you require a new OINSTALL group, then create it as follows. Inthe following command, use the group name dba unless a group with that name already exists.# /usr/sbin/groupadd oinstall

5

Note: The default OINSTALL group name is oinstall.2.1.5. Determining Whether an Oracle Software Owner User Exists

To determine whether an Oracle software owner user named oracle exists, enter the following command:# id oracle

If the oracle user exists, then the output from this command is similar to the following:

uid=440(oracle) gid=200(oinstall) groups=201(dba),202(oper)

If the user exists, then determine whether you want to use the existing user or create another oracle user. If you want to use the existing user, then ensure that the user’s primary group is the Oracle Inventory group and that it is a member of the appropriate OSDBA and OSOPER groups.

2.1.6. Creating an Oracle Software Owner UserIn the following procedure, use the user name oracle unless a user with that name already exists. If the Oraclesoftware owner user does not exist or if you require a new Oracle software owner user, then create it asfollows:1.To create the oracle user, enter a command similar to the following:# /usr/sbin/useradd -g oinstall -G dba[,oper] oracleIn this command:The -g option specifies the primary group, which must be the Oracle Inventory group, for exampleoinstallThe -G option specifies the secondary groups, which must include the OSDBA group and if required,the OSOPER group.dba or dba,oper2.Set the password of the oracle user:# passwd oracle

2.1.9. Checking Existing SSH Configuration on the System

To determine if SSH is running, enter the following command:$ pgrep sshd

If SSH is running, then the response to this command is one or more process ID numbers. In the homedirectory of the software owner that you want to use for the installation (crs, oracle), use the command ls -al to ensure that the .ssh directory is owned and writable only by the user.NOTE- If ssh is not configured properly as per capital One standards then get it installed by Unix team.

6

2.1.10 Set the Display properly

2.2. NetworkingWe need a total of 3 IP addresses per node:The public IP address, which should be recorded in hosts file on each node and, if available, DNS.This IP Address should be bound to the public adapter before starting the install. It should be a static,not DHCP, addressThe private IP address, which should be from a different subnet than the public IP address. Thisaddress does not require registering in DNS but you should place an entry in the hosts file on eachnode. This IP Address should be bound to the private adapter before starting the install. It should be astatic, not DHCP, addressA VIP address, which should be from the same subnet as the public IP address and should berecorded in DNS and the hosts file on each node. This IP Address should NOT be bound to the publicadapter before starting the install. Oracle Clusterware is responsible for binding this address. It shouldbe a static, not DHCP, address

2.2.1. Network Ping TestsThere are a series of 'ping' tests that should be completed, and then the network adapter binding order shouldbe checked. You should ensure that the public IP addresses resolve correctly and that the private addresses areof the form 'nodename-priv' and resolve on both nodes via the hosts file.Public Ping testPinging stnsp001 from stnsp001 should return stnsp001's public IP addressPinging stnsp002 from stnsp001 should return stnsp002's public IP addressPinging stnsp001 from stnsp002 should return stnsp001's public IP addressPinging stnsp002 from stnsp002 should return stnsp002's public IP addressPrivate Ping testPinging stnsp001 private from stnsp001 should return stnsp001's private IP addressPinging stnsp002 private from stnsp001 should return stnsp002's private IP addressPinging stnsp001 private from stnsp002 should return stnsp001's private IP addressPinging stnsp002 private from stnsp002 should return stnsp002's private IP addressVIP Ping test Pinging the VIP address at this point should fail. VIPs will be activated at the end of the

7

Oracle Clusterware install.2.3. Configuring Kernel ParametersOn all cluster nodes, verify that the kernel parameters shown in the following table are set to values greater than or equal to the recommended value shown. The procedure following the table describes how to verify and set the values.

2.4. Configuration of the Hangcheck-timer ModuleBefore installing Oracle Real Application Clusters on Linux systems, verify that the hangcheck-timer module(hangcheck-timer) is loaded and configured correctly. hangcheck-timer monitors the Linux kernel forextended operating system hangs that could affect the reliability of a RAC node and cause a databasecorruption. If a kernel/device driver hang occurs, then the module restarts the node in seconds. There are 3parameters used to control the behavior of the module:The hangcheck_tick parameter: it defines how often, in seconds, the hangcheck-timer checks thenode for hangs. The default value is 60 seconds. Oracle recommends to set it to 1(hangcheck_tick=1).

1.The hangcheck_margin parameter: it defines how long the timer waits, in seconds, for a responsefrom the kernel. The default value is 180 seconds. Oracle recommends to set it to 10(hangcheck_margin=10)

8

2.The hangcheck_reboot parameter: If the value of hangcheck_reboot is equal to or greater than 1,then the hangcheck-timer module restarts the system. If the hangcheck_ reboot parameter is set tozero, then the hangcheck-timer module will not restart the node. It should always be set to 1.

2.5. Stage the Oracle SoftwareIt is recommended that you stage the required software onto a local drive on Node 1 of your cluster.

NOTE-The 11gR1 now integrates the Oracle Clusterware, Database and Client install into one DVD with onerunInstaller program.If you download the software from OTN you will not get the integrated installer. You will have separatedownloads for :· Oracle Clusterware and· ASM/Database

2.7. Cluster Verification Utility stage check

Now run the CVU (Cluster Verification Utility) to check the state of the Operating Systemconfiguration. CVU can be run from the installation media, but it is recommended to download the latestversion from:http://www.oracle.com/technology/products/database/clustering/cvu/cvu_download_homepage.htmlAfter the hardware and OS has been configured, it is recommended to run CVU to verify the nodes areconfigured correctly:runcluvfy.sh stage -post hwos -n stnsp001,stnsp002 -verbose

9

CHAPTER 3

Prepare the shared storage for Oracle RAC

For all installations, you must choose the storage option that you want to use for Oracle Clusterware files,Automatic Storage Manager (ASM) and Oracle Real Application Clusters databases (Oracle RAC). You donot have to use the same storage option for each file type.Oracle Clusterware files include:· Voting disks, used to monitor cluster node status.· Oracle Cluster Registry (OCR) which contains configuration information about the cluster.There are two ways of storing Oracle Clusterware files:

* Block or Raw Devices: Oracle Clusterware files can be placed on either Block or RAW devices basedon shared disk partitions. Oracle recommends using Block devices for easier usage.

NOTE-When you create partitions with fdisk by specifying a device size, such as +256M, the actual device createdmay be smaller than the size requested, based on the cylinder geometry of the disk. This is due to current fdiskrestrictions. Oracle configuration software checks to ensure that devices contain a minimum of 256MB ofavailable disk space. Therefore, Oracle recommends using at least 280MB for the device size. You can checkpartition sizes by using the command syntax fdisk -s partition. For example: [root@node1]$ fdisk -s /dev/sdb1281106As root, now configure storage for cluster registry, voting disk and database files. You are presented with abunch of disks from the storage array. The output of fdisk –s /dev/sd[b-e] command may look as follows./dev/sdb: 116924416/dev/sdc: 116924416/dev/sdd: 116924416/dev/sde: 116924416In this example we have four 116Gig LUN’s:/dev/sdb 116G/dev/sdc 116G/dev/sdd 116G/dev/sde 116G

10

3.1. Example of Configuring Block Device Storage for OracleClusterwareThe procedure to create partitions for Oracle Clusterware files on block devices is asfollows:1. log in as root2. Enter the fdisk command to format a specific storage disk (for example,/sbin/fdisk /dev/sdb)3. Create a new partition, and make the partition 280 MB in size for both OCR andvoting disk partitions.Use the command syntax /sbin/partprobe diskpath on each node in thecluster to update the kernelpartition table for the shared storage device on eachnode.4.The following is an example of how to use fdisk to create one partition on a shared storage block disk devicefor an OCR file:[root@stnsp001] # /sbin/fdisk /dev/sdbThe number of cylinders for this disk is set to 1024.Command (m for help): nCommand actione extendedP primary partition (1-4)pPartition number (1-4): 1First cylinder (1-1024, default 1):Using default value 1

Last cylinder or +size or +sizeM or +sizeK (1-1024, default 1)Using default value 1Last cylinder or +size or +sizeM or +sizeK (1-1024, default 1024):+280mCommand (m for help):wThe partition table has been altered!Calling ioctl () to re-read partition table.Synching disks.Login as the root user on the remote nodes and execute the following:[root@stnsp002] # /sbin/partprobe /dev/sdb1Note: Oracle recommends that you create partitions for Oracle Clusterware files on physically separate disks.The user account with which you perform the installation (oracle or crs) must have write permissions to createthe files in the path that you specify.

3.1.1. Example of Creating a Udev Permissions File for Oracle ClusterwareThe procedure to create a permissions file to grant oinstall group members write privileges to block devices isas follows:

11

1. Log in as root.Change to the /etc/udev/permissions.d directory:# cd /etc/udev/permissions.d2.Start a text editor, such as vi, and enter the partition information where you want to place the OCRand voting disk files, using the syntaxdevice_partitions:root:oinstall:06403.Note that Oracle recommends that you place the OCR and the voting disk files on separate physical disks. Forexample, to grant oinstall members access to SCSI disks to place OCR files on sdb1 and sdc1, and to grant theOracle Clusterware owner (in this example crs) permissions to place voting disks on sdb5, sdc5 and sda5, addthe following information to the file:# OCR diskssdb1:root:oinstall:0640sdc1:root:oinstall:0640# Voting diskssdb5:crs:oinstall:0640sdc5:crs:oinstall:0640sdd5:crs:oinstall:0640

1. Save the file:.·On Asianux 3, Enterprise Linux 5, Red Hat Enterprise Linux 5, and SUSE Enterprise Server 10systems, save the file as 51-oracle.permissions.·1. Using the following command, assign the permissions in the udev file to the devices:

# /sbin/udevstartUse the following procedure above to create additional partitions to use for the OCR, Voting, and ASM disks.Refer to the OS documentation for additional information on using the fdisk command.

3.1.2. Platform Specific Settings

As per Metalink Note:357472.1, on SuSE Linux SLES9 disable HOTPLUG_USE_SUBFS by setting it to"no" in /etc/sysconfig/hotplug. The default setting of "yes" causes problems for multipath devices.

12

Chapter 4

Oracle Clusterware Installation and Configuration

Oracle Clusterware is an essential component of the OracleRAC database infrastructure.

4.1. CVU Pre Oracle Clusterware install check

Prior to installing CRS, verify the nodes are configured correctly for the CRS installruncluvfy.sh stage -pre crsinst -n stnsp001,stnsp002 -r 11gR1 -verbose

Verify the output . If it is successful then proceed further.

4.2. Oracle Clusterware Install

Start the installer by running "runInstaller" from the staged installation media../runInstaller

Step 1:NOTE-

Notice that the Oracle 11g Installer now combines the Oracle Database, Client andClusterware components¨·Action¨ Select the Oracle Clusterware radio button¨ Click Nex

Step 2:

The OUI will name the Oracle Clusterware Home'OraCrs11g_home'. If you change this youshould make sure that the name you use is unique¨·Actions¨ Specify a location for the Oracle Clusterware Home¨ Click Next·Rac11gR1OnLinux < RACGuides < TWiki4.2.

13

Step 3:

The installer will validate the state of the cluster before continuing. If there are issues youshould rectify them before continuing¨·Actions¨ Click Next·

Step 4:

The installer will validate the state of the cluster before continuing. If there are issues youshould rectify them before continuing¨·Actions¨ Click Next·Step 5:

NotesEach Cluster requires a name, this should be unique within your organisation, The default is asubstring of the node name followed by _cluster¨This is where you specify details of all the nodes in the cluster. The installer will defaultnames for the node it is running on. You must add other nodes manually¨¨ Oracle defaults the names to 'nodename', 'nodename-priv', 'nodename-vip'·Actions¨ Confirm the Cluster Name selected is acceptableConfirm the Details for the current node are OK. The defaults are:Public Node Name : must resolve via hosts and or DNS to the public IP address andmust be liveàPrivate Node Name : must resolve via hosts to the interconnect IP address and mustbe liveàVirtual Host Name : must resolve via hosts and or DNS to a new IP address and mustnot be liveàà If these are not correct select the node entry and click Edit... to modify OR¨¨ Click Add... to add more nodes to the cluster

14

Step 6:

Notes¨ Here you specify the details of the node you wish to add to the cluster nodes list·ActionsEnter the new node detailsPublic Node Name : must resolve via hosts and or DNS to the public IP address andmust be live

Private Node Name : must resolve via hosts to the interconnect IP address and mustbe live

Virtual Host Name : must resolve via hosts and or DNS to a new IP address and mustnot be live

¨¨ Click OK to return to the node list for the cluster

Step 7:

Notes¨ If you have more nodes repeast the Add... cycle·Actions¨ Click Next

Step 8:

The installer lists all the Network adapters. You should have one Adapter correctly identifiedas type 'Public' and at least one adapter correctly identified as type 'Private'. The installer willtry and guess the use of an adapter based on the IP address bound. If it guesses incorrectlyyou must change the usage. Here it has guessed that all adapters are Private, which isincorrect.¨Actions¨ Select the Adapter eth0¨ Click Edit...

Step 9:

The installer lists all the Network adapters. You should have one Adapter correctly identifiedas type 'Public' and at least one adapter correctly identified as type 'Private'. The installer willtry and guess the use of an adapter based on the IP address bound. If it guesses incorrectly

15

you must change the usage. Here it has guessed that all adapters are Private, which isincorrect.¨Actions¨ Select the Adapter eth0¨ Click Edit...

Step 10:

NotesHere you can see we have successfully configured the network adapter usage. Ideally you willhave only 1 adapter set as public and 1 adapter set as private. Other adapters, if available, setto ‘do not use’. If you have multiple public or multiple private adapters it is better to teamthem at the OS adapter driver level before commencing the install.Type : Private¨·Actions¨ Click Next

Step 11:

Here we specify the shared storage devices that will be used by Oracle Clusterware. Ideallyyou will have 2 devices for the OCR, Oracle will mirror to these devices to protect you from asingle OCR device failure. Also you will have an additional 3 vote devices to protect yourcluster from the failure of a single vote device.¨·Actions¨ Select the Normal radio button¨ Enter the device to be used for the First OCR¨ Enter the device to be used for the Second OCR¨ Click Next

Step 12:

Notes¨ Next we specify the devices to be used for the Oracle Clusterware vote disks·Actions¨ Select the Normal radio button¨ Enter the device to be used for the First vote disk¨ Enter the device to be used for the Second vote disk¨ Enter the device to be used for the Third vote disk¨ Click Next

16

Step 13:

Notes¨ the installer lists a summary of the planned actions·Actions¨ Click Install

Step 14:

¨ the installer installs the software onto the local nodeActions¨ none required

Step 15:

the installer installs the software onto the remote node·Actions¨ none required

Step 16:

The installer requires commands to be run as root on each f the nodes·Action¨ On the first node open a root shell window

Example :

bash-3.00# /scratch/11.1.0/crs/root.shWARNING: directory '/scratch/11.1.0' is not owned by rootChecking to see if Oracle CRS stack is already configuredSetting the permissions on OCR backup directorySetting up Network socket directoriesOracle Cluster Registry configuration upgraded successfullyThe directory '/scratch/11.1.0' is not owned by root. Changing owner to rootSuccessfully accumulated necessary OCR kegs.Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.node <nodenunber>: <nodenane> <private interconnect name> <hostnane>node 1: stnsp001 stnsp001-rac stnsp001node 2: stnsp002 stnsp002-rac stnsp002Creating OCR kegs for user 'root', privgrp 'root'..Operation successful.Now formatting voting device: /dev/sdb5

17

Now formatting voting device: /dev/sdc5Now formatting voting devices /dev/sdd5Format of 3 voting devices complete.Startup will be queued to init within 30 seconds.Adding daemons to inittabExpecting the CRS daemons to be up within 600 seconds.Cluster Synchronization Services is active on these nodes.stnsp001

Cluster Synchronization Services is inactive on these nodes.stnsp002Local node checking complete. Run root.sh on remaining nodes to start CRS

Notes¨ The output from the root.sh script should be similar to this¨ here the Oracle Clusterware is configured¨ This may take some time to run·Action¨ open a root shell on the first node¨ run the identified command as root on the first node¨ You must wait for the command to complete before continuing

bash-3.00# /scratch/11.1.0/crs/root.shWARNING: directory '/scratch/11.1.0' is not owned by rootChecking to see if Oracle CRS stack is already configuredSetting the permissions on OCR backup directorySetting up Network socket directoriesOracle Cluster Registry configuration upgraded successfullyThe directory '/scratch/11.1.0' is not owned be root. Changing owner to rootclscfg: EXISTING configuration version 4 detected.clscfg: version 4 is 11 Release 1.Successfully accumulated necessary OCR keys.Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.node <nodenunber>: <nodenane> <private interconnect name> <hostnane>node 1: stnsp001 stnsp001-rac stnsp001node 2: stnsp002 stnsp002-rac stnsp002clscfg: Arguments check out successfully.NO KEYS WERE WRITTEN. Supply -force parameter to override.-force is destructive and will destroy any previous clusterconfiguration.Oracle Cluster Registry for cluster has already been initializedStartup will be queued to init within 30 seconds.Adding deamons to inittabExpecting the CRS daemons to be up within 600 seconds.

18

Cluster Synchronization Services is active on these nodes.stnsp001stnsp002Cluster Synchronization Services is active on all the nodes.Waiting for the Oracle CRSD and EVMD to startWaiting for the Oracle CRSD and EVMD to startOracle CRS stack installed and running under init(lM)Running vipca(silent) for configuring nodeappsCreating VIP application resource on (2) nodes...Creating GSD application resource on (2) nodes...Creating ONS application resource on (2) nodes...Creating VIP application resource on (2) nodes...Creating GSD application resource on (2) nodes...Creating ONS application resource on (2) nodes...Done.

Step 17:

Notes¨ The output from the root.sh script should be similar to this¨ here the Oracle Clusterware is configured¨ This may take some time to run·Action¨ open a root shell on the second node¨ run the identified command as root on the second node¨ You must wait for the command to complete before continuing

Step 18:¨ You can now contnue with the install·Action¨ return to the installer¨ Click OK

Step 19

a series of configuration assistants are run at the end of the install·Actions¨ none required

Step 20

The installer will prompt for completed installation and configuration of Oracle Clusterware

19

Actions¨ Click Exit·Actions¨ Click Yes

4.2.0.1. Verify cluster resources are online

[oracle11@stnsp001 bin]$ cd /scratch/11.1.0/crs/bin[oracle11@stnsp001 bin]$ ./crs_stat -tName Type Target State Host------------------------------------------------------------ora....001.gsd application ONLINE ONLINE stnsp001ora....001.ons application ONLINE ONLINE stnsp001ora....001.vip application ONLINE ONLINE stnsp001ora....002.gsd application ONLINE ONLINE stnsp002ora....002.ons application ONLINE ONLINE stnsp002ora....002.vip application ONLINE ONLINE stnsp002Notes¨ You can see the resources configured inside Oracle Clusterware·Action¨ change directory to Oracle Clusterware home bin directory¨ run the ./crs_stat -t comand

4.2.0.2. Use CVU to verify the Oracle Clusterware installVerify the Oracle Clusterware installation using the Oracle Cluster Verification Utility:./cluvfy stage -post crsinst -n stnsp001,stnsp002 -verbose

You should get a meesage as given below

Post-check for cluster services setup was successful.

20

CHAPTER 5

Oracle Clusterware patching

At this point we have installed Oracle Clusterware 11.1.0.6. In this section we will patch the OracleClusterware to the latest release of Oracle 11gR1 - 11.1.0.7. The patchset can be downloaded from metalink

Before we start we can query the clusterware versions# /scratch/11.1.0/crs/bin/crsctl query crs softwareversionOracle Clusterware version on node [stnsp001] is [11.1.0.6.0]# /scratch/11.1.0/crs/bin/crsctl query crs activeversionOracle Clusterware active version on the cluster is [11.1.0.6.0]

Step 1

Enter the following commands to start Oracle Universal Installer, where patchset_directory is the directorywhere you unpacked the patch set software:$ cd patchset_directory/Disk1$ ./runInstaller

Notes¨ The installer appears

Actions¨ Click Next

Step 2

Notes¨ You should ensure that the Clusterware home is selected in the first drop down list box¨ The installer should default the directory to the correct location

Actions

21

¨ Ensure the correct home is selected¨ Click Next

Step 3

the installer detects that this is a clustered home and automatically selects all the nodes in thecluster

ActionsClick Next

Step 4Some parameters are validated by the installerActionsClick Next

Step 5

This screen is a summary of the actions the installer will completeActionsClick Install

Step 6

the installer stages the patch on all the nodes in the clusterActionsNo action required

Step 7

At the end the installer lists the mandatory steps that must be completed to apply this patchActionsLog in as the root user and enter the following command to shut down the OracleClusterware:# CRS_home/bin/crsctl stop crs

22

Run the root111.sh script. It will automatically start the Oracle Clusterware on the patchednode:# CRS_home/install/root111.sh

Example

bash-3.00# /scratch/11.1.0/crs/install/root111.shCreating pre-patch directory for saving pre-patch clusterware filesCompleted patching clusterware files to /opt/crsRelinking some shared libraries.Relinking of patched files is complete.Preparing to recopy patched init and RC scripts.Recopying init and RC scripts.Startup will be queued to init within 30 seconds.Starting up the CRS daemons.Waiting for the patched CRS daemons to start.This may take a while on some systems..11107 patch successfully applied.clscfg: EXISTING configuration version 4 detected.clscfg: version 4 is 11 Release 1.Successfully accumulated necessary OCR keys.Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.node : node 1: stnsp001 stnsp001-priv stnsp001 Creating OCR keys for user 'root

Continue with these two steps on all nodes, one by one, this will achieve a rolling upgrade of the OracleClusterware. When done, verify the Oracle Clusterware is running on all nodes before exiting the installer

$ CRS_home/bin/crsctl check crsCluster Synchronization Services appears healthyCluster Ready Services appears healthyEvent Manager appears healthy

# /scratch/11.1.0/crs/bin/crsctl query crs softwareversionOracle Clusterware version on node [stnsp001] is [11.1.0.7.0]

23

# /scratch/11.1.0/crs/bin/crsctl query crs activeversionOracle Clusterware active version on the cluster is [11.1.0.7.0

Once the Oracle Clusterware is running on all nodes, exit the installer.

CHAPTER 6

Oracle ASM Home Software Install

6.1. CVU check before creating ASM instanceVerify the cluster is configured correctly for an instance creation

./runcluvfy.sh stage -pre dbinst -n stnsp001,stnsp002 -r 11gR1 –verbose

You should get a message like below .

Result: CRS health check passed.CRS integrity check passed.Pre-check for database installation was successful.

6.2. Oracle ASM Home Software Install

Step 1

Start the installer by running "runInstaller" from the staged installation media../runInstaller

Here we will create a ASM Software home on all the nodes in the clusterActionsSelect the Oracle Database 11g radio buttonClick Next

Step 2

Here we can choose the type of database software to be installed

24

ActionsSelect the Enterprise Edition radio buttonClick Next

Step 3

Here we specify the location of various components. An 11g install makes more use of theORACLE_BASE. Most logs will be stored in sub directories under the oracle base. Theoracle base will be common to all installs. Modify as required. Also this is where you specifythe location of the ASM software home. We usually modify the Home name and Home Path toinclude the word ASM. This makes it easier to identify later on. If you change the path youshould ensure that you do not use the exact same path as the Oracle Clusterware homeActionsConfirm entries are OKClick Next

Step 4

The installer has detected the presence of Oracle Clusterware and uses this to populate thisdialog box. To build a cluster which includes all nodes you must ensure that there arecheck-boxes next to the node namesActionsClick Select AllClick Next

Step 5The installer will then complete some Product-Specific Prerequisite checks. These should allpass OK – as you have already run the CVU checkActionsClick Next

Step 6

25

We are going to install a Software only home and then subsequently configure the softwareActionsSelect the Install Software Only radio buttonClick NextNotesActionClick

Step 7

Here we can see a summary of the install.ActionsClick Install

Step 8

Here the installer copies the software to all nodes in the clusterActionsnone required

Step 9

The installer pauses, some scripts need to be run as root on both nodes of the clusterActionopen a shell window on each noderun the root.sh script

Example

bash-3.00# /scratch/product/11.1.0/asm/root.shRunning Oracle 11g root.sh script...The following environment variables are set as:ORACLE_OWNER= oracle11ORACLE_HOME= /scratch/product/11.1.0/asmEnter the full pathname of the local bin directory: [/usr/local/bin]:

26

/usr/local/bin is read only. Continue without copy (y/n) or retry (r)? [y]: Warning: /usr/local/bin is read only. No files will be copied.Creating /etc/oratab file...Entries will be added to the /etc/oratab file as needed byDatabase Configuration Assistant when a database is createdFinished running generic part of root.sh script

Now product-specific root actions will be performed.Finished product-specific root actions.NotesThe output from the root.sh script should be similar to thisit should only take a few seconds to run on each node

Actionrun the scripts on all nodesthen return to the installer and Click OK

Step 10

After the software install completes you will see this End of Installation dialogActionsClick Exit

27

CHAPTER 7

Oracle ASM Software Home Patching

Once ASM software has been installed, the next step in the process is to apply the 11.1.0.7 patchset to theASM ORACLE_HOME.7.1. Start the OUI for Oracle ASM Software Home PatchingEnter the following commands to start Oracle Universal Installer, where patchset_directory is the directorywhere you unpacked the patch set software:$ cd patchset_directory/Disk1$ ./runInstaller

Step 1

the Universal Installer screen appearsActionOn the Welcome screen, click Next.

28

Step 2

Specify the name and the location of the asm homeActionCheck that the name and location are correct

Step 3

here you can specify your metalik credentials for this install.If you leave both fields blank you can opt out of notifications (see next screen).

Step 4

the installer detects that this is a clustered home and automatically selects all the nodes in theclusterActionsClick Next

Step 5

Some parameters are validated by the installerActionClick Next

Step 6

This is a summary of the actions the installer will completeActionsClick Install

Step 7

The installer copies the patch to all the nodes in the clusterActionsNo action required

The installer pauses, root.sh need to be run as root on both nodes of the clusterAction

29

open a shell window on each node and run root.sh (one after the other)then return to the installer and Click OK

Step 8

NotesThe output from the root.sh script should be similar to thisit should only take a few seconds to run on each nodeActionrun the scripts on all nodesthen return to the installer and Click OK

Step 9

After the software install completes you will see this End of Installation dialog, exit theinstallerActionsClick Yes

CHAPTER 8

Oracle RAC Database Listener Creation

8.1. Create Node specific network listeners

The Oracle network listeners traditionally run from the ASM home. Here we are going to create the listenersusing netca from the ASM home.[oracle11@stnsp001 bin]$ cd /scratch/product/11.1.0/asm/bin[oracle11@stnsp001 bin]$ ./netca

Step 1

run the Network Configuration AssistantActionSet environment variable ORACLE_HOME to ASM home locationchange directory to the ASM home bin directoryrun ./netca

30

Netca detects that the Oracle Clusterware layer is running and offers Cluster or Single NodeconfigurationActionsSelect the Cluster configuration radio buttonClick Next

Step 2

Netca uses Oracle Clusterware to determine all the nodes in the clusterActionsClick Select all nodesClick Next

Step 3

You get various options – we need to configure listenersActionsSelect the Listener configuration radio buttonClick NextNotesWe need to add a listenerActionsSelect the Add radio buttonClick Next

Step 4

NotesWe need to add a listenerActionsSelect the Add radio buttonClick Next

Step 5

31

Here you get the opportunity to name the listener – Do not change this. The listeners willeventually be called LISTENER_nodename1 & LISTENER_nodename2. This is importantfor RAC

ActionsClick Next

Step 6

Oracle Net supports various network protocols, although TCP is the most common.ActionsEnsure the Selected Protocols list includes TCPClick Next

Step 7

It is possible to choose a non-default Port –

ActionsEnsure the Use the Standard port number of 1521 radio button is setClick Next

Step 8

After configuring the node listeners you get the opportunity to configure more networkcomponentsActionsSelect the No radio buttonClick Next

Step 9

You get the opportunity to configre other networking componentsAction

32

Click the Finish button to exit the tool

8.2. Verify the Listener resources are online[oracle11@stnsp001 bin]$ cd /scratch/11.1.0/crs/bin[oracle11@stnsp001 bin]$ ./crs_stat -tName Type Target State Host------------------------------------------------------------ora....01.lsnr application ONLINE ONLINE stnsp001ora....001.gsd application ONLINE ONLINE stnsp001ora....001.ons application ONLINE ONLINE stnsp001ora....001.vip application ONLINE ONLINE stnsp001ora....02.lsnr application ONLINE ONLINE stnsp002ora....002.gsd application ONLINE ONLINE stnsp002ora....002.ons application ONLINE ONLINE stnsp002ora....002.vip application ONLINE ONLINE stnsp002NotesYou can see the listener resources inside Oracle ClusterwareActionchange directory to the Oracle Clusterware home bin directoryrun the ./crs_stat -t command

CHAPTER 9

Oracle ASM Instance and diskgroup Creation

9.1. Create ASM Instance and add the +DATA and +FLASH diskgroups using the Database Assistant

Step 1

cd /scratch/product/11.1.0/asm/bin./dbca

We use the dbca from the ASM install to create the ASM instancesActions

33

ensure the ORACLE_HOME environment variable is set to the ASM home directoryrun ./dbca from the ASM home bin directory

dbca detects the Oracle Clusterware layer is running and offers to create either cluster orsingle instance databaseActionsSelect the Oracle Real Application Clusters database radio buttonClick Next

Step 2

ActionsSelect the Configure Automatic Storage Management radio buttonClick Next

Step 3

You need to make sure you create ASM instances on all the cluster nodesActionsClick Select AllClick Next

Step 4

Here we need to specify the password for the ASM Oracle SYS user

ActionsEnter the SYS passwordEnter the same password for the Confirm SYS passwordStep 5

dbca will create and start ASM instancesActionsClick OK

Step 6

34

ASM requires disks to be group together into diskgroups. This section will be used to create 2disk groups +DATA and +FLASHActionsClick Create New

Step 7

At the moment no disks are visibleActionsClick Change Discovery Path

Step 8

here we specifya filter to allow us to see the disks on the shared arrayActionEnter a filter to allow the installer to see the disksClick OK

Step 9

Now we will assign disks to specific disk groups and create the DATA diskgroupActionsIn the Disk Group Name enter DATASelect the External Redundancy radio buttonSelect the Show All radio buttonSelect the 6 disks to be used for the DATA diskgroupClick OK

Step 10

Here we can see the DATA diskgroup has been created and is mounted on 2/2 instances. Wenow need to create the FLASH diskgroupActions

35

Click Create New

Step 11

We need to allow the installer to see the disks we haev reserved for the FLASH disk groupActionClick Change Discovery Path

Step 12

We need to modify the disk discovery stringActionModify the stringClick OK

Step 13

Now we will assign disks to specific disk groups and create the FLASH diskgroupFor a change we will create this diskgroup with Normal redundancyActionsIn the Disk Group Name enter FLASHSelect the Normal radio buttonSelect the Show Candidate disks radio buttonSelect the remaining disks allocated for the FLASH diskgroupClick OK

Step 14

Here we can see the DATA and FLASH diskgroups have been created and are mounted on2/2 instances. This completed the ASM configuration.ActionsClick FinishA confirmation dialog box

ActionsClick No

36

9.2. Verify ASM instances are online

[oracle11@stnsp001 bin]$ cd /scratch/11.1.0/crs/bin[oracle11@stnsp001 bin]$ ./crs_stat -tName Type Target State Host------------------------------------------------------------ora....SM1.asm application ONLINE ONLINE stnsp001ora....01.lsnr application ONLINE ONLINE stnsp001ora....001.gsd application ONLINE ONLINE stnsp001ora....001.ons application ONLINE ONLINE stnsp001ora....001.vip application ONLINE ONLINE stnsp001ora....SM2.asm application ONLINE ONLINE stnsp002ora....02.lsnr application ONLINE ONLINE stnsp002ora....002.gsd application ONLINE ONLINE stnsp002ora....002.ons application ONLINE ONLINE stnsp002

ora....002.vip application ONLINE ONLINE stnsp002NotesHere we can see the asm instances now as part of Oracle ClusterwareActionchange directory to the Oracle Clusterware bin directoryrun ./crs_stat –t

Once this step is complete, please follow the 11g database build standards to continue the installation Of the database software and creation of the database .

37