Deploying Oracle 10gR2 on AIX 5.3 with an IBM DS4800 ... · Deploying Oracle RAC 10g Release 2 with...

26
Deploying Oracle RAC 10g Release 2 with ASM on AIX 5.3 with an IBM DS4700 or DS4800 Storage Subsystem Installation and Configuration Guide 34996-01 Rev A November 2008 © 2008, IBM Corporation

Transcript of Deploying Oracle 10gR2 on AIX 5.3 with an IBM DS4800 ... · Deploying Oracle RAC 10g Release 2 with...

Deploying Oracle RAC 10g Release 2 with ASM on AIX 53 with an IBM DS4700 or DS4800 Storage Subsystem

Installation and Configuration Guide34996-01 Rev ANovember 2008copy 2008 IBM Corporation

Copyright copy 2008 by International Business Machines Corporation All rights reserved

Contents

Introduction 1

Required Documents 1

Benefits 2

Reference Configuration 3

Configuring the Hosts 5

Configuring the IBM Storage 9

Installing Oracle 11

Creating Directories for Oracle12

Setting Up the Oracle Users Profile 12

Installing Oracle Clusterware 12

Updating Oracle CRS13

Installing and Configuring Oracle ASM13

Updating Oracle ASM 13

Installing an Oracle Database13

Updating the Database Home 14

Configuring Listener Names 14

Creating and Configuring a Database14

Conclusion 15

Contact Information15

Appendix A References A-1

AIX A-1

IBM Storage A-1

Oracle A-1

Appendix B Troubleshooting B-1

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem iiicopy Copyright 2008 IBM Corporation

iv Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Deploying Oracle RAC 10g Release 2 with ASM on AIX 53 with an IBM DS4700 or DS4800 Storage Subsystem

IntroductionOracle Real Application Clusters (Oracle RAC) is a high-availability option to Oracle Database Oracle RAC supports the deployment of a single database across a cluster of servers providing excellent fault tolerance performance and scalability The industry is taking note of Oracle RACs growing importance in a wide range of customer implementations across all industries for transaction processing and data warehousing applications

This document presents an overview and some tips and techniques in deploying the Oracle RAC 10g Release 2 database with Oracle RAC on the IBMreg AIXreg operating system using IBM storage The database storage is administered using the Oracle Automatic Storage Management (Oracle ASM) feature

The procedure of creating a simple configuration using a two-node cluster is used as an example to illustrate the principles of performing this type of implementation Actual implementations vary in complexity and require significant expertise in all of the technologies involved

Required DocumentsThis document is intended to illustrate any deviations from the Oracle installation guide and any enhancements in implementation This document does not provide a comprehensive reference for installing all of these components For full implementation instructions please refer to the product documentation for the hardware being used for the AIX operating system for the Oracle Database and for Oracle Clusterware

For IBM storage documentation contact your IBM business partner or service representative

For AIX documentation go to AIX Information

httppublibboulderibmcominfocentersystemsindexjsptopic=comibmaixdocdocbaseaixparenthtm

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 1copy Copyright 2008 IBM Corporation

Introduction

For Oracle documentation go to

httpwwworaclecomtechnologydocumentationindexhtml

Before installation you must know how to best configure the storage for enhanced performance For this information refer to Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to

wwwibmcomsupporttechdocs

BenefitsThe reference configuration provides insight into how to configure these products to work together to achieve numerous benefits The reference configuration uses Oracle ASM to manage the Oracle RAC 10g Release 2 database storage running under Oracle Clusterware on AIX with an IBM DS48001 storage subsystem The IBM DS4800 storage subsystem is a robust high-performance disk array that provides a scalable flexible and expandable storage foundation upon which to build a highly reliable database system using Oracle RAC

Oracle RAC offers these benefits

bull High availability ndash Provides enhanced uptime for database applications

bull Scalability ndash Expands capacity by adding servers to the cluster

bull Lower computing costs ndash Uses lower-cost hardware and reduces expenses associated with application downtime

bull Excellent performance ndash Spreads the database load across multiple clustered servers allowing performance to scale with the size of the cluster

The IBM DS4800 storage subsystem offers these benefits which go hand-in-hand with those of Oracle RAC

bull High availability ndash The IBM DS4800 storage subsystemsrsquo fully redundant hardware and redundant array of independent disks (RAID) functionality ensures that the storage is as available as the RAC database

bull Scalability ndash Additional storage capacity can be added to the storage subsystem while it continues operation

bull Lower costs ndash Shared storage within a storage area network (SAN) allows reduced cost compared to storage local to individual hosts and reduces management expenses associated with administering the storage

bull Excellent performance ndash The IBM DS4800 storage subsystem is one of the best performing disk arrays in its class Its full front-to-back 4 gigabit Fibre Channel interface architecture provides bandwidth for the most demanding applications

1Solution also works with a DS4700 storage subsystem

2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

The DS4000 Storage Manager software package configures and manages the IBMDS4800 storage subsystemrsquos fully redundant hardware and high-performance architecture with ease DS4000 Storage Manager allows in-band management from servers to attached storage or remotely through the Ethernet The storage subsystem offers several premium features that add functionality when used in a database environment Two of the most helpful features in an Oracle RAC environment are the Storage Partitioning premium feature and the Enhanced Remote Mirroring premium feature

The Storage Partitioning premium feature allows logical drives on the storage subsystem to be distributed to different hosts in your SAN associating each logical drive with one particular host or a group of hosts The logical drives are presented to each host according to the host type for the different operating systems that are running on those hosts The Storage Partitioning premium feature allows great flexibility in the use of the storage subsystem for storage consolidation

The Enhanced Remote Mirroring premium feature (RM) provides the ability to mirror logical drives between two IBM storage subsystems Mirroring provides an additional redundant copy of the data The mirror copy can be used for rapid disaster recovery and can be also used to create a copy of live data for test purposes

Although these two premium features are not used in the reference configuration there are many opportunities to use storage partitioning and Enhanced Remote Mirroring in a real-world Oracle implementation

For more information please discuss these premium features with your IBM business partner or sales representative

Reference ConfigurationThe reference configuration consists of the following equipment

bull 2 IBM P5 p630 servers

bull AIX 53 Technology Level 6

bull IBM DS4800 storage subsystem

bull Oracle Cluster Ready Services (Oracle CRS) 10201 upgraded to 10203

bull Oracle Database 10201 upgraded to 10203

The systems are configured as a two-node RAC cluster Oracle ASM is employed to simplify storage management The AIX RDAC multipathing driver provided failover support

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 3copy Copyright 2008 IBM Corporation

Introduction

Figure 1 Reference Configuration

4 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

Configuring the HostsThe hosts must be configured to function correctly with Oracle RAC The following steps describe in general terms how to prepare the systems for the Oracle installation

IMPORTANT Refer to Metalink Bulletin 2820361 Minimum Software Versions and Patches Required to Support Oracle Products on IBM Series for the latest requirements This bulletin is available to Metalink users athttpmetalinkoraclecom Metalink Bulletin 2820361 is regularly updated and the information it contains supersedes any information found in this document

1 Install the AIX operating system if it is not already installed The reference configuration uses AIX 53 Technology Level 6

2 Install additional software packages required for Oracle

bosadtbase

bosadtlib

bosadtlibm

bosperflibperfstat

bosperfperfstat

bosperfproctools

rsctbasicrte

rsctcompatclientsrte

xlCaix50rte7004 (See the following note)

xlCrte7001 (See the following note)

NOTE The versions of xlCaix50rte and xlCrte supplied on the AIX 53 distribution media are not compatible with Oracle RAC 10g Release 2 You must download newer versions from IBMs web site The reference configuration uses fileset xlcrteaix50apr2007ptftarZ

3 Download and install the updates to the AIX operating system from IBMs website

httpwww14softwareibmcomwebappset2sasfgenunix3aixfixeshtml

The reference configuration uses AIX 53 Technology Level 6

4 Configure the hosts network interfaces

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 5copy Copyright 2008 IBM Corporation

Configuring the Hosts

Use at least two ethernet interfaces

One for the Oracle public network

One for the private network used by Oracle CRS

Oracle recommends that the private interface be on a separate network isolated from all other traffic You should use a separate gigabit ethernet switch for the private network

Additional interfaces provide more flexibility in configuration and higher performance In the reference configuration the VIP network shares the interface used for the public network This is configured later in the installation process using the vipca utility Entries for the internet protocol (IP) address as well as the public and private networks are inserted into the etchosts file at this time

5 Create users and groups

a Create two groups dba and oinstall

b Add a user Oracle with the oinstall primary group and with the dba group

NOTE Make sure that the user ID for the Oracle user and the group numbers for the dba and oinstall groups are consistent on all nodes

6 Set the shell soft limits for the Oracle and root users using the smit chuser command

Soft FILE size = -1

Soft CPU time = -1

Soft DATA segment = -1

Soft STACK size = -1

7 Set system configuration parameters

a Use the smit chgsys command to set the maximum number of processes per user to at least 2048

b Use the vmo -p -o lru_file_repage=0 command to set the lru_file_repage parameter to 0

NOTE This setting instructs the AIX Virtual Memory Manager to take pages only from the file buffer cache allowing Oraclersquos System Global Area (SGA) to remain in memory for better performance

8 Use the usrsbinno -a command to configure the network tuning parameters

Use these minimum values for network tuning parameters

6 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

ipqmaxlen = 512rfc1323 = 1sb_max = 1310720 (2655360)tcp_recvspace = 65536 tcp_sendspace = 65536 udp_recvspace = 655360 (10 udp_sendspace must be less than sb_max)udp_sendspace = 65536

NOTE If you modify these parameters the method depends on whether the system is in compatibility mode For specific instructions refer to the AIX documentation for the no command

NOTE If you change the value of the ipqmaxlen parameter you must restart the system for the change to take effect

9 Download and install the Open Secure Shell (OpenSSH) utility from the IBM web site

httpwwwibmcomdeveloperworkseserverarticlesopenssh_updatedhtml

NOTE The reference configuration uses OpenSSH The SSH utility is not required for Oracle to function however SSH is recommended because it provides a more secure method of communication between nodes than does RSH If SSH is available Oracle will use it otherwise Oracle uses RSH Configuration of RSH user equivalence is outside the scope of this document

10 Configure SSH for root login permission

a Locate the sshd_config file (typically etc or usrlocaletc)

b Edit the sshd_config file changing PermitRootLogin to yes

c Use the startsrc -s sshd command to start sshd

11 Configure SSH for user equivalence for root and Oracle on all nodes

a Log on as ltrootgt

b Use the following commands to generate the SSH key

cd ~ssh

ssh-keygen -t dsa

cat id_dsapub gtgt authorized_keys

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 7copy Copyright 2008 IBM Corporation

Configuring the Hosts

c Copy the authorized_keys file to ~ssh on the next system

d Repeat step b and step c on each system

e After all systems have a public key in the file copy the file back to all of the previous systems so that they all have an identical file

f Test the SSH function from each system to itself and to each of the other systems by issuing such commands as the ssh host1 date command Make sure that the commands can be executed without using a password

g Log on as the Oracle user and repeat step a through step f so that Oracle can also perform remote commands without passwords

NOTE The permissions must restrict write access on both the base user directory and the ssh directory If the permissions are not correctly set OpenSSH requires a password even if the appropriate public key has been added to the authorized_keys file

12 Edit the profile file in the Oracle userrsquos home directory to set the Oracle users parameters

Table 1 shows the entries that you must add to the profile file

Table 1 Oracle Userrsquos Parameters

Parameter to Add Explanation

umask 022

AIXTHREAD_SCOPE=S

An environment variable that directs the system to place user threads within a system-wide rather than a process-wide contention scope This placement provides better performance and more efficient memory use

CRS_HOME A unique home directory for Oracle CRS binaries

ASM_HOME

Use this parameter if you prefer to install separate Oracle binaries for Oracle ASM This method is preferable if there will be more than one database instance supported by Oracle ASM

ORACLE_HOME A unique home for the database binaries

LD_LIBRARY_PATH The complete pathway is $ORACLE_HOMElib

ORACLE_BASE Identifies the Oracle base directory

ORACLE_SID Identifies the instance

PATH Include $ORACLE_HOMEbin

CLASSPATHInclude $ORACLE_HOMEjdbclibltclasses filesgt

8 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

13 Edit the profile file in the root userrsquos home directory to set the same parameters as set in the Oracle users profile

Be sure to add the following entry to the profile file

AIXTHREAD_SCOPE=S

14 Synchronize the system time between the hosts in the cluster using any effective method The reference configuration used Network Time Protocol (NTP) If an NTP server is available use this method to set up xntpd

a Edit etcntpconf with the nameIP of the NTP server(s)

b Add a line server lthostnamegt for each host

c Start the service using the smit xntpd command

15 Because some Oracle utilities expect lsattr in etc and do not function correctly without a link to it use the ln -s usrsbinlsattr etclsattr command to add a soft link

Configuring the IBM Storage

IMPORTANT You must know how to best configure the storage for enhanced performance For this information refer to Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to wwwibmcomsupporttechdocs

Use DS4000 Storage Manager and the AIX system utilities to configure the IBM storage subsystem

1 Use DS4000 Storage Manager to create logical drives on the storage subsystem to be shared for Oracle CRS communication and database storage

NOTE You also may use additional logical drives for Oracle binaries and similar purposes however for those logical drives on the storage subsystem that are used exclusively by one particular host you will need the Storage Partitioning premium feature

You will need to analyze your implementation to determine the appropriate logical drive sizes desired RAID levels and other parameters The reference configuration uses the following minimal configuration of shared logical drives

2 Oracle Cluster Registry (OCR) logical drives (300 MB RAID1)

3 voting disks (300MB RAID1)

1 logical drive for Oracle ASMs SPFILE (100MB RAID1)

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 9copy Copyright 2008 IBM Corporation

Configuring the IBM Storage

2 logical drives for Oracle ASM data storage (300 GB RAID5)

2 logical drives for Oracle ASM log storage (135GB RAID1)

The number and the size of the logical drives created for Oracle ASM storage varies based on the individual implementation but the sizes used for the OCR disks voting disks and the Oracle ASM SPFILE disks in the reference configuration are sufficient for any configuration

In the reference configuration the Oracle binaries are installed on internal disks within each individual server The reference configuration uses the JFS2 file system and the logical drive was mounted as u01

2 Configure the shared disks in the operating system

After you have created the logical drives on the storage subsystem the cfgmgr command rescans the AIX system for changes to the physical devices

The lspv command shows the physical devices that AIX has configured

The fget_config -Av command shows the LUN numbers the disk drive name and the associated logical drive names configured on the storage subsystem

3 If the hdisk names for each logical drive on the storage subsystem that will be shared in the cluster are not identical on all cluster nodes create new device nodes for each of the logical drives to give them a consistent name across all nodes

For example if the logical drive for the first OCR disk was hdisk2 on node1 and hdisk3 on node2 a new device node (devnode) called devORA_OCR1 could be created on each host using the major numbers and the minor numbers that correspond with the same logical drive on each system Show the major and the minor numbers with the command ls -l Use the command mknod devORA_OCR1 c ltmajor of OCRgt ltminor of OCRgt to create the new devnode

Even when the disk drive names correlate on all of the nodes you should still create these named device nodes for ease of location and administration Named device nodes were created on the reference configuration for all shared logical drives using the convention devORA_ltusegt

4 Set the reserve_policy attribute for each logical drive to be used for shared access for OCR logical drive voting logical drive or Oracle ASM logical drive

NOTE This operation must be performed on the hdisk device name rather than on the named devnode created in step 3

This operation allows the nodes to share the logical drives Otherwise the first system that accesses the logical drive reserves it for exclusive use Oracle coordinates access to the shared logical drives

10 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

For an IBM storage subsystem reserve_policy=no_reserve is the correct value

a Use the lsattr -E -l hdisk | grep reserve command to check the attribute

b Use the chdev -l hdisk -a reserve_policy=no_reserve command to change the policy

c Repeat step a and step b on each cluster node

5 Set ownership and permissions on devnodes to be used by Oracle

NOTE If you created new devnodes for the shared logical drives set the ownership and permissions on the new devnode that you created not on the original hdisk devnode

a Use the following commands to set the ownership for each OCR device file

chown rootoinstall devltdevnodegt

chmod 640 devltdevnodegt

b Use the following commands to set the ownership and permission on the voting disk and database device files

chown oracledba devltdevnodegt

chmod 660 devltdevnodegt

c Repeat step a and step b on each cluster node

NOTE Do not assign a physical volume identifier (PVID) to hdisks that will be used with Oracle ASM

Installing OracleHere are the basic steps for installing Oracle

bull Create directories for Oracle

bull Set up the Oracle userrsquos profile

bull Install Oracle Clusterware

bull Update Oracle CRS

bull Install and configure Oracle ASM

bull Update Oracle ASM

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 11copy Copyright 2008 IBM Corporation

Installing Oracle Clusterware

bull Install Oracle Database

bull Update the database home

bull Configure the listener names

bull Create and configure a database

Creating Directories for OracleAlthough you can create directory structures for Oracle files in many different ways for ease of support use a standard directory structure based on the Optimal Flexible Architecture (OFA) guidelines published by Oracle For more information about the OFA Guidelines see Appendix D of the Oracle Database Installation Guide 10gR2 for AIX 5L Based Systems (B19075-02) The reference configuration is based on the OFA guidelines

Follow these two important guidelines when creating directories for Oracle

bull You must separate the Oracle CRS home directory from the Oracle base and the database home directories The Oracle CRS installation procedure sets the ownership for the entire Oracle CRS directory tree to root while the database directory tree must be owned by oracle

bull If more than one database will be serviced by Oracle ASM it is recommended that you create a separate home directory for Oracle ASM An Oracle ASM home directory provides the capability to modify the Oracle release for Oracle ASM while not changing the release of any of the databases on the same server The directory tree must be owned by oracle

Setting Up the Oracle Users Profile1 Set up the Oracle users profile before installing the Oracle binaries See

Appendix A for an example profile

2 Add or modify entries for ORACLE_HOME ORACLE_BASE PATH LD_LIBRARY_PATH and similar files to reflect the directories where the Oracle binaries will be placed during the Oracle Clusterware installation

Installing Oracle ClusterwareAs you install the Oracle Clusterware remember the following important points

bull Do not make the CRS_HOME path a subdirectory of the Oracle RAC 10g Release 2 Oracle base directory due to permission changes required for Oracle CRS For more details refer to Appendix E of the Oracle Clusterware Installation Guide

bull The rootsh script must be run as the root user and must be run on only one node at a time not concurrently

12 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Updating Oracle CRS

bull After rootsh runs on the last node it will attempt to run the vipca utility to configure the VIP network interfaces If you have IP addresses that are not routable across the Internet for example 100 192168 the vipca configuration will fail If the automatic vipca configuration fails run the vipca utility manually using the $ORA_CRS_HOMEbinvipca command from the command line as root This requires a graphical display

bull You can do the manual configuration with vipca before clicking OK on the Oracle Universal Installer the Oracle CRS installation page The Oracle CRS installation page waits for you to run rootsh Do the manual configuration with vipca during this time This action allows the remainder of the post-installation checks to complete successfully

Updating Oracle CRSRefer to the Metalink Bulletin 2820361 (see important note on page 5) for required updates The reference configuration used the 10203 update to the Oracle CRS home Download the 10203 update and any other updates required from the Oracle Metalink web site (httpmetalinkoraclecom) and install according to the instructions on the web site

Installing and Configuring Oracle ASMBecause a single Oracle ASM instance can service many database instances Oracle recommends that you install Oracle ASM in its own home directory Follow the naming convention of the OFA guidelines for the directory path As you install and configure Oracle ASM remember the following important points

bull If the Oracle ASM instance was not correctly created by the Oracle Universal Installer Wizard during installation use the dbca utility to add the instance and create the disk group(s)

bull If you want additional disk groups create them using the dbca utility

Updating Oracle ASMUpdate the Oracle ASM instance with the 10203 patch set and any additional updates indicated in Metalink Bulletin 2820361 You can use the same update media for 10203 previously downloaded to update Oracle CRS (page 13) Carefully follow the instructions for updating this database instance

Installing an Oracle DatabaseUse the Oracle Universal Installer to install another database home for a database instance Select Install Database Software Only on the Create Database page

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 13copy Copyright 2008 IBM Corporation

Installing Oracle Clusterware

Updating the Database HomeInstall the 10203 update on the new database home Select the database home instead of Oracle ASM on the Specify Home Details page

Configuring Listener NamesIf your RAC environment consists of only Oracle CRS an Oracle ASM instance and a database instance on each node it is easier to support only one listenerora configuration file for the Oracle ASM and database instances The best way to accomplish this is to create an environment variable TNS_ADMIN and assign it to the $ORACLE_HOMEnetworkadmin directory of the database instance For example

export TNS_ADMIN=$ORA_ASM_HOMEnetworkadmin

This environment variable allows the DBA to maintain one set of tnsnamesora sqlnetora and listenerora files for the entire environment

Creating and Configuring a DatabaseUse the Database Configuration Assistant (DBCA) to create a database For more information refer to the Oracle documentation about using DBCA As you create the database remember the following important points

bull Using DBCA gives you the capability to create the Oracle ASM disk groups if you choose to use Oracle ASM

bull The Flash Recovery Area (FRA) option is highly recommended A FRA allows the setup of a separate area on disk for flashback capability area for the archive logs the second copy of the multiplexed redo log files and for RMAN backups Make sure that the FRA is correctly set up to handle the performance of the database and that the FRA contains enough space

Space to hold all of the data for the data retention periods that are set by the database

Space for the type of backups that will be used

Space for the number of backups that will be held

For further documentation about using the FRA refer to the Oracle documentation as well as the following white paper

Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to

wwwibmcomsupporttechdocs

14 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Creating and Configuring a Database

ConclusionAs you have followed the example of setting up a two-node cluster described in this document you have seen how simple and how effective a deployment of Oracle RAC 10g Release 2 can be When you are doing your own installation be sure to use the applicable installation information as the basis for your work This document describes only tips and techniques Your installer should have the expertise to know how and when to modify instructions as necessary If your environment is complex and you require outside help contact your IBM business partner or sales representative See ldquoContact Informationrdquo below

Contact InformationFor more information and sales office locations visit the IBM web site at

httpwwwibmcom

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 15copy Copyright 2008 IBM Corporation

Contact Information

16 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix A

References

AIXFor AIX operating system downloads go to Quick Links for AIX Fixes

httpwww-933ibmcomeserversupportfixesfixcentralmainpseriesaix

For AIX documentation go to AIX Information

httppublibboulderibmcominfocentersystemsindexjsptopic=comibmaixdocdocbaseaixparenthtm

IBM Corporation 2006 Get the Latest Version of OpenSSH for AIX

httpwwwibmcomdeveloperworkseserverarticlesopenssh_updatedhtml

IBM StorageFor information about the IBM DS4800 Storage Subsystem go to

httpwww-03ibmcomsystemsstoragediskds4000ds4800indexhtml

For information about IBM storage go to

httpwww-03ibmcomsystemsstoragediskcm_re=masthead-_-products-_-stg-disk

OracleFor Oracle documentation go to

httpwwworaclecomtechnologydocumentationindexhtml

IBM Corporation 2008 Tuning IBM AIX 5L for an Oracle Database

httpwwwibmcompartnerworldwpsservletContentHandlerVPAA-6V2C4E

IBM Corporation 2008 Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems

wwwibmcomsupporttechdocs

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem A-1copy Copyright 2008 IBM Corporation

A-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix B

Troubleshooting

CSS Node Evictions can affect your functionality

IO time-outs can cause inappropriate node evictions during multi-path failover or other problems Two parameters might affect this css_misscount and disktimeout With versions of Oracle prior to 102 this was primarily accommodated by increasing the value of the css_misscount parameter However the function of this parameter has changed as of Oracle 102 and changes to this parameter might not achieve the desired result The new disktimeout parameter should be used in most cases to make this adjustment In many cases the default values will work fine However you might find it necessary to increase the default value of this parameter if you are getting a lot of CSS node evictions

For information about how the parameters work and how to adjust them refer to the following two articles Go to httpmetalinkoraclecom

bull Metalink DocID 2944301 CSS Timeout Computation in RAC 10g

bull Metalink DocID 2847521 Steps to Increase CSS Misscount Reboottime and Disktimeout

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem B-1copy Copyright 2008 IBM Corporation

B-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Trademarks and special notices copy Copyright IBM Corporation 2008 All rights reserved

References in this document to IBM products or services do not imply that IBM intends to make them available in every country

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corporation in the United States other countries or both If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (reg or trade) these symbols indicate US registered or common law trademarks owned by IBM at the time this information was published Such trademarks may also be registered or common law trademarks in other countries A current list of IBM trademarks is available on the Web at Copyright and trademark information at wwwibmcomlegalcopytradeshtml

Intel Intel Inside (logos) MMX and Pentium are trademarks of Intel Corporation in the United States other countries or both

UNIX is a registered trademark of The Open Group in the United States and other countries

Linux is a trademark of Linus Torvalds in the United States other countries or both

Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both

Other company product or service names may be trademarks or service marks of others

Information is provided AS IS without warranty of any kind

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved Actual environmental costs and performance characteristics may vary by customer

Information concerning non-IBM products was obtained from a supplier of these products published announcement material or other publicly available sources and does not constitute an endorsement of such products by IBM Sources for non-IBM list prices and performance numbers are taken from publicly available information including vendor announcements and vendor worldwide homepages IBM has not tested these products and cannot confirm the accuracy of performance capability or any other claims related to non-IBM products Questions on the capability of non-IBM products should be addressed to the supplier of those products

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk

  • Title Page13
  • Contents
  • Deploying Oracle RAC 10g Release 2 with ASM on AIX 53 with an IBM DS4700 or DS4800 Storage Subsystem
    • Introduction
      • Required Documents
      • Benefits
      • Reference Configuration
        • Configuring the Hosts
        • Configuring the IBM Storage
        • Installing Oracle
          • Creating Directories for Oracle
          • Setting Up the Oracle Users Profile
            • Installing Oracle Clusterware
              • Updating Oracle CRS
              • Installing and Configuring Oracle ASM
              • Updating Oracle ASM
              • Installing an Oracle Database
              • Updating the Database Home
              • Configuring Listener Names
              • Creating and Configuring a Database
                • Conclusion
                • Contact Information
                  • Appendix A References
                    • AIX
                    • IBM Storage
                    • Oracle
                      • Appendix B Troubleshooting
                      • Copyrights13

Copyright copy 2008 by International Business Machines Corporation All rights reserved

Contents

Introduction 1

Required Documents 1

Benefits 2

Reference Configuration 3

Configuring the Hosts 5

Configuring the IBM Storage 9

Installing Oracle 11

Creating Directories for Oracle12

Setting Up the Oracle Users Profile 12

Installing Oracle Clusterware 12

Updating Oracle CRS13

Installing and Configuring Oracle ASM13

Updating Oracle ASM 13

Installing an Oracle Database13

Updating the Database Home 14

Configuring Listener Names 14

Creating and Configuring a Database14

Conclusion 15

Contact Information15

Appendix A References A-1

AIX A-1

IBM Storage A-1

Oracle A-1

Appendix B Troubleshooting B-1

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem iiicopy Copyright 2008 IBM Corporation

iv Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Deploying Oracle RAC 10g Release 2 with ASM on AIX 53 with an IBM DS4700 or DS4800 Storage Subsystem

IntroductionOracle Real Application Clusters (Oracle RAC) is a high-availability option to Oracle Database Oracle RAC supports the deployment of a single database across a cluster of servers providing excellent fault tolerance performance and scalability The industry is taking note of Oracle RACs growing importance in a wide range of customer implementations across all industries for transaction processing and data warehousing applications

This document presents an overview and some tips and techniques in deploying the Oracle RAC 10g Release 2 database with Oracle RAC on the IBMreg AIXreg operating system using IBM storage The database storage is administered using the Oracle Automatic Storage Management (Oracle ASM) feature

The procedure of creating a simple configuration using a two-node cluster is used as an example to illustrate the principles of performing this type of implementation Actual implementations vary in complexity and require significant expertise in all of the technologies involved

Required DocumentsThis document is intended to illustrate any deviations from the Oracle installation guide and any enhancements in implementation This document does not provide a comprehensive reference for installing all of these components For full implementation instructions please refer to the product documentation for the hardware being used for the AIX operating system for the Oracle Database and for Oracle Clusterware

For IBM storage documentation contact your IBM business partner or service representative

For AIX documentation go to AIX Information

httppublibboulderibmcominfocentersystemsindexjsptopic=comibmaixdocdocbaseaixparenthtm

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 1copy Copyright 2008 IBM Corporation

Introduction

For Oracle documentation go to

httpwwworaclecomtechnologydocumentationindexhtml

Before installation you must know how to best configure the storage for enhanced performance For this information refer to Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to

wwwibmcomsupporttechdocs

BenefitsThe reference configuration provides insight into how to configure these products to work together to achieve numerous benefits The reference configuration uses Oracle ASM to manage the Oracle RAC 10g Release 2 database storage running under Oracle Clusterware on AIX with an IBM DS48001 storage subsystem The IBM DS4800 storage subsystem is a robust high-performance disk array that provides a scalable flexible and expandable storage foundation upon which to build a highly reliable database system using Oracle RAC

Oracle RAC offers these benefits

bull High availability ndash Provides enhanced uptime for database applications

bull Scalability ndash Expands capacity by adding servers to the cluster

bull Lower computing costs ndash Uses lower-cost hardware and reduces expenses associated with application downtime

bull Excellent performance ndash Spreads the database load across multiple clustered servers allowing performance to scale with the size of the cluster

The IBM DS4800 storage subsystem offers these benefits which go hand-in-hand with those of Oracle RAC

bull High availability ndash The IBM DS4800 storage subsystemsrsquo fully redundant hardware and redundant array of independent disks (RAID) functionality ensures that the storage is as available as the RAC database

bull Scalability ndash Additional storage capacity can be added to the storage subsystem while it continues operation

bull Lower costs ndash Shared storage within a storage area network (SAN) allows reduced cost compared to storage local to individual hosts and reduces management expenses associated with administering the storage

bull Excellent performance ndash The IBM DS4800 storage subsystem is one of the best performing disk arrays in its class Its full front-to-back 4 gigabit Fibre Channel interface architecture provides bandwidth for the most demanding applications

1Solution also works with a DS4700 storage subsystem

2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

The DS4000 Storage Manager software package configures and manages the IBMDS4800 storage subsystemrsquos fully redundant hardware and high-performance architecture with ease DS4000 Storage Manager allows in-band management from servers to attached storage or remotely through the Ethernet The storage subsystem offers several premium features that add functionality when used in a database environment Two of the most helpful features in an Oracle RAC environment are the Storage Partitioning premium feature and the Enhanced Remote Mirroring premium feature

The Storage Partitioning premium feature allows logical drives on the storage subsystem to be distributed to different hosts in your SAN associating each logical drive with one particular host or a group of hosts The logical drives are presented to each host according to the host type for the different operating systems that are running on those hosts The Storage Partitioning premium feature allows great flexibility in the use of the storage subsystem for storage consolidation

The Enhanced Remote Mirroring premium feature (RM) provides the ability to mirror logical drives between two IBM storage subsystems Mirroring provides an additional redundant copy of the data The mirror copy can be used for rapid disaster recovery and can be also used to create a copy of live data for test purposes

Although these two premium features are not used in the reference configuration there are many opportunities to use storage partitioning and Enhanced Remote Mirroring in a real-world Oracle implementation

For more information please discuss these premium features with your IBM business partner or sales representative

Reference ConfigurationThe reference configuration consists of the following equipment

bull 2 IBM P5 p630 servers

bull AIX 53 Technology Level 6

bull IBM DS4800 storage subsystem

bull Oracle Cluster Ready Services (Oracle CRS) 10201 upgraded to 10203

bull Oracle Database 10201 upgraded to 10203

The systems are configured as a two-node RAC cluster Oracle ASM is employed to simplify storage management The AIX RDAC multipathing driver provided failover support

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 3copy Copyright 2008 IBM Corporation

Introduction

Figure 1 Reference Configuration

4 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

Configuring the HostsThe hosts must be configured to function correctly with Oracle RAC The following steps describe in general terms how to prepare the systems for the Oracle installation

IMPORTANT Refer to Metalink Bulletin 2820361 Minimum Software Versions and Patches Required to Support Oracle Products on IBM Series for the latest requirements This bulletin is available to Metalink users athttpmetalinkoraclecom Metalink Bulletin 2820361 is regularly updated and the information it contains supersedes any information found in this document

1 Install the AIX operating system if it is not already installed The reference configuration uses AIX 53 Technology Level 6

2 Install additional software packages required for Oracle

bosadtbase

bosadtlib

bosadtlibm

bosperflibperfstat

bosperfperfstat

bosperfproctools

rsctbasicrte

rsctcompatclientsrte

xlCaix50rte7004 (See the following note)

xlCrte7001 (See the following note)

NOTE The versions of xlCaix50rte and xlCrte supplied on the AIX 53 distribution media are not compatible with Oracle RAC 10g Release 2 You must download newer versions from IBMs web site The reference configuration uses fileset xlcrteaix50apr2007ptftarZ

3 Download and install the updates to the AIX operating system from IBMs website

httpwww14softwareibmcomwebappset2sasfgenunix3aixfixeshtml

The reference configuration uses AIX 53 Technology Level 6

4 Configure the hosts network interfaces

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 5copy Copyright 2008 IBM Corporation

Configuring the Hosts

Use at least two ethernet interfaces

One for the Oracle public network

One for the private network used by Oracle CRS

Oracle recommends that the private interface be on a separate network isolated from all other traffic You should use a separate gigabit ethernet switch for the private network

Additional interfaces provide more flexibility in configuration and higher performance In the reference configuration the VIP network shares the interface used for the public network This is configured later in the installation process using the vipca utility Entries for the internet protocol (IP) address as well as the public and private networks are inserted into the etchosts file at this time

5 Create users and groups

a Create two groups dba and oinstall

b Add a user Oracle with the oinstall primary group and with the dba group

NOTE Make sure that the user ID for the Oracle user and the group numbers for the dba and oinstall groups are consistent on all nodes

6 Set the shell soft limits for the Oracle and root users using the smit chuser command

Soft FILE size = -1

Soft CPU time = -1

Soft DATA segment = -1

Soft STACK size = -1

7 Set system configuration parameters

a Use the smit chgsys command to set the maximum number of processes per user to at least 2048

b Use the vmo -p -o lru_file_repage=0 command to set the lru_file_repage parameter to 0

NOTE This setting instructs the AIX Virtual Memory Manager to take pages only from the file buffer cache allowing Oraclersquos System Global Area (SGA) to remain in memory for better performance

8 Use the usrsbinno -a command to configure the network tuning parameters

Use these minimum values for network tuning parameters

6 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

ipqmaxlen = 512rfc1323 = 1sb_max = 1310720 (2655360)tcp_recvspace = 65536 tcp_sendspace = 65536 udp_recvspace = 655360 (10 udp_sendspace must be less than sb_max)udp_sendspace = 65536

NOTE If you modify these parameters the method depends on whether the system is in compatibility mode For specific instructions refer to the AIX documentation for the no command

NOTE If you change the value of the ipqmaxlen parameter you must restart the system for the change to take effect

9 Download and install the Open Secure Shell (OpenSSH) utility from the IBM web site

httpwwwibmcomdeveloperworkseserverarticlesopenssh_updatedhtml

NOTE The reference configuration uses OpenSSH The SSH utility is not required for Oracle to function however SSH is recommended because it provides a more secure method of communication between nodes than does RSH If SSH is available Oracle will use it otherwise Oracle uses RSH Configuration of RSH user equivalence is outside the scope of this document

10 Configure SSH for root login permission

a Locate the sshd_config file (typically etc or usrlocaletc)

b Edit the sshd_config file changing PermitRootLogin to yes

c Use the startsrc -s sshd command to start sshd

11 Configure SSH for user equivalence for root and Oracle on all nodes

a Log on as ltrootgt

b Use the following commands to generate the SSH key

cd ~ssh

ssh-keygen -t dsa

cat id_dsapub gtgt authorized_keys

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 7copy Copyright 2008 IBM Corporation

Configuring the Hosts

c Copy the authorized_keys file to ~ssh on the next system

d Repeat step b and step c on each system

e After all systems have a public key in the file copy the file back to all of the previous systems so that they all have an identical file

f Test the SSH function from each system to itself and to each of the other systems by issuing such commands as the ssh host1 date command Make sure that the commands can be executed without using a password

g Log on as the Oracle user and repeat step a through step f so that Oracle can also perform remote commands without passwords

NOTE The permissions must restrict write access on both the base user directory and the ssh directory If the permissions are not correctly set OpenSSH requires a password even if the appropriate public key has been added to the authorized_keys file

12 Edit the profile file in the Oracle userrsquos home directory to set the Oracle users parameters

Table 1 shows the entries that you must add to the profile file

Table 1 Oracle Userrsquos Parameters

Parameter to Add Explanation

umask 022

AIXTHREAD_SCOPE=S

An environment variable that directs the system to place user threads within a system-wide rather than a process-wide contention scope This placement provides better performance and more efficient memory use

CRS_HOME A unique home directory for Oracle CRS binaries

ASM_HOME

Use this parameter if you prefer to install separate Oracle binaries for Oracle ASM This method is preferable if there will be more than one database instance supported by Oracle ASM

ORACLE_HOME A unique home for the database binaries

LD_LIBRARY_PATH The complete pathway is $ORACLE_HOMElib

ORACLE_BASE Identifies the Oracle base directory

ORACLE_SID Identifies the instance

PATH Include $ORACLE_HOMEbin

CLASSPATHInclude $ORACLE_HOMEjdbclibltclasses filesgt

8 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

13 Edit the profile file in the root userrsquos home directory to set the same parameters as set in the Oracle users profile

Be sure to add the following entry to the profile file

AIXTHREAD_SCOPE=S

14 Synchronize the system time between the hosts in the cluster using any effective method The reference configuration used Network Time Protocol (NTP) If an NTP server is available use this method to set up xntpd

a Edit etcntpconf with the nameIP of the NTP server(s)

b Add a line server lthostnamegt for each host

c Start the service using the smit xntpd command

15 Because some Oracle utilities expect lsattr in etc and do not function correctly without a link to it use the ln -s usrsbinlsattr etclsattr command to add a soft link

Configuring the IBM Storage

IMPORTANT You must know how to best configure the storage for enhanced performance For this information refer to Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to wwwibmcomsupporttechdocs

Use DS4000 Storage Manager and the AIX system utilities to configure the IBM storage subsystem

1 Use DS4000 Storage Manager to create logical drives on the storage subsystem to be shared for Oracle CRS communication and database storage

NOTE You also may use additional logical drives for Oracle binaries and similar purposes however for those logical drives on the storage subsystem that are used exclusively by one particular host you will need the Storage Partitioning premium feature

You will need to analyze your implementation to determine the appropriate logical drive sizes desired RAID levels and other parameters The reference configuration uses the following minimal configuration of shared logical drives

2 Oracle Cluster Registry (OCR) logical drives (300 MB RAID1)

3 voting disks (300MB RAID1)

1 logical drive for Oracle ASMs SPFILE (100MB RAID1)

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 9copy Copyright 2008 IBM Corporation

Configuring the IBM Storage

2 logical drives for Oracle ASM data storage (300 GB RAID5)

2 logical drives for Oracle ASM log storage (135GB RAID1)

The number and the size of the logical drives created for Oracle ASM storage varies based on the individual implementation but the sizes used for the OCR disks voting disks and the Oracle ASM SPFILE disks in the reference configuration are sufficient for any configuration

In the reference configuration the Oracle binaries are installed on internal disks within each individual server The reference configuration uses the JFS2 file system and the logical drive was mounted as u01

2 Configure the shared disks in the operating system

After you have created the logical drives on the storage subsystem the cfgmgr command rescans the AIX system for changes to the physical devices

The lspv command shows the physical devices that AIX has configured

The fget_config -Av command shows the LUN numbers the disk drive name and the associated logical drive names configured on the storage subsystem

3 If the hdisk names for each logical drive on the storage subsystem that will be shared in the cluster are not identical on all cluster nodes create new device nodes for each of the logical drives to give them a consistent name across all nodes

For example if the logical drive for the first OCR disk was hdisk2 on node1 and hdisk3 on node2 a new device node (devnode) called devORA_OCR1 could be created on each host using the major numbers and the minor numbers that correspond with the same logical drive on each system Show the major and the minor numbers with the command ls -l Use the command mknod devORA_OCR1 c ltmajor of OCRgt ltminor of OCRgt to create the new devnode

Even when the disk drive names correlate on all of the nodes you should still create these named device nodes for ease of location and administration Named device nodes were created on the reference configuration for all shared logical drives using the convention devORA_ltusegt

4 Set the reserve_policy attribute for each logical drive to be used for shared access for OCR logical drive voting logical drive or Oracle ASM logical drive

NOTE This operation must be performed on the hdisk device name rather than on the named devnode created in step 3

This operation allows the nodes to share the logical drives Otherwise the first system that accesses the logical drive reserves it for exclusive use Oracle coordinates access to the shared logical drives

10 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

For an IBM storage subsystem reserve_policy=no_reserve is the correct value

a Use the lsattr -E -l hdisk | grep reserve command to check the attribute

b Use the chdev -l hdisk -a reserve_policy=no_reserve command to change the policy

c Repeat step a and step b on each cluster node

5 Set ownership and permissions on devnodes to be used by Oracle

NOTE If you created new devnodes for the shared logical drives set the ownership and permissions on the new devnode that you created not on the original hdisk devnode

a Use the following commands to set the ownership for each OCR device file

chown rootoinstall devltdevnodegt

chmod 640 devltdevnodegt

b Use the following commands to set the ownership and permission on the voting disk and database device files

chown oracledba devltdevnodegt

chmod 660 devltdevnodegt

c Repeat step a and step b on each cluster node

NOTE Do not assign a physical volume identifier (PVID) to hdisks that will be used with Oracle ASM

Installing OracleHere are the basic steps for installing Oracle

bull Create directories for Oracle

bull Set up the Oracle userrsquos profile

bull Install Oracle Clusterware

bull Update Oracle CRS

bull Install and configure Oracle ASM

bull Update Oracle ASM

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 11copy Copyright 2008 IBM Corporation

Installing Oracle Clusterware

bull Install Oracle Database

bull Update the database home

bull Configure the listener names

bull Create and configure a database

Creating Directories for OracleAlthough you can create directory structures for Oracle files in many different ways for ease of support use a standard directory structure based on the Optimal Flexible Architecture (OFA) guidelines published by Oracle For more information about the OFA Guidelines see Appendix D of the Oracle Database Installation Guide 10gR2 for AIX 5L Based Systems (B19075-02) The reference configuration is based on the OFA guidelines

Follow these two important guidelines when creating directories for Oracle

bull You must separate the Oracle CRS home directory from the Oracle base and the database home directories The Oracle CRS installation procedure sets the ownership for the entire Oracle CRS directory tree to root while the database directory tree must be owned by oracle

bull If more than one database will be serviced by Oracle ASM it is recommended that you create a separate home directory for Oracle ASM An Oracle ASM home directory provides the capability to modify the Oracle release for Oracle ASM while not changing the release of any of the databases on the same server The directory tree must be owned by oracle

Setting Up the Oracle Users Profile1 Set up the Oracle users profile before installing the Oracle binaries See

Appendix A for an example profile

2 Add or modify entries for ORACLE_HOME ORACLE_BASE PATH LD_LIBRARY_PATH and similar files to reflect the directories where the Oracle binaries will be placed during the Oracle Clusterware installation

Installing Oracle ClusterwareAs you install the Oracle Clusterware remember the following important points

bull Do not make the CRS_HOME path a subdirectory of the Oracle RAC 10g Release 2 Oracle base directory due to permission changes required for Oracle CRS For more details refer to Appendix E of the Oracle Clusterware Installation Guide

bull The rootsh script must be run as the root user and must be run on only one node at a time not concurrently

12 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Updating Oracle CRS

bull After rootsh runs on the last node it will attempt to run the vipca utility to configure the VIP network interfaces If you have IP addresses that are not routable across the Internet for example 100 192168 the vipca configuration will fail If the automatic vipca configuration fails run the vipca utility manually using the $ORA_CRS_HOMEbinvipca command from the command line as root This requires a graphical display

bull You can do the manual configuration with vipca before clicking OK on the Oracle Universal Installer the Oracle CRS installation page The Oracle CRS installation page waits for you to run rootsh Do the manual configuration with vipca during this time This action allows the remainder of the post-installation checks to complete successfully

Updating Oracle CRSRefer to the Metalink Bulletin 2820361 (see important note on page 5) for required updates The reference configuration used the 10203 update to the Oracle CRS home Download the 10203 update and any other updates required from the Oracle Metalink web site (httpmetalinkoraclecom) and install according to the instructions on the web site

Installing and Configuring Oracle ASMBecause a single Oracle ASM instance can service many database instances Oracle recommends that you install Oracle ASM in its own home directory Follow the naming convention of the OFA guidelines for the directory path As you install and configure Oracle ASM remember the following important points

bull If the Oracle ASM instance was not correctly created by the Oracle Universal Installer Wizard during installation use the dbca utility to add the instance and create the disk group(s)

bull If you want additional disk groups create them using the dbca utility

Updating Oracle ASMUpdate the Oracle ASM instance with the 10203 patch set and any additional updates indicated in Metalink Bulletin 2820361 You can use the same update media for 10203 previously downloaded to update Oracle CRS (page 13) Carefully follow the instructions for updating this database instance

Installing an Oracle DatabaseUse the Oracle Universal Installer to install another database home for a database instance Select Install Database Software Only on the Create Database page

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 13copy Copyright 2008 IBM Corporation

Installing Oracle Clusterware

Updating the Database HomeInstall the 10203 update on the new database home Select the database home instead of Oracle ASM on the Specify Home Details page

Configuring Listener NamesIf your RAC environment consists of only Oracle CRS an Oracle ASM instance and a database instance on each node it is easier to support only one listenerora configuration file for the Oracle ASM and database instances The best way to accomplish this is to create an environment variable TNS_ADMIN and assign it to the $ORACLE_HOMEnetworkadmin directory of the database instance For example

export TNS_ADMIN=$ORA_ASM_HOMEnetworkadmin

This environment variable allows the DBA to maintain one set of tnsnamesora sqlnetora and listenerora files for the entire environment

Creating and Configuring a DatabaseUse the Database Configuration Assistant (DBCA) to create a database For more information refer to the Oracle documentation about using DBCA As you create the database remember the following important points

bull Using DBCA gives you the capability to create the Oracle ASM disk groups if you choose to use Oracle ASM

bull The Flash Recovery Area (FRA) option is highly recommended A FRA allows the setup of a separate area on disk for flashback capability area for the archive logs the second copy of the multiplexed redo log files and for RMAN backups Make sure that the FRA is correctly set up to handle the performance of the database and that the FRA contains enough space

Space to hold all of the data for the data retention periods that are set by the database

Space for the type of backups that will be used

Space for the number of backups that will be held

For further documentation about using the FRA refer to the Oracle documentation as well as the following white paper

Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to

wwwibmcomsupporttechdocs

14 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Creating and Configuring a Database

ConclusionAs you have followed the example of setting up a two-node cluster described in this document you have seen how simple and how effective a deployment of Oracle RAC 10g Release 2 can be When you are doing your own installation be sure to use the applicable installation information as the basis for your work This document describes only tips and techniques Your installer should have the expertise to know how and when to modify instructions as necessary If your environment is complex and you require outside help contact your IBM business partner or sales representative See ldquoContact Informationrdquo below

Contact InformationFor more information and sales office locations visit the IBM web site at

httpwwwibmcom

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 15copy Copyright 2008 IBM Corporation

Contact Information

16 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix A

References

AIXFor AIX operating system downloads go to Quick Links for AIX Fixes

httpwww-933ibmcomeserversupportfixesfixcentralmainpseriesaix

For AIX documentation go to AIX Information

httppublibboulderibmcominfocentersystemsindexjsptopic=comibmaixdocdocbaseaixparenthtm

IBM Corporation 2006 Get the Latest Version of OpenSSH for AIX

httpwwwibmcomdeveloperworkseserverarticlesopenssh_updatedhtml

IBM StorageFor information about the IBM DS4800 Storage Subsystem go to

httpwww-03ibmcomsystemsstoragediskds4000ds4800indexhtml

For information about IBM storage go to

httpwww-03ibmcomsystemsstoragediskcm_re=masthead-_-products-_-stg-disk

OracleFor Oracle documentation go to

httpwwworaclecomtechnologydocumentationindexhtml

IBM Corporation 2008 Tuning IBM AIX 5L for an Oracle Database

httpwwwibmcompartnerworldwpsservletContentHandlerVPAA-6V2C4E

IBM Corporation 2008 Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems

wwwibmcomsupporttechdocs

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem A-1copy Copyright 2008 IBM Corporation

A-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix B

Troubleshooting

CSS Node Evictions can affect your functionality

IO time-outs can cause inappropriate node evictions during multi-path failover or other problems Two parameters might affect this css_misscount and disktimeout With versions of Oracle prior to 102 this was primarily accommodated by increasing the value of the css_misscount parameter However the function of this parameter has changed as of Oracle 102 and changes to this parameter might not achieve the desired result The new disktimeout parameter should be used in most cases to make this adjustment In many cases the default values will work fine However you might find it necessary to increase the default value of this parameter if you are getting a lot of CSS node evictions

For information about how the parameters work and how to adjust them refer to the following two articles Go to httpmetalinkoraclecom

bull Metalink DocID 2944301 CSS Timeout Computation in RAC 10g

bull Metalink DocID 2847521 Steps to Increase CSS Misscount Reboottime and Disktimeout

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem B-1copy Copyright 2008 IBM Corporation

B-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Trademarks and special notices copy Copyright IBM Corporation 2008 All rights reserved

References in this document to IBM products or services do not imply that IBM intends to make them available in every country

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corporation in the United States other countries or both If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (reg or trade) these symbols indicate US registered or common law trademarks owned by IBM at the time this information was published Such trademarks may also be registered or common law trademarks in other countries A current list of IBM trademarks is available on the Web at Copyright and trademark information at wwwibmcomlegalcopytradeshtml

Intel Intel Inside (logos) MMX and Pentium are trademarks of Intel Corporation in the United States other countries or both

UNIX is a registered trademark of The Open Group in the United States and other countries

Linux is a trademark of Linus Torvalds in the United States other countries or both

Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both

Other company product or service names may be trademarks or service marks of others

Information is provided AS IS without warranty of any kind

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved Actual environmental costs and performance characteristics may vary by customer

Information concerning non-IBM products was obtained from a supplier of these products published announcement material or other publicly available sources and does not constitute an endorsement of such products by IBM Sources for non-IBM list prices and performance numbers are taken from publicly available information including vendor announcements and vendor worldwide homepages IBM has not tested these products and cannot confirm the accuracy of performance capability or any other claims related to non-IBM products Questions on the capability of non-IBM products should be addressed to the supplier of those products

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk

  • Title Page13
  • Contents
  • Deploying Oracle RAC 10g Release 2 with ASM on AIX 53 with an IBM DS4700 or DS4800 Storage Subsystem
    • Introduction
      • Required Documents
      • Benefits
      • Reference Configuration
        • Configuring the Hosts
        • Configuring the IBM Storage
        • Installing Oracle
          • Creating Directories for Oracle
          • Setting Up the Oracle Users Profile
            • Installing Oracle Clusterware
              • Updating Oracle CRS
              • Installing and Configuring Oracle ASM
              • Updating Oracle ASM
              • Installing an Oracle Database
              • Updating the Database Home
              • Configuring Listener Names
              • Creating and Configuring a Database
                • Conclusion
                • Contact Information
                  • Appendix A References
                    • AIX
                    • IBM Storage
                    • Oracle
                      • Appendix B Troubleshooting
                      • Copyrights13

Contents

Introduction 1

Required Documents 1

Benefits 2

Reference Configuration 3

Configuring the Hosts 5

Configuring the IBM Storage 9

Installing Oracle 11

Creating Directories for Oracle12

Setting Up the Oracle Users Profile 12

Installing Oracle Clusterware 12

Updating Oracle CRS13

Installing and Configuring Oracle ASM13

Updating Oracle ASM 13

Installing an Oracle Database13

Updating the Database Home 14

Configuring Listener Names 14

Creating and Configuring a Database14

Conclusion 15

Contact Information15

Appendix A References A-1

AIX A-1

IBM Storage A-1

Oracle A-1

Appendix B Troubleshooting B-1

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem iiicopy Copyright 2008 IBM Corporation

iv Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Deploying Oracle RAC 10g Release 2 with ASM on AIX 53 with an IBM DS4700 or DS4800 Storage Subsystem

IntroductionOracle Real Application Clusters (Oracle RAC) is a high-availability option to Oracle Database Oracle RAC supports the deployment of a single database across a cluster of servers providing excellent fault tolerance performance and scalability The industry is taking note of Oracle RACs growing importance in a wide range of customer implementations across all industries for transaction processing and data warehousing applications

This document presents an overview and some tips and techniques in deploying the Oracle RAC 10g Release 2 database with Oracle RAC on the IBMreg AIXreg operating system using IBM storage The database storage is administered using the Oracle Automatic Storage Management (Oracle ASM) feature

The procedure of creating a simple configuration using a two-node cluster is used as an example to illustrate the principles of performing this type of implementation Actual implementations vary in complexity and require significant expertise in all of the technologies involved

Required DocumentsThis document is intended to illustrate any deviations from the Oracle installation guide and any enhancements in implementation This document does not provide a comprehensive reference for installing all of these components For full implementation instructions please refer to the product documentation for the hardware being used for the AIX operating system for the Oracle Database and for Oracle Clusterware

For IBM storage documentation contact your IBM business partner or service representative

For AIX documentation go to AIX Information

httppublibboulderibmcominfocentersystemsindexjsptopic=comibmaixdocdocbaseaixparenthtm

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 1copy Copyright 2008 IBM Corporation

Introduction

For Oracle documentation go to

httpwwworaclecomtechnologydocumentationindexhtml

Before installation you must know how to best configure the storage for enhanced performance For this information refer to Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to

wwwibmcomsupporttechdocs

BenefitsThe reference configuration provides insight into how to configure these products to work together to achieve numerous benefits The reference configuration uses Oracle ASM to manage the Oracle RAC 10g Release 2 database storage running under Oracle Clusterware on AIX with an IBM DS48001 storage subsystem The IBM DS4800 storage subsystem is a robust high-performance disk array that provides a scalable flexible and expandable storage foundation upon which to build a highly reliable database system using Oracle RAC

Oracle RAC offers these benefits

bull High availability ndash Provides enhanced uptime for database applications

bull Scalability ndash Expands capacity by adding servers to the cluster

bull Lower computing costs ndash Uses lower-cost hardware and reduces expenses associated with application downtime

bull Excellent performance ndash Spreads the database load across multiple clustered servers allowing performance to scale with the size of the cluster

The IBM DS4800 storage subsystem offers these benefits which go hand-in-hand with those of Oracle RAC

bull High availability ndash The IBM DS4800 storage subsystemsrsquo fully redundant hardware and redundant array of independent disks (RAID) functionality ensures that the storage is as available as the RAC database

bull Scalability ndash Additional storage capacity can be added to the storage subsystem while it continues operation

bull Lower costs ndash Shared storage within a storage area network (SAN) allows reduced cost compared to storage local to individual hosts and reduces management expenses associated with administering the storage

bull Excellent performance ndash The IBM DS4800 storage subsystem is one of the best performing disk arrays in its class Its full front-to-back 4 gigabit Fibre Channel interface architecture provides bandwidth for the most demanding applications

1Solution also works with a DS4700 storage subsystem

2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

The DS4000 Storage Manager software package configures and manages the IBMDS4800 storage subsystemrsquos fully redundant hardware and high-performance architecture with ease DS4000 Storage Manager allows in-band management from servers to attached storage or remotely through the Ethernet The storage subsystem offers several premium features that add functionality when used in a database environment Two of the most helpful features in an Oracle RAC environment are the Storage Partitioning premium feature and the Enhanced Remote Mirroring premium feature

The Storage Partitioning premium feature allows logical drives on the storage subsystem to be distributed to different hosts in your SAN associating each logical drive with one particular host or a group of hosts The logical drives are presented to each host according to the host type for the different operating systems that are running on those hosts The Storage Partitioning premium feature allows great flexibility in the use of the storage subsystem for storage consolidation

The Enhanced Remote Mirroring premium feature (RM) provides the ability to mirror logical drives between two IBM storage subsystems Mirroring provides an additional redundant copy of the data The mirror copy can be used for rapid disaster recovery and can be also used to create a copy of live data for test purposes

Although these two premium features are not used in the reference configuration there are many opportunities to use storage partitioning and Enhanced Remote Mirroring in a real-world Oracle implementation

For more information please discuss these premium features with your IBM business partner or sales representative

Reference ConfigurationThe reference configuration consists of the following equipment

bull 2 IBM P5 p630 servers

bull AIX 53 Technology Level 6

bull IBM DS4800 storage subsystem

bull Oracle Cluster Ready Services (Oracle CRS) 10201 upgraded to 10203

bull Oracle Database 10201 upgraded to 10203

The systems are configured as a two-node RAC cluster Oracle ASM is employed to simplify storage management The AIX RDAC multipathing driver provided failover support

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 3copy Copyright 2008 IBM Corporation

Introduction

Figure 1 Reference Configuration

4 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

Configuring the HostsThe hosts must be configured to function correctly with Oracle RAC The following steps describe in general terms how to prepare the systems for the Oracle installation

IMPORTANT Refer to Metalink Bulletin 2820361 Minimum Software Versions and Patches Required to Support Oracle Products on IBM Series for the latest requirements This bulletin is available to Metalink users athttpmetalinkoraclecom Metalink Bulletin 2820361 is regularly updated and the information it contains supersedes any information found in this document

1 Install the AIX operating system if it is not already installed The reference configuration uses AIX 53 Technology Level 6

2 Install additional software packages required for Oracle

bosadtbase

bosadtlib

bosadtlibm

bosperflibperfstat

bosperfperfstat

bosperfproctools

rsctbasicrte

rsctcompatclientsrte

xlCaix50rte7004 (See the following note)

xlCrte7001 (See the following note)

NOTE The versions of xlCaix50rte and xlCrte supplied on the AIX 53 distribution media are not compatible with Oracle RAC 10g Release 2 You must download newer versions from IBMs web site The reference configuration uses fileset xlcrteaix50apr2007ptftarZ

3 Download and install the updates to the AIX operating system from IBMs website

httpwww14softwareibmcomwebappset2sasfgenunix3aixfixeshtml

The reference configuration uses AIX 53 Technology Level 6

4 Configure the hosts network interfaces

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 5copy Copyright 2008 IBM Corporation

Configuring the Hosts

Use at least two ethernet interfaces

One for the Oracle public network

One for the private network used by Oracle CRS

Oracle recommends that the private interface be on a separate network isolated from all other traffic You should use a separate gigabit ethernet switch for the private network

Additional interfaces provide more flexibility in configuration and higher performance In the reference configuration the VIP network shares the interface used for the public network This is configured later in the installation process using the vipca utility Entries for the internet protocol (IP) address as well as the public and private networks are inserted into the etchosts file at this time

5 Create users and groups

a Create two groups dba and oinstall

b Add a user Oracle with the oinstall primary group and with the dba group

NOTE Make sure that the user ID for the Oracle user and the group numbers for the dba and oinstall groups are consistent on all nodes

6 Set the shell soft limits for the Oracle and root users using the smit chuser command

Soft FILE size = -1

Soft CPU time = -1

Soft DATA segment = -1

Soft STACK size = -1

7 Set system configuration parameters

a Use the smit chgsys command to set the maximum number of processes per user to at least 2048

b Use the vmo -p -o lru_file_repage=0 command to set the lru_file_repage parameter to 0

NOTE This setting instructs the AIX Virtual Memory Manager to take pages only from the file buffer cache allowing Oraclersquos System Global Area (SGA) to remain in memory for better performance

8 Use the usrsbinno -a command to configure the network tuning parameters

Use these minimum values for network tuning parameters

6 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

ipqmaxlen = 512rfc1323 = 1sb_max = 1310720 (2655360)tcp_recvspace = 65536 tcp_sendspace = 65536 udp_recvspace = 655360 (10 udp_sendspace must be less than sb_max)udp_sendspace = 65536

NOTE If you modify these parameters the method depends on whether the system is in compatibility mode For specific instructions refer to the AIX documentation for the no command

NOTE If you change the value of the ipqmaxlen parameter you must restart the system for the change to take effect

9 Download and install the Open Secure Shell (OpenSSH) utility from the IBM web site

httpwwwibmcomdeveloperworkseserverarticlesopenssh_updatedhtml

NOTE The reference configuration uses OpenSSH The SSH utility is not required for Oracle to function however SSH is recommended because it provides a more secure method of communication between nodes than does RSH If SSH is available Oracle will use it otherwise Oracle uses RSH Configuration of RSH user equivalence is outside the scope of this document

10 Configure SSH for root login permission

a Locate the sshd_config file (typically etc or usrlocaletc)

b Edit the sshd_config file changing PermitRootLogin to yes

c Use the startsrc -s sshd command to start sshd

11 Configure SSH for user equivalence for root and Oracle on all nodes

a Log on as ltrootgt

b Use the following commands to generate the SSH key

cd ~ssh

ssh-keygen -t dsa

cat id_dsapub gtgt authorized_keys

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 7copy Copyright 2008 IBM Corporation

Configuring the Hosts

c Copy the authorized_keys file to ~ssh on the next system

d Repeat step b and step c on each system

e After all systems have a public key in the file copy the file back to all of the previous systems so that they all have an identical file

f Test the SSH function from each system to itself and to each of the other systems by issuing such commands as the ssh host1 date command Make sure that the commands can be executed without using a password

g Log on as the Oracle user and repeat step a through step f so that Oracle can also perform remote commands without passwords

NOTE The permissions must restrict write access on both the base user directory and the ssh directory If the permissions are not correctly set OpenSSH requires a password even if the appropriate public key has been added to the authorized_keys file

12 Edit the profile file in the Oracle userrsquos home directory to set the Oracle users parameters

Table 1 shows the entries that you must add to the profile file

Table 1 Oracle Userrsquos Parameters

Parameter to Add Explanation

umask 022

AIXTHREAD_SCOPE=S

An environment variable that directs the system to place user threads within a system-wide rather than a process-wide contention scope This placement provides better performance and more efficient memory use

CRS_HOME A unique home directory for Oracle CRS binaries

ASM_HOME

Use this parameter if you prefer to install separate Oracle binaries for Oracle ASM This method is preferable if there will be more than one database instance supported by Oracle ASM

ORACLE_HOME A unique home for the database binaries

LD_LIBRARY_PATH The complete pathway is $ORACLE_HOMElib

ORACLE_BASE Identifies the Oracle base directory

ORACLE_SID Identifies the instance

PATH Include $ORACLE_HOMEbin

CLASSPATHInclude $ORACLE_HOMEjdbclibltclasses filesgt

8 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

13 Edit the profile file in the root userrsquos home directory to set the same parameters as set in the Oracle users profile

Be sure to add the following entry to the profile file

AIXTHREAD_SCOPE=S

14 Synchronize the system time between the hosts in the cluster using any effective method The reference configuration used Network Time Protocol (NTP) If an NTP server is available use this method to set up xntpd

a Edit etcntpconf with the nameIP of the NTP server(s)

b Add a line server lthostnamegt for each host

c Start the service using the smit xntpd command

15 Because some Oracle utilities expect lsattr in etc and do not function correctly without a link to it use the ln -s usrsbinlsattr etclsattr command to add a soft link

Configuring the IBM Storage

IMPORTANT You must know how to best configure the storage for enhanced performance For this information refer to Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to wwwibmcomsupporttechdocs

Use DS4000 Storage Manager and the AIX system utilities to configure the IBM storage subsystem

1 Use DS4000 Storage Manager to create logical drives on the storage subsystem to be shared for Oracle CRS communication and database storage

NOTE You also may use additional logical drives for Oracle binaries and similar purposes however for those logical drives on the storage subsystem that are used exclusively by one particular host you will need the Storage Partitioning premium feature

You will need to analyze your implementation to determine the appropriate logical drive sizes desired RAID levels and other parameters The reference configuration uses the following minimal configuration of shared logical drives

2 Oracle Cluster Registry (OCR) logical drives (300 MB RAID1)

3 voting disks (300MB RAID1)

1 logical drive for Oracle ASMs SPFILE (100MB RAID1)

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 9copy Copyright 2008 IBM Corporation

Configuring the IBM Storage

2 logical drives for Oracle ASM data storage (300 GB RAID5)

2 logical drives for Oracle ASM log storage (135GB RAID1)

The number and the size of the logical drives created for Oracle ASM storage varies based on the individual implementation but the sizes used for the OCR disks voting disks and the Oracle ASM SPFILE disks in the reference configuration are sufficient for any configuration

In the reference configuration the Oracle binaries are installed on internal disks within each individual server The reference configuration uses the JFS2 file system and the logical drive was mounted as u01

2 Configure the shared disks in the operating system

After you have created the logical drives on the storage subsystem the cfgmgr command rescans the AIX system for changes to the physical devices

The lspv command shows the physical devices that AIX has configured

The fget_config -Av command shows the LUN numbers the disk drive name and the associated logical drive names configured on the storage subsystem

3 If the hdisk names for each logical drive on the storage subsystem that will be shared in the cluster are not identical on all cluster nodes create new device nodes for each of the logical drives to give them a consistent name across all nodes

For example if the logical drive for the first OCR disk was hdisk2 on node1 and hdisk3 on node2 a new device node (devnode) called devORA_OCR1 could be created on each host using the major numbers and the minor numbers that correspond with the same logical drive on each system Show the major and the minor numbers with the command ls -l Use the command mknod devORA_OCR1 c ltmajor of OCRgt ltminor of OCRgt to create the new devnode

Even when the disk drive names correlate on all of the nodes you should still create these named device nodes for ease of location and administration Named device nodes were created on the reference configuration for all shared logical drives using the convention devORA_ltusegt

4 Set the reserve_policy attribute for each logical drive to be used for shared access for OCR logical drive voting logical drive or Oracle ASM logical drive

NOTE This operation must be performed on the hdisk device name rather than on the named devnode created in step 3

This operation allows the nodes to share the logical drives Otherwise the first system that accesses the logical drive reserves it for exclusive use Oracle coordinates access to the shared logical drives

10 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

For an IBM storage subsystem reserve_policy=no_reserve is the correct value

a Use the lsattr -E -l hdisk | grep reserve command to check the attribute

b Use the chdev -l hdisk -a reserve_policy=no_reserve command to change the policy

c Repeat step a and step b on each cluster node

5 Set ownership and permissions on devnodes to be used by Oracle

NOTE If you created new devnodes for the shared logical drives set the ownership and permissions on the new devnode that you created not on the original hdisk devnode

a Use the following commands to set the ownership for each OCR device file

chown rootoinstall devltdevnodegt

chmod 640 devltdevnodegt

b Use the following commands to set the ownership and permission on the voting disk and database device files

chown oracledba devltdevnodegt

chmod 660 devltdevnodegt

c Repeat step a and step b on each cluster node

NOTE Do not assign a physical volume identifier (PVID) to hdisks that will be used with Oracle ASM

Installing OracleHere are the basic steps for installing Oracle

bull Create directories for Oracle

bull Set up the Oracle userrsquos profile

bull Install Oracle Clusterware

bull Update Oracle CRS

bull Install and configure Oracle ASM

bull Update Oracle ASM

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 11copy Copyright 2008 IBM Corporation

Installing Oracle Clusterware

bull Install Oracle Database

bull Update the database home

bull Configure the listener names

bull Create and configure a database

Creating Directories for OracleAlthough you can create directory structures for Oracle files in many different ways for ease of support use a standard directory structure based on the Optimal Flexible Architecture (OFA) guidelines published by Oracle For more information about the OFA Guidelines see Appendix D of the Oracle Database Installation Guide 10gR2 for AIX 5L Based Systems (B19075-02) The reference configuration is based on the OFA guidelines

Follow these two important guidelines when creating directories for Oracle

bull You must separate the Oracle CRS home directory from the Oracle base and the database home directories The Oracle CRS installation procedure sets the ownership for the entire Oracle CRS directory tree to root while the database directory tree must be owned by oracle

bull If more than one database will be serviced by Oracle ASM it is recommended that you create a separate home directory for Oracle ASM An Oracle ASM home directory provides the capability to modify the Oracle release for Oracle ASM while not changing the release of any of the databases on the same server The directory tree must be owned by oracle

Setting Up the Oracle Users Profile1 Set up the Oracle users profile before installing the Oracle binaries See

Appendix A for an example profile

2 Add or modify entries for ORACLE_HOME ORACLE_BASE PATH LD_LIBRARY_PATH and similar files to reflect the directories where the Oracle binaries will be placed during the Oracle Clusterware installation

Installing Oracle ClusterwareAs you install the Oracle Clusterware remember the following important points

bull Do not make the CRS_HOME path a subdirectory of the Oracle RAC 10g Release 2 Oracle base directory due to permission changes required for Oracle CRS For more details refer to Appendix E of the Oracle Clusterware Installation Guide

bull The rootsh script must be run as the root user and must be run on only one node at a time not concurrently

12 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Updating Oracle CRS

bull After rootsh runs on the last node it will attempt to run the vipca utility to configure the VIP network interfaces If you have IP addresses that are not routable across the Internet for example 100 192168 the vipca configuration will fail If the automatic vipca configuration fails run the vipca utility manually using the $ORA_CRS_HOMEbinvipca command from the command line as root This requires a graphical display

bull You can do the manual configuration with vipca before clicking OK on the Oracle Universal Installer the Oracle CRS installation page The Oracle CRS installation page waits for you to run rootsh Do the manual configuration with vipca during this time This action allows the remainder of the post-installation checks to complete successfully

Updating Oracle CRSRefer to the Metalink Bulletin 2820361 (see important note on page 5) for required updates The reference configuration used the 10203 update to the Oracle CRS home Download the 10203 update and any other updates required from the Oracle Metalink web site (httpmetalinkoraclecom) and install according to the instructions on the web site

Installing and Configuring Oracle ASMBecause a single Oracle ASM instance can service many database instances Oracle recommends that you install Oracle ASM in its own home directory Follow the naming convention of the OFA guidelines for the directory path As you install and configure Oracle ASM remember the following important points

bull If the Oracle ASM instance was not correctly created by the Oracle Universal Installer Wizard during installation use the dbca utility to add the instance and create the disk group(s)

bull If you want additional disk groups create them using the dbca utility

Updating Oracle ASMUpdate the Oracle ASM instance with the 10203 patch set and any additional updates indicated in Metalink Bulletin 2820361 You can use the same update media for 10203 previously downloaded to update Oracle CRS (page 13) Carefully follow the instructions for updating this database instance

Installing an Oracle DatabaseUse the Oracle Universal Installer to install another database home for a database instance Select Install Database Software Only on the Create Database page

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 13copy Copyright 2008 IBM Corporation

Installing Oracle Clusterware

Updating the Database HomeInstall the 10203 update on the new database home Select the database home instead of Oracle ASM on the Specify Home Details page

Configuring Listener NamesIf your RAC environment consists of only Oracle CRS an Oracle ASM instance and a database instance on each node it is easier to support only one listenerora configuration file for the Oracle ASM and database instances The best way to accomplish this is to create an environment variable TNS_ADMIN and assign it to the $ORACLE_HOMEnetworkadmin directory of the database instance For example

export TNS_ADMIN=$ORA_ASM_HOMEnetworkadmin

This environment variable allows the DBA to maintain one set of tnsnamesora sqlnetora and listenerora files for the entire environment

Creating and Configuring a DatabaseUse the Database Configuration Assistant (DBCA) to create a database For more information refer to the Oracle documentation about using DBCA As you create the database remember the following important points

bull Using DBCA gives you the capability to create the Oracle ASM disk groups if you choose to use Oracle ASM

bull The Flash Recovery Area (FRA) option is highly recommended A FRA allows the setup of a separate area on disk for flashback capability area for the archive logs the second copy of the multiplexed redo log files and for RMAN backups Make sure that the FRA is correctly set up to handle the performance of the database and that the FRA contains enough space

Space to hold all of the data for the data retention periods that are set by the database

Space for the type of backups that will be used

Space for the number of backups that will be held

For further documentation about using the FRA refer to the Oracle documentation as well as the following white paper

Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to

wwwibmcomsupporttechdocs

14 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Creating and Configuring a Database

ConclusionAs you have followed the example of setting up a two-node cluster described in this document you have seen how simple and how effective a deployment of Oracle RAC 10g Release 2 can be When you are doing your own installation be sure to use the applicable installation information as the basis for your work This document describes only tips and techniques Your installer should have the expertise to know how and when to modify instructions as necessary If your environment is complex and you require outside help contact your IBM business partner or sales representative See ldquoContact Informationrdquo below

Contact InformationFor more information and sales office locations visit the IBM web site at

httpwwwibmcom

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 15copy Copyright 2008 IBM Corporation

Contact Information

16 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix A

References

AIXFor AIX operating system downloads go to Quick Links for AIX Fixes

httpwww-933ibmcomeserversupportfixesfixcentralmainpseriesaix

For AIX documentation go to AIX Information

httppublibboulderibmcominfocentersystemsindexjsptopic=comibmaixdocdocbaseaixparenthtm

IBM Corporation 2006 Get the Latest Version of OpenSSH for AIX

httpwwwibmcomdeveloperworkseserverarticlesopenssh_updatedhtml

IBM StorageFor information about the IBM DS4800 Storage Subsystem go to

httpwww-03ibmcomsystemsstoragediskds4000ds4800indexhtml

For information about IBM storage go to

httpwww-03ibmcomsystemsstoragediskcm_re=masthead-_-products-_-stg-disk

OracleFor Oracle documentation go to

httpwwworaclecomtechnologydocumentationindexhtml

IBM Corporation 2008 Tuning IBM AIX 5L for an Oracle Database

httpwwwibmcompartnerworldwpsservletContentHandlerVPAA-6V2C4E

IBM Corporation 2008 Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems

wwwibmcomsupporttechdocs

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem A-1copy Copyright 2008 IBM Corporation

A-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix B

Troubleshooting

CSS Node Evictions can affect your functionality

IO time-outs can cause inappropriate node evictions during multi-path failover or other problems Two parameters might affect this css_misscount and disktimeout With versions of Oracle prior to 102 this was primarily accommodated by increasing the value of the css_misscount parameter However the function of this parameter has changed as of Oracle 102 and changes to this parameter might not achieve the desired result The new disktimeout parameter should be used in most cases to make this adjustment In many cases the default values will work fine However you might find it necessary to increase the default value of this parameter if you are getting a lot of CSS node evictions

For information about how the parameters work and how to adjust them refer to the following two articles Go to httpmetalinkoraclecom

bull Metalink DocID 2944301 CSS Timeout Computation in RAC 10g

bull Metalink DocID 2847521 Steps to Increase CSS Misscount Reboottime and Disktimeout

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem B-1copy Copyright 2008 IBM Corporation

B-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Trademarks and special notices copy Copyright IBM Corporation 2008 All rights reserved

References in this document to IBM products or services do not imply that IBM intends to make them available in every country

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corporation in the United States other countries or both If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (reg or trade) these symbols indicate US registered or common law trademarks owned by IBM at the time this information was published Such trademarks may also be registered or common law trademarks in other countries A current list of IBM trademarks is available on the Web at Copyright and trademark information at wwwibmcomlegalcopytradeshtml

Intel Intel Inside (logos) MMX and Pentium are trademarks of Intel Corporation in the United States other countries or both

UNIX is a registered trademark of The Open Group in the United States and other countries

Linux is a trademark of Linus Torvalds in the United States other countries or both

Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both

Other company product or service names may be trademarks or service marks of others

Information is provided AS IS without warranty of any kind

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved Actual environmental costs and performance characteristics may vary by customer

Information concerning non-IBM products was obtained from a supplier of these products published announcement material or other publicly available sources and does not constitute an endorsement of such products by IBM Sources for non-IBM list prices and performance numbers are taken from publicly available information including vendor announcements and vendor worldwide homepages IBM has not tested these products and cannot confirm the accuracy of performance capability or any other claims related to non-IBM products Questions on the capability of non-IBM products should be addressed to the supplier of those products

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk

  • Title Page13
  • Contents
  • Deploying Oracle RAC 10g Release 2 with ASM on AIX 53 with an IBM DS4700 or DS4800 Storage Subsystem
    • Introduction
      • Required Documents
      • Benefits
      • Reference Configuration
        • Configuring the Hosts
        • Configuring the IBM Storage
        • Installing Oracle
          • Creating Directories for Oracle
          • Setting Up the Oracle Users Profile
            • Installing Oracle Clusterware
              • Updating Oracle CRS
              • Installing and Configuring Oracle ASM
              • Updating Oracle ASM
              • Installing an Oracle Database
              • Updating the Database Home
              • Configuring Listener Names
              • Creating and Configuring a Database
                • Conclusion
                • Contact Information
                  • Appendix A References
                    • AIX
                    • IBM Storage
                    • Oracle
                      • Appendix B Troubleshooting
                      • Copyrights13

iv Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Deploying Oracle RAC 10g Release 2 with ASM on AIX 53 with an IBM DS4700 or DS4800 Storage Subsystem

IntroductionOracle Real Application Clusters (Oracle RAC) is a high-availability option to Oracle Database Oracle RAC supports the deployment of a single database across a cluster of servers providing excellent fault tolerance performance and scalability The industry is taking note of Oracle RACs growing importance in a wide range of customer implementations across all industries for transaction processing and data warehousing applications

This document presents an overview and some tips and techniques in deploying the Oracle RAC 10g Release 2 database with Oracle RAC on the IBMreg AIXreg operating system using IBM storage The database storage is administered using the Oracle Automatic Storage Management (Oracle ASM) feature

The procedure of creating a simple configuration using a two-node cluster is used as an example to illustrate the principles of performing this type of implementation Actual implementations vary in complexity and require significant expertise in all of the technologies involved

Required DocumentsThis document is intended to illustrate any deviations from the Oracle installation guide and any enhancements in implementation This document does not provide a comprehensive reference for installing all of these components For full implementation instructions please refer to the product documentation for the hardware being used for the AIX operating system for the Oracle Database and for Oracle Clusterware

For IBM storage documentation contact your IBM business partner or service representative

For AIX documentation go to AIX Information

httppublibboulderibmcominfocentersystemsindexjsptopic=comibmaixdocdocbaseaixparenthtm

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 1copy Copyright 2008 IBM Corporation

Introduction

For Oracle documentation go to

httpwwworaclecomtechnologydocumentationindexhtml

Before installation you must know how to best configure the storage for enhanced performance For this information refer to Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to

wwwibmcomsupporttechdocs

BenefitsThe reference configuration provides insight into how to configure these products to work together to achieve numerous benefits The reference configuration uses Oracle ASM to manage the Oracle RAC 10g Release 2 database storage running under Oracle Clusterware on AIX with an IBM DS48001 storage subsystem The IBM DS4800 storage subsystem is a robust high-performance disk array that provides a scalable flexible and expandable storage foundation upon which to build a highly reliable database system using Oracle RAC

Oracle RAC offers these benefits

bull High availability ndash Provides enhanced uptime for database applications

bull Scalability ndash Expands capacity by adding servers to the cluster

bull Lower computing costs ndash Uses lower-cost hardware and reduces expenses associated with application downtime

bull Excellent performance ndash Spreads the database load across multiple clustered servers allowing performance to scale with the size of the cluster

The IBM DS4800 storage subsystem offers these benefits which go hand-in-hand with those of Oracle RAC

bull High availability ndash The IBM DS4800 storage subsystemsrsquo fully redundant hardware and redundant array of independent disks (RAID) functionality ensures that the storage is as available as the RAC database

bull Scalability ndash Additional storage capacity can be added to the storage subsystem while it continues operation

bull Lower costs ndash Shared storage within a storage area network (SAN) allows reduced cost compared to storage local to individual hosts and reduces management expenses associated with administering the storage

bull Excellent performance ndash The IBM DS4800 storage subsystem is one of the best performing disk arrays in its class Its full front-to-back 4 gigabit Fibre Channel interface architecture provides bandwidth for the most demanding applications

1Solution also works with a DS4700 storage subsystem

2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

The DS4000 Storage Manager software package configures and manages the IBMDS4800 storage subsystemrsquos fully redundant hardware and high-performance architecture with ease DS4000 Storage Manager allows in-band management from servers to attached storage or remotely through the Ethernet The storage subsystem offers several premium features that add functionality when used in a database environment Two of the most helpful features in an Oracle RAC environment are the Storage Partitioning premium feature and the Enhanced Remote Mirroring premium feature

The Storage Partitioning premium feature allows logical drives on the storage subsystem to be distributed to different hosts in your SAN associating each logical drive with one particular host or a group of hosts The logical drives are presented to each host according to the host type for the different operating systems that are running on those hosts The Storage Partitioning premium feature allows great flexibility in the use of the storage subsystem for storage consolidation

The Enhanced Remote Mirroring premium feature (RM) provides the ability to mirror logical drives between two IBM storage subsystems Mirroring provides an additional redundant copy of the data The mirror copy can be used for rapid disaster recovery and can be also used to create a copy of live data for test purposes

Although these two premium features are not used in the reference configuration there are many opportunities to use storage partitioning and Enhanced Remote Mirroring in a real-world Oracle implementation

For more information please discuss these premium features with your IBM business partner or sales representative

Reference ConfigurationThe reference configuration consists of the following equipment

bull 2 IBM P5 p630 servers

bull AIX 53 Technology Level 6

bull IBM DS4800 storage subsystem

bull Oracle Cluster Ready Services (Oracle CRS) 10201 upgraded to 10203

bull Oracle Database 10201 upgraded to 10203

The systems are configured as a two-node RAC cluster Oracle ASM is employed to simplify storage management The AIX RDAC multipathing driver provided failover support

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 3copy Copyright 2008 IBM Corporation

Introduction

Figure 1 Reference Configuration

4 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

Configuring the HostsThe hosts must be configured to function correctly with Oracle RAC The following steps describe in general terms how to prepare the systems for the Oracle installation

IMPORTANT Refer to Metalink Bulletin 2820361 Minimum Software Versions and Patches Required to Support Oracle Products on IBM Series for the latest requirements This bulletin is available to Metalink users athttpmetalinkoraclecom Metalink Bulletin 2820361 is regularly updated and the information it contains supersedes any information found in this document

1 Install the AIX operating system if it is not already installed The reference configuration uses AIX 53 Technology Level 6

2 Install additional software packages required for Oracle

bosadtbase

bosadtlib

bosadtlibm

bosperflibperfstat

bosperfperfstat

bosperfproctools

rsctbasicrte

rsctcompatclientsrte

xlCaix50rte7004 (See the following note)

xlCrte7001 (See the following note)

NOTE The versions of xlCaix50rte and xlCrte supplied on the AIX 53 distribution media are not compatible with Oracle RAC 10g Release 2 You must download newer versions from IBMs web site The reference configuration uses fileset xlcrteaix50apr2007ptftarZ

3 Download and install the updates to the AIX operating system from IBMs website

httpwww14softwareibmcomwebappset2sasfgenunix3aixfixeshtml

The reference configuration uses AIX 53 Technology Level 6

4 Configure the hosts network interfaces

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 5copy Copyright 2008 IBM Corporation

Configuring the Hosts

Use at least two ethernet interfaces

One for the Oracle public network

One for the private network used by Oracle CRS

Oracle recommends that the private interface be on a separate network isolated from all other traffic You should use a separate gigabit ethernet switch for the private network

Additional interfaces provide more flexibility in configuration and higher performance In the reference configuration the VIP network shares the interface used for the public network This is configured later in the installation process using the vipca utility Entries for the internet protocol (IP) address as well as the public and private networks are inserted into the etchosts file at this time

5 Create users and groups

a Create two groups dba and oinstall

b Add a user Oracle with the oinstall primary group and with the dba group

NOTE Make sure that the user ID for the Oracle user and the group numbers for the dba and oinstall groups are consistent on all nodes

6 Set the shell soft limits for the Oracle and root users using the smit chuser command

Soft FILE size = -1

Soft CPU time = -1

Soft DATA segment = -1

Soft STACK size = -1

7 Set system configuration parameters

a Use the smit chgsys command to set the maximum number of processes per user to at least 2048

b Use the vmo -p -o lru_file_repage=0 command to set the lru_file_repage parameter to 0

NOTE This setting instructs the AIX Virtual Memory Manager to take pages only from the file buffer cache allowing Oraclersquos System Global Area (SGA) to remain in memory for better performance

8 Use the usrsbinno -a command to configure the network tuning parameters

Use these minimum values for network tuning parameters

6 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

ipqmaxlen = 512rfc1323 = 1sb_max = 1310720 (2655360)tcp_recvspace = 65536 tcp_sendspace = 65536 udp_recvspace = 655360 (10 udp_sendspace must be less than sb_max)udp_sendspace = 65536

NOTE If you modify these parameters the method depends on whether the system is in compatibility mode For specific instructions refer to the AIX documentation for the no command

NOTE If you change the value of the ipqmaxlen parameter you must restart the system for the change to take effect

9 Download and install the Open Secure Shell (OpenSSH) utility from the IBM web site

httpwwwibmcomdeveloperworkseserverarticlesopenssh_updatedhtml

NOTE The reference configuration uses OpenSSH The SSH utility is not required for Oracle to function however SSH is recommended because it provides a more secure method of communication between nodes than does RSH If SSH is available Oracle will use it otherwise Oracle uses RSH Configuration of RSH user equivalence is outside the scope of this document

10 Configure SSH for root login permission

a Locate the sshd_config file (typically etc or usrlocaletc)

b Edit the sshd_config file changing PermitRootLogin to yes

c Use the startsrc -s sshd command to start sshd

11 Configure SSH for user equivalence for root and Oracle on all nodes

a Log on as ltrootgt

b Use the following commands to generate the SSH key

cd ~ssh

ssh-keygen -t dsa

cat id_dsapub gtgt authorized_keys

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 7copy Copyright 2008 IBM Corporation

Configuring the Hosts

c Copy the authorized_keys file to ~ssh on the next system

d Repeat step b and step c on each system

e After all systems have a public key in the file copy the file back to all of the previous systems so that they all have an identical file

f Test the SSH function from each system to itself and to each of the other systems by issuing such commands as the ssh host1 date command Make sure that the commands can be executed without using a password

g Log on as the Oracle user and repeat step a through step f so that Oracle can also perform remote commands without passwords

NOTE The permissions must restrict write access on both the base user directory and the ssh directory If the permissions are not correctly set OpenSSH requires a password even if the appropriate public key has been added to the authorized_keys file

12 Edit the profile file in the Oracle userrsquos home directory to set the Oracle users parameters

Table 1 shows the entries that you must add to the profile file

Table 1 Oracle Userrsquos Parameters

Parameter to Add Explanation

umask 022

AIXTHREAD_SCOPE=S

An environment variable that directs the system to place user threads within a system-wide rather than a process-wide contention scope This placement provides better performance and more efficient memory use

CRS_HOME A unique home directory for Oracle CRS binaries

ASM_HOME

Use this parameter if you prefer to install separate Oracle binaries for Oracle ASM This method is preferable if there will be more than one database instance supported by Oracle ASM

ORACLE_HOME A unique home for the database binaries

LD_LIBRARY_PATH The complete pathway is $ORACLE_HOMElib

ORACLE_BASE Identifies the Oracle base directory

ORACLE_SID Identifies the instance

PATH Include $ORACLE_HOMEbin

CLASSPATHInclude $ORACLE_HOMEjdbclibltclasses filesgt

8 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

13 Edit the profile file in the root userrsquos home directory to set the same parameters as set in the Oracle users profile

Be sure to add the following entry to the profile file

AIXTHREAD_SCOPE=S

14 Synchronize the system time between the hosts in the cluster using any effective method The reference configuration used Network Time Protocol (NTP) If an NTP server is available use this method to set up xntpd

a Edit etcntpconf with the nameIP of the NTP server(s)

b Add a line server lthostnamegt for each host

c Start the service using the smit xntpd command

15 Because some Oracle utilities expect lsattr in etc and do not function correctly without a link to it use the ln -s usrsbinlsattr etclsattr command to add a soft link

Configuring the IBM Storage

IMPORTANT You must know how to best configure the storage for enhanced performance For this information refer to Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to wwwibmcomsupporttechdocs

Use DS4000 Storage Manager and the AIX system utilities to configure the IBM storage subsystem

1 Use DS4000 Storage Manager to create logical drives on the storage subsystem to be shared for Oracle CRS communication and database storage

NOTE You also may use additional logical drives for Oracle binaries and similar purposes however for those logical drives on the storage subsystem that are used exclusively by one particular host you will need the Storage Partitioning premium feature

You will need to analyze your implementation to determine the appropriate logical drive sizes desired RAID levels and other parameters The reference configuration uses the following minimal configuration of shared logical drives

2 Oracle Cluster Registry (OCR) logical drives (300 MB RAID1)

3 voting disks (300MB RAID1)

1 logical drive for Oracle ASMs SPFILE (100MB RAID1)

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 9copy Copyright 2008 IBM Corporation

Configuring the IBM Storage

2 logical drives for Oracle ASM data storage (300 GB RAID5)

2 logical drives for Oracle ASM log storage (135GB RAID1)

The number and the size of the logical drives created for Oracle ASM storage varies based on the individual implementation but the sizes used for the OCR disks voting disks and the Oracle ASM SPFILE disks in the reference configuration are sufficient for any configuration

In the reference configuration the Oracle binaries are installed on internal disks within each individual server The reference configuration uses the JFS2 file system and the logical drive was mounted as u01

2 Configure the shared disks in the operating system

After you have created the logical drives on the storage subsystem the cfgmgr command rescans the AIX system for changes to the physical devices

The lspv command shows the physical devices that AIX has configured

The fget_config -Av command shows the LUN numbers the disk drive name and the associated logical drive names configured on the storage subsystem

3 If the hdisk names for each logical drive on the storage subsystem that will be shared in the cluster are not identical on all cluster nodes create new device nodes for each of the logical drives to give them a consistent name across all nodes

For example if the logical drive for the first OCR disk was hdisk2 on node1 and hdisk3 on node2 a new device node (devnode) called devORA_OCR1 could be created on each host using the major numbers and the minor numbers that correspond with the same logical drive on each system Show the major and the minor numbers with the command ls -l Use the command mknod devORA_OCR1 c ltmajor of OCRgt ltminor of OCRgt to create the new devnode

Even when the disk drive names correlate on all of the nodes you should still create these named device nodes for ease of location and administration Named device nodes were created on the reference configuration for all shared logical drives using the convention devORA_ltusegt

4 Set the reserve_policy attribute for each logical drive to be used for shared access for OCR logical drive voting logical drive or Oracle ASM logical drive

NOTE This operation must be performed on the hdisk device name rather than on the named devnode created in step 3

This operation allows the nodes to share the logical drives Otherwise the first system that accesses the logical drive reserves it for exclusive use Oracle coordinates access to the shared logical drives

10 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

For an IBM storage subsystem reserve_policy=no_reserve is the correct value

a Use the lsattr -E -l hdisk | grep reserve command to check the attribute

b Use the chdev -l hdisk -a reserve_policy=no_reserve command to change the policy

c Repeat step a and step b on each cluster node

5 Set ownership and permissions on devnodes to be used by Oracle

NOTE If you created new devnodes for the shared logical drives set the ownership and permissions on the new devnode that you created not on the original hdisk devnode

a Use the following commands to set the ownership for each OCR device file

chown rootoinstall devltdevnodegt

chmod 640 devltdevnodegt

b Use the following commands to set the ownership and permission on the voting disk and database device files

chown oracledba devltdevnodegt

chmod 660 devltdevnodegt

c Repeat step a and step b on each cluster node

NOTE Do not assign a physical volume identifier (PVID) to hdisks that will be used with Oracle ASM

Installing OracleHere are the basic steps for installing Oracle

bull Create directories for Oracle

bull Set up the Oracle userrsquos profile

bull Install Oracle Clusterware

bull Update Oracle CRS

bull Install and configure Oracle ASM

bull Update Oracle ASM

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 11copy Copyright 2008 IBM Corporation

Installing Oracle Clusterware

bull Install Oracle Database

bull Update the database home

bull Configure the listener names

bull Create and configure a database

Creating Directories for OracleAlthough you can create directory structures for Oracle files in many different ways for ease of support use a standard directory structure based on the Optimal Flexible Architecture (OFA) guidelines published by Oracle For more information about the OFA Guidelines see Appendix D of the Oracle Database Installation Guide 10gR2 for AIX 5L Based Systems (B19075-02) The reference configuration is based on the OFA guidelines

Follow these two important guidelines when creating directories for Oracle

bull You must separate the Oracle CRS home directory from the Oracle base and the database home directories The Oracle CRS installation procedure sets the ownership for the entire Oracle CRS directory tree to root while the database directory tree must be owned by oracle

bull If more than one database will be serviced by Oracle ASM it is recommended that you create a separate home directory for Oracle ASM An Oracle ASM home directory provides the capability to modify the Oracle release for Oracle ASM while not changing the release of any of the databases on the same server The directory tree must be owned by oracle

Setting Up the Oracle Users Profile1 Set up the Oracle users profile before installing the Oracle binaries See

Appendix A for an example profile

2 Add or modify entries for ORACLE_HOME ORACLE_BASE PATH LD_LIBRARY_PATH and similar files to reflect the directories where the Oracle binaries will be placed during the Oracle Clusterware installation

Installing Oracle ClusterwareAs you install the Oracle Clusterware remember the following important points

bull Do not make the CRS_HOME path a subdirectory of the Oracle RAC 10g Release 2 Oracle base directory due to permission changes required for Oracle CRS For more details refer to Appendix E of the Oracle Clusterware Installation Guide

bull The rootsh script must be run as the root user and must be run on only one node at a time not concurrently

12 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Updating Oracle CRS

bull After rootsh runs on the last node it will attempt to run the vipca utility to configure the VIP network interfaces If you have IP addresses that are not routable across the Internet for example 100 192168 the vipca configuration will fail If the automatic vipca configuration fails run the vipca utility manually using the $ORA_CRS_HOMEbinvipca command from the command line as root This requires a graphical display

bull You can do the manual configuration with vipca before clicking OK on the Oracle Universal Installer the Oracle CRS installation page The Oracle CRS installation page waits for you to run rootsh Do the manual configuration with vipca during this time This action allows the remainder of the post-installation checks to complete successfully

Updating Oracle CRSRefer to the Metalink Bulletin 2820361 (see important note on page 5) for required updates The reference configuration used the 10203 update to the Oracle CRS home Download the 10203 update and any other updates required from the Oracle Metalink web site (httpmetalinkoraclecom) and install according to the instructions on the web site

Installing and Configuring Oracle ASMBecause a single Oracle ASM instance can service many database instances Oracle recommends that you install Oracle ASM in its own home directory Follow the naming convention of the OFA guidelines for the directory path As you install and configure Oracle ASM remember the following important points

bull If the Oracle ASM instance was not correctly created by the Oracle Universal Installer Wizard during installation use the dbca utility to add the instance and create the disk group(s)

bull If you want additional disk groups create them using the dbca utility

Updating Oracle ASMUpdate the Oracle ASM instance with the 10203 patch set and any additional updates indicated in Metalink Bulletin 2820361 You can use the same update media for 10203 previously downloaded to update Oracle CRS (page 13) Carefully follow the instructions for updating this database instance

Installing an Oracle DatabaseUse the Oracle Universal Installer to install another database home for a database instance Select Install Database Software Only on the Create Database page

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 13copy Copyright 2008 IBM Corporation

Installing Oracle Clusterware

Updating the Database HomeInstall the 10203 update on the new database home Select the database home instead of Oracle ASM on the Specify Home Details page

Configuring Listener NamesIf your RAC environment consists of only Oracle CRS an Oracle ASM instance and a database instance on each node it is easier to support only one listenerora configuration file for the Oracle ASM and database instances The best way to accomplish this is to create an environment variable TNS_ADMIN and assign it to the $ORACLE_HOMEnetworkadmin directory of the database instance For example

export TNS_ADMIN=$ORA_ASM_HOMEnetworkadmin

This environment variable allows the DBA to maintain one set of tnsnamesora sqlnetora and listenerora files for the entire environment

Creating and Configuring a DatabaseUse the Database Configuration Assistant (DBCA) to create a database For more information refer to the Oracle documentation about using DBCA As you create the database remember the following important points

bull Using DBCA gives you the capability to create the Oracle ASM disk groups if you choose to use Oracle ASM

bull The Flash Recovery Area (FRA) option is highly recommended A FRA allows the setup of a separate area on disk for flashback capability area for the archive logs the second copy of the multiplexed redo log files and for RMAN backups Make sure that the FRA is correctly set up to handle the performance of the database and that the FRA contains enough space

Space to hold all of the data for the data retention periods that are set by the database

Space for the type of backups that will be used

Space for the number of backups that will be held

For further documentation about using the FRA refer to the Oracle documentation as well as the following white paper

Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to

wwwibmcomsupporttechdocs

14 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Creating and Configuring a Database

ConclusionAs you have followed the example of setting up a two-node cluster described in this document you have seen how simple and how effective a deployment of Oracle RAC 10g Release 2 can be When you are doing your own installation be sure to use the applicable installation information as the basis for your work This document describes only tips and techniques Your installer should have the expertise to know how and when to modify instructions as necessary If your environment is complex and you require outside help contact your IBM business partner or sales representative See ldquoContact Informationrdquo below

Contact InformationFor more information and sales office locations visit the IBM web site at

httpwwwibmcom

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 15copy Copyright 2008 IBM Corporation

Contact Information

16 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix A

References

AIXFor AIX operating system downloads go to Quick Links for AIX Fixes

httpwww-933ibmcomeserversupportfixesfixcentralmainpseriesaix

For AIX documentation go to AIX Information

httppublibboulderibmcominfocentersystemsindexjsptopic=comibmaixdocdocbaseaixparenthtm

IBM Corporation 2006 Get the Latest Version of OpenSSH for AIX

httpwwwibmcomdeveloperworkseserverarticlesopenssh_updatedhtml

IBM StorageFor information about the IBM DS4800 Storage Subsystem go to

httpwww-03ibmcomsystemsstoragediskds4000ds4800indexhtml

For information about IBM storage go to

httpwww-03ibmcomsystemsstoragediskcm_re=masthead-_-products-_-stg-disk

OracleFor Oracle documentation go to

httpwwworaclecomtechnologydocumentationindexhtml

IBM Corporation 2008 Tuning IBM AIX 5L for an Oracle Database

httpwwwibmcompartnerworldwpsservletContentHandlerVPAA-6V2C4E

IBM Corporation 2008 Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems

wwwibmcomsupporttechdocs

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem A-1copy Copyright 2008 IBM Corporation

A-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix B

Troubleshooting

CSS Node Evictions can affect your functionality

IO time-outs can cause inappropriate node evictions during multi-path failover or other problems Two parameters might affect this css_misscount and disktimeout With versions of Oracle prior to 102 this was primarily accommodated by increasing the value of the css_misscount parameter However the function of this parameter has changed as of Oracle 102 and changes to this parameter might not achieve the desired result The new disktimeout parameter should be used in most cases to make this adjustment In many cases the default values will work fine However you might find it necessary to increase the default value of this parameter if you are getting a lot of CSS node evictions

For information about how the parameters work and how to adjust them refer to the following two articles Go to httpmetalinkoraclecom

bull Metalink DocID 2944301 CSS Timeout Computation in RAC 10g

bull Metalink DocID 2847521 Steps to Increase CSS Misscount Reboottime and Disktimeout

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem B-1copy Copyright 2008 IBM Corporation

B-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Trademarks and special notices copy Copyright IBM Corporation 2008 All rights reserved

References in this document to IBM products or services do not imply that IBM intends to make them available in every country

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corporation in the United States other countries or both If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (reg or trade) these symbols indicate US registered or common law trademarks owned by IBM at the time this information was published Such trademarks may also be registered or common law trademarks in other countries A current list of IBM trademarks is available on the Web at Copyright and trademark information at wwwibmcomlegalcopytradeshtml

Intel Intel Inside (logos) MMX and Pentium are trademarks of Intel Corporation in the United States other countries or both

UNIX is a registered trademark of The Open Group in the United States and other countries

Linux is a trademark of Linus Torvalds in the United States other countries or both

Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both

Other company product or service names may be trademarks or service marks of others

Information is provided AS IS without warranty of any kind

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved Actual environmental costs and performance characteristics may vary by customer

Information concerning non-IBM products was obtained from a supplier of these products published announcement material or other publicly available sources and does not constitute an endorsement of such products by IBM Sources for non-IBM list prices and performance numbers are taken from publicly available information including vendor announcements and vendor worldwide homepages IBM has not tested these products and cannot confirm the accuracy of performance capability or any other claims related to non-IBM products Questions on the capability of non-IBM products should be addressed to the supplier of those products

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk

  • Title Page13
  • Contents
  • Deploying Oracle RAC 10g Release 2 with ASM on AIX 53 with an IBM DS4700 or DS4800 Storage Subsystem
    • Introduction
      • Required Documents
      • Benefits
      • Reference Configuration
        • Configuring the Hosts
        • Configuring the IBM Storage
        • Installing Oracle
          • Creating Directories for Oracle
          • Setting Up the Oracle Users Profile
            • Installing Oracle Clusterware
              • Updating Oracle CRS
              • Installing and Configuring Oracle ASM
              • Updating Oracle ASM
              • Installing an Oracle Database
              • Updating the Database Home
              • Configuring Listener Names
              • Creating and Configuring a Database
                • Conclusion
                • Contact Information
                  • Appendix A References
                    • AIX
                    • IBM Storage
                    • Oracle
                      • Appendix B Troubleshooting
                      • Copyrights13

Deploying Oracle RAC 10g Release 2 with ASM on AIX 53 with an IBM DS4700 or DS4800 Storage Subsystem

IntroductionOracle Real Application Clusters (Oracle RAC) is a high-availability option to Oracle Database Oracle RAC supports the deployment of a single database across a cluster of servers providing excellent fault tolerance performance and scalability The industry is taking note of Oracle RACs growing importance in a wide range of customer implementations across all industries for transaction processing and data warehousing applications

This document presents an overview and some tips and techniques in deploying the Oracle RAC 10g Release 2 database with Oracle RAC on the IBMreg AIXreg operating system using IBM storage The database storage is administered using the Oracle Automatic Storage Management (Oracle ASM) feature

The procedure of creating a simple configuration using a two-node cluster is used as an example to illustrate the principles of performing this type of implementation Actual implementations vary in complexity and require significant expertise in all of the technologies involved

Required DocumentsThis document is intended to illustrate any deviations from the Oracle installation guide and any enhancements in implementation This document does not provide a comprehensive reference for installing all of these components For full implementation instructions please refer to the product documentation for the hardware being used for the AIX operating system for the Oracle Database and for Oracle Clusterware

For IBM storage documentation contact your IBM business partner or service representative

For AIX documentation go to AIX Information

httppublibboulderibmcominfocentersystemsindexjsptopic=comibmaixdocdocbaseaixparenthtm

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 1copy Copyright 2008 IBM Corporation

Introduction

For Oracle documentation go to

httpwwworaclecomtechnologydocumentationindexhtml

Before installation you must know how to best configure the storage for enhanced performance For this information refer to Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to

wwwibmcomsupporttechdocs

BenefitsThe reference configuration provides insight into how to configure these products to work together to achieve numerous benefits The reference configuration uses Oracle ASM to manage the Oracle RAC 10g Release 2 database storage running under Oracle Clusterware on AIX with an IBM DS48001 storage subsystem The IBM DS4800 storage subsystem is a robust high-performance disk array that provides a scalable flexible and expandable storage foundation upon which to build a highly reliable database system using Oracle RAC

Oracle RAC offers these benefits

bull High availability ndash Provides enhanced uptime for database applications

bull Scalability ndash Expands capacity by adding servers to the cluster

bull Lower computing costs ndash Uses lower-cost hardware and reduces expenses associated with application downtime

bull Excellent performance ndash Spreads the database load across multiple clustered servers allowing performance to scale with the size of the cluster

The IBM DS4800 storage subsystem offers these benefits which go hand-in-hand with those of Oracle RAC

bull High availability ndash The IBM DS4800 storage subsystemsrsquo fully redundant hardware and redundant array of independent disks (RAID) functionality ensures that the storage is as available as the RAC database

bull Scalability ndash Additional storage capacity can be added to the storage subsystem while it continues operation

bull Lower costs ndash Shared storage within a storage area network (SAN) allows reduced cost compared to storage local to individual hosts and reduces management expenses associated with administering the storage

bull Excellent performance ndash The IBM DS4800 storage subsystem is one of the best performing disk arrays in its class Its full front-to-back 4 gigabit Fibre Channel interface architecture provides bandwidth for the most demanding applications

1Solution also works with a DS4700 storage subsystem

2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

The DS4000 Storage Manager software package configures and manages the IBMDS4800 storage subsystemrsquos fully redundant hardware and high-performance architecture with ease DS4000 Storage Manager allows in-band management from servers to attached storage or remotely through the Ethernet The storage subsystem offers several premium features that add functionality when used in a database environment Two of the most helpful features in an Oracle RAC environment are the Storage Partitioning premium feature and the Enhanced Remote Mirroring premium feature

The Storage Partitioning premium feature allows logical drives on the storage subsystem to be distributed to different hosts in your SAN associating each logical drive with one particular host or a group of hosts The logical drives are presented to each host according to the host type for the different operating systems that are running on those hosts The Storage Partitioning premium feature allows great flexibility in the use of the storage subsystem for storage consolidation

The Enhanced Remote Mirroring premium feature (RM) provides the ability to mirror logical drives between two IBM storage subsystems Mirroring provides an additional redundant copy of the data The mirror copy can be used for rapid disaster recovery and can be also used to create a copy of live data for test purposes

Although these two premium features are not used in the reference configuration there are many opportunities to use storage partitioning and Enhanced Remote Mirroring in a real-world Oracle implementation

For more information please discuss these premium features with your IBM business partner or sales representative

Reference ConfigurationThe reference configuration consists of the following equipment

bull 2 IBM P5 p630 servers

bull AIX 53 Technology Level 6

bull IBM DS4800 storage subsystem

bull Oracle Cluster Ready Services (Oracle CRS) 10201 upgraded to 10203

bull Oracle Database 10201 upgraded to 10203

The systems are configured as a two-node RAC cluster Oracle ASM is employed to simplify storage management The AIX RDAC multipathing driver provided failover support

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 3copy Copyright 2008 IBM Corporation

Introduction

Figure 1 Reference Configuration

4 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

Configuring the HostsThe hosts must be configured to function correctly with Oracle RAC The following steps describe in general terms how to prepare the systems for the Oracle installation

IMPORTANT Refer to Metalink Bulletin 2820361 Minimum Software Versions and Patches Required to Support Oracle Products on IBM Series for the latest requirements This bulletin is available to Metalink users athttpmetalinkoraclecom Metalink Bulletin 2820361 is regularly updated and the information it contains supersedes any information found in this document

1 Install the AIX operating system if it is not already installed The reference configuration uses AIX 53 Technology Level 6

2 Install additional software packages required for Oracle

bosadtbase

bosadtlib

bosadtlibm

bosperflibperfstat

bosperfperfstat

bosperfproctools

rsctbasicrte

rsctcompatclientsrte

xlCaix50rte7004 (See the following note)

xlCrte7001 (See the following note)

NOTE The versions of xlCaix50rte and xlCrte supplied on the AIX 53 distribution media are not compatible with Oracle RAC 10g Release 2 You must download newer versions from IBMs web site The reference configuration uses fileset xlcrteaix50apr2007ptftarZ

3 Download and install the updates to the AIX operating system from IBMs website

httpwww14softwareibmcomwebappset2sasfgenunix3aixfixeshtml

The reference configuration uses AIX 53 Technology Level 6

4 Configure the hosts network interfaces

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 5copy Copyright 2008 IBM Corporation

Configuring the Hosts

Use at least two ethernet interfaces

One for the Oracle public network

One for the private network used by Oracle CRS

Oracle recommends that the private interface be on a separate network isolated from all other traffic You should use a separate gigabit ethernet switch for the private network

Additional interfaces provide more flexibility in configuration and higher performance In the reference configuration the VIP network shares the interface used for the public network This is configured later in the installation process using the vipca utility Entries for the internet protocol (IP) address as well as the public and private networks are inserted into the etchosts file at this time

5 Create users and groups

a Create two groups dba and oinstall

b Add a user Oracle with the oinstall primary group and with the dba group

NOTE Make sure that the user ID for the Oracle user and the group numbers for the dba and oinstall groups are consistent on all nodes

6 Set the shell soft limits for the Oracle and root users using the smit chuser command

Soft FILE size = -1

Soft CPU time = -1

Soft DATA segment = -1

Soft STACK size = -1

7 Set system configuration parameters

a Use the smit chgsys command to set the maximum number of processes per user to at least 2048

b Use the vmo -p -o lru_file_repage=0 command to set the lru_file_repage parameter to 0

NOTE This setting instructs the AIX Virtual Memory Manager to take pages only from the file buffer cache allowing Oraclersquos System Global Area (SGA) to remain in memory for better performance

8 Use the usrsbinno -a command to configure the network tuning parameters

Use these minimum values for network tuning parameters

6 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

ipqmaxlen = 512rfc1323 = 1sb_max = 1310720 (2655360)tcp_recvspace = 65536 tcp_sendspace = 65536 udp_recvspace = 655360 (10 udp_sendspace must be less than sb_max)udp_sendspace = 65536

NOTE If you modify these parameters the method depends on whether the system is in compatibility mode For specific instructions refer to the AIX documentation for the no command

NOTE If you change the value of the ipqmaxlen parameter you must restart the system for the change to take effect

9 Download and install the Open Secure Shell (OpenSSH) utility from the IBM web site

httpwwwibmcomdeveloperworkseserverarticlesopenssh_updatedhtml

NOTE The reference configuration uses OpenSSH The SSH utility is not required for Oracle to function however SSH is recommended because it provides a more secure method of communication between nodes than does RSH If SSH is available Oracle will use it otherwise Oracle uses RSH Configuration of RSH user equivalence is outside the scope of this document

10 Configure SSH for root login permission

a Locate the sshd_config file (typically etc or usrlocaletc)

b Edit the sshd_config file changing PermitRootLogin to yes

c Use the startsrc -s sshd command to start sshd

11 Configure SSH for user equivalence for root and Oracle on all nodes

a Log on as ltrootgt

b Use the following commands to generate the SSH key

cd ~ssh

ssh-keygen -t dsa

cat id_dsapub gtgt authorized_keys

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 7copy Copyright 2008 IBM Corporation

Configuring the Hosts

c Copy the authorized_keys file to ~ssh on the next system

d Repeat step b and step c on each system

e After all systems have a public key in the file copy the file back to all of the previous systems so that they all have an identical file

f Test the SSH function from each system to itself and to each of the other systems by issuing such commands as the ssh host1 date command Make sure that the commands can be executed without using a password

g Log on as the Oracle user and repeat step a through step f so that Oracle can also perform remote commands without passwords

NOTE The permissions must restrict write access on both the base user directory and the ssh directory If the permissions are not correctly set OpenSSH requires a password even if the appropriate public key has been added to the authorized_keys file

12 Edit the profile file in the Oracle userrsquos home directory to set the Oracle users parameters

Table 1 shows the entries that you must add to the profile file

Table 1 Oracle Userrsquos Parameters

Parameter to Add Explanation

umask 022

AIXTHREAD_SCOPE=S

An environment variable that directs the system to place user threads within a system-wide rather than a process-wide contention scope This placement provides better performance and more efficient memory use

CRS_HOME A unique home directory for Oracle CRS binaries

ASM_HOME

Use this parameter if you prefer to install separate Oracle binaries for Oracle ASM This method is preferable if there will be more than one database instance supported by Oracle ASM

ORACLE_HOME A unique home for the database binaries

LD_LIBRARY_PATH The complete pathway is $ORACLE_HOMElib

ORACLE_BASE Identifies the Oracle base directory

ORACLE_SID Identifies the instance

PATH Include $ORACLE_HOMEbin

CLASSPATHInclude $ORACLE_HOMEjdbclibltclasses filesgt

8 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

13 Edit the profile file in the root userrsquos home directory to set the same parameters as set in the Oracle users profile

Be sure to add the following entry to the profile file

AIXTHREAD_SCOPE=S

14 Synchronize the system time between the hosts in the cluster using any effective method The reference configuration used Network Time Protocol (NTP) If an NTP server is available use this method to set up xntpd

a Edit etcntpconf with the nameIP of the NTP server(s)

b Add a line server lthostnamegt for each host

c Start the service using the smit xntpd command

15 Because some Oracle utilities expect lsattr in etc and do not function correctly without a link to it use the ln -s usrsbinlsattr etclsattr command to add a soft link

Configuring the IBM Storage

IMPORTANT You must know how to best configure the storage for enhanced performance For this information refer to Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to wwwibmcomsupporttechdocs

Use DS4000 Storage Manager and the AIX system utilities to configure the IBM storage subsystem

1 Use DS4000 Storage Manager to create logical drives on the storage subsystem to be shared for Oracle CRS communication and database storage

NOTE You also may use additional logical drives for Oracle binaries and similar purposes however for those logical drives on the storage subsystem that are used exclusively by one particular host you will need the Storage Partitioning premium feature

You will need to analyze your implementation to determine the appropriate logical drive sizes desired RAID levels and other parameters The reference configuration uses the following minimal configuration of shared logical drives

2 Oracle Cluster Registry (OCR) logical drives (300 MB RAID1)

3 voting disks (300MB RAID1)

1 logical drive for Oracle ASMs SPFILE (100MB RAID1)

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 9copy Copyright 2008 IBM Corporation

Configuring the IBM Storage

2 logical drives for Oracle ASM data storage (300 GB RAID5)

2 logical drives for Oracle ASM log storage (135GB RAID1)

The number and the size of the logical drives created for Oracle ASM storage varies based on the individual implementation but the sizes used for the OCR disks voting disks and the Oracle ASM SPFILE disks in the reference configuration are sufficient for any configuration

In the reference configuration the Oracle binaries are installed on internal disks within each individual server The reference configuration uses the JFS2 file system and the logical drive was mounted as u01

2 Configure the shared disks in the operating system

After you have created the logical drives on the storage subsystem the cfgmgr command rescans the AIX system for changes to the physical devices

The lspv command shows the physical devices that AIX has configured

The fget_config -Av command shows the LUN numbers the disk drive name and the associated logical drive names configured on the storage subsystem

3 If the hdisk names for each logical drive on the storage subsystem that will be shared in the cluster are not identical on all cluster nodes create new device nodes for each of the logical drives to give them a consistent name across all nodes

For example if the logical drive for the first OCR disk was hdisk2 on node1 and hdisk3 on node2 a new device node (devnode) called devORA_OCR1 could be created on each host using the major numbers and the minor numbers that correspond with the same logical drive on each system Show the major and the minor numbers with the command ls -l Use the command mknod devORA_OCR1 c ltmajor of OCRgt ltminor of OCRgt to create the new devnode

Even when the disk drive names correlate on all of the nodes you should still create these named device nodes for ease of location and administration Named device nodes were created on the reference configuration for all shared logical drives using the convention devORA_ltusegt

4 Set the reserve_policy attribute for each logical drive to be used for shared access for OCR logical drive voting logical drive or Oracle ASM logical drive

NOTE This operation must be performed on the hdisk device name rather than on the named devnode created in step 3

This operation allows the nodes to share the logical drives Otherwise the first system that accesses the logical drive reserves it for exclusive use Oracle coordinates access to the shared logical drives

10 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

For an IBM storage subsystem reserve_policy=no_reserve is the correct value

a Use the lsattr -E -l hdisk | grep reserve command to check the attribute

b Use the chdev -l hdisk -a reserve_policy=no_reserve command to change the policy

c Repeat step a and step b on each cluster node

5 Set ownership and permissions on devnodes to be used by Oracle

NOTE If you created new devnodes for the shared logical drives set the ownership and permissions on the new devnode that you created not on the original hdisk devnode

a Use the following commands to set the ownership for each OCR device file

chown rootoinstall devltdevnodegt

chmod 640 devltdevnodegt

b Use the following commands to set the ownership and permission on the voting disk and database device files

chown oracledba devltdevnodegt

chmod 660 devltdevnodegt

c Repeat step a and step b on each cluster node

NOTE Do not assign a physical volume identifier (PVID) to hdisks that will be used with Oracle ASM

Installing OracleHere are the basic steps for installing Oracle

bull Create directories for Oracle

bull Set up the Oracle userrsquos profile

bull Install Oracle Clusterware

bull Update Oracle CRS

bull Install and configure Oracle ASM

bull Update Oracle ASM

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 11copy Copyright 2008 IBM Corporation

Installing Oracle Clusterware

bull Install Oracle Database

bull Update the database home

bull Configure the listener names

bull Create and configure a database

Creating Directories for OracleAlthough you can create directory structures for Oracle files in many different ways for ease of support use a standard directory structure based on the Optimal Flexible Architecture (OFA) guidelines published by Oracle For more information about the OFA Guidelines see Appendix D of the Oracle Database Installation Guide 10gR2 for AIX 5L Based Systems (B19075-02) The reference configuration is based on the OFA guidelines

Follow these two important guidelines when creating directories for Oracle

bull You must separate the Oracle CRS home directory from the Oracle base and the database home directories The Oracle CRS installation procedure sets the ownership for the entire Oracle CRS directory tree to root while the database directory tree must be owned by oracle

bull If more than one database will be serviced by Oracle ASM it is recommended that you create a separate home directory for Oracle ASM An Oracle ASM home directory provides the capability to modify the Oracle release for Oracle ASM while not changing the release of any of the databases on the same server The directory tree must be owned by oracle

Setting Up the Oracle Users Profile1 Set up the Oracle users profile before installing the Oracle binaries See

Appendix A for an example profile

2 Add or modify entries for ORACLE_HOME ORACLE_BASE PATH LD_LIBRARY_PATH and similar files to reflect the directories where the Oracle binaries will be placed during the Oracle Clusterware installation

Installing Oracle ClusterwareAs you install the Oracle Clusterware remember the following important points

bull Do not make the CRS_HOME path a subdirectory of the Oracle RAC 10g Release 2 Oracle base directory due to permission changes required for Oracle CRS For more details refer to Appendix E of the Oracle Clusterware Installation Guide

bull The rootsh script must be run as the root user and must be run on only one node at a time not concurrently

12 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Updating Oracle CRS

bull After rootsh runs on the last node it will attempt to run the vipca utility to configure the VIP network interfaces If you have IP addresses that are not routable across the Internet for example 100 192168 the vipca configuration will fail If the automatic vipca configuration fails run the vipca utility manually using the $ORA_CRS_HOMEbinvipca command from the command line as root This requires a graphical display

bull You can do the manual configuration with vipca before clicking OK on the Oracle Universal Installer the Oracle CRS installation page The Oracle CRS installation page waits for you to run rootsh Do the manual configuration with vipca during this time This action allows the remainder of the post-installation checks to complete successfully

Updating Oracle CRSRefer to the Metalink Bulletin 2820361 (see important note on page 5) for required updates The reference configuration used the 10203 update to the Oracle CRS home Download the 10203 update and any other updates required from the Oracle Metalink web site (httpmetalinkoraclecom) and install according to the instructions on the web site

Installing and Configuring Oracle ASMBecause a single Oracle ASM instance can service many database instances Oracle recommends that you install Oracle ASM in its own home directory Follow the naming convention of the OFA guidelines for the directory path As you install and configure Oracle ASM remember the following important points

bull If the Oracle ASM instance was not correctly created by the Oracle Universal Installer Wizard during installation use the dbca utility to add the instance and create the disk group(s)

bull If you want additional disk groups create them using the dbca utility

Updating Oracle ASMUpdate the Oracle ASM instance with the 10203 patch set and any additional updates indicated in Metalink Bulletin 2820361 You can use the same update media for 10203 previously downloaded to update Oracle CRS (page 13) Carefully follow the instructions for updating this database instance

Installing an Oracle DatabaseUse the Oracle Universal Installer to install another database home for a database instance Select Install Database Software Only on the Create Database page

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 13copy Copyright 2008 IBM Corporation

Installing Oracle Clusterware

Updating the Database HomeInstall the 10203 update on the new database home Select the database home instead of Oracle ASM on the Specify Home Details page

Configuring Listener NamesIf your RAC environment consists of only Oracle CRS an Oracle ASM instance and a database instance on each node it is easier to support only one listenerora configuration file for the Oracle ASM and database instances The best way to accomplish this is to create an environment variable TNS_ADMIN and assign it to the $ORACLE_HOMEnetworkadmin directory of the database instance For example

export TNS_ADMIN=$ORA_ASM_HOMEnetworkadmin

This environment variable allows the DBA to maintain one set of tnsnamesora sqlnetora and listenerora files for the entire environment

Creating and Configuring a DatabaseUse the Database Configuration Assistant (DBCA) to create a database For more information refer to the Oracle documentation about using DBCA As you create the database remember the following important points

bull Using DBCA gives you the capability to create the Oracle ASM disk groups if you choose to use Oracle ASM

bull The Flash Recovery Area (FRA) option is highly recommended A FRA allows the setup of a separate area on disk for flashback capability area for the archive logs the second copy of the multiplexed redo log files and for RMAN backups Make sure that the FRA is correctly set up to handle the performance of the database and that the FRA contains enough space

Space to hold all of the data for the data retention periods that are set by the database

Space for the type of backups that will be used

Space for the number of backups that will be held

For further documentation about using the FRA refer to the Oracle documentation as well as the following white paper

Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to

wwwibmcomsupporttechdocs

14 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Creating and Configuring a Database

ConclusionAs you have followed the example of setting up a two-node cluster described in this document you have seen how simple and how effective a deployment of Oracle RAC 10g Release 2 can be When you are doing your own installation be sure to use the applicable installation information as the basis for your work This document describes only tips and techniques Your installer should have the expertise to know how and when to modify instructions as necessary If your environment is complex and you require outside help contact your IBM business partner or sales representative See ldquoContact Informationrdquo below

Contact InformationFor more information and sales office locations visit the IBM web site at

httpwwwibmcom

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 15copy Copyright 2008 IBM Corporation

Contact Information

16 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix A

References

AIXFor AIX operating system downloads go to Quick Links for AIX Fixes

httpwww-933ibmcomeserversupportfixesfixcentralmainpseriesaix

For AIX documentation go to AIX Information

httppublibboulderibmcominfocentersystemsindexjsptopic=comibmaixdocdocbaseaixparenthtm

IBM Corporation 2006 Get the Latest Version of OpenSSH for AIX

httpwwwibmcomdeveloperworkseserverarticlesopenssh_updatedhtml

IBM StorageFor information about the IBM DS4800 Storage Subsystem go to

httpwww-03ibmcomsystemsstoragediskds4000ds4800indexhtml

For information about IBM storage go to

httpwww-03ibmcomsystemsstoragediskcm_re=masthead-_-products-_-stg-disk

OracleFor Oracle documentation go to

httpwwworaclecomtechnologydocumentationindexhtml

IBM Corporation 2008 Tuning IBM AIX 5L for an Oracle Database

httpwwwibmcompartnerworldwpsservletContentHandlerVPAA-6V2C4E

IBM Corporation 2008 Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems

wwwibmcomsupporttechdocs

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem A-1copy Copyright 2008 IBM Corporation

A-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix B

Troubleshooting

CSS Node Evictions can affect your functionality

IO time-outs can cause inappropriate node evictions during multi-path failover or other problems Two parameters might affect this css_misscount and disktimeout With versions of Oracle prior to 102 this was primarily accommodated by increasing the value of the css_misscount parameter However the function of this parameter has changed as of Oracle 102 and changes to this parameter might not achieve the desired result The new disktimeout parameter should be used in most cases to make this adjustment In many cases the default values will work fine However you might find it necessary to increase the default value of this parameter if you are getting a lot of CSS node evictions

For information about how the parameters work and how to adjust them refer to the following two articles Go to httpmetalinkoraclecom

bull Metalink DocID 2944301 CSS Timeout Computation in RAC 10g

bull Metalink DocID 2847521 Steps to Increase CSS Misscount Reboottime and Disktimeout

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem B-1copy Copyright 2008 IBM Corporation

B-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Trademarks and special notices copy Copyright IBM Corporation 2008 All rights reserved

References in this document to IBM products or services do not imply that IBM intends to make them available in every country

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corporation in the United States other countries or both If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (reg or trade) these symbols indicate US registered or common law trademarks owned by IBM at the time this information was published Such trademarks may also be registered or common law trademarks in other countries A current list of IBM trademarks is available on the Web at Copyright and trademark information at wwwibmcomlegalcopytradeshtml

Intel Intel Inside (logos) MMX and Pentium are trademarks of Intel Corporation in the United States other countries or both

UNIX is a registered trademark of The Open Group in the United States and other countries

Linux is a trademark of Linus Torvalds in the United States other countries or both

Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both

Other company product or service names may be trademarks or service marks of others

Information is provided AS IS without warranty of any kind

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved Actual environmental costs and performance characteristics may vary by customer

Information concerning non-IBM products was obtained from a supplier of these products published announcement material or other publicly available sources and does not constitute an endorsement of such products by IBM Sources for non-IBM list prices and performance numbers are taken from publicly available information including vendor announcements and vendor worldwide homepages IBM has not tested these products and cannot confirm the accuracy of performance capability or any other claims related to non-IBM products Questions on the capability of non-IBM products should be addressed to the supplier of those products

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk

  • Title Page13
  • Contents
  • Deploying Oracle RAC 10g Release 2 with ASM on AIX 53 with an IBM DS4700 or DS4800 Storage Subsystem
    • Introduction
      • Required Documents
      • Benefits
      • Reference Configuration
        • Configuring the Hosts
        • Configuring the IBM Storage
        • Installing Oracle
          • Creating Directories for Oracle
          • Setting Up the Oracle Users Profile
            • Installing Oracle Clusterware
              • Updating Oracle CRS
              • Installing and Configuring Oracle ASM
              • Updating Oracle ASM
              • Installing an Oracle Database
              • Updating the Database Home
              • Configuring Listener Names
              • Creating and Configuring a Database
                • Conclusion
                • Contact Information
                  • Appendix A References
                    • AIX
                    • IBM Storage
                    • Oracle
                      • Appendix B Troubleshooting
                      • Copyrights13

Introduction

For Oracle documentation go to

httpwwworaclecomtechnologydocumentationindexhtml

Before installation you must know how to best configure the storage for enhanced performance For this information refer to Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to

wwwibmcomsupporttechdocs

BenefitsThe reference configuration provides insight into how to configure these products to work together to achieve numerous benefits The reference configuration uses Oracle ASM to manage the Oracle RAC 10g Release 2 database storage running under Oracle Clusterware on AIX with an IBM DS48001 storage subsystem The IBM DS4800 storage subsystem is a robust high-performance disk array that provides a scalable flexible and expandable storage foundation upon which to build a highly reliable database system using Oracle RAC

Oracle RAC offers these benefits

bull High availability ndash Provides enhanced uptime for database applications

bull Scalability ndash Expands capacity by adding servers to the cluster

bull Lower computing costs ndash Uses lower-cost hardware and reduces expenses associated with application downtime

bull Excellent performance ndash Spreads the database load across multiple clustered servers allowing performance to scale with the size of the cluster

The IBM DS4800 storage subsystem offers these benefits which go hand-in-hand with those of Oracle RAC

bull High availability ndash The IBM DS4800 storage subsystemsrsquo fully redundant hardware and redundant array of independent disks (RAID) functionality ensures that the storage is as available as the RAC database

bull Scalability ndash Additional storage capacity can be added to the storage subsystem while it continues operation

bull Lower costs ndash Shared storage within a storage area network (SAN) allows reduced cost compared to storage local to individual hosts and reduces management expenses associated with administering the storage

bull Excellent performance ndash The IBM DS4800 storage subsystem is one of the best performing disk arrays in its class Its full front-to-back 4 gigabit Fibre Channel interface architecture provides bandwidth for the most demanding applications

1Solution also works with a DS4700 storage subsystem

2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

The DS4000 Storage Manager software package configures and manages the IBMDS4800 storage subsystemrsquos fully redundant hardware and high-performance architecture with ease DS4000 Storage Manager allows in-band management from servers to attached storage or remotely through the Ethernet The storage subsystem offers several premium features that add functionality when used in a database environment Two of the most helpful features in an Oracle RAC environment are the Storage Partitioning premium feature and the Enhanced Remote Mirroring premium feature

The Storage Partitioning premium feature allows logical drives on the storage subsystem to be distributed to different hosts in your SAN associating each logical drive with one particular host or a group of hosts The logical drives are presented to each host according to the host type for the different operating systems that are running on those hosts The Storage Partitioning premium feature allows great flexibility in the use of the storage subsystem for storage consolidation

The Enhanced Remote Mirroring premium feature (RM) provides the ability to mirror logical drives between two IBM storage subsystems Mirroring provides an additional redundant copy of the data The mirror copy can be used for rapid disaster recovery and can be also used to create a copy of live data for test purposes

Although these two premium features are not used in the reference configuration there are many opportunities to use storage partitioning and Enhanced Remote Mirroring in a real-world Oracle implementation

For more information please discuss these premium features with your IBM business partner or sales representative

Reference ConfigurationThe reference configuration consists of the following equipment

bull 2 IBM P5 p630 servers

bull AIX 53 Technology Level 6

bull IBM DS4800 storage subsystem

bull Oracle Cluster Ready Services (Oracle CRS) 10201 upgraded to 10203

bull Oracle Database 10201 upgraded to 10203

The systems are configured as a two-node RAC cluster Oracle ASM is employed to simplify storage management The AIX RDAC multipathing driver provided failover support

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 3copy Copyright 2008 IBM Corporation

Introduction

Figure 1 Reference Configuration

4 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

Configuring the HostsThe hosts must be configured to function correctly with Oracle RAC The following steps describe in general terms how to prepare the systems for the Oracle installation

IMPORTANT Refer to Metalink Bulletin 2820361 Minimum Software Versions and Patches Required to Support Oracle Products on IBM Series for the latest requirements This bulletin is available to Metalink users athttpmetalinkoraclecom Metalink Bulletin 2820361 is regularly updated and the information it contains supersedes any information found in this document

1 Install the AIX operating system if it is not already installed The reference configuration uses AIX 53 Technology Level 6

2 Install additional software packages required for Oracle

bosadtbase

bosadtlib

bosadtlibm

bosperflibperfstat

bosperfperfstat

bosperfproctools

rsctbasicrte

rsctcompatclientsrte

xlCaix50rte7004 (See the following note)

xlCrte7001 (See the following note)

NOTE The versions of xlCaix50rte and xlCrte supplied on the AIX 53 distribution media are not compatible with Oracle RAC 10g Release 2 You must download newer versions from IBMs web site The reference configuration uses fileset xlcrteaix50apr2007ptftarZ

3 Download and install the updates to the AIX operating system from IBMs website

httpwww14softwareibmcomwebappset2sasfgenunix3aixfixeshtml

The reference configuration uses AIX 53 Technology Level 6

4 Configure the hosts network interfaces

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 5copy Copyright 2008 IBM Corporation

Configuring the Hosts

Use at least two ethernet interfaces

One for the Oracle public network

One for the private network used by Oracle CRS

Oracle recommends that the private interface be on a separate network isolated from all other traffic You should use a separate gigabit ethernet switch for the private network

Additional interfaces provide more flexibility in configuration and higher performance In the reference configuration the VIP network shares the interface used for the public network This is configured later in the installation process using the vipca utility Entries for the internet protocol (IP) address as well as the public and private networks are inserted into the etchosts file at this time

5 Create users and groups

a Create two groups dba and oinstall

b Add a user Oracle with the oinstall primary group and with the dba group

NOTE Make sure that the user ID for the Oracle user and the group numbers for the dba and oinstall groups are consistent on all nodes

6 Set the shell soft limits for the Oracle and root users using the smit chuser command

Soft FILE size = -1

Soft CPU time = -1

Soft DATA segment = -1

Soft STACK size = -1

7 Set system configuration parameters

a Use the smit chgsys command to set the maximum number of processes per user to at least 2048

b Use the vmo -p -o lru_file_repage=0 command to set the lru_file_repage parameter to 0

NOTE This setting instructs the AIX Virtual Memory Manager to take pages only from the file buffer cache allowing Oraclersquos System Global Area (SGA) to remain in memory for better performance

8 Use the usrsbinno -a command to configure the network tuning parameters

Use these minimum values for network tuning parameters

6 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

ipqmaxlen = 512rfc1323 = 1sb_max = 1310720 (2655360)tcp_recvspace = 65536 tcp_sendspace = 65536 udp_recvspace = 655360 (10 udp_sendspace must be less than sb_max)udp_sendspace = 65536

NOTE If you modify these parameters the method depends on whether the system is in compatibility mode For specific instructions refer to the AIX documentation for the no command

NOTE If you change the value of the ipqmaxlen parameter you must restart the system for the change to take effect

9 Download and install the Open Secure Shell (OpenSSH) utility from the IBM web site

httpwwwibmcomdeveloperworkseserverarticlesopenssh_updatedhtml

NOTE The reference configuration uses OpenSSH The SSH utility is not required for Oracle to function however SSH is recommended because it provides a more secure method of communication between nodes than does RSH If SSH is available Oracle will use it otherwise Oracle uses RSH Configuration of RSH user equivalence is outside the scope of this document

10 Configure SSH for root login permission

a Locate the sshd_config file (typically etc or usrlocaletc)

b Edit the sshd_config file changing PermitRootLogin to yes

c Use the startsrc -s sshd command to start sshd

11 Configure SSH for user equivalence for root and Oracle on all nodes

a Log on as ltrootgt

b Use the following commands to generate the SSH key

cd ~ssh

ssh-keygen -t dsa

cat id_dsapub gtgt authorized_keys

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 7copy Copyright 2008 IBM Corporation

Configuring the Hosts

c Copy the authorized_keys file to ~ssh on the next system

d Repeat step b and step c on each system

e After all systems have a public key in the file copy the file back to all of the previous systems so that they all have an identical file

f Test the SSH function from each system to itself and to each of the other systems by issuing such commands as the ssh host1 date command Make sure that the commands can be executed without using a password

g Log on as the Oracle user and repeat step a through step f so that Oracle can also perform remote commands without passwords

NOTE The permissions must restrict write access on both the base user directory and the ssh directory If the permissions are not correctly set OpenSSH requires a password even if the appropriate public key has been added to the authorized_keys file

12 Edit the profile file in the Oracle userrsquos home directory to set the Oracle users parameters

Table 1 shows the entries that you must add to the profile file

Table 1 Oracle Userrsquos Parameters

Parameter to Add Explanation

umask 022

AIXTHREAD_SCOPE=S

An environment variable that directs the system to place user threads within a system-wide rather than a process-wide contention scope This placement provides better performance and more efficient memory use

CRS_HOME A unique home directory for Oracle CRS binaries

ASM_HOME

Use this parameter if you prefer to install separate Oracle binaries for Oracle ASM This method is preferable if there will be more than one database instance supported by Oracle ASM

ORACLE_HOME A unique home for the database binaries

LD_LIBRARY_PATH The complete pathway is $ORACLE_HOMElib

ORACLE_BASE Identifies the Oracle base directory

ORACLE_SID Identifies the instance

PATH Include $ORACLE_HOMEbin

CLASSPATHInclude $ORACLE_HOMEjdbclibltclasses filesgt

8 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

13 Edit the profile file in the root userrsquos home directory to set the same parameters as set in the Oracle users profile

Be sure to add the following entry to the profile file

AIXTHREAD_SCOPE=S

14 Synchronize the system time between the hosts in the cluster using any effective method The reference configuration used Network Time Protocol (NTP) If an NTP server is available use this method to set up xntpd

a Edit etcntpconf with the nameIP of the NTP server(s)

b Add a line server lthostnamegt for each host

c Start the service using the smit xntpd command

15 Because some Oracle utilities expect lsattr in etc and do not function correctly without a link to it use the ln -s usrsbinlsattr etclsattr command to add a soft link

Configuring the IBM Storage

IMPORTANT You must know how to best configure the storage for enhanced performance For this information refer to Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to wwwibmcomsupporttechdocs

Use DS4000 Storage Manager and the AIX system utilities to configure the IBM storage subsystem

1 Use DS4000 Storage Manager to create logical drives on the storage subsystem to be shared for Oracle CRS communication and database storage

NOTE You also may use additional logical drives for Oracle binaries and similar purposes however for those logical drives on the storage subsystem that are used exclusively by one particular host you will need the Storage Partitioning premium feature

You will need to analyze your implementation to determine the appropriate logical drive sizes desired RAID levels and other parameters The reference configuration uses the following minimal configuration of shared logical drives

2 Oracle Cluster Registry (OCR) logical drives (300 MB RAID1)

3 voting disks (300MB RAID1)

1 logical drive for Oracle ASMs SPFILE (100MB RAID1)

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 9copy Copyright 2008 IBM Corporation

Configuring the IBM Storage

2 logical drives for Oracle ASM data storage (300 GB RAID5)

2 logical drives for Oracle ASM log storage (135GB RAID1)

The number and the size of the logical drives created for Oracle ASM storage varies based on the individual implementation but the sizes used for the OCR disks voting disks and the Oracle ASM SPFILE disks in the reference configuration are sufficient for any configuration

In the reference configuration the Oracle binaries are installed on internal disks within each individual server The reference configuration uses the JFS2 file system and the logical drive was mounted as u01

2 Configure the shared disks in the operating system

After you have created the logical drives on the storage subsystem the cfgmgr command rescans the AIX system for changes to the physical devices

The lspv command shows the physical devices that AIX has configured

The fget_config -Av command shows the LUN numbers the disk drive name and the associated logical drive names configured on the storage subsystem

3 If the hdisk names for each logical drive on the storage subsystem that will be shared in the cluster are not identical on all cluster nodes create new device nodes for each of the logical drives to give them a consistent name across all nodes

For example if the logical drive for the first OCR disk was hdisk2 on node1 and hdisk3 on node2 a new device node (devnode) called devORA_OCR1 could be created on each host using the major numbers and the minor numbers that correspond with the same logical drive on each system Show the major and the minor numbers with the command ls -l Use the command mknod devORA_OCR1 c ltmajor of OCRgt ltminor of OCRgt to create the new devnode

Even when the disk drive names correlate on all of the nodes you should still create these named device nodes for ease of location and administration Named device nodes were created on the reference configuration for all shared logical drives using the convention devORA_ltusegt

4 Set the reserve_policy attribute for each logical drive to be used for shared access for OCR logical drive voting logical drive or Oracle ASM logical drive

NOTE This operation must be performed on the hdisk device name rather than on the named devnode created in step 3

This operation allows the nodes to share the logical drives Otherwise the first system that accesses the logical drive reserves it for exclusive use Oracle coordinates access to the shared logical drives

10 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

For an IBM storage subsystem reserve_policy=no_reserve is the correct value

a Use the lsattr -E -l hdisk | grep reserve command to check the attribute

b Use the chdev -l hdisk -a reserve_policy=no_reserve command to change the policy

c Repeat step a and step b on each cluster node

5 Set ownership and permissions on devnodes to be used by Oracle

NOTE If you created new devnodes for the shared logical drives set the ownership and permissions on the new devnode that you created not on the original hdisk devnode

a Use the following commands to set the ownership for each OCR device file

chown rootoinstall devltdevnodegt

chmod 640 devltdevnodegt

b Use the following commands to set the ownership and permission on the voting disk and database device files

chown oracledba devltdevnodegt

chmod 660 devltdevnodegt

c Repeat step a and step b on each cluster node

NOTE Do not assign a physical volume identifier (PVID) to hdisks that will be used with Oracle ASM

Installing OracleHere are the basic steps for installing Oracle

bull Create directories for Oracle

bull Set up the Oracle userrsquos profile

bull Install Oracle Clusterware

bull Update Oracle CRS

bull Install and configure Oracle ASM

bull Update Oracle ASM

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 11copy Copyright 2008 IBM Corporation

Installing Oracle Clusterware

bull Install Oracle Database

bull Update the database home

bull Configure the listener names

bull Create and configure a database

Creating Directories for OracleAlthough you can create directory structures for Oracle files in many different ways for ease of support use a standard directory structure based on the Optimal Flexible Architecture (OFA) guidelines published by Oracle For more information about the OFA Guidelines see Appendix D of the Oracle Database Installation Guide 10gR2 for AIX 5L Based Systems (B19075-02) The reference configuration is based on the OFA guidelines

Follow these two important guidelines when creating directories for Oracle

bull You must separate the Oracle CRS home directory from the Oracle base and the database home directories The Oracle CRS installation procedure sets the ownership for the entire Oracle CRS directory tree to root while the database directory tree must be owned by oracle

bull If more than one database will be serviced by Oracle ASM it is recommended that you create a separate home directory for Oracle ASM An Oracle ASM home directory provides the capability to modify the Oracle release for Oracle ASM while not changing the release of any of the databases on the same server The directory tree must be owned by oracle

Setting Up the Oracle Users Profile1 Set up the Oracle users profile before installing the Oracle binaries See

Appendix A for an example profile

2 Add or modify entries for ORACLE_HOME ORACLE_BASE PATH LD_LIBRARY_PATH and similar files to reflect the directories where the Oracle binaries will be placed during the Oracle Clusterware installation

Installing Oracle ClusterwareAs you install the Oracle Clusterware remember the following important points

bull Do not make the CRS_HOME path a subdirectory of the Oracle RAC 10g Release 2 Oracle base directory due to permission changes required for Oracle CRS For more details refer to Appendix E of the Oracle Clusterware Installation Guide

bull The rootsh script must be run as the root user and must be run on only one node at a time not concurrently

12 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Updating Oracle CRS

bull After rootsh runs on the last node it will attempt to run the vipca utility to configure the VIP network interfaces If you have IP addresses that are not routable across the Internet for example 100 192168 the vipca configuration will fail If the automatic vipca configuration fails run the vipca utility manually using the $ORA_CRS_HOMEbinvipca command from the command line as root This requires a graphical display

bull You can do the manual configuration with vipca before clicking OK on the Oracle Universal Installer the Oracle CRS installation page The Oracle CRS installation page waits for you to run rootsh Do the manual configuration with vipca during this time This action allows the remainder of the post-installation checks to complete successfully

Updating Oracle CRSRefer to the Metalink Bulletin 2820361 (see important note on page 5) for required updates The reference configuration used the 10203 update to the Oracle CRS home Download the 10203 update and any other updates required from the Oracle Metalink web site (httpmetalinkoraclecom) and install according to the instructions on the web site

Installing and Configuring Oracle ASMBecause a single Oracle ASM instance can service many database instances Oracle recommends that you install Oracle ASM in its own home directory Follow the naming convention of the OFA guidelines for the directory path As you install and configure Oracle ASM remember the following important points

bull If the Oracle ASM instance was not correctly created by the Oracle Universal Installer Wizard during installation use the dbca utility to add the instance and create the disk group(s)

bull If you want additional disk groups create them using the dbca utility

Updating Oracle ASMUpdate the Oracle ASM instance with the 10203 patch set and any additional updates indicated in Metalink Bulletin 2820361 You can use the same update media for 10203 previously downloaded to update Oracle CRS (page 13) Carefully follow the instructions for updating this database instance

Installing an Oracle DatabaseUse the Oracle Universal Installer to install another database home for a database instance Select Install Database Software Only on the Create Database page

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 13copy Copyright 2008 IBM Corporation

Installing Oracle Clusterware

Updating the Database HomeInstall the 10203 update on the new database home Select the database home instead of Oracle ASM on the Specify Home Details page

Configuring Listener NamesIf your RAC environment consists of only Oracle CRS an Oracle ASM instance and a database instance on each node it is easier to support only one listenerora configuration file for the Oracle ASM and database instances The best way to accomplish this is to create an environment variable TNS_ADMIN and assign it to the $ORACLE_HOMEnetworkadmin directory of the database instance For example

export TNS_ADMIN=$ORA_ASM_HOMEnetworkadmin

This environment variable allows the DBA to maintain one set of tnsnamesora sqlnetora and listenerora files for the entire environment

Creating and Configuring a DatabaseUse the Database Configuration Assistant (DBCA) to create a database For more information refer to the Oracle documentation about using DBCA As you create the database remember the following important points

bull Using DBCA gives you the capability to create the Oracle ASM disk groups if you choose to use Oracle ASM

bull The Flash Recovery Area (FRA) option is highly recommended A FRA allows the setup of a separate area on disk for flashback capability area for the archive logs the second copy of the multiplexed redo log files and for RMAN backups Make sure that the FRA is correctly set up to handle the performance of the database and that the FRA contains enough space

Space to hold all of the data for the data retention periods that are set by the database

Space for the type of backups that will be used

Space for the number of backups that will be held

For further documentation about using the FRA refer to the Oracle documentation as well as the following white paper

Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to

wwwibmcomsupporttechdocs

14 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Creating and Configuring a Database

ConclusionAs you have followed the example of setting up a two-node cluster described in this document you have seen how simple and how effective a deployment of Oracle RAC 10g Release 2 can be When you are doing your own installation be sure to use the applicable installation information as the basis for your work This document describes only tips and techniques Your installer should have the expertise to know how and when to modify instructions as necessary If your environment is complex and you require outside help contact your IBM business partner or sales representative See ldquoContact Informationrdquo below

Contact InformationFor more information and sales office locations visit the IBM web site at

httpwwwibmcom

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 15copy Copyright 2008 IBM Corporation

Contact Information

16 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix A

References

AIXFor AIX operating system downloads go to Quick Links for AIX Fixes

httpwww-933ibmcomeserversupportfixesfixcentralmainpseriesaix

For AIX documentation go to AIX Information

httppublibboulderibmcominfocentersystemsindexjsptopic=comibmaixdocdocbaseaixparenthtm

IBM Corporation 2006 Get the Latest Version of OpenSSH for AIX

httpwwwibmcomdeveloperworkseserverarticlesopenssh_updatedhtml

IBM StorageFor information about the IBM DS4800 Storage Subsystem go to

httpwww-03ibmcomsystemsstoragediskds4000ds4800indexhtml

For information about IBM storage go to

httpwww-03ibmcomsystemsstoragediskcm_re=masthead-_-products-_-stg-disk

OracleFor Oracle documentation go to

httpwwworaclecomtechnologydocumentationindexhtml

IBM Corporation 2008 Tuning IBM AIX 5L for an Oracle Database

httpwwwibmcompartnerworldwpsservletContentHandlerVPAA-6V2C4E

IBM Corporation 2008 Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems

wwwibmcomsupporttechdocs

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem A-1copy Copyright 2008 IBM Corporation

A-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix B

Troubleshooting

CSS Node Evictions can affect your functionality

IO time-outs can cause inappropriate node evictions during multi-path failover or other problems Two parameters might affect this css_misscount and disktimeout With versions of Oracle prior to 102 this was primarily accommodated by increasing the value of the css_misscount parameter However the function of this parameter has changed as of Oracle 102 and changes to this parameter might not achieve the desired result The new disktimeout parameter should be used in most cases to make this adjustment In many cases the default values will work fine However you might find it necessary to increase the default value of this parameter if you are getting a lot of CSS node evictions

For information about how the parameters work and how to adjust them refer to the following two articles Go to httpmetalinkoraclecom

bull Metalink DocID 2944301 CSS Timeout Computation in RAC 10g

bull Metalink DocID 2847521 Steps to Increase CSS Misscount Reboottime and Disktimeout

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem B-1copy Copyright 2008 IBM Corporation

B-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Trademarks and special notices copy Copyright IBM Corporation 2008 All rights reserved

References in this document to IBM products or services do not imply that IBM intends to make them available in every country

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corporation in the United States other countries or both If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (reg or trade) these symbols indicate US registered or common law trademarks owned by IBM at the time this information was published Such trademarks may also be registered or common law trademarks in other countries A current list of IBM trademarks is available on the Web at Copyright and trademark information at wwwibmcomlegalcopytradeshtml

Intel Intel Inside (logos) MMX and Pentium are trademarks of Intel Corporation in the United States other countries or both

UNIX is a registered trademark of The Open Group in the United States and other countries

Linux is a trademark of Linus Torvalds in the United States other countries or both

Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both

Other company product or service names may be trademarks or service marks of others

Information is provided AS IS without warranty of any kind

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved Actual environmental costs and performance characteristics may vary by customer

Information concerning non-IBM products was obtained from a supplier of these products published announcement material or other publicly available sources and does not constitute an endorsement of such products by IBM Sources for non-IBM list prices and performance numbers are taken from publicly available information including vendor announcements and vendor worldwide homepages IBM has not tested these products and cannot confirm the accuracy of performance capability or any other claims related to non-IBM products Questions on the capability of non-IBM products should be addressed to the supplier of those products

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk

  • Title Page13
  • Contents
  • Deploying Oracle RAC 10g Release 2 with ASM on AIX 53 with an IBM DS4700 or DS4800 Storage Subsystem
    • Introduction
      • Required Documents
      • Benefits
      • Reference Configuration
        • Configuring the Hosts
        • Configuring the IBM Storage
        • Installing Oracle
          • Creating Directories for Oracle
          • Setting Up the Oracle Users Profile
            • Installing Oracle Clusterware
              • Updating Oracle CRS
              • Installing and Configuring Oracle ASM
              • Updating Oracle ASM
              • Installing an Oracle Database
              • Updating the Database Home
              • Configuring Listener Names
              • Creating and Configuring a Database
                • Conclusion
                • Contact Information
                  • Appendix A References
                    • AIX
                    • IBM Storage
                    • Oracle
                      • Appendix B Troubleshooting
                      • Copyrights13

Reference Configuration

The DS4000 Storage Manager software package configures and manages the IBMDS4800 storage subsystemrsquos fully redundant hardware and high-performance architecture with ease DS4000 Storage Manager allows in-band management from servers to attached storage or remotely through the Ethernet The storage subsystem offers several premium features that add functionality when used in a database environment Two of the most helpful features in an Oracle RAC environment are the Storage Partitioning premium feature and the Enhanced Remote Mirroring premium feature

The Storage Partitioning premium feature allows logical drives on the storage subsystem to be distributed to different hosts in your SAN associating each logical drive with one particular host or a group of hosts The logical drives are presented to each host according to the host type for the different operating systems that are running on those hosts The Storage Partitioning premium feature allows great flexibility in the use of the storage subsystem for storage consolidation

The Enhanced Remote Mirroring premium feature (RM) provides the ability to mirror logical drives between two IBM storage subsystems Mirroring provides an additional redundant copy of the data The mirror copy can be used for rapid disaster recovery and can be also used to create a copy of live data for test purposes

Although these two premium features are not used in the reference configuration there are many opportunities to use storage partitioning and Enhanced Remote Mirroring in a real-world Oracle implementation

For more information please discuss these premium features with your IBM business partner or sales representative

Reference ConfigurationThe reference configuration consists of the following equipment

bull 2 IBM P5 p630 servers

bull AIX 53 Technology Level 6

bull IBM DS4800 storage subsystem

bull Oracle Cluster Ready Services (Oracle CRS) 10201 upgraded to 10203

bull Oracle Database 10201 upgraded to 10203

The systems are configured as a two-node RAC cluster Oracle ASM is employed to simplify storage management The AIX RDAC multipathing driver provided failover support

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 3copy Copyright 2008 IBM Corporation

Introduction

Figure 1 Reference Configuration

4 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

Configuring the HostsThe hosts must be configured to function correctly with Oracle RAC The following steps describe in general terms how to prepare the systems for the Oracle installation

IMPORTANT Refer to Metalink Bulletin 2820361 Minimum Software Versions and Patches Required to Support Oracle Products on IBM Series for the latest requirements This bulletin is available to Metalink users athttpmetalinkoraclecom Metalink Bulletin 2820361 is regularly updated and the information it contains supersedes any information found in this document

1 Install the AIX operating system if it is not already installed The reference configuration uses AIX 53 Technology Level 6

2 Install additional software packages required for Oracle

bosadtbase

bosadtlib

bosadtlibm

bosperflibperfstat

bosperfperfstat

bosperfproctools

rsctbasicrte

rsctcompatclientsrte

xlCaix50rte7004 (See the following note)

xlCrte7001 (See the following note)

NOTE The versions of xlCaix50rte and xlCrte supplied on the AIX 53 distribution media are not compatible with Oracle RAC 10g Release 2 You must download newer versions from IBMs web site The reference configuration uses fileset xlcrteaix50apr2007ptftarZ

3 Download and install the updates to the AIX operating system from IBMs website

httpwww14softwareibmcomwebappset2sasfgenunix3aixfixeshtml

The reference configuration uses AIX 53 Technology Level 6

4 Configure the hosts network interfaces

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 5copy Copyright 2008 IBM Corporation

Configuring the Hosts

Use at least two ethernet interfaces

One for the Oracle public network

One for the private network used by Oracle CRS

Oracle recommends that the private interface be on a separate network isolated from all other traffic You should use a separate gigabit ethernet switch for the private network

Additional interfaces provide more flexibility in configuration and higher performance In the reference configuration the VIP network shares the interface used for the public network This is configured later in the installation process using the vipca utility Entries for the internet protocol (IP) address as well as the public and private networks are inserted into the etchosts file at this time

5 Create users and groups

a Create two groups dba and oinstall

b Add a user Oracle with the oinstall primary group and with the dba group

NOTE Make sure that the user ID for the Oracle user and the group numbers for the dba and oinstall groups are consistent on all nodes

6 Set the shell soft limits for the Oracle and root users using the smit chuser command

Soft FILE size = -1

Soft CPU time = -1

Soft DATA segment = -1

Soft STACK size = -1

7 Set system configuration parameters

a Use the smit chgsys command to set the maximum number of processes per user to at least 2048

b Use the vmo -p -o lru_file_repage=0 command to set the lru_file_repage parameter to 0

NOTE This setting instructs the AIX Virtual Memory Manager to take pages only from the file buffer cache allowing Oraclersquos System Global Area (SGA) to remain in memory for better performance

8 Use the usrsbinno -a command to configure the network tuning parameters

Use these minimum values for network tuning parameters

6 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

ipqmaxlen = 512rfc1323 = 1sb_max = 1310720 (2655360)tcp_recvspace = 65536 tcp_sendspace = 65536 udp_recvspace = 655360 (10 udp_sendspace must be less than sb_max)udp_sendspace = 65536

NOTE If you modify these parameters the method depends on whether the system is in compatibility mode For specific instructions refer to the AIX documentation for the no command

NOTE If you change the value of the ipqmaxlen parameter you must restart the system for the change to take effect

9 Download and install the Open Secure Shell (OpenSSH) utility from the IBM web site

httpwwwibmcomdeveloperworkseserverarticlesopenssh_updatedhtml

NOTE The reference configuration uses OpenSSH The SSH utility is not required for Oracle to function however SSH is recommended because it provides a more secure method of communication between nodes than does RSH If SSH is available Oracle will use it otherwise Oracle uses RSH Configuration of RSH user equivalence is outside the scope of this document

10 Configure SSH for root login permission

a Locate the sshd_config file (typically etc or usrlocaletc)

b Edit the sshd_config file changing PermitRootLogin to yes

c Use the startsrc -s sshd command to start sshd

11 Configure SSH for user equivalence for root and Oracle on all nodes

a Log on as ltrootgt

b Use the following commands to generate the SSH key

cd ~ssh

ssh-keygen -t dsa

cat id_dsapub gtgt authorized_keys

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 7copy Copyright 2008 IBM Corporation

Configuring the Hosts

c Copy the authorized_keys file to ~ssh on the next system

d Repeat step b and step c on each system

e After all systems have a public key in the file copy the file back to all of the previous systems so that they all have an identical file

f Test the SSH function from each system to itself and to each of the other systems by issuing such commands as the ssh host1 date command Make sure that the commands can be executed without using a password

g Log on as the Oracle user and repeat step a through step f so that Oracle can also perform remote commands without passwords

NOTE The permissions must restrict write access on both the base user directory and the ssh directory If the permissions are not correctly set OpenSSH requires a password even if the appropriate public key has been added to the authorized_keys file

12 Edit the profile file in the Oracle userrsquos home directory to set the Oracle users parameters

Table 1 shows the entries that you must add to the profile file

Table 1 Oracle Userrsquos Parameters

Parameter to Add Explanation

umask 022

AIXTHREAD_SCOPE=S

An environment variable that directs the system to place user threads within a system-wide rather than a process-wide contention scope This placement provides better performance and more efficient memory use

CRS_HOME A unique home directory for Oracle CRS binaries

ASM_HOME

Use this parameter if you prefer to install separate Oracle binaries for Oracle ASM This method is preferable if there will be more than one database instance supported by Oracle ASM

ORACLE_HOME A unique home for the database binaries

LD_LIBRARY_PATH The complete pathway is $ORACLE_HOMElib

ORACLE_BASE Identifies the Oracle base directory

ORACLE_SID Identifies the instance

PATH Include $ORACLE_HOMEbin

CLASSPATHInclude $ORACLE_HOMEjdbclibltclasses filesgt

8 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

13 Edit the profile file in the root userrsquos home directory to set the same parameters as set in the Oracle users profile

Be sure to add the following entry to the profile file

AIXTHREAD_SCOPE=S

14 Synchronize the system time between the hosts in the cluster using any effective method The reference configuration used Network Time Protocol (NTP) If an NTP server is available use this method to set up xntpd

a Edit etcntpconf with the nameIP of the NTP server(s)

b Add a line server lthostnamegt for each host

c Start the service using the smit xntpd command

15 Because some Oracle utilities expect lsattr in etc and do not function correctly without a link to it use the ln -s usrsbinlsattr etclsattr command to add a soft link

Configuring the IBM Storage

IMPORTANT You must know how to best configure the storage for enhanced performance For this information refer to Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to wwwibmcomsupporttechdocs

Use DS4000 Storage Manager and the AIX system utilities to configure the IBM storage subsystem

1 Use DS4000 Storage Manager to create logical drives on the storage subsystem to be shared for Oracle CRS communication and database storage

NOTE You also may use additional logical drives for Oracle binaries and similar purposes however for those logical drives on the storage subsystem that are used exclusively by one particular host you will need the Storage Partitioning premium feature

You will need to analyze your implementation to determine the appropriate logical drive sizes desired RAID levels and other parameters The reference configuration uses the following minimal configuration of shared logical drives

2 Oracle Cluster Registry (OCR) logical drives (300 MB RAID1)

3 voting disks (300MB RAID1)

1 logical drive for Oracle ASMs SPFILE (100MB RAID1)

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 9copy Copyright 2008 IBM Corporation

Configuring the IBM Storage

2 logical drives for Oracle ASM data storage (300 GB RAID5)

2 logical drives for Oracle ASM log storage (135GB RAID1)

The number and the size of the logical drives created for Oracle ASM storage varies based on the individual implementation but the sizes used for the OCR disks voting disks and the Oracle ASM SPFILE disks in the reference configuration are sufficient for any configuration

In the reference configuration the Oracle binaries are installed on internal disks within each individual server The reference configuration uses the JFS2 file system and the logical drive was mounted as u01

2 Configure the shared disks in the operating system

After you have created the logical drives on the storage subsystem the cfgmgr command rescans the AIX system for changes to the physical devices

The lspv command shows the physical devices that AIX has configured

The fget_config -Av command shows the LUN numbers the disk drive name and the associated logical drive names configured on the storage subsystem

3 If the hdisk names for each logical drive on the storage subsystem that will be shared in the cluster are not identical on all cluster nodes create new device nodes for each of the logical drives to give them a consistent name across all nodes

For example if the logical drive for the first OCR disk was hdisk2 on node1 and hdisk3 on node2 a new device node (devnode) called devORA_OCR1 could be created on each host using the major numbers and the minor numbers that correspond with the same logical drive on each system Show the major and the minor numbers with the command ls -l Use the command mknod devORA_OCR1 c ltmajor of OCRgt ltminor of OCRgt to create the new devnode

Even when the disk drive names correlate on all of the nodes you should still create these named device nodes for ease of location and administration Named device nodes were created on the reference configuration for all shared logical drives using the convention devORA_ltusegt

4 Set the reserve_policy attribute for each logical drive to be used for shared access for OCR logical drive voting logical drive or Oracle ASM logical drive

NOTE This operation must be performed on the hdisk device name rather than on the named devnode created in step 3

This operation allows the nodes to share the logical drives Otherwise the first system that accesses the logical drive reserves it for exclusive use Oracle coordinates access to the shared logical drives

10 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

For an IBM storage subsystem reserve_policy=no_reserve is the correct value

a Use the lsattr -E -l hdisk | grep reserve command to check the attribute

b Use the chdev -l hdisk -a reserve_policy=no_reserve command to change the policy

c Repeat step a and step b on each cluster node

5 Set ownership and permissions on devnodes to be used by Oracle

NOTE If you created new devnodes for the shared logical drives set the ownership and permissions on the new devnode that you created not on the original hdisk devnode

a Use the following commands to set the ownership for each OCR device file

chown rootoinstall devltdevnodegt

chmod 640 devltdevnodegt

b Use the following commands to set the ownership and permission on the voting disk and database device files

chown oracledba devltdevnodegt

chmod 660 devltdevnodegt

c Repeat step a and step b on each cluster node

NOTE Do not assign a physical volume identifier (PVID) to hdisks that will be used with Oracle ASM

Installing OracleHere are the basic steps for installing Oracle

bull Create directories for Oracle

bull Set up the Oracle userrsquos profile

bull Install Oracle Clusterware

bull Update Oracle CRS

bull Install and configure Oracle ASM

bull Update Oracle ASM

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 11copy Copyright 2008 IBM Corporation

Installing Oracle Clusterware

bull Install Oracle Database

bull Update the database home

bull Configure the listener names

bull Create and configure a database

Creating Directories for OracleAlthough you can create directory structures for Oracle files in many different ways for ease of support use a standard directory structure based on the Optimal Flexible Architecture (OFA) guidelines published by Oracle For more information about the OFA Guidelines see Appendix D of the Oracle Database Installation Guide 10gR2 for AIX 5L Based Systems (B19075-02) The reference configuration is based on the OFA guidelines

Follow these two important guidelines when creating directories for Oracle

bull You must separate the Oracle CRS home directory from the Oracle base and the database home directories The Oracle CRS installation procedure sets the ownership for the entire Oracle CRS directory tree to root while the database directory tree must be owned by oracle

bull If more than one database will be serviced by Oracle ASM it is recommended that you create a separate home directory for Oracle ASM An Oracle ASM home directory provides the capability to modify the Oracle release for Oracle ASM while not changing the release of any of the databases on the same server The directory tree must be owned by oracle

Setting Up the Oracle Users Profile1 Set up the Oracle users profile before installing the Oracle binaries See

Appendix A for an example profile

2 Add or modify entries for ORACLE_HOME ORACLE_BASE PATH LD_LIBRARY_PATH and similar files to reflect the directories where the Oracle binaries will be placed during the Oracle Clusterware installation

Installing Oracle ClusterwareAs you install the Oracle Clusterware remember the following important points

bull Do not make the CRS_HOME path a subdirectory of the Oracle RAC 10g Release 2 Oracle base directory due to permission changes required for Oracle CRS For more details refer to Appendix E of the Oracle Clusterware Installation Guide

bull The rootsh script must be run as the root user and must be run on only one node at a time not concurrently

12 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Updating Oracle CRS

bull After rootsh runs on the last node it will attempt to run the vipca utility to configure the VIP network interfaces If you have IP addresses that are not routable across the Internet for example 100 192168 the vipca configuration will fail If the automatic vipca configuration fails run the vipca utility manually using the $ORA_CRS_HOMEbinvipca command from the command line as root This requires a graphical display

bull You can do the manual configuration with vipca before clicking OK on the Oracle Universal Installer the Oracle CRS installation page The Oracle CRS installation page waits for you to run rootsh Do the manual configuration with vipca during this time This action allows the remainder of the post-installation checks to complete successfully

Updating Oracle CRSRefer to the Metalink Bulletin 2820361 (see important note on page 5) for required updates The reference configuration used the 10203 update to the Oracle CRS home Download the 10203 update and any other updates required from the Oracle Metalink web site (httpmetalinkoraclecom) and install according to the instructions on the web site

Installing and Configuring Oracle ASMBecause a single Oracle ASM instance can service many database instances Oracle recommends that you install Oracle ASM in its own home directory Follow the naming convention of the OFA guidelines for the directory path As you install and configure Oracle ASM remember the following important points

bull If the Oracle ASM instance was not correctly created by the Oracle Universal Installer Wizard during installation use the dbca utility to add the instance and create the disk group(s)

bull If you want additional disk groups create them using the dbca utility

Updating Oracle ASMUpdate the Oracle ASM instance with the 10203 patch set and any additional updates indicated in Metalink Bulletin 2820361 You can use the same update media for 10203 previously downloaded to update Oracle CRS (page 13) Carefully follow the instructions for updating this database instance

Installing an Oracle DatabaseUse the Oracle Universal Installer to install another database home for a database instance Select Install Database Software Only on the Create Database page

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 13copy Copyright 2008 IBM Corporation

Installing Oracle Clusterware

Updating the Database HomeInstall the 10203 update on the new database home Select the database home instead of Oracle ASM on the Specify Home Details page

Configuring Listener NamesIf your RAC environment consists of only Oracle CRS an Oracle ASM instance and a database instance on each node it is easier to support only one listenerora configuration file for the Oracle ASM and database instances The best way to accomplish this is to create an environment variable TNS_ADMIN and assign it to the $ORACLE_HOMEnetworkadmin directory of the database instance For example

export TNS_ADMIN=$ORA_ASM_HOMEnetworkadmin

This environment variable allows the DBA to maintain one set of tnsnamesora sqlnetora and listenerora files for the entire environment

Creating and Configuring a DatabaseUse the Database Configuration Assistant (DBCA) to create a database For more information refer to the Oracle documentation about using DBCA As you create the database remember the following important points

bull Using DBCA gives you the capability to create the Oracle ASM disk groups if you choose to use Oracle ASM

bull The Flash Recovery Area (FRA) option is highly recommended A FRA allows the setup of a separate area on disk for flashback capability area for the archive logs the second copy of the multiplexed redo log files and for RMAN backups Make sure that the FRA is correctly set up to handle the performance of the database and that the FRA contains enough space

Space to hold all of the data for the data retention periods that are set by the database

Space for the type of backups that will be used

Space for the number of backups that will be held

For further documentation about using the FRA refer to the Oracle documentation as well as the following white paper

Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to

wwwibmcomsupporttechdocs

14 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Creating and Configuring a Database

ConclusionAs you have followed the example of setting up a two-node cluster described in this document you have seen how simple and how effective a deployment of Oracle RAC 10g Release 2 can be When you are doing your own installation be sure to use the applicable installation information as the basis for your work This document describes only tips and techniques Your installer should have the expertise to know how and when to modify instructions as necessary If your environment is complex and you require outside help contact your IBM business partner or sales representative See ldquoContact Informationrdquo below

Contact InformationFor more information and sales office locations visit the IBM web site at

httpwwwibmcom

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 15copy Copyright 2008 IBM Corporation

Contact Information

16 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix A

References

AIXFor AIX operating system downloads go to Quick Links for AIX Fixes

httpwww-933ibmcomeserversupportfixesfixcentralmainpseriesaix

For AIX documentation go to AIX Information

httppublibboulderibmcominfocentersystemsindexjsptopic=comibmaixdocdocbaseaixparenthtm

IBM Corporation 2006 Get the Latest Version of OpenSSH for AIX

httpwwwibmcomdeveloperworkseserverarticlesopenssh_updatedhtml

IBM StorageFor information about the IBM DS4800 Storage Subsystem go to

httpwww-03ibmcomsystemsstoragediskds4000ds4800indexhtml

For information about IBM storage go to

httpwww-03ibmcomsystemsstoragediskcm_re=masthead-_-products-_-stg-disk

OracleFor Oracle documentation go to

httpwwworaclecomtechnologydocumentationindexhtml

IBM Corporation 2008 Tuning IBM AIX 5L for an Oracle Database

httpwwwibmcompartnerworldwpsservletContentHandlerVPAA-6V2C4E

IBM Corporation 2008 Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems

wwwibmcomsupporttechdocs

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem A-1copy Copyright 2008 IBM Corporation

A-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix B

Troubleshooting

CSS Node Evictions can affect your functionality

IO time-outs can cause inappropriate node evictions during multi-path failover or other problems Two parameters might affect this css_misscount and disktimeout With versions of Oracle prior to 102 this was primarily accommodated by increasing the value of the css_misscount parameter However the function of this parameter has changed as of Oracle 102 and changes to this parameter might not achieve the desired result The new disktimeout parameter should be used in most cases to make this adjustment In many cases the default values will work fine However you might find it necessary to increase the default value of this parameter if you are getting a lot of CSS node evictions

For information about how the parameters work and how to adjust them refer to the following two articles Go to httpmetalinkoraclecom

bull Metalink DocID 2944301 CSS Timeout Computation in RAC 10g

bull Metalink DocID 2847521 Steps to Increase CSS Misscount Reboottime and Disktimeout

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem B-1copy Copyright 2008 IBM Corporation

B-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Trademarks and special notices copy Copyright IBM Corporation 2008 All rights reserved

References in this document to IBM products or services do not imply that IBM intends to make them available in every country

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corporation in the United States other countries or both If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (reg or trade) these symbols indicate US registered or common law trademarks owned by IBM at the time this information was published Such trademarks may also be registered or common law trademarks in other countries A current list of IBM trademarks is available on the Web at Copyright and trademark information at wwwibmcomlegalcopytradeshtml

Intel Intel Inside (logos) MMX and Pentium are trademarks of Intel Corporation in the United States other countries or both

UNIX is a registered trademark of The Open Group in the United States and other countries

Linux is a trademark of Linus Torvalds in the United States other countries or both

Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both

Other company product or service names may be trademarks or service marks of others

Information is provided AS IS without warranty of any kind

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved Actual environmental costs and performance characteristics may vary by customer

Information concerning non-IBM products was obtained from a supplier of these products published announcement material or other publicly available sources and does not constitute an endorsement of such products by IBM Sources for non-IBM list prices and performance numbers are taken from publicly available information including vendor announcements and vendor worldwide homepages IBM has not tested these products and cannot confirm the accuracy of performance capability or any other claims related to non-IBM products Questions on the capability of non-IBM products should be addressed to the supplier of those products

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk

  • Title Page13
  • Contents
  • Deploying Oracle RAC 10g Release 2 with ASM on AIX 53 with an IBM DS4700 or DS4800 Storage Subsystem
    • Introduction
      • Required Documents
      • Benefits
      • Reference Configuration
        • Configuring the Hosts
        • Configuring the IBM Storage
        • Installing Oracle
          • Creating Directories for Oracle
          • Setting Up the Oracle Users Profile
            • Installing Oracle Clusterware
              • Updating Oracle CRS
              • Installing and Configuring Oracle ASM
              • Updating Oracle ASM
              • Installing an Oracle Database
              • Updating the Database Home
              • Configuring Listener Names
              • Creating and Configuring a Database
                • Conclusion
                • Contact Information
                  • Appendix A References
                    • AIX
                    • IBM Storage
                    • Oracle
                      • Appendix B Troubleshooting
                      • Copyrights13

Introduction

Figure 1 Reference Configuration

4 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

Configuring the HostsThe hosts must be configured to function correctly with Oracle RAC The following steps describe in general terms how to prepare the systems for the Oracle installation

IMPORTANT Refer to Metalink Bulletin 2820361 Minimum Software Versions and Patches Required to Support Oracle Products on IBM Series for the latest requirements This bulletin is available to Metalink users athttpmetalinkoraclecom Metalink Bulletin 2820361 is regularly updated and the information it contains supersedes any information found in this document

1 Install the AIX operating system if it is not already installed The reference configuration uses AIX 53 Technology Level 6

2 Install additional software packages required for Oracle

bosadtbase

bosadtlib

bosadtlibm

bosperflibperfstat

bosperfperfstat

bosperfproctools

rsctbasicrte

rsctcompatclientsrte

xlCaix50rte7004 (See the following note)

xlCrte7001 (See the following note)

NOTE The versions of xlCaix50rte and xlCrte supplied on the AIX 53 distribution media are not compatible with Oracle RAC 10g Release 2 You must download newer versions from IBMs web site The reference configuration uses fileset xlcrteaix50apr2007ptftarZ

3 Download and install the updates to the AIX operating system from IBMs website

httpwww14softwareibmcomwebappset2sasfgenunix3aixfixeshtml

The reference configuration uses AIX 53 Technology Level 6

4 Configure the hosts network interfaces

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 5copy Copyright 2008 IBM Corporation

Configuring the Hosts

Use at least two ethernet interfaces

One for the Oracle public network

One for the private network used by Oracle CRS

Oracle recommends that the private interface be on a separate network isolated from all other traffic You should use a separate gigabit ethernet switch for the private network

Additional interfaces provide more flexibility in configuration and higher performance In the reference configuration the VIP network shares the interface used for the public network This is configured later in the installation process using the vipca utility Entries for the internet protocol (IP) address as well as the public and private networks are inserted into the etchosts file at this time

5 Create users and groups

a Create two groups dba and oinstall

b Add a user Oracle with the oinstall primary group and with the dba group

NOTE Make sure that the user ID for the Oracle user and the group numbers for the dba and oinstall groups are consistent on all nodes

6 Set the shell soft limits for the Oracle and root users using the smit chuser command

Soft FILE size = -1

Soft CPU time = -1

Soft DATA segment = -1

Soft STACK size = -1

7 Set system configuration parameters

a Use the smit chgsys command to set the maximum number of processes per user to at least 2048

b Use the vmo -p -o lru_file_repage=0 command to set the lru_file_repage parameter to 0

NOTE This setting instructs the AIX Virtual Memory Manager to take pages only from the file buffer cache allowing Oraclersquos System Global Area (SGA) to remain in memory for better performance

8 Use the usrsbinno -a command to configure the network tuning parameters

Use these minimum values for network tuning parameters

6 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

ipqmaxlen = 512rfc1323 = 1sb_max = 1310720 (2655360)tcp_recvspace = 65536 tcp_sendspace = 65536 udp_recvspace = 655360 (10 udp_sendspace must be less than sb_max)udp_sendspace = 65536

NOTE If you modify these parameters the method depends on whether the system is in compatibility mode For specific instructions refer to the AIX documentation for the no command

NOTE If you change the value of the ipqmaxlen parameter you must restart the system for the change to take effect

9 Download and install the Open Secure Shell (OpenSSH) utility from the IBM web site

httpwwwibmcomdeveloperworkseserverarticlesopenssh_updatedhtml

NOTE The reference configuration uses OpenSSH The SSH utility is not required for Oracle to function however SSH is recommended because it provides a more secure method of communication between nodes than does RSH If SSH is available Oracle will use it otherwise Oracle uses RSH Configuration of RSH user equivalence is outside the scope of this document

10 Configure SSH for root login permission

a Locate the sshd_config file (typically etc or usrlocaletc)

b Edit the sshd_config file changing PermitRootLogin to yes

c Use the startsrc -s sshd command to start sshd

11 Configure SSH for user equivalence for root and Oracle on all nodes

a Log on as ltrootgt

b Use the following commands to generate the SSH key

cd ~ssh

ssh-keygen -t dsa

cat id_dsapub gtgt authorized_keys

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 7copy Copyright 2008 IBM Corporation

Configuring the Hosts

c Copy the authorized_keys file to ~ssh on the next system

d Repeat step b and step c on each system

e After all systems have a public key in the file copy the file back to all of the previous systems so that they all have an identical file

f Test the SSH function from each system to itself and to each of the other systems by issuing such commands as the ssh host1 date command Make sure that the commands can be executed without using a password

g Log on as the Oracle user and repeat step a through step f so that Oracle can also perform remote commands without passwords

NOTE The permissions must restrict write access on both the base user directory and the ssh directory If the permissions are not correctly set OpenSSH requires a password even if the appropriate public key has been added to the authorized_keys file

12 Edit the profile file in the Oracle userrsquos home directory to set the Oracle users parameters

Table 1 shows the entries that you must add to the profile file

Table 1 Oracle Userrsquos Parameters

Parameter to Add Explanation

umask 022

AIXTHREAD_SCOPE=S

An environment variable that directs the system to place user threads within a system-wide rather than a process-wide contention scope This placement provides better performance and more efficient memory use

CRS_HOME A unique home directory for Oracle CRS binaries

ASM_HOME

Use this parameter if you prefer to install separate Oracle binaries for Oracle ASM This method is preferable if there will be more than one database instance supported by Oracle ASM

ORACLE_HOME A unique home for the database binaries

LD_LIBRARY_PATH The complete pathway is $ORACLE_HOMElib

ORACLE_BASE Identifies the Oracle base directory

ORACLE_SID Identifies the instance

PATH Include $ORACLE_HOMEbin

CLASSPATHInclude $ORACLE_HOMEjdbclibltclasses filesgt

8 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

13 Edit the profile file in the root userrsquos home directory to set the same parameters as set in the Oracle users profile

Be sure to add the following entry to the profile file

AIXTHREAD_SCOPE=S

14 Synchronize the system time between the hosts in the cluster using any effective method The reference configuration used Network Time Protocol (NTP) If an NTP server is available use this method to set up xntpd

a Edit etcntpconf with the nameIP of the NTP server(s)

b Add a line server lthostnamegt for each host

c Start the service using the smit xntpd command

15 Because some Oracle utilities expect lsattr in etc and do not function correctly without a link to it use the ln -s usrsbinlsattr etclsattr command to add a soft link

Configuring the IBM Storage

IMPORTANT You must know how to best configure the storage for enhanced performance For this information refer to Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to wwwibmcomsupporttechdocs

Use DS4000 Storage Manager and the AIX system utilities to configure the IBM storage subsystem

1 Use DS4000 Storage Manager to create logical drives on the storage subsystem to be shared for Oracle CRS communication and database storage

NOTE You also may use additional logical drives for Oracle binaries and similar purposes however for those logical drives on the storage subsystem that are used exclusively by one particular host you will need the Storage Partitioning premium feature

You will need to analyze your implementation to determine the appropriate logical drive sizes desired RAID levels and other parameters The reference configuration uses the following minimal configuration of shared logical drives

2 Oracle Cluster Registry (OCR) logical drives (300 MB RAID1)

3 voting disks (300MB RAID1)

1 logical drive for Oracle ASMs SPFILE (100MB RAID1)

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 9copy Copyright 2008 IBM Corporation

Configuring the IBM Storage

2 logical drives for Oracle ASM data storage (300 GB RAID5)

2 logical drives for Oracle ASM log storage (135GB RAID1)

The number and the size of the logical drives created for Oracle ASM storage varies based on the individual implementation but the sizes used for the OCR disks voting disks and the Oracle ASM SPFILE disks in the reference configuration are sufficient for any configuration

In the reference configuration the Oracle binaries are installed on internal disks within each individual server The reference configuration uses the JFS2 file system and the logical drive was mounted as u01

2 Configure the shared disks in the operating system

After you have created the logical drives on the storage subsystem the cfgmgr command rescans the AIX system for changes to the physical devices

The lspv command shows the physical devices that AIX has configured

The fget_config -Av command shows the LUN numbers the disk drive name and the associated logical drive names configured on the storage subsystem

3 If the hdisk names for each logical drive on the storage subsystem that will be shared in the cluster are not identical on all cluster nodes create new device nodes for each of the logical drives to give them a consistent name across all nodes

For example if the logical drive for the first OCR disk was hdisk2 on node1 and hdisk3 on node2 a new device node (devnode) called devORA_OCR1 could be created on each host using the major numbers and the minor numbers that correspond with the same logical drive on each system Show the major and the minor numbers with the command ls -l Use the command mknod devORA_OCR1 c ltmajor of OCRgt ltminor of OCRgt to create the new devnode

Even when the disk drive names correlate on all of the nodes you should still create these named device nodes for ease of location and administration Named device nodes were created on the reference configuration for all shared logical drives using the convention devORA_ltusegt

4 Set the reserve_policy attribute for each logical drive to be used for shared access for OCR logical drive voting logical drive or Oracle ASM logical drive

NOTE This operation must be performed on the hdisk device name rather than on the named devnode created in step 3

This operation allows the nodes to share the logical drives Otherwise the first system that accesses the logical drive reserves it for exclusive use Oracle coordinates access to the shared logical drives

10 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

For an IBM storage subsystem reserve_policy=no_reserve is the correct value

a Use the lsattr -E -l hdisk | grep reserve command to check the attribute

b Use the chdev -l hdisk -a reserve_policy=no_reserve command to change the policy

c Repeat step a and step b on each cluster node

5 Set ownership and permissions on devnodes to be used by Oracle

NOTE If you created new devnodes for the shared logical drives set the ownership and permissions on the new devnode that you created not on the original hdisk devnode

a Use the following commands to set the ownership for each OCR device file

chown rootoinstall devltdevnodegt

chmod 640 devltdevnodegt

b Use the following commands to set the ownership and permission on the voting disk and database device files

chown oracledba devltdevnodegt

chmod 660 devltdevnodegt

c Repeat step a and step b on each cluster node

NOTE Do not assign a physical volume identifier (PVID) to hdisks that will be used with Oracle ASM

Installing OracleHere are the basic steps for installing Oracle

bull Create directories for Oracle

bull Set up the Oracle userrsquos profile

bull Install Oracle Clusterware

bull Update Oracle CRS

bull Install and configure Oracle ASM

bull Update Oracle ASM

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 11copy Copyright 2008 IBM Corporation

Installing Oracle Clusterware

bull Install Oracle Database

bull Update the database home

bull Configure the listener names

bull Create and configure a database

Creating Directories for OracleAlthough you can create directory structures for Oracle files in many different ways for ease of support use a standard directory structure based on the Optimal Flexible Architecture (OFA) guidelines published by Oracle For more information about the OFA Guidelines see Appendix D of the Oracle Database Installation Guide 10gR2 for AIX 5L Based Systems (B19075-02) The reference configuration is based on the OFA guidelines

Follow these two important guidelines when creating directories for Oracle

bull You must separate the Oracle CRS home directory from the Oracle base and the database home directories The Oracle CRS installation procedure sets the ownership for the entire Oracle CRS directory tree to root while the database directory tree must be owned by oracle

bull If more than one database will be serviced by Oracle ASM it is recommended that you create a separate home directory for Oracle ASM An Oracle ASM home directory provides the capability to modify the Oracle release for Oracle ASM while not changing the release of any of the databases on the same server The directory tree must be owned by oracle

Setting Up the Oracle Users Profile1 Set up the Oracle users profile before installing the Oracle binaries See

Appendix A for an example profile

2 Add or modify entries for ORACLE_HOME ORACLE_BASE PATH LD_LIBRARY_PATH and similar files to reflect the directories where the Oracle binaries will be placed during the Oracle Clusterware installation

Installing Oracle ClusterwareAs you install the Oracle Clusterware remember the following important points

bull Do not make the CRS_HOME path a subdirectory of the Oracle RAC 10g Release 2 Oracle base directory due to permission changes required for Oracle CRS For more details refer to Appendix E of the Oracle Clusterware Installation Guide

bull The rootsh script must be run as the root user and must be run on only one node at a time not concurrently

12 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Updating Oracle CRS

bull After rootsh runs on the last node it will attempt to run the vipca utility to configure the VIP network interfaces If you have IP addresses that are not routable across the Internet for example 100 192168 the vipca configuration will fail If the automatic vipca configuration fails run the vipca utility manually using the $ORA_CRS_HOMEbinvipca command from the command line as root This requires a graphical display

bull You can do the manual configuration with vipca before clicking OK on the Oracle Universal Installer the Oracle CRS installation page The Oracle CRS installation page waits for you to run rootsh Do the manual configuration with vipca during this time This action allows the remainder of the post-installation checks to complete successfully

Updating Oracle CRSRefer to the Metalink Bulletin 2820361 (see important note on page 5) for required updates The reference configuration used the 10203 update to the Oracle CRS home Download the 10203 update and any other updates required from the Oracle Metalink web site (httpmetalinkoraclecom) and install according to the instructions on the web site

Installing and Configuring Oracle ASMBecause a single Oracle ASM instance can service many database instances Oracle recommends that you install Oracle ASM in its own home directory Follow the naming convention of the OFA guidelines for the directory path As you install and configure Oracle ASM remember the following important points

bull If the Oracle ASM instance was not correctly created by the Oracle Universal Installer Wizard during installation use the dbca utility to add the instance and create the disk group(s)

bull If you want additional disk groups create them using the dbca utility

Updating Oracle ASMUpdate the Oracle ASM instance with the 10203 patch set and any additional updates indicated in Metalink Bulletin 2820361 You can use the same update media for 10203 previously downloaded to update Oracle CRS (page 13) Carefully follow the instructions for updating this database instance

Installing an Oracle DatabaseUse the Oracle Universal Installer to install another database home for a database instance Select Install Database Software Only on the Create Database page

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 13copy Copyright 2008 IBM Corporation

Installing Oracle Clusterware

Updating the Database HomeInstall the 10203 update on the new database home Select the database home instead of Oracle ASM on the Specify Home Details page

Configuring Listener NamesIf your RAC environment consists of only Oracle CRS an Oracle ASM instance and a database instance on each node it is easier to support only one listenerora configuration file for the Oracle ASM and database instances The best way to accomplish this is to create an environment variable TNS_ADMIN and assign it to the $ORACLE_HOMEnetworkadmin directory of the database instance For example

export TNS_ADMIN=$ORA_ASM_HOMEnetworkadmin

This environment variable allows the DBA to maintain one set of tnsnamesora sqlnetora and listenerora files for the entire environment

Creating and Configuring a DatabaseUse the Database Configuration Assistant (DBCA) to create a database For more information refer to the Oracle documentation about using DBCA As you create the database remember the following important points

bull Using DBCA gives you the capability to create the Oracle ASM disk groups if you choose to use Oracle ASM

bull The Flash Recovery Area (FRA) option is highly recommended A FRA allows the setup of a separate area on disk for flashback capability area for the archive logs the second copy of the multiplexed redo log files and for RMAN backups Make sure that the FRA is correctly set up to handle the performance of the database and that the FRA contains enough space

Space to hold all of the data for the data retention periods that are set by the database

Space for the type of backups that will be used

Space for the number of backups that will be held

For further documentation about using the FRA refer to the Oracle documentation as well as the following white paper

Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to

wwwibmcomsupporttechdocs

14 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Creating and Configuring a Database

ConclusionAs you have followed the example of setting up a two-node cluster described in this document you have seen how simple and how effective a deployment of Oracle RAC 10g Release 2 can be When you are doing your own installation be sure to use the applicable installation information as the basis for your work This document describes only tips and techniques Your installer should have the expertise to know how and when to modify instructions as necessary If your environment is complex and you require outside help contact your IBM business partner or sales representative See ldquoContact Informationrdquo below

Contact InformationFor more information and sales office locations visit the IBM web site at

httpwwwibmcom

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 15copy Copyright 2008 IBM Corporation

Contact Information

16 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix A

References

AIXFor AIX operating system downloads go to Quick Links for AIX Fixes

httpwww-933ibmcomeserversupportfixesfixcentralmainpseriesaix

For AIX documentation go to AIX Information

httppublibboulderibmcominfocentersystemsindexjsptopic=comibmaixdocdocbaseaixparenthtm

IBM Corporation 2006 Get the Latest Version of OpenSSH for AIX

httpwwwibmcomdeveloperworkseserverarticlesopenssh_updatedhtml

IBM StorageFor information about the IBM DS4800 Storage Subsystem go to

httpwww-03ibmcomsystemsstoragediskds4000ds4800indexhtml

For information about IBM storage go to

httpwww-03ibmcomsystemsstoragediskcm_re=masthead-_-products-_-stg-disk

OracleFor Oracle documentation go to

httpwwworaclecomtechnologydocumentationindexhtml

IBM Corporation 2008 Tuning IBM AIX 5L for an Oracle Database

httpwwwibmcompartnerworldwpsservletContentHandlerVPAA-6V2C4E

IBM Corporation 2008 Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems

wwwibmcomsupporttechdocs

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem A-1copy Copyright 2008 IBM Corporation

A-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix B

Troubleshooting

CSS Node Evictions can affect your functionality

IO time-outs can cause inappropriate node evictions during multi-path failover or other problems Two parameters might affect this css_misscount and disktimeout With versions of Oracle prior to 102 this was primarily accommodated by increasing the value of the css_misscount parameter However the function of this parameter has changed as of Oracle 102 and changes to this parameter might not achieve the desired result The new disktimeout parameter should be used in most cases to make this adjustment In many cases the default values will work fine However you might find it necessary to increase the default value of this parameter if you are getting a lot of CSS node evictions

For information about how the parameters work and how to adjust them refer to the following two articles Go to httpmetalinkoraclecom

bull Metalink DocID 2944301 CSS Timeout Computation in RAC 10g

bull Metalink DocID 2847521 Steps to Increase CSS Misscount Reboottime and Disktimeout

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem B-1copy Copyright 2008 IBM Corporation

B-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Trademarks and special notices copy Copyright IBM Corporation 2008 All rights reserved

References in this document to IBM products or services do not imply that IBM intends to make them available in every country

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corporation in the United States other countries or both If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (reg or trade) these symbols indicate US registered or common law trademarks owned by IBM at the time this information was published Such trademarks may also be registered or common law trademarks in other countries A current list of IBM trademarks is available on the Web at Copyright and trademark information at wwwibmcomlegalcopytradeshtml

Intel Intel Inside (logos) MMX and Pentium are trademarks of Intel Corporation in the United States other countries or both

UNIX is a registered trademark of The Open Group in the United States and other countries

Linux is a trademark of Linus Torvalds in the United States other countries or both

Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both

Other company product or service names may be trademarks or service marks of others

Information is provided AS IS without warranty of any kind

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved Actual environmental costs and performance characteristics may vary by customer

Information concerning non-IBM products was obtained from a supplier of these products published announcement material or other publicly available sources and does not constitute an endorsement of such products by IBM Sources for non-IBM list prices and performance numbers are taken from publicly available information including vendor announcements and vendor worldwide homepages IBM has not tested these products and cannot confirm the accuracy of performance capability or any other claims related to non-IBM products Questions on the capability of non-IBM products should be addressed to the supplier of those products

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk

  • Title Page13
  • Contents
  • Deploying Oracle RAC 10g Release 2 with ASM on AIX 53 with an IBM DS4700 or DS4800 Storage Subsystem
    • Introduction
      • Required Documents
      • Benefits
      • Reference Configuration
        • Configuring the Hosts
        • Configuring the IBM Storage
        • Installing Oracle
          • Creating Directories for Oracle
          • Setting Up the Oracle Users Profile
            • Installing Oracle Clusterware
              • Updating Oracle CRS
              • Installing and Configuring Oracle ASM
              • Updating Oracle ASM
              • Installing an Oracle Database
              • Updating the Database Home
              • Configuring Listener Names
              • Creating and Configuring a Database
                • Conclusion
                • Contact Information
                  • Appendix A References
                    • AIX
                    • IBM Storage
                    • Oracle
                      • Appendix B Troubleshooting
                      • Copyrights13

Reference Configuration

Configuring the HostsThe hosts must be configured to function correctly with Oracle RAC The following steps describe in general terms how to prepare the systems for the Oracle installation

IMPORTANT Refer to Metalink Bulletin 2820361 Minimum Software Versions and Patches Required to Support Oracle Products on IBM Series for the latest requirements This bulletin is available to Metalink users athttpmetalinkoraclecom Metalink Bulletin 2820361 is regularly updated and the information it contains supersedes any information found in this document

1 Install the AIX operating system if it is not already installed The reference configuration uses AIX 53 Technology Level 6

2 Install additional software packages required for Oracle

bosadtbase

bosadtlib

bosadtlibm

bosperflibperfstat

bosperfperfstat

bosperfproctools

rsctbasicrte

rsctcompatclientsrte

xlCaix50rte7004 (See the following note)

xlCrte7001 (See the following note)

NOTE The versions of xlCaix50rte and xlCrte supplied on the AIX 53 distribution media are not compatible with Oracle RAC 10g Release 2 You must download newer versions from IBMs web site The reference configuration uses fileset xlcrteaix50apr2007ptftarZ

3 Download and install the updates to the AIX operating system from IBMs website

httpwww14softwareibmcomwebappset2sasfgenunix3aixfixeshtml

The reference configuration uses AIX 53 Technology Level 6

4 Configure the hosts network interfaces

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 5copy Copyright 2008 IBM Corporation

Configuring the Hosts

Use at least two ethernet interfaces

One for the Oracle public network

One for the private network used by Oracle CRS

Oracle recommends that the private interface be on a separate network isolated from all other traffic You should use a separate gigabit ethernet switch for the private network

Additional interfaces provide more flexibility in configuration and higher performance In the reference configuration the VIP network shares the interface used for the public network This is configured later in the installation process using the vipca utility Entries for the internet protocol (IP) address as well as the public and private networks are inserted into the etchosts file at this time

5 Create users and groups

a Create two groups dba and oinstall

b Add a user Oracle with the oinstall primary group and with the dba group

NOTE Make sure that the user ID for the Oracle user and the group numbers for the dba and oinstall groups are consistent on all nodes

6 Set the shell soft limits for the Oracle and root users using the smit chuser command

Soft FILE size = -1

Soft CPU time = -1

Soft DATA segment = -1

Soft STACK size = -1

7 Set system configuration parameters

a Use the smit chgsys command to set the maximum number of processes per user to at least 2048

b Use the vmo -p -o lru_file_repage=0 command to set the lru_file_repage parameter to 0

NOTE This setting instructs the AIX Virtual Memory Manager to take pages only from the file buffer cache allowing Oraclersquos System Global Area (SGA) to remain in memory for better performance

8 Use the usrsbinno -a command to configure the network tuning parameters

Use these minimum values for network tuning parameters

6 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

ipqmaxlen = 512rfc1323 = 1sb_max = 1310720 (2655360)tcp_recvspace = 65536 tcp_sendspace = 65536 udp_recvspace = 655360 (10 udp_sendspace must be less than sb_max)udp_sendspace = 65536

NOTE If you modify these parameters the method depends on whether the system is in compatibility mode For specific instructions refer to the AIX documentation for the no command

NOTE If you change the value of the ipqmaxlen parameter you must restart the system for the change to take effect

9 Download and install the Open Secure Shell (OpenSSH) utility from the IBM web site

httpwwwibmcomdeveloperworkseserverarticlesopenssh_updatedhtml

NOTE The reference configuration uses OpenSSH The SSH utility is not required for Oracle to function however SSH is recommended because it provides a more secure method of communication between nodes than does RSH If SSH is available Oracle will use it otherwise Oracle uses RSH Configuration of RSH user equivalence is outside the scope of this document

10 Configure SSH for root login permission

a Locate the sshd_config file (typically etc or usrlocaletc)

b Edit the sshd_config file changing PermitRootLogin to yes

c Use the startsrc -s sshd command to start sshd

11 Configure SSH for user equivalence for root and Oracle on all nodes

a Log on as ltrootgt

b Use the following commands to generate the SSH key

cd ~ssh

ssh-keygen -t dsa

cat id_dsapub gtgt authorized_keys

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 7copy Copyright 2008 IBM Corporation

Configuring the Hosts

c Copy the authorized_keys file to ~ssh on the next system

d Repeat step b and step c on each system

e After all systems have a public key in the file copy the file back to all of the previous systems so that they all have an identical file

f Test the SSH function from each system to itself and to each of the other systems by issuing such commands as the ssh host1 date command Make sure that the commands can be executed without using a password

g Log on as the Oracle user and repeat step a through step f so that Oracle can also perform remote commands without passwords

NOTE The permissions must restrict write access on both the base user directory and the ssh directory If the permissions are not correctly set OpenSSH requires a password even if the appropriate public key has been added to the authorized_keys file

12 Edit the profile file in the Oracle userrsquos home directory to set the Oracle users parameters

Table 1 shows the entries that you must add to the profile file

Table 1 Oracle Userrsquos Parameters

Parameter to Add Explanation

umask 022

AIXTHREAD_SCOPE=S

An environment variable that directs the system to place user threads within a system-wide rather than a process-wide contention scope This placement provides better performance and more efficient memory use

CRS_HOME A unique home directory for Oracle CRS binaries

ASM_HOME

Use this parameter if you prefer to install separate Oracle binaries for Oracle ASM This method is preferable if there will be more than one database instance supported by Oracle ASM

ORACLE_HOME A unique home for the database binaries

LD_LIBRARY_PATH The complete pathway is $ORACLE_HOMElib

ORACLE_BASE Identifies the Oracle base directory

ORACLE_SID Identifies the instance

PATH Include $ORACLE_HOMEbin

CLASSPATHInclude $ORACLE_HOMEjdbclibltclasses filesgt

8 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

13 Edit the profile file in the root userrsquos home directory to set the same parameters as set in the Oracle users profile

Be sure to add the following entry to the profile file

AIXTHREAD_SCOPE=S

14 Synchronize the system time between the hosts in the cluster using any effective method The reference configuration used Network Time Protocol (NTP) If an NTP server is available use this method to set up xntpd

a Edit etcntpconf with the nameIP of the NTP server(s)

b Add a line server lthostnamegt for each host

c Start the service using the smit xntpd command

15 Because some Oracle utilities expect lsattr in etc and do not function correctly without a link to it use the ln -s usrsbinlsattr etclsattr command to add a soft link

Configuring the IBM Storage

IMPORTANT You must know how to best configure the storage for enhanced performance For this information refer to Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to wwwibmcomsupporttechdocs

Use DS4000 Storage Manager and the AIX system utilities to configure the IBM storage subsystem

1 Use DS4000 Storage Manager to create logical drives on the storage subsystem to be shared for Oracle CRS communication and database storage

NOTE You also may use additional logical drives for Oracle binaries and similar purposes however for those logical drives on the storage subsystem that are used exclusively by one particular host you will need the Storage Partitioning premium feature

You will need to analyze your implementation to determine the appropriate logical drive sizes desired RAID levels and other parameters The reference configuration uses the following minimal configuration of shared logical drives

2 Oracle Cluster Registry (OCR) logical drives (300 MB RAID1)

3 voting disks (300MB RAID1)

1 logical drive for Oracle ASMs SPFILE (100MB RAID1)

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 9copy Copyright 2008 IBM Corporation

Configuring the IBM Storage

2 logical drives for Oracle ASM data storage (300 GB RAID5)

2 logical drives for Oracle ASM log storage (135GB RAID1)

The number and the size of the logical drives created for Oracle ASM storage varies based on the individual implementation but the sizes used for the OCR disks voting disks and the Oracle ASM SPFILE disks in the reference configuration are sufficient for any configuration

In the reference configuration the Oracle binaries are installed on internal disks within each individual server The reference configuration uses the JFS2 file system and the logical drive was mounted as u01

2 Configure the shared disks in the operating system

After you have created the logical drives on the storage subsystem the cfgmgr command rescans the AIX system for changes to the physical devices

The lspv command shows the physical devices that AIX has configured

The fget_config -Av command shows the LUN numbers the disk drive name and the associated logical drive names configured on the storage subsystem

3 If the hdisk names for each logical drive on the storage subsystem that will be shared in the cluster are not identical on all cluster nodes create new device nodes for each of the logical drives to give them a consistent name across all nodes

For example if the logical drive for the first OCR disk was hdisk2 on node1 and hdisk3 on node2 a new device node (devnode) called devORA_OCR1 could be created on each host using the major numbers and the minor numbers that correspond with the same logical drive on each system Show the major and the minor numbers with the command ls -l Use the command mknod devORA_OCR1 c ltmajor of OCRgt ltminor of OCRgt to create the new devnode

Even when the disk drive names correlate on all of the nodes you should still create these named device nodes for ease of location and administration Named device nodes were created on the reference configuration for all shared logical drives using the convention devORA_ltusegt

4 Set the reserve_policy attribute for each logical drive to be used for shared access for OCR logical drive voting logical drive or Oracle ASM logical drive

NOTE This operation must be performed on the hdisk device name rather than on the named devnode created in step 3

This operation allows the nodes to share the logical drives Otherwise the first system that accesses the logical drive reserves it for exclusive use Oracle coordinates access to the shared logical drives

10 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

For an IBM storage subsystem reserve_policy=no_reserve is the correct value

a Use the lsattr -E -l hdisk | grep reserve command to check the attribute

b Use the chdev -l hdisk -a reserve_policy=no_reserve command to change the policy

c Repeat step a and step b on each cluster node

5 Set ownership and permissions on devnodes to be used by Oracle

NOTE If you created new devnodes for the shared logical drives set the ownership and permissions on the new devnode that you created not on the original hdisk devnode

a Use the following commands to set the ownership for each OCR device file

chown rootoinstall devltdevnodegt

chmod 640 devltdevnodegt

b Use the following commands to set the ownership and permission on the voting disk and database device files

chown oracledba devltdevnodegt

chmod 660 devltdevnodegt

c Repeat step a and step b on each cluster node

NOTE Do not assign a physical volume identifier (PVID) to hdisks that will be used with Oracle ASM

Installing OracleHere are the basic steps for installing Oracle

bull Create directories for Oracle

bull Set up the Oracle userrsquos profile

bull Install Oracle Clusterware

bull Update Oracle CRS

bull Install and configure Oracle ASM

bull Update Oracle ASM

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 11copy Copyright 2008 IBM Corporation

Installing Oracle Clusterware

bull Install Oracle Database

bull Update the database home

bull Configure the listener names

bull Create and configure a database

Creating Directories for OracleAlthough you can create directory structures for Oracle files in many different ways for ease of support use a standard directory structure based on the Optimal Flexible Architecture (OFA) guidelines published by Oracle For more information about the OFA Guidelines see Appendix D of the Oracle Database Installation Guide 10gR2 for AIX 5L Based Systems (B19075-02) The reference configuration is based on the OFA guidelines

Follow these two important guidelines when creating directories for Oracle

bull You must separate the Oracle CRS home directory from the Oracle base and the database home directories The Oracle CRS installation procedure sets the ownership for the entire Oracle CRS directory tree to root while the database directory tree must be owned by oracle

bull If more than one database will be serviced by Oracle ASM it is recommended that you create a separate home directory for Oracle ASM An Oracle ASM home directory provides the capability to modify the Oracle release for Oracle ASM while not changing the release of any of the databases on the same server The directory tree must be owned by oracle

Setting Up the Oracle Users Profile1 Set up the Oracle users profile before installing the Oracle binaries See

Appendix A for an example profile

2 Add or modify entries for ORACLE_HOME ORACLE_BASE PATH LD_LIBRARY_PATH and similar files to reflect the directories where the Oracle binaries will be placed during the Oracle Clusterware installation

Installing Oracle ClusterwareAs you install the Oracle Clusterware remember the following important points

bull Do not make the CRS_HOME path a subdirectory of the Oracle RAC 10g Release 2 Oracle base directory due to permission changes required for Oracle CRS For more details refer to Appendix E of the Oracle Clusterware Installation Guide

bull The rootsh script must be run as the root user and must be run on only one node at a time not concurrently

12 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Updating Oracle CRS

bull After rootsh runs on the last node it will attempt to run the vipca utility to configure the VIP network interfaces If you have IP addresses that are not routable across the Internet for example 100 192168 the vipca configuration will fail If the automatic vipca configuration fails run the vipca utility manually using the $ORA_CRS_HOMEbinvipca command from the command line as root This requires a graphical display

bull You can do the manual configuration with vipca before clicking OK on the Oracle Universal Installer the Oracle CRS installation page The Oracle CRS installation page waits for you to run rootsh Do the manual configuration with vipca during this time This action allows the remainder of the post-installation checks to complete successfully

Updating Oracle CRSRefer to the Metalink Bulletin 2820361 (see important note on page 5) for required updates The reference configuration used the 10203 update to the Oracle CRS home Download the 10203 update and any other updates required from the Oracle Metalink web site (httpmetalinkoraclecom) and install according to the instructions on the web site

Installing and Configuring Oracle ASMBecause a single Oracle ASM instance can service many database instances Oracle recommends that you install Oracle ASM in its own home directory Follow the naming convention of the OFA guidelines for the directory path As you install and configure Oracle ASM remember the following important points

bull If the Oracle ASM instance was not correctly created by the Oracle Universal Installer Wizard during installation use the dbca utility to add the instance and create the disk group(s)

bull If you want additional disk groups create them using the dbca utility

Updating Oracle ASMUpdate the Oracle ASM instance with the 10203 patch set and any additional updates indicated in Metalink Bulletin 2820361 You can use the same update media for 10203 previously downloaded to update Oracle CRS (page 13) Carefully follow the instructions for updating this database instance

Installing an Oracle DatabaseUse the Oracle Universal Installer to install another database home for a database instance Select Install Database Software Only on the Create Database page

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 13copy Copyright 2008 IBM Corporation

Installing Oracle Clusterware

Updating the Database HomeInstall the 10203 update on the new database home Select the database home instead of Oracle ASM on the Specify Home Details page

Configuring Listener NamesIf your RAC environment consists of only Oracle CRS an Oracle ASM instance and a database instance on each node it is easier to support only one listenerora configuration file for the Oracle ASM and database instances The best way to accomplish this is to create an environment variable TNS_ADMIN and assign it to the $ORACLE_HOMEnetworkadmin directory of the database instance For example

export TNS_ADMIN=$ORA_ASM_HOMEnetworkadmin

This environment variable allows the DBA to maintain one set of tnsnamesora sqlnetora and listenerora files for the entire environment

Creating and Configuring a DatabaseUse the Database Configuration Assistant (DBCA) to create a database For more information refer to the Oracle documentation about using DBCA As you create the database remember the following important points

bull Using DBCA gives you the capability to create the Oracle ASM disk groups if you choose to use Oracle ASM

bull The Flash Recovery Area (FRA) option is highly recommended A FRA allows the setup of a separate area on disk for flashback capability area for the archive logs the second copy of the multiplexed redo log files and for RMAN backups Make sure that the FRA is correctly set up to handle the performance of the database and that the FRA contains enough space

Space to hold all of the data for the data retention periods that are set by the database

Space for the type of backups that will be used

Space for the number of backups that will be held

For further documentation about using the FRA refer to the Oracle documentation as well as the following white paper

Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to

wwwibmcomsupporttechdocs

14 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Creating and Configuring a Database

ConclusionAs you have followed the example of setting up a two-node cluster described in this document you have seen how simple and how effective a deployment of Oracle RAC 10g Release 2 can be When you are doing your own installation be sure to use the applicable installation information as the basis for your work This document describes only tips and techniques Your installer should have the expertise to know how and when to modify instructions as necessary If your environment is complex and you require outside help contact your IBM business partner or sales representative See ldquoContact Informationrdquo below

Contact InformationFor more information and sales office locations visit the IBM web site at

httpwwwibmcom

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 15copy Copyright 2008 IBM Corporation

Contact Information

16 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix A

References

AIXFor AIX operating system downloads go to Quick Links for AIX Fixes

httpwww-933ibmcomeserversupportfixesfixcentralmainpseriesaix

For AIX documentation go to AIX Information

httppublibboulderibmcominfocentersystemsindexjsptopic=comibmaixdocdocbaseaixparenthtm

IBM Corporation 2006 Get the Latest Version of OpenSSH for AIX

httpwwwibmcomdeveloperworkseserverarticlesopenssh_updatedhtml

IBM StorageFor information about the IBM DS4800 Storage Subsystem go to

httpwww-03ibmcomsystemsstoragediskds4000ds4800indexhtml

For information about IBM storage go to

httpwww-03ibmcomsystemsstoragediskcm_re=masthead-_-products-_-stg-disk

OracleFor Oracle documentation go to

httpwwworaclecomtechnologydocumentationindexhtml

IBM Corporation 2008 Tuning IBM AIX 5L for an Oracle Database

httpwwwibmcompartnerworldwpsservletContentHandlerVPAA-6V2C4E

IBM Corporation 2008 Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems

wwwibmcomsupporttechdocs

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem A-1copy Copyright 2008 IBM Corporation

A-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix B

Troubleshooting

CSS Node Evictions can affect your functionality

IO time-outs can cause inappropriate node evictions during multi-path failover or other problems Two parameters might affect this css_misscount and disktimeout With versions of Oracle prior to 102 this was primarily accommodated by increasing the value of the css_misscount parameter However the function of this parameter has changed as of Oracle 102 and changes to this parameter might not achieve the desired result The new disktimeout parameter should be used in most cases to make this adjustment In many cases the default values will work fine However you might find it necessary to increase the default value of this parameter if you are getting a lot of CSS node evictions

For information about how the parameters work and how to adjust them refer to the following two articles Go to httpmetalinkoraclecom

bull Metalink DocID 2944301 CSS Timeout Computation in RAC 10g

bull Metalink DocID 2847521 Steps to Increase CSS Misscount Reboottime and Disktimeout

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem B-1copy Copyright 2008 IBM Corporation

B-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Trademarks and special notices copy Copyright IBM Corporation 2008 All rights reserved

References in this document to IBM products or services do not imply that IBM intends to make them available in every country

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corporation in the United States other countries or both If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (reg or trade) these symbols indicate US registered or common law trademarks owned by IBM at the time this information was published Such trademarks may also be registered or common law trademarks in other countries A current list of IBM trademarks is available on the Web at Copyright and trademark information at wwwibmcomlegalcopytradeshtml

Intel Intel Inside (logos) MMX and Pentium are trademarks of Intel Corporation in the United States other countries or both

UNIX is a registered trademark of The Open Group in the United States and other countries

Linux is a trademark of Linus Torvalds in the United States other countries or both

Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both

Other company product or service names may be trademarks or service marks of others

Information is provided AS IS without warranty of any kind

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved Actual environmental costs and performance characteristics may vary by customer

Information concerning non-IBM products was obtained from a supplier of these products published announcement material or other publicly available sources and does not constitute an endorsement of such products by IBM Sources for non-IBM list prices and performance numbers are taken from publicly available information including vendor announcements and vendor worldwide homepages IBM has not tested these products and cannot confirm the accuracy of performance capability or any other claims related to non-IBM products Questions on the capability of non-IBM products should be addressed to the supplier of those products

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk

  • Title Page13
  • Contents
  • Deploying Oracle RAC 10g Release 2 with ASM on AIX 53 with an IBM DS4700 or DS4800 Storage Subsystem
    • Introduction
      • Required Documents
      • Benefits
      • Reference Configuration
        • Configuring the Hosts
        • Configuring the IBM Storage
        • Installing Oracle
          • Creating Directories for Oracle
          • Setting Up the Oracle Users Profile
            • Installing Oracle Clusterware
              • Updating Oracle CRS
              • Installing and Configuring Oracle ASM
              • Updating Oracle ASM
              • Installing an Oracle Database
              • Updating the Database Home
              • Configuring Listener Names
              • Creating and Configuring a Database
                • Conclusion
                • Contact Information
                  • Appendix A References
                    • AIX
                    • IBM Storage
                    • Oracle
                      • Appendix B Troubleshooting
                      • Copyrights13

Configuring the Hosts

Use at least two ethernet interfaces

One for the Oracle public network

One for the private network used by Oracle CRS

Oracle recommends that the private interface be on a separate network isolated from all other traffic You should use a separate gigabit ethernet switch for the private network

Additional interfaces provide more flexibility in configuration and higher performance In the reference configuration the VIP network shares the interface used for the public network This is configured later in the installation process using the vipca utility Entries for the internet protocol (IP) address as well as the public and private networks are inserted into the etchosts file at this time

5 Create users and groups

a Create two groups dba and oinstall

b Add a user Oracle with the oinstall primary group and with the dba group

NOTE Make sure that the user ID for the Oracle user and the group numbers for the dba and oinstall groups are consistent on all nodes

6 Set the shell soft limits for the Oracle and root users using the smit chuser command

Soft FILE size = -1

Soft CPU time = -1

Soft DATA segment = -1

Soft STACK size = -1

7 Set system configuration parameters

a Use the smit chgsys command to set the maximum number of processes per user to at least 2048

b Use the vmo -p -o lru_file_repage=0 command to set the lru_file_repage parameter to 0

NOTE This setting instructs the AIX Virtual Memory Manager to take pages only from the file buffer cache allowing Oraclersquos System Global Area (SGA) to remain in memory for better performance

8 Use the usrsbinno -a command to configure the network tuning parameters

Use these minimum values for network tuning parameters

6 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

ipqmaxlen = 512rfc1323 = 1sb_max = 1310720 (2655360)tcp_recvspace = 65536 tcp_sendspace = 65536 udp_recvspace = 655360 (10 udp_sendspace must be less than sb_max)udp_sendspace = 65536

NOTE If you modify these parameters the method depends on whether the system is in compatibility mode For specific instructions refer to the AIX documentation for the no command

NOTE If you change the value of the ipqmaxlen parameter you must restart the system for the change to take effect

9 Download and install the Open Secure Shell (OpenSSH) utility from the IBM web site

httpwwwibmcomdeveloperworkseserverarticlesopenssh_updatedhtml

NOTE The reference configuration uses OpenSSH The SSH utility is not required for Oracle to function however SSH is recommended because it provides a more secure method of communication between nodes than does RSH If SSH is available Oracle will use it otherwise Oracle uses RSH Configuration of RSH user equivalence is outside the scope of this document

10 Configure SSH for root login permission

a Locate the sshd_config file (typically etc or usrlocaletc)

b Edit the sshd_config file changing PermitRootLogin to yes

c Use the startsrc -s sshd command to start sshd

11 Configure SSH for user equivalence for root and Oracle on all nodes

a Log on as ltrootgt

b Use the following commands to generate the SSH key

cd ~ssh

ssh-keygen -t dsa

cat id_dsapub gtgt authorized_keys

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 7copy Copyright 2008 IBM Corporation

Configuring the Hosts

c Copy the authorized_keys file to ~ssh on the next system

d Repeat step b and step c on each system

e After all systems have a public key in the file copy the file back to all of the previous systems so that they all have an identical file

f Test the SSH function from each system to itself and to each of the other systems by issuing such commands as the ssh host1 date command Make sure that the commands can be executed without using a password

g Log on as the Oracle user and repeat step a through step f so that Oracle can also perform remote commands without passwords

NOTE The permissions must restrict write access on both the base user directory and the ssh directory If the permissions are not correctly set OpenSSH requires a password even if the appropriate public key has been added to the authorized_keys file

12 Edit the profile file in the Oracle userrsquos home directory to set the Oracle users parameters

Table 1 shows the entries that you must add to the profile file

Table 1 Oracle Userrsquos Parameters

Parameter to Add Explanation

umask 022

AIXTHREAD_SCOPE=S

An environment variable that directs the system to place user threads within a system-wide rather than a process-wide contention scope This placement provides better performance and more efficient memory use

CRS_HOME A unique home directory for Oracle CRS binaries

ASM_HOME

Use this parameter if you prefer to install separate Oracle binaries for Oracle ASM This method is preferable if there will be more than one database instance supported by Oracle ASM

ORACLE_HOME A unique home for the database binaries

LD_LIBRARY_PATH The complete pathway is $ORACLE_HOMElib

ORACLE_BASE Identifies the Oracle base directory

ORACLE_SID Identifies the instance

PATH Include $ORACLE_HOMEbin

CLASSPATHInclude $ORACLE_HOMEjdbclibltclasses filesgt

8 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

13 Edit the profile file in the root userrsquos home directory to set the same parameters as set in the Oracle users profile

Be sure to add the following entry to the profile file

AIXTHREAD_SCOPE=S

14 Synchronize the system time between the hosts in the cluster using any effective method The reference configuration used Network Time Protocol (NTP) If an NTP server is available use this method to set up xntpd

a Edit etcntpconf with the nameIP of the NTP server(s)

b Add a line server lthostnamegt for each host

c Start the service using the smit xntpd command

15 Because some Oracle utilities expect lsattr in etc and do not function correctly without a link to it use the ln -s usrsbinlsattr etclsattr command to add a soft link

Configuring the IBM Storage

IMPORTANT You must know how to best configure the storage for enhanced performance For this information refer to Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to wwwibmcomsupporttechdocs

Use DS4000 Storage Manager and the AIX system utilities to configure the IBM storage subsystem

1 Use DS4000 Storage Manager to create logical drives on the storage subsystem to be shared for Oracle CRS communication and database storage

NOTE You also may use additional logical drives for Oracle binaries and similar purposes however for those logical drives on the storage subsystem that are used exclusively by one particular host you will need the Storage Partitioning premium feature

You will need to analyze your implementation to determine the appropriate logical drive sizes desired RAID levels and other parameters The reference configuration uses the following minimal configuration of shared logical drives

2 Oracle Cluster Registry (OCR) logical drives (300 MB RAID1)

3 voting disks (300MB RAID1)

1 logical drive for Oracle ASMs SPFILE (100MB RAID1)

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 9copy Copyright 2008 IBM Corporation

Configuring the IBM Storage

2 logical drives for Oracle ASM data storage (300 GB RAID5)

2 logical drives for Oracle ASM log storage (135GB RAID1)

The number and the size of the logical drives created for Oracle ASM storage varies based on the individual implementation but the sizes used for the OCR disks voting disks and the Oracle ASM SPFILE disks in the reference configuration are sufficient for any configuration

In the reference configuration the Oracle binaries are installed on internal disks within each individual server The reference configuration uses the JFS2 file system and the logical drive was mounted as u01

2 Configure the shared disks in the operating system

After you have created the logical drives on the storage subsystem the cfgmgr command rescans the AIX system for changes to the physical devices

The lspv command shows the physical devices that AIX has configured

The fget_config -Av command shows the LUN numbers the disk drive name and the associated logical drive names configured on the storage subsystem

3 If the hdisk names for each logical drive on the storage subsystem that will be shared in the cluster are not identical on all cluster nodes create new device nodes for each of the logical drives to give them a consistent name across all nodes

For example if the logical drive for the first OCR disk was hdisk2 on node1 and hdisk3 on node2 a new device node (devnode) called devORA_OCR1 could be created on each host using the major numbers and the minor numbers that correspond with the same logical drive on each system Show the major and the minor numbers with the command ls -l Use the command mknod devORA_OCR1 c ltmajor of OCRgt ltminor of OCRgt to create the new devnode

Even when the disk drive names correlate on all of the nodes you should still create these named device nodes for ease of location and administration Named device nodes were created on the reference configuration for all shared logical drives using the convention devORA_ltusegt

4 Set the reserve_policy attribute for each logical drive to be used for shared access for OCR logical drive voting logical drive or Oracle ASM logical drive

NOTE This operation must be performed on the hdisk device name rather than on the named devnode created in step 3

This operation allows the nodes to share the logical drives Otherwise the first system that accesses the logical drive reserves it for exclusive use Oracle coordinates access to the shared logical drives

10 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

For an IBM storage subsystem reserve_policy=no_reserve is the correct value

a Use the lsattr -E -l hdisk | grep reserve command to check the attribute

b Use the chdev -l hdisk -a reserve_policy=no_reserve command to change the policy

c Repeat step a and step b on each cluster node

5 Set ownership and permissions on devnodes to be used by Oracle

NOTE If you created new devnodes for the shared logical drives set the ownership and permissions on the new devnode that you created not on the original hdisk devnode

a Use the following commands to set the ownership for each OCR device file

chown rootoinstall devltdevnodegt

chmod 640 devltdevnodegt

b Use the following commands to set the ownership and permission on the voting disk and database device files

chown oracledba devltdevnodegt

chmod 660 devltdevnodegt

c Repeat step a and step b on each cluster node

NOTE Do not assign a physical volume identifier (PVID) to hdisks that will be used with Oracle ASM

Installing OracleHere are the basic steps for installing Oracle

bull Create directories for Oracle

bull Set up the Oracle userrsquos profile

bull Install Oracle Clusterware

bull Update Oracle CRS

bull Install and configure Oracle ASM

bull Update Oracle ASM

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 11copy Copyright 2008 IBM Corporation

Installing Oracle Clusterware

bull Install Oracle Database

bull Update the database home

bull Configure the listener names

bull Create and configure a database

Creating Directories for OracleAlthough you can create directory structures for Oracle files in many different ways for ease of support use a standard directory structure based on the Optimal Flexible Architecture (OFA) guidelines published by Oracle For more information about the OFA Guidelines see Appendix D of the Oracle Database Installation Guide 10gR2 for AIX 5L Based Systems (B19075-02) The reference configuration is based on the OFA guidelines

Follow these two important guidelines when creating directories for Oracle

bull You must separate the Oracle CRS home directory from the Oracle base and the database home directories The Oracle CRS installation procedure sets the ownership for the entire Oracle CRS directory tree to root while the database directory tree must be owned by oracle

bull If more than one database will be serviced by Oracle ASM it is recommended that you create a separate home directory for Oracle ASM An Oracle ASM home directory provides the capability to modify the Oracle release for Oracle ASM while not changing the release of any of the databases on the same server The directory tree must be owned by oracle

Setting Up the Oracle Users Profile1 Set up the Oracle users profile before installing the Oracle binaries See

Appendix A for an example profile

2 Add or modify entries for ORACLE_HOME ORACLE_BASE PATH LD_LIBRARY_PATH and similar files to reflect the directories where the Oracle binaries will be placed during the Oracle Clusterware installation

Installing Oracle ClusterwareAs you install the Oracle Clusterware remember the following important points

bull Do not make the CRS_HOME path a subdirectory of the Oracle RAC 10g Release 2 Oracle base directory due to permission changes required for Oracle CRS For more details refer to Appendix E of the Oracle Clusterware Installation Guide

bull The rootsh script must be run as the root user and must be run on only one node at a time not concurrently

12 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Updating Oracle CRS

bull After rootsh runs on the last node it will attempt to run the vipca utility to configure the VIP network interfaces If you have IP addresses that are not routable across the Internet for example 100 192168 the vipca configuration will fail If the automatic vipca configuration fails run the vipca utility manually using the $ORA_CRS_HOMEbinvipca command from the command line as root This requires a graphical display

bull You can do the manual configuration with vipca before clicking OK on the Oracle Universal Installer the Oracle CRS installation page The Oracle CRS installation page waits for you to run rootsh Do the manual configuration with vipca during this time This action allows the remainder of the post-installation checks to complete successfully

Updating Oracle CRSRefer to the Metalink Bulletin 2820361 (see important note on page 5) for required updates The reference configuration used the 10203 update to the Oracle CRS home Download the 10203 update and any other updates required from the Oracle Metalink web site (httpmetalinkoraclecom) and install according to the instructions on the web site

Installing and Configuring Oracle ASMBecause a single Oracle ASM instance can service many database instances Oracle recommends that you install Oracle ASM in its own home directory Follow the naming convention of the OFA guidelines for the directory path As you install and configure Oracle ASM remember the following important points

bull If the Oracle ASM instance was not correctly created by the Oracle Universal Installer Wizard during installation use the dbca utility to add the instance and create the disk group(s)

bull If you want additional disk groups create them using the dbca utility

Updating Oracle ASMUpdate the Oracle ASM instance with the 10203 patch set and any additional updates indicated in Metalink Bulletin 2820361 You can use the same update media for 10203 previously downloaded to update Oracle CRS (page 13) Carefully follow the instructions for updating this database instance

Installing an Oracle DatabaseUse the Oracle Universal Installer to install another database home for a database instance Select Install Database Software Only on the Create Database page

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 13copy Copyright 2008 IBM Corporation

Installing Oracle Clusterware

Updating the Database HomeInstall the 10203 update on the new database home Select the database home instead of Oracle ASM on the Specify Home Details page

Configuring Listener NamesIf your RAC environment consists of only Oracle CRS an Oracle ASM instance and a database instance on each node it is easier to support only one listenerora configuration file for the Oracle ASM and database instances The best way to accomplish this is to create an environment variable TNS_ADMIN and assign it to the $ORACLE_HOMEnetworkadmin directory of the database instance For example

export TNS_ADMIN=$ORA_ASM_HOMEnetworkadmin

This environment variable allows the DBA to maintain one set of tnsnamesora sqlnetora and listenerora files for the entire environment

Creating and Configuring a DatabaseUse the Database Configuration Assistant (DBCA) to create a database For more information refer to the Oracle documentation about using DBCA As you create the database remember the following important points

bull Using DBCA gives you the capability to create the Oracle ASM disk groups if you choose to use Oracle ASM

bull The Flash Recovery Area (FRA) option is highly recommended A FRA allows the setup of a separate area on disk for flashback capability area for the archive logs the second copy of the multiplexed redo log files and for RMAN backups Make sure that the FRA is correctly set up to handle the performance of the database and that the FRA contains enough space

Space to hold all of the data for the data retention periods that are set by the database

Space for the type of backups that will be used

Space for the number of backups that will be held

For further documentation about using the FRA refer to the Oracle documentation as well as the following white paper

Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to

wwwibmcomsupporttechdocs

14 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Creating and Configuring a Database

ConclusionAs you have followed the example of setting up a two-node cluster described in this document you have seen how simple and how effective a deployment of Oracle RAC 10g Release 2 can be When you are doing your own installation be sure to use the applicable installation information as the basis for your work This document describes only tips and techniques Your installer should have the expertise to know how and when to modify instructions as necessary If your environment is complex and you require outside help contact your IBM business partner or sales representative See ldquoContact Informationrdquo below

Contact InformationFor more information and sales office locations visit the IBM web site at

httpwwwibmcom

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 15copy Copyright 2008 IBM Corporation

Contact Information

16 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix A

References

AIXFor AIX operating system downloads go to Quick Links for AIX Fixes

httpwww-933ibmcomeserversupportfixesfixcentralmainpseriesaix

For AIX documentation go to AIX Information

httppublibboulderibmcominfocentersystemsindexjsptopic=comibmaixdocdocbaseaixparenthtm

IBM Corporation 2006 Get the Latest Version of OpenSSH for AIX

httpwwwibmcomdeveloperworkseserverarticlesopenssh_updatedhtml

IBM StorageFor information about the IBM DS4800 Storage Subsystem go to

httpwww-03ibmcomsystemsstoragediskds4000ds4800indexhtml

For information about IBM storage go to

httpwww-03ibmcomsystemsstoragediskcm_re=masthead-_-products-_-stg-disk

OracleFor Oracle documentation go to

httpwwworaclecomtechnologydocumentationindexhtml

IBM Corporation 2008 Tuning IBM AIX 5L for an Oracle Database

httpwwwibmcompartnerworldwpsservletContentHandlerVPAA-6V2C4E

IBM Corporation 2008 Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems

wwwibmcomsupporttechdocs

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem A-1copy Copyright 2008 IBM Corporation

A-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix B

Troubleshooting

CSS Node Evictions can affect your functionality

IO time-outs can cause inappropriate node evictions during multi-path failover or other problems Two parameters might affect this css_misscount and disktimeout With versions of Oracle prior to 102 this was primarily accommodated by increasing the value of the css_misscount parameter However the function of this parameter has changed as of Oracle 102 and changes to this parameter might not achieve the desired result The new disktimeout parameter should be used in most cases to make this adjustment In many cases the default values will work fine However you might find it necessary to increase the default value of this parameter if you are getting a lot of CSS node evictions

For information about how the parameters work and how to adjust them refer to the following two articles Go to httpmetalinkoraclecom

bull Metalink DocID 2944301 CSS Timeout Computation in RAC 10g

bull Metalink DocID 2847521 Steps to Increase CSS Misscount Reboottime and Disktimeout

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem B-1copy Copyright 2008 IBM Corporation

B-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Trademarks and special notices copy Copyright IBM Corporation 2008 All rights reserved

References in this document to IBM products or services do not imply that IBM intends to make them available in every country

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corporation in the United States other countries or both If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (reg or trade) these symbols indicate US registered or common law trademarks owned by IBM at the time this information was published Such trademarks may also be registered or common law trademarks in other countries A current list of IBM trademarks is available on the Web at Copyright and trademark information at wwwibmcomlegalcopytradeshtml

Intel Intel Inside (logos) MMX and Pentium are trademarks of Intel Corporation in the United States other countries or both

UNIX is a registered trademark of The Open Group in the United States and other countries

Linux is a trademark of Linus Torvalds in the United States other countries or both

Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both

Other company product or service names may be trademarks or service marks of others

Information is provided AS IS without warranty of any kind

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved Actual environmental costs and performance characteristics may vary by customer

Information concerning non-IBM products was obtained from a supplier of these products published announcement material or other publicly available sources and does not constitute an endorsement of such products by IBM Sources for non-IBM list prices and performance numbers are taken from publicly available information including vendor announcements and vendor worldwide homepages IBM has not tested these products and cannot confirm the accuracy of performance capability or any other claims related to non-IBM products Questions on the capability of non-IBM products should be addressed to the supplier of those products

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk

  • Title Page13
  • Contents
  • Deploying Oracle RAC 10g Release 2 with ASM on AIX 53 with an IBM DS4700 or DS4800 Storage Subsystem
    • Introduction
      • Required Documents
      • Benefits
      • Reference Configuration
        • Configuring the Hosts
        • Configuring the IBM Storage
        • Installing Oracle
          • Creating Directories for Oracle
          • Setting Up the Oracle Users Profile
            • Installing Oracle Clusterware
              • Updating Oracle CRS
              • Installing and Configuring Oracle ASM
              • Updating Oracle ASM
              • Installing an Oracle Database
              • Updating the Database Home
              • Configuring Listener Names
              • Creating and Configuring a Database
                • Conclusion
                • Contact Information
                  • Appendix A References
                    • AIX
                    • IBM Storage
                    • Oracle
                      • Appendix B Troubleshooting
                      • Copyrights13

Reference Configuration

ipqmaxlen = 512rfc1323 = 1sb_max = 1310720 (2655360)tcp_recvspace = 65536 tcp_sendspace = 65536 udp_recvspace = 655360 (10 udp_sendspace must be less than sb_max)udp_sendspace = 65536

NOTE If you modify these parameters the method depends on whether the system is in compatibility mode For specific instructions refer to the AIX documentation for the no command

NOTE If you change the value of the ipqmaxlen parameter you must restart the system for the change to take effect

9 Download and install the Open Secure Shell (OpenSSH) utility from the IBM web site

httpwwwibmcomdeveloperworkseserverarticlesopenssh_updatedhtml

NOTE The reference configuration uses OpenSSH The SSH utility is not required for Oracle to function however SSH is recommended because it provides a more secure method of communication between nodes than does RSH If SSH is available Oracle will use it otherwise Oracle uses RSH Configuration of RSH user equivalence is outside the scope of this document

10 Configure SSH for root login permission

a Locate the sshd_config file (typically etc or usrlocaletc)

b Edit the sshd_config file changing PermitRootLogin to yes

c Use the startsrc -s sshd command to start sshd

11 Configure SSH for user equivalence for root and Oracle on all nodes

a Log on as ltrootgt

b Use the following commands to generate the SSH key

cd ~ssh

ssh-keygen -t dsa

cat id_dsapub gtgt authorized_keys

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 7copy Copyright 2008 IBM Corporation

Configuring the Hosts

c Copy the authorized_keys file to ~ssh on the next system

d Repeat step b and step c on each system

e After all systems have a public key in the file copy the file back to all of the previous systems so that they all have an identical file

f Test the SSH function from each system to itself and to each of the other systems by issuing such commands as the ssh host1 date command Make sure that the commands can be executed without using a password

g Log on as the Oracle user and repeat step a through step f so that Oracle can also perform remote commands without passwords

NOTE The permissions must restrict write access on both the base user directory and the ssh directory If the permissions are not correctly set OpenSSH requires a password even if the appropriate public key has been added to the authorized_keys file

12 Edit the profile file in the Oracle userrsquos home directory to set the Oracle users parameters

Table 1 shows the entries that you must add to the profile file

Table 1 Oracle Userrsquos Parameters

Parameter to Add Explanation

umask 022

AIXTHREAD_SCOPE=S

An environment variable that directs the system to place user threads within a system-wide rather than a process-wide contention scope This placement provides better performance and more efficient memory use

CRS_HOME A unique home directory for Oracle CRS binaries

ASM_HOME

Use this parameter if you prefer to install separate Oracle binaries for Oracle ASM This method is preferable if there will be more than one database instance supported by Oracle ASM

ORACLE_HOME A unique home for the database binaries

LD_LIBRARY_PATH The complete pathway is $ORACLE_HOMElib

ORACLE_BASE Identifies the Oracle base directory

ORACLE_SID Identifies the instance

PATH Include $ORACLE_HOMEbin

CLASSPATHInclude $ORACLE_HOMEjdbclibltclasses filesgt

8 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

13 Edit the profile file in the root userrsquos home directory to set the same parameters as set in the Oracle users profile

Be sure to add the following entry to the profile file

AIXTHREAD_SCOPE=S

14 Synchronize the system time between the hosts in the cluster using any effective method The reference configuration used Network Time Protocol (NTP) If an NTP server is available use this method to set up xntpd

a Edit etcntpconf with the nameIP of the NTP server(s)

b Add a line server lthostnamegt for each host

c Start the service using the smit xntpd command

15 Because some Oracle utilities expect lsattr in etc and do not function correctly without a link to it use the ln -s usrsbinlsattr etclsattr command to add a soft link

Configuring the IBM Storage

IMPORTANT You must know how to best configure the storage for enhanced performance For this information refer to Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to wwwibmcomsupporttechdocs

Use DS4000 Storage Manager and the AIX system utilities to configure the IBM storage subsystem

1 Use DS4000 Storage Manager to create logical drives on the storage subsystem to be shared for Oracle CRS communication and database storage

NOTE You also may use additional logical drives for Oracle binaries and similar purposes however for those logical drives on the storage subsystem that are used exclusively by one particular host you will need the Storage Partitioning premium feature

You will need to analyze your implementation to determine the appropriate logical drive sizes desired RAID levels and other parameters The reference configuration uses the following minimal configuration of shared logical drives

2 Oracle Cluster Registry (OCR) logical drives (300 MB RAID1)

3 voting disks (300MB RAID1)

1 logical drive for Oracle ASMs SPFILE (100MB RAID1)

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 9copy Copyright 2008 IBM Corporation

Configuring the IBM Storage

2 logical drives for Oracle ASM data storage (300 GB RAID5)

2 logical drives for Oracle ASM log storage (135GB RAID1)

The number and the size of the logical drives created for Oracle ASM storage varies based on the individual implementation but the sizes used for the OCR disks voting disks and the Oracle ASM SPFILE disks in the reference configuration are sufficient for any configuration

In the reference configuration the Oracle binaries are installed on internal disks within each individual server The reference configuration uses the JFS2 file system and the logical drive was mounted as u01

2 Configure the shared disks in the operating system

After you have created the logical drives on the storage subsystem the cfgmgr command rescans the AIX system for changes to the physical devices

The lspv command shows the physical devices that AIX has configured

The fget_config -Av command shows the LUN numbers the disk drive name and the associated logical drive names configured on the storage subsystem

3 If the hdisk names for each logical drive on the storage subsystem that will be shared in the cluster are not identical on all cluster nodes create new device nodes for each of the logical drives to give them a consistent name across all nodes

For example if the logical drive for the first OCR disk was hdisk2 on node1 and hdisk3 on node2 a new device node (devnode) called devORA_OCR1 could be created on each host using the major numbers and the minor numbers that correspond with the same logical drive on each system Show the major and the minor numbers with the command ls -l Use the command mknod devORA_OCR1 c ltmajor of OCRgt ltminor of OCRgt to create the new devnode

Even when the disk drive names correlate on all of the nodes you should still create these named device nodes for ease of location and administration Named device nodes were created on the reference configuration for all shared logical drives using the convention devORA_ltusegt

4 Set the reserve_policy attribute for each logical drive to be used for shared access for OCR logical drive voting logical drive or Oracle ASM logical drive

NOTE This operation must be performed on the hdisk device name rather than on the named devnode created in step 3

This operation allows the nodes to share the logical drives Otherwise the first system that accesses the logical drive reserves it for exclusive use Oracle coordinates access to the shared logical drives

10 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

For an IBM storage subsystem reserve_policy=no_reserve is the correct value

a Use the lsattr -E -l hdisk | grep reserve command to check the attribute

b Use the chdev -l hdisk -a reserve_policy=no_reserve command to change the policy

c Repeat step a and step b on each cluster node

5 Set ownership and permissions on devnodes to be used by Oracle

NOTE If you created new devnodes for the shared logical drives set the ownership and permissions on the new devnode that you created not on the original hdisk devnode

a Use the following commands to set the ownership for each OCR device file

chown rootoinstall devltdevnodegt

chmod 640 devltdevnodegt

b Use the following commands to set the ownership and permission on the voting disk and database device files

chown oracledba devltdevnodegt

chmod 660 devltdevnodegt

c Repeat step a and step b on each cluster node

NOTE Do not assign a physical volume identifier (PVID) to hdisks that will be used with Oracle ASM

Installing OracleHere are the basic steps for installing Oracle

bull Create directories for Oracle

bull Set up the Oracle userrsquos profile

bull Install Oracle Clusterware

bull Update Oracle CRS

bull Install and configure Oracle ASM

bull Update Oracle ASM

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 11copy Copyright 2008 IBM Corporation

Installing Oracle Clusterware

bull Install Oracle Database

bull Update the database home

bull Configure the listener names

bull Create and configure a database

Creating Directories for OracleAlthough you can create directory structures for Oracle files in many different ways for ease of support use a standard directory structure based on the Optimal Flexible Architecture (OFA) guidelines published by Oracle For more information about the OFA Guidelines see Appendix D of the Oracle Database Installation Guide 10gR2 for AIX 5L Based Systems (B19075-02) The reference configuration is based on the OFA guidelines

Follow these two important guidelines when creating directories for Oracle

bull You must separate the Oracle CRS home directory from the Oracle base and the database home directories The Oracle CRS installation procedure sets the ownership for the entire Oracle CRS directory tree to root while the database directory tree must be owned by oracle

bull If more than one database will be serviced by Oracle ASM it is recommended that you create a separate home directory for Oracle ASM An Oracle ASM home directory provides the capability to modify the Oracle release for Oracle ASM while not changing the release of any of the databases on the same server The directory tree must be owned by oracle

Setting Up the Oracle Users Profile1 Set up the Oracle users profile before installing the Oracle binaries See

Appendix A for an example profile

2 Add or modify entries for ORACLE_HOME ORACLE_BASE PATH LD_LIBRARY_PATH and similar files to reflect the directories where the Oracle binaries will be placed during the Oracle Clusterware installation

Installing Oracle ClusterwareAs you install the Oracle Clusterware remember the following important points

bull Do not make the CRS_HOME path a subdirectory of the Oracle RAC 10g Release 2 Oracle base directory due to permission changes required for Oracle CRS For more details refer to Appendix E of the Oracle Clusterware Installation Guide

bull The rootsh script must be run as the root user and must be run on only one node at a time not concurrently

12 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Updating Oracle CRS

bull After rootsh runs on the last node it will attempt to run the vipca utility to configure the VIP network interfaces If you have IP addresses that are not routable across the Internet for example 100 192168 the vipca configuration will fail If the automatic vipca configuration fails run the vipca utility manually using the $ORA_CRS_HOMEbinvipca command from the command line as root This requires a graphical display

bull You can do the manual configuration with vipca before clicking OK on the Oracle Universal Installer the Oracle CRS installation page The Oracle CRS installation page waits for you to run rootsh Do the manual configuration with vipca during this time This action allows the remainder of the post-installation checks to complete successfully

Updating Oracle CRSRefer to the Metalink Bulletin 2820361 (see important note on page 5) for required updates The reference configuration used the 10203 update to the Oracle CRS home Download the 10203 update and any other updates required from the Oracle Metalink web site (httpmetalinkoraclecom) and install according to the instructions on the web site

Installing and Configuring Oracle ASMBecause a single Oracle ASM instance can service many database instances Oracle recommends that you install Oracle ASM in its own home directory Follow the naming convention of the OFA guidelines for the directory path As you install and configure Oracle ASM remember the following important points

bull If the Oracle ASM instance was not correctly created by the Oracle Universal Installer Wizard during installation use the dbca utility to add the instance and create the disk group(s)

bull If you want additional disk groups create them using the dbca utility

Updating Oracle ASMUpdate the Oracle ASM instance with the 10203 patch set and any additional updates indicated in Metalink Bulletin 2820361 You can use the same update media for 10203 previously downloaded to update Oracle CRS (page 13) Carefully follow the instructions for updating this database instance

Installing an Oracle DatabaseUse the Oracle Universal Installer to install another database home for a database instance Select Install Database Software Only on the Create Database page

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 13copy Copyright 2008 IBM Corporation

Installing Oracle Clusterware

Updating the Database HomeInstall the 10203 update on the new database home Select the database home instead of Oracle ASM on the Specify Home Details page

Configuring Listener NamesIf your RAC environment consists of only Oracle CRS an Oracle ASM instance and a database instance on each node it is easier to support only one listenerora configuration file for the Oracle ASM and database instances The best way to accomplish this is to create an environment variable TNS_ADMIN and assign it to the $ORACLE_HOMEnetworkadmin directory of the database instance For example

export TNS_ADMIN=$ORA_ASM_HOMEnetworkadmin

This environment variable allows the DBA to maintain one set of tnsnamesora sqlnetora and listenerora files for the entire environment

Creating and Configuring a DatabaseUse the Database Configuration Assistant (DBCA) to create a database For more information refer to the Oracle documentation about using DBCA As you create the database remember the following important points

bull Using DBCA gives you the capability to create the Oracle ASM disk groups if you choose to use Oracle ASM

bull The Flash Recovery Area (FRA) option is highly recommended A FRA allows the setup of a separate area on disk for flashback capability area for the archive logs the second copy of the multiplexed redo log files and for RMAN backups Make sure that the FRA is correctly set up to handle the performance of the database and that the FRA contains enough space

Space to hold all of the data for the data retention periods that are set by the database

Space for the type of backups that will be used

Space for the number of backups that will be held

For further documentation about using the FRA refer to the Oracle documentation as well as the following white paper

Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to

wwwibmcomsupporttechdocs

14 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Creating and Configuring a Database

ConclusionAs you have followed the example of setting up a two-node cluster described in this document you have seen how simple and how effective a deployment of Oracle RAC 10g Release 2 can be When you are doing your own installation be sure to use the applicable installation information as the basis for your work This document describes only tips and techniques Your installer should have the expertise to know how and when to modify instructions as necessary If your environment is complex and you require outside help contact your IBM business partner or sales representative See ldquoContact Informationrdquo below

Contact InformationFor more information and sales office locations visit the IBM web site at

httpwwwibmcom

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 15copy Copyright 2008 IBM Corporation

Contact Information

16 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix A

References

AIXFor AIX operating system downloads go to Quick Links for AIX Fixes

httpwww-933ibmcomeserversupportfixesfixcentralmainpseriesaix

For AIX documentation go to AIX Information

httppublibboulderibmcominfocentersystemsindexjsptopic=comibmaixdocdocbaseaixparenthtm

IBM Corporation 2006 Get the Latest Version of OpenSSH for AIX

httpwwwibmcomdeveloperworkseserverarticlesopenssh_updatedhtml

IBM StorageFor information about the IBM DS4800 Storage Subsystem go to

httpwww-03ibmcomsystemsstoragediskds4000ds4800indexhtml

For information about IBM storage go to

httpwww-03ibmcomsystemsstoragediskcm_re=masthead-_-products-_-stg-disk

OracleFor Oracle documentation go to

httpwwworaclecomtechnologydocumentationindexhtml

IBM Corporation 2008 Tuning IBM AIX 5L for an Oracle Database

httpwwwibmcompartnerworldwpsservletContentHandlerVPAA-6V2C4E

IBM Corporation 2008 Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems

wwwibmcomsupporttechdocs

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem A-1copy Copyright 2008 IBM Corporation

A-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix B

Troubleshooting

CSS Node Evictions can affect your functionality

IO time-outs can cause inappropriate node evictions during multi-path failover or other problems Two parameters might affect this css_misscount and disktimeout With versions of Oracle prior to 102 this was primarily accommodated by increasing the value of the css_misscount parameter However the function of this parameter has changed as of Oracle 102 and changes to this parameter might not achieve the desired result The new disktimeout parameter should be used in most cases to make this adjustment In many cases the default values will work fine However you might find it necessary to increase the default value of this parameter if you are getting a lot of CSS node evictions

For information about how the parameters work and how to adjust them refer to the following two articles Go to httpmetalinkoraclecom

bull Metalink DocID 2944301 CSS Timeout Computation in RAC 10g

bull Metalink DocID 2847521 Steps to Increase CSS Misscount Reboottime and Disktimeout

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem B-1copy Copyright 2008 IBM Corporation

B-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Trademarks and special notices copy Copyright IBM Corporation 2008 All rights reserved

References in this document to IBM products or services do not imply that IBM intends to make them available in every country

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corporation in the United States other countries or both If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (reg or trade) these symbols indicate US registered or common law trademarks owned by IBM at the time this information was published Such trademarks may also be registered or common law trademarks in other countries A current list of IBM trademarks is available on the Web at Copyright and trademark information at wwwibmcomlegalcopytradeshtml

Intel Intel Inside (logos) MMX and Pentium are trademarks of Intel Corporation in the United States other countries or both

UNIX is a registered trademark of The Open Group in the United States and other countries

Linux is a trademark of Linus Torvalds in the United States other countries or both

Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both

Other company product or service names may be trademarks or service marks of others

Information is provided AS IS without warranty of any kind

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved Actual environmental costs and performance characteristics may vary by customer

Information concerning non-IBM products was obtained from a supplier of these products published announcement material or other publicly available sources and does not constitute an endorsement of such products by IBM Sources for non-IBM list prices and performance numbers are taken from publicly available information including vendor announcements and vendor worldwide homepages IBM has not tested these products and cannot confirm the accuracy of performance capability or any other claims related to non-IBM products Questions on the capability of non-IBM products should be addressed to the supplier of those products

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk

  • Title Page13
  • Contents
  • Deploying Oracle RAC 10g Release 2 with ASM on AIX 53 with an IBM DS4700 or DS4800 Storage Subsystem
    • Introduction
      • Required Documents
      • Benefits
      • Reference Configuration
        • Configuring the Hosts
        • Configuring the IBM Storage
        • Installing Oracle
          • Creating Directories for Oracle
          • Setting Up the Oracle Users Profile
            • Installing Oracle Clusterware
              • Updating Oracle CRS
              • Installing and Configuring Oracle ASM
              • Updating Oracle ASM
              • Installing an Oracle Database
              • Updating the Database Home
              • Configuring Listener Names
              • Creating and Configuring a Database
                • Conclusion
                • Contact Information
                  • Appendix A References
                    • AIX
                    • IBM Storage
                    • Oracle
                      • Appendix B Troubleshooting
                      • Copyrights13

Configuring the Hosts

c Copy the authorized_keys file to ~ssh on the next system

d Repeat step b and step c on each system

e After all systems have a public key in the file copy the file back to all of the previous systems so that they all have an identical file

f Test the SSH function from each system to itself and to each of the other systems by issuing such commands as the ssh host1 date command Make sure that the commands can be executed without using a password

g Log on as the Oracle user and repeat step a through step f so that Oracle can also perform remote commands without passwords

NOTE The permissions must restrict write access on both the base user directory and the ssh directory If the permissions are not correctly set OpenSSH requires a password even if the appropriate public key has been added to the authorized_keys file

12 Edit the profile file in the Oracle userrsquos home directory to set the Oracle users parameters

Table 1 shows the entries that you must add to the profile file

Table 1 Oracle Userrsquos Parameters

Parameter to Add Explanation

umask 022

AIXTHREAD_SCOPE=S

An environment variable that directs the system to place user threads within a system-wide rather than a process-wide contention scope This placement provides better performance and more efficient memory use

CRS_HOME A unique home directory for Oracle CRS binaries

ASM_HOME

Use this parameter if you prefer to install separate Oracle binaries for Oracle ASM This method is preferable if there will be more than one database instance supported by Oracle ASM

ORACLE_HOME A unique home for the database binaries

LD_LIBRARY_PATH The complete pathway is $ORACLE_HOMElib

ORACLE_BASE Identifies the Oracle base directory

ORACLE_SID Identifies the instance

PATH Include $ORACLE_HOMEbin

CLASSPATHInclude $ORACLE_HOMEjdbclibltclasses filesgt

8 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

13 Edit the profile file in the root userrsquos home directory to set the same parameters as set in the Oracle users profile

Be sure to add the following entry to the profile file

AIXTHREAD_SCOPE=S

14 Synchronize the system time between the hosts in the cluster using any effective method The reference configuration used Network Time Protocol (NTP) If an NTP server is available use this method to set up xntpd

a Edit etcntpconf with the nameIP of the NTP server(s)

b Add a line server lthostnamegt for each host

c Start the service using the smit xntpd command

15 Because some Oracle utilities expect lsattr in etc and do not function correctly without a link to it use the ln -s usrsbinlsattr etclsattr command to add a soft link

Configuring the IBM Storage

IMPORTANT You must know how to best configure the storage for enhanced performance For this information refer to Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to wwwibmcomsupporttechdocs

Use DS4000 Storage Manager and the AIX system utilities to configure the IBM storage subsystem

1 Use DS4000 Storage Manager to create logical drives on the storage subsystem to be shared for Oracle CRS communication and database storage

NOTE You also may use additional logical drives for Oracle binaries and similar purposes however for those logical drives on the storage subsystem that are used exclusively by one particular host you will need the Storage Partitioning premium feature

You will need to analyze your implementation to determine the appropriate logical drive sizes desired RAID levels and other parameters The reference configuration uses the following minimal configuration of shared logical drives

2 Oracle Cluster Registry (OCR) logical drives (300 MB RAID1)

3 voting disks (300MB RAID1)

1 logical drive for Oracle ASMs SPFILE (100MB RAID1)

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 9copy Copyright 2008 IBM Corporation

Configuring the IBM Storage

2 logical drives for Oracle ASM data storage (300 GB RAID5)

2 logical drives for Oracle ASM log storage (135GB RAID1)

The number and the size of the logical drives created for Oracle ASM storage varies based on the individual implementation but the sizes used for the OCR disks voting disks and the Oracle ASM SPFILE disks in the reference configuration are sufficient for any configuration

In the reference configuration the Oracle binaries are installed on internal disks within each individual server The reference configuration uses the JFS2 file system and the logical drive was mounted as u01

2 Configure the shared disks in the operating system

After you have created the logical drives on the storage subsystem the cfgmgr command rescans the AIX system for changes to the physical devices

The lspv command shows the physical devices that AIX has configured

The fget_config -Av command shows the LUN numbers the disk drive name and the associated logical drive names configured on the storage subsystem

3 If the hdisk names for each logical drive on the storage subsystem that will be shared in the cluster are not identical on all cluster nodes create new device nodes for each of the logical drives to give them a consistent name across all nodes

For example if the logical drive for the first OCR disk was hdisk2 on node1 and hdisk3 on node2 a new device node (devnode) called devORA_OCR1 could be created on each host using the major numbers and the minor numbers that correspond with the same logical drive on each system Show the major and the minor numbers with the command ls -l Use the command mknod devORA_OCR1 c ltmajor of OCRgt ltminor of OCRgt to create the new devnode

Even when the disk drive names correlate on all of the nodes you should still create these named device nodes for ease of location and administration Named device nodes were created on the reference configuration for all shared logical drives using the convention devORA_ltusegt

4 Set the reserve_policy attribute for each logical drive to be used for shared access for OCR logical drive voting logical drive or Oracle ASM logical drive

NOTE This operation must be performed on the hdisk device name rather than on the named devnode created in step 3

This operation allows the nodes to share the logical drives Otherwise the first system that accesses the logical drive reserves it for exclusive use Oracle coordinates access to the shared logical drives

10 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

For an IBM storage subsystem reserve_policy=no_reserve is the correct value

a Use the lsattr -E -l hdisk | grep reserve command to check the attribute

b Use the chdev -l hdisk -a reserve_policy=no_reserve command to change the policy

c Repeat step a and step b on each cluster node

5 Set ownership and permissions on devnodes to be used by Oracle

NOTE If you created new devnodes for the shared logical drives set the ownership and permissions on the new devnode that you created not on the original hdisk devnode

a Use the following commands to set the ownership for each OCR device file

chown rootoinstall devltdevnodegt

chmod 640 devltdevnodegt

b Use the following commands to set the ownership and permission on the voting disk and database device files

chown oracledba devltdevnodegt

chmod 660 devltdevnodegt

c Repeat step a and step b on each cluster node

NOTE Do not assign a physical volume identifier (PVID) to hdisks that will be used with Oracle ASM

Installing OracleHere are the basic steps for installing Oracle

bull Create directories for Oracle

bull Set up the Oracle userrsquos profile

bull Install Oracle Clusterware

bull Update Oracle CRS

bull Install and configure Oracle ASM

bull Update Oracle ASM

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 11copy Copyright 2008 IBM Corporation

Installing Oracle Clusterware

bull Install Oracle Database

bull Update the database home

bull Configure the listener names

bull Create and configure a database

Creating Directories for OracleAlthough you can create directory structures for Oracle files in many different ways for ease of support use a standard directory structure based on the Optimal Flexible Architecture (OFA) guidelines published by Oracle For more information about the OFA Guidelines see Appendix D of the Oracle Database Installation Guide 10gR2 for AIX 5L Based Systems (B19075-02) The reference configuration is based on the OFA guidelines

Follow these two important guidelines when creating directories for Oracle

bull You must separate the Oracle CRS home directory from the Oracle base and the database home directories The Oracle CRS installation procedure sets the ownership for the entire Oracle CRS directory tree to root while the database directory tree must be owned by oracle

bull If more than one database will be serviced by Oracle ASM it is recommended that you create a separate home directory for Oracle ASM An Oracle ASM home directory provides the capability to modify the Oracle release for Oracle ASM while not changing the release of any of the databases on the same server The directory tree must be owned by oracle

Setting Up the Oracle Users Profile1 Set up the Oracle users profile before installing the Oracle binaries See

Appendix A for an example profile

2 Add or modify entries for ORACLE_HOME ORACLE_BASE PATH LD_LIBRARY_PATH and similar files to reflect the directories where the Oracle binaries will be placed during the Oracle Clusterware installation

Installing Oracle ClusterwareAs you install the Oracle Clusterware remember the following important points

bull Do not make the CRS_HOME path a subdirectory of the Oracle RAC 10g Release 2 Oracle base directory due to permission changes required for Oracle CRS For more details refer to Appendix E of the Oracle Clusterware Installation Guide

bull The rootsh script must be run as the root user and must be run on only one node at a time not concurrently

12 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Updating Oracle CRS

bull After rootsh runs on the last node it will attempt to run the vipca utility to configure the VIP network interfaces If you have IP addresses that are not routable across the Internet for example 100 192168 the vipca configuration will fail If the automatic vipca configuration fails run the vipca utility manually using the $ORA_CRS_HOMEbinvipca command from the command line as root This requires a graphical display

bull You can do the manual configuration with vipca before clicking OK on the Oracle Universal Installer the Oracle CRS installation page The Oracle CRS installation page waits for you to run rootsh Do the manual configuration with vipca during this time This action allows the remainder of the post-installation checks to complete successfully

Updating Oracle CRSRefer to the Metalink Bulletin 2820361 (see important note on page 5) for required updates The reference configuration used the 10203 update to the Oracle CRS home Download the 10203 update and any other updates required from the Oracle Metalink web site (httpmetalinkoraclecom) and install according to the instructions on the web site

Installing and Configuring Oracle ASMBecause a single Oracle ASM instance can service many database instances Oracle recommends that you install Oracle ASM in its own home directory Follow the naming convention of the OFA guidelines for the directory path As you install and configure Oracle ASM remember the following important points

bull If the Oracle ASM instance was not correctly created by the Oracle Universal Installer Wizard during installation use the dbca utility to add the instance and create the disk group(s)

bull If you want additional disk groups create them using the dbca utility

Updating Oracle ASMUpdate the Oracle ASM instance with the 10203 patch set and any additional updates indicated in Metalink Bulletin 2820361 You can use the same update media for 10203 previously downloaded to update Oracle CRS (page 13) Carefully follow the instructions for updating this database instance

Installing an Oracle DatabaseUse the Oracle Universal Installer to install another database home for a database instance Select Install Database Software Only on the Create Database page

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 13copy Copyright 2008 IBM Corporation

Installing Oracle Clusterware

Updating the Database HomeInstall the 10203 update on the new database home Select the database home instead of Oracle ASM on the Specify Home Details page

Configuring Listener NamesIf your RAC environment consists of only Oracle CRS an Oracle ASM instance and a database instance on each node it is easier to support only one listenerora configuration file for the Oracle ASM and database instances The best way to accomplish this is to create an environment variable TNS_ADMIN and assign it to the $ORACLE_HOMEnetworkadmin directory of the database instance For example

export TNS_ADMIN=$ORA_ASM_HOMEnetworkadmin

This environment variable allows the DBA to maintain one set of tnsnamesora sqlnetora and listenerora files for the entire environment

Creating and Configuring a DatabaseUse the Database Configuration Assistant (DBCA) to create a database For more information refer to the Oracle documentation about using DBCA As you create the database remember the following important points

bull Using DBCA gives you the capability to create the Oracle ASM disk groups if you choose to use Oracle ASM

bull The Flash Recovery Area (FRA) option is highly recommended A FRA allows the setup of a separate area on disk for flashback capability area for the archive logs the second copy of the multiplexed redo log files and for RMAN backups Make sure that the FRA is correctly set up to handle the performance of the database and that the FRA contains enough space

Space to hold all of the data for the data retention periods that are set by the database

Space for the type of backups that will be used

Space for the number of backups that will be held

For further documentation about using the FRA refer to the Oracle documentation as well as the following white paper

Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to

wwwibmcomsupporttechdocs

14 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Creating and Configuring a Database

ConclusionAs you have followed the example of setting up a two-node cluster described in this document you have seen how simple and how effective a deployment of Oracle RAC 10g Release 2 can be When you are doing your own installation be sure to use the applicable installation information as the basis for your work This document describes only tips and techniques Your installer should have the expertise to know how and when to modify instructions as necessary If your environment is complex and you require outside help contact your IBM business partner or sales representative See ldquoContact Informationrdquo below

Contact InformationFor more information and sales office locations visit the IBM web site at

httpwwwibmcom

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 15copy Copyright 2008 IBM Corporation

Contact Information

16 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix A

References

AIXFor AIX operating system downloads go to Quick Links for AIX Fixes

httpwww-933ibmcomeserversupportfixesfixcentralmainpseriesaix

For AIX documentation go to AIX Information

httppublibboulderibmcominfocentersystemsindexjsptopic=comibmaixdocdocbaseaixparenthtm

IBM Corporation 2006 Get the Latest Version of OpenSSH for AIX

httpwwwibmcomdeveloperworkseserverarticlesopenssh_updatedhtml

IBM StorageFor information about the IBM DS4800 Storage Subsystem go to

httpwww-03ibmcomsystemsstoragediskds4000ds4800indexhtml

For information about IBM storage go to

httpwww-03ibmcomsystemsstoragediskcm_re=masthead-_-products-_-stg-disk

OracleFor Oracle documentation go to

httpwwworaclecomtechnologydocumentationindexhtml

IBM Corporation 2008 Tuning IBM AIX 5L for an Oracle Database

httpwwwibmcompartnerworldwpsservletContentHandlerVPAA-6V2C4E

IBM Corporation 2008 Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems

wwwibmcomsupporttechdocs

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem A-1copy Copyright 2008 IBM Corporation

A-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix B

Troubleshooting

CSS Node Evictions can affect your functionality

IO time-outs can cause inappropriate node evictions during multi-path failover or other problems Two parameters might affect this css_misscount and disktimeout With versions of Oracle prior to 102 this was primarily accommodated by increasing the value of the css_misscount parameter However the function of this parameter has changed as of Oracle 102 and changes to this parameter might not achieve the desired result The new disktimeout parameter should be used in most cases to make this adjustment In many cases the default values will work fine However you might find it necessary to increase the default value of this parameter if you are getting a lot of CSS node evictions

For information about how the parameters work and how to adjust them refer to the following two articles Go to httpmetalinkoraclecom

bull Metalink DocID 2944301 CSS Timeout Computation in RAC 10g

bull Metalink DocID 2847521 Steps to Increase CSS Misscount Reboottime and Disktimeout

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem B-1copy Copyright 2008 IBM Corporation

B-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Trademarks and special notices copy Copyright IBM Corporation 2008 All rights reserved

References in this document to IBM products or services do not imply that IBM intends to make them available in every country

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corporation in the United States other countries or both If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (reg or trade) these symbols indicate US registered or common law trademarks owned by IBM at the time this information was published Such trademarks may also be registered or common law trademarks in other countries A current list of IBM trademarks is available on the Web at Copyright and trademark information at wwwibmcomlegalcopytradeshtml

Intel Intel Inside (logos) MMX and Pentium are trademarks of Intel Corporation in the United States other countries or both

UNIX is a registered trademark of The Open Group in the United States and other countries

Linux is a trademark of Linus Torvalds in the United States other countries or both

Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both

Other company product or service names may be trademarks or service marks of others

Information is provided AS IS without warranty of any kind

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved Actual environmental costs and performance characteristics may vary by customer

Information concerning non-IBM products was obtained from a supplier of these products published announcement material or other publicly available sources and does not constitute an endorsement of such products by IBM Sources for non-IBM list prices and performance numbers are taken from publicly available information including vendor announcements and vendor worldwide homepages IBM has not tested these products and cannot confirm the accuracy of performance capability or any other claims related to non-IBM products Questions on the capability of non-IBM products should be addressed to the supplier of those products

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk

  • Title Page13
  • Contents
  • Deploying Oracle RAC 10g Release 2 with ASM on AIX 53 with an IBM DS4700 or DS4800 Storage Subsystem
    • Introduction
      • Required Documents
      • Benefits
      • Reference Configuration
        • Configuring the Hosts
        • Configuring the IBM Storage
        • Installing Oracle
          • Creating Directories for Oracle
          • Setting Up the Oracle Users Profile
            • Installing Oracle Clusterware
              • Updating Oracle CRS
              • Installing and Configuring Oracle ASM
              • Updating Oracle ASM
              • Installing an Oracle Database
              • Updating the Database Home
              • Configuring Listener Names
              • Creating and Configuring a Database
                • Conclusion
                • Contact Information
                  • Appendix A References
                    • AIX
                    • IBM Storage
                    • Oracle
                      • Appendix B Troubleshooting
                      • Copyrights13

Reference Configuration

13 Edit the profile file in the root userrsquos home directory to set the same parameters as set in the Oracle users profile

Be sure to add the following entry to the profile file

AIXTHREAD_SCOPE=S

14 Synchronize the system time between the hosts in the cluster using any effective method The reference configuration used Network Time Protocol (NTP) If an NTP server is available use this method to set up xntpd

a Edit etcntpconf with the nameIP of the NTP server(s)

b Add a line server lthostnamegt for each host

c Start the service using the smit xntpd command

15 Because some Oracle utilities expect lsattr in etc and do not function correctly without a link to it use the ln -s usrsbinlsattr etclsattr command to add a soft link

Configuring the IBM Storage

IMPORTANT You must know how to best configure the storage for enhanced performance For this information refer to Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to wwwibmcomsupporttechdocs

Use DS4000 Storage Manager and the AIX system utilities to configure the IBM storage subsystem

1 Use DS4000 Storage Manager to create logical drives on the storage subsystem to be shared for Oracle CRS communication and database storage

NOTE You also may use additional logical drives for Oracle binaries and similar purposes however for those logical drives on the storage subsystem that are used exclusively by one particular host you will need the Storage Partitioning premium feature

You will need to analyze your implementation to determine the appropriate logical drive sizes desired RAID levels and other parameters The reference configuration uses the following minimal configuration of shared logical drives

2 Oracle Cluster Registry (OCR) logical drives (300 MB RAID1)

3 voting disks (300MB RAID1)

1 logical drive for Oracle ASMs SPFILE (100MB RAID1)

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 9copy Copyright 2008 IBM Corporation

Configuring the IBM Storage

2 logical drives for Oracle ASM data storage (300 GB RAID5)

2 logical drives for Oracle ASM log storage (135GB RAID1)

The number and the size of the logical drives created for Oracle ASM storage varies based on the individual implementation but the sizes used for the OCR disks voting disks and the Oracle ASM SPFILE disks in the reference configuration are sufficient for any configuration

In the reference configuration the Oracle binaries are installed on internal disks within each individual server The reference configuration uses the JFS2 file system and the logical drive was mounted as u01

2 Configure the shared disks in the operating system

After you have created the logical drives on the storage subsystem the cfgmgr command rescans the AIX system for changes to the physical devices

The lspv command shows the physical devices that AIX has configured

The fget_config -Av command shows the LUN numbers the disk drive name and the associated logical drive names configured on the storage subsystem

3 If the hdisk names for each logical drive on the storage subsystem that will be shared in the cluster are not identical on all cluster nodes create new device nodes for each of the logical drives to give them a consistent name across all nodes

For example if the logical drive for the first OCR disk was hdisk2 on node1 and hdisk3 on node2 a new device node (devnode) called devORA_OCR1 could be created on each host using the major numbers and the minor numbers that correspond with the same logical drive on each system Show the major and the minor numbers with the command ls -l Use the command mknod devORA_OCR1 c ltmajor of OCRgt ltminor of OCRgt to create the new devnode

Even when the disk drive names correlate on all of the nodes you should still create these named device nodes for ease of location and administration Named device nodes were created on the reference configuration for all shared logical drives using the convention devORA_ltusegt

4 Set the reserve_policy attribute for each logical drive to be used for shared access for OCR logical drive voting logical drive or Oracle ASM logical drive

NOTE This operation must be performed on the hdisk device name rather than on the named devnode created in step 3

This operation allows the nodes to share the logical drives Otherwise the first system that accesses the logical drive reserves it for exclusive use Oracle coordinates access to the shared logical drives

10 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

For an IBM storage subsystem reserve_policy=no_reserve is the correct value

a Use the lsattr -E -l hdisk | grep reserve command to check the attribute

b Use the chdev -l hdisk -a reserve_policy=no_reserve command to change the policy

c Repeat step a and step b on each cluster node

5 Set ownership and permissions on devnodes to be used by Oracle

NOTE If you created new devnodes for the shared logical drives set the ownership and permissions on the new devnode that you created not on the original hdisk devnode

a Use the following commands to set the ownership for each OCR device file

chown rootoinstall devltdevnodegt

chmod 640 devltdevnodegt

b Use the following commands to set the ownership and permission on the voting disk and database device files

chown oracledba devltdevnodegt

chmod 660 devltdevnodegt

c Repeat step a and step b on each cluster node

NOTE Do not assign a physical volume identifier (PVID) to hdisks that will be used with Oracle ASM

Installing OracleHere are the basic steps for installing Oracle

bull Create directories for Oracle

bull Set up the Oracle userrsquos profile

bull Install Oracle Clusterware

bull Update Oracle CRS

bull Install and configure Oracle ASM

bull Update Oracle ASM

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 11copy Copyright 2008 IBM Corporation

Installing Oracle Clusterware

bull Install Oracle Database

bull Update the database home

bull Configure the listener names

bull Create and configure a database

Creating Directories for OracleAlthough you can create directory structures for Oracle files in many different ways for ease of support use a standard directory structure based on the Optimal Flexible Architecture (OFA) guidelines published by Oracle For more information about the OFA Guidelines see Appendix D of the Oracle Database Installation Guide 10gR2 for AIX 5L Based Systems (B19075-02) The reference configuration is based on the OFA guidelines

Follow these two important guidelines when creating directories for Oracle

bull You must separate the Oracle CRS home directory from the Oracle base and the database home directories The Oracle CRS installation procedure sets the ownership for the entire Oracle CRS directory tree to root while the database directory tree must be owned by oracle

bull If more than one database will be serviced by Oracle ASM it is recommended that you create a separate home directory for Oracle ASM An Oracle ASM home directory provides the capability to modify the Oracle release for Oracle ASM while not changing the release of any of the databases on the same server The directory tree must be owned by oracle

Setting Up the Oracle Users Profile1 Set up the Oracle users profile before installing the Oracle binaries See

Appendix A for an example profile

2 Add or modify entries for ORACLE_HOME ORACLE_BASE PATH LD_LIBRARY_PATH and similar files to reflect the directories where the Oracle binaries will be placed during the Oracle Clusterware installation

Installing Oracle ClusterwareAs you install the Oracle Clusterware remember the following important points

bull Do not make the CRS_HOME path a subdirectory of the Oracle RAC 10g Release 2 Oracle base directory due to permission changes required for Oracle CRS For more details refer to Appendix E of the Oracle Clusterware Installation Guide

bull The rootsh script must be run as the root user and must be run on only one node at a time not concurrently

12 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Updating Oracle CRS

bull After rootsh runs on the last node it will attempt to run the vipca utility to configure the VIP network interfaces If you have IP addresses that are not routable across the Internet for example 100 192168 the vipca configuration will fail If the automatic vipca configuration fails run the vipca utility manually using the $ORA_CRS_HOMEbinvipca command from the command line as root This requires a graphical display

bull You can do the manual configuration with vipca before clicking OK on the Oracle Universal Installer the Oracle CRS installation page The Oracle CRS installation page waits for you to run rootsh Do the manual configuration with vipca during this time This action allows the remainder of the post-installation checks to complete successfully

Updating Oracle CRSRefer to the Metalink Bulletin 2820361 (see important note on page 5) for required updates The reference configuration used the 10203 update to the Oracle CRS home Download the 10203 update and any other updates required from the Oracle Metalink web site (httpmetalinkoraclecom) and install according to the instructions on the web site

Installing and Configuring Oracle ASMBecause a single Oracle ASM instance can service many database instances Oracle recommends that you install Oracle ASM in its own home directory Follow the naming convention of the OFA guidelines for the directory path As you install and configure Oracle ASM remember the following important points

bull If the Oracle ASM instance was not correctly created by the Oracle Universal Installer Wizard during installation use the dbca utility to add the instance and create the disk group(s)

bull If you want additional disk groups create them using the dbca utility

Updating Oracle ASMUpdate the Oracle ASM instance with the 10203 patch set and any additional updates indicated in Metalink Bulletin 2820361 You can use the same update media for 10203 previously downloaded to update Oracle CRS (page 13) Carefully follow the instructions for updating this database instance

Installing an Oracle DatabaseUse the Oracle Universal Installer to install another database home for a database instance Select Install Database Software Only on the Create Database page

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 13copy Copyright 2008 IBM Corporation

Installing Oracle Clusterware

Updating the Database HomeInstall the 10203 update on the new database home Select the database home instead of Oracle ASM on the Specify Home Details page

Configuring Listener NamesIf your RAC environment consists of only Oracle CRS an Oracle ASM instance and a database instance on each node it is easier to support only one listenerora configuration file for the Oracle ASM and database instances The best way to accomplish this is to create an environment variable TNS_ADMIN and assign it to the $ORACLE_HOMEnetworkadmin directory of the database instance For example

export TNS_ADMIN=$ORA_ASM_HOMEnetworkadmin

This environment variable allows the DBA to maintain one set of tnsnamesora sqlnetora and listenerora files for the entire environment

Creating and Configuring a DatabaseUse the Database Configuration Assistant (DBCA) to create a database For more information refer to the Oracle documentation about using DBCA As you create the database remember the following important points

bull Using DBCA gives you the capability to create the Oracle ASM disk groups if you choose to use Oracle ASM

bull The Flash Recovery Area (FRA) option is highly recommended A FRA allows the setup of a separate area on disk for flashback capability area for the archive logs the second copy of the multiplexed redo log files and for RMAN backups Make sure that the FRA is correctly set up to handle the performance of the database and that the FRA contains enough space

Space to hold all of the data for the data retention periods that are set by the database

Space for the type of backups that will be used

Space for the number of backups that will be held

For further documentation about using the FRA refer to the Oracle documentation as well as the following white paper

Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to

wwwibmcomsupporttechdocs

14 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Creating and Configuring a Database

ConclusionAs you have followed the example of setting up a two-node cluster described in this document you have seen how simple and how effective a deployment of Oracle RAC 10g Release 2 can be When you are doing your own installation be sure to use the applicable installation information as the basis for your work This document describes only tips and techniques Your installer should have the expertise to know how and when to modify instructions as necessary If your environment is complex and you require outside help contact your IBM business partner or sales representative See ldquoContact Informationrdquo below

Contact InformationFor more information and sales office locations visit the IBM web site at

httpwwwibmcom

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 15copy Copyright 2008 IBM Corporation

Contact Information

16 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix A

References

AIXFor AIX operating system downloads go to Quick Links for AIX Fixes

httpwww-933ibmcomeserversupportfixesfixcentralmainpseriesaix

For AIX documentation go to AIX Information

httppublibboulderibmcominfocentersystemsindexjsptopic=comibmaixdocdocbaseaixparenthtm

IBM Corporation 2006 Get the Latest Version of OpenSSH for AIX

httpwwwibmcomdeveloperworkseserverarticlesopenssh_updatedhtml

IBM StorageFor information about the IBM DS4800 Storage Subsystem go to

httpwww-03ibmcomsystemsstoragediskds4000ds4800indexhtml

For information about IBM storage go to

httpwww-03ibmcomsystemsstoragediskcm_re=masthead-_-products-_-stg-disk

OracleFor Oracle documentation go to

httpwwworaclecomtechnologydocumentationindexhtml

IBM Corporation 2008 Tuning IBM AIX 5L for an Oracle Database

httpwwwibmcompartnerworldwpsservletContentHandlerVPAA-6V2C4E

IBM Corporation 2008 Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems

wwwibmcomsupporttechdocs

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem A-1copy Copyright 2008 IBM Corporation

A-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix B

Troubleshooting

CSS Node Evictions can affect your functionality

IO time-outs can cause inappropriate node evictions during multi-path failover or other problems Two parameters might affect this css_misscount and disktimeout With versions of Oracle prior to 102 this was primarily accommodated by increasing the value of the css_misscount parameter However the function of this parameter has changed as of Oracle 102 and changes to this parameter might not achieve the desired result The new disktimeout parameter should be used in most cases to make this adjustment In many cases the default values will work fine However you might find it necessary to increase the default value of this parameter if you are getting a lot of CSS node evictions

For information about how the parameters work and how to adjust them refer to the following two articles Go to httpmetalinkoraclecom

bull Metalink DocID 2944301 CSS Timeout Computation in RAC 10g

bull Metalink DocID 2847521 Steps to Increase CSS Misscount Reboottime and Disktimeout

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem B-1copy Copyright 2008 IBM Corporation

B-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Trademarks and special notices copy Copyright IBM Corporation 2008 All rights reserved

References in this document to IBM products or services do not imply that IBM intends to make them available in every country

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corporation in the United States other countries or both If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (reg or trade) these symbols indicate US registered or common law trademarks owned by IBM at the time this information was published Such trademarks may also be registered or common law trademarks in other countries A current list of IBM trademarks is available on the Web at Copyright and trademark information at wwwibmcomlegalcopytradeshtml

Intel Intel Inside (logos) MMX and Pentium are trademarks of Intel Corporation in the United States other countries or both

UNIX is a registered trademark of The Open Group in the United States and other countries

Linux is a trademark of Linus Torvalds in the United States other countries or both

Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both

Other company product or service names may be trademarks or service marks of others

Information is provided AS IS without warranty of any kind

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved Actual environmental costs and performance characteristics may vary by customer

Information concerning non-IBM products was obtained from a supplier of these products published announcement material or other publicly available sources and does not constitute an endorsement of such products by IBM Sources for non-IBM list prices and performance numbers are taken from publicly available information including vendor announcements and vendor worldwide homepages IBM has not tested these products and cannot confirm the accuracy of performance capability or any other claims related to non-IBM products Questions on the capability of non-IBM products should be addressed to the supplier of those products

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk

  • Title Page13
  • Contents
  • Deploying Oracle RAC 10g Release 2 with ASM on AIX 53 with an IBM DS4700 or DS4800 Storage Subsystem
    • Introduction
      • Required Documents
      • Benefits
      • Reference Configuration
        • Configuring the Hosts
        • Configuring the IBM Storage
        • Installing Oracle
          • Creating Directories for Oracle
          • Setting Up the Oracle Users Profile
            • Installing Oracle Clusterware
              • Updating Oracle CRS
              • Installing and Configuring Oracle ASM
              • Updating Oracle ASM
              • Installing an Oracle Database
              • Updating the Database Home
              • Configuring Listener Names
              • Creating and Configuring a Database
                • Conclusion
                • Contact Information
                  • Appendix A References
                    • AIX
                    • IBM Storage
                    • Oracle
                      • Appendix B Troubleshooting
                      • Copyrights13

Configuring the IBM Storage

2 logical drives for Oracle ASM data storage (300 GB RAID5)

2 logical drives for Oracle ASM log storage (135GB RAID1)

The number and the size of the logical drives created for Oracle ASM storage varies based on the individual implementation but the sizes used for the OCR disks voting disks and the Oracle ASM SPFILE disks in the reference configuration are sufficient for any configuration

In the reference configuration the Oracle binaries are installed on internal disks within each individual server The reference configuration uses the JFS2 file system and the logical drive was mounted as u01

2 Configure the shared disks in the operating system

After you have created the logical drives on the storage subsystem the cfgmgr command rescans the AIX system for changes to the physical devices

The lspv command shows the physical devices that AIX has configured

The fget_config -Av command shows the LUN numbers the disk drive name and the associated logical drive names configured on the storage subsystem

3 If the hdisk names for each logical drive on the storage subsystem that will be shared in the cluster are not identical on all cluster nodes create new device nodes for each of the logical drives to give them a consistent name across all nodes

For example if the logical drive for the first OCR disk was hdisk2 on node1 and hdisk3 on node2 a new device node (devnode) called devORA_OCR1 could be created on each host using the major numbers and the minor numbers that correspond with the same logical drive on each system Show the major and the minor numbers with the command ls -l Use the command mknod devORA_OCR1 c ltmajor of OCRgt ltminor of OCRgt to create the new devnode

Even when the disk drive names correlate on all of the nodes you should still create these named device nodes for ease of location and administration Named device nodes were created on the reference configuration for all shared logical drives using the convention devORA_ltusegt

4 Set the reserve_policy attribute for each logical drive to be used for shared access for OCR logical drive voting logical drive or Oracle ASM logical drive

NOTE This operation must be performed on the hdisk device name rather than on the named devnode created in step 3

This operation allows the nodes to share the logical drives Otherwise the first system that accesses the logical drive reserves it for exclusive use Oracle coordinates access to the shared logical drives

10 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Reference Configuration

For an IBM storage subsystem reserve_policy=no_reserve is the correct value

a Use the lsattr -E -l hdisk | grep reserve command to check the attribute

b Use the chdev -l hdisk -a reserve_policy=no_reserve command to change the policy

c Repeat step a and step b on each cluster node

5 Set ownership and permissions on devnodes to be used by Oracle

NOTE If you created new devnodes for the shared logical drives set the ownership and permissions on the new devnode that you created not on the original hdisk devnode

a Use the following commands to set the ownership for each OCR device file

chown rootoinstall devltdevnodegt

chmod 640 devltdevnodegt

b Use the following commands to set the ownership and permission on the voting disk and database device files

chown oracledba devltdevnodegt

chmod 660 devltdevnodegt

c Repeat step a and step b on each cluster node

NOTE Do not assign a physical volume identifier (PVID) to hdisks that will be used with Oracle ASM

Installing OracleHere are the basic steps for installing Oracle

bull Create directories for Oracle

bull Set up the Oracle userrsquos profile

bull Install Oracle Clusterware

bull Update Oracle CRS

bull Install and configure Oracle ASM

bull Update Oracle ASM

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 11copy Copyright 2008 IBM Corporation

Installing Oracle Clusterware

bull Install Oracle Database

bull Update the database home

bull Configure the listener names

bull Create and configure a database

Creating Directories for OracleAlthough you can create directory structures for Oracle files in many different ways for ease of support use a standard directory structure based on the Optimal Flexible Architecture (OFA) guidelines published by Oracle For more information about the OFA Guidelines see Appendix D of the Oracle Database Installation Guide 10gR2 for AIX 5L Based Systems (B19075-02) The reference configuration is based on the OFA guidelines

Follow these two important guidelines when creating directories for Oracle

bull You must separate the Oracle CRS home directory from the Oracle base and the database home directories The Oracle CRS installation procedure sets the ownership for the entire Oracle CRS directory tree to root while the database directory tree must be owned by oracle

bull If more than one database will be serviced by Oracle ASM it is recommended that you create a separate home directory for Oracle ASM An Oracle ASM home directory provides the capability to modify the Oracle release for Oracle ASM while not changing the release of any of the databases on the same server The directory tree must be owned by oracle

Setting Up the Oracle Users Profile1 Set up the Oracle users profile before installing the Oracle binaries See

Appendix A for an example profile

2 Add or modify entries for ORACLE_HOME ORACLE_BASE PATH LD_LIBRARY_PATH and similar files to reflect the directories where the Oracle binaries will be placed during the Oracle Clusterware installation

Installing Oracle ClusterwareAs you install the Oracle Clusterware remember the following important points

bull Do not make the CRS_HOME path a subdirectory of the Oracle RAC 10g Release 2 Oracle base directory due to permission changes required for Oracle CRS For more details refer to Appendix E of the Oracle Clusterware Installation Guide

bull The rootsh script must be run as the root user and must be run on only one node at a time not concurrently

12 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Updating Oracle CRS

bull After rootsh runs on the last node it will attempt to run the vipca utility to configure the VIP network interfaces If you have IP addresses that are not routable across the Internet for example 100 192168 the vipca configuration will fail If the automatic vipca configuration fails run the vipca utility manually using the $ORA_CRS_HOMEbinvipca command from the command line as root This requires a graphical display

bull You can do the manual configuration with vipca before clicking OK on the Oracle Universal Installer the Oracle CRS installation page The Oracle CRS installation page waits for you to run rootsh Do the manual configuration with vipca during this time This action allows the remainder of the post-installation checks to complete successfully

Updating Oracle CRSRefer to the Metalink Bulletin 2820361 (see important note on page 5) for required updates The reference configuration used the 10203 update to the Oracle CRS home Download the 10203 update and any other updates required from the Oracle Metalink web site (httpmetalinkoraclecom) and install according to the instructions on the web site

Installing and Configuring Oracle ASMBecause a single Oracle ASM instance can service many database instances Oracle recommends that you install Oracle ASM in its own home directory Follow the naming convention of the OFA guidelines for the directory path As you install and configure Oracle ASM remember the following important points

bull If the Oracle ASM instance was not correctly created by the Oracle Universal Installer Wizard during installation use the dbca utility to add the instance and create the disk group(s)

bull If you want additional disk groups create them using the dbca utility

Updating Oracle ASMUpdate the Oracle ASM instance with the 10203 patch set and any additional updates indicated in Metalink Bulletin 2820361 You can use the same update media for 10203 previously downloaded to update Oracle CRS (page 13) Carefully follow the instructions for updating this database instance

Installing an Oracle DatabaseUse the Oracle Universal Installer to install another database home for a database instance Select Install Database Software Only on the Create Database page

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 13copy Copyright 2008 IBM Corporation

Installing Oracle Clusterware

Updating the Database HomeInstall the 10203 update on the new database home Select the database home instead of Oracle ASM on the Specify Home Details page

Configuring Listener NamesIf your RAC environment consists of only Oracle CRS an Oracle ASM instance and a database instance on each node it is easier to support only one listenerora configuration file for the Oracle ASM and database instances The best way to accomplish this is to create an environment variable TNS_ADMIN and assign it to the $ORACLE_HOMEnetworkadmin directory of the database instance For example

export TNS_ADMIN=$ORA_ASM_HOMEnetworkadmin

This environment variable allows the DBA to maintain one set of tnsnamesora sqlnetora and listenerora files for the entire environment

Creating and Configuring a DatabaseUse the Database Configuration Assistant (DBCA) to create a database For more information refer to the Oracle documentation about using DBCA As you create the database remember the following important points

bull Using DBCA gives you the capability to create the Oracle ASM disk groups if you choose to use Oracle ASM

bull The Flash Recovery Area (FRA) option is highly recommended A FRA allows the setup of a separate area on disk for flashback capability area for the archive logs the second copy of the multiplexed redo log files and for RMAN backups Make sure that the FRA is correctly set up to handle the performance of the database and that the FRA contains enough space

Space to hold all of the data for the data retention periods that are set by the database

Space for the type of backups that will be used

Space for the number of backups that will be held

For further documentation about using the FRA refer to the Oracle documentation as well as the following white paper

Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to

wwwibmcomsupporttechdocs

14 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Creating and Configuring a Database

ConclusionAs you have followed the example of setting up a two-node cluster described in this document you have seen how simple and how effective a deployment of Oracle RAC 10g Release 2 can be When you are doing your own installation be sure to use the applicable installation information as the basis for your work This document describes only tips and techniques Your installer should have the expertise to know how and when to modify instructions as necessary If your environment is complex and you require outside help contact your IBM business partner or sales representative See ldquoContact Informationrdquo below

Contact InformationFor more information and sales office locations visit the IBM web site at

httpwwwibmcom

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 15copy Copyright 2008 IBM Corporation

Contact Information

16 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix A

References

AIXFor AIX operating system downloads go to Quick Links for AIX Fixes

httpwww-933ibmcomeserversupportfixesfixcentralmainpseriesaix

For AIX documentation go to AIX Information

httppublibboulderibmcominfocentersystemsindexjsptopic=comibmaixdocdocbaseaixparenthtm

IBM Corporation 2006 Get the Latest Version of OpenSSH for AIX

httpwwwibmcomdeveloperworkseserverarticlesopenssh_updatedhtml

IBM StorageFor information about the IBM DS4800 Storage Subsystem go to

httpwww-03ibmcomsystemsstoragediskds4000ds4800indexhtml

For information about IBM storage go to

httpwww-03ibmcomsystemsstoragediskcm_re=masthead-_-products-_-stg-disk

OracleFor Oracle documentation go to

httpwwworaclecomtechnologydocumentationindexhtml

IBM Corporation 2008 Tuning IBM AIX 5L for an Oracle Database

httpwwwibmcompartnerworldwpsservletContentHandlerVPAA-6V2C4E

IBM Corporation 2008 Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems

wwwibmcomsupporttechdocs

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem A-1copy Copyright 2008 IBM Corporation

A-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix B

Troubleshooting

CSS Node Evictions can affect your functionality

IO time-outs can cause inappropriate node evictions during multi-path failover or other problems Two parameters might affect this css_misscount and disktimeout With versions of Oracle prior to 102 this was primarily accommodated by increasing the value of the css_misscount parameter However the function of this parameter has changed as of Oracle 102 and changes to this parameter might not achieve the desired result The new disktimeout parameter should be used in most cases to make this adjustment In many cases the default values will work fine However you might find it necessary to increase the default value of this parameter if you are getting a lot of CSS node evictions

For information about how the parameters work and how to adjust them refer to the following two articles Go to httpmetalinkoraclecom

bull Metalink DocID 2944301 CSS Timeout Computation in RAC 10g

bull Metalink DocID 2847521 Steps to Increase CSS Misscount Reboottime and Disktimeout

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem B-1copy Copyright 2008 IBM Corporation

B-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Trademarks and special notices copy Copyright IBM Corporation 2008 All rights reserved

References in this document to IBM products or services do not imply that IBM intends to make them available in every country

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corporation in the United States other countries or both If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (reg or trade) these symbols indicate US registered or common law trademarks owned by IBM at the time this information was published Such trademarks may also be registered or common law trademarks in other countries A current list of IBM trademarks is available on the Web at Copyright and trademark information at wwwibmcomlegalcopytradeshtml

Intel Intel Inside (logos) MMX and Pentium are trademarks of Intel Corporation in the United States other countries or both

UNIX is a registered trademark of The Open Group in the United States and other countries

Linux is a trademark of Linus Torvalds in the United States other countries or both

Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both

Other company product or service names may be trademarks or service marks of others

Information is provided AS IS without warranty of any kind

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved Actual environmental costs and performance characteristics may vary by customer

Information concerning non-IBM products was obtained from a supplier of these products published announcement material or other publicly available sources and does not constitute an endorsement of such products by IBM Sources for non-IBM list prices and performance numbers are taken from publicly available information including vendor announcements and vendor worldwide homepages IBM has not tested these products and cannot confirm the accuracy of performance capability or any other claims related to non-IBM products Questions on the capability of non-IBM products should be addressed to the supplier of those products

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk

  • Title Page13
  • Contents
  • Deploying Oracle RAC 10g Release 2 with ASM on AIX 53 with an IBM DS4700 or DS4800 Storage Subsystem
    • Introduction
      • Required Documents
      • Benefits
      • Reference Configuration
        • Configuring the Hosts
        • Configuring the IBM Storage
        • Installing Oracle
          • Creating Directories for Oracle
          • Setting Up the Oracle Users Profile
            • Installing Oracle Clusterware
              • Updating Oracle CRS
              • Installing and Configuring Oracle ASM
              • Updating Oracle ASM
              • Installing an Oracle Database
              • Updating the Database Home
              • Configuring Listener Names
              • Creating and Configuring a Database
                • Conclusion
                • Contact Information
                  • Appendix A References
                    • AIX
                    • IBM Storage
                    • Oracle
                      • Appendix B Troubleshooting
                      • Copyrights13

Reference Configuration

For an IBM storage subsystem reserve_policy=no_reserve is the correct value

a Use the lsattr -E -l hdisk | grep reserve command to check the attribute

b Use the chdev -l hdisk -a reserve_policy=no_reserve command to change the policy

c Repeat step a and step b on each cluster node

5 Set ownership and permissions on devnodes to be used by Oracle

NOTE If you created new devnodes for the shared logical drives set the ownership and permissions on the new devnode that you created not on the original hdisk devnode

a Use the following commands to set the ownership for each OCR device file

chown rootoinstall devltdevnodegt

chmod 640 devltdevnodegt

b Use the following commands to set the ownership and permission on the voting disk and database device files

chown oracledba devltdevnodegt

chmod 660 devltdevnodegt

c Repeat step a and step b on each cluster node

NOTE Do not assign a physical volume identifier (PVID) to hdisks that will be used with Oracle ASM

Installing OracleHere are the basic steps for installing Oracle

bull Create directories for Oracle

bull Set up the Oracle userrsquos profile

bull Install Oracle Clusterware

bull Update Oracle CRS

bull Install and configure Oracle ASM

bull Update Oracle ASM

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 11copy Copyright 2008 IBM Corporation

Installing Oracle Clusterware

bull Install Oracle Database

bull Update the database home

bull Configure the listener names

bull Create and configure a database

Creating Directories for OracleAlthough you can create directory structures for Oracle files in many different ways for ease of support use a standard directory structure based on the Optimal Flexible Architecture (OFA) guidelines published by Oracle For more information about the OFA Guidelines see Appendix D of the Oracle Database Installation Guide 10gR2 for AIX 5L Based Systems (B19075-02) The reference configuration is based on the OFA guidelines

Follow these two important guidelines when creating directories for Oracle

bull You must separate the Oracle CRS home directory from the Oracle base and the database home directories The Oracle CRS installation procedure sets the ownership for the entire Oracle CRS directory tree to root while the database directory tree must be owned by oracle

bull If more than one database will be serviced by Oracle ASM it is recommended that you create a separate home directory for Oracle ASM An Oracle ASM home directory provides the capability to modify the Oracle release for Oracle ASM while not changing the release of any of the databases on the same server The directory tree must be owned by oracle

Setting Up the Oracle Users Profile1 Set up the Oracle users profile before installing the Oracle binaries See

Appendix A for an example profile

2 Add or modify entries for ORACLE_HOME ORACLE_BASE PATH LD_LIBRARY_PATH and similar files to reflect the directories where the Oracle binaries will be placed during the Oracle Clusterware installation

Installing Oracle ClusterwareAs you install the Oracle Clusterware remember the following important points

bull Do not make the CRS_HOME path a subdirectory of the Oracle RAC 10g Release 2 Oracle base directory due to permission changes required for Oracle CRS For more details refer to Appendix E of the Oracle Clusterware Installation Guide

bull The rootsh script must be run as the root user and must be run on only one node at a time not concurrently

12 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Updating Oracle CRS

bull After rootsh runs on the last node it will attempt to run the vipca utility to configure the VIP network interfaces If you have IP addresses that are not routable across the Internet for example 100 192168 the vipca configuration will fail If the automatic vipca configuration fails run the vipca utility manually using the $ORA_CRS_HOMEbinvipca command from the command line as root This requires a graphical display

bull You can do the manual configuration with vipca before clicking OK on the Oracle Universal Installer the Oracle CRS installation page The Oracle CRS installation page waits for you to run rootsh Do the manual configuration with vipca during this time This action allows the remainder of the post-installation checks to complete successfully

Updating Oracle CRSRefer to the Metalink Bulletin 2820361 (see important note on page 5) for required updates The reference configuration used the 10203 update to the Oracle CRS home Download the 10203 update and any other updates required from the Oracle Metalink web site (httpmetalinkoraclecom) and install according to the instructions on the web site

Installing and Configuring Oracle ASMBecause a single Oracle ASM instance can service many database instances Oracle recommends that you install Oracle ASM in its own home directory Follow the naming convention of the OFA guidelines for the directory path As you install and configure Oracle ASM remember the following important points

bull If the Oracle ASM instance was not correctly created by the Oracle Universal Installer Wizard during installation use the dbca utility to add the instance and create the disk group(s)

bull If you want additional disk groups create them using the dbca utility

Updating Oracle ASMUpdate the Oracle ASM instance with the 10203 patch set and any additional updates indicated in Metalink Bulletin 2820361 You can use the same update media for 10203 previously downloaded to update Oracle CRS (page 13) Carefully follow the instructions for updating this database instance

Installing an Oracle DatabaseUse the Oracle Universal Installer to install another database home for a database instance Select Install Database Software Only on the Create Database page

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 13copy Copyright 2008 IBM Corporation

Installing Oracle Clusterware

Updating the Database HomeInstall the 10203 update on the new database home Select the database home instead of Oracle ASM on the Specify Home Details page

Configuring Listener NamesIf your RAC environment consists of only Oracle CRS an Oracle ASM instance and a database instance on each node it is easier to support only one listenerora configuration file for the Oracle ASM and database instances The best way to accomplish this is to create an environment variable TNS_ADMIN and assign it to the $ORACLE_HOMEnetworkadmin directory of the database instance For example

export TNS_ADMIN=$ORA_ASM_HOMEnetworkadmin

This environment variable allows the DBA to maintain one set of tnsnamesora sqlnetora and listenerora files for the entire environment

Creating and Configuring a DatabaseUse the Database Configuration Assistant (DBCA) to create a database For more information refer to the Oracle documentation about using DBCA As you create the database remember the following important points

bull Using DBCA gives you the capability to create the Oracle ASM disk groups if you choose to use Oracle ASM

bull The Flash Recovery Area (FRA) option is highly recommended A FRA allows the setup of a separate area on disk for flashback capability area for the archive logs the second copy of the multiplexed redo log files and for RMAN backups Make sure that the FRA is correctly set up to handle the performance of the database and that the FRA contains enough space

Space to hold all of the data for the data retention periods that are set by the database

Space for the type of backups that will be used

Space for the number of backups that will be held

For further documentation about using the FRA refer to the Oracle documentation as well as the following white paper

Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to

wwwibmcomsupporttechdocs

14 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Creating and Configuring a Database

ConclusionAs you have followed the example of setting up a two-node cluster described in this document you have seen how simple and how effective a deployment of Oracle RAC 10g Release 2 can be When you are doing your own installation be sure to use the applicable installation information as the basis for your work This document describes only tips and techniques Your installer should have the expertise to know how and when to modify instructions as necessary If your environment is complex and you require outside help contact your IBM business partner or sales representative See ldquoContact Informationrdquo below

Contact InformationFor more information and sales office locations visit the IBM web site at

httpwwwibmcom

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 15copy Copyright 2008 IBM Corporation

Contact Information

16 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix A

References

AIXFor AIX operating system downloads go to Quick Links for AIX Fixes

httpwww-933ibmcomeserversupportfixesfixcentralmainpseriesaix

For AIX documentation go to AIX Information

httppublibboulderibmcominfocentersystemsindexjsptopic=comibmaixdocdocbaseaixparenthtm

IBM Corporation 2006 Get the Latest Version of OpenSSH for AIX

httpwwwibmcomdeveloperworkseserverarticlesopenssh_updatedhtml

IBM StorageFor information about the IBM DS4800 Storage Subsystem go to

httpwww-03ibmcomsystemsstoragediskds4000ds4800indexhtml

For information about IBM storage go to

httpwww-03ibmcomsystemsstoragediskcm_re=masthead-_-products-_-stg-disk

OracleFor Oracle documentation go to

httpwwworaclecomtechnologydocumentationindexhtml

IBM Corporation 2008 Tuning IBM AIX 5L for an Oracle Database

httpwwwibmcompartnerworldwpsservletContentHandlerVPAA-6V2C4E

IBM Corporation 2008 Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems

wwwibmcomsupporttechdocs

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem A-1copy Copyright 2008 IBM Corporation

A-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix B

Troubleshooting

CSS Node Evictions can affect your functionality

IO time-outs can cause inappropriate node evictions during multi-path failover or other problems Two parameters might affect this css_misscount and disktimeout With versions of Oracle prior to 102 this was primarily accommodated by increasing the value of the css_misscount parameter However the function of this parameter has changed as of Oracle 102 and changes to this parameter might not achieve the desired result The new disktimeout parameter should be used in most cases to make this adjustment In many cases the default values will work fine However you might find it necessary to increase the default value of this parameter if you are getting a lot of CSS node evictions

For information about how the parameters work and how to adjust them refer to the following two articles Go to httpmetalinkoraclecom

bull Metalink DocID 2944301 CSS Timeout Computation in RAC 10g

bull Metalink DocID 2847521 Steps to Increase CSS Misscount Reboottime and Disktimeout

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem B-1copy Copyright 2008 IBM Corporation

B-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Trademarks and special notices copy Copyright IBM Corporation 2008 All rights reserved

References in this document to IBM products or services do not imply that IBM intends to make them available in every country

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corporation in the United States other countries or both If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (reg or trade) these symbols indicate US registered or common law trademarks owned by IBM at the time this information was published Such trademarks may also be registered or common law trademarks in other countries A current list of IBM trademarks is available on the Web at Copyright and trademark information at wwwibmcomlegalcopytradeshtml

Intel Intel Inside (logos) MMX and Pentium are trademarks of Intel Corporation in the United States other countries or both

UNIX is a registered trademark of The Open Group in the United States and other countries

Linux is a trademark of Linus Torvalds in the United States other countries or both

Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both

Other company product or service names may be trademarks or service marks of others

Information is provided AS IS without warranty of any kind

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved Actual environmental costs and performance characteristics may vary by customer

Information concerning non-IBM products was obtained from a supplier of these products published announcement material or other publicly available sources and does not constitute an endorsement of such products by IBM Sources for non-IBM list prices and performance numbers are taken from publicly available information including vendor announcements and vendor worldwide homepages IBM has not tested these products and cannot confirm the accuracy of performance capability or any other claims related to non-IBM products Questions on the capability of non-IBM products should be addressed to the supplier of those products

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk

  • Title Page13
  • Contents
  • Deploying Oracle RAC 10g Release 2 with ASM on AIX 53 with an IBM DS4700 or DS4800 Storage Subsystem
    • Introduction
      • Required Documents
      • Benefits
      • Reference Configuration
        • Configuring the Hosts
        • Configuring the IBM Storage
        • Installing Oracle
          • Creating Directories for Oracle
          • Setting Up the Oracle Users Profile
            • Installing Oracle Clusterware
              • Updating Oracle CRS
              • Installing and Configuring Oracle ASM
              • Updating Oracle ASM
              • Installing an Oracle Database
              • Updating the Database Home
              • Configuring Listener Names
              • Creating and Configuring a Database
                • Conclusion
                • Contact Information
                  • Appendix A References
                    • AIX
                    • IBM Storage
                    • Oracle
                      • Appendix B Troubleshooting
                      • Copyrights13

Installing Oracle Clusterware

bull Install Oracle Database

bull Update the database home

bull Configure the listener names

bull Create and configure a database

Creating Directories for OracleAlthough you can create directory structures for Oracle files in many different ways for ease of support use a standard directory structure based on the Optimal Flexible Architecture (OFA) guidelines published by Oracle For more information about the OFA Guidelines see Appendix D of the Oracle Database Installation Guide 10gR2 for AIX 5L Based Systems (B19075-02) The reference configuration is based on the OFA guidelines

Follow these two important guidelines when creating directories for Oracle

bull You must separate the Oracle CRS home directory from the Oracle base and the database home directories The Oracle CRS installation procedure sets the ownership for the entire Oracle CRS directory tree to root while the database directory tree must be owned by oracle

bull If more than one database will be serviced by Oracle ASM it is recommended that you create a separate home directory for Oracle ASM An Oracle ASM home directory provides the capability to modify the Oracle release for Oracle ASM while not changing the release of any of the databases on the same server The directory tree must be owned by oracle

Setting Up the Oracle Users Profile1 Set up the Oracle users profile before installing the Oracle binaries See

Appendix A for an example profile

2 Add or modify entries for ORACLE_HOME ORACLE_BASE PATH LD_LIBRARY_PATH and similar files to reflect the directories where the Oracle binaries will be placed during the Oracle Clusterware installation

Installing Oracle ClusterwareAs you install the Oracle Clusterware remember the following important points

bull Do not make the CRS_HOME path a subdirectory of the Oracle RAC 10g Release 2 Oracle base directory due to permission changes required for Oracle CRS For more details refer to Appendix E of the Oracle Clusterware Installation Guide

bull The rootsh script must be run as the root user and must be run on only one node at a time not concurrently

12 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Updating Oracle CRS

bull After rootsh runs on the last node it will attempt to run the vipca utility to configure the VIP network interfaces If you have IP addresses that are not routable across the Internet for example 100 192168 the vipca configuration will fail If the automatic vipca configuration fails run the vipca utility manually using the $ORA_CRS_HOMEbinvipca command from the command line as root This requires a graphical display

bull You can do the manual configuration with vipca before clicking OK on the Oracle Universal Installer the Oracle CRS installation page The Oracle CRS installation page waits for you to run rootsh Do the manual configuration with vipca during this time This action allows the remainder of the post-installation checks to complete successfully

Updating Oracle CRSRefer to the Metalink Bulletin 2820361 (see important note on page 5) for required updates The reference configuration used the 10203 update to the Oracle CRS home Download the 10203 update and any other updates required from the Oracle Metalink web site (httpmetalinkoraclecom) and install according to the instructions on the web site

Installing and Configuring Oracle ASMBecause a single Oracle ASM instance can service many database instances Oracle recommends that you install Oracle ASM in its own home directory Follow the naming convention of the OFA guidelines for the directory path As you install and configure Oracle ASM remember the following important points

bull If the Oracle ASM instance was not correctly created by the Oracle Universal Installer Wizard during installation use the dbca utility to add the instance and create the disk group(s)

bull If you want additional disk groups create them using the dbca utility

Updating Oracle ASMUpdate the Oracle ASM instance with the 10203 patch set and any additional updates indicated in Metalink Bulletin 2820361 You can use the same update media for 10203 previously downloaded to update Oracle CRS (page 13) Carefully follow the instructions for updating this database instance

Installing an Oracle DatabaseUse the Oracle Universal Installer to install another database home for a database instance Select Install Database Software Only on the Create Database page

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 13copy Copyright 2008 IBM Corporation

Installing Oracle Clusterware

Updating the Database HomeInstall the 10203 update on the new database home Select the database home instead of Oracle ASM on the Specify Home Details page

Configuring Listener NamesIf your RAC environment consists of only Oracle CRS an Oracle ASM instance and a database instance on each node it is easier to support only one listenerora configuration file for the Oracle ASM and database instances The best way to accomplish this is to create an environment variable TNS_ADMIN and assign it to the $ORACLE_HOMEnetworkadmin directory of the database instance For example

export TNS_ADMIN=$ORA_ASM_HOMEnetworkadmin

This environment variable allows the DBA to maintain one set of tnsnamesora sqlnetora and listenerora files for the entire environment

Creating and Configuring a DatabaseUse the Database Configuration Assistant (DBCA) to create a database For more information refer to the Oracle documentation about using DBCA As you create the database remember the following important points

bull Using DBCA gives you the capability to create the Oracle ASM disk groups if you choose to use Oracle ASM

bull The Flash Recovery Area (FRA) option is highly recommended A FRA allows the setup of a separate area on disk for flashback capability area for the archive logs the second copy of the multiplexed redo log files and for RMAN backups Make sure that the FRA is correctly set up to handle the performance of the database and that the FRA contains enough space

Space to hold all of the data for the data retention periods that are set by the database

Space for the type of backups that will be used

Space for the number of backups that will be held

For further documentation about using the FRA refer to the Oracle documentation as well as the following white paper

Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to

wwwibmcomsupporttechdocs

14 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Creating and Configuring a Database

ConclusionAs you have followed the example of setting up a two-node cluster described in this document you have seen how simple and how effective a deployment of Oracle RAC 10g Release 2 can be When you are doing your own installation be sure to use the applicable installation information as the basis for your work This document describes only tips and techniques Your installer should have the expertise to know how and when to modify instructions as necessary If your environment is complex and you require outside help contact your IBM business partner or sales representative See ldquoContact Informationrdquo below

Contact InformationFor more information and sales office locations visit the IBM web site at

httpwwwibmcom

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 15copy Copyright 2008 IBM Corporation

Contact Information

16 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix A

References

AIXFor AIX operating system downloads go to Quick Links for AIX Fixes

httpwww-933ibmcomeserversupportfixesfixcentralmainpseriesaix

For AIX documentation go to AIX Information

httppublibboulderibmcominfocentersystemsindexjsptopic=comibmaixdocdocbaseaixparenthtm

IBM Corporation 2006 Get the Latest Version of OpenSSH for AIX

httpwwwibmcomdeveloperworkseserverarticlesopenssh_updatedhtml

IBM StorageFor information about the IBM DS4800 Storage Subsystem go to

httpwww-03ibmcomsystemsstoragediskds4000ds4800indexhtml

For information about IBM storage go to

httpwww-03ibmcomsystemsstoragediskcm_re=masthead-_-products-_-stg-disk

OracleFor Oracle documentation go to

httpwwworaclecomtechnologydocumentationindexhtml

IBM Corporation 2008 Tuning IBM AIX 5L for an Oracle Database

httpwwwibmcompartnerworldwpsservletContentHandlerVPAA-6V2C4E

IBM Corporation 2008 Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems

wwwibmcomsupporttechdocs

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem A-1copy Copyright 2008 IBM Corporation

A-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix B

Troubleshooting

CSS Node Evictions can affect your functionality

IO time-outs can cause inappropriate node evictions during multi-path failover or other problems Two parameters might affect this css_misscount and disktimeout With versions of Oracle prior to 102 this was primarily accommodated by increasing the value of the css_misscount parameter However the function of this parameter has changed as of Oracle 102 and changes to this parameter might not achieve the desired result The new disktimeout parameter should be used in most cases to make this adjustment In many cases the default values will work fine However you might find it necessary to increase the default value of this parameter if you are getting a lot of CSS node evictions

For information about how the parameters work and how to adjust them refer to the following two articles Go to httpmetalinkoraclecom

bull Metalink DocID 2944301 CSS Timeout Computation in RAC 10g

bull Metalink DocID 2847521 Steps to Increase CSS Misscount Reboottime and Disktimeout

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem B-1copy Copyright 2008 IBM Corporation

B-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Trademarks and special notices copy Copyright IBM Corporation 2008 All rights reserved

References in this document to IBM products or services do not imply that IBM intends to make them available in every country

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corporation in the United States other countries or both If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (reg or trade) these symbols indicate US registered or common law trademarks owned by IBM at the time this information was published Such trademarks may also be registered or common law trademarks in other countries A current list of IBM trademarks is available on the Web at Copyright and trademark information at wwwibmcomlegalcopytradeshtml

Intel Intel Inside (logos) MMX and Pentium are trademarks of Intel Corporation in the United States other countries or both

UNIX is a registered trademark of The Open Group in the United States and other countries

Linux is a trademark of Linus Torvalds in the United States other countries or both

Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both

Other company product or service names may be trademarks or service marks of others

Information is provided AS IS without warranty of any kind

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved Actual environmental costs and performance characteristics may vary by customer

Information concerning non-IBM products was obtained from a supplier of these products published announcement material or other publicly available sources and does not constitute an endorsement of such products by IBM Sources for non-IBM list prices and performance numbers are taken from publicly available information including vendor announcements and vendor worldwide homepages IBM has not tested these products and cannot confirm the accuracy of performance capability or any other claims related to non-IBM products Questions on the capability of non-IBM products should be addressed to the supplier of those products

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk

  • Title Page13
  • Contents
  • Deploying Oracle RAC 10g Release 2 with ASM on AIX 53 with an IBM DS4700 or DS4800 Storage Subsystem
    • Introduction
      • Required Documents
      • Benefits
      • Reference Configuration
        • Configuring the Hosts
        • Configuring the IBM Storage
        • Installing Oracle
          • Creating Directories for Oracle
          • Setting Up the Oracle Users Profile
            • Installing Oracle Clusterware
              • Updating Oracle CRS
              • Installing and Configuring Oracle ASM
              • Updating Oracle ASM
              • Installing an Oracle Database
              • Updating the Database Home
              • Configuring Listener Names
              • Creating and Configuring a Database
                • Conclusion
                • Contact Information
                  • Appendix A References
                    • AIX
                    • IBM Storage
                    • Oracle
                      • Appendix B Troubleshooting
                      • Copyrights13

Updating Oracle CRS

bull After rootsh runs on the last node it will attempt to run the vipca utility to configure the VIP network interfaces If you have IP addresses that are not routable across the Internet for example 100 192168 the vipca configuration will fail If the automatic vipca configuration fails run the vipca utility manually using the $ORA_CRS_HOMEbinvipca command from the command line as root This requires a graphical display

bull You can do the manual configuration with vipca before clicking OK on the Oracle Universal Installer the Oracle CRS installation page The Oracle CRS installation page waits for you to run rootsh Do the manual configuration with vipca during this time This action allows the remainder of the post-installation checks to complete successfully

Updating Oracle CRSRefer to the Metalink Bulletin 2820361 (see important note on page 5) for required updates The reference configuration used the 10203 update to the Oracle CRS home Download the 10203 update and any other updates required from the Oracle Metalink web site (httpmetalinkoraclecom) and install according to the instructions on the web site

Installing and Configuring Oracle ASMBecause a single Oracle ASM instance can service many database instances Oracle recommends that you install Oracle ASM in its own home directory Follow the naming convention of the OFA guidelines for the directory path As you install and configure Oracle ASM remember the following important points

bull If the Oracle ASM instance was not correctly created by the Oracle Universal Installer Wizard during installation use the dbca utility to add the instance and create the disk group(s)

bull If you want additional disk groups create them using the dbca utility

Updating Oracle ASMUpdate the Oracle ASM instance with the 10203 patch set and any additional updates indicated in Metalink Bulletin 2820361 You can use the same update media for 10203 previously downloaded to update Oracle CRS (page 13) Carefully follow the instructions for updating this database instance

Installing an Oracle DatabaseUse the Oracle Universal Installer to install another database home for a database instance Select Install Database Software Only on the Create Database page

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 13copy Copyright 2008 IBM Corporation

Installing Oracle Clusterware

Updating the Database HomeInstall the 10203 update on the new database home Select the database home instead of Oracle ASM on the Specify Home Details page

Configuring Listener NamesIf your RAC environment consists of only Oracle CRS an Oracle ASM instance and a database instance on each node it is easier to support only one listenerora configuration file for the Oracle ASM and database instances The best way to accomplish this is to create an environment variable TNS_ADMIN and assign it to the $ORACLE_HOMEnetworkadmin directory of the database instance For example

export TNS_ADMIN=$ORA_ASM_HOMEnetworkadmin

This environment variable allows the DBA to maintain one set of tnsnamesora sqlnetora and listenerora files for the entire environment

Creating and Configuring a DatabaseUse the Database Configuration Assistant (DBCA) to create a database For more information refer to the Oracle documentation about using DBCA As you create the database remember the following important points

bull Using DBCA gives you the capability to create the Oracle ASM disk groups if you choose to use Oracle ASM

bull The Flash Recovery Area (FRA) option is highly recommended A FRA allows the setup of a separate area on disk for flashback capability area for the archive logs the second copy of the multiplexed redo log files and for RMAN backups Make sure that the FRA is correctly set up to handle the performance of the database and that the FRA contains enough space

Space to hold all of the data for the data retention periods that are set by the database

Space for the type of backups that will be used

Space for the number of backups that will be held

For further documentation about using the FRA refer to the Oracle documentation as well as the following white paper

Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to

wwwibmcomsupporttechdocs

14 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Creating and Configuring a Database

ConclusionAs you have followed the example of setting up a two-node cluster described in this document you have seen how simple and how effective a deployment of Oracle RAC 10g Release 2 can be When you are doing your own installation be sure to use the applicable installation information as the basis for your work This document describes only tips and techniques Your installer should have the expertise to know how and when to modify instructions as necessary If your environment is complex and you require outside help contact your IBM business partner or sales representative See ldquoContact Informationrdquo below

Contact InformationFor more information and sales office locations visit the IBM web site at

httpwwwibmcom

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 15copy Copyright 2008 IBM Corporation

Contact Information

16 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix A

References

AIXFor AIX operating system downloads go to Quick Links for AIX Fixes

httpwww-933ibmcomeserversupportfixesfixcentralmainpseriesaix

For AIX documentation go to AIX Information

httppublibboulderibmcominfocentersystemsindexjsptopic=comibmaixdocdocbaseaixparenthtm

IBM Corporation 2006 Get the Latest Version of OpenSSH for AIX

httpwwwibmcomdeveloperworkseserverarticlesopenssh_updatedhtml

IBM StorageFor information about the IBM DS4800 Storage Subsystem go to

httpwww-03ibmcomsystemsstoragediskds4000ds4800indexhtml

For information about IBM storage go to

httpwww-03ibmcomsystemsstoragediskcm_re=masthead-_-products-_-stg-disk

OracleFor Oracle documentation go to

httpwwworaclecomtechnologydocumentationindexhtml

IBM Corporation 2008 Tuning IBM AIX 5L for an Oracle Database

httpwwwibmcompartnerworldwpsservletContentHandlerVPAA-6V2C4E

IBM Corporation 2008 Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems

wwwibmcomsupporttechdocs

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem A-1copy Copyright 2008 IBM Corporation

A-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix B

Troubleshooting

CSS Node Evictions can affect your functionality

IO time-outs can cause inappropriate node evictions during multi-path failover or other problems Two parameters might affect this css_misscount and disktimeout With versions of Oracle prior to 102 this was primarily accommodated by increasing the value of the css_misscount parameter However the function of this parameter has changed as of Oracle 102 and changes to this parameter might not achieve the desired result The new disktimeout parameter should be used in most cases to make this adjustment In many cases the default values will work fine However you might find it necessary to increase the default value of this parameter if you are getting a lot of CSS node evictions

For information about how the parameters work and how to adjust them refer to the following two articles Go to httpmetalinkoraclecom

bull Metalink DocID 2944301 CSS Timeout Computation in RAC 10g

bull Metalink DocID 2847521 Steps to Increase CSS Misscount Reboottime and Disktimeout

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem B-1copy Copyright 2008 IBM Corporation

B-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Trademarks and special notices copy Copyright IBM Corporation 2008 All rights reserved

References in this document to IBM products or services do not imply that IBM intends to make them available in every country

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corporation in the United States other countries or both If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (reg or trade) these symbols indicate US registered or common law trademarks owned by IBM at the time this information was published Such trademarks may also be registered or common law trademarks in other countries A current list of IBM trademarks is available on the Web at Copyright and trademark information at wwwibmcomlegalcopytradeshtml

Intel Intel Inside (logos) MMX and Pentium are trademarks of Intel Corporation in the United States other countries or both

UNIX is a registered trademark of The Open Group in the United States and other countries

Linux is a trademark of Linus Torvalds in the United States other countries or both

Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both

Other company product or service names may be trademarks or service marks of others

Information is provided AS IS without warranty of any kind

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved Actual environmental costs and performance characteristics may vary by customer

Information concerning non-IBM products was obtained from a supplier of these products published announcement material or other publicly available sources and does not constitute an endorsement of such products by IBM Sources for non-IBM list prices and performance numbers are taken from publicly available information including vendor announcements and vendor worldwide homepages IBM has not tested these products and cannot confirm the accuracy of performance capability or any other claims related to non-IBM products Questions on the capability of non-IBM products should be addressed to the supplier of those products

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk

  • Title Page13
  • Contents
  • Deploying Oracle RAC 10g Release 2 with ASM on AIX 53 with an IBM DS4700 or DS4800 Storage Subsystem
    • Introduction
      • Required Documents
      • Benefits
      • Reference Configuration
        • Configuring the Hosts
        • Configuring the IBM Storage
        • Installing Oracle
          • Creating Directories for Oracle
          • Setting Up the Oracle Users Profile
            • Installing Oracle Clusterware
              • Updating Oracle CRS
              • Installing and Configuring Oracle ASM
              • Updating Oracle ASM
              • Installing an Oracle Database
              • Updating the Database Home
              • Configuring Listener Names
              • Creating and Configuring a Database
                • Conclusion
                • Contact Information
                  • Appendix A References
                    • AIX
                    • IBM Storage
                    • Oracle
                      • Appendix B Troubleshooting
                      • Copyrights13

Installing Oracle Clusterware

Updating the Database HomeInstall the 10203 update on the new database home Select the database home instead of Oracle ASM on the Specify Home Details page

Configuring Listener NamesIf your RAC environment consists of only Oracle CRS an Oracle ASM instance and a database instance on each node it is easier to support only one listenerora configuration file for the Oracle ASM and database instances The best way to accomplish this is to create an environment variable TNS_ADMIN and assign it to the $ORACLE_HOMEnetworkadmin directory of the database instance For example

export TNS_ADMIN=$ORA_ASM_HOMEnetworkadmin

This environment variable allows the DBA to maintain one set of tnsnamesora sqlnetora and listenerora files for the entire environment

Creating and Configuring a DatabaseUse the Database Configuration Assistant (DBCA) to create a database For more information refer to the Oracle documentation about using DBCA As you create the database remember the following important points

bull Using DBCA gives you the capability to create the Oracle ASM disk groups if you choose to use Oracle ASM

bull The Flash Recovery Area (FRA) option is highly recommended A FRA allows the setup of a separate area on disk for flashback capability area for the archive logs the second copy of the multiplexed redo log files and for RMAN backups Make sure that the FRA is correctly set up to handle the performance of the database and that the FRA contains enough space

Space to hold all of the data for the data retention periods that are set by the database

Space for the type of backups that will be used

Space for the number of backups that will be held

For further documentation about using the FRA refer to the Oracle documentation as well as the following white paper

Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems Go to

wwwibmcomsupporttechdocs

14 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Creating and Configuring a Database

ConclusionAs you have followed the example of setting up a two-node cluster described in this document you have seen how simple and how effective a deployment of Oracle RAC 10g Release 2 can be When you are doing your own installation be sure to use the applicable installation information as the basis for your work This document describes only tips and techniques Your installer should have the expertise to know how and when to modify instructions as necessary If your environment is complex and you require outside help contact your IBM business partner or sales representative See ldquoContact Informationrdquo below

Contact InformationFor more information and sales office locations visit the IBM web site at

httpwwwibmcom

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 15copy Copyright 2008 IBM Corporation

Contact Information

16 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix A

References

AIXFor AIX operating system downloads go to Quick Links for AIX Fixes

httpwww-933ibmcomeserversupportfixesfixcentralmainpseriesaix

For AIX documentation go to AIX Information

httppublibboulderibmcominfocentersystemsindexjsptopic=comibmaixdocdocbaseaixparenthtm

IBM Corporation 2006 Get the Latest Version of OpenSSH for AIX

httpwwwibmcomdeveloperworkseserverarticlesopenssh_updatedhtml

IBM StorageFor information about the IBM DS4800 Storage Subsystem go to

httpwww-03ibmcomsystemsstoragediskds4000ds4800indexhtml

For information about IBM storage go to

httpwww-03ibmcomsystemsstoragediskcm_re=masthead-_-products-_-stg-disk

OracleFor Oracle documentation go to

httpwwworaclecomtechnologydocumentationindexhtml

IBM Corporation 2008 Tuning IBM AIX 5L for an Oracle Database

httpwwwibmcompartnerworldwpsservletContentHandlerVPAA-6V2C4E

IBM Corporation 2008 Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems

wwwibmcomsupporttechdocs

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem A-1copy Copyright 2008 IBM Corporation

A-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix B

Troubleshooting

CSS Node Evictions can affect your functionality

IO time-outs can cause inappropriate node evictions during multi-path failover or other problems Two parameters might affect this css_misscount and disktimeout With versions of Oracle prior to 102 this was primarily accommodated by increasing the value of the css_misscount parameter However the function of this parameter has changed as of Oracle 102 and changes to this parameter might not achieve the desired result The new disktimeout parameter should be used in most cases to make this adjustment In many cases the default values will work fine However you might find it necessary to increase the default value of this parameter if you are getting a lot of CSS node evictions

For information about how the parameters work and how to adjust them refer to the following two articles Go to httpmetalinkoraclecom

bull Metalink DocID 2944301 CSS Timeout Computation in RAC 10g

bull Metalink DocID 2847521 Steps to Increase CSS Misscount Reboottime and Disktimeout

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem B-1copy Copyright 2008 IBM Corporation

B-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Trademarks and special notices copy Copyright IBM Corporation 2008 All rights reserved

References in this document to IBM products or services do not imply that IBM intends to make them available in every country

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corporation in the United States other countries or both If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (reg or trade) these symbols indicate US registered or common law trademarks owned by IBM at the time this information was published Such trademarks may also be registered or common law trademarks in other countries A current list of IBM trademarks is available on the Web at Copyright and trademark information at wwwibmcomlegalcopytradeshtml

Intel Intel Inside (logos) MMX and Pentium are trademarks of Intel Corporation in the United States other countries or both

UNIX is a registered trademark of The Open Group in the United States and other countries

Linux is a trademark of Linus Torvalds in the United States other countries or both

Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both

Other company product or service names may be trademarks or service marks of others

Information is provided AS IS without warranty of any kind

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved Actual environmental costs and performance characteristics may vary by customer

Information concerning non-IBM products was obtained from a supplier of these products published announcement material or other publicly available sources and does not constitute an endorsement of such products by IBM Sources for non-IBM list prices and performance numbers are taken from publicly available information including vendor announcements and vendor worldwide homepages IBM has not tested these products and cannot confirm the accuracy of performance capability or any other claims related to non-IBM products Questions on the capability of non-IBM products should be addressed to the supplier of those products

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk

  • Title Page13
  • Contents
  • Deploying Oracle RAC 10g Release 2 with ASM on AIX 53 with an IBM DS4700 or DS4800 Storage Subsystem
    • Introduction
      • Required Documents
      • Benefits
      • Reference Configuration
        • Configuring the Hosts
        • Configuring the IBM Storage
        • Installing Oracle
          • Creating Directories for Oracle
          • Setting Up the Oracle Users Profile
            • Installing Oracle Clusterware
              • Updating Oracle CRS
              • Installing and Configuring Oracle ASM
              • Updating Oracle ASM
              • Installing an Oracle Database
              • Updating the Database Home
              • Configuring Listener Names
              • Creating and Configuring a Database
                • Conclusion
                • Contact Information
                  • Appendix A References
                    • AIX
                    • IBM Storage
                    • Oracle
                      • Appendix B Troubleshooting
                      • Copyrights13

Creating and Configuring a Database

ConclusionAs you have followed the example of setting up a two-node cluster described in this document you have seen how simple and how effective a deployment of Oracle RAC 10g Release 2 can be When you are doing your own installation be sure to use the applicable installation information as the basis for your work This document describes only tips and techniques Your installer should have the expertise to know how and when to modify instructions as necessary If your environment is complex and you require outside help contact your IBM business partner or sales representative See ldquoContact Informationrdquo below

Contact InformationFor more information and sales office locations visit the IBM web site at

httpwwwibmcom

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem 15copy Copyright 2008 IBM Corporation

Contact Information

16 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix A

References

AIXFor AIX operating system downloads go to Quick Links for AIX Fixes

httpwww-933ibmcomeserversupportfixesfixcentralmainpseriesaix

For AIX documentation go to AIX Information

httppublibboulderibmcominfocentersystemsindexjsptopic=comibmaixdocdocbaseaixparenthtm

IBM Corporation 2006 Get the Latest Version of OpenSSH for AIX

httpwwwibmcomdeveloperworkseserverarticlesopenssh_updatedhtml

IBM StorageFor information about the IBM DS4800 Storage Subsystem go to

httpwww-03ibmcomsystemsstoragediskds4000ds4800indexhtml

For information about IBM storage go to

httpwww-03ibmcomsystemsstoragediskcm_re=masthead-_-products-_-stg-disk

OracleFor Oracle documentation go to

httpwwworaclecomtechnologydocumentationindexhtml

IBM Corporation 2008 Tuning IBM AIX 5L for an Oracle Database

httpwwwibmcompartnerworldwpsservletContentHandlerVPAA-6V2C4E

IBM Corporation 2008 Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems

wwwibmcomsupporttechdocs

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem A-1copy Copyright 2008 IBM Corporation

A-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix B

Troubleshooting

CSS Node Evictions can affect your functionality

IO time-outs can cause inappropriate node evictions during multi-path failover or other problems Two parameters might affect this css_misscount and disktimeout With versions of Oracle prior to 102 this was primarily accommodated by increasing the value of the css_misscount parameter However the function of this parameter has changed as of Oracle 102 and changes to this parameter might not achieve the desired result The new disktimeout parameter should be used in most cases to make this adjustment In many cases the default values will work fine However you might find it necessary to increase the default value of this parameter if you are getting a lot of CSS node evictions

For information about how the parameters work and how to adjust them refer to the following two articles Go to httpmetalinkoraclecom

bull Metalink DocID 2944301 CSS Timeout Computation in RAC 10g

bull Metalink DocID 2847521 Steps to Increase CSS Misscount Reboottime and Disktimeout

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem B-1copy Copyright 2008 IBM Corporation

B-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Trademarks and special notices copy Copyright IBM Corporation 2008 All rights reserved

References in this document to IBM products or services do not imply that IBM intends to make them available in every country

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corporation in the United States other countries or both If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (reg or trade) these symbols indicate US registered or common law trademarks owned by IBM at the time this information was published Such trademarks may also be registered or common law trademarks in other countries A current list of IBM trademarks is available on the Web at Copyright and trademark information at wwwibmcomlegalcopytradeshtml

Intel Intel Inside (logos) MMX and Pentium are trademarks of Intel Corporation in the United States other countries or both

UNIX is a registered trademark of The Open Group in the United States and other countries

Linux is a trademark of Linus Torvalds in the United States other countries or both

Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both

Other company product or service names may be trademarks or service marks of others

Information is provided AS IS without warranty of any kind

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved Actual environmental costs and performance characteristics may vary by customer

Information concerning non-IBM products was obtained from a supplier of these products published announcement material or other publicly available sources and does not constitute an endorsement of such products by IBM Sources for non-IBM list prices and performance numbers are taken from publicly available information including vendor announcements and vendor worldwide homepages IBM has not tested these products and cannot confirm the accuracy of performance capability or any other claims related to non-IBM products Questions on the capability of non-IBM products should be addressed to the supplier of those products

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk

  • Title Page13
  • Contents
  • Deploying Oracle RAC 10g Release 2 with ASM on AIX 53 with an IBM DS4700 or DS4800 Storage Subsystem
    • Introduction
      • Required Documents
      • Benefits
      • Reference Configuration
        • Configuring the Hosts
        • Configuring the IBM Storage
        • Installing Oracle
          • Creating Directories for Oracle
          • Setting Up the Oracle Users Profile
            • Installing Oracle Clusterware
              • Updating Oracle CRS
              • Installing and Configuring Oracle ASM
              • Updating Oracle ASM
              • Installing an Oracle Database
              • Updating the Database Home
              • Configuring Listener Names
              • Creating and Configuring a Database
                • Conclusion
                • Contact Information
                  • Appendix A References
                    • AIX
                    • IBM Storage
                    • Oracle
                      • Appendix B Troubleshooting
                      • Copyrights13

Contact Information

16 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix A

References

AIXFor AIX operating system downloads go to Quick Links for AIX Fixes

httpwww-933ibmcomeserversupportfixesfixcentralmainpseriesaix

For AIX documentation go to AIX Information

httppublibboulderibmcominfocentersystemsindexjsptopic=comibmaixdocdocbaseaixparenthtm

IBM Corporation 2006 Get the Latest Version of OpenSSH for AIX

httpwwwibmcomdeveloperworkseserverarticlesopenssh_updatedhtml

IBM StorageFor information about the IBM DS4800 Storage Subsystem go to

httpwww-03ibmcomsystemsstoragediskds4000ds4800indexhtml

For information about IBM storage go to

httpwww-03ibmcomsystemsstoragediskcm_re=masthead-_-products-_-stg-disk

OracleFor Oracle documentation go to

httpwwworaclecomtechnologydocumentationindexhtml

IBM Corporation 2008 Tuning IBM AIX 5L for an Oracle Database

httpwwwibmcompartnerworldwpsservletContentHandlerVPAA-6V2C4E

IBM Corporation 2008 Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems

wwwibmcomsupporttechdocs

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem A-1copy Copyright 2008 IBM Corporation

A-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix B

Troubleshooting

CSS Node Evictions can affect your functionality

IO time-outs can cause inappropriate node evictions during multi-path failover or other problems Two parameters might affect this css_misscount and disktimeout With versions of Oracle prior to 102 this was primarily accommodated by increasing the value of the css_misscount parameter However the function of this parameter has changed as of Oracle 102 and changes to this parameter might not achieve the desired result The new disktimeout parameter should be used in most cases to make this adjustment In many cases the default values will work fine However you might find it necessary to increase the default value of this parameter if you are getting a lot of CSS node evictions

For information about how the parameters work and how to adjust them refer to the following two articles Go to httpmetalinkoraclecom

bull Metalink DocID 2944301 CSS Timeout Computation in RAC 10g

bull Metalink DocID 2847521 Steps to Increase CSS Misscount Reboottime and Disktimeout

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem B-1copy Copyright 2008 IBM Corporation

B-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Trademarks and special notices copy Copyright IBM Corporation 2008 All rights reserved

References in this document to IBM products or services do not imply that IBM intends to make them available in every country

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corporation in the United States other countries or both If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (reg or trade) these symbols indicate US registered or common law trademarks owned by IBM at the time this information was published Such trademarks may also be registered or common law trademarks in other countries A current list of IBM trademarks is available on the Web at Copyright and trademark information at wwwibmcomlegalcopytradeshtml

Intel Intel Inside (logos) MMX and Pentium are trademarks of Intel Corporation in the United States other countries or both

UNIX is a registered trademark of The Open Group in the United States and other countries

Linux is a trademark of Linus Torvalds in the United States other countries or both

Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both

Other company product or service names may be trademarks or service marks of others

Information is provided AS IS without warranty of any kind

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved Actual environmental costs and performance characteristics may vary by customer

Information concerning non-IBM products was obtained from a supplier of these products published announcement material or other publicly available sources and does not constitute an endorsement of such products by IBM Sources for non-IBM list prices and performance numbers are taken from publicly available information including vendor announcements and vendor worldwide homepages IBM has not tested these products and cannot confirm the accuracy of performance capability or any other claims related to non-IBM products Questions on the capability of non-IBM products should be addressed to the supplier of those products

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk

  • Title Page13
  • Contents
  • Deploying Oracle RAC 10g Release 2 with ASM on AIX 53 with an IBM DS4700 or DS4800 Storage Subsystem
    • Introduction
      • Required Documents
      • Benefits
      • Reference Configuration
        • Configuring the Hosts
        • Configuring the IBM Storage
        • Installing Oracle
          • Creating Directories for Oracle
          • Setting Up the Oracle Users Profile
            • Installing Oracle Clusterware
              • Updating Oracle CRS
              • Installing and Configuring Oracle ASM
              • Updating Oracle ASM
              • Installing an Oracle Database
              • Updating the Database Home
              • Configuring Listener Names
              • Creating and Configuring a Database
                • Conclusion
                • Contact Information
                  • Appendix A References
                    • AIX
                    • IBM Storage
                    • Oracle
                      • Appendix B Troubleshooting
                      • Copyrights13

Appendix A

References

AIXFor AIX operating system downloads go to Quick Links for AIX Fixes

httpwww-933ibmcomeserversupportfixesfixcentralmainpseriesaix

For AIX documentation go to AIX Information

httppublibboulderibmcominfocentersystemsindexjsptopic=comibmaixdocdocbaseaixparenthtm

IBM Corporation 2006 Get the Latest Version of OpenSSH for AIX

httpwwwibmcomdeveloperworkseserverarticlesopenssh_updatedhtml

IBM StorageFor information about the IBM DS4800 Storage Subsystem go to

httpwww-03ibmcomsystemsstoragediskds4000ds4800indexhtml

For information about IBM storage go to

httpwww-03ibmcomsystemsstoragediskcm_re=masthead-_-products-_-stg-disk

OracleFor Oracle documentation go to

httpwwworaclecomtechnologydocumentationindexhtml

IBM Corporation 2008 Tuning IBM AIX 5L for an Oracle Database

httpwwwibmcompartnerworldwpsservletContentHandlerVPAA-6V2C4E

IBM Corporation 2008 Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 Storage Subsystems

wwwibmcomsupporttechdocs

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem A-1copy Copyright 2008 IBM Corporation

A-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix B

Troubleshooting

CSS Node Evictions can affect your functionality

IO time-outs can cause inappropriate node evictions during multi-path failover or other problems Two parameters might affect this css_misscount and disktimeout With versions of Oracle prior to 102 this was primarily accommodated by increasing the value of the css_misscount parameter However the function of this parameter has changed as of Oracle 102 and changes to this parameter might not achieve the desired result The new disktimeout parameter should be used in most cases to make this adjustment In many cases the default values will work fine However you might find it necessary to increase the default value of this parameter if you are getting a lot of CSS node evictions

For information about how the parameters work and how to adjust them refer to the following two articles Go to httpmetalinkoraclecom

bull Metalink DocID 2944301 CSS Timeout Computation in RAC 10g

bull Metalink DocID 2847521 Steps to Increase CSS Misscount Reboottime and Disktimeout

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem B-1copy Copyright 2008 IBM Corporation

B-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Trademarks and special notices copy Copyright IBM Corporation 2008 All rights reserved

References in this document to IBM products or services do not imply that IBM intends to make them available in every country

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corporation in the United States other countries or both If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (reg or trade) these symbols indicate US registered or common law trademarks owned by IBM at the time this information was published Such trademarks may also be registered or common law trademarks in other countries A current list of IBM trademarks is available on the Web at Copyright and trademark information at wwwibmcomlegalcopytradeshtml

Intel Intel Inside (logos) MMX and Pentium are trademarks of Intel Corporation in the United States other countries or both

UNIX is a registered trademark of The Open Group in the United States and other countries

Linux is a trademark of Linus Torvalds in the United States other countries or both

Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both

Other company product or service names may be trademarks or service marks of others

Information is provided AS IS without warranty of any kind

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved Actual environmental costs and performance characteristics may vary by customer

Information concerning non-IBM products was obtained from a supplier of these products published announcement material or other publicly available sources and does not constitute an endorsement of such products by IBM Sources for non-IBM list prices and performance numbers are taken from publicly available information including vendor announcements and vendor worldwide homepages IBM has not tested these products and cannot confirm the accuracy of performance capability or any other claims related to non-IBM products Questions on the capability of non-IBM products should be addressed to the supplier of those products

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk

  • Title Page13
  • Contents
  • Deploying Oracle RAC 10g Release 2 with ASM on AIX 53 with an IBM DS4700 or DS4800 Storage Subsystem
    • Introduction
      • Required Documents
      • Benefits
      • Reference Configuration
        • Configuring the Hosts
        • Configuring the IBM Storage
        • Installing Oracle
          • Creating Directories for Oracle
          • Setting Up the Oracle Users Profile
            • Installing Oracle Clusterware
              • Updating Oracle CRS
              • Installing and Configuring Oracle ASM
              • Updating Oracle ASM
              • Installing an Oracle Database
              • Updating the Database Home
              • Configuring Listener Names
              • Creating and Configuring a Database
                • Conclusion
                • Contact Information
                  • Appendix A References
                    • AIX
                    • IBM Storage
                    • Oracle
                      • Appendix B Troubleshooting
                      • Copyrights13

A-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Appendix B

Troubleshooting

CSS Node Evictions can affect your functionality

IO time-outs can cause inappropriate node evictions during multi-path failover or other problems Two parameters might affect this css_misscount and disktimeout With versions of Oracle prior to 102 this was primarily accommodated by increasing the value of the css_misscount parameter However the function of this parameter has changed as of Oracle 102 and changes to this parameter might not achieve the desired result The new disktimeout parameter should be used in most cases to make this adjustment In many cases the default values will work fine However you might find it necessary to increase the default value of this parameter if you are getting a lot of CSS node evictions

For information about how the parameters work and how to adjust them refer to the following two articles Go to httpmetalinkoraclecom

bull Metalink DocID 2944301 CSS Timeout Computation in RAC 10g

bull Metalink DocID 2847521 Steps to Increase CSS Misscount Reboottime and Disktimeout

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem B-1copy Copyright 2008 IBM Corporation

B-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Trademarks and special notices copy Copyright IBM Corporation 2008 All rights reserved

References in this document to IBM products or services do not imply that IBM intends to make them available in every country

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corporation in the United States other countries or both If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (reg or trade) these symbols indicate US registered or common law trademarks owned by IBM at the time this information was published Such trademarks may also be registered or common law trademarks in other countries A current list of IBM trademarks is available on the Web at Copyright and trademark information at wwwibmcomlegalcopytradeshtml

Intel Intel Inside (logos) MMX and Pentium are trademarks of Intel Corporation in the United States other countries or both

UNIX is a registered trademark of The Open Group in the United States and other countries

Linux is a trademark of Linus Torvalds in the United States other countries or both

Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both

Other company product or service names may be trademarks or service marks of others

Information is provided AS IS without warranty of any kind

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved Actual environmental costs and performance characteristics may vary by customer

Information concerning non-IBM products was obtained from a supplier of these products published announcement material or other publicly available sources and does not constitute an endorsement of such products by IBM Sources for non-IBM list prices and performance numbers are taken from publicly available information including vendor announcements and vendor worldwide homepages IBM has not tested these products and cannot confirm the accuracy of performance capability or any other claims related to non-IBM products Questions on the capability of non-IBM products should be addressed to the supplier of those products

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk

  • Title Page13
  • Contents
  • Deploying Oracle RAC 10g Release 2 with ASM on AIX 53 with an IBM DS4700 or DS4800 Storage Subsystem
    • Introduction
      • Required Documents
      • Benefits
      • Reference Configuration
        • Configuring the Hosts
        • Configuring the IBM Storage
        • Installing Oracle
          • Creating Directories for Oracle
          • Setting Up the Oracle Users Profile
            • Installing Oracle Clusterware
              • Updating Oracle CRS
              • Installing and Configuring Oracle ASM
              • Updating Oracle ASM
              • Installing an Oracle Database
              • Updating the Database Home
              • Configuring Listener Names
              • Creating and Configuring a Database
                • Conclusion
                • Contact Information
                  • Appendix A References
                    • AIX
                    • IBM Storage
                    • Oracle
                      • Appendix B Troubleshooting
                      • Copyrights13

Appendix B

Troubleshooting

CSS Node Evictions can affect your functionality

IO time-outs can cause inappropriate node evictions during multi-path failover or other problems Two parameters might affect this css_misscount and disktimeout With versions of Oracle prior to 102 this was primarily accommodated by increasing the value of the css_misscount parameter However the function of this parameter has changed as of Oracle 102 and changes to this parameter might not achieve the desired result The new disktimeout parameter should be used in most cases to make this adjustment In many cases the default values will work fine However you might find it necessary to increase the default value of this parameter if you are getting a lot of CSS node evictions

For information about how the parameters work and how to adjust them refer to the following two articles Go to httpmetalinkoraclecom

bull Metalink DocID 2944301 CSS Timeout Computation in RAC 10g

bull Metalink DocID 2847521 Steps to Increase CSS Misscount Reboottime and Disktimeout

Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystem B-1copy Copyright 2008 IBM Corporation

B-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Trademarks and special notices copy Copyright IBM Corporation 2008 All rights reserved

References in this document to IBM products or services do not imply that IBM intends to make them available in every country

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corporation in the United States other countries or both If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (reg or trade) these symbols indicate US registered or common law trademarks owned by IBM at the time this information was published Such trademarks may also be registered or common law trademarks in other countries A current list of IBM trademarks is available on the Web at Copyright and trademark information at wwwibmcomlegalcopytradeshtml

Intel Intel Inside (logos) MMX and Pentium are trademarks of Intel Corporation in the United States other countries or both

UNIX is a registered trademark of The Open Group in the United States and other countries

Linux is a trademark of Linus Torvalds in the United States other countries or both

Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both

Other company product or service names may be trademarks or service marks of others

Information is provided AS IS without warranty of any kind

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved Actual environmental costs and performance characteristics may vary by customer

Information concerning non-IBM products was obtained from a supplier of these products published announcement material or other publicly available sources and does not constitute an endorsement of such products by IBM Sources for non-IBM list prices and performance numbers are taken from publicly available information including vendor announcements and vendor worldwide homepages IBM has not tested these products and cannot confirm the accuracy of performance capability or any other claims related to non-IBM products Questions on the capability of non-IBM products should be addressed to the supplier of those products

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk

  • Title Page13
  • Contents
  • Deploying Oracle RAC 10g Release 2 with ASM on AIX 53 with an IBM DS4700 or DS4800 Storage Subsystem
    • Introduction
      • Required Documents
      • Benefits
      • Reference Configuration
        • Configuring the Hosts
        • Configuring the IBM Storage
        • Installing Oracle
          • Creating Directories for Oracle
          • Setting Up the Oracle Users Profile
            • Installing Oracle Clusterware
              • Updating Oracle CRS
              • Installing and Configuring Oracle ASM
              • Updating Oracle ASM
              • Installing an Oracle Database
              • Updating the Database Home
              • Configuring Listener Names
              • Creating and Configuring a Database
                • Conclusion
                • Contact Information
                  • Appendix A References
                    • AIX
                    • IBM Storage
                    • Oracle
                      • Appendix B Troubleshooting
                      • Copyrights13

B-2 Deploying Oracle RAC 10gR2 with ASM on AIX 53 with an IBM Storage Subsystemcopy Copyright 2008 IBM Corporation

Trademarks and special notices copy Copyright IBM Corporation 2008 All rights reserved

References in this document to IBM products or services do not imply that IBM intends to make them available in every country

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corporation in the United States other countries or both If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (reg or trade) these symbols indicate US registered or common law trademarks owned by IBM at the time this information was published Such trademarks may also be registered or common law trademarks in other countries A current list of IBM trademarks is available on the Web at Copyright and trademark information at wwwibmcomlegalcopytradeshtml

Intel Intel Inside (logos) MMX and Pentium are trademarks of Intel Corporation in the United States other countries or both

UNIX is a registered trademark of The Open Group in the United States and other countries

Linux is a trademark of Linus Torvalds in the United States other countries or both

Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both

Other company product or service names may be trademarks or service marks of others

Information is provided AS IS without warranty of any kind

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved Actual environmental costs and performance characteristics may vary by customer

Information concerning non-IBM products was obtained from a supplier of these products published announcement material or other publicly available sources and does not constitute an endorsement of such products by IBM Sources for non-IBM list prices and performance numbers are taken from publicly available information including vendor announcements and vendor worldwide homepages IBM has not tested these products and cannot confirm the accuracy of performance capability or any other claims related to non-IBM products Questions on the capability of non-IBM products should be addressed to the supplier of those products

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk

  • Title Page13
  • Contents
  • Deploying Oracle RAC 10g Release 2 with ASM on AIX 53 with an IBM DS4700 or DS4800 Storage Subsystem
    • Introduction
      • Required Documents
      • Benefits
      • Reference Configuration
        • Configuring the Hosts
        • Configuring the IBM Storage
        • Installing Oracle
          • Creating Directories for Oracle
          • Setting Up the Oracle Users Profile
            • Installing Oracle Clusterware
              • Updating Oracle CRS
              • Installing and Configuring Oracle ASM
              • Updating Oracle ASM
              • Installing an Oracle Database
              • Updating the Database Home
              • Configuring Listener Names
              • Creating and Configuring a Database
                • Conclusion
                • Contact Information
                  • Appendix A References
                    • AIX
                    • IBM Storage
                    • Oracle
                      • Appendix B Troubleshooting
                      • Copyrights13

Trademarks and special notices copy Copyright IBM Corporation 2008 All rights reserved

References in this document to IBM products or services do not imply that IBM intends to make them available in every country

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corporation in the United States other countries or both If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (reg or trade) these symbols indicate US registered or common law trademarks owned by IBM at the time this information was published Such trademarks may also be registered or common law trademarks in other countries A current list of IBM trademarks is available on the Web at Copyright and trademark information at wwwibmcomlegalcopytradeshtml

Intel Intel Inside (logos) MMX and Pentium are trademarks of Intel Corporation in the United States other countries or both

UNIX is a registered trademark of The Open Group in the United States and other countries

Linux is a trademark of Linus Torvalds in the United States other countries or both

Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both

Other company product or service names may be trademarks or service marks of others

Information is provided AS IS without warranty of any kind

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved Actual environmental costs and performance characteristics may vary by customer

Information concerning non-IBM products was obtained from a supplier of these products published announcement material or other publicly available sources and does not constitute an endorsement of such products by IBM Sources for non-IBM list prices and performance numbers are taken from publicly available information including vendor announcements and vendor worldwide homepages IBM has not tested these products and cannot confirm the accuracy of performance capability or any other claims related to non-IBM products Questions on the capability of non-IBM products should be addressed to the supplier of those products

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk

  • Title Page13
  • Contents
  • Deploying Oracle RAC 10g Release 2 with ASM on AIX 53 with an IBM DS4700 or DS4800 Storage Subsystem
    • Introduction
      • Required Documents
      • Benefits
      • Reference Configuration
        • Configuring the Hosts
        • Configuring the IBM Storage
        • Installing Oracle
          • Creating Directories for Oracle
          • Setting Up the Oracle Users Profile
            • Installing Oracle Clusterware
              • Updating Oracle CRS
              • Installing and Configuring Oracle ASM
              • Updating Oracle ASM
              • Installing an Oracle Database
              • Updating the Database Home
              • Configuring Listener Names
              • Creating and Configuring a Database
                • Conclusion
                • Contact Information
                  • Appendix A References
                    • AIX
                    • IBM Storage
                    • Oracle
                      • Appendix B Troubleshooting
                      • Copyrights13
  • Title Page13
  • Contents
  • Deploying Oracle RAC 10g Release 2 with ASM on AIX 53 with an IBM DS4700 or DS4800 Storage Subsystem
    • Introduction
      • Required Documents
      • Benefits
      • Reference Configuration
        • Configuring the Hosts
        • Configuring the IBM Storage
        • Installing Oracle
          • Creating Directories for Oracle
          • Setting Up the Oracle Users Profile
            • Installing Oracle Clusterware
              • Updating Oracle CRS
              • Installing and Configuring Oracle ASM
              • Updating Oracle ASM
              • Installing an Oracle Database
              • Updating the Database Home
              • Configuring Listener Names
              • Creating and Configuring a Database
                • Conclusion
                • Contact Information
                  • Appendix A References
                    • AIX
                    • IBM Storage
                    • Oracle
                      • Appendix B Troubleshooting
                      • Copyrights13