Installing Oracle Database 10g With Real Application Cluster (RAC) on Red Hat Enterprise Linux...

8/18/2019 Installing Oracle Database 10g With Real Application Cluster (RAC) on Red Hat Enterprise Linux Advanced Server 3

1/20

Installing Oracle Database 10g with Real Application Cluster (RAC) on Red Hat Enterprise LinuxAdvanced Server 3

Written by Werner Puschitz

www.puschitz.com

The following procedure is a step-by-step guide (Cookbook) with tips and information for installing Oracle Database 10g with Real Application Cluster (RAC) on RedHat Enterprise Linux Advanced Server 3. The primary objective of this article is to demonstrate a quick installation of Oracle 10g with RAC on RH AS 3. This articlecovers Oracle Cluster File System (OCFS), Oracle's Automatic Storage Management (ASM), and FireWire-based Shared Storage. Note that OCFS is not required for10g RAC. In fact, I never use OCFS for RAC systems. However, this article covers OCFS since some people want to know how to configure and use OCFS.

If you have never installed Oracle10g on Linux before, then I'd recommend that you first try to install an Oracle Database 10g on Linux by following my other guideInstalling Oracle Database 10g on Red Hat Linux.

I welcome emails from any readers with comments, suggestions, or corrections. You can find my email address at the bottom of this website.

This article covers the following subjects and steps:

Introduction

* General* Important Notes* Oracle 10g RAC Setup* Shared Disks Storage

GeneralFireWire-based Shared Storage for Linux

Pre-Installation Steps for All Clustered RAC Nodes

* Downloading Oracle 10g Software and Burning Oracle 10g CDs* Installing Red Hat Advanced Server

Installing Software Packages (RPMs)* Upgrading the Linux Kernel

GeneralUpgrading the Linux Kernel for FireWire Shared Disks Only

* Configuring the NetworkGeneralSetting Up the / etc/ hosts FileConfiguring the Network Interfaces (NICs)

* Configuring Shared Storage DevicesGeneralConfiguring FireWire-based Shared Storage

* Creating Oracle User Accounts* Setting Oracle Environments* Sizing Oracle Disk Space for Database Software* Creating Oracle Directories* Creating Partitions on Shared Storage Devices

GeneralCreating Partitions for OCFSCreating Partitions for Raw Devices

* Installing and Configuring Oracle Cluster File Systems (OCFS)Installing OCFSConfiguring and Loading OCFSCreating OCFS File SystemsMounting OCFS File SystemsConfiguring the OCFS File Systems to Mount Automatically at StartupInstalling and Configuring Automatic Storage Management (ASM) and Disks

* Installing and Configuring Automatic Storage Management (ASM) and DisksGeneralInstalling ASMConfiguring and Loading ASMCreating ASM Disks

* Configuring the "hangcheck-timer" Kernel Module

Network Monitoring Tool Remote Network Management Software Get a Free N-central® Trial Today! n-able.com/free-monitoring-s

Ad Sy stem Certi fic atio ns Interactive Advertising Measurement Certification Audits to IAB Stds www.imservicesgroup.com

ONLINE Cpr Certification Get Certified Online In Minutes. ECC-Compliant. Rated "Best" By CNN. www.CPRToday.com

Network Monitoring Tool Remote Network Management Software Try N-central® Free for 30 Days! n-able.com/free-monitoring-soft


Free TeamCity Server Total continuous integration tool for use with Team Foundation Server JetBrains.com/TeamCity


exstreamspeed Fast open-source in-memory database library and toolkit www.exstreamspeed.com

In-memory Database Embedded relational database for fault-tolerant and performance app. www.Enea.com/polyhedra

Ads b y Goog le

Rac

Linux

Oracle Rac Cluster

Oracle 10G DBA

Pagina 1 di 20Installing Oracle Database 10g Real Application Cluster (RAC) on Red Hat Linux A...

22/09/2011http://www.puschitz.com/InstallingOracle10gRAC.shtml


2/20


3/20

Shared Storage can be expensive. If you just want to check out the features of Oracle10g RAC without spending too much on cost, I'd recommend to buy an externalFireWire-based shared Storage for Oracle10g RAC.

NOTE: You can download a kernel from Oracle for FireWire-based shared storage for Oracle10g RAC, but Oracle does not provide support if you have problems. It isintended for testing and demonstration only! See Setting Up Linux with FireWire-based Shared Storage for Oracle Database 10g RAC for more information.

NOTE: It is very important to get an external FireWire drive that allows concurrent access for more than one server! Otherwise the disk(s) and partitions can only beseen by one server at a time. Therefore, make sure the FireWire drive(s) have a chipset that supports concurrent access for at least two servers or more. If you havealready a FireWire drive, you can check the maximum supported logins (concurrent access) by following the steps at Configuring FireWire-based Shared Storage.

For test purposes I used external 250 GB and 200 GB Maxtor hard drives which support a maximum of 3 concurrent logins. The technical specifications for these

FireWire drives are:

- Vendor: Maxtor- Model : OneTouch- Mf g. Part No. or KI T No. : A01A200 or A01A250- Capaci t y: 200 GB or 250 GB- Cache Buf f er: 8 MB- Spi n Rat e: 7200 RPM- "Combo" I nterf ace: I EEE 1394 and SPB- 2 compl i ant ( 100 t o 400 Mbi t s/ sec) pl us USB 2. 0 and USB 1. 1 compati bl e

Here are links where these Maxtor drives can be bought:Maxtor 200GB One Touch Personal Storage External USB 2.0/FireWire Hard DriveMaxtor 250GB One Touch Personal Storage External USB 2.0/FireWire Hard Drive

The FireWire adapters I'm using are StarTech 4 Port IEEE-1394 PCI Firewire Cards. Don't forget that you will also need a FireWire hub if you want to connect morethan 2 RAC nodes to the FireWire drive(s).

Pre-Installation Steps for All Clustered RAC Nodes

The following steps need to be performed on all nodes of the RAC cluster unless it says otherwise!

Downloading Oracle 10g Software and Burning Oracle 10g CDs

To install Oracle 10g with RAC, you will need the images " shi p. crs. cpi o. gz" (Cluster Ready Services 10.1.0.2) and "shi p. db.cpi o.gz" (Oracle Database 10g10.1.0.2).For more information on downloading the images and burning CDs, see Downloading Oracle10g Software and Burning Oracle10g CDs.

Installing Red Hat Advanced Server

You can find the installation guide for installing Red Hat Linux Advanced Server at Red Hat Enterprise Linux Manuals.

You cannot download Red Hat Linux Advanced Server, you can only download the source code. If you want to get the binary CDs, you can buy licenses athttp://www.redhat.com/software/rhel/.

Installing Software Packages (RPMs)

You don't have to install all RPMs when you want to run an Oracle Database 10g with RAC on Red Hat Linux Advanced Server. You are fine when you select theInstallation Type "Advanced Server" and when you don't select the Package Group "Software Development". There are only a few other RPMs that are required forinstalling Oracle 10g RAC, which are covered in this article.

Upgrading the Linux Kernel

General

It is recommended to use newer Red Hat Enterprise Linux kernels since newer kernels might fix known database performance problems and other issues. Unless you areusing FireWire-based shared drives (see below), I recommend to download the latest RHELAS3 kernel from Red Hat Network and to use Upgrading the Linux Kernelas a guide for upgrading the kernel. However, you also need to make sure that the OCFS and ASM drivers are compatible with the kernel version!

Upgrading the Linux Kernel for FireWire Shared Disks ONLY

You can download a kernel from Oracle for FireWire-Based Shared Storage for Oracle Database 10g RAC, but Oracle does not support it. It is intended for testing and

demonstration only! See Setting Up Linux with FireWire-based Shared Storage for Oracle10g RAC for more information.

Download the experimental kernel for FireWire shared drives from http://oss.oracle.com/projects/firewire/files .

There are two experimental kernels for FireWire shared drives, one for UP machines and one for SMP machines. To install the kernel for a single CPU machine, run thefollowing command:

su - r ootrpm- i vh kernel - 2. 4. 21- 15. ELorafw1. i 686. rpm

Note that the above command does not upgrade your existing kernel. This is the preferred method since you always want the option to go back to the old kernel if thenew kernel causes problems or doesn't come up.

To make sure that the right kernel is booted, check the / et c/grub. conf file if you use GRUB, and change the "default" attribute if necessary. Here is an example:

default=0t i meout=10spl ashi mage=( hd0, 0)/ grub/spl ash.xpm. gztitle Red Hat Enterprise Linux AS (2.4.21-15.ELorafw1)

r oot ( hd0,0) kernel / vml i nuz- 2. 4. 21- 15. ELor af w1 r o r oot=LABEL=/ i ni tr d / i ni tr d-2. 4. 21-15. ELorafw1. i mgti t l e Red Hat Ent erpri se Li nux AS ( 2.4. 21- 4.EL) r oot ( hd0,0) kernel / vml i nuz- 2. 4. 21- 4. EL r o r oot=LABEL=/ i ni t r d / i ni t r d-2.4.21-4.EL. i mg




4/20

In this example, the "def aul t " attribute is set to "0" which means that the the experimental FireWire kernel 2.4. 21- 9.0. 1.ELoraf w1 will be booted. If the "def aul t "attribute would be set to "1", the 2.4.21- 9.EL kernel would be booted.

After you installed the new kernel, reboot the server:

su - r ootreboot

Once you are sure that you don't need the old kernel anymore, you can remove the old kernel by running:

su - r ootrpm - e

When you remove the old kernel you shouldn't have to update def aul t parameter in the / etc/ grub. conf file. However, I have seen cases where this didn't work. So Irecommend to check the def aul t setting after you removed the old kernel.

Configuring the Network

General

Each RAC node should have at least one static IP address for the public network and one static IP address for the private cluster interconnect.

The private networks are critical components of a RAC cluster. The private networks should only be used by Oracle to carry Cluster Manager and Cache Fusion inter-node connection. A RAC database does not require a separate private network, but using a public network can degrade database performance (high latency, low

bandwidth). Therefore the private network should have high-speed NICs (preferably one gigabit or more) and it should only be used by Oracle .

You might want to manage the network addresses using the /etc/hosts file. This avoids the problem of making DNS, NIS, etc. a single point of failure for the databasecluster.

Make sure that no firewall is running or that it doesn't interfere with RAC, respectively.

Setting Up the /etc/hosts File

Here is an example how the / etc/ hosts file could look like:

# Publ i c hostnames f or e.g. eth0 i nterf aces (publ i c network)

192. 168. 1. 1 r ac1pub. puschi t z. comr ac1pub # RAC node 1192. 168. 1. 2 r ac2pub. puschi t z. comr ac2pub # RAC node 2192. 168. 1. 3 r ac3pub. puschi t z. comr ac3pub # RAC node 3

# Pri vat e hostnames, pri vat e network f or e. g. eth1 i nterf aces ( I nterconnect)

192. 168. 2. 1 r ac1prv. puschi t z. comr ac1prv # RAC node 1192. 168. 2. 2 r ac2prv. puschi t z. comr ac2prv # RAC node 2192. 168. 2. 3 r ac3prv. puschi t z. comr ac3prv # RAC node 3

# Publ i c vi rt ual I P address for e. g. eth0 i nterf aces ( publ i c Vi rt ual I nternet Protocol (VI P))

192. 168.1. 51 rac1vi p.puschi t z. comr ac1vi p # RAC node 1192. 168.1. 52 rac2vi p.puschi t z. comr ac2vi p # RAC node 2192. 168.1. 53 rac3vi p.puschi t z. comr ac3vi v # RAC node 3

The public virtual IP addressess are configured automatically by Oracle when you run OUI, which starts Oracle's Virtual Internet Protocol Configuration Assistant(VIPCA), see Installing Oracle Database 10g Software R1 (10.1.0.2) with Real Application Clusters (RAC)

NOTE:

Make sure that the name of the RAC node is not listed for the loopback address in the / etc/ hosts file similar to this example:

127.0. 0.1 r ac1pub l ocal host . l ocal domai n l ocal host

The entry should look like this:

127.0. 0.1 l ocal host . l ocal domai n l ocal host

If the RAC node is listed for the loopback address, you might later get the following errors:

ORA- 00603: ORACLE server sess i on t ermi nated by f at al err ororORA- 29702: er r or occurr ed i n Cl uster Gr oup Servi ce oper ati on

For more information, see Oracle 10g RAC Issues, Problems and Errors .

Configuring the Network Interfaces (NICs)

To configure the network interfaces (in this example eth0 and eth1), run the following command on each node.

su - r ootredhat- conf i g-net work

NOTE: You do not have to configure the network alias names for the public VIP. This will be done by Oracle's Virtual Internet Protocol Configuration Assistant(VIPCA).

NOTE: When the network configuration is done, it is important to make sure that the name of the public RAC nodes is displayed when you execute the following

command:

$ host namerac1pub




5/20

You can verify the new configured NICs by running the command:

/ sbi n/ i f conf i g

Configuring Shared Storage Devices

General

For instructions on how to setup a shared storage device on Red Hat Advanced Server, see the installation instructions of the manufacturer.

Configuring FireWire-based Shared Storage

First make sure that the experimental kernel for FireWire drives was installed and that the server was rebooted (see Upgrading the Linux Kernel for FireWire SharedDisks Only):

# uname - r2. 4.21-15. ELoraf w1

To load the kernel modules/drivers for the FireWire drive(s), add the following entry to the / etc/ modul es. conf file:

al i as i eee1394-cont r ol l er ohci 1394post- i nst al l ohci 1394 modpr obe sd_mod

The alias directive i eee1394-cont rol l er is used by Red Hat during the boot process. When you check the / etc/ r c.d/ rc.sysi ni t file, which is invoked by / e t c/ i ni t tab during the boot process, you will find the following code that searches for the i eee1394-cont rol l er stanza in / etc/ modul es. conf :

i f ! st rst r "$cmdl i ne" nofi rewi re ; thenaliases=̀ / sbi n/modprobe -c | awk ' / âl i as ieee1394-controller/ { pri nt $3 }' ` i f [ -n "$al i ases" -a "$al i ases" != "off " ] ; then for al i as i n $al i ases ; do [ "$al i as" = "off" ] && cont i nue act i on $"Ini t i al i z i ng f i rewi re contr ol l er ($al i as) : " modprobe $alias done LC_ALL=C grep - q "SBP2" / proc/bus/i eee1394/ devi ces 2>/ dev/nul l && \

modprobe sbp2 >/ dev/nul l 2>&1 f if i

This means that all the kernel modules for the FireWire drive(s) will be loaded automatically during the next reboot and your drive(s) should be ready for use.

To load the modules or the firewire stack right away without rebooting the server, execute the following commands:

su - r ootmodpr obe i eee1394-cont r ol l er; modpr obe sd_mod

If everything worked fine, the following modules should be loaded:

su - r oot

# l smod | egr ep "ohci 1394|s bp2| i eee1394|s d_mod| scs i _mod"sbp2 19724 0ohci 1394 28008 0 ( unused)i eee1394 62884 0 [ sbp2 ohci 1394]sd_ mod 13424 0scsi _mod 104616 5 [ sbp2 sd_mod sg sr _mod i de-scsi ]#

And when you run dmesg, you should see entries similar to this example:

# dmesg. . .ohci 1394_0: OHCI - 1394 1. 0 ( PCI ) : I RQ=[ 11] MMI O=[ f 2000000-f 20007ff ] Max Packet =[ 2048]i eee1394: Devi ce added: Node[00: 1023] GUI D[ 0010b9f 70089de1c] [ Maxtor]scsi 1 : SCSI emul ati on f or I EEE-1394 SBP-2 Devi cesbl k: queue cf 172e14, I / O l i mi t 4095Mb (mask 0xf f f f f f f f )i eee1394: Conf i gROM quadl et t r ansact i on err or f or node 01: 1023i eee1394: Host added: Node[02: 1023] GUI D[ 00110600000032a0] [ Li nux OHCI - 1394]i eee1394: sbp2: Quer y l ogi ns t o SBP- 2 devi ce successf ulieee1394: sbp2: Maximum concurrent logins supported: 3

ieee1394: sbp2: Number of active logins: 0i eee1394: sbp2: Logged i nto SBP- 2 devi cei eee1394: sbp2: Node[ 00: 1023]: Max speed [ S400] - Max payl oad [2048] Vendor: Maxt or Model : OneTouch Rev: 0200 Type: Di r ect - Access ANSI SCSI revi si on: 06bl k: queue cd0f b014, I / O l i mi t 4095Mb (mask 0xf f f f f f f f )At t ached scsi di sk sda at s csi 1, channel 0, i d 0, l un 0SCSI devi ce sda: 398295040 512-byte hdwr s ect or s ( 203927 MB) sda: sda1 sda2 sda3 sda4 < sda5 sda6 sda7 sda8 sda9 sda10 sda11 sda12 sda13 >

In this example, the kernel reported that the FireWire drive can be shared concurrently by 3 servers (see "Maximum concurrent logins supported:"). It is very importantthat you have a drive with a chipset and firmware that supports concurrent access for the nodes. The "Number of active logins:" shows how many servers are alreadysharing/using the drive before this server added the drive to its system.

If everything worked fine, you should be able to see now your FireWire drive(s):

su - r oot# fdi sk - l

Di sk / dev/ sda: 255 heads, 63 sector s, 30515 cyl i ndersUni t s = cyl i nders of 16065 * 512 bytes

Devi ce Boot Star t End Bl ocks I d Syst em/ dev/ sda1 1 6375 51207156 83 Li nux/ dev/ sda2 6376 12750 51207187+ 83 Li nux




6/20

. . .

And if you power off or power on your FireWire drive(s), respectively, the drives should be removed and added to the system automatically, which can take about 5-10seconds.

If everything worked fine without any errors or problems, I would recommend to reboot all RAC nodes to verify that all FireWire drive(s) are automatically added to thesystem during the next boot process:

su - r ootreboot

And after the reboot, execute the fd i sk command again to verify that the FireWire drive(s) were added to the system:

su - r oot# fdi sk - l

PROBLEMS:

Note that if you have a USB device attached, the system might not be able to recognice your FireWire drive!

If the i eee1394 module was not loaded, then your FireWire adapter might not be supported. I'm using the StarTech 4 Port IEEE-1394 PCI Firewire Card which worksfine:

# l spci. . .00:14.0 Fi reWi re ( I EEE 1394): VI A Technol ogi es, I nc. I EEE 1394 Host Contr ol l er ( rev 46). . .

Creating Oracle User Accounts

If you use OCFS, it is important that the UID of "oracle" and GID of "oinstall" are the same across all RAC nodes. Otherwise the Oracle files on the OCFS filesystemson some nodes could either be "unowned", or they could even be owned by another user account. In my setup the UID and GID of oracl e:dba is 700: 700.

su - r ootgroupadd - g 700 dba # group of users t o be gr ant ed wi t h SYSDBA syst empr i vi l egegroupadd - g 701 oi nstal l # group owner of Oracl e fi l esuser add - c "Or acl e sof t ware owner " - u 700 - g oi nstal l - G dba or acl epasswd oracl e

To verify the oracle account, enter the following command:

# i d oracleui d=700(oracl e) gi d=701(oi nstal l ) groups=701(oi nstal l ) , 700(dba)

For more information on the "oi nstal l " group account, see When to use "OINSTALL" group during install of oracle .

Setting Oracle Environments

Since the Oracle Universal Installer (OUI) "runI nstal l er " is executed from the oracl e account, some environment variables must be configured for the oracleaccount before OUI is started.

Note: When you set the Oracle environment variables for the RAC nodes, make sure to assign each RAC node a unique Oracle SID! In my test setup, the database nameis "orcl" and the Oracle SIDs are "orcl1" for RAC node one, "orcl2" for RAC node two, and so on.

If you use bash which is the default shell on Red Hat Linux (to verify your shell run: echo $SHELL), execute the following commands:

# Or acl e Envi r onmentexpor t ORACLE_BASE=/ u01/ app/ oracl eexport ORACLE_SI D=orcl 1 # Each RAC node must have a uni que Or acl e SI D! E. g. orcl 1, orcl 2, . . .expor t LD_LI BRARY_PATH=$ORACLE_HOME/ l i b

NOTE: If ORACLE_BASE is used, then Oracle recommends that you don't set the ORACLE_HOME environment variable but that you choose the default path suggested by

the OUI. You can set and use ORACLE_HOME after you finished installing the Oracle Database 10g Software with RAC, see Installing Oracle Database 10g Software R1(10.1.0.2) with Real Application Clusters (RAC).

The environment variables ORACLE_HOME and TNS_ADMI N should not be set. If you already set these environment variables, you can unset them by executing the

following commands:

unset ORACLE_HOMEunset TNS_ADMI N

To have these environment variables set automatically each time you login as oracle, you can add these environment variables to the ~oracl e/. bash_prof i l e file forthe Bash shell on Red Hat Linux. To do this you could simply copy/paste the following commands to make these settings permanent for the oracle Bash shell:

su - oracl ecat >> ~oracl e/. bash_prof i l e


7/20

Creating Oracle Directories

At the time of this writing, OCFS only supports Oracle Datafiles and a few other files. Therefore OCFS should not be used for Shared Oracle Home installs. SeeInstalling and Configuring Oracle Cluster File Systems (OCFS) for more information.

For Oracle10g you only need to create the directory for $ORACLE_BASE:

su - r ootmkdi r - p / u01/app/ oracl echown - R oracl e.oi nstal l / u01

But if you want to comply with Oracle's Optimal Flexible Architecture (OFA), then you don't want to place the database files in the / u01 directory but in anotherdirectory like / u02. This is not a requirement but if you want to comply with OFA, then you might want to create the following directories as well:

su - r ootmkdi r - p /u02/ oradat a/ orclchown - R oracl e.oi nstal l / u02

Here I would recommend to take a quick look at Oracle's new Optimal Flexible Architecture (OFA).

NOTE: In my example I will not place the database files into the OCFS directory / u02/oradat a/or cl since I will use Automatic Storage Management (ASM).However, I will use / u02/oradat a/or cl for the cluster manager files, see Installing Cluster Ready Services (CRS).

Creating Partitions on Shared Storage Devices

The partitioning of a shared disk needs to be performed on only one RAC node!

General

Note that it is important for the Redo Log files to be on the shared disks as well.

To partition the disks, you can use the fd i sk utility:

su - r ootfdi sk

For SCSI disks (including FireWire disks), stands for device names like / dev/s da, / dev/sdb, / dev/ sdc, dev/ sdd , etc. Be careful to use the rightdevice name!

Here is an example how to create a new 50 GB partition on drive / dev/ sda:

su - r oot# f di sk /dev/sda

The number of cyl i nders f or t hi s di sk i s set t o 30515. There i s nothi ng wr ong wi t h t hat, but t hi s i s l ar ger t han 1024,and coul d in cert ai n setups cause pr oblems wi t h:1) soft ware that runs at boot t i me (e. g., old versi ons of LI LO)

2) booti ng and part i ti oning sof tware f rom other OSs ( e. g. , DOS FDI SK, OS/ 2 FDI SK)

Command ( m f or hel p): p

Di sk / dev/sda: 255 heads, 63 sectors , 30515 cyli ndersUnit s = cyl i nders of 16065 * 512 bytes

Device Boot Start End Blocks I d System/ dev/ sda1 1 6375 51207156 83 Li nux

Command ( m f or hel p): nCommand ac t i on e extended p pri mary part i ti on (1-4)pParti ti on number (1- 4): 2Fi rst cyl i nder (6376- 30515, defaul t 6376):Usi ng defaul t value 6376Last cyl i nder or +si ze or +si zeM or +si zeK (6376- 30515, def aul t 30515): +50GB

Command ( m f or hel p): p

Di sk / dev/sda: 255 heads, 63 sectors , 30515 cyli ndersUnit s = cyl i nders of 16065 * 512 bytes

Device Boot Start End Blocks I d System/ dev/ sda1 1 6375 51207156 83 Li nux/ dev/ sda2 6376 12750 51207187+ 83 Li nux

Command ( m f or hel p): w The part i t i on t abl e has been al t er ed!

Cal l i ng i oct l ( ) to re- read part i t i on tabl e.

WARNING: I f you have cr eat ed or modi f i ed any DOS 6. xpart i ti ons, pl ease see the fdi sk manual page f or addi ti onali nformati on.

Syncing di sks.

#

For more information on fd i sk, see the fdisk(8) man page.

After you finished creating the partitions, inform the kernel of the partition table changes:

su - r ootpar t probe

Creating Partitions for OCFS

If you use OCFS for database files and other Oracle files, you can create several partitions on your shared storage for the OCFS filesystems. If you use a FireWire disk,you could create one large partition on the disk which should make things easier.

For more information on how to install OCFS and how to mount OCFS filesystems on partitions, see Installing and Configuring Oracle Cluster File Systems (OCFS).




8/20

Creating Partitions for Raw Devices

If you want to use raw devices, see Creating Partitions for Raw Devices for more information. This article does not cover raw devices.

Installing and Configuring Oracle Cluster File Systems (OCFS)

Note that OCFS is not required for 10g RAC. In fact, I never use OCFS for RAC systems. However, this article covers OCFS since some people want to know how toconfigure and use OCFS.

The Oracle Cluster File System (OCFS) was developed by Oracle to overcome the limits of Raw Devices and Partitions. It also eases administration of database files because it looks and feels just like a regular file sys tem.

At the time of this writing, OCFS only supports Oracle Datafiles and a few other files:- Redo Log files- Archive log files- Control files- Database datafiles- Shared quorum disk file for the cluster manager- Shared init file (srv)

Oracle says that they will support Shared Oracle Home installs in the future. So don't install the Oracle software on OCFS yet. See Oracle Cluster File System for moreinformation. In this article I'm creating a separate, individual ORACLE_HOME directory on local server storage for each and every RAC node.

NOTE:If files on the OCFS file system need to be moved, copied, tar'd, etc., or if directories need to be created on OCFS, then the standard file system commands mv, cp,t a r , . . . that come with the OS should not be used. These OS commands can have a major OS performance impact if they are being used on the OCFS file system.

Therefore, Oracle's patched file system commands should be used instead. It is also important to note that some 3rd vendor backup tools make use of standard OS commands like tar .

Installing OCFS

NOTE: In my example I will use OCFS only for the cluster manager files since I will use ASM for datafiles.

Download the OCFS RPMs (drivers, tools) for RHEL3 from http://oss.oracle.com/projects/ocfs/files/RedHat/RHEL3/i386/. (you can use the same RPMs for FireWireshared disks).

To find out which OCFS driver you need for your server, run:

$ uname - aLi nux rac1pub 2. 4. 21- 9. ELsmp #1 Thu J an 8 17: 24: 12 EST 2004 i686 i 686 i 386 GNU/ Li nux

To install the OCFS RPMs for SMP kernels (including FireWire SMP kernels), execute:

su - r ootrpm- Uvh ocf s- 2. 4. 21-EL- smp-1.0. 12- 1. i 686. rpm\ ocf s-t ool s-1.0.10- 1. i 386. rpm\ ocfs- support- 1. 0. 10- 1. i 386.r pm

To install the OCFS RPMs for uniprocessor kernels (including FireWire UP kernels), execute:

su - r ootrpm- Uvh ocf s- 2. 4. 21-EL- 1. 0.12- 1. i 686. rpm\ ocf s-t ool s-1.0.10- 1. i 386. rpm\ ocfs- support- 1. 0. 10- 1. i 386.r pm

Configuring and Loading OCFS

To generate the / etc/ ocfs. conf file, you can run the ocfst ool tool:

su - r ootocfst ool- Sel ect "Task" - Sel ect " Generate Conf i g"- Sel ect t he i nterconnect i nterf ace ( pri vat e net work i nterf ace)

I n my exampl e f or r ac1pub I sel ected: eth1, rac1prv

- Conf i r m t he val ues di spl ayed and exi t

The generated / etc/ ocfs. conf file will appear similar to the following example:

$ cat / et c/ocf s. conf ## ocfs conf i g# Ensure thi s f i l e exi sts i n /etc#

node_name = r ac1prv i p_address = 192.168. 2.1 i p_port = 7000 comm_vot i ng = 1 gui d = 84D43BC8FB7A2C1B88C3000D8821CC2C

The gui d entry is the unique group user ID. This ID has to be unique for each node. You can create the above file without the ocfst ool tool by editingthe / etc/ ocfs. conf file manually and by running ocfs_ui d_gen - c to assign/update the guid value in this file.

To load the ocfs .o kernel module, execute:

su - r oot# / sbi n/ l oad_ocfs/ sbi n/ i nsmod ocf s node_name=r ac1prv i p_address=192.168.2. 1 cs=1795 gui d=84D43BC8FB7A2C1B88C3000D8821CC2C comm_vot i ng=1 i p_por t =7000Usi ng / l i b/ modul es/ 2.4.21- EL- ABI / ocf s/ ocf s. o#




9/20

To verify if the ofcs module was loaded, execute:

# / sbi n/l smod | grep ocfsocf s 305920 0 ( unused)

Note that the load_ocfs command doest not have to be executed again once everything has been setup for the OCFS filesystems, see Configuring the OCFS File

Systems to Mount Automatically at Startup.

If you run l oad_ocf s on a system with the experimental FireWire kernel, you might get the following error message:

su - r oot# l oad_ocf s/ sbi n/ i nsmod ocf s node_name=r ac1prv i p_address=192.168.2. 1 c s=1843 gui d=AA12637FAABFB354371C000D8821CC2C comm_vot i ng=1 i p_por t =7000i nsmod: ocf s: no modul e by t hat name f oundl oad_ocf s: i nsmod fai l ed#

The ocfs .o module for the "FireWire kernel" can be found here:

su - r oot# rpm- ql ocfs- 2. 4. 21-EL-1. 0. 12-1/ l i b/ modules/ 2. 4.21- EL-ABI / ocf s/ l i b/ modules/ 2. 4.21- EL-ABI / ocf s/ ocf s. o#

So for the experimental kernel for FireWire drives, I manually created a link for the ocfs.o module file:

su - r ootmkdi r / l i b/modul es/`uname - r` / kernel / dri vers/ addon/ocf sl n - s `rpm- qa | grep ocf s- 2 | xargs rpm- ql | grep "/ ocf s. o$"` \

/ l i b/ modul es/ `uname - r`/ kernel / dri vers/ addon/ ocf s/ ocf s. o

Now you should be able to load the OCFS module using the "FireWire kernel", and the output should look similar to this example:

su - r oot# / sbi n/ l oad_ocfsl oad_ocf s / sbi n/ i nsmod ocf s node_name=r ac1prv i p_address=192.168.2. 1 cs=1843 gui d=AA12637FAABFB354371C000D8821CC2C comm_vot i ng=1 i p_porUsi ng / l i b/ modul es/ 2.4.21- EL- ABI / ocf s/ ocf s. oWarni ng: kernel - modul e ver si on mi smatch / l i b/modul es/2. 4.21-EL-ABI / ocfs/ ocfs. o was compi l ed f or kernel versi on 2.4. 21- 4.EL whi l e t hi s kernel i s versi on 2.4. 21- 15. ELoraf w1Warni ng: l oading /l i b/ modul es/ 2.4.21- EL- ABI / ocf s/ ocf s. o wi l l tai nt t he kernel : f orced load See ht t p:/ / www. t ux. org/l kml / #export - t ai nted f or i nformati on about t ai nted modul esModul e ocfs l oaded, wi t h warni ngs#

I would not worry about the above warning.

However, if you get the following error, then you have to upgrade the moduti l s RPM:

su - r oot# / sbi n/ l oad_ocfs/ sbi n/ i nsmod ocf s node_name=r ac2prv i p_address=192.168.2. 2 cs =1761 gui d=1815F1C57530339EA00E000D8825B058 comm_vot i ng=1 i p_por t =7000Usi ng / l i b/ modul es/ 2.4.21- EL- ABI / ocf s/ ocf s. o/ l i b/modul es/2. 4.21-EL-ABI / ocfs/ ocfs. o: kernel - modul e ver si on mi smatch / l i b/modul es/2. 4.21-EL-ABI / ocfs/ ocfs. o was compi l ed f or kernel versi on 2.4. 21- 4.EL whi l e t hi s kernel i s versi on 2.4. 21- 15. ELoraf w1.#

To remedy the "loading" problem, download the latest moduti l s RPM and enter e.g.:

rpm - Uvh moduti l s- 2.4. 25- 11. EL. i 386.r pm

To verify that the ofcs module was loaded, enter:

# / sbi n/l smod | grep ocfsocf s 305920 0 ( unused)

Note that the load_ocfs command doest not have to be executed again once everything has been setup for the OCFS filesystems, see Configuring the OCFS File

Systems to Mount Automatically at Startup.

Creating OCFS File Systems

Before you continue with the next steps, make sure you've created all needed partitions on your shared storage.

Under Creating Oracle Directories I created the / u02/oradat a/or cl mount directory for the cluster manager files. In the following example I will create one OCFSfilesystem and mount it on / u02/oradat a/or cl .

The following steps for creating the OCFS filesystem(s) should only be executed on one RAC node!

To create the OCFS filesystems, you can use the ocfst ool :

su - r ootocfst ool

- Sel ect "Task" - Sel ect "Format"

Alternatively, you can execute the "mkf s. ocfs " command to create the OCFS filesystems:

su - r ootmkfs. ocf s - F -b 128 -L / u02/oradata/orcl - m/ u02/ oradata/ orcl \




10/20

- u `i d -u oracl e` - g `i d -g oracl e` - p 0775 Cl eared vol ume header sect or sCl ear ed node conf i g sect orsCl ear ed publ i sh sect orsCl ear ed vote sector sCl ear ed bi t map sect orsCl ear ed dat a bl ockWr ot e vol ume header#

For SCSI disks (including FireWire disks), stands for devices like / dev/ sda, / dev/sdb, / dev/ sdc, dev/ sdd, etc. Be careful to use the right devicename! For this article I created an OCFS filesystem on / dev/sda1.

mkf s. ocfs options:

- F Forces to format existing OCFS volume- b Block size in kB. The block size must be a multiple of the Oracle block size. Oracle recommends to set the block size for OCFS to 128.

- L Volume label

- mMount point for the device (in this article "/ var/ opt/ oracl e/ oradata/orcl ")- u UID for the root directory (in this article "oracle")

- g GID for the root directory (in this article "oinstall")- p Permissions for the root directory

Mounting OCFS File Systems

As I mentioned previously, for this article I created one large OCFS fileystem on / dev/s da1. To mount the OCFS filesystem, I executed:

su - r oot# mount - t ocfs / dev/ sda1 / u02/oradat a/or clor# mount - t ocf s -L / u02/oradat a/or cl / u02/oradat a/or cl

Now run the l s command on all RAC nodes to check the ownership:

# l s -l d / u02/oradata/orcldrwxr wxr - x 1 oracle oinstall 131072 J ul 4 23: 25 / u02/ oradat a/ orcl#

NOTE: If the above l s command does not display the same ownership on all RAC nodes (oracle:oi nstal l ), then the "oracle" UID and the "oinstall" GID are not the

same accross the RAC nodes, see Creating Oracle User Accounts for more information.

Configuring the OCFS File Systems to Mount Automatically at Startup

To ensure the OCFS filesystems are mounted automatically during reboots, the OCFS mount points need to be added to the / etc/ f stab file.

Add lines to the /e tc/ f stab file similar to the following example:

/ dev/ sda1 / u02/oradat a/or cl ocfs _net dev 0 0

The " _net dev" option prevents the OCFS filesystem from being mounted until the network has first been enabled on the system, which provides access to the storagedevice (see mount(8)).

To make sure the ocfs . o kernel module is loaded and the OCFS file systems are mounted during the boot process, enter:

su - r oot# chkconf i g - - l i s t ocfsocfs 0: of f 1: off 2: off 3: on 4: on 5: on 6: off

If the flags are not set to "on" as marked in bold, run the following command:

su - r oot# chkconfi g ocf s on

You can also start the "ocfs" service manually by running:

su - r oot# servi ce ocf s start

When you run this command it will not only load the ocfs .o kernel module but it will also mount the OCFS filesystems as configured in / etc/ f stab.

At this point you might want to reboot all RAC nodes to ensure that the OCFS filesystems are mounted automatically after reboots:

su - r ootreboot

Installing and Configuring Automatic Storage Management (ASM) and Disks

General

For information about what Automatic Storage Management is, see Configuring and Using Automatic Storage Management.See also Installing Oracle ASMLib for Linux.

Installing ASM

Download the latest Oracle ASMRPMs from http://otn.oracle.com/tech/linux/asmlib/index.html .

Make sure that you download the right ASM driver for your kernel (UP or SMP).

To install the ASMRPMs on a UP server, run:

su - r ootrpm - Uvh oracl easm- 2.4. 21- EL- 1.0. 0-1. i 686.r pm\

Pagina 10 di 20Installing Oracle Database 10g Real Application Cluster (RAC) on Red Hat Linu...



11/20

oracl easm- support- 1. 0. 2- 1. i 386. rpm\ oracl easml i b- 1. 0. 0-1.i 386. rpm

To install the ASMRPMs on a SMP server, run:

su - r ootrpm - Uvh oracl easm- 2.4. 21- EL- smp-1. 0.0- 1.i 686.r pm\ oracl easm- support- 1. 0. 2- 1. i 386. rpm\ oracl easml i b- 1. 0. 0-1.i 386. rpm

Configuring and Loading ASM

To load the ASM driver oracl eams. o and to mount the ASM driver filesystem, enter:

su - r oot# / etc/ i ni t. d/ oracleasmconf i gureConf i guri ng the Oracl e ASM l i brary dri ver.

Thi s wi l l conf i gure t he on- boot propert i es of t he Or acl e ASM l i brarydr i ver. The f ol l owi ng questi ons wi l l det ermi ne whet her t he dri ver i sl oaded on boot and what permi ssi ons i t wi l l have. The curr ent val ueswi l l be shown i n bracket s ( ' [] ' ). Hi t ti ng wi thout typing ananswer wi l l keep that curr ent val ue. Ctr l - C wi l l abort.

Defaul t user to own t he dri ver i nterf ace [ ] : oracleDefaul t gr oup to own the dri ver i nterf ace [ ] : oi nstal lSt art Oracl e ASM l i brary dri ver on boot ( y/n) [ n] : yFi x permi ssi ons of Oracl e ASM di sks on boot ( y/ n) [ y]: yWri ti ng Oracl e ASM l i brary dri ver conf i gurati on [ OK ]Cr eat i ng / dev/ oracl easm mount poi nt [ OK ]Loadi ng modul e "oracl easm" [ OK ]Mounti ng ASMl i b dri ver f i l esyst em [ OK ]

Scanni ng system f or ASM di sks [ OK ]#

Creating ASM Disks

NOTE: Creating ASM disks is done on one RAC node! The following commands should only be executed on one RAC node!

I executed the following commands to create my ASM disks: (make sure to change the device names!)(In this example I used partitions (/ dev/sda2, / dev/ sda3, / dev/ sda5) instead of whole disks (/ dev/ sda, / dev/s db, / dev/ sdc,...))

su - r oot# / etc/ i ni t . d/or acl easm createdi sk VOL1 / dev/Marki ng di sk "/ dev/ sda2" as an ASM di sk [ OK ]# / etc/ i ni t . d/or acl easm createdi sk VOL2 / dev/Marki ng di sk "/ dev/ sda3" as an ASM di sk [ OK ]# / etc/ i ni t . d/or acl easm createdi sk VOL3 / dev/Marki ng di sk "/ dev/ sda5" as an ASM di sk [ OK ]## # Repl ace "s d??" wi t h the name of your devi ce. I used / dev/ sda2, / dev/ sda3, and / dev/ sda5

To list all ASM disks, enter:

# /etc/ i ni t . d/oracleasml i stdi sksVOL1VOL2VOL3#

On all other RAC nodes, you just need to notify the system about the new ASM disks:

su - r oot# / etc/ i ni t. d/ oracleasmscandi sksScanni ng system f or ASM di sks [ OK ]#

Configuring the "hangcheck-timer" Kernel Module

Oracle uses the Linux kernel module hangcheck- t i mer to monitor the system health of the cluster and to reset a RAC node in case of failures. The hangcheck-timer

module uses a kernel-based timer to periodically check the system task scheduler. This timer resets the node when the system hangs or pauses. This module uses theTime Stamp Counter (TSC) CPU register which is a counter that is incremented at each clock signal.

The hangcheck-timer module comes now with the kernel:

f i nd / l i b/ modul es - name "hangcheck- t i mer. o"

The hangcheck-timer module has the following two parameters:

hangcheck_t i ck

This parameter defines the period of time between checks of system health. The default value is 60 seconds. Oracle recommends to set it to 30 seconds.

hangcheck_mar gi n

This parameter defines the maximum hang delay that should be tolerated before

hangcheck-timer resets the RAC node. It defines the margin of error in seconds. The default value is 180 seconds. Oracle recommends to set it to 180 seconds.

These two parameters indicate how long a RAC node must hang before the hangcheck-timer module will reset the system. A node reset will occur when the following istrue:




12/20

syst emhang t i me > ( hangcheck_t i ck + hangcheck_margi n)

To load the module with the right parameter settings, make entries to the / etc/ modul es. conf file. To do that, add the following line to the / etc/ modul es. conf file:

# su - r oot# echo "opt i ons hangcheck- t i mer hangcheck_t i ck=30 hangcheck_margi n=180" >> / etc/ modul es. conf

Now you can run modpr obe to load the module with the configured parameters in / etc/ modul es. conf :

# su - r oot# modprobe hangcheck- t i mer

# grep Hangcheck / var/ l og/ messages | t ai l - 2 J ul 5 00: 46: 09 r ac1pub kernel : Hangcheck: st ar t i ng hangcheck t i mer 0. 8. 0 ( t i ck i s 180 seconds, mar gi n i s 60 seconds) . J ul 5 00: 46: 09 r ac1pub kernel : Hangcheck: Usi ng TSC.#

Note: To ensure the hangcheck-timer module is loaded after each reboot, add the modpr obe command to the /e tc/ rc. l ocal file.

Setting up RAC Nodes for Remote Access

When you run the Oracle Installer on a RAC node, it will use ssh to copy Oracle software and data to other RAC nodes. Therefore, the oracle user on the RAC nodewhere Oracle Installer is launched must be able to login to other RAC nodes without having to provide a password or passphrase.

The following procedure shows how ssh can be configured that no password is requested for oracle ssh logins.

To create an authentication key for oracle, enter the following command on all RAC node:(the ~/ . ssh directory will be created automatically if it doesn't exist yet)

su - oracl e$ ssh- keygen - t dsa -b 1024Generati ng publ i c/ pri vat e dsa key pai r .Enter f i l e i n whi ch t o save t he key (/ home/oracl e/. ssh/i d_dsa): Press ENTERCreat ed di rectory ' / home/oracle/ . ssh' .Ent er passphr ase (empty f or no passphr ase) : Ent er a passphraseEnt er s ame passphrase agai n: Etner a passphr ase

Your i denti f i cat i on has been s aved i n / home/ or acl e/ . ssh/ i d_dsa. Your publ i c key has been saved i n / home/ or acl e/ . ssh/ i d_dsa. pub. The key f i ngerpr i nt i s:e0: 71: b1: 5b: 31: b8: 46: d3: a9: ae: df : 6a: 70: 98: 26: 82 oracl e@r ac1pub

Copy the pulic key for oracle from each RAC node to all other RAC nodes.For example, run the following commands on all RAC nodes:

su - oracl essh r ac1pub cat ~/ . ssh/ i d_dsa. pub >> ~/ . ssh/ aut hor i zed_keysssh r ac2pub cat ~/ . ssh/ i d_dsa. pub >> ~/ . ssh/ aut hor i zed_keysssh r ac3pub cat ~/ . ssh/ i d_dsa. pub >> ~/ . ssh/ aut hor i zed_keys

Now verify that oracl e on each RAC node can login to all other RAC nodes without a password. Make sure that ssh only asks for the passphrase. Note, however, thatthe first time you ssh to another server you will get a message stating that the authenticity of the host cannot be established. Enter "yes" at the prompt to continue the

connection.For example, run the following commands on all RAC nodes to verify that no password is asked:

su - oracl essh r ac1pub host namessh r ac1pub host namessh r ac1pr v host namessh r ac2pub host namessh r ac2pr v host namessh r ac3pub host namessh r ac3pr v host name

And later, before runI nstal l er is launched, I will show how ssh can be configured that no passphrase has to be entered for oracl e ssh logins.

Checking Packages (RPMs)

Some packages will be missing when you selected the Installation Type "Advanced Server" during the Red Hat Advanced Server installation.

The following additional RPMs are required:

rpm - q gcc gl i bc-devel gl i bc-headers gl i bc-ker nheader s cpp compat - l i bstdc++

To install these RPMS, run:

su - r ootrpm- i vh gcc- 3. 2. 3-24. i 386. rpm\ gl i bc-devel -2.3. 2- 95. 6. i 386. rpm\ gl i bc- headers-2. 3. 2- 95.6.i 386.r pm \ gl i bc- kernheaders-2. 4- 8. 34.i 386. rpm\ cpp- 3. 2.3-24. i 386. rpm\ compat - l i bstdc++- 7.3- 2.96.123. i 386.r pm

The opemot i f RPM is also required, otherwise you won't pass Oracle's recommended operating system packages test. If it's not installed on your system, run

su - r ootrpm- i vh openmoti f - 2.2.2- 16.i 386. rpm

I recommend using the latest RPM version.

Adjusting Network Settings




13/20

Oracle now uses UDP as the default protocol on Linux for interprocess communication, such as cache fusion buffer transfers between the instances.It is strongly suggested to adjust the default and maximum send buffer size (SO_SNDBUF socket option) to 256 KB, and the default and maximum receive buffer size

(SO_RCVBUF socket option) to 256 KB. The receive buffers are used by TCP and UDP to hold received data until is is read by the application. The receive buffer cannotoverflow because the peer is not allowed to send data beyond the buffer size window. This means that datagrams will be discarded if they don't fit in the socket receive

buffer. This could cause the sender to overwhelm the receiver.

The default and maximum window size can be changed in the proc file system without reboot:

su - r ootsysct l - w net . core. r mem_defaul t =262144 # Def aul t sett i ng i n bytes of the socket recei ve buff ersysct l - w net . core. wmem_defaul t =262144 # Defaul t sett i ng i n bytes of t he socket send buf f ersysct l - w net . cor e. r mem_max=262144 # Maxi mumsocket r ecei ve buf f er s i ze whi ch may be set by usi ng t he SO_RCVBUF socket opt i on

sysct l - w net . cor e. wmem_max=262144 # Maxi mumsocket send buff er s i ze whi ch may be set by usi ng the SO_SNDBUF socket opti on

To make the change permanent, add the following lines to the / etc/ sysctl . conf file, which is used during the boot process:

net . cor e. r mem_def aul t =262144net. cor e. wmem_def aul t =262144net . cor e. r mem_max=262144net . cor e. wmem_max=262144

Sizing Swap Space

It is important to follow the steps as outlined in Sizing Swap Space.

Setting Shared Memory

It is important to follow the steps as outlined in Setting Shared Memory.

Checking /tmp Space

It is important to follow the steps as outlined in Checking /tmp Space.

Setting Semaphores

It is recommended to follow the steps as outlined in Setting Semaphores.

Setting File Handles

It is recommended to follow the steps as outlined in Setting File Handles.

Installing Cluster Ready Services (CRS)

General

Cluster Ready Services (CRS) contains cluster and database configuration information for RAC, and it provides many system management features. CRS acceptsregistration of Oracle instances to the cluster and it sends ping messages to other RAC nodes. If the heartbeat fails, CRS will use shared disk to distinguish between anode failure and a network failure.

Once CRS is running on all RAC nodes, OUI will automatically recognice all nodes on the cluster. This means that you can run OUI on one RAC node to install theOracle software on all other RAC nodes.

Note that Automatic Storage Management (ASM) cannot be used for the "Oracle Cluster Registry (OCR)" file or for the "CRS Voting Disk" file. These files must beaccessible before any Oracle instances are started. And for ASM to become available, the ASM instance needs to run first.

In the following example I will use OCFS for the "Oracle Cluster Registry (OCR)" file and for the "CRS Voting disk" file. The Oracle Cluster Registry file has a size ofabout 100 MB, and the CRS Voting Disk file has a size of about 20 MB. Tese files must reside on OCFS or on a shared raw device, or on any other clustered filesystem.

Automating Authentication for oracle ssh Logins

Make sure that the oracl e user can ssh to all RAC nodes without ssh asking for a passphrase. This is very important because otherwise OUI won't be able to install theOracle software on other RAC nodes. The following example shows how ssh- agent can do the authentication for you when the oracl e account logs in to other RACnodes using ssh.

Open a new terminal for the RAC node where you will execute runI nstal l er and use this terminal to login from your desktop using the following command:

$ ssh - X oracl e@r ac?pub

The "X11 forward" feature (-X option) of ssh will relink X to your local desktop. For more information, see Installing Oracle10g on a Remote Linux Server .

Now configure ssh- agent to handle the authentication for the oracl e account:

oracl e$ ssh- agent $SHELLoracl e$ ssh- addEnter passphr ase f or / home/oracl e/. ssh/i d_dsa: Ent er your passphraseI denti t y added: /home/oracl e/. ssh/i d_dsa ( / home/oracl e/. ssh/i d_dsa)oracl e$

Now make sure the oracle user can ssh into each RAC node. It is very important that NO text is displayed and that you are not asked for a passphrase. Only the servername of the remote RAC node should be displayed:

oracl e$ ssh r ac1pub host namerac1puboracl e$ ssh r ac1pr v host name




14/20

rac1puboracl e$ ssh r ac2pub host namerac2puboracl e$ ssh r ac2pr v host namerac2puboracl e$ ssh r ac3pub host namerac3puboracl e$ ssh r ac3pr v host namerac3pub

NOTE: Keep this terminal open since this is the terminal that will be used for running runInstaller !

Checking OCFS and Oracle Environment Variables

Checking OCFSs

Make sure the OCFS filesystem(s) are mounted on all RAC nodes:

oracl e$ ssh r ac1pub df | grep or adata/ dev/ sda1 51205216 33888 51171328 1%/ u02/ oradat a/ orc loracl e$ ssh r ac2pub df | grep or adata/ dev/ sda1 51205216 33888 51171328 1%/ u02/ oradat a/ orc loracl e$ ssh r ac3pub df | grep or adata/ dev/ sda1 51205216 33888 51171328 1%/ u02/ oradat a/ orc l

Checking Oracle Environment Variables

Run the following command on all RAC nodes:

su - oracl e

$ set | grep ORAORACLE_BASE=/ u01/app/ oracl eORACLE_SI D=or cl 1$

It is important that $ORACLE_SI D is different on each RAC node!It is also recommended that $ORACLE_ HOME is not set but that OUI selects the home directory.

Installing Oracle 10g Cluster Ready Services (CRS) R1 (10.1.0.2)

In order to install the Cluster Ready Services (CRS) R1 (10.1.0.2) on all RAC nodes, OUI has to be launched on only one RAC node. In my example I will run OUIalways on rac1pub.

To install CRS, insert the "Cluster Ready Services (CRS) R1 (10.1.0.2)" CD (downloadedd image name: " shi p. crs. cpi o. gz"), and mount it on e.g. rac1pub:

su - r ootmount / mnt / cdrom

Use the oracl e terminal that you prepared for ssh at Automating Authentication for oracle ssh Logins and execute runI nstal l er :

oracl e$ / mnt/ cdr om/ r unI nstal l er

- Wel come Screen: Cl i ck Next- I nvent ory di r ect ory and credenti al s:

Cl i ck Next- Uni x Gr oup Name: Use "oi nstal l ".- Root Scri pt Wi ndow: Open anot her wi ndow, l ogi n as r oot, and run / t mp/or ai nstRoot . sh

on t he node wher e you l aunched r unI nstal l er. Af ter you' ve run the scri pt, cl i ck Conti nue.- F i l e Locati ons: I used the r ecommended def aul t val ues:

Dest i nati on Name: Or aCr10g_home1 Desti nat i on Pat h: / u01/app/oracle/ product/ 10.1.0/ crs_1

Cl i ck Next- Language Sel ecti on: Cl i ck Next- Cl uster Conf i gurati on:

Cl ust er Name: cr s Cl ust er Nodes: Publ i c Node Name: r ac1pub Pr i vat e Node Name: r ac1prv Publ i c Node Name: r ac2pub Pr i vat e Node Name: r ac2prv Publ i c Node Name: r ac3pub Pr i vat e Node Name: r ac3prv

Cl i ck Next- Pri vat e I nterconnect Enf orcement :

I nter f ace Name: eth0 Subnet: 192.168. 1.0 I nterf ace Type: Public I nter f ace Name: eth1 Subnet: 192.168. 2.0 I nterf ace Type: Private

Cl i ck Next- Oracl e Cl uster Regi str y:

OCR Locat i on: / u02/oradata/orcl / OCRFi l eCl i ck Next

- Voti ng Di sk: Voti ng di sk f i l e name: / u02/ oradata/orcl / CSSFi l eCl i ck Next

- Root Scr i pt Wi ndow:Open another wi ndow, l ogi n as r oot, and execut e/ u01/ app/ oracl e/ oraI nventor y/ orai nstRoot . sh on ALL RAC Nodes!

NOTE: For any reason Or acl e does not cr eat e the l og di r ector y "/ u01/ app/ oracl e/ product/ 10. 1. 0/ crs_1/l og". I f there are probl ems wi th CRS, i t wi l l create l og f i l es i n th is di rectory, but onl y i f i t exi sts .

Theref or e make sur e t o cr eate t hi s di r ect ory as or acl e: oracl e$ mkdi r / u01/app/ oracl e/product / 10. 1.0/ crs_1/ l og

Af ter you' ve run the scri pt, cl i ck Conti nue.- Set up Pri vi l eges Scri pt Wi ndow:

Open another wi ndow, l ogi n as r oot, and execut e/ u01/ app/ oracl e/ product / 10. 1. 0/ cr s_1/ r oot . sh on ALL RAC Nodes one by one!

Note t hat hi s can t ake a whi l e. On t he l ast RAC node, t he out put of t hescri pt was as fol l ows:

. . . CSS i s acti ve on t hese nodes.




15/20

r ac1pub r ac2pub r ac3pub CSS i s acti ve on al l nodes. Or acl e CRS st ack i nstal l ed and r unni ng under i ni t ( 1M)

Cl i ck OK - Summar y: Cl i ck I nstal l- When i nstal l ati on i s compl et ed, cl i ck Exi t.

One way to verify the CRS installation is to display all the nodes where CRS was installed:

oracl e$ / u01/app/oracle/ product/ 10.1.0/ crs_1/bi n/ olsnodes - n

rac1pub 1rac2pub 2rac3pub 3

Installing Oracle Database 10g Software with Real Application Clusters (RAC)

General

The following procedure shows the installation of the software for Oracle Database 10g Software R1 (10.1.0.2) with Real Application Clusters (RAC).

Note that Oracle Database 10g R1 (10.1) OUI will not be able to discover disks that are marked as Linux ASMLib. Therefore it is recommended to complete thesoftware installation and then to use dbca to create the database, see http://otn.oracle.com/tech/linux/asmlib/install.html#10gr1 for more information.


Before you install the Oracle Database 10g Software with Real Application Clusters (RAC) R1 (10.1.0.2), it is important that you followed the steps as outlined inAutomating Authentication for oracle ssh Logins.

Checking Oracle Environment Variables

Run the following command on all RAC nodes:

su - oracl e$ set | grep ORAORACLE_BASE=/ u01/app/ oracl eORACLE_SI D=or cl 1$

It is important that $ORACLE_SI D is different on each RAC node!It is also recommended that $ORACLE_ HOME is not set but that OUI selects the home directory.

Installing Oracle Database 10g Software R1 (10.1.0.2) with Real Application Clusters (RAC)

In order to install the Oracle Database 10g R1 (10.1.0.2) Software with Real Application Clusters (RAC) on all RAC nodes, OUI has to be launched on only one RACnode. In my example I will run OUI on rac1pub.

To install the RAC Database software, insert the Oracle Database 10g R1 (10.1.0.2) CD (downloaded image name: "shi p. db. cpi o.gz"), and mount it on e.g. rac1pub:

su - r ootmount / mnt / cdrom

Use the oracl e terminal that you prepared for ssh at Automating Authentication for oracle ssh Logins, and execute runI nstal l er :

oracl e$ / mnt/ cdr om/ r unI nstal l er

- Wel come Screen: Cl i ck Next- F i l e Locati ons: I used the defaul t val ues:

Dest i nati on Name: r aDb10g_home1Desti nat i on Pat h: / u01/app/ oracl e/product/ 10. 1.0/ db_1Cl i ck Next .

- Hardware Cl uster I nstal l ati on Mode:Sel ect "Cluster I nstal l ati on"

Cl i ck "Sel ect Al l " t o sel ect all servers: rac1pub, rac2pub, rac3pubCl i ck Next

NOTE: I f i t st ops here and the st atus of a RAC node i s " Node not r eachabl e", t hen perf orm t he f ol l owi ng checks:

- Check i f t he node where you l aunched OUI i s abl e to do ssh wi t hout a passphrase t o t he RAC node wher e t he st atus i s set t o "Node not r eachabl e".

- Check i f t he CRS i s r unni ng t hi s RAC node.- I nstal l ati on Type:

I sel ected "Enterpri se Edi ti on".Cl i ck Next .

- Product- speci f i c Prerequisi t e Checks: Make sure that the status of each Check is set to "Succeeded".Cl i ck Next

- Database Conf i gur ati on:I sel ected "Do not cr eat e a st art er database" si nce we have t o create t hedatabase wi t h dbca. Or acl e Database 10g R1 (10. 1) OUI wi l l not be abl e todi scover di sks t hat are marked as Li nux ASMLi b. For more i nf ormati on, see

htt p: / / otn.oracl e.com/ tech/ l i nux/asml i b/i nstal l . html #10gr1Cl i ck Next

- Summar y: Cl i ck I nstal l- Set up Pri vi l eges Wi ndow:

Open anot her wi ndow, l ogi n as r oot, and execut e/ u01/app/or acl e/pr oduct/ 10. 1. 0/db_1/ r oot. sh on ALL RAC Nodes one by one!NOTE: Make al so sure t hat X i s rel i nked t o your l ocal desktop si nce t hi s

scri pt wi l l l aunch the "VI P Conf i gurati on Assi stant" tool whi ch i s aGUI based uti l i ty!

VI P Conf i gurat i on Assi st ant Tool :




16/20

( This Assistant tool will come up only once when root.sh is executed thefirst time in your RAC cluster )

- Wel come Cl i ck Next- Network I nterf aces: I sel ect ed bot h i nterf aces, eth0 and et h1.

Cl i ck Next- Vi rt ual I Ps f or cl uster nodes:

(for the alias names and IP address, see Setting Up the /etc/hosts File)

Node Name: r ac1pub I P Al i as Name: rac1vi p I P address: 192. 168. 1. 51 Subnet Mask: 255. 255. 255. 0

Node Name: r ac2pub

I P Al i as Name: rac2vi p I P address: 192. 168. 1. 52 Subnet Mask: 255. 255. 255. 0

Node Name: r ac3pub I P Al i as Name: rac3vi p I P address: 192. 168. 1. 53 Subnet Mask: 255. 255. 255. 0

Cl i ck Next- Summar y: Cl i ck Fi ni sh- Confi gurat i on Assi st ant Progress Di al og:

Cl i ck OK aft er conf i gurat i on i s compl ete.- Conf i gurati on Resul ts:

Cl i ck Exit

Cl i ck OK to cl ose the Setup Pr i vi l ege Wi ndow.

- End of I nstal l ati on:Cl i ck Exit

If OUI terminates abnormally (happend to me several times), or if anything else goes wrong, remove the following files/directories and start over again:

su - oracl erm - rf / u01/ app/ oracl e/ product/ 10. 1.0/db_1

Installing Oracle Database 10g with Real Application Cluster (RAC)

General

The following steps show how to use dbca to create the database and its instances. Oracle recommends to use dbca to create the RAC database since the preconfigured

databases are optimized for ASM, server parameter file, and automatic undo management. dbca also makes it much more easier to create new ASM disk groups etc.


Before you install a RAC database, it is important that you followed the steps as outlined in Automating Authentication for oracle ssh Logins.

Setting Oracle Environment Variables

Since the Oracle RAC software is already installed, $ORACLE_ HOME can now be set to the home directory that was choosen by OUI.

The following steps should now be performed on all RAC nodes! It is very important that these environment variables are set permanently for oracle on all RACnodes!

To make sure $ORACLE and $PATH are set automatically each time oracle logs in, add these environment variables to the ~oracl e/ . bash_prof i l e file which is the

user startup file for the Bash shell on Red Hat Linux. To do this you could simply copy/paste the following commands to make these settings permanent for youroracle's Bash shell (the path might differ on your system!):

su - oracl ecat >> ~oracl e/. bash_prof i l e


17/20

Cl i ck Next- Database Cr edent i al s:

I sel ected "Use t he Same Password f or Al l Accounts" . Ent er t he password andmake sur e the password does not st art wi t h a di gi t number .Cl i ck Next

- Storage Opti ons: I sel ect ed "Automat i c St orage Management ( ASM) ", seeI nst al l i ng and Conf i guri ng Automat i c Stor age Management ( ASM) and Di sksCl i ck Next

- Cr eat e ASM I nstance:Ent er t he SYS password f or t he ASM i nstance.I sel ected the def aul t paramet er f i l e (I FI LE):"{ORACLE_BASE}/ admi n/+ASM/ pfi l e/i ni t . ora"Cl i ck Next

At t hi s poi nt DBCA wi l l cr eat e and st art t he ASM i nstance on al l RAC nodes.Cl i ck OK to cr eat e and st art t he ASM i nstance.

An err or wi l l come up t hat oratab can' t be copi ed t o / t mp. I i gnored t hi s err or.

I f you get "ORACLE server sessi on t ermi nat ed by f atal err or" , t hen you probabl y di dn' t f ol l ow the steps at Setti ng Up the / etc/hosts Fi l e

- ASM Di sk Groups: - Cl i ck "Create New"

Creat e Di sk Gr oup Wi ndow:- Cl i ck "Change Di sk Di scover y Pat h".- Ent er " ORCL:VOL*" f or Di sk Di scovery Pat h.

The di scovery st ri ng f or f i ndi ng ASM di sks must be prefi xed wi t h "ORCL: ", and i n my exampl e I cal l ed t he ASM di sks VOL1, VOL2, VOL3.

- I enter ed an arbi t r at y Di sk Gr oup Name: ORCL_DATA1- I checked t he candi date: "ORCL:VOL1" and "ORCL:VOL2" whi ch have

t oget her about 60 GB space i n my confi gur at i on.- Cl i ck OK.

- Check the new cr eated di sk gr oup "ORCL_DATA1".

- Cl i ck Next- Database Fi l e Locati ons:Sel ect " Use Or acl e-Managed Fi l es"

Database Ar ea: +ORCL_ DATA1Cl i ck Next

- Recovery Conf i gur ati on: Usi ng recovery opt i ons l i ke Fl ash Recover y Ar ea i s out of scope f or t hi s art i cl e. So I di d not sel ect any recovery opt i ons.

Cl i ck Next- Database Cont ent : I di d not s el ect Sampl e Schemas or Cust omScr i pts.

Cl i ck Next- Database Ser vi ces: Cl i ck "Add" and enter a Ser vi ce Name: I ent ered "or cl t est" .

I sel ect ed TAF Pol i cy "Basi c".Cl i ck Next

- I ni t i al i zati on Parameters:Change sett i ngs as needed.Cl i ck Next

- Database Stor age: Change sett i ngs as needed.Cl i ck Next

- Creat i on Opti ons: Check "Create Dat abase"Cl i ck Fi ni sh

- Summar y: Cl i ck OK

Now t he dat abase i s bei ng cr eat ed.

The f ol l owi ng er r or message came up: "Unabl e t o copy t he f i l e "r ac2pub:/ etc/ oratab" t o "/ t mp/orat ab. rac2pub".

I cl i cked "I gnore". I have to i nvesti gate thi s.

Your RAC cluster should now be up and running. To verify, try to connect to each instance from one of the RAC nodes:

$ sql pl us syst em@orcl 1$ sql pl us syst em@orcl 2$ sql pl us syst em@orcl 3

After you connected to an instance, enter the following SQL command to verify your connection:

SQL> sel ect i nstance_name f r omv$i nstance;

Post-Installation Steps

Transparent Application Failover (TAF)

Introduction

Processes external to the Oracle 10g RAC cluster control the Transparent Application Failover (TAF). This means that the failover types and methods can be unique foreach Oracle Net client. The re-connection happens automatically within the OCI library which means that you do not need to change the client application to use TAF.

Setup

To test TAF on the new installed RAC cluster, configure the t nsnames. ora file for TAF on a non-RAC server where you have either the Oracle database software orthe Oracle client software installed.

Here is an example how my / opt / oracl e/product / 9.2. 0/net work/ admi n/t nsnames.or a: looks like:

ORCLTEST = ( DESCRI PTI ON = ( ADDRESS = ( PROTOCOL = TCP) ( HOST = r ac1vip) ( PORT = 1521) ) ( ADDRESS = ( PROTOCOL = TCP) ( HOST = r ac2vip) ( PORT = 1521)




18/20

( ADDRESS = ( PROTOCOL = TCP) ( HOST = r ac3vip) ( PORT = 1521) ) ( LOAD_BALANCE = yes) ( CONNECT_DATA = ( SERVER = DEDI CATED) ( SERVI CE_NAME = orcl ) ( FAI LOVER_MODE = ( TYPE = SELECT) ( METHOD = BASI C) ( RETRI ES = 180) ( DELAY = 5) ) ) )

The following SQL statement can be used to check the sessions's failover type, failover method, and if a failover has occured:

sel ect i nst ance_name, host _name, NULL AS f ai l over _t ype, NULL AS f ai l over_method, NULL AS f ai l ed_over FROM v$i nstance UNI ON SELECT NULL, NULL, f ai l over_t ype, f ai l over_method, f ai l ed_over FROM v$sessi on WHERE user name = ' SYSTEM' ;

Example of a Transparent Application Failover (TAF)

Here is an example of a Transparent Application Failover:

su - oracl e$ sql pl us system@orcltest

SQL> sel ect i nst ance_name, host _name, 2 NULL AS f ai l over _t ype, 3 NULL AS f ai l over_method, 4 NULL AS f ai l ed_over 5 FROM v$i nstance 6 UNI ON 7 SELECT NULL, NULL, f ai l over_t ype, f ai l over_method, f ai l ed_over 8 FROM v$sessi on 9 WHERE username = ' SYSTEM' ;

I NSTANCE_NAME HOST_NAME FAI LOVER_TYPE FAI LOVER_M FAI- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -orcl1 rac1pub SELECT BASIC NO

SQL>

The above SQL statement shows that I'm connected to " rac1pub" for instance "orcl1".

In this case, execute shut down abort on "rac1pub" for instance "orcl1":

SQL> shut down abortORACLE i nst ance shut down.SQL>

Now rerun the SQL statement:

SQL> sel ect i nst ance_name, host _name, 2 NULL AS f ai l over _t ype, 3 NULL AS f ai l over_method, 4 NULL AS f ai l ed_over 5 FROM v$i nstance 6 UNI ON 7 SELECT NULL, NULL, f ai l over_t ype, f ai l over_method, f ai l ed_over 8 FROM v$sessi on 9 WHERE username = ' SYSTEM' ;

I NSTANCE_NAME HOST_NAME FAI LOVER_TYPE FAI LOVER_M FAI

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -orcl2 rac2pub SELECT BASIC YES

SQL>

The SQL statement shows that the sessions has failed over to instance "orcl2". Note that this can take a few seconds.

Checking Automatic Storage Management (ASM)

Here are a couple SQL statements to verify ASM.

Run the following command to see which data files are in which disk group:

SQL> sel ect name f rom v$dat afi l e 2 uni on 3 sel ect name fr omv$control f i l e

4 uni on 5 sel ect member f rom v$l ogf i l e;

NAME- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -+ORCL_DATA1/orcl / cont rol f i l e/curr ent . 260.3+ORCL_DATA1/ orcl / dat af i l e/s ysaux. 257. 1




19/20

+ORCL_DATA1/or cl / dat af i l e/s yst em. 256. 1+ORCL_DATA1/or cl / dat af i l e/undotbs1. 258. 1+ORCL_DATA1/or cl / dat af i l e/undotbs2. 264. 1+ORCL_DATA1/or cl / dat af i l e/users . 259. 1+ORCL_DATA1/or cl / onl i nel og/ group_1. 261. 1+ORCL_DATA1/or cl / onl i nel og/ group_2. 262. 1+ORCL_DATA1/or cl / onl i nel og/ group_3. 265. 1+ORCL_DATA1/or cl / onl i nel og/ group_4. 266. 1

10 r ows sel ect ed.

SQL>

Run the following command to see which ASM disk(s) belong to the disk group 'ORCL_ DATA1':(ORCL_DATA1 was specified in Installing Oracle Database 10g with Real Application Cluster )

SQL> sel ect pat h fr om v$asm_di sk wher e gr oup_number i n 2 ( sel ect group_number f r om v$asm_di skgroup where name = ' ORCL_DATA1' ) ;

PATH- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -ORCL: VOL1ORCL: VOL2

SQL>

Oracle 10g RAC Issues, Problems and Errors

This section describes other issues, problems and errors pertaining to installing Oracle 10g with RAC which has not been covered so far.

Gt k-WARNI NG **: l i bgdk_pi xbuf. so. 2: cannot open shared obj ect f i l e: No such fi l e or di rect ory

This error can come up when you run ocfst ool . To fix this error, install the gdk- pi xbuf RPM:

rpm- i vh gdk- pi xbuf- 0. 18. 0-8.1. i 386. rpm

/ u01/app/ oracl e/product/ 10.1. 0/ crs_1/bi n/crs_st at. bin: err or whi l e l oading shared l i brari es: l i bstdc++- l i bc6. 2- 2.so. 3: cannot open/ u01/app/ oracl e/product/ 10.1. 0/ crs_1/bi n/crs_st at. bin: err or whi l e l oading shared l i brari es: l i bstdc++- l i bc6. 2- 2.so. 3: cannot openPRKR- 1061 : Fai l ed to r un remote command t o get node conf i gurat i on for node rac1pupPRKR- 1061 : Fai l ed to r un remote command t o get node conf i gurat i on for node rac1pup

This error can come up when you run root. sh. To fix this error, install the compat - l i bstdc++RPM and rerun root. sh:

rpm - i vh compat - l i bstdc++- 7.3- 2.96.122. i 386.r pm

mount: f s t ype ocf s not suppor t ed by kernel

The OCFS kernel module was not loaded. See Configuring and Loading OCFS for more information.

ORA- 00603: ORACLE server sess i on t ermi nated by f atal err ororSQL> st ar t up nomountORA- 29702: er r or occurr ed i n Cl uster Gr oup Servi ce oper ati on

If the trace file looks like this:

/ u01/app/ oracl e/product / 10. 1.0/ db_1/r dbms/ l og/ orcl 1_ora_7424. t rc. . .kgef ec: f atal err or 0*** 2004- 03- 13 20: 50: 28. 201ksedmp: i nternal or f atal err orORA- 00603: ORACLE server sess i on t ermi nated by f atal err orORA- 27504: I PC err or c r eati ng OSD context

ORA- 27300: OS syst emdependent operat i on: get host byname f ai l ed wi t h st atus: 0ORA- 27301: OS fai l ure message: Er r or 0ORA- 27302: f ai l ure occur r ed at: sskgxpmyi p4Curr ent SQL i nformati on unavai l abl e - no sessi on.- - - - - Cal l Stack Trace - - - - -cal l i ng cal l ent ry argument val ues i n hexl ocati on t ype poi nt ( ? means dubi ous val ue)- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -- - - - - - - - - - - -ksedmp( ) +493 cal l ksedst ( ) +0 0 ? 0 ? 0 ? 1 ? 0 ? 0 ?ksf dmp( ) +14 cal l ksedmp( ) +0 3 ? BFFF783C ? A483593 ? BF305C0 ? 3 ? BFFF8310 ?

Make sure that the name of the RAC node is not listed for the loopback address in the / etc/ hosts file similar to this example:

127.0. 0.1 rac1pub l ocal host . l ocal domai n l ocal host

The entry should rather look like this:

127.0. 0.1 l ocal host . l ocal domai n l ocal host

References

Oracle's Linux Center




20/20

Oracle Database 10g DocumentationInstalling Oracle9i Real Application Cluster (RAC) on Red Hat Linux Advanced Server 2.1Project Documentation: OCFSInstalling Oracle ASMLib

Copyright © 2007 PUSCHITZ.COM

The information provided on this website comes without warranty of any kind and is distributed AS IS. Every effort has been made to provide the information as accurate as possible, but no warranty orfitness is implied. The information may be incomplete, may contain errors or may have become out of date. The use of this information described herein is your responsibility, and to use it in your own

environments do so at your own risk.


Installing Oracle Database 10g With Real Application Cluster (RAC) on Red Hat Enterprise Linux...

Documents

Transcript of Installing Oracle Database 10g With Real Application Cluster (RAC) on Red Hat Enterprise Linux...