Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 1

Step By Step to Install Oracle Grid Infra-Structure 11.2.0.3 on Solaris 11.1

Introduction

Oracle Cluster ware is portable cluster software that allows clustering of independent servers so that they cooperate as a single system. Oracle

Cluster ware was first released with Oracle Database 10g Release 1 as the required cluster technology for Oracle Real Application Clusters

(RAC). Oracle Cluster ware is an independent cluster infrastructure, which is fully integrated with Oracle RAC, capable of protecting any kind of

application in a failover cluster.

Oracle Grid Infrastructure introduces a new server pool concept allowing the partitioning of the grid into groups of servers. “Role-separated

Management” can be used by organizations, in which cluster, storage, and database management functions are strictly separated. Cluster-aware

commands and an Enterprise Manager based cluster and resource management simplify grid management regardless of size. Further enhancements in Oracle ASM, like the new ASM cluster file system or the new dynamic

Volume manager, complete Oracle’s new Grid Infrastructure solution.

Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 2

About the Author

Osama Mustafa – Oracle ACE, a database specialist, Certified Oracle Professional (10g, 11g), Certified Ethical hacker (Penetration testing),

and Sun System Administrator, author of book Oracle Penetration Testing. Publishes many articles, including Oracle database articles in his blog.

Twitter: @OsamaOracle.

G+: Osama Mustafa.

SlidShare: Osama Mustafa. LinkedIn: http://www.linkedin.com/in/osamamustafa.

Blog: http://osamamustafa.blogpsot.com.

The Final Architecture:

The Below Diagram Describe what is my setup, Note there’s no IP , Hostname will be mentioned in this document and if you notice any It will

not be the real one, This Part of the document only Describe the Grid and Database installation for Fusion middlware part it will be Published

Separately. Also if you follow this document included with other document you can build full Environment Configured and ready to use with

Single-Sign-On.

Rac Node Number: 2

Database Service Name: TEST Database Location (Home): /u01/app/oracle/product/11.2.0/dbhome_1

Grid Location (Home): /u01/app/11.2.0/grid

Operating System (OS): Oracle SOLARIS SPARC 11.1

Note: All Software was downloaded from oracle OTN Here and e-delivery here.

Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 3

Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 4

Now after you already know what the setup will look like, Every Information about Locations, and Finally Operating system, during installation

for I faced lot of bugs Since Solaris 11.1 New But it was amazing experience and learn something new.

Let’s Start:

Step #1:

You need to know how the ect/hosts will look like after adding IP’s:

#########NODES######### 180.111.20.21 Test-db1

180.111.20.22 Test-db2

######################## #########NODE-One-IP###########

180.111.20.28 Test-db1-vip

10.0.0.1 Test-db1-priv ################################

#########NODE-Two-ip############

180.111.20.29 Test-db2-vip 10.0.0.2 Test-db2-priv

################################

######SCAN-IP################## 180.111.20.30 Test-db-scan

###############################

Step #2:

Check OS Version Using the below command:

/usr/platform/’uname –I’/sbin/prtdiag

Step #3 (Optional):

Because I was working remotely not directly from Data Center, I configure Vncserver to enable access to server GUI and Run the Installer from

there.

a. Install Required Package using

pkg info SUNWxvnc

b. Add the below line to /etc/services or use step “d” as command line

vnc-server 5900/tcp # Xvnc

c. Configure /etc/X11/gdm/custom/conf

[xdmcp]

Enable=true

[security] DisallowTCP=false

AllowRoot=true AllowRemoteRoot=true

d. Instead of step “b” you can do the below , I just want to mention both :

svccfg -s x11-server setprop options/tcp_listen=true

svccfg -s xvnc-inetd setprop inetd/wait=true svccfg -s xvnc-inetd

setprop inetd_start/exec=astring:"/usr/bin/Xvnc -geometry 1280x720 -inetd -query

localhost -once securitytypes=none" e. Finally Enable Services

svcadm restart gdm xvnc-inetd

svcadm disable gdm xvnc-inetd; svcadm enable gdm xvnc-inetd

Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 5

f. Just to make sure my work was Right I restart the Server and check vncsever again.

Step #4 (Optional):

1- I was thinking should I copy oracle software 4 times and it’s almost 35 GB so 35*4 you are talking about huge miss of time so

what I did here copy the files Once on one server and Configure NFS to share it between all Nodes, more easy and Save your

time.

a. First you have to enable NFS on All Server using the below command

svcadm enable nfs/server

svcadm restart nfs/server

b. If you copied files on server one then this command should be done on server one, if you copied the files on server 2

this command should be done on server 2. Depend where you copied software.

share -F nfs -o rw /base

c. On All Server you can run The Mount command to Share all the files and start setup, More Easy Save time.

mount -F nfs Base-Server-IP:software-location-on-remote-server mount-point-on-all-servers

Step #5: Prerequisite

As any Linux/Unix There is Prerequisite for Operating system follow the below

1. Users And Groups

Oracle Solaris 11 provide you with Command Called ZFS (amazing command to manage File system), I Create Oracle Home Using

this command, also create mount point /u01

This is For Oracle User

zfs create -o mountpoint=/u01 rpool/u01

Create Groups

groupadd -g 1001 oinstall groupadd -g 1002 dba

groupadd -g 1003 oper

Create Oracle User

zfs create -o mountpoint=/export/home/oracle rpool/export/home/oracle

useradd -g oinstall -G dba oracle

passwd oracle

Create necessary Directory for Our Installation

chown -R oracle:oinstall /export/home/oracle

mkdir -p /u01/app/oracle

chown oracle:oinstall /u01/app/oracle chmod -R 775 /u01/

chown -R oracle:oinstall /u01

Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 6

This is For Grid User

Create Group for Grid User

groupadd -g 1020 asmadmin

groupadd -g 1022 asmoper groupadd -g 1021 asmdba

Create Grid User

zfs create -o mountpoint=/export/home/grid rpool/export/home/grid

useradd -g oinstall -G dba grid usermod -g oinstall -G dba,asmdba grid

passwd grid

chown -R grid:oinstall /export/home/grid

Create necessary Directory for Our Installation

mkdir -p /u01/app/11.2.0/grid

mkdir -p /u01/app/grid

chown grid:oinstall /u01/app/11.2.0/grid chown grid:oinstall /u01/app/grid

#!important!# chown -R grid:oinstall /u01

Step #6:

Configure .Profile which is located in /export/home/oracle and /export/home/grid

export ORACLE_BASE=/u01/app/oracle/

export ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1 export GRID_HOME=/u01/app/11.2.0/grid/

export ORACLE_SID=TEST1

export PATH=$PATH:/usr/sbin:/usr/X11/bin:/usr/dt/bin:/usr/openwin/bin:/usr/sfw/bin:/usr/sfw/sbin: /usr/ccs/bin: /usr/local/bin:/usr/local/sbin:

$ORACLE_HOME /bin:$GRID_HOME/bin:.

Note: copy .profile to all Database Servers using scp command

cd /export/home/oracle

scp .profile oracle@Server-ip:/export/home/oracle/

Step #7:

Oracle Grid and Networking Notes:

1. The broadcast must work across any configured VLANs as used by the public or private interfaces.

2. Across the broadcast domain as defined for the private interconnect

3. On the IP address subnet ranges 224.0.0.0/24 and 230.0.1.0/24

Regarding to oracle to you need to check udp time using the below command

ndd /dev/udp udp_xmit_hiwat

ndd /dev/udp udp_recv_hiwat

To avoid reboot you can set it on memory and reboot time using the below

On Memory

ndd -set /dev/udp udp_xmit_hiwat 65536

ndd -set /dev/udp udp_recv_hiwat 65536

Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 7

On Reboot

ndd -set /dev/udp udp_xmit_hiwat 65536 ndd -set /dev/udp udp_recv_hiwat 65536

Step #8:

In this step you need to make sure of disks on both nodes, in my case I am using EMC storage.

List Disk Using

/usr/sbin/format

fdisk the raw disk using fdisk command.

Change owner to grid using Chown grid:asmadmin /dev/rdsk/Disk-name

Chmod 660 /dev/rdsk/disk-name

You can list your available using ls –ltr /dev/rdsk/emc*

Change owner for these disk Chown 660 emc*

Chown grid:asmadmin emc*

Note: In Unix Each Disk have slide from 0-6 like the below :

crw-rw---- 1 grid asmdba 302, 0 Apr 23 19:02 emcp@0:a,raw

crw-rw---- 1 grid asmdba 302, 8 Apr 23 20:38 emcp@1:a,raw

crw-rw---- 1 grid asmdba 302, 16 Apr 22 14:05 emcp@2:a,raw crw-rw---- 1 grid asmdba 302, 24 Apr 23 21:00 emcp@3:a,raw

crw-rw---- 1 grid asmdba 302, 32 Apr 23 21:00 emcp@4:a,raw

crw-rw---- 1 grid asmdba 302, 40 Apr 23 21:00 emcp@5:a,raw crw-rw---- 1 grid asmdba 302, 48 Apr 22 14:05 emcp@6:a,raw

Step #9:

Disable ntp using svcadm disable ntp

Step #10:

By Default Oracle Solaris SPARC prevent root access, for example we created oracle and grid user but we cannot access to root using su

command to enable root access for oracle and grid do below

Edit file /etc/user_attr

Add the below lines

oracle::::defaultpriv=basic,net_privaddr;roles=root

grid::::defaultpriv=basic,net_privaddr;roles=root

Step #11:

Now we need to configure memory Parameter for Oracle and Grid User and make permanent

To check memory Current Value for Both user

prctl -n project.max-shm-memory -i process $$

Modify memory values using

projmod -a -K "project.max-shm-memory=(privileged,32G,deny)" -U oracle default

projmod -a -K "project.max-shm-memory=(privileged,32G,deny)" -U grid default

projmod -s -K "project.max-shm-memory=(privileged,32G,deny)" default

Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 8

To make sure new memory value effect applied on oracle and grid user open new terminal and run prctl command again.

Step #12: During installation Oracle will check swap memory so you need to increase swap memory depend on your setup for sure I will use ZFS

command.

Check swap value

bash-3.00# swap -lh

swapfile dev swaplo blocks free

/dev/zvol/dsk/rpool/swap 256,1 16 4194288 4194288

Remove Swap using root user

bash-3.00# swap -d /dev/zvol/dsk/rpool/swap

Configure New Swap

bash-3.00# zfs set volsize=20G rpool/swap

bash-3.00# swap -a /dev/zvol/dsk/rpool/swap

Check New Value

bash-3.00# swap -lh

Step #13:

One more step Configure SSH Between Nodes You can do this Step During installation or Using the the Below command, Oracle Provide you

with new way, you can now configure SSH using sshsetup it’s already exists within media

For Oracle User

# ./sshUserSetup.sh -hosts " node1 node2 node3 node4" -user oracle -advanced –noPromptPassphrase

For Grid User

# ./sshUserSetup.sh -hosts " node1 node2 node3 node4" -user grid -advanced –noPromptPassphrase

Congratulations you can start your setup now!!

Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 9

Step #14:

Start Install Grid Infrastructure, in my case I choose to install Software only then Configure ASM in this way if error appeared I will know where

to start troubleshooting. Follow the screens ( this steps should be Done As Grid User )

Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 10

Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 11

Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 12

Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 13

Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 14

Step #15:

After success installation now I need to Configure ASM ( This Step should be Done as Grid User) , From One Node Only.

export ORACLE_HOME=/u01/app/11.2.0/grid/

cd $ORACLE_HOME/bin Run ./asmca

The Below Screen Should be Open

Three ASM Disks should be created:

1- DATA : DataFiles and Parameter files ( should be big )

2- FRA : FLASH RECOVERY AREA ( should be big )

3- CRS : this Disk for OCR and VOTING ( 4-5G will be enough )

After installation done you have to see MOUNTED on Both Node.

REDUNDANCY:

NORMAL REDUNDANCY - Two-way mirroring, requiring two failure groups.

HIGH REDUNDANCY - Three-way mirroring, requiring three failure groups.

EXTERNAL REDUNDANCY - No mirroring for disks that are already protected using hardware mirroring or RAID. If you have hardware RAID it should be used in preference to ASM redundancy, so this will be the standard option for most installations.

Step #16:

Finally You Are done with Grid Infrastructure and now you should configure Database on RAC (this step should be done as Oracle User),

Usually I am installing Software only then Called dbca to configure Instance Install Software.

Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 15

Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 16

Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 17

Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 18

Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 19

Done!!

You can reboot node to test your Setup.

Reference:

1- Oracle Documentation Here.

2- Oracle White Paper Here