Post on 15-Apr-2020
PART 1 of 4 - GRID INFRASTRUCTURE INSTALLATION
Environment details:
Red Hat Enterprise Linux Server release 6.4 (Santiago)
Oracle 11.2.0.3 PSU4
Preinstallation Steps
## Information provided by OS admin:
[root@laguna2:/u01/orainst]# nslookup laguna-scan.turkcell.tgc
Server: 10.210.154.150
Address: 10.210.154.150#53
Name: laguna-scan.turkcell.tgc
Address: 10.200.160.196
Name: laguna-scan.turkcell.tgc
Address: 10.200.160.197
Name: laguna-scan.turkcell.tgc
Address: 10.200.160.213
[root@laguna2:~]# oracleasm listdisks
CONF1
CONF2
CONF3
CONF4
CONF5
PROD1
PROD2
PROD3
Laguna1.turkcell.tgc
Root pass:123456
## OS version
[grid@laguna1 raw]$ uname -a
Linux laguna1 2.6.32-358.el6.x86_64 #1 SMP Tue Jan 29 11:47:41 EST 2013 x86_64
x86_64 x86_64 GNU/Linux
[grid@laguna1 raw]$ cat /etc/*-release
LSB_VERSION=base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-
4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Red Hat Enterprise Linux Server release 6.4 (Santiago)
root:oinstall /u01
root:oinstall /u01/app
oracle ORACLE_BASE
oracle:oinstall /u01/app/oracle
oracle ORACLE_HOME
oracle:oinstall /u01/app/oracle/product/11.2.0/dbhome_1
grid ORACLE_HOME
root:oinstall /u01/app/11.2.0/grid
grid $ORACLE_BASE
grid:oinstall /u01/app/grid
#### Creating required directories ####
mkdir /u01/app
chown root:oinstall /u01/app
## Create grid OB directory on both nodes.
mkdir /u01/app/grid
chown grid:oinstall /u01/app/grid
## Create grid OH directory on both nodes.
mkdir -p /u01/app/11.2.0/grid
chown -R grid:oinstall /u01/app/11.2.0/grid
## Create oracle OB directory on both nodes.
mkdir /u01/app/oracle
chown oracle:oinstall /u01/app/oracle
## Create oracle OH directory on both nodes.
mkdir -p /u01/app/oracle/product/11.2.0/dbhome_1
chown -R oracle:oinstall /u01/app/oracle/product/11.2.0/dbhome_1
## additioal rights &ownership
chmod -R 775 /u01
chown grid:oinstall /u01
[root@laguna2:/u01/orainst]# mkdir /u01/app
[root@laguna2:/u01/orainst]# chown root:oinstall /u01/app
[root@laguna2:/u01/orainst]#
[root@laguna2:/u01/orainst]# mkdir /u01/app/oracle
[root@laguna2:/u01/orainst]# chown oracle:oinstall /u01/app/oracle
[root@laguna2:/u01/orainst]#
[root@laguna2:/u01/orainst]# mkdir -p /u01/app/oracle/product/11.2.0/dbhome_1
[root@laguna2:/u01/orainst]# chown -R oracle:oinstall
/u01/app/oracle/product/11.2.0/dbhome_1
[root@laguna2:/u01/orainst]#
[root@laguna2:/u01/orainst]# mkdir -p /u01/app/11.2.0/grid
[root@laguna2:/u01/orainst]# chown -R grid:oinstall /u01/app/11.2.0/grid
[root@laguna2:/u01/orainst]#
[root@laguna2:/u01/orainst]# mkdir /u01/app/grid
[root@laguna2:/u01/orainst]# chown grid:oinstall /u01/app/grid
[root@laguna2:~]# chmod -R 775 /u01/
[root@laguna2:/u01/app]# chown grid:oinstall /u01
[root@laguna2:/u01/app]#
## Continue from below installation documentation.
##Oracle® Grid Infrastructure Installation Guide 11g Release 2 (11.2) for Linux
E22489-03
## Manual preinstallation checks, on both nodes.
#Verify System Requirements 1.3.1
grep MemTotal /proc/meminfo
grep SwapTotal /proc/meminfo
[root@laguna1:/u01/app]# grep MemTotal /proc/meminfo
MemTotal: 132141688 kB
[root@laguna1:/u01/app]#
[root@laguna1:/u01/app]# grep SwapTotal /proc/meminfo
SwapTotal: 4194296 kB
[root@laguna2:/u01/orainst]# grep MemTotal /proc/meminfo
MemTotal: 132141688 kB
[root@laguna2:/u01/orainst]# grep SwapTotal /proc/meminfo
SwapTotal: 4194296 kB
[root@laguna2:/u01/orainst]#
# Ensure you have at least 6.5 GB of space for the Oracle Grid Infrastructure /u01
disk
# Ensure that you have at least 1 GB of space in /tmp
[root@laguna1:/u01/app]# df -h /tmp
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vgroot-lvroot
20G 4.1G 15G 22% /
[root@laguna2:/u01/orainst]# df -h /tmp
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vgroot-lvroot
20G 4.1G 15G 22% /
[root@laguna1:/u01/app]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vgroot-lvroot
20G 4.1G 15G 22% /
tmpfs 64G 0 64G 0% /dev/shm
/dev/sda1 485M 87M 373M 19% /boot
/dev/mapper/oravg-u01lv
99G 7.0G 87G 8% /u01
[root@laguna2:/u01/orainst]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vgroot-lvroot
20G 4.1G 15G 22% /
tmpfs 64G 0 64G 0% /dev/shm
/dev/sda1 485M 87M 373M 19% /boot
/dev/mapper/oravg-u01lv
99G 5.9G 88G 7% /u01
## 2.Advanced Installation Oracle Grid Infrastructure for a Cluster Preinstallation
Tasks
Navigate to /u01/orainst/grid, unzipped GI installation files folder.
[grid@laguna1 grid]$ /u01/orainst/grid/runcluvfy.sh stage -pre crsinst -n
laguna1,laguna2 -fixup -verbose
/u01/orainst/cvu/bin/cluvfy stage -pre crsinst -n laguna1,laguna2 -fixup -verbose
Performing pre-checks for cluster services setup
Checking node reachability...
Check: Node reachability from node "laguna1"
Destination Node Reachable?
------------------------------------ ------------------------
laguna1 yes
laguna2 yes
Result: Node reachability check passed from node "laguna1"
Checking user equivalence...
Check: User equivalence for user "grid"
Node Name Status
------------------------------------ ------------------------
laguna2 failed
laguna1 failed
Result: PRVF-4007 : User equivalence check failed for user "grid"
ERROR:
User equivalence unavailable on all the specified nodes
Verification cannot proceed
Pre-check for cluster services setup was unsuccessful on all the nodes.
[grid@laguna1 grid]$ id grid
uid=1100(grid) gid=1000(oinstall)
groups=1000(oinstall),1020(asmadmin),1021(asmdba),1022(asmoper)
[grid@laguna2 grid]$ id grid
uid=1100(grid) gid=1000(oinstall)
groups=1000(oinstall),1020(asmadmin),1021(asmdba),1022(asmoper)
==> SSH connectivity issue
### setup grid passwordless ssh connection for each node. example below is for
grid@laguna1 to grid@laguna2. ###
[grid@laguna1 grid]$ cd ~
[grid@laguna1 ~]$ cd .ssh
[grid@laguna1 .ssh]$ ls -ltr
total 0
[grid@laguna1 .ssh]$
[grid@laguna1 .ssh]$ ls -las
total 8
4 drwx------ 2 grid oinstall 4096 May 8 14:30 .
4 drwx------ 5 grid oinstall 4096 May 8 14:40 ..
[grid@laguna1 .ssh]$
[grid@laguna1 .ssh]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/grid/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/grid/.ssh/id_rsa.
Your public key has been saved in /home/grid/.ssh/id_rsa.pub.
The key fingerprint is:
95:9f:af:20:5e:ef:28:22:bf:42:dc:b0:91:1b:c7:33 grid@laguna1
The key's randomart image is:
+--[ RSA 2048]----+
| |
| . |
| o o |
| = E . . . |
| . O oS o |
| = . . |
| . . o . |
| o ...o + . |
| +oo....+ |
+-----------------+
[grid@laguna1 .ssh]$ ls -las
total 16
4 drwx------ 2 grid oinstall 4096 May 8 14:54 .
4 drwx------ 5 grid oinstall 4096 May 8 14:40 ..
4 -rw------- 1 grid oinstall 1675 May 8 14:54 id_rsa
4 -rw-r--r-- 1 grid oinstall 394 May 8 14:54 id_rsa.pub
[grid@laguna1 .ssh]$ ssh
ssh ssh-add ssh-agent ssh-copy-id sshd ssh-keygen ssh-
keyscan
[grid@laguna1 .ssh]$ ssh-copy-id
Usage: /usr/bin/ssh-copy-id [-i [identity_file]] [user@]machine
[grid@laguna1 .ssh]$ ssh-copy-id -i id_rsa.pub grid@laguna2.turkcell.tgc
The authenticity of host 'laguna2.turkcell.tgc (10.200.160.194)' can't be
established.
RSA key fingerprint is e5:c8:18:3f:af:7b:a2:85:d0:25:21:da:f2:03:81:e2.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'laguna2.turkcell.tgc,10.200.160.194' (RSA) to the list
of known hosts.
..
grid@laguna2.turkcell.tgc's password:
Now try logging into the machine, with "ssh 'grid@laguna2.turkcell.tgc'", and check
in:
.ssh/authorized_keys
to make sure we haven't added extra keys that you weren't expecting.
[grid@laguna1 .ssh]$ ssh grid@laguna2
The authenticity of host 'laguna2 (10.200.160.194)' can't be established.
RSA key fingerprint is e5:c8:18:3f:af:7b:a2:85:d0:25:21:da:f2:03:81:e2.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'laguna2' (RSA) to the list of known hosts.
..
[grid@laguna2 ~]$
[grid@laguna1 grid]$ export DISPLAY=10.200.20.147:0.0
[grid@laguna1 grid]$
### setup oracle passwordless ssh connection for each node. example below is for
oracle@laguna2 to oracle@laguna1. ###
[oracle@laguna2 .ssh]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_rsa):
/home/oracle/.ssh/id_rsa already exists.
Overwrite (y/n)? y
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/oracle/.ssh/id_rsa.
Your public key has been saved in /home/oracle/.ssh/id_rsa.pub.
The key fingerprint is:
d8:31:d3:34:07:f5:58:3e:9e:5d:c4:99:cb:46:fc:bf oracle@laguna2
The key's randomart image is:
+--[ RSA 2048]----+
| +oo o.+|
| o o = *.|
| + . . * +|
| o + . Bo|
| . S + o|
| .|
| .|
| E |
| |
+-----------------+
[oracle@laguna2 .ssh]$
[oracle@laguna2 .ssh]$ ssh-copy-id -i id_rsa.pub oracle@laguna1.turkcell.tgc
The authenticity of host 'laguna1.turkcell.tgc (10.200.160.211)' can't be
established.
RSA key fingerprint is cf:28:91:95:74:1c:4e:eb:80:5f:02:22:a9:26:55:1a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'laguna1.turkcell.tgc,10.200.160.211' (RSA) to the list
of known hosts.
oracle@laguna1.turkcell.tgc's password:
Now try logging into the machine, with "ssh 'oracle@laguna1.turkcell.tgc'", and
check in:
.ssh/authorized_keys
to make sure we haven't added extra keys that you weren't expecting.
[oracle@laguna2 .ssh]$ ssh laguna1
The authenticity of host 'laguna1 (10.200.160.211)' can't be established.
RSA key fingerprint is cf:28:91:95:74:1c:4e:eb:80:5f:02:22:a9:26:55:1a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'laguna1' (RSA) to the list of known hosts.
[oracle@laguna1 ~]$
ssh-copy-id -i id_rsa.pub root@laguna1.turkcell.tgc
### setup root passwordless ssh connection for each node. example below is for
root@laguna1 to root@laguna2. ###
[root@laguna2:~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
06:f5:74:89:2b:6b:7e:3e:a9:54:4f:1f:15:e7:62:e9 root@laguna2
The key's randomart image is:
+--[ RSA 2048]----+
| . .... ..|
| . o... oo|
| . .. + o|
| .. . o o |
| So. . E |
| .o. o . . |
| o. .. . |
| .. + |
| .+.. |
+-----------------+
[root@laguna2:~]#
[root@laguna2:~]# cd .ssh
[root@laguna2:~/.ssh]# ls -las
total 20
4 drwx------ 2 root root 4096 May 21 13:50 .
4 dr-xr-x---. 5 root root 4096 May 21 00:46 ..
4 -rw------- 1 root root 1675 May 21 13:50 id_rsa
4 -rw-r--r-- 1 root root 394 May 21 13:50 id_rsa.pub
4 -rw-r--r-- 1 root root 629 May 8 17:19 known_hosts
[root@laguna2:~/.ssh]#
[root@laguna2:~/.ssh]# ssh-copy-id -i id_rsa.pub root@laguna1.turkcell.tgc
The authenticity of host 'laguna1.turkcell.tgc (10.200.160.211)' can't be
established.
RSA key fingerprint is cf:28:91:95:74:1c:4e:eb:80:5f:02:22:a9:26:55:1a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'laguna1.turkcell.tgc' (RSA) to the list of known hosts.
root@laguna1.turkcell.tgc's password:
Now try logging into the machine, with "ssh 'root@laguna1.turkcell.tgc'", and check
in:
.ssh/authorized_keys
to make sure we haven't added extra keys that you weren't expecting.
[root@laguna2:~/.ssh]#
### 2.8 additional manual package check ###
rpm -qa|grep elfutils-libelf-devel
rpm -qa|grep elfutils-libelf-devel
rpm -qa|grep gcc
rpm -qa|grep gcc-c++
rpm -qa|grep glibc
rpm -qa|grep glibc-common
rpm -qa|grep glibc-devel
rpm -qa|grep glibc-headers
rpm -qa|grep kernel-headers
rpm -qa|grep ksh-20060214
rpm -qa|grep libaio
rpm -qa|grep libaio-devel
rpm -qa|grep libgcc
rpm -qa|grep libgomp
rpm -qa|grep libstdc++
rpm -qa|grep libstdc++-devel
rpm -qa|grep make
rpm -qa|grep sysstat
rpm -qa|grep unixODBC
rpm -qa|grep unixODBC-devel
[root@laguna1:~]# rpm -qa|grep glibc-common
glibc-common-2.12-1.107.el6.x86_64
[root@laguna1:~]# rpm -qa|grep glibc-devel
glibc-devel-2.12-1.107.el6.x86_64
[root@laguna1:~]# rpm -qa|grep glibc-headers
glibc-headers-2.12-1.107.el6.x86_64
compat-glibc-headers-2.5-46.2.x86_64
[root@laguna1:~]# rpm -qa|grep kernel-headers
kernel-headers-2.6.32-358.el6.x86_64
[root@laguna1:~]# rpm -qa|grep ksh-20060214
[root@laguna1:~]# rpm -qa|grep ksh
mksh-39-7.el6.x86_64
ksh-20100621-19.el6.x86_64
[root@laguna1:~]# rpm -qa|grep libaio
libaio-devel-0.3.107-10.el6.x86_64
libaio-0.3.107-10.el6.x86_64
[root@laguna1:~]# rpm -qa|grep libaio-devel
libaio-devel-0.3.107-10.el6.x86_64
[root@laguna1:~]# rpm -qa|grep libgcc
libgcc-4.4.7-3.el6.i686
libgcc-4.4.7-3.el6.x86_64
[root@laguna1:~]# rpm -qa|grep libgomp
libgomp-4.4.7-3.el6.x86_64
[root@laguna1:~]# rpm -qa|grep libstdc++
compat-libstdc++-296-2.96-144.el6.i686
libstdc++-devel-4.4.7-3.el6.x86_64
compat-libstdc++-33-3.2.3-69.el6.x86_64
libstdc++-4.4.7-3.el6.x86_64
libstdc++-4.4.7-3.el6.i686
[root@laguna1:~]# rpm -qa|grep libstdc++-devel
libstdc++-devel-4.4.7-3.el6.x86_64
[root@laguna1:~]# rpm -qa|grep make
makebootfat-1.4-10.el6.x86_64
make-3.81-20.el6.x86_64
[root@laguna1:~]# rpm -qa|grep sysstat
sysstat-9.0.4-20.el6.x86_64
[root@laguna1:~]# rpm -qa|grep unixODBC
unixODBC-devel-2.2.14-12.el6_3.x86_64
unixODBC-2.2.14-12.el6_3.x86_64
[root@laguna1:~]# rpm -qa|grep unixODBC-devel
unixODBC-devel-2.2.14-12.el6_3.x86_64
[root@laguna1:~]#
## 2.10. Verifying UDP and TCP Kernel Parameters
cat /proc/sys/net/ipv4/ip_local_port_range
[root@laguna1:~]# cat /proc/sys/net/ipv4/ip_local_port_range
9000 65500
## 2.15.3 Check Resource Limits for the Oracle Software Installation Users ##
grid@laguna1 grid]$ ulimit -Sn
131072
[grid@laguna1 grid]$ ulimit -Hn
131072
[grid@laguna1 grid]$ ulimit -Su
8192
[grid@laguna1 grid]$ ulimit -Hu
16384
[grid@laguna1 grid]$ ulimit -Ss
10240
[grid@laguna1 grid]$ ulimit -Hs
unlimited
[grid@laguna1 grid]$
[root@laguna1:~]# oracleasm listdisks
CONF1
CONF2
CONF3
CONF4
CONF5
PROD1
PROD2
PROD3
PROD4
[root@laguna1:~]#
[root@laguna1:~]#
[root@laguna1:~]# oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@laguna1:~]#
[grid@laguna1 grid]$ ./runInstaller
Starting Oracle Universal Installer...
Checking Temp space: must be greater than 120 MB. Actual 15005 MB Passed
Checking swap space: must be greater than 150 MB. Actual 4095 MB Passed
Checking monitor: must be configured to display at least 256 colors. Actual
16777216 Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2013-05-21_05-
54-43PM. Please wait ...[grid@laguna1 grid]$
Installation Steps
Admin123
[root@laguna1:~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@laguna1:~]# /u01/app/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file:
/u01/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
OLR initialization - successful
root wallet
root wallet cert
root cert export
peer wallet
profile reader wallet
pa wallet
peer wallet keys
pa wallet keys
peer cert request
pa cert request
peer cert
pa cert
peer root cert TP
profile reader root cert TP
pa root cert TP
peer pa cert TP
pa peer cert TP
profile reader pa cert TP
profile reader peer cert TP
peer user cert
pa user cert
Adding Clusterware entries to upstart
CRS-2672: Attempting to start 'ora.mdnsd' on 'laguna1'
CRS-2676: Start of 'ora.mdnsd' on 'laguna1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'laguna1'
CRS-2676: Start of 'ora.gpnpd' on 'laguna1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'laguna1'
CRS-2672: Attempting to start 'ora.gipcd' on 'laguna1'
CRS-2676: Start of 'ora.cssdmonitor' on 'laguna1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'laguna1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'laguna1'
CRS-2672: Attempting to start 'ora.diskmon' on 'laguna1'
CRS-2676: Start of 'ora.diskmon' on 'laguna1' succeeded
CRS-2676: Start of 'ora.cssd' on 'laguna1' succeeded
ASM created and started successfully.
Disk Group CONFIG created successfully.
clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4256: Updating the profile
Successful addition of voting disk 6dae7976ff034f6abf23a9eeb07866c5.
Successful addition of voting disk 1d807080e86c4fe2bf0acf6d2f3dc3fe.
Successful addition of voting disk 453f08f20d4b4f92bff4d6bcf979b655.
Successful addition of voting disk 4fe8507cb22f4ff6bf2a6934eb56a330.
Successful addition of voting disk 832af65698704fd0bf2e8cf4a865b9b7.
Successfully replaced voting disk group with +CONFIG.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 6dae7976ff034f6abf23a9eeb07866c5 (ORCL:CONF1) [CONFIG]
2. ONLINE 1d807080e86c4fe2bf0acf6d2f3dc3fe (ORCL:CONF2) [CONFIG]
3. ONLINE 453f08f20d4b4f92bff4d6bcf979b655 (ORCL:CONF3) [CONFIG]
4. ONLINE 4fe8507cb22f4ff6bf2a6934eb56a330 (ORCL:CONF4) [CONFIG]
5. ONLINE 832af65698704fd0bf2e8cf4a865b9b7 (ORCL:CONF5) [CONFIG]
Located 5 voting disk(s).
CRS-2672: Attempting to start 'ora.asm' on 'laguna1'
CRS-2676: Start of 'ora.asm' on 'laguna1' succeeded
CRS-2672: Attempting to start 'ora.CONFIG.dg' on 'laguna1'
CRS-2676: Start of 'ora.CONFIG.dg' on 'laguna1' succeeded
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@laguna1:~]#
[root@laguna2:~]# /u01/app/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file:
/u01/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
OLR initialization - successful
Adding Clusterware entries to upstart
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS
daemon on node laguna1, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the
cluster
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@laguna2:~]#
### Oracle Cluster Verification Utility NTP related error:
There are three problems:
1) ntpd daemon is dead.
Restarting ntpd process resolved the problem.
service ntpd status
service ntpd stop
service ntpd start
2) ntp server entry is not valid /etc/ntp.conf
Deleting invalid server names, inserting valid server, resolved.
3) time is beyond drift value:
adjust date according to ntp server.
ntpdate <ntp server ip>
ntpdate 10.200.100.12
After getting NTP related error:
INFO: CTSS is in Observer state. Switching over to clock synchronization checks
using NTP
INFO: Starting Clock synchronization checks using Network Time Protocol(NTP)...
INFO: NTP Configuration file check started...
INFO: NTP Configuration file check passed
INFO: Checking daemon liveness...
INFO: Liveness check passed for "ntpd"
INFO: Check for NTP daemon or service alive passed on all nodes
INFO: NTP daemon slewing option check failed on some nodes
INFO: Check failed on nodes:
INFO: laguna2
INFO: PRVF-5436 : The NTP daemon running on one or more nodes lacks the slewing
option "-x"
[root@laguna2:~/.ssh]# service ntpd status
ntpd dead but pid file exists
[root@laguna2:~/.ssh]# ^C
[root@laguna2:~/.ssh]# service ntpd stop
Shutting down ntpd: [FAILED]
[root@laguna2:~/.ssh]# service ntpd start
Starting ntpd: [ OK ]
[root@laguna2:~/.ssh]#
[root@laguna2:~/.ssh]# service ntpd status
ntpd (pid 47784) is running...
[root@laguna2:~/.ssh]# service ntpd stop
Shutting down ntpd: [ OK ]
[root@laguna2:~/.ssh]# service ntpd start
Starting ntpd: [ OK ]
[root@laguna2:~/.ssh]# service ntpd status
ntpd (pid 48323) is running...
[root@laguna2:~/.ssh]# vi /etc/ntp.conf
### NTP server ntp.conf file was containing illegal server entries.
[root@laguna2:~/.ssh]# service ntpd stop
Shutting down ntpd: [ OK ]
### ntpdate command for updating date accoring to supplied ntp server.
[root@laguna2:~/.ssh]# ntpdate
22 May 09:26:29 ntpdate[61843]: no servers can be used, exiting
[root@laguna2:~/.ssh]# ntpdate 10.200.100.12
22 May 09:26:40 ntpdate[62471]: step time server 10.200.100.12 offset 2.175496 sec
[root@laguna2:~/.ssh]#
### After pressing “Retry” it successfully finished the step: Oracle Cluster
Verification Utility
[It is not over, continueing in other page...]
### It worths seeing the change in ifconfig output before and after GI
installation:
###BEFORE
[root@laguna1:~]# ifconfig
bond0 Link encap:Ethernet HWaddr AC:16:2D:B0:29:74
inet addr:10.200.160.211 Bcast:10.200.160.255 Mask:255.255.255.0
inet6 addr: fe80::ae16:2dff:feb0:2974/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:300879 errors:0 dropped:0 overruns:0 frame:0
TX packets:68500 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:71872763 (68.5 MiB) TX bytes:67371829 (64.2 MiB)
bond1 Link encap:Ethernet HWaddr AC:16:2D:B0:29:76
inet addr:90.90.90.91 Bcast:90.90.90.255 Mask:255.255.255.0
inet6 addr: fe80::ae16:2dff:feb0:2976/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:69690 errors:0 dropped:0 overruns:0 frame:0
TX packets:224 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:4181476 (3.9 MiB) TX bytes:10704 (10.4 KiB)
eth0 Link encap:Ethernet HWaddr AC:16:2D:B0:29:74
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:199869 errors:0 dropped:0 overruns:0 frame:0
TX packets:68500 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:65440529 (62.4 MiB) TX bytes:67371829 (64.2 MiB)
Interrupt:69
eth1 Link encap:Ethernet HWaddr AC:16:2D:B0:29:74
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:101010 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:6432234 (6.1 MiB) TX bytes:0 (0.0 b)
Interrupt:73
eth2 Link encap:Ethernet HWaddr AC:16:2D:B0:29:76
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:34843 errors:0 dropped:0 overruns:0 frame:0
TX packets:224 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2090656 (1.9 MiB) TX bytes:10704 (10.4 KiB)
Interrupt:77
eth3 Link encap:Ethernet HWaddr AC:16:2D:B0:29:76
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:34847 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2090820 (1.9 MiB) TX bytes:0 (0.0 b)
Interrupt:81
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:96832 errors:0 dropped:0 overruns:0 frame:0
TX packets:96832 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:23219019 (22.1 MiB) TX bytes:23219019 (22.1 MiB)
[root@laguna1:~]#
### AFTER
[root@laguna1:~]# ifconfig
bond0 Link encap:Ethernet HWaddr AC:16:2D:B0:29:74
inet addr:10.200.160.211 Bcast:10.200.160.255 Mask:255.255.255.0
inet6 addr: fe80::ae16:2dff:feb0:2974/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:2291765 errors:0 dropped:0 overruns:0 frame:0
TX packets:611590 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:251820629 (240.1 MiB) TX bytes:5014093262 (4.6 GiB)
bond0:1 Link encap:Ethernet HWaddr AC:16:2D:B0:29:74
inet addr:10.200.160.212 Bcast:10.200.160.255 Mask:255.255.255.0
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
bond0:3 Link encap:Ethernet HWaddr AC:16:2D:B0:29:74
inet addr:10.200.160.196 Bcast:10.200.160.255 Mask:255.255.255.0
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
bond0:4 Link encap:Ethernet HWaddr AC:16:2D:B0:29:74
inet addr:10.200.160.197 Bcast:10.200.160.255 Mask:255.255.255.0
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
bond1 Link encap:Ethernet HWaddr AC:16:2D:B0:29:76
inet addr:90.90.90.91 Bcast:90.90.90.255 Mask:255.255.255.0
inet6 addr: fe80::ae16:2dff:feb0:2976/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:1973966 errors:0 dropped:0 overruns:0 frame:0
TX packets:1436794 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1974215073 (1.8 GiB) TX bytes:1467319327 (1.3 GiB)
bond1:1 Link encap:Ethernet HWaddr AC:16:2D:B0:29:76
inet addr:169.254.79.25 Bcast:169.254.255.255 Mask:255.255.0.0
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
eth0 Link encap:Ethernet HWaddr AC:16:2D:B0:29:74
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:2093220 errors:0 dropped:0 overruns:0 frame:0
TX packets:611590 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:239175211 (228.0 MiB) TX bytes:5014093262 (4.6 GiB)
Interrupt:69
eth1 Link encap:Ethernet HWaddr AC:16:2D:B0:29:74
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:198545 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:12645418 (12.0 MiB) TX bytes:0 (0.0 b)
Interrupt:73
eth2 Link encap:Ethernet HWaddr AC:16:2D:B0:29:76
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:1905257 errors:0 dropped:0 overruns:0 frame:0
TX packets:1436794 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1970081759 (1.8 GiB) TX bytes:1467319327 (1.3 GiB)
Interrupt:77
eth3 Link encap:Ethernet HWaddr AC:16:2D:B0:29:76
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:68709 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:4133314 (3.9 MiB) TX bytes:0 (0.0 b)
Interrupt:81
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:433336 errors:0 dropped:0 overruns:0 frame:0
TX packets:433336 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:236033817 (225.0 MiB) TX bytes:236033817 (225.0 MiB)
[root@laguna1:~]#
### 2 scan names and vip is up in first node.
[root@laguna1:~]# nslookup laguna1vip.turkcell.tgc
Server: 10.210.154.150
Address: 10.210.154.150#53
Name: laguna1vip.turkcell.tgc
Address: 10.200.160.212
[root@laguna1:~]# nslookup laguna-scan
Server: 10.210.154.150
Address: 10.210.154.150#53
Name: laguna-scan.turkcell.tgc
Address: 10.200.160.213
Name: laguna-scan.turkcell.tgc
Address: 10.200.160.196
Name: laguna-scan.turkcell.tgc
Address: 10.200.160.197
[root@laguna1:~]#
### The orher san ip and laguna2vip is up in laguna2 node.
[root@laguna2:~]# nslookup laguna2vip.turkcell.tgc
Server: 10.210.154.150
Address: 10.210.154.150#53
Name: laguna2vip.turkcell.tgc
Address: 10.200.160.195
[root@laguna2:~]# ifconfig
bond0 Link encap:Ethernet HWaddr AC:16:2D:BC:8D:50
inet addr:10.200.160.194 Bcast:10.200.160.255 Mask:255.255.255.0
inet6 addr: fe80::ae16:2dff:febc:8d50/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:2831324 errors:0 dropped:0 overruns:0 frame:0
TX packets:1018048 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:3541772648 (3.2 GiB) TX bytes:71026022 (67.7 MiB)
bond0:1 Link encap:Ethernet HWaddr AC:16:2D:BC:8D:50
inet addr:10.200.160.213 Bcast:10.200.160.255 Mask:255.255.255.0
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
bond0:2 Link encap:Ethernet HWaddr AC:16:2D:BC:8D:50
inet addr:10.200.160.195 Bcast:10.200.160.255 Mask:255.255.255.0
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
bond1 Link encap:Ethernet HWaddr AC:16:2D:BC:8D:52
inet addr:90.90.90.92 Bcast:90.90.90.255 Mask:255.255.255.0
inet6 addr: fe80::ae16:2dff:febc:8d52/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:1639471 errors:0 dropped:0 overruns:0 frame:0
TX packets:1803434 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1488826628 (1.3 GiB) TX bytes:1977180379 (1.8 GiB)
bond1:1 Link encap:Ethernet HWaddr AC:16:2D:BC:8D:52
inet addr:169.254.187.201 Bcast:169.254.255.255 Mask:255.255.0.0
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
eth0 Link encap:Ethernet HWaddr AC:16:2D:BC:8D:50
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:199106 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:12680250 (12.0 MiB) TX bytes:0 (0.0 b)
Interrupt:71
eth1 Link encap:Ethernet HWaddr AC:16:2D:BC:8D:50
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:2632219 errors:0 dropped:0 overruns:0 frame:0
TX packets:1018050 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3529092452 (3.2 GiB) TX bytes:71027162 (67.7 MiB)
Interrupt:75
eth2 Link encap:Ethernet HWaddr AC:16:2D:BC:8D:52
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:1570562 errors:0 dropped:0 overruns:0 frame:0
TX packets:1803434 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1484681314 (1.3 GiB) TX bytes:1977180379 (1.8 GiB)
Interrupt:79
eth3 Link encap:Ethernet HWaddr AC:16:2D:BC:8D:52
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:68909 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:4145314 (3.9 MiB) TX bytes:0 (0.0 b)
Interrupt:83
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:333267 errors:0 dropped:0 overruns:0 frame:0
TX packets:333267 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:100537601 (95.8 MiB) TX bytes:100537601 (95.8 MiB)
[root@laguna2:~]#
Post Installation Steps
--5.2.1 Back Up the root.sh Script for both nodes.
[root@laguna1:~]# cp /u01/app/11.2.0/grid/root.sh /u01/orainst/root.sh-
backup_GI_inst
--5.2.3.2 Creating the Fast Recovery Area Disk Group
[grid@laguna1 grid]$ cd /u01/app/11.2.0/grid/bin
[grid@laguna1 bin]$
[grid@laguna1 bin]$
[grid@laguna1 bin]$ ./asmca
A.7 About Using CVU Cluster Healthchecks After Installation [grid@laguna1 bin]$ cd /u01/orainst/cvu/bin
[grid@laguna1 bin]$ ./cluvfy comp healthcheck -collect cluster -bestpractice –
deviations
Verifying OS Best Practice
Verifying Hardware Clock synchronization at shutdown ...not met
Verifying Clusterware Best Practice
***********************************************************************************
*******
Summary of environment
***********************************************************************************
*******
Date (mm/dd/yyyy) : 05/23/2013
Time (hh:mm:ss) : 16:40:45
Cluster name : laguna-cluster
Clusterware version : 11.2.0.3.0
Grid home : /u01/app/11.2.0/grid
Grid User : grid
Operating system : Linux2.6.32-358.el6.x86_64
***********************************************************************************
*******
System recommendations
***********************************************************************************
*******
Verification Check : Hardware Clock synchronization at shutdown
Verification Description : Checks whether Hardware Clock is synchronized with the
system clock during system shutdown
Verification Result : NOT MET
Verification Summary : Check for Hardware Clock synchronization at shutdown
failed
Node Status Expected Value Actual Value
-----------------------------------------------------------------------------------
-------
laguna2 FAILED HWCLOCKSYNC=1 HWCLOCKSYNC=0
laguna1 FAILED HWCLOCKSYNC=1 HWCLOCKSYNC=0
___________________________________________________________________________________
_______
[grid@laguna1 bin]$
[grid@laguna1 bin]$ ./cluvfy comp healthcheck -collect cluster -mandatory -save -
savedir /u01/orainst/cvu/bin
[grid@laguna1 bin]$ ./cluvfy comp healthcheck -collect cluster -mandatory -save -
savedir /u01/orainst/cvu/bin
cvucheckreport_5232013165416.txt