Oracle RAC – adding and removing cluster nodes

CERN - IT DepartmentCH-1211 Genève 23

Switzerlandwww.cern.ch/it

Oracle RAC – adding and removing

cluster nodes

WLCG Service Reliability Workshop

CERN, November 30th, 2007

Jacek Wojcieszuk, CERN IT

LCG



RAC – adding and removing nodes – WLCG Service Reliability Workshop, Nov 2007 - 2

Part I

Node addition




Methods

• Silent cloning procedures• Enterprise Manager Grid Control cloning and

adding instance• addNode.sh and DBCA• Manual

DEMOIn the presented demo as well as in the command examples appearing in the presentation, the following names are being used:demorac – cluster nameitrac18 (demorac1, +ASM1), itrac19 (demorac2, +ASM2) - existing cluster nodes/instancesitrac20 (demorac3, +ASM3) – node/instances that are being added to the cluster




Steps

• Setup hardware for a new node• Install and configure OS• Configure account equivalency• Add Oracle Clusterware to the new node• Configure ONS for the new node• Add ASM and DB home to the new node• Configure a listener on the new node• Add an ASM and a database instances




Hardware setup

• New node of the cluster should be:– of similar type as other cluster nodes (the same

architecture e.g. x86_64)– of similar size to other cluster nodes– physically connected to the shared storage used

by the cluster– physically connected to the cluster private

network(s)




OS installation and configuration

• Install OS on the new node– The same flavor and version as installed on other nodes (e.g.

RHEL 4.0)– The same kernel version recommended

• Make sure that all required packages are installed– Ideally the list of software packages should be identical on all

cluster nodes• Configure kernel parameters and shell limits• Create Oracle-related OS groups and users• Configure network interfaces

– On one of the old cluster nodes edit /etc/hosts file adding entries related to the new node; distribute this file to all cluster nodes

• Configure hangcheck timer module

# as root on the new node:echo "/sbin/modprobe hangcheck-timer" >> /etc/rc.d/rc.local echo "session required /lib/security/pam_limits.so" >> /etc/pam.d/login /sbin/modprobe hangcheck-timer




Account equivalency

• Passwordless access from one Oracle software owner account to another has to be configured– Needed only during software installation– The easiest is to re-configure the equivalency for the

whole cluster– Use sshUserSetup.sh script available on metalink

# as oracle user from one of the cluster nodes./sshUserSetup.sh -hosts „itrac18 itrac18.cern.ch itrac19 itrac19.cern.ch itrac20 itrac20.cern.ch" -advanced

• One can use clufvy tool to verify if the new machine is ready for clusterware installation

# as oracle user from one of the existing cluster nodes$ORA_CRS_HOME/bin/cluvfy stage -pre crsinst -n itrac18,itrac19,itrac20 –r 10gR2 –osdba ci –orainv ci




Adding Oracle Clusterware and configuring ONS• Run addNode.sh script from an existing cluster node

cd $ORA_CRS_HOME/oui/bin; ./addNode.sh

• After Oracle Universal Installer is started, follow the wizard:– Provide information related to the node being added– Launch the installation– Run the scripts as root on new and old nodes as requested by

the pop-up window

• Close OUI and verify if nodeapps associated with the new node have been properly added to the Cluster registry– Use crs_stat command

• Configure the new ONS:

# From one of the old cluster nodes run:cat $ORA_CRS_HOME/opmn/conf/ons.config |grep remoteportremoteport=6200cd $ORA_CRS_HOME/bin./racgons add_config itrac20:6200




Adding ASM and DB home (addNode script)• Depending on the chosen configuration can be

done in either 1 or 2 steps• Run addNode script on an existing cluster node

cd $ORACLE_HOME/oui/bin; ./addNode.sh

• After Oracle Universal Installer is started, follow the wizard– Tick node that you want to add (should be ticked by

default)– Install the software– Run root.sh script as requested by the pop-up window




Adding ASM and DB home (cloning)• Go to an existing cluster node and tar ORACLE_HOME

directory# on the first node of the clustercd $ORACLE_HOME/..tar cvfp rdbms.tar rdbms

• Copy created tar file to the new node and unpack it

# on the new nodecd $ORACLE_HOME/..scp oracle@itrac18:$PWD/rdbms.tar .tar xvfp rdbms.tar

• Remove from the $ORACLE_HOME/dbs directory instance specific files (pfiles and password files)

• Run cloning procedure

# on the new nodecd $ORACLE_HOME/clone/binperl clone.pl ORACLE_HOME="/ORA/dbs01/oracle/product/10.2.0/rdbms" ORACLE_HOME_NAME="OraDb10g_rdbms" '-O"CLUSTER_NODES={itrac18,itrac19,itrac20}"' '-O"LOCAL_NODE=itrac20"'




Listener configuration

• From the new node run netca to configure a listener:– Choose ‘Cluster configuration’– Select the new node only– Choose ‘Listener configuration’– Choose ‘Add’– Leave the default listener name– Leave intact the default protocol choice– Specify proper port– Complete the configuration

• Check with the crs_stat command if the new listener has been registered to clusterware

• Edit listener.ora and tnsnames.ora files on the new node:– You may want to remove extproc related entries– In tnsnames.ora add/edit entries related to the new listener

• Distribute updated tnsnames.ora file to all nodes of the cluster




Adding ASM and database instances (with DBCA)

• Although this procedure is recommended by Oracle we have seen that it is quite unreliable and therefore we strongly recommend using manual procedure described later

• Run DBCA from an old cluster node:– Choose ‘Oracle Real Application Clusters database’– Choose ‘Instance Management’– Choose ‘Add an instance’– Specify SYS username and password– Select name of the new node– Modify custom service definitions– Agree to extend ASM to the new node– Complete the operation




Adding ASM and database instances (manually)• Edit /etc/oratab file on the new node

– add entries describing new ASM and DB instances• Create dump directories for ASM and DB instances as defined

by initialization parameters• Create ASM pfile in the default location

($ORACLE_HOME/dbs)• Create a password file for ASM

orapwd file=$ORACLE_HOME/dbs/orapw+ASM3 password=xxxx

• Define ASM instance in the cluster registry and start it up

srvctl add asm -n itrac20 -i +ASM3 -o $ORACLE_HOMEsrvctl start asm -n itrac20

• Connect to an existing ASM instance and define values for init parameters specific for the new instance

export ORACLE_SID=+ASM1; sqlplus / as sysdbaASM> alter system set instance_number=3 scope=spfile sid=‘+ASM3’;ASM> alter system set local_listener=‘LISTENER_+ASM3’ scope=spfile sid=‘+ASM3’;




Adding ASM and database instances (manually) (2)• Connect to an existing database instance

– Create an undo tablespace and a redo log thread for the new RAC node

– Set properly init parameters specific to the new instance

export ORACLE_SID=demorac1; sqlplus / as sysdbaSQL> create undo tablespace undotbs3 datafile size 1G autoextend on next 1G maxsize 30G;SQL> alter database add logfile thread 3 group 31 size 512m;SQL> alter database add logfile thread 3 group 32 size 512m;SQL> alter database enable public thread 3;SQL> alter system set instance_number=3 scope=spfile sid=‘demorac3’;SQL> alter system set thread=3 scope=spfile sid=‘demorac3’;SQL> alter system set undo_tablespace=‘UNDOTBS3’ scope=spfile sid=‘demorac3’SQL> alter system set local_listener=‘LISTENER_DEMORAC3’ scope=spfile sid=‘demorac3’;

• Create DB pfile in the default location ($ORACLE_HOME/dbs)• Create a password file for the DB instance

orapwd file=$ORACLE_HOME/dbs/orapwdemorac3 password=xxxx




Adding ASM and database instances (manually) (3)• Define clusterware targets related to the new

database instancesrvctl add instance -d demorac -i demorac3 –n itrac20srvctl modify instance -d demorac –i demorac3 -s +ASM3

• Start the new instance using srvctl toolsrvctl start instance -d demorac -i demorac3

• Verify with crs_stat tool if all the clusterware targets are up




Final steps

• Install OEM agent• Redefine services if needed

srvctl stop service -d demorac -s demorac_lb srvctl modify service -d demorac -s demorac_lb –i demorac1,demorac2,demorac3 -nsrvctl start service -d demorac -s demorac_lb

• Update your public tnsnames.ora or LDAP• During next maintenance you may want to increase

value of the cluster_database_instances initialization parameter




Part II

Node removal




Steps

• Stop and delete the DB instance on the node to be deleted

• Clean up the ASM instance• Remove the listener from the node to be

deleted• Remove RAC and ASM software from

oraInventory• Remove ONS configuration from the node• Remove the node from the clusterware




Deleting DB instance on the node to be deleted from the cluster

• Redefine services in a way that they do not use the node being deleted (using srvctl tool)

• From the node that is not being deleted run DBCA:– Choose ‘Oracle Real Application Clusters database’– Choose ‘Instance Management’– Choose ‘Delete an instance’– Provide SYS credentials– Select the instance to delete– Proceed with the deletion

• Verify with the crs_stat command that the instance has been removed from the cluster registry




Cleaning up the ASM instance

• Stop the ASM instance# on any cluster nodesrvctl stop asm -n itrac20

• Remove the ASM instance# on any cluster nodesrvctl remove asm -n itrac20

• Check with crs_stat command if the ASM instance has been removed from OCR

• On the node being deleted edit /etc/oratab file– remove entries related to removed DB and ASM instances




Removing the listener

• Run netca on the node being deleted:– Choose ‘Cluster configuration’– Select the node being deleted– Choose ‘Listener configuration’– Choose ‘Delete’– Complete the deletion

• Check with crs_stat tool if the listener has disappeared from OCR




Removing database software

• Update oraInventory on the node being removed# on the node being deleted:$ORACLE_HOME/oui/bin/runInstaller –updateNodeList ORACLE_HOME=$ORACLE_HOME CLUSTER_NODES=itrac20 -local

• Uninstall the DB software using OUI# on the node being deleted:$ORACLE_HOME/oui/bin/runInstaller

• Update oraInventory on other cluster nodes# on the first node of the cluster $ORACLE_HOME/oui/bin/runInstaller –updateNodeList ORACLE_HOME=$ORACLE_HOME CLUSTER_NODES=itrac18,itrac19

• If you use separate software installation for ASM repeat the steps above for this ASM_HOME




Removing ONS configuration and cluster software

• Remove ONS configuration# From the first node of the cluster run:$ORA_CRS_HOME/bin/racgons remove_config itrac20:6200

• Delete clusterware software from the node# On the node to be removed as root run:$ORA_CRS_HOME/install/rootdelete.sh

• Run rootdeletenode.sh script as root from the first node of the cluster to complete clusterware removal

# On first node of the cluster as root:$ORA_CRS_HOME/install/rootdeletenode.sh itrac20




Removing cluster software (2)

• Update oraInventory on the node being removed# on the node being deleted:$ORA_CRS_HOME/oui/bin/runInstaller –updateNodeList ORACLE_HOME=$ORA_CRS_HOME CLUSTER_NODES=itrac20 CRS=true -local

• Uninstall the Clusterware software using OUI# on the node being deleted:$ORA_CRS_HOME/oui/bin/runInstaller

• Update oraInventory on other cluster nodes# on the first node of the cluster:$ORA_CRS_HOME/oui/bin/runInstaller -updateNodeList ORACLE_HOME=$ORA_CRS_HOME CLUSTER_NODES=itrac18,itrac19 CRS=true

• Verify with the crs_stat command that the node has been properly removed from the cluster




Final steps (optional)

• Remove binaries left on the removed node• Deconfigure OS oracle account equivalency• Stop and remove OEM agent from the node• Deconfigure private network interfaces

Oracle RAC – adding and removing cluster nodes

Documents

Transcript of Oracle RAC – adding and removing cluster nodes