SC338student Guide MAIN

Sun Microsystems, Inc.UBRM05-104

500 Eldorado Blvd.Broomfield, CO 80021

U.S.A.

Revision A.2.1

StudentGuide

Sun™ Cluster 3.1 AdministrationES-338

July 28, 2004 9:55 am

Please

Recycle

Copyright 2004 Sun Microsystems, Inc. 4150 Network Circle, Santa Clara, California 95054, U.S.A. All rights reserved.

This product or document is protected by copyright and distributed under licenses restricting its use, copying, distribution, anddecompilation. No part of this product or document may be reproduced in any form by any means without prior written authorization ofSun and its licensors, if any.

Third-party software, including font technology, is copyrighted and licensed from Sun suppliers.

Sun, Sun Microsystems, the Sun logo, Sun Fire, Solaris, Sun Enterprise, Solstice DiskSuite, JumpStart, Sun StorEdge, SunPlex, SunSolve,Java, Sun BluePrints, and OpenBoot are trademarks or registered trademarks of Sun Microsystems, Inc. in the U.S. and other countries.

All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. in the U.S. andother countries. Products bearing SPARC trademarks are based upon an architecture developed by Sun Microsystems, Inc.

UNIX is a registered trademark in the U.S. and other countries, exclusively licensed through X/Open Company, Ltd.

ORACLE is a registered trademark of Oracle Corporation.

Xylogics product is protected by copyright and licensed to Sun by Xylogics. Xylogics and Annex are trademarks of Xylogics, Inc.

Federal Acquisitions: Commercial Software – Government Users Subject to Standard License Terms and Conditions

Export Laws. Products, Services, and technical data delivered by Sun may be subject to U.S. export controls or the trade laws of othercountries. You will comply with all such laws and obtain all licenses to export, re-export, or import as may be required after delivery toYou. You will not export or re-export to entities on the most current U.S. export exclusions lists or to any country subject to U.S. embargoor terrorist controls as specified in the U.S. export laws. You will not use or provide Products, Services, or technical data for nuclear, missile,or chemical biological weaponry end uses.

DOCUMENTATION IS PROVIDED “AS IS” AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS, ANDWARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSEOR NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BELEGALLY INVALID.

THIS MANUAL IS DESIGNED TO SUPPORT AN INSTRUCTOR-LED TRAINING (ILT) COURSE AND IS INTENDED TO BEUSED FOR REFERENCE PURPOSES IN CONJUNCTION WITH THE ILT COURSE. THE MANUAL IS NOT A STANDALONETRAINING TOOL. USE OF THE MANUAL FOR SELF-STUDY WITHOUT CLASS ATTENDANCE IS NOT RECOMMENDED.

Export Commodity Classification Number (ECCN) assigned: 24 February 2003

Please

Recycle

Copyright 2004 Sun Microsystems Inc. 4150 Network Circle, Santa Clara, California 95054, Etats-Unis. Tous droits réservés.

Ce produit ou document est protégé par un copyright et distribué avec des licences qui en restreignent l’utilisation, la copie, la distribution,et la décompilation. Aucune partie de ce produit ou document ne peut être reproduite sous aucune forme, par quelque moyen que ce soit,sans l’autorisation préalable et écrite de Sun et de ses bailleurs de licence, s’il y en a.

Le logiciel détenu par des tiers, et qui comprend la technologie relative aux polices de caractères, est protégé par un copyright et licenciépar des fournisseurs de Sun.

Sun, Sun Microsystems, le logo Sun, Sun Fire, Solaris, Sun Enterprise, Solstice DiskSuite, JumpStart, Sun StorEdge, SunPlex, SunSolve, Java,Sun BluePrints, et OpenBoot sont des marques de fabrique ou des marques déposées de Sun Microsystems, Inc. aux Etats-Unis et dansd’autres pays.

Toutes les marques SPARC sont utilisées sous licence sont des marques de fabrique ou des marques déposées de SPARC International, Inc.aux Etats-Unis et dans d’autres pays. Les produits portant les marques SPARC sont basés sur une architecture développée par SunMicrosystems, Inc.

UNIX est une marques déposée aux Etats-Unis et dans d’autres pays et licenciée exclusivement par X/Open Company, Ltd.

ORACLE est une marque déposée registre de Oracle Corporation.

Législation en matière dexportations. Les Produits, Services et données techniques livrés par Sun peuvent être soumis aux contrôlesaméricains sur les exportations, ou à la législation commerciale dautres pays. Nous nous conformerons à lensemble de ces textes et nousobtiendrons toutes licences dexportation, de ré-exportation ou dimportation susceptibles dêtre requises après livraison à Vous. Vousnexporterez, ni ne ré-exporterez en aucun cas à des entités figurant sur les listes américaines dinterdiction dexportation les plus courantes,ni vers un quelconque pays soumis à embargo par les Etats-Unis, ou à des contrôles anti-terroristes, comme prévu par la législationaméricaine en matière dexportations. Vous nutiliserez, ni ne fournirez les Produits, Services ou données techniques pour aucune utilisationfinale liée aux armes nucléaires, chimiques ou biologiques ou aux missiles.

LA DOCUMENTATION EST FOURNIE “EN L’ETAT” ET TOUTES AUTRES CONDITIONS, DECLARATIONS ET GARANTIESEXPRESSES OU TACITES SONT FORMELLEMENT EXCLUES, DANS LA MESURE AUTORISEE PAR LA LOI APPLICABLE, YCOMPRIS NOTAMMENT TOUTE GARANTIE IMPLICITE RELATIVE A LA QUALITE MARCHANDE, A L’APTITUDE A UNEUTILISATION PARTICULIERE OU A L’ABSENCE DE CONTREFAÇON.

CE MANUEL DE RÉFÉRENCE DOIT ÊTRE UTILISÉ DANS LE CADRE D’UN COURS DE FORMATION DIRIGÉ PAR UNINSTRUCTEUR (ILT). IL NE S’AGIT PAS D’UN OUTIL DE FORMATION INDÉPENDANT. NOUS VOUS DÉCONSEILLONS DEL’UTILISER DANS LE CADRE D’UNE AUTO-FORMATION.

vCopyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1

Table of Contents

About This Course .............................................................Preface-xxiCourse Goals........................................................................ Preface-xxiCourse Map.........................................................................Preface-xxiiiTopics Not Covered...........................................................Preface-xxivHow Prepared Are You?....................................................Preface-xxvIntroductions ......................................................................Preface-xxviHow to Use Course Materials .........................................Preface-xxviiConventions ..................................................................... Preface-xxviii

Icons ......................................................................... Preface-xxviiiTypographical Conventions ................................... Preface-xxix

Introducing Sun™ Cluster Hardware and Software ......................1-1Objectives ........................................................................................... 1-1Relevance............................................................................................. 1-2Additional Resources ........................................................................ 1-3Defining Clustering ........................................................................... 1-4

High-Availability Platforms .................................................... 1-4Scalability Platforms ................................................................. 1-6High Availability and Scalability Are Not Mutually

Exclusive.................................................................................. 1-6Describing the Sun Cluster 3.1 Hardware and Software

Environment .................................................................................... 1-7Introducing the Sun Cluster 3.1 Hardware Environment............ 1-9

Cluster Host Systems.............................................................. 1-10Cluster Transport Interface.................................................... 1-10Public Network Interfaces ..................................................... 1-11Cluster Disk Storage .............................................................. 1-12Boot Disks................................................................................. 1-12Terminal Concentrator ........................................................... 1-12Administrative Workstation.................................................. 1-13Clusters With More Than Two Nodes ................................. 1-13Sun Cluster Hardware Redundancy Features .................... 1-14Cluster in a Box ....................................................................... 1-15

vi Sun™ Cluster 3.1 AdministrationCopyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1

Viewing Sun Cluster 3.1 Software Support.................................. 1-17Describing the Types of Applications in the Sun Cluster

Software Environment ................................................................. 1-19Cluster-Unaware (Off-the-Shelf) Applications................... 1-19Cluster-Aware Applications................................................. 1-22

Identifying Sun Cluster 3.1 Software Data Service Support...... 1-24HA and Scalable Data Service Support ............................... 1-24

Exploring the Sun Cluster Software HA Framework................. 1-26Node Fault Monitoring and Cluster Membership ............. 1-26Network Fault Monitoring .................................................... 1-26Disk Path Monitoring ............................................................. 1-27Application Traffic Striping................................................... 1-27Cluster Reconfiguration Notification Protocol ................... 1-27Cluster Configuration Repository ........................................ 1-28

Defining Global Storage Service Differences ............................... 1-29Disk ID Devices (DIDs) .......................................................... 1-29Global Devices......................................................................... 1-30Device Files for Global Devices............................................. 1-32Global File Systems................................................................. 1-33Local Failover File Systems in the Cluster........................... 1-34

Exercise: Guided Tour of the Training Lab.................................. 1-35Preparation............................................................................... 1-35Task ........................................................................................... 1-35

Exercise Summary............................................................................ 1-36

Exploring Node Console Connectivity and the ClusterConsole Software............................................................................. 2-1

Objectives ........................................................................................... 2-1Relevance............................................................................................. 2-2Additional Resources ........................................................................ 2-3Accessing the Cluster Node Consoles ............................................ 2-4

Accessing Serial Port Consoles on Traditional Nodes......... 2-4Access Serial Port Node Consoles Using a Terminal

Concentrator ........................................................................... 2-5Alternatives to a Terminal Concentrator............................... 2-8

Accessing the Node Consoles on Domain-based Servers ............ 2-9Sun Enterprise™ 10000 Servers .............................................. 2-9Sun Fire™ 15K/12K Servers.................................................... 2-9Sun Fire 3800–6800 Servers...................................................... 2-9

Describing Sun Cluster Console Software forAdministration Workstation ....................................................... 2-10

Console Software Installation ............................................... 2-10Cluster Console Window Variations.................................... 2-10Cluster Console Tools Look and Feel................................... 2-12Start the Tools Manually....................................................... 2-13Cluster Control Panel ............................................................. 2-14

viiCopyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1

Configuring Cluster Console Tools............................................... 2-15Configuring the /etc/clusters File .................................. 2-15Configuring the /etc/serialports File........................... 2-16Configuring Multiple Terminal Concentrators ................. 2-18

Exercise: Configuring the Administrative Console .................... 2-19Preparation............................................................................... 2-19Task 1 – Updating Host Name Resolution......................... 2-20Task 2 – Installing the Cluster Console Software............... 2-20Task 3 – Verifying the Administrative Console

Environment ........................................................................ 2-21Task 4 – Configuring the /etc/clusters File................... 2-21Task 5 – Configuring the /etc/serialports File ........... 2-22Task 6 – Starting the cconsole Tool .................................... 2-22Task 7 – Using the ccp Control Panel .................................. 2-23


Preparing for Installation and Understanding QuorumDevices ..............................................................................................3-1

Objectives ........................................................................................... 3-1Relevance............................................................................................. 3-2Additional Resources ........................................................................ 3-3Configuring Cluster Servers............................................................. 3-4

Boot Device Restrictions .......................................................... 3-4Server Hardware Restrictions ................................................. 3-5

Configuring Cluster Storage Connections ..................................... 3-7Cluster Topologies .................................................................... 3-7Clustered Pairs Topology ........................................................ 3-8Pair+N Topology...................................................................... 3-9N+1 Topology......................................................................... 3-10Scalable Storage Topology.................................................... 3-11Non-Storage Topology .......................................................... 3-12Single-Node Cluster Topology ............................................. 3-12Clustered Pair Topology for x86........................................... 3-12

Describing Quorum Votes and Quorum Devices ....................... 3-14Why Have Quorum Voting at All?....................................... 3-14Failure Fencing ........................................................................ 3-15Amnesia Prevention ............................................................... 3-15Quorum Device Rules ........................................................... 3-16Quorum Mathematics and Consequences .......................... 3-16Two-Node Cluster Quorum Devices ................................... 3-17Clustered-Pair Quorum Disks............................................... 3-17Pair+N Quorum Disks ........................................................... 3-18N+1 Quorum Disks................................................................. 3-19Quorum Devices in the Scalable Storage Topology........... 3-20

viii Sun™ Cluster 3.1 AdministrationCopyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1

Preventing Cluster Amnesia With PersistentReservations................................................................................... 3-21

Persistent Reservations and Reservation Keys.................. 3-22SCSI-2 Persistent Reservation Emulation............................ 3-24SCSI-3 Persistent Group Reservation................................... 3-24SCSI-3 PGR Scenario............................................................... 3-25Intentional Reservation Delays for Partitions With

Fewer Than Half of the Nodes ........................................... 3-27Configuring a Cluster Interconnect............................................... 3-28

Point-to-Point Cluster Interconnect...................................... 3-28Junction-based Cluster Interconnect .................................... 3-29Cluster Transport Interface Addresses ................................ 3-29Identify Cluster Transport Interfaces................................... 3-30

Configuring a Public Network....................................................... 3-32Exercise: Preparing for Installation ............................................... 3-33

Preparation............................................................................... 3-33Task 1 – Verifying the Solaris™ Operating System ........... 3-33Task 2 – Identifying a Cluster Topology ............................ 3-34Task 3 – Selecting Quorum Devices ..................................... 3-34Task 4 – Verifying the Cluster Interconnect

Configuration ...................................................................... 3-35Task 5 – Selecting Public Network Interfaces ..................... 3-37


Installing the Sun Cluster Host Software ...................................... 4-1Objectives ........................................................................................... 4-1Relevance............................................................................................. 4-2Additional Resources ........................................................................ 4-3Summarizing Sun Cluster Software ................................................ 4-4

Sun Cluster Framework Software ......................................... 4-6Sun Cluster 3.1 Software Licensing....................................... 4-7Sun Cluster Software Basic Installation Process................... 4-7Sun Cluster Software Alternative Installation

Processes................................................................................. 4-8Configuring the Sun Cluster Software Node

Environment .................................................................................. 4-11Configuring the User root Environment............................ 4-11Configuring Network Name Resolution ............................. 4-11

Installing Sun Cluster Software Node Patches ............................ 4-12Patch Installation Warnings .................................................. 4-12Obtaining Patch Information................................................. 4-12

Installing the Sun Cluster 3.1 Software......................................... 4-13Understanding the installmode Flag ................................ 4-13Installing on All Nodes .......................................................... 4-13Installing One Node at a Time .............................................. 4-19Installing Additional Nodes.................................................. 4-31Installing a Single-Node Cluster........................................... 4-33

ixCopyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1

Performing Post-Installation Configuration ................................ 4-34Verifying DID Devices ........................................................... 4-34Resetting the installmode Attribute and Choosing

Quorum ................................................................................. 4-35Configuring NTP..................................................................... 4-37

Performing Post-Installation Verification..................................... 4-38Verifying General Cluster Status .......................................... 4-38Verifying Cluster Configuration Information ................... 4-39

Correcting Minor Configuration Errors ....................................... 4-41Exercise: Installing the Sun Cluster Server Software.................. 4-42

Preparation............................................................................... 4-42Task 1 – Verifying the Boot Disk .......................................... 4-42Task 2 – Verifying the Environment .................................... 4-43Task 3 – Updating the Name Service .................................. 4-44Task 4a – Establishing a New Cluster – All Nodes............ 4-44Task 4b – Establishing a New Cluster – First Node........... 4-45Task 5 – Adding Additional Nodes..................................... 4-47Task 6 – Configuring a Quorum Device ............................. 4-48Task 7 – Configuring the Network Time Protocol ............ 4-49Task 8 – Testing Basic Cluster Operation ............................ 4-50


Performing Basic Cluster Administration ......................................5-1Objectives ........................................................................................... 5-1Relevance............................................................................................. 5-2Additional Resources ........................................................................ 5-3Identifying Cluster Daemons ........................................................... 5-4Using Cluster Status Commands..................................................... 5-6

Checking Status Using the scstat Command..................... 5-6Viewing the Cluster Configuration ........................................ 5-6Disk Path Monitoring ............................................................. 5-10Checking the Status Using the scinstall Utility ............. 5-11Examining the /etc/cluster/release File ..................... 5-12

Controlling Clusters ........................................................................ 5-13Starting and Stopping Cluster Nodes .................................. 5-13Booting Nodes in Non-Cluster Mode ................................. 5-14Placing Nodes in Maintenance Mode ................................. 5-15Maintenance Mode Example................................................. 5-15

Introducing Cluster Administration Utilities .............................. 5-17Introducing the Low-level Administration

Commands............................................................................ 5-17Using the scsetup Command .............................................. 5-17Comparing Low-level Command and scsetup Usage..... 5-18Introducing the SunPlex™ Manager.................................... 5-19Logging In to SunPlex Manager ........................................... 5-20

x Sun™ Cluster 3.1 AdministrationCopyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1

Navigating the SunPlex Manager Main Window............. 5-22Viewing a SunPlex Manager Graphical Topology

Example ................................................................................. 5-23Exercise: Performing Basic Cluster Administration ................... 5-24

Preparation............................................................................... 5-24Task 1 – Verifying Basic Cluster Status................................ 5-24Task 2 – Starting and Stopping Cluster Nodes................... 5-25Task 3 – Placing a Node in Maintenance State ................... 5-25Task 4 – Booting Nodes in Non-Cluster Mode................... 5-26Task 5 – Navigating SunPlex Manager................................ 5-27


Using VERITAS Volume Manager for Volume Management........ 6-1Objectives ........................................................................................... 6-1Relevance............................................................................................. 6-2Additional Resources ........................................................................ 6-3Introducing VxVM in the Sun Cluster Software

Environment .................................................................................... 6-4Exploring VxVM Disk Groups and the rootdg Disk Group ...... 6-5

Shared Storage Disk Groups ................................................... 6-5The rootdg Disk Group on Each Node................................. 6-5Shared Storage Disk Group Ownership ............................... 6-6Sun Cluster Management of Disk Groups ............................ 6-6Sun Cluster Global Devices in Disk Group........................... 6-6VxVM Cluster Feature Used Only for ORACLE®

Parallel Server and ORACLE Real ApplicationCluster...................................................................................... 6-7

Initializing the VERITAS Volume Manager Disk ......................... 6-8Reviewing the Basic Objects in a Disk Group................................ 6-9

Disk Names or Media Names ................................................. 6-9Subdisk ....................................................................................... 6-9Plex .............................................................................................. 6-9Volume ..................................................................................... 6-10Layered Volume...................................................................... 6-10

Exploring Volume Requirements in the Sun ClusterSoftware Environment ................................................................. 6-11

Simple Mirrors......................................................................... 6-11Mirrored Stripe (“Mirror-Stripe”) ........................................ 6-12Striped Mirrors (“Stripe-Mirror”)......................................... 6-13Dirty Region Logs for Volumes in the Cluster ................... 6-14

Viewing the Installation and rootdg Requirements inthe Sun Cluster Software Environment ..................................... 6-15

Describing DMP Restrictions ......................................................... 6-17DMP Restrictions in Sun Cluster 3.1 (VxVM 3.2

and 3.5) ................................................................................. 6-18

xiCopyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1

Installing VxVM in the Sun Cluster 3.1 SoftwareEnvironment .................................................................................. 6-20

Describing the scvxinstall Utility .................................... 6-20Examining an scvxinstall Example ................................ 6-21Installing and Initializing VxVM Manually ........................ 6-22Configuring a Pre-existing VxVM for Sun Cluster 3.1

Software................................................................................. 6-23Creating Shared Disk Groups and Volumes................................ 6-25

Listing Available Disks .......................................................... 6-25Initializing Disks and Putting Them in a New Disk

Group..................................................................................... 6-25Verifying Disk Groups Imported on a Node ...................... 6-26Building a Simple Mirrored Volume................................... 6-27Building a Mirrored Striped Volume (RAID 0+1).............. 6-27Building a Striped Mirrored Volume (RAID 1+0).............. 6-28

Examining Hot Relocation.............................................................. 6-29Registering VxVM Disk Groups .................................................... 6-30

Using the scconf Command to Register Disk Groups..... 6-31Using the scsetup Command to Register Disk

Groups ................................................................................... 6-31Verifying and Controlling Registered Device Groups ...... 6-32

Managing VxVM Device Groups .................................................. 6-33Resynchronizing Device Groups .......................................... 6-33Making Other Changes to Device Groups .......................... 6-33Using the Maintenance Mode ............................................... 6-33

Using Global and Local File Systems on VxVM Volumes ......... 6-34Creating File Systems ............................................................. 6-34Mounting File Systems........................................................... 6-34

Mirroring the Boot Disk With VxVM............................................ 6-35Performing Disk Replacement With VxVM in the Cluster........ 6-36

Repairing Physical Device Files on Multiple Nodes.......... 6-36Repairing the DID Device Database..................................... 6-36Analyzing a Full Scenario of a Sun StorEdge™

A5200 Array Disk Failure ................................................... 6-37Exercise: Configuring Volume Management............................... 6-39

Preparation............................................................................... 6-39Task 1 – Selecting Disk Drives .............................................. 6-40Task 2 – Using the scvxinstall Utility to Install

and Initialize VxVM Software............................................ 6-41Task 3 – Configuring Demonstration Volumes.................. 6-42Task 4 – Registering Demonstration Disk Groups............. 6-44Task 5 – Creating a Global nfs File System ........................ 6-45Task 6 – Creating a Global web File System ........................ 6-46Task 7 – Testing Global File Systems ................................... 6-46

xii Sun™ Cluster 3.1 AdministrationCopyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1

Task 8 – Managing Disk Device Groups ............................. 6-47Task 9 – Viewing and Managing VxVM Device

Groups Through SunPlex Manager .................................. 6-49Exercise Summary............................................................................ 6-50

Managing Volumes With Solaris™ Volume Manager Software(Solstice DiskSuite™ Software)...................................................... 7-1

Objectives ........................................................................................... 7-1Relevance............................................................................................. 7-3Additional Resources ........................................................................ 7-4Describing Solaris Volume Manager Software and Solstice

DiskSuite Software.......................................................................... 7-5Viewing Solaris Volume Manager Software in the Sun

Cluster Software Environment ..................................................... 7-6Exploring Solaris Volume Manager Software Disk Space

Management .................................................................................... 7-7Solaris Volume Manager Software Partition-Based Disk

Space Management ................................................................ 7-7Solaris Volume Manager Software Disk Space

Management With Soft Partitions ....................................... 7-8Exploring Solaris Volume Manager Software Shared

Disksets and Local Disksets......................................................... 7-11Using Solaris Volume Manager Software Volume

Database Replicas (metadb) ........................................................ 7-13Local Replicas Management.................................................. 7-13Local Replica Mathematics ................................................... 7-14Shared Diskset Replicas Management................................. 7-14Shared Diskset Replica Quorum Mathematics................... 7-14Shared Diskset Mediators ..................................................... 7-15

Installing Solaris Volume Manager Software andTuning the md.conf File .............................................................. 7-16

Modifying the md.conf File .................................................. 7-16Initializing the Local Metadbs on Boot Disk and Mirror ........... 7-17

Using DIDs Compared to Using Traditional c#t #d# ........ 7-17Adding the Local Metadb Replicas to the Boot Disk......... 7-17Repartitioning a Mirror Boot Disk and Adding

Metadb Replicas ................................................................... 7-18Using the metadb or metadb -i Command to Verify

Metadb Replicas ................................................................... 7-18Creating Shared Disksets and Mediators ..................................... 7-19

Shared Diskset Disk Automatic Repartitioning andMetadb Placement ............................................................... 7-20

Using Shared Diskset Disk Space .................................................. 7-21Building Volumes in Shared Disksets With Soft

Partitions of Mirrors ..................................................................... 7-22

xiiiCopyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1

Using Solaris Volume Manager Software StatusCommands..................................................................................... 7-23

Checking Volume Status........................................................ 7-23Using Hot Spare Pools..................................................................... 7-25Managing Solaris Volume Manager Software Disksets

and Sun Cluster Device Groups.................................................. 7-26Managing Solaris Volume Manager Software Device

Groups ............................................................................................ 7-28Device Groups Resynchronization....................................... 7-28Other Changes to Device Groups ......................................... 7-28Maintenance Mode ................................................................. 7-28

Using Global and Local File Systems on Shared DisksetDevices............................................................................................ 7-29

Creating File Systems ............................................................. 7-29Mounting File Systems........................................................... 7-29

Using Solaris Volume Manager Software to Mirror theBoot Disk ........................................................................................ 7-30

Verifying Partitioning and Local Metadbs.......................... 7-30Building Volumes for Each Partition Except for Root ....... 7-31Building Volumes for Root Partition ................................... 7-31Running the Commands........................................................ 7-32Rebooting and Attaching the Second Submirror ............... 7-32

Performing Disk Replacement With Solaris VolumeManager Software in the Cluster ................................................ 7-34

Repairing Physical Device Files on Multiple Nodes.......... 7-34Repairing the DID Device Database..................................... 7-34Analyzing a Full Scenario of a Sun StorEdge

A5200 Array Disk Failure ................................................... 7-35Exercise: Configuring Solaris Volume Manager Software......... 7-36

Preparation............................................................................... 7-36Task 1 – Initializing the Solaris Volume Manager

Software Local Metadbs...................................................... 7-37Task 2 – Selecting the Solaris Volume Manager

Software Demo Volume Disk Drives ................................ 7-38Task 3 – Configuring the Solaris Volume Manager

Software Demonstration Disksets ..................................... 7-39Task 4 – Configuring Solaris Volume Manager

Software Demonstration Volumes .................................... 7-40Task 5 – Creating a Global nfs File System ........................ 7-41Task 6 – Creating a Global web File System ........................ 7-42Task 7 – Testing Global File Systems ................................... 7-42Task 8 – Managing Disk Device Groups ............................. 7-43Task 9 – Viewing and Managing Solaris Volume

Manager Software Device Groups .................................... 7-44Task 10 – (Optional) Mirroring the Boot Disk With

Solaris Volume Manager Software.................................... 7-44Exercise Summary............................................................................ 7-47

xiv Sun™ Cluster 3.1 AdministrationCopyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1

Managing the Public Network With IPMP ...................................... 8-1Objectives ........................................................................................... 8-1Relevance............................................................................................. 8-2Additional Resources ........................................................................ 8-3Introducing IPMP .............................................................................. 8-4Describing General IPMP Concepts ................................................ 8-5

Defining IPMP Group Requirements..................................... 8-5 Configuring Standby Interfaces in a Group........................ 8-6

Examining IPMP Group Examples ................................................. 8-7Single IPMP Group With Two Members and No

Standby.................................................................................... 8-7Single IPMP Group With Three Members Including

a Standby................................................................................. 8-8Two IPMP Groups on Different Subnets............................... 8-8Two IPMP Groups on the Same Subnet ................................ 8-9

Describing IPMP .............................................................................. 8-10Network Path Failure Detection ........................................... 8-10Network Path Failover ........................................................... 8-11Network Path Failback........................................................... 8-11

Configuring IPMP............................................................................ 8-12Examining New ifconfig Options for IPMP.................... 8-12Putting Test Interfaces on Physical or Virtual

Interfaces ............................................................................... 8-13Using ifconfig Commands to Configure IPMP

Examples ............................................................................... 8-13Configuring the /etc/hostname. xxx Files for IPMP....... 8-14Multipathing Configuration File ......................................... 8-15

Performing Failover and Failback Manually ............................... 8-16Configuring IPMP in the Sun Cluster 3.1 Software

Environment .................................................................................. 8-17Configuring IPMP Before or After Cluster

Installation............................................................................. 8-17Using Same Group Names on Different Nodes ................. 8-17Understanding Standby and Failback ................................. 8-17

Integrating IPMP Into the Sun Cluster 3.1 SoftwareEnvironment .................................................................................. 8-18

Capabilities of the pnmd Daemon in Sun Cluster 3.1Software................................................................................. 8-18

Summary of IPMP Cluster Integration ................................ 8-19Viewing Cluster-Wide IPMP Information With

the scstat Command........................................................ 8-20Exercise: Configuring and Testing IPMP ..................................... 8-21

Task 1 – Verifying the local-mac-address? Variable .... 8-21Task 2 – Verifying the Adapters for the IPMP Group ....... 8-21Task 3 – Verifying or Entering Test Addresses in

the /etc/inet/hosts File ................................................ 8-22Task 4 – Creating /etc/hostname. xxx Files ..................... 8-22

xvCopyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1

Task 5 – Rebooting and Verifying That IPMP IsConfigured ............................................................................ 8-23

Task 6 – Verifying IPMP Failover and Failback ................. 8-23Exercise Summary............................................................................ 8-24

Introducing Data Services, Resource Groups, and HA-NFS ........9-1Objectives ........................................................................................... 9-1Relevance............................................................................................. 9-2Additional Resources ........................................................................ 9-3Introducing Data Services in the Cluster........................................ 9-4

Off-the-Shelf Applications ....................................................... 9-4Sun Cluster 3.1 Software Data Service Agents ..................... 9-5

Reviewing Components of a Data Service Agent ......................... 9-6Data Service Agent Fault-Monitor Components.................. 9-6

Introducing Data Service Packaging, Installation, andRegistration ...................................................................................... 9-7

Data Service Packages and Resource Types.......................... 9-7Introducing Resources, Resource Groups, and the

Resource Group Manager.............................................................. 9-8Resources.................................................................................... 9-8Resource Groups ....................................................................... 9-9Resource Group Manager...................................................... 9-11

Describing Failover Resource Groups .......................................... 9-12Resources and Resource Types ............................................. 9-13Resource Type Versioning..................................................... 9-13Data Service Resource Types................................................. 9-14

Using Special Resource Types........................................................ 9-15The SUNW.LogicalHostname Resource Type..................... 9-15The SUNW.SharedAddress Resource Type ......................... 9-15The SUNW.HAStorage Resource Type.................................. 9-15The SUNW.HAStoragePlus Resource Type ......................... 9-16Guidelines for Using the SUNW.HAStorage and

SUNW.HAStoragePlus Resource Types ............................ 9-17The SUNW.Event Resource Type........................................... 9-18Generic Data Service............................................................... 9-19Monitoring OPS/RAC............................................................ 9-19

Understanding Resource Dependencies and ResourceGroup Dependencies .................................................................... 9-21

Resource Dependencies ......................................................... 9-21Resource Group Dependencies............................................. 9-22

Configuring Resource and Resource Groups ThroughProperties ....................................................................................... 9-23

Standard Resource Properties ............................................... 9-24Extension Properties .............................................................. 9-25Resource Group Properties.................................................... 9-25

xvi Sun™ Cluster 3.1 AdministrationCopyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1

Examining Some Interesting Properties ....................................... 9-26Some Standard Resource Properties .................................... 9-26Some Resource Group Properties......................................... 9-27

Using the scrgadm Command ....................................................... 9-29Show Current Configuration................................................. 9-29Registering, Changing, or Removing a Resource

Type........................................................................................ 9-29Adding, Changing, or Removing Resource Groups.......... 9-29Adding a LogicalHostname or a SharedAddress

Resource ................................................................................ 9-30Adding, Changing, or Removing All Resources ................ 9-30Using the scrgadm Command ............................................. 9-31Setting Standard and Extension Properties......................... 9-31

Controlling Resources and Resource Groups by Using thescswitch Command .................................................................... 9-32

Resource Group Operations .................................................. 9-32Resource Operations.............................................................. 9-32Fault Monitor Operations ..................................................... 9-33

Confirming Resource and Resource Group Status byUsing the scstat Command ...................................................... 9-34

Using the scstat Command for a Single FailoverApplication ........................................................................... 9-34

Using the scsetup Utility for Resource and ResourceGroup Operations ......................................................................... 9-36

Exercise: Installing and Configuring the Sun Cluster HAfor NFS Software Package ........................................................... 9-37

Preparation............................................................................... 9-37Task 1 – Preparing to Register and Configure the

Sun Cluster HA for NFS Data Service ............................. 9-38Task 2 – Registering and Configuring the Sun Cluster

HA for NFS Data Service Software Package.................... 9-39Task 3 – Verifying Access by NFS Clients........................... 9-40Task 4 – Observing Sun Cluster HA for NFS Failover

Behavior................................................................................. 9-41Task 5 – Generating Cluster Failures and Observing

Behavior of the NFS Failover ............................................. 9-42Task 6 – Configuring NFS to Use a Local Failover

File System ............................................................................ 9-42Task 7 – Viewing and Managing Resources and

Resource Groups Through SunPlex Manager ................. 9-43Exercise Summary............................................................................ 9-44

Configuring Scalable Services ..................................................... 10-1Objectives ......................................................................................... 10-1Relevance........................................................................................... 10-2Additional Resources ...................................................................... 10-3

xviiCopyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1

Using Scalable Services and Shared Addresses........................... 10-4Exploring Characteristics of Scalable Services............................. 10-5

File and Data Access............................................................... 10-5File Locking for Writing Data ............................................... 10-5

Using the SharedAddress Resource as IP andLoad-Balancer................................................................................ 10-6

Client Affinity.......................................................................... 10-6Load-Balancing Weights ........................................................ 10-6

Exploring Resource Groups for Scalable Services....................... 10-7Resources and Their Properties in the Resource

Groups .................................................................................. 10-8Understanding Properties for Scalable Groups and

Services ........................................................................................... 10-9The Desired_primaries and Maximum_primaries

Properties .............................................................................. 10-9The Load_balancing_policy Property ............................. 10-9The Load_balancing_weights Property........................... 10-9The Network_resources_used Property......................... 10-10

Adding Auxiliary Nodes for a SharedAddress Property ....... 10-11Reviewing scrgadm Command Examples for Scalable

Service........................................................................................... 10-12Controlling Scalable Resources and Resource Groups

With the scswitch Command.................................................. 10-13Resource Group Operations ................................................ 10-13Resource Operations............................................................. 10-13Fault Monitor Operations .................................................... 10-14

Using the scstat Command for a Scalable Application......... 10-15Describing the RGOffload Resource: Clearing the Way

for Critical Resources.................................................................. 10-16Example of Using the RGOffload Resource .................... 10-17

Exercise: Installing and Configuring Sun ClusterScalable Service for Apache....................................................... 10-18

Preparation............................................................................. 10-18Task 1 – Preparing for HA-Apache Data Service

Configuration .................................................................... 10-19Task 2 – Registering and Configuring the Sun

Cluster HA for Apache Data Service .............................. 10-21Task 3 – Verifying Apache Web Server Access ................ 10-22Task 4 – Observing Cluster Failures................................... 10-23Task 5 (Optional) – Configuring the RGOffload

Resource in Your NFS Resource Group to“Push Your Apache Out of the Way” ............................. 10-23

Exercise Summary.......................................................................... 10-24

xviii Sun™ Cluster 3.1 AdministrationCopyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1

Performing Supplemental Exercises for Sun Cluster 3.1Software.......................................................................................... 11-1

Objectives ......................................................................................... 11-1Relevance........................................................................................... 11-2Additional Resources ...................................................................... 11-3Exploring the Exercises ................................................................... 11-4

Configuring HA-LDAP.......................................................... 11-4Configuring HA-ORACLE .................................................... 11-5Configuring ORACLE RAC .................................................. 11-5Reinstalling Sun Cluster 3.1 Software.................................. 11-5

Exercise 1: Integrating Sun ONE Directory Server 5Software Into Sun Cluster 3.1...................................................... 11-6

Preparation............................................................................... 11-6Task 1 – Creating nsuser and nsgroup Accounts............ 11-7Task 2 – Registering the SUNW.HAStoragePlus

Resource Type ...................................................................... 11-7Task 3 – Installing and Registering the SUNW.nsldap

Resource Type ...................................................................... 11-7Task 4 – Creating a Global File System for Server

Root ........................................................................................ 11-7Task 5 – Configuring Mount Information for the

File System ............................................................................ 11-8Task 6 – Creating an Entry in the Name Service for

the Logical Host ................................................................... 11-9Task 7 – Running directoryserver Setup Script on

the First Node ....................................................................... 11-9Task 8 – Verifying That Directory Server Is Running

Properly ............................................................................... 11-10Task 9 – Stopping the Directory Service on the

First Node............................................................................ 11-10Task 10 – Running directoryserver Setup Script

on the Second Node........................................................... 11-11Task 11 – Stopping the Directory Service on the

Second Node....................................................................... 11-12Task 12 – Replacing the /var/ds5 Directory on the

Second Node...................................................................... 11-12Task 13 – Verifying That the Directory Server

Runs on the Second Node................................................. 11-12Task 14 – Stopping the Directory Service on the

Second Node Again ........................................................... 11-13Task 15 – Repeating Installation Steps on

Remaining Nodes............................................................... 11-13Task 16 – Creating Failover Resource Group and

Resources for Directory Server ........................................ 11-13Task 17 – Verifying HA-LDAP Runs Properly in

the Cluster ........................................................................... 11-14

xixCopyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1

Exercise 2: Integrating ORACLE 9i Into Sun Cluster 3.1Software........................................................................................ 11-15

Task 1 – Creating ha-ora-lh Entry in the/etc/inet/hosts File ...................................................... 11-15

Task 2 – Creating oracle and dba Accounts .................. 11-16Task 3 – Creating Global File System for ORACLE

Software............................................................................... 11-16Task 4 – Preparing the oracle User Environment .......... 11-17Task 5 – Disabling Access Control of X Server on

Admin Workstation........................................................... 11-17Task 6 – Editing the /etc/system File for ORACLE

Parameters........................................................................... 11-18Task 7 – Running the runInstaller Installation

Script .................................................................................... 11-18Task 8 – Preparing ORACLE Instance for Cluster

Integration........................................................................... 11-22Task 9 – Stopping the ORACLE Software ......................... 11-23Task 10 – Verifying ORACLE Software Runs on

Other Nodes........................................................................ 11-24Task 11 – Registering the SUNW.HAStoragePlus Type... 11-25Task 12 – Installing and Registering ORACLE Data

Service.................................................................................. 11-25Task 13 – Creating Resources and Resource

Groups for ORACLE ......................................................... 11-25Task 14 – Verifying ORACLE Runs Properly in

the Cluster ........................................................................... 11-26Exercise 3: Running ORACLE 9i RAC in Sun Cluster 3.1

Software........................................................................................ 11-28Preparation............................................................................. 11-28Task 1 – Creating User and Group Accounts .................. 11-29Task 2 – Installing Cluster Packages for ORACLE

RAC...................................................................................... 11-29Task 3 – Installing ORACLE Distributed Lock

Manager............................................................................... 11-29Task 4 – Modifying the /etc/system File ........................ 11-30Task 5 – Installing the CVM License Key .......................... 11-30Task 6 – Rebooting the Cluster Nodes............................... 11-31Task 7 – Creating Raw Volumes ......................................... 11-31Task 8 – Configuring oracle User Environment ............ 11-33Task 9 – Creating the dbca_raw_config File................... 11-34Task 10 – Disabling Access Control on X Server of

Admin Workstation........................................................... 11-34Task 11 – Installing ORACLE Software ............................. 11-35Task 12 – Configuring ORACLE Listener ........................ 11-38Task 13 – Starting the Global Services Daemon .............. 11-39

xx Sun™ Cluster 3.1 AdministrationCopyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1

Task 14 – Creating ORACLE Database.............................. 11-39Task 15 – Verifying That ORACLE RAC Works

Properly in Cluster............................................................ 11-41Exercise 4: Reinstalling Sun Cluster 3.1 Software ..................... 11-43

Terminal Concentrator ....................................................................A-1Viewing the Terminal Concentrator .............................................. A-2

Operating System Load.......................................................... A-3Setup Port.................................................................................. A-3Terminal Concentrator Setup Programs............................... A-3

Setting Up the Terminal Concentrator........................................... A-4Connecting to Port 1 ................................................................ A-4Enabling Setup Mode .............................................................. A-4Setting the Terminal Concentrator IP Address................... A-5Setting the Terminal Concentrator Load Source ................. A-5Specifying the Operating System Image .............................. A-6Setting the Serial Port Variables............................................. A-6Setting the Port Password....................................................... A-7

Setting a Terminal Concentrator Default Route ........................... A-9Creating a Terminal Concentrator Default Route ............ A-10

Using Multiple Terminal Concentrators...................................... A-11Troubleshooting Terminal Concentrators ................................... A-12

Using the telnet Command to Manually Connectto a Node .............................................................................. A-12

Using the telnet Command to Abort a Node.................. A-12Connecting to the Terminal Concentrator

Command-Line Interpreter .............................................. A-13Using the Terminal Concentrator help Command .......... A-13Identifying and Resetting a Locked Port ............................ A-13Erasing Terminal Concentrator Settings............................ A-14

Configuring Multi-Initiator SCSI .....................................................B-1Multi-Initiator Overview ..................................................................B-2Installing a Sun StorEdge Multipack Device .................................B-3Installing a Sun StorEdge D1000 Array ..........................................B-7The nvramrc Editor and nvedit Keystroke Commands...........B-11

Role-Based Access Control Authorizations..................................C-1RBAC Cluster Authorizations......................................................... C-2

Preface-xxiCopyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1

Preface

About This Course

Rebranding Notice: Recently, Sun rebranded its popular Sun™ Open NetEnvironment (Sun ONE) product line under the Sun Java™ System brand.Some course materials might contain the former brand names until Sunreleases new versions of the course materials. In addition, you may seereference to either name within the course description and coursematerials. Nevertheless, the training you are about to receive provides themost current information available regarding Sun Java System products.

Course Goals

Upon completion of this course, you should be able to:

● Describe the major Sun™ Cluster software components andfunctions

● Configure different methods of connecting to node consoles

● Configure a cluster administrative workstation

● Install the Sun Cluster 3.1 software

● Configure Sun Cluster 3.1 software quorum devices

● Configure VERITAS Volume Manager in the Sun Cluster softwareenvironment

● Configure Solaris™ Volume Manager software in the Sun Clustersoftware environment

● Create Internet Protocol Multipathing (IPMP) failover groups in theSun Cluster 3.1 software environment

● Understand resources and resource groups

Course Goals

Preface-xxii Sun™ Cluster 3.1 AdministrationCopyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1

● Configure a failover data service resource group (NFS)

● Configure a scalable data service resource group (Apache)

● Configure Sun Java™ System Directory Server (formerly Sun™ OpenNet Environment [Sun ONE] Directory Server) and ORACLE® in theSun Cluster software environment

Course Map

About This Course Preface-xxiiiCopyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1

Course Map

The following course map enables you to see what you haveaccomplished and where you are going in reference to the course goals.

Performing SupplementalExercises for Sun

Cluster 3.1 Software

Product Introduction

Supplemental Exercises

IntroducingSun™ Cluster

Hardware and Software

Customization

Using VERITAS

Volume Management

Managing Volumes With

Software (Solstice

Managing the

Introducing Data Configuring ScalableServices

Volume Manager for Solaris™ Volume Manager

DiskSuite™ Software)

Public NetworkWith IPMP

Services, ResourceGroups, and HA-NFS

Operation

Performing BasicCluster

Administration

Installation

Exploring Node Console Preparing for Installationand UnderstandingQuorum Devices

Installing the SunCluster Host Software

Connectivity and theCluster Console Software

Topics Not Covered

Preface-xxiv Sun™ Cluster 3.1 AdministrationCopyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1

Topics Not Covered

This course does not cover the topics shown on the overhead. Many of thetopics listed on the overhead are described in other courses offered by SunServices:

● Database installation and management – Described in databasevendor courses

● Network administration – Described in SA-399: NetworkAdministration for the Solaris™ 9 Operating Environment

● Solaris™ Operating System (Solaris OS) administration – Describedin SA-239: Intermediate System Administration for the Solaris™ 9Operating Environment and SA-299: Advanced System Administrationfor the Solaris™ 9 Operating Environment

● Disk storage management – Described in ES-220: Disk ManagementWith DiskSuite and ES-310: Sun StorEdge™ Volume ManagerAdministration

Refer to the Sun Services catalog for specific information and registration.

How Prepared Are You?

About This Course Preface-xxvCopyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1

How Prepared Are You?

To be sure you are prepared to take this course, can you answer yes to thefollowing questions?

● Can you explain virtual volume management terminology, such asmirroring, striping, concatenation, volumes, and mirrorsynchronization?

● Can you perform basic Solaris OS administration tasks, such as usingtar and ufsdump commands, creating user accounts, formatting diskdrives, using vi , installing the Solaris OS, installing patches, andadding packages?

● Do you have prior experience with Sun hardware and theOpenBoot™ programmable read-only memory (PROM) technology?

● Are you familiar with general computer hardware, electrostaticprecautions, and safe-handling practices?

Introductions

Preface-xxvi Sun™ Cluster 3.1 AdministrationCopyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1

Introductions

Now that you have been introduced to the course, introduce yourself toeach other and the instructor, addressing the items shown in the followingbullets.

● Name

● Company affiliation

● Title, function, and job responsibility

● Experience related to topics presented in this course

● Reasons for enrolling in this course

● Expectations for this course

How to Use Course Materials

About This Course Preface-xxviiCopyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1

How to Use Course Materials

To enable you to succeed in this course, these course materials use alearning model that is composed of the following components:

● Goals – You should be able to accomplish the goals after finishingthis course and meeting all of its objectives.

● Objectives – You should be able to accomplish the objectives aftercompleting a portion of instructional content. Objectives supportgoals and can support other higher-level objectives.

● Lecture – The instructor will present information specific to theobjective of the module. This information should help you learn theknowledge and skills necessary to succeed with the activities.

● Activities – The activities take on various forms, such as an exercise,self-check, discussion, and demonstration. Activities help to facilitatemastery of an objective.

● Visual aids – The instructor might use several visual aids to convey aconcept, such as a process, in a visual form. Visual aids commonlycontain graphics, animation, and video.

Conventions

Preface-xxviii Sun™ Cluster 3.1 AdministrationCopyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1

Conventions

The following conventions are used in this course to represent varioustraining elements and alternative learning resources.

Icons

Additional resources – Indicates other references that provide additionalinformation on the topics described in the module.

?!

Discussion – Indicates a small-group or class discussion on the currenttopic is recommended at this time.

Note – Indicates additional information that can help students but is notcrucial to their understanding of the concept being described. Studentsshould be able to understand the concept or complete the task withoutthis information. Examples of notational information include keywordshortcuts and minor system adjustments.

Caution – Indicates that there is a risk of personal injury from anonelectrical hazard, or risk of irreversible damage to data, software, orthe operating system. A caution indicates that the possibility of a hazard(as opposed to certainty) might happen, depending on the action of theuser.

Conventions

About This Course Preface-xxixCopyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1

Typographical Conventions

Courier is used for the names of commands, files, directories,programming code, and on-screen computer output; for example:

Use ls -al to list all files.system% You have mail .

Courier is also used to indicate programming constructs, such as classnames, methods, and keywords; for example:

Use the getServletInfo method to get author information.The java.awt.Dialog class contains Dialog constructor.

Courier bold is used for characters and numbers that you type; forexample:

To list the files in this directory, type:# ls

Courier bold is also used for each line of programming code that isreferenced in a textual description; for example:

1 import java.io.*;2 import javax.servlet.*;3 import javax.servlet.http.*;Notice the javax.servlet interface is imported to allow access to itslife cycle methods (Line 2).

Courier italic is used for variables and command-line placeholdersthat are replaced with a real name or value; for example:

To delete a file, use the rm filename command.

Courier italic bold is used to represent variables whose values are tobe entered by the student as part of an activity; for example:

Type chmod a+rwx filename to grant read, write, and executerights for filename to world, group, and users.

Palatino italic is used for book titles, new words or terms, or words thatyou want to emphasize; for example:

Read Chapter 6 in the User’s Guide.These are called class options.

1-1Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1

Module 1

Introducing Sun™ Cluster Hardware andSoftware

Objectives

Upon completion of this module, you should be able to:

● Define the concept of clustering

● Describe the Sun™ Cluster 3.1 hardware and software environment

● Introduce the Sun Cluster 3.1 hardware environment

● View the Sun Cluster 3.1 software support

● Describe the types of applications in the Sun Cluster 3.1 softwareenvironment

● Identify the Sun Cluster 3.1 software data service support

● Explore the Sun Cluster 3.1 software high availability framework

● Define global storage services differences

Relevance

1-2 Sun™ Cluster 3.1 AdministrationCopyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1

Relevance

?!

Discussion – The following questions are relevant to understanding thecontent of this module:

● Are there similarities between the redundant hardware componentsin a standalone server and the redundant hardware components in acluster?

● Why is it impossible to achieve industry standards of highavailability on a standalone server?

● Why are a variety of applications that run in the Sun Clustersoftware environment said to be cluster unaware?

● How do cluster-unaware applications differ from cluster-awareapplications?

● What services does the Sun Cluster software framework provide forall the applications running in the cluster?

● Do global devices and global file systems have usage other than forscalable applications in the cluster?

Additional Resources

Introducing Sun™ Cluster Hardware and Software 1-3Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1


Additional resources – The following references provide additionalinformation on the topics described in this module:

● Sun Cluster 3.1 4/04 Software Collection for Solaris™ OperatingSystem (x86 Platform Edition)

● Sun Cluster 3.1 4/04 Software Collection for Solaris OperatingSystem (SPARC® Platform Edition)

● Sun Cluster 3.1 4/04 Hardware Collection for Solaris OperatingSystem (x86 Platform Edition)

● Sun Cluster 3.1 4/04 Hardware Collection for Solaris OperatingSystem (SPARC Platform Edition)

● Sun Cluster 3.x Hardware Collection for Solaris OS (SPARC PlatformEdition)

Defining Clustering


Defining Clustering

Clustering is a general terminology that describes a group of two or moreseparate servers operating as a “harmonious unit.” Clusters generallyhave the following characteristics:

● Separate server nodes, each booting from its own, non-shared, copy ofthe operating system.

● Dedicated hardware interconnect, providing private transport onlybetween the nodes of the same cluster.

● Multi-ported storage, providing paths from at least two nodes in thecluster to each physical storage device storing data for theapplications running in the cluster.

● Cluster software “framework,” providing cluster-specific knowledgeto the nodes in the cluster about the health of the hardware and thehealth and state of their peer nodes.

● General goal of providing platform for high availability andscalability for the applications running in the cluster.

● Support for a variety of cluster-unaware applications and cluster-awareapplications.

High-Availability Platforms

Clusters are generally marketed as the only way to provide highavailability (HA) for the applications that run on them.

HA can be defined as a minimization of downtime rather than thecomplete elimination of down time. Many standalone servers themselvesare marketed as “providing higher levels of availability than ourcompetitors (or predecessors).” Most true standards of HA cannot beachieved in a standalone server environment.

HA Standards

HA standards are usually phrased with wording such as “provides5 nines availability.” This means 99.999 percent uptime for the applicationor about 5 minutes of downtime per year. One clean server reboot oftenexceeds that amount of downtime.

Defining Clustering


How Clusters Provide HA (Inter-Node Failover)

Cluster environments provide an environment where, in the case of anysingle hardware or software failure in the cluster, application services anddata are recovered automatically (without human intervention) andquickly (faster than a server reboot). This is done by taking advantage ofthe existence of the redundant servers in the cluster and redundantserver-storage paths.

HA Benefits for Unplanned and Planned Outages

The HA benefit that cluster environments provide involves not onlyhardware and software failures, but also “planned outages.” Where acluster has the ability to automatically relocate applications inside thecluster in the case of failures, it also provides the ability to manuallyrelocate services for planned outages. As such, normal reboots orhardware maintenance in the cluster only affects the uptime of theapplications for as much time as it takes to manually relocate them todifferent servers in the cluster.

Fault-Tolerant Servers Are Not an Alternative to HA Clusters

Many vendors provide servers that are marketed as fault-tolerant. Theseservers are designed to be able to tolerate any single hardware failure(including memory failure, central processing unit (CPU) failure, and thelike) without any down time whatsoever.

There is a common misconception that fault-tolerant servers are analternative to HA clusters, or that a fault-tolerant server supersedes HA insome way. In fact, while fault-tolerant servers can hide any hardwarefailure, they are not designed to provide especially fast recovery in thecase of a software failure, such as a Solaris OS kernel panic or anapplication failure. Recovery in these circumstances on a singlefault-tolerant server might still require a full OS reboot, which, aspreviously stated, might already exceed maximum downtime permittedby the HA standards to which you aspire.

Defining Clustering


Scalability Platforms

Clusters also provide an integrated hardware and software environmentfor scalability. Scalability is defined as the ability to increase applicationperformance by supporting multiple instances of applications on differentnodes in the cluster. These instances are generally accessing the same dataas each other.

High Availability and Scalability Are Not MutuallyExclusive

Clusters generally do not require a choice between availability andperformance. HA is generally built-in to scalable applications as well asnon-scalable ones. In scalable applications, you might not need to relocatefailed applications because other instances are already running on othernodes. You might still need to perform recovery on behalf of failedinstances.

Describing the Sun Cluster 3.1 Hardware and Software Environment


Describing the Sun Cluster 3.1 Hardware and SoftwareEnvironment

The Sun Cluster 3.1 hardware and software environment is Sun’s latestgeneration clustering product. The following features distinguish the SunCluster hardware and software product from competitors in the field:

● Supports from two to eight nodes – Nodes can be added, removed, andreplaced from the cluster without any interruption of applicationservice.

● Global device implementation – While data storage must be physicallyconnected on paths from at least two different nodes in the SunCluster 3.1 hardware and software environment, all the storage inthe cluster is logically available from every node in the cluster usingstandard device semantics.

This provides a huge amount of flexibility in being able to runapplications on nodes that use data that is not even physicallyconnected to the nodes. More description of global device filenaming and location is in ‘‘Defining Global Storage ServiceDifferences’’ on page 1-29.

● Global file system implementation – The Sun Cluster softwareframework provides a global file service independent of anyparticular application running in the cluster, so that the same filescan be accessed on every node of the cluster, regardless of thestorage topology.

● Cluster framework services implemented in the kernel – The Sun Clustersoftware is tightly integrated with the Solaris 8 OS and Solaris 9 OSkernels. Node monitoring capability, transport monitoring capability,and the global device and file system implementation among others,are all implemented in the kernel to provide higher reliability andperformance.

Describing the Sun Cluster 3.1 Hardware and Software Environment


● Support for a large variety of “off-the-shelf” applications – The SunCluster hardware and software product includes data service agentsfor a large variety of cluster-unaware applications. These are testedscripts and fault monitors that make applications run properly in thecluster environment with the inherent high availability andscalability that entails. A full list of data service agents is in‘‘Identifying Sun Cluster 3.1 Software Data Service Support’’ onpage 1-24.

● Support for some “off-the-shelf “applications as scalable applications withbuilt-in load balancing (global interfaces) – The global interface featureprovides a single IP address and load-balancing service for somescalable services, for example, Apache Web Server and Sun Java™System Web and Application Services (formerly known as Sun ONEWeb and Application Services). Clients outside the cluster see themultiple node instances of the service as a single service with asingle Internet Protocol (IP) address.

Introducing the Sun Cluster 3.1 Hardware Environment



The Sun Cluster 3.1 hardware environment supports a maximum of eightnodes. Figure 1-1 demonstrates the hardware components of a typicaltwo-node cluster:

● Cluster nodes running Solaris 8 OS (Update 7 or higher) orSolaris 9 OS

● Separate boot disks on each node (with a preference for boot disksmirrored on each node)

● One or more public network interfaces per system per subnet (with apreferred minimum of at least two public network interfaces)

● A redundant private cluster transport interface

● Dual-hosted, mirrored disk storage

● One terminal concentrator (or any other console access method)

● Administrative console

Figure 1-1 Minimal Sun Cluster 3.1 Hardware Environment

Network

Node 1 Node 2

IPMPgroup

Serialport A

Multihost StorageMultihost Storage

Public net

AdministrationWorkstation

TerminalConcentrator

RedundantTransport

Boot DisksBoot Disks



Cluster Host Systems

A wide range of Sun hardware platforms are supported for use in theclustered environment. These range from small rackmounted servers(SPARC and x86 servers) up to Sun’s largest enterprise-level servers,including Sun Fire™ 15K servers.

Many heterogeneous environments are supported (that is, different typesof servers used as nodes in the same cluster). It depends on the networkand storage host adapters used, not on the servers themselves. The servermust be a standalone (the server, the network, and the storage) supportedserver to be used as a cluster node.

Cluster Transport Interface

All nodes in a cluster are linked by a private cluster transport. Thetransport is redundant and can be used for the following purposes:

● Cluster-wide monitoring and recovery

● Global data access (transparent to applications)

● Application specific transport for cluster-aware applications (such asORACLE® Parallel Server)

Clusters must be defined with a minimum of two separate “privatenetworks” that form the cluster transport. You can have more than two(and can add more later) that would be a performance benefit in certaincircumstances, because global data access traffic is “striped” across all ofthe transports.

Crossover cables are often used in a two-node cluster. Switches areoptional when you have two nodes and required for more than twonodes.

The following are the types of cluster transport hardware supported:

● Fast Ethernet (100BASE-T). This is the most common type oftransport.



● Gigabit Ethernet (1000BASE-SX). This is used when a higher clustertransport speed is required for proper functioning of a data service.

● Scalable coherent interface (PCI SCI) intended for remote sharedmemory (RSM) applications. These applications are described in‘‘Other Remote Shared Memory (RSM) Applications’’ on page 1-23.

● Sun Fire™ Link hardware intended for RSM applications andavailable in the Sun Fire 3800–6800 servers and Sun Fire 15K/12Kservers.

Public Network Interfaces

Each node must have public network interfaces under control of theSolaris OS IPMP software. It is recommended, though not required, that atleast two interfaces in an IPMP group connect each node to each subnetused. The cluster requires that all nodes on which an application mightrun be on the same subnet.

You can have connectivity to as many subnets as you want, but a SunCluster hardware host is not allowed to act as a router. Not all availablenetwork interface cards are supported in the Sun Cluster 3.1 hardwareenvironment.



Cluster Disk Storage

The Sun Cluster hardware environment can use several Sun storagemodels. They must all accept multihost connections. The Sun StorEdge™T3 array has a single connection and must be used with a hub or a switch.

Most data storage in the cluster has exactly two hosts (in other words, thenodes connected to it). Certain storage configurations support more thantwo nodes connected to the storage.

VERITAS Volume Manager or Solaris Volume Manager (SolsticeDiskSuite™) software must be used to mirror the storage acrosscontrollers.

Boot Disks

The Sun Cluster 3.1 environment requires that boot disks for each node belocal to the node. That is, not in any of the multi-ported storage arrays.

Having two local drives for disks and mirroring with VERITAS VolumeManager or Solaris Volume Manager software is recommended.

Terminal Concentrator

The server nodes in the cluster typically run without a graphics console,although the Sun graphics console (keyboard and monitor) is allowedwhen supported by the base server configuration. If the server nodes donot use a Sun graphics console, you can use any method you want toconnect to the serial (ttyA ) consoles of the nodes.

A terminal concentrator (TC) is a device that provides data translationfrom the network to serial port interfaces. Each of the serial port outputsconnects to a separate node in the cluster through serial port A.



Another way of getting the convenience of remote console access is to justuse a workstation that has serial ports connected to serial port A of eachnode (which is perfect for workstations with two serial ports in a two-node cluster). You could access this workstation remotely and then usethe tip command to access the node consoles.

There is always a trade-off between convenience and security. You mightprefer to have only “dumb-terminal” console access to the cluster nodes,and keep these terminals behind locked doors requiring stringent securitychecks to open. That would be acceptable (although less convenient toadminister) for Sun Cluster 3.1 hardware as well.

Administrative Workstation

Included with the Sun Cluster software is the administration consolesoftware, which can be installed on any SPARC or x86 Solaris OSworkstation. The software can be a convenience in managing the multiplenodes of the cluster from a centralized location. It does not affect thecluster in any other way.

Clusters With More Than Two Nodes

In clusters with more than two nodes:

● Switches are required for the transport. There must be at least twoswitches. Each node is connected to each switch to form a redundanttransport.

● A variety of storage topologies are supported. Figure 1-2 onpage 1-14 shows an example of a “Pair + N” topology. All the nodesin the cluster have access to the data in the multihost storagethrough the global devices and global file system features of the SunCluster environment.



See Module 3, “Preparing for Installation and UnderstandingQuorum Devices,” for more information on these subjects.

Figure 1-2 “Pair + N” Topology

Sun Cluster Hardware Redundancy Features

The following items summarize the generally required and optionalhardware redundancy features in the Sun Cluster hardware environment:

● Redundant server nodes are required.

● Redundant transport is required.

● Redundant storage arrays are required.

● Hardware redundant array of independent disks (RAID) storagearrays are optional.

● Software mirroring across data controllers is required.

● Redundant public network interfaces per subnet are recommended.

● Redundant boot disks are recommended.

Node Node

RedundantTransport

Public net

NodeNode



It is always recommended, although not required, to have redundantcomponents as far apart as possible. For example, on a system withmultiple input/output (I/O) boards, you would want to put theredundant transport interfaces, the redundant public nets, and theredundant storage array controllers on two different I/O boards.

There are specific exceptions to some of these requirements. Moreconfigurations are available as supported configurations evolve.Verification that a specific hardware configuration is supported must beperformed.

Cluster in a Box

The Sun Cluster 3.1 hardware environment supports clusters with both(or all) nodes in the same physical box for the following domain-basedservers (current at the time of writing of this course):

● Sun Fire 15K/12K servers

● Sun Fire 3800–6800 servers

● Sun Enterprise™ 10000 servers

You should take as many redundancy precautions as possible, forexample, running multiple domains in different segments onSun Fire 3800–6800 servers and segmenting across the power plane onSun Fire 6800 servers.



If a customer chooses to configure the Sun Cluster 3.1 hardwareenvironment in this way, there must be a realization that there could be agreater chance of a single hardware failure causing down time for theentire cluster than there could be when clustering independent standaloneservers, or domains from domain-based servers across different physicalplatforms (Figure 1-3).

Figure 1-3 Domain-based Cluster in a Box

domain-A Window

ok boot

domain-B Window

ok bootEthernet

System Unit

SystemController

Board

domain-A domain-B

System Boards

I/O Board BootDisk

System Boards

I/O Board BootDisk

Viewing Sun Cluster 3.1 Software Support



To function as a cluster member, the following types of software must beinstalled on every Sun Cluster hardware node:

● Solaris OS software

● Sun Cluster software

● Data service application software

● Logical volume management

An exception is a configuration that uses hardware redundant array ofindependent disks (RAID). This configuration might not require asoftware volume manager.

Figure 1-4 provides a high-level overview of the software componentsthat work together to create the Sun Cluster software environment.

Figure 1-4 Sun Cluster Software Layered Model

Data Service Software

User

Kernel

Volume Management Software

Solaris Operating System

Sun Cluster Software



The following software revisions are supported by Sun Cluster 3.1software:

● Solaris OS versions (each node must run same version)

● Solaris 8 OS Update 7 (02/02)

● Solaris 8 OS (12/02, 5/03, 7/03)

● Solaris 9 OS (all updates)

● VERITAS Volume Manager versions

● VERITAS Volume Manager 3.2 (for the Solaris 8 OS)

● VERITAS Volume Manager 3.5 (for the Solaris 8 OS orSolaris 9 OS)

● Solstice DiskSuite software revisions

● Solstice DiskSuite 4.2.1 software for the Solaris 8 OS

● Soft partitions supported with patch 108693-06 or greater

● Solaris Volume Manager software revisions – Solaris VolumeManager software (part of base OS in the Solaris 9 OS)

Note – New Solaris 9 OS updates have some lag before being officiallysupported in the cluster. If there is any question, consult Sun supportpersonnel who have access to the latest support matrices.

Describing the Types of Applications in the Sun Cluster Software Environment


Describing the Types of Applications in the Sun ClusterSoftware Environment

The Sun Cluster software environment supports applications with HighAvailability (HA) and scalability.

Cluster-Unaware (Off-the-Shelf) Applications

The majority of applications in the cluster are in the cluster-unawarecategory and are part of the main focus of this course.

Two main categories of cluster-unaware applications can run in the SunCluster 3.1 Environment. These are described on the following pages andin detail in Module 9, “Introducing Data Services, Resource Groups, andHA-NFS,” and Module 10, “Configuring Scalable Services.” Theapplications include:

● Failover applications

● Scalable applications

The common elements of all of these applications are:

● The cluster’s resource group manager (RGM) coordinates all stoppingand starting of the applications. They are never started and stoppedby traditional Solaris OS run-control (rc) scripts.

● A data service agent for the application provides the “glue pieces” tomake it work properly in the Sun Cluster software environment. Thisincludes methods to start and stop the application appropriately inthe cluster, as well as fault monitors specific for that application inthe cluster.



Failover Applications

The failover model is the easiest to provide for in the cluster. Failoverapplications run on only one node of the cluster at a time. The clusterprovides high-availability by providing automatic restart on the same or adifferent node of the cluster.

Failover services are usually paired with an application IP address. This isan IP address that always fails over from node to node along with theapplication. In this way, clients outside the cluster see a logical host namewith no knowledge on which node a service is running or evenknowledge that a service is running in the cluster.

Multiple failover applications in the same resource group can share an IPaddress, with the restriction that they must all fail over to the same nodetogether as shown in Figure 1-5.

Figure 1-5 Multiple Failover Applications in the Same Resource Group

Node 1 Node 2

Resource Group(Local File System Mounts)ApplicationsApplication IPs



Scalable Applications

Scalable applications involve running multiple instances of an applicationin the same cluster and making it look like a single service by means of aglobal interface that provides a single IP address and load balancing.

While scalable applications (Figure 1-6) are still off-the-shelf, not everyapplication can be made to run as a scalable application in the SunCluster 3.1 software environment. Applications that write data withoutany type of locking mechanism might work as failover applications butdo not work as scalable applications.

Figure 1-6 Scalable Application Work Flow

<HTML>

</HTML>

<HTML>

</HTML>

<HTML>

</HTML>

Node 1

Node 2 Node 3

Globally Available HTML Documents

HTTP Application HTTP

ApplicationHTTP

Application

Global Interfacexyz.com

Request Distribution

Transport

WebPage

Client 1 Client 2 Client 3

Requests RequestsNetwork



Cluster-Aware Applications

Cluster-aware applications are applications where knowledge of the clusteris built-in to the software. They differ from cluster-unaware applications inthe following ways:

● Multiple instances of the application running on different nodes areaware of each other and communicate across the private transport.

● It is not required that the Sun Cluster software framework RGM startand stop these applications. Because these applications arecluster-aware, they could be started in their own independentscripts, or by hand.

● Applications are not necessarily logically grouped with externalapplication IP addresses. If they are, the network connections can bemonitored by cluster commands. It is also possible to monitor thesecluster-aware applications with Sun Cluster 3.1 software frameworkresource types.

Parallel Database Applications

Parallel databases are a special type of cluster application. Multipleinstances of the database server cooperate in the cluster, handlingdifferent queries on the same database and even providing parallel querycapability on large queries. The following are supported in the SunCluster 3.1 software environment:

● ORACLE 8i Parallel Server (ORACLE 8i OPS)

● ORACLE 9i Real Application Cluster (ORACLE 9i RAC)



Other Remote Shared Memory (RSM) Applications

Applications that run on Sun Cluster 3.1 hardware, that also contain theSCI or Sun Fire Link special interconnects, can make use of an applicationprogramming interface (API) called RSM. This maps data from anapplication instance running on one node into the address space of aninstance running on another node. This can be a highly efficient way forcluster-aware applications to share large amounts of data across thetransport.

ORACLE 9i RAC is the only application that is supported in the SunCluster 3.1 software environment that can make use of RSM (if you havethe right interconnect).

Note – This course focuses on the cluster-unaware applications ratherthan the types presented on this page. Module 11, “PerformingSupplemental Exercises for Sun Cluster 3.1 Software,” containsprocedures and an optional exercise for running ORACLE 9i RAC(without RSM).

Identifying Sun Cluster 3.1 Software Data Service Support



This section contains a list of Sun-supported data service agents that makecluster-unaware applications highly available, either in failover or scalableconfigurations.

HA and Scalable Data Service Support

The Sun Cluster software provides preconfigured components thatsupport the following HA data services. The release of new data serviceagents does not correspond exactly with updates of the Sun Clustersoftware framework. Some agents are available from various divisionswithin Sun, rather than being on the Sun Cluster software data servicesCD-ROM. These were the components available at the time of writing ofthis course:

● Sun Cluster HA for ORACLE Server (failover)

● Sun Cluster HA for ORACLE E-Business Suite (failover)

● Sun Cluster HA for Sun Java System Web Server (failover orscalable)

● Sun Cluster HA for Sun Java System Web Proxy Server (failover)

● Sun Cluster HA for Sun Java System Application Server SE/PE(failover)

● Sun Cluster HA for Sun Java System Directory Server (failover)

● Sun Cluster HA for Sun Java System Message Queue software(failover)

● Sun Cluster HA for Sun StorEdge™ Availability Suite (failover)

● Sun Cluster HA for Sun StorEdge Enterprise Backup Software (EBS)(failover)

● Sun Cluster HA for Apache Web Server (failover or scalable)

● Sun Cluster HA for Apache Proxy Server (failover)

● Sun Cluster HA for Apache Tomcat (failover or scalable)

● Sun Cluster HA for Domain Name System (DNS) (failover)

● Sun Cluster HA for Network File System (NFS) (failover)



● Sun Cluster HA for Dynamic Host Configuration Protocol (DHCP)(failover)

● Sun Cluster HA for SAP (failover or scalable)

● Sun Cluster HA for SAP LiveCache (failover)

● Sun Cluster HA for Sybase ASE (failover)

● Sun Cluster HA for VERITAS NetBackup (failover)

● Sun Cluster HA for BroadVision (scalable)

● Sun Cluster HA for Siebel (failover)

● Sun Cluster HA for Samba (failover)

● Sun Cluster HA for BEA WebLogic Application Server (failover)

● Sun Cluster HA for IBM Websphere MQ (failover)

● Sun Cluster HA for IBM Websphere MQ Integrator (failover)

● Sun Cluster HA for MySQL (failover)

● Sun Cluster HA for SWIFTalliance (failover)

Exploring the Sun Cluster Software HA Framework



The Sun Cluster software framework is the software layer that providesgeneric cluster services to the nodes in the cluster, regardless of whichapplications are running in the cluster. The Sun Cluster softwareframework is implemented as a series of daemons and kernel modules.One of the advantages of the Sun Cluster software environment is that somuch of the framework is in the kernel, where it is fast, alwaysmemory-resident, and reliable. Some of the services provided by theframework are described in the following sections.

Node Fault Monitoring and Cluster Membership

The cluster membership monitor (CMM) is kernel-resident on each nodeand detects major cluster status changes, such as loss of communicationbetween one or more nodes. The CMM relies on the transport kernelmodule to generate heartbeats across the transport medium to othernodes in the cluster. If the heartbeat from any node is not detected withina defined time-out period, it is considered as having failed, and a clusterreconfiguration is initiated to renegotiate cluster membership.

Network Fault Monitoring

Both the public network interfaces and the cluster transport interfaces aremonitored for potential failures.

Public Network Management

The Sun Cluster 3.1 software environment requires the use of IPMP, astandard Solaris OS feature, to control interface failures on a node. TheSun Cluster software adds a layer of monitoring to detect total networkfailure on one node and drive possible failover of applications to anothernode.

Cluster Transport Monitoring

The cluster transport interfaces are monitored on each node. If an activecluster transport interface on any node is determined to be inoperative, allnodes route interconnect traffic to functional transport interfaces. Thefailure is transparent to Sun Cluster software applications.



Disk Path Monitoring

Disk path monitoring (DPM) provides the capability to monitor diskpaths with a new cluster daemon process and a CLI command. DPM isnot supported on nodes that run versions that were released prior to SunCluster 3.1 5/03 software.

DPM improves the overall reliability of failover and switchover bymonitoring the secondary disk-path availability. It is possible to monitordisk paths to a single node or to all nodes in the cluster. The disk pathfailure detection mechanism generates an event through the Cluster EventFramework and allows manual intervention.

Application Traffic Striping

Applications written correctly can use the transport for data transfer. Thisfeature stripes IP traffic sent to the per-node logical IP addresses across allprivate interconnects. Transmission Control Protocol (TCP) traffic isstriped on a per connection granularity. User Datagram Protocol (UDP)traffic is striped on a per packet basis. The cluster framework uses thenetwork device clprivnet0 for these transactions. This network interfaceis visible with ifconfig . No configuration is required as part of the SunCluster 3.1 software framework.

Cluster Reconfiguration Notification Protocol

Cluster Reconfiguration Notification Protocol (CRNP) provides amechanism for applications to register for, and receive, subsequentasynchronous notification of Sun Cluster reconfiguration events. Bothdata services running on the cluster and applications running outside ofthe cluster can register for event notification. Notifications includechanges in cluster membership, resource groups, and resource state. ASun Cluster 3.1 software framework resource type is available to make thenotification daemon highly available.



Cluster Configuration Repository

General cluster configuration information is stored in global configurationfiles collectively referred to as the cluster configuration repository (CCR).The CCR must be kept consistent between all nodes and is a criticalelement that enables each node to be aware of its potential role as adesignated backup system.

Caution – Never attempt to modify any of the CCR-related files. The filescontain generation number information that is critical to the operation ofthe cluster software. The CCR information is automatically modified asthe result of administrative command execution and cluster statuschanges.

The CCR structures contain the following types of information:

● Cluster and node names

● Cluster transport configuration

● The names of registered VERITAS disk groups or Solaris VolumeManager software disksets

● A list of nodes that can master each disk group

● Data service operational parameter values (timeouts)

● Paths to data service callback methods

● Disk ID (DID) device configuration

● Current cluster status

The CCR is accessed when error or recovery situations occur or whenthere has been a general cluster status change, such as a node leaving orjoining the cluster.

Defining Global Storage Service Differences



The Sun Cluster software framework provides global storage services, afeature which greatly distinguishes the Sun Cluster software product. Notonly do these features enable scalable applications to run in the cluster,they also provide a much more flexible environment for failover servicesby freeing applications to run on nodes that are not physically connectedto the data.

It is important to understand the differences and relationships betweenthe following services:

● Global naming (DID devices)

● Global devices

● Global file system

Disk ID Devices (DIDs)

The DID feature provides a unique device name for every disk drive,CD-ROM drive, or tape drive in the cluster. Multiported disks that mighthave different logical names on different nodes (different controllernumbers) are given a cluster-wide unique DID instance number. Differentdisks that may, each on its own node, use the same logical name (forexample, c0t0d0 for each node’s root disk) are each given differentunique DID instance numbers.



Figure 1-7 demonstrates the relationship between normal Solaris OSlogical path names and DID instances.

Figure 1-7 DID Driver Devices

Device files are created for each of the normal eight Solaris OS diskpartitions in both the /dev/did/dsk and /dev/did/rdsk directories; forexample: /dev/did/dsk/d2s3 and /dev/did/rdsk/d2s3 .

It is important to note that DIDs themselves are just a global namingscheme and not a global access scheme.

DIDs are used as components of Solaris Volume Manager softwarevolumes and in choosing cluster quorum devices, as described inModule 3, “Preparing for Installation and Understanding QuorumDevices.”

DIDs are not used as components of VERITAS Volume Manager volumes.

Global Devices

The global devices feature of Sun Cluster 3.1 software providessimultaneous access to all storage devices from all nodes, regardless ofwhere the storage is physically attached.

This includes individual DID disk devices, CD-ROMs and tapes, as wellas VERITAS Volume Manager volumes and Solaris Volume Managersoftware metadevices.

Junction

Node 1 Node 2 Node 3

d2=c2t18d0d2=c5t18d0

d4=c0t0d0d5=c1t6d0

d8191=/dev/rmt/0

c2 c5 c0 c1

t18d0Array Disk

t0d0Boot Disk

t6d0CD-ROM

t0d0Boot Disk

t0d0Boot Disk

c0c0

d1=c0t0d0d3=c0t0d0

rmt/0



Global devices are used most often in the cluster with VERITAS VolumeManager and Solaris Volume Manager software devices. The volumemanagement software is unaware of the global device serviceimplemented on it.

The Sun Cluster 3.1 software framework manages automatic failover ofthe primary node for global device groups. All nodes use the same devicepath, but only the primary node for a particular device actually talksthrough the storage medium to the disk device. All other nodes access thedevice by communicating with the primary node through the clustertransport. In Figure 1-8, all nodes have simultaneous access to the device/dev/vx/rdsk/nfsdg/nfsvol . Node 2 becomes the primary node ifNode 1 fails.

Figure 1-8 Node Access Diagram

Node 1Primary

for nfsdgNode 2 Node 3 Node 4

nfsdg/nfsvol



Device Files for Global Devices

The Sun Cluster 3.1 software maintains a special file system on each node,completely dedicated to storing the device files for global devices. Thisfile system has the mount point /global/.devices/node@ nodeID , wherenodeID is an integer representing a node in the cluster. This requires adedicated disk partition on each node, usually on the boot disk.

All of the /global/.devices file systems—one for each node—arevisible from each node. In other words, they are examples of global filesystems. The global file system feature is described more on the followingpages.

# df -k/dev/vx/dsk/rootdisk_25vol

95702 5026 81106 6% /global/.devices/node@2/dev/vx/dsk/rootdisk_15vol

95702 5026 81106 6% /global/.devices/node@1

The device names under the /global/.devices/node@ nodeID arenacould be used directly, but, because they are clumsy, the Sun Cluster 3.1environment provides symbolic links into this namespace.

For VERITAS Volume Manager and Solaris Volume Manager software, theSun Cluster software links the standard device access directories into theglobal namespace:

proto192:/dev/vx# ls -l /dev/vx/dsk/nfsdglrwxrwxrwx 1 root root 40 Nov 25 03:57

/dev/vx/dsk/nfsdg ->/global/.devices/node@1/dev/vx/dsk/nfsdg/

For individual DID devices, the standard directories /dev/did/dsk ,/dev/did/rdsk , and /dev/did/rmt are not global access paths. Instead,Sun Cluster software creates alternate path names under the/dev/global directory that link into the global device space:

proto192:/dev/md/nfsds# ls -l /dev/globallrwxrwxrwx 1 root other 34 Nov 6 13:05

/dev/global -> /global/.devices/node@1/dev/global/proto192:/dev/md/nfsds# ls -l /dev/global/dsk/d3s0lrwxrwxrwx 1 root root 39 Nov 4 17:43

/dev/global/dsk/d3s0 -> ../../../devices/pseudo/did@0:3,3s0,blkproto192:/dev/md/nfsds# ls -l /dev/global/rmt/1lrwxrwxrwx 1 root root 39 Nov 4 17:43

/dev/global/rmt/1 -> ../../../devices/pseudo/did@8191,1,tp



Note – You do have access from one node, through the /dev/globaldevice paths, to the boot disks of other nodes. It is not necessarilyadvisable to take advantage of this feature.

Global File Systems

The global file system feature makes file systems simultaneously availableon all nodes, regardless of their physical location.

The global file system capability is independent of the structure of theactual file system layout on disk. The UNIX® File System (ufs ), VERITASFile System (VxFS) and High Sierra File System (hsfs ) are supported.

The Sun Cluster software makes a file system global with a global mountoption. This is normally in the /etc/vfstab file but can be put on thecommand line of a standard mount command:

# mount -o global,logging /dev/vx/dsk/nfs-dg/vol-01 /global/nfs

The equivalent mount entry in the /etc/vfstab file is:

/dev/vx/dsk/nfs-dg/vol-01 /dev/vx/rdsk/nfs-dg/vol-01 \/global/nfs ufs 2 yes global ,logging

The global file system works on the same principle as the global devicefeature. That is, only one node at a time is the primary and actually talksto the underlying file system. All other nodes use normal file semanticsbut actually communicate with the primary over the cluster transport. Theprimary for a global file system built on a global device is always thesame as the primary for the global device.

The global file system is also known by the following names:

● Cluster file system

● Proxy file system



Local Failover File Systems in the Cluster

Sun Cluster 3.1 software also has support in the cluster for local failover filesystem access. That is, file systems that are not globally, simultaneously,available from every node, but available only one node at a time, on anode which is running a service and has a physical connection to thestorage in question.

Local Failover File System Access Is Not for Scalable Services

Local failover file system access is appropriate for failover services thatwill run only on the nodes physically connected to storage devices.

Local failover file system access is not usable for scalable services thatrequire global file system access.

Local failover file system access, when used appropriately, can have aperformance benefit over global file system access. There is overhead inthe global file system infrastructure of maintaining replicated stateinformation on multiple nodes simultaneously.

Exercise: Guided Tour of the Training Lab


Exercise: Guided Tour of the Training Lab

At the end of this exercise you should be able to identify Sun Clusterhardware components located in the training lab.

Preparation

No special preparation is needed for this lab.

Task

You participate in a guided tour of the training lab. While participating inthe guided tour, you identify the Sun Cluster hardware componentsincluding the cluster nodes, terminal concentrator, and administrativeworkstation.

Exercise Summary


Exercise Summary

?!

Discussion – Take a few minutes to discuss what experiences, issues, ordiscoveries you had during the lab exercises.

● Experiences

● Interpretations

● Conclusions

● Applications


Module 2

Exploring Node Console Connectivity andthe Cluster Console Software

Objectives


● Describe the difference in node access between domain-based clusternodes and traditional standalone cluster nodes

● Describe the different access methods for accessing a console in atraditional node

● Connect to the console for a domain-based cluster node

● Set up the administrative console environment

● Configure the Sun Cluster console software on the administrationworkstation

Relevance


Relevance

?!


● What is the trade-off between convenience and security in differentconsole access methods?

● How do you reach the node console on domain-based clusters?

● What benefits does using a terminal concentrator give you?

● Is installation of the administration workstation software essentialfor proper cluster operation?


Exploring Node Console Connectivity and the Cluster Console Software 2-3Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1



● Sun Cluster 3.1 4/04 Software Collection for Solaris OS (x86 PlatformEdition)

● Sun Cluster 3.1 4/04 Software Collection for Solaris OS (SPARCPlatform Edition)

Accessing the Cluster Node Consoles



This section describes different methods for achieving access to the SunCluster 3.1 node consoles. It is expected that a Sun Cluster 3.1environment administrator:

● Does not require node console access for most operations describedduring the duration of the course. Most cluster operations requireonly that you be logged in on a cluster node as root or as a userwith cluster authorizations in the Role-Based Access Control (RBAC)subsystem. It is acceptable to have direct telnet , rlogin , or sshaccess to the node.

● Must have console node access for certain emergency andinformational purposes. If a node is failing to boot, the clusteradministrator will have to access the node console to figure out why.The cluster administrator might like to observe boot messages evenin normal, functioning clusters.

The following pages describe the essential difference between consoleconnectivity on “traditional” nodes that are not domain based andconsole connectivity on the domain-based nodes (Sun Enterprise 10000servers, Sun Fire 3800–6800 servers, and Sun Fire 15K/12K servers).

Accessing Serial Port Consoles on Traditional Nodes

Traditional Sun Cluster 3.1 nodes (which are not those listed on page 2-3)usually use serial port ttyA as the console. Using a graphics console witha Sun keyboard and monitor is supported, as long as it is supported onthe base server platform.

The rule for console connectivity is simple. You can connect to the nodettyA interfaces any way you like, as long as whatever device you haveconnected directly to them does not spuriously issue BREAKsignals on theserial line.

BREAKsignals on the serial port will bring a cluster node to the OKprompt,killing all cluster operations on that node.



It is possible to disable node recognition of a BREAKsignal, either by ahardware keyswitch position (on some nodes), a software keyswitchposition (on the Sun Fire 3800–6800 servers and Sun Fire 15K/12Kservers), or a file setting (on all nodes).

For those servers with a hardware keyswitch, turn the key to the thirdposition to power the server on and disable the BREAKsignal.

For those servers with a software keyswitch, issue the setkeyswitchcommand with the secure option to power the server on and disable theBREAK signal.

For all servers, while running Solaris OS, uncomment the lineKEYBOARD_ABORT=alternate in /etc/default/kbd to disable receipt ofthe normal BREAKsignal through the serial port. This setting takes effecton boot, or by running the kbd -i command as root .

The Alternate Break signal is defined by the serial driver (zs ), and iscurrently carriage return, tilde (~), and control-B: CR ~ CTRL-B. When theAlternate Break sequence is in effect, only serial console devices areaffected.

Access Serial Port Node Consoles Using a TerminalConcentrator

One of the popular ways of accessing traditional node consoles is througha TC, a device which listens for connections on the network and passesthrough traffic (unencapsulating and reencapsulating all the TCP/IPheaders) to the various serial ports.

A TC is also known as a Network Terminal Server (NTS).



Figure 2-1 shows a terminal concentrator network and serial portinterfaces. The node public network interfaces are not shown. While youcan attach the TC to the public net, most security-consciousadministrators would attach it to a private management network.

Figure 2-1 TC Network and Serial Port Interfaces

Most TCs enable you to administer TCP pass-through ports on the TC.When you connect with telnet to the TC’s IP address and pass throughport, the TC transfers traffic directly to the appropriate serial port(perhaps with an additional password challenge).

Sun Terminal Concentrator (Sun NTS)

Sun rebrands a terminal concentrator, the Xylogics™ NTS. Sun guaranteesthat this TC does not spuriously issue BREAKsignals when it is poweredon and off, for example.

The Sun TC supports telnet and rlogin access. The TCP ports it uses aspass-through ports for telnet access are 5002 for serial port 2, 5003 forserial port 3, and so forth.

Administrative Console


Network

Cluster ConsoleSoftware

Setup port

Setup Device

Serial port A

Node 1 Node 2

NetworkInterface Serial Ports

1 2 3 4 5 6 7 8



When you use telnet to connect to the Sun TC from an administrativeworkstation using, for example, the following command, the Sun TCpasses you through directly to serial port 2, connected to your first clusternode.

# telnet tc-ipname 5002

The Sun TC supports an extra level of password challenge, a per-portpassword which you would have to enter before going through to theserial port.

Full instructions on installing and configuring the Sun TC are inAppendix A, “Terminal Concentrator.”

Other Terminal Concentrators

You can choose any type of TC as long as it does not issue BREAKsignalson the serial ports when it is powered on, powered off, reset, or any othertime that might be considered spurious.

If your TC cannot meet that requirement, you could still disablerecognition of the BREAKsignal or enable an alternate abort signal for yournode.

Some terminal concentrators support Secure Shell (the Sun NTS does notsupport Secure Shell). This might influence your choice, if you areconcerned about passing TC traffic in the clear on the network.



Alternatives to a Terminal Concentrator

Some possible alternatives to the TC include:

● Use a Sun graphics console. This requires physical access to the nodefor console work. This is most secure, but least convenient.

● Use dumb terminals for each node console. If these are in a securephysical environment, this would certainly be your most secure butleast convenient method.

● Use a workstation that has two serial ports as a “tip launchpad,”especially for a cluster with only two nodes

You could attach a workstation on the network exactly as you wouldplace a TC, and attach its serial ports to the node consoles. Youshould then add lines to the /etc/remote file of the Solaris OSworkstation as follows:

node1:\:dv=/dev/term/a:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:

node2:\:dv=/dev/term/b:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:

This would allow you to access node consoles by accessing thelaunchpad workstation and manually typing tip node1 or tipnode2 .

One advantage of using a Solaris OS workstation instead of a TC isthat it is easier to tighten the security therein. You could easily, forexample, disable telnet and rlogin access and require thatadministrators access the tip launchpad by means of Secure Shell.

● Use Remote System Control (RSC) on servers that support it (such asthe Sun Enterprise 250 server, Sun Fire V280 server, and Sun FireV480 server)

Servers with RSC have a separate, dedicated Ethernet interface andserial port for remote system control. The RSC software has acommand line interface which gives you console access and access tothe server’s hardware. Refer to the Sun Remote System Control(RSC) 2.2 User's Guide for additional information.

If you use RSC on cluster nodes you can have console connectivitydirectly through TCP/IP to the RSC hardware on the node. Adedicated IP address, which is different from any Solaris OS IPaddress, is used.

Accessing the Node Consoles on Domain-based Servers


Accessing the Node Consoles on Domain-based Servers

The domain-based servers have no serial port console access for thedomains. Instead, you access the main system support processor (SSP, forSun Enterprise 10000 servers) or system controller (SC, for Sun Fireservers) and use a virtual console protocol to access the console for eachnode.

Sun Enterprise™ 10000 Servers

The SSP and its backup sit physically outside the server frame and run theSolaris OS and SSP software.

To access the node console, you will always just have to log in manuallyto the SSP as the user named ssp and manually invoke the netconcommand for the appropriate domain.

Sun Fire™ 15K/12K Servers

The SC and its backup sit physically inside the server frame but act muchlike the Sun Enterprise 10000 system SSP. They run the Solaris OS withSystem Management Services (SMS) software.

To access the node console, you have to log in manually to the SC as thedomain administrator for the node in question (this could be a differentuser for each domain) and manually invoke the console command forthe appropriate domain.

Sun Fire 3800–6800 Servers

The SC and its backup sit physically inside the server frame. They run aunique, firmware-based operating environment known as VxWorks. TheSCs run an application, based on Java technology, known as ScApp that isdedicated to managing the server.

The SC runs a TC-like emulation where it listens on TCP ports 5001, 5002,and so forth. Accessing the SC with one of these ports passes you through(perhaps with an extra password challenge) to a domain console ordomain shell.

Describing Sun Cluster Console Software for Administration Workstation


Describing Sun Cluster Console Software forAdministration Workstation

The Sun Cluster 3.1 software includes a small amount of software to beinstalled on an administrative workstation. This software can be installedon any Solaris OS workstation running a minimum of Solaris OSversion 2.6.

The console software is a convenience which displays new windows foryou to the cluster nodes and issues telnet or rlogin commands toconnect you either directly to the nodes or to the node console accesspoint.

The console software includes a common window feature that enables youissue keystrokes simultaneously to all the nodes in the cluster (or even toall the nodes in multiple clusters).

The Sun Cluster 3.1 software environment itself has no dependency on theadministration workstation’s console software. It is just an administrativeconvenience.

Console Software Installation

The administrative console software is contained in a single package:SUNWccon. The SUNWcconpackage is installed manually from the SunCluster 3.1 software distribution CD-ROM.

Cluster Console Window Variations

The three variations of the cluster console tool each use a different methodto access the cluster hosts. They all look and behave the same way.

● Cluster console (cconsole )

The cconsole program accesses the node consoles through the TCor other remote console access method. Depending on your accessmethod, you might be directly connected to the console node, or youmight have to respond to additional password challenges, or youmight have to log in to a SSP or SC and manually connect to thenode console.



● Cluster console (crlogin )

The crlogin program accesses the nodes directly using the rlogincommand.

● Cluster console (ctelnet )

The ctelnet program accesses the nodes directly using the telnetcommand.

Note that it is possible in certain scenarios that some tools might beunusable, but you might want to install this software to use other tools.

For example, if you are using dumb terminals to attach to the nodeconsoles, you will not be able to use the cconsole variation, but youcould use the other two after the nodes are booted.

Alternatively, you could have, for example, a Sun Fire 6800 server wheretelnet and rlogin are disabled in the domains, and only ssh is workingas a remote login method. You might be able to use only the cconsolevariation, but not the other two variations.



Cluster Console Tools Look and Feel

All the tools have the same general look and feel. The tool automaticallyshows one new window for each node, and a small common keystrokewindow (Figure 2-2). You can type in each individual window as desired.Input directed to the common window is automatically replicated to allthe other windows.

Figure 2-2 Cluster Console Windows



Start the Tools Manually

As shown, you can use the tools manually to connect to a single clusternode or to the entire cluster.

# /opt/SUNWcluster/bin/cconsole node1 &# /opt/SUNWcluster/bin/ctelnet my-cluster &# /opt/SUNWcluster/bin/crlogin node3 &

Cluster Console Host Windows

Each node in the cluster has a host window. You can enter commands ineach host window separately.

Set the TERMenvironment variable to vt100 or dtterm for best operation.

Cluster Console Common Window

The common window, shown in Figure 2-3, allows you to entercommands to all host system windows at the same time. All of thewindows are tied together, so when you move the common window, thehost windows follow. The Options menu allows you to ungroup thewindows, move them into a new arrangement, and group them again.

Figure 2-3 Cluster Console Common Window



Cluster Control Panel

As shown in Figure 2-4, the Cluster Control Panel provides centralizedaccess to three variations of the cluster console tool.

Figure 2-4 Cluster Control Panel

Starting the Cluster Control Panel

To start the Cluster Control Panel, type the following command:

# /opt/SUNWcluster/bin/ccp [ clustername ] &

Configuring Cluster Console Tools



All of the necessary information needed for the cluster administrationtools that run on the administrative console is configured in two files. Thefiles are:

● The /etc/clusters file

● The /etc/serialports file (for cconsole variation only)

When you install the Sun Cluster console software on the administrativeconsole, you must manually create the clusters and serialports filesand populate them with the necessary information.

Configuring the /etc/clusters File

The /etc/clusters file contains the name of a cluster followed by thenames of the nodes that are part of the cluster.

The following is a typical entry in the /etc/clusters file:

sc-cluster sc-node1 sc-node2

The single-line entry defines a cluster named sc-cluster , which has twonodes named sc-node1 and sc-node2 .

Note – The cluster name is purely arbitrary. The Sun Cluster 3.1 softwareitself has you define a cluster name, which likely agrees with this one,although nothing breaks if it does not agree.

You can define many different clusters in a single /etc/clusters file, soyou can administer several clusters from a single administrative console.



Configuring the /etc/serialports File

The /etc/serialports file defines the remote path to the console foreach node defined in the /etc/clusters file. You must enter the paths toall nodes in all of your clusters in this file. This file uses three columns: thenode name listed in /etc/inet/hosts , the host name of the workstationor device providing the connection as listed in /etc/inet/hosts , and theport for telnet to use for the connection to or through this device.

The following are typical entries in the /etc/serialports file whenutilizing a terminal concentrator to pass through to the physical serialconnection (ttyA) on each cluster node:

sc-node1 sc-tc 5002sc-node2 sc-tc 5003

For the Sun Enterprise 10000 server, the columns are the name of thedomain, the name of the SSP workstation, and port 23. When thecconsole cluster command is executed, you are connected to port 23on the SSP. This opens up a telnet session for each domain listed. Theactual login to the SSP is done manually in each window. Once logged in,a netcon session is manually started for the domain in each window.

sc-10knode1 sc10k-ssp 23sc-10knode2 sc10k-ssp 23

For the Sun Fire 15K/12K server, the columns are the name of the domain,the name of the main system controller, and port 23. When the cconsolecluster command is executed, you are connected to port 23 on the mainsystem controller. This opens up a telnet session for each domain listed.The actual login to the main system controller is done manually in eachwindow using the login of the domain administrator for that domain.Once logged in, a console session is manually started for that domain.

sc-15knode1 sf15k-mainsc 23sc-15knode2 sf15k-mainsc 23



For a system using RSC access instead of a TC, the columns display thename of the node, the name of the RSC IP, and port 23. When thecconsole cluster command is executed, you are connected to port 23on the RSC port for that node. This opens up a telnet session for eachnode. The actual login to the node console is done manually in eachwindow.

e250node1 RSCIPnode1 23e250node2 RSCIPnode2 23

If you are using a tip launchpad, the columns display the name of thenode, the name of the launchpad, and port 23. The actual login to thelaunchpad is done manually in each window. The actual connection toeach node console is made by executing the tip command and specifyingthe serial port used for that node in each window.

node1 sc-tip-ws 23node2 sc-tip-ws 23

For the Sun Fire 3800–6800 servers, the columns display the domainname, the Sun Fire server main system controller name, and a TCP portnumber similar to those used with a terminal concentrator. Port 5000connects you to the platform shell, ports 5001, 5002, 5003, or 5004 connectyou to the domain shell or domain console for domains A, B, C, or D,respectively. Connection to the domain shell or console is dependent uponthe current state of the domain.

sf1_node1 sf1_mainsc 5001sf1_node2 sf1_mainsc 5002



Configuring Multiple Terminal Concentrators

If you have widely separated nodes or many nodes in your cluster, youmight need more than one TC to manage a single cluster. The followingexamples demonstrate how the /etc/clusters and /etc/serialportsfiles might appear for an eight-node cluster.

The following entry in the /etc/clusters file represents the nodes in aneight-node cluster:

sc-cluster sc-node1 sc-node2 sc-node3 sc-node4 \sc-node5 sc-node6 sc-node7 sc-node8

The single-line entry defines a cluster named sc-cluster , which haseight nodes.

The following entries in the /etc/serialports file define the TC pathsto each node in an eight-node cluster:

sc-node1 sc-tc1 5002sc-node2 sc-tc1 5003sc-node3 sc-tc1 5004sc-node4 sc-tc1 5005sc-node5 sc-tc2 5002sc-node6 sc-tc2 5003sc-node7 sc-tc2 5004sc-node8 sc-tc2 5005

Each cluster host has a line that describes the name of the host, the nameof the TC, and the TC port to which the host is attached.

Exercise: Configuring the Administrative Console



In this exercise, you complete the following tasks:

● Install and configure the Sun Cluster console software on anadministrative console

● Configure the Sun Cluster administrative console environment forcorrect Sun Cluster console software operation

● Start and use the basic features of the Cluster Control Panel and theCluster Console

Preparation

This exercise assumes that the Solaris 9 OS software is already installed onall of the cluster systems. Perform the following steps to prepare forthe lab:

1. Ask your instructor for the name assigned to your cluster.

Cluster name: _______________________________

2. Record the information in Table 2-1 about your assigned clusterbefore proceeding with this exercise.

3. Ask your instructor for the location of the Sun Cluster software.

Software location: _______________________________

Table 2-1 Cluster Names and Addresses

System Name IP Address

Administrativeconsole

TC

Node 1

Node 2

Node 3 (if any)



Task 1 – Updating Host Name Resolution

Even though your site might use Network Information Service (NIS) orDNS to resolve host names, it can be beneficial to resolve the nameslocally on the administrative console and cluster hosts. This can bevaluable in the case of naming service failures. The cconsole programdoes not start unless it can first resolve the host names in the/etc/clusters file.

1. If necessary, edit the /etc/inet/hosts file on your administrativeconsole, and add the IP addresses and names of the TC and the hostsystems in your cluster.

2. Verify that the /etc/nsswitch.conf file entry for hosts hascluster listed first.

hosts: cluster files nis

Task 2 – Installing the Cluster Console Software

Perform the following steps to install the cluster console software:

1. Log in to your administrative console as user root .

2. Move to the Sun Cluster 3.1 packages directory.

Note – Either load the Sun Cluster 3.1 CD or move to the locationprovided by your instructor.

3. Verify that you are in the correct location.

# ls SUNWcconSUNWccon

4. Install the cluster console software package.

# pkgadd -d . SUNWccon



Task 3 – Verifying the Administrative ConsoleEnvironment

Perform the following steps to verify the administrative console:

1. Verify that the following search paths and variables are present inthe /.profile file:

PATH=$PATH:/opt/SUNWcluster/binMANPATH=$MANPATH:/opt/SUNWcluster/manEDITOR=/usr/bin/viexport PATH MANPATH EDITOR

Note – Create the .profile file in the / directory if necessary, and addthe changes.

2. Execute the .profile file to verify changes that have been made.

# . /.profile

Note – You can also log out and log in again to set the new variables.

Task 4 – Configuring the /etc/clusters File

The /etc/clusters file has a single line entry for each cluster you intendto monitor. The entries are in the form:

clustername host1name host2name host3name host4name

Sample /etc/clusters File

sc-cluster pnode1 pnode2 pnode3

Perform the following step to configure the /etc/clusters file:

Edit the /etc/clusters file, and add a line using the cluster andnode names assigned to your system.

Note – Your administrative workstation must be able to resolve the hostnames.



Task 5 – Configuring the /etc/serialports File

The /etc/serialports file has an entry for each cluster host describingthe connection path. This example is for a three node cluster using aSun NTS.

hostname tcname tcport

Sample /etc/serialports File

pnode1 cluster-tc 5002pnode2 cluster-tc 5003pnode3 cluster-tc 5004

Perform the following step to configure the /etc/serialports file:

Edit the /etc/serialports file, and add lines using the node andTC names assigned to your system.

Task 6 – Starting the cconsole Tool

This section provides a good functional verification of the TC in additionto the environment configuration. Perform the following steps to start thecconsole tool:

1. Make sure power is on for the TC and all of the cluster hosts.

2. Start the cconsole tool on the administrative console.

# cconsole clustername &

Note – Substitute the name of your cluster for clustername .

3. Place the cursor in the cconsole Common window, and press theReturn key several times. You should see a response on all of thecluster host windows. If not, ask your instructor for assistance.

4. If the cluster host systems are not booted, boot them now.

ok boot

5. After all cluster host systems have completed their boot, log in asuser root .

6. Practice using the Common window Group Term Windows featureunder the Options menu. You can ungroup the cconsole windows,rearrange them, and then group them together again.



Task 7 – Using the ccp Control Panel

The ccp control panel can be useful if you must use the console toolvariations crlogin and ctelnet . Perform the following steps to use theccp control panel:

1. Start the ccp tool (# ccp clustername &).

2. Practice using the crlogin and ctelnet console tool variations.

3. Quit the crlogin , ctelnet , and ccp tools.

Exercise Summary


Exercise Summary

?!


● Experiences

● Interpretations

● Conclusions

● Applications


Module 3

Preparing for Installation and UnderstandingQuorum Devices

Objectives


● List the Sun Cluster software boot disk requirements

● Identify typical cluster storage topologies

● Identify the quorum devices needed for selected cluster topologies

● Describe persistent quorum reservations and cluster amnesia

● Configure a supported cluster interconnect system

● Physically configure a public network group

Relevance


Relevance

?!


● Is cluster planning required even before installation of the Solaris OSon a node?

● Do certain cluster topologies enforce which applications are going torun on which nodes, or do they just suggest this relationship?

● Why is a quorum device absolutely required in a two-node cluster?

● What is meant by a cluster amnesia problem?


Preparing for Installation and Understanding Quorum Devices 3-3Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1



● Sun Cluster 3.1 4/04 Software Collection for Solaris OperatingSystem (x86 Platform Edition)

● Sun Cluster 3.1 4/04 Software Collection for Solaris OperatingSystem (SPARC Platform Edition)



● Sun Cluster 3.x Hardware Collection for Solaris Operating System(SPARC Platform Edition)

● Sun System Handbook at[http://sunsolve.sun.com/handbook_pub ]

Configuring Cluster Servers



The servers you use in a Sun Cluster 3.1 hardware configuration mustconform to a number of general software and hardware requirements toqualify for support. The requirements include both hardware andsoftware in the following areas:

● Boot device restrictions

● Server hardware restrictions

Boot Device Restrictions

With the Sun Cluster 3.1 software release, there are several restrictions onboot devices including:

● You cannot use a shared storage device as a boot device. If a storagedevice is connected to more than one host, it is shared.

● The Solaris 8 OS (Update 7 or greater) and Solaris 9 OS aresupported with the following restrictions:

● The same version of the Solaris OS, including the maintenanceupdate, must be installed on all nodes in the cluster.

● Alternate Pathing is not supported.

● VERITAS Dynamic Multipathing (DMP) is not supported.

See Module 6, “Using VERITAS Volume Manager for VolumeManagement,” for more details about this restriction.

The supported versions of VERITAS Volume Manager musthave the DMP device drivers enabled, but Sun Cluster 3.1software still does not support an actual multipathing topologyunder control of Volume Manager DMP.

● The boot disk partitions have several requirements:

● Sun Cluster 3.1 software requires a minimum swap partition of750 Mbytes.

● There must be a minimum 100 Mbytes per globaldevices filesystem for the training environment. It is recommended that thisfile system be 512 Mbytes for a production environment.

● Solaris Volume Manager software requires a 20 Mbyte partitionfor its meta state databases.

● VERITAS Volume Manager software requires two unusedpartitions for encapsulation of the boot disk.



Note – The /globaldevices file system is modified during the SunCluster 3.1 software installation. It is automatically renamed to/global/.devices/node@ nodeid , where nodeid represents thenumber that is assigned to a node when it becomes a cluster member. Theoriginal /globaldevices mount point is removed. The/globaldevices file system must have ample space and ample inodecapacity for creating both block special devices and character specialdevices. This is especially important if a large number of disks are in thecluster. A file system size of 512 Mbytes should suffice for most clusterconfigurations.

Boot Disk JumpStart™ Software Profile

If you decide to configure your cluster servers using the JumpStart™software, the following JumpStart software profile represents a startingpoint for boot disk configuration:

install_type initial_installsystem_type standalonepartitioning explicitcluster SUNWCXallfilesys c0t0d0s0 free / loggingfilesys c0t0d0s1 1024 swapfilesys c0t0d0s5 512 /globaldevicesfilesys c0t0d0s7 20

Server Hardware Restrictions

To be a supported Sun Cluster configuration, the configuration ofhardware components in a cluster must first be supported by thecorresponding base product groups for each hardware component. Forexample, for a Sun Cluster configuration composed of two Sun Fire V880servers connected to two Sun StorEdge 3510 FC arrays to be supported,the server and storage base product groups must support connecting aSun StorEdge 3510 FC array to a Sun Fire V880 server in a standaloneconfiguration.



All cluster servers must meet the following minimum hardwarerequirements:

● Each server must have a minimum of 512 Mbytes of memory.

● Each server must have a minimum of 750 Mbytes of swap space. Ifyou are running applications on the cluster nodes that require swapspace, you should allocate at least 512 Mbytes over that requirementfor the cluster.

● Servers in a cluster can be heterogeneous with certain restrictions onmixing certain types of host adapters in the same cluster. Forexample, PCI and Sbus adapters cannot be mixed in the same clusterif attached to Small Computer System Interface (SCSI) storage.

Configuring Cluster Storage Connections



While previous versions of the Sun Cluster software had strict rulesregarding how many nodes were supported in various disk topologies,the only rules in the Sun Cluster 3.1 software regarding the data storagefor the cluster are:

● Sun Cluster software supports a maximum of eight nodes in acluster, regardless of the storage configurations that areimplemented. Some storage configurations have restrictions on thetotal number of nodes supported.

● A shared storage device can connect to as many nodes as the storagedevice supports.

● Shared storage devices do not need to connect to all nodes of thecluster. However, these storage devices must connect to at least twonodes.

Cluster Topologies

Cluster topologies describe typical ways in which cluster nodes can beconnected to data storage devices. While Sun Cluster does not require youto configure a cluster by using specific topologies, the followingtopologies are described to provide the vocabulary to discuss a cluster’sconnection scheme. The following are some typical topologies:

● Clustered pairs topology

● Pair+N topology

● N+1 topology

● Multi-ported (more than two node) N*N scalable topology



Clustered Pairs Topology

As shown in Figure 3-1, a clustered pairs topology is two or more pairs ofnodes with each pair physically connected to some storage. Because of theglobal device and global file system infrastructure, this does not restrictwhere applications can fail over to and run. Still, it is likely you willconfigure applications to fail over in pairs of nodes attached to the samestorage.

Figure 3-1 Clustered Pairs Topology Configuration

The features of clustered pairs configurations are:

● Nodes are configured in pairs. You could have any even number ofnodes from two to eight.

● Each pair of nodes shares storage. Storage is connected to both nodesin the pair.

● All nodes are part of the same cluster.

You are likely to design applications that run on the same pair ofnodes physically connected to the data storage for that application,but you are not restricted to this design.

● Because each pair has its own storage, no one node must have asignificantly higher storage capacity than the others.

● The cost of the cluster interconnect is spread across all the nodes.

● This configuration is well suited for failover data services.

Junction

Junction

Node 1 Node 2 Node 3 Node 4

Storage StorageStorage Storage



Pair+N Topology

As shown in Figure 3-2, the Pair+N topology includes a pair of nodesdirectly connected to shared storage and nodes that must use the clusterinterconnect to access shared storage because they have no directconnection themselves.

Figure 3-2 Pair+N Topology

The features of the Pair+N configurations are:

● All shared storage is connected to a single pair.

● Additional cluster nodes support scalable data services or failoverdata services with the global device and file system infrastructure.

● A maximum of eight nodes is supported.

● There are common redundant interconnects between all nodes.

● The Pair+N configuration is well suited for scalable data services.

The limitations of a Pair+N configuration is that there can be heavy datatraffic on the cluster interconnects. You can increase bandwidth by addingmore cluster transports.

Junction

Junction


Storage Storage



N+1 Topology

The N+1 topology, shown in Figure 3-3, enables one system to act as thestorage backup for every other system in the cluster. All of the secondarypaths to the storage devices are connected to the redundant or secondarysystem, which can be running a normal workload of its own.

Figure 3-3 N+1 Topology Configuration

The features of the N+1 configurations are:

● The secondary node is the only node in the configuration that isphysically connected to all the multihost storage.

● The backup node can take over without any performancedegradation.

● The backup node is more cost effective because it does not requireadditional data storage.

● This configuration is best suited for failover data services.

A limitation of a N+1 configuration is that if there is more than oneprimary node failure, you can overload the secondary node.

Junction

Junction

Node 1Primary

Node 2Primary

Node 3Primary

Node 4Secondary

StorageStorage Storage



Scalable Storage Topology

In a scalable, or N*N topology, shown in Figure 3-4, more than two nodescan be physically connected to the same storage.

Figure 3-4 Scalable Storage Topology

This configuration is required for running the ORACLE ParallelServer/Real Application Clusters (OPS/RAC) across more than twonodes.

For ordinary, cluster-unaware applications, each particular disk group ordiskset in the shared storage will still only support physical traffic fromone node at a time. Still, having more than two nodes physicallyconnected to the storage adds flexibility and reliability to the clusterconfiguration.

Junction

Junction


StorageStorage

Logical diagrams, physical connections, and number of units are dependenton storage interconnect used.



Non-Storage Topology

In clusters with more than two nodes, it is not required to have any sharedstorage at all. This could be suitable for an application that is purelycompute-based and requires no data storage. Two-node clusters requireshared storage because they require a quorum device.

Single-Node Cluster Topology

In this configuration, one node or domain forms the entire cluster. Thisconfiguration allows for a single node to run as a functioning cluster. Itoffers users the benefits of having application management functionalityand application restart functionality. The cluster starts and is fullyfunctional with just one node.

Single-node clusters are ideal for users learning how to manage a cluster,to observe cluster behavior (possibly for agent development purposes), orto begin a cluster with the intention of adding nodes to it as time goes on.

The limitation of a single-node cluster configuration is that true failover isimpossible because there is only one node.

Clustered Pair Topology for x86

Sun Cluster 3.1 version 4/04 software supports cluster nodes based on thex86 architecture. Supported configurations are constantly being approved.Check with your service provider or http://www.sun.com/clusters forthe latest supported x86 configurations.

A cluster that is composed of x86 based systems supports a maximum oftwo nodes in the cluster. Shared storage devices must connect to bothnodes. In this configuration, failover occurs only between the pair.However, both nodes are connected by the cluster interconnect andoperate under Sun Cluster software control.



You might use this topology to run a failover or scalable application onthe pair. Figure 3-5 illustrates the supported clustered pair topology.

Figure 3-5 Clustered Pair Topology for x86

Node 2Node 1

Public

Storage Storage

Private

Describing Quorum Votes and Quorum Devices



The cluster membership subsystem of the Sun Cluster 3.1 softwareframework operates on a “voting system.”

● Each node is assigned exactly one vote.

● Certain disks can be identified as “quorum devices” and areassigned votes.

● There must be majority (more than 50 percent of all possible votespresent) to form a cluster or remain in the cluster.

Why Have Quorum Voting at All?

Given the rules present in the bullet items listed in the previous section, itis clear by looking at a simple two-node cluster why you would needextra quorum disk votes. If a two-node cluster had only “node votes,”then you would need to have both nodes booted to run the cluster. Thiswould defeat one of the major goals of the cluster, which is to be able tosurvive node failure.

But why have quorum voting at all? If there were no quorum rules, youcould happily run as many nodes in the cluster as were able to boot at anypoint in time. However, the quorum vote and quorum devices solve twomajor problems:

● Failure fencing

● Amnesia prevention

These are two distinct problems and it is actually quite clever that they aresolved by the same quorum mechanism in the Sun Cluster 3.x software.Other vendors’ cluster implementations have two distinct mechanisms forsolving these problems, making cluster management more complicated.



Failure Fencing

As shown in Figure 3-6, if interconnect communication between nodesceases, either because of a complete interconnect failure or a nodecrashing, each node must assume the other is still functional. This is calledsplit-brain operation. Two separate clusters cannot be allowed to existbecause of the potential for data corruption. Each node tries to establish acluster by gaining another quorum vote. Both nodes attempt to reservethe designated quorum device. The first node to reserve the quorum diskestablishes a majority and remains as a cluster member. The node thatfails the race to reserve the quorum device aborts the Sun Cluster softwarebecause it does not have a majority of votes.

Figure 3-6 Failure Fencing

Amnesia Prevention

A “cluster amnesia” scenario involves one or more nodes being able toform the cluster (boot first in the cluster) with a “stale” copy of the clusterconfiguration. Imagine the following scenario:

1. In a two node cluster (Node 1 and Node 2), Node 2 is halted formaintenance or crashes

2. Cluster configuration changes are made on Node 1

3. Node 1 is shut down

4. You try to boot Node 2 to form a new cluster

There is more information about how quorum votes and quorum devicesprevent amnesia in ‘‘Preventing Cluster Amnesia With PersistentReservations’’ on page 3-21.

CCR Database

Node 1 (1)

Quorumdevice =

CCR Database

Node 2 (1)

Quorumdevice =

Storage array

QD(1)

Interconnect



Quorum Device Rules

The general rules for quorum devices are:

● A quorum device must be available to both nodes in a two-nodecluster.

● Quorum device information is maintained globally in the ClusterConfiguration Repository (CCR) database.

● A quorum device should contain user data.

● The maximum and optimal number of votes contributed by quorumdevices should be the number of node votes minus one (N-1).

If the number of quorum devices equals or exceeds the number ofnodes, the cluster cannot come up if too many quorum devices fail,even if all nodes are available. Clearly this is unacceptable.

● Quorum devices are not required in clusters with greater than twonodes, but they are recommended for higher cluster availability.

● Quorum devices are manually configured after the Sun Clustersoftware installation is complete.

● Quorum devices are configured using DID devices

Quorum Mathematics and Consequences

When the cluster is running, it always is aware of the following:

● The total possible quorum votes (number of nodes plus the number ofdisk quorum votes defined in the cluster)

● The total present quorum votes (number of nodes booted in thecluster plus the number of disk quorum votes physically accessibleby those nodes)

● The total needed quorum votes equal greater than 50 percent of thepossible votes

The consequences are quite simple:

● A node that cannot find the needed number of votes at boot timefreezes waiting for other nodes to join to up the vote count.

● A node that is booted in the cluster but can no longer find the needednumber of votes kernel panics.



Two-Node Cluster Quorum Devices

As shown in Figure 3-7, a two-node cluster requires a single quorum disk.The total votes are three. With the quorum disk, a single node can startclustered operation with a majority of votes (two votes, in this example).

Figure 3-7 Two-Node Cluster Quorum Devices

Clustered-Pair Quorum Disks

A clustered-pairs configuration, shown in Figure 3-8, always has an evennumber of cluster nodes. The nodes in each pair usually provide dataservice failover backup for one another.

Figure 3-8 Clustered-Pair Quorum Devices

Node 1 (1) Node 2 (1)

Storage Array

QD(1)

Interconnect

Junction

Junction

Node 1(1)

Node 2(1)

Node 3(1)

Node 4(1)

QD(1) QD(1)

QD(1)

Sun StorEdgeMultiPack System



There are many possible split-brain scenarios. Not all of the possiblesplit-brain combinations allow the continuation of clustered operation.The following is true for a clustered pair configuration without the “extra”quorum device shown in Figure 3-8.

● There are six possible votes.

● A quorum is four votes.

● If both quorum devices fail, the cluster can still come up.

The nodes wait until all are present (booted).

● If Nodes 1 and 2 fail, there are not enough votes for Nodes 3 and 4 tocontinue running.

A token quorum device between Nodes 2 and 3 can eliminate thisproblem. A Sun StorEdge MultiPack desktop array could be used forthis purpose.

● An entire pair of nodes can fail, and there are still four votes out ofseven.

Pair+N Quorum Disks

Figure 3-9 shows a typical quorum disk configuration in a Pair+2configuration. Three quorum disks are used.

Figure 3-9 Pair+N Quorum Devices

Junction

Junction

Node 1(1)

Node 2(1)

Node 3(1)

Node 4(1)

QD(1)

QD(1)

QD(1)



The following is true for the Pair+N configuration shown in Figure 3-9:

● There are three quorum disks.

● There are seven possible votes.

● A quorum is four votes.

● Nodes 3 and 4 do not have access to any quorum devices.

● Nodes 1 or 2 can start clustered operation by themselves.

● Up to three nodes can fail (Nodes 1, 3, and 4 or Nodes 2, 3, and 4),and clustered operation can continue.

N+1 Quorum Disks

The N+1 configuration shown in Figure 3-10 requires a differentapproach. Node 3 is the failover backup for both Node 1 and Node 2.

Figure 3-10 N+1 Quorum Devices

The following is true for the N+1 configuration shown in Figure 3-10:

● There are five possible votes.

● A quorum is three votes.

● If Nodes 1 and 2 fail, Node 3 can continue.

Junction

Junction

Node 1(1)

Node 2(1)

Node 3(1)

QD(1)QD(1)



Quorum Devices in the Scalable Storage Topology

Quorum Devices in the scalable storage topology, as shown in Figure 3-11,differ significantly from any other topology.

Figure 3-11 Quorum Devices in the Scalable Storage Topology

The following is true for the quorum devices shown in Figure 3-11:

● The single quorum device has a vote count of number of nodesdirectly attached to it minus one.

● The mathematics and consequences still apply.

● The reservation is done using a SCSI-3 Persistent Group Reservation(see ‘‘SCSI-3 Persistent Group Reservation’’ on page 3-24).

If, for example, Nodes 1 and 3 can intercommunicate but Node 2 isisolated, Node 1 or 3 can reserve the quorum device on behalf ofboth of them.

Note – It would seem that in the same race, Node 2 could “win” andeliminate both Nodes 2 and 3. ‘‘Intentional Reservation Delays forPartitions With Fewer Than Half of the Nodes’’ on page 3-27 shows whythis is unlikely.

Junction

Junction

Node 1(1)

Node 2(1)

Node 3(1)

QD(2)

Preventing Cluster Amnesia With Persistent Reservations



Cluster devices in the Sun Cluster software environment are used notonly as a means of failure fencing but also as a means to prevent clusteramnesia.

Earlier you reviewed the following scenario:

1. In a two-node cluster (Node 1 and Node 2), Node 2 is halted formaintenance.

2. Meanwhile Node 1, running fine in the cluster, makes all sorts ofcluster configuration changes (new device groups, resource groups).

3. Now Node 1 is shut down.

4. You try to boot Node 2 to form a new cluster.

In this simple scenario, the problem is that is you boot Node 2 at the end.It does not have the “correct” copy of the CCR. But, if it were allowed toboot, Node 2 would have to use the copy that it has (as there is no othercopy available) and you would “lose” the changes to the clusterconfiguration made in Step 2.

The Sun Cluster software quorum involves persistent reservations thatprevent Node 2 from booting into the cluster. It is not able to count thequorum device as a vote. Node 2, therefore, waits until the other nodeboots to achieve the correct number of quorum votes.



Persistent Reservations and Reservation Keys

A persistent reservation means that reservation information on a quorumdevice:

● Survives even if all nodes connected to the device are reset

● Survives even after the quorum device itself is powered on and off

Clearly this involves writing some type of information on the disk itself.The information is called a reservation key.

● Each node is assigned a unique 64-bit reservation key value.

● Every node that is physically connected to a quorum device has itsreservation key physically written onto the device.

Figure 3-12 and Figure 3-13 on page 3-23 consider two nodes booted into acluster connected to a quorum device:

Figure 3-12 Two Nodes Booted and Connected to a Quorum Device

Node 1 Node 2

Quorum Device

Node 1 keyNode 2 key



If Node 2 leaves the cluster for any reason, Node 1 will remove or preemptNode 2’s key off of the quorum device. This would include if there were asplit brain and Node 1 won the race to the quorum device.

Figure 3-13 Node 2 Leaves the Cluster

Now the rest of the equation is clear. If you are booting into the cluster, anode cannot count the quorum device vote unless its reservation key isalready on the quorum device. Thus, in the scenario illustrated in theprevious paragraph, if Node 2 tries to boot first into the cluster, it will notbe able to count the quorum vote, and must wait for Node 1 to boot.

After Node 1 joins the cluster, it can detect Node 2 across the transportand add Node 2’s reservation key back to the quorum device so thateverything is equal again.

A reservation key only gets added back to a quorum device by anothernode in the cluster whose key is already there.

Node 1 Node 2

Quorum Device

Node 1 keyNode 2 key



SCSI-2 Persistent Reservation Emulation

The scenario presented in the previous section was actually using amethod called SCSI-2 Persistent Reservation Emulation (PRE) toimplement the reservation keys. PREs have the following characteristics.

● The persistent reservations are not supported directly by the SCSIcommand set.

● Reservation keys are written on private cylinders of the disk (cylindersthat are not visible in the format command, but are still directlywritable by the Solaris OS).

The reservation keys have no impact on using the disk as a regulardata disk, where you will not see the private cylinders.

● The race (for example, in a split brain scenario) is still decided by anormal SCSI-2 disk reservation.

When you add a quorum device, the Sun Cluster software automaticallywill use SCSI-2 PRE with that quorum device if there are exactly twosub-paths for the DID associated with that device.

SCSI-3 Persistent Group Reservation

When there are more than two nodes physically connected to the disk,SCSI-3 Persistent Group Reservations (PGRs) are used. PGRs have thefollowing characteristics:

● The persistent reservations are implemented directly by the SCSI-3command set. Disk firmware itself must be fully SCSI-3 compliant.

● “Racing” to remove another node’s reservation key does not involvephysical reservation of the disk.



SCSI-3 PGR Scenario

In Figure 3-14, four nodes are all physically connected to a quorum drive.Remember, the single quorum drive has three quorum votes.

Figure 3-14 Four Nodes Physically Connected to a Quorum Drive

Now imagine Nodes 1 and 3 can see each other over the transport, andNodes 2 and 4 can see each other over the transport.


Quorum DeviceNode 1 keyNode 2 keyNode 3 keyNode 4 key



In each pair, the node with the lower reservation key tries to eliminate thereservation key of the other pair. The SCSI-3 protocol assures that onlyone pair will win. This is shown in Figure 3-15.

Figure 3-15 One Pair of Nodes Eliminating the Other

Because Nodes 2 and 4 have their reservation key eliminated, they cannotcount the three votes of the quorum device. Because they fall below theneeded quorum, they will kernel panic.

Cluster amnesia is avoided in the exact same way as in a two-nodequorum device. If you now shut down the whole cluster, Node 2 andNode 4 could not count the quorum device because their reservation keyis eliminated. They would have to wait for either Node 1 or Node 3 tojoin. One of those nodes could then add back reservation keys for Node 2and Node 4.


Quorum Device(3 votes)Node 1 key

Node 2 keyNode 3 keyNode 4 key



Intentional Reservation Delays for Partitions WithFewer Than Half of the Nodes

Imagine the same scenario just presented, but three nodes can talk to eachother while the fourth is “isolated” on the cluster transport, as shown inFigure 3-16.

Figure 3-16 Node 3 Is Isolated on the Cluster Transport

Is there anything to prevent the lone node from eliminating the clusterkeys of the other three and making them all kernel panic?

In this configuration, the lone node will intentionally delay before racingfor the quorum device. The only way it could “win” is if the other threenodes are really dead (or each of them is also isolated, which would makethem all delay the same amount).

The delay is implemented when the number of nodes that a node can seeon the transport (including itself) is fewer than half the total nodes.

Node 1 Node 2Node 3

(isolated)(delay)

Node 4

Quorum DeviceNode 1 keyNode 2 keyNode 3 keyNode 4 key

Configuring a Cluster Interconnect



There are two variations of cluster interconnects: point-to-point andjunction-based. In a junction-based interconnect, the junctions must beswitches and not hubs.

Point-to-Point Cluster Interconnect

In a two-node cluster, you can directly connect interconnect interfacesusing crossover cables. Figure 3-17 shows a point-to-point interconnectconfiguration using 100BASE-T interfaces.

Figure 3-17 Point-to-Point Cluster Interconnect

During the Sun Cluster 3.1 software installation, you must provide thenames of the end-point interfaces for each cable.

Caution – If you provide the wrong interconnect interface names duringthe initial Sun Cluster software installation, the first node is installedwithout errors, but when you try to manually install the second node, theinstallation hangs. You have to correct the cluster configuration error onthe first node and then restart the installation on the second node.

System board hme1

System board hme2

hme1 System board

hme2 System board

Node 1 Node 2



Junction-based Cluster Interconnect

In cluster configurations greater than two nodes, you must join theinterconnect interfaces using switches. You can also use switches to jointwo-node cluster interconnects to prepare for the expansion of the numberof nodes at a later time. A typical junction-based interconnect is shown inFigure 3-18.

During the Sun Cluster 3.1 software installation, you are asked whetherthe interconnect system uses junctions. If you answer yes, you mustprovide names for each of the switches.

Figure 3-18 Junction-based Cluster Interconnect

Note – If you specify more than two nodes during the initial portion ofthe Sun Cluster software installation, the use of junctions is assumed.

Cluster Transport Interface Addresses

During the Sun Cluster software installation, the cluster interconnects areassigned IP addresses based on a base address of 172.16.0.0 . Ifnecessary, you can override the default address, but this is notrecommended. Uniform addresses can be a benefit during problemisolation.

You are required to reserve at least a class B-like namespace for the clusterinterconnect IP addresses. This is documented in the Sun™ Cluster 3.1System Administration Guide but is not well presented or enforced by theinstallation script.

Switch

Switch




Identify Cluster Transport Interfaces

Identifying network interfaces is not an easy task. To accurately determinethe logical name of each interface on a system, use the prtconf commandto complete the following steps:

1. Look for network interfaces in the prtconf command output.

Typical instances you might see are network instance #0 ,SUNW,hme instance #1 , and SUNW,hme instance #2 .

2. Verify which interfaces are already up (in other words, plumbed).

# ifconfig -alo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu8232 index 1 inet 127.0.0.1 netmask ff000000hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4>mtu 1500 index 2 inet 129.200.9.2 netmask ffffff00broadcast 129.200.9.255ether 8:0:20:96:3:86

3. Bring up the unplumbed interfaces for testing. Make up someunused subnet address just to test out interconnectivity acrossprivate net.

# ifconfig hme1 plumb# ifconfig hme1 192.168.1.1 up# ifconfig hme2 plumb# ifconfig hme2 192.168.2.1 up

4. Perform Step 3 on the other node. Choose corresponding IPaddresses, for example, 192.168.1.2 and 192.168.2.2 .

5. Verify that the new interfaces are up.

# ifconfig -a

6. Test that the nodes can ping across each private network, forexample:

# ping 192.168.1.2192.168.1.2 is alive# ping 192.168.2.2192.168.2.2 is alive



7. Make sure that the interfaces you are looking at are not actually onthe public net. Do the following for each set of interfaces:

a. In one window, run the snoop command for the interface inquestion:

# snoop -d hme1hope to see no output here

b. In another window, use the ping command to probe the publicbroadcast address, and make sure no traffic is seen by the“candidate” interface.

# ping public_net_broadcast_address

Note – You should create configuration worksheets and drawings of yourcluster configuration.

8. After you have identified the new network interfaces, bring themdown again. Cluster installation fails if your transport networkinterfaces are still up from testing.

# ifconfig hme1 down unplumb# ifconfig hme2 down unplumb

Configuring a Public Network


Configuring a Public Network

You are not asked about public network configuration at the time you areinstalling the cluster.

The public network interfaces must be managed by IPMP, which can beadministered either before or after cluster installation. In this course, youdo it after cluster installation in Module 8, “Managing the Public NetworkWith IPMP.”

Because you are identifying your private transport interfaces beforecluster installation, it can be useful to identify your public networkinterfaces at the same time, so as to avoid confusion.

Your primary public network adapter should be the only one currentlyconfigured on the public net. You can verify this with:

# ls -l /etc/hostname.*# ifconfig -a

You can verify your secondary public network adapter, if applicable, by:

● Making sure it is not one of those you identified to be used as theprivate transport

● Making sure it can snoop public network broadcast traffic:

# ifconfig ifname plumb# snoop -d ifname(other window or node)# ping -s pubnet_broadcast_addr

Exercise: Preparing for Installation




● Identify a cluster topology

● Identify quorum devices needed

● Verify the cluster interconnect cabling

Preparation

To begin this exercise, you must be connected to the cluster hosts throughthe cconsole tool, and you are logged into them as user root .

Note – During this exercise, when you see italicized names, such asIPaddress , enclosure_name , node1 , or clustername imbedded in acommand string, substitute the names appropriate for your cluster.

Task 1 – Verifying the Solaris™ Operating System

In this section, you verify that the boot disk is correctly partitioned on allnodes.

1. Type the /etc/prtconf command on each node, and record the sizeof physical memory (/etc/prtconf | grep Memory ).

Node 1 memory: _______________

Node 2 memory: _______________

2. Type the df -kl command on each node, and verify that there is a100-Mbyte per globaldevices file system mounted.

3. Type swap -l on each node, and verify it has at least 750 Mbytes ofswap space.



Task 2 – Identifying a Cluster Topology

1. Record the desired topology configuration of your cluster inTable 3-1.

2. Verify that the storage arrays in your cluster are properly connectedfor your target topology. Recable the storage arrays if necessary.

Task 3 – Selecting Quorum Devices

1. Record the number of quorum devices you must configure after thecluster host software installation.

Number of quorum devices: __________

Note – Consult with your instructor if you are not sure about yourquorum device configuration.

2. Type the format command, and record the logical path to a suitablequorum disk drive in one of your storage arrays.

Quorum disks: ________________________________

3. Type Control-D to cleanly exit the format utility.

Table 3-1 Topology Configuration

Number of nodes

Number of storage arrays

Types of storage arrays



Task 4 – Verifying the Cluster InterconnectConfiguration

This task describes how to verify the cluster interconnect configuration.

Configuring a Point-to-Point Ethernet Interconnect

Skip this section if your cluster interconnect is not point-to-point.

1. Determine the names of your cluster interconnect adapters.

2. Complete the form in Figure 3-19 if your cluster uses anEthernet-based point-to-point interconnect configuration.

Figure 3-19 Ethernet Interconnect Point-to-Point Form

PrimaryInterconnectInterface

SecondaryInterconnectInterface



Node 1 Node 2



Configuring a Switch-Based Ethernet Interconnect

Perform the following steps to configure a switch-based Ethernetinterconnect:

1. Complete the form in Figure 3-20 if your cluster uses anEthernet-based cluster interconnect with switches:

a. Record the logical names of the cluster interconnect interfaces(hme2, qfe1 , and so forth).

b. Add or delete nodes to the diagram as appropriate.

Figure 3-20 Ethernet Interconnect With Switches Form

2. Verify that each Ethernet interconnect interface is connected to thecorrect switch.

Note – If you have any doubt about the interconnect cabling, consult withyour instructor now. Do not continue this exercise until you are confidentthat your cluster interconnect system is cabled correctly.





Node 1 Node 2

Switch 1

Switch 2



Node 3



Node 4



Task 5 – Selecting Public Network Interfaces

Ask for help from your instructor in identifying public network interfaceson each node that can be used in IPMP groups.

Perform the following steps to select public network interfaces:

1. Record the logical names of potential IPMP Ethernet interfaces oneach node in Table 3-2.

Note – It is important that you are sure about the logical name of eachpublic network interface (hme2, qfe3 , and so on).

2. Verify that the target IPMP interfaces on each node are connected toa public network.

Table 3-2 Logical Names of Potential IPMP Ethernet Interfaces

System Primary IPMPInterface

Backup IPMPInterface

Node 1

Node 2

Node 3 (if any)

Node 4 (if any)

Exercise Summary


Exercise Summary

?!


● Experiences

● Interpretations

● Conclusions

● Applications


Module 4

Installing the Sun Cluster Host Software

Objectives


● Install the Sun Cluster host system software

● Interpret configuration questions during the Sun Cluster softwareinstallation

● Describe the Sun Cluster Software node patches

● Perform the interactive installation process for the Sun Cluster 3.1software

● Perform post-installation configuration

● Verify the general cluster status and cluster configurationinformation

Relevance


Relevance

?!


● What configuration issues might control how the Sun Clustersoftware is installed?

● What type of post-installation tasks might be necessary?

● What other software might you need to finish the installation?


Installing the Sun Cluster Host Software 4-3Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1







● Sun Cluster 3.x Hardware Collection for Solaris OS (SPARC PlatformEdition)

Summarizing Sun Cluster Software



Sun Cluster software is installed on a Sun Cluster hardware platform. Thecomplete Sun Cluster 3.1 software collection, shown in Figure 4-1, isreleased on the Sun Java System CD-ROM or as a standalone CD-ROM.

Figure 4-1 Sun Cluster Software

For the Sun Java System, the Sun Cluster 3.1 software is located at:

/ mnt/jes_ Rel _sparc/Solaris_sparc/Product/sun_cluster/Solaris_ Rel

Where:

● / mnt – The mount point for the software. If the software is on aCD-ROM, this is typically /cdrom . If the software has been spooled,it is the directory containing the software.

● /jes_ Rel _sparc – This is the CD-ROM title. Rel is the releaseyear and quarter for this version of the software. For example, 03q4is June 2003. If the software is on a CD-ROM under control of thevolume management daemon, this portion of the directory structurecan be referenced by /cdrom0 .

SUNWccon SUNWscscnSUNWmdm SUNWscshlSUNWscdev SUNWscssvSUNWscfab SUNWscuSUNWscman SUNWscvaSUNWscr SUNWscvmSUNWscsal SUNWscvrSUNWscsam SUNWscvw

Packages

SunCluster_3.1/Sol_9

Tools

defaultslib

scinstall

Sun ClusterData Services



● /Solaris_sparc/Product/sun_cluster – This is the path to theSun Cluster 3.1 software. The _sparc is replaced with x86 if thissoftware has been created for that platform.

● /Solaris_ Rel – This is the release of Solaris for which this SunCluster 3.1 can be used. Rel is either 8 for the Solaris 8 OS or 9 forthe Solaris 9 OS.

The directory below the Solaris release contains the actual packagescomposing the Sun Cluster 3.1 software in the directory Packages , andthe tools to install the software in the Tools directory. Use the scinstallutility in the Tools directory to install the Sun Cluster software on clusternodes. For example, to start the Sun Cluster 3.1 installation process fromthe Sun Java System CD-ROM for a Sun SPARC platform, mounted withvolume management, with the cluster node running the Solaris 9 OS, typethe following commands as the root user:

# cd /cdrom/cdrom0/Solaris_sparc/Product/sun_cluster/Solaris_9/Tools# ./scinstall

For a standalone CD-ROM, the Sun Cluster 3.1 software is located at/ mnt/SunCluster_3.1/Sol_rel/ .

The directory below the Solaris release contains the actual packagescomposing the Sun Cluster 3.1 software in the directory Packages , andthe tools to install the software in the Tools directory. Use the scinstallutility in the Tools directory to install the Sun Cluster software on clusternodes. For example, to start the Sun Cluster 3.1 installation process from astandalone CD-ROM, mounted with volume management, with thecluster node running the Solaris 9 OS, type the following commands asthe root user:

# cd /cdrom/cdrom0/SunCluster_3.1/Sol_9/Tools# ./scinstall

Note – You might also need Solstice DiskSuite software or VERITASVolume Manager software. Solstice DiskSuite software was renamed asSolaris Volume Manager software in the Solaris 9 OS and is part of thebase OS. Installation and configuration of these software products isdescribed later in this course.



Sun Cluster Framework Software

Running the installation script for the Sun Cluster 3.1 software installs thefollowing framework software packages:

● SUNWscr– Sun Cluster Framework software

● SUNWscu– Sun Cluster (Usr)

● SUNWscsck – Sun Cluster vulnerability checking

● SUNWscnm– Sun Cluster name

● SUNWscdev– Sun Cluster Developer support

● SUNWscgds– Sun Cluster Generic Data Service

● SUNWscman– Sun Cluster manual pages

● SUNWscsal – Sun Cluster SyMON agent library

● SUNWscsam– Sun Cluster SyMON modules

● SUNWscvm– Sun Cluster VxVM support

● SUNWmdm– Solstice DiskSuite support (Mediator software)

● SUNWscva– Apache SSL components

● SUNWscvr – SunPlex™ manager root components

● SUNWscvw– SunPlex manager core components

● SUNWfsc – Sun Cluster messages (French)

● SUNWfscvw – SunPlex manager online help (French)

● SUNWjsc – Sun Cluster messages (Japanese)

● SUNWjscman– Sun Cluster online manual pages (Japanese)

● SUNWjscvw – SunPlex manager online help (Japanese)

● SUNWksc– Sun Cluster messages (Korean)

● SUNWkscvw– SunPlex manager online help (Korean)

● SUNWcsc– Sun Cluster messages (Simplified Chinese)

● SUNWcscvw– SunPlex manager online help (Simplified Chinese)

● SUNWhsc– Sun Cluster messages (Traditional Chinese)

● SUNWhscvw– SunPlex manager online help (Traditional Chinese)

● SUNWexplo – Sun Explorer software

There are additional software packages on the CD-ROM that providesupplemental features.



Sun Cluster 3.1 Software Licensing

No license keys are required for the Sun Cluster software. You must,however, furnish paper license numbers to obtain service.

VERITAS Volume Manager is automatically licensed when used withsome Sun storage arrays, such as the Sun StorEdge A5x00 arrays. If youuse any other storage array, you need a special license for VERITASVolume Manager.

Sun Cluster Software Basic Installation Process

The basic Sun Cluster 3.1 host system (node) installation process iscompleted in several major steps. The general process is:

1. Repartition boot disks to meet Sun Cluster 3.1 softwarerequirements.

2. Install the Solaris OS software.

3. Configure the environment for the cluster host systems.

4. Install Solaris 8 OS or Solaris 9 OS patches.

5. Install hardware-related patches.

6. Install the Sun Cluster 3.1 software.

Do this as specified on page 4-8 if you want to be sure about whichnode gets which nodeid in the cluster.

7. Install any Sun Cluster software patches not installed during theprevious step.

8. Perform post-installation checks and configuration.



Sun Cluster Software Alternative InstallationProcesses

There are three different methods for installing the Sun Cluster software.For instructional purposes, the exercises in this course use the manualmethod. You must have a good understanding of the basic installationprocess before attempting the more advanced methods.

Manual Installation

Use the Sun Cluster software scinstall utility to manually install theSun Cluster software on each cluster host system. The Solaris OS softwaremust be previously installed. There are multiple manual installationmethodologies available.

● Install the Sun Cluster 3.1 software on each node individually.

This option gives you the most control over the installation. Youchoose which nodes start with which node ID, the transportconnections, and so on. Use this if you need to explicitly answer allthe installation questions.

● Install the Sun Cluster 3.1 software on all nodes at once.

This option requires some pre-configuration on the servers. It is runon the node that receives the highest node ID. The other servers areinstalled in the order specified in the node list. You can makeminimal modifications to the default values.

● Install the Sun Cluster 3.1 software on a single node to create asingle-node cluster.

This option can be used in a development or test environment whereactual node failover is not necessary. It sets up the node to use thefull Sun Cluster 3.1 software framework in a standalone mode. Thisallows the addition of data services, or the development of dataservices, or a server to learn the administration of the clusterframework.



● Upgrade an existing cluster installation.

There are many requirements and restrictions involved with anupgrade of Sun Cluster software. These are specific to the currentconfiguration of your cluster and what is being upgraded to whatlevel. There are two basic types of upgrades: rolling or non-rolling.

● Nonrolling upgrade – In a nonrolling upgrade, you shut downthe cluster before you upgrade the cluster nodes. You return thecluster to production after all nodes are fully upgraded.

● Rolling upgrade – In a rolling upgrade, you upgrade one node ofthe cluster at a time. The cluster remains in production withservices running on the other nodes.

For complete information about upgrading your version of Sun Clustersoftware, read Chapter 3 in the Sun Cluster 3.1 Software Installation Guidefor the version of Sun Cluster software that will be your version at thecompletion of the upgrade. Additionally, upgrades are performed as partof the Sun Cluster 3.1 Advanced Administration course, ES-438.

JumpStart™ Software Installation

If you have an existing JumpStart™ software server, you can use thescinstall utility to include the Sun Cluster software installation in theexisting JumpStart software configuration. The Sun Cluster 3.1 software iscopied to the JumpStart software server. You can then automaticallyinstall the Solaris OS and the Sun Cluster software in a single operation.This process is performed on the JumpStart software install server.

Note – You must answer all of the scinstall questions just as you doduring a manual installation. The resulting information is stored and usedduring the JumpStart software installation.



SunPlex Manager Installation

You can use the SunPlex manager application to install the Sun Clustersoftware. You must first install the Solaris OS and the SunPlex managersoftware on each cluster node. The packages that must be installed oneach node first are: SUNWscva, SUNWscvr, and SUNWscvw.

Note – The SunPlex manager software packages are automaticallyinstalled on each node during the manual Sun Cluster softwareinstallation process.

Configuring the Sun Cluster Software Node Environment


Configuring the Sun Cluster Software Node Environment

You can configure the Sun Cluster software environment before youinstall the Sun Cluster 3.1 software, providing you have documented yourintended cluster configuration.

You should configure the user root environment on each cluster nodeand also configure local network name resolution. If your environmentrequires RBAC, the administrative user accounts can be added at thistime. The RBAC configuration must be completed after the installation ofthe Sun Cluster 3.1 software. Additional information on the SunCluster 3.1 RBAC requirements can be found in Appendix C, “Role-BasedAccess Control Authorizations.” All tasks in this course are performed asthe user root .

Configuring the User root Environment

The root login environment should include the following search path andman page information:

PATH=$PATH:/usr/cluster/bin

MANPATH=$MANPATH:/usr/cluster/man:/usr/share/man

Note – Some of the path information depends on which virtual volumemanagement software your cluster uses. Add the location of theexecutables for your volume manager of choice to the PATHvariable. Addthe location of the manual pages for your volume manager of choice tothe MANPATH.

Configuring Network Name Resolution

The names of cluster nodes should be resolved locally so that the cluster isnot disabled in the event of a naming service failure. Several changes tothe /etc/nsswitch.conf file are mandatory and are performedautomatically during the Sun Cluster software installation. You shouldverify the changes after completing the Sun Cluster software installationconfiguration.

Installing Sun Cluster Software Node Patches


Installing Sun Cluster Software Node Patches

The Sun Cluster software nodes might require patches in the followingareas:

● Solaris OS patches

● Storage array interface firmware patches

● Storage array disk drive firmware patches

● VERITAS Volume Manager patches

● Solaris Volume Manager software patches

You cannot install some patches, such as those for VERITAS VolumeManager, until after the volume management software installation iscompleted. To install patches during cluster node installation, place thepatches in the directory /var/cluster/patches .

Patch Installation Warnings

Before installing any patches, always do the following:

● Make sure all cluster nodes have the same patch levels.

● Do not install any firmware-related patches without qualifiedassistance.

● Always obtain the most current patch information.

● Read all patch README notes carefully.

Obtaining Patch Information

You should always consult with your Sun field representative aboutpossible patch installations. You can obtain current patch information asfollows:

● Consult http://sunsolve.sun.com.

● Use the SunSolveSM PatchPro tool interactively

● Read current Sun Cluster software release notes

Installing the Sun Cluster 3.1 Software



Although there are three different methods to install the Sun Cluster 3.1software on the cluster nodes, only the interactive method is described inthis module. You must understand the interactive installation processbefore you attempt a more advanced method.

Understanding the installmode Flag

As you install Sun Cluster software on cluster nodes and reboot the nodesinto the cluster, a special flag called the installmode flag is set in thecluster CCR. When this flag is set:

● The first node installed (node ID 1) has a quorum vote of 1

● All other nodes have a quorum vote of zero.

This allows you to complete rebooting of the second node into the clusterwhile maintaining the quorum mathematics rules. If the second node hada vote (making a total of two in the cluster), the first node would kernelpanic when the second node was rebooted after the cluster software wasinstalled, because the first node would lose operational quorum!

One important side-effect of the installmode flag is that you must becareful not to reboot the first node (node ID 1) until you can choosequorum devices and eliminate (reset) the installmode flag. If youaccidentally reboot the first node, all the other nodes will kernel panic (asthey will be left with zero votes out of a possible total of one).

If the installation is a single-node cluster, the installmode flag is not set.Post-installation steps to choose a quorum device and reset theinstallmode flag are unnecessary.

Installing on All Nodes

The Sun Cluster 3.1 10/03 software and higher allows running theinstallation program on one node of the cluster, and doing the installationof the software on all nodes with that one command. The cluster must bephysically constructed according to the information presented earlier inthis module. The installation of the Sun Cluster 3.1 software must beperformed as the user root . The user root must be able to log in to eachnode of the cluster without a password, using rsh or ssh .



Using the Initial scinstall Menu

You use the scinstall menu to install a cluster of two or more nodes.Installation of a single-node cluster or upgrading an existing cluster mustbe done from the scinstall command line. Examples are presentedlater in this module.

The following example shows a multiple node cluster being installed froma single node. The scinstall program is located in the Tools directoryon the Sun Cluster 3.1 software CD-ROM.

# / mnt / product_path / Solaris_release /Tools/scinstall*** Main Menu ***

Please select from one of the following (*) options:

* 1) Install a cluster or cluster node 2) Configure a cluster to be JumpStarted from this install server 3) Add support for new data services to this cluster node * 4) Print release information for this cluster node

* ?) Help with menu options * q) Quit

Option: 1

The balance of the Sun Cluster 3.1 software installation is excerpted in thefollowing sections with comments for each section.

*** Install Menu ***

Please select from any one of the following options:

1) Install all nodes of a new cluster 2) Install just this machine as the first node of a new cluster 3) Add this machine as a node in an existing cluster

?) Help with menu options q) Return to the Main Menu

Option: 1



Note – You can type Control-D at any time to return to the mainscinstall menu and restart the installation. Your previous answers toquestions become the default answers.

Establishing a New Cluster and Package Installation

Selecting option 1 from the scinstall install menu presents a descriptionof the requirements necessary for installation of the entire cluster from asingle node.

*** Installing all Nodes of a New Cluster ***

This option is used to install and configure a new cluster with a set of two or more member nodes.

If either remote shell (see rsh(1)) or secure shell (see ssh(1)) root access is enabled to all of the new member nodes from this node, the Sun Cluster framework software will be installed on each node. Otherwise, the Sun Cluster software must already be pre-installed on each node with the "remote configuration" option enabled.

The Sun Cluster installation wizard (see installer(1M)) can be used to install the Sun Cluster framework software with the "remote configuration" option enabled. Since the installation wizard does not yet include support for cluster configuration, you must still use scinstall to complete the configuration process.

Press Control-d at any time to return to the Main Menu.

Do you want to continue (yes/no) [yes]?

>>> Type of Installation <<<

There are two options for proceeding with cluster installation. For most clusters, a Typical installation is recommended. However, you might need to select the Custom option if not all of the Typical defaults can be applied to your cluster.

For more information about the differences between the Typical and Custom installation methods, select the Help option from the menu.

Please select from one of the following options:



1) Typical 2) Custom

?) Help q) Return to the Main Menu

Option [1]: 1

Begin the installation by answering questions about the installation. Ifroot access by way of rsh or ssh is not enabled on each node, this mustbe done before the installation can proceed.

Establishing Cluster Configuration

You provide a cluster name which all nodes agree on. This is not a nameassociated with an IP address in the /etc/inet/hosts file. After thecluster name is decided, the node names are provided. The node namesare associated with an IP address in the /etc/inet/hosts file.

>>> Cluster Name <<<

Each cluster has a name assigned to it. The name can be made up of any characters other than whitespace. Each cluster name should be unique within the namespace of your enterprise.

What is the name of the cluster you want to establish [proto]?

>>> Cluster Nodes <<<

This Sun Cluster release supports a total of up to 16 nodes.

Please list the names of the other nodes planned for the initial cluster configuration. List one node name per line. When finished, type Control-D:

Node name: proto193 Node name (Control-D to finish): ^D



This is the complete list of nodes:

proto192proto193

Is it correct (yes/no) [yes]? y

Attempting to contact "proto193" ... done

Searching for a remote install method ... done

The remote shell (see rsh(1)) will be used to install and configure this cluster.

Press Enter to continue:

The nodes identified previously are installed in the order shown, with thenode running the installation being the last, or highest, numbered node.

The next section asks for the physical network adapters that are used toconstruct the cluster private interconnect. Two network adapters for theprivate interconnect are required. You are specifying the adapters to beused on the node running the scinstall program and can only specifytwo. The adapters on the other nodes are determined during theinstallation process. The default IP network address and netmask areused.

>>> Cluster Transport Adapters and Cables <<<

You must identify the two cluster transport adapters which attach this node to the private cluster interconnect.

Select the first cluster transport adapter for "proto192":

1) hme0 2) qfe0 3) qfe2 4) qfe3 5) Other

Option: 1 Searching for any unexpected network traffic on "hme0" ... done Verification completed. No traffic was detected over a 10 second sample period.

Select the second cluster transport adapter for "proto192":




Option: 2 Searching for any unexpected network traffic on "qfe0" ... done Verification completed. No traffic was detected over a 10 second sample period.

Is it okay to begin the installation (yes/no) [yes]?

During the installation process, sccheck(1M) is run on each of the new cluster nodes. If sccheck(1M) detects problems, you can either interrupt the installation process or check the log files after installation has completed.

Interrupt the installation for sccheck errors (yes/no) [no]?Installation and Configuration

Log file - /var/cluster/logs/install/scinstall.log.1527

Testing for "/globaldevices" on "proto192" ... done Testing for "/globaldevices" on "proto193" ... done

Checking installation status ... done

The Sun Cluster software is not installed on "proto192". The Sun Cluster software is not installed on "proto193".

Generating the download files ... done

Started download of the Sun Cluster software to "proto193".

Finished download to "proto193".

Started installation of the Sun Cluster software on "proto192". Started installation of the Sun Cluster software on "proto193".

Finished software installation on "proto192". Finished software installation on "proto193".

Starting discovery of the cluster transport configuration.



The following connections were discovered:

proto192:hme0 switch1 proto193:hme0proto192:qfe0 switch2 proto193:qfe0

Completed discovery of the cluster transport configuration.

Started sccheck on "proto192". Started sccheck on "proto193".

sccheck completed with no errors or warnings for "proto192". sccheck completed with no errors or warnings for "proto193".

Removing the downloaded files ... done

Configuring "proto193" ... done Rebooting "proto193" ... done

Configuring "proto192" ... done Rebooting "proto192" ...

Log file - /var/cluster/logs/install/scinstall.log.1527

Installing One Node at a Time

For a traditional installation, the nodes can be installed one at a time. Thefirst node is installed, then each subsequent node is installed using thefirst node as a “sponsoring node.” You are given the opportunity on eachnode to override the default values. The first node is installed usingOption 2 from the Install Menu. The second node and subsequent nodesare installed using Option 3 from the Install Menu.

Selecting From the scinstall Initial Menus

*** Main Menu ***

Please select from one of the following (*) options:

* 1) Install a cluster or cluster node 2) Configure a cluster to be JumpStarted from this install server 3) Add support for new data services to this cluster node * 4) Print release information for this cluster node




Option: 1


Please select from any one of the following options:

1) Install all nodes of a new cluster 2) Install just this machine as the first node of a new cluster 3) Add this machine as a node in an existing cluster


Option: 2

*** Installing just the First Node of a New Cluster ***

This option is used to establish a new cluster using this machine as the first node in that cluster.

Once the cluster framework software is installed, you will be asked for the name of the cluster. Then, you will have the opportunity to run sccheck(1M) to test this machine for basic Sun Cluster pre-configuration requirements.

After sccheck(1M) passes, you will be asked for the names of the other nodes which will initially be joining that cluster. In addition, you will be asked to provide certain cluster transport configuration information.

Press Control-d at any time to return to the Main Menu.

Do you want to continue (yes/no) [yes]?

>>> Software Package Installation <<<

Installation of the Sun Cluster framework software packages will take a few minutes to complete.

Is it okay to continue (yes/no) [yes]?

** Installing SunCluster 3.1 framework **



SUNWscr.....doneSUNWscu.....doneSUNWscsck...doneSUNWscnm....doneSUNWscdev...doneSUNWscgds...doneSUNWscman...doneSUNWscsal...doneSUNWscsam...doneSUNWscvm....doneSUNWmdm.....doneSUNWscva....doneSUNWscvr....doneSUNWscvw....doneSUNWscrsm...doneSUNWcsc.....doneSUNWcscvw...doneSUNWdsc.....doneSUNWdscvw...doneSUNWesc.....doneSUNWescvw...doneSUNWfsc.....doneSUNWfscvw...doneSUNWhsc.....doneSUNWhscvw...doneSUNWjsc.....doneSUNWjscman..doneSUNWjscvw...doneSUNWksc.....doneSUNWkscvw...doneSUNWexplo...done




Installing Patches

The installation program allows installation of Solaris OS or SunCluster 3.1 patches during the installation. The installation program looksin /var/cluster/patches for these patches.

>>> Software Patch Installation <<<

If there are any Solaris or Sun Cluster patches that need to be added as part of this Sun Cluster installation, scinstall can add them for you. All patches that need to be added must first be downloaded into a common patch directory. Patches can be downloaded into the patch directory either as individual patches or as patches grouped together into one or more tar, jar, or zip files.

If a patch list file is provided in the patch directory, only those patches listed in the patch list file are installed. Otherwise, all patches found in the directory will be installed. Refer to the patchadd(1M) man page for more information regarding patch list files.

Do you want scinstall to install patches for you (yes/no) [yes]?

Choosing the Cluster Name

>>> Cluster Name <<<

Each cluster has a name assigned to it. The name can be made up of any characters other than whitespace. Each cluster name should be unique within the namespace of your enterprise.

What is the name of the cluster you want to establish? proto

Running the sccheck Program

The sccheck program has been greatly enhanced in a minimum of SunCluster 3.1 10/03 software. It now performs a more in-depth test of thecluster node, running with the Sun Explorer software. If you have aversion of the Sun Explorer software installed on your system, ensure thatit meets the version requirements for sccheck . If it does not, it must beremoved and installed from the Sun Cluster 3.1 software CD-ROM.



>>> Check <<<

This step allows you to run sccheck(1M) to verify that certain basic hardware and software pre-configuration requirements have been met. If sccheck(1M) detects potential problems with configuring this machine as a cluster node, a report of failed checks is prepared and available for display on the screen. Data gathering and report generation can take several minutes, depending on system configuration.

Do you want to run sccheck (yes/no) [yes]? Running sccheck ...

sccheck: Requesting explorer data and node report from proto192.sccheck: proto192: Explorer finished.sccheck: proto192: Starting single-node checks.sccheck: proto192: Single-node checks finished.


Choosing Cluster Node Names

You enter names of other nodes that you know you are installing into thesame cluster.

>>> Cluster Nodes <<<

This release of Sun Cluster supports a total of up to 16 nodes.

Please list the names of the other nodes planned for the initial cluster configuration. List one node name per line. When finished, type Control-D:

Node name: proto193Node name (Ctrl-D to finish): ^D

This is the complete list of nodes:

proto192proto193

Is it correct (yes/no) [yes]?



Note – Use local /etc/inet/hosts name resolution to preventinstallation difficulties, and to increase cluster availability by eliminatingthe cluster’s dependence on a naming service.

Selecting Data Encryption Standard (DES) Authentication

As shown in the following example, you can select DES authentication foruse only during the remainder of the cluster installation. This can preventunauthorized modification of the CCR contents during installation. This isnot likely but might be a requirement in high security environments.

>>> Authenticating Requests to Add Nodes <<<

Once the first node establishes itself as a single node cluster, other nodes attempting to add themselves to the cluster configuration must be found on the list of nodes you just provided. You can modify this list using scconf(1M) or other tools once the cluster has been established.

By default, nodes are not securely authenticated as they attempt to add themselves to the cluster configuration. This is generally considered adequate, since nodes which are not physically connected to the private cluster interconnect will never be able to actually join the cluster. However, DES authentication is available. If DES authentication is selected, you must configure all necessary encryption keys before any node will be allowed to join the cluster (see keyserv(1M), publickey(4)).

Do you need to use DES authentication (yes/no) [no]?

Configuring the Cluster Interconnect

The following examples summarize the process of configuring the clusterinterconnect. You must approve the default network address(172.16.0.0 ) and netmask (255.255.0.0 ). If the default base address isin use elsewhere at your site, you may have to supply a different address,a different netmask, or both.

If your cluster is a two-node cluster, you are asked if switches are used.The connections can be point-to-point in a two-node cluster, but you canalso use switches. If the cluster is greater than two nodes, switches areassumed. The switches are assigned arbitrary names.



You must also furnish the names of the cluster transport adapters.

You might choose to have your cluster emulate that it uses switches evenif it does not. Your cluster appears as if it had switches. The advantage isthat it becomes easier to add nodes to your two-node cluster later.

>>> Network Address for the Cluster Transport <<<

The private cluster transport uses a default network address of 172.16.0.0. But, if this network address is already in use elsewhere within your enterprise, you may need to select another address from the range of recommended private addresses (see RFC 1597 for details).

If you do select another network address, bear in mind that the Sun Cluster software requires that the rightmost two octets always be zero.

The default netmask is 255.255.0.0. You can select another netmask, as long as it minimally masks all bits given in the network address.

Is it okay to accept the default network address (yes/no) [yes]? Is it okay to accept the default netmask (yes/no) [yes]?

>>> Point-to-Point Cables <<<

The two nodes of a two-node cluster may use a directly-connected interconnect. That is, no cluster transport junctions are configured. However, when there are greater than two nodes, this interactive form of scinstall assumes that there will be exactly two cluster transport junctions.

Does this two-node cluster use transport junctions (yes/no) [yes]?

>>> Cluster Transport Junctions <<<

All cluster transport adapters in this cluster must be cabled to a transport junction, or "switch". And, each adapter on a given node must be cabled to a different junction. Interactive scinstall requires that you identify two switches for use in the cluster and the two transport adapters on each node to which they are cabled.

What is the name of the first junction in the cluster [switch1]? What is the name of the second junction in the cluster [switch2]?



Selecting Cluster Transport Adapters

The scinstall process presents you with menus of all your possibletransport adapters. Be careful because the menu numbers do not changeafter you choose the first adapter. The process does try to determine if itsees network traffic normally found on a public network for you, andensures that you are not accidentally picking a public network interface. Iftraffic is found, you must select another adapter.

>>> Cluster Transport Adapters and Cables <<<

You must configure at least two cluster transport adapters for each node in the cluster. These are the adapters which attach to the private cluster interconnect.

Select the first cluster transport adapter:


Option: 1 Adapter "hme0" is an Ethernet adapter.

Searching for any unexpected network traffic on "hme0" ... done Verification completed. No traffic was detected over a 10 second sample period.

The "dlpi" transport type will be set for this cluster.

Name of the junction to which "hme0" is connected [switch1]? Each adapter is cabled to a particular port on a transport junction. And, each port is assigned a name. You can explicitly assign a name to each port. Or, for Ethernet switches, you can choose to allow scinstall to assign a default name for you. The default port name assignment sets the name to the node number of the node hosting the transport adapter at the other end of the cable.

For more information regarding port naming requirements, refer to the scconf_transp_jct family of man pages (e.g., scconf_transp_jct_dolphinswitch(1M)).

Use the default port name for the "hme0" connection (yes/no) [yes]? Select the second cluster transport adapter:




Option: 2 Adapter "qfe0" is an Ethernet adapter.

Searching for any unexpected network traffic on "qfe0" ... done Verification completed. No traffic was detected over a 10 second sample period.

Name of the junction to which "qfe0" is connected [switch2]? Use the default port name for the "qfe0" connection (yes/no) [yes]?

Configuring the Global Devices File System

If a /globaldevices file system already exists on the boot disk of eachcluster node, the scinstall utility asks if you want to use it for the globaldevices file system. You can also use a different file system or have thescinstall utility create one for you.

>>> Global Devices File System <<<

Each node in the cluster must have a local file system mounted on /global/.devices/node@<nodeID> before it can successfully participate as a cluster member. Since the "nodeID" is not assigned until scinstall is run, scinstall will set this up for you.

You must supply the name of either an already-mounted file system or raw disk partition which scinstall can use to create the global devices file system. This file system or partition should be at least 512 MB in size.

If an already-mounted file system is used, the file system must be empty. If a raw disk partition is used, a new file system will be created for you.

The first example is for an existing /globaldevices file system. Thismust be a mounted file system containing only the lost+found directory.

The default is to use /globaldevices.



Is it okay to use this default (yes/no) [yes]?

The second example shows how you can use a file system other than thedefault /globaldevices file system.

Is it okay to use this default (yes/no) [yes]? noDo you want to use an already existing file system (yes/no) [yes]? yesWhat is the name of the file system /sparefs

Note – If you use a file system other than the default, it must be an emptyfile system containing only the lost+found directory.

The third example shows how to create a new file system for the globaldevices using an available disk partition.

Is it okay to use this default (yes/no) [yes]? noDo you want to use an already existing file system (yes/no) [yes]? noWhat is the name of the disk partition you want to use /dev/rdsk/c0t0d0s4

Note – You should use the default /globaldevices file system.Standardization helps during problem isolation.

Selecting Automatic Reboot

To complete the basic installation of the first node, you must decidewhether you want the system to reboot automatically.

The reboot question should be considered carefully, since you might needto install Sun Cluster software patches before rebooting. If you chose toinstall patches using scinstall earlier, or if there are no patches thatmust be installed prior to the first boot, it is safe to have scinstallperform the reboot

>>> Automatic Reboot <<<

Once scinstall has successfully installed and initialized the Sun Cluster software for this machine, it will be necessary to reboot. After the reboot, this machine will be established as the first node in the new cluster.

Do you want scinstall to reboot for you (yes/no) [yes]?




Note – If for any reason the reboot fails, you can use a special boot option(ok boot -x ) to disable the cluster software startup until you can fix theproblem.

Confirming the Final Configuration

Before starting the Sun Cluster software installation, the scinstall utilityshows the command line equivalent of the installation for your approval.This information is recorded in the installation log file and can beexamined in the /var/cluster/logs/install directory.

A typical display is as follows:

>>> Confirmation <<<

Your responses indicate the following options to scinstall:

scinstall -ik \ -C proto \ -F \ -T node=proto192,node=proto193,authtype=sys \ -A trtype=dlpi,name=hme0 -A trtype=dlpi,name=qfe0 \ -B type=switch,name=switch1 -B type=switch,name=switch2 \ -m endpoint=:hme0,endpoint=switch1 \ -m endpoint=:qfe0,endpoint=switch2 \ Are these the options you want to use (yes/no) [yes]? Do you want to continue with the install (yes/no) [yes]?

Reviewing Installation Operations

During the final installation process, the scinstall utility performs thefollowing operations on the first cluster node:

● Initializes the CCR with the information entered

● Modifies the vfstab file for the /global/.devices/node@1

● Sets up Network Time Protocol (NTP); you can modify this later



● Inserts the cluster keyword for hosts and netmasks in thensswitch.conf file

● Disables power management

● Sets the electrically erasable programmable read-only memory(EEPROM) local-mac-address? variable to true

● Disables routing on the node (touch /etc/notrouter )

● Creates an installation log (/var/cluster/logs/install )

Checking device to use for global devices file system ... done

Initializing cluster name to "cl1" ... doneInitializing authentication options ... doneInitializing configuration for adapter "hme0" ... doneInitializing configuration for adapter "qfe0" ... doneInitializing configuration for junction "switch1" ... doneInitializing configuration for junction "switch2" ... doneInitializing configuration for cable ... doneInitializing configuration for cable ... done

Setting the node ID for "proto192" ... done (id=1)

Setting the major number for the "did" driver ... done"did" driver major number set to 300

Checking for global devices global file system ... doneUpdating vfstab ... done

Verifying that NTP is configured ... doneInstalling a default NTP configuration ... donePlease complete the NTP configuration after scinstall has finished.

Verifying that "cluster" is set for "hosts" in nsswitch.conf ... doneAdding the "cluster" switch to "hosts" in nsswitch.conf ... done

Verifying that "cluster" is set for "netmasks" in nsswitch.conf ... doneAdding the "cluster" switch to "netmasks" in nsswitch.conf ... done

Verifying that power management is NOT configured ... doneUnconfiguring power management ... done/etc/power.conf has been renamed to /etc/power.conf.021404103637Power management is incompatible with the HA goals of the cluster.Please do not attempt to re-configure power management.



Ensure that the EEPROM parameter "local-mac-address?" is set to "true"... done

Ensure network routing is disabled ... done


Rebooting the First Node

You must reboot the first node completely into the clustered mode tocontinue installing Sun Cluster 3.1 software on other nodes. Theinstallmode flag is automatically set, indicating you are in a“provisional” mode where you can add more cluster nodes.

Note – It is normal to see some DID error messages during the reboot.You should not see any such messages during later system reboots. Atypical error message is did_instances, no such file .

Installing Additional Nodes

While you could run the scinstall process simultaneously on allremaining nodes, run it one node at a time, so that the node IDassignment is predictable.

Most of the interactive dialog for installing additional nodes looks muchlike the first node, so not all of it appears here. The biggest differences arein the initial windows and in entering transport information, where thereis an “autodiscovery” feature.

Reviewing Additional Node Initial Windows

The initial windows for the scinstall process are shown. The sponsoringnode is the first node already installed and rebooted into the cluster. Later,if you want to add more nodes into the cluster, any existing node runningin the cluster can be a sponsoring node.

*** Main Menu ***

Please select from one of the following (*) options: * 1) Install a cluster or cluster node 2) Configure a cluster to be JumpStarted from this install server 3) Add support for new data services to this cluster node



* 4) Print release information for this cluster node


Option: 1


Please select from any one of the following options: 1) Install all nodes of a new cluster 2) Install just this machine as the first node of a new cluster 3) Add this machine as a node in an existing cluster


Option: 3

[[ Welcome and Confirmation Windows as in First Node]][[ Installation of the framework packages as in the First Node ]]

>>> Sponsoring Node <<<. What is the name of the sponsoring node? proto192>>> Cluster Name <<<. What is the name of the cluster you want to join? proto

Attempting to contact "proto192" ... done Cluster name "proto" is correct.

[[ sccheck screens same as in the First Node ]]

Using Transport Auto-Discovery on Additional Nodes

You should use the transport auto-discovery on the additional nodes. Thescinstall process can automatically understand which adapters to use(through a series of ifconfig , ping , and snoop commands).

>>> Autodiscovery of Cluster Transport <<<

If you are using ethernet adapters as your cluster transport adapters, autodiscovery is the best method for configuring the cluster transport.



Do you want to use autodiscovery (yes/no) [yes]? yes

Probing ........

The following connections were discovered:

proto192:qfe0 switch1 proto193:qfe0 proto192:hme0 switch2 proto193:hme0

Is it okay to add these connections to the configuration (yes/no)[yes]? yes

Finishing Installation on Additional Nodes

The rest of the cluster installation on additional nodes is identical to thefirst node. As the installation on each new node is complete, each nodereboots and joins the cluster without a quorum vote. If you reboot the firstnode at this point, all the other nodes would panic because they cannotobtain a quorum. However, you can reboot the second or later nodes.They should come up and join the cluster without errors.

Cluster nodes remain in install mode until you use the scsetupcommand to reset the install mode.

You must perform post-installation configuration to take the nodes out ofinstall mode and also to establish quorum disks.

Note – If you must reboot the first node before performingpostinstallation procedures, you can first shut down the entire clusterusing the scshutdown command. You can also shut down single nodesusing the standard init 0 command.

Installing a Single-Node Cluster

Installation of the Sun Cluster 3.1 software on a single cluster node cannotbe performed from the scinstall menus. It must be performed from thecommand line. The following command installs a single-node cluster:

# / mnt / product_path / Solaris_release /Tools/scinstall -iFo

Performing Post-Installation Configuration



Post-installation can include a number of complex tasks, such as installinga volume manager, database software, or both. There are less complextasks that you must complete first.

This section focuses on the following post-installation tasks:

● Viewing DID device names to choose quorum devices

● Resetting the install mode of operation

● Configuring the NTP

Verifying DID Devices

The scdidadm command shows the DID numbers assigned to the disks inthe cluster. In this excerpt, c1 on proto192 is connected to the samestorage as c4 on proto193 .

You need to know the DID device number for the quorum device youchoose in the next step. You can choose any multi-ported disk.

Note – The local disks (single-ported) appear at the beginning and end ofthe output and cannot be chosen as quorum devices.

proto192:/# scdidadm -L1 proto192:/dev/rdsk/c0t0d0 /dev/did/rdsk/d12 proto192:/dev/rdsk/c0t6d0 /dev/did/rdsk/d23 proto192:/dev/rdsk/c0t8d0 /dev/did/rdsk/d34 proto192:/dev/rdsk/c0t9d0 /dev/did/rdsk/d45 proto192:/dev/rdsk/c0t10d0 /dev/did/rdsk/d56 proto192:/dev/rdsk/c1t0d0 /dev/did/rdsk/d66 proto193:/dev/rdsk/c4t0d0 /dev/did/rdsk/d67 proto192:/dev/rdsk/c1t1d0 /dev/did/rdsk/d77 proto193:/dev/rdsk/c4t1d0 /dev/did/rdsk/d7.22 proto193:/dev/rdsk/c0t0d0 /dev/did/rdsk/d2223 proto193:/dev/rdsk/c0t6d0 /dev/did/rdsk/d23



Resetting the installmode Attribute and ChoosingQuorum

Before a new cluster can operate normally, you must reset theinstallmode attribute on all nodes. You must also define a quorumdevice at the same time. You can do this automatically using the scsetuputility or manually with the scconf command.

Resetting Install Mode Using the scsetup Utility

The scsetup utility is a menu-driven interface that prompts for quorumdevice information the first time it is run on a new cluster installation.When the quorum device is defined, the install mode attribute is reset forall nodes.

# /usr/cluster/bin/scsetup>>> Initial Cluster Setup <<<

This program has detected that the cluster "installmode" attribute is still enabled. As such, certain initial cluster setup steps will be performed at this time. This includes adding any necessary quorum devices, then resetting both the quorum vote counts and the "installmode" property.

Please do not proceed if any additional nodes have yet to join the cluster.

Is it okay to continue (yes/no) [yes]?

Do you want to add any quorum disks (yes/no) [yes]? Dual-ported SCSI-2 disks may be used as quorum devices in two-node clusters. However, clusters with more than two nodes require that SCSI-3 PGR disks be used for all disks with more than two node-to-disk paths. You can use a disk containing user data or one that is a member of a device group as a quorum device.

Each quorum disk must be connected to at least two nodes. Please refer to the Sun Cluster documentation for more information on supported quorum device topologies.

Which global device do you want to use (d<N>)? d19 Is it okay to proceed with the update (yes/no) [yes]?scconf -a -q globaldev=d19

Command completed successfully.




Do you want to add another quorum disk (yes/no)? no Once the "installmode" property has been reset, this program will skip "Initial Cluster Setup" each time it is run again in the future. However, quorum devices can always be added to the cluster using the regular menu options. Resetting this property fully activates quorum settings and is necessary for the normal and safe operation of the cluster.

Is it okay to reset "installmode" (yes/no) [yes]?

scconf -c -q resetscconf -a -T node=.

Cluster initialization is complete.

Type ENTER to proceed to the main menu:

*** Main Menu ***


1) Quorum 2) Resource groups 3) Data Services 4) Cluster interconnect 5) Device groups and volumes 6) Private hostnames 7) New nodes 8) Other cluster properties

?) Help with menu options q) Quit

Note – Although it appears that the scsetup utility uses two simplescconf commands to define the quorum device and reset install mode,the process is more complex. The scsetup utility performs numerousverification checks for you. It is recommended that you do not use scconfmanually to perform these functions.

Use the scconf -p command to verify the install mode status.



Configuring NTP

Modify the NTP configuration file, /etc/inet/ntp.conf.cluster , on allcluster nodes. Remove all private host name entries that are not beingused by the cluster. Also, if you changed the private host names of thecluster nodes, update this file accordingly.

You can also make other modifications to meet your NTP requirements.

You can verify the current ntp.conf.cluster file configuration asfollows:

# grep clusternode /etc/inet/ntp.conf.clusterpeer clusternode1-priv preferpeer clusternode2-privpeer clusternode3-privpeer clusternode4-privpeer clusternode5-priv..peer clusternode16-priv

If your cluster ultimately has two nodes, remove the entries for Node 3through Node 16 from the ntp.conf.cluster file on all nodes. Do notchange the “prefer” keyword. All nodes must have Node 1 as thepreferred peer.

Until the ntp.conf.cluster file configuration is corrected, you see aboot error messages similar to the following:

May 2 17:55:28 proto192 xntpd[338]: couldn’t resolve‘clusternode3-priv’, giving up on it

Performing Post-Installation Verification



When you have completed the Sun Cluster software installation on allnodes, verify the following information:

● General cluster status

● Cluster configuration information

Verifying General Cluster Status

The scstat utility shows the current status of various clustercomponents. You use it to display the following information:

● The cluster name and node names

● Names and status of cluster members

● Status of resource groups and related resources

● Cluster interconnect status

The following scstat-q command option shows the cluster membershipand quorum vote information.

# /usr/cluster/bin/scstat -q

-- Quorum Summary --

Quorum votes possible: 3 Quorum votes needed: 2 Quorum votes present: 3

-- Quorum Votes by Node --

Node Name Present Possible Status --------- ------- -------- ------ Node votes: proto192 1 1 Online Node votes: proto193 1 1 Online

-- Quorum Votes by Device --

Device Name Present Possible Status ----------- ------- -------- ------ Device votes: /dev/did/rdsk/d19s2 1 1 Online



Verifying Cluster Configuration Information

Cluster configuration information is stored in the CCR on each node. Youshould verify that the basic CCR values are correct. The scconf -pcommand shows general cluster information along with detailedinformation about each node in the cluster.

The following example shows the nature of the scconf output for atwo-node cluster following installation and post-installation. Some outputis deleted.

# scconf -pCluster name: protoCluster ID: 0x3DC71034Cluster install mode: disabledCluster private net: 172.16.0.0Cluster private netmask: 255.255.0.0Cluster new node authentication: unixCluster new node list: <. - Exclude allnodes>Cluster nodes: proto192 proto193

Cluster node name: proto192 Node ID: 1 Node enabled: yes Node private hostname: clusternode1-priv Node quorum vote count: 1 Node reservation key: 0x3DC7103400000001 Node transport adapters: qfe0 hme0

Node transport adapter: qfe0 Adapter enabled: yes Adapter transport type: dlpi Adapter property: device_name=qfe Adapter property: device_instance=0 Adapter property:dlpi_heartbeat_timeout=10000 Adapter property:dlpi_heartbeat_quantum=1000 Adapter property: nw_bandwidth=80 Adapter property: bandwidth=10 Adapter property:netmask=255.255.255.128 Adapter property: ip_address=172.16.1.1 Adapter port names: 0



Adapter port: 0 Port enabled: yes

Node transport adapter: hme0 Adapter enabled: yes Adapter transport type: dlpi Adapter property: device_name=hme Adapter property: device_instance=0 Adapter property: dlpi_heartbeat_timeout=10000 Adapter property: dlpi_heartbeat_quantum=1000 Adapter property: nw_bandwidth=80 Adapter property: bandwidth=10 Adapter property: netmask=255.255.255.128 Adapter property: ip_address=172.16.0.129 Adapter port names: 0

Adapter port: 0 Port enabled: yes...Information about the other node removed here for brevity..

Cluster transport cables

Endpoint Endpoint State -------- -------- ----- Transport cable: proto192:qfe0@0 switch1@1 Enabled Transport cable: proto192:hme0@0 switch2@1 Enabled Transport cable: proto193:qfe0@0 switch1@2 Enabled Transport cable: proto193:hme0@0 switch2@2 Enabled

Quorum devices: d19

Quorum device name: d19 Quorum device votes: 1 Quorum device enabled: yes Quorum device name: /dev/did/rdsk/d19s2 Quorum device hosts (enabled): proto192 proto193 Quorum device hosts (disabled): Quorum device access mode: scsi2

Correcting Minor Configuration Errors


Correcting Minor Configuration Errors

When you install the Sun Cluster software, some common mistakes are:

● Using the wrong cluster name

● Using incorrect cluster interconnect interface assignments

You can resolve these mistakes using the scsetup utility.

You can run the scsetup utility on any cluster node. It performs thefollowing tasks:

● Adds or removes quorum disks

● Adds, removes, enables, or disables cluster transport components

● Registers or unregisters VERITAS Volume Manager disk groups

● Adds or removes node access from a VERITAS Volume Managerdisk group

● Changes the cluster private host names

● Prevents or permits the addition of new nodes

● Changes the name of the cluster

Note – You should use the scsetup utility instead of manual commands,such as scconf . The scsetup utility is less prone to errors and, in somecases, performs complex verification before proceeding.

Exercise: Installing the Sun Cluster Server Software




● Configure environment variables

● Install the Sun Cluster server software

● Perform post-installation configuration

Preparation

Obtain the following information from your instructor if you have notalready done so in a previous exercise:

Ask your instructor about the location of the Sun Cluster 3.1software. Record the location.

Software location: _____________________________


Task 1 – Verifying the Boot Disk

Perform the following steps on all nodes to verify that the boot disks havea minimum 100-Mbyte /globaldevices partition and a small partitionfor use by Solstice DiskSuite software replicas.

1. Type the mount command, and record the logical path to the bootdisk on each node (typically /dev/dsk/c0t0d0 ).

Node 1 boot device: _______________

Node 2 boot device: _______________

2. Type the prtvtoc command to verify each boot disk meets the SunCluster software partitioning requirements.

# /usr/sbin/prtvtoc /dev/dsk/c0t0d0s2

Note – Append slice 2 (s2) to the device path. The sector count for the/globaldevices partition should be at least 200,000.



Task 2 – Verifying the Environment

Perform the following steps on all nodes:

1. Verify that the /.profile file on each cluster node contains thefollowing environment variables:

PATH=$PATH:/usr/cluster/bin

MANPATH=$MANPATH:/usr/cluster/man:/usr/share/man

TERM=dttermEDITOR=viexport PATH MANPATH TERM EDITOR

Note – If necessary, create the .profile login file as follows:cp /etc/skel/local.profile /.profile .

2. If you edit the file, verify the changes are correct by logging out andin again as user root .

3. On all nodes, create a .rhosts file in the root directory. Edit the file,and add a single line with a plus (+) sign.

4. On all cluster nodes, edit the /etc/default/login file andcomment out the CONSOLE=/dev/console line.

Note – The .rhosts and /etc/default/login file modifications shownhere can be a security risk in some environments. They are used here tosimplify some of the lab exercises.



Task 3 – Updating the Name Service

Perform the following steps to update the name service:

1. Edit the /etc/inet/hosts file on the administrative workstationand all cluster nodes, and add the IP addresses and host names ofthe administrative workstation and cluster nodes.

2. If you are using a naming service, add the IP addresses and hostnames to the name service.

Note – Your lab environment might already have all of the IP addressesand host names entered in the /etc/inet/hosts file.

Task 4a – Establishing a New Cluster – All Nodes

If you want to install all nodes in the cluster using default entries,perform the following installation. If you want to install each node inyour cluster individually, skip this task and begin with ‘‘Task 4b –Establishing a New Cluster – First Node’’ on page 4-45.

Perform the following steps to establish your cluster:

1. In the cconsole window, complete the following steps:

a. Log in to a node as user root .

This node will become the highest numbered node in thecluster.

b. Verify that you can log into each node without a password.

2. Change to the location of the Sun Cluster 3.1 software furnished byyour instructor.

3. Change to the Tools directory.

4. Start the scinstall script.

5. As the installation proceeds, make the following choices:

a. From the Main Menu, choose Option 1, Establish a new cluster.

b. From the Install Menu, choose Option 1, Install all nodes of anew cluster.

c. From the Type of Installation Menu, choose Option 1, Typical.



d. Furnish your assigned cluster name.

e. Enter the names of the other nodes in your cluster.

The first name you supply is Node 1, and so on.

f. Select the adapters that will be used for the cluster transport.

The cluster installation will now be performed on all nodes enteredpreviously. At the completion of the installation on each node, the node isrebooted. When all nodes have been completed, go to ‘‘Task 6 –Configuring a Quorum Device’’ on page 4-48.

Task 4b – Establishing a New Cluster – First Node

If you chose to install your cluster in “Task 4a – Establishing a NewCluster – All Nodes,” do not do this task. You should proceed to ‘‘Task 6 –Configuring a Quorum Device’’ on page 4-48 to complete the installationof your cluster.

Perform the following steps to establish the first node in your cluster:

1. In the cconsole window, log in to Node 1 as user root .



4. Start the scinstall script on Node 1 only.


a. Choose Option 1 from the Main Menu, Establish a new cluster.

b. Choose Option 2 from the Install Menu, Install just this machineas the first node of a new cluster.

c. Allow the framework packages to be added.

d. Add any required patches.

e. Furnish your assigned cluster name.

f. Allow sccheck to run.

g. Furnish the name of the second node that will be added later.

h. Verify the list of node names.

i. Reply no to using DES authentication.

j. Unless your instructor has stated otherwise, accept the defaultcluster transport base address and netmask values.



k. Configure the cluster transport based on your clusterconfiguration. Accept the default names for switches if yourconfiguration uses them.

l. Accept the default global device file system.

m. Reply yes to the automatic reboot question.

n. Examine the scinstall command options for correctness.Accept them if they seem appropriate.

You must wait for this node to complete rebooting to proceed tothe second node.

6. You may observe the following messages during the node reboot:

/usr/cluster/bin/scdidadm: Could not load DID instancelist.Cannot open /etc/cluster/ccr/did_instances.

Booting as part of a cluster

NOTICE: CMM: Node proto192(nodeid = 1) with votecount =1 added.NOTICE: CMM: Node proto192: attempting to join cluster.NOTICE: CMM: Cluster has reached quorum.NOTICE: CMM: Node proto192 (nodeid = 1) is up; newincarnation number = 975797536.NOTICE: CMM: Cluster members: proto192 .NOTICE: CMM: node reconfiguration #1 completed.NOTICE: CMM: Node proto192: joined cluster.

Configuring DID devices

Configuring the /dev/global directory (global devices)May 2 17:55:28 proto192 xntpd[338]: couldn’t resolve‘clusternode2-priv’, giving up on it



Task 5 – Adding Additional Nodes

Perform the following steps on each additional node in the cluster. If youhave more than two nodes, you could do all the other nodessimultaneously, although then it will be hard to predict which node getswhich node id. If you want this to be predictable, do one node at a time.

1. In the cconsole window, log in to the node as user root .



4. Start the scinstall script on the new node.


a. Choose Option 1 from the Main Menu.

b. Choose Option 2 from the Install Menu, Add this machine as anode in an established cluster.

c. Allow the packages to be installed and add the Sun Cluster 3.1software framework patches.

d. Provide the name of a sponsoring node.

e. Provide the name of the cluster you want to join.

Type scconf -p | more on the first node (the sponsoringnode) if you have forgotten the name of the cluster.

f. Use auto-discovery for the transport adapters.

g. Select the default for the global device directory.

h. Reply yes to the automatic reboot question.

i. Examine and approve the scinstall command-line options.

Note – You see interconnect-related errors on the existing nodes until thenew node completes the first portion of its reboot operation.



Task 6 – Configuring a Quorum Device

Perform the following steps to finish initializing your new cluster.

1. On either node, type the scstat -q command.

The cluster is still in install mode. Only the first node has a vote. Noquorum device has been assigned.

2. On Node 1, type the scdidadm -L command, and record the DIDdevices you intend to configure as a quorum disk. For example, ifyou had three nodes in a “Pair + 1” configuration, you would wanttwo quorum devices.

Quorum disks: _______ (d4, d6, and so on)

Note – Pay careful attention. The first few DID devices might be localdisks, such as the boot disk and a CD-ROM (target 6). Examine thestandard logical path to make sure the DID devices you select is a disk ina storage array and is connected to more than one nodes.

3. On Node 1, type the scsetup command, and supply the name ofthe DID device (global device) you selected in the previous step. Youshould see output similar to the following.

scconf -a -q globaldev=d12May 3 22:29:13 proto192 cl_runtime: NOTICE: CMM:Cluster members: proto192 proto193 .May 3 22:29:13 proto192 cl_runtime: NOTICE: CMM: nodereconfiguration #4 completed.

4. Do not add a second quorum disk on a two-node cluster. If you havethree nodes in a “Pair +N” configuration, for example, you shouldadd a second quorum device.

5. Reply yes to the reset installmode question.

A “Cluster initialization is complete” message is displayed.

Now that install mode has been reset and quorum devices defined,the scsetup utility shows its normal menu selections.

6. Type q to quit the scsetup utility.



Task 7 – Configuring the Network Time Protocol

Perform the following steps on all nodes to complete the NTPconfiguration:

1. On all nodes, edit the /etc/inet/ntp.conf.cluster file, andremove configuration entries for node instances that are notconfigured. In a two-node cluster, remove the following lines:

peer clusternode3-privpeer clusternode4-priv..peer clusternode16-priv

2. On all nodes, type the scstat -q command.

You should see three quorum votes present and a quorum device.

3. On all nodes, type the scdidadm -L command.

Each shared (dual-ported) DID device should show a logical pathfrom each cluster node.

4. On either node, type the scconf -p command.

The cluster status, node names, transport configuration, and quorumdevice information should be complete.



Task 8 – Testing Basic Cluster Operation

Perform the following steps to verify the basic cluster software operation:

Note – You are using a command that has not yet been presented in thecourse. If you have any questions, consult with your instructor.

1. Ensure that you are still connected to the console of all your clusternodes.

● If you are connected, go to the next step.

● If you are not connected, connect to all nodes with thecconsole clustername command.

2. Log in to one of your cluster host systems as user root .

3. On one node, issue the command to shut down all cluster nodes.

# scshutdown -y -g 15

Note – The scshutdown command completely shuts down all clusternodes, including the Solaris OS.

4. Ensure that all nodes of your cluster are at the OpenBoot™programmable read-only memory (PROM) ok prompt. Consult yourinstructor if this is not the case.

5. Boot each node in turn.

Each node should come up and join the cluster. Consult yourinstructor if this is not the case.

6. Verify the cluster status.

# scstat

Exercise Summary


Exercise Summary

?!


● Experiences

● Interpretations

● Conclusions

● Applications


Module 5

Performing Basic Cluster Administration

Objectives


● Identify the cluster daemons

● Verify cluster status from the command line

● Perform basic cluster startup and shutdown operations, includingbooting nodes in non-cluster mode and placing nodes in amaintenance state

● Describe the Sun Cluster administration utilities

Relevance


Relevance

?!


● What must be monitored in the Sun Cluster software environment?

● How current does the information need to be?

● How detailed does the information need to be?

● What types of information are available?


Performing Basic Cluster Administration 5-3Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1







● Sun Cluster 3.x Hardware Collection for Solaris Operating System(SPARC Platform Edition)

Identifying Cluster Daemons



When a cluster node is fully booted into cluster node, there are severalcluster daemons that will be added to the traditional Solaris OS.

root 4 0 0 03:52:15 ? 0:14 clusterroot 80 1 0 03:52:50 ? 0:00 /usr/cluster/lib/sc/clexecdroot 81 80 0 03:52:50 ? 0:00 /usr/cluster/lib/sc/clexecdroot 679 1 0 03:56:51 ? 0:00 /usr/cluster/lib/sc/cl_eventdroot 680 1 0 03:56:51 ? 0:00 /usr/cluster/lib/sc/rpc.fedroot 734 1 0 03:56:57 ? 0:00 /var/cluster/spm/bin/scguieventd -droot 661 1 0 03:56:48 ? 0:00 /usr/cluster/lib/sc/sparcv9/rpc.pmfdroot 694 1 0 03:56:53 ? 0:00 /usr/cluster/bin/pnmdroot 707 1 0 03:56:55 ? 0:00 /usr/cluster/lib/sc/rgmd

● cluster – This is a system process (created by the kernel) toencapsulate the kernel threads that make up the core kernel range ofoperations.

It directly panics the kernel if it is sent a KILL signal. Other signalshave no effect.

● clexecd – This is used by cluster kernel threads to executeuserland commands (such as the run_reserve and dofsckcommand). It is also used to run cluster commands remotely (likethe scshutdown command).

A failfast driver panics the kernel if this daemon is killed and notrestarted in 30 seconds.

● cl_eventd – This daemon registers and forwards cluster events(such as nodes entering and leaving the cluster). With a minimum ofSun Cluster 3.1 10/03 software, user applications can registerthemselves to receive cluster events.

The daemon automatically gets respawned by rpc.pmfd if it iskilled.

● rgmd – This is the resource group manager, which manages the stateof all cluster-unaware applications.


● rpc.fed – This is the “fork-and-exec” daemon, which handlesrequests from rgmd to spawn methods for specific data services.




● scguieventd – This daemon processes cluster events for the SunPlexManager graphical user interface (GUI), so that the display can beupdated in real time.

It is not automatically restarted if it stops. If you are having problemswith SunPlex Manager, you might need to restart this daemon orreboot the node in question.

● rpc.pmfd – This is the process monitoring facility. It is used as ageneral mechanism to initiate restarts and failure action scripts forsome cluster framework daemons, and for most applicationdaemons and application fault monitors.

A failfast driver panics the kernel if this daemon is stopped andnot restarted in 30 seconds.

● pnmd – This is the public network management daemon, whichmanages network status information received from the local IPMPrunning on each node and facilitates application failovers caused bycomplete public network failures on nodes.

It is automatically restarted by rpc.pmfd if it is stopped.

● scdpmd – A multithreaded DPM daemon runs on each node. TheDPM daemon is started by an rc x.d script when a node boots. TheDPM daemon monitors the availability of the logical path that isvisible through multipath drivers such as the Sun StorEdge TrafficManager software (formerly known as MPxIO), Hitachi DynamicLink Manager (HDLM), and EMC PowerPath.

It is automatically restarted by rpc.pmfd if it is stopped.

Using Cluster Status Commands



There are several cluster status commands. Some of the commands haveuses other than status reporting. Many of the commands, or specificoptions to the commands, must be run as the user root or with clusterRBAC authorizations. See Appendix C, “Role-Based Access ControlAuthorizations,” for more information about RBAC and theSun Cluster 3.1 RBAC authorizations.

Checking Status Using the scstat Command

The scstat utility displays the current state of the Sun Cluster 3.1software and its components. Run scstat on any one node. When runwithout any options, scstat displays the status for all components of thecluster. This display includes the following information:

● A list of cluster members

● The status of each cluster member

● The status of resource groups and resources

● The status of every path on the cluster interconnect

● The status of every disk device group

● The status of every quorum device

● The status of every Internet Protocol Network Multipathing groupand public network adapter

Viewing the Cluster Configuration

Display information about the overall cluster configuration with thescconf command. You do not need to be the user root to run this formof the command. For additional information, include the verbose options.

$ scconf -pCluster name: protoCluster ID: 0x3DC71034Cluster install mode: disabledCluster private net: 172.16.0.0Cluster private netmask: 255.255.0.0Cluster new node authentication: unixCluster new node list: <. - Exclude allnodes>



Cluster nodes: proto192 proto193

Cluster node name: proto192 Node ID: 1 Node enabled: yes Node private hostname: clusternode1-priv Node quorum vote count: 1 Node reservation key: 0x3DC7103400000001 Node transport adapters: qfe0 hme0

Node transport adapter: qfe0 Adapter enabled: yes Adapter transport type: dlpi Adapter property: device_name=qfe Adapter property: device_instance=0 Adapter property:dlpi_heartbeat_timeout=10000 Adapter property:dlpi_heartbeat_quantum=1000 Adapter property: nw_bandwidth=80 Adapter property: bandwidth=10 Adapter property:netmask=255.255.255.128 Adapter property: ip_address=172.16.1.1 Adapter port names: 0

Adapter port: 0 Port enabled: yes

Node transport adapter: hme0 Adapter enabled: yes Adapter transport type: dlpi Adapter property: device_name=hme Adapter property: device_instance=0 Adapter property: dlpi_heartbeat_timeout=10000 Adapter property: dlpi_heartbeat_quantum=1000 Adapter property: nw_bandwidth=80 Adapter property: bandwidth=10 Adapter property: netmask=255.255.255.128 Adapter property: ip_address=172.16.0.129 Adapter port names: 0

Adapter port: 0 Port enabled: yes..



.Information about the other node removed here for brevity..

Cluster transport cables

Endpoint Endpoint State -------- -------- ----- Transport cable: proto192:qfe0@0 switch1@1 Enabled Transport cable: proto192:hme0@0 switch2@1 Enabled Transport cable: proto193:qfe0@0 switch1@2 Enabled Transport cable: proto193:hme0@0 switch2@2 Enabled

Quorum devices: d19

Quorum device name: d19 Quorum device votes: 1 Quorum device enabled: yes Quorum device name: /dev/did/rdsk/d19s2 Quorum device hosts (enabled): proto192 proto193 Quorum device hosts (disabled): Quorum device access mode: scsi2

Validating a Basic Cluster Configuration

The sccheck utility examines Sun Cluster 3.1 nodes for knownvulnerabilities and configuration problems. It also delivers reports thatdescribe all failed checks, if any. The reports are placed into the/var/cluster/sccheck directory by default. Run the sccheck utilityany time configuration changes are performed.



The utility runs one of these two sets of checks, depending on the state ofthe node that issues the command:

● Pre-installation checks – When issued from a node that is notrunning as an active cluster member, the sccheck utility runspre-installation checks on that node. These checks ensure that thenode meets the minimum requirements to be successfully configuredwith Sun Cluster 3.1 software.

● Cluster configuration checks – When issued from an active memberof a running cluster, the sccheck utility runs configuration checks onthe specified or default set of nodes. These checks ensure that thecluster meets the basic configuration required for a cluster to befunctional. The sccheck utility produces the same results for this setof checks, regardless of which cluster node issues the command.

The sccheck utility runs configuration checks and uses theexplorer (1M) utility to gather system data for check processing. Thesccheck utility first runs single-node checks on each nodename specified,then runs multiple-node checks on the specified or default set of nodes.

The sccheck command runs in two steps: data collection and analysis.Data collection can be time consuming, depending on the systemconfiguration. You can invoke sccheck in verbose mode with the -v1flag to print progress messages, or you can use the -v2 flag to runsccheck in highly verbose mode, which prints more detailed progressmessages, especially during data collection.

Each configuration check produces a set of reports that are saved. Foreach specified node name, the sccheck utility produces a report of anysingle-node checks that failed on that node. Then, the node from whichsccheck was run produces an additional report for the multiple-nodechecks. Each report contains a summary that shows the total number ofchecks executed and the number of failures, grouped by check severitylevel.

Each report is produced in both ordinary text and in Extensible MarkupLanguage (XML). The reports are produced in English only.

Run the sccheck command with or without options. Running thecommand from an active cluster node without options causes it togenerate reports on all the active cluster nodes and to place the reportsinto the default reports directory.



Disk Path Monitoring

Disk path monitoring (DPM) administration commands enable you toreceive notification of secondary disk-path failure. The scdpm commandmanages the disk-path monitoring daemon,/usr/cluster/lib/sc/scdpmd , in the Sun Cluster 3.1 environment. Thiscommand is used to monitor and unmonitor disk paths. You can also usethe scdpm command to display the status of disk paths. All of theaccessible disk paths in the cluster or on a specific node are printed to thestandard output. The scdpm command must be run from a cluster nodethat is online in cluster mode. You can specify either a global name or aUNIX name when you monitor a new disk path. Additionally, you canforce the daemon to reread the entire disk configuration.

The disk path is represented by a node name and a disk name. The nodename must be the host name or the word all to address all of the nodesin the cluster. The disk name must be the global disk name, a UNIX pathname, or the word all to address all the disks in the node. The disk namecan be either the full global path name or just the disk name, for example,/dev/did/dsk/d3 or d3 . The disk name can also be the full UNIX pathname, for example, /dev/rdsk/c0t0d0s0 .

Disk path status changes are logged into /var/adm/messages with thesyslogd LOG_INFO facility level. All failures are logged by usingLOG_ERRfacility level.

The following command monitors a new disk path. In the followingexample, all nodes monitor /dev/did/dsk/d3 where this path is valid.

# scdpm -m /dev/did/dsk/d3

The following command monitors a new path on a single node. Thedaemon on the proto192 node monitors paths to the /dev/did/dsk/d4and /dev/did/dsk/d5 disks.

# scdpm -m proto192:d4 -m proto192:d5

The following command prints all disk paths in the cluster and theirstatus.

# scdpm -p all:all



Checking the Status Using the scinstall Utility

During the Sun Cluster software installation, the scinstall utility iscopied into the /usr/cluster/bin directory. You can run the scinstallutility with options that display the Sun Cluster software revision, thenames and revision of installed packages, or both. The displayedinformation is for the local node only. This is the only form of scinstallthat you can run as a non-root user. A typical scinstall status outputfollows.

# scinstall -pvSun Cluster 3.1u2 for Solaris 9 sparcSUNWscr: 3.1.0,REV=2003.03.25.13.14, 113801-06 115939-07 115364-05SUNWscu: 3.1.0,REV=2003.03.25.13.14, 113801-06 115939-07 115364-05SUNWscsck: 3.1.0,REV=2003.09.10.18.59SUNWscnm: 3.1.0,REV=2003.10.27.19.55SUNWscdev: 3.1.0,REV=2003.03.25.13.14, 113801-06 115939-07SUNWscgds: 3.1.0,REV=2003.03.25.13.14SUNWscman: 3.1.0,REV=2003.03.25.13.14, 113801-06 115939-07SUNWscsal: 3.1.0,REV=2003.03.25.13.14, 115939-07SUNWscsam: 3.1.0,REV=2003.03.25.13.14, 113801-06SUNWscvm: 3.1.0,REV=2003.03.25.13.14, 113801-06 115939-07SUNWmdm: 3.1.0,REV=2003.03.25.13.14, 115945-01SUNWscva: 3.1.0,REV=2003.03.25.13.14SUNWscvr: 3.1.0,REV=2003.03.25.13.14, 113801-06 115939-07 115364-05SUNWscvw: 3.1.0,REV=2003.03.25.13.14, 113801-06 115939-07 115364-05SUNWscrsm: 3.1.0,REV=2003.09.10.18.59, 115939-07SUNWfsc: 3.1.0,REV=2003.10.10.19.43SUNWfscvw: 3.1.0,REV=2003.10.10.19.43SUNWjsc: 3.1.0,REV=2003.10.10.19.59SUNWjscman: 3.1.0,REV=2003.10.10.19.59SUNWjscvw: 3.1.0,REV=2003.10.10.19.59SUNWksc: 3.1.0,REV=2003.10.10.19.56SUNWkscvw: 3.1.0,REV=2003.10.10.19.56SUNWcsc: 3.1.0,REV=2003.10.10.19.57SUNWcscvw: 3.1.0,REV=2003.10.10.19.57SUNWhsc: 3.1.0,REV=2003.10.10.19.58SUNWhscvw: 3.1.0,REV=2003.10.10.19.58SUNWexplo: 4.0, FCS

Caution – Use the scinstall utility carefully. It is possible to createserious cluster configuration errors using the scinstall utility.



Examining the /etc/cluster/release File

The /etc/cluster/release file gives you specific information about therelease of the Sun Cluster software framework that is installed on yournode:

# cat /etc/cluster/releaseSun Cluster 3.1u2 for Solaris 9 sparcCopyright 2003 Sun Microsystems, Inc. All Rights Reserved.

Controlling Clusters



Basic cluster control includes starting and stopping clustered operation onone or more nodes and booting nodes in non-cluster mode.

Starting and Stopping Cluster Nodes

The Sun Cluster software starts automatically during a system bootoperation. Use the init command to shut down a single node. You usethe scshutdown command to shut down all nodes in the cluster.

Before shutting down a node, you should switch resource groups to thenext preferred node and then run init 0 on the node.

Note – After an initial Sun Cluster software installation, there are noconfigured resource groups with which to be concerned.

Shutting Down a Cluster

You can shut down the entire cluster with the scshutdown commandfrom any active cluster node. A typical cluster shutdown example follows.

# scshutdown -y -g 30Broadcast Message from root (???) on proto192 Wed May 2017:43:17... The cluster scdev will be shutdown in 30 secondsMay 20 17:43:38 proto192 cl_runtime: WARNING: CMM:Monitoring disabled.INIT: New run level: 0The system is coming down. Please wait.System services are now being stopped./etc/rc0.d/K05initrgm: Calling scswitch -S (evacuate)Print services stopped.May 20 17:43:54 proto192 syslogd: going down on signal 15The system is down.syncing file systems... doneProgram terminatedok



Booting Nodes in Non-Cluster Mode

Occasionally, you might want to boot a node without it joining in aclustered operation. This is common when installing software patches.Some patches cannot be installed on an active cluster member. Anexample follows.

ok boot -xRebooting with command: boot -xBoot device:/pci@1f,4000/scsi@3/disk@0,0 File and args: -xSunOS Release 5.9 Version Generic_112233-07 64-bitCopyright 1983-2003 Sun Microsystems, Inc. All rightsreserved.configuring IPv4 interfaces: hme0.Hostname: proto192Not booting as part of a clusterThe system is coming up. Please wait.Starting IPv4 routing daemon.starting rpc services: rpcbind done.Setting netmask of hme0 to 255.255.255.0Setting default IPv4 interface for multicast: add net224.0/4: gateway proto193syslog service starting.Print services started.May 20 17:51:02 proto192 xntpd[195]: couldn’t resolve‘clusternode1-priv’, giving up on itMay 20 17:51:02 proto1922 xntpd[195]: couldn’t resolve‘clusternode2-priv’, giving up on itvolume management starting.The system is ready.

proto192 console login:

Other active cluster nodes display transport-related errors because theytry to establish a transport path to the node running in non-cluster mode.An example of typical errors on active nodes follows.

# May 4 21:09:42 proto192 cl_runtime: WARNING: Pathproto192:hme1 - proto193:hme1 initiation encounterederrors, errno = 62. Remote node may be down or unreachablethrough this path.May 4 21:09:42 proto192 cl_runtime: WARNING: Pathproto192:hme2 - proto193:hme2 initiation encounterederrors, errno = 62. Remote node may be down or unreachablethrough this path.



Placing Nodes in Maintenance Mode

If you anticipate a node will be down for an extended period, you canplace the node in a maintenance state from an active cluster node. Themaintenance state disables the node’s quorum vote. You cannot place anactive cluster member in a maintenance state. A typical command follows.

# scconf -c -q node=proto193,maintstate

The scstat command shows that the possible vote for proto193 is nowset to 0.

In addition, the vote count for any SCSI-2 (dual ported) quorum devicephysically attached to the node is also set to 0.

You can reset the maintenance state for a node by rebooting the node intothe cluster, or by running the command scconf -c -q reset . The nodeand any directly-attached SCSI-2 quorum devices regain their votes.

Maintenance Mode Example

To see the value of placing a node in maintenance state, consider thefollowing topology (shown in Figure 5-1), which was described inModule 3, “Preparing for Installation and Understanding QuorumDevices.”

Figure 5-1 Clustered-Pair Quorum Devices

Junction

Junction

Node 1(1)

Node 2(1)

Node 3(1)

Node 4(1)

QD(1) QD(1)

QD(1)

Sun StorEdgeMultiPack System



Now, imagine Nodes 3 and 4 are down. The cluster can still continue withNodes 1 and 2, because you still have four out the possible seven quorumvotes.

Now imagine you wanted to halt Node 1 and keep Node 2 running as theonly node in the cluster. If there were no such thing as maintenance state,this would be impossible. Node 2 would panic with only three out ofseven quorum votes. But if you put Nodes 3 and 4 into maintenance state,you reduce the total possible votes to three, and Node 2 can continue withtwo out of three votes.

Introducing Cluster Administration Utilities



There are three general approaches to performing cluster administrationtasks:

● Use the low-level administration commands

● Use the scsetup utility

● Use the SunPlex Manager web-based GUI

Introducing the Low-level Administration Commands

There are a surprisingly small number of commands used to administerthe cluster, but some of the commands have a large number of options.The commands are:

● scconf – For configuration of hardware and device groups

● scrgadm – For configuration of resource groups

● scswitch – For switching device groups and resource groups

● scstat – For printing the status of anything

● scdpm – For monitoring disk paths

As this course presents device groups and resource groups in thefollowing four modules, it will focus on the using the low-levelcommands for administration. Seeing and understanding all the properoptions to these commands is the best way to understand the capability ofthe cluster.

Using the scsetup Command

The scsetup command is a menu-driven utility meant to guide youthrough a dialogue, rather than you having to remember the options tothe commands listed in the previous section. The scsetup utility does allof its actual work by calling the previously listed commands. It alsoalways shows the lower-level commands as it runs them, so it haseducational value as well:

# scsetup*** Main Menu ***




1) Quorum 2) Resource groups 3) Data services 4) Cluster interconnect 5) Device groups and volumes 6) Private hostnames 7) New nodes 8) Other cluster properties

?) Help with menu options q) Quit

Option:

Comparing Low-level Command and scsetup Usage

There are many examples of the usage of the commands in the sectionsabout device groups and resource groups in the following four modules.

As a comparison, assume your task was to add a new cluster transport.You have a third private transport physically connected in a two-nodecluster, without switches, and you need to configure it into the cluster.

Note that the scsetup utility does not save you from needing theknowledge about “how things are done” in the Sun Cluster softwareenvironment. To configure a transport into Sun Cluster software without aswitch, you must:

1. Add a transport adapter definition on each node.

2. Add a transport cable definition between them.

Now, given that knowledge alone and without knowing any options tothe scconf command, you could easily follow the menus of the scsetupcommand to accomplish the task. You would choose Menu Option 4,Cluster interconnect, to see:

*** Cluster Interconnect Menu ***


1) Add a transport cable 2) Add a transport adapter to a node 3) Add a transport junction 4) Remove a transport cable



5) Remove a transport adapter from a node 6) Remove a transport junction 7) Enable a transport cable 8) Disable a transport cable


Option:

Menu Option 2 guides you through adding the adapters, and then MenuOption 1 guides you through adding the cables.

If you were to use the scconf command “by hand” to accomplish thesame tasks, the commands you would need to run would look like thefollowing:

# scconf -a -A trtype=dlpi,name=qfe0,node=proto192# scconf -a -A trtype=dlpi,name=qfe0,node=proto193# scconf -a -m endpoint=proto192:qfe0,endpoint=proto193:qfe0

These are the same commands eventually called by scsetup .

Introducing the SunPlex™ Manager

The SunPlex Manager is a web-based administrative tool for the SunCluster environment.

The server portions of SunPlex manager are automatically installed aspart of the scinstall of the Sun Cluster software framework. The clientis run in any web browser enabled with Java technology, from anywherethat can connect to the node (using the physical node name address).



Usage of the SunPlex Manager is much like the usage of scsetup in that:

● You still must understand the “nature” of the cluster tasks beforeaccomplishing anything.

● It saves you from having to remember options to the lower-levelcommands

● It accomplishes all its actions through the lower-level commands,and shows the lower-level commands as it runs them.

SunPlex Manager has the following additional benefits:

● You can use it to install the entire Sun Cluster software.

In that case, you would manually install the SunPlex Managerpackages on a server. The software would recognize that it wasrunning on a node with no other Sun Cluster software and guideyou through installing the cluster.

● It offers some high-level operations, such as building entire dataservice applications (LDAP, ORACLE, Apache, NFS) without havingto know the terminology of resources and resource groups.

● It offers some graphical “topology” views of the Sun Clustersoftware nodes with respect to device groups, resource groups, andthe like.

Many of the exercises in the modules following this one will offer you theopportunity to view and manage device and resource groups throughSunPlex Manager. However, this course will not focus on the details ofusing SunPlex Manager as the method of accomplishing the task. Likescsetup , after you understand the nature of a task, SunPlex Manager canguide you through it. This is frequently easier than researching andtyping all the options correctly to the lower level commands.

Logging In to SunPlex Manager

Use the following URL on a web browser enabled with Java technology toaccess SunPlex Manager. SunPlex Manager is available on a secure-socket(https) connection to the physical node name port 3000:

https:// nodename:3000



Figure 5-2 shows the login window. Cookies must be enabled on yourweb browser to allow SunPlex Manager login and usage.

Figure 5-2 SunPlex Manager Login Window

Log in with the user name of root and the password of root.

Alternatively, you can create a non-root user with thesolaris.cluster.admin authorization using RBAC. This user has fullcontrol of the cluster using SunPlex Manager.



Navigating the SunPlex Manager Main Window

SunPlex Manager navigation is simple, with the tree-like navigation baron the left, and the display corresponding to your current selection in themain frame.

Actions are selected from a context-sensitive pull-down menu that, whenappropriate, is at the upper-right of the main frame.

Topological views are selected from the View menu a the upper-left of themain frame.

Figure 5-3 shows the window after login.

Figure 5-3 SunPlex Manager Main Window



Viewing a SunPlex Manager Graphical TopologyExample

Figure 5-4 demonstrates one of the graphical topologies available inSunPlex Manager. This example shows the nodes and transport topology.

Figure 5-4 Graphic Topology Window

Exercise: Performing Basic Cluster Administration




● Perform basic cluster startup and shutdown operations

● Boot nodes in non-cluster mode

● Place nodes in a maintenance state

● Verify cluster status from the command line

Preparation

Join all nodes in the cluster, and run the cconsole tool on theadministration workstation.


Task 1 – Verifying Basic Cluster Status

Perform the following steps to verify the basic status of your cluster:

1. Use the scstat command to verify the current cluster membership.

# scstat -q

2. Record the quorum configuration from the previous step.

Quorum votes possible: _____

Quorum votes needed: _____

Quorum votes present: _____

3. Verify that the global device vfstab entries are correct on all nodes.

# sccheck -v

4. Monitor and verify all cluster disk paths.

# scdpm -m all:all# scdpm -p all:all

5. Verify the revision of the currently installed Sun Cluster software oneach cluster node.

# scinstall -pv



Task 2 – Starting and Stopping Cluster Nodes

Perform the following steps to start and stop configured cluster nodes:

1. Verify that all nodes are active cluster members.

2. Shut down Node 2.

# init 0

Note – You might see remote procedure call-related (RPC-related) errorsbecause the NFS server daemons, nfsd and mountd , are not running. Thisis normal if you are not sharing file systems in /etc/dfs/dfstab .

3. Join Node 2 into the cluster again by performing a boot operation.

4. When all nodes are members of the cluster, run scshutdown on onenode to shut down the entire cluster.

# scshutdown -y -g 60 Log off now!!

5. Join Node 1 into the cluster by performing a boot operation.

6. When Node 1 is in clustered operation again, verify the clusterquorum configuration again.

# scstat -q | grep “Quorum votes” Quorum votes possible: 3 Quorum votes needed: 2 Quorum votes present: 2

7. Leave other nodes down for now.

Task 3 – Placing a Node in Maintenance State

Perform the following steps to place a node in the maintenance state:

1. On Node 1, use the scconf command to place Node 2 into amaintenance state.

# scconf -c -q node= node2 ,maintstate

Note – Substitute the name of your node for node2 .

2. Verify the cluster quorum configuration again.

# scstat -q | grep “Quorum votes”



Note – The number of possible quorum votes should be reduced by two.The quorum disk drive vote is also removed.

3. Boot Node 2 again to reset its maintenance state.

The following message is displayed on both nodes:

NOTICE: CMM: Votecount changed from 0 to 1 for nodeproto193

4. Boot any additional nodes into the cluster.

5. Verify the cluster quorum configuration again. The number ofpossible quorum votes should be back to normal.

Task 4 – Booting Nodes in Non-Cluster Mode

Perform the following steps to boot a cluster node so that it does notparticipate in clustered operation:


2. Boot Node 2 in non-cluster mode.

ok boot -x

Note – You should see a message similar to: Not booting as part ofa cluster . You can also add the single-user mode option: boot -sx .

3. Verify the quorum status again.

4. Return Node 2 to clustered operation.

# init 6



Task 5 – Navigating SunPlex Manager

1. In a web browser window on your workstation, type the followingURL:


where nodename is the name of any of your nodes currently bootedinto the cluster.

Note – If you do not succeed in reaching the SunPlex Manager, you mightneed to disable or set exceptions for the proxy settings in your webbrowser. The node name and IP address must be in the name service, orthe connection will not be allowed. Verify the server portion has startedon the cluster node (pgrep -l http ). This server requires a fullyqualified domain name (FQDN) in /etc/inet/hosts that matches/etc/defaultdomain to start. If FQDN has not been set, set it, and thenrun /etc/init.d/initspm start . You might also want to set yourbrowser to accept cookies without prompting you. Consult yourinstructor if you need help with this.

2. Log in as the user root with the root password.

3. Familiarize yourself with SunPlex Manager navigation and thetopological views.

Exercise Summary


Exercise Summary

?!


● Experiences

● Interpretations

● Conclusions

● Applications


Module 6

Using VERITAS Volume Manager forVolume Management

Objectives


● Describe the most important concepts of VERITAS Volume Manager(VxVM) and issues involved in using VxVM in the Sun Cluster 3.1software environment

● Differentiate between rootdg and shared storage disk groups

● Explain VxVM disk groups

● Describe the basic objects in a disk group

● Describe the types of volumes that will be created for Sun Cluster 3.1software environments

● Describe the general procedures for installing and administeringVxVM in the cluster

● Describe the DMP restrictions

● Install and initialize VxVM using the scvxinstall command

● Use basic commands to put disks in disk groups and build volumes

● Describe the two flags used to mark disks under the normal hotrelocation

● Register VxVM disk groups with the cluster

● Resynchronize VxVM disk groups with the cluster

● Create global and local file systems

● Describe the procedure used for mirroring the boot drive

● Describe disk repair and replacement issues

Relevance


Relevance

?!


● Which VxVM features are the most important to clustered systems?

● Are there any VxVM feature restrictions when VxVM is used in theSun Cluster software environment?


Using VERITAS Volume Manager for Volume Management 6-3Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1




● Sun Cluster 3.1 4/04 Hardware Collection for Solaris OS (SPARCPlatform Edition)

● VERITAS Volume Manager™ 3.5 Administrator's Guide (Solaris)

● VERITAS Volume Manager™ 3.5 Installation Guide (Solaris)

● VERITAS Volume Manager™ 3.5 Release Notes (Solaris)

● VERITAS Volume Manager™ 3.5 Hardware Notes (Solaris)

● VERITAS Volume Manager™ 3.5 Troubleshooting Guide (Solaris)

Introducing VxVM in the Sun Cluster Software Environment


Introducing VxVM in the Sun Cluster SoftwareEnvironment

This module does not intend to replace a full four-day class in VxVMadministration. Rather, it briefly introduces the most important conceptsof VxVM and focuses on issues involved in using VxVM in theSun Cluster 3.1 environment.

Only the following versions of VxVM are supported in Sun Cluster 3.1software:

● VxVM 3.2, for Solaris 8 OS (Update 7) only

● VxVM 3.5, for Solaris 8 OS (Update 7) and Solaris 9 OS

Exploring VxVM Disk Groups and the rootdg Disk Group



The assignment of disk drives or logical unit numbers (LUNs) into diskgroups is the most important organizational concept to understand tomanage VxVM in the cluster environment.

Every disk or LUN used by VxVM must be a member of exactly one diskgroup. Disks in the same disk group act as an organizational unit.

Shared Storage Disk Groups

All of the data associated with applications that run in the Sun Cluster 3.1software environment must be in storage that is physically ported to atleast two nodes. Disk groups are created using only disks in multiplearrays that are connected to the same collection of nodes.

The rootdg Disk Group on Each Node

VxVM requires that each node have an independent disk group namedrootdg . You will not be able to configure anything else in VxVM untilrootdg is configured. Figure 6-1 shows a typical organization of disks intodisk groups in a simple Sun Cluster software environment.

Figure 6-1 Typical Organization of Disks in a Sun Cluster SoftwareEnvironment

Boot Disks(rootdg)

Node 1

Boot Disks(rootdg)

Node 2

Diskgroup webdgDiskgroup nfsdg



Shared Storage Disk Group Ownership

While the disks in disk groups used for Sun Cluster software applicationdata must be physically connected to at least two nodes, disk groups areowned, or imported, by only one node at a time. The node that is currentlyimporting a disk group will be the one that:

● Physically reads and writes data to the drives of the disk group.

● Manages the disk group at that time. VxVM commands pertaining tothat diskgroup can be issued only from the node importing the diskgroup.

● Can voluntarily give up ownership of the disk group by deporting thedisk group.

Sun Cluster Management of Disk Groups

Registering VxVM disk groups so that the Sun Cluster software hasknowledge of them is described in‘‘Registering VxVM Disk Groups’’ onpage 6-30.

After a disk group is managed by the cluster, that cluster is responsible forissuing all VxVM commands to import and deport disk groups. The nodecurrently importing the disk group becomes the primary device groupserver.

Sun Cluster Global Devices in Disk Group

While the fact that only one node imports a disk group remains true in theSun Cluster software environment, Sun Cluster software’s global deviceinfrastructure will make it appear that devices in the disk group areaccessible from all nodes in the cluster, including nodes that are not evenphysically connected to the disks in the disk group.

All nodes that are not the current primary for a particular disk group,even other nodes that are physically connected to the disks, actually accessthe device data through the cluster transport.



VxVM Cluster Feature Used Only for ORACLE®

Parallel Server and ORACLE Real Application Cluster

You may see references throughout the VERITAS documentation to aCluster Volume Manager (CVM) feature, and creating disk groups with aspecific shared flag.

This is a separately licensed feature of VxVM that allows simultaneousownership of a disk group by multiple nodes simultaneously.

In the Sun Cluster 3.1 software, this VxVM feature can be used only by theORACLE Parallel Server (OPS) and ORACLE Real Application Cluster(RAC) application.

Do not confuse the usage of shared storage disk group throughout thismodule with the CVM feature of VxVM.

Initializing the VERITAS Volume Manager Disk


Initializing the VERITAS Volume Manager Disk

Before a disk can be put into a VxVM disk group, it must be initialized byVxVM. The disk is divided into two sections called the private region andthe public region.

The private and public regions are used for different purposes.

● The private region is used for configuration information.

● The public region is used for data storage.

As shown in Figure 6-2, the private region is small. It is usuallyconfigured as slice 3 on the disk and is, at most, a few cylinders in size.

Figure 6-2 VxVM Disk Initialization

The public region is the rest of the disk drive. It is usually configured asslice 4 on the disk.

While initializing a disk and putting it in a disk group are two separateoperations, the vxdiskadd utility, demonstrated later in this module, canperform both steps for you.

Configuration andManagement Information

Public region

Private region

Data storage

Reviewing the Basic Objects in a Disk Group



This section reviews some of the basic naming terminology of objects indisk groups. Many of the objects (such as subdisks and plexes) are oftenautomatically created for you when you use the recommended commands(such as vxassist ) to create volumes.

Although the GUI for VxVM furnishes useful visual status information,the most reliable and the quickest method of checking status is from thecommand line. Command line status tools are easy to use in script files,cron jobs, and remote logins.

Disk Names or Media Names

Each disk that you put in a diskgroup has a logical name that you canassign. This logical name can be anything, and is independent of itsSolaris OS logical device name (c#t#d# ).

VxVM does not use the Sun Cluster Disk IDs (DIDs). Its own logicalnaming scheme serves a similar purpose.

Subdisk

A subdisk is a contiguous piece of a physical disk that is used as abuilding block for a volume. The smallest possible subdisk is one blockand the biggest is the whole public region of the disk.

Plex

A plex, or data plex, has the following characteristics:

● A plex is a copy of the data in the volume

● A volume with one data plex is not mirrored

● A volume with two data plexes is mirrored

● A volume with five data plexes is mirrored five ways



Volume

The volume is the actual logical storage device created and used by alldisk consumers (for example, file system, swap space, and ORACLE rawspace) above the volume management layer.

Only volumes are given block and character device files.

The basic volume creation method (vxassist ) lets you specify parametersfor your volume. This method automatically creates the subdisks andplexes for you first.

Layered Volume

A layered volume is a technique used by VxVM that lets you createRAID 1+0 configuration. For example, you can create two mirroredvolumes, and then use those as components in a “larger” striped volume.

While this sounds complicated, the automation involved in therecommended vxassist command makes such configurations easy tocreate.

Exploring Volume Requirements in the Sun Cluster Software Environment


Exploring Volume Requirements in the Sun ClusterSoftware Environment

The only absolute requirement for creating a volume in shared storagedisk groups for the Sun Cluster software environment is that you mustmirror across controllers.

With that in mind, any of the following are acceptable volumes to holdyour Sun Cluster software data.

Simple Mirrors

Figure 6-3 demonstrates subdisks from disks in different storage arraysforming the plexes of a mirrored volume:

Figure 6-3 Forming the Plexes of a Mirrored Volume

c1t0d0disk01

c2t0d0disk02disk01-01 disk02-01

plex1 plex2

volume nfsvol



Mirrored Stripe (“Mirror-Stripe”)

Figure 6-4 demonstrates subdisks from disks in the same array stripedtogether. This is mirrored with a configuration similar to the other array.

Figure 6-4 Mirror-Stripe Volume

c1t0d0(disk01)

c2t0d0(disk02)

c1t1d0(disk03)

c2t1d0(disk04)

disk01-01 disk02-01

plex1 plex2

disk03-01 disk04-01

volume nfsvol



Striped Mirrors (“Stripe-Mirror”)

Figure 6-5 shows two mirrored sub-volumes, both mirrored across storagearrays, striped together to form the final volume.

Figure 6-5 Stripe-Mirror Volume

Fortunately, as demonstrated later in the module, this is easy to create.

Often this configuration (RAID 1+0) is preferred because there might beless data to resynchronize after disk failure, and you could suffersimultaneous failures in each storage array. For example, you could loseboth disk01 and disk04 , and your volume is still available.

c1t0d0disk01

c2t0d0disk02

c1t1d0disk03

c2t1d0disk04single striped plex

disk01-01 disk02-01

plex1 plex2

plex1 plex2

disk03-01 disk04-01

subvolume

subvolume



Dirty Region Logs for Volumes in the Cluster

Dirty region log (DRL) is an optional feature you can add on aper-volume basis. The DRL is an extra plex for the volume that does nothold data but rather holds bitmaps of which regions of the data may be“dirty.” That is, they may have to be resynchronized in the volume after aSolaris OS crash.

DRLs have no effect whatsoever on resynchronization behavior followingdisk failure.

Without a DRL, a mirrored volume must be completely resynchronizedafter a Solaris OS crash. That is because on recovery (on another node, inthe cluster environment), VxVM has no way of knowing which bits ofdata it may have been “in the middle of writing” when the crashoccurred.

With the DRL, VxVM knows exactly which regions of the data marked inthe DRL bitmap may need to be resynchronized. Regions not marked inthe bitmap are known to be clean, and do not need to be resynchronized.

You use a DRL for any large volume in the cluster. The whole purpose ofthe cluster is making everything behave as well as possible after crashes.

Size of DRL

DRLs are small. Use the vxassist command to size them for you. Thesize of the logs that the command chooses depends on the version ofVxVM, but might be, for example, 26 Kbytes of DRL per 1 Gbyte ofstorage.

If you grow a volume, just delete the DRL, and then use the vxassistcommand to add it back again so that it gets sized correctly.

Location of DRL

You can put DRL subdisks on the same disk as the data if the volume willbe “read mostly.” An example of this is web data.

For a heavily written volume, performance is better with a DRL on adifferent disk than the data. For example, you could dedicate one disk toholding all the DRLs for all the volumes in the group. They are so smallthat performance on this one disk is still fine.

Viewing the Installation and rootdg Requirements in the Sun Cluster Software


Viewing the Installation and rootdg Requirements in theSun Cluster Software Environment

The rest of this module is dedicated to general procedures for installingand administering VxVM in the cluster. It describes both easy, automaticways, and more difficult, manual ways for meeting the requirementsdescribed in the following list.

● The vxio major number identical on all nodes – This is the entry inthe /etc/name_to_major file for the vxio device driver. If the majornumber is not the same on all nodes, the global device infrastructurecannot work on top of VxVM.

● VxVM installed on all nodes physically connected to sharedstorage – On non-storage nodes, you can install VxVM if you willuse it to encapsulate and mirror the boot disk.

If not (for example, you may be using Solaris Volume Managersoftware to mirror root storage), then a non-storage node requiresonly that the vxio major number be added to the/etc/name_to_major file.

● License required on all nodes not attached to Sun StorEdge A5x00array – This includes nodes not physically attached to any sharedstorage, where you just want to use VxVM to mirror boot storage.

● Standard rootdg created on all nodes where VxVM is installed – Theoptions to initialize rootdg on each node are:

● Encapsulate the boot disk so it can be mirrored – Encapsulatingthe boot disk preserves its data, creating volumes in rootdg foryour root disk partitions, including/global/.devices/node@#.

You cannot encapsulate the boot disk if it has more than fiveslices on it already. VxVM needs two free slice numbers tocreate the private and public regions of the boot disk.

● Initialize other local disks into rootdg – These could be used tohold other local data, such as application binaries.

● Unique volume name and minor number across nodes for the/global/.devices/node@# files system if boot disk isencapsulated – The /global/.devices/node@# files system must beon devices with a unique name on each node, because it is mountedas a global file system. It also must have a different minor number oneach node for the same reason. The normal Solaris OS /etc/mnttablogic predates global file systems and still demands that each deviceand major/minor combination be unique.

Viewing the Installation and rootdg Requirements in the Sun Cluster Software


VxVM cannot change the minor number of a single volume, but doessupport changing the minor numbers of all volumes in a disk group.You will have to perform (either automatically or manually) this“reminor” operation on the whole rootdg .

Describing DMP Restrictions



The DMP driver is a VxVM product feature. As shown in Figure 6-6, theDMP driver can access the same storage array through more than onepath. The DMP driver automatically configures multiple paths to thestorage array if they exist. Depending on the storage array model, thepaths are used either simultaneously for load-balancing or one at a time ina primary/backup configuration.

Figure 6-6 Dynamic Multipath Driver

The Sun Cluster software environment is incompatible with multipatheddevices under control of DMP device driver. In VxVM versions 3.1 andearlier (which are not supported by Sun Cluster 3.1 software), DMP couldbe permanently disabled by laying down some dummy symbolic linksbefore installing the VxVM software. You will notice in examples later inthis module that the scvxinstall utility actually still tries to do this,although it does not have any effect on VxVM versions actually supportedin the Sun Cluster 3.1 software.

Host system

Storage array

Controller

Controller

SOCCard C1

SOCCard C2

DMPDriver

Fiber-opticinterface

Driv

eD

rive

Driv

eD

rive

Driv

eD

rive



DMP Restrictions in Sun Cluster 3.1 (VxVM 3.2and 3.5)

In versions of VxVM supported by Sun Cluster 3.1 (VxVM 3.2 and 3.5),you cannot permanently disable the DMP feature. Even if you take stepsto disable DMP, it automatically re-enables itself each time the system isbooted.

Having multiple paths from the same node to the same storage undercontrol of VxVM DMP is still not supported. You can do one of thefollowing:

● Do not have any multiple paths at all from a node to the samestorage

● Have multiple paths from a node under the control of Sun StorEdgeTraffic Manager software or EMC PowerPath software



Supported Multipathing Under Sun StorEdge Traffic ManagerSoftware

Multipathing is supported using Sun StorEdge Traffic Manager software,formerly known as mpxio . Volume Manager detects Sun StorEdge TrafficManager software devices and does not use DMP on those paths.

Figure 6-7 shows an example of multipathing using Sun StorEdge T3array partner pairs.

Figure 6-7 Example of Multipathing Using Sun StorEdge T3 ArrayPartner-Pair

Sun StorEdge T3 System

Data

Sun StorEdge T3 System

Mirror

FC Hub/Switch FC Hub/Switch

HBA 2

0 1 0 1

HBA 1 HBA 2

0 1 0 1

HBA 1

Installing VxVM in the Sun Cluster 3.1 Software Environment


Installing VxVM in the Sun Cluster 3.1 SoftwareEnvironment

The following sections describe the following:

● Using the scvxinstall utility to install VxVM after the cluster isinstalled

● Installing VxVM manually after the cluster is installed

● Addressing VxVM cluster-specific issues if you installed VxVMbefore the cluster was installed

Describing the scvxinstall Utility

The scvxinstall utility is a shell script included with the cluster thatautomates VxVM installation in an environment where the SunCluster 3.1 environment is already running.

Wait to install VxVM until after the cluster is operational. Usescvxinstall to do it. This utility automates all of the following tasks:

● It tries to disable DMP. (As previously mentioned, this is ineffectiveon the VxVM versions supported in Sun Cluster 3.1 software.)

● It installs the correct cluster packages.

● It automatically negotiates a vxio major number and properly editsthe /etc/name_to_major file.

● If you choose to encapsulate the boot disk, it performs the wholerootdg initialization process.

● It gives different device names for the/global/.devices/node@# volumes on each node.

● It edits the vfstab file properly for this same volume. Theproblem is that this particular line currently has a DID deviceon it, and the normal VxVM does not understand DID devices.

● It installs a script to “reminor” the rootdg on the reboot

● It gets you rebooted so VxVM on your node is fully operational

● If you choose not to encapsulate the boot disk, it exits and lets yourun vxinstall yourself

● You may also choose to run pkgadd manually to install clusterpackages and then run scvxinstall . This action lets you chooseyour packages and lets you add patches immediately, if you need to.



Examining an scvxinstall Example

The following is sample output from running the scvxinstall utility:

proto193:/# scvxinstallDo you want Volume Manager to encapsulate root [no]? yes

Where is the Volume Manager cdrom? /net/proto196/jumpstart/vxvm3.5

Disabling DMP.Installing packages from /net/proto196/jumpstart/vxvm3.5.Installing VRTSvlic.Installing VRTSvxvm.Installing VRTSvmman.Obtaining the clusterwide vxio number...Using 240 as the vxio major number.Volume Manager installation is complete.Dec 12 17:28:17 proto193 vxdmp: NOTICE: vxvm:vxdmp: added disk arrayOTHER_DISKS, datype = OTHER_DISKS..Please enter a Volume Manager license key [none]: RRPH-PKC3-DRPC-DCUP-PRPP-PPO6-P6Verifying encapsulation requirements.

The Volume Manager root disk encapsulation step will begin in 20 seconds.Type Ctrl-C to abort ....................Arranging for Volume Manager encapsulation of the root disk.The vxconfigd daemon has been started and is in disabled mode...Reinitializied the volboot file...Created the rootdg...Added the rootdisk to the rootdg...The setup to encapsulate rootdisk is complete...The /etc/vfstab file already includes the necessary updates.

This node will be re-booted in 20 seconds.Type Ctrl-C to abort ....................Rebooting with command: boot -x

The node reboots two times. The first reboot completes the VERITAS bootdisk encapsulation process, and the second reboot brings the node up inclustered operation again. After the configuration of the first node hascompleted, you run the scvxinstall utility on the next node.



Installing and Initializing VxVM Manually

Use the scvxinstall utility to install and initialize VxVM in analready-clustered environment, especially if you are encapsulating yourboot disk.

Manually Installing VxVM Software

The following are the only VxVM packages required on all the nodes:

● VRTSvxvm

● VRTSvlic (version 3.5) or VRTSlic (version 3.2)

● VRTSvmdev(version 3.2 only)

You are also free to install the packages for the Volume Manager SystemAdministrator GUI (VMSA, version 3.2) or Volume Manager EnterpriseAdministrator (VEA, version 3.5).

Manually Checking and Fixing vxio Major Numbers

Immediately after software installation, you must check on all the nodesand make sure the major number for vxio is identical across all of them:

# grep vxio /etc/name_to_majorvxio 240

If it is not the same on all nodes, pick a number to which to change it thatdoes not appear anywhere else in any node’s file already. You might haveto edit the file on one or more nodes.

Manually Using vxinstall to Encapsulate Boot

Use the following steps to make boot-disk encapsulation work correctlyon the cluster:

1. Make sure you select Custom Install.

2. Give a different name for the root disk on each node. Do not use thedefault name rootdisk on each node.

The result is that the volume name for the/global/.devices/node@# file system is different on each node.

3. Make sure you do not enter any shared disk storage drives in therootdg .



4. Do not let the vxinstall utility reboot for you.

5. Before you reboot manually, put back the normal/dev/dsk/c #t #d#s# and /dev/rdsk/c #t #d#s# on the line in the/etc/vfstab file for the /global/.devices/node@ # file system.

Without making this change, VxVM cannot figure out that it mustedit that line after the reboot to put in the VxVM volume name.

6. Do a manual reboot with boot -x . VxVM. It arranges a secondreboot after that which will bring you back into the cluster.

7. That second reboot will fail on all but one of the nodes because of theconflict in minor numbers for rootdg (which is really just for the/global/.devices/node@# volume).

On those nodes, give the root password to go into single user mode.Use the following command to fix the problem:

# vxdg reminor rootdg 100*nodeid

For example, use 100 for Node 1, 200 for Node 2, and so forth. Thiswill provide a unique set of minor numbers for each node.

8. Reboot the nodes for which you had to repair the minor numbers.

Configuring a Pre-existing VxVM for Sun Cluster 3.1Software

The last possibility is that you have a potential Sun Cluster 3.1 softwarenode where the proper version of VxVM was already installed andinitialized before the Sun Cluster environment was installed. The followingsubsections detail how to provision your VxVM to deal with thecluster-specific issues mentioned on the previous pages. They assume youwill deal with these issues before you install the Sun Cluster software.

Fixing the vxio Major Number

If you need to change the vxio major number to make it agree with theother nodes, do the following:

1. Manually edit the /etc/name_to_major file.

2. Unmirror and unencapsulate the root disk, if it is encapsulated.

3. Reboot.

4. Reencapsulate and remirror the boot disk, if desired.



Fixing the /globaldevices Volume Name and Minor Number

If you are still at a point before cluster installation, you need to have a/globaldevices placeholder file system or at least a correctly sizeddevice ready for cluster installation.

If this is on a volume manager device:

1. Rename the volume if it has the same name as the /globaldevicesor /global/.devices/node@# volume of any other node. Youmight need to manually edit the /etc/vfstab file.

2. Perform a reminor operation on the rootdg if it is using the same setof minor numbers as another node. Follow the same procedure asStep 7 on page 6-23.

Creating Shared Disk Groups and Volumes



The examples in this section are not trying to exhaustively cover all thepossible ways of initializing disks and creating them in volumes. Rather,these are demonstrations of some simple ways of initializing disks,populating disk groups, and creating the mirrored configurationsdescribed earlier in the module.

Listing Available Disks

The following command lists all of the disks visible to VxVM:

# vxdisk listDEVICE TYPE DISK GROUP STATUSc0t0d0s2 sliced rootdisk rootdg onlinec0t8d0s2 sliced - - errorc1t0d0s2 sliced - - errorc1t1d0s2 sliced - - errorc1t2d0s2 sliced - - errorc2t0d0s2 sliced - - errorc2t1d0s2 sliced - - errorc2t2d0s2 sliced - - error

Note – Disk group names are listed only for disk groups imported on thisnode, even after disk groups are under Sun Cluster software control.

A disk in an error state is likely not yet initialized for VxVM use.

Initializing Disks and Putting Them in a New DiskGroup

One of the simplest ways to do this is using the dialogue presented by thevxdiskadd command. It guides you through initializing disks and puttingthem in existing or new groups.

# vxdiskadd c1t0d0 c1t1d0 c2t0d0 c2t1d0..



Verifying Disk Groups Imported on a Node

The following command shows which groups are imported:

# vxdg listNAME STATE IDrootdg enabled 1039669868.1025.proto192nfsdg enabled 1039815054.1153.proto192

Note – The ID of a disk group contains the name of the node on which itwas created. Later you could see the exact same disk group with the sameID imported on a different node.

# vxdisk listDEVICE TYPE DISK GROUP STATUSc0t0d0s2 sliced rootdisk rootdg onlinec0t8d0s2 sliced - - errorc1t0d0s2 sliced nfs1 nfsdg onlinec1t1d0s2 sliced nfs3 nfsdg onlinec1t2d0s2 sliced - - errorc2t0d0s2 sliced nfs2 nfsdg onlinec2t1d0s2 sliced nfs4 nfsdg onlinec2t2d0s2 sliced - - error

The following shows printing out the status of an entire group where novolumes have been created yet:

# vxprint -g nfsdgTY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0dg nfsdg nfsdg - - - - - -

dm nfs1 c1t0d0s2 - 71118513 - - - -dm nfs2 c2t0d0s2 - 71118513 - - - -dm nfs3 c1t1d0s2 - 71118513 - - - -dm nfs4 c2t1d0s2 - 71118513 - - - -



Building a Simple Mirrored Volume

The following example shows building a simple mirror, with a subdiskfrom a disk in one controller mirrored with one from another controller.

# vxassist -g nfsdg make nfsvol 100m layout=mirror nfs1 nfs2# vxprint -g nfsdgTY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0dg nfsdg nfsdg - - - - - -


v nfsvol fsgen ENABLED 204800 - ACTIVE - -pl nfsvol-01 nfsvol ENABLED 205119 - ACTIVE - -sd nfs1-01 nfsvol-01 ENABLED 205119 0 - - -pl nfsvol-02 nfsvol ENABLED 205119 - ACTIVE - -sd nfs2-01 nfsvol-02 ENABLED 205119 0 - - -

Building a Mirrored Striped Volume (RAID 0+1)

The following example rebuilds the mirror. Each plex is striped betweentwo disks in the same array so mirroring is still across controllers.

# vxassist remove volume nfsvol# vxassist -g nfsdg make nfsvol 100m layout=mirror-stripe mirror=ctlr# vxprint -g nfsdgTY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0dg nfsdg nfsdg - - - - - -


v nfsvol fsgen ENABLED 204800 - ACTIVE - -pl nfsvol-01 nfsvol ENABLED 208068 - ACTIVE - -sd nfs1-01 nfsvol-01 ENABLED 104004 0 - - -sd nfs3-01 nfsvol-01 ENABLED 104004 0 - - -pl nfsvol-02 nfsvol ENABLED 208068 - ACTIVE - -sd nfs2-01 nfsvol-02 ENABLED 104004 0 - - -sd nfs4-01 nfsvol-02 ENABLED 104004 0 - - -



Building a Striped Mirrored Volume (RAID 1+0)

The final example rebuilds the volume as a stripe of two mirrors. Asmentioned earlier in this module, this layout is often preferred for fasterresynchronization on disk failure and greater resiliency.

# vxassist remove volume nfsvol# vxassist -g nfsdg make nfsvol 100m layout=stripe-mirror mirror=ctlr# vxprint -g nfsdgTY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0dg nfsdg nfsdg - - - - - -


v nfsvol fsgen ENABLED 204800 - ACTIVE - -pl nfsvol-03 nfsvol ENABLED 204800 - ACTIVE - -sv nfsvol-S01 nfsvol-03 ENABLED 102400 0 - - -sv nfsvol-S02 nfsvol-03 ENABLED 102400 0 - - -

v nfsvol-L01 fsgen ENABLED 102400 - ACTIVE - -pl nfsvol-P01 nfsvol-L01 ENABLED 102400 - ACTIVE - -sd nfs1-02 nfsvol-P01 ENABLED 102400 0 - - -pl nfsvol-P02 nfsvol-L01 ENABLED 102400 - ACTIVE - -sd nfs2-02 nfsvol-P02 ENABLED 102400 0 - - -v nfsvol-L02 fsgen ENABLED 102400 - ACTIVE - -pl nfsvol-P03 nfsvol-L02 ENABLED 102400 - ACTIVE - -sd nfs3-02 nfsvol-P03 ENABLED 102400 0 - - -pl nfsvol-P04 nfsvol-L02 ENABLED 102400 - ACTIVE - -sd nfs4-02 nfsvol-P04 ENABLED 102400 0 - - -

Examining Hot Relocation


Examining Hot Relocation

Hot relocation is controlled by the vxrelocd daemon, which is started bydefault after you initialize volume manager.

When a mirrored volume breaks under hot relocation, VxVM looks forspace to substitute for the broken half of the mirror. If one large diskbreaks, VxVM could concatenate together ten new subdisks from otherdisks in the diskgroup to recreate the broken plex. It will never use anydisk space from disks in the surviving plex.

The SPAREand NOHOTUSEFlags

Under the normal hot relocation, disks can be marked with one of twoflags, but not both:

● SPAREflag

Disks marked with this flag are preferred as disk space to use for hotrelocation. These disks can still be used to build normal volumes.

● NOHOTUSEflag

Disks marked with this flag are excluded from consideration for usefor hot relocation.

In the following example, disk nfs2 is set as a preferred disk for hotrelocation, and disk nfs1 is excluded from hot relocation usage:

# vxedit -g nfsdg set spare=on nfs2# vxedit -g nfsdg set nohotuse=on nfs1

The flag settings are visible in the output of the vxdisk list andvxprint commands.

If the new plex is being used in a volume with specific applicationownership credentials, you might need to complete this action manuallyafter the relocation has occurred.

Registering VxVM Disk Groups



After you create a new VxVM disk group and volumes, you mustmanually register the disk group using either the scsetup utility or thescconf command for the disk group to be managed by the cluster. Thescsetup utility is recommended.

When a VxVM disk group is registered in the Sun Cluster softwareenvironment, it is referred to as a device group.

Until a VxVM disk group is registered, the cluster does not detect it.While you can build volumes and perform all normal VxVMadministration tasks, you are not able to use the volume manager devicesin the cluster.

If you change any configuration information for a VxVM disk group orvolume that is registered with the cluster, you must reregister the diskdevice group by using scsetup . Such configuration changes includeadding or removing volumes, as well as changing the group, owner, orpermissions of existing volumes. Reregistration after configurationchanges ensures that the global namespace is in the correct state.

The following example shows a disk group that is known to VxVM, butnot yet known to the cluster.

# vxdg listNAME STATE IDrootdg enabled 1039669868.1025.proto192nfsdg enabled 1039815054.1153.proto192

# scstat -D

-- Device Group Servers --

Device Group Primary Secondary ------------ ------- ---------

-- Device Group Status --

Device Group Status ------------ ------



Using the scconf Command to Register Disk Groups

The following command registers a VxVM disk group as a cluster devicegroup:

# scconf -a -D type=vxvm,name=nfsdg,\nodelist=proto192:proto193,\preferenced=true,failback=enabled

This command uses the following parameters:

● The nodelist should contain all the nodes and only the nodesphysically connected to the disk group’s disks

● The preferenced=true/false affects whether the nodelistindicates an order of failover preference. On a two-node cluster, thisoption is only meaningful if failback is enabled.

● The failback=disabled/enabled affects whether a preferred node“takes back” its device group when it joins the cluster. The defaultvalue is disabled. When failback is disabled, preferenced is set tofalse . If it is enabled, then you must also have preferenced set totrue .

Using the scsetup Command to Register Disk Groups

The scsetup utility can guide you through the previous options in amenu-driven environment. From the main menus, selecting MenuOption 4 (Device Groups and Volumes) gives you the following submenu:

*** Device Groups Menu ***


1) Register a VxVM disk group as a device group 2) Synchronize volume information for a VxVM device group 3) Unregister a VxVM device group 4) Add a node to a VxVM device group 5) Remove a node from a VxVM device group 6) Change key properties of a device group




Verifying and Controlling Registered Device Groups

Use the scstat -D command to verify registered device groups:

# scstat -D


Device Group Primary Secondary ------------ ------- --------- Device group servers: nfsdg proto192 proto193


Device Group Status ------------ ------ Device group status: nfsdg Online

Note – By default, even if there are more than two nodes in the nodelistfile for a device group, only one node shows up as secondary. If theprimary fails, the secondary becomes primary and another (spare) nodewill become secondary.

There is another parameter that when set on the scconf command lineexplained previously, numsecondaries=# , allows you to have more thanone secondary in this case. You would have to do this, for example, tosupport more than two-way failover applications using local failover filesystems.

When VxVM disk groups are registered as Sun Cluster software devicegroups and have the status Online, never use the vxdg import and vxdgdeport commands to control ownership of the disk group. This will causethe cluster to treat the device group as failed.

Instead, use the following command syntax to control disk groupownership:

# scswitch -z -D nfsdg -h node_to_switch_to

Managing VxVM Device Groups


Managing VxVM Device Groups

Use the scconf -c command to perform cluster-specific changes toVxVM device groups.

Resynchronizing Device Groups

After a VxVM device group is registered with the cluster, it must beresynchronized any time a new volume is created or deleted in the diskgroup. This instructs the Sun Cluster software to scan the disk group andbuild and remove the appropriate global device files:

# scconf -c -D name=nfsdg,sync

Making Other Changes to Device Groups

The properties of existing VxVM device groups can also be changed. Forexample, the failback and preferenced properties of a group could bemodified after it is registered.

# scconf -c -D \name=nfsdg,preferenced=false,failback=disabled

Using the Maintenance Mode

You can take a VxVM device group “out of service,” as far as the cluster isconcerned, for emergency repairs.

To put the device group in maintenance mode, all of the VxVM volumesmust be unused (unmounted or otherwise not open). Then you can issue:

# scswitch -m -D nfsdg

It is rare that you will ever want to do this because almost all repairs canstill be done while the device group is in service.

To come back out of maintenance mode, just switch the device group ontoa node:

# scswitch -z -D nfsdg -h new_primary_node

Using Global and Local File Systems on VxVM Volumes


Using Global and Local File Systems on VxVM Volumes

Sun Cluster 3.1 data services support running data services on thefollowing categories of file systems:

● Global file systems – Accessible to all cluster nodes simultaneously,even those not physically connected to the storage

● Local failover file systems – Mounted only on the node running thefailover data service, which must be physically connected to thestorage

The file system type can be UFS or VxFS, regardless of whether you areusing global or local file systems. These examples and the exercisesassume you are using UFS.

Creating File Systems

The distinction between global and local failover file system is not made atthe time of file system creation. Use newfs as normal to create a UFS filesystem for the cluster on a registered disk group:

# newfs /dev/vx/rdsk/nfsdg/nfsvol

Mounting File Systems

The distinction between global and local failover file system is made inthe /etc/vfstab “mount-at-boot” and “options” columns.

A global file system entry should look like the following, and it should beidentical on all nodes who might run services that access the file system(including nodes not physically connected to the storage):

/dev/vx/dsk/nfsdg/nfsvol /dev/vx/rdsk/nfsdg/nfsvol /global/nfs ufs 2 yes global ,logging

A local failover file system entry looks like the following, and it should beidentical on all nodes who might run services that access the file system(can only be nodes physically connected to the storage):

/dev/vx/dsk/nfsdg/nfsvol /dev/vx/rdsk/nfsdg/nfsvol /localnfs ufs 2 no logging

Mirroring the Boot Disk With VxVM


Mirroring the Boot Disk With VxVM

Assuming you have encapsulated your boot drive with VxVM, mirroringthe boot drive is a simple procedure which can be undertaken any timewhile the cluster is online, without any reboot.

1. Initialize and add another local drive to rootdg , if this has not beendone yet. This drive must be at least as big as the boot drive. Givethe drive the VxVM logical name rootmir , for example. This is doneas part of the vxdiskadd dialogue.

# vxdiskadd c0t8d0

2. Use the vxmirror command, or the vxdiskadm “Mirror volumes ona disk” option to mirror all the volumes on the boot drive.

Caution – Do not mirror each volume separately with vxassist . Thisleaves you in a state that the mirrored drive is “unencapsulatable.” Thatis, if you ever need to unencapsulate the second disk you will not be ableto.

If you mirror the root drive correctly as shown, the second drive becomesjust as “unencapsulatable” as the original. That is because correct“restricted subdisks” are laid down onto the new drive. These can later beturned into regular Solaris OS partitions because they are on cylinderboundaries.

# vxmirror rootdisk_1 rootmir..

3. The previous command also creates aliases for both the original rootpartition and the new mirror. You must manually set your boot-device variable so you can boot off both the original and the mirror.

# eeprom|grep vxdevalias vx-rootdisk_1 /pci@1f,4000/scsi@3/disk@0,0:adevalias vx-rootmir /pci@1f,4000/scsi@3/disk@8,0:a

# eeprom boot-device=”vx-rootdisk_1 vx-rootmir”

4. Verify the system has been instructed to use these device aliases onboot:

# eeprom|grep use-nvramrcuse-nvramrc?=true

Performing Disk Replacement With VxVM in the Cluster



The disk repair issues that are strictly cluster-specific are actuallyunrelated to VxVM. That is, you have the same issues regardless of whichvolume manager you choose to use.

This section assumes you are working on a non-hardware RAID just abunch of disks (JBOD) disk replacement. Assume that spindlereplacement in hardware-RAID protected arrays have no specific clusterimages, nor cause any breakages that are visible at the software volumemanagement or cluster software layer.

Repairing Physical Device Files on Multiple Nodes

For disk devices where a disk replacement requires repairing Solaris OSphysical device files and the /etc/path_to_inst file, such as SunStorEdge A5x00 storage arrays, you will have to remember to do theSolaris OS repair on all nodes physically connected to the storage.

For a Sun StorEdge A5x00 array, for example, you are used to running theluxadm remove_dev and luxadm insert_dev commands to manageboth the physical replacement of the disk and the repair of the Solaris OSdevice files. Remember to also run the devfsadm command on the othernode physically connected to the storage.

Note – In the Solaris 9 OS, physically replacing such disks is actuallyautomatically detected for you (on both nodes) and the device files andpath_to_inst file are automatically updated.

Repairing the DID Device Database

When any JBOD disks are replaced in the cluster environment, the DIDdatabase must be updated with new physical device information for theunderlying disk.

Make sure you know which disk to update:

# scdidadm -l c1t1d07 proto192:/dev/rdsk/c1t1d0 /dev/did/rdsk/d7# scdidadm -l d77 proto192:/dev/rdsk/c1t1d0 /dev/did/rdsk/d7



Then, use the following commands to update and verify the DIDinformation:

# scdidadm -R d7# scdidadm -l -o diskid d746554a49545355204d4146333336344c2053554e33364720303....

Analyzing a Full Scenario of a Sun StorEdge™ A5200Array Disk Failure

The following procedure combines the cluster-specific repair issues withstandard disk repair procedures.

1. Identify the failed drive. This is often easiest to do through thevolume management software.

# vxdisk list# vxprint -g dgname

2. On the node hosting the device group to which the disk belongs,replace the failed disk. Let the luxadm command guide you throughremoving the failed disk and inserting a new one.

# luxadm remove enclosure,position# luxadm insert enclosure,position

3. On the other node connected to the array, run the devfsadmcommand.

4. On either node, reconfigure the DID information.

# scdidadm -l c #t #d## scdidadm -R d#

5. On the node hosting the device group, run:

# vxdctl enable

6. On the node hosting the device group, run:

# vxdiskadmReplace a failed or removed disk

Let the vxdiskadm command guide you through the process. You donot need to physically replace the drive again if it prompts you to doso, because it has already been replaced.



7. Verify in volume manager that the failed mirrors are beingresynchronized, or that they were previously reconstructed by thevxrelocd command:

# vxprint -g grpname# vxtask list

8. Remove any submirrors (plexes) that might look wrong (mirrored inthe same array because of hot relocation) and remirror correctly:

# vxunreloc repaired-diskname

Exercise: Configuring Volume Management




● Install the VxVM software

● Initialize the VxVM software

● Create demonstration volumes

● Perform Sun Cluster software disk group registration

● Create global file systems

● Modify device group policies

Preparation

Record the location of the VxVM software you will install during thisexercise.

Location: _____________________________

During this exercise, you create a private rootdg disk group for eachnode and two data service disk groups. Each data service disk groupcontains a single mirrored volume, as shown in Figure 6-8.

Figure 6-8 Configuring Volume Management

Node 1 Node 2Boot Disk Boot Disk

c1 c1

c0 c0

A B A B

Quorum Disk

Primary Disk

Primary Disk

Disk Groupand Volumes

nfsdg

nfsvol

webdg

webvol

Array A Array B

(Encapsulated)

rootdgrootdg

Mirror Disk

Mirror Disk



Note – During this exercise, when you see italicized names, such asIPaddress , enclosure_name , node1 , or clustername embedded in acommand string, substitute the names appropriate for your cluster.

Task 1 – Selecting Disk Drives

Before proceeding with this exercise, you must have a clear picture of thedisk drives that are used throughout this course. You have alreadyconfigured one disk drive as a quorum disk. It is perfectly acceptable touse this as one of your disk group drives as well. In this exercise, youmust identify the boot disks to be encapsulated and two disks in eachstorage array for use in the two demonstration disk groups, nfsdg andwebdg.

Perform the following steps to select disk driver:

1. Use the scdidadm -l and format commands to identify and recordthe logical addresses of disks for use during this exercise. Use theaddress format: c0t2d0 .

Quorum disk: _________

Node 1 Node 2

Boot disks: _________ _________

Array A Array B

nfsdg disks: _________ _________

webdg disks: _________ _________



Task 2 – Using the scvxinstall Utility to Install andInitialize VxVM Software

Perform the following steps to install and initialize the VxVM software onyour cluster host systems:

1. Type the scstat -q command to verify that both nodes are joinedas cluster members. Do not proceed until this requirement is met.

2. Start the /usr/cluster/bin/scvxinstall utility on Node 1.

3. Respond to the scvxinstall questions as follows:

a. Reply yes to root disk encapsulation.

b. Type the full path name to the VxVM software packages.

c. Type none for the license key if you are using Sun StorEdgeA5200 arrays. If not, your instructor will provide you a licensekey.

4. After the installation is complete, log in as user root on Node 1.

5. Type the vxprint command to verify the rootdg disk group isconfigured and operational.

6. Start the scvxinstall utility on Node 2.

7. Respond to the scvxinstall questions as follows:

a. Reply yes to boot disk encapsulation.

b. Type the full path name to the VxVM software packages.

c. Type none for the license key, if you are using Sun StorEdgeA5200 arrays or Sun StorEdge T3 arrays. If not, your instructorprovides you with a license key.

8. Use scvxinstall to install VxVM on any additional nodes.

Note you may need a license key for a third node in a “Pair + 1”configuration even if you did not need one on the first two nodes.

9. After the VxVM configuration is complete on all nodes, perform thefollowing steps on all nodes:

a. Type scstat -q , and verify all nodes are cluster members.

b. Type vxprint on all nodes and verify that the names of therootdg volumes are unique between nodes.

c. Type ls -l /dev/vx/rdsk/rootdg on all nodes to verifythat the rootdg disk group minor device numbers are differenton each node.

d. Type scvxinstall -s on all nodes, and verify all of theinstallation and configuration steps have completed.



Task 3 – Configuring Demonstration Volumes

Perform the following steps to configure two demonstration disk group,with each containing a single mirrored volume:

1. On Node 1, create the nfsdg disk group with your previouslyselected logical disk addresses.

# vxdiskadd c #t #d# c #t #d#Which disk group [<group>,none,list,q,?](default: rootdg) nfsdgCreate a new group named nfsdg? [y,n,q,?](default: y) yUse default disk names for these disks? [y,n,q,?](default: y) yAdd disks as spare disks for nfsdg? [y,n,q,?] (default:n) nExclude disks from hot-relocation use? [y,n,q,?](default: n) y

Caution – If you are prompted about encapsulating the disk, you shouldreply no. If you are prompted about clearing old disk usage status from aprevious training class, you should reply yes . In your work environment,be careful when answering these questions because you can destroycritical data or cluster node access information.

2. Verify the status of the disk group and the names and ownership ofthe disks in the nfsdg disk group.

# vxdg list# vxdisk list

3. On Node 1, verify that the new nfsdg disk group is globally linked.

# ls -l /dev/vx/dsk/nfsdglrwxrwxrwx 1 root root 12 Dec 6 2002/dev/vx/dsk/nfsdg ->/global/.devices/node@1/dev/vx/dsk/nfsdg

4. On Node 1, create a 500-Mbyte mirrored volume in the nfsdg diskgroup.

# vxassist -g nfsdg make nfsvol 500m layout=mirror

5. On Node 1, create the webdg disk group with your previouslyselected logical disk addresses.

# vxdiskadd c #t #d# c #t #d#Which disk group [<group>,none,list,q,?](default: rootdg) webdg



6. On Node 1, create a 500-Mbyte mirrored volume in the webdg diskgroup.

# vxassist -g webdg make webvol 500m layout=mirror

7. Type the vxprint command on both nodes. Notice that Node 2 doesnot see the disk groups created and still imported on Node 1.

8. Prior to registration of the new disk groups, verify that all steps havebeen completed properly for VxVM by importing and deportingboth disk groups on each node without the cluster running.

a. Shut down the cluster.

# scshutdown -y -g0

b. Perform the following on both nodes in turn:

ok boot -x# vxdg -g nfsdg import# vxdg -g nfsdg deport# vxdg -g webdg import# vxdg -g webdg deport

c. After you have successfully imported and deported both diskgroups on both nodes, bring node1, then node 2, back into thecluster:

# reboot

Attempting to create a file system on the demonstration volumeshould generate an error message because the cluster cannot see theVxVM device groups.

9. Issue the following command on the node that currently owns thedisk group:

# newfs /dev/vx/rdsk/webdg/webvol

It fails with a device not found error.



Task 4 – Registering Demonstration Disk Groups

Perform the following steps to register the two new disk groups with theSun Cluster framework software:

1. On Node 1, use the scconf utility to manually register the nfsdgdisk group.

# scconf -a -D type=vxvm,name=nfsdg,\nodelist= node1 : node2 ,preferenced=true,failback=enabled

Note – Put the local node (Node 1) first in the node list.

2. On Node 1, use the scsetup utility to register the webdg disk group.

# scsetup

3. From the main menu, complete the following steps:

a. Select option 4, “Device groups and volumes.”

b. From the device groups menu, select option 1, “Register aVxVM disk group as a device group.”

4. Answer the scsetup questions as follows:

Name of the VxVM disk group you want to register? webdgDo you want to configure a preferred ordering (yes/no)[yes]? yes

Are both nodes attached to all disks in this group(yes/no) [yes]? yes

Note – Read the previous question carefully. On a cluster with more thantwo nodes it will ask if all nodes are connected to the disks. In a “Pair +1”configuration, for example, you must answer no and let it ask you whichnodes are connected to the disks.

Which node is the preferred primary for this devicegroup? node2

Enable "failback" for this disk device group (yes/no)[no]? yes



Note – Make sure you specify node2 as the preferred primary node. Youmight see warnings about disks configured by a previous class that stillcontain records about a disk group named webdg. You can safely ignorethese warnings.

5. From either node, verify the status of the disk groups.

# scstat -D

Note – Until a disk group is registered, you cannot create file systems onthe volumes. Even though vxprint shows the disk group and volume asbeing active, the newfs utility cannot detect it.

Task 5 – Creating a Global nfs File System

Perform the following steps on Node 1 to create and mount ademonstration file system on the nfsdg disk group volume:

1. On Node 1, create a file system on nfsvol in the nfsdg disk group.

# newfs /dev/vx/rdsk/nfsdg/nfsvol

2. On all nodes, create a global mount point for the new file system.

# mkdir /global/nfsOn all nodes , add a mount entry in the /etc/vfstab filefor the new file system with the global and loggingmount options./dev/vx/dsk/nfsdg/nfsvol /dev/vx/rdsk/nfsdg/nfsvol \/global/nfs ufs 2 yes global,logging

Note – Do not use the line continuation character (\) in the vfstab file.

3. On Node 1, mount the /global/nfs file system.

# mount /global/nfs

4. Verify that the file system is mounted and available on all nodes.

# mount# ls /global/nfslost+found



Task 6 – Creating a Global web File System

Perform the following steps on Node 2 to create and mount ademonstration file system on the webdg disk group volume:

1. On Node 2, create a file system on webvol in the webdg disk group.

# newfs /dev/vx/rdsk/webdg/webvol


# mkdir /global/web

3. On all nodes, add a mount entry in the /etc/vfstab file for the newfile system with the global and logging mount options.

/dev/vx/dsk/webdg/webvol /dev/vx/rdsk/webdg/webvol \/global/web ufs 2 yes global,logging


4. On Node 2, mount the /global/web file system.

# mount /global/web


# mount# ls /global/weblost+found

Task 7 – Testing Global File Systems

Perform the following steps to confirm the general behavior of globallyavailable file systems in the Sun Cluster 3.1 software environment:

1. On Node 2, move into the /global/nfs file system.

# cd /global/nfs

2. On Node 1, try to unmount the /global/nfs file system (umount/global/nfs ). You should get an error that the file system is busy.

3. On Node 2, move out of the /global/nfs file system (cd / ) and tryto unmount it again on Node 1.

4. Mount the /global/nfs file system again on Node 1.

5. Try unmounting and mounting /global/nfs from all nodes.



Task 8 – Managing Disk Device Groups

In the Sun Cluster 3.1 software environment, VERITAS disk groupsbecome disk device groups when they are registered. In most cases, theyshould not be managed using VERITAS commands. Some administrativetasks are accomplished using a combination of Sun Cluster and VERITAScommands. Common tasks are:

● Adding volumes to a disk device group

● Removing volume from a disk device group

Adding a Volume to a Disk Device Group

Perform the following steps to add a volume to an existing device group:

1. Make sure the device group is online (to the Sun Cluster software).

# scstat -D

Note – You can bring it online to a selected node as follows:# scswitch -z -D nfsdg -h node1

2. Determine which node is currently mastering the related VERITASdisk group (that is, has it imported).

# vxdg list# vxprint

3. On Node 1, create a 50-Mbyte test volume in the nfsdg disk group.

# vxassist -g nfsdg make testvol 50m layout=mirror

4. Verify the status of the volume testvol .

# vxprint testvol

5. Perform the following steps to register the changes to the nfsdg diskgroup configuration.

a. Start the scsetup utility.

b. Select menu item 4, “Device groups and volumes.”

c. Select menu item 2, “Synchronize volume information.”

d. Supply the name of the disk group, and quit the scsetup utilitywhen the operation is finished.

Note – The command-line equivalent is:scconf -c -D name=nfsdg,sync



Removing a Volume From a Disk Device Group

To remove a volume from a disk device group, perform the followingsteps on the node that currently has the related disk group imported:

1. Unmount any file systems that are related to the volume.

2. On Node 1, recursively remove the test volume, testvol , from thenfsdg disk group.

# vxedit -g nfsdg -rf rm testvol

3. Synchronize the nfsdg disk group configuration change with theSun Cluster software framework.

Note – You can use either the scsetup utility or scconf as follows:scconf -c -D name=nfsdg,sync

Migrating Device Groups

The scconf -p command is the best method of determining currentdevice group configuration parameters. Perform the following steps toverify device group behavior:

1. Verify the current demonstration device group configuration.

# scconf -p |grep groupDevice group name: nfsdg Device group type: VxVM Device group failback enabled: yes Device group node list: node1,node2 Device group ordered node list: yes Device group desired number of secondaries: 1 Device group diskgroup name: nfsdg

Device group name: webdg Device group type: VxVM Device group failback enabled: yes Device group node list: node2,node1 Device group ordered node list: yes Device group desired number of secondaries: 1 Device group diskgroup name: webdg



2. From either node, switch the nfsdg device group to Node 2.

# scswitch -z -D nfsdg -h node2 (use your node name)

Type the init 0 command to shut down Node 1.

3. Boot Node 1. The nfsdg disk group should automatically migrateback to Node 1.

Task 9 – Viewing and Managing VxVM Device GroupsThrough SunPlex Manager

Perform the following steps on your administration workstation:

1. In a web browser, log in to SunPlex manager on any cluster node:


2. Click on the Global Devices folder on the left.

3. Investigate the status information and graphical topologyinformation that you can see regarding VxVM Device Groups.

4. Investigate the commands you can run through the Action Menu.

5. Use the Action Menu to switch the primary for one of your devicegroups.

Exercise Summary


Exercise Summary

?!


● Experiences

● Interpretations

● Conclusions

● Applications


Module 7

Managing Volumes With Solaris™ VolumeManager Software (Solstice DiskSuite™Software)

Objectives


● Explain Solaris Volume Manager software disk space management

● Describe the most important concepts of Solaris Volume Managersoftware

● Describe Solaris Volume Manager software soft partitions

● Differentiate between shared disksets and local disksets

● Describe volume database (metadb ) management issues and the roleof Solaris Volume Manager software mediators

● Install the Solaris Volume Manager software

● Create the local metadbs

● Add disks to shared disksets

● Use shared diskset disk space

● Build Solaris Volume Manager software mirrors with soft partitions indisksets

● Use Solaris Volume Manager software status commands

● Use hot spare pools in the disk set

● Perform Sun Cluster software level device group management

● Perform cluster-specific changes to Solaris Volume Manager softwaredevice groups

Objectives



● Mirror the boot disk with Solaris Volume Manager software

● Perform disk replacement and repair

Relevance

Managing Volumes With Solaris™ Volume Manager Software (Solstice DiskSuite™ Software) 7-3Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1

Relevance

?!


● Is it easier to manage Solaris Volume Manager software devices withor without soft partitions?

● How many different collections of metadbs will you need tomanage?

● What is the advantage of using DID devices as Solaris VolumeManager software building blocks in the cluster?

● How do Solaris Volume Manager software disksets get registered inthe cluster?







● Sun Microsystems, Inc. Solaris Volume Manager Administration Guide,part number 816-4519.

Describing Solaris Volume Manager Software and Solstice DiskSuite Software


Describing Solaris Volume Manager Software and SolsticeDiskSuite Software

Solaris 9 OS integrates the Solaris Volume Manager software into the baseSolaris OS. If you are running Solaris 8 OS Update 7 (02/02) in the SunCluster 3.1 software environment, installing Solstice DiskSuite 4.2.1software with a minimum of patch 108693-06 gives you the equivalentcapabilities of Solaris Volume Manager software. This includes softpartitioning, which is described in the module and used in the labs. Youmust have a SunSolve program account to access this patch.

In the Solstice DiskSuite software, virtual volume structures have alwaysbeen referred to as metadevices.

As its new name indicates, the Solaris Volume Manager software prefers theuse of the word volumes to refer to the same structures. This modulefollows this latter convention.

Viewing Solaris Volume Manager Software in the Sun Cluster Software Environment


Viewing Solaris Volume Manager Software in theSun Cluster Software Environment

This module does not intend to replace the full three-day course in diskmanagement using Solaris Volume Manager software. Instead, this modulebriefly introduces the most important concepts of Solaris Volume Managersoftware, and then focuses on the issues for using Solaris Volume Managersoftware to manage disk space in the cluster environment. Among thetopics highlighted in this module are:

● Using Solaris Volume Manager software with and without softpartitions

● Identifying the purpose of Solaris Volume Manager software disksetsin the cluster

● Managing local volume database replicas (metadb ) and shareddiskset metadb s

● Initializing Solaris Volume Manager software to build cluster disksets

● Building cluster disksets

● Mirroring with soft partitions in cluster disksets

● Mirroring the cluster boot disk with Solaris Volume Manager software

Exploring Solaris Volume Manager Software Disk Space Management


Exploring Solaris Volume Manager Software Disk SpaceManagement

Solaris Volume Manager software has two distinct ways of managing diskspace. Until patch 108693-06, the Solstice DiskSuite software (SolarisVolume Manager software’s precursor product) had the restriction thatSolaris OS partitions were the smallest granularity building blocks forvolumes. The current Solaris Volume Manager software supports both thetraditional way of managing space and the new method called softpartitioning.

Solaris Volume Manager Software Partition-Based DiskSpace Management

Figure 7-1 demonstrates Solaris Volume Manager software disk spacemanagement that equates standard disk partitions with building blocksfor virtual volumes.

Figure 7-1 Solstice Volume Manager Software Space Management

Physical Disk Drive

Slice 7 (metadb)

Slice 3

Slice 4

Slice 6

Volume d18 submirror

Volume d6 submirror




The following are the limitations with using only partitions as SolarisVolume Manager software building blocks:

● The number of building blocks per disk or LUN is limited to thetraditional seven (or eight, if you want to push it) partitions. This isparticularly restrictive for large LUNs in hardware RAID arrays.

● Disk repair is harder to manage because an old partition table on areplacement disk must be recreated or replicated manually.

Solaris Volume Manager Software Disk SpaceManagement With Soft Partitions

Soft partitioning supports the creation of virtual partitions within SolarisOS disk partitions or within other volumes. These virtual partitions canrange in size from one block to the size of the base component. You arelimited in the number of soft partitions only by the size of the basecomponent.

Soft partitions add an enormous amount of flexibility to the capability ofSolaris Volume Manager software:

● They allow you to manage volumes whose size meets your exactrequirements without physically repartitioning Solaris OS disks.

● They allow you to create way more than seven volumes using spacefrom a single large drive or LUN.

● They can grow as long as there is space left inside the parentcomponent. The space does not have to be physically contiguous.Solaris Volume Manager software finds whatever space it can in thesame parent component to allow your soft partition to grow.



There are two ways to deal with soft partitions in Solaris Volume Managersoftware:

● Make soft partitions on disk partitions, and then use those to buildyour submirrors (Figure 7-2):

Figure 7-2 Building Submirrors From Soft Partitions

● Build mirrors using entire partitions, and then use soft partitions tocreate the volumes with the size you want (Figure 7-3):

Figure 7-3 Using Soft Partitions to Create Volumes

Physical Disk Drive




Slice 7 (metadb)

soft partition d18

soft partition d28

soft par tition d38

slice s0

Physical disk drivePhysical disk drive

Slice 7 (metadb )

slice s0

Slice 7 (metadb )

slice s0

submirrorsubmirror

soft partition

volume



All the examples later in this module use the second strategy. Theadvantages of this are:

● Volume management is simplified.

● Hot spare management is consistent with how hot spare pools work.

The only disadvantage of this scheme is that you have large mirrordevices even if you are only using a small amount of space. This slowsdown resynching of mirrors.

Exploring Solaris Volume Manager Software Shared Disksets and Local Disksets


Exploring Solaris Volume Manager Software SharedDisksets and Local Disksets

When using Solaris Volume Manager software to manage data in the SunCluster 3.1 software environment, all disks that hold the data for clusterdata services must be members of Solaris Volume Manager software shareddisksets.

Only disks that are physically located in the multi-ported storage will bemembers of the shared disksets. Only disks that are in the same disksetoperate as a unit; they can be used together to build mirrored volumes,and primary ownership of the diskset transfers as a whole from node tonode. You will always have disks from different controllers (arrays) in thesame diskset, so that you can mirror across controllers.

Shared disksets are given a name that often reflects the intended usage ofthe diskset (for example, nfsds ).

Exploring Solaris Volume Manager Software Shared Disksets and Local Disksets


To create any shared disksets, Solaris Volume Manager software requiresthat each host (node) must have a local diskset on non-shared disks. Theonly requirement for these local disks is that they have local disksetmetadb s, which are described later. It is likely, however, that if you areusing Solaris Volume Manager software for your data service data, that youwill also mirror your boot disk using Solaris Volume Manager software.Figure 7-4 shows the contents of two shared disksets in a typical cluster,along with the local disksets on each node.

Figure 7-4 Two Shared Disksets

Boot Disks(Local Diskset)

Node 1

Boot Disks(Local Diskset)

Node 2

Diskset webdsDiskset nfsds

Using Solaris Volume Manager Software Volume Database Replicas (metadb)


Using Solaris Volume Manager Software VolumeDatabase Replicas (metadb)

Solaris Volume Manager software requires that certain partitions serve asvolume database replicas, storing the Solaris Volume Manager softwareconfiguration and state information in a raw (non-file system) format.These are normally small dedicated Solaris OS partitions. Formerly, in thelocal disksets, Solaris Volume Manager software allowed the creation ofmetadbs on a large partition and then using the same partition as acomponent in a volume. Reasons why you would never want to do thison the boot disks are described later in this module.

There are separate sets of metadbs for the local disksets (local metadbs)and for each shared diskset. For example, in Figure 7-4 on page 7-12(which shows two nodes and two shared disksets), there are four distinctcollections of metadbs.

It is possible to put several copies of the metadb on the same partition.You might do this to balance the numbers of metadbs across disks orpartitions, for reasons described in the following subsections.

Local Replicas Management

When managing local replicas:

● You must add local replicas manually.

● You can put local replicas on any dedicated partitions on the localdisks. You might prefer to use s7 as convention, because the shareddiskset replicas have to be on that partition.

● You must spread local replicas evenly across disks and controllers.

● If you have less than three local replicas, Solaris Volume Managersoftware logs warnings. You can put more than one copy in the samepartition to satisfy this requirement. For example, with two localdisks, set up a dedicated partition on each disk and put three copieson each disk.



Local Replica Mathematics

The math for local replicas is as follows:

● If less than 50 percent of the defined metadb replicas are available,Solaris Volume Manager software will cease to operate.

● If less than or equal to 50 percent of the defined metadb replicas areavailable at boot time, you cannot boot the node. However, you canboot to single user mode, and just use the metadb command todelete ones that are not available.

Shared Diskset Replicas Management

Consider the following issues when managing shared diskset replicas:

● There are separate metadb replicas for each diskset.

● They are automatically added to disks as you add disks to disk sets.They will be, and must remain, on slice 7.

● You must use the metadb command to remove and add replicas ifyou repair a disk containing one (remove broken ones and add backonto repaired disk).

Shared Diskset Replica Quorum Mathematics

The math for shared diskset replica quorums is as follows:

● If less than 50 percent of the defined replicas for a diskset areavailable, the diskset ceases to operate!

● If less than or equal to 50 percent of the defined replicas for a disksetare available, the diskset cannot be “taken” or switched over.



Shared Diskset Mediators

When you have two nodes connected to two storage arrays, theimplication of the replica mathematics described in the previous section isthat your diskset could keep operating but could not transfer primarycontrol from node to node.

This is unacceptable in the Sun Cluster software environment, because itcould take a while to fix a broken array or controller, and you still want tobe able to gracefully survive a node failure during that period.

The Sun Cluster 3.1 software environment includes special add-ons toSolaris Volume Manager software called mediators. Mediators allow you toidentify the nodes themselves as “tie-breaking votes” in the case of failureof exactly 50 percent of the metadbs of a diskset. The mediator data isstored in the memory of a running Solaris OS process on each node. If youlose an array, the node mediators transfer to “golden” status, indicatingthey count as two extra votes for the shared diskset quorum mathematics.This allows you to maintain normal diskset operations with exactly 50percent of the metadbs surviving. You could also lose a node at this point(you still have one tie-breaking, “golden” mediator).

Installing Solaris Volume Manager Software and Tuning the md.conf File


Installing Solaris Volume Manager Software and Tuningthe md.conf File

In Solaris 9 OS, the packages SUNWmdr, SUNWmdu, and SUNWmdxare part ofthe base operating environment.

In Solaris 8 OS Update 7 (02/02), the only release earlier than the Solaris 9OS on which Sun Cluster 3.1 software is supported, these packages mustbe installed manually. Make sure you also install a minimum ofpatch 108693-06 so the soft partitioning feature is enabled.

Support for shared diskset mediators is in the package SUNWmdm. Thispackage is automatically installed as part of the cluster framework.

Modifying the md.conf File

Based on your planned implementation, you might need to update SolarisVolume Manager software’s kernel configuration file,/kernel/drv/md.conf . There are two variables that might need to beupdated. These maximums include your local disksets. The modificationsare summarized in Table 7-1.

Keep this file identical on all nodes of the cluster. Changes to this file takeeffect after you call devfsadm or perform a boot -r .

Table 7-1 Modifications to the md.conf File

Variable DefaultValue Description

nmd 128 The maximum number of volumes. Solaris Volume Managersoftware uses this setting to limit the names of the volumes aswell. Note that setting this number too high can use a lot ofinodes for device files in your /global/.devices/node@# filesystem.

md_nsets 4 The maximum number of disksets. This includes the localdiskset. Set this number should be set to the number of shareddisksets you plan to create in your cluster plus one. Themaximum value for md_nsets is 32.

Initializing the Local Metadbs on Boot Disk and Mirror



No other Solaris Volume Manager software management can be done untilyou initialize the local metadbs on each node.

These instructions assume that you have a small partition to use for themetadbs on your boot disk. In this example, the partition is “s7.”

The instructions presented in the following section assume that you areeventually mirroring your boot drive. As such, they instruct you topartition the eventual boot mirror identically to the boot drive and to addlocal metadbs there as well.

Make sure you initialize local metadbs correctly and separately on eachnode.

Using DIDs Compared to Using Traditional c#t #d#

In the Sun Cluster environment, you can add any metadb or partitioncomponent either using its cluster Device ID (/dev/did/rdsk/d #s#) orby using the traditional c#t #d#.

Use the DID naming for all shared disksets. Without it you would berestricted to having identical controller numbers on each node. You mightassume this will always be true, then be surprised when you add a newnode or repair a node.

Use the traditional c#t #d# naming scheme for local metadbs and devices.This will make recovery easier in the case that you need to access thesestructures when booted in non-clustered mode. Omit the /dev/rdsk toabbreviate traditional names in all of the Solaris Volume Manager softwarecommands.

Adding the Local Metadb Replicas to the Boot Disk

Use the metadb -a command to add local metadbs. As you add the firstones, you must use the -f (force) option. The following example usesthree copies:

# metadb -a -f -c 3 c0t0d0s7



Repartitioning a Mirror Boot Disk and Adding MetadbReplicas

This procedure assumes you have a second local disk that has identicalgeometry to the first. In that case, you can use the fmthard command toreplicate the boot disks’s partition table onto the mirror:

# prtvtoc /dev/rdsk/c0t0d0s0|fmthard -s - /dev/rdsk/c0t8d0s0fmthard: New volume table of contents now in place.

# metadb -a -c 3 c0t8d0s7

Using the metadb or metadb -i Command to VerifyMetadb Replicas

The output of the metadb command includes one line for each replica.The -i option adds the legend information about the status abbreviations.

# metadb -i flags first blk block count

a m p luo 16 8192 /dev/dsk/c0t0d0s7a p luo 8208 8192 /dev/dsk/c0t0d0s7a p luo 16400 8192 /dev/dsk/c0t0d0s7a p luo 16 8192 /dev/dsk/c0t8d0s7a p luo 8208 8192 /dev/dsk/c0t8d0s7a p luo 16400 8192 /dev/dsk/c0t8d0s7

r - replica does not have device relocation information o - replica active prior to last mddb configuration change u - replica is up to date l - locator for this replica was read successfully c - replica's location was in /etc/lvm/mddb.cf p - replica's location was patched in kernel m - replica is master, this is replica selected as input W - replica has device write errors a - replica is active, commits are occurring to this replica M - replica had problem with master blocks D - replica had problem with data blocks F - replica had format problems S - replica is too small to hold current data base R - replica had device read errors

Creating Shared Disksets and Mediators



You use the metaset command to both create new empty disksets and toadd disk drives into and existing diskset. You must use the -a -h optionsof the metaset command first to create an empty diskset. Then you canadd disks. The first host listed as you create a new diskset is the first oneto be the owner of the diskset. You can add mediators with the -a -moptions.

All the examples shown in this module of diskset operations use the DIDnames for disks instead of the c#t #d# names. You might have to run thescdidadm command often to map between the two.


# metaset -s nfsds -a -h proto192 proto193# metaset -s nfsds -a -m proto192 proto193

# metaset -s nfsds -a /dev/did/rdsk/d9 /dev/did/rdsk/d17# metaset

Set name = nfsds, Set number = 1

Host Owner proto192 Yes proto193

Drive Dbased9 Yesd17 Yes

# metadb -s nfsds flags first blk block count a u r 16 8192 /dev/did/dsk/d9s7 a u r 16 8192 /dev/did/dsk/d17s7

# medstat -s nfsdsMediator Status Goldenproto192 Ok Noproto193 Ok No

Mediators become golden if exactly 50 percent of the metadbs fail.



Shared Diskset Disk Automatic Repartitioning andMetadb Placement

When a disk is added to a diskset, it is automatically repartitioned asfollows:

● A small portion of the drive (starting at cylinder 0) is placed intoslice 7 to be used for volume state database replicas (usually at least2 Mbytes in size).

● Metadbs are added to slice 7 as appropriate to maintain the balanceacross controllers.

● The rest of the drive is placed into slice 0 (even slice 2 is deleted).

The drive is not repartitioned if the disk already has no slice 2 and slice 7already has the following characteristics:

● It starts at cylinder 0

● It has at least 2 Mbytes (large enough to hold a state database)

● It has the V_UNMTflag set (unmountable flag)

● It is not read only

Regardless of whether the disk is repartitioned, diskset metadbs areadded automatically to slice 7 as appropriate as shown in Figure 7-5. Ifyou have exactly two disk controllers you should always add equivalentnumbers of disks or LUNs from each controller to each diskset tomaintain the balance of metadbs in the diskset across controllers.

Figure 7-5 Automatically Adding Metadbs to Slice 7

Physical disk drive

Slice 7 (metadb )

Slice 0

Using Shared Diskset Disk Space


Using Shared Diskset Disk Space

This section shows the commands that implement the strategy describedearlier in the module:

● Always use slice 0 “as is” in the diskset (almost the entire drive orLUN)

● Build mirrors out of slice 0 of disks across two different controllers

● Use soft partitioning of the large mirrors to size the volumesaccording to your needs

Figure 7-6 demonstrates the strategy again in terms of volume (d#) andDID (d#s#). While Solaris Volume Manager software would allow you touse c#t #d#, always use DID numbers in the cluster to guarantee aunique, agreed-upon device name from the point of view of all nodes.

Figure 7-6 Strategy for Building Volumes

d10 (soft partition) d11 (soft partition)

d101 (stripe / concat) d102 (stripe / concat)

/dev/did/rdsk/d9s0 /dev/did/rdsk/d17s0

d100 (mirror)

Volumes

DIDs

Building Volumes in Shared Disksets With Soft Partitions of Mirrors


Building Volumes in Shared Disksets With Soft Partitionsof Mirrors

There are two ways to indicate which diskset you are referring to for eachmetainit command:

● Use -s disksetname with the command

● Use disksetname/d# for volume operands in the command.

This module uses the former model for all the examples.

The following is an example of the commands that are used to build theconfiguration described in Figure 7-6 on page 7-21:

# metainit -s nfsds d101 1 1 /dev/did/rdsk/d9s0nfsds/d101: Concat/Stripe is setup# metainit -s nfsds d102 1 1 /dev/did/rdsk/d17s0nfsds/d102: Concat/Stripe is setup# metainit -s nfsds d100 -m d101nfsds/d100: Mirror is setup# metattach -s nfsds d100 d102nfsds/d100: submirror nfsds/d102 is attached# metainit -s nfsds d10 -p d100 200md10: Soft Partition is setup# metainit -s nfsds d11 -p d100 200md11: Soft Partition is setup

The following commands show how a soft partition can be grown as longas there is any (even non-contiguous) space in the parent volume. Youwill see in the output of metastat on the next page how both softpartitions contain noncontiguous space, because of the order in whichthey are created and grown.

# metattach -s nfsds d10 400mnfsds/d10: Soft Partition has been grown/ metattach -s nfsds d11 400mnfsds/d11: Soft Partition has been grown

Note – The volumes are being increased by 400 Mbytes, to a total size of600 Mbytes.

Using Solaris Volume Manager Software Status Commands


Using Solaris Volume Manager Software StatusCommands

There are no commands that display information about volumes ormetadbs in multiple disksets at the same time. If you do not specify adiskset name (with -s ), you will (possibly inadvertently) get output onlyabout the local diskset on the node on which you entering the command.

Checking Volume Status

The following metastat command output is for the mirrored volume andsoft partitions built in the previous example:

# metastat -s nfsdsnfsds/d10: Soft Partition Device: nfsds/d100 State: Okay Size: 1228800 blocks Extent Start Block Block count 0 1 409600 1 819203 819200

nfsds/d11: Soft Partition Device: nfsds/d100 State: Okay Size: 1228800 blocks Extent Start Block Block count 0 409602 409600 1 1638404 819200

nfsds/d100: Mirror Submirror 0: nfsds/d101 State: Okay Submirror 1: nfsds/d102 State: Resyncing Resync in progress: 51 % done Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 71118513 blocks

nfsds/d101: Submirror of nfsds/d100 State: Okay Size: 71118513 blocks

Using Solaris Volume Manager Software Status Commands


Stripe 0: Device Start Block Dbase State Reloc Hot Spare d9s0 0 No Okay

nfsds/d102: Submirror of nfsds/d100 State: Resyncing Size: 71118513 blocks Stripe 0: Device Start Block Dbase State Reloc Hot Spare d17s0 0 No Okay

Using Hot Spare Pools


Using Hot Spare Pools

The hot spare pool feature allows you to put unused diskset partitionsinto a named pool. These partitions cannot then be used as buildingblocks to manually create volumes. Instead, the hot spare pool isassociated with submirrors. The first partition in a pool that is big enoughcan replace a broken submirror with which it is associated.

Hot spares must be entire partitions; they cannot be soft partitions.Therefore, the model demonstrated in this module (using entire partitionsas submirrors, building soft partitions on top of mirrors) works best withthe hot spare model.

The following example adds a disk to the diskset, puts its partition s0 in ahot spare pool, and then associates the pool with a submirror. Theassociation also could have been made when the submirror was built:

# metaset -s nfsds -a /dev/did/dsk/d13# metainit -s nfsds hsp001 /dev/did/dsk/d13s0nfsds/hsp001: Hotspare pool is setup

# metaparam -s nfsds -h hsp001 d101# metastat -s nfsds d101nfsds/d101: Concat/Stripe Hot spare pool: nfsds/hsp001 Size: 71118513 blocks Stripe 0: Device Start Block Dbase State Reloc Hot Spare d9s0 0 No Okay

A good strategy, if you have extra drives in each array for hot sparing, isto do the following procedure (with different drives for each diskset):

1. Put extra drives from each array in diskset

2. Put s0 from each extra drive in two hot spare pools:

a. In pool one, list drives from Array 1 first

b. In pool two, list drives from Array 2 first

c. Associate pool one with all submirrors from Array 1

d. Associate pool two with all submirrors from Array 2

The result is in sparing so that you are still mirrored across controllers, ifpossible, and sparing with both submirrors in the same controller as a lastresort.

Managing Solaris Volume Manager Software Disksets and Sun Cluster DeviceGroups


Managing Solaris Volume Manager Software Disksets andSun Cluster Device Groups

When Solaris Volume Manager software is run in the cluster environment,the Solaris Volume Manager software commands are tightly integrated intothe cluster environment. Creation of a Solaris Volume Manager softwareshared diskset automatically registers that diskset as a cluster-manageddevice group:

# scstat -D


Device Group Primary Secondary ------------ ------- --------- Device group servers: nfsds proto193 proto192


Device Group Status ------------ ------ Device group status: nfsds Online

# scconf -pv|grep nfsdsDevice group name: nfsds (nfsds) Device group type: SVM (nfsds) Device group failback enabled: no (nfsds) Device group node list: proto192, proto193 (nfsds) Device group ordered node list: yes (nfsds) Device group desired number of secondaries: 1 (nfsds) Device group diskset name: nfsds

Addition or removal of a directly connected node into the diskset usingmetaset -s nfsds -a -h newnode would automatically update thecluster to add or remove the node from its list. There is no need to usecluster commands to resynchronize the group if volumes are added anddeleted.

In the Sun Cluster software environment, use only the scwitch command(rather than the metaset -[rt] commands) to change physicalownership of the group. Note, as demonstrated in this example, if amirror is in the middle of resynching, this will force a switch anyway andrestart the resynch of the mirror all over again:

Managing Solaris Volume Manager Software Disksets and Sun Cluster DeviceGroups


# scswitch -z -D nfsds -h proto192Dec 10 14:06:03 proto193 Cluster.Framework: stderr: metaset: proto193:Device busy

(A mirror was still synching; it restarts on the other node.)

# scstat -D


Device Group Primary Secondary ------------ ------- --------- Device group servers: nfsds proto192 proto193


Device Group Status ------------ ------ Device group status: nfsds Online

Managing Solaris Volume Manager Software Device Groups


Managing Solaris Volume Manager Software DeviceGroups

While there is no need to register Solaris Volume Manager software disksetswith the scconf command (the registration is performed by the metasetcommand), the scconf -c command can be used to performcluster-specific changes to Solaris Volume Manager software device groups.

Device Groups Resynchronization

Solaris Volume Manager software device groups are automaticallyresyncronized when new volumes are added to an existing diskset orvolumes are deleted.

Other Changes to Device Groups

The properties of existing Solaris Volume Manager software device groupscan be changed. For example, the failback property of a group could bemodified after it is registered.

# scconf -c -D name=nfsds,failback=enabled

Maintenance Mode

You can take a Solaris Volume Manager software device group “out ofservice,” as far as the cluster is concerned, for emergency repairs.

To put the device group in maintenance mode, all of the Solaris VolumeManager software volumes must be unused (unmounted, or otherwise notopen). Then you can issue:

# scswitch -m -D nfsds

It is a rare event that causes you to do this, because almost all repairs canstill be done while the device group is in service.

To come back out of maintenance mode, just switch the device group ontoa node:

# scswitch -z -D nfsds -h new_primary_node

Using Global and Local File Systems on Shared Diskset Devices


Using Global and Local File Systems on Shared DisksetDevices

Sun Cluster 3.1 software data services support running data services onthe following categories of file systems:

● Global file systems – These are accessible to all cluster nodessimultaneously, even those not physically connected to the storage.

● Local failover file systems – These are mounted only on the noderunning the failover data service, which must be physicallyconnected to the storage

The file system type can be UFS or VxFS regardless of whether you areusing global or local file systems. These examples and the lab exercisesassume you are using UFS.

Creating File Systems

The distinction between global and local failover file system is not made atthe time of file system creation. Use newfs as normal to create a UFS filesystem for the cluster:

# newfs /dev/md/nfsds/rdsk/d10

Mounting File Systems

The distinction between global and local failover file system is made inthe /etc/vfstab “mount-at-boot” and “options” columns.

A global file system entry should look like the following, and it should beidentical on all nodes that might run services which access the file system(including nodes not physically connected to the storage):

/dev/md/nfsds/dsk/d10 /dev/md/nfsds/rdsk/d10 /global/nfs ufs 2 yes global ,logging

A local failover file system entry looks like the following, and it should beidentical on all nodes that might run services that access the file system(they can only be nodes that are physically connected to the storage).

/dev/md/nfsds/dsk/d10 /dev/md/nfsds/rdsk/d10 /global/nfs ufs 2 no logging

Using Solaris Volume Manager Software to Mirror the Boot Disk


Using Solaris Volume Manager Software to Mirror the BootDisk

If Solaris Volume Manager software is your volume manager of choice forthe shared storage, you should—although this is not required—use SolarisVolume Manager software to mirror the boot drive. The following examplecontains the following scenario:

● The boot drive is mirrored after cluster installation.

● All partitions on the boot drive are mirrored. The example has root ,swap, /var , and /global/.devices/node@1 . That is, you aremanually creating four separate mirror devices.

● The geometry and partition tables of the boot disk and the newmirror are identical. Copying the partition table from one disk to theother was previously demonstrated in ‘‘Repartitioning a Mirror BootDisk and Adding Metadb Replicas’’ on page 7-18.

● Both the boot disk and mirror have two copies of the local metadbs.This was done previously.

● Soft partitions are not used on the boot disk. That way, if you need toback out, you can just go back to mounting the standard partitionsby editing the /etc/vfstab manually.

● The /global/.devices/node@1 is a special, cluster-specific case. Forthat one device you must use a different volume d# for the top-levelmirror on each node. The reason is that each of these file systemsappear in the /etc/mnttab of each node (as global file systems) andthe Solaris OS does not allow duplicate device names.

Verifying Partitioning and Local Metadbs

The bold sections of the following output emphasize the partitions thatyou will mirror:

# df -kFilesystem kbytes used avail capacity Mounted on/dev/dsk/c0t0d0s0 26843000 1064410 25510160 5% //proc 0 0 0 0% /procmnttab 0 0 0 0% /etc/mnttabfd 0 0 0 0% /dev/fd/dev/dsk/c0t0d0s3 4032504 43483 3948696 2% /varswap 5471480 152 5471328 1% /var/runswap 5471328 0 5471328 0% /tmp



/dev/did/dsk/d1s5 95702 5317 80815 7% /global/.devices/node@1/dev/did/dsk/d22s5 95702 5010 81122 6% /global/.devices/node@2

# swap -lswapfile dev swaplo blocks free/dev/dsk/c0t0d0s1 32,1 16 8170064 8170064

# metadb flags first blk block count

a m p luo 16 8192 /dev/dsk/c0t0d0s7a p luo 8208 8192 /dev/dsk/c0t0d0s7a p luo 16400 8192 /dev/dsk/c0t0d0s7a p luo 16 8192 /dev/dsk/c0t8d0s7a p luo 8208 8192 /dev/dsk/c0t8d0s7a p luo 16400 8192 /dev/dsk/c0t8d0s7

Building Volumes for Each Partition Except for Root

For all boot disk partitions except root, follow the same general strategy:

1. Create simple volumes for the existing partition and the othersubmirror. Use the -f option for the existing partition.

2. Create a mirror using only the existing partition. Do not attach theother half of the mirror until after a reboot. Make sure the d# chosenfor the /global/.devices/node@# is different across your nodes.

3. Edit the /etc/vfstab file manually to use the new volume (mirror)instead of the original partition.

4. Wait to reboot until you do all partitions.

Building Volumes for Root Partition

For the root boot disk partition, follow the same general strategy:

1. Create simple volumes for the existing partition and the othersubmirror. Use the -f option for the existing partition.

2. Create a mirror using only the existing partition. Do not attach theother half of the mirror until after a reboot.

3. Use metaroot to edit the vfstab and system files.



Running the Commands

Examples are shown for root and swap. The procedure for the otherpartitions would be just like the one for swap.

# metainit -f d11 1 1 c0t0d0s0d11: Concat/Stripe is setup# metainit d12 1 1 c0t8d0s0d12: Concat/Stripe is setup# metainit d10 -m d11d10: Mirror is setup# metaroot d10

# metainit -f d21 1 1 c0t0d0s1d11: Concat/Stripe is setup# metainit d22 1 1 c0t8d0s1d12: Concat/Stripe is setup# metainit d20 -m d21d10: Mirror is setup# vi /etc/vfstab(change correct line manually)

/dev/md/dsk/d20 - - swap - no -

Rebooting and Attaching the Second Submirror

After a reboot, you can attach the second submirror to each volume. Thesynchronizing of the mirrors runs in the background and can take a longtime.

# init 6..# df -k/dev/md/dsk/d10 26843000 1064163 25510407 5% //proc 0 0 0 0% /proc



mnttab 0 0 0 0% /etc/mnttabfd 0 0 0 0% /dev/fd/dev/md/dsk/d30 4032504 43506 3948673 2% /varswap 5550648 152 5550496 1% /var/runswap 5550496 0 5550496 0% /tmp/dev/md/dsk/d50 95702 5317 80815 7%/global/.devices/node@1/dev/did/dsk/d22s5 95702 5010 81122 6%/global/.devices/node@2

proto192:/# swap -lswapfile dev swaplo blocks free/dev/md/dsk/d20 85,20 16 8170064 8170064

# metattach d10 d12# metattach d20 d22# metattach d30 d32# metattach d50 d52

Performing Disk Replacement With Solaris Volume Manager Software in the Cluster


Performing Disk Replacement With Solaris VolumeManager Software in the Cluster

The disk repair issues that are strictly cluster-specific are actuallyunrelated to Solaris Volume Manger software. That is, you have the sameissues regardless of which volume manager you choose to use.

This section assumes that you are working on a non-hardware-RAID justa bunch of disks (JBOD) disk replacement. Assume that spindlereplacement in hardware-RAID protected arrays have no specific clusterimages, nor cause any breakages that are visible at the software volumemanagement or cluster software layer.

Repairing Physical Device Files on Multiple Nodes

For disk devices where a disk replacement requires repairing Solaris OSphysical device files, such as Sun StorEdge A5x00 arrays, you mustremember to do the Solaris OS repair on all nodes physically connected tothe storage.

For Sun StorEdge A5x00 array, for example, you are used to running theluxadm remove_dev and luxadm insert_dev commands to manageboth the physical replacement of the disk and the repair of the Solaris OSdevice files. You must remember to also run the devfsadm command onthe other node physically connected to the storage.

Repairing the DID Device Database

When any JBOD disks are replaced in the cluster environment, the DIDdatabase must be updated with new physical device information for theunderlying disk.

Make sure you know which disk to update:


Performing Disk Replacement With Solaris Volume Manager Software in the Cluster


Then use the following commands to update and verify the DIDinformation:

# scdidadm -R d7

# scdidadm -l -o diskid d746554a49545355204d4146333336344c2053554e33364720303....

Analyzing a Full Scenario of a Sun StorEdge A5200Array Disk Failure

The following procedure combines the cluster-specific disk replacementissues with the standard Solaris Volume Manager software diskreplacement issues.

1. Identify the failed drive. This is often easiest to do through thevolume management software.

# metastat -s setname# scdidadm -L did-number

2. On the node hosting the device group to which the disk belongs,replace the failed disk. Let the luxadm command guide you throughremoving the failed disk and inserting a new one.

# luxadm remove enclosure,position# luxadm insert enclosure,position

3. On the other node connected to the array, run the devfsadmcommand.

4. On either node, reconfigure the DID information:

# scdidadm -l c#t#d## scdidadm -R d #

5. On any node connected to the storage, copy the partition table froma surviving disk already in the disk set to the replaced disk.

# prtvtoc /dev/did/rdsk/ unbroken-did s0|fmthard -s - \/dev/did/rdsk/ repaired-did s0

6. On any node, delete and add back any broken metadbs.

# metadb -s setname# metadb -s setname -d /dev/did/rdsk/ repaired-did s7# metadb -s setname -a [ -c copies ] /dev/did/rdsk/ repaired-did s7

7. Repair your failed mirror. You use the same command whether a hotspare has been substituted.

# metareplace -s setname d # -e /dev/did/rdsk/ repaired-did s0

Exercise: Configuring Solaris Volume Manager Software




● Initialize the Solaris Volume Manager software local metadbs

● Create and manage disksets and diskset volumes

● Create dual-string mediators if appropriate


● Mirror the boot disk (optional)

Preparation

This exercise assumes you are running the Sun Cluster 3.1 softwareenvironment on Solaris 9 OS.

If you are using Solaris 8 OS Update 7 (02/02), you might have tomanually install Solstice DiskSuite software (packages SUNWmdu, SUNWmdr,SUNWmdx) as well as patch 108693-06 or higher.

Ask your instructor about the location of the software that is needed ifyou are running Solaris 8 OS Update 7.

Each of the cluster host boot disks must have a small unused slice thatyou can use for the local metadbs. The examples in this exercise useslice 7.



During this exercise, you create two data service disksets that eachcontain a single mirrored volume as shown in Figure 7-7.

Figure 7-7 Configuring Solaris Volume Manager Software


Task 1 – Initializing the Solaris Volume ManagerSoftware Local Metadbs

Before you can use Solaris Volume Manager software to create disksets andvolumes, you must initialize the state database and create one or morereplicas.

Configure the system boot disk on each cluster host with a small unusedpartition. This should be slice 7.

Node 1 Node 2Boot Disk Boot Disk

c1 c1

c0 c0

A B A B

Quorum Disk

Primary Disk

Primary Disk

Disksetand Volumes

nfsds

d100

webds

d100

Array A Array B

(replicas)(replicas)

Mirror Disk

Mirror Disk



Perform the following steps:

1. On each node in the cluster, verify that the boot disk has a smallunused slice available for use. Use the format command to verifythe physical path to the unused slice. Record the paths of the unusedslice on each cluster host. A typical path is c0t0d0s7 .

Node 1 replica slice: _____________

Node 2 replica slice: ________________

Caution – You must ensure that you are using the correct slice. A mistakecan corrupt the system boot disk. Check with your instructor.

2. On each node in the cluster, use the metadb command to create threereplicas on the unused boot disk slice.

# metadb -a -c 3 -f replica_slice

Caution – Make sure you reference the correct slice address on each node.You can destroy your boot disk if you make a mistake.

3. On all nodes, verify that the replicas are configured and operational.

# metadbflags first blk block count a u 16 8192 /dev/dsk/c0t0d0s7 a u 8208 8192 /dev/dsk/c0t0d0s7 a u 16400 8192 /dev/dsk/c0t0d0s7

Task 2 – Selecting the Solaris Volume ManagerSoftware Demo Volume Disk Drives

Perform the following steps to select the Solaris Volume Manager softwaredemo volume disk drives:

1. On Node 1, type the scdidadm -L command to list all of theavailable DID drives.

2. Record the logical path and DID path numbers of four disks that youuse to create the demonstration disk sets and volumes in Table 7-2 onpage 7-39. Remember to mirror across arrays.



Note – You need to record only the last portion of the DID path. The firstpart is the same for all DID devices: /dev/did/rdsk .

Note – Make sure the disks you select are not local devices. They must bedual hosted and available to more than one cluster host.

Task 3 – Configuring the Solaris Volume ManagerSoftware Demonstration Disksets

Perform the following steps to create demonstration disksets and volumesfor use in later exercises:

1. On Node 1, create the nfsds diskset, and configure the nodes thatare physically connected to it.

# metaset -s nfsds -a -h node1 node2

2. On Node 1, create the webds diskset and configure the nodes that arephysically connected to it.

# metaset -s webds -a -h node1 node2

3. Add the diskset mediators to each diskset, using the IP addressesfrom the previous example:

# metaset -s nfsds -a -m node1 node2# metaset -s webds -a -m node1 node2

4. Add the primary and mirror disks to the nfsds diskset.

# metaset -s nfsds -a /dev/did/rdsk/ primary \/dev/did/rdsk/ mirror

Table 7-2 Logical Path and DID Numbers

Diskset Volumes PrimaryDisk

MirrorDisk

example d400 c2t3d0 d4 c3t18d0 d15

nfsds d100

webds d100



5. Add the primary and mirror disks to the webds diskset.

# metaset -s webds -a /dev/did/rdsk/ primary \/dev/did/rdsk/ mirror

6. Verify the status of the new disksets.

# metaset -s nfsds# metaset -s webds# medstat -s nfsds# medstat -s webds

# scstat -D

Task 4 – Configuring Solaris Volume Manager SoftwareDemonstration Volumes

Perform the following steps on Node 1 to create a 500-Mbyte mirroredvolume in each diskset, as shown for the two following disksets.

The nfsds Diskset

Follow these steps to create the volume on the nfsds diskset:

1. Create a submirror on each of your disks in the nfsds diskset.

# metainit -s nfsds d0 1 1 /dev/did/rdsk/ primary s0# metainit -s nfsds d1 1 1 /dev/did/rdsk/ mirror s0

2. Create a volume, d99 , and add the d0 submirror to it.

# metainit -s nfsds d99 -m d0

3. Attach the second submirror, d1, to the volume d99 .

# metattach -s nfsds d99 d1

4. Create a 500 Mbyte soft partition, d100 , on top of your mirror. This isthe volume you will actually use for your file system data.

# metainit -s nfsds d100 -p d99 500m

5. Verify the status of the new volume

# metastat -s nfsds



The webds Diskset

Follow these steps to create the volume on the webds diskset:

1. Create a submirror on each of your disks in the webds diskset.

# metainit -s webds d0 1 1 /dev/did/rdsk/ primary s0# metainit -s webds d1 1 1 /dev/did/rdsk/ mirror s0

2. Create a volume, d99 , and add the d0 submirror to it.

# metainit -s webds d99 -m d0

3. Attach the second submirror, d1, to the volume d99 .

# metattach -s webds d99 d1

4. Create a 500-Mbyte soft partition on top of your mirror. This is thevolume you will actually use for your file system data.

# metainit -s webds d100 -p d99 500m

5. Verify the status of the new volume

# metastat -s webds

Task 5 – Creating a Global nfs File System

Perform the following steps on Node 1 to create a global file system in thenfsds diskset:

1. On Node 1, create a file system on d100 in the nfsds diskset.

# newfs /dev/md/nfsds/rdsk/d100


# mkdir /global/nfs


/dev/md/nfsds/dsk/d100 /dev/md/nfsds/rdsk/d100 \/global/nfs ufs 2 yes global,logging


4. On Node 1, mount the /global/nfs file system.

# mount /global/nfs


# mount# ls /global/nfslost+found



Task 6 – Creating a Global web File System

Perform the following steps on Node 1 to create a global file system in thewebds diskset:

1. On Node 1, create a file system on d100 in the webds diskset.

# newfs /dev/md/webds/rdsk/d100


# mkdir /global/web


/dev/md/webds/dsk/d100 /dev/md/webds/rdsk/d100 \/global/web ufs 2 yes global,logging


4. On Node 1, mount the /global/web file system.

# mount /global/web


# mount# ls /global/weblost+found

Task 7 – Testing Global File Systems

Perform the following steps to confirm the general behavior of globallyavailable file systems in the Sun Cluster 3.1 software environment:

1. On Node 2, move into the /global/nfs file system.

# cd /global/nfs

2. On Node 1, try to unmount the /global/nfs file system.

An error is displayed informing you that the file system is busy.

3. On Node 2, move out of the /global/nfs file system (cd / ), and tryto unmount it again on Node 1.

4. Mount the /global/nfs file system on Node 1.

5. Try unmounting and mounting /global/nfs from all nodes.



Task 8 – Managing Disk Device Groups

Perform the following steps to migrate a disk device group (diskset)between cluster nodes:

1. Make sure the device groups are online (to the Sun Cluster software).

# scstat -D

Note – You can bring a device group online to a selected node as follows:# scswitch -z -D nfsds -h node1

2. Verify the current demonstration device group configuration.

proto193:/# scconf -p |grep groupproto193:/# scconf -p|grep groupDevice group name: nfsds Device group type: SVM Device group failback enabled: no Device group node list: proto192, proto193 Device group ordered node list: yes Device group desired number of secondaries: 1 Device group diskset name: nfsds

Device group name: webds Device group type: SVM Device group failback enabled: no Device group node list: proto192, proto193 Device group ordered node list: yes Device group desired number of secondaries: 1 Device group diskset name: webds


The nfsds and webds disksets should automatically migrate toNode 2 (verify with the scstat -D command).

4. Boot Node 1. Both disksets should remain mastered by Node 2.

5. Use the scswitch command from either node to migrate the nfsdsdiskset to Node 1.

# scswitch -z -D nfsds -h node1



Task 9 – Viewing and Managing Solaris VolumeManager Software Device Groups

In this task, you view and manage Solaris Volume Manager softwaredevice groups through SunPlex Manager. Perform the following steps onyour administration workstation:

1. In a web browser, log in to SunPlex Manager on any cluster node:


2. Click on the Global Devices folder on the left.

3. Investigate the status information and graphical topologyinformation that you can see regarding Solaris Volume Managersoftware device groups.


5. Use the Action Menu to switch the primary for one of your devicegroups.

Task 10 – (Optional) Mirroring the Boot Disk WithSolaris Volume Manager Software

You can try this procedure if you have time. Normally, you would run iton all nodes, but feel free to run it on one node only as a “proof ofconcept.”

This task assumes you have a second drive with identical geometry to thefirst. If you do not, you could still perform the parts of the lab that do notrefer to the second disk; that is, create mirrors with only one sub-mirrorfor each partition on your boot disk.

The example uses c0t8d0 as the second drive.

Repartitioning the Second Drive and Adding metadb Replicas

Use the following commands to repartition the second drive and addmetadbs:

# prtvtoc /dev/rdsk/c0t0d0s0|fmthard -s -/dev/rdsk/c0t8d0s0fmthard: New volume table of contents now in place.

# metadb -a -c 3 c0t8d0s7



Using the metadb Command to Verify Metadb Replicas

Use the following command to verify the metadbs:

# metadb flags first blk block count a u 16 8192 /dev/dsk/c0t0d0s7 a u 8208 8192 /dev/dsk/c0t0d0s7

a u 16400 8192 /dev/dsk/c0t0d0s7a u 16 8192 /dev/dsk/c0t8d0s7

a u 8208 8192 /dev/dsk/c0t8d0s7a u 16400 8192 /dev/dsk/c0t8d0s7

Making Mirrors for Non-Root Slices

For each non-root slice on your drive (including swap but not includingthe metadb partition), make submirrors from the partition on each of yourtwo disks. Do not attach the second submirror. Edit the vfstab filemanually to reference the volume instead of the partition or did device.Do not reboot yet.

The example shows the procedure for the swap partition. If you are doingthis on two nodes, make sure the volume number chosen for the/global/.devices/node@# mirror is different on each node.

# metainit -f d21 1 1 c0t0d0s1d11: Concat/Stripe is setup# metainit d22 1 1 c0t8d0s1d12: Concat/Stripe is setup# metainit d20 -m d21d10: Mirror is setup# vi /etc/vfstab(change correct line manually)

/dev/md/dsk/d20 - - swap - no -



Making the Mirror for the Root Slice

You do not have to edit the vfstab or system files manually here; themetaroot command takes care of it for you.

# metainit -f d11 1 1 c0t0d0s0d11: Concat/Stripe is setup# metainit d12 1 1 c0t8d0s0d12: Concat/Stripe is setup# metainit d10 -m d11d10: Mirror is setup# metaroot d10

Rebooting and Attaching the Second Submirror

Use the following commands to reboot and attach the second submirror:

# init 6# df -kproto192:/# swap -l# metattach d10 d12(etc)# metattach d50 d52

Exercise Summary


Exercise Summary

?!


● Experiences

● Interpretations

● Conclusions

● Applications


Module 8

Managing the Public Network With IPMP

Objectives


● Define the purpose of IPMP

● Define the requirements for an IPMP group

● List examples of network adapters in IPMP groups on a singleSolaris OS server

● Describe the operation of the in.mpathd daemon

● List the new options to the ifconfig command that support IPMP,and configure IPMP with /etc/hostname. xxx files

● Perform a “forced failover” of an adapter in an IPMP group

● Configure IPMP manually with IPMP commands

● Describe the integration of IPMP into the Sun Cluster softwareenvironment

Relevance


Relevance

?!


● Should you configure IPMP before or after you install the SunCluster software?

● Why is IPMP required even if you do not have redundant networkinterfaces?

● Is the configuration of IPMP any different in the Sun Clustersoftware environment than on a standalone Solaris OS?

● Is the behavior of IPMP any different in the Sun Cluster softwareenvironment?


Managing the Public Network With IPMP 8-3Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1





● Sun Microsystems, Inc. IP Network Multipathing Administration Guide,part number 816-5249.

Introducing IPMP


Introducing IPMP

IPMP has been a standard part of the base Solaris OS since Solaris 8 OSUpdate 3 (01/01).

IPMP enables you to configure redundant network adapters, on the sameserver (node) and on the same subnet, as an IPMP failover group.

IPMP detects failures and repairs of network connectivity for adapters,and provides failover and failback of IP addresses among members of thesame group. Existing TCP connections to the IP addresses that fail over aspart of an IPMP group are interrupted for short amounts of time, and arenot disconnected.

In the Sun Cluster 3.1 software environment, it is required to use IPMP tomanage any public network interfaces on which you will place the IPaddresses associated with applications running in the cluster.

These application IP addresses are implemented by mechanisms knownas LogicalHostname and SharedAddress resources. The validationmethods associated with these resources will not allow their configurationunless the public network adapters being used are already under controlof IPMP. Module 9, “Introducing Data Services, Resource Groups, andHA-NFS,” and Module 10, “Configuring Scalable Services,” provide detailabout the proper configuration of applications and their IP addresses inthe Sun Cluster software environment.

Describing General IPMP Concepts



IPMP allows you to group network adapters for redundancy. Members ofthe same IPMP group are identified by a common group name. This canbe any alphanumeric name. The group name is only meaningful inside asingle Solaris OS server.

Defining IPMP Group Requirements

You must observe the following rules when configuring an IPMP group:

● A network adapter can be a member of only one group.

● All members of a group must be on the same subnet. The members,if possible, should be connected to physically separate switches onthe same subnet.

● Adapters on different subnets must be in different groups.

● It is possible to have multiple groups on the same subnet.

● You can have a group with only one member; however, there cannotbe any failover associated with this adapter.

● Each Ethernet network interface must have a unique MAC address.This is achieved by setting the local-mac-address? variable totrue in the OpenBoot PROM.

● Network adapters in the same group must be the same type. Forexample, you cannot combine Ethernet interfaces and AsynchronousTransfer Mode (ATM) interfaces in the same group.

● Each adapter assigned to an IPMP group requires a dedicated testinterface. This is an extra static IP for each group member specificallyconfigured for the purposes of testing the health of the adapter.

The test interface enables test traffic on all the members of the IPMPgroup. This is reason that local-mac-address?=true is required forIPMP.



Configuring Standby Interfaces in a Group

In an IPMP group with two or more members, it is optional to configurean adapter as a standby adapter for the group. Standby adapters have thefollowing properties:

● You are allowed (and must) configure only the test interface on theadapter. Attempt to manually configure any other interfaces will fail.

● Additional IP addresses will be added to the adapter only as a resultof failure of another member of the group.

● The standby interface will be preferred as a failover target of anothermember of the group fails.

● You must have at least one member of the group that is not astandby adapter.

Note – The examples of two-member IPMP groups in the Sun Clustersoftware environment do not use any standby adapters. This allows theSun Cluster software to place additional IP addresses associated withapplications on both members of the group simultaneously.

If you had a three-member IPMP group in the Sun Cluster softwareenvironment, setting up one of the members as a standby is still a validoption.

Examining IPMP Group Examples



The following sections describe and illustrate examples of networkadapters in IPMP groups on a single Solaris OS server.

Single IPMP Group With Two Members and NoStandby

Figure 8-1 shows a server with two member adapters in a single IPMPgroup. These two adapters must be on the same subnet and providefailover for each other.

Figure 8-1 Server With Two Member Adapters in a Single IPMP Group

Solaris™ OS Server

qfe0 qfe4

Group:Therapy

mutual failover



Single IPMP Group With Three Members Including aStandby

Figure 8-2 shows a server with three members of a single IPMP group.One of the adapters in the group is configured as a standby interface.

Figure 8-2 Server With Three Members of a Single IPMP Group

Two IPMP Groups on Different Subnets

Figure 8-3 shows how different IPMP groups must be used for adapterson different subnets.

Figure 8-3 Two IPMP Groups on Different Subnets

Solaris™ Operating System Server

qfe0 qfe4

Group:Therapy

failover

qfe8 (standby)


qfe0 qfe4

qfe1

qfe5

Group:Mentality

Group:Therapy



Two IPMP Groups on the Same Subnet

Figure 8-4 shows two different IPMP groups configured on the samesubnet. Failover still occurs only within each particular group.

Figure 8-4 Two IPMP Groups on the Same Subnet


qfe0 qfe4 qfe1 qfe5

Group:Therapy

Group:Mentality

Describing IPMP


Describing IPMP

The in.mpathd daemon controls the behavior of IPMP. This behavior canbe summarized as a three-part scheme:

● Network path failure detection

● Network path failover

● Network path failback

The in.mpathd daemon starts automatically when an adapter is made amember of an IPMP group through the ifconfig command.

Network Path Failure Detection

The in.mpathd daemon sends Internet Control Message Protocol (ICMP)echo probes (pings) to the targets connected to the link on all adaptersthat belong to a group to detect failures and repair. The test interface isused as the source address of these pings.

Because the in.mpathd daemon determines what targets to probedynamically, you cannot configure the targets. Targets are chosen in thisorder:

1. Default router (if you have a single default router, it is the onlytarget)

2. Other routers

3. Other targets discovered by a ping command to the 224.0.0.1(all-hosts) multicast IP address

To ensure that each adapter in the group functions properly, thein.mpathd daemon probes all the targets separately through all theadapters in the multipathing group, using each adapter’s test interface. Ifthere are no replies to five consecutive probes, the in.mpathd daemonconsiders the adapter as having failed. The probing rate depends on thefailure detection time (FDT). The default value for failure detection time is10 seconds. For a failure detection time of 10 seconds, the probing rate isapproximately one probe every two seconds.

Describing IPMP


Network Path Failover

After a failure is detected, failover of all network access occurs from thefailed adapter to another functional adapter in the group. If you haveconfigured a standby adapter, the in.mpathd daemon chooses that forfailover of IP addresses and multicast memberships. If you have notconfigured a standby adapter, in.mpathd chooses the adapter with theleast number of IP addresses. The new adapter assumes all of the IPaddresses of the failed adapter, except for the test interface.

Network Path Failback

The in.mpathd daemon uses the fixed test interface to determine whetherthe failed path has been repaired. If the in.mpathd daemon determinesthat the path has been repaired, the IP addresses moved in the failover arereturned to their original path, assuming failback is enabled.

Configuring IPMP


Configuring IPMP

This section outlines how IPMP is configured directly through options tothe ifconfig command.

Most of the time, you just specify the correct IPMP-specific ifconfigoptions in the /etc/hostname. xxx files. After these are correct, you willnever have to change them again.

Examining New ifconfig Options for IPMP

Several new ifconfig options have been added for use with IPMP. Theyare:

● group groupname – Adapters on the same subnet are placed in afailover group by configuring them with the same groupname. Thegroup name can be any name and must be unique only within asingle Solaris OS (node). It is irrelevant if you have the same ordifferent group names across the different nodes of a cluster.

● -failover – Use this option to demarcate the test interface for eachadapter. The test interface is the only interface on an adapter thatdoes not fail over to another adapter when a failure occurs.

● deprecated – This option, while not required, is generally alwaysused on the test interfaces. Any IP address marked with this optionis not used as the source IP address for any client connectionsinitiated on this machine.

● standby – This option, when used with the physical adapter(-failover deprecated standby ), turns that adapter into astandby-only adapter. No other virtual interfaces can be configuredon that adapter until a failure occurs on another adapter on thegroup.

● addif – Use this option to create the next available virtual interfacefor the specified adapter. The maximum number of virtual interfacesper physical interface is 8192.

The Sun Cluster 3.1 software automatically uses the ifconfig xxxaddif command to add application-specific IP addresses as virtualinterfaces on top of the physical adapters that you have placed inIPMP groups. Details about configuring these IP addresses are inModule 9, “Introducing Data Services, Resource Groups, and HA-NFS,” and Module 10, “Configuring Scalable Services.”

Configuring IPMP


Putting Test Interfaces on Physical or Virtual Interfaces

IPMP was designed so that the required test interface for each member ofan IPMP group could be either on the physical interface (for example,qfe0 ) or on a virtual interface (for example, qfe0:1 , created with theifconfig xxx addif command).

There are many existing IPMP documents written with the test IP on thephysical interface because it looks “cleanest” to have only the virtualinterfaces failing over to other adapters and the IP on the physicalinterface always staying where it is.

However, at the time of writing of this course, there is a relatively obscurebug that manifests itself when the test IP is on the physical interface.

The examples in this module follow the convention of placing the testinterface on a virtual interface, when given the choice.

Using ifconfig Commands to Configure IPMPExamples

While most of the time you will set up your /etc/hostname. xxx filesonce and never have to worry about IPMP configuration again, theseexamples show using the ifconfig command directly with theIPMP-specific options.

The /etc/inet/hosts fragment shows the dedicated IP address for thetest interface for each adapter. While it is not required to have host namesfor these, remember that these IPs are reserved.

# cat /etc/inet/hosts. #physical host address 172.20.4.192 proto192

# test addresses for proto192 (node 1) 172.20.4.194 proto192-qfe1-test 172.20.4.195 proto192-qfe2-test

# ifconfig qfe1 proto192 group therapy netmask + broadcast + up

# ifconfig qfe1 addif proto192-qfe1-test -failover deprecated \netmask + broadcast + up

Configuring IPMP


# ifconfig qfe2 plumb# ifconfig qfe2 proto192-qfe2-test group therapy -failover deprecated \netmask + broadcast + up

# ifconfig -a.qfe1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 inet 172.20.4.192 netmask ffffff00 broadcast 172.20.4.255

groupname therapy ether 8:0:20:f1:2b:dqfe1:1:flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAI

LOVER> mtu 1500 index 2 inet 172.20.4.194 netmask ffffff00 broadcast 172.20.4.255qfe2:flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILO

VER> mtu 1500 index 3 inet 172.20.4.195 netmask ffffff00 broadcast 172.20.4.255

groupname therapy ether 8:0:20:f1:2b:e

Configuring the /etc/hostname. xxx Files for IPMP

You are unlikely to want to actually type ifconfig commands as theyappear on the previous page on any type of regular basis.

Instead, the ifconfig commands get run automatically at Solaris OS boottime by way of their presence in the /etc/hostname. xxx files.

The “new format” of these files (since Solaris 8 OS) allows you to put allthe options to the ifconfig command in these files.

You can omit the netmask + broadcast + from these files as this isalways automatically applied by the Solaris OS boot process as a groupfor all of the interfaces that have been configured.

# cat /etc/hostname.qfe1 proto192 group therapy up addif proto192-qfe1-test -failover deprecated up

# cat /etc/hostname.qfe2 proto192-qfe2-test group therapy -failover deprecated up

Configuring IPMP


Multipathing Configuration File

The in.mpathd daemon uses the settings in the /etc/default/mpathdconfiguration file to invoke multipathing. Changes to this file are read bythe in.mpathd daemon at startup and on a SIGHUP. This file contains thefollowing default settings and information:

## Time taken by mpathd to detect a NIC failure in ms.# The minimum time that can be specified is 100 ms.#FAILURE_DETECTION_TIME=10000## Failback is enabled by default. To disable failback# turn off this option#FAILBACK=yes## By default only interfaces configured as part of# multipathing groups are tracked. Turn off this# option to track all network interfaces on the system#TRACK_INTERFACES_ONLY_WITH_GROUPS=yes

Note – Generally, you do not need to edit the default/etc/default/mpathd configuration file.

The three settings you can alter in this file are:

● FAILURE_DETECTION_TIME– You can lower the value of thisparameter. If the load on the network is too great, the system cannotmeet the failure detection time value. Then the in.mpathd daemonprints a message on the console, indicating that the time cannot bemet. It also prints the time that it can meet currently. If the responsecomes back correctly, the in.mpathd daemon meets the failuredetection time provided in this file.

● FAILBACK – After a failover, failbacks take place when the failedadapter is repaired. However, the in.mpathd daemon does not failback the addresses if the FAILBACK option is set to no.

● TRACK_INTERFACES_ONLY_WITH_GROUPS– In standalone servers,you can set this to no so that in.mpathd monitors traffic even onadapters not in groups. In the cluster environment make sure youleave the value yes so that private transport adapters are not probed.

Performing Failover and Failback Manually


Performing Failover and Failback Manually

You can do a “forced failover” of an adapter in an IPMP group. Thefollowing is an example:

# if_mpadm -d qfe1

This command causes IPMP to behave as if qfe1 had failed. All IPaddresses, except the test interface, are failed over to another member ofthe group. The interface is marked down and is not accessed byin.mpathd until re-enabled.

To re-enable an adapter after this operation, use:

# if_mpadm -r qfe1

This command allows the adapter to take back the IP addresses to whichit was originally assigned, assuming FAILBACK=yes is set in the/etc/default/mpathd file.

Configuring IPMP in the Sun Cluster 3.1 Software Environment


Configuring IPMP in the Sun Cluster 3.1 SoftwareEnvironment

There are no special cluster-related tools for configuring IPMP in theSun Cluster 3.1 software environment. IPMP is configured on each nodeexactly as in a non-clustered environment.

Configuring IPMP Before or After Cluster Installation

It makes no difference whether IPMP is configured before or after thescinstall utility is run to configure the cluster framework. Thescinstall utility does automatically set the local-mac-address?variable to true for you.

You might choose to make IPMP configuration part of your Solaris OSJumpStart software installation, by copying in the correct/etc/hostname. xxx files as part of a JumpStart software finish script.

Using Same Group Names on Different Nodes

It makes no difference to IPMP whether you use the same group names ordifferent group names for IPMP across nodes. IPMP itself is aware only ofwhat is going on in the local Solaris OS.

It is a helpful convention to use the same group names to indicate groupsof adapters on different nodes that are connected to the same subnet.

Understanding Standby and Failback

It is unlikely that you would want to use a standby interface in atwo-member IPMP group in the cluster environment. If you do, the SunCluster software is unable to load-balance additional application-relatedIP addresses across the members of the group.

Keep FAILBACK=yes in the /etc/default/mpathd file. Because the SunCluster software is automatically trying to load-balance additional IPaddresses across the members of the group, it makes sense that youwould want them to “rebalance” when a repair is detected.

Integrating IPMP Into the Sun Cluster 3.1 Software Environment


Integrating IPMP Into the Sun Cluster 3.1 SoftwareEnvironment

There is nothing special about IPMP itself in the cluster. In the clusterenvironment, as in a non-cluster environment, in.mpathd is concernedwith probing network adapters only on a single Solaris OS, and utilityfailovers and failbacks between the adapters in a group only on a singleSolaris OS.

The Sun Cluster software environment, however, needs additionalcapability wrapped around IPMP to:

● Store cluster-wide status of IPMP groups on each node on the CCR,and enable retrieval of this status with the scstat command.

● Facilitate application failover in the case where all members of anIPMP group on one node have failed, but the corresponding groupon the same subnet on another node has a healthy adapter.

Clearly IPMP itself is unaware of any of these cluster requirements.Instead, the Sun Cluster 3.1 software uses a cluster-specific publicnetwork management daemon (pnmd) to perform this cluster integration.

Capabilities of the pnmdDaemon in Sun Cluster 3.1Software

In the cluster environment, the pnmd daemon has the followingcapabilities:

● Populate CCR with public network adapter status

● Facilitate application failover

When pnmd detects that all members of a local IPMP group have failed, itconsults a file named /var/cluster/run/pnm_callbacks . This filecontains entries that would have been created by the activation ofLogicalHostname and SharedAddress resources. (There is moreinformation about this in Module 9, “Introducing Data Services, ResourceGroups, and HA-NFS,” and Module 10, “Configuring Scalable Services.”)



It is the job of the hafoip_ipmp_callback , in the following example, todecide whether to migrate resources to another node.

# cat /var/cluster/run/pnm_callbackstherapy proto-nfs.mon \/usr/cluster/lib/rgm/rt/hafoip/hafoip_ipmp_callback mon nfs-rg proto-nfs

Summary of IPMP Cluster Integration

Figure 8-5 summarizes the IPMP and pnmd elements of public networkmanagement in the cluster.

Figure 8-5 IPMP Cluster Integration

failover resources

hafoip_ipmp_callback

in.mpathd in.mpathd

pnmd pnmd



Viewing Cluster-Wide IPMP Information With thescstat Command

In the Sun Cluster 3.1 software environment, running the scstat -icommand from any node shows the status of IPMP group members on allthe nodes. If you use the scstat command without any options, then theIPMP information is included among all the other cluster statusinformation.

# scstat -i

-- IPMP Groups --

Node Name Group Status Adapter Status --------- ----- ------ ------- ------ IPMP Group: proto192 therapy Online qfe2 Online IPMP Group: proto192 therapy Online qfe1 Online

IPMP Group: proto193 therapy Online qfe2 Online IPMP Group: proto193 therapy Online qfe1 Online

Exercise: Configuring and Testing IPMP



Perform the following exercises on all cluster nodes. It is assumed that theSun Cluster software is installed and operational, but that IPMP has notbeen previously set up on the nodes.

Task 1 – Verifying the local-mac-address? Variable

Perform the following steps on each node in the cluster:

1. Verify that the EEPROM local-mac-address? variable is set totrue :

# eeprom “local-mac-address?”

It is set to true by the scinstall utility at cluster install time, so theonly reason it should be false now is if someone changed it backmanually.

2. If for some reason you need to modify the variable, do so, and rebootthe node.

Task 2 – Verifying the Adapters for the IPMP Group

Perform the following steps on each node of the cluster:

1. Make sure you know which are the redundant adapters on thepublic network. You might have already written this down in theexercises for Module 3, “Preparing for Installation andUnderstanding Quorum Devices.”

Your primary public network adapter should be the only onecurrently configured on the public net. You can verify this with:

# ls -l /etc/hostname.*# ifconfig -a

You can verify your secondary public net by:

● Making sure it is not configured as a private transport

● Making sure it can snoop public network broadcast traffic:

# ifconfig ifname plumb# snoop -d ifname(other window or node)# ping -s pubnet_broadcast_addr



Task 3 – Verifying or Entering Test Addresses in the/etc/inet/hosts File

It is a good idea, although not required, to have test IP addresses in thehosts file. While one node does not really need to know anything aboutanother node’s test addresses, it is advisable to have all test addresses inthe hosts file for all nodes to indicate that these addresses are reservedand what they are reserved for.

1. Verify with your instructor which IP addresses should be used forthe test interfaces for each adapter on each node.

2. Enter the IP addresses in /etc/inet/hosts on each node if they arenot already there.

The following is only an example. It really does not matter if you usethe same names as another group working on another cluster, aslong as the IP addresses are different:

# IPMP TEST ADDRESSES172.20.4.194 proto192-qfe1-test172.20.4.195 proto192-qfe2-test172.20.4.197 proto193-qfe1-test172.20.4.198 proto193-qfe2-test

Task 4 – Creating /etc/hostname. xxx Files

On each node, create the appropriate /etc/hostname. xxx files to placeadapters in IPMP groups. The group name is unimportant. The followingare just examples:

# vi /etc/hostname.qfe1 proto192 group therapy up addif proto192-qfe1-test -failover deprecated up# vi /etc/hostname.qfe2 proto192-qfe2-test group therapy -failover deprecated up



Task 5 – Rebooting and Verifying That IPMP IsConfigured

The following can be performed on one node at a time, so that the clusterstays active the entire time:

1. Reboot the node.

2. Verify the new IPMP configuration with ifconfig -a .

3. Verify IPMP cluster-wide status with scstat -i .

Task 6 – Verifying IPMP Failover and Failback

Test the following on at least one of your nodes:

1. From outside of the cluster, launch ping -s nodename, and keep itrunning.

2. If you have physical access to your cluster hardware, unplug theEthernet cable from the network adapter that currently has the nodephysical interface on it.

If you have no physical access, you can sabotage your adapter with:

# ifconfig adapter_name modinsert ldterm@2

3. Observe the node messages (on the console or in the/var/adm/messages file).

4. Observe the output of the scstat -i command.

5. Observe the behavior of your command from Step 1 (keep itrunning).

If you have physical access, reattach the broken cable. If you have nophysical access, use the following to repair your sabotage:

# ifconfig adapter_name modremove ldterm@2

6. Observe the messages and the behavior of your command fromStep 1.

7. Observe the output of the scstat -i command.

Exercise Summary


Exercise Summary

?!


● Experiences

● Interpretations

● Conclusions

● Applications


Module 9

Introducing Data Services, ResourceGroups, and HA-NFS

Objectives


● Describe how data service agents enable a data service in a cluster tooperate properly

● List the components of a data service agent

● Describe data service packaging, installation, and registration

● Describe the primary purpose of resource groups

● Differentiate between failover and scalable data services

● Configure and run NFS in the cluster

● Describe how to use special resource types

● List the components of a resource group

● Differentiate standard, extension, and resource group properties

● Describe the resource group configuration process

● List the primary functions of the scrgadm command

● Use the scswitch command to control resources and resourcegroups

● Use the scstat command to view resource and group status

● Use the scsetup utility for resources and resource group operations

Relevance


Relevance

?!


● What is a data service agent for the Sun Cluster 3.1 software?

● What is involved in fault monitoring a data service?

● What is the difference between a resource and a resource group?

● What are the different types of properties?

● What are the specific requirements for setting up NFS in the SunCluster 3.1 software environment?


Introducing Data Services, Resource Groups, and HA-NFS 9-3Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1


Additional resources – The following reference provides additionalinformation on the topics described in this module:

Sun Cluster 3.1 Data Services 4/04 Collection

Introducing Data Services in the Cluster



The Sun Cluster 3.1 software framework makes applications highlyavailable, minimizing application interruptions after any single failure inthe cluster.

In addition, some applications, such as Apache Web Server software, aresupported not only with high availability features but also in a scalableconfiguration. This configuration allows the service to run on multiplenodes of the cluster simultaneously while providing a single IP address tothe client.

Off-the-Shelf Applications

For most applications supported in the Sun Cluster software environment,software customization is not required to enable an application to runcorrectly in the cluster. A Sun Cluster 3.1 software agent enables the dataservice to run correctly in the cluster environment.

Application Requirements

Identify requirements for all of the data services before you begin Solarisand Sun Cluster software installation. Failure to do so might result ininstallation errors that require you to completely reinstall the Solaris andSun Cluster software.

Determining the Location of the Application Binaries

You can install the application software and application configuration fileson one of the following locations.

● The local disks of each cluster node

Placing the software and configuration files on the individual clusternodes lets you upgrade application software later without shuttingdown the service. The disadvantage is that you have several copiesof the software and configuration files to maintain and administer.



● The cluster file system

If you put the application binaries on the cluster file system, youhave only one copy to maintain and manage. However, you mustshut down the data service in the entire cluster to upgrade theapplication software. If you can spare a small amount of downtimefor upgrades, place a single copy of the application andconfiguration files on the cluster file system.

● Highly available local file system

Using HAStoragePlus , you can integrate your local file system intothe Sun Cluster environment, making the local file system highlyavailable. HAStoragePlus provides additional file systemcapabilities, such as checks, mounts, and unmounts, enablingSun Cluster to fail over local file systems. To failover, the local filesystem must reside on global disk groups with affinity switchoversenabled.

Sun Cluster 3.1 Software Data Service Agents

The word agent is an informal name for a set of components, writtenspecifically for Sun Cluster 3.1 software, that enable a data service in acluster to operate properly. Figure 9-1 shows the relationship between astandard application and the data service agent.

Figure 9-1 Standard Application and Data Service Agent

StandardApplication

Data ServiceAgent

startstop

faultmonitors

Reviewing Components of a Data Service Agent


Reviewing Components of a Data Service Agent

Typical components of a data service agent include:

● Fault monitors for the service

● Methods to start and stop the service in the cluster

● Methods to start and stop the fault monitoring

● Methods to validate configuration of the service in the cluster

● A registration information file that allows the Sun Cluster softwareto store all the information about the methods for you. You then onlyneed to reference a resource type to refer to all the components of theagent.

Sun Cluster software provides an API that can be called from shellprograms, and an API which can be called from C or C++ programs. Allof the program components of data service methods that are supported bySun products are actually compiled C and C++ programs.

Data Service Agent Fault-Monitor Components

Fault-monitoring components specific to data services in Sun Cluster 3.1software are run on the local node only. This is the same node that isrunning the data service. Fault-monitoring components are intended todetect application failure, and can suggest either application restarts orfailovers in the cluster. Unlike Sun Cluster 2.x software, no remote faultmonitoring is running on backup nodes that could suggest a takeover.

While the actual capabilities of these fault monitors is application specificand often poorly documented, the general strategy for fault monitors inthe Sun Cluster 3.1 environment is to monitor the health of the following:

● The daemons by placing them under control of the process monitoringfacility (rpc.pmfd ). This facility calls action scripts if data servicedaemons stop unexpectedly.

● The service by using client commands.

Note – Data service fault monitors do not need to monitor the health ofthe public network itself, as this is already done by the combination ofin.mpathd and pnmd, as described in Module 8, “Managing the PublicNetwork With IPMP.”

Introducing Data Service Packaging, Installation, and Registration


Introducing Data Service Packaging, Installation, andRegistration

You install most agents for Sun Cluster 3.1 software data services asseparate Solaris OS packages by using the pkgadd command.

Many of these agents are on the Sun Java System CD in thesun_cluster_agents directory. The scinstall utility includes a menuitem that allows you to install these packages from the menu interfacerather than by using pkgadd command.

Some agents are supplied with the application software, rather than withthe cluster software.

Data Service Packages and Resource Types

Each data service agent encapsulates all the information about the agentas a resource type.

When this resource type is registered with the cluster software, you do notneed to know the location or names of the components of the agent. Youonly need to reference an application instance’s resource type todetermine all the correct information about methods and fault monitorsfor that component.

Note – The package that you add by using the pkgadd command to installan agent, and the corresponding resource type usually have differentnames. For example, when you install the SUNWschtt package, a resourcetype called SUNW.iws becomes available.

Introducing Resources, Resource Groups, and the Resource Group Manager


Introducing Resources, Resource Groups, and theResource Group Manager

Data services are placed under the control of the cluster by configuringthe services as resources in resource groups. The rgmd daemon is theresource group manager, which controls all activity having to do withresources and resource groups as shown in Figure 9-2. That is, the rgmddaemon controls all data service activity in the cluster.

Figure 9-2 Resource Group Manager

Resources

In the context of clusters, the word resource refers to any element abovethe layer of the cluster framework which can be turned on or off, and canbe monitored in the cluster.

The most obvious example of a resource is an instance of a running dataservice. For example, an Apache Web Server, with a single httpd.conffile, counts as one resource.

ResourceGroup

Manager

ResourceGroup

Resource Resource

Resource Type Resource Type

controls

contains contains

is a is a



Each resource has a specific type. For example, the type for ApacheWeb Server is SUNW.apache.

Other types of resources represent IP and storage resources that arerequired by the cluster.

A particular resource is identified by:

● Its type, however, the type is not unique for each instance

● A unique name, which is used as input and output in utilities

● A set of properties, which are parameters that define a particularresource

You can configure multiple resources of the same type in the cluster, eitherin the same or different resource groups.

For example, you might want to run two Apache Web Server applicationservices on different nodes in the cluster. In this case, you have tworesources, both of type SUNW.apache, in two different resource groups.

Resource Groups

Resource groups are collections of resources. Resource groups are eitherfailover or scalable.

Failover Resource Groups

For failover applications in the cluster, the resource group becomes the unitof failover. That is, the resource group is the collection of services thatalways run together on one node of the cluster at one time, andsimultaneously failover or switchover to another node.

Scalable Resource Groups

For scalable applications, the resource group describes the collection ofservices that simultaneously run on one or more nodes.

You must configure two separate resource groups for scalableapplications. Scalable applications are described in detail in Module 10,“Configuring Scalable Services.”



Number of Data Services in a Resource Group

You can put multiple data services in the same resource group.

For example, you could run two Sun Java System Web Server (formerlyknown as Sun ONE Web Server) application services, a Sun Java SystemDirectory Server (formerly known as Sun ONE Directory Server) softwareidentity management service, and a Sybase database as four separateresources in the same failover resource group. These applications alwaysrun on the same node at the same time, and always simultaneouslymigrate to another node.

On the other hand, you could put all four of the preceding services inseparate resource groups. The services could still run on the same node,but would failover and switchover independently.

Benefits of Putting All Services in a Single Resource Group

If you put all data services in the same failover resource group of atwo-node cluster, the node that is not currently hosting the resource groupis in a pure backup mode. Different services do not run on differentnodes.

Some customers prefer to deploy services in a single resource groupbecause this configuration provides more predictable behavior. If the nodecurrently running all the data services is performing optimally, and thereis a failover, you can predict that the new node has the same performance(assuming equivalent servers) because the node was not doing anythingelse to start with.



Resource Group Manager

The rgmd daemon maintains all information about data services that areknown to the cluster as resources in resource groups. The rgmd daemonlaunches all scripts that start resources in resource groups and performsall failovers.

Unlike the rest of the cluster framework, the rgmd daemon is a typicaldaemon that does not add any special range of operations to the SolarisOS kernel. Most switching and failing services occur through typicalSolaris OS activity. For example, the Solaris OS calls methods that directlyor indirectly launch daemons. In addition, the ifconfig addifcommand and the ifconfig removeif command migrate IP addressesthat are used for services.

Describing Failover Resource Groups



The following example is provided to help you understand the generalconcept of resources as part of specific resource groups.

A resource name must be globally unique, not merely unique inside aparticular resource group (Figure 9-3).

Figure 9-3 Failover Resource Group

The previous failover resource group has been defined with a uniquename in the cluster. Configured resource types are placed into the emptyresource group. These are known as resources. By placing these resourcesinto the same failover resource group, they fail over together.

Application Resource

Resource Type:SUNW.nfs

Data Storage Resource

Resource Type:SUNW.HAStoragePlus

Logical Host Resource

Resource Type:SUNW.LogicalHostname

Failover Resource Group



Resources and Resource Types

Each instance of a data service under cluster control is represented by aresource in a resource group.

Resources exist only inside resource groups. There is no such thing as adisembodied resource that is not a member of a resource group.

Each resource has a resource type that describes the type of resource it is(for example, SUNW.nfs for an NFS resource).

At least one defined resource type exists for each supported service in thecluster. Some applications that are typically considered to be a singleentity, such as an instance of the Oracle database, actually require twodifferent resources with different types: the Oracle server and the Oraclelistener.

In addition to resource types that relate to data services, there are a fewother special resource types that relate to IP addresses and storage. Theseresource types are described in ‘‘Using Special Resource Types’’ onpage 9-15.

Resource Type Versioning

Sun Cluster 3.1 software gives you the ability to use different versions ofresource types. For example, old and new versions of data service agentscan co-exist as separate types. Individual resources of an original resourcetype can be upgraded to a new type on a resource-by-resource basis.

Most data service resources in the Sun Cluster 3.1 software have 3.1 astheir version number. In fact, this version number is appended to theresource type name. For example, the official name of the NFS resourcetype is SUNW.nfs:3.1 .

When using a resource type name, the version suffix can be dropped ifthere is no ambiguity. For that matter, the vendor prefix can also bedropped. So, for example, when you initially install Sun Cluster 3.1software, all of the following names can be used to refer to the nfsresource type:

● SUNW.nfs:3.1

● SUNW.nfs

● nfs:3.1

● nfs



Resource Type Version 1.0

If a resource type in the initial release of the Sun Cluster 3.1 software isunchanged from the Sun Cluster 3.0 software release, then the versionnumber is 1.0.

In the case where a resource type is unchanged from Sun Cluster 3.0, thesuffix is not included as part of the type name.

Unchanged resource types include some data service resource types, suchas SUNW.bv, as well as special resource types that are described in ‘‘UsingSpecial Resource Types’’ on page 9-15.

Data Service Resource Types

Table 9-1 lists some of the resource types that are associated withSun Cluster 3.1 software data services.

Table 9-1 Data Service Resource Types

Data Service Resource Type

Sun Cluster HA for NFS SUNW.nfs:3.1

Sun Cluster HA for Oracle SUNW.oracle_server:3.1SUNW.oracle_listener:3.1

Sun Cluster HA for Sun JavaSystem Directory Server

SUNW.nsldap:3.1

Sun Cluster HA for Apache SUNW.apache:3.1

Sun Cluster HA for DNS SUNW.dns:3.1

Sun Cluster HA for Sun JavaSystem Web Server

SUNW.iws:3.1

Sun Cluster HA for SAP SUNW.sap_ciSUNW.sap_as

Sun Cluster HA for BroadVision SUNW.bv

Sun Cluster HA for Sybase SUNW.sybase:3.1

Using Special Resource Types



The following sections describe special resources that you can include inresource groups to complete the capability of those groups.

The SUNW.LogicalHostname Resource Type

Resources of type SUNW.LogicalHostname represent one or more IPaddresses on a particular subnet that will be the logical IP addresses forservices in that resource group.

That is, each IP address described by a SUNW.LogicalHostname resourcemigrates from node to node, along with the services for that group. Theclient uses these IP addresses to access the services in the cluster.

A large part of what makes cluster failover relatively transparent to theclient is that IP addresses migrate along with services of the group. Theclient always uses the same logical IP address to contact a particularinstance of a data service, regardless of which physical node is actuallyrunning the service.

The SUNW.SharedAddress Resource Type

Resources of type SUNW.SharedAddress represent a special type of IPaddress that is required by scalable services. This IP address is configuredon the public net of only one node with failover capability but provides aload-balanced IP address that supports scalable applications that run onmultiple nodes simultaneously. This subject is described in greater detailin Module 10, “Configuring Scalable Services.”

The SUNW.HAStorage Resource Type

The SUNW.HAStorage resource type is the original storage managementtype available in all versions of Sun Cluster 3.0 software.

Global device and file system management, including failover from afailed node, are part of the Sun Cluster software framework. So, it mightseem redundant to provide a resource type that also manages globaldevices and file systems. In fact it is not redundant.



The resource type SUNW.HAStorage serves the following purposes:

● You use the STARTmethod of SUNW.HAStorage to check if the globaldevices or file systems in question are accessible from the nodewhere the resource group is going online.

● You almost always use the Resource_dependencies standardproperty to place a dependency so that the real data services dependon the SUNW.HAStorage resource type. In this way, the resourcegroup manager does not try to start services if the storage theservices depend on is not available.

● You set the AffinityOn resource property to True , so that theSUNW.HAStorage resource type attempts co-location of resourcegroups and disk device groups on the same node, thus enhancingthe performance of disk-intensive data services.

If movement of the device group along with the resource group isimpossible (moving service to a node not physically connected to thestorage), that is still fine. AffinityOn=true moves the device groupfor you whenever it can.

The SUNW.HAStorage resource type provides support only for globaldevices and global file systems. The SUNW.HAStorage resource type doesnot include the ability to unmount file systems on one node and mountthem on another. The STARTmethod of SUNW.HAStorage is only a checkthat a global device or file system is already available.

The SUNW.HAStoragePlus Resource Type

The SUNW.HAStoragePlus resource type supersedes the olderSUNW.HAStorage type because it incorporates all of the older type’scapabilities and adds support for a local, failover file system.

A local failover file system must fail over to a node as the service failsover, but can failover only to nodes physically connected to the storage.

The advantage of a local failover file system is performance. Thedisadvantage is that the local failover file system cannot be used forscalable services or for failover services that need to failover to a node thatis not physically connected to the storage.

The SUNW.HAStoragePlus resource type still supports global devices andfile systems. The capability of SUNW.HAStoragePlus is a proper supersetof SUNW.HAStorage .



The SUNW.HAStoragePlus resource type uses theFilesystemMountpoints property for global and local file systems. TheSUNW.HAStoragePlus resource type understands the difference betweenglobal and local file systems by looking in the vfstab file, where:

● Global file systems have yes in the mount at boot column andglobal (usually global,logging ) in the mount options column. Inthis case, SUNW.HAStoragePlus behaves exactly likeSUNW.HAStorage . The STARTmethod is just a check. The STOPmethod does nothing except return the SUCCESS return code.

● Local file systems have no in the mount at boot column, and do nothave the mount option global (along with logging ). In this case,the STOPmethod actually unmounts the file system on one node, andthe STARTmethod mounts the file system on another node.

Guidelines for Using the SUNW.HAStorage andSUNW.HAStoragePlus Resource Types

Although the SUNW.HAStorage resource type is still available forbackward compatibility, you should use SUNW.HAStoragePlus in a freshSun Cluster 3.1 software installation because it includes all the capabilitiesof SUNW.HAStorage . Eventually, SUNW.HAStorage will be removed fromSun Cluster software.

When to Use a Global File System

Use a global file system if you want to support:

● A scalable service

● A failover service that must failover to a node not physicallyconnected to the storage

● Data for different failover services that are contained in differentresource groups

● Data from a node that is not currently running the service

You can still use AffinityOn=true to migrate the storage along with theservice, if that makes sense, as a performance benefit.



When to Use a Local File System

Use a local failover file system if:

● The file system is for failover service only

● The Nodelist for the resource group contains only nodes that arephysically connected to the storage; that is, if the Nodelist for theresource groups is the same as the Nodelist for the device group

● Only services in a single resource group are using the file system

If these conditions are true, a local file system provides a higher level ofperformance than a global file system, especially for services that are filesystem intensive.

The SUNW.Event Resource Type

The SUNW.Event resource type implementation provides highly availableCluster Reconfiguration Notification Protocol (CRNP) services on SunCluster 3.1 software. This implementation makes the notification daemon(/usr/cluster/lib/sc/cl_apid ) highly available by managing it as aresource under the Sun Cluster resource group manager. The resourcegroup that contains the SUNW.Event resource must have a networkresource configured in the same resource group. Only a single resource oftype SUNW.Event should exist on a cluster.

An application that wants to be notified of cluster reconfiguration eventsmust register with the service. The cl_apid daemon uses XML overTCP/IP to communicate with interested clients. The clients may resideinside or outside of the cluster.



Generic Data Service

The Generic Data Service (GDS) is a resource type for making simplenetwork-aware applications highly available or scalable using theresource type SUNW.gds by plugging them into the Sun Cluster RGMframework. The GDS composes a fully functional Sun Cluster ResourceType complete with callback methods and a Resource Type Registrationfile.

The GDS is a single, precompiled data service. You cannot modify theprecompiled data service and its components, the callback methodimplementations, or the resource type registration file.

Monitoring OPS/RAC

Oracle Parallel Server (OPS) and Oracle Real Application Clusters (RAC)are cluster-aware services. They are managed and maintained by adatabase administrator outside of the Sun Cluster 3.1 softwareenvironment. This makes it difficult to effectively manage the cluster,because the status of OPS/RAC must be determined outside the clusterframework.

The RAC framework resource group enables Oracle OPS/RAC to bemanaged by using Sun Cluster 3.1 commands. This resource groupcontains an instance of the following single-instance resource types:

● The SUNW.rac_framework resource type represents the frameworkthat enables Sun Cluster Support for OPS/RAC. This resource typeenables you to monitor the status of this framework.

● The SUNW.rac_udlm resource type enables the management of theUNIX Distributed Lock Manager (Oracle UDLM) component ofSun Cluster Support for OPS/RAC. The management of thiscomponent involves the following activities:

● Setting the parameters of the Oracle UDLM component

● Monitoring the status of the Oracle UDLM component



In addition, the RAC framework resource group contains an instance of asingle-instance resource type that represents the storage managementscheme that you are using.

● The SUNW.rac_cvm resource type represents the VxVM componentof Sun Cluster Support for OPS/RAC.

You can use the SUNW.rac_cvm resource type to represent thiscomponent only if the cluster feature of VxVM is enabled. Instancesof the SUNW.rac_cvm resource type hold VxVM componentconfiguration parameters. Instances of this type also show the statusof a reconfiguration of the VxVM component.

● The SUNW.rac_hwraid resource type represents the hardware RAIDcomponent of Sun Cluster Support for OPS/RAC.

● The cluster file system is not represented by a resource type.

Usage of the OPS/RAC Monitoring Resource Types

To utilize the Sun Cluster 3.1 RAC resource types, Oracle OPS/RAC mustbe successfully installed in the cluster. The RAC resource types must thenbe installed. The software is located with the Sun Cluster 3.1 softwareframework.

Note – Because of the preparation that is required before installation, thescinstall utility does not support automatic installation of the dataservice packages. You must manually use pkgadd to install thosepackages needed for your installation. Not all packages are needed for allinstallations. The packages are: SUNWscucm, SUNWudlm, SUNWudlmr,SUNWcvmr, and SUNWcvm.

After the software is installed on each node that will be runningOPS/RAC, the resource types must be registered and configured witheither the scsetup command or the scrgadm command. After a resourcegroup is created, configure the rac_framework resource and therac_udlm . Then, configure the appropriate storage resource type. Whencomplete, verify the resource group is online in the cluster.

It is now possible to monitor the status of OPS/RAC using the scstatcommand. Normal operation is indicated by the online status of the SunCluster Support for OPS/RAC resource group. If there are errors in theOPS/RAC configuration or processing, error messages appear in thisoutput.

Understanding Resource Dependencies and Resource Group Dependencies


Understanding Resource Dependencies and ResourceGroup Dependencies

You can declare dependency relationships between resources in the samegroup and between different groups. This implies ordering relationshipsbetween resources and between groups. Dependencies are set throughresource and resource group properties as shown in Figure 9-4.

Figure 9-4 Resource Dependencies

Resource Dependencies

You can set resource dependencies for two resources in the same group. IfResource A depends on Resource B, then:

● Resource B must be added to the group first.

● Resource B must be started first.

● Resource A must be stopped first.

● Resource A must be deleted from the group first.

● The rgmd daemon does not try to start resource A if resource B failsto go online.

Client Workstation

# ping nfs-server# mount nfs-server:/global/nfs/data /export/home

Resource Group: nfs-rg

NAFO Group: nafo2Logical hostname: nfs-server 129.50.20.3

Disk group: nfs-dg

Global Administration and Data Paths

/global/nfs/data/global/nfs/admin

dfstab.nfs-rsshare -F nfs -o rw /global/nfs/data

Data Storage Resource: SUNW.HAStoragePlus

Primary: Node 1 Secondary: Node 2

Node 1

Network

Understanding Resource Dependencies and Resource Group Dependencies


Weak Dependencies

If resource A has a weak dependency on resource B, all of the previousbulleted items apply except for the last one.

Resource Group Dependencies

Resource group dependencies imply an ordering relationship betweentwo groups: If resource group G has a group dependency on resourcegroup H, group G cannot be brought online unless group H is online.Group H cannot be brought offline unless group G is offline.

Note – Resource group dependencies are somewhat limited. They areenforced between groups running on different nodes when you manuallytry to start or stop resource groups in the wrong dependency order,regardless of which nodes the groups are mastered. However, whenfailovers occur, resource group dependencies are only strictly enforcedbetween groups that are stopping and starting on the same node.

Configuring Resource and Resource Groups Through Properties


Configuring Resource and Resource Groups ThroughProperties

You configure specific resources into the cluster by using resourceproperties and resource group properties. Properties consist of a set ofname=value pairs.

These properties are used by the data service agent and by rgmd. There isno way they could be accessed directly by the application software itself,as it is cluster-unaware software.

Some properties are essential for running the service in the cluster. Scriptsspecifically used to run the service in the cluster can read these propertyvalues and use them to locate configuration files, or can use them to passcommand line parameters to the actual services.

Other properties are essential only for service-fault monitoring.Misconfiguring these properties might leave the service running fine, butcause fault monitoring to fail.

Multiple properties can be assigned to different resources, as shown inFigure 9-5 on page 9-24.



Each resource can literally have dozens of properties. Fortunately, manyimportant properties for particular types of resource are automaticallyprovided with reasonable default values. Therefore, most administratorsof Sun Cluster software environments never need to deal with theproperties.

Figure 9-5 Standard and Extension Properties

Standard Resource Properties

The names of standard resource properties can be used with any type ofresource.

You can access a full list of standard resource properties and their generalmeanings by typing:

# man r_properties

Of the dozens of properties listed, only a handful are critical for aparticular resource type. Other properties can be ignored, or can havedefault values that are unlikely to be changed.

Resource Types

SUNW.LogicalHostname

SUNW.SharedAddress

SUNW.HAStoragePlus

SUNW.nfs

SUNW.apache

SUNW.oracle_serverSUNW.oracle_listener

SUNW.dns

SUNW.nsldap

SUNW.iws

Standardresource type

properties

Resourceproperties

Resourcegroup

properties

Extensionresource type

properties

Resource group

Name

Resource

Name

Resource

Name

Resource



Extension Properties

Names of extension properties are specific to resources of a particulartype. You can get information about extension properties from the manpage for a specific resource type. For example:

# man SUNW.apache# man SUNW.HAStoragePlus

If you are setting up the Apache Web Server, for example, you must createa resource to point to the Apache binaries, which, in turn, point to theBin_dir=/usr/apache/bin configuration.

The storage resource has the following important extension properties:

FilesystemMountpoints=/global/nfsAffinityOn=True

Use the first of these extension properties to identify which storageresource is being described. The second extension property is a parameterwhich tells the cluster framework to switch physical control of the storagegroup to the node running the service. Switching control to the nodewhich is running the service optimizes performance when services in asingle failover resource group are the only services accessing the storage

Many resource types (including LDAP and DNS) have an extensionproperty called Confdir_list which points to the configuration.

Confdir_list=/global/dnsconflocation

Many have other ways of identifying their configuration and data. Thereis no hard and fast rule about extension properties.

Resource Group Properties

Certain properties apply to an entire resource group. You can getinformation about these properties by typing:

# man rg_properties

Examining Some Interesting Properties



The following sections examine the usage of some of the more interestingstandard resource properties and resource group properties.

Some Standard Resource Properties

The following properties are standard resource properties, that is, theycan have meaningful values for many different types of resources.

The Resource_dependencies Property

Resource dependencies, which imply ordering, are configured usingstandard properties.

Resource_dependencies=nfs-stor

The Resource_dependencies_weak Property

To set up a weak dependency, so that resource A has a weak dependencyon resource B, set the Resource_dependencies_weak property onresource A. (Recall that this implies an ordering, but not a realdependency):

Resource_dependencies_weak=resB

The Failover_mode Property

The Failover_mode property describes what should happen to a resourceif the resource fails to start up or shut down properly. Table 9-2 describeshow values of the Failover_mode property work.

Table 9-2 The Failover_mode Value Operation

Value of theFailover_modeProperty

Failure to Start Failure to Stop

NONE Other resources in thesame resource groupcan still start (ifnon-dependent).

The STOP_FAILEDflag isset on the resource.



If the STOP_FAILEDflag is set, it must be manually cleared by using thescswitch command before the service can start again.

# scswitch -c -j resource -h node -f STOP_FAILED

Some Resource Group Properties

The following properties are associated with an entire resource group,rather than an individual resource:

The RG_dependencies Property

The RG_dependencies property is set on a whole resource group todescribe its dependency on another group. If resource group A has theproperty RG_dependencies=rgB , then resource group A cannot bebrought online unless resource group B is online. Resource group B mustbe brought online somewhere, but not necessarily in the same node.

The Nodelist Property

The Nodelist property for a resource group describes what node ornodes the resource group can run on. If you have more than two nodes inthe cluster, you can have as many or as few (but probably no less thantwo) potential nodes where a resource group can run.

The property is set up in order, from most preferred node to leastpreferred node.

SOFT The whole resourcegroup is switched toanother node.

The STOP_FAILEDflag isset on the resource.

HARD The whole resourcegroup is switched toanother node.

The node reboots.

Table 9-2 The Failover_mode Value Operation (Continued)

Value of theFailover_modeProperty

Failure to Start Failure to Stop



The Failback Property

If the Failback property is TRUE(not the default), the resource groupautomatically switches back when a preferred node (earlier in theNodelist) joins the cluster.

The Implicit_network_dependencies Property

The Implicit_network_dependencies property is TRUEby default. Thisproperty makes all the services in the resource group dependent on all theLogicalHostName and SharedAddress resources. In other words, noservices can start if logical IP addresses cannot be brought online. Youcould set the Implicit_Network_Dependencies property to FALSEif youhave a service which does not depend on logical IP addresses.

The Pingpong_interval Property

This property, whose default value is 3600 seconds (one hour), determinesthe interval inside which the cluster will refuse to start a resource groupon a particular node if the resource group has previously failed on thatnode or failed to start on that node. The nomenclature derives from thefact that this property prevents rapid “ping-ponging” of a resource groupback and forth between or among the nodes if something ismisconfigured.

The Pathprefix Property

The Pathprefix property points to a directory in a shared-storage filesystem which will be used for the administrative purposes of services inthe resource group.

Currently, the only data service that must have the Pathprefix propertyset is NFS, which uses the property to find its dfstab file. NFS needs thedfstab file so that it knows what it is supposed to be sharing. NFS alsouses the same arena to store file lock state information. File lock stateinformation is typically stored in the /etc directory on a standalone NFSserver, but it must be in the shared storage in the cluster.

An NFS resource must have its dfstab file named:

value_of_Pathprefix_for_RG /SUNW.nfs/dfstab. resource_name

When you create an NFS resource, the resource checks that thePathprefix property is set on its resource group, and that the dfstab fileexists.

Using the scrgadm Command



You use the scrgadm command to set and view configuration of allresources, resource types, and resource groups. You run each commandon any one node that is booted into the cluster.

Note – The scrgadm command does not show any status. Use thescstat -g command to the show status of resources in the cluster.

Show Current Configuration

Use the following command syntax to show the current resources that areconfigured:

scrgadm -p[v[v]] [-t resource_type_name ] [-g resource_group_name ] \[-j resource_name ]

Registering, Changing, or Removing a Resource Type

You use the following command syntax to register (-a ), change (-c ), orremove (-r ) a resource type:

scrgadm -a -t resource_type_name [-h RT_installed_node_list ] \[-f registration_file_path ]scrgadm -c -t resource_type_name -h RT_installed_node_listscrgadm -r -t resource_type_name

Adding, Changing, or Removing Resource Groups

You must add a resource group before putting resources in it. In addition,you can only delete a resource group that has no resources in it. Use thefollowing command syntax to add, change or remove a resource group:

scrgadm -a -g RG_name [-h nodelist ] [-y property […]]scrgadm -c -g RG_name [-h nodelist ] -y property [-y property […]]scrgadm -r -g RG_name

The Nodelist property is set with the -h option.



Adding a LogicalHostname or a SharedAddressResource

Use the following command syntax to add a LogicalHostname (-L )resource or a SharedAddress (-S ) resource:

scrgadm -a -L -g RG_name [-j resource_name ] -l hostnamelist \[-n netiflist ] [-y property […]]scrgadm -a -S -g RG_name -l hostnamelist [-j resource_name ] \[-n netiflist ][-X auxnodelist ] [-y property […]]

Note – The LogicalHostname and SharedAddress resources can havemultiple IP addresses associated with them on the same subnet. Multiplelogical IP resources on different subnets must be separate resources,although they can still be in the same resource group. If you do notspecify a resource name, it will be the same as the first logical host namefollowing the -l argument.

Adding, Changing, or Removing All Resources

Use the following command syntax to add and remove all other resourcetypes, as well as to change properties of all resource types.

scrgadm -a -j resource_name -t resource_type_name -g RG_name \[-y property […]] [-x extension_property […]]scrgadm -c -j resource_name [-y property […]] [-x extension_property […]]scrgadm -r -j resource_name




The following example is based on the assumption that the NFS agentfrom the Sun Cluster 3.1 Data Services CD has been added by using thescinstall or pkgadd command. You should also assume that the dfstabfile has been set up under the /global/nfs/admin directory.

1. Add (register) the resource types by typing:

# scrgadm -a -t SUNW.nfs# scrgadm -a -t SUNW.HAStoragePlus

2. Add the resource group by typing:

# scrgadm -a -g nfs-rg -h proto192,proto193 \-y PathPrefix=/global/nfs/admin

3. Add the logical host name resource by typing:

# scrgadm -a -L -l proto-nfs -g nfs-rg

4. Add the SUNW.HAStoragePlus resource by typing:

# scrgadm -a -j nfs-stor -t SUNW.HAStoragePlus \-g nfs-rg -x FilesystemMountpoints=/global/nfs \-x AffinityOn=true

5. Add the NFS service. Although many services have properties thatpoint to configuration files or binaries, NFS does not because it usesthe Pathprefix property of the resource group. Dependency on thestorage is expressed by using the standard property. Dependency onthe logicalhostname resource is implied.

# scrgadm -a -j nfs-res -t SUNW.nfs -g nfs-rg \-y Resource_dependencies=nfs-stor

Setting Standard and Extension Properties

Sun Cluster 3.1 software allows the developer of a resource type to placerestrictions on when particular properties can be set or changed. Eachproperty for a particular type has a “tunability” characteristic that can beone of the following:

● at_creation – You can only set the property as you do the scrgadm-a command to add the resource

● when_disabled – You can change with the scrgadm -c command ifthe resource is disabled

● anytime – You can change anytime with the scrgadm -c command

Controlling Resources and Resource Groups by Using the scswitch Command


Controlling Resources and Resource Groups by Using thescswitch Command

The following examples demonstrate how to use the scswitch commandto perform advanced operations on resource groups, resources, and dataservice fault monitors.

Resource Group Operations

Use the following commands to:

● Shut down a resource group:

# scswitch -F -g nfs-rg

● Turn on a resource group:

# scswitch -Z -g nfs-rg

● Switch a failover resource group to another node:

# scswitch -z -g nfs-rg -h node

● Restart a resource group:

# scswitch -R -h node -g nfs-rg

● Evacuate all resources and resource groups from a node:

# scswitch -S -h node

Resource Operations


● Disable a resource and its fault monitor:

# scswitch -n -j nfs-res

● Enable a resource and its fault monitor:

# scswitch -e -j nfs-res

● Clear the STOP_FAILEDflag:

# scswitch -c -j nfs-res -h node -f STOP_FAILED

Controlling Resources and Resource Groups by Using the scswitch Command


Fault Monitor Operations


● Disable the fault monitor for a resource:

# scswitch -n -M -j nfs-res

● Enable a resource fault monitor:

# scswitch -e -M -j nfs-res

Note – Do not manually stop any resource group operations that areunderway. Operations, such as those initiated by the scswitch command,must be allowed to complete.

Confirming Resource and Resource Group Status by Using the scstat Command


Confirming Resource and Resource Group Status byUsing the scstat Command

The scstat -g command and option show the state of all resourcegroups and resources. If you use the scstat command without the -g , itshows status of everything, including disk groups, nodes, quorum votes,and transport. For failover resources or resource groups, the outputalways shows offline for a node which the group is not running on, even ifthe resource is a SUNW.HAStorage global storage resource type that isaccessible everywhere.

Using the scstat Command for a Single FailoverApplication

Use the scstat -g command as follows to show the state of resourcegroups and resources for a single failover application:

# scstat -g-- Resource Groups and Resources --

Group Name Resources---------- ---------

Resources: nfs-rg proto-nfs nfs-stor nfs-res

-- Resource Groups --

Group Name Node Name State---------- --------- -----

Group: nfs-rg proto192 OfflineGroup: nfs-rg proto193 Online

Confirming Resource and Resource Group Status by Using the scstat Command


-- Resources --

Resource Name Node Name State Status Message------------- --------- ----- --------------

Resource: proto-nfs proto192 Offline OfflineResource: proto-nfs proto193 Online Online -LogicalHostname online.

Resource: nfs-stor proto192 Offline OfflineResource: nfs-stor proto193 Online Online

Resource: nfs-res proto192 Offline OfflineResource: nfs-res proto193 Online Online -Service is online.

Using the scsetup Utility for Resource and Resource Group Operations


Using the scsetup Utility for Resource and ResourceGroup Operations

The scsetup utility has an extensive set of menus pertaining to resourceand resource group management. These menus are accessed by choosingOption 2 (Resource Group) from the main menu of the scsetup utility.

The scsetup utility is intended to be an intuitive, menu-driven interface,which guides you through the options without having to remember theexact command line syntax. The scsetup utility calls the scrgadm ,scswitch , and scstat commands that have been described in previoussections of this module.

The Resource Group menu for the scsetup utility looks like thefollowing:

*** Resource Group Menu ***


1) Create a resource group 2) Add a network resource to a resource group 3) Add a data service resource to a resource group 4) Resource type registration 5) Online/Offline or Switchover a resource group 6) Enable/Disable a resource 7) Change properties of a resource group 8) Change properties of a resource 9) Remove a resource from a resource group 10) Remove a resource group 11) Clear the stop_failed error flag from a resource

?) Help s) Show current status q) Return to the main menu

Option:

Exercise: Installing and Configuring the Sun Cluster HA for NFS Software Package


Exercise: Installing and Configuring the Sun Cluster HA forNFS Software Package


● Prepare to register and configure the Sun Cluster HA for NFS dataservice

● Use the scrgadm command to register and configure the Sun ClusterHA for NFS data service

● Verify that the Sun Cluster HA for NFS data service is registered andfunctional

● Verify that the Sun Cluster HA for NFS file system is mounted andexported

● Verify that clients can access Sun Cluster HA for NFS file systems

● Switch the Sun Cluster HA for NFS data services from one server toanother

● Observe NFS behavior during failure scenarios

● Configure NFS with a local failover file system

Preparation

The following tasks are explained in this section:

● Preparing to register and configure the Sun Cluster HA for NFS dataservice

● Registering and configuring the Sun Cluster HA for NFS data service

● Verifying access by NFS clients

● Observing Sun Cluster HA for NFS failover behavior

Note – During this exercise, when you see italicized terms, such asIPaddress , enclosure_name , node1 , or clustername , imbedded in acommand string, substitute the names appropriate for your cluster.



Task 1 – Preparing to Register and Configure the SunCluster HA for NFS Data Service

In earlier exercises in Module 6, “Using VERITAS Volume Manager forVolume Management,” or Module 7, “Managing Volumes With Solaris™Volume Manager Software (Solstice DiskSuite™ Software),” you createdthe global file system for NFS. Confirm that this file system is availableand ready to configure for Sun Cluster HA for NFS. Perform the followingsteps:

1. Install the Sun Cluster HA for NFS data service software package onall nodes by running the scinstall command. Select Option 4 fromthe interactive menus.

Note – You must furnish the full path to the data service software. Thefull path name is the location of the .cdtoc file and not the location of thepackages.

2. Log in to Node 1 as user root .

3. Verify that your cluster is active.

# scstat -p

4. Verify that the global/nfs file system is mounted and ready for use.

# df -k

5. Verify that the hosts line in the /etc/nsswitch.conf file iscluster files nis . Depending on your situation, nis might notbe included.

6. Add an entry to the /etc/inet/hosts file on each cluster node andon the administrative workstation for the logical host name resourceclustername -nfs . Substitute the IP address supplied by yourinstructor.

IP_address clustername -nfs

Perform the remaining steps on just one node of the cluster.

7. Create the administrative directory that will contain thedfstab.nfs-res file for the NFS resource.

# cd /global/nfs# mkdir admin# cd admin# mkdir SUNW.nfs



8. Create the dfstab.nfs-res file in the/global/nfs/admin/SUNW.nfs directory. Add the entry to share the/global/nfs/data directory.

# cd SUNW.nfs# vi dfstab.nfs-res

share -F nfs -o rw /global/nfs/data

9. Create the directory specified in the dfstab.nfs-res file.

# cd /global/nfs# mkdir /global/nfs/data# chmod 777 /global/nfs/data# touch /global/nfs/data/sample.file

Note – You are changing the mode of the data directory only for thepurposes of this lab. In practice, you would be more specific about theshare options in the dfstab.nfs-res file.

Task 2 – Registering and Configuring the Sun ClusterHA for NFS Data Service Software Package

Perform the following steps to register the Sun Cluster HA for NFS dataservice software package:

1. From one node, register the NFS and SUNW.HAStorage resourcetypes.

# scrgadm -a -t SUNW.nfs# scrgadm -a -t SUNW.HAStoragePlus# scrgadm -p

Note – The scrgadm command only produces output to the window ifthere are errors executing the commands. If the command executes andreturns, that indicates successful creation.

2. Create the failover resource group. Note that on a three-node clusterthe nodelist (specified with -h ) can include nodes not physicallyconnected to the storage if you are using a global file system.

# scrgadm -a -g nfs-rg -h node1 , node2 \-y Pathprefix=/global/nfs/admin



3. Add the logical host name resource to the resource group.

# scrgadm -a -L -g nfs-rg -l clustername-nfs

4. Create the SUNW.HAStoragePlus resource.

# scrgadm -a -j nfs-stor -g nfs-rg \-t SUNW.HAStoragePlus \-x FilesystemMountpoints=/global/nfs -x AffinityOn=True

5. Create the SUNW.nfs resource.

# scrgadm -a -j nfs-res -g nfs-rg -t SUNW.nfs \-y Resource_dependencies=nfs-stor

6. Enable the resources and the resource monitors, manage the resourcegroup, and switch the resource group into the online state.


7. Verify that the data service is online.

# scstat -g

Task 3 – Verifying Access by NFS Clients

Perform the following steps to verify that NFS clients can access the filesystem of the Sun Cluster HA for NFS software package:

1. On the administration workstation, verify that you can access thecluster file system.

# ls /net/ clustername -nfs/global/nfs/datasample.file

2. On the administration workstation, copy the Scripts/test.nfs fileinto the root directory.

3. Edit the /test.nfs script, and verify that the logical host name andNFS file system names are correct.

When this script is running, it creates and writes a file containing atimestamp to your NFS-mounted file system. The script also shows thefile to standard output (stdout ). This script times how long the NFS dataservice is interrupted during switchovers and takeovers.



Task 4 – Observing Sun Cluster HA for NFS FailoverBehavior

Now that the Sun Cluster HA for NFS environment is working properly,test its high-availability operation by performing the following steps:

1. On the administration workstation, start the test.nfs script.

2. On one node of the cluster, determine the name of the node currentlyhosting the data services of the Sun Cluster HA for NFS softwarepackage.

3. On one node of the cluster, use the scswitch command to transfercontrol of the NFS service from one node to another.

# scswitch -z -h dest-node -g nfs-rg

Substitute the name of your offline node for dest-node .

4. Observe the messages displayed by the test.nfs script.

5. How long was the data service interrupted during the switchoverfrom one physical host to another?

__________________________________________________

6. Use the mount and share commands on all nodes to verify whichfile systems the nodes are now mounting and exporting.

__________________________________________________

__________________________________________________

__________________________________________________

7. Use the ifconfig command on all nodes to observe the additionalIP address associated with the Logical Hostname resourceconfigured as a virtual interface on one of the adapters in your IPMPgroup.

# ifconfig -a

8. On one node of the cluster, use the scswitch command to transfercontrol of the NFS service back to its preferred host.

# scswitch -z -h dest-node -g nfs-rg



Task 5 – Generating Cluster Failures and ObservingBehavior of the NFS Failover

Generate failures in your cluster to observe the recovery features of SunCluster 3.1 software. If you have physical access to the cluster, you canphysically pull out network cables or power down a node. If you do nothave physical access to the cluster, you can still bring a node to the OKprompt using a break signal through your terminal concentrator.

Try to generate the following failures:

● Single public network interface failure (observe IPMP failover)

● Total public network failure on a single node (pull both publicnetwork interfaces)

● Single transport failure

● Dual transport failure

● Node failure (power down a node or bring it to the ok prompt)

Task 6 – Configuring NFS to Use a Local Failover FileSystem

In this exercise, you configure NFS to use a local failover file system ratherthan a global file system.

1. Disable the NFS resource group.

# scswitch -F -g nfs-rg

2. If you have more than two nodes, make sure that the NFS resourcegroup is enabled to run only on the nodes physically connected tothe storage.

# scrgadm -c -g nfs-rg -y Nodelist= node1,node2

3. Unmount the global NFS file system.

# umount /global/nfs

4. On each node, edit the /etc/vfstab file to make /global/nfs a localfile system.

a. Change yes to no in the mount-at-boot column.

b. Remove the word global from the mount options (keep theword logging , or add logging if you forgot before).



5. Restart the NFS resource group.


6. Observe the file system behavior. The file system should be mountedonly on the node running the NFS service, and should fail overappropriately.

Task 7 – Viewing and Managing Resources andResource Groups Through SunPlex Manager

Perform the following steps on your administration workstation:

1. In a web browser, log in to SunPlex Manager on any cluster node:


2. Click on the Resource Groups folder on the left.

3. Investigate the status information and graphical topologyinformation that you can see regarding resource groups.


5. Use the Action Menu to switch the primary for one of your resourcegroups.

Exercise Summary


Exercise Summary

?!


● Experiences

● Interpretations

● Conclusions

● Applications


Module 10

Configuring Scalable Services

Objectives


● Describe the characteristics of scalable services

● Describe the function of the load balancer

● Create the failover resource group for the SharedAddress resource

● Create the scalable resource group for the scalable service

● Use important resource group and resource properties for scalableservices

● Describe how the SharedAddress resource works with scalableservices

● Use the scrgadm command to create these resource groups

● Use the scswitch command to control scalable resources andresource groups

● Use the scstat command to view scalable resource and group status

● Configure and run Apache as a scalable service in the cluster

Relevance


Relevance

?!


● What type of services can be made scalable?

● What type of storage must be used for scalable services?

● How does the load balancing work?

Using Scalable Services and Shared Addresses


Using Scalable Services and Shared Addresses

The scalable service architecture allows some services, such as ApacheWeb Server, to run simultaneously on multiple nodes as if it were a singleserver.

Clients connect to such a service using a single IP address called a sharedaddress. Clients are not normally aware which node they connect to, nordo they care.

Figure 10-1 shows the architecture of scalable services and sharedaddresses.

Figure 10-1 Scalable Services, Shared Addresses Architecture

<HTML>

</HTML>

<HTML>

</HTML>

<HTML>

</HTML>

Node 1

Node 2 Node 3

Globally Available HTML Documents

HTTP Application HTTP

ApplicationHTTP

Application

Global Interfacexyz.com

Request Distribution

Transport

WebPage

Client 1 Client 2 Client 3

Requests RequestsNetwork

Exploring Characteristics of Scalable Services


Exploring Characteristics of Scalable Services

Like the failover services described in the previous modules, theapplications used as scalable services in Sun Cluster 3.1 are generallyoff-the-shelf applications that are not specifically compiled or released torun in the cluster. The application binaries running on the various nodesare unaware of each other; all the “magic” that makes this work is in theSun Cluster 3.1 SharedAddress mechanism.

File and Data Access

Like the failover services, all the data for the scalable service must be inthe shared storage.

Unlike the failover service, a file system-oriented scalable service must usethe global file system. A SUNW.HAStoragePlus resource is still set up tomanage dependencies between the service and the storage.

File Locking for Writing Data

One of the big barriers to taking “just any application” and turning it intoa scalable service is that a service that is cluster-unaware might have beenwritten in such a way as to ignore file-locking issues.

In Sun Cluster 3.1 software, an application that does data modificationwithout any type of locking or file synchronization mechanism generallycannot be used as a scalable service.

While Sun Cluster 3.1 software provides the global data access methods, itdoes not automatically call any file-locking primitives for you.

Web servers that do file writing in common gateway interface (cgi) scriptsthat, although not written specifically for a scalable platform, are usuallywritten in such a way that multiple instances can be launchedsimultaneously even on a standalone server. Thus, they already have thelocking in place to make them ideal to work as scalable services in SunCluster 3.1 software.

Web servers that do file writing using Java servlets must be examinedmuch more closely. A servlet might use thread synchronization ratherthan file locking to enforce serial write access to a critical resource. Thiswill not translate properly into a scalable service.

Using the SharedAddress Resource as IP and Load-Balancer


Using the SharedAddress Resource as IP andLoad-Balancer

The “glue” that holds a scalable service together in Sun Cluster 3.1software is the SharedAddress resource.

This resource provides not only a single IP address that makes thescalable service look like a “single server” from the point of view of theclient, but also provides the load balancing of requests to all the nodes onwhich a Scalable Service is active.

Client Affinity

Certain types of applications require that load balancing in a scalableservice be on a per-client basis, rather than a per-connection basis. That is,they require that the same client IP always have its requests forwarded tothe same node. The prototypical example of an application requiring suchClient Affinity is a shopping cart application, where the state of the client’sshopping cart is recorded only in the memory of the particular nodewhere the cart was created.

A single SharedAddress resource can provide standard (per connectionload balancing) and the type of sticky load balancing required byshopping carts. A scalable service’s agent “registers” which type of loadbalancing is required, based on a property of the data service.

Load-Balancing Weights

Sun Cluster 3.1 software also lets you control, on a scalable resource byscalable resource basis, the weighting that should be applied for loadbalancing. The default weighting is equal (connections or clients,depending on the stickiness) per node, but through the use of propertiesdescribed in the following sections, a “better” node can be made to handlea greater percentage of requests.

Exploring Resource Groups for Scalable Services



A scalable service requires the creation of two resource groups. A failoverresource group holds the SharedAddress resource; it is online, ormastered, by only one node at a time. The node that masters theSharedAddress resource is the only one which receives incoming packetsfor this address from the public network. A scalable resource group holdsan HAStoragePlus resource and the actual service.

Remember, the HAStoragePlus resource is there to guarantee that thestorage is accessible on each node before the service is started on that node.Figure 10-2 is a block diagram showing the relationship between scalableand failover resource groups.

Figure 10-2 Scalable and Failover Resource Groups

Scalable Resource Group Failover Resource Group

SUNW.HAStoragePlusResource

SUNW.SharedAddressResource

SUNW.apache Resource

Resource_dependencies

Network_resources_Used

RG_dependencies



Resources and Their Properties in the ResourceGroups

Table 10-1 and Table 10-2 demonstrate the properties and contents of thetwo resource groups required to implement a scalable service.

● Resource Group Name: sa-rg

● Properties: [Nodelist=proto192,proto193 Mode=FailoverFailback=False...]

● Resource Group Contents: See Table 10-1

● Resource Group Name: apache-rg

● Properties: [Nodelist=proto192,proto193 Mode=ScalableDesired_primaries=2 Maximum_primaries=2]

RG_dependencies=sa-rg

● Resource Group Contents: See Table 10-2

Table 10-1 sa-rg Resource Group Contents

Resource Name Resource Type Properties

apache-lh SUNW.SharedAddress HostnameList=apache-lhNetiflist=therapy@proto192,therapy@proto193

Table 10-2 apache-rg Resource Group Contents

Resource Name Resource Type Properties

web-stor SUNW.HAStoragePlus FilesystemMountpoints=/global/web

AffinityOn = [ignored]

apache-res SUNW.apache Bin_dir=/usr/apache/binLoad_balancing_policy=LB_WEIGHTEDLoad_balancing_weights=3@1,1@2Scalable=TRUEport_list=80/tcpNetwork_resources_used=apache-lhResource_dependencies=web-stor

Understanding Properties for Scalable Groups and Services


Understanding Properties for Scalable Groups andServices

There are certain resource group properties and certain resourceproperties of particular importance for scalable services.

The Desired_primaries and Maximum_primariesProperties

These are group properties that indicate how many nodes the serviceshould run on. The rgmd tries to run on the service onDesired_primaries nodes, but you can manually switch it on usingscswitch , up to Maximum_primaries .

Note that if these values are greater than 1, the Mode=Scalable propertyis automatically set.

The Load_balancing_policy Property

This is a property of the data service resource. It has one of the followingvalues:

● Lb_weighted – Client connections are all load-balanced. This is thedefault. Repeated connections from the same client might beserviced by different nodes.

● Lb_sticky – Connections from the same client IP to the same serverport all go to the same node. Load balancing is only for differentclients. This is only for all ports listed in the Port_list property.

● Lb_sticky_wild – Connections from the same client to any serverport go to the same node. This is good when port numbers aregenerated dynamically and not known in advance.

The Load_balancing_weights Property

This property controls the weighting of the load balancer. Default is evenfor all nodes. A value such as 3@1,1@2indicates three times as manyconnections (or clients, with one of the “sticky” policies) serviced byNode 1 in the cluster compared to the number serviced by Node 2.

Understanding Properties for Scalable Groups and Services


The Network_resources_used Property

It is required to set this property on a scalable resource; its value must pointto the SharedAddress resource (in the other resource group). This is usedby the data service agent to “register” the data service with the loadbalancer associated with the SharedAddress resource.

Adding Auxiliary Nodes for a SharedAddress Property


Adding Auxiliary Nodes for a SharedAddress Property

The SharedAddress logic includes a routine whereby the actual IPaddress associated with the resource is configured on the public network(IPMP groups) on the primary node and is configured as a virtual addresson the loopback network on all other nodes in its Nodelist.

This enables scalable services that are running on nodes other than theprimary SharedAddress node to still bind to the IP address used for theSharedAddress .

However, the SharedAddress resource must know about all possiblenodes on which any scalable service associated with it might possibly run.

If the Nodelist for the SharedAddress resource group is a superset of theNodelist of every scalable service that might run on it, you are fine. Butyou might want to restrict which nodes might actually be the primary forthe SharedAddress , while still allowing a larger Nodelist for the scalableservices dependent on it.

The SharedAddress resource has a special “auxiliary nodes” propertythat allows you to augment the Nodelist of its group, just for the purposesof supporting more nodes for scalable services. This is set with the -Xoption to scrgadm .

In the following example, you want only nodes proto192 and proto193to be the primaries for the SharedAddress (to actually host it on thepublic net and do the load balancing). But you might be supportingscalable services that run on proto192 , proto193 , and proto194 :

# scrgadm -a -g sa-rg -h proto192,proto193# scrgadm -a -S -l apache-lh -g sa-rg -X proto194

Reviewing scrgadm Command Examples for Scalable Service


Reviewing scrgadm Command Examples for ScalableService

The following examples assume that the Apache Web Server agent wasadded from the Sun Cluster 3.1 Data Services CD using the scinstall orpkgadd commands.

1. Add (register) the resource types by typing:

# scrgadm -a -t SUNW.apache# scrgadm -a -t SUNW.HAStoragePlus

2. Add the failover resource group for the SharedAddress by typing:

# scrgadm -a -g sa-rg -h proto192,proto193

3. Add the SharedAddress resource by typing:

# scrgadm -a -S -l apache-lh -g sa-rg

4. Add the scalable resource group by typing:

# scrgadm -a -g apache-rg \-y RG_dependencies=sa-rg \-y Desired_primaries=2 -y Maximum_primaries=2

5. Add the HAStorage resource by typing:

# scrgadm -a -j web-stor -t SUNW.HAStoragePlus \-g apache-rg -x FilesystemMountpoints=/global/web

6. Add the Apache Service by typing:

# scrgadm -a -j apache-res -t SUNW.apache \-g apache-rg \

-y Resource_dependencies=web-stor \-x Bin_dir=/usr/apache/bin \-x Port_list=80/tcp \-y Network_resources_used=apache-lh \-y Scalable=TRUE

Controlling Scalable Resources and Resource Groups With the scswitchCommand


Controlling Scalable Resources and Resource GroupsWith the scswitch Command

The following examples demonstrate how to use the scswitch commandto perform advanced operations on resource groups, resources, and dataservice fault monitors.

Resource Group Operations


● Shut down a resource group:

# scswitch -F -g apache-rg

● Turn on a resource group:

# scswitch -Z -g apache-rg

● Switch the scalable resource group to these specific nodes (on othernodes it is turned off):

# scswitch -z -g apache-rg -h node,node,node

● Restart a resource group:

# scswitch -R -h node , node -g apache-rg

● Evacuate all resources and resource groups from a node:

# scswitch -S -h node

Resource Operations


● Disable a resource and its fault monitor:

# scswitch -n -j apache-res

● Enable a resource and its fault monitor:

# scswitch -e -j apache-res

● Clear the STOP_FAILEDflag:

# scswitch -c -j apache-res -h node -f STOP_FAILED

Controlling Scalable Resources and Resource Groups With the scswitch Command


Fault Monitor Operations


● Disable the fault monitor for a resource:

# scswitch -n -M -j apache-res

● Enable a resource fault monitor:

# scswitch -e -M -j apache-res

Note – You should not manually stop any resource group operations thatare underway. Operations, such as scswitch , must be allowed tocomplete.

Using the scstat Command for a Scalable Application


Using the scstat Command for a Scalable Application

Use the scstat -g command and option as follows to show the state ofresource groups and resources for a scalable application:

# scstat -g-- Resource Groups and Resources --

Group Name Resources---------- ---------

Resources: sa-rg apache-lhResources: apache-rg apache-res web-stor

-- Resource Groups --

Group Name Node Name State---------- --------- -----

Group: sa-rg proto192 OfflineGroup: sa-rg proto193 Online

Group: apache-rg proto192 OnlineGroup: apache-rg proto193 Online

-- Resources --

Resource Name Node Name State Status Message------------- --------- ----- --------------

Resource: apache-lh proto192 Offline OfflineResource: apache-lh proto193 Online Online -SharedAddress online.

Resource: web-stor proto192 Online OnlineResource: web-stor proto193 Online Online

Resource: apache-res proto192 Online OnlineResource: apache-res proto193 Online Online -Service is online.

Describing the RGOffload Resource: Clearing the Way for Critical Resources


Describing the RGOffload Resource: Clearing the Way forCritical Resources

The RGOffload resource is an alternative (not required) method formanaging the relationship between resource groups.

The general idea is that you want to have a “critical” failover resourcegroup. When that resource group goes online to any particular node, orfails over to any particular node, there are other “less critical” groups thatcan be migrated off of that node. While the “less critical” group could bea failover group or a scalable group, it is often a scalable group (where itis just one instance that must migrate).

To use SUNW.RGOffload , do the following:

1. Add an RGOffload resource to your “critical” resource group.

2. The rg_to_offload property (only required property) would pointto the “less critical” scalable groups that you want to “evacuate” to“make way” for your critical group

3. The Desired_primaries of each of these “less critical” groups mustbe set to 0. This sounds strange, as if the RGM will never directly callSTARTon your scalable group anywhere. What happens is that theSTARTand STOPmethods of the RGOffload resource actuallymanage all the STARTand STOPmethods for the “less critical”resource, as shown in Figure 10-3.

Figure 10-3 Using the RGOffload Resource

LogicalHostname

HAStoragePlus

Critical Data Service

RGOffloadrg_to_offload

HAStoragePlus

Scalable Data Service

"Less Critical" Scalable Group

Desired_primaries=0Maximum_primaries=5

"Critical" Resource Group

Describing the RGOffload Resource: Clearing the Way for Critical Resources


Example of Using the RGOffload Resource

The following example adds the RGOffload resource to a critical NFSresource group. The “less critical” group is the apache-rg of the previousexamples.

# scrgadm -a -g nfs-rg -j rg-res -t SUNW.RGOffload \-x rg_to_offload=apache-rg

# scrgadm -c -g apache-rg -y Desired_primaries=0

Exercise: Installing and Configuring Sun Cluster Scalable Service for Apache


Exercise: Installing and Configuring Sun Cluster ScalableService for Apache


● Prepare for Sun Cluster HA for Apache data service registration andconfiguration

● Use the scrgadm command to register and configure the Sun ClusterHA for Apache data service

● Verify that the Sun Cluster HA for Apache data service is registeredand functional

● Verify that clients can access the Apache Web Server

● Verify the capability of the scalable service

● Observe network and node failures

● Optionally, use the RGOffload resource type

Preparation

The following tasks are explained in this section:

● Preparing for Sun Cluster HA for Apache registration andconfiguration

● Registering and configuring the Sun Cluster HA for Apache dataservice

● Verifying Apache Web Server access and scalable capability




Task 1 – Preparing for HA-Apache Data ServiceConfiguration

Perform the following steps on each node of the cluster:

1. Install the Sun Cluster Apache data service software package byrunning scinstall on each node. Use Option 4.

# scinstall

2. Create the following:

a. Create an entry in /etc/inet/hosts for the shared addressyou will be configuring with the Apache Web server.

b. Create the entry on the administration workstation. Substitutethe IP address supplied by your instructor.

IP_address clustername -web

3. Edit the /usr/apache/bin/apachectl script.

a. Locate the line:

HTTPD=/usr/apache/bin/httpd

b. Change it to:

HTTPD=”/usr/apache/bin/httpd -f /global/web/conf/httpd.conf”

On (any) one node of the cluster, perform the following steps:

4. Copy the sample /etc/apache/httpd.conf-example to/global/web/conf/httpd.conf .

# mkdir /global/web/conf# cp /etc/apache/httpd.conf-example /global/web/conf/httpd.conf

5. Edit the /global/web/conf/httpd.conf file, and change thefollowing entries as shown.

From:

ServerName 127.0.0.1KeepAlive On#BindAddress *DocumentRoot "/var/apache/htdocs"<Directory "/var/apache/htdocs">ScriptAlias /cgi-bin/ "/var/apache/cgi-bin/"<Directory "/var/apache/cgi-bin">

to:

ServerName clustername -webKeepAlive Off



BindAddress clustername -web (Uncomment the line)DocumentRoot " /global/web/htdocs "<Directory " /global/web/htdocs ">ScriptAlias /cgi-bin/ " /global/web/cgi-bin/ "<Directory " /global/web/cgi-bin ">

6. Create directories for the Hypertext Markup Language (HTML)and CGI files.

# mkdir /global/web/htdocs# mkdir /global/web/cgi-bin

7. Copy the sample HTML documents to the htdocs directory.

# cp -rp /var/apache/htdocs /global/web# cp -rp /var/apache/cgi-bin /global/web

8. Copy the file called “test-apache.cgi ” from the classroom serverto /global/web/cgi-bin . You use this file to test the scalableservice. Make sure that test-apache.cgi is executable by all users.

# chmod 755 /global/web/cgi-bin/test-apache.cgi

Test the server on each node before configuring the data service resources.

9. Start the server.

# /usr/apache/bin/apachectl start

10. Verify that the server is running.

# ps -ef | grep httpdnobody 490 488 0 15:36:27 ? 0:00 /usr/apache/bin/httpd -f/global/web/conf/httpd.conf root 488 1 0 15:36:26 ? 0:00 /usr/apache/bin/httpd -f/global/web/conf/httpd.conf nobody 489 488 0 15:36:27 ? 0:00 /usr/apache/bin/httpd -f/global/web/conf/httpd.conf nobody 491 488 0 15:36:27 ? 0:00 /usr/apache/bin/httpd -f/global/web/conf/httpd.conff nobody 492 488 0 15:36:27 ? 0:00 /usr/apache/bin/httpd -f/global/web/conf/httpd.conf nobody 493 488 0 15:36:27 ? 0:00 /usr/apache/bin/httpd -f/global/web/conf/httpd.conf



11. Connect to the server from the Web browser on your console server.Use http:// nodename where nodename is the name of one of yourcluster nodes (Figure 10-4).

Figure 10-4 Apache Server Test Page

12. Stop the Apache Web server.

# /usr/apache/bin/apachectl stop

13. Verify that the server has stopped.

# ps -ef | grep httpdroot 8394 8393 0 17:11:14 pts/6 0:00 grep httpd

Task 2 – Registering and Configuring the Sun ClusterHA for Apache Data Service

Perform the following steps only on (any) one node of the cluster.

1. Register the resource type required for the Apache data service.

# scrgadm -a -t SUNW.apache

2. Create a failover resource group for the shared address resource. Usethe appropriate node names for the -h argument. If you have morethan two nodes, you can include more than two nodes after the -h .

# scrgadm -a -g sa-rg -h node1,node2



3. Add the SharedAddress logical host name resource to the resourcegroup.

# scrgadm -a -S -g sa-rg -l clustername -web

4. Create a scalable resource group to run on all nodes of the cluster.(The example assumes two nodes.)

# scrgadm -a -g web-rg -y Maximum_primaries=2 \-y Desired_primaries=2 -y RG_dependencies=sa-rg

5. Create a storage resource in the scalable group.

# scrgadm -a -g web-rg -j web-stor -t SUNW.HAStoragePlus \-x FilesystemMountpoints=/global/web -x AffinityOn=false

6. Create an application resource in the scalable resource group.

# scrgadm -a -j apache-res -g web-rg \-t SUNW.apache -x Bin_dir=/usr/apache/bin \-y Scalable=TRUE -y Network_resources_used= clustername -web \-y Resource_dependencies=web-stor

7. Bring the failover resource group online.

# scswitch -Z -g sa-rg

8. Bring the scalable resource group online.

# scswitch -Z -g web-rg

9. Verify that the data service is online.

# scstat -g

Task 3 – Verifying Apache Web Server Access

Perform the following steps to verify Apache Web Server access andscalable utility:

1. Connect to the web server using the browser on the administratorworkstation using http:// clustername -web/cgi-bin/test-apache.cgi.

2. Repeatedly press the Refresh or Reload button on the browser. Thetest-apache.cgi script shows the name of the cluster node that iscurrently servicing the request. It might take several iterations beforethe packet is distributed to a new node.



Task 4 – Observing Cluster Failures

Cause as many of the following failures (one at a time, and fix that onebefore going on to the next one) as you can, given your physical access tothe cluster.

Observe the behavior of the scalable service:

1. Fail a single public network interface on the node where theSharedAddress resource is online

2. Fail all public network interfaces on that node

3. Reboot one of the nodes running the scalable service

4. Fail (or bring to the ok prompt) one of the nodes running the scalableservice

Task 5 (Optional) – Configuring the RGOffloadResource in Your NFS Resource Group to “Push YourApache Out of the Way”

In this task, you add an RGOffload resource to your NFS resource groupfrom the previous module. You observe how your scalable Apacheresource group “moves out of the way” for your NFS group. Perform thefollowing steps:

1. Change Desired_primaries for your scalable group:

# scrgadm -c -g apache-rg -y Desired_primaries=0

2. Add and enable the RGOffload resource:

# scrgadm -a -g nfs-rg -j rg-res -t SUNW.RGOffload-x rg_to_offload=apache-rg

# scswitch -e -j rg-res

3. Observe the behavior of the Apache scalable resource group as yourNFS resource group switches from node to node.

Exercise Summary


Exercise Summary

?!


● Experiences

● Interpretations

● Conclusions

● Applications


Module 11

Performing Supplemental Exercises for SunCluster 3.1 Software

Objectives


● Configure HA-LDAP in a Sun Cluster 3.1 software environment

● Configure HA-ORACLE in a Sun Cluster 3.1 software environment

● Configure ORACLE Real Application Cluster (RAC) in a SunCluster 3.1 software environment

● Reinstall Sun Cluster 3.1 software

Relevance


Relevance

?!

Discussion – The following questions are relevant to understanding thecontent this module:

● Why would it be more difficult to install ORACLE software on thelocal disks of each node rather than installing it in the sharedstorage?

● Would there be any advantage to installing software on the localdisks of each node?

● How is managing ORACLE RAC different that managing the othertypes of failover and scalable services presented in this course?


Performing Supplemental Exercises for Sun Cluster 3.1 Software 11-3Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1







● Sun Microsystems, Inc. Solaris Volume Manager Administration Guide,part number 816-4519.

● VERITAS Volume Manager™ 3.5 Administrator's Guide (Solaris)

● VERITAS Volume Manager™ 3.5 Installation Guide (Solaris)

● VERITAS Volume Manager™ 3.5 Release Notes (Solaris)

● VERITAS Volume Manager™ 3.5 Hardware Notes (Solaris)

● VERITAS Volume Manager™ 3.5 Troubleshooting Guide (Solaris)

● Sun Microsystems, Inc. IP Network Multipathing Administration Guide,part number 816-5249.

Exploring the Exercises



This module consists of exercises for you to perform on the last day ofclass. You should select one or more of these exercises to do, but bear inmind that it is not possible to complete all of them. If you do want toperform more than one exercise, sort them in order of importance so thatyou at least have the opportunity to perform the ones most important toyou. The time estimates given do not include the installation of theSolaris OS or the Sun Cluster 3.1 framework.

The exercises include the following:

● Configuring HA-LDAP

● Configuring HA-ORACLE

● Configuring ORACLE RAC

● Reinstalling Sun Cluster 3.1 software

Configuring HA-LDAP

Sun ONE Directory Server 5.1 software is included in the standardSolaris 9 OS installation. The software is also obtained as unbundledpackages. The manner in which these two sources of the software getsinstalled is different. Specifically, the bundled version does not allow theadministrator to specify the location for the LDAP data, whereas theunbundled version does.

The Sun ONE Directory Server 5.1 software can be used for applicationsthat use LDAP as a directory service, as a name service, or both. This labexercise installs the Sun ONE Directory Server 5.1 software as a directoryservice using the software bundled with Solaris 9 OS and integrates it intothe cluster.

The estimated total time for this lab is one hour.



Configuring HA-ORACLE

ORACLE software may run as either a failover or parallel service in SunCluster 3.1 software. This exercise sets up the ORACLE software to run asa failover (HA) data service in the cluster. The estimated total time for thisexercise is two hours.

Much of the time required to perform this lab is the ORACLE installationprogram which takes up to 60 minutes. Therefore, begin the installationprogram just before going to lunch. After the installation is complete, afew configuration assistants are run which takes about 15 minutes total tofinish. The remaining 45 minutes estimated for this lab are used up in allthe preparation, verification, or otherwise supporting steps.

Configuring ORACLE RAC

ORACLE RAC software runs as a parallel service in the cluster. It wasformerly known as ORACLE Parallel Server (OPS). The estimated totaltime for this exercise is two hours.

Reinstalling Sun Cluster 3.1 Software

Over the course of this week, you have installed the Sun Cluster 3.1software on your administrative workstation and cluster nodes. There areno instructions for this exercise in this module because the steps to installand configure the cluster software are included in previous exercises.Consider using the volume management software that you did not useover the course of this week when performing this exercise.

There are a variety of approaches you could take in preparing for this lab.One approach is to start from scratch by performing a JumpStart softwareinstallation on all the cluster nodes and the admin workstation. A secondapproach is to manually uninstall the cluster and volume managementsoftware. A third approach is to uninstall only the cluster software(leaving the volume management software installed).

Depending on which approach you select, the lab takes anywhere fromone to three hours to complete.

Exercise 1: Integrating Sun ONE Directory Server 5 Software Into Sun Cluster 3.1


Exercise 1: Integrating Sun ONE Directory Server 5Software Into Sun Cluster 3.1


● Create user and group accounts for Sun ONE Directory Server 5software

● Register SUNW.HAStoragePlus resource type, if necessary

● Install and register SUNW.nsldap resource type

● Create a global file system for the directory data

● Configure mount information for the file system

● Create an entry in /etc/inet/hosts for the logical host name

● Run the directoryserver setup script on Node 1

● Verify the directory service is running properly

● Stop the directory service on Node 1

● Run the directoryserver setup script on Node 2

● Stop the directory service on Node 2

● Replace the /var/ds5 directory with a symbolic link

● Verify the directory service runs properly on Node 2

● Stop the directory service on Node 2 again

● Prepare the resource group and resources for the Sun ONE DirectoryServer software

● Verify the Sun ONE Directory Server software runs properly in thecluster

Preparation

This lab assumes that your cluster nodes belong to a name service domaincalled suned.sun.com . If your cluster nodes already belong to a domain,replace suned.sun.com with your domain’s name. If not, perform thefollowing on all nodes to set up the suned.sun.com domain name.

# echo suned.sun.com > /etc/defaultdomain# domainname suned.sun.com



Task 1 – Creating nsuser and nsgroup Accounts

To create nsuser and nsgroup accounts, perform the following steps onall cluster nodes:

1. Create a group account for the nsuser to belong to by typing:

# groupadd -g 9999 nsgroup

2. Create a user account for the directory server daemon’s identity bytyping:

# useradd -g nsgroup -u 9999 nsuser

Task 2 – Registering the SUNW.HAStoragePlusResource Type

If necessary (if it was not done in a previous exercise), register theSUNW.HAStoragePlus type from any one node in the cluster:

# scrgadm -a -t SUNW.HAStoragePlus

Task 3 – Installing and Registering the SUNW.nsldapResource Type

To install and register SUNW.nsldap :

1. On all cluster nodes, install the SUNWscnsl resource type by typing:

# cd sc-dataservice-software-location# pkgadd -d . SUNWscnsl

2. From one node only, register the SUNW.nsldap by typing:

# scrgadm -a -t SUNW.nsldap

Task 4 – Creating a Global File System for Server Root

To accomplish this task, perform the following steps from one nodeattached to storage:

1. Select two disks (cAtAdA and cBtBdB ) from shared storage for a diskgroup by typing:

# vxdisk list



2. Initialize these disks as VM disks by typing:

# /etc/vx/bin/vxdisksetup -i cAtAdA# /etc/vx/bin/vxdisksetup -i cBtBdB

3. Create a disk group by typing:

# vxdg init ids-dg ids1=cAtAdA ids2=cBtBdB

4. Create a 1-Gbyte volume by typing:

# vxassist -g ids-dg make idsvol 1024m layout=mirror

5. Register the disk group (and synchronize volume info) to the clusterby typing:

# scconf -a -D name=ids-dg,type=vxvm,nodelist= X:Y

Note – The Nodes X and Y in the preceding command are attached to thestorage arrays containing the ids-dg disk group. If there are more nodesattached to these storage arrays, then include them in the command.

6. Create a UFS file system on the volume by typing:

# newfs /dev/vx/rdsk/ids-dg/idsvol

Task 5 – Configuring Mount Information for the FileSystem

Perform the following steps as indicated:

1. On all cluster nodes, create the mount point for the local file systemby typing:

# mkdir /ldap

2. On all cluster nodes, create an entry in /etc/vfstab for a localfailover file system by typing:

/dev/vx/dsk/ids-dg/idsvol /dev/vx/rdsk/ids-dg/idsvol /ldap ufs 1 no logging

3. From the cluster node attached to storage, mount the local filesystem by typing:

# mount /ldap



Task 6 – Creating an Entry in the Name Service for theLogical Host

To create an entry in the name service for the logical host, enter thefollowing on all cluster nodes and the administrative workstation:

Note – Your instructor might have you use different hostnames and IPaddresses. This exercise consistently refers to the host name and IPaddress as listed in the following.

# vi /etc/inet/hosts10.10.1.1 ids-lh ids-lh.suned.sun.com

Task 7 – Running directoryserver Setup Script onthe First Node

Perform the following steps on the first node:

1. Plumb and configure the virtual interface by typing:

# ifconfig hme0 addif ids-lh netmask + broadcast + up

2. Create a symbolic link for the /var/ds5 directory by typing:

# ln -s /ldap /var/ds5

3. Run the directoryserver setup script by typing:

# /usr/sbin/directoryserver setupWould you like to continue with installation? [Yes]:Select the component you want to configure [1]:Choose a configuration type [2]: 3Specify the components you wish to configure [All]:Specify the components you wish to install [1, 2]:Specify the components you wish to install [1, 2]:Computer name [mynode]: ids-lh.suned.sun.comSystem User [nobody]: nsuserSystem Group [nobody]: nsgroupDo you want to register this software with an existing iPlanetconfiguration directory server? [No]:Do you want to use another directory to store your data? [No]:Directory server network port [389]:Directory server identifier [mynode]: ids-lhiPlanet configuration directory server administrator ID [admin]:Password: adminPassword (again): admin



Suffix [dc=suned, dc=sun, dc=com]:Directory Manager DN [cn=Directory Manager]:Password: dirmanagerPassword (again): dirmanagerAdministration Domain [suned.sun.com]:Do you want to install the sample entries? [No]: yesType the full path and filename, the word suggest, or the word none[suggest]:Do you want to disable schema checking? [No]:Administration port [18733]: 10389IP address [10.10.1.204]: 10.10.1.1Run Administration Server as [root]:

Task 8 – Verifying That Directory Server Is RunningProperly


1. Query the directory service by typing:

# /usr/bin/ldapsearch -h ids-lh -p 389 -b "dc=suned,dc=sun,dc=com" \-s base "objectClass=*"

2. Connect to the admin server (use Control-C to terminate connection)by typing:

# telnet ids-lh 10389

Task 9 – Stopping the Directory Service on the FirstNode


1. Stop the admin server by typing:

# /usr/iplanet/ds5/stop-admin

2. Stop the LDAPserver by typing:

# /ldap/slapd-ids-lh/stop-slapd

3. Unconfigure the ids-lh interface by typing:

# ifconfig hme0 removeif ids-lh



Task 10 – Running directoryserver Setup Script onthe Second Node

Perform the following steps on the second node:

1. Plumb and configure the virtual interface by typing:

# ifconfig hme0 addif ids-lh netmask + broadcast + up

2. Run the directoryserver setup script by typing:

# /usr/sbin/directoryserver setupWould you like to continue with configuration? [Yes]:Select the component you want to configure [1]:Choose a configuration type [2]: 3Specify the components you wish to configure [All]:Specify the components you wish to configure [1, 2]:Specify the components you wish to configure [1, 2]:Computer name [mynode]: ids-lh.suned.sun.comSystem User [nobody]: nsuserSystem Group [nobody]: nsgroupDo you want to register this software with an existing iPlanetconfiguration directory server? [No]:Do you want to use another directory to store your data? [No]:Directory server network port [389]:Directory server identifier [mynode]: ids-lhiPlanet configuration directory server administrator ID [admin]:Password: adminPassword (again): adminSuffix [dc=suned, dc=sun, dc=com]:Directory Manager DN [cn=Directory Manager]:Password: dirmanagerPassword (again): dirmanagerAdministration Domain [suned.sun.com]:Do you want to install the sample entries? [No]: yesType the full path and filename, the word suggest, or the word none[suggest]:Do you want to disable schema checking? [No]:Administration port [18733]: 10389IP address [10.10.1.204]: 10.10.1.1Run Administration Server as [root]:



Task 11 – Stopping the Directory Service on theSecond Node

Perform the following steps on the second node:



2. Stop the ldap server by typing:

# /var/ds5/slapd-ids-lh/stop-slapd

Task 12 – Replacing the /var/ds5 Directory on theSecond Node

Perform the following steps on the other node:

1. Rename the /var/ds5 directory by typing:

# mv /var/ds5 /var/ds5.old

2. Create a symbolic link for /var/ds5 by typing:

# ln -s /ldap /var/ds5

Task 13 – Verifying That the Directory Server Runs onthe Second Node


1. Start the ldap server by typing:

# /var/ds5/slapd-ids-lh/start-slapd

2. Query the directory service by typing:


3. Start the admin server by typing:

# /usr/iplanet/ds5/start-admin

4. Query the admin server by typing:

# telnet ids-lh 10389



Task 14 – Stopping the Directory Service on theSecond Node Again




2. Stop the ldap server by typing:

# /var/ds5/slapd-ids-lh/stop-slapd

3. Unconfigure the ids-lh interface by typing:

# ifconfig hme0 removeif ids-lh

Task 15 – Repeating Installation Steps on RemainingNodes

Repeat Tasks 10 through 14 on any remaining nodes in the cluster.

Task 16 – Creating Failover Resource Group andResources for Directory Server

To create a failover resource group and resources for the directory server,perform the following steps on any cluster node:

1. Create an empty resource group by typing:

# scrgadm -a -g ids-rg -h node1,node2[,node3]

2. Add a LogicalHostname resource to the resource group by typing:

# scrgadm -a -L -g ids-rg -l ids-lh

3. Add an HAStoragePlus resource to the resource group by typing:

# scrgadm -a -j ids-hasp-res -t SUNW.HAStoragePlus -g ids-rg \-x FilesystemMountpoints=/ldap -x AffinityOn=true

4. Bring the resource group online by typing:

# scswitch -Z -g ids-rg

5. Add the SUNW.nsldap resource to the resource group by typing:

# scrgadm -a -j ids-res -g ids-rg -t SUNW.nsldap \-y resource_dependencies=ids-hasp-res -y network_resources_used=ids-lh \-y port_list=389/tcp -x confdir_list=/ldap/slapd-ids-lh

6. Enable the ids-res resource by typing:

# scswitch -e -j ids-res



Task 17 – Verifying HA-LDAP Runs Properly in theCluster

Perform the following steps as indicated:

1. From the administrative workstation, verify that the directoryservice is working properly by typing:


2. From any cluster node, switch the resource group to the second nodeby typing:

# scswitch -z -g ids-rg -h second-node

3. From the administrative workstation, verify the directory service isworking properly on the second node by typing:


4. Repeat Steps 2 and 3 for all remaining nodes in the cluster.

Exercise 2: Integrating ORACLE 9i Into Sun Cluster 3.1 Software


Exercise 2: Integrating ORACLE 9i Into Sun Cluster 3.1Software


● Create a logical host name entry in /etc/inet/hosts

● Create oracle user and group accounts

● Create a global file system for the ORACLE software

● Prepare the ORACLE user environment

● Disable access control for the X server on the admin workstation

● Modify /etc/system with kernel parameters for ORACLE software

● Install ORACLE 9i software from one node

● Prepare the ORACLE instance for cluster integration

● Stop the ORACLE instance on the installing node

● Verify the ORACLE software runs properly on the other node

● Install the Sun Cluster 3.1 data service for ORACLE

● Register the SUNW.HAStoragePlus resource type

● Create a failover resource group for the ORACLE service

● Verify that ORACLE runs properly in the cluster

Task 1 – Creating ha-ora-lh Entry in the/etc/inet/hosts File

Create an entry in the /etc/inet/hosts file of all cluster nodes asfollows:

# vi /etc/inet/hosts...10.1.1.2 ha-ora-lh



Task 2 – Creating oracle and dba Accounts

To create the oracle user and dba group accounts, perform the followingsteps on all cluster nodes:

1. Create the dba group account by typing:

# groupadd -g 8888 dba

2. Create the oracle user account by typing:

# useradd -m -c "oracle user" -g dba -u 8888 -d /global/oracle oracle

3. Create the oracle user password by typing:

# passwd oracleNew password: oracleRe-enter new Password: oracle

Task 3 – Creating Global File System for ORACLESoftware

To create the global file system for the ORACLE software, perform thefollowing steps as indicated:

1. Select two disks (cAtAdA and cBtBdB ) from shared storage for a diskgroup by typing (on only one node attached to storage):

# vxdisk list

2. Initialize these disks as VM disks by typing (on only one nodeattached to storage):

# /etc/vx/bin/vxdisksetup -i cAtAdA# /etc/vx/bin/vxdisksetup -i cBtBdB

3. Create a disk group by typing (on only one node attached tostorage):

# vxdg init ha-ora-dg ha-ora1=cAtAdA ha-ora2=cBtBdB

4. Create a 4-Gbyte volume by typing (on only one node attached tostorage):

# vxassist -g ha-ora-dg make oravol 4096m layout=mirror

5. Register the disk group (and synchronize volume information) to thecluster by typing (on only one node attached to storage):

# scconf -a -D name=ha-ora-dg,type=vxvm,nodelist= X:Y

6. Create a UFS file system on the volume by typing (on only one nodeattached to storage):

# newfs /dev/vx/rdsk/ha-ora-dg/oravol



7. On all nodes in the cluster, create an entry in /etc/vfstab for theglobal file system by typing:

/dev/vx/dsk/ha-ora-dg/oravol /dev/vx/rdsk/ha-ora-dg/oravol /global/oracleufs 1 yes global,logging

8. On any cluster node, mount the file system by typing:

# mount /global/oracle

9. From any cluster node, modify the owner and group associated withthe newly mounted global file system by typing:

# chown oracle:dba /global/oracle

Task 4 – Preparing the oracle User Environment

To configure environment variables required for the ORACLE software,run the following commands from the cluster node on which theORACLE software installation will be performed:

1. Switch user to the oracle user by typing:

# su - oracle

2. Edit the .profile file by typing:

$ vi .profileORACLE_BASE=/global/oracleORACLE_HOME=$ORACLE_BASE/products/9.2ORACLE_SID=MYORAPATH=$PATH:$ORACLE_HOME/binDISPLAY=<aws>:0.0export ORACLE_BASE ORACLE_HOME ORACLE_SID PATH DISPLAY

Task 5 – Disabling Access Control of X Server onAdmin Workstation

To allow client GUIs to be displayed, run the following command on theadmin workstation:

# /usr/openwin/bin/xhost +



Task 6 – Editing the /etc/system File for ORACLEParameters

Perform the following steps on all cluster nodes as user root to set SolarisOS kernel parameters required for ORACLE software.

1. Edit /etc/system (or append the HAoracle.systemfile file yourinstructor has prepared for you) by typing:

# vi /etc/system...set shmsys:shminfo_shmmax=4294967295set shmsys:shminfo_shmmin=1set shmsys:shminfo_shmmni=100set shmsys:shminfo_shmseg=10set semsys:seminfo_semmni=100set semsys:seminfo_semmsl=1000set semsys:seminfo_semmns=3000set semsys:seminfo_semopm=100set semsys:seminfo_semvmx=32767

2. Reboot by typing:

# init 6

Task 7 – Running the runInstaller Installation Script

To install the ORACLE software on the local file system, run the followingcommands on the node that has been selected as the one to run theORACLE installation program.

Note – After you have responded to all the dialog boxes in the installationprogram (specifically, you have clicked “Install” in the “Summary” dialogbox), it takes about one hour for the runInstaller program to finish.

1. Unmount the /global/oracle directory as a global file system bytyping:

# umount /global/oracle

2. Mount the /global/oracle directory as a local file system—local tothe node from which the ORACLE installation is to be performed(that is, without the global option so that the installation isexpedited)—by typing:

# mount -o noglobal,logging /dev/vx/dsk/ha-ora-dg/oravol /global/oracle



3. switch user to the oracle user by typing:

# su - oracle

4. Change directory to the location of the ORACLE software by typing:

$ cd ORACLE-software-location /Disk1

5. Run the runInstaller script by typing:

$ ./runInstaller

6. Respond to the dialogs using Table 11-1.

Table 11-1 The runInstaller Script Dialog Answers

Dialog Action

Welcome Click Next.

Inventory Location Verify and click OK.

UNIX Group Name Type dba in the text field, and click Next.

Oracle Universal InstallerorainstRoot.sh script

Open a terminal window as user root on theinstalling cluster node, and run the/tmp/orainstRoot.sh script. When the scriptfinishes, click Continue.

File Locations Verify source and destination, and click Next.

Available Products Verify that the ORACLE9i Database radio buttonis selected, and click Next.

Installation Types Select the Custom radio button, and click Next.

Available Product Components Deselect the following:

● Enterprise Edition Options

● Oracle Enterprise Manager Products

● Oracle9i Development Kit

● Oracle9i for UNIX Documentation

● Oracle HTTP Server

● Oracle Transparent Gateways

and click Next.

Component Locations Verify, and click Next.

Privileged Operating SystemGroups

Verify, and click Next.



Create Database Verify that the Yes radio button is selected, andclick Next.

Summary Verify, and click Install.

Note – This might be a good time to go to lunch.The installation program takes from 30 to60 minutes to finish. When the Install program isalmost complete (approximately 97 percent), youwill be prompted to run a script.

Setup Privileges configuration script Open a terminal window as user root on theinstalling node and run the root.sh script (usedefault values when prompted), and click OKwhen script finishes.

Configuration Tools NetConfiguration Assistant Welcome

Verify that the Perform typical configurationcheck box is not selected, and click Next.

Directory Usage Configuration Select the No, I want to defer this configuration toanother time radio button, and click Next.

Listener Configuration Verify the Listener name is LISTENER, and clickNext.

Select Protocols Verify that TCP is among the Selected Protocols,and click Next.

TCP/IP Protocol Configuration Verify that the Use the standard port number of1521 radio button is selected, and click Next.

More Listeners Verify that the No radio button is selected, andclick Next.

Listener Configuration Done Click Next.

Naming Methods Configuration Verify that the No, I do not want to change thenaming methods configured radio button isselected, and click Next.

Database Configuration Assistant:Welcome

Click Next.

Step 1 of 8 Verify that the Create a database radio button isselected, and click Next.

Table 11-1 The runInstaller Script Dialog Answers (Continued)

Dialog Action



Step 2 of 8 Select the General Purpose radio button, and clickNext.

Step 3 of 8 Type MYORA in the Global Database Name textfield, observe that it is echoed in the SID text field,and click Next.

Step 4 of 7 Verify that the Dedicated Server Mode radiobutton is selected, and click Next.

Step 5 of 7 Verify that the Custom radio button is selectedand that the custom initialization parameters areproperly set. Click Next.

Step 6 of 7 Verify the Storage data (control files, data files,redo logs), and click Next.

Step 7 of 7 Verify that the Create Database check box isselected, and click Finish.

Summary Verify summary of database creation options,parameters, character sets, and data files, andclick OK.

Note – This might be a good time to take a break.The database configuration assistant takes 15 to20 minutes to complete.

Passwords Type SYSas the SYS Password and SYSTEMas theSYSTEM Password, and click Exit.

End of Installation Click Exit, and click Yes (you really want to exit).

Table 11-1 The runInstaller Script Dialog Answers (Continued)

Dialog Action



Task 8 – Preparing ORACLE Instance for ClusterIntegration

On the same node from which the ORACLE software was installed, runthe following commands as indicated to prepare the ORACLE instance forSun Cluster 3.x software integration:

1. Configure an ORACLE user for the fault monitor by typing (as useroracle ):

$ sqlplus /nologSQL> connect / as sysdbaSQL> create user sc_fm identified by sc_fm;SQL> grant create session, create table to sc_fm;SQL> grant select on v_$sysstat to sc_fm;SQL> alter user sc_fm default tablespace users quota 1m on users;SQL> quit

2. Test the new ORACLE user account used by the fault monitor bytyping (as user oracle ):

$ sqlplus sc_fm/sc_fmSQL> select * from sys.v_$sysstat;SQL> quit

3. Create a sample table by typing (as user oracle ):

$ sqlplus /nologSQL> connect / as sysdbaSQL> create table mytable (mykey VARCHAR2(10), myval NUMBER(10));SQL> insert into mytable values (‘off’, 0);SQL> insert into mytable values (‘on’, 1);SQL> commit;SQL> select * from mytable;SQL> quit

Note – This sample table is created only for a quick verification that thedatabase is running properly.

4. Configure the ORACLE listener by typing (as user oracle ):

$ vi $ORACLE_HOME/network/admin/listener.ora

Modify the HOSTvariable to match logical host name ha-ora-lh .

5. Configure the tnsnames.ora file by typing (as user oracle ):

$ vi $ORACLE_HOME/network/admin/tnsnames.ora

Modify the HOSTvariables to match logical host name ha-ora-lh .



6. Rename the parameter file by typing (as user oracle ):

$ cd /global/oracle/admin/MYORA/pfile$ mv initMYORA.ora.* initMYORA.ora

7. On all cluster nodes, allow the user root to use the FTP service bytyping (as user root ):

# vi /etc/ftpd/ftpusers

Delete the entry for root .

8. Replicate the contents of /var/opt/oracle and /usr/local/bin tothe other nodes by typing (as user root ):

# tar cvf /tmp/varoptoracle.tar /var/opt/oracle# tar cvf /tmp/usrlocalbin.tar /usr/local/bin# ftp other-nodeuser: rootpassword:ftp> binftp> put /tmp/varoptoracle.tar /tmp/varoptoracle.tarftp> put /tmp/usrlocalbin.tar /tmp/usrlocalbin.tarftp> quit

9. On the other nodes, extract the tar files by typing (as user root ):

# tar xvf /tmp/varoptoracle.tar# tar xvf /tmp/usrlocalbin.tar

Task 9 – Stopping the ORACLE Software

Stop the ORACLE software on the node from which the software wasinstalled by performing the following steps:

1. Switch user to oracle user (if necessary) by typing:

# su - oracle

2. Shut down the ORACLE instance by typing:

$ sqlplus /nologSQL> connect / as sysdbaSQL> shutdownSQL> quit

3. Stop the ORACLE listener by typing:

$ lsnrctl stop



4. Unmount the /global/oracle file system, and remount it using theglobal option (as user root ) by typing:

# umount /global/oracle# mount /global/oracle

5. Remove the virtual interface configured for ha-ora-lh by typing:

# ifconfig hme0 removeif ha-ora-lh

Task 10 – Verifying ORACLE Software Runs on OtherNodes

Start the ORACLE software on the second node by performing thefollowing steps:

1. Configure a virtual interface with the ha-ora-lh IP address bytyping:

# ifconfig hme0 addif ha-ora-lh netmask + broadcast + up

2. Switch user to oracle user by typing:

# su - oracle

3. Start the ORACLE instance by typing:

$ sqlplus /nologSQL> connect / as sydbaSQL> startupSQL> quit

4. Start the ORACLE listener by typing:

$ lsnrctl start

5. Query the database and then shut it down by typing:

$ sqlplus /nologSQL> connect / as sysdbaSQL> select * from mytable;SQL> shutdownSQL> quit

6. Stop the ORACLE listener by typing:

$ lsnrctl stop

7. Remove the virtual interface for ha-ora-lh (as user root ) by typing:

# ifconfig hme0 removeif ha-ora-lh

8. Repeat Steps 1–7 on all remaining cluster nodes.



Task 11 – Registering the SUNW.HAStoragePlus Type

If necessary (if not done in a previous exercise), register theSUNW.HAStoragePlus type from any one node in the cluster:

# scrgadm -a -t SUNW.HAStoragePlus

Task 12 – Installing and Registering ORACLE DataService

Run the following commands as indicated to install, and register theORACLE data service:

1. Install the SUNW.oracle_server and SUNW.oracle_listenerresource types on all nodes by typing:

# cd sc-ds-location /components/SunCluster_HA_ORACLE_3.1/Sol_9/Packages# pkgadd -d . SUNWscor

2. Register the SUNW.oracle_server and SUNW.oracle_listenerresource types from one cluster node by typing:

# scrgadm -a -t SUNW.oracle_server# scrgadm -a -t SUNW.oracle_listener

Task 13 – Creating Resources and Resource Groupsfor ORACLE

Run the following commands from one cluster node to create the resourcegroup and resources necessary for the ORACLE data service:

1. Create an empty resource group by typing:

# scrgadm -a -g ha-ora-rg -h node1,node2[,node3]

2. Create a LogicalHostname resource by typing:

# scrgadm -a -L -g ha-ora-rg -l ha-ora-lh

3. Create an HAStoragePlus resource by typing:

# scrgadm -a -j hasp-ora-res -g ha-ora-rg -t HAStoragePlus \-x FilesystemMountpoints=/global/oracle -x AffinityOn=true



4. Create an oracle_server resource by creating and running a scriptas follows:

a. Write a script containing the scrgadm command by typing:

# vi /var/tmp/setup-ora-res.sh#!/bin/sh/usr/cluster/bin/scrgadm -a -j ora-server-res -g ha-ora-rg \-t oracle_server \-y resource_dependencies=hasp-ora-res \-x Oracle_sid=MYORA -x Oracle_home=/global/oracle/products/9.2 \-x Alert_log_file=/global/oracle/admin/MYORA/bdump/alert_MYORA.log \-x Parameter_file=/global/oracle/admin/MYORA/pfile/initMYORA.ora \-x Connect_string=sc_fm/sc_fm

b. Make the script executable by typing:

# chmod a+x /var/tmp/setup-ora-res.sh

c. Run the script by typing:

# /var/tmp/setup-ora-res.sh

5. Create an oracle_listener resource by typing:

# scrgadm -a -j ora-listener-res -g ha-ora-rg -t oracle_listener \-x Oracle_home=/global/oracle/products/9.2 -x Listener_name=LISTENER

6. Bring the resource group online by typing:

# scswitch -Z -g ha-ora-rg

Task 14 – Verifying ORACLE Runs Properly in theCluster

Run the following commands as indicated to verify the ORACLE softwareproperly runs in the cluster:

1. Switch user to oracle user on the cluster node not currently runningthe ha-ora-rg resource group by typing:

# su - oracle

2. Query the database from a cluster node not currently running theha-ora-rg resource group by typing:

$ sqlplus sc_fm/sc_fm@MYORASQL> select * from sys.v_$sysstat;SQL> quit

3. As user root (from any cluster node), switch the resource group toanother node by typing:

# scswitch -z -g ha-ora-rg -h other-node



4. Query the database from a cluster node not currently running theha-ora-rg resource group (as user oracle ) by typing:

$ sqlplus sc_fm/sc_fm@MYORASQL> select * from sys.v_$sysstatSQL> quit

Exercise 3: Running ORACLE 9i RAC in Sun Cluster 3.1 Software


Exercise 3: Running ORACLE 9i RAC in Sun Cluster 3.1Software


● Create oracle user and group accounts

● Install cluster packages for ORACLE 9i RAC

● Install the Distributed Lock Manager

● Modify /etc/system for ORACLE software

● Install the CVM license key

● Reboot the cluster nodes

● Create raw volumes required for ORACLE data and logs

● Configure the oracle user environment

● Create the dbca_raw_config file

● Disable access control of the X server on the admin workstation

● Install ORACLE software from Node 1

● Configure the ORACLE listener

● Start the Global Services daemon

● Create an ORACLE database

● Verify ORACLE 9i RAC works properly in cluster

Preparation

Select the nodes on which the ORACLE 9i RAC software will be installedand configured. Note that the nodes you select must all be attached toshared storage.



Task 1 – Creating User and Group Accounts

Perform the following steps on the selected cluster nodes:

Note – If you have already done the HA-ORACLE lab, then delete theoracle user and dba groups from all nodes before proceeding.

1. Create the dba group account by typing:

# groupadd -g 7777 dba

2. Create the oracle user account by typing:

# useradd -g dba -u 7777 -d /oracle -m -s /bin/ksh oracle

3. Provide a password for the oracle user account by typing:

# passwd oracleNew Password: oracleRe-enter new Password: oracle

Task 2 – Installing Cluster Packages for ORACLE RAC

Perform the following commands on the selected cluster nodes:

# cd sc31-framework-software-location# pkgadd -d . SUNWscucm SUNWudlm SUNWudlmr SUNWcvm SUNWcvmr

Task 3 – Installing ORACLE Distributed Lock Manager

Perform the following steps on the selected cluster nodes as user root :

1. Copy the ORCLudlm.tar.Z file by typing:

# cd Oracle9i-software-location /Disk1/racpatch# cp ORCLudlm.tar.Z /var/tmp

2. Uncompress and extract the compressed tar file by typing:

# uncompress /var/tmp/ORCLudlm.tar.Z# tar xvf /var/tmp/ORCLudlm.tar

3. Install the ORCLudlmpackage by typing:

# pkgadd -d /var/tmp ORCLudlm

4. The pkgadd command will prompt you for the group that is to beDBA for the database. Respond by typing:



Please enter the group which should be able to act asthe DBA of the database (dba): [?] dba

Task 4 – Modifying the /etc/system File

Edit the /etc/system file on the selected cluster nodes. You can eithermake the edits manually as described in the following or you canconcatenate a OracleRAC.systemfile provided by your instructor.

Note – If you have already performed the HA-ORACLE lab, then youmust delete the /etc/system entries created for that lab beforeproceeding.

# vi /etc/system...set shmsys:shminfo_shmmax=268435456set shmsys:shminfo_shmmin=200set shmsys:shminfo_shmmni=200set shmsys:shminfo_shmseg=200set semsys:seminfo_semmap=1024set semsys:seminfo_semmni=2048set semsys:seminfo_semmns=2048set semsys:seminfo_semmsl=2048set semsys:seminfo_semmnu=2048set semsys:seminfo_semume=200forceload: sys/shmsysforceload: sys/semsysforceload: sys/msgsys

Task 5 – Installing the CVM License Key

Install the VERITAS license key that enables CVM capabilities on theselected cluster nodes by typing:

# /opt/VRTSvlic/bin/vxlicinst -c

Note – Ask your instructor for the license key as the preceding commandprompts you for one.



Task 6 – Rebooting the Cluster Nodes

Run the following commands as indicated:

1. Shut the cluster down from one cluster node by typing:

# /usr/cluster/bin/scshutdown -y -g0

2. Boot all cluster nodes by typing:

ok boot

Task 7 – Creating Raw Volumes

To create the raw volumes for the ORACLE 9i RAC database, run thefollowing commands from one of the selected cluster nodes:

1. Select four disks (cAtAdA , cBtBdB , cCtCdC, cDtDdD) from sharedstorage (two from one array and two from the other array) for a diskgroup by typing:

# vxdisk list

2. Initialize these disks as VM disks by typing:

# /etc/vx/bin/vxdisksetup -i cAtAdA# /etc/vx/bin/vxdisksetup -i cBtBdB# /etc/vx/bin/vxdisksetup -i cCtCdC# /etc/vx/bin/vxdisksetup -i cDtDdD

3. Create a shared disk group by typing:

# vxdg -s init ora-rac-dg data=cAtAdA log=cBtBdB datamir=cCtCdC \logmir=cDtDdD

4. The volumes must be created for each raw device of the ORACLE 9iRAC database. You can either run a script that your instructor hasprepared for you or create them manually. The OracleRACvolumesscript contains all of the commands for Steps 4 and 5.

# vxassist -g ora-rac-dg make sun_raw_system_400m 400m \layout=mirror data datamir# vxassist -g ora-rac-dg make sun_raw_spfile_100m 100m \layout=mirror data datamir# vxassist -g ora-rac-dg make sun_raw_users_120m 120m \layout=mirror data datamir# vxassist -g ora-rac-dg make sun_raw_temp_100m 100m \layout=mirror data datamir# vxassist -g ora-rac-dg make sun_raw_undotbs1_312m 312m \layout=mirror log logmir# vxassist -g ora-rac-dg make sun_raw_undotbs2_312m 312m \



layout=mirror log logmir# vxassist -g ora-rac-dg make sun_raw_example_160m 160m \layout=mirror data datamir# vxassist -g ora-rac-dg make sun_raw_cwmlite_100m 100m \layout=mirror data datamir# vxassist -g ora-rac-dg make sun_raw_indx_70m 70m \layout=mirror data datamir# vxassist -g ora-rac-dg make sun_raw_tools_12m 12m \layout=mirror data datamir# vxassist -g ora-rac-dg make sun_raw_drsys_90m 90m \layout=mirror data datamir# vxassist -g ora-rac-dg make sun_raw_controlfile1_110m \110m layout=mirror data datamir# vxassist -g ora-rac-dg make sun_raw_controlfile2_110m \110m layout=mirror data datamir# vxassist -g ora-rac-dg make sun_raw_xdb_100m 100m \layout=mirror data datamir# vxassist -g ora-rac-dg make sun_raw_log11_120m 120m \layout=mirror log logmir# vxassist -g ora-rac-dg make sun_raw_log12_120m 120m \layout=mirror log logmir# vxassist -g ora-rac-dg make sun_raw_log21_120m 120m \layout=mirror log logmir# vxassist -g ora-rac-dg make sun_raw_log22_120m 120m \layout=mirror log logmir

5. Change the owner and group associated with the newly-createdvolumes. These commands are part of the OracleRACvolumes scriptprovided by your instructor.

# vxedit -g ora-rac-dg set user=oracle group=dba sun_raw_system_400m# vxedit -g ora-rac-dg set user=oracle group=dba sun_raw_spfile_100m# vxedit -g ora-rac-dg set user=oracle group=dba sun_raw_users_120m# vxedit -g ora-rac-dg set user=oracle group=dba sun_raw_temp_100m# vxedit -g ora-rac-dg set user=oracle group=dba sun_raw_undotbs1_312m# vxedit -g ora-rac-dg set user=oracle group=dba sun_raw_undotbs2_312m# vxedit -g ora-rac-dg set user=oracle group=dba sun_raw_example_160m# vxedit -g ora-rac-dg set user=oracle group=dba sun_raw_cwmlite_100m# vxedit -g ora-rac-dg set user=oracle group=dba sun_raw_indx_70m# vxedit -g ora-rac-dg set user=oracle group=dba sun_raw_tools_12m# vxedit -g ora-rac-dg set user=oracle group=dba sun_raw_drsys_90m# vxedit -g ora-rac-dg set user=oracle group=dba \sun_raw_controlfile1_110m# vxedit -g ora-rac-dg set user=oracle group=dba \sun_raw_controlfile2_110m# vxedit -g ora-rac-dg set user=oracle group=dba sun_raw_xdb_100m# vxedit -g ora-rac-dg set user=oracle group=dba sun_raw_log11_120m



# vxedit -g ora-rac-dg set user=oracle group=dba sun_raw_log12_120m# vxedit -g ora-rac-dg set user=oracle group=dba sun_raw_log21_120m# vxedit -g ora-rac-dg set user=oracle group=dba sun_raw_log22_120m

Task 8 – Configuring oracle User Environment

Perform the following steps on the selected cluster nodes:

1. switch user to oracle user by typing:

# su - oracle

2. Edit .profile (or copy an already-existing file your instructor hasprepared for you) to include environment variables for ORACLE 9iRAC by typing:

$ vi .profileORACLE_BASE=/oracleORACLE_HOME=$ORACLE_BASE/products/9.2TNS_ADMIN=$ORACLE_HOME/network/adminDBCA_RAW_CONFIG=$ORACLE_BASE/dbca_raw_configSRVM_SHARED_CONFIG=/dev/vx/rdsk/ora-rac-dg/sun_raw_spfile_100mDISPLAY=<admin-workstation>:0.0if [ ̀ /usr/sbin/clinfo -n ̀-eq 1 ]; then

ORACLE_SID=sun1fiif [ ̀ /usr/sbin/clinfo -n ̀= 2 ]; then

ORACLE_SID=sun2fiPATH=/usr/ccs/bin:$ORACLE_HOME/bin:/usr/binexport ORACLE_BASE ORACLE_HOME TNS_ADMIN DBCA_RAW_CONFIGexport SRVM_SHARED_CONFIG ORACLE_SID PATH DISPLAY

3. Edit the .rhosts file to allow rsh commands in the ORACLE 9iRAC installation to be performed by typing:

$ echo + > /oracle/.rhosts



Task 9 – Creating the dbca_raw_config File

The dbca_raw_config file defines the locations of the raw devices of theORACLE 9i RAC database. You must either copy an existing file that yourinstructor has prepared for you or manually create the file as user oracleon the selected cluster nodes by typing:

$ vi /oracle/dbca_raw_configsystem=/dev/vx/rdsk/ora-rac-dg/sun_raw_system_400mspfile=/dev/vx/rdsk/ora-rac-dg/sun_raw_spfile_100musers=/dev/vx/rdsk/ora-rac-dg/sun_raw_users_120mtemp=/dev/vx/rdsk/ora-rac-dg/sun_raw_temp_100mundotbs1=/dev/vx/rdsk/ora-rac-dg/sun_raw_undotbs1_312mundotbs2=/dev/vx/rdsk/ora-rac-dg/sun_raw_undotbs2_312mexample=/dev/vx/rdsk/ora-rac-dg/sun_raw_example_160mcwmlite=/dev/vx/rdsk/ora-rac-dg/sun_raw_cwmlite_100mindx=/dev/vx/rdsk/ora-rac-dg/sun_raw_indx_70mtools=/dev/vx/rdsk/ora-rac-dg/sun_raw_tools_12mdrsys=/dev/vx/rdsk/ora-rac-dg/sun_raw_drsys_90mcontrol1=/dev/vx/rdsk/ora-rac-dg/sun_raw_controlfile1_110mcontrol2=/dev/vx/rdsk/ora-rac-dg/sun_raw_controlfile2_110mredo1_1=/dev/vx/rdsk/ora-rac-dg/sun_raw_log11_120mredo1_2=/dev/vx/rdsk/ora-rac-dg/sun_raw_log12_120mredo2_1=/dev/vx/rdsk/ora-rac-dg/sun_raw_log21_120mredo2_2=/dev/vx/rdsk/ora-rac-dg/sun_raw_log22_120mxdb=/dev/vx/rdsk/ora-rac-dg/sun_raw_xdb_100m

Task 10 – Disabling Access Control on X Server ofAdmin Workstation

To allow client GUIs to be displayed, run the following command on theadmin workstation:

# /usr/openwin/bin/xhost +



Task 11 – Installing ORACLE Software

Install the ORACLE software on the first selected cluster node. TheORACLE 9i RAC software automatically copies over the binaries to theother nodes using rsh at the end of the installation.

Perform the following steps on the first selected cluster node as theoracle user:

1. Change directory to the ORACLE9i RAC software location bytyping:

$ cd ORACLE9i-RAC-software-location /Disk1

2. Run the runInstaller installation program by typing:

$ ./runInstaller

3. Respond to the dialog boxes using Table 11-2.

Table 11-2 ORACLE Installation Dialog Actions

Dialog Action

Welcome Click Next.

Inventory Location Verify, and click OK.

UNIX Group Name Type dba in text field, and click Next.

Oracle Universal Installerscript

Open a terminal as the root user on all selected clusternodes, run the /tmp/orainstRoot.sh script, and clickContinue when script is finished.

Cluster Node Selection Use SHIFT-mouse-click to highlight all the nodes youselected at the beginning of the labs, and click Next.

File Locations Verify, and click Next.

Available Products Verify that the Oracle9i Database radio button isselected, and click Next.

Installation Types Select the Custom radio button, and click Next.



Available ProductComponents

Deselect the following components:• Oracle Advanced Security

• Legato Networker Single Server

• Oracle Enterprise Manager Products

• Oracle9i Development Kit

• Oracle9i for UNIX Documentation

• Oracle HTTP Server

• Oracle Transparent Gateways

Click Next.

Component Locations Verify, and click Next.

Privileged Operating SystemGroups

Verify that both text fields have dba as the group, andclick Next.

Create Database Select the No radio button, and click Next.

Summary Verify, and click Install.

Note – This might be a good time to take a break or goto lunch. The installation takes from 30 to 60 minutesto finish. The installation status bar remains at 99percent while the Oracle binaries are being copiedfrom the installing node to the other nodes. When theinstallation is almost complete, you are prompted torun a script on the selected cluster nodes.

Script Open a terminal window as user root on the selectedcluster nodes, and run the script/oracle/products/9.2/root.sh . Specify/oracle/local/bin as the local bin directory whenprompted. Click OK when the script is finished.

Oracle Net ConfigurationAssistant Welcome

Verify that the Perform typical configuration check boxis not selected, and click Next.

Directory UsageConfiguration

Select No, I want to defer this configuration to anothertime, and click Next.

Table 11-2 ORACLE Installation Dialog Actions (Continued)

Dialog Action



Listener Name Click Cancel and click Yes to indicate you are sure wantto cancel the Oracle Net Configuration Assistant.

Error The installation programs considers the cancellation ofthe listener configuration to be a failure. You configurethe listener in the next task—so it is not really a failure.Click OK.

Configuration Tools Verify that the Oracle Cluster Configuration Assistantand Agent Configuration Assistant succeeded, andclick Next.

End of Installation Click Exit and Yes to indicate you really want to exit.

Table 11-2 ORACLE Installation Dialog Actions (Continued)

Dialog Action



Task 12 – Configuring ORACLE Listener

On one cluster node, configure the ORACLE listener by performing thefollowing steps as user oracle :

1. Run the Net Configuration Assistant by typing:

$ /oracle/products/9.2/bin/netca

2. Respond to the dialog boxes using Table 11-3.

Table 11-3 ORACLE Listener Dialog Answers

Dialog Action

TOPSWelcome Verify that the Cluster configuration radio button isselected, and click Next.

TOPSNodes Verify that all selected cluster nodes are highlighted, andclick Next.

Welcome Verify that the Listener configuration radio button isselected, and click Next.

Listener Configuration,Listener

Verify that the Add radio button is selected, and clickNext.

Listener Configuration,Listener Name

Verify that the Listener name text field contains the wordLISTENER, and click Next.

Listener Configuration, SelectProtocols

Verify that TCP is in the Selected Protocols text area, andclick Next.

Listener Configuration,TCP/IP Protocol

Verify that the Use the standard port number of 1521radio button is selected, and click Next.

Listener Configuration, MoreListeners?

Verify that the No radio button is selected, and clickNext.

Listener Configuration Done Click Next.

Welcome Click Finish.



Task 13 – Starting the Global Services Daemon

On the selected cluster nodes, run the following commands as the rootuser:

1. Verify that the gsd Java program is running by typing:

# ps -ef | grep gsd

2. If the gsd Java technology program is not running, start it by typing:

# ORACLE_HOME=/oracle/products/9.2# export ORACLE_HOME# /oracle/products/9.2/bin/gsdctl start

Task 14 – Creating ORACLE Database

Perform the following steps on one of the selected cluster nodes:

1. Run the Database Configuration Assistant as user oracle by typing:

$ /oracle/products/9.2/bin/dbca

2. Use Table 11-4 to respond to the dialog boxes.

Table 11-4 ORACLE Database Creation Dialog Answers

Dialog Action

Welcome Verify that the Oracle cluster database radio buttonis selected, and click Next.

Step 1 of 9 Verify that the Create a database radio button isselected, and click Next.

Step 2 of 9 Click Select All, verify that the selected clusternodes are highlighted, click any cluster node thatshould not be highlighted, and click Next.

Step 3 of 9 Select the New Database radio button, and clickNext.

Step 4 of 9 Type sun in the Global Database Name text field(notice that your keystrokes are echoed in the SIDPrefix text field), and click Next.



Step 5 of 9 Deselect all check box options except Exampleschemas and its check boxes Human Resources andSales History. Click Yes to pop-up dialogs(indicating you want to delete those table spaces).Click the Standard database features... button.Deselect Oracle JVM and Oracle XML DB checkboxes, and click Yes to all pop-up dialogs. Deselectthe Oracle Text check box, and click No to theassociated dialog. Click OK, and click Next.

Step 6 of 9 Verify that the Dedicated server mode radio buttonis selected, and click Next.

Step 7 of 9 Click the File Locations tab, and verify that theCreate server parameters file check box is selected.Verify that the Server Parameters Filename text fieldcorrectly identifies the SRVM_SHARED_CONFIGenvironment variable. Click Next.

Step 8 of 9 Verify that the database storage file locations arecorrect. Note that these file locations are determinedby the contents of the /oracle/dbca_raw_configfile that you prepared in a previous task. Click Next.

Step 9 of 9 Verify that the Create Database check box isselected, and click Finish.

Summary Verify and click OK.

Note – The database creation takes from 20 to30 minutes.

Database ConfigurationAssistant passwords

Type SYS in the SYS Password and Confirm SYSPassword text fields. Enter SYSTEM in the SYSTEMPassword and Confirm SYSTEM Password textfields. Click Exit.

Table 11-4 ORACLE Database Creation Dialog Answers (Continued)

Dialog Action



Task 15 – Verifying That ORACLE RAC Works Properlyin Cluster

Run the following commands as indicated to verify the ORACLE softwareproperly runs in the cluster:

1. Switch user to the oracle user by typing (all selected cluster nodes):

# su - oracle

2. Rename the ORACLE RAC parameter file by typing (all selectedcluster nodes):

$ cd /oracle/admin/mysun/pfile$ mv init.ora* init.ora

3. Query the database by typing (all selected cluster nodes):

$ sqlplus /nologSQL> connect / as sysdbaSQL> select * from V$ACTIVE_INSTANCES;

4. Create a table by typing (from the first selected cluster node):

SQL> create table mytable (mykey VARCHAR2(10), myval NUMBER(10));SQL> insert into mytable values (‘off’, 0);SQL> insert into mytable values (‘on’, 1);SQL> commit;SQL> select * from mytable;SQL> quit

5. Query the database by typing (from the remaining selected clusternodes):

SQL> select * from mytable;SQL> quit

6. Determine which node is currently the master of the shared diskgroup by typing (from all selected cluster nodes):

# vxdctl -c mode

7. Send the break signal to the console of the current master of theshared disk group by typing:

# Control-]telnet> send break

8. Wait for the reconfiguration to complete on the remaining clusternodes. Verify that another node has been promoted to master of theshared disk group from all selected cluster nodes by typing:

# vxdctl -c mode



9. Boot the former master of the shared disk group back into the clusterby typing:

ok sync

Note – This command panics the operating system which, in turn, rebootsthe node.

10. Start the RAC software on the node you just booted by logging in asroot and performing the following steps:

a. Set the ORACLE_HOMEenvironment variable for user root bytyping:

# ORACLE_HOME=/oracle/products/9.2# export ORACLE_HOME

b. Start the gsd daemon as user root by typing:

# /oracle/products/9.2/bin/gsdctl start

c. Switch user to the oracle user by typing:

# su - oracle

d. Start the listener daemon as user oracle by typing:

$ /oracle/products/9.2/bin/lsnrctl start

e. Start the ORACLE instance as user oracle by typing:

$ sqlplus /nologSQL> connect / as sysdbaSQL> startup pfile=/oracle/admin/mysun/pfile/init.ora

Exercise 4: Reinstalling Sun Cluster 3.1 Software


Exercise 4: Reinstalling Sun Cluster 3.1 Software

As indicated at the beginning of this module, there are no instructionsincluded for this exercise because they are described in the previousmodule exercises of this course.

A-1Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1

Appendix A


This appendix describes the configuration of the Sun TerminalConcentrator (TC) as a remote connection mechanism to the serialconsoles of the nodes in the Sun Cluster software environment.

Viewing the Terminal Concentrator

A-2 Sun™ Cluster 3.1 AdministrationCopyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1


The Sun Terminal Concentrator (Annex NTS) has its own internaloperating system and resident administration programs.

Note – If any other TC is substituted, it must not send an abort signal tothe attached host systems when it is powered on.

Figure A-1 shows the TC is a self-contained unit with its own operatingsystem.

Figure A-1 Terminal Concentrator Functional Diagram

Note – If the programmable read-only memory (PROM) operating systemis older than version 52, you must upgrade it.

Setup port

Setup Device

Serial port A

Node 1 Node 2

Memory

Self -Load Power-On

Operating System

PROM

oper.52.enet

NetworkInterface

Serial Ports

1 2 3 4 5 6 7 8


Terminal Concentrator A-3Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1

Operating System Load

You can set up the TC to load its operating system either internally fromthe resident PROM or externally from a server. In the cluster application,it is always set to load internally. Placing the operating system on anexternal server can decrease the reliability of the terminal server.

When power is first applied to the TC, it performs the following steps:

1. It runs a PROM-based self-test and displays error codes.

2. It loads a resident PROM-based operating system into the TCmemory.

Setup Port

Serial port 1 on the TC is a special purpose port that is used only duringinitial setup. It is used primarily to set up the IP address and loadsequence of the TC. You can access port 1 from either a tip connection orfrom a locally connected terminal.

Terminal Concentrator Setup Programs

You must configure the TC nonvolatile random access memory (NVRAM)with the appropriate IP address, boot path, and serial port information.Use the following resident programs to specify this information:

● addr

● seq

● image

● admin

Setting Up the Terminal Concentrator



The TC must be configured for proper operation. Although the TC setupmenus seem simple, they can be confusing and it is easy to make amistake. You can use the default values for many of the prompts.

Connecting to Port 1

To perform basic TC setup, you must connect to the TC setup port.Figure A-2 shows a tip hardwire connection from the administrativeconsole. You can also connect an American Standard Code forInformation Interchange (ASCII) terminal to the setup port.

Figure A-2 Setup Connection to Port 1

Enabling Setup Mode

To enable Setup mode, press the TC Test button shown in Figure A-3 untilthe TC power indicator begins to blink rapidly, then release the Testbutton and press it again briefly.

Figure A-3 Terminal Concentrator Test Button

After you have enabled Setup mode, a monitor:: prompt appears on thesetup device. Use the addr , seq , and image commands to complete theconfiguration.

1 2 3 4 5 6 7 8

STATUS

1 2 3 4 5 6 7 8 POWER UNIT NET ATTN LOAD ACTIVE

Power indicator Test button



Setting the Terminal Concentrator IP Address

The following example shows how to use the addr program to set the IPaddress of the TC. Usually this is set correctly when your cluster arrives,but you should always verify that it is correct.

monitor:: addrEnter Internet address [192.9.22.98]:: 129.150.182.100Enter Subnet mask [255.255.255.0]::Enter Preferred load host Internet address[192.9.22.98]:: 129.150.182.100Enter Broadcast address [0.0.0.0]:: 129.150.182.255Enter Preferred dump address [192.9.22.98]::129.150.182.100Select type of IP packet encapsulation(ieee802/ethernet) [<ethernet>]:: Type of IP packet encapsulation: <ethernet>

Load Broadcast Y/N [Y]:: y

Setting the Terminal Concentrator Load Source

The following example shows how to use the seq program to specify thetype of loading mechanism to be used.

monitor:: seqEnter a list of 1 to 4 interfaces to attempt to use fordownloading code or upline dumping. Enter them in theorder they should be tried, separated by commas orspaces. Possible interfaces are:

Ethernet: net SELF: self

Enter interface sequence [self]::

The self response configures the TC to load its operating systeminternally from the PROM when you turn on the power. The PROM imageis currently called OPER_52_ENET.SYS.

Enabling the self-load feature negates other setup parameters that refer toan external load host and dump host, but you must still define theseparameters during the initial setup sequence.



Note – Although you can load the TC’s operating system from an externalserver, this introduces an additional layer of complexity that is prone tofailure.

Specifying the Operating System Image

Even though the self-load mode of operation negates the use of anexternal load and dump device, you should still verify the operatingsystem image name as shown by the following:

monitor:: image

Enter Image name ["oper.52.enet"]:: Enter TFTP Load Directory ["9.2.7/"]:: Enter TFTP Dump path/filename["dump.129.150.182.100"]::

monitor::

Note – Do not define a dump or load address that is on another networkbecause you receive additional questions about a gateway address. If youmake a mistake, you can press Control-C to abort the setup and startagain.

Setting the Serial Port Variables

The TC port settings must be correct for proper cluster operation. Thisincludes the type and mode port settings. Port 1 requires different typeand modesettings. You should verify the port settings before installing thecluster host software. The following is an example of the entire procedure:

admin-ws# telnet terminal_concentrator_nameTrying terminal concentrator IP address ...Connected to terminal concentrator IP address .Escape character is '̂ ]'.Rotaries Defined: cli -Enter Annex port name or number: cliAnnex Command Line Interpreter * Copyright 1991Xylogics, Inc.annex: su



Password: type_the_passwordannex# adminAnnex administration MICRO-XL-UX R7.0.1, 8 portsadmin: show port=1 type modePort 1:type: hardwired mode: cliadmin: set port=1 type hardwired mode cliadmin: set port=2-8 type dial_in mode slaveadmin: set port=1-8 imask_7bits Yadmin: quitannex# bootbootfile: <CR>warning: <CR>

Note – Do not perform this procedure through the special setup port; usepublic network access.

Setting the Port Password

An optional and recommended security feature is to set per-portpasswords. These provides an extra password challenge as you access aserial port in slave mode remotely through the telnet command.

You can set different (or the same) port passwords on each port. You mustset the enable_security parameter to Y to enable all the passwords.

If you ever forget a port password but know the TC root password, youcan just reset the passwords to whatever you want.

admin-ws# telnet terminal_concentrator_nameTrying terminal concentrator IP address . . .Connected to terminal concentrator IP address .Escape character is '̂ ]'.Rotaries Defined: cli -Enter Annex port name or number: cliAnnex Command Line Interpreter * Copyright 1991Xylogics, Inc.annex: suPassword: type the passwordannex# adminAnnex administration MICRO-XL-UX R7.0.1, 8 ports



admin: set port=2 port_password homeradmin: set port=3 port_password margeadmin: reset 2-3admin: set annex enable_security Yadmin: reset annex security

Setting a Terminal Concentrator Default Route



If you access a TC from a host that is on a different network, the TC’sinternal routing table can overflow. If the TC routing table overflows,network connections can be intermittent or lost completely.

Figure A-4 shows that you can correct this problem by setting a defaultroute within the TC config.annex configuration file.

Figure A-4 Terminal Concentrator Routing

Node 2Node 1 Node 3 Node 4

Router Administrative Console

Network

hme0 hme1

129.50.1.23 129.50.2.17

129.50.1 129.50.2

Terminal Concentrator (129.50.1.35)

%gatewaynet default gateway 129.50.1.23 metric 1 active

config.annex

129.50.2.12



Creating a Terminal Concentrator Default Route

To create a default route for the TC, you must edit an EEPROM file in theTC named config.annex . You must also disable the TC routing function.The following is a summary of the general process.

admin-ws# telnet tc1.centralTrying 129.50.1.35 ...Connected to 129.50.1.35.Escape character is ’̂ ]’.[Return] [Return]Enter Annex port name or number: cli...annex: suPassword: root_passwordannex# edit config.annex( Editor starts )Ctrl-W:save and exit Ctrl-X:exit Ctrl-F:page downCtrl-B:page up%gatewaynet default gateway 129.50.1.23 metric 1 active ̂ Wannex# admin set annex routed nYou may need to reset the appropriate port, Annexsubsystem or reboot the Annex for changes to takeeffect.annex# boot

Note – You must enter an IP routing address appropriate for your site.While the TC is rebooting, the node console connections are not available.

Using Multiple Terminal Concentrators


Using Multiple Terminal Concentrators

A single Sun TC can provide serial port service to a maximum of eightnodes. If it is necessary to reconfigure the TC, the node connection toport 1 must be switched with a serial connection to the configurationdevice, and the TC placed into setup mode. After configuration iscomplete, the normal node connection to port 1 is replaced, and the TCrebooted.

The maximum length for a TC serial port cable is approximately 348 feet.As shown in Figure A-5, it might be necessary to have cluster hostsystems separated by more than the serial port cable limit. You mightneed a dedicated TC for each node in a cluster.

Figure A-5 Multiple Terminal Concentrators

Node 2

Node 1

Router

Administrative Console

TC

TC

Network

Network

Troubleshooting Terminal Concentrators



Occasionally, it is useful to be able to manually manipulate the TC. Thecommands to do this are not well documented in the cluster manuals.

Using the telnet Command to Manually Connect to aNode

If the cconsole tool is not using the TC serial ports, you can use thetelnet command to connect to a specific serial port as follows:

# telnet tc_name 5002

You can then log in to the node attached to port 5002. After you havefinished and logged out of the node, you must break the telnetconnection with the Control-] keyboard sequence and then type quit . Ifyou do not, the serial port remains locked and cannot be used by otherapplications, such as the cconsole tool.

Using the telnet Command to Abort a Node

If you have to abort a cluster node, you can either use the telnetcommand to connect directly to the node and use the Control-] keyboardsequence, or you can use the Control-] keyboard sequence in a clusterconsole window. When you have the telnet prompt, you can abort thenode with the following command:

telnet > send brkok

Note – You might have to repeat the command multiple times.



Connecting to the Terminal ConcentratorCommand-Line Interpreter

You can use the telnet command to connect directly to the TC, and thenuse the resident command-line interface (CLI) to perform status andadministration procedures.

# telnet IPaddressTrying 129.146.241.135...Connected to 129.146.241.135Escape character is '̂ ]'.

Enter Annex port name or number: cliAnnex Command Line Interpreter * Copyright 1991Xylogics, Inc.annex:

Using the Terminal Concentrator help Command

After you connect directly into a terminal concentrator, you can get onlinehelp as follows:

annex: helpannex: help hangup

Identifying and Resetting a Locked Port

If a node crashes, it can leave a telnet session active that effectively locksthe port from further use. You can use the who command to identify whichport is locked, and then use the admin program to reset the locked port.The command sequence is as follows:

annex: whoannex: suPassword:annex# adminAnnex administration MICRO-XL-UX R7.0.1, 8 portsadmin : reset 6admin : quitannex# hangup



Erasing Terminal Concentrator Settings

Using the TC erase command can be dangerous. Use it only when youhave forgotten the superuser password. It returns all settings to theirdefault values. When the addr command is then run to give the TC its IPaddress, the password is set to this IP. For security reasons, the erasecommand is available only through the port 1 interface. A typicalprocedure is as follows:

monitor :: erase

1) EEPROM(i.e. Configuration information)2) FLASH(i.e. Self boot image)

Enter 1 or 2 :: 1

Caution – Do not use option 2 of the erase command; it destroys the TCboot PROM-resident operating system.

B-1Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1

Appendix B

Configuring Multi-Initiator SCSI

This appendix contains information that can be used to configuremulti-initiator SCSI storage devices including the Sun StorEdge MultiPackdesktop array and the Sun StorEdge D1000 array.

Multi-Initiator Overview

B-2 Sun™ Cluster 3.1 AdministrationCopyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1

Multi-Initiator Overview

This section applies only to SCSI storage devices and not to Fibre Channelstorage used for the multihost disks.

In a standalone server, the server node controls the SCSI bus activitiesusing the SCSI host adapter circuit connecting this server to a particularSCSI bus. This SCSI host adapter circuit is referred to as the SCSI initiator.This circuit initiates all bus activities for this SCSI bus. The default SCSIaddress of SCSI host adapters in Sun systems is 7.

Cluster configurations share storage between multiple server nodes.When the cluster storage consists of singled-ended or differential SCSIdevices, the configuration is referred to as multi-initiator SCSI. As thisterminology implies, more than one SCSI initiator exists on the SCSI bus.

The SCSI specification requires that each device on a SCSI bus has aunique SCSI address. (The host adapter is also a device on the SCSI bus.)The default hardware configuration in a multi-initiator environmentresults in a conflict because all SCSI host adapters default to 7.

To resolve this conflict, on each SCSI bus leave one of the SCSI hostadapters with the SCSI address of 7, and set the other host adapters tounused SCSI addresses. Proper planning dictates that these “unused”SCSI addresses include both currently and eventually unused addresses.An example of addresses unused in the future is the addition of storageby installing new drives into empty drive slots. In most configurations,the available SCSI address for a second host adapter is 6.

You can change the selected SCSI addresses for these host adapters bysetting the scsi-initiator-id OpenBoot PROM property. You can setthis property globally for a node or on a per-host-adapter basis.Instructions for setting a unique scsi-initiator-id for each SCSI hostadapter are included in the chapter for each disk enclosure in the Sun™Cluster 3.1 Hardware Guide.

Installing a Sun StorEdge Multipack Device

Configuring Multi-Initiator SCSI B-3Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1


This section provides the procedure for an initial installation of aSun StorEdge MultiPack device.

Use this procedure to install a Sun StorEdge MultiPack device in a clusterprior to installing the Solaris OS and Sun Cluster software. Perform thisprocedure with the procedures in the Sun Cluster 3.1 Installation Guide andyour server hardware manual.

1. Ensure that each device in the SCSI chain has a unique SCSI address.

The default SCSI address for host adapters is 7. Reserve SCSI address7 for one host adapter in the SCSI chain. This procedure refers to thehost adapter you choose for SCSI address 7 as the host adapter onthe second node. To avoid conflicts, in Step 7 you change thescsi-initiator-id of the remaining host adapter in the SCSI chainto an available SCSI address. This procedure refers to the hostadapter with an available SCSI address as the host adapter on thefirst node. Depending on the device and configuration settings ofthe device, either SCSI address 6 or 8 is usually available.

Caution – Even though a slot in the device might not be in use, youshould avoid setting scsi-initiator-id for the first node to the SCSIaddress for that disk slot. This precaution minimizes future complicationsif you install additional disk drives.

For more information, refer to the OpenBoot™ 3.x Command ReferenceManual and the labels inside the storage device.

2. Install the host adapters in the nodes that will be connected to thedevice.

For the procedure on installing host adapters, refer to the documentationthat shipped with your nodes.

3. Connect the cables to the device, as shown in Figure B-1 on page B-4.



Make sure that the entire SCSI bus length to each device is less than6 meters. This measurement includes the cables to both nodes, aswell as the bus length internal to each device, node, and hostadapter. Refer to the documentation which shipped with the devicefor other restrictions regarding SCSI operation.

Figure B-1 Example of a Sun StorEdge MultiPack Desktop ArrayEnclosure Mirrored Pair

4. Connect the AC power cord for each device of the pair to a differentpower source.

5. Without allowing the node to boot, power on the first node. Ifnecessary, abort the system to continue with OpenBoot PROMMonitor tasks.

6. Find the paths to the host adapters.

ok show-disks

Identify and record the two controllers that are connected to thestorage devices, and record these paths. Use this information tochange the SCSI addresses of these controllers in the nvramrc script.Do not include the /sd directories in the device paths.

7. Edit the nvramrc script to set the scsi-initiator-id for the hostadapter on the first node.

For a list of nvramrc editor and nvedit keystroke commands, seethe ‘‘The nvramrc Editor and nvedit Keystroke Commands’’ onpage B-11.

Node 1Host Adapter A Host Adapter B

SCSI IN

SCSI OUT

Enclosure 1

SCSIcables

9-14

1-6

IN

OUT

Node 2Host adapter B Host Adapter A

SCSI IN

SCSI OUT

Enclosure 2

9-14

1-6

IN

OUT



The following example sets the scsi-initiator-id to 6. TheOpenBoot PROM Monitor prints the line numbers (0: , 1: , and soon).

nvedit0: probe-all1: cd /sbus@1f,0/2: 6 encode-int " scsi-initiator-id" property3: device-end4: cd /sbus@1f,0/SUNW,fas@2,88000005: 6 encode-int " scsi-initiator-id" property6: device-end7: install-console8: banner <Control-C>ok

Note – Insert exactly one space after the first double quotation mark andbefore scsi-initiator-id .

8. Store the changes.

The changes you make through the nvedit command are done on atemporary copy of the nvramrc script. You can continue to edit thiscopy without risk. After you complete your edits, save the changes.If you are not sure about the changes, discard them.

● To discard the changes, type:

ok nvquitok

● To store the changes, type:

ok nvstoreok

9. Verify the contents of the nvramrc script you created in Step 7.

ok printenv nvramrcnvramrc = probe-allcd /sbus@1f,0/6 encode-int " scsi-initiator-id" propertydevice-endcd /sbus@1f,0/SUNW,fas@2,88000006 encode-int " scsi-initiator-id" propertydevice-endinstall-consolebannerok



10. Instruct the OpenBoot PROM Monitor to use the nvramrc script.

ok setenv use-nvramrc? trueuse-nvramrc? = trueok

11. Without allowing the node to boot, power on the second node. Ifnecessary, abort the system to continue with OpenBoot PROMMonitor tasks.

12. Verify that the scsi-initiator-id for the host adapter on thesecond node is set to 7.

ok cd /sbus@1f,0/SUNW,fas@2,8800000ok .propertiesscsi-initiator-id 00000007...

13. Continue with the Solaris OS, Sun Cluster software, and volumemanagement software installation tasks.

For software installation procedures, refer to the Sun™ Cluster 3.1Installation Guide.

Installing a Sun StorEdge D1000 Array



This section provides the procedure for an initial installation of a SunStorEdge D1000 array.

Use this procedure to install a Sun StorEdge D1000 array in a cluster priorto installing the Solaris Operating Environment and Sun Cluster software.Perform this procedure with the procedures in the Sun™ Cluster 3.1Installation Guide and your server hardware manual.

1. Ensure that each device in the SCSI chain has a unique SCSI address.

The default SCSI address for host adapters is 7. Reserve SCSIaddress 7 for one host adapter in the SCSI chain. This procedurerefers to the host adapter you choose for SCSI address 7 as the hostadapter on the second node. To avoid conflicts, in Step 7 youchange the scsi-initiator-id of the remaining host adapter in theSCSI chain to an available SCSI address. This procedure refers to thehost adapter with an available SCSI address as the host adapter onthe first node. SCSI address 6 is usually available.

Note – Even though a slot in the device might not be in use, you shouldavoid setting the scsi-initiator-id for the first node to the SCSIaddress for that disk slot. This precaution minimizes future complicationsif you install additional disk drives.

For more information, refer to the OpenBoot™ 3.x Command ReferenceManual and the labels inside the storage device.

2. Install the host adapters in the node which will be connected to thearray.

For the procedure on installing host adapters, refer to thedocumentation that shipped with your nodes.

3. Connect the cables to the arrays, as shown in Figure B-2 on page B-8.



Make sure that the entire bus length connected to each array is lessthan 25 meters. This measurement includes the cables to both nodes,as well as the bus length internal to each array, node, and the hostadapter.

Figure B-2 Example of a Sun StorEdge D1000 Array Mirrored Pair

4. Connect the AC power cord for each array of the pair to a differentpower source.

5. Power on the first node and the arrays.

6. Find the paths to the host adapters.

ok show-disks

Identify and record the two controllers that will be connected to thestorage devices and record these paths. Use this information tochange the SCSI addresses of these controllers in the nvramrc script.Do not include the /sd directories in the device paths.

7. Edit the nvramrc script to change the scsi-initiator-id for thehost adapter on the first node.

For a list of nvramrc editor and nvedit keystroke commands, see‘‘The nvramrc Editor and nvedit Keystroke Commands’’ onpage B-11.

The following example sets the scsi-initiator-id to 6. TheOpenBoot PROM Monitor prints the line numbers (0: , 1: , andso on).

nvedit0: probe-all1: cd /sbus@1f,0/QLGC,isp@3,10000

Node 1Host Adapter A Host Adapter B

Node 2Host Adapter B Host Adapter A

Disk array 1

Disk array 2



2: 6 encode-int " scsi-initiator-id" property3: device-end4: cd /sbus@1f,0/5: 6 encode-int " scsi-initiator-id" property6: device-end7: install-console8: banner <Control-C>ok

Note – Insert exactly one space after the first double quotation mark andbefore scsi-initiator-id .

8. Store or discard the changes.

The edits are done on a temporary copy of the nvramrc script. Youcan continue to edit this copy without risk. After you complete youredits, save the changes. If you are not sure about the changes,discard them.

● To store the changes, type:

ok nvstoreok

● To discard the changes, type:

ok nvquitok

9. Verify the contents of the nvramrc script you created in Step 7.

ok printenv nvramrcnvramrc = probe-allcd /sbus@1f,0/QLGC,isp@3,100006 encode-int " scsi-initiator-id" propertydevice-endcd /sbus@1f,0/6 encode-int " scsi-initiator-id" propertydevice-endinstall-consolebannerok

10. Instruct the OpenBoot PROM Monitor to use the nvramrc script.

ok setenv use-nvramrc? trueuse-nvramrc? = trueok



11. Without allowing the node to boot, power on the second node. Ifnecessary, abort the system to continue with OpenBoot PROMMonitor tasks.

12. Verify that the scsi-initiator-id for each host adapter on thesecond node is set to 7.

ok cd /sbus@1f,0/QLGC,isp@3,10000ok .propertiesscsi-initiator-id 00000007differentialisp-fcode 1.21 95/05/18device_type scsi...

13. Continue with the Solaris OS, Sun Cluster software, and volumemanagement software installation tasks.

For software installation procedures, refer to the Sun™ Cluster 3.1Installation Guide.

The nvramrc Editor and nvedit Keystroke Commands



The OpenBoot PROM Monitor builds its own device tree based on thedevices attached to the system when the boot sequence is invoked. TheOpenBoot PROM Monitor has a set of default aliases for the commonlyoccurring devices in the system.

An nvramrc script contains a series of OpenBoot PROM commands thatare executed during the boot sequence. The procedures in this guideassume that this script is empty. If your nvramrc script contains data, addthe entries to the end of the script. To edit an nvramrc script or mergenew lines in an nvramrc script, you must use nvedit editor and nveditkeystroke commands.

Table B-1 and Table B-2 below list useful nvramrc editor and nveditkeystroke commands, respectively. For an entire list of nvramrc editorand nvedit keystroke commands, refer to the OpenBoot™ 3.x CommandReference Manual.

Table B-1 The nvramrc Editor Commands

Command Description

nvedit Enter the nvramc editor. If the data remains in thetemporary buffer from a previous nvedit session, resumeediting previous contents. Otherwise, read the contents ofnvramrc into the temporary buffer and begin editing it.This command works on a buffer, and you can save thecontents of this buffer by using the nvstore command.

nvstore Copy the contents of the temporary buffer to nvramrc , anddiscard the contents of the temporary buffer.

nvquit Discard the contents of the temporary buffer, withoutwriting it to nvramrc . Prompt for confirmation.

nvrecover Attempts to recover the content of the nvramrc if thecontent was lost as a result of the execution ofset-defaults , then enters the nvramrc editors as withnvedit . This command fails if nvedit is executed betweenthe time the content of nvramrc was lost and the time thecontent of the nvramrc was executed.

nvrun Executes the contents of the temporary buffer.



Table B-2 lists more useful nvedit commands.

Table B-2 The nvedit Keystroke Commands

Keystroke Description

^A Moves to the beginning of the line.

^B Moves backward one character.

^C Exits the script editor.

^F Moves forward one character.

^K Deletes until end of line.

^L Lists all lines.

^N Moves to the next line of the nvramrc editing buffer.

^O Inserts a new line at the cursor position and stay on thecurrent line.

^P Moves to the previous line of the nvramrc editing buffer.

^R Replaces the current line.

Delete Deletes previous character.

Return Inserts a new line at the cursor position and advances tothe next line.

C-1Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1

Appendix C

Role-Based Access Control Authorizations

This document describes the Sun Cluster 3.1 software RBACauthorizations required for the Sun Cluster 3.1 software status andmaintenance commands.

RBAC Cluster Authorizations

C-2 Sun™ Cluster 3.1 AdministrationCopyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1


SunPlex Manager and selected Sun Cluster commands and options thatyou issue on the command line use RBAC for authentication. SeveralRBAC rights profiles are included in Sun Cluster. You can assign theserights profiles to users or to roles to give them different levels of access toSun Cluster. Sun provides the rights profiles shown in Table C-1 with SunCluster software.

Table C-1 Rights Profiles

Rights Profile Includes AuthorizationsThis AuthorizationPermits the RoleIdentity to . . .

Sun ClusterCommands

None, but includes a listof Sun Clustercommands that run witheuid=0

Execute selected SunCluster commands thatyou use to configure andmanage a cluster,including:

scgdevs (1M)

scswitch (1M) (selectedoptions)

scha_control (1HA)

scha_resource_get(1HA)

scha_resource_setstatus (1HA)

scha_resourcegroup_get (1HA)

scha_resourcetype_get (1HA)


Role-Based Access Control Authorizations C-3Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1

Basic SolarisUser

This existing Solarisrights profile containsSolaris authorizations, aswell as:

Perform the sameoperations that the BasicSolaris User role identitycan perform, as well as:

solaris.cluster.device.read

Read information aboutdevice groups

solaris.cluster.gui

Access SunPlex Manager

solaris.cluster.network.read

Read information about IPNetwork MultipathingNote – This authorizationdoes not apply to SunPlexManager.

solaris.cluster.node.read

Read information aboutattributes of nodes

solaris.cluster.quorum.read

Read information aboutquorum devices and thequorum state

solaris.cluster.resource.read

Read information aboutresources and resourcegroups

solaris.cluster.system.read

Read the status of thecluster

solaris.cluster.transport.read

Read information abouttransports

Table C-1 Rights Profiles (Continued)




ClusterOperation

solaris.cluster.appinstall

Install clusteredapplications

solaris.cluster.device.admin

Perform administrativetasks on device groupattributes

solaris.cluster.device.read

Read information aboutdevice groups

solaris.cluster.gui


solaris.cluster.install

Install clustering softwareNote – This authorizationdoes not apply to SunPlexManager.

solaris.cluster.network.admin

Perform administrativetasks on IP NetworkMultipathing attributesNote – This authorizationdoes not apply to SunPlexManager.

solaris.cluster.network.read

Read information about IPNetwork MultipathingNote – This authorizationdoes not apply to SunPlexManager.

solaris.cluster.node.admin

Perform administrativetasks on node attributes

solaris.cluster.node.read

Read information aboutattributes of nodes

solaris.cluster.quorum.admin

Perform administrativetasks on quorum devicesand quorum stateattributes




Role-Based Access Control Authorizations C-5Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision A.2.1

ClusterOperation

solaris.cluster.quorum.read

Read information aboutquorum devices and thequorum state

solaris.cluster.resource.admin

Perform administrativetasks on resource attributesand resource groupattributes

solaris.cluster.resource.read

Read information aboutresources and resourcegroups

solaris.cluster.system.admin

Administer the systemNote – This authorizationdoes not apply to SunPlexManager.

solaris.cluster.system.read

Read the status of thecluster

solaris.cluster.transport.admin

Perform administrativetasks on transportattributes

solaris.cluster.transport.read

Read information abouttransports

SystemAdministrator

This existing Solarisrights profile contains thesame authorizations thatthe Cluster Managementprofile contains.

Perform the sameoperations that the ClusterManagement role identitycan perform, in addition toother systemadministration operations.





ClusterManagement

This rights profilecontains the sameauthorizations that theCluster Operation profilecontains, as well as thefollowing authorizations:

Perform the sameoperations that the ClusterOperation role identity canperform, as well as:

solaris.cluster.device.modify

Modify device groupattributes

solaris.cluster.gui


solaris.cluster.network.modify

Modify IP NetworkMultipathing attributesNote – This authorizationdoes not apply to SunPlexManager.

solaris.cluster.node.modify

Modify node attributesNote – This authorizationdoes not apply to SunPlexManager.

solaris.cluster.quorum.modify

Modify quorum devicesand quorum stateattributes

solaris.cluster.resource.modify

Modify resource attributesand resource groupattributes

solaris.cluster.system.modify

Modify system attributesNote – This authorizationdoes not apply to SunPlexManager.

solaris.cluster.transport.modify

Modify transport attributes



SC338student Guide MAIN

Documents

Transcript of SC338student Guide MAIN