CA eHealth High Availability and Disaster Recovery...

CA eHealth®

High Availability and Disaster Recovery Administration Guide

r6.1

This documentation and any related computer software help programs (hereinafter referred to as the “Documentation”) is for the end user’s informational purposes only and is subject to change or withdrawal by CA at any time.

This Documentation may not be copied, transferred, reproduced, disclosed, modified or duplicated, in whole or in part, without the prior written consent of CA. This Documentation is confidential and proprietary information of CA and protected by the copyright laws of the United States and international treaties.

Notwithstanding the foregoing, licensed users may print a reasonable number of copies of the Documentation for their own internal use, and may make one copy of the related software as reasonably required for back-up and disaster recovery purposes, provided that all CA copyright notices and legends are affixed to each reproduced copy. Only authorized employees, consultants, or agents of the user who are bound by the provisions of the license for the Product are permitted to have access to such copies.

The right to print copies of the Documentation and to make a copy of the related software is limited to the period during which the applicable license for the Product remains in full force and effect. Should the license terminate for any reason, it shall be the user’s responsibility to certify in writing to CA that all copies and partial copies of the Documentation have been returned to CA or destroyed.

EXCEPT AS OTHERWISE STATED IN THE APPLICABLE LICENSE AGREEMENT, TO THE EXTENT PERMITTED BY APPLICABLE LAW, CA PROVIDES THIS DOCUMENTATION “AS IS” WITHOUT WARRANTY OF ANY KIND, INCLUDING WITHOUT LIMITATION, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NONINFRINGEMENT. IN NO EVENT WILL CA BE LIABLE TO THE END USER OR ANY THIRD PARTY FOR ANY LOSS OR DAMAGE, DIRECT OR INDIRECT, FROM THE USE OF THIS DOCUMENTATION, INCLUDING WITHOUT LIMITATION, LOST PROFITS, BUSINESS INTERRUPTION, GOODWILL, OR LOST DATA, EVEN IF CA IS EXPRESSLY ADVISED OF SUCH LOSS OR DAMAGE.

The use of any product referenced in the Documentation is governed by the end user’s applicable license agreement.

The manufacturer of this Documentation is CA.

Provided with “Restricted Rights.” Use, duplication or disclosure by the United States Government is subject to the restrictions set forth in FAR Sections 12.212, 52.227-14, and 52.227-19(c)(1) - (2) and DFARS Section 252.227-7014(b)(3), as applicable, or their successors.

All trademarks, trade names, service marks, and logos referenced herein belong to their respective companies.

Copyright © 2008 CA. All rights reserved.

Contact CA Contact Technical Support

For online technical assistance and a complete list of locations, primary service hours, and telephone numbers, contact Technical Support at http://ca.com/support.

Provide Feedback

If you have comments or questions about CA product documentation, you can send a message to [email protected].

If you would like to provide feedback about CA product documentation, please complete our short customer survey, which is also available on the CA Support website.

http://www.ca.com/support

mailto:[email protected]

http://www.casurveys.com/wsb.dll/166/TIPO_2008_Survey.htm

http://www.casurveys.com/wsb.dll/166/TIPO_2008_Survey.htm

CA Product References This document may reference the following CA products:

■ CA eHealth® AdvantEDGE View

■ CA eHealth® Application Response

■ CA eHealth® Business Service Console (eHealth BSC)

■ CA eHealth® Distributed eHealth

■ CA eHealth® Fault Manager

■ CA eHealth® Live Health® Application

■ CA eHealth® Response

■ CA eHealth® Service Availability

■ CA eHealth® SystemEDGE

■ CA eHealth® TrapEXPLODER

■ CA eHealth® Voice Quality Monitor (VQM)

■ CA eHealth® AIM for Apache

■ CA eHealth® AIM for Microsoft Exchange

■ CA eHealth® AIM for Microsoft IIS

■ CA eHealth® AIM for Microsoft SQL Server

■ CA eHealth® AIM for Oracle

■ CA Insight™ AIM for CA eHealth®

■ CA Insight™ Database Performance Monitor for Distributed Databases (CA Insight DPM for Distributed Databases)

■ CA eHealth® Integration for Alcatel (eHealth - Alcatel)

■ CA eHealth® Integration for Cisco IP Solution Center (eHealth - Cisco ISC)

■ CA eHealth® Integration for Cisco WAN Manager (eHealth - Cisco WAN Manager)

■ CA eHealth® Integration for HP OpenView (eHealth - OpenView)

■ CA eHealth® Integration for Lucent (eHealth - Lucent)

■ CA eHealth® Integration for Netcool (eHealth - Netcool)

■ CA eHealth® Integration for Nortel Preside (eHealth - Nortel Preside)

■ CA eHealth® Integration for Nortel Shasta SCS GGSN (eHealth - Nortel GGSN)

■ CA eHealth® Integration for Psytechnics (eHealth - Psytechnics)

■ CA eHealth® Integration for Starent (eHealth - Starent)

■ CA SPECTRUM®

■ CA Unicenter® Network and Systems Management (Unicenter NSM)

■ CA eTrust® Identity and Access Management (eTrust IAM)

■ CA Embedded Entitlements Manager (CA EEM)

Note: CA Embedded Entitlements Manager (CA EEM) is the new name for eTrust IAM. This product will be rebranded throughout the documentation in a future release.

Contents

Chapter 1: Introduction to High Availability and Disaster Recovery 11 Overview .................................................................................... 11 High Availability .............................................................................. 12

Clusterware .............................................................................. 12 High Availability Clusters .................................................................. 12 High Availability Clusters and Distributed eHealth Clusters................................... 13 Failures .................................................................................. 13 Fail over ................................................................................. 14

How eHealth Works with High Availability ...................................................... 17 How Monitoring Works ........................................................................ 17

What Clusterware Monitors ................................................................ 18

Chapter 2: Sun Cluster High Availability Configuration 21 How to Configure Sun Cluster with High Availability ............................................. 21

Step 1: Install and Configure Sun Cluster Software ......................................... 22 Step 2: Register Clusterware Resource Types............................................... 22 Step 3: Create Resource Types and the Resource Group..................................... 23 Step 4: Install eHealth .................................................................... 24 Step 5: Configure eHealth to Run in the High Availability Cluster ............................. 25 Step 6: (Optional) Install CA XOsoft Replication Software.................................... 26 Step 7: Create, Install, and Register eHealth Resource Types ................................ 26 Step 8: Create and Add Clusterware Resources to the Resource Group ....................... 28 Step 9: Create and Add eHealth Resources to the Resource Group ........................... 29 Step 10: Start the High Availability Service ................................................. 30

How to Configure Multiple Instances of eHealth ................................................. 31 High Availability Cluster Management Tasks .................................................... 32

Install a Patch, Service Pack, or Upgrade ................................................... 32 Upgrade the Sun Cluster High Availability Package .......................................... 33 Load the Database after High Availability Configuration...................................... 34 Initiate a Planned Failover ................................................................. 34 How to Enable or Disable SSL for Sun Cluster............................................... 35

Chapter 3: Veritas High Availability Configuration 37 Critical Resource Functionality ................................................................. 37

Critical Resources......................................................................... 37 Non-critical Resources..................................................................... 38

Contents 7

How to Configure Veritas with High Availability ................................................. 38 Step 1: Install and Configure VCS Software................................................. 39 Step 2: Install eHealth .................................................................... 40 Step 3: Configure eHealth to Run in the High Availability Cluster ............................. 40 Step 4: Create a Service Group ............................................................ 42 Step 5: (Optional) Install CA XOsoft Replication Software.................................... 43 Step 6: Create and Install eHealth Resource Types.......................................... 43 Step 7: Start the High Availability Service .................................................. 46

Multiple Instances of eHealth in the HA Cluster ................................................. 46 How to Add Multiple Instances of eHealth................................................... 47

High Availability Cluster Management Tasks .................................................... 48 Install a Patch or Service Pack ............................................................. 49 Load the Database after High Availability Configuration...................................... 50 Initiate a Planned Failover ................................................................. 51 How to Enable or Disable SSL for Veritas ................................................... 51

Chapter 4: Disaster Recovery 53 How eHealth Works with Disaster Recovery .................................................... 53 System Guidelines ........................................................................... 54

System Terminology ...................................................................... 54 How CA XOsoft Replication Software Works .................................................... 55

Scenarios ................................................................................ 55 Replication ............................................................................... 55

Failover in a Disaster Recovery Environment ................................................... 56 Downtime and Data Loss...................................................................... 56 Configure eHealth for Disaster Recovery ....................................................... 56

Before You Configure eHealth for DR ....................................................... 56 Windows Disaster Recovery Configuration .................................................. 57 Solaris Disaster Recovery Configuration .................................................... 62

Start the Failover Process ..................................................................... 67 How to Upgrade Your Disaster Recovery Environment .......................................... 71 Disaster Recovery Management Tasks ......................................................... 72

Initiate Failover from the Standby (Active) System to the Primary System.................... 72 Install a Patch or Service Pack after Configuring Disaster Recovery........................... 72

Appendix A: Disaster Recovery in a High Availability Cluster 75 Configure eHealth for Disaster Recovery in a High Availability Cluster ............................ 75 Failover from High Availability Cluster to Standby Disaster Recovery System ..................... 78 Failover From Standby System to a System in a High Availability Cluster ......................... 80

8 High Availability and Disaster Recovery Administration Guide

Contents 9

Appendix B: Sun Cluster Commands 87

Appendix C: Veritas Cluster Service Commands 89

Index 91

Chapter 1: Introduction to High Availability and Disaster Recovery

This section contains the following topics:

Overview (see page 11) High Availability (see page 12) How eHealth Works with High Availability (see page 17) How Monitoring Works (see page 17)

Overview eHealth can be configured to run in the following environments:

■ High availability (Solaris 2.9 or 2.10)

■ Disaster recovery (Windows 2003 and Solaris 2.9 or 2.10)

Running eHealth in these environments helps ensure that internal system failures (such as hardware or software malfunctions) and external failures (such as a natural disaster or operator error) will minimize eHealth system downtime and data loss.

High Availability-High availability (HA) is a system implementation based on levels of redundancy that helps ensure a system or application can quickly come back online in the event of a failure. Highly available systems are often characterized by the ability of their components to fail over to backup systems in the event of a failure.

For a strong HA solution, many large system environments are configured into clusters using clusterware or other application support. These HA clusters consist of two or more nodes that work together to build hardware and software redundancy: if one system fails, another can become active and resume its duties.

This functionality should not be confused with load balance (or web farm) clusters, where multiple machines run concurrently with identical hardware and software configurations. When one system fails, the others continue to function as before, picking up the load dropped by the failed machine.

Chapter 1: Introduction to High Availability and Disaster Recovery 11

High Availability

Disaster Recovery-Disaster recovery (DR) is a network configuration that becomes vital when systems fail due to disasters like floods or fires. The DR service helps recover critical data after such events occur. DR includes the continual backup of data from a primary system to a standby system in a geographically separate location. After a catastrophic failure causes the primary system to go down, the standby system is manually started and resumes the first system’s duties.

When planning your DR environments, keep in mind that to configure eHealth for DR, you must first install CA XOsoft Replication software. You can also configure an eHealth Solaris HA cluster for DR using either Sun Cluster software or Veritas Cluster Server software.

Note: In the DR environment, failover is a manual process.

High Availability This section describes:

■ HA clusterware and associated concepts, including clusters, failures, and failover.

■ The process to implement HA in your infrastructure so eHealth can function as a highly available data service.

Clusterware

Clusterware is software that monitors hardware and connectivity issues and builds redundancy into a system cluster to eliminate single points of failure. Clusterware must be installed and configured on all machines in the environment before you install eHealth.

High Availability Clusters

Clusterware enables you to configure several similar systems that use shared storage into a highly available cluster, and deploy instances of applications into that cluster. When a failure occurs, the clusterware helps ensure the applications continue to run by restarting the instances on that system or failing over to a backup system.


High Availability

Build the High Availability Cluster

You begin building an HA cluster when you install clusterware on the first machine, or node. When you install clusterware on the second node, you tell it to join the group that the first node is a member of. Continue adding nodes to the group until your cluster is built. You need at least two nodes in a cluster to achieve redundancy.

HA clusters are considered failover pools; machines that operate, fail over, and share resources together. Each cluster node is a stand-alone server that runs its own processes.

This HA cluster solution uses a shared-storage model so that all applications connected to the cluster have read and write access to the stored data.

High Availability Clusters and Distributed eHealth Clusters

eHealth systems are configured to run as stand-alone machines (a single workstation) or part of a Distributed eHealth Cluster (multiple eHealth systems located centrally or across a geographic range). For distributed eHealth members to be highly available, they can be configured to be members of an HA cluster.

While they may work in the same environment, each of these cluster types has its own job, separate from the other. For example, the nodes of an HA cluster function normally (as described in High Availability Clusters (see page 12)) whether or not they share members with a Distributed eHealth Cluster. Also, a Distributed eHealth Cluster is not aware when its members become part of an HA cluster and continue to poll and process data across multiple eHealth systems.

Failover behavior does not change in a Distributed eHealth Cluster sharing membership with an HA cluster. Systems can be added or removed from either cluster type without interrupting functionality.

Note: For more information about Distributed eHealth Clusters, see the Distributed eHealth Administration Guide.

Failures

No matter how well a system is designed or maintained, there will always be system failures. There are various types of failures that can have an impact on the availability of eHealth servers and applications. These include hardware problems such as a failed CPU, a corrupted disk, and bad memory; also software problems such as mis-configuration of the operating system and incorrectly specified networking parameters.


High Availability

eHealth failures can cause the following types of outages:

■ eHealth is inaccessible, with eHealth administrators unable to access it through the OneClick for eHealth interface.

■ eHealth is unable to poll or save data to the eHealth database.

■ eHealth fails if all elements return a bad poll.

■ eHealth reports run through the web are inaccessible.

■ Live Health and reporting applications are inaccessible.

■ Scheduled eHealth reports are not run.

■ Reports being run at the time of failure are lost.

When you configure your eHealth environment to be highly available, eHealth servers and applications continue to run and provide access to data with minimal interruption, even when failures occur.

Fail over

Failover is a service that initiates when a node or application failure is detected. To achieve failover, an automatic series of steps is performed by clusterware to migrate eHealth resources and applications from a failing node to a backup node in the HA cluster, providing HA through redundancy.

During failover in an eHealth environment, the clusterware will either try to restart the eHealth application or start and run the application on the backup system, making the backup system active. After the failed application or system recovers, the backup system can be manually failed back to the primary system, and the primary system becomes active again.

Note: For information about failover in a DR environment, see Failover in a Disaster Recovery Environment.

eHealth continually monitors the status of the poller. Depending on a default time-out threshold, eHealth determines when to restart or failover its monitored applications.

High Availability Failover Configurations

In general, the clusterware you use determines which HA configurations you can have in your eHealth environment. Sun Cluster and Veritas software support the following cluster configurations:

■ Two nodes (1 + 1); one primary and one backup.

■ Multiple nodes (N + 1); several primary with a single shared backup.

■ Multiple nodes (N * N) with multiple backups.


High Availability

The following graphic details how eHealth is configured in an HA cluster with two nodes. As shown, the cluster is running properly, ready in the event of failover.


High Availability

Downtime and Data Loss

There may be some downtime and data loss when your eHealth system fails over.

Downtime-Downtime is a period of time when you cannot access the eHealth system. For example, you cannot run a report or the system does not register a poll.

Note: If you are connected to an eHealth client application using a shared (virtual) hostname, such as OneClick for eHealth, after the failover you can use the session you were using before failover without having to log back in to the application.

Data Loss- Data loss is a period of time when the eHealth system does not collect data, or the data that was collected is lost. After an eHealth system fails over there will be a short data gap as the backup servers start and the poller initializes. If eHealth fails over while running a report, the process will be interrupted and the data will be lost. Scheduled jobs in the run queue will also be lost during failover.

After Failover

After failover, the system on which the failure occurred is offline and its processes have been migrated to a backup system and started there, making the backup system active. While the backup system is active, you should perform any necessary maintenance in order to bring the failed system back online. You can then keep the system as a backup system or plan to fail back to the original system.

Planned Failover

You can manually bring eHealth online and offline using the command line interface. This is helpful when you perform planned maintenance on a system running eHealth. You can essentially force a failover at a convenient time allowing you to perform maintenance to the hardware or operating system without interrupting HA functionality. After the primary system is functional, bring the system back online and transfer operations back to the original system.

When you bring eHealth up on the backup system, eHealth will experience missed polls causing some downtown, often minimal, depending on the number of elements you are polling and your system configuration.

Note: When a system or application is considered online, it is turned on and active. When a system or application is considered offline, it is physically shut down.


How eHealth Works with High Availability

How eHealth Works with High Availability eHealth can be configured with HA clusterware to run as a data service in a highly available cluster, where, on a separate disk, there is one instance of the eHealth server and Oracle. In an HA configuration, a number of different machines will be able to mount the disk and have access to eHealth. This disk can be RAID 1 configured: if one part of the disk fails, another takes over and continues the eHealth service without interruption. This adds another level of redundancy, buffering your system from potential downtime and data loss.

The eHealth HA cluster functionality includes:

■ The nhHaSetup command, which allows an existing eHealth system to be managed by third-party clusterware.

■ The ability for an HA cluster to join an eHealth system in a Distributed eHealth Cluster.

■ The ability for the clusterware to speak with eHealth so it can monitor the eHealth applications and let eHealth monitor polling status.

■ The use of shared hostnames in the eHealth environment. During failover, the clusterware moves the shared hostname (or names) from the primary system to the backup system; this lets you continue accessing eHealth applications with very little interruption.

Note: When you connect to the eHealth server, Oracle, or any eHealth applications in an HA cluster, you must use the shared hostname (or names) confirmed when you initially configured eHealth for HA using the nhHaSetup command.

■ The use of one eHealth license across multiple systems in an HA cluster. When a failover occurs, you will not have to change or update any license files.

How Monitoring Works After eHealth is configured for HA, all systems and applications in the HA cluster are monitored, helping ensure early detection of hardware and software failures. eHealth and the clusterware have specific applications and systems they monitor.


How Monitoring Works

To communicate with the clusterware, all eHealth applications and services must be organized using the following:

Resource Type

For monitoring purposes, all eHealth applications and processes must be configured as resource types to be managed by the clusterware. A resource type is a group of parameters that helps define the application in an HA cluster, defining where it is and what to monitor. For example, when a resource fails it can either be restarted on the same system or all the resources in a group can be failed over, depending on the group’s configuration.

Resource

A resource is one instance of a resource type defined across an HA cluster.

Resource/Service Group

A resource group or service group is a collection of resources that should all fail over at once. When all eHealth applications reside in a group, an association of dependence is created. For example, eHealth startup may depend on the Oracle startup. Also, if a resource has to restart on a backup system, all resources in the group will restart on that backup system.

What Clusterware Monitors

Clusterware monitors the HA cluster to determine if all cluster nodes are functioning properly. It also monitors most eHealth resources.

High Availability Cluster Node Failure

Clusterware monitors hardware and network connectivity problems in the HA cluster. Each node contacts the other nodes with a heartbeat message to verify that it is connected and functioning properly. When the nodes get a response, the cluster continues functioning normally. If a node does not respond, each node continues to send a monitored heartbeat message to the other nodes, confirming that there is a unresponsive node. It then fails over the node and migrates resources to a backup node.



Clusterware Resources

The HA clusterware monitors the following resources:

Note: Resource failures result in failover, unless otherwise noted.

■ Host (IP address)-The clusterware validates that the shared hostname that you specify is connected properly to the network. A failure occurs if the local machine thinks the IP address is no longer valid.

■ Disk-The eHealth and Oracle processes reside on one disk in the HA cluster. The clusterware confirms that the disk is correctly mounted. A failure occurs if your disk connection fails.

■ Oracle and Oracle Listener-The clusterware validates that Oracle and Oracle Listener are running and the processes are up. A failure occurs if the Oracle processes stop running. See the clusterware documentation regarding any additional monitoring capability.

■ Apache Web Server-The clusterware validates that the Apache processes are up and running. A failure occurs if Apache processes stop running. See the clusterware documentation regarding any additional monitoring capability.

■ WANSync (if you configure the primary HA system to be configured for DR)-The clusterware validates that the principal (eHealth data) scenario is running and the WANSync daemons exist in the process table. A failure occurs if the principal scenario is stopped or in an otherwise non-running state, or the daemons are not in place. A WANSync resource failure results in the following actions, depending on your clusterware:

– Veritas Cluster Service-The WANSync resource shows a fault but does not initiate failover.

– Sun Cluster-The WANSync resource initiates failover.

Note: When you use Sun Cluster software, the Host and Disk resources must be set up before eHealth is installed. Oracle, Oracle Listener, and Apache resources must be set up after eHealth is installed.

■ Licensing-The clusterware validates that the license processes are running. A failure occurs is the processes shut down.

■ TrapEXPLODER-The clusterware validates that the trap processes are running. A failure occurs if these processes shut down. A TrapEXPLODER resource failure results in the following actions, depending on your clusterware:

– Veritas Cluster Service-The TrapEXPLODER resource shows a fault but does not initiate failover.

– Sun Cluster-The TrapEXPLODER resource initiates failover.




■ Servlet (Tomcat)-The clusterware validates that the Tomcat processes are running. A failure occurs if these processes stop. A Tomcat resource failure results in the following actions, depending on your clusterware:

– Veritas Cluster Service-The Tomcat resource shows a fault but does not initiate failover.

– Sun Cluster-The Tomcat resource initiates failover.

■ Report Center (if installed)-The clusterware confirms that Report Center has a valid connection. A failure occurs when the clusterware does not detect a connection. A Report Center resource failure results in the following actions, depending on your clusterware:

– Veritas Cluster Service-The Report Center resource shows a fault but does not initiate failover.

– Sun Cluster-The Report Center resource initiates failover.

What eHealth Monitors

eHealth monitors itself for poller status as well as the status of the eHealth startup server. The monitoring reads a configuration file to determine what thresholds should be used to control monitoring.

The Poller Monitor

An integral part of the eHealth program is its ability to poll devices in your infrastructure and write data to the eHealth database. In an HA cluster, eHealth runs its monitoring process once every minute and can detect and validate serious polling errors.

Failure of the eHealth system to write data to the database by the statistics, traffic, or import poller within a certain time frame will trigger the system to restart or failover. For example, if the poller is configured to poll once every five minutes and after fifteen minutes has not written data to the database, the eHealth monitoring program will return a failed message.

Failure Thresholds

Thresholds, parameters by which failure is measured, are used as part of the monitoring process. When you configure eHealth for HA, eHealth loads threshold defaults for each application. The thresholds determine when an eHealth system or application has failed. Thresholds should only be adjusted by Technical Support.

Chapter 2: Sun Cluster High Availability Configuration


How to Configure Sun Cluster with High Availability (see page 21) How to Configure Multiple Instances of eHealth (see page 31) High Availability Cluster Management Tasks (see page 32)

How to Configure Sun Cluster with High Availability The following list outlines the process to configure eHealth for HA with Sun Cluster:

1. Install and configure Sun Cluster software.

2. Register clusterware resource types.

3. Create resource types and the resource group.

4. Install eHealth.

5. Configure eHealth to run in the HA cluster.

6. (Optional) Install CA XOsoft Replication software.

7. Create, install, and register eHealth resource types.

8. Create and add clusterware resources to the resource group.

9. Create and add eHealth resources to the resource group.

10. Start the high availability service.

Note: In the following procedures, rgroup-1 represents the name of the unique resource group.

Chapter 2: Sun Cluster High Availability Configuration 21

How to Configure Sun Cluster with High Availability

Step 1: Install and Configure Sun Cluster Software

For eHealth to be configured for HA, Sun Cluster software version 3.1 or 3.2 must first be installed and configured, creating an HA cluster that eHealth can be managed by. When planning your HA cluster, follow these guidelines:

■ Configure the primary system to use the shared (or virtual) hostname. Add the shared hostname and IP address to the /etc/hosts file on the primary system by doing the following:

1. Log on to the primary system as the root user and open a command line interface.

2. In the /etc/inet/hosts and /etc/inet/ipnodes files, add the shared IP address and hostname.

These files map IP addresses to hostnames.

■ Use the procedures in the Sun Cluster Software Installation Guide to do the following:

– Install the Sun Cluster software on all systems that will make up the HA Cluster.

– Configure the HA cluster and setup disk sets for locally mounted, highly available storage.

■ Configure IP Multipath Networking in the HA cluster so that shared hostnames can be used.

■ Install the Sun Cluster Apache agent and the Sun Cluster Oracle (not Oracle RAC) agent on each node in the HA cluster.

Step 2: Register Clusterware Resource Types

Before you install eHealth you must register the resource types that the clusterware will monitor.

Note: Complete all the procedures in this chapter from any node in the HA cluster.



To register clusterware resource types

1. From any system in the HA cluster, open a CLI.

2. Enter the following commands to install the Sun Cluster resource types:

> su root

# scrgadm -a -t SUNW.apache

# scrgadm -a -t SUNW.oracle_server

# scrgadm -a -t SUNW.oracle_listener

# scrgadm -a -t SUNW.HAStoragePlus

The following resource types are installed and registered: Apache, Oracle, Oracle Listener, and Disk (HAStoragePlus).

Step 3: Create Resource Types and the Resource Group

When you create the resource group it must point to the instance of eHealth you will be installing. You must supply the name of the directory where eHealth will be installed so that the resource group can indicate the location of support tools to the rest of the eHealth resources. During this step the following actions occur:

■ The Host (IP) resource type is created.

■ The Disk resource type is created. This will be a highly available local file system. Remember this location as you will need to install eHealth on the path you specify for this resource.

■ The eHealth resource types are created.

■ The resource group is brought online so that the disks are accessible.

Choose a unique name for the resource group. If you are configuring multiple instances of eHealth to run in the HA cluster, each must reside in a separate resource group with a unique name. Also, each resource type you create must be given a unique name.



To create the resource types and resource group

Note: The file system should not already be mounted. The file system will be mounted during this step when you run the Sun command scswitch.

1. Open a CLI.

2. Enter the following command:

> su root

# scrgadm -a -g rgroup-1 -y PathPrefix=/mountpoint

# scrgadm -a -j ehealth-host -g rgroup-1 -L -l sharedhost

# scrgadm -a -j ehealth-storage -g rgroup-1 -t SUNW.HAStoragePlus \

-x FilesystemMountPoints=/mountpoint \

-x AffinityOn=True

# scswitch -Z -g rgroup-1

rgroup-1

Represents the unique resource group name.

mountpoint

Represents the file system mountpoint

sharedhost

Represents the shared hostname that must always be used whenever you access the eHealth server and applications.

Step 4: Install eHealth

Install eHealth using the eHealth Installation Guide and the following guidelines:

■ Install eHealth into the storage directory you created in Step 3: Create Resource Types and the Resource Group (see page 23).

■ Confirm that you install eHealth, Oracle, and create the eHealth database on the path specified in the Disk resource in Step 3. For example, install eHealth in /mountpoint/software, install Oracle in /mountpoint/oracle, and create the database in /mountpoint/database.

■ Before you install the eHealth license file, license.dat, you need to configure the file to include all host IDs, including all possible backup systems by doing the following:

1. Send all host IDs to the eHealth product licensing team. You will receive back one license file.

2. Copy and paste the license keys into the license.dat file located on the mountpoint/software/lmgr directory. It will reside on the disk to be accessed by all the eHealth systems in the HA cluster.



Note: If you are configuring eHealth for both HA and DR, to save time you can install the WANSync software concurrent to eHealth. Do not create the replication scenarios until eHealth has been configured for HA. For complete steps, see Configuring eHealth for Disaster Recovery in a High Availability Cluster (Solaris Only).

Step 5: Configure eHealth to Run in the High Availability Cluster

After you install eHealth, you must run the nhHaSetup command to configure eHealth to be started and managed by Sun Cluster.

To configure eHealth to run in the HA cluster

1. (Optional) If your system is already running as part of a Distributed eHealth Cluster, you must complete this step before you run nhHaSetup. From the primary system, run the following command:

nhModifyClusterMember -cluster -name physicalHost -newName sharedhost

physicalHost

Represents the name of the primary HA system prior to being HA enabled

sharedhost

Represents the shared hostname.

This prepares the Distributed eHealth Cluster to allow the cluster members to communicate with the primary system using the shared hostname.

Note: If your system is already running as part of a Distributed eHealth Cluster and you have already run nhHaSetup, you must enter the following commands to establish communication between all eHealth cluster members, from a system in the Distributed eHealth Cluster that nhHaSetup was not run on: nhModifyClusterMember -all -name physicalHost -newName sharedhost

You will receive an error from the Distributed eHealth Cluster member configured for HA because the other systems cannot communicate with it until the command finishes running.

2. Open a CLI and start the eHealth database by entering the following command:

$NH_HOME/bin/nhStartDb



3. Enter the following command to configure TrapEXPLODER to run in the HA cluster by copying /opt/trapx to a shared file system:

> su root

# mv /opt/trapx /mountpoint

mountpoint

Represents a shared file system mountpoint.

4. Enable and verify the shared hostname on the system:

a. As root, in the command line interface, enter the following:

ifconfig XXXX:n plumb

ifconfig XXXX:n ipaddress up

b. Verify the shared hostname is properly set by entering the following:

ifconfig -a

The names of the system interfaces should appear in the output.

5. Enter the following command to configure eHealth to support shared hostnames:

> $NH_HOME/bin/nhHaSetup -hostname sharedhost

On WANSync (if installed):

# rm /etc/rc*.d/[SK]*wansync

6. Enter the following command to remove the startup files to prevent eHealth from starting automatically at startup time:

> su root

# rm /etc/rc*.d/[SK]*httpd

# rm /etc/rc*.d/[SK]*nethealth

# rm /etc/rc*.d/[SK]*nhreportcenter

# rm /etc/rc*.d/[SK]*trapexploder

Step 6: (Optional) Install CA XOsoft Replication Software

Complete this step only if you want your HA cluster to be configured for DR. For the complete procedure, see Configuring eHealth for Disaster Recovery in a High Availability Cluster (Solaris Only).

Step 7: Create, Install, and Register eHealth Resource Types

The resource types necessary to monitor eHealth must be installed on every node in the HA cluster. They also need to be registered.



To create, install, and register eHealth resource types

1. As the eHealth user, stop the eHealth processes by entering the following commands:

$NH_HOME/bin/nhServer stop

$NH_HOME/bin/nhReportCenter stop

$NH_HOME/bin/nhHttpd stop

$NH_HOME/bin/nhLmgr stop

$NH_HOME/bin/nhStopDb

The eHealth processes are stopped.

2. On the machine on which the resource group is currently online, enter the following command to access the Solaris package: > su root

# cd $NH_HOME/sys

# pkgadd -d CAehealthHA.pkg

3. Change to the next machine on which you want to install the resources by entering the following commands:

> su root

# scswitch -z -g rgroup-1 -h system_name

system_name

Represents the name of the machine on which you want to install the resource types.

4. Log on to the next machine and repeat Step 2 and 3. Continue this until the resource types have been installed on every node in the cluster.

5. On any machine, enter the following commands to register the eHealth resource types:

> su root

# scrgadm -a -t CA.ehealthLicense

# scrgadm -a -t CA.ehealth

# scrgadm -a -t CA.ehealthRpt

# scrgadm -a -t CA.ehealthServlet

# scrgadm -a -t CA.ehealthTrap

# scrgadm -a -t CA.ehealthWansync

# scrgadm -a -t CA.ehealthWansync is only needed if you are setting up HA and DR.

The eHealth resource types are now created, installed, and registered.

Note: To view all registered resource types, enter the following command: scrgadm -p



Step 8: Create and Add Clusterware Resources to the Resource Group

The clusterware resources must be created. To do this, you create the Apache and Oracle resources and add them to the resource group.

To configure clusterware resources

1. If you have not done so, source the nethealthrc eHealth file as root for your shell environment using one of the following commands:

Bourne: . ./nethealthrc.sh

C: source nethealthrc.csh

Korn: ./nethealthrc.ksh

The eHealth environment is set.

2. Enter the following command to create an Apache resource to monitor the web server:

> su root

# scrgadm -a -j ehealth-web -g rgroup-1 \

-t SUNW.apache -y Network_resources_used=ehealth-host \

-y Scalable=False -y Port_list=$NH_HTTP_PORT/tcp \

-x Bin_dir=$NH_HOME/web/httpd/bin \

-y resource_dependencies=ehealth-storage

Note: If SSL has been enabled, create the ‘ehealth-web’ resource by using the NH_HTTP_PORT value found in the $NH_HOME/web/webCfg/servers.properties file. For example, if the Port value for SSL is now 443, the command to create the resource should read ‘scrgadm –a –j ehealth-web…..Port_list=443/tcp’. If SSL is not enabled, continue to use ‘Port_list=$NH_HTTP_Port/tcp’.

3. Enter the following command to create an Oracle resource to monitor the database:

# scrgadm -a -j ehealth-db -g rgroup-1 -t SUNW.oracle_server \

-x Connect_string=$NH_USER/$NH_USER -x ORACLE_SID=$ORACLE_SID \

-x ORACLE_HOME=$ORACLE_HOME \

-x Alert_log_file=$ORACLE_HOME/admin/bdump/

$ORACLE_SID/alert_$ORACLE_SID.log \

-x Restart_type=RESOURCE_GROUP_RESTART \


When you create the Oracle resource, you may receive errors indicating that the ORACLE_HOME directory is not currently accessible on all of the nodes in the HA cluster. This is normal when using locally mounted, highly available file systems and can be ignored.



Note: When you enter this command, you set the password for the Oracle database. If you plan to change this password at a future date, use the following command:

scrgadm -c -j ehealth-db -x Connect_string=new_user/new_password

4. Enter the following command to create a Listener resource to monitor the Oracle Listener:

# scrgadm -a -j ehealth-db-listener -g rgroup-1 \

-t SUNW.oracle_listener \

-x LISTENER_NAME=listener \

-x ORACLE_HOME=$ORACLE_HOME \


5. Enter the following command to bring the new resources online:

# scswitch -Z -g rgroup-1

The clusterware resources are created and added to the resource group.

Step 9: Create and Add eHealth Resources to the Resource Group

The eHealth resources must be created and added to the resource group so that when failover occurs, all eHealth processes, not just individual applications, start on the backup system. Adding resources to a group also establishes a dependency relationship between the associated eHealth processes that help determine how they are managed by the clusterware.

The resources in this procedure are the default eHealth resources. For example, you can choose not to add the WANSync resource to the eHealth resource group at this time but at a later date.

To add eHealth resources to the group

1. As root, specify the time zone environment variable (TZ) in your nethealthrc file.

Example:

Bourne: TZ=”US/Eastern”; export TZ

C: setenv TZ “US/Eastern”

Korn: TZ=”US/Eastern” export TZ

Note: Do not omit this step or errors will occur.



2. Enter the following commands to add the eHealth resources to the resource group:

> su root

# scrgadm -a -j ehealth-license -g rgroup-1 -t CA.ehealthLicense \

-y Network_resources_used=ehealth-host \

-x NH_HOME=$NH_HOME \

-x NH_USER=$NH_USER \


# scrgadm -a -j ehealth -g rgroup-1 -t CA.ehealth \




-y resource_dependencies=ehealth-db,ehealth-db-listener,

ehealth-license,ehealth-storage

# scrgadm -a -j ehealth-rptctr -g rgroup-1 -t CA.ehealthRpt \





ehealth-storage,ehealth-web

# scrgadm -a -j ehealth-servlet -g rgroup-1 -t CA.ehealthServlet \





ehealth-storage

# scrgadm -a -j ehealth-trap -g rgroup-1 -t CA.ehealthTrap \



-x Trap_dir=/mountpoint/trapx \

-y resource_dependencies=ehealth-storage,ehealth-license

Step 10: Start the High Availability Service

To enable the HA cluster and bring it online, start the fully-configured HA service by enabling the resource group and all resources in the group.

To start the HA service

Enter the following command to bring the resource group online:

> su root

scswitch -Z -g rgroup-1

Note: Running the scswitch command with the -Z argument brings the resource group online and enables every resource within the group, even if those resources were previously disabled using the -n argument.


How to Configure Multiple Instances of eHealth

Using the -z Argument

Use the following guidelines when using the -z (lowercase) argument:

– -z lets you to choose on which machine the resource group is brought online.

– -z starts only those resources currently enabled within the group, ignoring the resources that have been disabled using the -n argument.

How to Configure Multiple Instances of eHealth If you have more than two nodes in your HA cluster, you can configure more than one instance eHealth, creating a Distributed eHealth Cluster that is highly available. For example, if you have three nodes, you can run two instances of eHealth on two different primary nodes while one backup node sits in reserve, ready to take over active duties if either primary machine fails.

Note: When multiple instances of eHealth are running in the same HA cluster, you must confirm that no two instances of eHealth are active on the same node at the same time. eHealth depends on certain global environment settings, therefore running multiple eHealth systems simultaneously on the same node will cause conflicts and crashes.

Only one eHealth resource group should be run at a time on a single machine. You can control which resource groups (and therefore which instances of eHealth) are run on a particular system. You do this by configuring a strong negative affinity between each resource group using the RG_affinities property.

For example, if you have three eHealth resource groups: rgroup-1, rgroup-2, and rgroup-3, the correct affinities could be configured using the following commands:

# scrgadm -c -g rgroup-1 -y RG_affinities=--rgroup-2,--rgroup-3



Following the above example:

■ rgroup-2 and rgroup-3 can never be online at the same time on the same machine as rgroup-1.




High Availability Cluster Management Tasks

High Availability Cluster Management Tasks Use the following tasks to help manage eHealth in an HA Sun cluster.

Install a Patch, Service Pack, or Upgrade

After eHealth is fully integrated into the HA cluster you may have to run a patch or service pack installation program to update the eHealth and Oracle services. You must tell the clusterware that the system on which you want to install the patch will be shut down and the monitoring stopped. If you do not do this, the clusterware will continue to monitor the system, misread system and application failures, and induce failover. You use clusterware commands to achieve this functionality.

Note: An alternate procedure to installing a patch or service pack is to run the nhHaSetup -disable command to avoid resrouces being monitored during the upgrades and updates.

To install a patch or service pack

1. Enter the following commands, including the name of the service as appropriate:

> su root

scswitch -n -j ehealthLicense

scswitch -n -j ehealth

scswitch -n -j ehealthRpt

scswitch -n -j ehealthServlet

scswitch -n -j ehealthTrap

scswitch -n -j ehealthdb

scswitch -n -j ehbdlistener

All eHealth services are brought offline.

2. Start the eHealth server manually. Enter the following command:

nhServer start

3. Start all other eHealth services.

4. Run the patch or service pack installer as you normally would on a stand-alone system.

Note: After the patch or service patch completes installing, confirm that eHealth and all applications are working properly before restarting the HA service.



5. Shut down any services started by the patch or service pack installation program.

6. Start the HA service again. See Step 10: Start the High Availability Service (see page 30).

The patch or service pack is installed.

Upgrade the Sun Cluster High Availability Package

If you have already configured eHealth for HA with Sun Cluster software, this release of eHealth upgrades CAehealthHA, the Sun Cluster Solaris package that allows eHealth to be integrated and managed by the Sun clusterware. The new version of this package includes new command attributes, and clusterware and eHealth resources.

Note: You must upgrade to eHealth r6.1 before you can upgrade CAehealthHA.

To upgrade the CAehealthHA cluster package

1. Enter the following command to disable resource monitoring, including the name of the service as appropriate, for example:

> su root

# scswitch -M -n -j ehealth-license

# scswitch -M -n -j ehealth

# scswitch -M -n -j ehealth-web

# scswitch -M -n -j ehealth-servlet

# scswitch -M -n -j ehealth-trap

2. On the machine on which the resources are currently online, enter the following command to access and upgrade the Solaris package:

> su root

# cd $NH_HOME/sys

# pkgadd -d CAehealthHA.pkg

3. Re-enable monitoring for each resource that you disabled by entering the following command, including the name of the service as appropriate:

> su root

# scswitch -M -e -j ehealth-license

# scswitch -M -e -j ehealth

# scswitch -M -e -j ehealth-web

# scswitch -M -e -j ehealth-servlet

# scswitch -M -e -j ehealth-trap

Note: If you have multiple instances of eHealth installed in your HA environment, upgrade the Solaris package on each system.



Load the Database after High Availability Configuration

After eHealth is fully integrated into the HA cluster you may need to load a saved eHealth database. Before you do this, the clusterware on the system on which you want to load the database must be shut down and the monitoring stopped. If you do not do this, the clusterware will continue to monitor the system, misread system and application failures, and induce failover. You use clusterware commands to achieve this functionality.

To load the database

1. Shut down eHealth. As root, enter the following command, including the name of the service as appropriate. For example:

scswitch -n -j ehealth-License

scswitch -n -j ehealth

scswitch -n -j ehealth-Rpt

scswitch -n -j ehealth-Servlet

scswitch -n -j ehealth-Trap


2. Load the saved database.

Note: For more information about the eHealth database and reasons for using the data restore feature, see the eHealth Administration Guide. When you start or stop servers in any eHealth database procedures on machines in an HA cluster environment, always shut down the clusterware and stop the monitoring first.


nhServer start



The database is loaded and HA service is started.

Initiate a Planned Failover

When you initiate a planned failover, you manually migrate eHealth resources from one system to another.




To manually failover

Enter the following command from the CLI of the target system:

scswitch -z -g rgroup-1 -h hostToMigrateTo

hostToMigrateTo

Represents the system to which you want to migrate the eHealth resources.

How to Enable or Disable SSL for Sun Cluster

When you want to enable or disable SSL after eHealth and HA are already implemented and you decide to change HTTP/S modes, you must use the following process:

1. Disable all resources for Sun Cluster using the following:

scswitch -n -j each resource name

2. Run nhWebProtocol.

3. Delete and recreate your web resources.

4. Re-enable all the resources.

Chapter 3: Veritas High Availability Configuration


Critical Resource Functionality (see page 37) How to Configure Veritas with High Availability (see page 38) Multiple Instances of eHealth in the HA Cluster (see page 46) High Availability Cluster Management Tasks (see page 48)

Critical Resource Functionality In a Veritas HA cluster, resources have two default classifications:

■ Critical resources

■ Non-critical resources

Critical Resources

A critical resource causes a failover when it experiences a failure. Failures can cause outtages in eHealth accessability, polling, and other functionality. The following resources are considered critical:

■ eHealth

■ Oracle

■ Oracle listener

■ IP address

■ Mount

■ Disk Group (and Volume Manager, if applicable)

■ License Manager

Chapter 3: Veritas High Availability Configuration 37

How to Configure Veritas with High Availability

Non-critical Resources

A resource that is not designated as critical but belongs to a service group will come online or offline whenever the service group is taken online or offline. Though these resources may be necessary to provide the service, they will not fail over if they experience a fault. Instead, the resource appears as having experienced a fault and will need maintenance. The following resources are considered non-critical:

■ TrapEXPLODER

■ Report Center (if installed)

■ Apache

■ Servlet (Tomcat)

■ WANSync (if installed)

Note: The critical and non-critical resource functionality can be customized in the VCS Java Console. You can also use the CLI to specify a resource as critical by entering the following command: hares –modify resource Critical 1

How to Configure Veritas with High Availability The following list outlines the process you must follow to configure eHealth for HA with VCS:

1. Install and configure VCS software.

2. Install eHealth.

3. Configure eHealth to run in the HA cluster.

4. Create a resource group.

5. (Optional) Install CA XOsoft Replication software.

6. Create and install eHealth resource types.

7. Start the HA service.



Step 1: Install and Configure VCS Software

For eHealth to be configured for HA with Veritas, VCS software version 5.0 must first be installed and configured, creating an HA cluster by which eHealth can be managed. When planning your HA cluster with Veritas, follow these guidelines:

■ Configure the primary system to use the shared (virtual) hostname when prompted. The system maps the IP addresses to hostnames.

■ Configure IP Multipath Networking in the HA cluster so that shared hostnames can be used.

■ Use the procedures in the Veritas Cluster Server Installation Guide 5.0 for Solaris to do the following:

– Install the VCS software on all systems that will make up the HA Cluster.

– Configure the HA cluster and setup disk sets for locally mounted, highly available storage.

■ Confirm that you have the following installed on each system:

– Veritas packages VRTScscw and VRTScsocw

– Oracle agent and Oracle Listener agent

During installation, a ClusterService group is created, along with all the resources needed to support the HA Cluster Manager Console and HA Cluster Manager Java User Interface. You will be prompted to supply a shared IP address. This address will be used to access the ClusterService group only.

Note: Supply a unique shared IP address for each service group created. This address will be used whenever you access client applications in the eHealth cluster, such as OneClick for eHealth and Live Health.



Step 2: Install eHealth

Install eHealth r6.1 following the procedures in the eHealth Installation Guide and these guidelines:

■ Mount the shared storage disk manually and install eHealth and Oracle on the device. Use the following command to mount the storage disk:

mount devicetomount mount_point

devicetomount

Represents the shared storage disk device.

mount_point

Represents the system on which you are installing eHealth.

For example, if you mount the device on /shared, eHealth may be installed under /shared/ehealth61, Oracle under /shared/oracle, and the Oracle database under /shared/oracleData.

■ During installation, the Spool and Report directories are placed on the shared storage. You can move these to a separate, dedicated 73 GB disk.

■ If you are configuring eHealth for both HA and DR, to save time you can install the WANSync software concurrent to eHealth. Do not create the replication scenarios until after eHealth has been configured for HA. For complete steps, see Configuring eHealth for Disaster Recovery in a High Availability Cluster (Solaris Only).

■ Before you install the eHealth license file, license.dat, configure the file to include all host IDs including all possible backup systems by doing the following:

a. Send all host IDs to the product licensing team. You will receive back one license file.

b. Copy and paste the license keys into the license.dat file located on the mountpoint/lmgr directory. It will reside on the disk to be accessed by all the eHealth systems in the HA cluster.

Note: If you are installing multiple instances of eHealth in the HA cluster, see Multiple Instances of eHealth in the HA Cluster (see page 46) for guidelines.

Step 3: Configure eHealth to Run in the High Availability Cluster

After you install eHealth, you must run the nhHaSetup command to configure eHealth to be started and managed by VCS software.



To configure eHealth to run in the HA cluster

1. (Optional) If your system is already running as part of a Distributed eHealth Cluster, you must complete this step before you run nhHaSetup. From the primary system, run the following commands:

nhServer start


physicalHost

Represents the name of the primary HA system prior to being HA enabled.

sharedhost


This prepares the Distributed eHealth Cluster to allow the eHealth cluster members to communicate with the primary system using the shared hostname.

Note: If your system is already running as part of a Distributed eHealth Cluster and you have run nhHaSetup, you must enter the following command to establish communication between all eHealth cluster members, from a system in the Distributed eHealth Cluster that nhHaSetup was not run on:

nhModifyClusterMember -all -name physicalHost -newName sharedhost

You will receive an error from the Distributed eHealth Cluster member configured for HA because the other systems cannot communicate with it until the command finishes running.

2. Enable and verify the shared hostname:





ifconfig -a


3. Enter the following command to configure eHealth to support shared hostnames:

> $NH_HOME/bin/nhHaSetup -hostname sharedhost

sharedhost

Specifies the name of the shared host where eHealth is running.



4. Enter the following command to configure TrapEXPLODER to run in the HA cluster by copying /opt/trapx to a shared file system:

> su root

# mv /opt/trapx /mountpoint

mountpoint

Represents a shared file system mountpoint.

5. Enter the following command to remove the startup files preventing eHealth from starting automatically at boot time:

> su root

# rm /etc/rc*.d/[SK]*httpd

# rm /etc/rc*.d/[SK]*nethealth

# rm /etc/rc*.d/[SK]*nhreportcenter

# rm /etc/rc*.d/[SK]*trapexploder

# rm /etc/rc*.d/[SK]*wansync (if installed)

Step 4: Create a Service Group

Veritas provides basic monitoring for the Oracle resources. Complete this step on each node in the HA cluster to create a service group that contains the following storage-related resources:

■ Oracle

■ Oracle Listener

■ IP Address

■ Network interface card (NIC)

■ Disk

Depending on the type of HA storage devices in your environment, additional storage resources may be created. For example, DiskGroup and Volume Manager.

To create and configure the resource group

1. Open a CLI and start the eHealth database by entering the following command:

$NH_HOME/bin/nhStartDb

2. Install the VCS Enterprise Oracle Agent on each node in the HA cluster if you have not already done so by entering the following command:

> su root pkgadd -d VRTSvcsor



3. Enter the following command to start the VCS Orac le Configuration Wizard. You will use this wizard to configure Oracle:

> hawizard oracle

Note: If the VCS Oracle Configuration Wizard does not run, confirm that the VRTScscw and VRTScsocw packages are installed on each system. Also, confirm that the 'had' daemon has started and the Oracle server and Listener are running.

4. Create a service group in the VCS Oracle Configuration Wizard. This group will contain all the resources that are required to fail over eHealth. Enter a unique group name, for example ehealth-sg-1. If you are configuring multiple instances of eHealth to run in the HA cluster, each must reside in a separate service group with a unique name. Also, each resource type you create must be given a unique name.

Note: You can add these or additional resources to the group at a later time.

The wizard discovers the running Oracle instance and Oracle Listener.

5. In the Instance Selection menu, select the Oracle instance and Oracle Listener as items to be configured.

6. Click next.

7. Accept the defaults for the remaining menus, then close the wizard.

The service group with Oracle, Oracle listener, IP, NIC, and Mount resources is created and online.

Note: To enable detail monitoring, follow the instructions in the Veritas High Availability Agent for Oracle Installation and Configuration Guide.

Step 5: (Optional) Install CA XOsoft Replication Software

Complete this step only if you want your HA cluster to be configured for DR. For the complete procedure, see Configuring eHealth for Disaster Recovery in a High Availability Cluster (Solaris Only).

Step 6: Create and Install eHealth Resource Types

The eHealth resource types must be created and added to the resource group so that the clusterware monitosr eHealth on every node in the HA cluster and so that when failover occurs, all eHealth processes, not just individual applications, start on the backup system. Adding resource types to a group also establishes a dependency relationship between the associated eHealth processes that help determine how they are managed by the clusterware.



To install the resource types

1. As root, specify the time zone environment variable (TZ) in your nethealthrc file.

Examples:

Bourne: TZ=”US/Eastern”; export TZ

C: setenv TZ “US/Eastern”

Korn: TZ=”US/Eastern” export TZ

Note: Do not omit this step or errors will occur.

2. As the eHealth user, stop the eHealth processes by entering the following commands:

$NH_HOME/bin/nhServer stop

$NH_HOME/bin/nhHttpd stop

$NH_HOME/bin/nhLmgr stop

If Report Center is installed, run:

$NH_HOME/bin/nhReportCenter stop

If WANSync is installed, run:

/etc/init.d/WANSync stop

The eHealth processes are stopped.

3. As root, enter the following command to stop the TrapEXPLODER process:

/etc/init.d/trapexploder stop

TrapEXPLODER is stopped.

4. Confirm that the service group created in Step 4: Create a Service Group (see page 42) appears in the VCS Java Console.

5. On the machine on which the storage service group is currently online, enter the following commands to access the Veritas Solaris package:

> su root

# cd $NH_HOME/sys

# pkgadd -d CAehealthVcsHA.pkg

The resources are created.

Note: A Report Center resource is created by default. If you are not installing Report Center, you can remove or disable this resource.



6. Change to the next machine on which you want to install the resources by entering the following commands (to help speed this process, take the Oracle and Oracle Listener resources offline):

> su root

# hagrp -switch ehealth-sg-1 -to system_name

ehealth-sg-1

Represents the unique name given to the service group.

system_name

Represents the name of the machine on which you want to install the resource types.

7. Log on to the next machine and repeat 5 and 6. Continue until the resource types have been installed on every node in the HA cluster.

8. Change back to the system on which you first installed the eHealth resources.

9. If you have not done so, source the nethealthrc eHealth file as root for your shell environment using one of the following commands:

Bourne: . ./nethealthrc.sh

C: source nethealthrc.csh

Korn: . ./nethealthrc.ksh

10. Open and make writable the VCS configuration file (main.cf) by entering the following command:

# haconf -makerw

11. Verify that /opt/VRTSvcs/bin is in the PATH environment and launch the nhAddVcsResources script to add eHealth resources by entering the following commands:

# cd /opt/CAehealthVcsHA/bin

# ./nhAddVcsResources

Note: To make WANSync highly available, execute the nhDrConfigScenario script before the nhAddVcsResources script. If you already ran nhAddVcsResources, you can run it again to configure the WANSync resource.

The following occurs:

– You are prompted to enter the name of the storage service group and some of the resources you have created. For example, the Oracle database resource, the IP resource, and the Mount resource.

– The script validates the entries. As each resource is added, output appears showing the status of the installation.


Multiple Instances of eHealth in the HA Cluster

– The script terminates if it encounters a problem running the VCS commands. A message alerts you to reasons for the termination. After you resolve the issues, re-run the script.

– For the question about the trap exploder you must enter the full path of the trap exploder binary file: /shared_mountpoint/trapx/bin.

12. Save the configuration by entering the following command:

# haconf -dump -makero

Note: If the VCS restarts and you have not saved the configuration, repeat Step 11.

Step 7: Start the High Availability Service

To enable the HA cluster and bring it online, start the fully-configured HA service by enabling the service group and all resources in the group.

To start the HA service

Enter the following command on any system in the HA cluster to bring the service group online:

> su root

# hagrp -online ehealth-sg-1

This brings up the service group on the system that has priority in the SystemList attribute.

Note: To bring a service group online on a particular HA cluster node, run the following command:

# hagrp -online ehealth-sg-1 -sys system_name

Multiple Instances of eHealth in the HA Cluster If you have more than two nodes in your HA cluster, you can configure more than one instance eHealth, creating a Distributed eHealth Cluster that is highly available. For example, if you have three nodes, you can run two instances of eHealth on two different primary nodes while one backup node sits in reserve, ready to take over active duties if either primary machine fails.


Multiple Instances of eHealth in the HA Cluster

Pre-Online Trigger-Only one eHealth service group should be run at a time on a single machine. Veritas uses a pre-online trigger feature to control which resource groups (and therefore, which instances of eHealth) can be running simultaneously on a single system. The trigger is a script called by the Veritas engine before it attempts to bring a service group online. The script checks whether any eHealth processes are already running. If so, it prevents a second service group from coming online.

Note: eHealth depends on certain global environment settings, therefore running multiple eHealth systems simultaneously on the same node causes conflicts and crashes.

How to Add Multiple Instances of eHealth

When you have more than one instance of eHealth in an HA cluster you create a service group, then create and add resources to it. The following process outlines the steps needed to help ensure your environment is correctly configured for eHealth on two or more HA nodes:

1. Confirm that the first service group has been created and all resources are created and installed.

2. On the second node, install eHealth and Oracle. See Step 2: Install eHealth (see page 40).

Note: The VCS Oracle Configuration Wizard uses the Oracle SID and Listener name to generate the resource names for Oracle and Oracle Listener. Therefore, when installing an additional eHealth instance you must specify a SID that is different from that of the first eHealth instance.

3. Change the Oracle Listener name (see procedure below).

4. Configure eHealth to run in a HA cluster. See Step 3: Configure eHealth to Run in the High Availability Cluster (see page 40).

5. Set the DISPLAY environment variable to a running X Server, if necessary, by entering the following command:

setenv DISPLAY hostname:0.0

6. Use the VCS Oracle Configuration Wizard to create the second service group and create and add the Oracle and storage-related resources. See Step 4: Create a Service Group (see page 42).

7. Use the CLI to create and install the eHealth resources and start the HA service. See Step 6: Create and Install eHealth Resource Types (see page 43) and Step 7: Start the High Availability Service (see page 46).



Use the following guidelines:

■ All service groups created with the Veritas software must have unique names. All resource names must be uniquely named as well, whether in the same service group or a different group. Resource names are based on the name of the group they are in. As long as the service group names are unique, the resource names will be unique.

■ If you get a Disk Resource Type error when using the wizard, this message can be ignored.

Change the Oracle Listener Name

After eHealth and Oracle are installed, you must alter the listener name; you cannot have two instances of the Oracle Listener with the same name. This can be done anytime after Oracle is installed, but before the group is brought online and the HA service is started.

To change the Oracle Listener name

1. Start the Oracle database if is not running by entering the following command:

nhStartDb

2. Stop the current Oracle Listener process by entering the following command:

nhConfigDbNet -stopListener

3. Alter the Oracle Listener name by entering the following command:

nhConfigDbNet -addListener -listenerName

listenerName

Represents the new name for Oracle Listener.

4. Set the NH_ORA_LISTENER_NAME to the listenerName in the following files:

nethealthrc.sh.usr

nethealthrc.csh.usr

High Availability Cluster Management Tasks Use the following tasks to help manage eHealth in an HA cluster. For additional information about Veritas HA cluster commands, see Clusterware Commands Used to Manage eHealth and the VCS software documentation.



Install a Patch or Service Pack

After eHealth is fully integrated into the HA cluster you may have to run a patch or service pack installation program to update the eHealth and Oracle services. To do this you tell the clusterware that the system on which you want to install the patch is going to be shut down and the monitoring stopped. If you do not do this, the clusterware will continue to monitor the system, misread system and application failures, and induce failover.


1. Shut down eHealth. Enter the following command:

> su root

hares -offline ehealth-license -sys system_name

hares -offline ehealth -sys system_name

hares -offline ehealth-rptctr -sys system_name

hares -offline ehealth-servlet -sys system_name

hares -offline ehealth-trap -sys system_name

hares -offline ehealth-wansync -sys system_name

hares -offline ehealth-web -sys system_name

hares -offline ehealth-lmgr -sys system_name

hares -offline ehealth-db -sys system_name

hares -offline ehealth-db-listener -sys system_name

system_name

Represents the name of the system on which the resource is currently running.


2. Stop the Oracle resources. Enter the following command:

nhStopDb


nhServer start


5. Run the patch or service pack installer as you normally would on a stand-alone system.

Note: After the patch or service patch completes installing, confirm that eHealth and all applications are working properly before restarting the HA service.

6. Shut down any services started by the patch or service pack installation program.


The patch or service pack is installed.



Load the Database after High Availability Configuration

After eHealth is fully integrated into the HA cluster you may need to load a saved eHealth database. Before you load the database, the clusterware on the system on which you want to load the database must be shut down and the monitoring stopped. If you do not do this, the clusterware will continue to monitor the system, misread system and application failures, and induce failover. You use clusterware commands to achieve this functionality.

To load the database

1. Shut down eHealth. Enter the following command:

> su root

hares -offline ehealth-license -sys system_name

hares -offline ehealth -sys system_name

hares -offline ehealth-rptctr -sys system_name

hares -offline ehealth-servlet -sys system_name

hares -offline ehealth-trap -sys system_name

hares -offline ehealth-wansync -sys system_name

hares -offline ehealth-web -sys system_name

hares -offline ehealth-lmgr -sys system_name

system_name

Represents the name of the system on which the resource is currently running.


2. Load the saved database.

Note: For more information about the eHealth database and reasons for using the data restore feature, see the eHealth Administration Guide. When you start or stop servers in any eHealth database procedures on machines in an HA cluster environment, always shut down the clusterware and stop the monitoring.


nhServer start



The database is loaded and HA service is started.




Initiate a Planned Failover

You use the same command whenever you initiate a planned failover, as they amount to the same action in the HA cluster: manually migrating eHealth resources from one system to another.

To manually failover

Enter the following command from the CLI of the target system:

hagrp –switch ehealth-sg-1 –to hostToMigrateTo

hostToMigrateTo

Represents the system to which you want to migrate the eHealth resources.

How to Enable or Disable SSL for Veritas

When you want to enable or disable SSL after eHealth and HA are already implemented and you decide to change HTTP/S modes, you must use the following process:

1. Run the following command:

nhAddVcsResources -replace

2. Enable the HA system first if you are enabling SSL.

3. Enable or disable SSL by running nhWebProtocol –mode https (or http for non-SSL) –hostname <shared hostname>.

4. Re-run /opt/CAehealthVcsHA/bin/nhAddVcsResources.

Chapter 4: Disaster Recovery


How eHealth Works with Disaster Recovery (see page 53) System Guidelines (see page 54) How CA XOsoft Replication Software Works (see page 55) Failover in a Disaster Recovery Environment (see page 56) Downtime and Data Loss (see page 56) Configure eHealth for Disaster Recovery (see page 56) Start the Failover Process (see page 67) How to Upgrade Your Disaster Recovery Environment (see page 71) Disaster Recovery Management Tasks (see page 72)

How eHealth Works with Disaster Recovery You can configure your eHealth environment to integrate with the DR replication software CA XOsoft Replication. This chapter assumes that you are installing eHealth for the first time. If you have already installed eHealth, you must upgrade to eHealth r6.1 and then configure your system for DR.

The replication software copies all eHealth, Oracle, and eHealth database files over the network from the active eHealth system to a standby system, often in another physical location. When you update a file or directory on the active system, the changes are automatically replicated to the standby system. When there is a critical failure (data file corruption) or a disaster (hurricane, earthquake) on the active system, you manually failover to the standby system, which then becomes the active system.

Your eHealth systems can be stand-alone or configured in a Distributed eHealth Cluster.

Note: For information about how to install and configure the CA XOsoft Replication software, see the WANSync User Guide, the Disaster Recovery and High Availability with WANSync Architectural Overview guide, and the WANSync Solaris Server Operations Guide available from the CA XOsoft Replication product website at www.xosoft.com. You will need these guides to install the software and help create replication scenarios.

Chapter 4: Disaster Recovery 53

System Guidelines

System Guidelines ■ For replication, an eHealth system that is polling and regularizing data for

50K elements while also running reports will require network bandwidth of 16 Mbits/sec.

■ While there is no specific bandwidth requirement for the initial synchronization, the time it takes to complete will depend on the volume of data to be transmitted and the available bandwidth.

■ As a best practice, when you configure your eHealth environment for DR, keep the spool and report directories on a separate, dedicated disk.

■ (Solaris only) Confirm that your /opt directory is at least 1GB in size. WANSync software stores log files in this directory.

■ Confirm that the user ID (UID) and group identifier (GID) are the same on the primary and standby systems.

■ Synchronize the time on your primary and standby systems using time synchronization software such as NTP or XNTP. Time sync software should be installed with your operating system.

■ You can install WANSync on Windows 2003 servers and Solaris 2.9 and 2.10 systems as follows:

– Windows: Install eHealth on both the active system and the standby system.

– Solaris: Install eHealth on only the active system.

Note: DR scenarios cannot be configured from an active Windows system to a standby UNIX system, or conversely from an active UNIX system to a standby Windows system.

■ Add the shared hostname and IP to the Windows\system32\drivers\etc\hosts to avoid getting error messages about no web server connection when using the Business Service COnsole (BSC) with DR enabled.

System Terminology

The following describes common terms for DR systems:

■ Primary-The machine normally used as the active eHealth machine. Also known as the master machine.

■ Standby-An idle machine that takes over for the primary machine in the event of a disaster. Also known as the replica machine.

■ Active-The machine currently running eHealth. May be the primary machine (normal case) or the standby machine (in the event of a disaster).


How CA XOsoft Replication Software Works

How CA XOsoft Replication Software Works The CA XOsoft Replication software continuously replicates eHealth files from the primary active system to a remote standby system. In the event of a disaster, the standby system is manually started to become active, with a current configuration and complete database.

The following WANSync components must be installed in your eHealth environment:

■ XOsoft Manager-XOsoft Manager is an interface you use to create replication scenarios, and monitor and manage the individual servers in the DR configuration. Install this software on a Windows system separate from your DR configuration but with an TCP/IP connection to the primary and standby systems.

■ XOsoft Engine-XOsoft Engine is software used to help support the administration of the DR environment. Install this software on the primary and standby DR systems.

Note: For information about where to find the latest version of XOsoft software, see the Release Notes.

Scenarios

Scenarios are definitions you create specifying which data to replicate. Scenarios contain information about which files and directories are to be copied, and where and when replication occurs. You create scenarios to synchronize and replicate the data that you want to copy from a primary system to the standby system. To create scenarios, use the WANSync Manager software after the WANSync Engine and eHealth have been installed and configured.

Replication

Replication is the copying of data from the primary (active) system to the standby system and is the basis of the eHealth DR service. This replication is continuous and helps ensure the standby system remains current. Any changes made on the primary system are automatically propagated to the standby system.


Failover in a Disaster Recovery Environment

Failover in a Disaster Recovery Environment Failover in the DR environment is a manual process that involves starting eHealth on the standby system when the primary system goes down. After failover, the standby system becomes the active system and runs the eHealth services and collects data. When problems on the primary system have been resolved, another DR scenario can be created and replication can begin in the opposite direction: from the standby (active) system to the primary system.

Downtime and Data Loss Downtime and data loss in the event of a disaster cannot be avoided. When a disaster brings the primary system offline, the extent of downtime and data loss you experience depends on how quickly you start the eHealth server, the Oracle database, and all eHealth applications on the standby system. After the standby system becomes active, eHealth resumes normal service, polling devices and writing data to the database.

Configure eHealth for Disaster Recovery The procedures in this section must be followed precisely to successfully configure eHealth for DR on Windows and Solaris systems.

Before You Configure eHealth for DR

Before you install the WANSync software and eHealth, you must secure a shared (or virtual) IP address and shared hostname which you will use during the configuration process. After you complete the configuration, use the shared hostname and IP address whenever you access eHealth applications such as OneClick for eHealth and Report Center.

If the primary system goes down, you must enable the shared IP address on the standby system. This lets you continue to access eHealth applications. During failover, both the shared hostname and IP address may need to be changed depending on your system configurations.

Note: In most DR configurations you will use the same shared hostname on both primary and standby systems. Because of the standard physical distance between systems, they will usually have different IP addresses. The configurations described in these procedures assume this scenario.


Configure eHealth for Disaster Recovery

Windows Disaster Recovery Configuration

This procedure includes the tasks to configure eHealth for DR on Windows systems.

To configure disaster recovery on Windows

1. Install WANSync Manager and WANSync Engine software. Use the following guidelines:

– As a best practice, install the WANSync Manager software on a machine that will not have an eHealth server installed: the WANSync Manager software should be separate from eHealth so that in the event of a disaster you can access the WANSync user interface.

– Install the WANSync Engine software on the primary and standby systems.

– Do not create any DR scenarios until after eHealth is installed.

2. Add the shared IP address and subnet mask to the IP settings to configure the primary system to use the shared IP address:

a. On the primary system, open Control Panel. Double-click Network Connections, Local Area Connection Properties.

b. Select Internet Protocol (TCP/IP), Properties.

c. The Internet Protocol (TCP/IP) Properties dialog opens. Click Advanced.

The Advanced TCP/IP Settings dialog opens.

d. Click the IP Settings tab. Click Add and enter the shared IP address and subnet mask. Click add, and then OK, and close the remaining dialogs.

The primary system is configured to use the shared IP address.

3. Install eHealth r6.1. Use the following guidelines:

– Install eHealth on the primary and standby machines. It is important to install eHealth on both machines so that the necessary registries, services, and supported software is configured correctly.

– During installation, select the same options on both machines. For example, the eHealth, Oracle, and Oracle database directories must be located in the same directory on the standby system and the primary system.

– The eHealth Administrator must be the same on both machines.



– Before you install the eHealth license file, license.dat, configure the file to include all host IDs, including all possible standby systems, by doing the following:

a. Send all host IDs to the eHealth product licensing team. You will receive back one license file.

b. Copy and paste the license keys into the license.dat file located on the NH_HOME/Imgr directory.

Note: The Enter License Interface is no longer used to install eHealth licenses on your eHealth system.

– Modify the standby machine configuration:

a. On the Standby system, click Start, Control Panel; double-click Administrative Tools, Services.

The Services window opens.

b. Right-click the following services to stop them, then double-click each service and set the Startup type to manual:

– ehealth

– ehealth httpd

– ehealth tomcat

– ehealth report center

– The following Oracle services: OracleEHORA10TNSListener and OracleServiceehealth

– Microsoft Distributed Transaction Coordinator

– FLEXlm license server

– TrapEXPLODER

Note: For more information about installing and licensing eHealth, see the Installation Guide.



4. Configure eHealth to use a shared hostname:

a. (Optional) If your system is already running as part of a Distributed eHealth Cluster, you must complete this step before you run nhDrSetup. From the primary system, run the following command:


physicalHost

Represents the name of the primary system prior to being DR enabled.

sharedhost

Represents the shared hostname. This prepares the Distributed eHealth Cluster to allow the eHealth cluster members to communicate with the primary system using the shared hostname.

Note: If your system is already running as part of a Distributed eHealth Cluster and you have run nhDrSetup, you must enter the following command to establish communication between all eHealth cluster members, from a system in the Distributed eHealth Cluster that nhDrSetup was not run on: nhModifyClusterMember -all -name physicalHost -newName sharedhost You will receive an error from the Distributed eHealth Cluster member configured for DR because the other systems cannot communicate with it until the command finishes running. Ignore the message.

a. On the primary system, start the eHealth database (if not already started), and stop the eHealth server and Report Center.

b. Open a command prompt window and enter the following command:

nhDrSetup -hostname sharedhostname

sharedhostname

Represents your shared hostname.

Messages appear confirming when each eHealth interface has been changed to use the shared hostname.

a. Start the eHealth server and all applications.

b. (Optional) If your system will be part of a Distributed eHealth Cluster, run the nhJoinClusterMember command and use the shared hostname.

5. Create DR scenarios using the WANSync Manager software. For eHealth to be properly configured for DR, you must create three scenarios:

■ A principal scenario that synchronizes and continuously replicates all of the eHealth data.

■ A secondary scenario that synchronizes the libraries and executables.



■ A third scenario that synchronizes only the TrapEXPLODER file.

Use the following guidelines when creating scenarios:

■ To avoid confusion and help ensure a smooth failover process, directories must be replicated from the primary system to the same disk names and eHealth, eHealth database, and Oracle sub directory names on the standby system.

■ In WANSync Manager, choose Create New Scenario and Next. On the Select Scenario Type screen, under the Tasks on Replica section, choose None.

– Create the Principal Scenario-To create the principal scenario, in WANSync Manager do the following:

■ Select the top-level eHealth directory. Expand this directory and deselect the following directories to exclude them from this scenario:

$NH_HOME/bin

$NH_HOME/jre

$NH_HOME/lib

$NH_HOME/lmgr/bin

■ Select the top-level Oracle software directory. Expand this directory and deselect the following directories to exclude them from this scenario:

$ORACLE_HOME/bin

$ORACLE_HOME/ctx

$ORACLE_HOME/lib

■ Select the top-level Oracle database directory or directories: all the files in this directory (or directories) are included in this scenario and should be selected.

■ Select the following registry settings and all environment variables under the registry paths:

/HKEY_LOCAL_MACHINE/SOFTWARE/Apache SoftwareFoundation/ Tomcat Service Manager /HKEY_LOCAL_MACHINE/SOFTWARE/ConcordCommunications/eHealth /HKEY_LOCAL_MACHINE/SOFTWARE/FirstSense /HKEY_LOCAL_MACHINE/SOFTWARE/ORACLE /HKEY_LOCAL_MACHINE/SYSTEM/ControlSet001/Control/Session Manager/Environment /HKEY_LOCAL_MACHINE/SYSTEM/ControlSet002/Control/Session Manager/Environment /HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/Control/Session Manager/Environment



■ In the Host Replication Tree, select the primary system. In the Property pane under Replication, select Automatic Synchronization Type and choose Block Synchronization.

The primary scenario is created.

– Create the Secondary Scenario-To create the secondary scenario, in WANSync Manager do the following:

■ Select the top-level eHealth directory. Expand this directory and deselect all the directories excluding the following:

$NH_HOME/bin

$NH_HOME/jre

$NH_HOME/lib

$NH_HOME/lmgr/bin

■ Select the top-level Oracle database directory, or directories if more than one, and deselect all the directories excluding the following:

$ORACLE_HOME/bin

$ORACLE_HOME/ctx

$ORACLE_HOME/lib

■ Set the Replication properties for the primary system by doing the following:

– Set the Replication Mode property value to Scheduling.

– Set the Attach to Open Files on Run property value to Off.

The secondary scenario is created.

– Create the Third Scenario-To create the third scenario, in WANSync Manager do the following:

■ Select the C:\WINDOWS\system32\trapexploder.cf file.

■ Select the standby machine in the middle pane.

In the right pane, under Properties, turn on the following options:

– Keep deleted files during synchronization

– Keep deleted files during replication

All scenarios are created.

6. Use WANSync Manager to run the scenarios you created in Step 5 in the following order:

a. Run the secondary scenario: select Tools, Synchronize, then select File Synchronization. This sets the secondary scenario to synchronize the directories in the scenario.

b. Run the third scenario: select Tools, Synchronize, then select File Synchronization. This sets the third scenario to synchronize the file in the scenario.



c. Run the principal scenario: select Tools, Run, then select Block Synchronization. This sets the principal scenario to synchronize and continually replicate its contents to the standby system.

An initial synchronization and replication process begins. The duration of this process depends on the speed of the network between the primary and standby systems and the amount of data in your database.

Solaris Disaster Recovery Configuration

This procedure includes the tasks to configure eHealth for DR on Solaris systems.

To configure disaster recovery on Solaris

1. Install the WANSync Manager (on a separate Windows system) and WANSync Engine software. Use the following guidelines:

– Install the WANSync Manager software on a machine that will not have an eHealth server installed: the WANSync Manager software should be separate from eHealth so that in the event of a disaster you can access the WANSync user interface and begin failover.

– Install the WANSync Engine software on the primary and standby systems.

– Do not create any DR scenarios until after eHealth is installed.

2. Configure the primary system to use the shared hostname. Add the shared hostname and IP address to the /etc/hosts file on the primary system.

a. Log on the primary system as the root user and open a command line interface.

b. In the /etc/hosts file, add the shared IP address and hostname as the last entry in the file.

3. Create the following file: /etc/hostname.XXXX:n

XXXX

Represents the interface name.

n

Represents the number of IP addresses (usually n=1 because counting starts at 0 and :0 is not shown).

4. Add the shared hostname as the file contents. To determine the interface name, use the following command:

ls /etc/hostname*



5. Enable the shared hostname:

a. As root, in the command line interface, enter the following: ifconfig XXXX:n plumb


XXXX

Represents is the interface name.

n

Represents the number of IP addresses from Step 3.

ipaddress

Represents is the shared IP address.

a. Verify the shared hostname is properly set up by entering the following: ifconfig -a


6. Install eHealth r6.1. Use the following guidelines:

– Install eHealth on the primary system. You need only one instance of eHealth when configuring for DR on Solaris systems.

– The mount points on both the primary and standby systems must be named the same.

– Configure both the primary and standby systems to have the same kernel parameters.

– Before you install the eHealth license file, license.dat, configure the file to include all host IDs, including all possible standby systems, by doing the following:

a. Send all host IDs to the eHealth product licensing team. You will receive back one license file.

b. Copy and paste the license keys into the license.dat file located on the NH_HOME/Imgr directory.

Note: The Enter License Interface is no longer used to install eHealth licenses on your eHealth system.

– For more information about installing and licensing eHealth, see the eHealth Installation Guide.

7. Configure eHealth to use the shared hostname:

a. (Optional) If your system is already running as part of a Distributed eHealth Cluster, you must complete this step before you run nhDrSetup. From the primary system, run the following command:




physicalHost

Represents the name of the primary system prior to being DR enabled.

sharedhost

Represents the shared hostname. This prepares the Distributed eHealth Cluster to allow the eHealth cluster members to communicate with the primary system using the shared hostname.

Note: If your system is already running as part of a Distributed eHealth Cluster and you have run nhDrSetup, you must enter the following command to establish communication between all eHealth cluster members, from a system in the Distributed eHealth Cluster that nhDrSetup was not run on: nhModifyClusterMember -all -name physicalHost -newName sharedhost You will receive an error from the Distributed eHealth Cluster member configured for DR because the other systems cannot communicate with it until the command finishes running. Ignore the message.

a. On the primary system, start the eHealth database (if not already started), and stop the eHealth server and Report Center.

b. Enter the following command: nhDrSetup -hostname sharedhostname

sharedhostname


Messages appear confirming when each eHealth interface has been changed to use the shared hostname.

a. Start the eHealth server and all applications.



■ A secondary scenario that synchronizes the eHealth and Oracle libraries and executables.

■ A third scenario that synchronizes the /etc files only.




■ To avoid confusion and help ensure a smooth failover process, directories must be replicated from the primary system to the same disk names and eHealth, eHealth database, and Oracle sub directory names on the standby system.

■ In WANSync Manager, choose Create New Scenario and Next. On the Select Scenario Type screen, under the Tasks on Replica section, choose None.



$NH_HOME/bin

$NH_HOME/jre

$NH_HOME/lib

$NH_HOME/lmgr/bin


$ORACLE_HOME/bin

$ORACLE_HOME/ctx

$ORACLE_HOME/lib



The primary scenario is created.

– Create the Secondary Scenario. To create the secondary scenario, in WANSync Manager do the following:


$NH_HOME/bin

$NH_HOME/jre

$NH_HOME/lib

$NH_HOME/lmgr/bin

■ Select the contents of the following directory (all files should be selected):

/opt/trapx ()



■ Select the top-level Oracle database directory, or directories if more than one, and deselect all the directories, excluding the following:

$ORACLE_HOME/bin

$ORACLE_HOME/ctx

$ORACLE_HOME/lib

■ Set the Replication property for the primary system by doing the following:

– Set Replication Mode to Scheduling.

– Set the Attach to Open Files on Run to Off.


– Create the Third Scenario. To create the third scenario, in WANSync Manager do the following:

■ Expand the /etc/init.d directory. In the right pane, select the following files in the following order to add them to the scenario:

httpd httpd.sh nethealth nethealth.sh nhreportcenter trapexploder

■ Expand the /etc directory. In the right pane, select the following files in the following order to add them to the scenario:

/etc/nh.install.cfg /etc/trapexploder.cf

These configuration files are used to automatically start eHealth and Oracle.

■ Edit the scenario by deleting the /etc/init.d sub directory from the scenario list.


■ In the right pane, under Properties, turn on the following options:



Note: If you rerun the nhDrSetup -disable or nhDrSetup -hostname commands again for any reason, you must rerun this third scenario to synchronize the /etc files.


Start the Failover Process

9. Use WANSync Manager to run the scenarios you created in the following order:

■ Run the secondary scenario: select Tools, Synchronize, then select File Synchronization. This sets the secondary scenario to synchronize the directories in the scenario.

■ Run the third scenario: select Tools, Synchronize, then select File Synchronization. This sets the third scenario to synchronize the files in the scenario.

■ Run the principal scenario: select Tools, Run, then select Block Synchronization. This sets the principal scenario to synchronize and continually replicate its contents to the standby system.

An initial synchronization and replication process begins. The duration of this process depends on the speed of the network between the primary and standby systems and the amount of data in your database.

Start the Failover Process When your primary system goes offline because of a critical failure or disaster, perform the tasks in this section to manually bring the standby system online.

Note: If your primary system is part of an HA cluster, see Appendix A for the procedure to manually bring the standby system online.

As a best practice, test the manual failover procedure before a disaster occurs. This helps confirm the following:

■ The eHealth servers and the database start without a problem.

■ The replication scenarios are complete and all the necessary files are being copied. In WANSync Manager, you can turn on synchronization and replication scenario reports for the primary system. These reports list all replicated files.

■ You can log on to the eHealth clients after failover.



To start failover on a Windows system

1. Stop the replication scenario on WANSync Manager, or confirm that it has been stopped on the primary and standby systems.

2. Stop the replication scenario on the primary and standby systems as follows:

a. Open Control Panel. In Administration Tools, open the Services window.

b. Right-click the XOsoft Engine service and stop it.

c. Double-click the XOsoft Engine service and set the Startup type to manual.

d. Delete the config_25000 subdirectory from both the primary and standby systems or the replication will start as soon as the system is restarted.

3. If the primary system is functional, stop the eHealth servers and applications including: FLEXlm, Oracle, Report Center, TrapEXPLODER, and the web server.

4. If the primary system is functional, do the following:

a. Remove the shared IP address and subnet mask from the IP settings.

b. In the system Services window, right-click the following services to stop them, then double-click each service and set the Startup type to manual:

■ ehealth

■ ehealth httpd

■ ehealth tomcat

■ ehealth report center (if installed)

■ The following Oracle services: OracleEHORA10TNSListener and OracleServiceehealth

■ Microsoft Distributed Transaction Coordinator

■ FLEXlm license server

■ TrapEXPLODER

Note: If the primary system is not functional, you must still perform this step after the system has been repaired but before it becomes the active system.



5. On the standby system, add the shared IP address and subnet mask to the IP settings to configure the system to use the shared IP address:

a. Open Control Panel. Double-click Network Connections, Local Area Connection Properties.

b. Select Internet Protocol (TCP/IP), Properties.

The Internet Protocol (TCP/IP) Properties dialog opens.

c. Click Advanced.

The Advanced TCP/IP Settings dialog opens.

d. Click the IP Settings tab. Click Add and enter the shared IP address and subnet mask. Click add, and then OK, and close the remaining dialogs.

The Standby system is configured to use the shared IP address.

6. On the standby system, start the eHealth database.

7. Update the Domain Name System (DNS) of the shared hostname to the correct IP address, if it is in a different subnet than the primary system.

8. Enter the following command on the standby system to configure it to run eHealth:

nhDrSetup -configureStandby

9. On the standby system, start the eHealth servers, the web server, and the FLEXlm and TrapEXPLODER applications.

10. On the standby system, set all services Startup type to Automatic so they will restart.

The standby system is now online and failover is complete.

To start failover on a Solaris system

1. On the standby system, create a symbolic link from /opt/eHealth to the location where eHealth has been replicated to by entering the following command:

ln -s $NH_HOME /opt/eHealth

2. If the primary system is functional, stop the eHealth servers and applications including: FLEXlm, Oracle, Report Center, TrapEXPLODER, and the web server.

3. Stop the replication scenario on WANSync Manager, or confirm that it has been stopped. To confirm the scenario has stopped, enter the following command on the primary system, if it is functional, and the standby system:




4. If the primary system is functional, remove the shared hostname and IP address entries from the /etc/hosts file and remove the /etc/hostname.XXXX:n file.

5. If the primary system is functional, remove the symbolic link from the following files in the /etc/rc0.d, /etc/rc1.d, /etc/rc2.d, /etc/rc3.d, and /etc/rcS.d directories:

■ nethealth

■ nhreportcenter

■ trapexploder

■ httpd

Note: If the primary system is not functional, you must still perform Steps 4 and 5 after the system has been repaired but before it becomes the active system.

6. On the primary system, disable the shared hostname, as root, by entering the following commands:

ifconfig XXXX:n ipaddress down

ifconfig XXXX:n unplumb

XXXX


n

Represents the number of IP addresses (usually n=1 because counting starts at 0 and :0 is not shown).

ipaddress

Represents the shared IP address.

Verify the shared hostname is disabled by entering the following:

ifconfig -a

Confirm that the shared hostname is not in the output.

7. On the standby system, log on as the root user. In the /etc/hosts file, add the shared IP address and hostname.

8. On the standby system, create the following file:

/etc/hostname.XXXX:n

Add the shared hostname as the file contents. To determine the interface name, enter the following command:

ls /etc/hostname*


How to Upgrade Your Disaster Recovery Environment

9. On the standby system, enable and verify the shared hostname:





ifconfig -a


10. On the standby system, start the eHealth database.

11. Update the Domain Name System (DNS) of the shared hostname to the correct IP address, if it is in a different subnet than the primary system.

12. On the standby system, enter the following command to configure the system to run eHealth:


13. On the standby system, start the eHealth servers, the web server, and the FLEXlm and TrapEXPLODER applications.

How to Upgrade Your Disaster Recovery Environment When you upgrade from eHealth r6.0 service packs to eHealth r6.1, the WANSync scenario should be stopped before the upgrade or activation(if the system is part of an eHealth cluster).

To upgrade DR systems to eHealth r6.1, follow this process:

1. Stop the WANSync scenario.

2. Stop the WANSync engine on both the primary and standby systems.

3. Delete the config_2500 subdirectory where WANSync was installed.

4. Upgrade the primary system by making certain the primary system has been upgraded and activated properly.

5. If your Disaster Recovery systems are on Windows servers, perform the following steps:

a. Uninstall eHealth on your standby system.

b. Perform a fresh eHealth r6.1 installation of the standby machine.

6. Upgrade your XOSoft Replication Software:

a. On the primary and standby systems upgrade the WANSync software to the appropriate release.

b. Verify WANsyn engine is running.


Disaster Recovery Management Tasks

7. Upgrade the XOSoft Manager.

8. Modify the WANSync scenarios:

a. Modify the scenarios to copy the files under the new $NH_HOME and oracle 10G.

b. Start the WANSync scenarios as specified when preparing a system for DR. (see page 62) (All three scenarios must be run after a system is upgraded.)

Disaster Recovery Management Tasks After you configure your eHealth environment for DR, you may need to perform these additional tasks.

Initiate Failover from the Standby (Active) System to the Primary System

After you fail over from the primary system to the standby system, the standby system becomes active, running the eHealth server and applications. During this time you can do the following:

1. Perform maintenance on the primary system to resolve problems related to the disaster instance and confirm the primary system is running properly.

2. Create new scenarios to replicate data from the standby (active) system back to the primary system.

3. Fail over from the standby (active) system back to the primary system, thereby making the primary system the active system.

Note: On primary Windows systems, eHealth must be reinstalled so that there are no conflicts when the standby system begins replication back to the primary system.

Install a Patch or Service Pack after Configuring Disaster Recovery

When you need to install an eHealth or Oracle patch or service pack in your DR environment, use the following guidelines:

■ Install the patch or service pack to the primary system only. The WANSync replication software will propagate the changes to the standby system.

■ If the standby system is currently the active system, it is recommended that you wait until the disaster instance on the primary system has been repaired and the system is active again before installing patches and service packs.


Disaster Recovery Management Tasks



1. Stop eHealth, Oracle, the network connection, and all replication scenarios.

2. Stop the WANSync Engine software on both the primary and standby systems by doing the following:

■ Windows-

– In the Services window, right-click the XOsoft Engine service and stop it

– Delete the scenario subdirectory generally stored in C:\Program Files\XOsoft\WANSync\config_25000.

■ Solaris-

– As root user, enter the following command:


– Delete the scenario subdirectory /opt/WANSync/bin/config_25000.

3. Perform the patch or service pack installation.

4. Start the eHealth server and applications and confirm they are running properly.

5. Start the network connection.

6. Start the WANSync Engine software on both the primary and standby systems by doing the following:

■ Windows-In the Services window, right-click the XOsoft Engine service and start it.

■ Solaris-As root user, enter the following command:

/etc/init.d/WANSync start

7. Run the scenarios.

8. Start the WANSync scenarios as specified when preparing a system for DR. (see page 62)(All three scenarios must be run after a system is upgraded.)

Appendix A: Disaster Recovery in a High Availability Cluster


Configure eHealth for Disaster Recovery in a High Availability Cluster (see page 75) Failover from High Availability Cluster to Standby Disaster Recovery System (see page 78) Failover From Standby System to a System in a High Availability Cluster (see page 80)

Configure eHealth for Disaster Recovery in a High Availability Cluster

This procedure includes the steps to configure the following:

■ DR on a primary Solaris eHealth system that is part of an HA cluster.

■ A standby DR system outside the HA cluster to receive data replicated from the HA cluster.

To configure disaster recovery on Solaris

1. Install and configure eHealth r6.1 in the HA cluster (if not already installed).

2. Install the WANSync Engine software. Use the following guidelines:

■ Install the WANSync Engine software on all primary systems in the HA cluster, and the backup system in the DR environment.

■ (Optional) Install the WANSync Manager software on a separate Windows system that is outside the HA cluster and will not have an eHealth server installed. Note that you use only the CLI to start, stop, and edit DR replication scenarios, unless you need to replicate data to an HA cluster after failing over to a DR cluster, in which case you use the WANSync Manager. See Failover From Standby System to a System in a High Availability Cluster (see page 80) for more information.

■ Do not start and stop WANSync processes as root. For security purposes, choose another user account.

■ Enter Yes at the following prompt: Create WANSync group?

■ Enter No at the following prompt: Enable Oracle Support?

Appendix A: Disaster Recovery in a High Availability Cluster 75


3. Configure the primary and standby systems using the following guidelines:

■ Configure the standby DR system on a stand-alone machine outside of the HA cluster.

■ Configure both the primary HA and standby DR systems to have the same kernel parameters.

■ Confirm that the user ID (UID) and group identifier (GID) are the same on both the primary HA and standby DR systems.

4. Bring the Mount and IP resources online on the primary system by entering the following commands:

> su root

■ On Veritas:

hares -online Mount -shared -sys system_name

hares -online ipaddress -sys system_name

shared

Represents the name of the shared directory.

system_name

Represents the name of the primary system.

ipaddress


■ On Sun Cluster :

# scswitch -z -g rgroup-1 -h system_name

rgroup1

Represents the resource group name created earlier.

system_name

Represents the name of the machine in the HA cluster on which you want to install the WANSync software.

5. Install and license the WANSync package on all nodes in the HA cluster.

6. Edit the /etc/group file by appending the $NH_USER user to the end of the /etc/group "wansync" entry.

Example:

If NH_USER is ehuser, then the entry in /etc/group would be:

wansync::60000:ehuser



7. Configure the WANSync DR replication scenarios for eHealth by entering the nhDrConfigScenario command. The command is located under one of the following directories, depending on your clusterware:

■ Sun Cluster: /opt/CAehealthHA/bin

■ Veritas: /opt/CAehealthVcsHA/bin

Enter the WANSync user password for the user that you created in Step 2.

8. Run nhAddVcsResources.

Note: If WANSync has been installed and you try to run nhAddVcsResources before running nhDrConfigScenario, the script will exit with the following error, "Failed to determine the WANSync scenario directory. Please run nhDrConfigScenario before running nhAddVcsResources."

9. Enter the name of the DR standby system.

The scenarios are created and placed on the shared storage area.

Note: By default, the reports and spool directories are placed in the shared storage area. You can use the CLI to change this location, for example, to a dedicated disk on the standby system.

10. (Optional) If you plan to configure eHealth for DR in an HA cluster, enter the following command to add the WANSync resource to the resource group:

# scrgadm -a -j ehealthWanSync -g eHealthGrp-1 -t CA.ehealthWansync \

-x WS_HOME=/opt/WANSync \

-x NH_HOME=$NH_HOME

-x NH_USER="$NH_USER" \

-x NHDR_FILE="haWsUser.dat"

-x NHDR_SCENARIO_P="ehealthWansync-Primary" \

-x NHDR_SCENARIO_S="ehealthWansync-Secondary" \

-x NHDR_SCENARIO_T="ehealthWansync-Third" \



The eHealth resources now reside in the eHealth resource group.

11. Create a symbolic link for the Disaster Recovery scenarios directory in /opt/WANSync/bin to the Disaster Recovery directory (ws_scenarios) under the shared mount point.

12. Stop WANSync from starting the ws_rep process by entering the following command on each node:



Failover from High Availability Cluster to Standby Disaster Recovery System

13. Repeat Steps 4, 5, 10 and 11 for the remaining nodes in the HA cluster.

eHealth is configured for DR in an HA cluster.

14. Add the WANSync resource on Veritas using the command nhAddVcsResources.


When the primary system in the HA cluster goes offline because of a critical failure or disaster, complete this procedure to manually bring the standby system in the DR environment online.

As a best practice, test the manual failover procedure before a disaster occurs. This helps confirm the following:

■ The eHealth servers and the database start without a problem.

■ The replication scenarios are complete and all the necessary files are being copied. You can turn on synchronization and replication scenario reports for the primary system; these reports list all replicated files.

■ You can log on to the eHealth clients after failover.

To fail over from an HA system to a DR environment

1. On the standby DR system, create a symbolic link from /opt/eHealth to the location where eHealth has been replicated to by entering the following command:


2. If the primary HA system is functional, stop the eHealth servers and applications including: FLEXlm, Oracle, Report Center, TrapEXPLODER, WANSync, and the web server by entering one of the following commands for each, depending on your clusterware:

■ Sun Cluster: scswitch -F -g rgroup-1

■ Veritas: hagrp –offline ehealth-sg-1 –sys system_name

Enter one of the following commands to verify the resources are offline, depending on your clusterware:

■ Sun Cluster: scstat

■ Veritas: hagrp –state ehealth-sg-1

3. On the standby DR system, log on as the root user. In the /etc/hosts file, add the shared IP address and hostname.



4. On the standby DR system, create the following file:

/etc/hostname.XXXX:n

XXXX


n

Represents the number of IP addresses.

Note: n usually equals 1 because counting starts at 0 and :0 is not shown.

Add the shared hostname as the file contents. To determine the interface name, enter the following command:

ls /etc/hostname*

5. On the standby DR system, enable and verify the shared hostname:

a. As root, in the command line interface, enter the following commands:



b. Verify the shared hostname is properly set up by entering the following command:

ifconfig -a


6. On the standby DR system, log in as $NH_USER and source the $NH_HOME/nethealthrc.csh file.

7. Start the eHealth database by entering the following command:

nhStartDb

8. Update the Domain Name System (DNS) of the shared hostname to the correct IP address if it is in a different subnet than the HA cluster. Your system administrator may need to complete this step for you.

9. On the standby DR system, enter the following commands to configure the standby system to run eHealth:

nhHaSetup -disable

nhDrSetup -hostname shared_hostname


10. On the standby DR system, start the eHealth servers by entering the nhServer start command.

11. Start the following applications on the standby DR system: FLEXlm (nhLmgr start), TrapEXPLODER (/etc/init.d/trapexploder start), and the web server (nhHttpd start). Do not start Report Center, if installed.


Failover From Standby System to a System in a High Availability Cluster

12. (Optional) If Report Center is installed, rebuild the Report Center configuration file and start Report Center on the standby DR system by entering the following commands:

nhRptCtrConfig -action importConfig

nhRptCtrConfig -action updateDatabaseAccess


After failover from the primary HA system to the standby DR system, the standby DR system becomes active, running the eHealth server and applications. You may choose to keep this active DR system primary, or fail back to the standby system in the HA cluster after problems on the original system are resolved.

Use the following procedure to manually failover from the standby DR system back to a system in the HA cluster. You must fail back if you want to resume replicating data from a primary system in the HA cluster to the standby DR system.

To fail over from an active standby DR system to an HA system

1. Verify that all clusterware and eHealth resources are offline. If they are still online, enter one of the following commands, depending on your clusterware:

■ Sun Cluster: scswitch -F -g rgroup-1

■ Veritas: hagrp –offline ehealth-sg-1 –sys system_name

2. On any system in the HA cluster, as root user, mount the directory or directories (for example, /export/share2) that contain the following:

$NH_HOME $ORACLE_HOME Oracle datafiles



■ A secondary scenario that synchronizes the eHealth and Oracle libraries and executables.



■ A third scenario that synchronizes the /etc files only.


– All scenarios must be configured using the HA cluster's shared (logical) hostname as the DR primary system and the physical node name as the DR standby system.

– When you create the scenarios, you must use the physical node names (not the shared name) of the standby DR system and the primary HA system where you have mounted the shared file system.

– To avoid confusion and help ensure a smooth failover process, directories must be replicated from the primary system to the same disk names and eHealth, eHealth database, and Oracle sub directory names on the standby system.

– When creating the scenarios, choose Create New Scenario and Next. On the Select Scenario Type screen, under the Tasks on Replica section, choose None.



$NH_HOME/bin

$NH_HOME/jre

$NH_HOME/lib

$NH_HOME/lmgr/bin


$ORACLE_HOME/bin

$ORACLE_HOME/ctx

$ORACLE_HOME/lib



■ You must define what directories to use on the HA cluster machine by selecting the HA machine in the middle pane (under Hosts - Replication Tree), and then selecting which directories to use in the right-pane.

The principal scenario is created.



– Create the Secondary Scenario-To create the secondary scenario, in WANSync Manager do the following:


$NH_HOME/bin

$NH_HOME/jre

$NH_HOME/lib

$NH_HOME/lmgr/bin

■ Select the contents of the trap exploder directory (all files should be selected).

■ Select the top-level Oracle database directory (or directories, if more than one) and deselect all the directories excluding the following:

$ORACLE_HOME/bin

$ORACLE_HOME/ctx

$ORACLE_HOME/lib

■ Set the Replication property for the primary system by doing the following:

– Set Replication Mode to Scheduling.

– Set the Attach to Open Files on Run to Off.



– Create the Third Scenario-To create the third scenario, in WANSync Manager do the following:

■ Expand the /etc/init.d directory. In the right pane, select the following files in the following order to add them to the scenario:

httpd httpd.sh nethealth nethealth.sh nhreportcenter trapexploder

■ Expand the /etc directory. In the right pane, select the following files in the following order to add them to the scenario:

/etc/nh.install.cfg /etc/trapexploder.cf

These configuration files are used to automatically start eHealth and Oracle.



■ Right-click on the /etc/init.d directory, and select Remove Directory from the righthand pane.


■ In the right pane, under Properties, turn on the following options:




Note: If you rerun the nhDrSetup -disable or nhDrSetup -hostname commands again for any reason, you must also recreate this third scenario to synchronize the /etc files again.

All scenarios are created.

4. Use WANSync Manager to run the scenarios you created in the following order:

■ Run the secondary scenario: select Tools, Synchronize, then select File Synchronization. This sets the secondary scenario to synchronize the directories in the scenario.

■ Run the third scenario: select Tools, Synchronize, then select File Synchronization. This sets the third scenario to synchronize the files in the scenario.

■ Run the principal scenario: select Tools, Run, then select Block Synchronization. This sets the principal scenario to synchronize and continually replicate its contents to the standby system.

An initial synchronization and replication process begins. The duration of this process depends on the speed of the network between the primary and standby systems and how much data is in your database.

■ On the HA cluster system, create a symbolic link from /opt/eHealth to the location where eHealth has been replicated to by entering the following command:


5. If the standby DR system is functional, stop the eHealth servers by entering the nhServer stop command. Stop the following applications: FLEXlm (nhLmgr stop), Report Center (nhReportCenter stop), Oracle (nhStopDb), TrapEXPLODER (/etc/init.d/trapexploder stop), and the web server (nhHttpd stop).



6. Stop the replication scenario on WANSync Manager, or confirm that it has been stopped. To verify WANSync has stopped, enter the following command on both the HA cluster system used in Step 2, and the standby DR system:


Delete the scenario subdirectory – found in /opt/WANSync/bin/config_25000.

Note: Perform Steps 7 and 8 after the HA system has been repaired but before it becomes the active system.

7. On the standby DR system, remove the shared hostname and IP address entries from the /etc/hosts file and remove the /etc/hostname.XXXX:n file.

8. On the standby DR system, remove the symbolic link for the following files in the /etc/rc0.d, /etc/rc1.d, /etc/rc2.d, /etc/rc3.d, and /etc/rcS.d directories:

nethealth nhreportcenter trapexploder httpd

9. On the standby DR system, disable the shared hostname, as root, by entering the following commands:

ifconfig XXXX:n ipaddress down

ifconfig XXXX:n unplumb

XXXX


n

Represents the number of IP addresses.

ipaddress


Note: n usually equals 1 because counting starts at 0 and :0 is not shown.

10. Verify the shared hostname is disabled by entering the following command:

ifconfig -a

Confirm that the shared hostname is not in the output.



11. Unmount the directory (or directories) mounted in Step 2.

For example, as root, enter the following command depending on the directory:

umount /export/share2

If the umount command is not allowed because it is busy, enter the following command, as root:

fuser -ck /export/share2; umount -f /export/share2

12. On the HA cluster system, the following resources must be disabled before the eHealth resource group is started. To do this, enter the scswitch -n -j resource command for each resource in the following order:

eHealth report center eHealth trap eHealth servlet eHealth eHealth license eHealth web (Apache)

To determine the resource names, enter the scstat command.

13. On the HA cluster system, start the eHealth group by entering one of the following commands, depending on your clusterware:

■ Sun Cluster: scswitch -z -g rgroup-1 -h hostToMigrateTo

■ Veritas: hagrp –online ehealth-sg-1 –sys system_name

Enter one of the following commands to confirm that the resources are online:

■ Sun Cluster: scstat

■ Veritas: hagrp –state ehealth-sg-1

14. On the standby DR system, update the DNS system to map the hostname to the correct IP address if it is in a different subnet than the HA cluster. Your system administrator may need to complete this step for you.

15. From the HA cluster system, configure the standby DR system to run eHealth by enter the following commands:

nhParameter -set highAvailabilityEnabled yes

$NH_HOME/bin/sys/nhiHttpdCfg -user $NH_USER -grp group of NH_USER -nhDir

$NH_HOME -outFile $NH_HOME/web/httpd/conf/httpd.conf

nhWebUtil -setupTomcat

Note: Confirm that the active session has been sourced so that the sourced version of eHealth matches the /opt/eHealth link set in Step 5.




16. On the HA cluster system, re-enable the following resources. To do this, enter the scswitch -e -j resource command for each of the following resources in the following order:

eHealth web (Apache) eHealth license eHealth servet eHealth eHealth trap eHealth report center

To determine the resource names, enter the scstat command.

Failover to a primary HA system is complete.

Appendix B: Sun Cluster Commands

The commands in the following table are supplied by the Sun Cluster software and are used by the HA administrator to manage eHealth in the HA cluster.

Command Description

scswitch -z -g rgroup-1 -h hostToMigrateTo

Brings the resource group online on a particular cluster node, where hostToMigrateTo is the target machine you want to bring online. Use this command to manually transfer operations from a backup system to the original primary system, or to initiate a planned failover for maintenance purposes.

scswitch -F -g rgroup-1 Shuts down the resource group.

scswitch -S -h hostToVacate

Moves all resource groups off of a cluster node, where hostToVacate is the name of the machine you are moving from.

scswitch -n -j resource Brings a resource in the resource group offline, leaving all other resources enabled. For example, use this command to shut down the servers for maintenance or upgrade the Solaris package.

scswitch -n -M -j resource

Disables monitoring of a resource in a resource group without bringing that resource offline.

scswitch -e -j resource Re-enables a resource in a resource group after it has been taken offline.

scswitch -e -M -j resource

Re-enables monitoring of a resource in a resource group.

scstat Lists the current status in the HA cluster including which nodes are part of the cluster and which applications are online and offline.

scrgadm -p Lists all registered resource types in the HA cluster.

Appendix B: Sun Cluster Commands 87

Appendix C: Veritas Cluster Service Commands 89

Appendix C: Veritas Cluster Service Commands

The commands in the following table are supplied by the Veritas Cluster Service software and are used by the HA administrator to manage eHealth in the HA cluster. These actions can also be accomplished using the VCS Java Console.

Command Description

hagrp –offline ehealth-sg-1 –sys system_name

Brings a service group offline on a particular cluster node, where ehealth-sg-1 is the service group and system_name is the name of the system.

hagrp –online ehealth-sg-1 –sys system_name

Brings a service group online on a particular cluster node.

hagrp –online ehealth-sg-1

Brings a service group online on the system that has priority in the SystemList attribute.

haconf -makerw Opens the VCS configuration file. Use when adding resources to a service group.

haconf -dump -makero Saves the VCS configuration file with all resource information.

hagrp –state ehealth-sg-1

Verifies the current status of the service group.

hares –modify resource Critical 1

Specifies a resource as critical. A critical resource causes a failover when it experiences a failure.

hares -offline resource -sys system_name

Brings a resource offline, leaving all other resources enabled. system_name is the name of the system on which the resource is currently running.

hares -online resource -sys system_name

Brings a resource online. system_name is the name of the system on which the resource is currently running.

hagrp –switch ehealth-sg-1 –to hostToMigrateTo

Brings the resource group online on a particular cluster node, where hostToMigrateTo is the target machine you want to bring online. Use this command to manually transfer operations from a backup system to the original primary system, or to initiate a planned failover for maintenance purposes.

Index 91

Index

C Commands

nhHaSetup • 17

D Disaster Recovery

configuration, in a high availability clust • 75

failover from standby to primary • 72 introduction • 11

E eHealth

configuration, disaster recovery in a high availabil • 75

installation guidelines, disaster recovery on Solari • 62

installation guidelines, disaster recovery on Window • 57

F Failover

from standby to primary, disaster recovery • 72

Failures thresholds • 20

G Guidelines

eHealth installation on Solaris, disaster recovery • 62

eHealth installation on Windows, disaster recovery • 57

XOsoft Replication software installation, Solaris • 62

XOsoft Replication software installation, Windows • 57

H High Availability

resource groups • 17 resource types • 17 resources • 17

R Resource Groups • 17 Resource Types • 17 Resources • 17

T Thresholds

failure, high availability • 20

X XOsoft Replication software

installation guidelines, Solaris • 62 installation guidelines, Windows • 57

CA eHealth High Availability and Disaster Recovery...

Documents

Transcript of CA eHealth High Availability and Disaster Recovery...