How to set up, Configure and Manage VCS Virtual Business...

SM L09 –How to Set up, Configure and Manage VERITAS Cluster Server Virtual Business Services across a Multi-Tier Application environment

How to set up, Configure and Manage VCS Virtual Business Services across a

Multi-tier Application environment

Hands-On Lab

Description VERITAS Cluster Server (VCS) 6.0 introduces a new feature known as Virtual Business Service (VBS). VBS ensures high availability and manageability across multiple tiers, hardware architectures, virtualization technologies and operating systems. Gain hands-on experience how to use VCS and ApplicationHA together with VBS to ensure maximum availability protection for a complete business service. Attendees will learn how VBS integrates with different components in a datacenter, such as virtualization technologies. Attendees should have a working knowledge of VCS/ApplicationHA and preferably VERITAS Operations Manager (VOM). They should be able to install and configure both products. However, these components are already installed.

At the end of this lab,

you should be able to Explain the relationship between VBS, VCS, ApplicationHA

and VOM

Install the VBS infrastructure from VOM

Configure Virtual Business Services

Understand VBS dependency types and fault management model

Environment The labs are based on 6 Virtual Machines (2-node VCS cluster on

Solaris, 2-node VCS cluster on Linux Redhat, 1-node with

ApplicationHA and a VOM Central Server).

A VCS cluster (dbsolclus) running on Solaris simulates a

database cluster with three Oracle service groups.

The VCS cluster (applinuxclus) running on Redhat simulates a

application cluster with three WebSphere service groups.

The ApplicationHA node simulates a VMware Virtual Machine

with Windows Server running an IIS Web Service

Lab Student’s Guide

Symantec Vision 2012 – MGM, Las Vegas SM L09 - Lab Student’s Guide / Last Update: Saturday, May 05, 2012 2

Feature Descriptions

This section contains a short introduction of the features that we will configure in this lab session.

Virtual Business Services

Virtual Business Services (VBS) is an add-on to Symantec’s leading clustering software VERITAS Cluster Server (VCS). VBS combines the proven High Availability technology from VCS, as well as ApplicationHA and the power of the VERITAS Operations Manager (VOM) management interface.

VBS allows you to manage multiple Service Groups (applications), grouping them into a single entity. The service groups can be located in different clusters, with a different operating system and hardware architecture. Key features of VBS:

One-click start/stop of applications in different tiers

Fault propagation between tiers, providing maximum availability and automation

One-stop shop for HA management, regardless of operating system, hardware platform or virtualization technology

Start/Stop Virtual Machines as a part of a VBS start/stop operation (6.0 supports VMware only. Start/stop operations for other virtualization platforms may be implemented in future releases)

Note: Virtual Machine start/stop is not covered in this lab.

VBS architectural features:

No central brain orchestrates HA operations. Each tier is managing its own High Availability. The VOM server is required for initial configuration. VOM can also visualization and management, but is not required for VBS to operate.

Backward compatible with VCS 5.1 and 5.1 SP1 and ApplicationHA 5.1 SP2

Support all platforms that VCS and ApplicationHA supports Supports operations using the VOM Central Server or CLI.

Start/stop of multiple tiers

Coordination to start/stop a complete multi-tier application has been a pain point since the

introduction of multi-tiered concepts. Usually, these procedures are carried out manually, which

involves interaction from several different teams within an organization. These procedures are time

consuming and there are a lot of room for human errors.

VBS introduces the ability to start/stop a complete multi-tier application with a single operation. This is

especially useful in Disaster Recovery scenarios.



Start/Stop of Virtual Machines

VBS includes the capability to start and stop individual VMware Virtual Machines. In a disaster

recovery scenario, VBS can spin up “cold” VMs running on a DR site.

Fault Management

If a failure occurs in one tier, this fault may need to be propagated to another tier. For example, an

application in a middle tier may need to be restarted following a failure on a lower tier. VBS introduces

fault propagation between different clusters running on different platforms.

Symantec ApplicationHA 6.0

ApplicationHA was developed in partnership with VMware and was introduced at VMWorld in late

2010. VMwareHA provides infrastructure resilience and ApplicationHA providing application

availability in VMware environments, together they provide protection for application failures, virtual

machine failures and physical machine failures. The solution provides full support for vMotion, DRS

and FT and supports storage interconnects such as NFS, iSCSI and Fiber Channel.

ApplicationHA is also available for other virtualization platforms, such as Oracle VM Server for

SPARC (LDOMs), IBM AIX Micro partitions (DLPARs) and Redhat Kernel Virtual Machines.

In this lab, a virtual machine running Windows Server 2008 and ApplicationHA is available. This VM

simulates an IIS web server, and is located in the top tier.



Lab Exercises

Deploying VBS infrastructure

In this exercise, you will deploy the VBS packages to support this new feature. It is required to install

specific packages on cluster nodes and on the VOM Central Management Server. In later releases of

VCS and VOM, packages may be included in the base product.

Installing the VBS packages is done in two steps. First, we will install the Virtual Business Services

Add-on. This add-on is installed onto the VOM Central Management Server only.

In the second step, we will deploy the VBS infrastructure to all VCS cluster nodes / ApplicationHA

nodes. This package is known as VRTSvbs.

NOTE 1: The Virtual Business Services Add-on and the VRTSvbs packages have already been

uploaded to the VOM Central Server to save time. This will allow more time for labs. In a real-world

environment, the VBS Add-On and the VRTSvbs package are required to be uploaded to the VOM

Central Server.

Installing the VBS Add-on onto the VOM Central Management Server

1. Click on the “vomserver” virtual machine. Click on the VOM Central Management Console short cut. The short cut will take you directly to the VOM splash screen. Login credentials are available on the presentation screen or in the last page of this document

2. In the VOM console, select Settings > Deployment Management

3. In the Deployment Management view, select the Virtual Business Services Availability add-on and click Actions, Install

4. In the Install Solution panel, you will get prompted to install the VBS Availability add-on on the “vomserver” Central Management Server. Click Install. In this step, certain parts of the VOM Central Management Server will be restarted automatically. Click OK.

5. Login to the VOM Central Management Server again using the browser (refresh the sesson)

6. After the session to the VOM Central Management Server is restored, the Deployment Summary is displayed. Verify that the add-on is installed correctly (This may take some time)

Deploying the VRTSvbs package onto the cluster nodes

The VRTSvbs package is required on ALL cluster nodes in clusters that participate in a VBS

configuration.



NOTE: In VCS 6.0, the VRTSvbs package is included in the base product. Hence, there is no need to

deploy VRTSvbs package onto nodes running VCS 6.0. In this lab, the following nodes are running

VCS 6.0:

dbsol1

dbsol2

However, earlier releases of VCS and ApplicationHA will require installation of the VRTSvbs package.

In this lab, we will deploy the VRTSvbs package onto required nodes:

1. In the VOM console, select Settings > Deployment Management

2. In the Deployment Management view, select VRTSvbs and click Actions, Install

3. In the Install Solution panel, select all hosts that will participate in the VBS configuration. Select the following nodes:

applinux1

applinux2

webwin1

Click Install

NOTE: The VRTSvbs package should NOT be deployed on the VOM Central Management

Server (vomserver)

4. In the Result panel, click OK

5. In the Deployment Summary view, verify that the VRTSvbs package installs correctly on all nodes. This may take some time

Virtual Business Services Setup and Management

Initial configuration of a VBS is performed from the VOM Console. After the VBS has been created, it

can be managed and displayed using VOM or using the CLI. In this lab, we will configure three

different Virtual Business Services, one for each dependency type. The dependency types are

explained in a later lab.

The above picture displays the lab environment. The following VBS names will be used: VBS_soft,

VBS_firm and VBS_restart.



The table below describes the VBS configuration and dependencies.

VBS Name Lowest Tier /

Child Service Group Dependency Depending Tier 1 /

Parent Service Group Dependency Depending Tier 2 /

Parent Service Group

VBS_soft oracle1 Soft websphere1 N/A N/A

VBS_firm oracle2 Firm websphere2 N/A N/A

VBS_restart oracle3 Restart websphere3

(child SG for IIS_SG)

Soft IIS_SG

The following picture shows an example of a parent/child relationship

Virtual Business Services creation

A VBS is created in two steps. First, a Business Entity is created that includes the service group you

want to have in the VBS. Secondly, VBS functionality such as dependencies and fault propagation is

configured and enabled.

Repeat the following steps for each VBS using the details provided in the above table:

1. Click Manage, Business Entities

2. Click Actions, Create Business Entity (VBS/OE)

3. Type in a Name from the above table. Make sure that Virtual Business Service is selected as Type and click Next

4. In the Select Base Object Types panel, select Service Groups and click Next

5. In the Select Base Service Groups panel, select Service Groups according to the above table and click Next



6. In the Business Entity Summary panel, click Finish

7. Right-click on the newly created VBS, click VBS Availability, Configure Service Group Dependencies

8. In the Set Start order panel, configure Service Group dependencies according to the above table. Select Parent and Child, and click Link. (for the VBS_restart configuration, you will configure two service group dependencies) NOTE: For all dependency types except soft, the Fault Management checkbox is required to be

true. If CLI support is desired, this checkbox should be set to true for the soft dependency as well.

Click Next

9. Validate the dependencies and click Finish

10. In the Results panel, click OK

11. Repeat the above steps for each VBS (VBS_soft, VBS_firm and VBS_restart)

Start/Stop and visualization of Virtual Business Services

The objective in this lab is to visualize the state of a VBS and to test start/stop functionality.

VBS can be managed using VOM or CLI.

Start/stop using VOM:

1. Click Manage, Business Entities

2. Right-click on a VBS, select VBS Availability, Start VBS or Stop VBS

3. Click OK to confirm that you want to start up the VBS

4. The Service Group Order panel is displayed. Examine and validate start/stop of the VBS

Start/stop using CLI:

1. Open a terminal session as “root” to any cluster node (the “vomserver” host is not part of a VBS configuration)

2. Use the following commands to start/stop and visualize a VBS from CLI:

# vbssvc –start/stop <vbs>

# vbssvc –state <vbs> <vbs> entry is optional. If not specified, all VBS

configured in the cluster are displayed

# vbssbc –grpstate <vbs> Displays the consolidated status of all servicegroups within

the VBS



# vbsgrp –state <vbs> <vbs> entry is optional. If not specified, groups from all

VBS configured on the cluster will be displayed

Understanding VBS Fault Management

High Availability decisions, such as failing over a service group from one node to another, are handled on the local cluster level. This is the default VCS behaviour, and VBS does not change this. However, VBS provides the ability to propagate an event from one tier to another (between clusters and/or ApplicationHA instances) Fault propagation works autonomously of the VOM Central Management Server. Even if the VOM Central Management Server is unavailable, fault propagation is still in effect. Note that VBS will propagate events upon failures only. If fault propagation is enabled for a VBS, and a service group fails, a fault propagation event will be triggered. However, if a service group is switched over manually, a fault will not be propagated to another tier. In our examples, we have one 3-tier VBS (VBS_restart) and two 2-tier VBS (VBS_soft and VBS_firm) The following table describes the VBS dependency types:

Dependency Type

Explanation Use case

Soft When a fault occurs, the event is not propagated to another

tier Used for start/stop orchestration/ordering

Firm

When the child faults, the parent is taken

OFFLINE. When the child recovers, the parent is

brought ONLINE

Some applications have a strict

dependency on lower level components.

For example, the firm dependency is useful

when an application requires a NFS share.

If the NFS share goes offline, the

application will be brought offline

Restart When the child faults, the parent ignores the event.

When the child recovers, the parent is taken OFFLINE and

then brought ONLINE.

Some applications requires a restart following a failure of let’s say a database. Application may need to restart to reconnect to the database. The Restart dependency resolves this situation and will provide

another layer of High Availability.

Understanding the “soft” service group dependency

1. Open a Terminal Session to “applinux1” and “dbsol1” as “root”. Issue the following command in both sessions:

# tail –f /var/VRTSvcs/log/engine_A.log

2. Start the VBS_soft VBS using VOM. Make sure that the “Service Group order” panel is displayed. This panel is displayed automatically after VBS start has been initiated



3. Open another terminal session to “dbsol1”

4. Issue the following # hastatus –sum command to locate the oracle1 service group. # hastatus -sum

-- SYSTEM STATE

-- System State Frozen

A dbsol1 RUNNING 0

A dbsol2 RUNNING 0

-- GROUP STATE

-- Group System Probed AutoDisabled State

B ClusterService dbsol1 Y N ONLINE

B ClusterService dbsol2 Y N OFFLINE

B oracle1 dbsol1 Y N ONLINE Online on dbsol1

B oracle1 dbsol2 Y N OFFLINE

B oracle2 dbsol1 Y N ONLINE




5. If the oracle1 service group is online on dbsol1, continue with the next step. If the oracle1 service group is online on dbsol2, login to dbsol2 and then continue with the next step

6. Simulate a failure of the oracle1 service group by removing the following lock file: # rm /testfiles/oracle1

Examine the output in the terminal sessions started in step 1, as well as the visualization in the

VOM console. The oracle1 service group should be failed over from one system to another,

and the fault should NOT be propagated to another tier, as the soft dependency doesn’t

perform fault propagation.

7. Clear the resource failure using the following command on dbsol1 or dbsol2: # hagrp –clear oracle1

Understanding the “Firm” service group dependency

To demo the Firm dependency type, a double failure is required. We will fail the oracle2 service group

twice (once on node dbsol1 and once on node dbsol2)



1. Open a Terminal Session to “applinux1” and “dbsol1” as “root”. Issue the following command in both sessions: # tail –f /var/VRTSvcs/log/engine_A.log

2. Start the VBS_firm Virtual Business Service using VOM. Make sure that the “Service Group order” panel is displayed. This panel is displayed automatically after VBS start has been initiated

3. Open a terminal session to “dbsol1”

4. Issue the following # hastatus -sum command to locate the oracle2 service group # hastatus -sum

-- SYSTEM STATE


A dbsol1 RUNNING 0

A dbsol2 RUNNING 0

-- GROUP STATE










5. If the oracle2 service group is online on dbsol1, continue with the next step. If the oracle1 service group is online on dbsol2, login to dbsol2 and then continue with the next step


Use # hastatus –sum to determine that the service group has failed over from one system to

another. Login to the other system and issue the above command again:

# rm /testfiles/oracle2



and the fault is propagated to the application tier (applinuxclus).

In the Firm case, this means that the dependent service groups (websphere2) is stopped.

7. Clear the resource failure and online the database (commands can be executed from dbsol1 or dbsol2) # hagrp –clear oracle2



# hagrp –online oracle2 –sys dbsol1

This step simulates that an administrator are resolving a non-recoverable issue with the

database, and then starting up the database again.

Examine the output in the terminal sessions started in step 1 and the visualization in the VOM

console again. The dependent service groups are now started by VBS.

Understanding the “Restart” service group dependency

1. Open a Terminal Session to “applinux1” and “dbsol1” as “root”. Issue the following command in both sessions: # tail –f /var/VRTSvcs/log/engine_A.log

2. Start the VBS_restart Virtual Business Service using the VOM console. Make sure that the “Service Group order” panel is displayed. This panel is displayed automatically after VBS start has been initiated

3. Open a terminal session to “dbsol1”

4. Issue the following hastatus -sum command to locate the oracle3 service group

# hastatus -sum

-- SYSTEM STATE


A dbsol1 RUNNING 0

A dbsol2 RUNNING 0

-- GROUP STATE










5. If oracle3 service group is online on dbsol1, continue with the next step. If oracle3 service group is online on dbsol2, login to dbsol2 and then continue with the next step






and the fault is propagated to the Websphere tier. When the service group is failed over in the

database tier, the Websphere tier is restarted.

8. Clear the resource failure using the following command on dbsol1 or dbsol2:

# hagrp –clear oracle3

Creating shared VBS dependencies

You have deployed the VBS infrastructure, learned how to manage VBS from VOM and from the

command line, and also understood how VBS fault management works.

VBS also support “shared service groups”. A shared service group is shared between two or more

VBS entities. VBS applies a specific logic to shared service groups.

Startup of a VBS with one or more shared service groups:

VBS will start up all service groups in the configured order.

Shutdown of a VBS with one or more shared service groups:

VBS will stop all non-shared service groups. By default, shared service groups will not be stopped. However, there is an option to offline shared service groups from the VOM Console and from the CLI

You will reconfigure the environment according to the picture below:



Removing the VBS_Firm VBS and configuring a new shared VBS

In this lab, we will remove the VBS_Firm VBS from the configuration and create a new shared VBS.

1. In the VOM console, select Manage > Business Entities

2. Right-click on the VBS_Firm VBS and click Delete Business Entity

3. Click Yes to confirm that you want to remove the VBS. Click OK

4. Click Actions, Create Business Entity (VBS/OE)

5. In the Name field, type VBS_shared. Make sure that Virtual Business Service is selected as Type and click Next

6. In the Select Base Object Types panel, select “Service Groups” and click Next

7. In the Select Base Service Groups panel, select oracle3 and websphere2

8. In the Business Entity Summary panel, click Finish

9. Right-click on the VBS_shared VBS, click VBS Availability, Configure Service Group Dependencies

10. In the Set Start order panel, configure the oracle3 service group as child, and the websphere2 service group as parent. Select Dependency Type Soft. Click Next

11. Validate the dependencies and click Finish



Start/Stop of a VBS with shared service groups

In this lab, we will validate start and stop behavior when one VBS has a shared service group

configured.

1. Stop VBS_shared from the VOM Console. Note the warning about shared service groups. Do not select “Offline shared service groups”

2. Stop VBS_restart. Select Offline shared service groups (If you prefer CLI operations, you can use # vbssvc –stop –force VBS_restart

3. Start VBS_restart and VBS_shared again and confirm that they online correctly

4. Induce a failure on oracle3 (remove the lock file as instructed earlier). Examine the difference and note that fault propagation only is effective on the websphere3 service group.

Lab Configuration Information



Hostname IP Cluster

dbsol1 169.254.128.11 Dbsolclus

dbsol2 169.254.128.12 Dbsolclus

applinux1 169.254.128.13 Applinuxclus

applinux2 169.254.128.14 Applinuxclus

webwin1 169.254.128.15 ApplicationHA

instance

vom 169.254.128.31 N/A

Service Group IP Cluster

ClusterService 169.254.128.21 Dbsolcluster

ClusterService 169.254.128.22 Applinuxclus

Credentials: vom: root/symc4now dbsol1/dbsol2 root/symc4now applinux1/applinux2 root/symc4now webwin1 administrator/symc4now

How to set up, Configure and Manage VCS Virtual Business...

Documents

Transcript of How to set up, Configure and Manage VCS Virtual Business...