FlexFrame Agents Guide -...
Transcript of FlexFrame Agents Guide -...
FlexFrame® Orchestrator
Version 1.2A
FlexFrame Agents Installation and Administration
Edition January 2017 Document Version 1.2
Fujitsu Limited
© Copyright Fujitsu Technology Solutions 2015 - 2016
PRIMEFLEX® is a registered trademark of Fujitsu Limited in Europe and other countries.
FlexFrame® and PRIMERGY™ are trademarks or registered trademarks of Fujitsu Lim-
ited in Japan and other countries.
SAP® and NetWeaver™ are trademarks or registered trademarks of SAP AG in Germa-
ny and in several other countries
Linux® is a registered trademark of Linus Torvalds
SUSE® Linux is a registered trademark of Novell, Inc., in the United States and other
countries
Oracle™ and Java™ are trademarks of ORACLE Corporation and/or its affiliates
Intel® and PXE® are registered trademarks of Intel Corporation in the United States and
other countries
MaxDB® is a registered trademark of MySQL AB, Sweden
MySQL® is a registered trademark of MySQL AB, Sweden
NetApp® and the Network Appliance® logo are registered trademarks and Network Ap-
pliance™ and Data ONTAP™ are trademarks of NetApp, Inc. in the U.S. and other coun-
tries.VMware®, ESX®, ESXi, VMware vCenter, VMware vSphere are registered trade-
marks or trademarks of VMware, Inc. in the United States and/or other jurisdictions.
Ethernet® is a registered trademark of XEROX, Inc., Digital Equipment Corporation and
Intel Corporation
All other hardware and software names used are trademarks of their respective compa-
nies.
All rights, including rights of translation, reproduction by printing, copying or similar meth-
ods, in part or in whole, are reserved.
Offenders will be liable for damages.
All rights, including rights created by patent grant or registration of a utility model or de-
sign, are reserved.
Delivery subject to availability. Right of technical modification reserved.
3 FlexFrame Agents Guide
5 FlexFrame Agents Guide
Contents
General product information and naming .................................................................... 11
1 Introduction ................................................................................................... 13 1.1 FlexFrame Agents Suite .................................................................................. 14 1.2 Additional Documentation ............................................................................... 14 1.3 Target Group ................................................................................................... 15 1.4 Notational Conventions ................................................................................... 15 1.5 Document History ............................................................................................ 16 1.6 Changes and Enhancements in this Version ................................................... 17 1.7 Related Documents ......................................................................................... 18 1.8 Naming Information ......................................................................................... 18
2 First Steps ...................................................................................................... 19 2.1 General Information ......................................................................................... 19 2.2 Installation and Startup .................................................................................... 20 2.2.1 The FlexFrame Solution .................................................................................. 20 2.2.2 Installation ....................................................................................................... 21 2.2.3 Configuration ................................................................................................... 33 2.2.4 Starting and Stopping ...................................................................................... 34 2.2.5 Configuration of generic .................................................................................. 35 2.3 FlexFrame Agents WebInterface ..................................................................... 37 2.3.1 Function .......................................................................................................... 37 2.3.2 Installation ....................................................................................................... 37 2.3.3 Configuration ................................................................................................... 37 2.3.4 Starting and Stopping ...................................................................................... 37 2.3.5 FlexFrame Agents WebGui and database ...................................................... 38 2.4 FlexFrame DomainManager ............................................................................ 40 2.4.1 Database connection ....................................................................................... 40 2.5 FlexFrame Agents Suite Messenger ............................................................... 42 2.5.1 Database connection ....................................................................................... 42 2.6 FlexFrame Agents Suite version management................................................ 44 2.6.1 Lifecycle process ............................................................................................. 44 2.6.2 Lifecycle management files ............................................................................. 45 2.7 Usage of Pool-Master ...................................................................................... 51 2.7.1 Create Pool-Master from existing pool ............................................................ 52 2.7.2 Create/Update pool from Pool-Master ............................................................. 52 2.7.3 Usage of ‘fa_MasterPoolMgr.sh’ ..................................................................... 54 2.7.4 Automation of steps during pool creation ........................................................ 54
3 Introduction ................................................................................................... 55 3.1 FlexFrame Agents Suite Architecture .............................................................. 55
Introduction
FlexFrame Agents Guide 6
3.1.1 FlexFrame Agents Suite Visualization and Reporting ...................................... 57 3.1.2 FlexFrame Agents Suite virtual server and hypervisor nodes.......................... 58 3.2 Pool Creation and Grouping ............................................................................ 60 3.2.1 Virtual FlexFrame Pools ................................................................................. 60 3.2.2 Grouping .......................................................................................................... 62 3.2.3 Pool Types (Pool Independent Spare) ............................................................. 62 3.3 Service Classes ............................................................................................... 66 3.3.1 Service Priority ................................................................................................ 66 3.3.2 Service Power Value ....................................................................................... 66 3.3.3 Class Creation Rules ....................................................................................... 67 3.3.4 Testament Types ............................................................................................. 67 3.4 FlexFrame Agents Configuration, Work and LogFiles ..................................... 68 3.5 Systems ........................................................................................................... 69 3.5.1 Service Types .................................................................................................. 69 3.6 Generic Services ............................................................................................. 73 3.6.1 Service State Model ........................................................................................ 73 3.6.2 Service Detection Model .................................................................................. 74 3.6.3 Service Reaction Model ................................................................................... 74 3.7 FlexFrame Performance and Accounting Option ............................................ 75 3.7.1 Performance Option ........................................................................................ 76 3.7.2 Accounting Option ........................................................................................... 77 3.7.3 Billing ............................................................................................................... 78 3.8 Parametricable Value Observation and Service Level Profile ......................... 80 3.8.1 Monitoring of file system and directoy utilization .............................................. 80 3.8.2 Performance Profile, Capacity Profile, Accounting Profile ............................... 80 3.9 FlexFrame File System Utilization and File Space Accounting Option ............ 81 3.9.1 File System Detector and Directory Detector ................................................... 84 3.10 Domain Manager for Performance and File System Collections ..................... 85 3.10.1 Data collection ................................................................................................. 85 3.10.2 Data retentiontime and compression ............................................................... 86 3.11 Customized Detector engine (service ping) ..................................................... 87 3.11.1 ServicePingDb.sh ............................................................................................ 87 3.11.2 ServicePingDef.sh ........................................................................................... 87 3.11.3 ServicePingErs.sh ........................................................................................... 87 3.11.4 ServicePingHDB.sh ......................................................................................... 87
4 FlexFrame Agents ......................................................................................... 89 4.1 FlexFrame Agents Reactions .......................................................................... 90 4.1.1 Restart ............................................................................................................. 90 4.1.2 Reboot ............................................................................................................. 90 4.1.3 SwitchOver ...................................................................................................... 90 4.1.4 Special case “fast start VM” ............................................................................. 93 4.1.5 Maintenance .................................................................................................... 94 4.2 Self-Repair Strategies ..................................................................................... 94 4.2.1 Self-Repair in the Event of a Service Failure ................................................... 94
Introduction
7 FlexFrame Agents Guide
4.2.2 Self-Repair in the Event of a Node Failure ...................................................... 95 4.2.3 Takeover by a Spare Node (TakeOver) .......................................................... 95 4.2.4 Multi Node Failure ........................................................................................... 95 4.3 Escalation with partner service ........................................................................ 97 4.3.1 SCS / ASCS in conjunction with ERS .............................................................. 97 4.3.2 Service with assigned SMD-service ................................................................ 98 4.4 Takeover Rules ............................................................................................. 100 4.4.1 Overview ....................................................................................................... 100 4.4.2 TakeOver Strategy ........................................................................................ 100 4.4.3 TakeOver Rule .............................................................................................. 101 4.4.4 Spare Nodes from the Adminpool (pool-independent spare node)................ 108 4.5 Management of Performance, File System and Accounting .......................... 116 4.5.1 Performance and Accounting Management .................................................. 116 4.5.2 File System Utilization ................................................................................... 116 4.5.3 Monitoring Cycles .......................................................................................... 116 4.6 Operating Mode ............................................................................................. 117 4.6.1 Event Mode ................................................................................................... 117 4.6.2 Local Reaction Mode ..................................................................................... 117 4.6.3 Central Reaction Mode .................................................................................. 117 4.7 Autonomous Operation of a FlexFrame Infrastructure ................................. 119 4.7.1 FlexFrame Agents and Fujitsu FlexFrame Scripts ......................................... 119 4.8 FlexFrame Agents and User Interactions ...................................................... 120 4.8.1 Starting/Stopping/Status of FlexFrame Agents.............................................. 121 4.8.2 Starting/Stopping an SAP Instance ............................................................... 123 4.8.3 NoWatch/Watch an SAP Instance ................................................................. 125 4.8.4 Suspend Autonomy ....................................................................................... 131 4.9 Use cases...................................................................................................... 136 4.9.1 General ......................................................................................................... 136 4.9.2 Semi-autonomous operation ......................................................................... 137 4.9.3 Autonomy for Application Instances .............................................................. 137 4.9.4 Management of Performance, Capacity and Accounting .............................. 140 4.9.5 File System Utilization ................................................................................... 140 4.9.6 High Availability and Autonomy Rules ........................................................... 141 4.10 FlexFrame Agents Work and Log Files ......................................................... 152 4.11 Migration of FlexFrame Agent Versions on Pool Level .................................. 153 4.12 The FlexFrame Migration Tool ...................................................................... 157 4.12.1 Pool Mode ..................................................................................................... 157 4.12.2 File Mode....................................................................................................... 157 4.12.3 Usage ............................................................................................................ 158 4.13 Command Line Interface fa_list_services.sh ................................................. 160 4.14 Command Execution at All Nodes of a Pool / ALL_POOLS .......................... 161
5 WebInterface ................................................................................................ 163 5.1 Installation / Configuration ............................................................................. 163 5.1.1 Prerequisites ................................................................................................. 163
Introduction
FlexFrame Agents Guide 8
5.1.2 Installation ..................................................................................................... 163 5.1.3 Configuration ................................................................................................. 163 5.2 Visualization .................................................................................................. 169 5.2.1 Starting the Client .......................................................................................... 169 This browser will open a new JNLP client wich can be used independently
from the browser. ............................................................................................................ 169 5.2.2 Login .............................................................................................................. 169 5.2.3 Overview of Elements .................................................................................... 170 5.2.4 Pool / Group Tree .......................................................................................... 171 5.2.5 Status Display ................................................................................................ 177 5.2.6 Message Display ........................................................................................... 184 5.2.7 Configuration of FlexFrame Agents with the Webinterface ............................ 188 5.2.8 Special Signals .............................................................................................. 199 5.3 Interaction ...................................................................................................... 200 5.3.1 Updates ......................................................................................................... 200 5.4 Info and Help ................................................................................................. 202 5.5 FlexFrame Performance and Accounting Plug-in .......................................... 203 5.6 FlexFrame Reporting Plug-in ......................................................................... 203 5.7 Customization of Icons in the FlexFrame Agents WebGUI ............................ 204
6 FlexFrame Agents Power Shutdown Concept .......................................... 205 6.1 General .......................................................................................................... 205 6.2 Power Shutdown Architecture ....................................................................... 206 6.3 Basics ............................................................................................................ 207 6.3.1 Powershutdown methods for use with FlexFrame generic shutdown facility . 207 6.4 Power Shutdown Configuration ..................................................................... 209 6.4.1 Switchover Control Parameters ..................................................................... 210 6.4.2 User, Password and Community ................................................................... 212 6.4.3 Management Blades ...................................................................................... 215 6.4.4 Application Nodes .......................................................................................... 217 6.4.5 Default Shutdown Mode ................................................................................ 222 6.4.6 Management blades with SNMP-community different from default ............... 222 6.5 Possible Errormessages with old Configuration ............................................ 223
7 Parameter Reference ................................................................................... 225 7.1 FlexFrame Agents ......................................................................................... 225 7.1.1 FlexFrame Agent Configuration Files ............................................................ 225 7.2 SNMP Traps .................................................................................................. 226 7.2.1 General .......................................................................................................... 226 7.2.2 Structure ........................................................................................................ 226 7.2.3 Default Parameter File ................................................................................... 228 7.3 Pooling and Grouping .................................................................................... 229 7.3.1 Pooling........................................................................................................... 229 7.3.2 Grouping ........................................................................................................ 229 7.3.3 Default Parameter File ................................................................................... 234
Introduction
9 FlexFrame Agents Guide
7.4 Service Classes ............................................................................................. 253 7.4.1 Service Priority .............................................................................................. 253 7.4.2 Service Power Value ..................................................................................... 253 7.4.3 Class Creation Rules ..................................................................................... 253 7.4.4 Example ........................................................................................................ 254 7.5 FlexFrame Agents ......................................................................................... 254 7.5.1 General Parameters ...................................................................................... 255 7.5.2 Parameters for the Performance and Accounting Option .............................. 257 7.5.3 Parameters for the File System Utilization..................................................... 258 7.5.4 Configuration of the Spaceutil Functionality .................................................. 258 7.5.5 Parameters for the Admin Pool Option .......................................................... 268 7.5.6 Parameters for ServicePing........................................................................... 269 7.5.7 Node-related Parameters .............................................................................. 269 7.5.8 Service-related Parameters ........................................................................... 270 7.5.9 Parameters for the Definition of a Generic Service ....................................... 272 7.5.10 Path Configuration ......................................................................................... 277 7.5.11 Shutdown Configuration ................................................................................ 279 7.5.12 Default Parameter File .................................................................................. 280 7.5.13 FlexFrame FrameAgent ................................................................................ 312 7.5.14 Adjustment of the FlexFrame Agents WebGUI.............................................. 312 7.5.15 FlexFrame Webservices and Hypervisor Monitor configuration .................... 313 7.6 FlexFrame Domain Manager ........................................................................ 314 7.6.1 Common ........................................................................................................ 314 7.6.2 Configuration ................................................................................................. 314 7.6.3 Configuration of DomainManager with FA-ConfigGUI ................................... 315
8 BlackBoard .................................................................................................. 316 8.1 General ......................................................................................................... 316 8.2 Implementation .............................................................................................. 316 8.3 Generating BlackBoard Commands .............................................................. 319 8.3.1 WebInterface ................................................................................................. 319 8.3.2 Interactive ...................................................................................................... 320 8.4 Commands .................................................................................................... 321
9 FlexFrame Agents Traps............................................................................ 323 9.1 General ......................................................................................................... 323 9.2 Structur of FlexFrame Agents Traps ............................................................. 324 9.3 Overview of the FlexFrame Agents SNMP Traps .......................................... 327
10 FlexFrame Troubleshooting ....................................................................... 345 10.1 General ......................................................................................................... 345 10.2 Incident Diagnosis..................................................................................... 345 10.3 FlexFrame Agents Work and Log Files ......................................................... 351 10.3.1 General ......................................................................................................... 351 10.3.2 Overview, Principal Directories, Files ............................................................ 351
Introduction
FlexFrame Agents Guide 10
10.3.3 Collecting Diagnostic Information for Support Assistance ............................. 357 10.3.4 Selected Files ................................................................................................ 359 10.3.5 FlexFrame Diagnostic Tool ............................................................................ 360 10.3.6 Collecting Diagnostic Information for Support Assistance ............................. 361
11 Abbreviations ............................................................................................... 363
12 Glossary ....................................................................................................... 367
13 Index ............................................................................................................. 373
11 FlexFrame Agents Guide
General product information and naming
The name FlexFrame®
is a generic term for both „FlexFrame®
for SAP®“ (FF4S) and
„FlexFrame®
Orchestrator“ (FFO).
The trademark myAMC® is not being used any more in the context of FlexFrame. The
brand myAMC® will be replaced by the brand FlexFrame
®, e.g. instead of myAMC Ac-
counting we now speak about FlexFrame Accounting.
The abbreviation FA for FlexFrame Autonomy now denotes the FlexFrame Agents
(Application Agents, Control Agents and the Frame Agent). The FA Suite now denotes
the FlexFrame Agents Suite.
The notation “myAMC” continues to be used in the context of pathnames, configfiles, rpm
package names or as part of scriptnames.
13 FlexFrame Agents Guide
1 Introduction
For many companies, applications such as SAP® today provide the basis for handling all
important business processes. Failure of these components therefore results in consider-
able costs. Nowadays companies must be able to react very rapidly to changing market
and organizational demands, which also means that it must be possible to adapt the
capacity of existing IT resources very quickly to the changing requirements.
The FlexFrame Agents Suite for monitoring the availability and utilization of IT systems
with their intelligent automated facility for responding to system failures are the answer to
these demands. The FlexFrame® Agents Suite offer powerful monitoring and manage-
ment functions with functions which permit the autonomous operation of a distributed
applications environment. These functions reduce the number of manual interventions
and make the operation of your business critical applications more efficient.
FlexFrame offers a flexible hardware architecture which can be adapted to altered re-
quirements and, together with management components, permits highly available opera-
tion of this infrastructure. Partial outages are automatically repaired or compensated for.
FlexFrame Agents Suite is an integral component of every FlexFrame solution and pro-
vides the functions for implementing operation with considerably reduced operator inter-
ventions through the built-in autonomy functions, right up to a high-availability solution.
This manual describes the functional concepts and the application scenarios for the Flex-
Frame Agents Suite.
Introduction
FlexFrame Agents Guide 14
1.1 FlexFrame Agents Suite
The FlexFrame Agents Suite is a solution for monitoring and managing FlexFrame and
FlexFrame Orchestrator infrastructures. In addition to monitoring, this solution suite also
provides the option of implementing automatic restoration of failed services autonomous-
ly. These self-repair mechanisms are not just effective locally for one system, however;
they also permit a failed service to be moved automatically to another resource which, in
line with a defined rule for operation, is suitable for operating the service.
This function permits a considerable reduction in the number of manual interventions by
an administrator. Availability is increased, and the costs for operating a complex applica-
tions environment are reduced.
For this functionality, the FlexFrame Agents Suite uses its agents and management com-
ponents to detect, collect and analyze the information. Autonomous functions can be
configured for various tasks and requirements by combining different detectors and man-
ager components and by defining and selecting the reaction and decision rules. In con-
junction with the powerful FlexFrame Agents WebGUI, the entire infrastructure can be
presented in a straightforward manner in an IT cockpit.
1.2 Additional Documentation
Further application options for other FlexFrame Agents Suite components are described
in special documents.
Use of the FlexFrame Messenger for editing and forwarding the FlexFrame Agents’ mes-
sages is described in the documentation “FlexFrame Messenger Concept and Usage".
Use of the FlexFrame LogAgent for forwarding of log-entries is described in the docu-
mentation “FlexFrame LogAgent Concept and Usage“.
Introduction
15 FlexFrame Agents Guide
1.3 Target Group
This documentation is intended to support both users of FlexFrame Agents Suite and
administrators who wish to integrate this solution in an enterprise IT management solu-
tion.
1.4 Notational Conventions
The following conventions are used in this manual:
Additional information that should be observed.
Warning that must be observed.
fixed font Names of paths, files, commands, and system output.
<fixed font> Names of variables.
fixed font User input in command examples
(if applicable using <> with variables)
Introduction
FlexFrame Agents Guide 16
1.5 Document History
Document Version Changes Date
1.0 First Edition for FlexFrame Orchestrator
1.2A
2015-12-01
1.1 Update with FFO 1.2A N2 2016-06-01
1.2 Update with FFO Patch 019 2017-01-25
Introduction
17 FlexFrame Agents Guide
1.6 Changes and Enhancements in this Version
IBM JAVA 7 for FlexFrame WebGui
Upgraded usage of ESX and KVM hypervisor management service
SAP HANA as single instance or as multi instance installation
Upgraded configuration of the service priority with the FlexFrame WebGui
Upgrade Persistant “NoWatch” for SIDs and for non running instances
EzSetup-parameter for cDOT migration steps
Output of script fa_list_services.sh modified.
Introduction
FlexFrame Agents Guide 18
1.7 Related Documents
FlexFrame® – Administration and Operation
FlexFrame® – HW Characteristics Quickguides
FlexFrame® – Installation and Configuration LVM 2.1 Standard Edition
FlexFrame® – Installation Guide for SAP Solutions
FlexFrame® – Installation of a FlexFrame Environment
FlexFrame® – Management Tool
FlexFrame®
– Messenger Concept and Usage
FlexFrame®
– LogAgent Concept and Usage
FlexFrame®
– Agents Accounting Guide
FlexFrame® – Network Design and Configuration Guide
FlexFrame® – Security Guide
FlexFrame® – Technical White Paper
FlexFrame® – Upgrading FlexFrame Orchestrator 1.0A or 1.1A to 1.2A
ServerView Documentation
SUSE Linux Enterprise Server Documentation
1.8 Naming Information
PRIMEFLEX for SAP Landscapes enables simplified, fast and secure implementation
and operation of SAP applications and databases. The infrastructure solution is designed,
delivered and supported as one product and supplemented by a broad services portfolio.
The integrated FlexFrame Orchestrator software offers consistent and standardized ad-
ministration of infrastructure, databases and applications.
First Steps
19 FlexFrame Agents Guide
2 First Steps
2.1 General Information
This chapter describes how you start and stop the FlexFrame Agents Suite. It also de-
scribes how to install the FlexFrame Agents Suite and its basic configuration.
The FlexFrame Agents Suite provides a comprehensive, flexible and scalable solution for
setting up semi-autonomous IT processes. Its functionality falls into these subareas:
FlexFrame AppAgents: FlexFrame Application Agents for monitoring, checking and
controlling instances. An AppAgent runs at every application node.
FlexFrame CtrlAgent: FlexFrame Control Agent for monitoring, checking and control-
ling Application Nodes with a separate Control Node. A CtrlAgent runs for every pool
at the active control node.
FlexFrame FrameAgent: FlexFrame FrameAgent for monitoring, checking of central
resources. FrameAgent is an AppAgent at the Control Node. A FrameAgent runs at
every control node.
FlexFrame Agent WebInterface: A component for displaying the active services on a
web front-end.
FlexFrame Messenger: A component for receiving and forwarding messages.
FlexFrame LogAgent: A component for controlling logfiles.
FlexFrame DomainManager: A component for performance-, accounting- and file-
utilisation options. The use of this component requires a special license.
FlexFrame Agents Suite rpm: common component for example version manage-
ment.
First Steps
FlexFrame Agents Guide 20
2.2 Installation and Startup
2.2.1 The FlexFrame Solution
The FlexFrame Agents Suite solution was conceived and developed especially for the
“FlexFrame® for SAP
® “ solution from Fujitsu. Consequently the FlexFrame solution with
the components Shared OS, Virtualized SAP Application and NetApp Storage on the
target computers is a prerequisite for the procedure described in the following.
Further details on FlexFrame configurations can be found in the FlexFrame manual “In-
stallation of a FlexFrame Environment“. Use of FlexFrame Agents Suite on other Linux
architectures (e.g. standalone systems or for monitoring processes which do not belong
to SAP R/3) is not described in this manual and is not supported.
The following prerequisites are thus particularly important:
Server architecture with IP storage (NetApp Filer) and client, server and storage
LANs.
Paths for read-only and read/write Root images.
SAP start scripts from Fujitsu
Operating system SUSE Linux Enterprise Server (SLES)
FlexFrame Agents are installed in a directory on the storage system which is reachable
and available to all nodes in accordance with the FlexFrame rules for jointly used pro-
grams. Programs are always accessed and installed via a Control Node.
The FlexFrame Agents Suite components are installed using RPM packages. Normally the agents are stored in the directories /opt/myAMC/FA_AppAgent,
/opt/myAMC/FA_CtrlAgent and /opt/local/myAMC/FA_FrameAgent.
The /opt/myAMC directory is located in a FlexFrame environment on the Filer and is
available from every Application Node and Control Node.
Multiple FlexFrame Agents Suite versions can be installed simultaneously. Installation,
configuration and activation of a version are three separate activities.
Installation, parameterization and configuration of a new version can thus be performed
during ongoing operation. Only when all preparations have been completed is the active
version deactivated and the new version activated.
Deactivation and activation of a version always takes place on a pool-specific basis. In
this way new agent versions can, for example, first be activated in a pool with test sys-
tems.
First Steps
21 FlexFrame Agents Guide
2.2.2 Installation
2.2.2.1 Requirements
In the case of a FlexFrame standard installation, new software components are installed
via one of the Control Nodes.
The FlexFrame Agents Suite software is contained in the /opt/myAMC resp.
/opt/local/myAMC directory. Ensure that all servers (Control Nodes and Application
Nodes) use the same directories. FlexFrame Agents Suite is thus also installed in a tree
on a Filer (NFS share).
The NFS file systems used have to support NFS file locking.
control1:/opt/myAMC # mount
filer1_qa:/vol/volFF/FlexFrame/myAMC on /FlexFrame/myAMC type nfs
(rw,nfsvers=3,intr,noac,wsize=32768,rsize=32768,addr=172.16.1.204)
filer1_qa:/vol/volFF/FlexFrame/scripts on /FlexFrame/scripts type nfs
(rw,nfsvers=3,intr,nolock,noac,wsize=32768,rsize=32768,addr=172.16.1.204)
control1:/opt/myAMC # ls -al /opt/myAMC
lrwxrwxrwx 1 root root 16 Dec 2 18:35 /opt/myAMC ->
/FlexFrame/myAMC
2.2.2.2 Installation Packages
The following packages must be installed:
FlexFrame Agents Suite; the installation package for this is called
myAMC.FA_Suite-<X.Y-Z>.i586.rpm
FlexFrame Application Agent; the installation package for this is called
myAMC.FA_AppAgent-<X.Y-Z>.i586.rpm
FlexFrame Control Agent; the installation package for this is called
myAMC.FA_CtrlAgent-<X.Y-Z>.i586.rpm
FlexFrame Frame Agent; the installation package for this is called
myAMC.FA_FrameAgent-<X.Y-Z>.i586.rpm
FlexFrame Agents WebInterface; the installation package for this is called
myAMC.FA_WebGui-<X.Y-Z>.i586.rpm
FlexFrame Messenger; the installation package for this is called
myAMC.FA_Messenger-<X.Y-Z>.i586.rpm
FlexFrame LogAgent; the installation package for this is called
First Steps
FlexFrame Agents Guide 22
myAMC.FA_LogAgent-<X.Y-Z>.i586.rpm
FlexFrame DomainManager (optional, e.g.for the performance and accounting op-
tion), The installation package for this is called
myAMC.FA_DomainManager-X.Y-Z.i586.rpm
where X.Y-Z stands for the version number.
2.2.2.3 rpm-Installation
Standard installation is implemented from a completely writeable (as the user root)
directory tree.
1. Log onto the target computer as root and copy the rpm packages to a tempo-
rary directory.
2. Install the required package with
rpm –ihv --nodeps myAMC.FA_AppAgent-<X.Y-Z>.i586.rpm
After all the required packages have been installed, the start scripts may need to be
copied to the root images of the various node types (Application / Control).
The FlexFrame CtrlAgent and FlexFrame FrameAgent run on the Control Node, and only
the FlexFrame AppAgent may run on the Application Nodes.
The rpm-installation will be only used for explicit change of single packages. Thereby no
automatic migration steps will be done.
For installation or patch of an existing FlexFrame Agents Suite in a FlexFrame solution
only EzSetup will be used (see following chapters).
First Steps
23 FlexFrame Agents Guide
2.2.2.4 EzSetup-Installation
2.2.2.4.1 General
The installation or patch of an existing FlexFrame Agents Suite in a Flexframe solution
will be supported with a backup-script as of FlexFrame V5.2A00-018.
The backup script allows the backup of config and dynamic data from FA Messenger
stored in database.
The data of Accounting (FA DomainManager) can also be backed up with the backup
script if the component is active in the FlexFrame.
Also the config files of the FlexFrame Agents can be saved.
With parameters of the backup script can be selected which data will be saved.
The backup functionality can be used before any config changes to save the actual con-
figuration. But also before every change of the FlexFrame during an update or patch.
The config and dynamic data of FA Messenger and the optional FA DomainManager will
be exported as XML datafiles.
Before export of the dynamic accounting data from FA DomainManager the size of the
data must be respected.
Bevor using the backup script it must be checked if there is enough space left at storage
for the exported data.
The fa_ezsetup has a new integration with the FlexFrame Installer environment as of
FlexFrame V5.2A00-018.
The new installation mode is described in the FlexFrame manual “Installation of a Flex-
Frame Environment”.
The fa_ezsetup tool supports now the following installation processes.
new FlexFrame installations as of FlexFrame 5.2A00-018
upgrade FlexFrame from FlexFrame < 5.2A to FlexFrame ≥ V5.2A00-018
patch FlexFrame V5.2A to FlexFrame V5.2A00-n with n ≥ 018
new FlexFrame installations as of FlexFrame Orchestrator 1.0A
upgrade FlexFrame from FlexFrame Orchestrator 1.0A to FlexFrame Orchestra-
tor 1.1A
The FlexFrame preinstaller and the FlexFrame installer together with fa_ezbackup and
the fa_ezsetup automate the database migration process from mySQL to postgreSQL.
First Steps
FlexFrame Agents Guide 24
Attention: A patch installation from FlexFrame V5.2A to FlexFrame V5.2A00-n with n ≥
018 needs a manually triggered fa_ezbackup before you can start fa_ezsetup.
Check the fa_ezbackup logfile before you start the fa_ezsetup!
2.2.2.4.2 Backup with EzBackup
With EzBackup the config and dynamic data of an existing database, if it is running inside
of FlexFrame, will be saved.
The data will be stored in the following directory:
/opt/myAMC/Backup/[Komponente]/[VersionString]
Every component will be named with version number and stored in an own subdirectory.
Usage:
./fa_ezbackup.sh -h
Usage: ./fa_ezbackup.sh [ -x ] [ -a | -c | -d ] [ -m | -u ] [ -p |
-y ] [ -h ] [ -o dirname ] [ -l logfile ]
-x execute export
-a export agent config only
-c export database config only
-d export dynamic data only
-m use messenger database
-u use accounting database
-p use postgreSQL
-y use mySQL
-h display this help
-o dirname specify output directory
-l logfile write all logs into this file
First Steps
25 FlexFrame Agents Guide
Example sequence of backup:
1. Change to directory containing the update script
cn1: # cd /media/dvd/FA_Suite/myAMC_FA
2. Run a test without permanent changes:
This option determines which data is involved and the estimated time for the export.
cn1: # ./fa_ezbackup.sh -l /tmp/myAMC_FA.log
If this test did not report errors continue with step #3.
If this test failed you can't continue since the upgrade will fail. Please fix the problem
by help of the mentioned README and start again with step #2.
3. Run the backup.
cn1: # ./fa_ezbackup.sh -l /tmp/myAMC_FA.log -x
4. Check the results.
Check the data in backup-directory: /opt/myAMC/backup
2.2.2.4.3 Standard-Installation with EzSetup
A detailed description of the installation with EzSetup und possible changes and hints can
be found in the readme-file “fa_ezsetup.README.txt”.
1. Change to directory containing the update script:
cn1: # cd /media/dvd/FA_Suite/myAMC_FA
1.1. Make a backup: (Only needed for FlexFrame versions < 5.2A00-018 for database-
migration to PostgreSQL)
Check for errors and estimated time:
cn1: # ./fa_ezbackup.sh
Do the backup:
First Steps
FlexFrame Agents Guide 26
cn1: # ./fa_ezbackup.sh -x
2. Clear the state:
cn1: # ./fa_ezsetup.sh clear
3. Run a test using the -n option of fa_ezsetup.sh:
This option prevents any permanent changes, but simply shows
what would be done.
cn1: # ./fa_ezsetup.sh -n -v -A install
If this test did not report errors continue with step #4.
If this test failed you can't continue since the upgrade will fail.
Please fix the problem by help of the mentioned README and start again
with step #3.
4. Run the actual upgrade:
cn1: # ./fa_ezsetup.sh -L /tmp/myAMC_FA.log -S
/tmp/myAMC_FA.state clear
cn1: # ./fa_ezsetup.sh -v -A activate
5. Check the results:
cn1: # ./fa_ezsetup.sh -v status
Hint for configfiles:
During installation with EzSetup the poolspecifig config-files will be back up, so customer
specific configuration was saved.
The backup of the old version can be found as follows:
/opt/myAMC/vFF/vFF_<POOL>/Migration.<NEWVERSION>.<DATE>/Backup.<OLDVERSION>/config
The migrated files of the new version can be found as follows:
/opt/myAMC/vFF/vFF_<POOL>/Migration.<NEWVERSION>.<DATE>/config
Example:
First Steps
27 FlexFrame Agents Guide
/opt/myAMC/vFF/vFF_p1/Migration.V90A00K27.2012-08-23/MIGRATION-INSTRUCTIONS.txt
/opt/myAMC/vFF/vFF_p1/Migration.V90A00K27.2012-08-23/Backup.V90A00K26
/opt/myAMC/vFF/vFF_p1/Migration.V90A00K27.2012-08-23/Backup.V90A00K26/.info
/opt/myAMC/vFF/vFF_p1/Migration.V90A00K27.2012-08-23/Backup.V90A00K26/config
TrapTargets.xml
config-specs.xml
myAMC_FA.xml
myAMC_FA_GUI.xml
myAMC_FA_Groups.xml
myAMC_FA_Rules.xml
myAMC_FA_SD_Sec.xml
/opt/myAMC/vFF/vFF_p1/Migration.V90A00K27.2012-08-23/.info
/opt/myAMC/vFF/vFF_p1/Migration.V90A00K27.2012-08-23/config
TrapTargets.xml
config-specs.xml
myAMC_FA.xml
myAMC_FA_GUI.xml
myAMC_FA_Groups.xml
myAMC_FA_Rules.xml
myAMC_FA_SD_Sec.xml
2.2.2.4.4 Important for Installation with fa_ezsetup
The installation/upgrade using fa_ezsetup must be performed on the active control node.
2.2.2.4.4.1 For FlexFrame < V 5.2A00-018
For FlexFrame < V5.2A00-018 within an update or patch installation of FlexFrame the
migration of the used database will not be done automatically.
The database creation / updates must be performed manually before using fa_ezsetup.
Necessary databases:
Necessary creation and updates of FlexFrame Messenger and FlexFrame Domain-
Manager database won't be done automatically.
First Steps
FlexFrame Agents Guide 28
The database creation / updates must be performed before using fa_ezsetup.
This must be done in maintenance mode of an HA-Software.
The FlexFrame Messenger / FlexFrame DomainManager must be stopped.
Creation of database:
Syntax for mySQL:
mysql -u [user] -p[passwort] < SQL-script.sql
Details:
For FlexFrame Messenger database:
cd /media/cdrom/FA_Suite/myAMC_FA/Messenger_Scripts/
mysql -u [user] -p[passwort] < CreateAmcMsgmySQL_FF.9.0-03.sql
Always use the newest version of script (example above ‘9.0-03’).
For FlexFrame DomainManager database:
cd /opt/myAMC/FA_DomainManager/sql /
mysql -u [user] -p[passwort] < create-amc-MySQL.sql
Update of database:
Syntax for mySQL:
mysql -u [user] -p[passwort] < UpdateAmcMsgmySQL_FF.6.0A12-KXXTo6.0A12-KYY.sql
or
mysql -u [user] -p[passwort] < UpdateAmcMsgmySQL_FF.9.0-XXTo9.0-YY.sql
The Values for ‘XX’ and ‘YY’ must be detected in customer installation.
Value ‘YY’ is from the newest database update script.
For FlexFrame Messenger:
cd /media/cdrom/FA_Suite/myAMC_FA/Messenger_Scripts/
First Steps
29 FlexFrame Agents Guide
For FlexFrame DomainManager:
cd /opt/myAMC/FA_DomainManager/sql
Value ‘XX’ is from the installed database version value ‘DBVERSION’.
It can be detected from database like the following example value ‘DBVERSION’:
For FlexFrame Messenger: /usr/bin/mysql --user=root --password=passwort --
database=messenger --execute="select * from CNF_CONFIGGLOBAL where
APPLICATION like 0;"
+-------------+-----------------+---------+
| APPLICATION | TYPES | STRINGS |
+-------------+-----------------+---------+
| 0 | DBVERSION | 9.0-03 |
| 0 | DBCONTENT | 9.0-03 |
| 0 | POLLEVTMANCYCLE | 5 |
+-------------+-----------------+---------+
Or:
/usr/bin/mysql --user=root --password=passwort --
database=messenger --execute="select STRINGS from CNF_CONFIGGLOBAL
where TYPES='DBVERSION';"
+---------+
| STRINGS |
+---------+
| 9.0-03 |
+---------+
For FlexFrame DomainManager: /usr/bin/mysql --user=root --password=passwort --database=amc --
execute="select * from CNF_CONFIGGLOBAL where APPLICATION like 0;"
+-------------+-----------+---------+
| APPLICATION | TYPES | STRINGS |
+-------------+-----------+---------+
| 0 | DBVERSION | 9.0-15 |
| 0 | DBCONTENT | 9.0-15 |
+-------------+-----------+---------+
Or:
/usr/bin/mysql --user=root --password=passwort --database=amc --
execute="select STRINGS from CNF_CONFIGGLOBAL where
TYPES='DBVERSION';"
+---------+
| STRINGS |
+---------+
| 9.0-15 |
+---------+
First Steps
FlexFrame Agents Guide 30
All scripts between ‘XX’ and ‘YY’ must be executed.
Example for update:
/opt/myAMC/Messenger/Messenger/bin/myAMC.MessengerSrv stop
mysql -u [user] -p[passwort] < UpdateAmcMsgmySQL_FF.6.0A12-K02To6.0A12-K03.sql
mysql -u [user] -p[passwort] < UpdateAmcMsgmySQL_FF.6.0A12-K03To6.0A12-K04.sql
…
mysql -u [user] -p[passwort] < UpdateAmcMsgmySQL_FF.6.0A12-K06To6.0A12-K07.sql
mysql -u [user] -p[passwort] < UpdateAmcMsgmySQL_FF.6.0A12-K07To8.0-00.sql
mysql -u [user] -p[passwort] < UpdateAmcMsgmySQL_FF.8.0-00To8.0-01.sql
mysql -u [user] -p[passwort] < UpdateAmcMsgmySQL_FF.8.0-01To8.0-02.sql
…
mysql -u [user] -p[passwort] < UpdateAmcMsgmySQL_FF.8.0-06To9.0-00.sql
mysql -u [user] -p[passwort] < UpdateAmcMsgmySQL_FF.9.0-00To9.0-01.sql
…
mysql -u [user] -p[passwort] < UpdateAmcMsgmySQL_FF.9.0-02To9.0-03.sql
/opt/myAMC/Messenger/Messenger/bin/myAMC.MessengerSrv start
2.2.2.4.4.2 For FlexFrame ≥ 5.2A00-018
With FlexFrame ≥ 5.2A00-018 within an update or patch installation of FlexFrame there
was done a migration of the used database.
This will be done as follows:
With EzBackup the config and dynamic data of an existing database, if it is running inside
of FlexFrame, will be saved.
The data will be stored in the following directory:
/opt/myAMC/Backup/[Komponente][VersionString]
Every component will be named with version number and stored in an own subdirectory.
EzSetup as of FlexFrame V5.2A00-018 has an additional functionality for creation and
initialization of a PostgreSQL database server and to import the data saved with EzBack-
up.
The data for import will be read from the backup directory (last version) and/or from a
template directory. In the template directory as example the trap-rules of FlexFrame Mes-
senger will be stored as xml file.
First Steps
31 FlexFrame Agents Guide
2.2.2.4.5 Usage
./fa_ezsetup.sh -h
usage: fa_ezsetup.sh <general options> <command> <command specific args>
fa_ezsetup.sh 1.121
general options:
-h (show this help)
-q (quiet mode)
-v (verbose mode, may be specified multiple times)
-L <file> use specified logfile.
If <file> is a directory, the default file name will be appended.
-n do not perform permanent changes
-A automatically perform required steps
-R <dir> FA root dir (default: /opt/myAMC)
-F <dir> FF root dir (default: /opt/FlexFrame)
-D <dir> release dir
-r <release> release to install
-P <properties> release property file
-S <statefile> state file
-s do not save state
-C clear state
-f force initialisation of databases, discard existing data (messenger
and accounting) -i force installation of packages, even if same version is already
installed
commands:
clear clear installation state
info show info on current environment
status show installation status
check check pre requisites
install install packages to be updated
migrate migrate current settings to new version
autoconfig automatic configuration (if possible)
activate activate new version(s)
version show version of fa_ezsetup.sh
run run commands on application or control nodes
startapps start all FlexFrame Agents components
This will start all FlexFrame Agents applications on
the control node
as well as all agents for all pools on the application
nodes
stopapps stop all FlexFrame Agents Suite components
This will stop all FlexFrame Agents Suite applications on
the control node
as well as all agents for all pools on the application
nodes
restartapps restart all FlexFrame Agents Suite components
This will restart all FA applications on the control node
as well as all agents for all pools on the application
nodes
First Steps
FlexFrame Agents Guide 32
run fa_ezsetup.sh help <command> for command specific help
First Steps
33 FlexFrame Agents Guide
2.2.3 Configuration
The FlexFrame Agents do in most cases not require any additional configuration for use
in productive operation.
The myAMC_FA.xml file is stored when installation takes place. This file already contains
a complete parameter set for the operation of the FlexFrame AppAgents and CtrlAgents.
The services to be monitored and the reaction scenarios which run in the event of prob-
lems are parameterized in this file. The parameters and their default values are described
in section 7.5. The mode in which the agents are to operate is also configured here.
In the course of the startup, in particular the start times and stop times, the function of the
MonitorAlerts, and the times for a reboot and switchover need to be checked. The Mon-
itorAlerts are a component part of the FlexFrame basic installation.
The MonitorAlertTime must always be at least three times as great as the parameter-
ized CheckCycleTime.
In the startup scenarios, the real start, stop, restart and reboot times must be determined
individually for each service type. If the times specified for start, restart, reboot or switch-
over are not sufficient, this can result in unwanted reaction escalations.
Changes in the parameter file become effective only after the agents have been restart-
ed.
The FA migration tool enables a configuration file of an existing installation to transfer the
data automatically to a new configuration file. Parameters which, for example, were not
present in an older version of the configuration file are then initially automatically set to
their default values.
First Steps
FlexFrame Agents Guide 34
2.2.4 Starting and Stopping
During installation, links to the FlexFrame Agents’ start/stop scripts were set in
/etc/init.d/. Run this script without any options so that all available options are dis-
played, e.g. start or stop.
Example: Starting the FlexFrame AppAgent:
/etc/init.d/myAMC.FA_AppAgent start
/opt/myAMC/FA_AppAgent/myAMC.FA_AppAgent start
Example: Starting the FlexFrame CtrlAgent:
/etc/init.d/myAMC.FA_CtrlAgent start
/opt/myAMC/FA_CtrlAgent/myAMC.FA_CtrlAgent start
Example: Starting the FlexFrame FrameAgent:
/etc/init.d/myAMC.FA_FrameAgent start
/opt/local/myAMC/FA_FrameAgent/myAMC.FA_FrameAgent start
First Steps
35 FlexFrame Agents Guide
2.2.5 Configuration of generic
Generic services can be configured in two different files:
myAMC_FA_Rules.xml: Generic rules general available as best practice which
is central delivered from FlexFrame.
myAMC_FA_Rules_Custom.xml: Customer specific rules.
Therefore the customer specific rules, if used in myAMC_FA.xml, once must be migrated
in an present installation to the custom-rules.
Attention:
The whole pool specific ‚myAMC_FA_Rules.xml‘-files should be saved before call of
EzSetup, if customer specific rules are used.
During migration the ‚myAMC._FA_Rules.xml‘-file will be taken automatically from tem-
plate.
EzSetup assume that customer specific generic rules were transfered to custom-rules-file
before calling.
Case 1: Customer has no changes in generic service configuration:
The configuration of the generic services ‘myAMC_FA_Rules.xml’ must be cop-
ied manually from template in the pool configuration.
Hint: Within installation with EzSetup this will be done automatically.
Example:
cp /opt/myAMC/vFF/Common/.vFF_template/config/myAMC_FA_Rules.xml
/opt/myAMC/vFF/vFF_<POOL>/config
Case 2: Customer has changes in generic service configuration:
The configuration of generic services must be manually migrated:
Save the config of the (old) genreric services (old-‚myAMC_FA_Rules.xml‘).
First Steps
FlexFrame Agents Guide 36
Copy the file from template in the pool-config (new-‚myAMC_FA_Rules.xml‘).
Add / migrate the customer specific changes to the customer specific generic
trules in pool-config (‚myAMC_FA_Rules_Custom.xml‘).
Hint: Also within installation with EzSetup this will must be done manually.
Example:
cp /opt/myAMC/vFF/vFF_<POOL>config/myAMC_FA_Rules.xml
/opt/myAMC/vFF/vFF_<POOL>/config/myAMC_FA_Rules_OLD_with_CustomChanges.xml
cp /opt/myAMC/vFF/Common/.vFF_template/config/myAMC_FA_Rules.xml
/opt/myAMC/vFF/vFF_<POOL>/config
vi /opt/myAMC/vFF/vFF_<POOL>config/myAMC_FA_Rules_Custom.xml
First Steps
37 FlexFrame Agents Guide
2.3 FlexFrame Agents WebInterface
2.3.1 Function
The FlexFrame Agents WebInterface visualizes all nodes and services which exist in a
FlexFrame system insofar as these are monitored by an FlexFrame AppAgent. The sta-
tus, availability and messages of the FlexFrame AppAgents and FlexFrame CtrlAgents
are displayed.
2.3.2 Installation
The installation package is called myAMC.FA_WebGui-<X.Y-Z>.i386.rpm.
A prerequisite here is that an Apache Tomcat Servlet Container is installed. Currently
Tomcat >= 6.0.x is supported.
2.3.3 Configuration
Further settings can be made in the files /opt/myAMC/config/FA_WebGui.conf
(general settings, paths, cycle tymes, database settings) and
/opt/myAMC/config/amc-users.xml (user administration).
Changes require the FlexFrame Agents WebInterface to be restarted or reloaded (e.g. via
the Tomcat Service Manager) or Tomcat to be restarted or reloaded.
2.3.4 Starting and Stopping
The WebInterface can always be reached if the Apache Tomcat is running. This can
generally be started using the script /etc/init.d/tomcat6 start.
The WebInterface can then be reached at the following address:
http://<contol_node>:8080/FAwebgui/
The specified port can be changed in the Tomcat configuration file server.xml.
Prerequisites here are Mozilla >= 1.4.1 or Internet Explorer >= 6.0 and the latest Java
plugin / JRE for Oracle or IBM.
By the first start of WebGui as java web application ‘JNLP’ a warning message can ap-
pear “Java Version 1.7+ required” this message can be ignored and after accepting it the
WebGui application client will start normally.
First Steps
FlexFrame Agents Guide 38
2.3.5 FlexFrame Agents WebGui and database
The web interfaces use a connection to a database for displaying the message and per-
formance data.
FlexFrame Messenger and the FlexFrame Accounting Option ( FlexFrame DomainMan-
ager) use an own database in a database server. The FlexFrame DomainManager data-
base can be run optionally at a system outside of the FlexFrame, for not influencing the
productive FlexFrame.
Cardinally the FlexFrame Agents WebGui can work with different types of database. This
depends on the selected database in the connection property.
In FlexFrame version < V5.2A00-018 normally a mySQL database is used.
In FlexFrame ≥ 5.2A00-018 a PostgreSQL database will be used as default.
2.3.5.1 Installation and configuration of a mySQL connection
The database connection will be configured in the following file:
/opt/myAMC/config/connector.properties
Example:
#connector
#Tue Jan 29 16:16:52 CET 2013
jdbc.url.Accounting_Database=jdbc\:mysql\://localhost\:3306/amc?useGmtMillisForDatetimes\=true
connector.type.Accounting_Database={jdbc}jdbc
jdbc.url.Messenger_Database=jdbc\:mysql\:// localhost \:3306/messenger
connector.priority.Accounting_Database=10
jdbc.user.Messenger_Database=myAMC
connector.type.Messenger_Database={jdbc}jdbc
connector.active.Accounting_Database=true
jdbc.password.Accounting_Database=FlexFrame
connector.name.Accounting_Database=Accounting Database
connector.name.Messenger_Database=Messenger Database
jdbc.driver.Messenger_Database=com.mysql.jdbc.Driver
jdbc.driver.Accounting_Database=com.mysql.jdbc.Driver
connector.active.Messenger_Database=true
jdbc.user.Accounting_Database=myAMC
jdbc.password.Messenger_Database=FlexFrame
First Steps
39 FlexFrame Agents Guide
2.3.5.2 Installation and configuration of a PostgreSQL connection
The database connection will be configured in the following file:
/opt/myAMC/config/connector.properties
Example:
#connector
#Thu Jan 31 09:48:31 CET 2013
jdbc.url.Accounting_Database=jdbc\:postgresql\://localhost\:5432/amc
connector.type.Accounting_Database={jdbc}jdbc
jdbc.url.Messenger_Database=jdbc\:postgresql\:// localhost \:5432/messenger
connector.priority.Accounting_Database=10
jdbc.user.Messenger_Database=myAMC
connector.type.Messenger_Database={jdbc}jdbc
connector.active.Accounting_Database=true
jdbc.password.Accounting_Database=FlexFrame
connector.name.Accounting_Database=Accounting Database
connector.name.Messenger_Database=Messenger Database
jdbc.driver.Messenger_Database=org.postgresql.Driver
jdbc.driver.Accounting_Database=org.postgresql.Driver
connector.active.Messenger_Database=true
jdbc.user.Accounting_Database=myAMC
jdbc.password.Messenger_Database=FlexFrame
First Steps
FlexFrame Agents Guide 40
2.4 FlexFrame DomainManager
The FlexFrame DomainManager is installed on the Control Node.
The accounting and performance data collected by the FlexFrame AppAgents is automat-
ically adopted by the ITDW and can be visualized and evaluated with the help of the
FlexFrame Agents WebGUI with the Accounting and Performance management plugin.
The FlexFrame DomainManager is configured via the file
/opt/myAMC/DomainManager/config/ DomainManager.xml. Pool-specific con-
figuration is also possible. Changes to parameters in the DomainManager configuration
are dynamically recognized and adopted.
Alternatively to processing through the DomainManager, the files can also be accessed
by an external DomainManager which runs outside of FlexFrame.
2.4.1 Database connection
The FlexFrame DomainManager uses a connection to a database to store the perfor-
mance data.
In FlexFrame version < V5.2A00-018 normally a mySQL database is used.
In FlexFrame ≥ 5.2A00-018 a PostgreSQL database will be used as default.
The FlexFrame DomainManager and its database can be run integrated in FlexFrame or
optionally at a system outside of the FlexFrame, for not influencing the productive Flex-
Frame.
First Steps
41 FlexFrame Agents Guide
2.4.1.1 Integrated operation of FlexFrame Accounting option
2.4.1.1.1 mySQL database server connection
The database connection will be configured in the following file:
/root/.odbc.ini
or/and
/etc/odbc.ini
Example:
[AMC_PERF]
Driver = /usr/lib/libmyodbc3.so
SOCKET = /opt/myAMC/mySQLDBs/mysql.sock
Server = localhost
User =
Password =
Database = amc
Port = 3306
2.4.1.2 PostgreSQL database server connection
The database connection will be configured in the following file:
/root/.odbc.ini
or/and
/etc/odbc.ini
Example:
[AMC_PERF]
Driver = /usr/lib/unixODBC/libodbcpsql.so
Database = amc
Servername = localhost
Username = postgres
Password =
Port = 5432
#ksqo = No
ReadOnly = No
RowVersioning = No
ShowSystemTables = No
ShowOidColumn = No
FakeOidIndex = No
ConnSettings =
Debug = 0
CommLog = 0
First Steps
FlexFrame Agents Guide 42
2.5 FlexFrame Agents Suite Messenger
The FlexFrame Messenger will be installed at control node.
For FlexFrame Messenger there is an own documentation. Here only the database con-
nection will be documented.
2.5.1 Database connection
The FlexFrame Messenger uses a connection to a database to store the message data.
In FlexFrame version < V5.2A00-018 normally a mySQL database is used.
In FlexFrame ≥ 5.2A00-018 a PostgreSQL database will be used as default.
2.5.1.1 mySQL database server connection
The database connection will be configured in the following file:
/root/.odbc.ini
or/and
/etc/odbc.ini
Example:
[AMCMSG_V10]
Driver = /usr/lib/libmyodbc3.so
SOCKET = /opt/myAMC/mySQLDBs/mysql.sock
Server = localhost
User =
Password =
Database = messenger
Port = 3306
2.5.1.2 PostgreSQL database server connection
The database connection will be configured in the following file:
/root/.odbc.ini
or/and
/etc/odbc.ini
Example:
[AMCMSG_V10]
Driver = /usr/lib/unixODBC/libodbcpsql.so
Database = messenger
Servername = localhost
Username = postgres
Password =
Port = 5432
First Steps
43 FlexFrame Agents Guide
#ksqo = No
ReadOnly = No
RowVersioning = No
ShowSystemTables = No
ShowOidColumn = No
FakeOidIndex = No
ConnSettings =
Debug = 0
CommLog = 0
First Steps
FlexFrame Agents Guide 44
2.6 FlexFrame Agents Suite version management
2.6.1 Lifecycle process
The FlexFrame Agents Suite disposes of an version management component. The ver-
sion management component alleviate the lifecycle process of a FlexFrame Agents Suite
installation in FlexFrame and delivers information about the installed and active versions
of the individual components of the FlexFrame Agents Suite.
The version management was bundled and installed in the rpm “myAMC.FA-Suite”.
This rpm represent the build version oft the installed FlexFrame Agents Suite. So with
linux commands there can be get the info about the installed version of FlexFrame
Agents Suite.
It contains the program ‚versionmanager‘ as a script and resource files with information
for the version management.
For improvement of the lifecycle management there are the following measures:
FlexFrame Agents -Suite with build version
Versioning of the FlexFrame Agents Suite config files
Versioning of the FlexFrame Agents Suite databases
Move of versions for the central FlexFrame Agents Suite components
Version management for the FlexFrame Agents-Suite
Update- and migrations-handling for optional FlexFrame Agents Suite compo-
nents
The FlexFGrame Agents Suite consists of different components:
Components, which are running once in a FlexFrame:
FlexFrame FrameAgent
FlexFrame Agents WebGui
FlexFrame Messenger
FlexFrame LogAgent
FlexFrame DomainManager
Components, which are running once per pool:
FlexFrame CtrlAgent
First Steps
45 FlexFrame Agents Guide
Components, which are running on every application node:
FlexFrame AppAgent
The installation and activation of a new FlexFrame Agent-Suite is divided in 3 phases, to
avoid downtime and to switch back to an older version easily.
Installation of the rpm’s of a suite
Backup of the der existing config files
Migration of the config files if necassary (EzSetup)
Update of the databases (update-scripts)
2.6.2 Lifecycle management files
With the rpm of FlexFrame Agents. Suite for the version management the following com-
ponents will be installed:
Version-Dependency Datei
Version-Component Datei
Versionmanager
o Output des Versionmanagers
The rpm will install these components in the directory ‚/opt/myAMC/VersionInfo’.
2.6.2.1 Version-Dependency file
In this file the dependency to other FlexFrame Agents Suite components will be found.
First Steps
FlexFrame Agents Guide 46
The Version-Dependency file shows tabulary the components version and the dependend
component with version information. The file is a csv-file.
Example:
If the version or the dependency of a component has changed, this will be added in this
file.
First Steps
47 FlexFrame Agents Guide
2.6.2.2 Version-Components file
The management of FlexFrame Agents Suite components which belongs to a FlexFrame
Agents Suite and there version numbers will be done in the version-components file.
The file is build tabulary as csv-file and can be imported easily in Excel.
File structure of version-componens file.
MYAMCSUITE version und build number of the FlexFrame Agents -Suite
MYAMCCOMPONENT name of the associated FlexFrame Agents Suite component
VERSION_KLEVEL version with K-level
POOL_FRAME central or pool specific FlexFrame Agents Suite component
PARAM_CONFIG version of he associated config files
BASIC_CONFIG version of he associated config files
RULEFILE version of he associated myAMC_FA_Rules.xml
BINARY version for the database (binary)
DBCFG version for the database (config)
DBSTRUCT version for the database (structure)
DBCONTENT version for the database (content)
FF4S_FROM FlexFrame versionen for the FlexFrame Agents Suite compo-
nent (from)
FF4S_TO FlexFrame versionen for the FlexFrame Agents Suite compo-
nent (to)
There are many of config files in a FlexFrame Agents -Suite.
The versioning will be done in groups:
Group of base config files: Config files which have only rare changes.
Group of param config files: Config files, which include parameters for the func-
tionality of the FlexFrame Agents Suite. Example: ‚myAMC_FA.xml‘
First Steps
FlexFrame Agents Guide 48
2.6.2.3 Version manager (from FlexFrame V5.2)
2.6.2.3.1 Mission of version manager
The central mission of the version manager, which simplifies the processing and support
of a FlexFrame landscape, is the dynamic detection of all active FlexFrame Agents Suite
components in the different pools of a FlexFrame landscape. There Info will be written to
standard output, or can be redirected into a file.
The version manager (script) will be installed with the FlexFrame Agents Suite rpm.
The version manager shows the active FlexFrame Agents Suite components, the version
of the FlexFrame Agents Suite config files und the database version.
It determines if optional components of the FlexFrame Agents Suite are active.
The info is shown at standard output and can be redirected to a file.
This report can be send to FlexFrame- or FlexFrame Agents Suite Support as info about
the customers system.
Usage:
cn1-a:/opt/myAMC/scripts # ./fa_versionsMgr.sh -?
Usage of fa_versionsMgr.sh:
./fa_versionsMgr.sh -h shows this help
./fa_versionsMgr.sh -? shows this help
./fa_versionsMgr.sh [-s separator] [-f outputfile]
default separator is ;
default outputfile is /opt/myAMC/scripts/fa_versionsCustomer.csv and stdout
cn1-a:/opt/myAMC/scripts #
First Steps
49 FlexFrame Agents Guide
2.6.2.3.2 Output of version manager
The output has the following structure:
FRAMEID Frame-ID
FFVERSION FlexFrame-Release
POOL Poolname
COMPONENT FlexFrame Agents Suite component
VERSION Version and k-level
CNFPARA Version of parameter files
CNFBASIC Version of basic configuration files
RULES Version of myAMC_FA_Rules.xml resp. Logdefini-
tions.xml
CUSTOM_RULES Customer specific Rules active ?
CUSTOMIZED Customer specific changes ?
VERSIONBIN „VersionBin“ from FlexFrame Messenger.vrs resp. Flex-
Frame DomainManager.vrs
VERSIONCFG „VersionCfg“ from FlexFrame Messenger.vrs resp. Flex-
Frame DomainManager.vrs
VERSIONDBSTRUCT „VersionDbStruct“ from FlexFrame Messenger.vrs resp.
FlexFrame DomainManager.vrs
VERSIONDBCONTENT „VersionDbContent“ from FlexFrame Messenger.vrs
resp. FlexFrame DomainManager.vrs
CHK_CNFPARA Check of „CNFPARA“
CHK_CNFBASIC Check of „CNFBASIC“
CHK_RULES Check of „RULES“
CHK_VERSIONBIN Check of „VERSIONBIN“
CHK_VERSIONCFG Check of „VERSIONCFG“
CHK_VERSIONDBSTRUCT Check of „VERSIONDBSTRUCT“
CHK_VERSIONDBCONTENT Check of „VERSIONDBCONTENT“
CHK_RPM Check of „VERSION“
CHK_SUMMARY overall summary
First Steps
FlexFrame Agents Guide 50
Example: Output of version manager:
fa_versionsCustomer.csv
First Steps
51 FlexFrame Agents Guide
2.7 Usage of Pool-Master
A FlexFrame as a physical unit of servers, storage and network can be logically divided
into separate pools. Each pool has its own set of configuration data and is thus inde-
pendent of any other pool in a FlexFrame. For each pool a number of parameters needed
to configure a FlexFrame individually for each pool.
With the pool master functionality, the configuration data from a pool that has been con-
figured individually for a FlexFrame system, are copied to a so-called pool master.
When creating a new pool, the configuration data can be used from the pool master to
generate a new pool.
The usage of Pool-Master allows the automation of steps for FlexFrame Agents Suite
after pool creation.
Therefore a script can be used to create a Pool-Master from an existing pool or to
write/update a pool configuration from Pool-Master.
The script provides the following commands:
Create Pool-Master (directories/files) from existing Pool
Create/Update Pool (directories/files) from Pool-Master
Delete Pool-Master
Important hint:
The Pool-Master contains config files which may differ from version to version, as well as
the info which version of FlexFrame AppAgent and FlexFrame CtrlAgent will be started.
Therefore it is highly recommended after each update/upgrade of FlexFrame orFlex-
Frame Agents Suite to update the Pool-Master from an active pool which was migrated
during update/upgrade process.
Poolspecific configuration should be not passed to the Pool-Master, because they will
force needless queries and error messages in the FlexFrame Agents.
Example: „Individual Nodes“, „Management-Blades“, „Filer-Check“, …
First Steps
FlexFrame Agents Guide 52
2.7.1 Create Pool-Master from existing pool
Example: Create Pool-Master from Pool1
In this step the config files of pool ‚Pool1‘ will be copied in the Pool-Master directory
(‚/opt/myAMC/vFF/Common/.vFF_Master‘) as well as the necessary directory structures
will be created.
After that, the pool depending config values will be replaced in Pool-Master:
“TrapTargets” of CN will be replaced with placeholders.
File: TrapTargets.xml
“Managementblades” remain (default) or will be deleted (‘-b’).
File: myAMC_FA.xml
“Individual nodes” will be detected from LDAP (default) or will be deleted (‘-i’).
File: myAMC_FA.xml
“Filer-check” The filer name will be replaced with placeholders.
File: myAMC_FA.xml
2.7.2 Create/Update pool from Pool-Master
Example: Deploy Pool_Master to Pool NEU
vFF
opt/myAMC
config
Create Pool-Master from Pool1
Common/.vFF_template/config
vFF_Pool
1
Config
Files
Pool Pool1
Config
Files
Pool-
Master
Config
Files
Template
Common/.vFF_Master/config
First Steps
53 FlexFrame Agents Guide
In this step the config files of Pool-Master will be copied to pool ‚NEU‘in the directory
(‚/opt/myAMC/vFF/ vFF_NEU‘)as well as the necessary directory structures will be creat-
ed.
After that, the pool depending config values will be replaced by the pool specific values:
“TrapTargets” of CN will be detected from LDAP.
File: TrapTargets.xml
“Managementblades” remain (default) or will be deleted (‘-b’).
File: myAMC_FA.xml
“Individual nodes” will be detected from LDAP (default) or will be deleted (‘-i’).
File: myAMC_FA.xml
“Filer-check” The filer name will be detected from LDAP.
File: myAMC_FA.xml
vFF
opt/myAMC
config Create Pool NEU from Pool-Master
Common/.vFF_template/config
vFF_Pool1
Config
Files
Pool NEU
Config
Files
Pool-
Master
Config
Files
Template
Common/.vFF_Master/config
config
vFF_NEU
First Steps
FlexFrame Agents Guide 54
2.7.3 Usage of ‘fa_MasterPoolMgr.sh’
Usage:
fa_MasterPoolMgr.sh -h
usage: fa_MasterPoolMgr.sh <args>
fa_MasterPoolMgr.sh -h
fa_MasterPoolMgr.sh -r
fa_MasterPoolMgr.sh -d toMaster|fromMaster -p <poolname> [-b] [-i] [-m writeTrapTar-
gets|modifyTrapTargets]
Description of arguments:
-h (show this help)
-r remove Pool-Master
-d toMaster|fromMaster specify direction (toMaster, fromMaster)
-p <poolname> specify poolname
[-b] Remove the management blade configuration.
(default = Not remove the management blade configuration).
[-i] Remove the 'individual nodes' (Hypervisor-server) configuration.
(default = Get the 'individual nodes' (Hypervisor-server) configura-
tion from LDAP.
[-m writeTrapTargets|modifyTrapTargets] specify the mode for TrapTargets.xml (default=write)
'modifyTrapTargets' requires correct configuration of trap targets in pool before make a Pool-
Master.
Please ensure, that this script will be started at Ctrl-Node.
2.7.4 Automation of steps during pool creation
The script for Pool-Master management can be used for automation of steps during the
pool creation process.
The script ‚fa_MasterPoolMgr.sh‘ can be called during pool creation from „ff_pool_adm.pl
–op add”. Before calling it should be checked whether a Pool-Master exists. If necessary
the Pool-Master could be created from first pool.
The script ‚fa_MasterPoolMgr.sh‘ can also be used for configuration of several pools of
an existing FlexFrame, for deploying the Pool-Master configuration to the existing pools.
Keep in mind, that the script only makes automated adaptions for the pool specific values
described above. Other pool specific things must be changed manually.
55 FlexFrame Agents Guide
3 Introduction
3.1 FlexFrame Agents Suite Architecture
FlexFrame Agents Suite is a powerful component for high-availability operation of sys-
tems with distributed instances. A FlexFrame solution consists of storage, application
servers and redundant Control Nodes. This product has been implemented for this solu-
tion comprising storage, servers and connectivity. It enables fast and flexible setup of
solutions which offer autonomous functions to simplify and provide flexibility for operating
applications. The figure below shows an overview of the FlexFrame architecture and the
associated FlexFrame Agents Suite components:
The benefit of the FlexFrame Agents Suite solution lies in the flexibility for integrating new
nodes and instances without changing the configuration.
Components of FlexFrame Agents Suite:
FlexFrame Application-Agents (FF AppAgent)
FlexFrame Control-Agents (FF CtrlAgent)
FlexFrame Frame-Agents (FF FrameAgent)
FlexFrame Agents WebGui (FA WebGui)
FlexFrame Messenger (FF Messenger)
Introduction
FlexFrame Agents Guide 56
FlexFrame DomainManager (FF DomainManager)
FlexFrame LogAgent (FF LogAgent)
The FlexFrame Agents Suite components permit highly available, semi-autonomous
operation of distributed applications. In principle the instances can be distributed to any
number of nodes within a FlexFrame solution. The individual services are monitored via
FlexFrame Agents. By default, the FF AppAgents currently support SAP central instanc-
es and SAP application instances, as well as SAPDB and Oracle database instances.
The FF FrameAgents are used for monitoring at the control node. The central monitoring
of filesystems and the performance monitoring of the control node requires the FF
FrameAgents which are installed at the control nodes.
Introduction
57 FlexFrame Agents Guide
3.1.1 FlexFrame Agents Suite Visualization and Reporting
The FlexFrame Agents WebGui and the FlexFrame Agents Viewer are the components
for the visualization of the availability and performance data. The FA WebGUI is an applet
to be used in a webbrowser. The FA Viewer is a stand-alone application, which can run
on any Linux or Windows PC. The two components are necessary because both con-
cepts add their own advantages. The applet component can be used on every PC with
connection to the FlexFrame Agents Suite server component without any installation
steps, but applets have some restrictions. An applet can not use the reporting functionali-
ty and can not connect to multiple frames with one applet.
The visualization architecture is shown in the following picture.
Introduction
FlexFrame Agents Guide 58
3.1.2 FlexFrame Agents Suite virtual server and hypervisor nodes
Since FlexFrame version V5. 0 virtual servers and hypervisor nodes can be visualized in
the FlexFrame viewer and WebGUI. The existing web service component running in the
Tomcat server periodically collects data of all virtual servers and their hosts.The infor-
mation is shown in the system pool “FlexFrame”.
As of FlexFrame V5.3 or FlexFrame Orchestrator V1.0 ESX or KVM can be used as hy-
pervisor.
On the hypervisor the virtual servers ESX-VM and KVM-VM are running.
The representation and visualization is basically identical for both types of hypervisors.
The corresponding overlay icon is depending on type "ESX" or "KVM".
The FlexFrame CtrlAgent monitors the virtual servers belonging to its pool. If a virtual
server is not available any more, it is switched offwith the shutdown facility for virtual
servers.
If powering down a virtual server fails, there are several cases:
Introduction
59 FlexFrame Agents Guide
the hypervisor node runs other VMs without problems
the hypervisor node does not host any other VMs
other VMS on the hypervisor node also failed.
An hypervisor node may host virtual servers from multiple pools.
As the FlexFrame CtrlAgent knows only nodes running in its own pool, but the hypervisor
node possibly also operates virtual nodes for other pools, the permission to turn off the
hypervisor nodes from the central hypervisor nodes monitoring service (part of the Flex-
Frame Agents WebGui server is part of the Tomcat) is requested. If the permission could
be obtained, the external switchover can be performed as planned. Can the hypervisor
node not be switched off, for example because of running other VMs, the external switch-
over is not performed.
Introduction
FlexFrame Agents Guide 60
3.2 Pool Creation and Grouping
FlexFrame permits pool creation and grouping functions to be implemented.
3.2.1 Virtual FlexFrame Pools
A pool is the assignment of hardware resources to a virtual FlexFrame pool. From the
viewpoint of autonomy and of the high-availability functions, an Autonomy pool is an
independent structure. In a standard installation of Version 1, all resources of a Flex-
Frame solution are managed in a single pool. Configuration of the pools takes place di-
rectly with the configuration of FlexFrame in the LDAP. The FlexFrame AppAgents ascer-
tain the pool affiliation at startup. Configuration of the FlexFrame agents always relates to
one pool, i.e. there is one directory tree with the parameters and configuration data for
each pool.
In a pool, the FlexFrame Agents provide the autonomous functions restart, reboot and
switchover of services and nodes. These reactions no longer relate to all nodes of a
FlexFrame solution, but only to the set of nodes which belong to the same pool.
Pool creation results in virtual FlexFrame Agent Suite pools being created, each of which
performs autonomous functions independently of other pools which exist in the same
FlexFrame solution.
A FlexFrame pool always consists of one Control Agent and n Application Agents. Each
Control Agent is responsible only for the Application Nodes which belong to its pool and
shares a joint config and data directory with its Application Nodes. For each pool it is thus
possible to parameterize autonomous behavior which is independent of other pools.
The flexibility and security of a virtual FlexFrame pool is based on two major features
which the FlexFrame Agents provide:
A Control Agent for each virtual FlexFrame pool
Each Application Agent is provided with a flexible assignment to the pool and
thus to the Control Agent with which it interworks.
The use and interleaving of these two new options with the FlexFrame basic functionality
offers a large number of new options to enhance the flexibility in server farms.
The virtual FlexFrame pools provide the option of simple and secure operation of multiple
clusters which run in parallel and simultaneously in a distributed IT infrastructure.
Introduction
61 FlexFrame Agents Guide
A virtual FlexFrame pool offers the advantage of complete separation of all reactions and
the associated parameter sets for the start and stop times. This can also be completely
disabled for a virtual pool (e.g. for service and maintenance) without affecting any other
virtual pool which is running in parallel.
A virtual FlexFrame pool does not share its FlexFrame Agents with any other virtual Flex-
Frame. In this way, depending on the configuration, the virtual FlexFrame pools could
use different binary statuses.
The pools FlexFrame and Adminpool are special pools.
The Adminpool is a pool for spare nodes which are pool independent..
The pool FlexFrame is a system pool established on the control node to collect infor-
mation for the whole frame.
Introduction
FlexFrame Agents Guide 62
3.2.2 Grouping
FlexFrame offers grouping functions for flexible server farming. Grouping enables nodes
and services within a pool to be assigned to different groups. A group is therefore always
a part of a virtual FlexFrame pool.
The configuration information for the groups is stored in the myAMC_FA_Group.xml file.
The entries in this file are taken them over from the LDAP directory. The configuration is
always done via LDAP by specific assignment. The ability of the generic assignment of
the FlexFrame Agent is not supported in FlexFrame.
3.2.2.1 Configuration in the LDAP Directory
The group information are stored in the LDAP directory. When the agents are started, the
group information is read directly from the LDAP directory and be stored in an xml file myAMC_FA_Group.xml.
3.2.3 Pool Types (Pool Independent Spare)
Pools can differ in admin pools and productive pools. The FlexFrame Agents have differ-
ent rules depending on in which pool type the corresponding FF CtrlAgent is acitve.
The admin pool in contradiction to the productive pool has the function to provide spare
nodes for productive pools. If a node fails, the FlexFrame Agent requires a spare node
from spare pool. Saving spare nodes and provide them pool overlapping for multiple
pools represents the ability of that functionality.
The property of pool-overlapping spare nodes can be activated/deactivated for each pool.
The pool type is needed for the use and control of admin pools.
Pool type
productive pool (default value)
Adminpool
With a further parameter can be controlled after which time period the spare nodes in the
admin pool start cyclicly with the check of availability and performance and stop again
automatically.
Both Adminpool and productive pool in principle use the same versions of FlexFrame
Agents. The behavior of the agents depends on the configuration.
3.2.3.1 Spare Node from Adminpool
In FlexFrame pools on the level of the FlexFrame Agents there is no information ex-
change. By the implementation of pool-overlapping spare nodes they can operate in a
Introduction
63 FlexFrame Agents Guide
collective pool. When a pool needs a spare node, it will be requested from the spare-pool
(Adminpool) and add into the pool/group, which need it.
In a standard FlexFrame installation each pool has one FF CtrlAgent. Therefore this FF
CtrlAgent is only concerned for one pool. There is no interaction with FF CtrlAgents of
other pools.
With the attachment of admin pools, which provides spare for other pools, there is an
information channel in form of a collective directory. In this directory the FF CtrlAgents
can achieve information of the order of the spare nodes.
Analogical to the so far implemented procedure for reboot and switchover, there will be a
pool-overlapping directory, in which all FF CtrlAgents from each pool can read and write.
Every pool has its own FF CtrlAgent. This agents work focuses after rules. With the im-
plementation of productive and admin pools as pool type parameters the differing rules of
the FF CtrlAgents become activated dependent on which role the FF CtrlAgent is used.
3.2.3.2 Architecture and Sequence - Pool Independent Spare Node
Procedure spare node from spare pool :
In a productive pool a node fails. The productive pool has no available spare node and no
other node applies for the task. That is proved from the FF CtrlAgent by means of the
configured switchover rules.
DB
SCS
copy Testament
Testament
SCS
SpareAP
Spare
Spare
Spare
Controlnode
Cont Cont Cont Cont
no local candidate
SparePool eintragen
Sparenode
liste
Inite_sparenode
Delete_sparenode
Change_sparenode
Init_Sparegroup
Introduction
FlexFrame Agents Guide 64
The check can have the following results:
The failed node or service has low prior and there is no need to make the node/service
available again.
The services on the failed node have high prior, there is a need for a spare node to make
the services available again. If in the pool/group of the failed noted exist no spare node,
that means there is no applicant, then the FF CtrlAgent decides to require a spare node
from the spare pool.
The FF CtrlAgent in the admin pool has a selection rule to identify the spare node to
transfer to the productive pool. For that the FF CtrlAgent manages a dynamic list with all
activated spare nodes and their selection attributes. A node in the adminpool is activated
as spare node through a successful boot process in the spare group of the adminpool.
The spare node list is not persistend. FF CtrlAgent initiates every cycle time a boot test of
every node in the spare group of the adminpool. Only nodes which can be booted suc-
cessfully in the MaxReboottime remain in the spare pool list.
The FA WebGui visualizes all nodes in the admin pool. To differentiate activated spare
nodes there is an additional overlay icon in the node tree. Nodes without such an overlay
icon in the tree are not useable as pool-overlapping spare nodes in the admin pool.
3.2.3.3 Competing Access to Spare Nodes
The FF CtrlAgent of a productive pool transmits all requirements which can not be ac-
complished locally to the spare pool. For the present the FF CtrlAgent does not imple-
ment a further preselection.
As long as no specific parameter exists all pools have priority 1 through default. Thereby
applies first come, first served. With heavier failure scenarios etc. demands for spare
nodes will come from many systems simultaneously. If thus all pools have the same prior-
ity then the system or the service priority applies which already can be configurated to-
day.
3.2.3.4 Providing of Spare Nodes in the Adminpool
The installation and configuration of a spare node in an admin pool for the use as pool-
overlapping spare node is described in the FlexFrame Admininistration and Operation
guide. A spare node in the spare pool is only visible and usable if it has been running at
least once and thus enabled the FlexFrame Agents to dynamically determine the output
data of the node. The node will be recorded in a list administered by the FF CtrlAgent
during the initialization.
Using a parameter the FlexFrame Agents are able to cyclicly check and then stopp again
the availability and the output data of the spare nodes through booting.This guarantees
the actuality of the sparenodelist and the belonging outputdata.
Introduction
65 FlexFrame Agents Guide
This init- or checkrun can be accomplished for all spare nodes of the spare pools through
a script. Depending on the parameter one or all nodes of the spare group will be initia-
lized.
Introduction
FlexFrame Agents Guide 66
3.3 Service Classes
The service classes are required for the prioritized operation of individual services or
systems. A service is defined as an application instance which must be identified unam-
biguously and which can be started and stopped individually, e.g. central instance, appli-
cation instance or database instance (CI, AP DB).
A service class defines the minimum requirements which must be provided when services
are taken over in the event of a switchover.
When multiple nodes fail simultaneously, the spare nodes in the group take into account
the priorities of the services which have failed. First all services with priority 1 are taken
over, and only then all services with a lower priority. It will be possible to extend the at-
tributes of a service in the future, as already shown in the examples (e.g. operating sys-
tem).
A system is a logical unit consisting of multiple service instances which together define a
system. In an SAP system these comprise the database instance, central instance and
application instances.
The following attributes are defined in the service classes:
service-priority
service-powervalue
In the future it will be possible to enhance such a service class by further attributes which,
for example, define the operating system required by a service or the number of CPUs or
the performance requirement of the service.
3.3.1 Service Priority
The highest service priority is 1. Every service is assigned this priority by default, i.e. if no
service classes are defined, all services have the priority 1. The higher the number, the
lower the priority of a service.
Priority 0 has a special status. Setting priority 0 for a service class enables the autono-
mous functions to be disabled for a service.
The service priority is evaluated for all autonomous reactions. If, for example, a service of
a productive system and a service of a test system are running on the same node and the
test system’s service is assigned priority 5, this service is not executed because the pro-
ductive system’s service which is functioning without error has the higher priority of 1.
3.3.2 Service Power Value
The service power value specifies for a service a performance number which defines the
maximum performance (powervalue [SAPs-äquivalent]) required by this service.
Introduction
67 FlexFrame Agents Guide
This value is used for takeover scenarios; the add rule requires the service power value
A failed service with a performance value of 50 can, for example, also be taken over by a
node which still has at least 50 of its maximum performance number free.
3.3.3 Class Creation Rules
A service belongs either to the default class which always exists or it can be assigned
unambiguously to another class by evaluating the aforementioned variables.
3.3.4 Testament Types
The switchover scenarios use testaments to transport the service information to other
nodes. The creation of the testaments can be node-based or service-based. With node-
based testaments, all services of a node always come together to the takeover node.
With service-based testaments the services could be taken by different nodes.
The parameter for the testament type and the takeover rules therefore strongly influence
the possible takeover scenarios.
3.3.4.1 Service-specific Testaments
Service-specific testaments are used for services which require individual takeover sce-
narios.
3.3.4.1.1 Enqueue Service in Case of Replicated Enqueue
The enqueue service with replicated enqueue service has its own service type. This spe-
cial testament is built dynamically if a replicated enqueue service exists. For this service-
specific testament, the service-based takeover rule applies. Only nodes with a replicated
enqueue service can apply.
Introduction
FlexFrame Agents Guide 68
3.4 FlexFrame Agents Configuration, Work and LogFiles
The figure below provides an overview of the configuration and log files which are gener-
ated by FlexFrame Agents components and stored on the common file system. These
files also form the permanent memory which is required, for example, to restore the ser-
vices needed when a system is rebooted.
In a standard FlexFrame installation the following directories exist for FlexFrame Agents
Suite. All the directories and files required for the FlexFrame Agents Suite software are
located here.
directory structure: /opt/myAMC/
and /opt/local/myAMC/
./FAwebgui FA web interface
./vFF/Common/.vFF_template.<version> Template for pool directories
./vFF/Common/.vFF_Master Pool-Master
./vFF/SpareRequest Requests for spares from sparepool
./vFF/vFF_<pool_name> Pool directory
config Pool-specific configuration data
log Log files
log/AppAgt Log files of Application Agents
log/CtrlAgt Log files of Control Agent
log/FrameAgt Log files of Frame Agent
log.common Common log files
data/FA/ FA data directory
data/FA/blackboard Blackboard directory
data/FA/livelist Live list
data/FA/servicelists Service files of all nodes
data/FA/servicelogs Service files of all nodes (history)
data/FA/xmlrepository XML files for the web interface
data/FA/reboot Reboot files for all nodes
data/FA/switchover Switchover files for all nodes
data/FA/performance Performance and accounting files
A detailed description can be found in chapter 10.3.
Introduction
69 FlexFrame Agents Guide
3.5 Systems
A system is based on several services which belong to a logical group. SAP systems are
an example of logical systems. The services of such a system can be distributed in one
pool on several Application Nodes.
The FlexFrame AppAgents identify the services and the system they belong to autono-
mously and they identify standard SAP services automatically.
3.5.1 Service Types
The FlexFrame AppAgents are able to identify standard SAP services and the hierarchy
in a logical SAP system. For these services the FlexFrame AppAgents do the autono-
mous reactions restart, reboot and switchover.
DB, CI, APP, J, JC, SCS, ASCS, ERS, LC, MDS, MDIS, MDSS, MDS71, XPR, ...
Version 3.0 of the FlexFrame Agents can monitor the service types SCS and ASCS with
replicated enqueue service (ERS). The detection of SCS/ASCS with or without replicated
enqueue service is done automatically.
As of FlexFrame version 5.0A and FlexFrame Agent version 9.0, the additional service
types MDS, MDIS, MDSS, MDS71, XPR, ... are supported.
As of FlexFrame 5.3 / FFO 1.0 the additional service types BOBJ 4.1 and SAP HANA are
supported.
3.5.1.1 Replicated Enqueue Service ERS
Version 3.0 of the FlexFrame Agents can monitor the service types SCS and ASCS with
replicated enqueue service (ERS) scenarios.
For an SCS or ASCS service there is an ERS on which the enqueue table is replicated.
The detection of SCS/ASCS with or without ERS is done automatically.
Details of the reaction rules are described in chapter 4.
3.5.1.2 Solution Manager Diagnostic (SMD)
Version 9.0 of the FlexFrame Agents can monitor the Solution Manager Diagnostic
(SMD).
As of FlexFrame V5.2A00-018 the FlexFrame Agents support host based SMD agents of
version 7.3.
SMD 7.3 agents can work as host based and as service based agents. These require the
existence of a FlexFrame version ≥ 5.2A00-018.
Introduction
FlexFrame Agents Guide 70
In FlexFrame < V5.2A00-018 the SMD 7.3 agent can only be used with project specific
release and a special configuration. Details of the reaction rules are described in chapter
4.
3.5.1.3 Live Cache
With version 3.0 of the FlexFrame Agents it is possible to integrate the live cache into the
standard autonomy scenarios. The FlexFrame Agents offer the standard autonomy func-
tions restart, reboot and switchover for the live cache.
A speciality of the live cache is the possibility to stop it from the SAP GUI. For this restart
scenario you have to check the restart times of the live cache, otherwise this scenario
cannot be diagnosed (recognized) as a fault of the live cache.
3.5.1.4 BOBJ 3.2 and BOBJ 4.0
The service BOBJ 4.0 can be detected since FlexFrame V5.1A.
The FlexFrame Agents therefore has in FlexFrame V5.1 an external rule in file my-
AMC_FA_Rule.xml.
Since FlexFrame V5.2A this rule for BOBJ 4.0 is no more in file, it is an internal rule.
At migration from FlexFrame V5.1A to V5.2A the rule file myAMC_FA_Rule.xml must be
replaced with the rule file for FlexFrame V5.2A.
This will be done automatically during installation with EzSetup.
Important hint:
Also if FlexFrame Agents for FlexFrame 5.2 are used in a FlexFrame 5.1 the my-
AMC_FA_Rules.xml may not include an active BOBJ40 rule.
A potential includes rule must be inactive.
Example for active BOBJ40-rule:
<section type="Service" name="SRV_BOE4">
<description> Created: Feb 2012 </description>
<attribute name="Active">1</attribute>
Example for inactive BOBJ40-rule:
Introduction
71 FlexFrame Agents Guide
<section type="Service" name="SRV_BOE4">
<description> Created: Feb 2012 </description>
<attribute name="Active">0</attribute>
3.5.1.5 SAP HANA
The SAP HANA database server needs a lot of memory and might need a very long time
for getting started.
Therefore, the default start-/stop-/restart- monitoring times were significantly increased.
Service_MaxRestartTime = 3100 Sek
Service_MaxStartTime = 2700 Sek
Service_MaxStopTime = 400 Sek
In case of SwitchOver there are different TakeOver strategies (see chapter 4.4.2) without
hardware check.
In case of "pool independent spare" procedure (see chapter 4.4.4) a similar AN was se-
lected according to certain rules (CPU-arch, HBA, 10GBES, powervalue [SAPs-
äquivalent]).
In both cases there is no check of the necessary memory expansion. Thus, it is quite
possible that an AN with too little memory takes over the failed services and therefore
SAP HANA may not start.
Hint:
In case of switch over an AN (SwitchOver or pool independent spare) it is not guaranteed
that a node with similar memory expansion takes over the services of the failed node.
There is also the possibility that an AN takes over with too little memory.
Therefore, the user must ensure that a suitably powerful node takes over the services of
the failed node.
This can be done by appropriate grouping of SAP HANA services, or general use of simi-
lar nodes.
Homogeneous hardware with SAP HANA certified node should be used.
Introduction
FlexFrame Agents Guide 72
3.5.1.5.1 SAP HANA Single Instance Installation
SAP HANA as a single installation is monitored by the FlexFrame Agents and in case of
failure of the node it is switched to another node according to the set of high availability
rules.
On the failure of individual subservice of a SAP Hana instance there is not responding by
FlexFrame Agents, because the SAP Hana internal audit rules automatically detect the
problem and even respond.
A SAP HANA installation can not be started, stopped or restarted from the context-
sensitive menu in FlexFrame Agents WebGui.
3.5.1.5.2 SAP HANA Master Replika Installation
SAP HANA as the master replica installation with one or more instances of replication is
monitored by the FlexFrame Agents and in case of failure of the node, the node is turned
off according to the set of high availability rules with the usual methods and the Switch-
Over file is published.
The SAP HANA service is not started from the FF Agents on another server, since taking
over the master role with SAP HANA resources automatically, according to one of the
HANA replica instances.
The failed SAP HANA node is set at shutdown cycle process, reboots and is back as a
spare node available.
In FlexFrame Agents WebGui a SAP HANA master / replica installation is displayed with
extended attributes, so that in the instance line, the respective roles is displayed directly.
In Systemtree the SAP HANA replica an overlay icon "R" Obtaining instances, the corre-
sponding master instance receives the overlay icon "M".
In Systemtree the SAP HANA replica instances get an overlay icon "R", the correspond-
ing master instance receives the overlay icon "M".
In the instance view the Replica-ID, the Node-ID and the respective role is displayed.
A SAP HANA installation can not be started, stopped or restarted from the context-
sensitive menu in FlexFrame Agents WebGui.
Introduction
73 FlexFrame Agents Guide
3.6 Generic Services
Generic services are services which are not integrated into the autonomy rules. With
generic services it is now possible to integrate other virtualized services into the autono-
my monitoring and reaction scenarios.
A generic service is a logic application suite consisting of one or more subservices.
For this purpose a generic service is defined through a set of parameters which are used
for its identification and which generate the service states. The description and definition
of a service is arranged in several models:
Service state model
Service detection model
Service reaction model
3.6.1 Service State Model
The autonomy scenarios are based on an defined state model. The standard service
state model uses the following states:
Starting
Running
Restarting
Stopping
Error
Warning
NoWatch (Attribute of a service)
Reboot
SwitchOver
Some states are detected automatically, but for setting other states there is an external
event neccessary. This will be done due to an eventscript, which will be started during
start of the startscript of SAP service. The eventscript will be visible for a defined time (30
sec) in the process list.
The eventscript is called monitor_alert / monitor_info and will be called with pa-
rameters which indicate the service calling for the external state (start, stopp, restart,
watch, nowatch).
The state changes are initiated through events from an event script or through detection.
For a generic service, implementation and integration in the standard start/stop procedure
of the service is necessary. The standard state model knows the following events:
Start
Stop
Introduction
FlexFrame Agents Guide 74
Restart
Error
Watch
Nowatch (Attribute of a service)
3.6.2 Service Detection Model
The service detection model provides the basis for identifying the service and building the
state model. A service detection model needs the parameters for the identification of the
service components. The parameters are the subservice and the processes of the sub-
service. For this there are parameters for hierarchy and process count. There is also a
process filter and exception rules, to avoid ambiguities.
3.6.3 Service Reaction Model
The service reaction model defines the reaction and the connection to the start, stop and
restart scripts. The reaction API has the parameters Script and Parameter:
Script The call reference for the script
Parameter Set of parameters for the script
The FA-AppAgents reaction API provides a set of parameters, which can be used as call
parameters in service-specific scripts:
@{SIDENT}@ Parameter for the (SID, in upper case)
@{sident}@ Parameter for the (SID, in lower case)
@{SRV}@ Parameter for service name (in upper case)
@{srv}@ Service name
@{SRVDISP}@ Display service name (in upper case)
@{srvdisp}@ Display service name
@{NIDENT:2@ Instance number (two-digits)
Introduction
75 FlexFrame Agents Guide
3.7 FlexFrame Performance and Accounting Option
The FlexFrame Agents provide optional performance and accounting data. The agents
collect node-, service- and group-based information.
The FlexFrame performance and accounting option requires the activation of additional
services on the Control Node. This service does a performance and accounting calcula-
tion of the raw data.
The FlexFrame Agents produce performance and accounting collections in the data di-
rectory of the pool. There are 3 types of collect data
Collections per node with the name pattern Perf_Node~<node_name>.prf.<number>.col
Collections per service group with the name pattern Perf_Group~<node_name>.prf.<number>.col
Collections per service with the name pattern Perf_Service~<SrvType>~<SID>~<ID>.prf.<number>.col
The number and size of the collections produced by the FlexFrame Agents can be ad-
justed. In the standard adjustment there are in each case 10 collections per service or
node installed. This results in a ring buffer of data automatically reorganized by the
agents. For the sizing it is possible to calculate the required storage size through the
number of nodes and the size of the report cycle.
The parameters of the DomainManager and of the backup routine have to be configured
in a way that the raw data can be safely processed before being overwritten by the Flex-
Frame Agents.
Introduction
FlexFrame Agents Guide 76
The following graphic shows the architecture of the performance and accounting option.
3.7.1 Performance Option
The performance option measures several performance values. For all measured values
there is a minimum, average, maximum and total value. This data is supplied in absolute
as well as relative form. The performance option enables monitoring and evaluation of the
server and services over a longer period of time. For every node the following data are
available as a minimal, average and maximum value:
load of SAP-, database- or generic services
other services
Machine idle
By using the generic services, the granularity of the performance values will be
increased. The data of the performance and accounting option can be directly visualized
with the FlexFrame Agents WebGUI with performance and accounting management
plugin. The granularity of the view and the timespan can be freely defined.
Service Groups
Services are combined to form groups through specific criteria. This enables the group-
aggregated evaluation of the recorded data.The collected data is aggregated per report
cycle and is created for every node. By default the following groups exist:
SAP SAP services
DB Database services
IDLE Share (proportion) of the free CPU capacity
Introduction
77 FlexFrame Agents Guide
OTHER Sum of all processes not belonging to a defined group
It is possible to define further services and assign them to existing or new groups.
3.7.2 Accounting Option
The accounting option is, like the performance option, an optionally activatable part of the
FlexFrame Agents. The production of the accounting data is a multistage process deter-
mining accounting data through aggregation and analysis of the recorded raw data.
The accounting data is determined on the basis of powervalue [SAPs-äquivalent] values.
SAPs is the measured size used for the sizing of a server for the SAP operation. SAPs
values are only available within the scope of a defined benchmark with defined SAP
transactions.Therefore only SAPs equivalents can be produced and calculated during the
operation. For this purpose the agents dynamically evaluate information on the SAP
version and hardware-workload data and use this to calculate the powervalue [SAPs-
äquivalent] values.
Important parameters for the accounting are detection and report cycles. The detection
cycle defines the number of measurements within a report cycle. The minimum, maxi-
mum and average values are calculated on the basis of individual measurements for a
report cycle. The detection cycle therefore always corresponds to the detection cycle of
the FlexFrame Agents, which is also parametered for the autonomy function.
The following figure shows the ascertainment and calculation of values with regard to the
detection cycle and report cycle.
Time-
stamp
Host 1Backup
Host 7ASCSP22
Host 3
Host 6
Host 5
Host 4
Host 3
Host 2
Host 1
Hostname
xy
SCSP22
JCP22
JP22
APPP22
CIP22
DBP22
SAPS
%
SAPS
abs
Mem
%
Mem
Kb
CPU
%
CPU
ms
ServiceSystem
SID
Time-
stamp
Host 1Backup
Host 7ASCSP22
Host 3
Host 6
Host 5
Host 4
Host 3
Host 2
Host 1
Hostname
xy
SCSP22
JCP22
JP22
APPP22
CIP22
DBP22
SAPS
%
SAPS
abs
Mem
%
Mem
Kb
CPU
%
CPU
ms
ServiceSystem
SID
Min, Max, Avg, Totalper Report-cycle
Introduction
FlexFrame Agents Guide 78
3.7.2.1 Automatic Calculation of powervalue [SAPs-äquivalent] Values
The powervalue [SAPs-äquivalent] calculation is based on the automatic and dynamically
determined workload ability of a node. Based on a variety of technical features such as
cache, CPU, hyperthreating etc. and the possibilities of the operating system to use
these, modern servers can, come to wrong assumptions concerning the workload abilities
of a node. In these cases the automatic valuation can result in defective workload calcu-
lations.
If the internal automatic ascertainment of the powervalue [SAPs-äquivalent] value results
in defective values, the manual powervalue [SAPs-äquivalent] calculation can be used.
3.7.2.2 Manual Calculation of powervalue [SAPs-äquivalent] Values
If the maximum workload number of a server could not be correctly determined via the
FlexFrame AppAgent the workload number can be individually defined for each node.
The workload values are then calculated using the prepared workload data. In this way
the individual particularities of the workload abilities of a node can be taken into consider-
ation. For this purpose, however, the workload values for each node have to be entered
manually.
3.7.3 Billing
Using another calculation stage, chargeable workload units can be calculated from the
powervalue [SAPs-äquivalent]-based accounting data. For the calculation, a range of
parameters enabling differentiated pricing of the workload used can be set.
In the default mode, all systems and services are charged at the same value every time.
Detection cycle
Default 10 sec
Max
workload
Min
workload
Report-cycle 1 min
SAPs
t
Total work
Server capacity
Introduction
79 FlexFrame Agents Guide
With the help of the FlexFrame Control Center accounting plug-ins, the pricing can be
determined through additional configuration settings.
Therefore the following statements are necessary:
Service contract no.
System ID
ServiceID
Date range
Day type, e.g. weekday, holiday, weekend
Time of day, e.g. daytime, nighttime operation.
The billing table enables very differentiated billing of accounting data as far as the service
contract level. By way of system, service time, time of day and time types, different
service contract items with various workload prices can be used.
CPU/ SAPs Values
0,15€
0,20€
0,30€
0,15€
0,25 €
Accounting
Price
Sapsrule
Sapsrule
Sapsrule
Sapsrule
Sapsrule
Service-
levelrule
SC_12345
SC_12345
SC_12345
SC_12345
SC_12345
Service-
Contract
Standard
Standard
Standard
Standard
Standard
Accounting
rule
24:0000:00daily01.01.210001.01.2006DBP23
24:0000:00daily01.01.210001.01.2006otherP23
24:0000:00daily01.01.210001.01.2006allQ22
P22
P22
System
ID
all
all
Service
typ
00:00
00:00
from
Time
24:00
24:00
to
Time
weekend01.01.210001.01.1900
workday01.01.210001.01.1900
UnitdayTypetoDatefromDate
0,15€
0,20€
0,30€
0,15€
0,25 €
Accounting
Price
Sapsrule
Sapsrule
Sapsrule
Sapsrule
Sapsrule
Service-
levelrule
SC_12345
SC_12345
SC_12345
SC_12345
SC_12345
Service-
Contract
Standard
Standard
Standard
Standard
Standard
Accounting
rule
24:0000:00daily01.01.210001.01.2006DBP23
24:0000:00daily01.01.210001.01.2006otherP23
24:0000:00daily01.01.210001.01.2006allQ22
P22
P22
System
ID
all
all
Service
typ
00:00
00:00
from
Time
24:00
24:00
to
Time
weekend01.01.210001.01.1900
workday01.01.210001.01.1900
UnitdayTypetoDatefromDate
Aggregationcycle
AccountingcycleAccountingreport
Introduction
FlexFrame Agents Guide 80
3.8 Parametricable Value Observation and Service Level Profile
The FlexFrame Agentssuite offers the possibility of generating the production of events
e.g. on transgression of a defined limiting value. Parametricable events are directly eval-
uated and dispatched by the FlexFrame Agents.In contrary profiles are evaluated by the
domain manager during processing of the performance-, capacity- and accounting-
collections, generated on transgression of adjusted levels of belonging events and dis-
patched as SNMP trap.
3.8.1 Monitoring of file system and directoy utilization
From version 8.0 on the FlexFrame Agents are able to generate parametricable events.
On those terms observable objects and their limiting values can be registerd in a file. This
function exists for the observation of file systems. Obervable file systems as well as the
limiting values for size and increase are registered in a regulation file.On transgression of
the limits a belonging event is generated and dispatched.
3.8.2 Performance Profile, Capacity Profile, Accounting Pro-file
Profiles in which limiting values for the CPU consumption and powervalue [SAPs-
äquivalent] consumption can be defined, are defined for the performance data and ac-
counting data of the FlexFrame Agents. On transgression of the limiting values a SNMP
trap is generated.The definition of the profiles follows with the standard FlexFrame Con-
figGui.
Performanceprofile
Capacityprofile
Accountingprofile
The profiles can be defined on service level, system level and node level. The user can
individually decide which profiles exist. A profile can apply to all systems, nodes and
services or pool-, system-, service-, group- or node-specific, depending on the hierachical
level of the profile.
The evaluation of the performance, capacity and accounting profiles ensues through the
domain manager.
The domain manager reads the collections in. The addition and condensation to higher
condensation-based or system-based values follows through the aggregation in the do-
mainmanager.
Introduction
81 FlexFrame Agents Guide
3.9 FlexFrame File System Utilization and File Space Accounting Option
This function is available from version 8.0 of the FlexFrame Agents on and enables the
observation of file systems and the node- and system- or service-based accounting of the
file spaces which is needed because of defined files or registers. The fill degree of file
systems on local servers and also on storage systems (NAS and SAN) can be supervised
through the file space accounting system. On reaching a specific percentual threshold
this system will generate a report.
The tasks of the file space management are on the one hand the monotoring which ena-
bles the alerting and reacting in due time if not enough file space is at disposal and on the
other hand the processing of the file system data to the accounting and to the evaluation
for the storage capacity planning.
The following graphic shows the architecture of the file system utilization:
The data for file system utilization and file space accounting can be determined with the
FlexFrame AppAgents and the FlexFrame FrameAgents.The FlexFrame FrameAgents
are installed on the Control Node which allows locally personal parameter files for each.
The FlexFrame AppAgents have a pool-specific parametrization meaning all FlexFrame
Agents of one pool will be using the same parameter file.
The collecting and processing of the data can be done frame-, node- and system- or
service-specific. The parameter sets and rule sets for the file system utilization allow very
granular definition in which cycles and for which file system or file the storage consump-
tion and the increase will be ascertained.
Introduction
FlexFrame Agents Guide 82
Statistic definition in the ruleset
The observed file systems, registers and files can be staticly defined in the rulest, i.e. the
start of the FlexFrame Agents activates the collection of the data and the calculation for
every report cycle.
Dynamic definition in the ruleset
The ascertainment of the file systems, directories and file data can also be achieved in
dependency to systems and/or specific service types. The generation of the file system
utilization data ensues only if the defined system or a specific service type actually runs
on a node. The FlexFrame AppAgents ascertain through the dynamic triggering.
The ressource requirements and time requirements for the ascertainment vary depend-
ing on the size and the number of the monitored file systems, directories and the number
of files. To ensure the use of the functionality in very big SAN-based or NAS-based stor-
age infrastructures a convenient and adapted configuration of the rules and cycles for
collecting and processing of the file system data and file data is necessary. For that the
FlexFrame Agents offer the configuration of two different cycles for collecting of the data
and a file system detector which ascertains resource savingly only the consumptions on
file system level and does not execute file-based evaluations.
Another aspect is the visibility of the file systems and directories from the view of one or
more Application Nodes. Normally all nodes of a frame or pool use the same file system.
It is recommended to collect this information with the FlexFrame FrameAgent which will
be installed on the Control Node and which is able to execute this task centrally for a
whole frame.
As soon as a file-based or directory-based supervision or accounting is exspected, the
configurated rules have to be fitted to size and number of the storage systems. Otherwise
a successful implementation and use is not possible.
The directory detector and file detector is a dependent detector, i.e. it requieres infor-
mation of the file systems to supervise the values for individually defined directories or
files.
With that the register detector is able to work from the view of the Application Nodes or
from the view of the Control Nodes. If used on the Application Node generic information
from the services can be used to ascertain the directories, i.e. this function is better suited
for the use of the directory monitored. With that the monitored directories can be generi-
cally derived from the service information. Thereby e.q. data directories and log directo-
ries of a data bank can be generically dynamically ascertained.
File systems and directories normally are not in a 1:1 relation to nodes and services. A
file system e.q. may be used by all nodes of a pool, directories on the other hand are only
available in dependency to specific services. The architecture of the FA file utilization
management offers therefore the possibility to specifically ascertain the data on frame,
node or service.
Introduction
83 FlexFrame Agents Guide
The FlexFrame Agent takes over the data for every cycle from the detector and com-
presses the results to a report cycle. The results are written in a collection file for every
report cycle. The collection files develop service-, node-, pool- and frame-specific.
Service-specific or node-specific information can only be generated by FlexFrame Ap-
pAgents.
During the collection of the data on the Control Node frame-specific storage data can be
collected.
For the administration of the storage consumption information, collections of variable
type are written:
Exampes:
Perf_FSUtil_Node
The collection develops for every node and contains a record with correspond-
ing timestamp of the report cycle to the report cycle for every file system to be
supervised.
Perf_DirUtil_Service
The collection develops for an SID service and contains a record with corre-
sponding timestamp of the report cycle to the report cycle for every directory or
file to be supervised.
Perf_FSUtil_Frame
The collection develops for every node and contains a record with correspond-
ing timestamp of the report cycle to the report cycle for every file system to be
supervised.
Introduction
FlexFrame Agents Guide 84
3.9.1 File System Detector and Directory Detector
As of version 8.0 FlexFrame Agents have two additional detector components for the
supervision of file systems and directories.
1. File system detector
2. Directory detector
The file system detector can supervise file systems or mount points and supplies the
existing and employed capacity of a file system. A file system corresponds to a mount
point in a Linux system.
The directory detector enables the monitoring of individual files or directory. The directory
detector ascertains the absolute quantity.
Absolute storage space z
Total capacity of the file system
Usage of file system and directory
Usage %
The delta to the last cycle kb%delta and additionally as value per second
The resource consumption or the footprint of the directory detector is distinctly higher
then the file system detector consumption. Therefore the directory detector can only be
used for individual directories and should not be configurated for all directories.
Interesting is for instance the supervision of files and directories where significant
increase is possible for example logfiles or database directories.
Variable cycle times can be defined for collecting of the file system data and directory
data:
Perfdata_SpaceUtil_FSCycleTime
specifies how often (in seconds) FlexFrame Agents checks the file system utili-
zation for performance data.
Perfdata_SpaceUtil_DirCycleTime
specifies how often (in seconds) FlexFrame Agents checks the directory utiliza-
tion for performance data.
The FlexFrame Agent is able to calculate from the raw values the increase rates through
consi-deration of several cycles and to dispatch alerts as soon as these and/or the fill
degree of the file system exceed specific values.
Values for the generation of the file monitoring alerts receives the FlexFrame Agent from
the configuration file myAMC.FA-Check.xml in which limiting values for every super-
vised file system can be configurated.
This function enables for example the supervision of the availability and the capacity of
the voIFF filesystem from FlexFrame.
Introduction
85 FlexFrame Agents Guide
3.10 Domain Manager for Performance and File System Collections
3.10.1 Data collection
The domain manager is an independent component of the FA suite. The domain manag-
er cyclically processes the collection files for the performance, capacity, accounting and
file utilization management. The FA domain manager is used on the Control Node.
To that purpose the FlexFrame Agents generate so called file system file collections and
accoun-ting collections each in the data directory of a pool. The following types are dis-
tinguished:
Collections for every node with the name pattern Perf_Node~<node_name>.prf.<number>.col
Collections for every servicegroup with the name pattern Perf_Group~<node_name>.prf.<number>.col
Collections for every service with the name pattern Perf_Service~<SrvType>~<SID>~<ID>.prf.<number>.col
Collections for the file system supervision Perf_FsUtil_Frame~< frame_name>.prg.<number>.col
Perf_FsUtil_Pool~<pool_name>.prg.<number>.col
Perf_FsUtil_Group~<group_name>.prg.<number>.col
Perf_FsUtil_Node~<node_name>.prg.<number>.col
Perf_FsUtil_System~<SID>.prg.<number>.col
Perf_FsUtil_Service~<SrvType>~<SID>~<ID>.prg.<number>.col
Collections for the directory and file monitoring Perf_DirUtil_Frame~< frame_name>.prg.<number>.col
Perf_DirUtil_Pool~<pool_name>.prg.<number>.col
Perf_DirUtil_Group~<group_name>.prg.<number>.col
Perf_DirUtil_Node~<node_name>.prg.<number>.col
Perf_DirUtil_System~<SID>.prg.<number>.col
Perf_DirUtil_Service~<SrvType>~<SID>~<ID>.prg.<number>.col
Number and size of the collections generated by the FlexFrame Agents can be adjusted.
The standard setup enables the generating of 10 collections each per service or node.
With that a ringbuffer of data which can be automatically reorganized by the agents de-
velops. The for the sizing required storage size can be calculated from the number of the
nodes and the size of the report cycle.
The parameter of the performance manager and the backup routine have to be adjusted
in such a way that the raw data can be securely further processed bevor being super-
scribed again by the agents.
Introduction
FlexFrame Agents Guide 86
3.10.2 Data retentiontime and compression
The domain manager has the task to process new data and write it to the database.
However, the domain manager also monitors the retention time of the individual data in
the database. After expiration of the retention time he automatically deletes older data
sets from the database.
From FlexFrame Orchestrator V5.3 or V1.0 the DomainManager can also compress data
cyclically. From the minute values, e.g. be used for online monitoring, which can lead to
very large amounts of data very quickly, is cyclically generated a compression stage (eg
hourly values). The amount of data is reduced by a factor of 60. From the minimum, max-
imum and average values based on minutes, the corresponding values are calculated on
an hourly basis. For this compression not the collets, but the data from the repository are
required. The erase cycles and compression cycles can be configured independently of
each other. Prerequisite for a higher compression level is, of course, that the deletion of
the data needed for the compression will be done at a later time.
Introduction
87 FlexFrame Agents Guide
3.11 Customized Detector engine (service ping)
The Agents support an API to integrate specific detector modules. This API allows to
integrate external modules into the detection- and state- model of a service. The agent
calls this scripts in defined cycles. Included as standard are some scripts that contain
function as an API to the internal rules.
These scripts are specialized detection modules for specific services (db, ers, hana, ...)
and allow enhanced detection- and display options
The scripts have the following return codes:
0 : ok
1: not ok
99: unknown
2-98: reserved for future use
100-999: free for customer specific detection scripts
The use of this service ping scripts can be configured service specifically for certain types
of services or special services with the WebGUI.
3.11.1 ServicePingDb.sh
This script allows the availability check of the DB service with database resources.
3.11.2 ServicePingDef.sh
This script allows you to check availability of service with sapcontrol.
This is the default scripts for common services.
3.11.3 ServicePingErs.sh
This script allows you to check availability of ERS service with sapcontrol.
3.11.4 ServicePingHDB.sh
This script allows you to check availability of HANA database.
Introduction
FlexFrame Agents Guide 88
Furthermore, it retrieves information for HANA replica, the replica ID and the information
'primary' or 'secondary'. This information is passed on by the Flex Frame agents and
displayed in the FlexFrame Agents WebGUI.
89 FlexFrame Agents Guide
4 FlexFrame Agents
The operation of SAP systems is becoming increasingly complex, the number of compo-
nents required is constantly rising.
Installation, configuration and operation of a distributed SAP installation consequently
involve considerable administrative effort. The demands on the systems change rapidly,
and it must be possible to expand an existing configuration of replace failed components
both quickly and flexibly.
Through the use of FlexFrame Agents, FlexFrame enables the number of operator inter-
ventions to be reduced and availability to be increased. This chapter describes the appli-
cation scenarios for the FlexFrame Agents functions.
Installation and startup of the agents is described in chapter 2.
To permit active operation of a FlexFrame Agents installation, the FlexFrame Agents
must run on the Application Nodes and a Control Agent on the active Control Node.
Use of the FA Messenger component is optional and is required only for displaying and
forwarding events of the FlexFrame Agents and for integration into Enterprise Event
Management Systems.
The FlexFrame Application Agent is used to monitor SAP central instances, SAP appli-
cation instances and database instances. In the event of a problem, so-called self-repair
mechanisms are used for these services. Execution of these self-repair mechanisms can
be triggered locally or centrally. For each service/node these mechanisms can be divided
into the following categories:
Monitoring of a service
Restarting of a service if it was down
Rebooting of a node if a service could not be started again after one or more re-
starts
Switchover (automatic change) to another node if the reboot could not be per-
formed or was not successful
Detecting of START, STOP and maintenance situations
Control functions for displaying activities and statuses, sending mails and SMSs,
configurable in conjunction with time, contact and problem situation
Performance, capacity and accounting management
File system monitoring
FlexFrame Agents
FlexFrame Agents Guide 90
4.1 FlexFrame Agents Reactions
FlexFrame Agents detects problems and decides autonomously on the reactions to be
implemented after evaluating rules which can be controlled via parameters. FlexFrame
Agents knows the following basic reactions:
Restart
Reboot
SwitchOver (internal / external)
These basic reactions, combined with pool creation, grouping, the service classes, and
the service priorities result in a large number of reaction scenarios.
4.1.1 Restart
The FlexFrame AppAgent restarts a service if a required subservice is down or no longer
avai-lable. In this case it checks whether the service is available again after the restart on
the basis of a configurable time interval. The restart is not performed if any service which
runs on the node has already triggered a reboot. Furthermore, failure of multiple sub-
services of a service leads to a restart within the configured time interval only until service
availability has been restored.
The number of restart attempts for restoring service availability can be configured. If the
number of parameterized restart attempts is 0, failure of the service results directly in a
reboot attempt. If the number of reboots permitted for the nodes is 0, a switchover is
initiated.
The restart reactions are not affected by pool creation, grouping or service classification.
4.1.2 Reboot
A node is rebooted if an monitored service has failed and could not be made available
again after the configured number of restart attempts, or if no restarts are allowed.
The autonomous reaction also evaluates the service priority of the service which is dis-
turbed. If multiple services are running simultaneously on the nodes, it will be checked
whether services with higher priority are still running. If this is the case, the reboot is not
performed.
4.1.3 SwitchOver
A SwitchOver always leads to all the monitored services of a node being moved to an-
other server node.
The decision to move to another node can be taken locally by the FlexFrame Agent (
internal SwitchOver), or by the Control Agent on one of the Control Nodes (external
SwitchOver).
FlexFrame Agents
91 FlexFrame Agents Guide
4.1.3.1 General Rules
Takeover in the event of a node failure is implemented using what is termed an applicant
rule. The applicant rule states that each spare node may apply to take over the services
of a failed node.
Pool creation, grouping and service classes permit new switchover scenarios which can
satisfy different availability requirements depending on the parameterization.
This results in the following scenarios:
Pool-dependent switchover
Group-dependent switchover
Service-prioritized switchover
The failure of a node is only reacted to within a virtual FlexFrame pool.
Groups can be defined within a virtual FlexFrame pool. The applicant rule states that a
node only issues an application when a node in its own group fails.
The granularity of the reaction to a system failure can be further refined by prioritizing
individual services.
The applicant rule states that in the event of simultaneous failure of multiple services, the
application is first issued for the switchover file (testament) of the service with the higher
priority. Only if all higher-priority services have been taken over by another node and free
spare nodes exist do these apply for the switchover files of lower-priority services which
still need to be taken over.
When services with priority 0 fail, no applications are made by spare nodes. This pre-
vents spare nodes being used up by the failure of unimportant test systems.
The parameter file also contains a “minimum priority” parameter. This parameter provides
a very simple way to define, for example, that spare nodes only apply to take over the
services of a node if none of the failed services has a lower priority than that entered
there.
In conjunction with the basic rule “by default all services have priority 1“, a lower priority
can be configured for individual services, thus providing a simple way to prevent valuable
spare nodes being used up by the failure of test systems.
FlexFrame Agents
FlexFrame Agents Guide 92
4.1.3.2 Internal SwitchOver
In the case of an internal SwitchOver the Application Agent recognizes that a service is
down and cannot (or depending on the configuration may not) be restored using a restart
or reboot. The FlexFrame Agent then initiates an internal SwitchOver.
The autonomous reaction also evaluates the service priority of the service which is dis-
turbed. If multiple services are running simultaneously on the nodes, it will be checked
whether services with higher priority are still running. If this is the case, the internal
SwitchOver is not performed.
The actual takeover by another node begins with the transfer message. Only spare nodes
can apply and take over these services. The node which takes over control starts the
required services. If, after the maximum switchover time, the FlexFrame AppAgent on the
system that is to take over control is not able to start the services, it reports this by means
of an SNMP trap. The switchover is aborted and must be processed further by the admin-
istrator.
4.1.3.3 External SwitchOver
In contrast to the internal SwitchOver, the external SwitchOver is detected and initiated
by the Control Agent on a Control Node. This is required if the system is showing no sign
of life or can no longer be reached in the network. Reachability is tested using Ping or
SSH tests. The user decides whether to perform only Ping, only SSH or both kinds of
test. Additionally the Ping requests may be configured for client LAN, server LAN, and/or
storage LAN interfaces.
The takeover is performed in the same way as for the internal SwitchOver.
In order to enable user-specific actions before or after a node was powered down, the
CtrlAgent calls hook scripts, which may be customized by the user. The scripts are pro-
vided with the return code of the previously executed action.
Pre-PowerOff hook script: Called with return code 0 as argument, as there was no previ-
ously executed action.
Post-PowerOff hook: Called with the return code of the Pre-PowerOff hook script (if it
failed, i.e. the return code was != 0) or with the return code of the power off script.
If the configuration value IgnorePoffHookResult is set to true, the return codes of
the hook scripts are ignored. If set to false, they are used as hints on how to proceed in
case of errors: if the Pre-PowerOff hook script returns a value != 0, power off will not be
performed, if the Pre-PowerOff hook script returns a value != 0, the switchover will not
proceed. This enables the user to customize the external switch over and power off pro-
cesses based on additional information or rules or to perform additional actions, e.g.
mounting SAN devices.
FlexFrame Agents
93 FlexFrame Agents Guide
4.1.4 Special case “fast start VM”
In case of failure of an AN (VM), it may happen that the VM is available again after a very
short time. In this case, the CtrlAgent not detect the failure as such, because the AN has
been available back when accessibility test (PING / SSH).
In this case, it is checked whether the AN had services before the failure, and then is
indeed available but has no services.
If such a case is detected, the testament of the AN (before failure) is placed as Reboot-
file in the appropriate directory, and the AN (VM) is switched (cycle) from CtrlAgenten per
shutdown facility.
In this, by the CtrlAgenten hard initiated reboot of the AN (VM), the reboot file is recog-
nized by AppAgenten, and the enclosed services are started.
There is also a configuration parameter "MaxHeartbeatQuickStartTime" (default = 10 s).
Even after this short time is performed to check whether the situation of a "Fast Start VM"
described above is present.
For this special case "FastStart VM" occurs, the VM must be made within a very short
time, less than "MaxHeartbeatTime" (default = 60 s), be available again.
Requires the VM longer until it starts up again, occur the normal switchover sequences
(expiration of "MaxHeartbeatTime", test with ping / ssh, then if unreachable done a
switchover).
FlexFrame Agents
FlexFrame Agents Guide 94
4.1.5 Maintenance
The autonomous functions and reactions of the FlexFrame Agents can be disabled for
individual services by calling a maintenance script. This is always required when applica-
tion instances are to be started and stopped without autonomous reactions.
A service is set to nowatch using the following scripts on the relevant Application Node:
ff_service.sh -s <SID> -t <SERVICETYPY -a nowatch
ff_service.sh -s ml5 -t db -a nowatch
...
A service is reincluded in monitoring using the following scripts on the relevant Applica-
tion Node:
ff_service.sh -s <SID> -t <SERVICETYPY -a watch
ff_service.sh -s ml5 -t db -a watch
...
4.2 Self-Repair Strategies
In terms of the strategy for restoring a failed service, a distinction must be made between
the following failures:
Service failure
Node failure
A detailed description of the procedure for the subsequent autonomous reaction was
provided in the preceding chapter.
4.2.1 Self-Repair in the Event of a Service Failure
If a service failure occurs, this is detected by the FlexFrame AppAgent and an attempt
is made to make the service available again using the following autonomous reactions
and their escalations:
Restart of the service
Reboot of the node
SwitchOver (internal)
Implementation and the number of the above-mentioned autonomous reactions and the
escalations can be affected by the configuration.
FlexFrame Agents
95 FlexFrame Agents Guide
4.2.2 Self-Repair in the Event of a Node Failure
If a node failure occurs, this is detected by the FlexFrame CtrlAgent and an attempt is
made to make the service available again using the following autonomous reactions:
SwitchOver (external)
4.2.3 Takeover by a Spare Node (TakeOver)
The standard rule in the FlexFrame concept for taking over the services of a failed node
is to have them taken over by a spare node.
Every Application Node in a standard FlexFrame installation on which an FlexFrame
AppAgent is running and none of the monitored services exists is automatically a spare
node.
If a switchover is started as a result of a node failure or escalation of a service failure, all
spare nodes apply to take over the services. The quickest node in the application proce-
dure is chosen and takes over the tasks.
4.2.4 Multi Node Failure
4.2.4.1 Common
The simultaneous failure of multiple systems or nodes is called “Multi Node Failure“. This
indicates a different kind of failure than a single node or system failure, where the cause
is usually more complex. As of version V30A10, the FlexFrame Agents offer support for
the automatic detection of Multi Node Failures with different reactions and additional
alarms. This allows the recognition of failure states, which require the attention and deci-
sion of an administrator. Several new parameters allow the modification of the usual
behaviour, like delaying or skipping reactions. Additionally a set of new alarms triggered
by user-configurable indicators inform the administrator in case of of a multi node failure,
so he may take apprioriate actions. The configuration of these indicators can be per-
formed per pool.
There are two different Multi Node Failure scenarios:
1. Simultaneous failure of multiple nodes, systems or services, e.g. due to a power
outage in a blade cabinet, which shows an affect within a short period of time
(e.g. one minute)
2. Failure of several nodes, systems or services, within a specific timerange, which
is bigger than the one specified above (e.g. one hour)
These scenarios are called “ShortTime Failure” and “LongTime Failure”.
The CtrlAgent keeps a list of all failures, with each entry containing node name and
timestamp. If the number of entries within a scenario-specific timerange exceeds the limit,
the CtrlAgent assumes a Multi Node Failure.
FlexFrame Agents
FlexFrame Agents Guide 96
4.2.4.2 Case 1: ShortTime Failure
Simultaneous failure of multiple nodes, systems or services, e.g. due to a power outage
in a blade cabinet, which shows an affect within a short period of time (e.g. one minute).
MultiNodeFailure_ShortTime_FailureCount
Specifies the number of failures within a certain time range, which leads to a Multi
Node Failure state.
MultiNodeFailure_ShortTime_FailureTime
Specifies the time range (in seconds) to be used for failure aggregation.
MultiNodeFailure_ShortTime_ReactionDelay
Specifies a delay time (in seconds) before the CtrlAgent reacts on failures.
MultiNodeFailure_ShortTime_ReactionAction (for future use)
Specifies a reaction different from the normal modus of operation.
In case of a Short Time Multi Node Failure the CtrlAgent sends an emergency alarm.
Additionally the usual autonomous reactions can be delayed or skipped (by setting MultiNodeFailure_ShortTime_ReactionDelay to a very big value).
4.2.4.3 Case 2: LongTime Failure
Failure of several nodes, systems or services, within a specific timerange, which is bigger
than the one specified above (e.g. one hour).
MultiNodeFailure_LongTime_FailureCount
Specifies the number of failures within a certain time range, which leads to a Multi
Node Failure state.
MultiNodeFailure_LongTime_FailureTime
Specifies the time range (in seconds) to be used for failure aggregation.
MultiNodeFailure_LongTime_ReactionDelay
Specifies a delay time (in seconds) before the CtrlAgent reacts on failures.
MultiNodeFailure_LongTime_ReactionAction (for future use)
Specifies a reaction different from the normal modus of operation.
In case of a Long Time Multi Node Failure the CtrlAgent sends an emergency alarm.
Additionally the usual autonomous reactions can be delayed or skipped (by setting
MultiNodeFailure_LongTime_ReactionDelay to a very big value).
FlexFrame Agents
97 FlexFrame Agents Guide
4.3 Escalation with partner service
There are special rules for some service types which have a partner-service.
There are differences if the partners run at the same or at another node as well as the
behavior at escalation and SwitchOver.
There are special rules for the following service-types:
• SCS / ASCS in conjunction with ERS
• Service with assigned SMD-service
4.3.1 SCS / ASCS in conjunction with ERS
For an SCS or ASCS service there is a replicated enqueue service (ERS) on which the
enqueue table is replicated.
The FlexFrame Agent has to check is the replication was active.
Therefore the status of replication will be detected with the script (‚my-
AMC.DetTestEnsmon.sh‘).
Possible results:
# no replication service configured
ENSMON_RESULT="replication is disabled in server"
# replication service configured, but no replica running
#ENSMON_RESULT="replication is enabled in server"
# replication service configured, replica running
#ENSMON_RESULT="replication is enabled in server\nReplication is
active"
Only if the replication ist enabled and acrive, there is a special rule as VIP-service for
SCS/ASCS.
Up to now it was usual run the SCS / ASCS and the ERS at different nodes, so in cause
of failure there are special scenarios for TakeOver.
By now it is allowed to run SCS/ASCS and ERS at the same node.
These involved adapted rules for escalation.
4.3.1.1 SCS/ASCS with partner ERS at various nodes
Aim:
FlexFrame Agents
FlexFrame Agents Guide 98
If the SCS/ASCS fails, this service must be restarted on that node where the associated
ERS is running.
The SCS/ASCS takes over the enqueue table in shared memory and stops the ERS.
Once the ERS is stopped, the testament is published and, with the autonomy scenarios
for internal SwitchOver, the ERS gets a new node and starts up. So if the SCS/ASCS
fails again, there is another ERS for a new takeover scenario.
For switchover of SCS/ASCS there was written a service based VIP-testament, which
contains only the SCS/ASCS and may only take over from the node running the associ-
ated ERS.
For switchover of ERS there was written a service based testament, which contains only
the ERS. For this TakeOver the normal takeover rules being valid.
Sequence of steps:
Node A Node B Node C
SCS/ASCS
running
ERS
running
SCS/ASCS
fail
SCS/ASCS
VIP-Testament
TakeOver
ERS
Switch
ERS-
Testament
TakeOver
These behavior with the VIP-testament will be done if configured both „Node_SwitchOverTyp=node“ and „Node_SwitchOverTyp=service“.
This scenario works with one or more replicated enqueue services in one system.
4.3.1.2 SCS/ASCS with partner ERS at same node
If SCS/ASCS and associated ERS are running at the same node, there is no shared
memory at another node which can be taken over.
Therefore the SCS/ASCS and ERS will be written in a common SwitchOver-file and will
be taken over to the same node.
These common switch of the partner services will be done if configured both
„Node_SwitchOverTyp=node“ and „Node_SwitchOverTyp=service“.
4.3.2 Service with assigned SMD-service
Service based SMD:
FlexFrame Agents
99 FlexFrame Agents Guide
If a service has an assigned SMD running at the same node, in case of failure the ser-
vices should be switch to the same node.
Therefore the service and the assigned SMD will be written in a common SwitchOver-file
and will be taken over to the same node.
These common switch of the partner services will be done if configured both
„Node_SwitchOverTyp=node“ and „Node_SwitchOverTyp=service“.
Host based SMD:
If the SMD is host based there is no fix mapping to the service.
If a host based SMD is the only running service at a node, the FlexFrame Agents will be
recognized the node as spare node.
During the FlexFrame Agentsreactions Restart and Reboot the SMD will be handled as
every other service.
FlexFrame Agents
FlexFrame Agents Guide 100
4.4 Takeover Rules
4.4.1 Overview
Rule based high availability for nodes and services is performed by evaluating rule sets,
which control the take over of services from a failed node. They consist of qualification
rules, take over strategy and take over rules.
The qualification rules specifies, which nodes may apply for the services of a failed node.
The take over strategy defines the conflict resolution mode to be used, when more than
one node applies for a node testament.
The take over rules controls the actual take over, i.e. service start order and possibly
service displacement or replacement.
The parameters for this TakeOver rules always apply to the entire pool (Up Flexframe
V5.2).
From FlexFrame V5.3 or FlexFrame Orchestrator V1.0 these rules can be defined differ-
ently in each group of a pool.
In the default case, there is a set of rules that applies to the entire pool. In addition,
group-specific rule sets can be configured for one or more groups. For all groups for
which there is no separate set of rules, the default rule applies.
4.4.2 TakeOver Strategy
4.4.2.1 Overview
The qualification rules specifies, which nodes may apply for the services of a failed node.
When performing a switchover, all nodes may apply for take over of the failed node’s
services by taking part in an auction. As long as the auction lasts, all nodes, which
match the requirements as specified in the failed node’s testament may apply. When it is
finished, the take over strategy is used to decide which node won the auction.
4.4.2.2 FirstFit
FirstFit specifies that the first node, which applied for a testament, is the winner. This
is the default strategy.
4.4.2.3 LowPrioFit
As of version V30A10, the FlexFrame Agents provide a new strategy: LowPrioFit. The
application node containing the services with the lowest priority wins the auction. It there-
fore has the best chance to replace or displace some running services in order to take
over the failed services.
FlexFrame Agents
101 FlexFrame Agents Guide
By definition a spare node is considered to have the lowest priority, so it will win an auc-
tion over a node with running services. A node with only services of priority 0 will win over
a node with services of priority 1 and higher and so on.
This strategy can be used as an alternative to FirstFit. This changes only the behaviour of
the new take over rules: add rule, replace rule and substitution rule. If only the spare
node rule is used, the behaviour is the same as with the FirstFit strategy, because all
spare nodes have the same priority and the first one wins the auction.
4.4.3 TakeOver Rule
4.4.3.1 Overview
In version 3 and higher, the FlexFrame Agents offer the option of configuring various
takeover rules. It is now possible to replace or supplement the previously available spare
node rule with further alternatives.
Generally it is possible to differentiate between a static takeover rule and a dynamic
takeover rule.
These takeover rules can be applied not only for nodes but also for service-based testa-
ments. With node-based testaments it is evaluated in each case on the basis of the high-
est priority in the testament or the highest priority of a current service on an Application
Node.
4.4.3.2 Takeover Rules
The takeover rule is referred to as static if one of the possible takeover rules is param-
eterized. The spare node rule available until version 3 is a static takeover rule.
With version 3.0 of the agents the following static parameterized takeover rules will be
available:
Spare node rule
Add rule
Replace rule
Substitution rule
Dynamic
These rules allow very granular reaction to the breakdown of Application Nodes and
services. It is no longer necessary to always keep spare nodes on hand in case of a
breakdown of services for high availability.This function can now also be performed by
Application Nodes which already possess services.
4.4.3.2.1 Takeover through Spare Nodes (TakeOver)
The takeover by way of a spare node is the standard rule in the FlexFrame concept for
taking over the services of the defective node.
FlexFrame Agents
FlexFrame Agents Guide 102
Every Application Node in a standard FlexFrame installation on which the FlexFrame
AppAgent runs and on which none of the controlled services exist, is automatically a
spare node.
If, through the breakdown of a node or the escalation of a service disturbance, switchover
occurs, all spare nodes apply for the takeover of the services. The quickest node in the
application procedure receives the job and will take over those services. The application
takes place at group level.
In a default configuration, the spare node rule always applies.
4.4.3.2.2 Add Rule
With this rule all spare nodes of a group can apply but so too can all nodes which still
have sufficient workload reserves. The add rule only uses a node if the priority of the
services in the testament is equal to or higher than the priority already running on the
node. This prevents a high-priority node from taking over lower-priority services.
With add –rule only the nodes which possess services with lower or equal priority
apply (from pool, group).
If (OWN_prio_max >= SWO_prio_max ) apply
In addition to the already running services, the services that have been taken over are
started.
Thus the add rule is employed if running services are supposed to be stopped by the
takeover rule. This normally gives rise to a performance disadvantage after the taking
over of the services for the services taken over as well as the services already running on
the Application Node.
The add rule expects for the services a configured maximum powervalue [SAPs-
äquivalent] value reserved for a service. The node only applies if its powervalue [SAPs-
äquivalent] workload suffices for the operation of the already running services and the
services taken over.
4.4.3.2.3 Replace Rule
This rule enables all spare nodes of a group to apply, but also all other nodes on which
services with lower priority run. The replace rule only uses a node if the priority of the
services in the testament is higher than the priority of the services already running.
If ( OWN_prio_max > SWO_prio_max ) apply
Attention: Prio 1 is the highest priority: numbers > 1 decrease in priority.
Services which are already running are displaced. The adopted services are started.
A displaced service is transferred via the internal SwitchOver.
FlexFrame Agents
103 FlexFrame Agents Guide
The other nodes of the group apply according to the application procedure and the ser-
vices can again be made available for other nodes, provided that lower-priority nodes still
exist within the group.
This rule defines which services may be stopped within the context of a takeover scenario
so that higher-priority services can be adopted and can possibly be made available on
another node.
4.4.3.2.4 Substitution Rule
This rule enables all spare nodes of a group to apply, but also all other nodes on which
lower-priority services run.
With the substitution rule, only nodes (from pool, group) possessing services with a lower
priority than the high priority of the switchover file apply.
If ( OWN_prio_max > SWO_prio_max ) apply
Attention: Prio 1 is the highest priority numbers > 1 decrease in priority
Services which are already running are stopped according to the rules of the stop hierar-
chy. The adopted services are started.
FlexFrame Agents
FlexFrame Agents Guide 104
4.4.3.3 Dynamic TakeOver Rules
4.4.3.3.1 Overview
In the static mode only one of the possible takeover rules can count. With the parameter dynamic as static takeover rule the decision which rule ist used is made dynamicaliy,
depending on the priorities of the services on the Application Nodes and the highest prior-
ity of the defective services. For this purpose, the priorities table exists in the parameter
file, which is only used in the case of a static rule dynamically parameterized.
The dynamic takeover rule allows disjunct as well as overlapping priority domains be-
tween the spare node rule and one of the other static takeover rules. For the takeover rules add rule, replace rule and substitution rule, the priority domains must defi-
nitely belong to a rule. If a priority domain is defective then the first fitting is used.
The dynamic takeover rules are best understood from the view of a service on one of the
Application Nodes.
The add rule replace rule and the substitution rule can be seen from the view of a running
service on an Application Node as escalation stages which, within the context of a takeo-
ver scenario, increasingly handicap it unless it can avoid impairment through a higher
priority.
As soon as a testament is published for takeover, all application nodes of the group
check if they can improve their status (services with higher priority). An Application Node
applies if, on successful application, an improvement to its service priority status is possi-
ble.
The successful applicant determines the taking over dynamically according to the para-
meterized takeover rule. The rule results from the evaluation of the dynamic takeover
table and the highest priority of the running services.
From this view, naturally the spare node has a special role, as it has no services it could
handicap. It has the function of protecting its resources for special challenges. The pa-
rameters registered for the spare node result in principle in another evaluation. The pa-
rameters determine the priority domains in which the spare node may apply.The priority
domain for which a spare node is responsible can overlap with the priority domains of the
other rules. A spare node now no longer applies for defective services, as was the case
with simple spare node rules, but only for services in the priority domain. This prevents
spare nodes from beingwasted on operating defective lower priority services.
The rules are called dynamic takeover rules if, depending on the highest service priority
that an application server already has, it is dynamically decided according to which rule
an Application Node takes part in the application procedure on breakdown of other Appli-
cation Nodes or services .The decision as to which rule is applied depends on the ser-
vices running on an Application Node at that time.
The dynamic takeover rules are supposed to ensure that a node is always available for
the highest priority services. Lower priority services can be operated in parallel. On
FlexFrame Agents
105 FlexFrame Agents Guide
breakdown of higher priority systems, those (the lower priority services) are displaced or
replaced.
The dynamic rules also have the function of reserving valuable spare nodes for high
priority services. This allows the use of different workloads and priorities in a FlexFrame
pool service. In case of hardware breakdowns very different disturbance scenarios, it is
now possible to use the remaining hardware optimally.
The application itself does not change. Of all the Application Nodes that applied, the one
which registered first wins.This is determined by the configuration parameter qualification
rule first fit.
The rules to be applied are parametered through a table with the following structure:
Spare node rule Prio >= 1 < 2
Add rule Prio > = 2 < 4
Replace rule Prio >= 4 < 6
Substitution rule Prio >= 6
The interpretation of this table depends on the dynamic evaluation implemented by an
Application Node for itself. An Application Node without services is a spare node. A spare
node applies only for node or service testaments with a service belonging to a priority
domain defined for spare nodes. This rule enables the reservation of spare nodes for high
priority services, without them being lost through lower priority services. This is shown by
the example with the following parametering for the spare node rule in the dynamic
takeover table.
Spare node rule Prio >= 1 < = 2
The spare nodes only apply for defective services which have a priority of 1 or 2. This
rule allows the restriction of the use of spare nodes on high-priority services.
Spare node rule Prio >=1 <= 2 exclusive
The additional attribute exclusive prevents Application Nodes which already possess
services from applying for services with the priority 1 or 2.
For Application Nodes which are not spare nodes, the rule for application is a bit more
complex. Here, two different dynamic influencing factors are at work: the highest priority
of a service in a testament for which it is possible to apply, and the highest priority of a
service already running on the Application Node.
For the application rule, the principle applies that an Application Node does not apply for
new services if this would cause already-running services with equal or higher priority to
be handicapped.
Example of a dynamic takeover rule:
FlexFrame Agents
FlexFrame Agents Guide 106
Individual dynamic switchover rules per pool
1
1
2
2
Spare
3
4
Pool n
Group 1
Group 2
Sparenode >=1 < 2
Add rule >= 2 < 3
Replace rule >= 3 < 4
Substitute rule >= 4
Testament
Prio 2
5000 SAPs
2
2
3
4
Candidate with add rule
Candidate with repluce rule
Candidate with substitution rule
Group specific application process
FlexFrame Agents
107 FlexFrame Agents Guide
4.4.3.3.2 Dynamic Example
In the Dynamic rule the highest priority of services in the switchover file and the highest
priority of the services at the own node will be used for further decision criteria.
For this, there is a further configuration:
Testament High Prio
TakeOver rule (min) (max)
SpareNode 1 4
Own High Prio
(min) (max)
Add 3 4
Replace 5 6
Substitute 8 20
Additionally there is the setting Dyn_Spare_exclusive, which means that the priority
range for SpareNode is exclusively reserved for the spare nodes.
The table above means the following:
High Prio in testament == 1 to 4:
Dyn_Spare_exclusive = true: Only spare nodes may apply.
Dyn_Spare_exclusive = false: All nodes may apply.
High-Prio in testament > 4:
Spare nodes may not apply.
All nodes with Own High Prio 3-4, 5-6, and 8-20 may apply.
All nodes with Own High Prio 1, 2, 7, and >20 may not apply.
Own High-Prio == 3 or 4: The TakeOver rule Add will be used.
Own High-Prio == 5 or 6: The TakeOver rule Replace will be used.
Own High-Prio == 8 to 20: The TakeOver rule Substitute will be used.
FlexFrame Agents
FlexFrame Agents Guide 108
4.4.4 Spare Nodes from the Adminpool (pool-independent spare node)
As of version V8.0 the agents on spare nodes can be requested from another pool. The
CtrlAgent of a pool monitors the takeover of failed nodes and services. Are no applicants
for failed nodes available can the CtrlAgent request spare nodes from the admin pool if it
is configured
A CtrlAgent is able to recognize through evaluation of its own pool name and the config-
urated parameter, which ruleset is needed for its operation
The following configuration parameters are used:
CtrlAgentRole
Defines the role of the CtrlAgent
Normal Normal function of this CtrlAgent
(don't work with a caretaker of the spare pool). WorkWithSparePool CtrlAgent work with a caretaker of the spare pool
SparePoolName
Name of the spare pool
<name> Name of the spare pool
NONE No spare pool available
SpareGroup
Name of the spare node group
<name> Name of the spare group
NONE No spare group available
The following parameters for the pool-overlapping spare node are also required:
SpareRequestMoveTimeout
Timeout in seconds for move a node from spare pool to work pool.
SpareNodePowercycleTime
Time in minutes how often the spare nodes in caretaker will be powered ON and
OFF. Values <= 1 (60sec) are not allowed and switched of this functionality.
SpareRequestMoveNodeMaxRetry
Max retries to call move-node script in case of error (LDAP-locks).
SpareRequestMoveNodeRetryDelay
Delay-Time (in seconds) between retries the call of move-node sript.
4.4.4.1 Productive Pool
Productive pools are all those pools whose poolname does not correspond with the con-
figurated spare pool name.
FlexFrame Agents
109 FlexFrame Agents Guide
4.4.4.2 SparePool (Adminpool)
4.4.4.2.1 General
The spare pool is the pool whose pool name corresponds to the configurated spare pool
name.
The configuration parameters CtrlAgentRole, SparePoolName and SpareGroup
must be set to the same values as in the productive pools using the spare pool.
An adminpool is a special pool whose nodes are not used actively. A special group inside
an adminpool is the group SpareGroup. The spare group holds all nodes which are
available as spare nodes for other nodes.
4.4.4.2.2 Ascertainment of the Node Information
The spare nodes are normally turned off and are only cyclically turned on and off by the
CtrlAgents. The cycle time is defined through a parameter.
Initially the spare nodes have to be configurated manually and be available for at least
one detection cycle in the pool. With that the spare nodes with their capacity data are
automatically recorded and registered in a spare node list.
The cyclic turn on of the spare nodes actualizes the list every time. Nodes which can´t be
reached or started during that are then not in the list anymore.
4.4.4.3 Sparenode Selection Rule
To replaice a spare node of the productive pool with a spare node of the admin pool the
CtrlAgent selects a suitable node with a configurateable ruleset. The node which fits the
best with the failed node is selected.
The following attributes are considered for the selection of a failed node:
Attribute Meaning Examples
CPU-HT States
(if hyperthreading is active)
-1: not ascertainable
0: no hyperthreading
1: hyperthreading active
0
CPU-arch Processor architecture i686 sun4us
CPU-cnt-core Number of CPU cores 1
CPU-cnt-phys Number of CPU bases 1
CPU-cnt-siblings Number of CPU cores per
base
1
FlexFrame Agents
FlexFrame Agents Guide 110
Attribute Meaning Examples
CPU-frequency(MHz) CPU tact frequncy 2993
OS-typ Operating system Linux SunOS
OS-version Version of the operating sys-
tem
SUSE LINUX Enter-
prise Server 9 (i586),
VERSION = 9,
PATCHLEVEL = 3
5.8
cache-size(kB) Size of the CPU caches (in
KB)
1024
hasHba States, if the node has a HBA
(host adapter for SAN) (true
or false)
false
hbaType Typ of the HBA (host adapter
for SAN)
none unknown
is10GbES States, if the node has a 10
GBit/s Ethernet interface (true
or false)
false
location Position of the node deposited
in LDAP
default
mem-total(MB) Main storagesize (in MB) 1001
saps Specified powervalue [SAPs-
äquivalent] value of the node
1042
node-product-name CPU term Intel(R) Pentium(R)
4 CPU 3.00GHz
FJSV,
GPUZC-M
node-vendor CPU producer GenuineIntel Sun_Micros
ystems
The concrete values of the attributes of the spare nodes are determined from the LDAP
and direct dynamically from each node by the FlexFrame Agents of the admin pool.
Some attributes (hasHba, hbaType, is10GbES and location) are determined from the
LDAP.
4.4.4.3.1 Configuration
The selection rules are defined in the config directory of the admin pool in the file my-
AMC_FA_SelectionRules.xml. They are organised in rule sets. Per inquiry only one
rule set is used: firstly the control agent searches a rule set with the name of the inquiring
pool. If such a ruleset does not exist, the default rule set is used.
FlexFrame Agents
111 FlexFrame Agents Guide
A rule set contains a lot of selection rules which have to be checked from the CtrlAgent
for every possible substitute node. As a result every candidate has a weight function. The
candidate with the highest weight function is transfered as a substitute node to the inquir-
ing pool.
Example of the rule file:
<?xml version="1.0" encoding="iso-8859-1"?>
<rules xmlns="myAMC/SelectRules/1.0" defaultSet="default">
<ruleset schema="default">
<select attribute="OS-typ" match="equal" required="true">
<select attribute="CPU-arch" match="equal"/>
</select>
<select attribute="hasHba" match="equalOrMore"/>
<select attribute="hasHba" match="more"
required="false" weight="0">
<select attribute="location" match="equal"/>
<select attribute="hbaType" match="equal" weight="2"/>
</select>
<select attribute="is10GbES" match="equalOrMore"/>
<select attribute="saps" match="range" weight="1"
lowerBound="20" upperBound="50" required="false">
<!-- even more weight for better range match -->
<select attribute="saps" match="range" weight="2"
lowerBound="10" upperBound="20" required="false"/>
<select attribute="saps" match="range" weight="5"
lowerBound="5" upperBound="5" required="false"/>
<select attribute="saps" match="range" weight="10"
lowerBound="1" upperBound="1" required="false"/>
</select>
</ruleset>
<ruleset schema="Pool1">
<!-- ... -->
</ruleset>
<ruleset schema="Pool2">
<!-- ... -->
</ruleset>
</rules>
4.4.4.3.2 Rule evaluation
A single selection rule compares an attribute of the reference node (the failed node) with
that of a substitute candidate. The comparison can ensue through different operators.If a
rule applies the substitute candidate receives a certain amount of points. A score is de-
FlexFrame Agents
FlexFrame Agents Guide 112
termined for the substitute candidate through evaluation of all rules of the ruleset. With
this score the most suitabel node is selected: the node with the highest score “wins”.
The following table describes the possible comparison operators:
equal Check of equality of an attribute. If both values are numbers a numeric
comparison is implemented; otherwise a textual comparison is imple-
mented.
more If the value of the candidate is more than the reverence value the rule
applies. If both values are numbers a numeric comparison is imple-
mented, otherwise a textual comparison is implemented. With boolean
status values (true, false) true in relation to false means “more”.
less If the value of the candidate is less than the reverence value the rule
applies. If both values are numbers a numeric comparison is imple-
mented, otherwise a textual comparison is implemented. With boolean
status values (true, false) false in relation to true means “less”.
equalOrMore If the value of the candidate is more or equal than or to the reverence
value the rule applies. If both values are numbers a numeric compari-
son is implemented, otherwise a textual comparison is implemented.
With boolean status values (true, false) true in relation to false means
“more”
equalOrLess If the value of the candidate is less or equal than or to the reverence
value the rule applies.I If both values are numbers a numeric compari-
son is implemented, otherwise a textual comparison is implemented.
With boolean status values (true, false) false in relation to true means
“less”.
range Checks if the value of the candidate in relation to the reverence value is
situated inside a certain area. The area limits aregiven as percentual
deviation with the help of the parameters “upperBound” and “lower-
Bound”. Differs the value of the candidate maximally over the given
limits from the reverence value the rule applies.
This comparison operator is only convenient with numeric attributes
e.g. the powervalue [SAPs-äquivalent] value (attribute “saps”)
The selection rules can be staggered hierachically. Subordinate rules are only evaluated
if the higher rule applies.
Example:
Rule 1
Rule 1.1
Rule 1.1
FlexFrame Agents
113 FlexFrame Agents Guide
Rule 1.2.1
Rule 2
Rule 3
Rule 3.1
Every selection rule has its own weight function (parameter weight). If a rule applies its
weight function is added to the score of the selected candidate. Its possible to distribute
negative weight functions if an attribute rather serves to prevent a selection.
If the parameter weight is not given, the weight function is, depending on its hierachy
level, automatically put to the following value:
1.Level: weight=5
2.Level: weight=3
And all further Levels: weight=1
A rule can be marked as necessary (parameter required) i.e. a candidate is seen as
not fitting if a necessary rule is not fulfilled. If the parameter required is not given rules
on the first hierachy level are automatically considered necessary and all others as opio-
nal i.e. if they are not fulfilled the candidate can still be selectet if the score is sufficient.
4.4.4.3.3 Test of the Selection Rules
A simple test of the selection rules can be exeuted with fa_selectionRuleTest.sh
(in the directory /opt/myAMC/FA_AppAgent). The script evaluates matching candidates
to a reference node and issues the most fitting candidate.
fa_selectionRuleTest.sh
{-n|--source-node} <nodename>
{-p|--source-pool} <poolname>
{-P|--source-livelist} <file>
{-t|--target-pool} <poolname>
{-T|--target-livelist} <file>
[{-f|--rule-file} <file>] [{-r|--rule-set} <name>]
[{-B|--pools-basedir} <directory>]
[{-l|--log-level} <level number>] [{-o|--print}]
[{-a|--trace}] [{-c|--candidates}] [{-w|--weight}]
{-n|--source-node} <nodename>
Name of source node (mandatory)
{-p|--source-pool} <poolname>
Name of source pool (required if source-livelist is not set)
{-P|--source-livelist} <file>
Name of source input file (required if source-pool is not set)
FlexFrame Agents
FlexFrame Agents Guide 114
{-t|--target-pool} <poolname>
Name of target pool (required if target-livelist is not set)
{-T|--target-livelist} <file>
Name of target input file (required if target-pool is not set)
{-f/--rule-file} <file>
Name of the rule file to use (optional)
Default: myAMC_FA_SelectionRules.xml in the target pool
{-r/--rule-set} <name>
Name of rule set to evaluate (optional)
Default: ruleset as specified in rule file
{-B/--pools-basedir} <directory>
Base directory for pools (optional)
Default: /opt/myAMC/vFF
{-l/--log-level <level number>}
Log level (optional)
Default: 5
-v|--verbose
Increase log level (optional)
-o|--print
Print node lists (optional)
-a|--trace
Trace selection rule evaluation (optional)
-c|--candidates
Print replacement candidates (optional)
-w|--weight
Print results including their weight (optional)
The tool determines the data of the reference node ( given with -n) from the accompany-
ing pool (-p)and searchs the fitting candidate in another pool (-t), the admin pool.
If additional options –a, -c and –w are given the results of all individual evaluations are
given. With this the correct parametricaton can be checked.
Example with several nodes:
cn1 # fa_selectionRuleTest.sh -a -w -c -n sno2bx1 -p pool2 -t adminpool
node "sno2bx3" does not match
sno2bx1 [36]
sno2bx2 [36]
Example with tracing:
FlexFrame Agents
115 FlexFrame Agents Guide
cn1 # fa_selectionRuleTest.sh -a -w -c -n sno1apl5p1 -p pool1 -t adminpool
evaluating node sno1apl5p1
object : attribute OS-typ matched and added weight 5
matched 'SunOS' against 'SunOS' using operator 'equal'
object : attribute CPU-arch matched and added weight 3
matched 'sun4u' against 'sun4u' using operator 'equal'
object : attribute hasHba matched and added weight 5
matched 'false' against 'false' using operator 'equalOrMore'
object : optional attribute hasHba did not match
matched 'false' against 'false' using operator 'more'
object : attribute is10GbES matched and added weight 5
matched 'true' against 'true' using operator 'equalOrMore'
object : attribute node-powervalue matched and added weight 1
matched '7705' against '7705' using operator 'range'
object : attribute node-powervalue matched and added weight 2
matched '7705' against '7705' using operator 'range'
object : attribute node-powervalue matched and added weight 5
matched '7705' against '7705' using operator 'range'
object : attribute node-powervalue matched and added weight 10
matched '7705' against '7705' using operator 'range'
node "sno1apl5p1" matches with weight 36
evaluating node sno1pw1
object : attribute OS-typ matched and added weight 5
matched 'SunOS' against 'SunOS' using operator 'equal'
object : optional attribute CPU-arch did not match
matched 'sun4u' against 'sun4us' using operator 'equal'
object : attribute hasHba matched and added weight 5
matched 'false' against 'false' using operator 'equalOrMore'
object : optional attribute hasHba did not match
matched 'false' against 'false' using operator 'more'
object : attribute is10GbES did not match
matched 'true' against 'false' using operator 'equalOrMore'
node "sno1pw1" does not match
sno1apl5p1 [36]
FlexFrame Agents
FlexFrame Agents Guide 116
4.5 Management of Performance, File System and Accounting
FlexFrame AppAgents and FlexFrame FrameAgents have the possibility of collecting
performance data and file system data and generate cyclically defined value tuples with
them. The result tuples are frame-, pool-, group-, node-, system- and service-specific.
The result of every cycle is stored in a collection file. Collection files are temporary data
containers for the storing of the data. A collection file enables the storing of data from
several report cycles.
The collection files themselves are organised in a ring buffer which enables the ascer-
tainment of the number and size of the collection files in a sizing process.
4.5.1 Performance and Accounting Management
The performance- and accounting management is already prepared in the standard con-
figuration. The performance and accounting functions are therefore in form of colletfiles
directly at disposal in supportcases as long as the necessary licence has been acquired.
The domain manager is necessary for the further processing and storing of the raw data
in the data bank.
4.5.2 File System Utilization
FlexFrame AppAgents and FlexFrame FrameAgents are able to determine the data for
the file system utilization. A parameter file defines, which file systems and directories are
monitored. The file system monitoring as well as the writing of the corresponding collec-
tions is directly activated by entry of file systems or directories which have to be moni-
tored in the pool specific configuration file myAMC_FA_Checks.xml.
4.5.3 Monitoring Cycles
For the collection of file system data and directory data variable cycle times can be de-
fined:
Perfdata_SpaceUtil_FSCycleTime
specifies how often (in seconds) FlexFrame Agents checks the file system utilization
for performance data.
Perfdata_SpaceUtil_DirCycleTime
specifies how often (in seconds) FlexFrame Agents checks the directory utilization for
performance data.
FlexFrame Agents
117 FlexFrame Agents Guide
4.6 Operating Mode
FlexFrame Agents has various operating modes. The operating modes enable the Flex-
Frame Agents to be used for simple monitoring tasks with automatic alarms, through to
semi or fully autonomous operation of a service.
Particularly in the startup and learning phases, this flexible configurability permits succes-
sive replacement of manual interventions by autonomous reactions.
The following operating modes are provided to implement the reactions:
Event mode
Local reaction mode
Central reaction mode
The operating modes can be defined at service level. This enables the degree of auton-
omous functions to be configured on an individual basis in accordance with the priority
and importance of a service. Parameterization of these modes takes place using the
parameters
Service_MaxRestartNumber
Node_MaxRebootNumber
Node_MaxSwitchOverNumber
4.6.1 Event Mode
No autonomous functions are performed in event mode. The events are just reported to
an event console in the form of messages. The administrator can then decide whether
the reactions proposed by the FlexFrame Agents are appropriate and then execute these
manually. This mode is particularly recommended in the introductory phase. For this
purpose the parameter variables for the number of permissible restarts, reboots and
switchovers is set to 0.
4.6.2 Local Reaction Mode
Local reaction mode is the standard operating mode for a FlexFrame installation.
The reactions for restart, reboot and switchover are activated for all services in the my-
AMC.FA.xml parameter file, and at the same time the number of reaction attempts for a
reaction type is defined.
4.6.3 Central Reaction Mode
In central reaction mode the reactions are not initiated by the FlexFrame AppAgents.
Here each of the local parameter variables for restart, reboot and switchover is set to 0.
The reactions are initiated from a central position.
FlexFrame Agents
FlexFrame Agents Guide 118
An option is that the reactions are created externally on the basis of the traps sent and
forwarded to the FlexFrame AppAgents via the BlackBoard.
FlexFrame Agents
119 FlexFrame Agents Guide
4.7 Autonomous Operation of a FlexFrame Infrastructure
In every application environment the user needs an option for starting and stopping new
application instances. This will be done with the Fujitsu FlexFrame scripts.
4.7.1 FlexFrame Agents and Fujitsu FlexFrame Scripts
The reactions and user interactions take place by direct use of the Fujitsu FlexFrame
scripts.
The Fujitsu FlexFrame scripts are responsible for starting and stopping the SAP instanc-
es. They permit the SAP instances to be visualized and also supply the information re-
quired for the Agents to detect the user interactions.
FlexFrame Agents
FlexFrame Agents Guide 120
4.8 FlexFrame Agents and User Interactions
The autonomous reactions and the user interactions influence the status of FlexFrame
Agents. Status changes and user interactions are logged in the log files and, if so config-
ured, also sent as traps.
Status creation begins with starting up the agents and the evaluation of the parameteriza-
tion information.
When the FlexFrame AppAgents are started up they read the parameterization file and
subsequently know their job and operating mode. The FlexFrame Agents send a startup
trap when they start up.
Service instances that are already running are automatically recognized and managed.
The most important commands, reactions and their events in the various startup situa-
tions of the FlexFrame Agent are described in the following.
Restart, no services running:
The FlexFrame AppAgent sends only a node startup alert but no alerts for ser-
vices which do not exist.
Restart of the FlexFrame AppAgent when services are running:
The agent reports its own restart and availability and the various SAP services
which are running.
After reboot:
The agent reports its own startup and availability and the successful startup of
each individual service.
After switchover:
The agent that takes over control starts the services described in the testament
and send events corresponding to these.
FlexFrame Agents
121 FlexFrame Agents Guide
4.8.1 Starting/Stopping/Status of FlexFrame Agents
4.8.1.1 Starting the FlexFrame Agents Manually
When the FlexFrame Agents are started an Agent start trap is sent.
A FlexFrame AppAgent can be started manually using the following command:
/etc/init.d/myAMC.FA_AppAgent start
/opt/myAMC/FA_AppAgent/myAMC.FA_AppAgent start
A FlexFrame CtrlAgent can be started manually using the following command:
/etc/init.d/myAMC.FA_CtrlAgent start
/opt/myAMC/FA_CtrlAgent/myAMC.FA_CtrlAgent start
or
/opt/myAMC/FA_CtrlAgent/myAMC.FA_CtrlAgent start <pool_name>
/opt/myAMC/FA_CtrlAgent/myAMC.FA_CtrlAgent start Cust_1
A FlexFrame FrameAgent can be started manually using the following command:
/etc/init.d/myAMC.FA_FrameAgent start
/opt/local/myAMC/FA_FameAgent/myAMC.FA_FrameAgent start
4.8.1.2 Stopping the FlexFrame Agents Manually
When the FlexFrame Agents are stopped a Agent stop trap is sent.
A FlexFrame AppAgent can be stopped manually using the following command:
/etc/init.d/myAMC.FA_AppAgent stop
/opt/myAMC/FA_AppAgent/myAMC.FA_AppAgent stop
A FlexFrame CtrlAgent can be stopped manually using the following command:
/etc/init.d/myAMC.FA_CtrlAgent stop
/opt/myAMC/FA_CtrlAgent/myAMC.FA_CtrlAgent stop
or
/opt/myAMC/FA_CtrlAgent/myAMC.FA_CtrlAgent stop <pool_name>
/opt/myAMC/FA_CtrlAgent/myAMC.FA_CtrlAgent stop Cust_1
A FlexFrame FrameAgent can be stopped manually using the following command:
/etc/init.d/myAMC.FA_FrameAgent stop
/opt/local/myAMC/FA_FrameAgent/myAMC.FA_FrameAgent stop
FlexFrame Agents
FlexFrame Agents Guide 122
4.8.1.3 Status of the FlexFrame Agents
The status of the FlexFrame AppAgents can be inquired using the following command:
/etc/init.d/myAMC.FA_AppAgent status
/opt/myAMC/FA_AppAgent/myAMC.FA_AppAgent status
The status of the FlexFrame CrtlAgents can be inquired using the following command:
/etc/init.d/myAMC.FA_CtrlAgent status
/opt/myAMC/FA_CtrlAgent/myAMC.FA_CtrlAgent status
or
/opt/myAMC/FA_CtrlAgent/myAMC.FA_CtrlAgent status <pool_name>
/opt/myAMC/FA_CtrlAgent/myAMC.FA_CtrlAgent status Cust_1
The status of the FlexFrame FrameAgents can be inquired using the following command:
/etc/init.d/myAMC.FA_FrameAgent status
/opt/local/myAMC/FA_FrameAgent/myAMC.FA_FrameAgent status
FlexFrame Agents
123 FlexFrame Agents Guide
4.8.2 Starting/Stopping an SAP Instance
4.8.2.1 Starting an SAP Instance
A further SAP instance of the type DB, CI, APP, J, JC, SCS, ASCS, LC, ERS, … can be
started at any time. Use of the FlexFrame start scripts in a version which has been re-
leased for the FlexFrame Agents is mandatory.
An instance which is to be monitored via the Agents is started using the following script
calls on the Application Node:
ff_service.sh -s <SID> -t <SERVICETYPY -a start
ff_service.sh -s ml5 -t db -a start
...
The startup of an instance is documented by the agents using the following traps:
ServiceIsStarting trap,
ServiceIsStarted trap,
or
ErrorStartingService trap
FlexFrame Agents
FlexFrame Agents Guide 124
4.8.2.2 Stopping an SAP Instance
An active SAP instance of the type DB, CI, APP, J, JC, SCS, ASCS, LC, ERS, … can be
stopped at any time. Use of the FlexFrame stop scripts in a version which has been re-
leased for the FlexFrame Agents is mandatory.
Service instances which are stopped while the FlexFrame AppAgent is running are de-
tected by the agent and acknowledged with corresponding SNMP traps. In this case the
FlexFrame Agent can use mechanisms integrated into the FlexFrame solution to distin-
guish between an instance being intentionally stopped and a service or instance crash-
ing.
ff_service.sh -s <SID> -t <SERVICETYP> -a stop
ff_service.sh -s ml5 -t db -a stop
...
The following traps are sent:
ServiceIsStopping trap,
ServiceHasStopped trap,
or
ErrorStoppingService trap
FlexFrame Agents
125 FlexFrame Agents Guide
4.8.3 NoWatch/Watch an SAP Instance
The “NoWatch” is managed as an attribute of a service.
The attribute "No-Watch" causes a so attributierter Service:
continue to be monitored and thus is visible in the monitor information and the
GUI with its states.
no longer in autonomy actions / reactions will be included, ie e.g.:
o not restart the service if it is disturbed or fails.
o Do not restart the service in the internal reaction Reboot.
o Do not restart the service in internal or exteren Switchover.
Setting the attribute "nowatch" or the "Watch" can be done in various ways and is de-
scribed in the following chapters.
The attribute “NoWatch” of a service is stored persistently, so that also receive the re-
mains if an AppAgent is stopped / started.
Thus, the “NoWatch” is stored as persistent information, an AppAgent must be running on
the AN.
The persistent “NoWatch” is stored in the following file:
/opt/myAMC/vFF/vFF_<POOLNAME>/data/FA/reboot/Persist_FA_Info.txt
The persistent “NoWatch” can be canceled with a “Watch”.
Several services can be set by using wildcards in “NoWatch”. Thus, e.g. a complete SID
are set in “NoWatch”.
The use of wildcards implies, that future running services which match the wildcards, will
be in “NoWatch” state.
This is done via the Blackboard, which is supplied via the WebGUI or a Blackboard tool
script.
Hint:
A “NoWatch” with wildcards can be only repealed by a “Watch” with exact the
same wildcards.
When an AN fails, it will be done the reaction "external SwitchOver" by the CtrlA-
gent independent of a “NoWatch” of a service. This is necessary because it may be
possible, that are running services at the failed node, which are not in "nowatch".
FlexFrame Agents
FlexFrame Agents Guide 126
Therefore there will be also services in "nowatch" published in Swichtover-file,
however these services will not be started on the taking over node.
Services that are provided with an external switchover with the attribute
"nowatch", are published also in the SwitchOver-file. But they will not start on the
acquiring AN.
If a "nowatch" canceled, the current state of the service is reassessed and possibly re-
sumed in the monitoring.
So if this service is still delivered as faulty by the detector, thus in the state 'error', 'restart',
... is, then is subsequently reacted according to reaction rules.
If this service, which is in a fault condition is no longer supplied by the detector, since all
processes are gone, he is not reinstated in the monitoring. That such a service is to be
started in order to be readmitted to the autonomy with rules opponent.
Example:
Service 'A' is the state 'error', since some but not all of the necessary processes are miss-
ing.
Service 'B' is in the state 'error', because all the necessary processes are missing.
Service 'C' is in the state 'restart', since the reaction Restart not yet over, but there are
already processes there and be recognized by the detector.
"Nowatch" is taken back:
For Service 'A', the FA-agent will react, because the detector will continue to be reported
to the state 'error'.
For Service 'B' the FA Agent will not respond because the detector does not provide any
information about this service, since no processes of this service (more) are active.
For Service 'C' will be escalated according to the rules (Restart -> Reboot -> Switchover).
4.8.3.1 NoWatch an SAP Instance
An instance which is to be monitored via the FlexFrame Agents is “NoWatch” using the
following script calls on the Application Node:
ff_service.sh -s <SID> -t <SERVICETYPY -a nowatch
ff_service.sh -s ml5 -t db -a nowatch
Alternatively, the WebGUI or a Blackboard tool script can be used.
FlexFrame Agents
127 FlexFrame Agents Guide
4.8.3.2 Watch an SAP Instance
An instance which is to be monitored via the FlexFrame Agents is “Watch” using the
following script calls on the Application Node:
ff_service.sh -s <SID> -t <SERVICETYP> -a watch
ff_service.sh -s ml5 -t db -a watch
Alternatively, the WebGUI or a Blackboard tool script can be used.
FlexFrame Agents
FlexFrame Agents Guide 128
4.8.3.3 NoWatch / Watch with WebGUI
With the WebGUI a “Watch” / “NoWatch” can be sent for services, certain SIDs or even
an entire pool.
Whether a “Watch” / “NoWatch” to be sent for a single service, a SID or a whole pool, it
depends on which part of the system tree, the "Set to Watch" / "Set to NoWatch" is start-
ed.
“NoWatch” a service:
“NoWatch” a SID:
“NoWatch” a pool:
Afterwards:
FlexFrame Agents
129 FlexFrame Agents Guide
FlexFrame Agents
FlexFrame Agents Guide 130
4.8.3.4 NoWatch / Watch with BlackBoard-Tool
With the BlackBoard-Tool a “Watch” / “NoWatch” can be sent for services, certain SIDs or
even an entire pool.
Example:
bx902:~ # /opt/myAMC/FA_AppAgent/BBT_dialog.sh
BLACKBOARD V1.0 (c) FUJITSU Limited
-----------------------------------
Add command (y/n)? y
Enter PATH to BlackBoard (trailing '/', default
'/opt/myAMC/vFF/vFF_p1/data/FA/blackboard/'):
Enter BLACKBOARD (default 'blackboard.txt'):
Enter MD5-Key: (default 'myAMC.FA_BBTool')
Enter Command:
--------------
TimeStamp =19-08-14 13:20:18 (write-time)
TimeLong =1408447218 (write-time)
SRC-ID (default ‘myAMC.FA_BBTool’) =myAMC.FA_BBTool
SRC-Hostname=bx902
CMD-ID =
<<< EMPTY >>> ( 0) = default
…
Watch ( 6)
Nowatch ( 7)
…
Enter command-id:7
CMD-ID =Nowatch
Pool =p1
Group =ernst
SID =NL*
Service =*
Inst-Nr =*
Node =*
Value =
Validity(s) =600
Verify COMMAND: 'TimeStamp=19-08-14 13:20:18;TimeLong=1408447218;CMD-
ID=Nowatch;vFF=p1;Group=ernst;SID=NL*;Service=*;Node=*;Validity=600;Key=myAMC.FA_BBTo
ol;'
Continue writing COMMAND to BlackBoard (y/n)? y
bx902:~ #
FlexFrame Agents
131 FlexFrame Agents Guide
4.8.4 Suspend Autonomy
After an ‘SuspendAutonomy’ no actions will be done by the FA-agents.
The ‘SuspendAutonomy’ affects all pools, AppAgents and CtrlAgents.
‘SuspendAutonomy’ causes for an entire FlexFrame Orchestrator system autonomy reac-
tions of any kind will be disabled. So it is quasi a "emergency stop for automatic fault
responses of FlexFrame agents".
However, the system continues to be monitored and thus is visible in the monitor infor-
mation and the GUI with its states.
Setting or resetting of the ‘SuspendAutonomy’ can be done in various ways and is de-
scribed in the following chapters.
The ‘SuspendAutonomy’ is passed through the Blackboard to the FA-Agents. How quick-
ly the FA-Agents recognize this, depends on the "BlackboardCheckCycleTime".
If this 'BlackboardCheckCycleTime' for example, is 30 seconds, so it can take this time
until this Blackboard command is read from the FA agents and can be realised.
The ‘SuspendAutonomy’ is initiated and withdrawn from the FA WebGUI.
In addition, a command line interface available.
The ‘SuspendAutonomy’ is stored persistently, so that also receive the remains if an FA-
Agent is stopped / started.
The persistent ‘SuspendAutonomy’ is stored in the following file:
/opt/myAMC/VFF/vFF_<POOLNAME>/data/FA/reboot/Persist_SuspendAutono
my.marker
The persistent ‘SuspendAutonomy’ can be canceled with a ‘SuspendAutonomy’ reset .
Hint:
A ‘SuspedAutnomy’ stops the reactions of the FA-Agents, so with an active ‘Sus-
pendAutonomy’ no reactions will be done. But the systems continue to be moni-
tored and their statuses are displayed.
If, at the time of the taking back of a "SuspendAutonomy", for example via WebGUI
or CLI, even fault conditions exists, then will be responded by the FA-agents ac-
cording to the reaction rules.
FlexFrame Agents
FlexFrame Agents Guide 132
Example:
Service 'A' is the state 'error', since some but not all of the necessary processes are miss-
ing.
Service 'B' is in the state 'error', because all the necessary processes are missing.
Service 'C' is in the state 'restart', since the reaction Restart not yet ended.
"SuspendAutonomy" is taken back:
For service 'A', 'B' and 'C', the FA-agents will react according to the reaction rules.
'A' & 'B' will be restarts and 'C' will be escalated according to the rules (Restart -> Reboot
-> Switchover).
FlexFrame Agents
133 FlexFrame Agents Guide
4.8.4.1 Suspend Autonomy with WebGUI
With the WebGUI a ‘SuspendAutonomy’ can be sent and reset.
Set a ‘SuspendAutonomy’:
In the WebGUI far right at the top, press the button .
Then, a dialog appears in which the action has to be confirmed:
Reset a ‘SuspendAutonomy’:
In the WebGUI far right at the top, press the button .
Then, a dialog appears in which the action has to be confirmed:
A detected ‘SuspendAutonomy’ at AN/CN is visualized by an overlay icon in the WebGUI:
and/or
FlexFrame Agents
FlexFrame Agents Guide 134
FlexFrame Agents
135 FlexFrame Agents Guide
4.8.4.2 Suspend Autonomy per CLI (script)
To set and reset an ‘SuspendAutonomy’ there is a command line interface available.
Usage:
cn2:/opt/myAMC/scripts # ./fa_SuspendAutonomy.sh
Usage: ./fa_SuspendAutonomy.sh { set } to initiate an ‘SuspendAutonomy’
Usage: ./fa_SuspendAutonomy.sh { reset } to reset an ‘SuspendAutonomy’
cn2:/opt/myAMC/scripts #
Example:
cn2:/opt/myAMC/scripts # ./fa_SuspendAutonomy.sh set
Set an SuspendAutonomy in pool='Cust_2'.
Set an SuspendAutonomy in pool='Cust_3'.
Set an SuspendAutonomy in pool='SparePool'.
cn2:/opt/myAMC/scripts #
FlexFrame Agents
FlexFrame Agents Guide 136
4.9 Use cases
4.9.1 General
The FlexFrame Agents Suite solution offers various configuration and application options.
Application and Control Agents share the tasks for implementing autonomy in a Flex-
Frame solution.
The tasks of the Control and Application Nodes vary according to their configuration.
Some typical autonomy scenarios for the most important applications are presented in
the following:
Passive monitoring of your instances (notification mode); no reactions take place
(event mode).
Control of the instances’ availability using active FlexFrame AppAgents (local
reaction mode); the reactions are triggered by the FlexFrame AppAgent.
Control of the instances’ availability using passive FlexFrame AppAgents and
active High-Autonomy Control Agents (central reaction mode); the reactions are
triggered by the FlexFrame CtrlAgent.
The settings for notifications and reactions can be configured independently.
FlexFrame Agents
137 FlexFrame Agents Guide
4.9.2 Semi-autonomous operation
In certain situations it may make sense only to use notification functions and initially to
dispense with active intervention in the system. This scenario is practical, for example, for
using a central position to monitor various systems and analyze failure frequencies.
4.9.2.1 Monitoring of Application Instances
In order to monitor application instances, an FlexFrame AppAgent must run on every
agent which is equipped with detectors for the application instance to be monitored and
has been configured for this. The monitoring and reaction can be configured individually
for each application instance.
The service-specific parameters are set in the configuration section of the services. A
configuration for the event mode can achieved by the following settings in the Configsection Services Default section:
Service_SendTraps : true
Service_EnableMonitoring : true
Node_MaxRebootNumber : 0
Node_MaxSwitchOverNumber : 0
Service_MaxRestartNumber : 0
Servcice_TrapSendDelayTime : 0 or greater
With this configuration no reactions take place for any service type. However, event mes-
sages are sent if services are not available or have failed.
The parameters can, for example, be set on a service-specific basis for the semi-
autonomous operation of services.
In the Configsection Services APP section:
Service_SendTraps : true
Service_EnableMonitoring : true
Node_MaxRebootNumber : 0
Node_MaxSwitchOverNumber : 0
Service_MaxRestartNumber : 3
Service_MaxRestartTime : 120
With these setttings for the APP service, up to three restart attempts are made to render
an application service available again after it has failed. No reactions are implemented for
the other service types CI and DB. However, event messages are sent if services are not
available or have failed.
4.9.3 Autonomy for Application Instances
To permit autonomous operation of applications, FlexFrame Agents Suite provides the
option of monitoring instances and reacting actively to the failure of a service. The type of
reaction depends on the configuration set.
FlexFrame Agents
FlexFrame Agents Guide 138
The following parameters can be used for this purpose in the parameter file:
Service_SendTraps : true
Service_EnableMonitoring : true
Node_MaxRebootNumber : 2
Node_MaxSwitchOverNumber : 1
Service_MaxRestartNumber : 3
Service_TrapSendDelayTime : 0 or greater
Service_ReactionDelayTime : 0 or greater
MaxRebootTime : 120
4.9.3.1 Restart
Restarting a service is the first option FlexFrame Agents have of reacting to a service
failure.
For this purpose a Service_MaxRestartNumber greater than 0 is specified and also
the Service_MaxRestartTime. The Service_MaxRestartTime is evaluated by the
FF AppAgent.
Service_MaxRestartNumber : 10
Service_MaxRest artTime : 240
The time by which the service must be available again is the Service_MaxRestartTime of this service.
A maximum of Service_MaxRestartNumber attempts are made to restore availability
through this reaction; escalation to the next escalation level then follows.
FlexFrame Agents
139 FlexFrame Agents Guide
4.9.3.2 Reboot
Rebooting a node is a further option for an FlexFrame Agent to react and the next escala-
tion level after a restart.
For this purpose a Node_MaxRebootNumber greater than 0 is specified and also the
Node_MaxRebootTime. The Node_MaxRebootTime here is evaluated by both the
FlexFrame AppAgent and the FlexFrame CtrlAgent. After the Node_MaxRebootTime
has elapsed the Control Agent uses the Node_CheckAvailabilty command to check
the availability of the system. If the RebootNumber is set to two or three, this value is
increased by the corresponding factor.
Node_MaxRebootNumber : 2
MaxRebootTime : 120
The time by which a server and all its services must be available again is calculated by adding the Node_MaxRebootTime and the greatest Node_MaxRestartTime of all
services to be started on this server.
A maximum of Service_MaxRestartNumber attempts are made to restore availability
through this reaction; escalation to the next escalation level then follows.
4.9.3.3 Switchover
A distinction is made between the internal and external SwitchOvers.
4.9.3.3.1 Switchover intern
The internal SwitchOver of all a node’s services is a further reaction option for the Flex-
Frame Agent and the next escalation level after a reboot. An internal SwitchOver is initi-
ated by a failed node which actively wishes to transfer its services. This is therefore a
further reaction option for the FlexFrame Agent and the next escalation level after a re-
boot.
For this purpose a Node_MaxSwitchOverNumber greater than 0 is specified.
Node_MaxSwitchOverNumber : 1
The time by which a server and all its services must be available again is calculated by adding the Service_MaxRestartTime of all services to be started on this server.
4.9.3.3.2 Switchover extern
An external SwitchOver is initiated by a FlexFrame CtrlAgent which determines that an
Application Node is no longer working.
FlexFrame Agents
FlexFrame Agents Guide 140
4.9.4 Management of Performance, Capacity and Account-ing
The mangement of performance, capacity and accounting of the FlexFrame Agents Suite
is an optional function. The use of this component requires a standard installation for the
FlexFrame Agents and additionally the installation of the FlexFrame Domainmanager.
The function has to be activated in the configfile of the FlexFrame Agents.
Depending on the demands for mangement of performance, capacity and accounting the
report cycle and the size of the ringbuffer for the each performance collection type has to
be defined. Additionally the FlexFrame Domain manager has to be installed and started
on the active Control Nnode. The FlexFrame Agents supply the performance data nodes
and capacity data nodes service and SAP system specific as is standard without having
to use individual configurations.
The following example describes the default configuration. The default configuration sup-
plies the performance data nodes and capacity data nodes system specific and service
specific. Also the following groups are already predefined.
The groups supply a node specific view of the portions on the whole output consumption
of a node.
SAP
NONSAP
FA
Idle
4.9.5 File System Utilization
The management for the supervision of the fill state and the increase of file systems can
be activated through simple activation in the FA config file. The data for the file system
expenditure can be supplied by the FlexFrame AppAgents or the FlexFrame
FrameAgents.
The activation of the filesystem expenditure activates the default configuration. The de-
fault configuration supplies throught the FlexFrame FrameAgent the supervision of the
voIFF and dispatches an alert if this is filled more than 90% or the increase rate inside a
report cyle is more than 5%.
The ascertainment of the fill state of a file system and its increase rate can be ascer-
tained with relativly minor time expense and footprint.
FlexFrame Agents
141 FlexFrame Agents Guide
4.9.6 High Availability and Autonomy Rules
The FlexFrame Agents suite can be configured through a set of parameters for different
high availability and autonomy situations. In the following examples the interaction and
the different reaction scenarios of some parameters is shown.
The examples also show how the interplay of the parameters in special high availablity
situations an emergency event is triggered and how the service priority and the service
capacity requirements influence the availability in a failure scenario.
Possible application scenario:
In a data center, there are one or more spare nodes within a pool. In case of failure of an
application server spare nodes take over the services that are running on the failed appli-
cation server.
Simultaneous failure multiple servers, only the number of application nodes can be made
available again on a Sparenode as there are free spare nodes within the pool, where are
the failures. If more than two nodes have failed within 5 minutes, then is this to be report-
ed over an emergency event. If more than 4 nodes have failed within 2 days, then this
should also be reported over an emergency event to a central console.
In general, the rules for each pool can be parameterized by the other pools within a Flex
Frame installation independently.
The following parameters will be used:
Node_SwitchOverTyp: service or node (default = node):
Depending on the setting of this Paramter node or service based SwitchOver-Files
will be written.
EscalationType: Service or Node (default = Node):
Depending on the setting of this Paramter the escalation will be done for all services
on a node (restart reboot switchover), or the escalation will be done for a single
service (restart swtchover (single service)).
TakeOverStrategy: FirstFit or LowPrioFit (default = FirstFit):
Depending on the setting of this Paramters wins in a failure scenario, the server has
become the first candidate for the switchover file of the failed node or the server that
had the lowest priority of its services.
Pool independet spare (default = no pool independent spare):
If no spare nodes are available, in failure scenarios in a pool, so there is the possibil-
ity to define a Spare Pool. Then, when applying no Sparenode in the pool for a
switchover file, then tries to get a Sparenode from the spare pool.
FlexFrame Agents
FlexFrame Agents Guide 142
Parameter must be set in Pool and Sparepool: CtrlAgentRole: WorkWithSparePool (default = Normal)
SparePoolName: <Name of SparePool> (default = NONE)
SpareGroupName: <Name of SpareGroup> (default = NONE)
TakeOverRule (default = SpareNode):
This parameter determines the type of the adoption of a switchover file.
possible settings: SpareNode: Only a spare node will Start the services from taken SwitchOver-File.
add (Add, addition): Additional start the services from taken SwitchOver-File.
replace (move): Running services will be stopped and a SwitchOver-File will be
written therefore, Start the services from taken SwitchOver-File. substitute (displacement): Running services will be stopped, start the services
from taken SwitchOver-File.
dynamic: Cause of prios there will be done one of the TakeOver-rules SpareNode,
add, replace, substitute, or none.
For dynamic rule the following parameters are important: Dyn_Spare_min, Dyn_Spare_max; Prio-range
Dyn_Add_min, Dyn_Add_max; Prio- range
Dyn_Replace_min, Dyn_Replace_max; Prio- range
Dyn_Substitute_min, Dyn_Substitute_max; Prio-Bereich
Dyn_Spare_exclusive: Switch for „exclusive for spare“
Vip Services: VIP Services are services for which there is a special takeover scenario
outside of the standard. That is, for example, the ERS (enque Replica service).
The High Availability Rules can be divided into static and dynamic rules.
FlexFrame Agents
143 FlexFrame Agents Guide
4.9.6.1 High Availability: Static Rules
In the Static Rules that apply generally to any failure in the affected pool. The basic rules
Sparenode-rule, Replace rule, add-rule-rule and substitution of the static rules.
Example: On an application server to run multiple Appliaktionsservices.
Sparenode-rule:
P01 with ci (priority 2) and the associated database db P01 (priority 1) fails. In the pool
there is no spare nodes. Thus, there is no high availability for the canceled services be-
cause there is no free node for the acquisition of the failed service.
Replace-rule:
P01 with ci (priority 2) and the associated database db P01 (priority 1) fails. Each applica-
tion node in the same pool with lower service priorities applying for the acquisition of the
switchover file. Own services with worse priority (ie 3 and greater) are stopped and then
published as the switchover file.
4.9.6.2 High Availability: Dynamic Rules
Upon failure of Services, Dynamic Rule allows a differentiated use of available spare
nodes and free capacity on application nodes which are not yet fully utilized, or run on
those with lower priority services.
For this use the parameterization of service priority and a service associated capacity
(power value) is important. When not only the spare node rule is used, but one of the
other rules, individual prioritization of services is necessary.
FlexFrame Agents
FlexFrame Agents Guide 144
4.9.6.3 High Availability: Examples
In the following examples we have a production system with P01 3 instances and a quali-
ty assurance system with 3 instances and a development system. The production system
has the highest priority, then comes the quality assurance system and the development
system with the lowest priority. The services are operated in a flexible frame with a pool,
which consists of 12 application nodes.
The application node Node_10, Node_11 and Node_12 are spare nodes, since no ser-
vices running on that node.
4.9.6.3.1 Example 1 (SpareNode)
System ID
Servicetyp Prio Power value
Applikation node
Pool Group Scenario Node- capacity
P01 DB 1 6000 Node_01 Pool_1 Default t 10000
P01 CI 1 3000 Node_02 Pool_1 Default t 10000
P01 App 3 2000 Node_03 Pool_1 Default t 10000
Q01 DB 10 3000 Node_04 Pool_1 Default 10000
Q01 CI 10 1000 Node_05 Pool_1 Default 10000
Q01 App 30 1000 Node_06 Pool_1 Default 10000
E01 DB 20 500 Node_07 Pool_1 Default 10000
E01 CI 20 500 Node_08 Pool_1 Default 10000
E01 App 40 500 Node_09 Pool_1 Default 10000
Node_10 Pool_1 Default 10000
Node_11 Pool_1 Default 10000
Node_12 Pool_1 Default
Default parameter settings:
Parameter Value Value Options
Node_TakeOverRule SpareNode SpareNode,Addrule,Displacement, Supplementation, dynamic
Noed_SwitchOverTyp Node Node, Service
Take_OverStrategy FirstFit FirstFit, LowPrioFit
The failure of the 3 node with the productive instances of the SAP system P01 is as-
sumed in this scenario of the 3 spare nodes.
FlexFrame Agents
145 FlexFrame Agents Guide
FlexFrame Agents
FlexFrame Agents Guide 146
4.9.6.3.2 Example 2 (SpareNode)
System ID
Service- typ
Prio Power value
Applikation node
Pool Group Scenario Node- capacity
P01 DB 1 6000 Node_01 Pool_1 Default t+60 10000
P01 CI 1 3000 Node_02 Pool_1 Default 10000
P01 App 3 2000 Node_03 Pool_1 Default 10000
Q01 DB 10 3000 Node_04 Pool_1 Default 10000
Q01 CI 10 1000 Node_05 Pool_1 Default 10000
Q01 App 10 1000 Node_06 Pool_1 Default 10000
E01 DB 10 500 Node_07 Pool_1 Default t 10000
E01 CI 10 500 Node_08 Pool_1 Default t 10000
E01 App 10 500 Node_09 Pool_1 Default t 10000
Node_10 Pool_1 Default 10000
Node_11 Pool_1 Default 10000
Node_12 Pool_1 Default 10000
Default parameter settings:
Parameter Value Value Options
Node_TakeOverRule SpareNode SpareNode,Addrule,Displacement, Supplementation, dynamic
Noed_SwitchOverTyp Node Node, Service
Take_OverStrategy FirstFit FirstFit, LowPrioFit
In this scenario at time t the 3 nodes from the development system fails. The at time t
existing 3 spare nodes take over these services and they are available again. Cause of
the consumption of the existing spare capacity, there is at time t+x no high availability for
the failure of an productive services of the P01 system.
FlexFrame Agents
147 FlexFrame Agents Guide
4.9.6.3.3 Example 3 (dynamic)
System ID
Service- typ
Prio Power value
Applikation node
Pool Group Scenario Node- capacity
P01 DB 1 6000 Node_01 Pool_1 Default 10000
P01 CI 1 3000 Node_02 Pool_1 Default 10000
P01 App 3 2000 Node_03 Pool_1 Default 10000
Q01 DB 10 3000 Node_04 Pool_1 Default 10000
Q01 CI 10 1000 Node_05 Pool_1 Default 10000
Q01 App 10 1000 Node_06 Pool_1 Default 10000
E01 DB 10 500 Node_07 Pool_1 Default 10000
E01 CI 10 500 Node_08 Pool_1 Default 10000
E01 App 10 500 Node_09 Pool_1 Default 10000
Node_10 Pool_1 Default 10000
Node_11 Pool_1 Default 10000
Node_12 Pool_1 Default 10000
Default parameter settings:
Parameter Value Value Options
Node_TakeOverRule dynamic SpareNode,Addrule,Displacement, Supplementation, dynamic
Noed_SwitchOverTyp Node Node, Service
Take_OverStrategy FirstFit FirstFit, LowPrioFit
Additional parameters that are needed along with the dynamic Node_TakeOverRule:
Paramater Value
Dyn_Spare_min 1
Dyn_Spare_max 2
Dyn_Add_min 3
Dyn_Add_max 30
Dyn_Replace_min 100
Dyn_Replace_max 100
Dyn_Substitute_min 200
FlexFrame Agents
FlexFrame Agents Guide 148
Dyn_Substitute_max 200
Dyn_Spare_exclusive true
By using the Node_TakeOverRule in dynamic mode, the spare nodes for the productive
system is protected because the spare nodes are used with these parameters only for
services with priority <= 2.
Upon failure of the Q or E systems, these are added by other systems. With add–rule
only the nodes which possess services with lower or equal priority apply (from pool,
group).
In our example the node with P instances would not accept any services from Q and E
systems and therefore do not affect their performance.
The example also shows that the failure of the node node_3, with the productive applica-
tion instance of the P01 system, is not the high availability of this service through the use
of an idea node, but by the add rule. I.e. this app instance would run after a failure sce-
nario on one node with other services.
Note:
If new SAP services added to a pool, these have default values first.
The priority assigned to a service and the power value can be clearly visualized with the
FlexFrame ConfigGui, and be changed as needed directly in this view.
FlexFrame Agents
149 FlexFrame Agents Guide
4.9.6.3.4 Example 4 (sercive specific reaction)
Failure scenarios ERS service
The enqueue replication service has a special role in a high-availability scenarios.
The ERS belongs to an SCS or ASCS service only and assumes the failure of the asso-
ciated SCS or ASCS its task and thus becomes the new SCS / ASCS. Thus, in a second
failure of the SCS / ASCS another ERS instance is available it must be started on another
server. Therefore, this ERS is made available again by switchover to another node.
In the following example the ERS will be available at the spare node Node_11, but in
dynamic mode it my also be available will add- or other rule.
System ID
Service- typ
Prio Power value
Applikation node
Pool Group Scenario Node- capacity
P01 DB 1 6000 Node_01 Pool_1 Default 10000
P01 SCS 1 3000 Node_02 Pool_1 Default t 10000
P01 App 3 2000 Node_03 Pool_1 Default 10000
P01 ERS 3 2000 Node_03 Pool_1 Default t+1 10000
Q01 DB 10 3000 Node_04 Pool_1 Default 10000
FlexFrame Agents
FlexFrame Agents Guide 150
Q01 CI 10 1000 Node_05 Pool_1 Default 10000
Q01 App 10 1000 Node_06 Pool_1 Default 10000
E01 DB 10 500 Node_07 Pool_1 Default 10000
E01 CI 10 500 Node_08 Pool_1 Default 10000
E01 App 10 500 Node_09 Pool_1 Default 10000
Node_10 Pool_1 Default 10000
Node_11 Pool_1 Default 10000
Node_12 Pool_1 Default 10000
FlexFrame Agents
151 FlexFrame Agents Guide
4.9.6.3.5 Example 5 (group based rules)
The rules for the autonomy and high availability (Sparenode, Add, Replace and Dynamic
Rule) usually always apply to the entire pool.
If this high availability rules are defined at the group level, the response for the same
service type can be very different depending on which group the service is running on.
Defining rules on group level which together with the dynamic rule a plurality of different
reaction scenarios for failure.
FlexFrame Agents
FlexFrame Agents Guide 152
4.10 FlexFrame Agents Work and Log Files
The functions of the FlexFrame Agents are documented in various files. These files may
not be changed manually as this can impair error free operation of the FlexFrame Agents
or result in errored reactions.
These files are created dynamically during ongoing operation. Deleting these files leads
to a status in which the FlexFrame Agents reorganize themselves, and from this point
they reevaluate the situation from the current viewpoint without any previous knowledge.
Base directory: /opt/myAMC/
Base directory: /opt/local/myAMC/
A detailed description can be found in chapter 3.4 resp. 10.3.
FlexFrame Agents
153 FlexFrame Agents Guide
4.11 Migration of FlexFrame Agent Versions on Pool Level
The FlexFrame Agents offer a whole raft of strategies and functionalities for installing and
activating patches and new release versions for a wide range of security, test and release
scenarios.
The administrator can use the update and activation functionality provided by the agents
in line with his/her requirements. The following basic functions are available:
A Reading and observing update, patch and release notes
B Installation of a new FlexFrame Agent version parallel to an operating FlexFrame
Agent version. All data and configuration information for the operating FlexFrame
Agent version are retained.
C Taking over of the configuration data for the new FlexFrame Agent version using the
FlexFrame migration tool
D Pool-by-pool configuration/parameterization and activation of the new FlexFrame
Agent version
E Testing of a new FlexFrame Agent version, e.g. in a separate test pool, if required by
deactivating the autonomous reactions for test operation
The following activities are required to install or update a FlexFrame Agent patch or a
newer release version.
1. Reading of the update, patch and release notes and observation of any required
modifications and special features, in particular in the event of simultaneous updating
of FlexFrame and operating system versions and patches.
2. Installation of a patch or a new release version (FlexFrame CtrlAgent and FlexFrame
AppAgent).
FlexFrame Agents
FlexFrame Agents Guide 154
3. Parameterization/configuration, possibly using the FA migration tool.
3.1. Copying the parameters from the active agent version to the migration configu-
ration directory using the FlexFrame migration tool. Normally the following call
is sufficient to do this:
/opt/myAMC/FA_CtrlAgent/MGRTool.sh --target-release=<release>
--migrate-pool=<pool> --backup
This following call migrates the configuration of the pool1 pool to the version
V20K23 for the associated FlexFrame Agents.
/opt/myAMC/FA_CtrlAgent/MGRTool.sh --target-release=V20K23
--migrate-pool=pool1 --backup
Starting pool migration
pool: pool1
source release: V20K22
target release: V20K23
migration dir: /opt/myAMC//vFF/vFF_pool1/Migration.V20K23.2005-05-10
migration succeeded
see file /opt/myAMC/vFF//vFF_pool1/Migration.V20K23.2005-05-10/
MIGRATION-INSTRUCTIONS.txt
for details and installation instructions.
The modified files are written into a subdirectory and no current files are modi-
fied. In addition, the migration directory contains a backup of all current files. All
new/modified parameters are listed in the MIGRATION-INSTRUCTIONS.txt
file.
3.2. Testing, parameterizing and configuring the parameters taken over in the TrapTargets.xml, myAMC_FA_Groups.xml, myAMC_FA_Rules.xml,
myAMC_FA.xml, myAMC_FA_GUI.xml and myAMC_FA_SD_Sec.xml files.
3.3. Testing any new parameters and, if necessary, modifying the default values en-
tered.
3.4. Parameterizing and configuring of FlexFrame/operating system version de-
pendencies if the FlexFrame basis is updated at the same time.
3.5. Check the modifications made by the migration tool, according to the file
MIGRATION-INSTRUCTIONS.txt in result directory of migration.
FlexFrame Agents
155 FlexFrame Agents Guide
4. Pool-specific deactivation of the active FlexFrame Agent.
4.1. Stopping the FlexFrame CtrlAgent for the pool whose agents are to be updated.
/etc/init.d/myAMC.FA_CtrlAgent stop <pool>
4.2. Stopping the FlexFrame AppAgents on all nodes of the pool whose agents are
to be updated.
/etc/init.d/myAMC.FA_AppAgent stop
5. Pool-specific activation of the new FlexFrame Agents.
5.1. Modifying the active agent version in the .info file in the associated pool di-
rectory. To do this the pool.release.current entry must be adapted ac-
cordingly. The syntax is VxxKyy. This syntax is mandatory.
control1:/ # cat /opt/myAMC/vFF/vFF_pool1/.info
# Version V20K17
pool.release.base=V20K17
pool.release.current=V20K23
This pool will use the FlexFrame Agents of the versions V20K23.
Alternately this file can be transferred from the migrated configuration to the
configuration directory:
cd /opt/myAMC/vFF/vFF_pool1
cp ./Migration.<version>.<date>/.info .
5.2. Transferring the migrated configuration to the configuration directory.
cd /opt/myAMC/vFF/vFF_pool1
cp ./Migration.<version>.<date>/config/* ./config
5.3. Starting the FlexFrame AppAgents on all Application Nodes of the updated
pool.
/etc/init.d/myAMC.FA_AppAgent start
5.4. Starting the FlexFrame CtrlAgent.
/etc/init.d/myAMC.FA_CtrlAgent start pool1
FlexFrame Agents
FlexFrame Agents Guide 156
6. Checking the new active FlexFrame Agent version.
6.1. Checking the agent processes.
Output for the example of the pool pool1:
control1:/ # /etc/init.d/myAMC.FA_CtrlAgent status pool1
Status of myAMC.FA_CtrlAgent ( myAMC_FA_CtrlAgent ) in vFF='pool1' at
host 'control1' ...
root 17608 1 0 13:41 pts/5 00:00:00 ./myAMC_FA_CtrlAgent
vFF=pool1 -lf=/opt/myAMC/vFF/vFF_pool1/log/CtrlAgt/
root 17610 17608 1 13:41 pts/5 00:00:00 ./myAMC_FA_CtrlAgent
vFF=pool1 -lf=/opt/myAMC/vFF/vFF_pool1/log/CtrlAgt/
root 17664 17610 0 13:42 pts/5 00:00:00 ./myAMC_FA_CtrlAgent
vFF=pool1 -lf=/opt/myAMC/vFF/vFF_pool1/log/CtrlAgt/
root 17665 17664 0 13:42 pts/5 00:00:00 ./myAMC_FA_CtrlAgent
vFF=pool1 -lf=/opt/myAMC/vFF/vFF_pool1/log/CtrlAgt/
root 17668 17664 0 13:42 pts/5 00:00:00 ./myAMC_FA_CtrlAgent
vFF=pool1 -lf=/opt/myAMC/vFF/vFF_pool1/log/CtrlAgt/
control1:/ #control1:/ # echo $?
0
6.2. Checking agent messages at startup.
control1:/ # /etc/init.d/myAMC.FA_CtrlAgent start pool1
Found vFF='pool1'.
Checking the files for vFF='pool1'...
Starting myAMC.FA_CtrlAgent ( myAMC_FA_CtrlAgent ) Version ( V20K23 ) in
vFF='pool1' at host 'control1' ...
6.3. Diagnosis and checking that the shown data is correct
6.4. Performing FlexFrame Agents tests (restart, reboot, etc.).
Steps 1, 2 and 3 can take place while FlexFrame Agents are active. The FlexFrame
Agents functions are not available only for a brief period between deactivation of the
active agent version and activation of the new agent version.
Note that only version-compatible FlexFrame CtrlAgents, FlexFrame AppAgents and
FlexFrame versions can be used. Compatibility of the agent versions with various Flex-
Frame versions results in dependencies which must be taken into account.
FlexFrame Agents
157 FlexFrame Agents Guide
4.12 The FlexFrame Migration Tool
The FlexFrame migration tool is used to migrate configurations of a selected pool to and
from a particular FlexFrame Agent version. The FlexFrame migration tool also enables
you to merge configuration files.
4.12.1 Pool Mode
Pool mode generates a migrated configuration in the subdirectory Migration.<version>_<timestamp>, including the backup of the current files. To
enable the migrated configuration to be used it must be copied into the relevant configu-
ration directory of the pool concerned.
Required / useful parameters:
-p/--migrate-pool=<pool>
-r/--target-release=<release>
-b/--backup
[-V/]--verbose]
[-d/--pools-basedir=<dir>] (optional)
[-c/--clean] optional
[-s/--source-release=<release>] (optional)
Example:
MGRTool.sh --migrate-pool=<pool> --target-release=<release> --backup
4.12.2 File Mode
File mode merges two files which are in FlexFrame Agents config format. The two files are defined with the parameters merge-file and template.
File mode can only be used on files which are in FlexFrame Agents config format (e.g.
myAMC_FA.xml). The myAMC_Pools.xml and myAMC_FA_Groups.xml files are not in
this format. These files can therefore not be migrated using file mode, but only in pool
mode.
Required / useful parameters:
-m/--merge-file=<file>
-t/--template=<template>
-o/--out-file=<file>
[-V/--verbose]
[-c/--clean] (optional)
FlexFrame Agents
FlexFrame Agents Guide 158
Example:
MGRTool.sh –-merge-file=File1.xml --template=File1-default.xml
--out-file=File-out.xml
4.12.3 Usage
Usage:
/opt/myAMC/FA_CtrlAgent/MGRTool.sh --help
myAMC_FA_MGR_Tool 1.0 (May 4 2005)
usage:
-m/--merge-file=<file> ...
-p/--migrate-pool=<pool> ...
-l/--list-releases
-h/--help print help
-v/--version print version info
command line reference:
-m/--merge-file=<file> merge this file with a template
-t/--template=<template> template to be used for merging
-o/--out-file=<file> write merged output to this file. (use '-' for std
output)
-p/--migrate-pool=<pool> migrate specified pool
-s/--source-release=<release> migrate from specified release [optional, deter-
mined automatically]
-r/--target-release=<release> migrate to specified release
-b/--backup create backup of all files
-d/--pools-basedir=<dir> base dir of all pools (default: /opt/myAMC/vFF)
-l/--list-releases list all available releases
misc options:
-c/--clean remove unnecessary files and config settings
-V/--verbose be verbose
-lf/--logfile <file> write to this log file
-lp/--logpath <path> create log file in this directory
FlexFrame Agents
159 FlexFrame Agents Guide
Explanation of the Parameters of the FlexFrame Migration Tool
-m/--merge-file=<file> Merges the specified file with the template
-t/--template=<template> Specifies the template to be used for the
merge
-o/--out-file=<file> Write merged results into the specified file
(use '-' for standard output)
-p/--migrate-pool=<pool> Specifies the pool to be migrated
-s/--source-release=<release> Migrates from specified release (parameter
is optional). If parameter is not specified, the
version of the pool to be converted is used.
-r/--target-release=<release> Migration to the specified release
-b/--backup Generates a backup of all files
-d/--pools-basedir=<dir> Basic directory of the pools (default: /opt/myAMC/vFF)
-l/--list-releases Lists all available (installed) releases
-c/--clean Removes unnecessary files and configura-
tion settings
-V/--verbose Detailed output during migration
-lf/--logfile <file> Writes log messages to the specified log file
-lp/--logpath <path> Generates the log file in the specified direc-
tory
-h/--help Prints usage, as shown above
FlexFrame Agents
FlexFrame Agents Guide 160
4.13 Command Line Interface fa_list_services.sh
The command line interface is used to list the states of all FlexFrame services.
Usage:
/opt/myAMC/scripts/fa_list_services.sh -h
usage: ./fa_list_services.sh [-iIvcChH] [-o services|nodes|spares] [<pool> ...]
-o: opmode:
services: show services
nodes: show nodes
spares: show spare nodes
-i: show active and inactive services and nodes
-I: show only inactive services and nodes
-v: verbose mode
-c force use of colors for state information
-C suppress colors for state information
-H suppress headers
-h show usage
-? show usage
if no pools are specified, services of all pools will be shown.
Example:
/opt/myAMC/scripts # ./fa_list_services.sh -C p1
Pool |Group |Hostname |SID |Type |Id|Node|H-Repl|H-MNrole|State
p1 |Trauben |pw250a |OEC |db | | | | |RUNNING
p1 |g1 |rx600a |OEP |db | | | | |RUNNING
The fields Node, H-Repl (HANA Replica) and H-MNrole (HANA Multi Node Role) are
for HANA systems only.
FlexFrame Agents
161 FlexFrame Agents Guide
4.14 Command Execution at All Nodes of a Pool / ALL_POOLS
For execution of a command at all nodes of a pool you can use the script allnodes:
Usage:
cn2:/opt/myAMC/scripts # ./allnodes
allnodes <pool> "<command>"
.
allnodes ALL_POOLS "<command>"
allnodes ALLPOOLS "<command>"
allnodes ALL "<command>"
cn2:/opt/myAMC/scripts #
Example:
cn2:/opt/myAMC/scripts # ./allnodes FCK date
Pool: FCK
node pw250c:
Tue Jun 27 10:28:21 CEST 2006
node pw250d:
Tue Jun 27 10:28:21 CEST 2006
node pw650a:
Tue Jun 27 10:28:21 CEST 2006
node blade4:
Tue Jun 27 10:28:21 CEST 2006
node rx300a:
Tue Jun 27 10:28:21 CEST 2006
node rx300b:
Tue Jun 27 10:28:22 CEST 2006
cn2:/opt/myAMC/scripts #
163 FlexFrame Agents Guide
5 WebInterface
The FlexFrame WebGui visualizes all nodes and services present in a FlexFrame system
insofar as they are monitored by an FlexFrame AppAgent. The display shows the status,
availablity and messages of the Application and Control Agents.
5.1 Installation / Configuration
5.1.1 Prerequisites
On the Control Node an Apache-Tomcat Servlet Container must be installed. Currently
Tomcat >= 5.5.x is supported. Additionally a Oracle Java JRE or SDK >= 1.6.0 must be
installed.
Prerequisites for the clients are FireFox >= 3.0 or Internet Explorer >= 6.0 and the Oracle
Java plugin >= 1.6.0.
5.1.2 Installation
The installation package is called myAMC.FA_WebGui-<X.Y-Z>.i386.rpm.
Install the relevant package on the Control Node with
rpm –i myAMC.FA_WebGui-<X.Y-Z>.i386.rpm
The files are installed under /opt/myAMC/FA_WebGui.
5.1.3 Configuration
5.1.3.1 Web Server
Provided no paths have been changed in the FlexFrame Agentsconfiguration, the config-
uration runs out of the box.
Changes to the web server require Tomcat to be restarted or reloaded.
5.1.3.2 Login IDs
Login IDs are stored in the file /opt/myAMC/config/amc-users.xml. User names
and passwords can be entered simply here. No administration of rights is provided in the
current version; each user has the same rights and sees all pools, groups and systems.
WebInterface
FlexFrame Agents Guide 164
The file’s as-supplied status is as follows:
<?xml version="1.0"?>
<users>
<user name="admin" password="FlexFrame" role="admin, user"/>
<user name="myAMC" password="FlexFrame" role="user"/>
<user name="ByD" password="FSC" role="user"/>
</users>
Each user is assigned an entry in the following format:
<user name="admin" password="FlexFrame" role="admin, user"/>
The name field contains the user name and the id field the password. The remaining
fields must be taken over as they have been defined.
The default values for user and password are:
User: myAMC
PW: FlexFrame
The configuration file is an XML file whose format must be valid, otherwise no
user can log in. You can check this by opening the file in Internet Explorer, for
example. If the file is displayed without an error message, it is also accepted by
the WebGui.
If changes are made, the FlexFrame WebGui must be restarted or reloaded (e.g. using
the Tomcat Service Manager) or Tomcat must be restarted or reloaded.
To add/remove a user for the WebGui in FlexFrame or to change a password of one
user, the file
‘/opt/myAMC/config/amc-users.xml’ should be used. Only in this file defined users are
able to login through the WebGui. The login dialog will be shown in the WebGui if the
program is run as an Applet in the web browser or as a JNLP java application.
In case that the WebGui is started as a normal java application with the commando
‘webgui.sh’ or if the configuration WebGui is started with the commando ‘configgui.sh’,
there will be no login dialog to ask for the credential of the user. The user in this case
should be defined in the file
‘/opt/myAMC/FA_WebGui91/locations/local/location.properties’. The username/password
defined in this file should map to one of the username/password pairs defined in the file
‘config/amc-users.xml’.
5.1.3.3 Link to the Accounting and Messenger Databases
The FlexFrame Agents WebInterface enables messages of the Application and Control
Agents to be displayed. These are sent from the Agents to the FlexFrame Messenger,
WebInterface
165 FlexFrame Agents Guide
which filters them, automatically triggers reactions as required, and writes them into a
database.
Additionally the WebGui supports showing performance graphs and accounting data,
which is read from the accounting database.
The database access parameters must be specified in the configuration file
/opt/myAMC/config/connector.properties. If changes are made, the FlexFrame
WebGui must be restarted or reloaded (e.g. using the Tomcat Service Manager) or
Tomcat must be restarted or reloaded.
Configuration of the connector properties see chapter 2.3.5.
5.1.3.4 LDAP Options
The graphical user interface provides structural information of pools, nodes and services
even if they are not running. They are gained by accessing the information stored in the
system-wide LDAP database.
LDAP access is preconfigured for standard installations and works out of the box. In
order to allow fine-grained control, the following options can be specified in the configura-tion file /opt/myAMC/config/FA_WebGui.conf. If changes are made, the WebInter-
face must be restarted or reloaded (e.g. using the Tomcat Service Manager) or Tomcat
must be restarted or reloaded.
The table below includes all LDAP-specific configuration settings, their default values and
a description.
LDAP database Default value and description
ldap.config.file /etc/openldap/ldap.conf
Specifies the name of the system-wide LDAP config file,
which contains parameters for all services requiring LDAP
access, e.g. automount, etc. This file will be read in order to
get the necessary settings.
ldap.username <empty>
User name for LDAP access. This parameter may contain
the BindDN (equivalent of user name).
ldap.password <empty>
Password for LDAP access.
ldap.update-cycletime 3600
Specifies how often (in seconds) structural information is
updated.
WebInterface
FlexFrame Agents Guide 166
LDAP database Default value and description
ldap.enabled true
Specifies whether LDAP access is enabled (true, false)
5.1.3.5 GUI Options
The graphical user interface provides some options which can be set using parameters.
These permit the window size to be changed, automatic login, and the activation of the
context menus.
For the JNLP GUI setting are changed using the paramenters in the
/opt/myAMC/FA_WebGui91/FlexFrameViewer.properties file.
Fot the applet GUI settings are changed using applet parameters in the
/opt/myAMC/FA_WebGui91/index.html file.
For the applet autlogin and activation of the context menus are implemented using applet
parameters, which are set as follows:
///////////////////////////////////////////////////
// user changable paramaters
// set to 'yes' to enable context menues
allowContextMenues:'no',
// change to enable auto-login
//user: 'MYUSER',
//password: 'MYPASSWD',
///////////////////////////////////////////////////
Parameter Possible values and description
allowContextMenues yes or true and no or false (no = default)
Specifies whether context menus (and thus actions) are per-
mitted.
user User name for Autologin; must be uncommented to be used
password Password for Autologin; must be uncommented to be used
5.1.3.6 Other Settings
Further settings can be made in the file /opt/myAMC/config/FA_WebGui.conf. If
changes are made, the WebInterface must be restarted or reloaded (e.g. using the
Tomcat Service Manager) or Tomcat must be restarted or reloaded.
WebInterface
167 FlexFrame Agents Guide
BlackBoard settings Default value and description
The BlackBoard is an interface for transferring commands
to the Application and Control Agents (see chapter 8).
blackboard.toolpath /opt/myAMC/FA_CtrlAgent
Path containing the BlackBoard command line tool.
blackboard.toolfile BBTool.sh
Name of the BlackBoard command line tool. This is nor-
mally a shell script which starts the correct program in
accordance with the operating system .
blackboard.path /opt/myAMC/vFF/
Path for the BlackBoard file (without pool name!)
blackboard.file data/FA/blackboard/blackboard.txt
Name of the BlackBoard file (without pool name!)
blackboard.md5key myAMC.FA_BBTool
MD5 key which is used for saving.
Paths and file names Default value and description
path.flexframe.data /opt/myAMC/vFF
Location of the data directories (pools)
path.flexframe.config /opt/myAMC/config
Configuration directory
file.amc-roles amc-users.xml
Name of the file (in the above-mentioned configuration
directory) which contains the user definitions.
Hypervisor Monitor Default value and description
builder.enabled.vmhost true
Specifies whether Hypervisor monitoring service is ena-
bled (true or commented) or disabled.
builder.update-
cycletime.vmhost
60
Cycle time (in seconds) for Hypervisor monitor service
Reading out the FA data Default value and description
WebInterface
FlexFrame Agents Guide 168
pools.update-cycletime 3
Specifies how often the data of all pools is to be read out
(in seconds).
WebInterface
169 FlexFrame Agents Guide
5.2 Visualization
5.2.1 Starting the Client
The WebInterface is always reachable when the Apache Tomcat is running. The Apache
Tomcat will be started by the script /etc/init.d/ff_jakarta-tomcat start.
The WebInterface can be reached at the following URL:
http://<control_node>:8080/FAwebgui/index_plugin.html
The port specified can be changed in the Tomcat configuration files server.xml. . In
this case the WebGui client will be started as an applet in browser.
To start the client as Web Service client (JNLP client) and not as an applet, the address
above should be called without the suffix index_plugin.html at the end:
http://<control_node>:8080/
5.2.2 This browser will open a new JNLP client wich can be used independently from the browser.Login
The user name and password must be entered in the login mask to permit authentication.
You can use the WebInterface only with a valid combination of user name and password
(for details on configuring users see section 5.1.3.2).
WebInterface
FlexFrame Agents Guide 170
5.2.3 Overview of Elements
The WebInterface provides a clearly structured display of all the elements in a FlexFrame
system. The left-hand side shows pools, groups and nodes in a tree structure. The main
panel provides and overview of the status of all instances and displays messages from
the various agents. The panels at the lower edge show all the nodes and systems and
their current statuses.
WebInterface
171 FlexFrame Agents Guide
5.2.4 Pool / Group Tree
5.2.4.1 Overview
The pool and group tree shows all elements of a FlexFrame system in a hierarchical
structure. Each pool has, child elements, groups, which in turn are used as containers for
systems, nodes or instances.
5.2.4.2 Status
The status of an element is indicated by its color.
The colors have the following meanings:
Red Critical
Yellow Warning
Green Normal, everything OK.
White Not active or no further information
Black Not active or shut down
Blue Managed Spare Node in Adminpool
Color inheritance takes place according to the worst-case principle. Here an element
displays the worst value for its own status or for the status of its child elements.
5.2.4.3 Selecting an Element
When an element is selected, the associated instances, nodes and systems are dis-
played in the panels on the right-hand side:
When a pool is selected, the instance view shows all instances which are running on
nodes of this pool, the node panel shows all nodes belonging to the selected pool,
and the system panel shows all systems which have instances on nodes of this pool.
When a group is selected, the instance view shows all instances which are running on
nodes of this group, the node panel shows all nodes belonging to the selected
group, and the system panel shows all systems which have instances on nodes of
this group.
When an Application Node is selected, the instance view shows all instances which are
running on this node, the node panel shows the selected node and the system panel
shows all systems which have instances on this node.
When a system is selected, the instance view shows all instances belonging to this sys-
tem, the node panel shows the nodes on which the associated instances are run-
ning, and the system panel shows the selected system.
When an instance is selected, the instance view shows the selected instance, the node
panel shows the node on which the instance is running, and the system panel shows
the system to which the instance belongs.
WebInterface
FlexFrame Agents Guide 172
5.2.4.4 Different Tree Presentations
The tree view offers different presentations for displaying the various elements.
The Nodes tree displays the pool, group and node elements.
Example 1 Example 2
The nodes displayed in black are not currently running, the ones in white are
running, but are not currently hosting services, i.e. they are spare nodes.
Virtual nodes will be marked with a “V”.
Spare noes will be marked with a “S”.
WebInterface
173 FlexFrame Agents Guide
The system pool “FlexFrame” shows Hypervisor, virtual machines and their state.
The state is Hypervisor-specific and does not show FlexFrame or SAP service state.
The type of Hypervisor (VMware/ESX or KVM) will be shown als overlay icon.
State: RUNNING as soon as VM is started/powered-on on Hypervisor level.
State is not related to ping-response or FA_AppAgent state
WebInterface
FlexFrame Agents Guide 174
The Systems tree displays the pool, system and service elements.
The All tree displays all elements, from pool to service.
WebInterface
175 FlexFrame Agents Guide
5.2.4.5 Visualization of host systems and virtual servers
As of version 9.0 the WebGUI displays the hypervisor of virtual servers.
For each hypervisor the running virtual machines are shown. Virtual servers are marked
with an overlay icon “V”.
Hypervisor are shown together with the running virtual servers across the pools in the
pesudo pool "FlexFrame".
The running virtual servers of a hypervisor server can be in different FlexFrame groups
as well as in different pools.
In addition, the virtual server can be seen as any physical application nodes in their re-
spective pool.
5.2.4.5.1 Virtual server
A virtual machine is marked with an overlay icon „V“ in the tree and the node panel.
5.2.4.5.2 Hypervisor
A hypervisor is a physical server, running on the virtual node in a virtualized environment.
With FlexFrame 5.3 or FlexFrame Ochestrator V1.0 ESX and KVM hypervisors are sup-
ported.
Each Frame has a pool “FlexFrame”, which is a special system pool for information about
the whole frame. The data of the pool FlexFrame is located on the control node and not
on the storage system. The pool “FlexFrame” contains a group ”Hypervisor Nodes”, which
shows all physical servers (Hypervisor) with the currently existing virtual machines.
The shown state in pool FlexFrame is the state seen from Hypervisor / interface to Hy-
pervisor. A running VM means that from the perspective of Hypervisor the virtual machine
is started. VM state RUNNING is shown even if the operating system and application
software is not completely active.
The state of the VM with operating system and application software as seen from Flex-
Frame Agents is visible in normal pool. So the state of the same VM might be different
between pool FlexFrame and normal pool.
WebInterface
FlexFrame Agents Guide 176
The following illustration shows the pool “FlexFrame” with Hypervisor and virtual servers
and the pool “pool3” with physical, virtual and sparenodes.
WebInterface
177 FlexFrame Agents Guide
5.2.5 Status Display
5.2.5.1 Node Panel
The node panel displays all nodes on which instances belonging to the currently selected
element are running. Each node is shown with its name, icon and status, the node status
corresponding to the worst instance status.
In the node panel virtual servers are marked with an overlay icon “V”.
Nodes whose shutdown facility may not be configured right or is unknown will be shown
with a red triangle in node and tree-panel (from FlexFrame V5.2).
The order of the elements in the panel can be changed by pressing the right button and
choosing the sort order.
Example: Sorting by colours, i.e. critical nodes with problems can be recognized directly,
even if many nodes are visible in the panel.
If you leave the mouse cursor over a node icon, a tool tip is displayed specifying the
name, group and pool.
Example 1 Example 2
WebInterface
FlexFrame Agents Guide 178
5.2.5.2 System Panel
The system panel displays all systems of the instances belonging to the current-
ly selected element. Each system is shown with its name, icon and status, the
system status corresponding to the worst instance status.
The order of the elements in the panel can be changed by pressing the right button and
choosing thesort order.
WebInterface
179 FlexFrame Agents Guide
5.2.5.3 Instance Panel
The instance panel displays all instances belonging to the currently selected element in a
list view containing the following fields:
Severity
Current instance status: red = critical, yellow = warning, green = everything OK,
white = no information or not active.
Pool
Name of the pool incorporating the node on which the instance runs.
Group
Name of the group incorporating the node on which the instance runs.
System
Name of the system to which the instance belongs.
ID Instance number.
Service
Generic instance name.
Type
Type of instance: database (DB), central instance (CI), application instance (APP)
Priority
Priority of the instance (see section 3.3.1)
Node
Node on which the instance is running
State
Current status of the instance
HANA Master/Rep
Role of the SAP HANA master replica intstallation
HANA Node Id
Node-ID of SAP HANA
HANA Replica Id
Replica-ID of SAP HANA
NoWatch
NoWatch attribute (true/false)
WebInterface
FlexFrame Agents Guide 180
The fields in the list can be arranged in any order by left-clicking on the column header.
Clicking on it again reverses the sort order, and a third click cancels the sort again. An
arrow appears next to the header which specifies the current sort order and sort direction.
If sorting is to take place according to several columns, you click on the first column
(twice if required, depending on whether sorting is to take place in ascending or descend-
ing order). You select all other columns with CTRL + mouse click. The lower the sort
priority of a column becomes, the smaller the arrows next to the column header become.
In order, for example, to sort by pool and within a pool by group, you first click on Pool,
hold down the CTRL key, and then click on Group.
WebInterface
181 FlexFrame Agents Guide
5.2.5.4 Assigning States to Colors
As described above, color inheritance takes place according to the worst-case principle.
Here an element displays the worst value for its own status or for the status of its child
elements.
The states are assigned to the color display as follows:
White:
inactive or no
further information
Green:
normal,
everything ok
Yellow:
warning
Red:
critical
Black:
critical
Blue:
Info
Node states
RUNNING
SWITCH_INT
SWITCH_EXT
PowerOff
DOWN
Managed Spare
Node in Ad-
minpool
Service states
SHUTDOWN
DOWN
WATCH
UNKNOWN
NULL
RUNNING
STOPPING
STARTING
REBOOT
RESTART
RESTARTING
REBOOT
REBOOTING
RBGET
SWITCH
SWITCHOVER
SWGET
ERROR
The keywords for the states are accepted in either upper or lower case.
WebInterface
FlexFrame Agents Guide 182
5.2.5.5 Overlay Icons
Additional of the colored visualization there are overlay icons with the following sense:
Nodestate is ‘unknown’
ESX Hypervisor
KVM Hypervisor
Virtual node
Spare node
Virtual spare node
Shutdown facility detect that shutdown for this node should not work (shutdown
is invalid)
NoWatch attribute of a service
SAP HANA is Master
SAP HANA is Replica
SuspendAutonomy is active
These overlay icons will be shown in tree and in node panel.
WebInterface
183 FlexFrame Agents Guide
WebInterface
FlexFrame Agents Guide 184
5.2.6 Message Display
The message display provides an overview of all messages which come from the Appli-
cation and Control Agents of the FlexFrame system. The messages are prefiltered in
accordance with the element currently selected in the tree. If the highest element with the name FlexFrame is selected, all messages are displayed. If a specific node is selected,
only messages of this node are displayed.
Scrolling to reg right shown additional informative columns:
5.2.6.1 Fields
The following fields are available in the message panel:
WebInterface
185 FlexFrame Agents Guide
Severity
Message severity: red = critical, yellow = warning, green = everything OK, white = no
information or not active
Timestamp
Message timestamp
Pool
Pool name
Group
Group name
System
System name
Id
Instance number
Type
Type of service affected (DB, APP, CI, …)
Node
Node name
ShortMessage
Abbreviated message
Category
Category
Info
Service-Type of service affected (SRV_DB, SRV_APP, SRV_CI, …)
MessageId
Unique message ID
Application
Application which sent the message
LongMessage
Detailed message
WebInterface
FlexFrame Agents Guide 186
5.2.6.2 Navigation
The navigation bar enables you to navigate through all messages:
With the message view will be closed.
With and the following drop-down-menue the refresh cycle can be
adjusted.
The fields “From” and “to” specify the currently selected timerange. They can be changed
as well by manually entering a valid date. Start and end time are synchronized, i.e. the
timerange will always be the one specified below.
The buttons and can be used to scroll within the messages.
fast-forwards toward the most recent messages. The button updates the mes-
sages in the current time range. If the update will take to much time, it can be interrupted
with .
A manual refresh with is necessary after the first time the message view has been
opened, and whenever start date, end date, or the time span has been changed.
Pushing a drop-down menue will be shown, where ‚page-size‘ and ‚cache‘ could be
adjusted.
Using cache the message view must not be load all data from database for a refresh. The
amount of data loaded from database during refresh and will be hold in cache, depends
from ‚page size‘ and the desired timerange. If timerange is smaller as page-size. It may
be that more than one page size will be loaded, because the message view will use
page.size with special rules. If the page-size is smaller as the timerange, there will be
loaded as many page-sizes that the whole timerange is filled.
With “Clear cache” the cache will be cleared.
WebInterface
187 FlexFrame Agents Guide
5.2.6.3 Sorting
The message window allows sorting of the currently displayed messages.
The fields in the list can be arranged in any order by left-clicking on the column header.
Clicking on it again reverses the sort order, and a third click cancels sorting. An arrow
next to the header specifies the current sort order and direction.
If sorting is to take place according to several columns, click on the first column (twice if
required, depending on whether sorting is to take place in ascending or descending or-
der). Additional columns can be selected with CTRL + mouse click. The size of the arrow
next to the column header increases and decreases with sort priority.
In order, for example, to sort by group and within a group by nodes, click on “Group”, hold
down the CTRL key, and then click on “Node”.
By default messages are displayed in descending chronological order, i.e. new messages
are shown at the top.
WebInterface
FlexFrame Agents Guide 188
5.2.7 Configuration of FlexFrame Agents with the Webinter-face
The FlexFrame Agents can be configured using the web interface.
The FlexFrame WebGui is a display option for the visualization of Flexframe infrastruc-
ture and the services running on it. The configuration with this frontend is not possible in
the standard fashion. Reason for this is that in normal operation important configuration
settings can not be changed by an operator error. An administrator can enable the con-
figuration dialogs by following change.
In order to use the configuration editor, the context menus must be enabled. This can be
achieved by setting the option allowContextMenues in the file
/opt/myAMC/FA_WebGui91/index.html to yes.
Caution: Turning on the context menu to make from the pure observation GUI (sta-
tus-and event-monitor) a limited ADMIN GUI.
The configuration editor can be started by right-clicking on a pool element and selecting Edit configuration from the context menu.
The normal way to configure the FlexFrame Agents Suite is the FlexFrame ConfigGui.
Calling this configuration Gui should be allowed only to authorized persons.
WebInterface
189 FlexFrame Agents Guide
The following descriptions and screenshots show a matter of principle, but not the whole
configuration menus.
Each parameter offers a detailed description, when the mouse pointer is positioned over
the parameter name:
The values can be modified using the input fields, changes are indicated by a an icon:
All parameters are validated, whenever a change occurs. Validation failures are indicated
by an icon, and offer a detailed error description as a tooltip:
Parameter lists can be extended using the add button, unnecessary entries can be
removed using the remove button. When a new entry is created, an input dialog asks for
a name, which uniquely identifies this entry. The suggested name is composed of a
template name and the current date:
Some additional elements can be used to control the configuration process:
Save: Stores the modified values in the config files.
Revert: Reverts all changes.
WebInterface
FlexFrame Agents Guide 190
Close: Closes the configuration dialog.
5.2.7.1 Configuration of priorities and powervalue
The high availability and autonomy rules also depend on the priorities and the configured
power requirements of a service. This parameter priority or power value to a service are
stored in the pool-specific myAMC_FA_Group.xml file.
From FlexFrame V5.3 or FlexFrame Orchestrator V1.0 there is a graphical dialog with
which these parameters can be edited in FlexFrame ConfigGui.
An understanding of the configuration of a service associated priority value and the power
is very important because the behavior is controlled depending upon failure.
It is called in the main menu in the View menu-> Service Priority View.
The configuration of the rules for priorities and power values is pool-specific. Unless a
pool is selected in the tree, a grid appears empty priorities. The DataGrid for priority con-
figuration has its own menu bar with its own menu button.
The buttons shown in the image above from left to right have the following meaning:
Priorities display and close the dialog
Save button to save the changes.
InsertBefore A new rule is inserted before the selected rule
Insert After: A new rule by selecting the rule is placed.
Edit: A selected rule can be edited
Delete: You can delete a selected rule
WebInterface
191 FlexFrame Agents Guide
Filters for display of associated services
Selection of the service schema
Add a service schema
Delete a service schema
Once you select a pool, the stored rules appear. Rules are shown in a different color than
the services that are derived from this rule, their priority or power value.
The picture shows the rules concerning the currently selected tree pool "default".
WebInterface
FlexFrame Agents Guide 192
All parameters are arranged hierarchically. I.e. the rule tree is processed from top to
bottom. A service derives from its priority rule from the first hit. In the rules table is there-
fore always, and that should not be changed, as the last rule *. * Rule, which applies to
each service and defines the default priority and the default power value for a service. Is
not this a general rule, the internal default values are set for a service matched with any
rule entry.
The default values are 998 for the priority and 100000 for the Power value. If you want a
service of a particular SAP system has to be another priority is always to insert this rule
before the general rule.
By using the filter button, the currently existing services may be displayed. The services
are always displayed immediately after the rule has matched in the determination of ser-
vice priority and power value.
The display of the rules along with the to the Related Services is shown in the next pic-
ture. Together with the button 'follow selction' can be controlled so that only the services
for a group within a pool are displayed for clarity.
Insert a new rule.:
The configuration of a new rule can be made in the filtered or unfiltered display. Before
creating a new rule, the new rule must be considered where in the rules table must be
inserted. Accordingly, there is a menu button to insert a new rule before or after a select-
ed rule or to edit the selected rule.
WebInterface
193 FlexFrame Agents Guide
The definition of such a rule is shown below:
Group: Defines the group of the pool in which the rule should apply. No entry or *
is called the rule is applied Across group within the pool.
SID: System ID to apply for a rule. No entry or * means that the rule applies to
all systems.
Service Type: The service type.
ServiceID Instancenummer a service.
The individual Attritbute generally be linked via "AND" in the evaluation. So a service is
assigned the priorities defined in the rule and power values if its attributes Group, SID,
service type, service ID match on the rule.
Evaluation is carried out hierarchically from top to bottom. The position of a rule in the
rule tree therefore has significant influence on which rule is applied to a service at run
time. For a more specific rule is applied before a general rule, this rule must be in the
table above.
The following figure shows the dialog to add a new rule. This dialog is identical in princi-
ple for 'insertBefore', 'insertAfter' and 'edit'.
So that new rules can be added to the control hierarchy at the right places, there are two
menu buttons 'insert before' and 'after insert'. Therefore it is necessary that a control is
selected.
WebInterface
FlexFrame Agents Guide 194
Furthermore, there is still the option to edit an existing rule. A new or modified rule is
initially only in the memory of FlexFrame ConfigGuis only on the save button, the rule
change is stored in the associated file.
Activation of a rule change:
Note: A modified rule is active after a restart of the FlexFrame agents.
The display of the Services always displays the current values of the services as they
currently are, according to the live list file.
In the rule table, a service usually is indicating he will belong to from rule activation.
The user can therefore see the possible effects of a rule change right on the screen.
Display the rule table with and without services:
This can be set via the filter button. The filter button controls whether only the rules or the
rules derived from the service are shown.
The line filter always applies only to the selected line. The button for the row filter is only
visible when switched filter function.
WebInterface
195 FlexFrame Agents Guide
5.2.7.2 Configuration of ServicePing
A ServicePing determines the availability of a service by querying the service.
This is done by special ServicePing scripts (eg 'ServicePingDb.sh', 'ServicePingDef.sh',
'ServicePingErs.sh', 'ServicePingHDB.sh', ...) See section 3.11.
ServicePing is currently available for the following service types:
SRV_DBORA, SRV_DBSAP, SRV_LC, SRV_APP, SRV_CI, SRV_PAI, SRV_SCS,
SRV_JC, SRV_J, SRV_ASCS, SRV_ERS, SRV_HANA_REP, SRV_HANA.
The ServicePing can be turned on for specific services.
This can be done via the pool configuration “Autonomy” “Servce Ping”.
Description:
Name
Name of this entry. Assignd when adding a new entry.
Active:
Specifies whether this entry has been active or not.
ServiceType
Service-Type for which the service ping is enabled (wildcards allowed).
SID
SID for which the service ping is enabled (wildcards allowed).
instance number
Instance number for which the service ping is enabled (wildcards allowed).
WebInterface
FlexFrame Agents Guide 196
5.2.7.3 Configuration of highavailability
The high availability and autonomy rules depend on the parameters for the control of high
availability. These parameters are configured at the pool level.
As of version V5.3 FlexFrame, in addition to the default set of parameters that applies to
the entire pool, a separate set of rules for a group within a pool can be configured.
The group-specific ruleset contains only the important SwitchOver- and TakeOver- pa-
rameter, which controls the highavailability.
The image shows the dialog for configuring the parameter settings for the Takeover
Rules.
In addition to the default configuration that applies to all groups of a pool, a rule set that
deviates can be configured for a group within a pool.
WebInterface
197 FlexFrame Agents Guide
The configuration dialog is shown in the following image.
To add a neu group specific configuration please press „Add“.
Then the following dialog will appear:
Here you must enter the name of the group for which the new rule should apply:
After pressing "OK" the new group appears in the list and can be customized:
WebInterface
FlexFrame Agents Guide 198
WebInterface
199 FlexFrame Agents Guide
5.2.8 Special Signals
5.2.8.1 Overview
The pool and group tree shows all elements of a FlexFrame system in a hierarchical
structure. Each pool has, as child elements, groups, which in turn are used as containers
for systems, nodes or instances. In addition to the standard visualization of the elements
their some signaling feature for special functions or situtations.
5.2.8.2 Spare Nodes in a Spare Pool
The visualization of spare node in an admin pool uses additional signals to differentiate
between active and inacitive spare nodes. The status of an element is indicated by its
color.
A spare node in the SparePool/SpareGroup is white or black because the node is un-
known and down. To use a spare node from the spare pool in an other pool the spare
node has to be activated. In addition to the standard signaling an activated spare node in
a spare pool has an overlay icon. Spare nodes in the admin pool without overlay icon are
not activated.
WebInterface
FlexFrame Agents Guide 200
5.3 Interaction
5.3.1 Updates
The WebInterface regularly updates all the displayed elements, states and messages.
5.3.1.1 Update Interval
The interval between two updates can be set using the menu ”Refresh”:
An update interval of between three seconds and two hours can be selected.
5.3.1.2 Manual Update
You can initiate a manual update outside the update cycle using the button.
WebInterface
201 FlexFrame Agents Guide
, ,
WebInterface
FlexFrame Agents Guide 202
5.4 Info and Help
The button enables you to access the information dialogs.
Message History
The Message History displays the last messages of the WebInterface.
WebInterface
203 FlexFrame Agents Guide
5.5 FlexFrame Performance and Accounting Plug-in
The FlexFrame Agents Suite contains, as optional components, plug-ins for the views of
the performance and accounting data.
The plug-in consists of a graphical display in which the performance or accounting values
are shown in the form of graphs. The temporal view domain can be freely chosen and is
only limited through the data timespans stored in the Repository.
The GUI uses a powerful cache algorithm to optimize the display, which minimizes the
waste of resources and optimizes the response time for the data request. The cache
works through page size and number of pages, meaning that, according to the efficiency
of the front-end system used, the default parameters can be temporarily altered to adjust
the signal performance to individual demands.
Pagesize
Pagecount
These two parameters influence the size of the display cache and the quantity of data to
be transferred at the request of the GUI to its server.
The cache algorithm erases old pages only after obtaining the page count. Then the
pages not used for the longest time are erased.
The performance plug-in shows the CPU and memory values for a chosen service, in
each case in individual graphs. Every graph shows minimum, average and maximum
values.
The displayed timespan can be shifted using the two arrows at the top of the plug-in.
5.6 FlexFrame Reporting Plug-in
The FlexFrame Reporting plug-in contains the production, export and print of reports from
the FlexFrame Autonomous Management Center. This plug-in is only available in the
standalone variant of the management center; display within a web browser is not possi-
ble.
The plug-in shows a selection of available reports. New predefined or user-specified
reports can simply be filed in the report directory and are immediately applicable.
All reports have a standardized parameter interface. A new report can easily be derived
from one of the existing reports.
WebInterface
FlexFrame Agents Guide 204
5.7 Customization of Icons in the FlexFrame Agents WebGUI
The FA WebGUI uses icons to represent company and product information as well as
nodes and systems. As of version 8.0 of the FA WebGUI, these icons can be changed by
the user.
User-defined icons can be deposited in directory /opt/myAMC/FA_WebGui/icons. The
file /opt/myAMC/FA_WebGui/FlexFrameViewer.properties contains names of
heading, logo, node, and system icons as well as the window title and can be customized
as necessary.
The information stored in this directory (logos and icons) are loaded each time the FA
WebGUI, and then stored locally for the duration of the use of the FA WebGUI. I.e. the
start of the FA WebGUI will ever slow down on the number of individualized icons and
logos, due to the additional load times slightly.
The behavior and display of the FA WebGUI remain unchanged if the directory for the
logos and icons contains no additional logos and icons. The directory and the standard
logos and icons are already components of the distributed FA WebGUI versions. The
exact procedure is described in section 7.5.14.
205 FlexFrame Agents Guide
6 FlexFrame Agents Power Shutdown Concept
6.1 General
The FlexFrame Agents power shutdown function is designed to provide a simple, easy-
to-configure method for implementing secure shutdown of various hardware platforms.
For this pupose the FlexFrame Control Agents use the generic interface with a set of
predefined shutdown methods.
A new parameter defines the check sequence to find the appropriate shutdown method
for a server.
For external SwitchOver of application instances, FlexFrame Agents requires a secure
option for shutting-down nodes simply and securely.
The FlexFrame Agents ascertain this lack of configuration information at runtime and
then generate the configuration information required for the agents.
FA_Blade
FA_IPMI
FA_VM
FA_Generic
Reserved
FlexFrame Agents Power Shutdown Concept
FlexFrame Agents Guide 206
6.2 Power Shutdown Architecture
The figure below provides an overview of the components involved and how these inter-
work in a FlexFrame environment.
The FlexFrame Control Agent runs on the active Control Node. The FlexFrame Applica-
tion Agents provide the Control Agent with information on the computer type and further
information which is required for generic detection of shutdown information, insofar as
this is technically possible and the information is unambiguous and can be ascertained
securely. Information which cannot be ascertained generically must be entered manually
in the Powershutdown section of the myAMC_FA.xml file.
FlexFrame Agents Power Shutdown Concept
207 FlexFrame Agents Guide
6.3 Basics
6.3.1 Powershutdown methods for use with FlexFrame ge-
neric shutdown facility
As of FlexFrame version 4.2B, the generic FlexFrame Powershutdown facility uses
wrapper scripts to integrate server specific shutdown tools.
The standard path for the FlexFrame Agents shutdown wrapper scripts is:
/opt/myAMC/scripts/PowerMng/<type>_wrapper.sh
6.3.1.1 Power Shutdown for Blade Systems
Power shutdown for the Blade systems is implemented via the wrapper script blade_wrapper.sh. In order to shutdown a compute blade, the IP addresses, SNMP
communities and password (usually the same as the SNMP community) of the Manage-
ment Blade must be configured.
The hosting blade chassis, MAC addresses and slot numbers of compute blades are
determined automatically.
The wrapper script is: blade_wrapper.sh
6.3.1.2 Power Shutdown for PRIMERGY Systems
By default power shutdown for the PRIMERGY systems in FlexFrame is performed via
the IPMI interface using the Linux IPMI tool. Necessary configuration settings are user
name and password, if they are different from the default user name and password.
The wrapper scipt is: ipmi_wrapper.sh
FlexFrame Agents Power Shutdown Concept
FlexFrame Agents Guide 208
6.3.1.3 Power Shutdown for virtual VM Server
Power shutdown for virtual servers is performed via the VM shutdown script.
The wrapper scripts are: vm_vm_wrapper.sh and vm_hv_wrapper.sh
6.3.1.4 Power Shutdown for other Server (e.g. PrimeQuest PQ2800
Power shutdown for other servers like PrimeQuest PQ2800 is performed via the generic
shutdown script.
The wrapper script is: generic_wrapper.sh
6.3.1.5 Power Shutdown for Host systems of Virtual Servers
The usage of virtual servers on a host system (ESX server) requires the configuration of
the shutdown interface of the host system. The host system can be a blade system or a
rack server, therefore there are different shutdown parameter.
The shutdown parameter of the host systems have to be configured manually, since the
FlexFrame AppAgents cannot provide information about these hostsytems.
The following values must be known for the hostsystem (ESX server):
Hostname: hostname of the host system
If the hostsystem is a bladesystem, additional information is necessary:
MAC address: mac address of the host system
The requirement is that the authentication defaults and the authentication settings (SNMP
community, user name, password) and the management blade are configured correctly.
FlexFrame Agents Power Shutdown Concept
209 FlexFrame Agents Guide
6.4 Power Shutdown Configuration
The following sections describe the configuration of the power shutdown. This can also
be done in the FlexFrame Webinterface.
See Pool-Konfiguration „Shutdown Settings“
For general information about usage and configuring with the FlexFrame Webinterface
please see section 5.
FlexFrame Agents Power Shutdown Concept
FlexFrame Agents Guide 210
6.4.1 Switchover Control Parameters
The normal lapse on breakdown of a node requires the disconnecting of this node to
avoid a disturbance caused by the existence of duplicate services. If it is not possible to
disconnect the node, the move breaks up with a failure notice. The services concerned
are not restarted.
With the help of the parameter IgnoreShutdownFailure, this failure can be influenced
in that, if a shutdown fault occurs, for example because a shutdown parameter was not
configured for a node, the relevant node is separated from the network.
If a node is separated from the network, the network must be reactivated manually. For
this, the following command must be executed on the Control Node before the defective
node is restarted:
ff_sw_ports.pl --op up --name <hostname>
If the network interfaces of the node are not reactivated, it’s not possible to start the node.
See Pool-Konfiguration „Shutdown Settings“ „Shutdown Parameters “.
FlexFrame Agents Power Shutdown Concept
211 FlexFrame Agents Guide
Description:
• Default ShutdownMode
Default mode for Shutdown (leave-off / cycle).
• Cycletime
Cycletime (ms) of Shutdown-Facility.
• Execution timeout
Timeout (ms) for the Shutdown-Agents.
• Poweroff try count
Maximum number of PowerOff attempts.
• Renew config cycletime when invalid
Cycletime (ms) to renew invalid Shutdown-configuration.
• Renew config cycletime when valid
Cycletime (ms) to renew valid Shutdown-configuration.
• Resend trap if shutdown valitity is valid
Resend time (s) of the traps for valid Shutdown-configuration.
• Resend trap if shutdown valitity is invalid
Resend time (s) of the traps for invalid Shutdown-configuration.
• SNMP timeout
Timeout (ms) for SNMP-Requests.
• SNMP try count
Number of attempts for SNNMP requests.
• Ignore Shutdown failure
Specifies if a shutdown error should be igoriert.
• PrePoffHookPath
Path to PrePoffHook-Script.
• PostPoffHookPath
Path to PostPoffHook-Script.
• Ignore PrePoffHook failure
Specifies if a PrePoffHook error should be igoriert.
• VMHostPoffCheckPath
Path to VMHostPoffCheck-Script.
• VMHostPoffCheck timeout
Timeout (s) of VMHostPoffCheck-Script.
FlexFrame Agents Power Shutdown Concept
FlexFrame Agents Guide 212
6.4.2 User, Password and Community
To use agent power shutdown, user, password and community must be defined in the
configuration of the FlexFrame Agent. This configuration is specified in the pool-specific
configuration file.
The entries for user and password must be the same as those configured for IPMI (rack
server).
The entries for the community must be the same as those configured in the Management
Blade (blade server).
Further information on the configuration of user, password and community in the Applica-
tion Nodes can be found in the ‘Hardware Characteristics Quickguides for the rack serv-
ers resp. management blades’.
The configuration of default user, password and community can be found in the file my-
AMC_FA_SD_Sec.xml at config section Security_default.
In addition, individual Application Nodes can be defined with other values for user, pass-
word and community. To permit this, a corresponding configuration section must be de-
fined for the Application Node.
See Pool-Konfiguration „Shutdown Settings“ „Authentification Defaults“.
Description:
• SNMP Community
Default SNMP community for SNMP requests.
• User name
Default user name used for Shutdown.
FlexFrame Agents Power Shutdown Concept
213 FlexFrame Agents Guide
• Password
Default password used for Shutdown.
See Pool-Konfiguration „Shutdown Settings“ „Authentification Settings“.
Description:
With "Add", a new list item is added.
With "Remove", the selected list entry will be removed.
• Name
Name of ths list entry.
• Hostname
Hostname of the host for this ‘Authentication Settings’.
• SNMP Community
SNMP community for SNMP requests.
• User name
User name used for Shutdown.
• Password
Password used for Shutdown.
FlexFrame Agents Power Shutdown Concept
FlexFrame Agents Guide 214
Example file myAMC_FA_SD_Sec.xml:
<!-- DEFAULT entry -->
<configsection name="Security_default">
<configentry name="Default_User">
<value type="String">OEM</value>
</configentry>
<configentry name="Default_Passwort">
<value type="String">password/value>
</configentry>
<configentry name="Default_Community">
<value type="String">public</value>
</configentry>
</configsection>
<!-- first entry -->
<configsection name="Sec_pq28">
<configentry name="Sec_Hostname">
<value type="String">pq28</value>
</configentry>
<configentry name="Sec_User">
<value type="String">Administrator</value>
</configentry>
<configentry name="Sec_Passwort">
<value type="String">passwort</value>
</configentry>
<configentry name="Sec_Community">
<value type="String">public</value>
</configentry>
</configsection>
<!-- second entry -->
<configsection name="Sec_rx600s6">
<configentry name="Sec_Hostname">
<value type="String">rx600s6</value>
</configentry>
<configentry name="Sec_User">
<value type="String">OEM</value>
</configentry>
<configentry name="Sec_Passwort">
<value type="String">password</value>
</configentry>
<configentry name="Sec_Community">
<value type="String">public</value>
</configentry>
</configsection>
FlexFrame Agents Power Shutdown Concept
215 FlexFrame Agents Guide
6.4.3 Management Blades
As the Management Blades cannot be detected by the FlexFrame CtrlAgent automatical-
ly, they must be configured. This is done in the Managementblades configuration sec-
tion of the myAMC_FA.xml configuration file. Further the SNMP community should be
configured.
The example below illustrates the configuration of the Management Blade BX900-
control. The name of the configuration section (Mgmt_Blade_1) is arbitrary. A condi-
tion is that this configuration section must be contained within the Managementblades
configuration section. Further Management Blades can be entered using the relevant
configuration sections (see the otherMgmtBlade example).
See Pool-Konfiguration „Shutdown Settings“ „Management blades“.
Description:
With "Add", a new list item is added.
With "Remove", the selected list entry will be removed.
• Name
Name of ths list entry.
• Hostname
Hostname (-co interface) of the management blade of the blade-chassis.
FlexFrame Agents Power Shutdown Concept
FlexFrame Agents Guide 216
Example from File myAMC_FA.xml:
<configsection name="Managementblades">
<!-- Here the 'Hostname's of the management-blades must be configured. -->
<!-- first entry -->
<configsection name="MgmtBl_bx900">
<configentry name="Hostname">
<value type="String">bx900-co</value>
</configentry>
</configsection>
<!-- second entry -->
<configsection name="MgmtBl_bx600">
<configentry name="Hostname">
<value type="String">bx600-co</value>
</configentry>
</configsection>
</configsection>
FlexFrame Agents Power Shutdown Concept
217 FlexFrame Agents Guide
6.4.4 Application Nodes
Normally the shutdown procedures for all Application Nodes are determined automatical-
ly. However, it may be necessary to configure individual systems for power management
yourself. To permit this, the values required must be entered in the pool-specific configu-
ration file.
See Pool-Konfiguration „Shutdown Settings“ „Individual Nodes“.
Description:
With "Add", a new list item is added.
With "Remove", the selected list entry will be removed.
• Name
Name of ths list entry.
• Hostname
Hostname of the host with indifidual settings.
• IP Address
IP address of the host.
• MAC Address
MAC address of the host.
No ‘:’, to mac’s separated by blank (aabbcc112233 aabbcc112233)
• Machine
Machine (empty).
• Port
Port (empty).
• ShutdownMode
Mode for Shutdown (leave-off / cycle).
• Shutdown typ
Type of Shutdown (FA_BLADE, FA_IPMI, FA_GENERIC, …).
FlexFrame Agents Power Shutdown Concept
FlexFrame Agents Guide 218
• Hardware type
Hardware type (LINUX, UNKNOWN).
• Console
Console (empty).
Which values must be specified for which power shutdown type is shown in the table
below.
In this table there are some shutdown types resp. server types which are not supported in
the actual FlexFrame Release. For compatibility, they are included in FlexFrame Agents
and will be documented here.
Yes Must be specified.
No Can be left empty. configentry is mandatory, however!
Default (Def) Is read from the Security_default or
Default_ShutdownMode configuration section if it is empty. If a value
is entered, it overrides the default value.
Table see next page:
(for the up-to-date types of server models see FlexFrame Support-Matrix)
FlexFrame Agents Power Shutdown Concept
219 FlexFrame Agents Guide
Linux Linux Linux Linux ESX ESX
Value BLADE IPMI VM BLADE IPMI
Server
type
BX600
BX900
RX300Sn
RX600Sn
PQ2800B ESXVM
BX600
BX900
RX300Sn
RX600Sn
Host
name Yes Yes Yes Yes Yes Yes
Shutdown
type FA_BLADE FA_IPMI FA_GENERIC FA_VM FA_BLADE FA_IPMI
MAC
address Yes No No No Yes No
Manage-
ment
Blade IP
No No No No No No
Hardware No No No No No No
Shutdown
mode Yes/Def. Yes/Def. Yes/Def.
Yes/Def
. Yes/Def. Yes/Def.
IP ad-
dress
(Control
LAN)
No No No No No No
Console No No No No No No
Machine No No No No No No
Port No No No No No No
SNMP
communi-
ty
Yes/Def. No No Yes/Def
. Yes/Def. No
User No Yes/Def. Yes/Def. Yes/Def
. Yes/Def. Yes/Def.
Password No Yes/Def. Yes/Def. Yes/Def
. Yes/Def. Yes/Def.
FlexFrame Agents Power Shutdown Concept
FlexFrame Agents Guide 220
6.4.4.1 Configuration of Hypervisor
The shutdown methods are usually found out automatically for all application nodes. On
the Application Nodes there is running an FlexFrame AppAgent, which determines some
values required for shutdown facility (e.g. MAC-Addr of a blade) and forward them to the
FlexFrame CtrlAgent.
Hint:
MAC address (2 macs separated by blank) Format: aabbcc112233 aabbcc112233
No AppAgent runs on an Hypervisor and the automatic determination of some values
can’t be done.
For this reason the Hypervisor must be configured manually.
Prerequisite for the configuration is, that the "Authentication Defauls" and "Authentication
setting" (SNMP-Community, user, PW) as well as the "management blades" are config-
ured correctly.
The configuration of the Hypervisor will be done in the pool configuration.
See Pool-Konfiguration „Shutdown Settings“ „Individual Nodes“.
FlexFrame Agents Power Shutdown Concept
221 FlexFrame Agents Guide
6.4.4.2 Example for configuration of a node from file myAMC_FA.xml
<!-- 2. entry -->
<configsection name="ddc1esx1">
<configentry name="Hostname">
<value type="String">ddc1esx1</value>
</configentry>
<configentry name="ShutdownTyp">
<value type="String">FA_BLADE</value>
</configentry>
<configentry name="MacAddress">
<value type="String">0016363fcc08 0016363fcc0a</value>
</configentry>
<configentry name="Hardware">
<value type="String">LINUX</value>
</configentry>
<configentry name="ShutdownMode">
<value type="String">leave-off</value>
</configentry>
<configentry name="IP_Address">
<value type="String"></value>
</configentry>
<configentry name="Slot">
<value type="String"></value>
</configentry>
<configentry name="MgmtBladeIp">
<value type="String"></value>
</configentry>
<configentry name="Console">
<value type="String"></value>
</configentry>
<configentry name="Machine">
<value type="String"></value>
</configentry>
<configentry name="Port">
<value type="UnsignedInteger"></value>
</configentry>
</configsection>
FlexFrame Agents Power Shutdown Concept
FlexFrame Agents Guide 222
6.4.5 Default Shutdown Mode
A default shutdown mode is defined for the power shutdown. This can be changed by the
configuration entry ShutdownMode.
The possible values are cycle or leave-off.
See Pool-Konfiguration „Shutdown Settings“ „Shutdown Parameters“.--> “De-
fault Shutdown mode”.
You are advised not to change the default shutdown mode so as to guarantee that Appli-
cation Nodes are shut down securely.
<configentry name="Default_ShutdownMode">
<value type="String">leave-off</value>
</configentry>
6.4.6 Management blades with SNMP-community different from default
For management bades with a SNMP community different from default the authentica-
tionn settings ‚comunity‘, ‚user‘, ‚pasword‘ must be explicitly configured at „Authentica-
tion Settings“.
This is independent from the discosure of the management blade in menu „Management
Blades“.
FlexFrame Agents Power Shutdown Concept
223 FlexFrame Agents Guide
6.5 Possible Errormessages with old Configura-tion
The shutdown methods which are not supported in FF5.1A are removed from configura-
tion.
When running with old config you can ignore hourly messages in
/opt/local/myAMC/vFF/vFF_<pool>/log/CtrlAgt/log_user_cn1.log like:
2012-02-27 07:48:16; 1330325296; KEY_CRITICAL ; cn1-a; LogEx-
tern(); LogEx-Severity = '13' ==> log_key = '1', Fkt = 'CSHUT-
Down::WriteIpmiCfg', text = 'file /etc/opt/SMAW/SMAWsf/SA_ipmi.cfg
can not been opened for writing'.
These messages may be ignored and have no effect to the functionality.
If the customer wants to remove the old shutdown methods from existing pools there may
be changes in the pool specific configuration as follows:
To obtain the changes for existing pools in GUI, the file 'config-specs.xml' must
be copied from template-config to pool-config.
See in template:
/opt/myAMC/vFF/Common/.vFF_template.V90A00K21/config/config-specs.xml
The whole configsection ‘ <configsection name="Shutdown_Facility">’ in file
myAMC_FA.xml may be changed to that from template.
See in template:
/opt/myAMC/vFF/Common/.vFF_template.V90A00K21/config/myAMC_FA.xml
part ‘ <configsection name="Shutdown_Facility">’.
The Versionstring in path of template („V90A00K21“) is only an example. Just use the
newest version.
225 FlexFrame Agents Guide
7 Parameter Reference
7.1 FlexFrame Agents
Operation of the FlexFrame Agents does not necessarily require individual parameteriza-
tion. After installation executable parameter files are available. For productive use, the
values must have been tested and, if necessary, adjusted to the requirements and the
start, stop, ping and restart times called for in the customer system for the services moni-
tored in myAMC.FA.
The monitoring of additional services which are not included in the standard rules is per-
formed in the rule file myAMC.FA_Rules.xml.
The default parameters of the FlexFrame Agents are set such that, when the perfor-
mance and accounting option is activated, no changes to the parameter settings are
necessary.
7.1.1 FlexFrame Agent Configuration Files
The configuration of FlexFrame Agents is stored in several configuration files that can be
found in /opt/myAMC/vFF/vFF_<poolname>/config. The files are in XML format.
TrapTargets.xml
Trap targets (pool-specific). The recipients of SNMP trap messages can be defined.
myAMC_FA.xml
FlexFrame Agents(pool-specific). Settings for autonomous reactions.
myAMC_FA_Checks.xml
Configuration of SpaceUtil-functionality..
myAMC_FA_Groups.xml
Groups (pool-specific). The group memberships are configured.
It is used as LDAP cache.
myAMC_FA_Hosts.xml
Hosts (pool-specific). The host configuration is configured.
Generated from script ‚fa_createHostFileFromLDAP.sh’.
It is used as LDAP cache.
myAMC_FA_Rules.xml
Generic service configuration
myAMC_FA_SD_Sec.xml
FlexFrame Agents Shutdown Security (pool-specific). Settings for the Power-
Shutdown.
Parameter Reference
FlexFrame Agents Guide 226
myAMC_FA_SelectionRules.xml
Definition of the „SelectionRules“ for „Pool independent spare node“.
For most configuration files, there is an additional default file named <filename>-default.xml which contains the as-supplied status is available after
installation. The myAMC_FA_default.xml file is an exact copy of myAMC_FA.xml in its
original state.
7.2 SNMP Traps
7.2.1 General
The TrapTargets.xml file contains all the information which is needed to send SNMP
traps. Two parameters are required for each target:
Host name or IP address
SNMP community
The community corresponds roughly to a password. Mostly the default value public
suffices. For FlexFrame at least the two Control Nodes have to be set as trap targets.
7.2.2 Structure
The file is in XML format and has the following structure:
<?xml version="1.0" encoding="ISO-8859-1"?>
<configuration>
<configsection name="SnmpConnector">
<configsection name="TrapSender">
<configsection name="TrapTargets">
<!-- List of trap targets. For each host/community combination
to send traps to a config section containing the two config
entries Host and Community has to exist -->
<!-- first trap target -->
<configsection name="control1">
<!—- host name of ip address or name of host to send SNMP trap to -->
<configentry name="Host">
<value type="String">control1</value>
</configentry>
<!-- community string to use for a SNMP trap -->
<configentry name="Community">
<value type="String">public</value>
</configentry>
</configsection>
Parameter Reference
227 FlexFrame Agents Guide
</configsection>
</configsection>
</configsection>
</configuration>
The name or IP address of the node which is to receive the traps is specified as the value of the parameter Host (<configentry name="Host">), the SNMP community as the
value of the parameter Community (<configentry name="Community">).
A trap can be sent to any number of targets. If further targets are to be defined, the en-
closing ConfigSection must be copied, renamed and adapted accordingly.
Example:
[header, see above]
<!-- first trap target -->
<configsection name="control1">
<!-- host name of ip address or name of host to send SNMP trap to -->
<configentry name="Host">
<value type="String">control1</value>
</configentry>
<!-- community string to use for a SNMP trap -->
<configentry name="Community">
<value type="String">public</value>
</configentry>
</configsection>
<!-- second trap target -->
<configsection name="control2">
<!-- host name of ip address or name of host to send SNMP trap to -->
<configentry name="Host">
<value type="String">control2</value>
</configentry>
<!-- community string to use for a SNMP trap -->
<configentry name="Community">
<value type="String">public</value>
</configentry>
</configsection>
<!-- next trap target goes here -->
[footer, see above]
Parameter Reference
FlexFrame Agents Guide 228
7.2.3 Default Parameter File
<?xml version="1.0" encoding="ISO-8859-1"?>
<configuration>
<configsection name="SnmpConnector">
<configsection name="TrapSender">
<configsection name="TrapTargets">
<!-- List of trap targets. For each host/community combination
to send traps to a config section containing the two config
entries Host and Community has to exist -->
<!-- first trap target -->
<configsection name="control1">
<!-- host name of ip address or name of host to send SNMP trap to -->
<configentry name="Host">
<value type="String">control1</value>
</configentry>
<!-- community string to use for a SNMP trap -->
<configentry name="Community">
<value type="String">public</value>
</configentry>
</configsection>
<!-- second trap target -->
<configsection name="control2">
<!-- host name of ip address or name of host to send SNMP trap to -->
<configentry name="Host">
<value type="String">control2</value>
</configentry>
<!-- community string to use for a SNMP trap -->
<configentry name="Community">
<value type="String">public</value>
</configentry>
</configsection>
<!-- next trap target goes here -->
</configsection>
</configsection>
</configsection>
</configuration>
Parameter Reference
229 FlexFrame Agents Guide
7.3 Pooling and Grouping
7.3.1 Pooling
Pool creation means the flexible assignment of Application Nodes to a pool. The Flex-
Frame Agents take over the pool information from the FlexFrame configuration in the
LDAP.
When an Application Node is restarted, a start script for the FlexFrame Agents is called.
This start script determines the pool to which this Application Node belongs and starts the
FlexFrame AppAgent accordingly.
7.3.2 Grouping
FlexFrame offers grouping functions for flexible server farming. Grouping enables nodes
and services within a pool to be assigned to different groups. A group is therefore always
a part of a virtual FlexFrame pool.
FlexFrame Agents grouping is a function which is configured via LDAP. Individual param-
eterization is not required for the FlexFrame Agents.
The Groups configuration will be read from LDAP during start of FlexFrame Agents and
be stored in the myAMC_FA_Groups.xml file. It is used as LDAP cache.
An absolute group configuration can be implemented in this file, as in the LDAP configu-
ration. However, generic parameterization is also possible as an alternative. This can
also be used if absolute grouping is not to be used via the LDAP.
7.3.2.1 LDAP Grouping
LDAP grouping is performed in conjunction with the general configuration of the nodes in
the LDAP directory. Here the FlexFrame Agents take over the group information from the
LDAP directory at startup, as they do the pool information. Individual group configuration
in the myAMC_FA_Groups.xml file is thus no longer required.
7.3.2.1.1 Manual Group Assignment
The group assignment is entered manually in the configuration file.
7.3.2.1.2 Example
<gr:group schema="default" name="GR_All">
<gr:description></gr:description>
<match category="${node-hostname}">
<value>vade*</value>
<value>yod*</value>
<value>blade_a*</value>
<value>blade_b*</value>
Parameter Reference
FlexFrame Agents Guide 230
<value>server*</value>
</match>
</gr:group>
<gr:group schema="default_OS" name="GR_Solaris">
<gr:description></gr:description>
<match category="${node-hostname}">
<value>vade*</value>
<value>blade_sol*</value>
<value>server*</value>
</match>
</gr:group>
7.3.2.2 Generic Grouping
Generic group creation is implemented on the basis of generic information which the
Application Agents can obtain automatically. For generic group creation it makes sense to
use the host names, the IP addresses or the operating system employed.
The group name is also created generically. For this purpose each schema is assigned a
group naming rule which combines a fixed component with a variable component.
The following base values are available for creating the generic rules:
Operating system (os)
Network (network)
Number of CPUs (cpu)
The following generic group creation rules result:
Group definition by operating system, network and number of CPUs schema="os_network_cpu"
name="AutoGroup_${OS-typ}_${CPU-cnt}cpu_${ip-adr:netmask=/24}"
Group definition by operating system and number of CPUs schema="os_cpu"
name="AutoGroup_${OS-typ}_${CPU-cnt}cpu"
Group definition by operating system and network schema="os_network"
name="AutoGroup_${OS-typ}_${ip-adr:netmask=/24}"
Group definition by network and number of CPUs schema="network_cpu"
name="AutoGroup_${CPU-cnt}cpu_${ip-adr:netmask=/24}"
Group definition by operating system schema="os"
name="AutoGroup_${OS-typ}"
Parameter Reference
231 FlexFrame Agents Guide
Group definition by network schema="network"
name="AutoGroup_${ip-adr:netmask=/24}"
Group definition by number of CPUs schema="cpu"
name="AutoGroup_${CPU-cnt}cpu"
The parameters are combined via a group schema. The parameters for multiple group
schemas can thus be stored in one myAMC_FA_Groups.xml file. A schema is activated
in myAMC_FA.xml.
7.3.2.2.1 Logical operation
The following logical operations are possible:
OR (through multiple value lines within a match condition)
<match category="${node-hostname}">
<value>vade*</value>
<value>yod*</value>
<value>blade_a*</value>
<value>blade_b*</value>
<value>server*</value>
</match>
AND (through multiple match conditions within a group definition)
<match category="${node-powervalue}">
<value>1000</value>
</match>
<match category="${node-hostname}">
<value>blade_a*</value>
<value>blade_b*</value>
<value>server*</value>
</match>
7.3.2.2.2 Wildcards
The following wildcards can be used:
* “*“ is equivalent to any number of arbitrary characters.
? “?“is equivalent to one arbitrary character.
7.3.2.2.3 Min / Max values
For numeric variables, the following syntax can be used to inquire value ranges:
syntax="min" (in this case a variable is used as a minimum)
Parameter Reference
FlexFrame Agents Guide 232
<match category="${CPU-cnt}" syntax="min">
<value>4</value>
</match>
syntax="max" (in this case a variable is used as a maximum)
<match category="${ip-adr}" syntax="max">
<value>192.168.10.21</value>
</match>
7.3.2.2.4 Masking with IP addresses
With IP addresses, a type of masking similar to the netmask can be used in order to se-
lect network segments:
${ip-adr:netmask=/24} (in this case only the first 24 bits of the IP address are tak-
en into account)
<match category="${ip-adr:netmask=/24}">
<value>192.168.10.0</value>
</match>
7.3.2.2.5 Example
<!-- auto groups -->
<!—
DO NOT EDIT !!!
If you need your own group definition, please use the section
before !!!
-->
<gr:group schema="os_network_cpu" name="AutoGroup_${OS-typ}_
${CPU-cnt}cpu_${ip-adr:netmask=/24}">
<gr:description>Group defined by number of CPUs, network and
operating system</gr:description>
</gr:group>
<gr:group schema="os_cpu" name="AutoGroup_${OS-typ}_${CPU-cnt}cpu">
<gr:description>Group defined by number of CPUs and operating
system</gr:description>
</gr:group>
<gr:group schema="os_network" name="AutoGroup_${OS-typ}_
${ip-adr:netmask=/24}">
<gr:description>Group defined by number of CPUs and
network</gr:description>
</gr:group>
<gr:group schema="network_cpu" name="AutoGroup_${CPU-cnt}cpu_
${ip-adr:netmask=/24}">
<gr:description>Group defined by number of CPUs</gr:description>
Parameter Reference
233 FlexFrame Agents Guide
</gr:group>
<gr:group schema="os" name="AutoGroup_${OS-typ}">
<gr:description>Group defined by operating system</gr:description>
</gr:group>
<gr:group schema="network" name="AutoGroup_${ip-adr:netmask=/24}">
<gr:description>Group defined by network</gr:description>
</gr:group>
<gr:group schema="cpu" name="AutoGroup_${CPU-cnt}cpu">
<gr:description>Group defined by number of CPUs</gr:description>
</gr:group>
Parameter Reference
FlexFrame Agents Guide 234
7.3.3 Default Parameter File
Hint:
As default there is a "service-powervalue" of 100000 for some services (DBORA,
DBSAP, DB2, CI, JC, J, LC, APP, and unknown Service-Types) for safety.
A DB (and services-types above) should not be added with add-rule to a node with other
services, cause it may have too much load.
This may be changed for customer needs.
<?xml version="1.0" encoding="ISO-8859-1"?>
<!-- @@@MYAMC.VERSION@@@ 9.0-23 -->
<defs:definitions
xmlns:defs="myAMC/Definitions/1.0"
xmlns:gr="myAMC/Groups/1.0"
xmlns:attr="myAMC/Attribute/1.0">
<gr:services>
<!--
"service schema" is specified in myAMC_FA.xml
allowed categories:
"system-id" ("P46", "O20", ...)
"service-type" ("SRV_DBORA", "SRV_SAPSYBASE",
"SRV_DBSAP", "SRV_DB2", "SRV_APP", "SRV_CI",
"SRV_SCS", "SRV_ASCS", "SRV_JC",
"SRV_J", "SRV_LC",
"SRV_ERS", SRV_MDS, SRV_MDIS, SRV_MDSS,
SRV_MDS71, SRV_MDIS71, SRV_MDSS71,
"SRV_XPR", ...)
"service-id" ("00", ...)
...
The lowest Service-priority you may use is '9999' (As default a
SpareNode has a prio of '10000').
See 'CNF_TakeOverStrategy_LowPrioFit_LowestPrio'.
-->
<!-- Anfang: ServiceSchema default_trivial -->
<gr:service schema="default_trivial" name="Default">
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="1"/>
<attr:attribute name="service-powervalue" value="2201"/>
</gr:service>
<!-- Ende: ServiceSchema default_trivial -->
<!-- Anfang: ServiceSchema default -->
<gr:service schema="default" name="Default">
<match category="${service-type}">
<value>SRV_DBORA</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="2"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
Parameter Reference
235 FlexFrame Agents Guide
<gr:service schema="default" name="Default">
<match category="${service-type}">
<value>SRV_SAPSYBASE</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="2"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
<gr:service schema="default" name="Default">
<match category="${service-type}">
<value>SRV_SAPSYB_IQ</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="2"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
<gr:service schema="default" name="Default">
<match category="${service-type}">
<value>SRV_DBSAP</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="2"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
<gr:service schema="default" name="Default">
<match category="${service-type}">
<value>SRV_DB2</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="2"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
<gr:service schema="default" name="Default">
<match category="${service-type}">
<value>SRV_SCS</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="3"/>
<attr:attribute name="service-powervalue" value="0"/>
</gr:service>
<gr:service schema="default" name="Default">
<match category="${service-type}">
<value>SRV_ASCS</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="3"/>
<attr:attribute name="service-powervalue" value="0"/>
</gr:service>
<gr:service schema="default" name="Default">
<match category="${service-type}">
<value>SRV_CI</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="4"/>
<attr:attribute name="service-powervalue" value="100000"/>
Parameter Reference
FlexFrame Agents Guide 236
</gr:service>
<gr:service schema="default" name="Default">
<match category="${service-type}">
<value>SRV_JC</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="5"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
<gr:service schema="default" name="Default">
<match category="${service-type}">
<value>SRV_J</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="6"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
<gr:service schema="default" name="Default">
<match category="${service-type}">
<value>SRV_LC</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="6"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
<gr:service schema="default" name="Default">
<match category="${service-type}">
<value>SRV_APP</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="7"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
<gr:service schema="default" name="Default">
<match category="${service-type}">
<value>SRV_ERS</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="8"/>
<attr:attribute name="service-powervalue" value="0"/>
</gr:service>
<gr:service schema="default" name="Default">
<match category="${service-type}">
<value>SRV_MDS71</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="9"/>
<attr:attribute name="service-powervalue" value="0"/>
</gr:service>
<gr:service schema="default" name="Default">
<match category="${service-type}">
<value>SRV_MDIS71</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="9"/>
Parameter Reference
237 FlexFrame Agents Guide
<attr:attribute name="service-powervalue" value="0"/>
</gr:service>
<gr:service schema="default" name="Default">
<match category="${service-type}">
<value>SRV_MDSS71</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="9"/>
<attr:attribute name="service-powervalue" value="0"/>
</gr:service>
<gr:service schema="default" name="Default">
<match category="${service-type}">
<value>SRV_XPR</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="9"/>
<attr:attribute name="service-powervalue" value="0"/>
</gr:service>
<gr:service schema="default" name="Default">
<match category="${service-type}">
<value>SRV_CMS</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="2"/>
<attr:attribute name="service-powervalue" value="0"/>
</gr:service>
<gr:service schema="default" name="Default">
<match category="${service-type}">
<value>SRV_TRX</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="10"/>
<attr:attribute name="service-powervalue" value="0"/>
</gr:service>
<gr:service schema="default" name="Default">
<match category="${service-type}">
<value>SRV_BOE4</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="2"/>
<attr:attribute name="service-powervalue" value="0"/>
</gr:service>
<gr:service schema="default" name="Default">
<match category="${service-type}">
<value>SRV_BOE_SRV</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="2"/>
<attr:attribute name="service-powervalue" value="0"/>
</gr:service>
<gr:service schema="default" name="Default">
<match category="${service-type}">
<value>SRV_SMD</value>
</match>
<!-- attributes for selected services -->
Parameter Reference
FlexFrame Agents Guide 238
<attr:attribute name="service-priority" value="8"/>
<attr:attribute name="service-powervalue" value="0"/>
</gr:service>
<gr:service schema="default" name="Default">
<match category="${service-type}">
<value>SRV_PAI</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="4"/>
<attr:attribute name="service-powervalue" value="0"/>
</gr:service>
<gr:service schema="default" name="Default">
<match category="${service-type}">
<value>SRV_VMWARE</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="2"/>
<attr:attribute name="service-powervalue" value="0"/>
</gr:service>
<gr:service schema="default" name="Default">
<match category="${service-type}">
<value>SRV_ORACLE_RAC_CRS</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="2"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
<gr:service schema="default" name="Default">
<match category="${service-type}">
<value>SRV_ORACLE_RAC_DB</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="2"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
<gr:service schema="default" name="Default">
<match category="${service-type}">
<value>SRV_CTM</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="2"/>
<attr:attribute name="service-powervalue" value="0"/>
</gr:service>
<gr:service schema="default" name="Default">
<match category="${service-type}">
<value>SRV_WEBDISP</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="9"/>
<attr:attribute name="service-powervalue" value="0"/>
</gr:service>
<gr:service schema="default" name="Default">
<match category="${service-type}">
<value>SRV_SMD73</value>
</match>
Parameter Reference
239 FlexFrame Agents Guide
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="8"/>
<attr:attribute name="service-powervalue" value="0"/>
</gr:service>
<gr:service schema="default" name="Default">
<match category="${service-type}">
<value>SRV_HANA</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="2"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
<gr:service schema="default" name="Default">
<match category="${service-type}">
<value>SRV_HANA_REP</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="1"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
<gr:service schema="default" name="Default">
<match category="${service-type}">
<value>SRV_HANA_MN</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="1"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
<gr:service schema="default" name="Default">
<match category="${service-type}">
<value>*</value>
</match>
<match category="${system-id}">
<value>*</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="998"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
<!-- Ende: ServiceSchema default -->
<!-- Anfang: ServiceSchema Example_service_type -->
<gr:service schema="Example_service_type" name="Service_${service-
type}">
<match category="${service-type}">
<value>SRV_DBORA</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="1"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
<gr:service schema="Example_service_type" name="Service_${service-
type}">
<match category="${service-type}">
Parameter Reference
FlexFrame Agents Guide 240
<value>SRV_CI</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="3"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
<gr:service schema="Example_service_type" name="Service_${service-
type}">
<match category="${service-type}">
<value>SRV_APP</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="6"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
<!--
...
-->
<gr:service schema="Example_service_type" name="other">
<match category="${service-type}">
<value>*</value>
</match>
<match category="${system-id}">
<value>*</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="998"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
<!-- Ende: ServiceSchema Example_service_type -->
<!-- Anfang: ServiceSchema static -->
<gr:service schema="Example_static" name="ProdDB">
<gr:description></gr:description>
<match category="${system-id}">
<value>P*</value>
</match>
<match category="${service-type}">
<value>SRV_DBORA</value>
<value>SRV_SAPSYBASE</value>
<value>SRV_DBSAP</value>
<value>SRV_DB2</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="1"/>
<attr:attribute name="service-powervalue" value="1000"/>
</gr:service>
<!-- Ende: ServiceSchema static -->
<!-- Anfang: ServiceSchema Example_Customer -->
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
<value>SRV_APP</value>
</match>
Parameter Reference
241 FlexFrame Agents Guide
<match category="${system-id}">
<value>PRE</value>
</match>
<match category="${service-id}">
<value>24</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="37"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
<value>SRV_APP</value>
</match>
<match category="${system-id}">
<value>PRE</value>
</match>
<match category="${service-id}">
<value>26</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="37"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
<value>SRV_DB*</value>
</match>
<match category="${system-id}">
<value>PPI</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="12"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
<value>SRV_CI</value>
</match>
<match category="${system-id}">
<value>PPI</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="14"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
<value>SRV_SCS</value>
</match>
<match category="${system-id}">
<value>PPI</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="13"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
Parameter Reference
FlexFrame Agents Guide 242
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
<value>SRV_APP</value>
</match>
<match category="${system-id}">
<value>PRE</value>
</match>
<match category="${service-id}">
<value>27</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="37"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
<value>SRV_DB*</value>
</match>
<match category="${system-id}">
<value>PBW</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="12"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
<value>SRV_CI</value>
</match>
<match category="${system-id}">
<value>PBW</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="14"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
<value>SRV_SCS</value>
</match>
<match category="${system-id}">
<value>PBW</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="13"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
<value>SRV_DB*</value>
</match>
<match category="${system-id}">
<value>PRE</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="12"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
Parameter Reference
243 FlexFrame Agents Guide
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
<value>SRV_CI</value>
</match>
<match category="${system-id}">
<value>PRE</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="14"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
<value>SRV_DB*</value>
</match>
<match category="${system-id}">
<value>P10</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="22"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
<value>SRV_CI</value>
</match>
<match category="${system-id}">
<value>P10</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="24"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
<value>SRV_DB*</value>
</match>
<match category="${system-id}">
<value>PSM</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="52"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
<value>SRV_CI</value>
</match>
<match category="${system-id}">
<value>PSM</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="54"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
<gr:service schema="Example_Customer" name="Example_Customer">
Parameter Reference
FlexFrame Agents Guide 244
<match category="${service-type}">
<value>SRV_SCS</value>
</match>
<match category="${system-id}">
<value>PSM</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="53"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
<value>SRV_APP</value>
</match>
<match category="${system-id}">
<value>PBW</value>
</match>
<match category="${service-id}">
<value>28</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="47"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
<value>SRV_DB*</value>
</match>
<match category="${system-id}">
<value>QRE</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="102"/>
<attr:attribute name="service-powervalue" value="100"/>
</gr:service>
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
<value>SRV_CI</value>
</match>
<match category="${system-id}">
<value>QRE</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="104"/>
<attr:attribute name="service-powervalue" value="100"/>
</gr:service>
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
<value>SRV_DB*</value>
</match>
<match category="${system-id}">
<value>TRE</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="102"/>
<attr:attribute name="service-powervalue" value="100"/>
</gr:service>
Parameter Reference
245 FlexFrame Agents Guide
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
<value>SRV_CI</value>
</match>
<match category="${system-id}">
<value>TRE</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="104"/>
<attr:attribute name="service-powervalue" value="100"/>
</gr:service>
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
<value>SRV_DB*</value>
</match>
<match category="${system-id}">
<value>Q10</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="102"/>
<attr:attribute name="service-powervalue" value="100"/>
</gr:service>
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
<value>SRV_CI</value>
</match>
<match category="${system-id}">
<value>Q10</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="104"/>
<attr:attribute name="service-powervalue" value="100"/>
</gr:service>
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
<value>SRV_DB*</value>
</match>
<match category="${system-id}">
<value>T10</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="102"/>
<attr:attribute name="service-powervalue" value="100"/>
</gr:service>
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
<value>SRV_CI</value>
</match>
<match category="${system-id}">
<value>T10</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="104"/>
<attr:attribute name="service-powervalue" value="100"/>
</gr:service>
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
Parameter Reference
FlexFrame Agents Guide 246
<value>SRV_DB*</value>
</match>
<match category="${system-id}">
<value>QBW</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="102"/>
<attr:attribute name="service-powervalue" value="100"/>
</gr:service>
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
<value>SRV_CI</value>
</match>
<match category="${system-id}">
<value>QBW</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="104"/>
<attr:attribute name="service-powervalue" value="100"/>
</gr:service>
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
<value>SRV_SCS</value>
</match>
<match category="${system-id}">
<value>QBW</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="103"/>
<attr:attribute name="service-powervalue" value="100"/>
</gr:service>
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
<value>SRV_DB*</value>
</match>
<match category="${system-id}">
<value>TBW</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="102"/>
<attr:attribute name="service-powervalue" value="100"/>
</gr:service>
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
<value>SRV_CI</value>
</match>
<match category="${system-id}">
<value>TBW</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="104"/>
<attr:attribute name="service-powervalue" value="100"/>
</gr:service>
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
<value>SRV_SCS</value>
</match>
Parameter Reference
247 FlexFrame Agents Guide
<match category="${system-id}">
<value>TBW</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="103"/>
<attr:attribute name="service-powervalue" value="100"/>
</gr:service>
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
<value>SRV_DB*</value>
</match>
<match category="${system-id}">
<value>QPI</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="102"/>
<attr:attribute name="service-powervalue" value="100"/>
</gr:service>
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
<value>SRV_CI</value>
</match>
<match category="${system-id}">
<value>QPI</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="104"/>
<attr:attribute name="service-powervalue" value="100"/>
</gr:service>
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
<value>SRV_SCS</value>
</match>
<match category="${system-id}">
<value>QPI</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="103"/>
<attr:attribute name="service-powervalue" value="100"/>
</gr:service>
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
<value>SRV_DB*</value>
</match>
<match category="${system-id}">
<value>TPI</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="102"/>
<attr:attribute name="service-powervalue" value="100"/>
</gr:service>
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
<value>SRV_CI</value>
</match>
<match category="${system-id}">
<value>TPI</value>
Parameter Reference
FlexFrame Agents Guide 248
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="104"/>
<attr:attribute name="service-powervalue" value="100"/>
</gr:service>
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
<value>SRV_SCS</value>
</match>
<match category="${system-id}">
<value>TPI</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="103"/>
<attr:attribute name="service-powervalue" value="100"/>
</gr:service>
<gr:service schema="Example_Customer" name="Example_Customer">
<match category="${service-type}">
<value>*</value>
</match>
<match category="${system-id}">
<value>*</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="998"/>
<attr:attribute name="service-powervalue" value="100000"/>
</gr:service>
<!-- Ende: ServiceSchema Example_Customer -->
</gr:services>
<!-- ********************************************************** -->
<!-- ********************************************************** -->
<!-- ********************************************************** -->
<gr:groups>
<!--
"group schema" is specified in myAMC_FA.xml
allowed categories:
"hostname" ("vader", ...)
"ip-adr" ("192.168.1.47", ...)
"OS-typ" ("SunOS", ...)
"OS-version" ("5.8", "SuSE SLES-8
(i386); VERSION = 8.1", ...)
"OS-bits" (for future use)
"CPU-arch" ("sun4u", ...)
"CPU-cnt-siblings" ("2")
(min-, max-)
"CPU-cnt-core" ("2")
(min-, max-)
"CPU-cnt-phys" ("2")
(min-, max-)
"CPU-frequency(MHz)" ("333")
(min-, max-)
"cache-size(kB)" ("2048") (min-,
max-)
"mem-total(MB)" ("1024")
Parameter Reference
249 FlexFrame Agents Guide
(min-, max-)
"node-hostname" ("vader", ...)
"node-product-name" ("SUNW,Ultra-5_10", ...)
"node-vendor" ("Sun_Microsystems", ...)
"node-powervalue" ("1000")
(min-, max-)
-->
<!-- ********************************************************** -->
<!-- ********************************************************** -->
<!--
-->
<gr:group schema="default" name="GR_All">
<gr:description></gr:description>
<match category="${node-hostname}">
<value>vade*</value>
<value>yod*</value>
<value>blade_a*</value>
<value>blade_b*</value>
<value>server*</value>
</match>
</gr:group>
<gr:group schema="default_LDAP" name="LD_Solaris_8">
<gr:description></gr:description>
<match category="${OS-version}">
<value>5.8</value>
</match>
</gr:group>
<gr:group schema="default_LDAP" name="LD_SLES_8">
<gr:description></gr:description>
<match category="${OS-version}">
<value>SuSE SLES-8*</value>
</match>
</gr:group>
<gr:group schema="default_LDAP" name="LD_Solaris_9">
<gr:description></gr:description>
<match category="${OS-version}">
<value>5.9</value>
</match>
</gr:group>
<gr:group schema="default_LDAP" name="LD_SLES_9">
<gr:description></gr:description>
<match category="${OS-version}">
<value>*SUSE LINUX Enterprise Server 9*</value>
</match>
</gr:group>
<gr:group schema="default_OS" name="Solaris_8">
<gr:description></gr:description>
<match category="${OS-version}">
<value>5.8</value>
</match>
Parameter Reference
FlexFrame Agents Guide 250
</gr:group>
<gr:group schema="default_OS" name="SLES_8">
<gr:description></gr:description>
<match category="${OS-version}">
<value>SuSE SLES-8*</value>
</match>
</gr:group>
<gr:group schema="default_OS" name="Solaris_9">
<gr:description></gr:description>
<match category="${OS-version}">
<value>5.9</value>
</match>
</gr:group>
<gr:group schema="default_OS" name="SLES_9">
<gr:description></gr:description>
<match category="${OS-version}">
<value>*SUSE LINUX Enterprise Server 9*</value>
</match>
</gr:group>
<gr:group schema="BSP_OStyp_1" name="BSP_GR_default_${OS-typ}">
<gr:description></gr:description>
<match category="${OS-typ}">
<value>${OS-typ}</value>
</match>
<match category="${CPU-cnt-phys}" syntax="min">
<value>0</value>
</match>
<match category="${mem-total(MB)}" syntax="min">
<value>20</value>
</match>
<match category="${ip-adr:netmask=/24}">
<value>158.92.4.0</value>
</match>
<match category="${node-powervalue}" syntax="min">
<value>0</value>
</match>
<match category="${node-hostname}">
<value>vade*</value>
<value>blade_a*</value>
<value>blade_b*</value>
<value>server*</value>
</match>
</gr:group>
<gr:group schema="BSP_static_1" name="BSP_MyCompany_${OS-typ}">
<gr:description></gr:description>
<match category="${OS-typ}">
<value>${OS-typ}</value>
</match>
<match category="${CPU-cnt-phys}" syntax="min">
<value>4</value>
</match>
<match category="${mem-total(MB)}" syntax="min">
<value>4000</value>
Parameter Reference
251 FlexFrame Agents Guide
</match>
<match category="${ip-adr:netmask=/24}">
<value>192.168.10.0</value>
</match>
<match category="${node-powervalue}">
<value>1000</value>
</match>
<match category="${node-hostname}">
<value>blade_a*</value>
<value>blade_b*</value>
<value>server*</value>
</match>
</gr:group>
<gr:group schema="BSP_static_2" name="BSP_MyCompany">
<gr:description></gr:description>
<match category="${CPU-cnt-phys}" syntax="max">
<value>2</value>
</match>
<match category="${ip-adr:netmask=/24}">
<value>192.168.10.0</value>
</match>
<match category="${node-powervalue}">
<value>1000</value>
</match>
<match category="${node-hostname}">
<value>blade_a*</value>
<value>blade_b*</value>
<value>server*</value>
</match>
</gr:group>
<gr:group schema="BSP_static_3" name="BSP_3">
<gr:description></gr:description>
<match category="${ip-adr}" syntax="min">
<value>192.168.10.17</value>
</match>
<match category="${ip-adr}" syntax="max">
<value>192.168.10.21</value>
</match>
<match category="${node-powervalue}">
<value>1000</value>
</match>
<match category="${node-hostname}">
<value>vad*</value>
<value>blade_a*</value>
<value>blade_b*</value>
<value>server*</value>
</match>
</gr:group>
<!-- ********************************************************** -->
<!-- ********************************************************** -->
<!-- auto groups -->
<!--
Parameter Reference
FlexFrame Agents Guide 252
DO NOT EDIT !!!
If you need your own group definition, please use the
section befor !!!
-->
<gr:group schema="os_network_cpu" name="AutoGroup_${OS-typ}_${CPU-
cnt-phys}cpu_${ip-adr:netmask=/24}">
<gr:description>Group defined by number of CPUs, network and
operating system</gr:description>
</gr:group>
<gr:group schema="os_cpu" name="AutoGroup_${OS-typ}_${CPU-cnt-
phys}cpu">
<gr:description>Group defined by number of CPUs and
operating system</gr:description>
</gr:group>
<gr:group schema="os_network" name="AutoGroup_${OS-typ}_${ip-
adr:netmask=/24}">
<gr:description>Group defined by number of CPUs and
network</gr:description>
</gr:group>
<gr:group schema="network_cpu" name="AutoGroup_${CPU-cnt-
phys}cpu_${ip-adr:netmask=/24}">
<gr:description>Group defined by number of
CPUs</gr:description>
</gr:group>
<gr:group schema="os" name="AutoGroup_${OS-typ}">
<gr:description>Group defined by operating
system</gr:description>
</gr:group>
<gr:group schema="network" name="AutoGroup_${ip-adr:netmask=/24}">
<gr:description>Group defined by network</gr:description>
</gr:group>
<gr:group schema="cpu" name="AutoGroup_${CPU-cnt-phys}cpu">
<gr:description>Group defined by number of
CPUs</gr:description>
</gr:group>
<!-- ********************************************************** -->
</gr:groups>
</defs:definitions>
Parameter Reference
253 FlexFrame Agents Guide
7.4 Service Classes
The service classes are defined and parameterized in the group configuration file (my-
AMC_FA_Groups.xml) of a virtual FlexFrame pool.
A sevice class is defined by the following variables:
"system-id" ("P46", "O20", ...)
"service-type" ("db", "app", "ci", ...)
"service-id" ("00", ...)
The attributes service-priority and service-powervalue are defined in accord-
ance with these variables.
In the future it will be possible to extend such a service class by further attributes which,
for example, define the operating system required by a service or the number of
CPUs/performance requirement of the service.
7.4.1 Service Priority
The highest service priority is 1. Every service is assigned this priority by default, i.e. if no
service classes are defined, all services have the priority 1. The higher the number, the
lower the priority of a service.
Priority 0 has a special status. Setting priority 0 for a service class enables the autono-
mous functions to be disabled for a service.
The service priority is evaluated for all autonomous reactions. If, for example, a service of
a productive system and a service of a test system are running on the same node and the
test system’s service is assigned priority 5, this service is not executed because the pro-
ductive system’s service which is functioning without error has the higher priority of 1.
7.4.2 Service Power Value
The service power value specifies for a service a performance number which defines the
maximum performance (powervalue [SAPs-äquivalent]) required by this service. This is
provided for future enhancements in the field of “load distribution and load transfer“.
A failed service with a performance value of 50 can, for example, also be taken over by a
node which still has at least 50 of its maximum performance number free.
7.4.3 Class Creation Rules
A service belongs either to the default class which always exists or it can be assigned
unambiguously to another class by evaluating the aforementioned variables.
Parameter Reference
FlexFrame Agents Guide 254
7.4.4 Example
<gr:services>
<!—
"service schema" is specified in myAMC_FA.xml
allowed categories:
"system-id" ("P46", "O20", ...)
"service-type" ("db", "app", "ci", ...)
"service-id" ("00", ...)
...
-->
<gr:service schema="default" name="Default">
<match category="${service-type}">
<value>DB</value>
<value>db</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="2"/>
<attr:attribute name="service-powervalue" value="2202"/>
</gr:service>
<gr:service schema="default" name="Default">
<match category="${service-type}">
<value>CI</value>
<value>ci</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="3"/>
<attr:attribute name="service-powervalue" value="2203"/>
</gr:service>
<gr:service schema="default" name="Default">
<match category="${service-type}">
<value>APP</value>
<value>app</value>
</match>
<!-- attributes for selected services -->
<attr:attribute name="service-priority" value="4"/>
<attr:attribute name="service-powervalue" value="2204"/>
</gr:service>
... ...
7.5 FlexFrame Agents
The base parameterization is always required.
The parameterization is organized hierarchically, i.e. information can be configured iden-
tically for all services or specifically for individual services.
myAMC_FA.xml
Parameter Reference
255 FlexFrame Agents Guide
The information to be configured relates to the following components:
General parameters
Parameters for the Performance and Accounting option
Node-related parameters
Service-related parameters
Path configurations
7.5.1 General Parameters
CheckCycleTime
Defines the cycles in which the detector modules supply results and the rule modules
evaluate the status derived from these. The parameter value may not be less than
the minimum processing time which the detector modules, rule modules and reaction
modules require to process a cycle. The default value in the as-supplied status is 10
seconds. The parameter value must also always be at least 1/3 of the lifetime of the
MonitorAlerts. In the FlexFrame standard installation the lifetime of the MonitorAlerts
is 30 seconds.
LivelistWriterTime
Defines the intervals at which the FlexFrame Agents must generate a Livelist. It is
specified in seconds.
ControlAgentTime
Specifies how often the Control Agent checks the Livelists of the Application Agents. The parameter should thus be about the same as the LiveListWriterTime.
MaxHeartbeatTime
Specifies the maximum time which may elapse between two Livelist entries of an Application Agent before the Control Agent intervenes. The MaxHeartbeatTime
must therefore always be greater than the ControlAgentTime and the Livelist-
WriterTime. In practice the factor of 3 between LivelistWriterTime and
MaxHeartbeatTime has proved practical.
MaxRebootTime
Specifies the maximum time which may elapse between two Livelist entries of an
Application Agent before the Control Agent intervenes if the latter is rebooting.
MaxFailedReachNumber
Specifies how often the Control Agent attempts to reach a node after the MaxHeart-
beatTime has been exceeded before an external SwitchOver is initiated.
MaxAgeSwitchOverFile
This parameter specifies the max age (in seconds) of SwitchOver-File.
If the age of a SwitchOver-file exceeds this value, the SwitchOver-file is ignored.
Node_SwitchOverTyp
Specifies the mode according to which the testaments are created: node-based or
service-based. The following keywords are currently valid for this parameter:
Parameter Reference
FlexFrame Agents Guide 256
service
node
TakeOverStrategy
Defines how the application to take over services of defective nodes occurs and how
the winner is determined.
Allowed values are:
FirstFit
The first application node, which makes a entry to the switchover file, is
the winner and is allowed to take over the services.
LowPrioFit
The application node with the worst priority wins and is allowed to take
over the services. The priority of the application node will be taken from
the service with the highest priority on this application node.
Node_TakeOverRule
Defines which takeover rule is to be used. The following values are possible:
SpareNode
Substitution (add rule)
Displacement (replace rule)
Supplementation (supplementation)
Dynamic
Parameter Reference
257 FlexFrame Agents Guide
Dyn_Spare_min
Dyn_Spare_max
Dyn_Add_min
Dyn_Add_max
Dyn_Replace_min
Dyn_Replace_max
Dyn_Substitute_mi
Dyn_Substitute_max
Dyn_Spare_exclusive
Defines the range of takeover rule “dynamic”.
CtrlAgent_WriteOnlyChanges
Controls the writing of output files.
The following values are possible:
false: files will be written every cycle (default) .
true: files will be written only if there was changes.
With this parameter load can be reduced on control node at great installations
and on the Storagesystem since changes are only written persistently at rele-
vant changes. The net load to the FA WebGUI and the load the the server part
of the FA WebGUI is also reduced.
7.5.2 Parameters for the Performance and Accounting Op-tion
FlexFrame Agent:
PerfdataReportCycleTime (myAMC_FA.xml)
The report cycle specifies the cycle of creating a performance and accounting value,
which is written to the collet.
max_colletcount_performance_files (myAMC_FA_AppAgent_spec.xml)
Number of collet generations that are stored before being rewritten.
max_filesize_performance_files (myAMC_FA_AppAgent_spec.xml)
Maximum size of the performance files. This parameter serves to limit the size of the
files.
collet_switch_start_performance_files (myAMC_FA_AppAgent_spec.xml)
Defines the date and time when new collet files are written.
Eexample: 2006-01-01 00:00:00
Parameter Reference
FlexFrame Agents Guide 258
collet_switch_cycle_performance_files (myAMC_FA_AppAgent_spec.xml)
Cycle time when new collets are written. Example: 3600
By combining collet_switch_start_performance_files and
collet_switch_cycle_performance_files, it is possible to specify that, starting at
0:00, new collets are written every 3600 seconds, i.e. every hour.
It is also possible to implement different cycle times.
7.5.3 Parameters for the File System Utilization
Perfdata_FSUtil_ReportCycleTime
Specifies the report cycle time for FsUtil checks.
FSUtilScriptFilePath
Specifies the path to the FsUtil scripts.
Script file names are myAMC.DetGetDirData.sh, myAMC.DetGetFSData.sh.
Performance files Perf_FsUtil~<hostname>.prf.
Line-Identifier in this file is FSUTIL.
7.5.4 Configuration of the Spaceutil Functionality
The monitored directories and file systems will be pool specific configured in the configu-ration file myAMC_FA_Checks.xml in the directory
/opt/myAMC/vFF/vFF_<poolname>/config. The configuration consists of tem-
plates, depending on the actual conditions to be activated.
7.5.4.1 General structure
The configuration file for monitoring the filesystem utilization or the directory utilization
use a common structure. and a specific structure. For every Filesystem or directory to
monitor you have to define an entry in theConfigSection like “TrapTargets with the follow-
ing ConfigEntries :
Active
Indicates whether this entry is currently active or not. In case of inactive the complete
entry is ignored.
Name
Symbolic display name
The symbolic name can be freely assigned with a maximum lenth of 50 characters
and no special characters.
Group
Group name for logical grouping, that allows group based aggregation. The group
name can be freely assigned with a maximum length of 50 characters and no special
characters.
Parameter Reference
259 FlexFrame Agents Guide
ContextObject
Reference object. Possible values are Frame, Pool, Group, Node, System, or
Service.
FrameActivationPattern
Wildcard pattern for frame-specific activation
PoolActivationPattern
Wildcard patern for pool specific activation.
GroupActivationPattern
Wildcard Pattern for group specific activation.
NodeActivationPattern
Wildcard pattern for node specific activation
SystemActivationPattern
Wildcard pattern for system specific activation
ServiceTypeActivationPattern
Wildcard pattern for service specific activation
InstNumActivationPattern
Wildcard pattern for instance specific activation
The values in the reference object define the aggregation type of the data.
7.5.4.2 File System Utilization
In addition to the common values there are some attributes to define the parameters for
file system utilization.
Configuration set to monitor a file system (ConfigSection, like TrapTargets)
/myAMC.FA/FilesystemUtilization.
(ConfigEntries):
FilesystemNamePattern
Wildcard pattern to identify the file system.
If this value is missing or empty, this pattern is always true. This parameter can be definied with placeholder like $(name).
MountPointPattern
Wildcard pattern to select certain files using mount points.
If this value is missing or empty this value is always true. This parameter can be de-finied with placeholder like $(name).
MaxUtilization
Thresholding (maximum utilization in percent) to trigger utilization events. The value
0 disables this threshold monitoring.
Parameter Reference
FlexFrame Agents Guide 260
MaxGrowthRate
Threshold for maximum growth rate in percent per second. to trigger a trap as re-
sponse to large growth rates. The value 0 disables this threshold
7.5.4.3 Directory Utilization
Every directory to monitor need a config entry with following values (ConfigSection,
like TrapTargets) bellow the key /myAMC.FA/DirectoryUtilization.
In addition to the standard values (ConfigEntries) :
Paths
Absolute path of the directory being monitored.
This parameter can include placeholder in the form $(name).
7.5.4.4 Activation of Configration Entries
The directory and file system monitoring is based on a reference object. A reference
object can be of the type Frame, Pool, Group, Node, System or Service.
The activation depends on the entry in the config files and dynamic values from the
agent.
Parameter name Activation depends on Example
AgentTypeActivationPattern Agent type (“myAMC.AppAgent”,
“myAMC.FrameAgent”)
*Agent*
FrameActivationPattern Frame name
(allways local)
Local
PoolActivationPattern Pool name MyPool*
GroupActivationPattern Group name GR_Lin*
NodeActivationPattern Node name bx*
SystemActivationPattern SID P*
ServiceTypeActivationPattern Service type, possible
values: SRV_DBORA,
RV_DB2, SRV_DBSAP,
SRV_CI, SRV_APP,
SRV_J, SRV_JC,
SRV_SCS, SRV_ERS,
SRV_ASCS
SRV_DB*
InstNumActivationPattern Instance number
to definie a special in-
stance
02
Parameter Reference
261 FlexFrame Agents Guide
For every config entry the FlexFrame AppAgent or FlexFrame FrameAgent checks
together with the dynamical information Framename, Poolname, Groupname,
Nodename, Servicename if there is a match with a config entry. So managed and
monitored sets of objects are dynamicly defined.
Reference objects Frame, Pool, Group, Node
Frame, pool, group or node specific rules to monitor file system utilization. Reference object: Service.
The Service type ist the activator for a rule, for example monitor only spezial files if
their is an db service on the node.
Reference object System
The system name is activator for a rule
Examples:
AppAgent Active Services: dbo46 (ora), dbp52 (sapdb), app02p52, cip55
Reference object: Group
Active: false
The entry is ignored because active is false.
AppAgent Active Services: dbo46 (ora), dbp52 (sapdb), app02p52, cip55
Reference object: Pool
Active: true
ServiceTypeActivationPattern: SRV_DBORA
other ActivationPatterns: empty or „*“
The FF AppAgent activates a rule for the defined pool, but only for DB services.
AppAgent
Active Services: dbo46 (ora), dbp52 (sapdb), app02p52, cip55
Reference objekt: System
Active: true
SystemActivationPattern: P*
other ActivationPatterns: empty or „*“
The FF AppAgent activates a rule for SAP SIDs which match with P*.
FrameAgent
Active Pools: Cust_1, pool2, Cust_3
Reference objekt: Pool
Active: true
FrameActivationPattern: A00*
PoolActivationPattern: Cust_*
other ActivationPatterns: empty or „*“
The FF FrameAgent activates a monitoring rule for the pools Cust_1 and Cust_3,
but only if the name of the frames correlates with the FrameActivationPattern
Parameter Reference
FlexFrame Agents Guide 262
7.5.4.5 Replacing of Placeholders
The configuration section allows for some parameters the use of placeholders. The
placeholders a resolved from the FlexFrame AppAgent or FlexFrame FrameAgent. If a
placeholder is not resolved, an empty string is used.
The following placeholder are available:
${framename} (currently always Local)
Example: Local, A00001
${poolname}
Example: Cust_3, pool1
${groupname}
Example: GR_Linux
${nodename}
Example: tombx2, sno1aplp2
${systemname}
Example: P46, ABC
${servicetype}
Example: SRV_APP, SRV_CI, SRV_DBORA, SRV_DBSAP, SRV_SCS, ...
${servicedisplaytype} Example: app, ci, db, scs, ...
${instnum}
The instance number will only be dissolved if it is valid (>=0). The number is always used
as double digit.
Example: 7 07
-1 “”
42 42
7.5.4.6 Wildcard Patterns
Wildcard patterns allow comparison wtih variable texts. Unknown elements can be se-
lected by the wildcard “*” and “?”.
“*” stands for an unlimited quantity of any character that may be present or not.
“?” stands for a single character
Example:
Parameter Reference
263 FlexFrame Agents Guide
*.doc
matches with abc.doc, .doc and aabbccdd.doc , but not with abcdoc,
abc.doc2 or abc.xdoc
app??O46
matches with app1O46, app02O46, appO46 but not with app123O46, ciO46 or
ap02O46
app??P*
matches with app1P52, app02P5, appPPP but not with app123O46, app1O52, or
ap02O46
Parameter Reference
FlexFrame Agents Guide 264
7.5.4.7 Configuration File myAMC_FA_Checks.xml
<?xml version="1.0" encoding="ISO-8859-1"?>
<!-- @@@MYAMC.VERSION@@@ 9.0-23 -->
<configuration>
<configsection name="myAMC.FA">
<configsection name="Additional_Checks">
<!-- SpaceUtil: DirectoryUtilization -->
<!-- ############################### -->
<configsection name="DirectoryUtilization">
<!-- Pool specific directory -->
<configsection name="FAPoolData">
<configentry name="Active">
<value
type="Boolean">false</value>
</configentry>
<!-- identification -->
<configentry name="Name">
<value
type="String">fapooldata</value>
</configentry>
<configentry name="Group">
<value
type="String">fapooldata</value>
</configentry>
<configentry name="ContextObject">
<value type="String">Pool</value>
</configentry>
<!-- activation -->
<configentry
name="AgentTypeActivationPattern">
<value
type="String">*Agent*</value>
</configentry>
<configentry
name="FrameActivationPattern">
<value type="String">*</value>
</configentry>
<configentry name="PoolActivationPattern">
<value type="String">*</value>
</configentry>
<configentry
name="GroupActivationPattern">
<value type="String">*</value>
</configentry>
<configentry name="NodeActivationPattern">
<value type="String">*</value>
</configentry>
<configentry
name="SystemActivationPattern">
<value type="String">*</value>
</configentry>
<configentry
name="ServiceTypeActivationPattern">
<value type="String">*</value>
</configentry>
<configentry
name="InstNumActivationPattern">
Parameter Reference
265 FlexFrame Agents Guide
<value type="String">*</value>
</configentry>
<!-- paths to check -->
<configentry name="Paths">
<value
type="String">/opt/myAMC/vFF/vFF_${poolname}/data</value>
</configentry>
</configsection>
<!-- Service specific directory -->
<configsection name="OracleDB">
<configentry name="Active">
<value
type="Boolean">false</value>
</configentry>
<!-- identification -->
<configentry name="Name">
<value
type="String">oradata</value>
</configentry>
<configentry name="Group">
<value
type="String">database</value>
</configentry>
<configentry name="ContextObject">
<value
type="String">Service</value>
</configentry>
<!-- activation -->
<configentry
name="AgentTypeActivationPattern">
<value
type="String">*AppAgent*</value>
</configentry>
<configentry
name="FrameActivationPattern">
<value type="String">*</value>
</configentry>
<configentry name="PoolActivationPattern">
<value type="String">*</value>
</configentry>
<configentry
name="GroupActivationPattern">
<value type="String">*</value>
</configentry>
<configentry name="NodeActivationPattern">
<value type="String">*</value>
</configentry>
<configentry
name="SystemActivationPattern">
<value type="String">*</value>
</configentry>
<configentry
name="ServiceTypeActivationPattern">
<value
type="String">SRV_DBORA</value>
</configentry>
<configentry
name="InstNumActivationPattern">
<value type="String">*</value>
</configentry>
<!-- paths to check -->
Parameter Reference
FlexFrame Agents Guide 266
<configentry name="Paths">
<value
type="String">/oracle/${systemname}/sapdata*</value>
</configentry>
</configsection>
</configsection>
<!-- SpaceUtil: FilesystemUtilization -->
<!-- ################################ -->
<configsection name="FilesystemUtilization">
<!-- Frame specific filesystem -->
<configsection name="volFF">
<configentry name="Active">
<value type="Boolean">true</value>
</configentry>
<!-- identification -->
<configentry name="Name">
<value type="String">volFF</value>
</configentry>
<configentry name="Group">
<value
type="String">storage</value>
</configentry>
<configentry name="ContextObject">
<value type="String">Frame</value>
</configentry>
<!-- activation -->
<configentry
name="AgentTypeActivationPattern">
<value
type="String">*FrameAgent*</value>
</configentry>
<configentry
name="FrameActivationPattern">
<value type="String">*</value>
</configentry>
<configentry name="PoolActivationPattern">
<value type="String">*</value>
</configentry>
<configentry
name="GroupActivationPattern">
<value type="String">*</value>
</configentry>
<configentry name="NodeActivationPattern">
<value type="String">*</value>
</configentry>
<configentry
name="SystemActivationPattern">
<value type="String">*</value>
</configentry>
<configentry
name="ServiceTypeActivationPattern">
<value type="String">*</value>
</configentry>
<configentry
name="InstNumActivationPattern">
<value type="String">*</value>
Parameter Reference
267 FlexFrame Agents Guide
</configentry>
<!-- filesystem information -->
<configentry name="FilesystemNamePattern">
<value
type="String">*:/*volFF*</value>
</configentry>
<configentry name="MountPointPattern">
<value type="String">*</value>
</configentry>
<!-- threshold values -->
<configentry name="MaxUtilization">
<value type="Float">90.0</value>
</configentry>
<configentry name="MaxGrowthRate">
<value type="Float">5.0</value>
</configentry>
</configsection>
</configsection>
</configsection>
</configsection>
</configuration>
Parameter Reference
FlexFrame Agents Guide 268
7.5.5 Parameters for the Admin Pool Option
CtrlAgentRole
Allowed values are:
Normal Normal function of this CtrlAgent (don't work with a
caretaker of the admin pool).
WorkWithSparePool CtrlAgent work with a caretaker of the admin pool.
SparePoolName
Specifies the name of spare pool.
Allowed values are:
mySparePoolName Name of admin pool is mySparePoolName.
NONE No Admin Pool available.
If the own pool name is identical to the name of the admin pool, CtrlAgent of this pool
is the admin pool caretaker.
SpareGroupName
Specifies the name of the group in the admin pool.
Allowed values are:
mySpareGroupName Name of the group in admin pool is mySpareGroupName.
NONE No group in admin pool available.
SpareNodeRequestPath
Specifies the path for spare node requests.
MaxAgeResponseFile
Specifies the max age (in seconds) of response file (used by the AppAgent).
SpareRequestMoveTimeout
Timeout (in seconds) for moving an Application Node from the admin pool to work
pool (fa_move_node.sh -> ff_an_adm.pl).
SpareRequestMoveNodeMaxRetry
Max retries to call move-node script in case of error (LDAP-locks).
SpareRequestMoveNodeRetryDelay
Delay-Time (in seconds) between retries the call of move-node sript.
SpareNodePowercycleTime
Cycle time (in minutes) how often the spare node in spare-pool will be switches ON
and OFF.
Values <= 1 are forbidden and switch off this functionality.
SelectionRuleConfigFile
Path name and file name of the config file for he selection of the spare nodes from
the admin pool.
Parameter Reference
269 FlexFrame Agents Guide
7.5.6 Parameters for ServicePing
ServicePingScriptFilePath
This parameter specifies the path of the ServicePing scripts. ServicePing scripts see
section 3.11.
CheckCycle_of_ServicePing
This parameter defines the check cycle (in seconds) of the ServicePing functionality.
7.5.7 Node-related Parameters
Node_MaxRebootNumber
Specifies how many consecutive reboots may be performed to restore a service. If 3
is specified, the Application Agent attempts to make the system available again with
up to three reboots. Here you must bear in mind that a reboot is also unsuccsessful if the system could not be restored within the MaxRebootTime set. In the event of re-
boot problems, the MaxRebootTime parameter must therefore also always be
checked and compared with the reboot time actually needed.
Node_MaxSwitchOverNumber
Specifies how many consecutive switchovers may be performed to restore a service.
Node_SwitchOverServiceStartDelayTime
After an internal SwitchOver this parameter defines how long will be waited until the
services to be taken over are started on the node that is taking over control. This val-
ue is needed since, depending on the switches used, the virtual IP addresses cannot
be released directly owing to the devices’ internal caching. This delay time must
therefore be greater than the devices’ caching time otherwise the service startup on
the node which is taking over control will fail.
Node_SendTrapsAllowed
Releases or blocks the sending of node traps.
Node_RebootCommand
Specifies which command is executed when the Application Agent initiates a reboot.
Normally this is a shutdown with a subsequent reboot.
Node_ShutdownCommand
Specifies which command is executed when the Application Agent initiates a switch-
over. Normally this is a shutdown without a subsequent reboot.
Node_PowerDownCommand
Specifies which command is executed by the Control Agent before an external
SwitchOver is initiated. In this way it is ensured that the services on the node being
switched over are really stopped and can be taken over without any problem by other
nodes. The Control Agent waits at most for the period specified with the
Parameter Reference
FlexFrame Agents Guide 270
ShutdownFac_Shut_Exec_Timeout parameter before it continues with the
switchover.
Node_CheckAvailabilityCommand
Specifies which command is executed by the Control Agent to check the availability
of a node. A return value of 0 is interpreted as a positive result, every other return
value as negative. The Control Agent waits at most for the period of 5 seconds. If the
command has not been executed completely by then, it is assumed that the test is
negative, i.e. the node is no longer available, resulting in an external SwitchOver.
Node_RemoteExecutionCommand
Specifies which command the Control Agent puts ahead of a command to be execut-
ed on another node. This is used, for example, to start or stop a service remotely on
an Application Node. Usually ssh is used here.
SwitchOver_ext_Unavailability_check
Specifies which unavailability check will be performed before external SwitchOver.
Possible values are:
PING_and_SSH ping and ssh must fail for Poff (default)
PING_or_SSH ping or ssh must fail for Poff
PING_only ping only must fail for Poff
SSH_only ssh only must fail for Poff
SwitchOver_ext_Unavailability_check_PING
Specifies which ping unavailability check will be performed before external Switch-
Over. Possible values are:
1 ping the normal host name (blade2) (default)
2 ping the server LAN host name (blade2-se)
4 ping the storage LAN host name (blade2-st)
3 ping normal and server LAN
5 ping normal and storage LAN
6 ping server-lan and storage LAN
7 ping normal and server LAN and storage LAN
7.5.8 Service-related Parameters
The following parameters can be set individually for each service type or for multiple
services simultaneously. This is also a result of the hierarchical structure of the parameter
file. In the parameter file there is also an option for configuring the values for the DB, CI
and APP services individually. The value of the default service is used for any value
which is not service-specific.
Service_EnableMonitoring
Service_SendTraps
Service_MaxRestartNumber
Service_TrapSendDelayTime
Parameter Reference
271 FlexFrame Agents Guide
Service_ReactionDelayTime
Service_MaxStartTime
Service_MaxStopTime
Service_PingVirtualServiceInterface
Service_ReactionScript
The dynamic behavior of the AppAgents and CtrlAgents depends very much on the val-
ues in the configuration file and the physical conditions. You must therefore check very
carefully that the relation between certain values is secure and application-oriented.
Service_EnableMonitoring
Defines whether monitoring is enabled or disabled for the service type in question.
Service_SendTraps
Releases or blocks the sending of service traps.
Service_MaxRestartNumber
Defines how many attempts are made to restart a failed service. This value can be
configured individually for each service type. The value is typically in the range 1 to
10. The value 0 means that no attempt is made to restart a failed service. If reboots
are permitted on the node, failure of a service leads directly to a reboot.
Service_TrapSendDelayTime
Defines the send delay time for the service traps.
Service_ReactionDelayTime
Interworks directly with CheckCycleTime. It can be set individually for each service
type. This time defines how long the triggering of a reaction is delayed after a failure
has been detected.
Examples:
CheckCycleTime = 10 sec; ServiceReactionDelayTime = 30 sec
In this example a failed service is detected in a cycle. However, the reaction only
takes place after 30 seconds. The failure must therefore have been identified as a
failure over at least three detection cycles. This allows you to prevent a detection er-
ror resulting in an incorrect reaction.
CheckCycleTime = 10 sec; ServiceReactionDelayTime = 0 sec
In this example the required reaction takes place immediately in the cycle in which
the problem was detected.
Service_MaxRestartTime
Defines the maximum time which may be required for a service type in the event of a
restart. If this time is exceeded, a second or nth attempt is made in accordance with
Service_MaxRestartNumber. Thus if too short a time is selected for the service
Parameter Reference
FlexFrame Agents Guide 272
to be monitored and the hardware used, i.e. the service requires longer to restart
than permitted by Service_MaxRestartTime, a problem situation is triggered in-
correctly.
Service_MaxStartTime
Defines how long a service may take to start up. If this time is exceeded, the Agent
interprets the service as not started and initiates further reactions.
Service_MaxStopTime
Defines how long a service may take to stop. If this time is exceeded, the Agent in-
terprets the service as not stopped and initiates appropriate reactions.
Service_PingVirtualServiceInterface
Defines whether the associated virtual FlexFrame service interface is pinged to de-
termine the availability of a service. If it is set to 0 the virtual LAN interfaces of the cli-
ent and server network are not queried. Interface availability then has no influence on
the status change of a service. Switching off this parameter causes a defined wait
time (which is parameter Node_SwitchOverServiceStartDelayTime) before
starting services in a takeover (switchover).
Service_ReactionScript
Defines the interace to the FF-SAP-Start-/Stop-Scripts.
<!-- SAP-scripts used before FF V5.0A like sapdb, sapci, sapapp, sapj,
sapacc, ... -->
<value
type="String">${ScriptPath}/${ScriptPrefix}${monparam}${ScriptPostfix}
${BYNIDENT} ${SIDENT} ${ACTION}</value>
<!-- SAP-script used from FF V5.0A (ff_service.sh) -->
<value type="String">${ScriptPath}/ff_service.sh -s ${SIDENT} -t
${monparam} ${BYKEYNIDENT} -a ${ACTION}</value>
7.5.9 Parameters for the Definition of a Generic Service
By default, the FlexFrame Agents can supervise a defined quantity of services, depend-
ing on the version. For these services, the rules for the detection and the rules for the
autonomous reaction are fixed components of the FlexFrame Agents. Generic services
are services which are not considered in the standard scope of the FlexFrame Agents but
which will also be considered within the context of supervision and of the autonomous
reaction.
This section describes the parameters that are used to enable monitoring by the Flex-
Frame Agents as well as the autonomous reactions.
In the parameter file myAMC_FA_Rules.xml, the necessary information for the detection
of the service is registered. Most of the parameters are optional, meaning they are not
normally needed for monitoring. In the following the mandatory parameters are indicated
by an asteriks (*). The optional parameters are set with default values if not used.
Parameter Reference
273 FlexFrame Agents Guide
The parameter file for the definition of generic services is organized in two main parts: the
parametering of the detection and of the reactions which are required for a service.
For the configuration of a generic service, this generic service must be described in the
detection section.The reaction is then defined in the second section.
Therefore, ideally, in each case one of the templates in the detection and reaction block
is copied. This must then be individualized with the specific parameters in the appropiate
places. The referencing between the detection and reaction blocks occurs via the defined
service name.
Attention: The templates are set to ‘not active’. Therefore, in principle, the individualiza-tion of the parameter Active must be changed to 1.
Furthermore, the standard service parameters (see section above) must be defined such
that they are also valid for built-in services. If service-specified parameters are not set,
the default values will be used.
Service_EnableMonitoring
Service_SendTraps
Service_MaxRestartNumber
Service_TrapSendDelayTime
Service_ReactionDelayTime
Service_MaxStartTime
Service_MaxStopTime
7.5.9.1 Parametering of the Service Detection
The parameter set for the detection consists of a header which defines the service as a
whole. A service itself can consist of one or more subservices. For every subservice a
separate subservice parameter block must be created.
7.5.9.1.1 Parameters of the service header
*Name
Unique symbolic servicename inside the rulefile. The maximum string length is 50
characters. It does not have to be identical with the technical service name. The ser-
vice name is the reference to the reaction.
Description
Used for documentation purposes only.
*Displayname
The display name is used by the FlexFrame ControlCenter. If it is not defined, the
Parameter Reference
FlexFrame Agents Guide 274
symbolic name is used, but it should be defined. It should be as short as possible be-
cause this service name will be displayed in the FlexFrame ControlCenter (limited
space available).
Active
Activates (1) or deactivates (0) the parameter block.
MonitorParam
Defines the required parameters that produce a state-altering event during starting or
stopping of a service. The event script is called MonitorFlag and is doing nothing.
It remains in the pro-cess list for a defined timeperiod (normally 30 seconds).To at-tach the MonitorFlag to a service, the following call syntax is prescribed:
MonitorFlag <RefSrv> <State> <InstNo> <SID>
RefSrv
This is a reference to a service and is defined under MonitorParam.
State
start, stop, restart, watch or nowatch
InstNo
A positive instance number; this information is optional
SID
The system id. This information is optional
Orderprio
When there are several services, the order priority defines within a system the order
in which they will be started. Stopping will be done in the reverse order.
PowerValue
Workload demands of a service in powervalue [SAPs-äquivalent].
ServicePrio
Priority of the service to be defined.
*Group
Accounting group name for the performance and accounting management. A new
group can be defined for a service or its values appear in an existing group.
For example, SAP: If the backup server should be monitored and its workload con-
sumption should appear in the group SAP, then it should be configured to the group
SAP.
7.5.9.1.2 Subservice parametering
A service consists of a minimum of one or more subservices. For every subservice the
following parameters must be defined:
Parameter Reference
275 FlexFrame Agents Guide
Subservice
Subservice name
Symbolic name for the subservice. This parameter is optional.
Display name
This parameter is for example used as service name in failure traps or error messag-
es.
Active
Active (1) or inactive (0). The default is active.
7.5.9.1.3 Subservice detector parametering
For every subservice one or more detectors can be activated. In version 3.0 of the Flex-
Frame Agents, only detectors of the type process exist. In future versions more detectors
will be added. A detector requires the following parameters:
Detector
Detector type
Process
Active
Switches the rule on or off
ProcessName
The process name to be detected
CountMin
Minimum number of required processes of this type
CountMax
Maximum number of required processes of this type
HierachyMin
Minimum process hierachy of the subservices
Severity
Determined Severity (warning or critical) if the affiliated subservice has a fault.
7.5.9.2 Parametering of the Service Reactions
The second big block to be defined is the reactions required for a service. For this the
commands for starting, stopping and restarting the services must be defined.
Parameter Reference
FlexFrame Agents Guide 276
The reaction parameter block for a service also has the service name, meaning the same
name. In a reaction the determining of a program or script to be called occurs in the case
of each start, stop or restart of the services.
The first command specifies the start call, the second the stop call and the third the re-
start call.
Each command is composed of the attributes script and parameter. script defines
the called program or script. If a parameter is omitted, no parameter is used.
Parameter Reference
277 FlexFrame Agents Guide
7.5.10 Path Configuration
The path configuration is used to define the directories in which the FlexFrame Agents
Suite components store their various work files. A FlexFrame Agents Suite solution stores
a range of information, such as files with display information for the WebInterface and
logging information to be used for support when this is required, in various files.
To ensure performance and clarity are retained even in larger configurations, we
recommend that you do not modify these settings! If the suggested path configura-
tion is changed, though, make sure that clatiy is still retained and no problems
arise with regard to performance and accessibility.
LiveListLogFilePath
This parameter specifies the directory in which the Livelist is stored.
LiveListXmlFilePath
This parameter specifies the directory in which the XML representation of the Livelist
is stored. This file is required by the FlexFrame Agents WebInterface. The parameter
should contain the same path as ServicesXmlFilePath.
ServicesXmlFilePath
This parameter specifies the directory in which the XML representation of the ser-
vices list is stored. These files are required by the FlexFrame Agents WebInterface.
The parameter should contain the same path as LiveListXmlFilePath.
ServicesListFilePath
This parameter specifies the directory in which the services list files are stored.
ServicesLogFilePath
This parameter specifies the directory in which the services log files are stored.
RebootListFilePath
This parameter specifies the directory in which the reboot files are stored. These files
contain a list of all services which must be restored after a reboot.
SwitchOverListFilePath
This parameter specifies the directory in which the switchover files are stored. These
files contain a list of all services which must be restored on another node after a
switchover.
PerformanceFilePath
This parameter specifies the directory in which the performance files are stored.
These files contain measured values for performance data.
FSUtilScriptFilePath
Specifies the path to the FsUtil-scripts.
Script-file names are 'myAMC.DetGetDirData.sh', 'myAMC.DetGetFSData.sh'.
Performance files 'Perf_FsUtil~<hostname>.prf'.
Line-Identifier in this file is 'FSUTIL'.
Parameter Reference
FlexFrame Agents Guide 278
SAPScriptFilePath
This parameter specifies the directory in which the start and stop scripts for the SAP
services (sapdb, sapci, sapapp, etc.) can be found. The default path
(/opt/myAMC/scripts/sap) is normally a symbolic link to the actual script directo-
ry.
ControlFilePath
This parameter specifies the directory in which the control files (<service
type><service id><service sid>_host) generated by the start/stop scripts
are contained.
BlackboardFilePath
This parameter specifies the directory in which the BlackBoard file can be found.
Commands can be entered in it which are executed by the FlexFrame AppAgents.
GroupConfigFile
This parameter specifies the file in which the group affiliation is configured.
PrePoffHookPath
This parameter specifies the script, which is execute before powering off a node. If
IgnorePoffHookResult = true, the return code will be ignored, otherwise the
node will be powered off only if this script returns 0.
PostPoffHookPath
This parameter specifies the script, which is execute after powering off a node and
before performing a SwitchOver. If IgnorePoffHookResult = true, the return
code will be ignored, otherwise the SwitchOver will be performed only if this script re-
turns 0.
ServicePingScriptFilePath
This parameter specifies the path of the serviceping scripts. ServicePing scripts see
section 3.11.
Parameter Reference
279 FlexFrame Agents Guide
7.5.11 Shutdown Configuration
The shutdown feature is described in detail in section 6.4.
IgnoreShutdownFailure
The parameter IgnoreShutdownFailure defines whether, after a failed shut-
down/powerdown, the network interfaces of the relevant node are to be deactivated,
which would ensure that all services running on this node are shut down and the ser-
vices can be switched over to an other Application Node.
Shut_DiagTrapRepeat_valid
Time in seconds to resend the „shutdown validity is valid“ – trap.
(‚0‘ == no resend)
Shut_DiagTrapRepeat_invalid
Time in seconds to resend the „shutdown validity is invalid“ – trap.
(‚0‘ == no resend)
Parameter Reference
FlexFrame Agents Guide 280
7.5.12 Default Parameter File
<?xml version="1.0" encoding="ISO-8859-1"?>
<!-- @@@MYAMC.VERSION@@@ 9.0-23 -->
<configuration>
<configsection name="myAMC.FA">
<!-- *** timing parameters *** -->
<!-- specifies how often (in seconds) myAMC.FA checks process states.
This value should not exceed a third of the time interval specified
in the script monitor-alert (which is part of the FlexFrame installation).
The default value for this parameter is 10 seconds (Note: 'monitor_alert'
is 30).
-->
<configentry name="CheckCycleTime">
<value type="UnsignedInteger">10</value>
</configentry>
<!-- Specifies how often (in seconds) myAMC.FA checks the BlackBoard.
This value should be >= 'CheckCycleTime'.
A value of '0' disables the reading.
-->
<configentry name="BlackBoardCheckCycleTime">
<value type="UnsignedInteger">20</value>
</configentry>
<!-- Specifies that myAMC.FA checks the persistent FA-Info file.
This value should be > '0' to enable reading.
A value of '0' disables the reading.
-->
<configentry name="PersistFAInfoCheckCycleTime">
<value type="UnsignedInteger">30</value>
</configentry>
<!-- specifies how often (in seconds) myAMC.FA writes live list entries
This value should not exceed a third of the time interval specified by the
parameter MaxHeartbeatTime.
-->
<configentry name="LivelistWriterTime">
<value type="UnsignedInteger">10</value>
</configentry>
<!-- specifies the dT (in sec) for reading in past in case of a CtrlAgent
switch.
For activation set this value for example to 600 sec.
If this value is '0' this feature is switched off.
-->
<configentry name="LivelistRead_dT">
<value type="UnsignedInteger">0</value>
</configentry>
<!-- specifies how often (in seconds) myAMC.FA tries a service ping.
This value should be >= 60 sec because of performance consumption.
-->
<configentry name="CheckCycle_of_ServicePing">
<value type="UnsignedInteger">60</value>
</configentry>
<!-- specifies whether ServicePing is active or not. -->
<!--
<configentry name="ServicePing_is_active">
<value type="Boolean">false</value>
</configentry>
-->
Parameter Reference
281 FlexFrame Agents Guide
<!-- specifies whether CtrlAgent writes only to files if there was changes.
'true': Output files will be written if there was changes.
'false': Output files will be written every cycle.
-->
<configentry name="CtrlAgent_WriteOnlyChanges">
<value type="Boolean">false</value>
</configentry>
<!-- Enable the usage of virtual cold standby sparenodes.
'true': Virtual cold standby sparenodes are used before pool
independent spare.
'false': Virtual cold standby sparenodes are not used before pool
independent spare.
-->
<configentry name="VirtualColdStandbySpareNode_use">
<value type="Boolean">false</value>
</configentry>
<!-- Specify the role of CtrlAgent.
Allowed values are:
= 'Normal': Normal function of this Ctrl-Agent. (don't
work with a caretaker of the 'SparePool').
= 'WorkWithSparePool': Ctrl-Agent work with a caretaker of the
'SparePool'.
-->
<configentry name="CtrlAgentRole">
<!--
<value type="String">WorkWithSparePool</value>
-->
<value type="String">Normal</value>
</configentry>
<!-- Specify the name of 'SparePool'.
Allowed values are:
= '*': Name of the 'SparePool'.
= 'NONE': No 'SparePool' available.
-->
<configentry name="SparePoolName">
<value type="String">NONE</value>
</configentry>
<!-- Specify the name of 'SpareGroup'.
Allowed values are:
= '*': Name of the 'SpareGroup'.
= 'NONE': No 'SpareGroup' available.
-->
<configentry name="SpareGroupName">
<value type="String">SpareGrp</value>
</configentry>
<!-- Values for 'fa_move_node.sh' calls 'ff_an_adm.pl -op move ...' -->
<!-- Timeout (in seconds) for move a node from SparePool to work pool. -->
<configentry name="SpareRequestMoveTimeout">
<value type="UnsignedInteger">900</value>
</configentry>
<!-- Max retries to call move-node script in case of error (LDAP-locks). -->
<configentry name="SpareRequestMoveNodeMaxRetry">
<value type="UnsignedInteger">10</value>
</configentry>
<!-- Delay-Time (in seconds) between retries the call of move-node sript. -->
<configentry name="SpareRequestMoveNodeRetryDelay">
<value type="UnsignedInteger">30</value>
</configentry>
<!-- Time (in minutes) how often the spare nodes in caretaker will be powered
ON and OFF.
Parameter Reference
FlexFrame Agents Guide 282
Values <= 1 (60sec) are not allowed and switched of this functionality.
-->
<configentry name="SpareNodePowercycleTime">
<value type="UnsignedInteger">1440</value>
</configentry>
<!-- Timeout (in seconds) for NASCheck_Script.
-->
<configentry name="SRDF_NAS_FH_NASCheck_Timeout">
<value type="UnsignedInteger">1800</value>
</configentry>
<!-- Maximal repair time of a SRDF-NAS failure.
Used in 'manual' and 'automatic' case.
If a potential storage problem persits, a new check will be done
after exceeding this time.
-->
<configentry name="SRDF_NAS_FH_MaxSRDF_RepairTime">
<value type="UnsignedInteger">1800</value>
</configentry>
<!-- Specify the trigger for the SRDF-NAS-Check.
Allowed values are:
= 'NO_Check':
NO SRDF-NAS-Check will be done.
= 'RequestMA_only':
'Request per Monitor-Alert' only triggers the SRDF-NAS-Check.
= 'MountFail_only':
'Mount-Failed' only triggers the SRDF-NAS-Check.
= 'FilerFail_only':
'Filer-Failed' only triggers the SRDF-NAS-Check.
= 'MountFail_and_RequestMA': 'Mount-
Failed' and 'Request per Monitor-Alert' triggers the SRDF-NAS-Check.
= 'MountFail_or_RequestMA':
'Mount-Failed' or 'Request per Monitor-Alert' triggers the SRDF-NAS-Check.
= 'FilerFail_and_RequestMA': 'Filer-
Failed' and 'Request per Monitor-Alert' triggers the SRDF-NAS-Check.
= 'FilerFail_or_RequestMA':
'Filer-Failed' or 'Request per Monitor-Alert' triggers the SRDF-NAS-Check.
= 'MountFail_and_FilerFail': 'Mount-
Failed' and 'Filer-Failed' triggers the SRDF-NAS-Check.
= 'MountFail_or_FilerFail':
'Mount-Failed' or 'Filer-Failed' triggers the SRDF-NAS-Check.
= 'FilerFail_and_MountFail_and_RequestMA': 'Filer-Failed' and
'Mount-Failed' and 'Request per Monitor-Alert' triggers the SRDF-NAS-Check.
= 'FilerFail_or_MountFail_or_RequestMA': 'Filer-Failed' or
'Mount-Failed' or 'Request per Monitor-Alert' triggers the SRDF-NAS-Check.
-->
<configentry name="SRDF_NAS_FH_CheckTrigger">
<!--
<value type="String">NO_Check</value>
<value type="String">RequestMA_only</value>
<value type="String">MountFail_only</value>
<value type="String">FilerFail_only</value>
<value type="String">MountFail_and_RequestMA</value>
<value type="String">MountFail_or_RequestMA</value>
<value type="String">FilerFail_and_RequestMA</value>
<value type="String">FilerFail_or_RequestMA</value>
<value type="String">MountFail_and_FilerFail</value>
<value type="String">MountFail_or_FilerFail</value>
<value type="String">FilerFail_and_MountFail_and_RequestMA</value>
<value type="String">FilerFail_or_MountFail_or_RequestMA</value>
-->
<value type="String">NO_Check</value>
</configentry>
<!-- Specify the SRDF NAS failure reaction.
Allowed values are:
= 'manual': Manual SRDF switchover. Administrator must do it.
= 'automatic': Automatic SRDF switchover.
-->
<configentry name="SRDF_NAS_FH_Reaction">
Parameter Reference
283 FlexFrame Agents Guide
<!--
<value type="String">manual</value>
<value type="String">automatic</value>
-->
<value type="String">manual</value>
</configentry>
<!-- specifies how often (in seconds) myAMC.FA looks for new live list
entries.
This value should not exceed a third of the time interval specified by the
parameter MaxHeartbeatTime.
-->
<configentry name="ControlAgentTime">
<value type="UnsignedInteger">10</value>
</configentry>
<!-- Maximum time (in seconds) between two application agent heartbeat
messages in live list -->
<configentry name="MaxHeartbeatTime">
<value type="UnsignedInteger">60</value>
</configentry>
<!-- Maximum time (in seconds) between two application agent heartbeat
messages in live list,
used for checking quick starting AN's (VM's).
A value of '0' will disable this feature.
-->
<configentry name="MaxHeartbeatQuickStartTime">
<value type="UnsignedInteger">10</value>
</configentry>
<!-- Parameters for 'MultiNodeFailure_ShortTime' -->
<!-- Number of nodes failed for detection of 'MultiNodeFailure_ShortTime'. -->
<configentry name="MultiNodeFailure_ShortTime_FailureCount">
<value type="UnsignedInteger">4</value>
</configentry>
<!-- Timeframe for detection of 'MultiNodeFailure_ShortTime'. -->
<configentry name="MultiNodeFailure_ShortTime_FailureTime">
<value type="UnsignedInteger">120</value>
</configentry>
<!-- Maximum time (in seconds) witch is added as delay (to 'MaxHeartbeatTime')
in case of 'MultiNodeFailure_ShortTime'. -->
<configentry name="MultiNodeFailure_ShortTime_ReactionDelay">
<value type="UnsignedInteger">0</value>
</configentry>
<!-- Specify which reaction in case of 'MultiNodeFailure_ShortTime'
Allowed values are:
= 'None': No special reaction jet (will be used for future use).
-->
<configentry name="MultiNodeFailure_ShortTime_ReactionAction">
<!--
<value type="String">ffu</value>
-->
<value type="String">None</value>
</configentry>
<!-- Parameters for 'MultiNodeFailure_LongTime' -->
<!-- Number of nodes failed for detection of 'MultiNodeFailure_LongTime'. -->
<configentry name="MultiNodeFailure_LongTime_FailureCount">
<value type="UnsignedInteger">10</value>
</configentry>
<!-- Timeframe for detection of 'MultiNodeFailure_LongTime'. -->
<configentry name="MultiNodeFailure_LongTime_FailureTime">
<value type="UnsignedInteger">86400</value>
</configentry>
<!-- Maximum time (in seconds) witch is added as delay (to 'MaxHeartbeatTime')
in case of 'MultiNodeFailure_LongTime'. -->
<configentry name="MultiNodeFailure_LongTime_ReactionDelay">
<value type="UnsignedInteger">0</value>
</configentry>
<!-- Specify which reaction in case of 'MultiNodeFailure_LongTime'
Parameter Reference
FlexFrame Agents Guide 284
Allowed values are:
= 'None': No special reaction jet (will be used for future use).
-->
<configentry name="MultiNodeFailure_LongTime_ReactionAction">
<!--
<value type="String">ffu</value>
-->
<value type="String">None</value>
</configentry>
<!-- Parameters for 'MultiNodeFailure_Heartbeat'. Used for initiate a SRDF-
NAS-Check. -->
<!-- Number of nodes failed for detection of 'MultiNodeFailure_Heartbeat'. -->
<configentry name="MultiNodeFailure_Heartbeat_FailureCount">
<value type="UnsignedInteger">4</value>
</configentry>
<!-- Timeframe for detection of 'MultiNodeFailure_Heartbeat'. -->
<configentry name="MultiNodeFailure_Heartbeat_FailureTime">
<value type="UnsignedInteger">600</value>
</configentry>
<!-- Maximum time (in seconds) witch is added as delay (to 'MaxHeartbeatTime')
in case of 'MultiNodeFailure_Heartbeat'. -->
<configentry name="MultiNodeFailure_Heartbeat_ReactionDelay">
<value type="UnsignedInteger">0</value>
</configentry>
<!-- Specify which reaction in case of 'MultiNodeFailure_Heartbeat'
Allowed values are:
= 'None': No special reaction jet (will be used for future use).
-->
<configentry name="MultiNodeFailure_Heartbeat_ReactionAction">
<!--
<value type="String">ffu</value>
-->
<value type="String">None</value>
</configentry>
<!-- Maximum time (in seconds) a machine takes to reboot -->
<configentry name="MaxRebootTime">
<value type="UnsignedInteger">1200</value>
</configentry>
<!-- Maximum time (in seconds) a machine takes to reboot (special for windows)
-->
<configentry name="MaxUpTime_for_Reboot_detection">
<value type="UnsignedInteger">900</value>
</configentry>
<!-- Maximum number of failed reach attempts -->
<configentry name="MaxFailedReachNumber">
<value type="UnsignedInteger">1</value>
</configentry>
<!-- Specify which unavailability check will be done before external
SwitchOver
Allowed values are:
= 'PING_and_SSH': 'ping' and 'ssh' must fail for Poff (= default)
= 'PING_or_SSH': 'ping' or 'ssh' must fail for Poff
= 'PING_only': 'ping' only must fail for Poff
= 'SSH_only': 'ssh' only must fail for Poff
-->
<configentry name="SwitchOver_ext_Unavailability_check">
<!--
<value type="String">PING_and_SSH</value>
<value type="String">PING_or_SSH</value>
<value type="String">PING_only</value>
<value type="String">SSH_only</value>
-->
<value type="String">PING_and_SSH</value>
</configentry>
Parameter Reference
285 FlexFrame Agents Guide
<!-- Specify which PING unavailability check will be done before external
SwitchOver
Allowed values are:
= '1': 'ping' the normal hostname ('blade2')
= '2': 'ping' the server-lan hostname ('blade2-se')
= '4': 'ping' the storage-lan hostname ('blade2-st')
= '3': 'ping' normal and server-lan
= '5': 'ping' normal and storage-lan
= '6': 'ping' server-lan and storage-lan
= '7': 'ping' normal and server-lan and storage-lan
-->
<configentry name="SwitchOver_ext_Unavailability_check_PING">
<value type="UnsignedInteger">1</value>
</configentry>
<!-- specifies how often (in seconds) myAMC.FA writes performance data.
This value should be a multiple of 'CheckCycleTime'.
The default value for this parameter is 60 seconds.
-->
<configentry name="PerfdataReportCycleTime">
<value type="UnsignedInteger">600</value>
</configentry>
<!-- specifies how often (in seconds) myAMC.FA checks the filesystem
utilization for performance data.
This value should be a multiple of 'CheckCycleTime'.
-->
<configentry name="Perfdata_SpaceUtil_FSCycleTime">
<value type="UnsignedInteger">1800</value>
</configentry>
<!-- specifies how often (in seconds) myAMC.FA checks the directory
utilization for performance data.
This value should be a multiple of 'CheckCycleTime'.
-->
<configentry name="Perfdata_SpaceUtil_DirCycleTime">
<value type="UnsignedInteger">3600</value>
</configentry>
<!-- specifies the max age (in seconds) of SwitchOver file.
If the age of a SwitchOver file exceeds this value, it is markes as 'too
old' and will be ignored.
-->
<configentry name="MaxAgeSwitchOverFile">
<value type="UnsignedInteger">240</value>
</configentry>
<!-- specifies the max age (in seconds) of Response file.
If the age of a Response file exceeds this value, it will be ignored.
-->
<configentry name="MaxAgeResponseFile">
<value type="UnsignedInteger">1200</value>
</configentry>
<!-- specifies the communication method(s) of FA -->
<!-- Allowed values are:
= 'File-based':
Use file livelist.log for communication.
= 'MPI-based':
Use message pasing interface MPI_AL for communication.
= 'MPI-fallback-File':
Use message pasing interface MPI_AL for communication and as
fallback the file based communication.
-->
<configentry name="ComMethod_LiveList">
<value type="String">File-based</value>
<!--
<value type="String">File-based</value>
<value type="String">MPI-based</value>
<value type="String">MPI-fallback-File</value>
-->
Parameter Reference
FlexFrame Agents Guide 286
</configentry>
<!-- *** node parameters *** -->
<configsection name="Node">
<!-- specifies the minimal service prio for take over a SwitchOver-
file -->
<!--
<configentry name="Node_MinServicePrio">
<value type="UnsignedInteger">1</value>
</configentry>
-->
<!-- specifies the 'service schema' with controls 'priority' and
'load' -->
<!-- See file 'myAMC_FA_Groups.xml'
-->
<configentry name="Node_ServiceSchema">
<value type="String">default</value>
</configentry>
<!-- specifies the 'group schema' with controls grouping.
-->
<!-- See file 'myAMC_FA_Groups.xml'
-->
<configentry name="Node_GroupSchema">
<value type="String">default_LDAP</value>
</configentry>
<!-- specifies the interface used for start/stop/restart SAP-
services -->
<configentry name="Node_SAP_Interface">
<value type="String">SAPscripts</value>
<!--
<value type="String">ACC</value>
-->
<!--
<value type="String">SAPscripts, ACC</value>
-->
</configentry>
<!-- specifies the replace strings for generation of full specified
script file path (DO NOT CHANGE) -->
<configentry name="Node_SAP_Interface_ReplaceSTR_ScriptPath">
<value type="String">${ScriptPath}</value>
</configentry>
<configentry name="Node_SAP_Interface_ReplaceSTR_ScriptPrefix">
<value type="String">${ScriptPrefix}</value>
</configentry>
<configentry name="Node_SAP_Interface_ReplaceSTR_ScriptPostfix">
<value type="String">${ScriptPostfix}</value>
</configentry>
<!-- specifies the replace string for
'Node_SAP_Interface_ReplaceSTR_ScriptPrefix' -->
<configentry name="Node_SAP_Interface_ScriptPrefix_SAP">
<value type="String">sap</value>
</configentry>
<configentry name="Node_SAP_Interface_ScriptPrefix_ACC">
<value type="String">acc</value>
</configentry>
<!-- specifies the replace string for
'Node_SAP_Interface_ReplaceSTR_ScriptPostfix' -->
<configentry name="Node_SAP_Interface_ScriptPostfix_SAP">
<value type="String"></value>
</configentry>
<configentry name="Node_SAP_Interface_ScriptPostfix_ACC">
<value type="String"></value>
</configentry>
<!-- specifies whether SwitchOver is service based or node based -->
<!-- Allowed values are 'service' and 'node' --
>
Parameter Reference
287 FlexFrame Agents Guide
<configentry name="Node_SwitchOverTyp">
<!--
<value type="String">service</value>
-->
<value type="String">node</value>
</configentry>
<!-- specifies the rule to take over a SwitchOver-file -->
<!-- Allowed values are:
= 'SpareNode':
Only a spare node will Start the services from taken
SwitchOver-File.
= 'add' (Hinzufuegen, Ergaenzung):
Additional start the services from taken SwitchOver-File.
= 'replace' (Verschieben):
Running services will be stopped and a SwitchOver-File
will be written therefore,
Start the services from taken SwitchOver-File.
= 'substitute' (Ersetzung):
Running services will be stopped,
Start the services from taken SwitchOver-File.
= 'dynamic':
Cause of prios there will be done one of the TakeOver-
rules
'SpareNode', 'add', 'replace', 'substitute', or none.
-->
<configentry name="Node_TakeOverRule">
<!--
<value type="String">add</value>
<value type="String">replace</value>
<value type="String">substitute</value>
<value type="String">dynamic</value>
-->
<value type="String">SpareNode</value>
</configentry>
<!-- specifies the ranges for take over rule 'dynamic'.
For 'Dyn_Spare_*' it is the high prio of SwitchOver-file.
For others it is the high prio of own node.
The lowest Service-priority you may use is '9999' (As
default a SpareNode has a prio of '10000').
-->
<!-- 'Spare': -->
<configentry name="Dyn_Spare_min"> <value
type="UnsignedInteger">1</value> </configentry>
<configentry name="Dyn_Spare_max"> <value
type="UnsignedInteger">4</value> </configentry>
<!-- 'Add' (Hinzufuegen, Ergaenzen): -->
<configentry name="Dyn_Add_min"> <value
type="UnsignedInteger">3</value> </configentry>
<configentry name="Dyn_Add_max"> <value
type="UnsignedInteger">4</value> </configentry>
<!-- 'Replace' (Verschieben): -->
<configentry name="Dyn_Replace_min"> <value
type="UnsignedInteger">5</value> </configentry>
<configentry name="Dyn_Replace_max"> <value
type="UnsignedInteger">6</value> </configentry>
<!-- 'Substitute' (Ersetzen): -->
<configentry name="Dyn_Substitute_min"> <value
type="UnsignedInteger">7</value> </configentry>
<configentry name="Dyn_Substitute_max"> <value
type="UnsignedInteger">20</value> </configentry>
<!-- specifies whether the min- max- range of spare node is
exclusive for spare nodes or not. -->
<configentry name="Dyn_Spare_exclusive">
<value type="Boolean">true</value>
</configentry>
Parameter Reference
FlexFrame Agents Guide 288
<!-- specifies the escalation type -->
<!-- Allowed values are:
= 'Node':
Node-escalation: "Restart" => "Reboot" =>
"SwitchOver"
The escalation will be done for all services on
a node.
= 'Service':
Service-escalation: "Restart" =>
"SwitchOver"(single service)
The escalation will be done for a single
service.
-->
<configentry name="EscalationType">
<!--
<value type="String">Service</value>
-->
<value type="String">Node</value>
</configentry>
<!-- specifies the TakeOver strategy -->
<!-- Allowed values are:
= 'FirstFit':
The first node, who applies for TakeOver, wins
and gets the SwitchOver-File.
= 'LowPrioFit':
The node with the minimal prio, wins and gets
the SwitchOver-File.
The lowest Service-priority you may use is
'9999' (As default a SpareNode has a prio of '10000').
-->
<configentry name="TakeOverStrategy">
<value type="String">FirstFit</value>
<!--
<value type="String">FirstFit</value>
<value type="String">LowPrioFit</value>
-->
</configentry>
<!-- specifies the maximum number of reboots -->
<configentry name="Node_MaxRebootNumber">
<value type="UnsignedInteger">0</value>
</configentry>
<!-- specifies the maximum number of switch overs -->
<configentry name="Node_MaxSwitchOverNumber">
<value type="UnsignedInteger">1</value>
</configentry>
<!-- if Service_PingVirtualServiceInterface is set to true,
this value specifies the time window, in which a node
tries
to take over a service.
if Service_PingVirtualServiceInterface is set to false,
the service will be started after this time interval to
ensure, that is is actually down.
-->
<configentry name="Node_SwitchOverServiceStartDelayTime">
<value type="UnsignedInteger">120</value>
</configentry>
<!-- specifies whether traps are allowed -->
<configentry name="Node_SendTrapsAllowed">
<value type="Boolean">true</value>
</configentry>
<!-- specifies the command to be executed apon reboot -->
<configentry name="Node_RebootCommand">
Parameter Reference
289 FlexFrame Agents Guide
<value
type="String">/opt/myAMC/scripts/ShutDown_Node/shutdown_node.sh reboot</value>
</configentry>
<!-- specifies the command to be executed apon shutdown -->
<configentry name="Node_ShutdownCommand">
<value
type="String">/opt/myAMC/scripts/ShutDown_Node/shutdown_node.sh shutdown</value>
</configentry>
<!-- specifies the command to be executed to power down a node
before
another node is allowed to take over its services. The
variable
${node-name} can be used to specifiy the name of the node
the actual
command should be executed on. The actual command will
be appended to this command -->
<configentry name="Node_PowerDownCommand">
<!--
<value type="String">/bin/true</value>
-->
<value type="String">/bin/su - root -c
"/opt/myAMC/scripts/PowerMng/powerOnOff.sh ${node-name} down"</value>
</configentry>
<!-- specifies the command to be executed to in order to
determine whether a node is still available -->
<configentry name="Node_CheckAvailabilityCommand">
<value type="String">/usr/bin/ssh -q ${node-name}
/bin/uname -a</value>
</configentry>
<!-- specifies the command to be used when a command is to be
executed on another node. The variable ${node-name} can
be used to specifiy the name of the node the actual
command should be executed on. The actual command will
be appended to this command -->
<configentry name="Node_RemoteExecutionCommand">
<value type="String">/usr/bin/ssh -q root@${node-name}
</value>
</configentry>
<!-- specifies the command to be executed for a stonith-responce -->
<configentry name="Node_HANAStonithResponceCommand">
<value
type="String">/FlexFrame/scripts/HanaConnectorApiNAS/ff_read_agents_reply.sh --opcode reply --
requested-from ${Request-Node} --host-to-shutdown ${Shutdown-Node} --action-id ${Action-ID} --
state ${State}</value>
</configentry>
</configsection>
<!-- *** service parameters *** -->
<configsection name="Services">
<configsection name="Default">
<!-- specifies whether to monitor a service -->
<configentry name="Service_EnableMonitoring">
<value type="Boolean">true</value>
</configentry>
<!-- specifies whether traps are allowed -->
<configentry name="Service_SendTraps">
<value type="Boolean">true</value>
</configentry>
<!-- specifies the maximum number of restarts -->
<configentry name="Service_MaxRestartNumber">
<value type="UnsignedInteger">3</value>
</configentry>
<!-- specifies how long (in seconds) to delay a trap in
case of an error -->
Parameter Reference
FlexFrame Agents Guide 290
<configentry name="Service_TrapSendDelayTime">
<value type="UnsignedInteger">15</value>
</configentry>
<!-- specifies how long (in seconds) to delay a reaction
in case of an error
This value should be at least three times as
high as the time interval
specified in CheckCycleTime.
-->
<configentry name="Service_ReactionDelayTime">
<value type="UnsignedInteger">45</value>
</configentry>
<!-- specifies maximum service restart time (in seconds) -
->
<configentry name="Service_MaxRestartTime">
<value type="UnsignedInteger">600</value>
</configentry>
<!-- specifies maximum service start time (in seconds) -->
<configentry name="Service_MaxStartTime">
<value type="UnsignedInteger">300</value>
</configentry>
<!-- specifies maximum service stop time (in seconds) -->
<configentry name="Service_MaxStopTime">
<value type="UnsignedInteger">300</value>
</configentry>
<!-- specifies whether to ping virtual service interface
in order to detect whether a service is up and
running. -->
<configentry name="Service_PingVirtualServiceInterface">
<value type="Boolean">true</value>
</configentry>
<!-- specifies name and parameter of reaction script. -->
<configentry name="Service_ReactionScript">
<!-- SAP-scripts used before FF V5.0 like sapdb,
sapci, sapapp, sapj, sapacc, ...
<value
type="String">${ScriptPath}/${ScriptPrefix}${monparam}${ScriptPostfix} ${BYNIDENT} ${SIDENT}
${ACTION}</value>
-->
<!-- SAP-script used from FF V5.0
(ff_service.sh) -->
<value type="String">${ScriptPath}/ff_service.sh
-s ${SIDENT} -t ${monparam} ${BYKEYNIDENT} -a ${ACTION}</value>
</configentry>
</configsection>
<configsection name="SRV_ORACLE_RAC_CRS">
<!-- specifies the maximum number of restarts -->
<configentry name="Service_MaxRestartNumber">
<value type="UnsignedInteger">100000000</value>
</configentry>
<configentry name="Service_ReactionDelayTime">
<value type="UnsignedInteger">100000000</value>
</configentry>
<!-- specifies maximum service restart time (in seconds) -
->
<configentry name="Service_MaxRestartTime">
<value type="UnsignedInteger">100000000</value>
</configentry>
<!-- specifies maximum service start time (in seconds) -->
<configentry name="Service_MaxStartTime">
<value type="UnsignedInteger">100000000</value>
</configentry>
<!-- specifies maximum service stop time (in seconds) -->
<configentry name="Service_MaxStopTime">
<value type="UnsignedInteger">100000000</value>
</configentry>
</configsection>
<configsection name="SRV_ORACLE_RAC_DB">
<!-- specifies the maximum number of restarts -->
Parameter Reference
291 FlexFrame Agents Guide
<configentry name="Service_MaxRestartNumber">
<value type="UnsignedInteger">100000000</value>
</configentry>
<configentry name="Service_ReactionDelayTime">
<value type="UnsignedInteger">100000000</value>
</configentry>
<!-- specifies maximum service restart time (in seconds) -
->
<configentry name="Service_MaxRestartTime">
<value type="UnsignedInteger">100000000</value>
</configentry>
<!-- specifies maximum service start time (in seconds) -->
<configentry name="Service_MaxStartTime">
<value type="UnsignedInteger">100000000</value>
</configentry>
<!-- specifies maximum service stop time (in seconds) -->
<configentry name="Service_MaxStopTime">
<value type="UnsignedInteger">100000000</value>
</configentry>
</configsection>
<!-- *** timing parameters for DB services *** -->
<configsection name="SRV_DBORA">
<!-- specifies maximum service restart time -->
<configentry name="Service_MaxRestartTime">
<value type="UnsignedInteger">600</value>
</configentry>
<!-- specifies maximum service start time -->
<configentry name="Service_MaxStartTime">
<value type="UnsignedInteger">300</value>
</configentry>
<!-- specifies maximum service stop time -->
<configentry name="Service_MaxStopTime">
<value type="UnsignedInteger">300</value>
</configentry>
</configsection>
<!-- *** timing parameters for DB services *** -->
<configsection name="SRV_DBSAP">
<!-- specifies maximum service restart time -->
<configentry name="Service_MaxRestartTime">
<value type="UnsignedInteger">600</value>
</configentry>
<!-- specifies maximum service start time -->
<configentry name="Service_MaxStartTime">
<value type="UnsignedInteger">300</value>
</configentry>
<!-- specifies maximum service stop time -->
<configentry name="Service_MaxStopTime">
<value type="UnsignedInteger">300</value>
</configentry>
</configsection>
<!-- *** timing parameters for DB services *** -->
<configsection name="SRV_DB2">
<!-- specifies maximum service restart time -->
<configentry name="Service_MaxRestartTime">
<value type="UnsignedInteger">600</value>
</configentry>
<!-- specifies maximum service start time -->
<configentry name="Service_MaxStartTime">
<value type="UnsignedInteger">300</value>
</configentry>
<!-- specifies maximum service stop time -->
<configentry name="Service_MaxStopTime">
<value type="UnsignedInteger">300</value>
</configentry>
<!-- specifies name and parameter of reaction script. -->
<configentry name="Service_ReactionScript">
<!-- SAP-scripts used before FF V5.0 like sapdb,
sapci, sapapp, sapj, sapacc, ...
<value
type="String">${ScriptPath}/${ScriptPrefix}db${ScriptPostfix} ${BYNIDENT} ${SIDENT}
Parameter Reference
FlexFrame Agents Guide 292
${ACTION}</value>
-->
<!-- SAP-script used from FF V5.0
(ff_service.sh) -->
<value type="String">${ScriptPath}/ff_service.sh
-s ${SIDENT} -t db ${BYKEYNIDENT} -a ${ACTION}</value>
</configentry>
</configsection>
<!-- *** timing parameters for SMD73 services *** -->
<configsection name="SRV_SMD73">
<!-- specifies maximum service restart time -->
<configentry name="Service_MaxRestartTime">
<value type="UnsignedInteger">600</value>
</configentry>
<!-- specifies maximum service start time -->
<configentry name="Service_MaxStartTime">
<value type="UnsignedInteger">300</value>
</configentry>
<!-- specifies maximum service stop time -->
<configentry name="Service_MaxStopTime">
<value type="UnsignedInteger">300</value>
</configentry>
<!-- specifies name and parameter of reaction script. -->
<configentry name="Service_ReactionScript">
<!-- SAP-script used for SMD73 (ff_smd_agent.sh)
-->
<value
type="String">${ScriptPath}/ff_smd_agent.sh -s ${SIDENT} -t ${monparam} ${BYKEYNIDENT} -a
${ACTION}</value>
</configentry>
</configsection>
<!-- *** timing parameters for CI services *** -->
<configsection name="SRV_CI">
<!-- specifies maximum service restart time -->
<configentry name="Service_MaxRestartTime">
<value type="UnsignedInteger">600</value>
</configentry>
<!-- specifies maximum service start time -->
<configentry name="Service_MaxStartTime">
<value type="UnsignedInteger">300</value>
</configentry>
<!-- specifies maximum service stop time -->
<configentry name="Service_MaxStopTime">
<value type="UnsignedInteger">300</value>
</configentry>
</configsection>
<!-- *** timing parameters for APP services *** -->
<configsection name="SRV_APP">
<!-- specifies maximum service restart time -->
<configentry name="Service_MaxRestartTime">
<value type="UnsignedInteger">600</value>
</configentry>
<!-- specifies maximum service start time -->
<configentry name="Service_MaxStartTime">
<value type="UnsignedInteger">300</value>
</configentry>
<!-- specifies maximum service stop time -->
<configentry name="Service_MaxStopTime">
<value type="UnsignedInteger">300</value>
</configentry>
</configsection>
<!-- *** timing parameters for SCS services *** -->
<configsection name="SRV_SCS">
<!-- specifies maximum service restart time -->
<configentry name="Service_MaxRestartTime">
<value type="UnsignedInteger">600</value>
</configentry>
<!-- specifies maximum service start time -->
<configentry name="Service_MaxStartTime">
Parameter Reference
293 FlexFrame Agents Guide
<value type="UnsignedInteger">300</value>
</configentry>
<!-- specifies maximum service stop time -->
<configentry name="Service_MaxStopTime">
<value type="UnsignedInteger">300</value>
</configentry>
</configsection>
<!-- *** timing parameters for ASCS services *** -->
<configsection name="SRV_ASCS">
<!-- specifies maximum service restart time -->
<configentry name="Service_MaxRestartTime">
<value type="UnsignedInteger">600</value>
</configentry>
<!-- specifies maximum service start time -->
<configentry name="Service_MaxStartTime">
<value type="UnsignedInteger">300</value>
</configentry>
<!-- specifies maximum service stop time -->
<configentry name="Service_MaxStopTime">
<value type="UnsignedInteger">300</value>
</configentry>
</configsection>
<!-- *** timing parameters for JC services *** -->
<configsection name="SRV_JC">
<!-- specifies maximum service restart time -->
<configentry name="Service_MaxRestartTime">
<value type="UnsignedInteger">600</value>
</configentry>
<!-- specifies maximum service start time -->
<configentry name="Service_MaxStartTime">
<value type="UnsignedInteger">300</value>
</configentry>
<!-- specifies maximum service stop time -->
<configentry name="Service_MaxStopTime">
<value type="UnsignedInteger">300</value>
</configentry>
</configsection>
<!-- *** timing parameters for J services *** -->
<configsection name="SRV_J">
<!-- specifies maximum service restart time -->
<configentry name="Service_MaxRestartTime">
<value type="UnsignedInteger">600</value>
</configentry>
<!-- specifies maximum service start time -->
<configentry name="Service_MaxStartTime">
<value type="UnsignedInteger">300</value>
</configentry>
<!-- specifies maximum service stop time -->
<configentry name="Service_MaxStopTime">
<value type="UnsignedInteger">300</value>
</configentry>
</configsection>
<!-- *** timing parameters for livecache (LC) services *** -->
<configsection name="SRV_LC">
<!-- specifies maximum service restart time -->
<configentry name="Service_MaxRestartTime">
<value type="UnsignedInteger">600</value>
</configentry>
<!-- specifies maximum service start time -->
<configentry name="Service_MaxStartTime">
<value type="UnsignedInteger">300</value>
</configentry>
<!-- specifies maximum service stop time -->
<configentry name="Service_MaxStopTime">
<value type="UnsignedInteger">300</value>
</configentry>
</configsection>
<!-- *** timing parameters for enqueue replication (ERS) services
Parameter Reference
FlexFrame Agents Guide 294
*** -->
<configsection name="SRV_ERS">
<!-- specifies maximum service restart time -->
<configentry name="Service_MaxRestartTime">
<value type="UnsignedInteger">600</value>
</configentry>
<!-- specifies maximum service start time -->
<configentry name="Service_MaxStartTime">
<value type="UnsignedInteger">300</value>
</configentry>
<!-- specifies maximum service stop time -->
<configentry name="Service_MaxStopTime">
<value type="UnsignedInteger">300</value>
</configentry>
</configsection>
<!-- *** timing parameters for MDM services *** -->
<configsection name="SRV_MDS71">
<!-- specifies maximum service restart time -->
<configentry name="Service_MaxRestartTime">
<value type="UnsignedInteger">600</value>
</configentry>
<!-- specifies maximum service start time -->
<configentry name="Service_MaxStartTime">
<value type="UnsignedInteger">300</value>
</configentry>
<!-- specifies maximum service stop time -->
<configentry name="Service_MaxStopTime">
<value type="UnsignedInteger">300</value>
</configentry>
</configsection>
<!-- *** timing parameters for MDM Import services *** -->
<configsection name="SRV_MDIS71">
<!-- specifies maximum service restart time -->
<configentry name="Service_MaxRestartTime">
<value type="UnsignedInteger">600</value>
</configentry>
<!-- specifies maximum service start time -->
<configentry name="Service_MaxStartTime">
<value type="UnsignedInteger">300</value>
</configentry>
<!-- specifies maximum service stop time -->
<configentry name="Service_MaxStopTime">
<value type="UnsignedInteger">300</value>
</configentry>
</configsection>
<!-- *** timing parameters for MDM Export services *** -->
<configsection name="SRV_MDSS71">
<!-- specifies maximum service restart time -->
<configentry name="Service_MaxRestartTime">
<value type="UnsignedInteger">600</value>
</configentry>
<!-- specifies maximum service start time -->
<configentry name="Service_MaxStartTime">
<value type="UnsignedInteger">300</value>
</configentry>
<!-- specifies maximum service stop time -->
<configentry name="Service_MaxStopTime">
<value type="UnsignedInteger">300</value>
</configentry>
</configsection>
<!-- *** timing parameters for xprint services *** -->
<configsection name="SRV_XPR">
<!-- specifies maximum service restart time -->
<configentry name="Service_MaxRestartTime">
<value type="UnsignedInteger">600</value>
</configentry>
<!-- specifies maximum service start time -->
<configentry name="Service_MaxStartTime">
<value type="UnsignedInteger">300</value>
Parameter Reference
295 FlexFrame Agents Guide
</configentry>
<!-- specifies maximum service stop time -->
<configentry name="Service_MaxStopTime">
<value type="UnsignedInteger">300</value>
</configentry>
<configentry name="Service_ReactionScript">
<!-- SAP-script used for XPR -->
<value
type="String">${ScriptPath}/${ScriptPrefix}xprint${ScriptPostfix} ${BYNIDENT} ${SIDENT}
${ACTION}</value>
</configentry>
</configsection>
<!-- *** timing parameters for content management server services
*** -->
<configsection name="SRV_CMS">
<!-- specifies maximum service restart time -->
<configentry name="Service_MaxRestartTime">
<value type="UnsignedInteger">600</value>
</configentry>
<!-- specifies maximum service start time -->
<configentry name="Service_MaxStartTime">
<value type="UnsignedInteger">300</value>
</configentry>
<!-- specifies maximum service stop time -->
<configentry name="Service_MaxStopTime">
<value type="UnsignedInteger">300</value>
</configentry>
</configsection>
<!-- *** timing parameters for TREX (Text Retrieval and information
EXtraction) services *** -->
<configsection name="SRV_TRX">
<!-- specifies maximum service restart time -->
<configentry name="Service_MaxRestartTime">
<value type="UnsignedInteger">600</value>
</configentry>
<!-- specifies maximum service start time -->
<configentry name="Service_MaxStartTime">
<value type="UnsignedInteger">300</value>
</configentry>
<!-- specifies maximum service stop time -->
<configentry name="Service_MaxStopTime">
<value type="UnsignedInteger">300</value>
</configentry>
</configsection>
<!-- *** timing parameters for Busines Object Enterprice services
*** -->
<configsection name="SRV_BOE4">
<!-- specifies maximum service restart time -->
<configentry name="Service_MaxRestartTime">
<value type="UnsignedInteger">600</value>
</configentry>
<!-- specifies maximum service start time -->
<configentry name="Service_MaxStartTime">
<value type="UnsignedInteger">300</value>
</configentry>
<!-- specifies maximum service stop time -->
<configentry name="Service_MaxStopTime">
<value type="UnsignedInteger">300</value>
</configentry>
</configsection>
<!-- *** timing parameters for Busines Object Enterprice Server
services *** -->
<configsection name="SRV_BOE_SRV">
<!-- specifies maximum service restart time -->
<configentry name="Service_MaxRestartTime">
<value type="UnsignedInteger">600</value>
</configentry>
<!-- specifies maximum service start time -->
<configentry name="Service_MaxStartTime">
Parameter Reference
FlexFrame Agents Guide 296
<value type="UnsignedInteger">300</value>
</configentry>
<!-- specifies maximum service stop time -->
<configentry name="Service_MaxStopTime">
<value type="UnsignedInteger">300</value>
</configentry>
</configsection>
<!-- *** timing parameters for Solution Manager Diagnostics services
*** -->
<configsection name="SRV_SMD">
<!-- specifies maximum service restart time -->
<configentry name="Service_MaxRestartTime">
<value type="UnsignedInteger">600</value>
</configentry>
<!-- specifies maximum service start time -->
<configentry name="Service_MaxStartTime">
<value type="UnsignedInteger">300</value>
</configentry>
<!-- specifies maximum service stop time -->
<configentry name="Service_MaxStopTime">
<value type="UnsignedInteger">300</value>
</configentry>
</configsection>
<!-- *** timing parameters for Primary Application Server/Instance
services *** -->
<configsection name="SRV_PAI">
<!-- specifies maximum service restart time -->
<configentry name="Service_MaxRestartTime">
<value type="UnsignedInteger">600</value>
</configentry>
<!-- specifies maximum service start time -->
<configentry name="Service_MaxStartTime">
<value type="UnsignedInteger">300</value>
</configentry>
<!-- specifies maximum service stop time -->
<configentry name="Service_MaxStopTime">
<value type="UnsignedInteger">300</value>
</configentry>
</configsection>
<!-- *** timing parameters for VMware services *** -->
<configsection name="SRV_VMWARE">
<!-- specifies maximum service restart time -->
<configentry name="Service_MaxRestartTime">
<value type="UnsignedInteger">600</value>
</configentry>
<!-- specifies maximum service start time -->
<configentry name="Service_MaxStartTime">
<value type="UnsignedInteger">300</value>
</configentry>
<!-- specifies maximum service stop time -->
<configentry name="Service_MaxStopTime">
<value type="UnsignedInteger">300</value>
</configentry>
</configsection>
<!-- *** timing parameters for Control-M Service services *** -->
<configsection name="SRV_CTM">
<!-- specifies maximum service restart time -->
<configentry name="Service_MaxRestartTime">
<value type="UnsignedInteger">600</value>
</configentry>
<!-- specifies maximum service start time -->
<configentry name="Service_MaxStartTime">
<value type="UnsignedInteger">300</value>
</configentry>
<!-- specifies maximum service stop time -->
<configentry name="Service_MaxStopTime">
<value type="UnsignedInteger">300</value>
</configentry>
<configentry name="Service_ReactionScript">
Parameter Reference
297 FlexFrame Agents Guide
<!-- SAP-script used for CTM -->
<value type="String">/FlexFrame/scripts/ctms
${ACTION}</value>
</configentry>
</configsection>
<!-- *** timing parameters for Web Dspatcher services *** -->
<configsection name="SRV_WEBDISP">
<!-- specifies maximum service restart time -->
<configentry name="Service_MaxRestartTime">
<value type="UnsignedInteger">600</value>
</configentry>
<!-- specifies maximum service start time -->
<configentry name="Service_MaxStartTime">
<value type="UnsignedInteger">300</value>
</configentry>
<!-- specifies maximum service stop time -->
<configentry name="Service_MaxStopTime">
<value type="UnsignedInteger">300</value>
</configentry>
<configentry name="Service_ReactionScript">
<!-- SAP-script used for WEBDISP -->
<value type="String">${ScriptPath}/ff_service.sh
-s ${SIDENT} -t ${monparam} ${BYKEYNIDENT} -a ${ACTION}</value>
</configentry>
</configsection>
<!-- *** timing parameters for Database SAPSYBASE services *** -->
<configsection name="SRV_SAPSYBASE">
<!-- specifies maximum service restart time -->
<configentry name="Service_MaxRestartTime">
<value type="UnsignedInteger">600</value>
</configentry>
<!-- specifies maximum service start time -->
<configentry name="Service_MaxStartTime">
<value type="UnsignedInteger">300</value>
</configentry>
<!-- specifies maximum service stop time -->
<configentry name="Service_MaxStopTime">
<value type="UnsignedInteger">300</value>
</configentry>
</configsection>
<!-- *** timing parameters for Database SAPSYBASE IQ services *** --
>
<configsection name="SRV_SAPSYB_IQ">
<!-- specifies maximum service restart time -->
<configentry name="Service_MaxRestartTime">
<value type="UnsignedInteger">600</value>
</configentry>
<!-- specifies maximum service start time -->
<configentry name="Service_MaxStartTime">
<value type="UnsignedInteger">300</value>
</configentry>
<!-- specifies maximum service stop time -->
<configentry name="Service_MaxStopTime">
<value type="UnsignedInteger">300</value>
</configentry>
</configsection>
<!-- *** timing parameters for Database SAP-HANA Real-Time Data
Platform services *** -->
<configsection name="SRV_HANA">
<!-- specifies maximum service restart time -->
<configentry name="Service_MaxRestartTime">
<value type="UnsignedInteger">3100</value>
</configentry>
<!-- specifies maximum service start time -->
<configentry name="Service_MaxStartTime">
<value type="UnsignedInteger">2700</value>
</configentry>
<!-- specifies maximum service stop time -->
<configentry name="Service_MaxStopTime">
Parameter Reference
FlexFrame Agents Guide 298
<value type="UnsignedInteger">400</value>
</configentry>
<configentry name="Service_ReactionScript">
<!-- SAP-script used for HANA -->
<value type="String">${ScriptPath}/ff_service.sh
-s ${SIDENT} -t ${monparam} ${BYKEYNIDENT} --nodeid ${NODEID} -a ${ACTION}</value>
</configentry>
</configsection>
<!-- *** timing parameters for Database SAP-HANA Replica Real-Time
Data Platform services *** -->
<configsection name="SRV_HANA_REP">
<!-- specifies maximum service restart time -->
<configentry name="Service_MaxRestartTime">
<value type="UnsignedInteger">3100</value>
</configentry>
<!-- specifies maximum service start time -->
<configentry name="Service_MaxStartTime">
<value type="UnsignedInteger">2700</value>
</configentry>
<!-- specifies maximum service stop time -->
<configentry name="Service_MaxStopTime">
<value type="UnsignedInteger">400</value>
</configentry>
<configentry name="Service_ReactionScript">
<!-- SAP-script used for HANA-REP -->
<value type="String">${ScriptPath}/ff_service.sh
-s ${SIDENT} -t ${monparam} ${BYKEYNIDENT} -N ${NODEID} -r ${REPID} -a ${ACTION}</value>
</configentry>
</configsection>
<!-- *** timing parameters for Database SAP-HANA multi node Real-
Time Data Platform services *** -->
<configsection name="SRV_HANA_MN">
<!-- specifies maximum service restart time -->
<configentry name="Service_MaxRestartTime">
<value type="UnsignedInteger">3100</value>
</configentry>
<!-- specifies maximum service start time -->
<configentry name="Service_MaxStartTime">
<value type="UnsignedInteger">2700</value>
</configentry>
<!-- specifies maximum service stop time -->
<configentry name="Service_MaxStopTime">
<value type="UnsignedInteger">400</value>
</configentry>
<configentry name="Service_ReactionScript">
<!-- SAP-script used for HANA-MN -->
<value type="String">${ScriptPath}/ff_service.sh
-s ${SIDENT} -t ${monparam} ${BYKEYNIDENT} -N ${NODEID} ${BYREPID} -a ${ACTION}</value>
</configentry>
</configsection>
</configsection>
<!-- *** path parameters *** -->
<!-- specifies path to FA scripts -->
<configentry name="FAScriptFilePath">
<value type="String">/opt/myAMC/scripts</value>
</configentry>
<!-- specifies path to FA scripts for local installation -->
<configentry name="FAScriptFilePathLocal">
<value type="String">/opt/local/myAMC/scripts</value>
</configentry>
<!-- Specifies path to spare node request directory. -->
<configentry name="SpareNodeRequestPath">
<value type="String">/opt/myAMC/vFF/SpareRequest</value>
</configentry>
Parameter Reference
299 FlexFrame Agents Guide
<!-- Specifies path to list of virtual cold standby sparenodes. -->
<configentry name="VirtualColdStandbySpareNodeFilePath">
<value type="String">/opt/myAMC/vFF/vFF_${vFF}/config</value>
</configentry>
<!-- specifies path to live list file -->
<configentry name="LiveListLogFilePath">
<value
type="String">/opt/myAMC/vFF/vFF_${vFF}/data/FA/livelist</value>
</configentry>
<!-- specifies path to live list file
In order to use the FlexWeb web interface this path must be
the same as specified in <install-path>/web/myamc-flexweb.conf
and in ServicesXmlFilePath
-->
<configentry name="LiveListXmlFilePath">
<value
type="String">/opt/myAMC/vFF/vFF_${vFF}/data/FA/xmlrepository</value>
</configentry>
<!-- specifies path to service xml file
In order to use the FlexWeb web interface this path must be
the same as specified in <install-path>/web/myamc-flexweb.conf
and in LiveListXmlFilePath
-->
<configentry name="ServicesXmlFilePath">
<value
type="String">/opt/myAMC/vFF/vFF_${vFF}/data/FA/xmlrepository</value>
</configentry>
<!-- specifies path to services list file -->
<configentry name="ServicesListFilePath">
<value
type="String">/opt/myAMC/vFF/vFF_${vFF}/data/FA/servicelists</value>
</configentry>
<!-- specifies path to services log file -->
<configentry name="ServicesLogFilePath">
<value
type="String">/opt/myAMC/vFF/vFF_${vFF}/data/FA/servicelogs</value>
</configentry>
<!-- specifies path to reboot file -->
<configentry name="RebootListFilePath">
<value
type="String">/opt/myAMC/vFF/vFF_${vFF}/data/FA/reboot</value>
</configentry>
<!-- specifies path to switch over file -->
<configentry name="SwitchOverListFilePath">
<value
type="String">/opt/myAMC/vFF/vFF_${vFF}/data/FA/switchover</value>
</configentry>
<!-- specifies path to performance data -->
<configentry name="PerformanceFilePath">
<value
type="String">/opt/myAMC/vFF/vFF_${vFF}/data/FA/performance</value>
</configentry>
<!-- specifies path to blackboard file -->
<configentry name="BlackboardFilePath">
<value
type="String">/opt/myAMC/vFF/vFF_${vFF}/data/FA/blackboard</value>
</configentry>
<!-- specifies path to PersistFAInfoFile file -->
<configentry name="PersistFAInfoFilePath">
<value
type="String">/opt/myAMC/vFF/vFF_${vFF}/data/FA/reboot</value>
</configentry>
Parameter Reference
FlexFrame Agents Guide 300
<!-- specifies path to shutdown config files -->
<configentry name="ShutdownCfgFilePath">
<value
type="String">/opt/myAMC/vFF/vFF_${vFF}/data/FA/shutdown</value>
</configentry>
<!-- specifies path to ACC script files
<install-path>/scripts/acc should either contain
the acc scripts or must be a link to the correct
directory.
-->
<configentry name="ACCScriptFilePath">
<value type="String">/opt/myAMC/scripts/acc</value>
</configentry>
<!-- specifies path to SAP script files
<install-path>/scripts/sap should either contain
the sap scripts or must be a link to the correct
directory.
-->
<configentry name="SAPScriptFilePath">
<value type="String">/opt/myAMC/scripts/sap</value>
</configentry>
<!-- specifies path to control file -->
<configentry name="ControlFilePath">
<value type="String">/opt/myAMC/scripts/sap/log</value>
</configentry>
<!-- specifies path to ServicePing scripts.
-->
<configentry name="ServicePingScriptFilePath">
<value type="String">/opt/myAMC/scripts</value>
</configentry>
<!-- specifies path to FSUtil scripts.
-->
<configentry name="FSUtilScriptFilePath">
<value type="String">/opt/myAMC/scripts</value>
</configentry>
<!-- specifies path to DetTest - scripts, output and timeout [s].
-->
<configentry name="DetTest_Sapparar_ScriptFilePath">
<!--
<value
type="String">/opt/myAMC/FA_AppAgent/myAMC.DetTestSapparar.sh</value>
-->
<value type="String">../myAMC.DetTestSapparar.sh</value>
</configentry>
<configentry name="DetTest_Sapparar_ScriptTimeout">
<value type="UnsignedInteger">60</value>
</configentry>
<configentry name="DetTest_Sapparar_OutputFilePath">
<value type="String">/tmp/myAMC.DetTestOutSapparar.txt</value>
</configentry>
<configentry name="DetTest_Ensmon_ScriptFilePath">
<!--
<value
type="String">/opt/myAMC/FA_AppAgent/myAMC.DetTestEnsmon.sh</value>
-->
<value type="String">../myAMC.DetTestEnsmon.sh</value>
</configentry>
<configentry name="DetTest_Ensmon_ScriptTimeout">
<value type="UnsignedInteger">60</value>
</configentry>
<configentry name="DetTest_Ensmon_OutputFilePath">
<value type="String">/tmp/myAMC.DetTestOutEnsmon.txt</value>
</configentry>
Parameter Reference
301 FlexFrame Agents Guide
<configentry name="DetTest_Dbmcli_ScriptFilePath">
<!--
<value
type="String">/opt/myAMC/FA_AppAgent/myAMC.DetTestDbmcli.sh</value>
-->
<value type="String">../myAMC.DetTestDbmcli.sh</value>
</configentry>
<configentry name="DetTest_Dbmcli_ScriptTimeout">
<value type="UnsignedInteger">60</value>
</configentry>
<configentry name="DetTest_Dbmcli_OutputFilePath">
<value type="String">/tmp/myAMC.DetTestOutDbmcli.txt</value>
</configentry>
<!-- specifies name of HostInfoLDAP config file -->
<!-- The part '-HOSTNAME' will be replaced at runtime with '' or with the
hostname of the node. -->
<configentry name="HostInfoLDAPFile">
<value
type="String">/opt/myAMC/vFF/vFF_${vFF}/config/myAMC_FA_Hosts-HOSTNAME.xml</value>
</configentry>
<!-- specifies name of group config file -->
<configentry name="GroupConfigFile">
<value
type="String">/opt/myAMC/vFF/vFF_${vFF}/config/myAMC_FA_Groups.xml</value>
</configentry>
<!-- specifies name of rule config file.
If the file does not exist, it will be ignored.
-->
<configentry name="RuleConfigFile">
<value
type="String">/opt/myAMC/vFF/vFF_${vFF}/config/myAMC_FA_Rules.xml</value>
</configentry>
<!-- specifies name of custom rule config file.
If the file does not exist, it will be ignored.
-->
<configentry name="RuleCustomConfigFile">
<value
type="String">/opt/myAMC/vFF/vFF_${vFF}/config/myAMC_FA_Rules_Custom.xml</value>
</configentry>
<!-- specifies name of selection-rule config file.
-->
<configentry name="SelectionRuleConfigFile">
<value
type="String">/opt/myAMC/vFF/vFF_${vFF}/config/myAMC_FA_SelectionRules.xml</value>
</configentry>
<!-- *** misc parameters *** -->
<!-- specifies the postfix of server-lan -->
<configentry name="LanPostfixServer">
<value type="String">-se</value>
</configentry>
<!-- specifies the postfix of client-lan -->
<configentry name="LanPostfixClient">
<value type="String"></value>
</configentry>
<!-- specifies the postfix of storage-lan -->
<configentry name="LanPostfixStorage">
<value type="String">-st</value>
</configentry>
Parameter Reference
FlexFrame Agents Guide 302
<!-- specifies the postfix of control-lan -->
<configentry name="LanPostfixControl">
<value type="String">-co</value>
</configentry>
<!-- specifies whether or not to respect the service dependencies -->
<configentry name="RespectServiceDependencies">
<value type="Boolean">true</value>
</configentry>
<configsection name="Shutdown_Facility">
<configsection name="Executable">
<configentry name="Shut_Ex_FA_BLADE">
<value
type="String">/opt/myAMC/scripts/PowerMng/blade_wrapper.sh</value>
</configentry>
<configentry name="Shut_Ex_FA_IPMI">
<value
type="String">/opt/myAMC/scripts/PowerMng/ipmi_wrapper.sh</value>
</configentry>
<configentry name="Shut_Ex_FA_XSCF2">
<value type="String"></value>
</configentry>
<configentry name="Shut_Ex_FA_XSCF">
<value type="String"></value>
</configentry>
<configentry name="Shut_Ex_FA_RPS">
<value type="String"></value>
</configentry>
<configentry name="Shut_Ex_FA_VM">
<value
type="String">/opt/myAMC/scripts/PowerMng/vm_vm_wrapper.sh</value>
</configentry>
<configentry name="Shut_Ex_FA_GENERIC">
<value
type="String">/opt/myAMC/scripts/PowerMng/generic_wrapper.sh</value>
</configentry>
<configentry name="Shut_Ex_RESERVED_1">
<value type="String"></value>
</configentry>
<configentry name="Shut_Ex_RESERVED_2">
<value type="String"></value>
</configentry>
<configentry name="Shut_Ex_RESERVED_3">
<value type="String"></value>
</configentry>
<configentry name="Shut_Ex_RESERVED_4">
<value type="String"></value>
</configentry>
<configentry name="Shut_Ex_RESERVED_5">
<value type="String"></value>
</configentry>
<configentry name="Shut_Ex_WINSTD">
<value type="String">Shutdown.exe</value>
</configentry>
<configentry name="Shut_Ex_WIN_RESERVED_1">
<value type="String"></value>
</configentry>
<configentry name="Shut_Ex_WIN_RESERVED_2">
<value type="String"></value>
</configentry>
<configentry name="Shut_Ex_WIN_RESERVED_3">
<value type="String"></value>
</configentry>
<configentry name="Shut_Ex_WIN_RESERVED_4">
<value type="String"></value>
</configentry>
<configentry name="Shut_Ex_WIN_RESERVED_5">
<value type="String"></value>
</configentry>
Parameter Reference
303 FlexFrame Agents Guide
</configsection>
<configsection name="Configuration">
<configentry name="Shut_Prio_FA_BLADE">
<value type="UnsignedInteger">1</value>
</configentry>
<configentry name="Shut_Prio_FA_IPMI">
<value type="UnsignedInteger">2</value>
</configentry>
<configentry name="Shut_Prio_FA_XSCF2">
<value type="UnsignedInteger">111</value>
</configentry>
<configentry name="Shut_Prio_FA_XSCF">
<value type="UnsignedInteger">112</value>
</configentry>
<configentry name="Shut_Prio_FA_RPS">
<value type="UnsignedInteger">113</value>
</configentry>
<configentry name="Shut_Prio_FA_VM">
<value type="UnsignedInteger">3</value>
</configentry>
<configentry name="Shut_Prio_FA_GENERIC">
<value type="UnsignedInteger">5</value>
</configentry>
<configentry name="Shut_Prio_RESERVED_1">
<value type="UnsignedInteger">6</value>
</configentry>
<configentry name="Shut_Prio_RESERVED_2">
<value type="UnsignedInteger">7</value>
</configentry>
<configentry name="Shut_Prio_RESERVED_3">
<value type="UnsignedInteger">8</value>
</configentry>
<configentry name="Shut_Prio_RESERVED_4">
<value type="UnsignedInteger">9</value>
</configentry>
<configentry name="Shut_Prio_RESERVED_5">
<value type="UnsignedInteger">10</value>
</configentry>
<configentry name="Shut_Prio_WINSTD">
<value type="UnsignedInteger">11</value>
</configentry>
<configentry name="Shut_Prio_WIN_RESERVED_1">
<value type="UnsignedInteger">12</value>
</configentry>
<configentry name="Shut_Prio_WIN_RESERVED_2">
<value type="UnsignedInteger">13</value>
</configentry>
<configentry name="Shut_Prio_WIN_RESERVED_3">
<value type="UnsignedInteger">14</value>
</configentry>
<configentry name="Shut_Prio_WIN_RESERVED_4">
<value type="UnsignedInteger">15</value>
</configentry>
<configentry name="Shut_Prio_WIN_RESERVED_5">
<value type="UnsignedInteger">16</value>
</configentry>
<configentry name="Shut_UseIfIP_FA_BLADE">
<value type="Boolean">true</value>
</configentry>
<configentry name="Shut_UseIfIP_FA_IPMI">
<value type="Boolean">false</value>
</configentry>
<configentry name="Shut_UseIfIP_FA_XSCF2">
<value type="Boolean">true</value>
</configentry>
<configentry name="Shut_UseIfIP_FA_XSCF">
<value type="Boolean">true</value>
</configentry>
<configentry name="Shut_UseIfIP_FA_RPS">
<value type="Boolean">false</value>
Parameter Reference
FlexFrame Agents Guide 304
</configentry>
<configentry name="Shut_UseIfIP_FA_VM">
<value type="Boolean">false</value>
</configentry>
<configentry name="Shut_UseIfIP_FA_GENERIC">
<value type="Boolean">false</value>
</configentry>
<configentry name="Shut_UseIfIP_RESERVED_1">
<value type="Boolean">false</value>
</configentry>
<configentry name="Shut_UseIfIP_RESERVED_2">
<value type="Boolean">false</value>
</configentry>
<configentry name="Shut_UseIfIP_RESERVED_3">
<value type="Boolean">false</value>
</configentry>
<configentry name="Shut_UseIfIP_RESERVED_4">
<value type="Boolean">false</value>
</configentry>
<configentry name="Shut_UseIfIP_RESERVED_5">
<value type="Boolean">false</value>
</configentry>
<configentry name="Shut_UseIfIP_WINSTD">
<value type="Boolean">false</value>
</configentry>
<configentry name="Shut_UseIfIP_WIN_RESERVED_1">
<value type="Boolean">false</value>
</configentry>
<configentry name="Shut_UseIfIP_WIN_RESERVED_2">
<value type="Boolean">false</value>
</configentry>
<configentry name="Shut_UseIfIP_WIN_RESERVED_3">
<value type="Boolean">false</value>
</configentry>
<configentry name="Shut_UseIfIP_WIN_RESERVED_4">
<value type="Boolean">false</value>
</configentry>
<configentry name="Shut_UseIfIP_WIN_RESERVED_5">
<value type="Boolean">false</value>
</configentry>
</configsection>
<!-- Full specified path to pre poff hook script. -->
<configentry name="PrePoffHookPath">
<value
type="String">/opt/myAMC/scripts/PrePoffHook.sh</value>
</configentry>
<!-- Full specified path to post poff hook script. -->
<configentry name="PostPoffHookPath">
<value
type="String">/opt/myAMC/scripts/PostPoffHook.sh</value>
</configentry>
<!-- Ignore the result of the pro or post hook scripts. -->
<configentry name="IgnorePoffHookResult">
<value type="Boolean">true</value>
</configentry>
<!-- Full specified path to script to check for VM host POFF. -->
<configentry name="VMHostPoffCheckPath">
<value
type="String">/opt/myAMC/scripts/VMHostPoffCheck.sh</value>
</configentry>
<!-- Timeout in sec for the script to check for VM host POFF. -->
<configentry name="VMHostPoffCheckTimeout">
<value type="UnsignedInteger">60</value>
</configentry>
Parameter Reference
305 FlexFrame Agents Guide
<configsection name="Managementblades">
<!-- Here the 'Hostname's of the management-blades must be
configured. -->
<!-- first entry -->
<!-- this is in comment, because it is only an example !!!
<configsection name="Mgmt_Blade_1">
<configentry name="Hostname">
<value type="String">vader</value>
</configentry>
</configsection>
-->
<!-- 2. entry -->
<!-- this is in comment, because it is only an example !!!
<configsection name="Mgmt_Blade_2">
<configentry name="Hostname">
<value type="String">yoda</value>
</configentry>
</configsection>
-->
</configsection>
<!-- Here the default 'ShutdownMode' must be configured. -->
<!--
Allowed values for:
'ShutdownMode': cycle, leave-off
-->
<configentry name="Default_ShutdownMode">
<value type="String">leave-off</value>
</configentry>
<!-- Here the values for timing may be configured. -->
<!--
Shut_Cycle [ms]: Cycletime of Shutdown-functionality. Must
be > 10 Sec.
Shut_SNMP_Timeout [ms]: SNMP-timeout
Shut_SNMP_Tries []: SNMP-tries
Shut_Exec_Timeout [ms]: Timeout for execute the SA-agents
Shut_PowerOff_Try []: Poff-tries
Shut_ReNew_validity_1 [ms]: renew the SA-config if
hostdata are valid
Shut_ReNew_validity_0 [ms]: renew the SA-config if
hostdata are invalid
Shut_DiagTrapRepeat_valid [s]: resend trap if shutdown
validity is valid ('0' == no resend)
Shut_DiagTrapRepeat_invalid [s]: resend trap if shutdown
validity is invalid ('0' == no resend)
-->
<configentry name="Shut_Cycle">
<value type="UnsignedInteger">60000</value>
</configentry>
<configentry name="Shut_SNMP_Timeout">
<value type="UnsignedInteger">3000</value>
</configentry>
<configentry name="Shut_SNMP_Tries">
<value type="UnsignedInteger">3</value>
</configentry>
<configentry name="Shut_Exec_Timeout">
<value type="UnsignedInteger">30000</value>
</configentry>
<configentry name="Shut_PowerOff_Try">
<value type="UnsignedInteger">1</value>
</configentry>
<configentry name="Shut_ReNew_validity_1">
<value type="UnsignedInteger">3600000</value>
</configentry>
<configentry name="Shut_ReNew_validity_0">
<value type="UnsignedInteger">600000</value>
Parameter Reference
FlexFrame Agents Guide 306
</configentry>
<configentry name="Shut_DiagTrapRepeat_valid">
<value type="UnsignedInteger">86400</value>
</configentry>
<configentry name="Shut_DiagTrapRepeat_invalid">
<value type="UnsignedInteger">3600</value>
</configentry>
<configentry name="IgnoreShutdownFailure">
<value type="Boolean">false</value>
</configentry>
<configsection name="BrutForceShutdown">
<!-- Here all parameters for a 'brut force shutdown' must be
configured. -->
<!--
Allowed values for:
'ShutdownTyp': UNKNOWN, BLADE, IPMI, RSB, XSCF, RPS,
SCON, XSCF2, ALOM, FA_BLADE, FA_IPMI,
FA_XSCF2, FA_XSCF,
FA_RPS, FA_VM, FA_GENERIC, RESERVED_1, RESERVED_2, RESERVED_3, RESERVED_4, RESERVED_5,
WINSTD,
WIN_RESERVED_1, WIN_RESERVED_2, WIN_RESERVED_3, WIN_RESERVED_4, WIN_RESERVED_5
'Hardware': UNKNOWN, LINUX, SOLARIS, WIN
'ShutdownMode': cycle, leave-off
-->
<!-- first entry -->
<!-- this is in comment, because it is only an example !!!
<configsection name="BFSD_Blade_1">
<configentry name="Hostname">
<value type="String">vader</value>
</configentry>
<configentry name="ShutdownTyp">
<value type="String">XSCF</value>
</configentry>
<configentry name="MacAdress">
<value
type="String">00c00d0032f7</value>
</configentry>
<configentry name="Hardware">
<value type="String">SOLARIS</value>
</configentry>
<configentry name="ShutdownMode">
<value type="String">cycle</value>
</configentry>
<configentry name="IP_Adress">
<value
type="String">158.92.4.214</value>
</configentry>
<configentry name="Console">
<value type="String">console_1</value>
</configentry>
<configentry name="Machine">
<value type="String">i686</value>
</configentry>
<configentry name="Port">
<value
type="UnsignedInteger">2</value>
</configentry>
</configsection>
-->
<!-- 2. entry -->
<!-- this is in comment, because it is only an example !!!
<configsection name="BFSD_Blade_2">
<configentry name="Hostname">
<value type="String">yoda</value>
</configentry>
<configentry name="ShutdownTyp">
<value type="String">BLADE</value>
</configentry>
Parameter Reference
307 FlexFrame Agents Guide
<configentry name="MacAdress">
<value
type="String">00c00d0032f7</value>
</configentry>
<configentry name="Hardware">
<value type="String">LINUX</value>
</configentry>
<configentry name="ShutdownMode">
<value type="String">cycle</value>
</configentry>
<configentry name="IP_Adress">
<value
type="String">192.168.102.104</value>
</configentry>
<configentry name="Console">
<value type="String">console_1</value>
</configentry>
<configentry name="Machine">
<value type="String">i686</value>
</configentry>
<configentry name="Port">
<value
type="UnsignedInteger">2</value>
</configentry>
</configsection>
-->
</configsection>
</configsection>
<configsection name="Additional_Checks">
<!-- specifies how often (in seconds) myAMC.FA make the additional
checks.
Addition Checks: 'Mount', 'Filers', 'Files', 'Lock'.
The default value for this parameter is 60 seconds.
-->
<configentry name="AddCheckCycleTime">
<value type="UnsignedInteger">60</value>
</configentry>
<!-- specifies the filename for 'lock'-checks checks.
"<hostname>.log.lock" will be added by program.
-->
<configentry name="Lock_file">
<value
type="String">/opt/myAMC/vFF/vFF_${vFF}/log/AppAgt/CHK_lock_</value>
</configentry>
<configsection name="File_check">
<!-- For the FileCheck there must be configured 'Filename'
and 'permissions'. -->
<!-- 'Filename' must be full specified or accassable from
'bin_xxx'-directory. -->
<!-- 'permissions': Allowed are 'F'=exists, 'R'=readable,
'W'=writable, 'X'=executable -->
<!-- first entry -->
<configsection name="File_monitor_alert">
<configentry name="Filename">
<value
type="String">/opt/myAMC/scripts/sap/monitor_alert</value>
</configentry>
<configentry name="permissions">
Parameter Reference
FlexFrame Agents Guide 308
<value type="String">FRX</value>
</configentry>
</configsection>
<!-- next entry -->
<configsection name="File_ff_service.sh">
<configentry name="Filename">
<value
type="String">/opt/myAMC/scripts/sap/ff_service.sh</value>
</configentry>
<configentry name="permissions">
<value type="String">FRX</value>
</configentry>
</configsection>
</configsection>
<configsection name="Mount_check">
<!-- For the Mount_check there must be configured
'Mountpoint' and 'permissions'. -->
<!-- 'Mountpoint' must be full specified. -->
<!-- 'permissions': Allowed are 'R'=readable, 'W'=writable
-->
<!-- FF5.3/FFO1.0 : /FlexFrame/myAMC
-->
<!-- FFO1.1 : CN /FlexFrame/volFF -->
<!-- FFO1.1 : AN /myAMC/vFF_pool -->
<!-- first entry -->
<configsection name="Mount_1">
<configentry name="Mountpoint">
<value
type="String">/myAMC/vFF_pool</value>
</configentry>
<configentry name="permissions">
<value type="String">RW</value>
</configentry>
</configsection>
</configsection>
<configsection name="Filer_check">
<!-- For the Filer_check there must be configured
'IP_addr' and 'permissions'. -->
<!-- 'IP_Hostname': IP-address of filer, or hostname. -->
<!-- 'ffu': parmaeter for future use -->
<!-- first entry -->
<!-- this is in comment, because it is only an example !!!
<configsection name="Filer_1">
<configentry name="IP_Hostname">
<value type="String">filer</value>
</configentry>
<configentry name="ffu">
<value type="String">xxx</value>
</configentry>
</configsection>
-->
<!-- 2. entry -->
<!-- this is in comment, because it is only an example !!!
<configsection name="Filer_2">
<configentry name="IP_Hostname">
<value type="String">filer2</value>
</configentry>
<configentry name="ffu">
<value type="String">xxx</value>
Parameter Reference
309 FlexFrame Agents Guide
</configentry>
</configsection>
-->
</configsection>
<configsection name="TimeSyncFB">
<!-- The TimeSyncFB-check checks the time synchronicity
against the filer. -->
<!-- The TimeSyncFB-check needs the configuration of a
seperate cycle-time for thread and file-based-check, and max difference for warning and critical
level. -->
<!-- 'CycleTime_THR' Cycletime-Thread [s] how often the
thread will work. -->
<!-- 'CycleTime_FBC' Cycletime-FBC [s] to perform the file
based check ('0' switch off this check). -->
<!-- 'MaxDiff_Warning': Max time difference [s] for
warning-level. -->
<!-- 'MaxDiff_Critical': Max time difference [s] for
critical-level. -->
<configentry name="CycleTime_THR">
<value type="UnsignedInteger">10</value>
</configentry>
<configentry name="CycleTime_FBC">
<value type="UnsignedInteger">3600</value>
</configentry>
<configentry name="MaxDiff_Warning">
<value type="UnsignedInteger">30</value>
</configentry>
<configentry name="MaxDiff_Critical">
<value type="UnsignedInteger">80</value>
</configentry>
</configsection>
<configsection name="ServicePing_Switch">
<!-- Service ping can be switched on for special services
selected with wildcards. -->
<!-- ServicePing_active: boolen to activate the
entry. -->
<!-- ServicePing_ServiceTyoe: Service-type of service
'SRV_HANA', 'SRV_DB', 'SRV_APP', ... . -->
<!-- ServicePing_SIdent: SID of service (with
wildcards). -->
<!-- ServicePing_NIdent: instance-number of service
(with wildcards). -->
<!-- first entry -->
<configsection name="HANA_Replica">
<configentry name="ServicePing_active">
<value type="Boolean">true</value>
</configentry>
<configentry name="ServicePing_ServiceType">
<value
type="String">SRV_HANA_REP</value>
</configentry>
<configentry name="ServicePing_SIdent">
<value type="String">*</value>
</configentry>
<configentry name="ServicePing_NIdent">
<value type="String">*</value>
</configentry>
</configsection>
<!-- next entry -->
<configsection name="HANA_MN">
<configentry name="ServicePing_active">
<value type="Boolean">true</value>
Parameter Reference
FlexFrame Agents Guide 310
</configentry>
<configentry name="ServicePing_ServiceType">
<value
type="String">SRV_HANA_MN</value>
</configentry>
<configentry name="ServicePing_SIdent">
<value type="String">*</value>
</configentry>
<configentry name="ServicePing_NIdent">
<value type="String">*</value>
</configentry>
</configsection>
<!-- next entry -->
<configsection name="HANA">
<configentry name="ServicePing_active">
<value type="Boolean">false</value>
</configentry>
<configentry name="ServicePing_ServiceType">
<value type="String">SRV_HANA</value>
</configentry>
<configentry name="ServicePing_SIdent">
<value type="String">*</value>
</configentry>
<configentry name="ServicePing_NIdent">
<value type="String">*</value>
</configentry>
</configsection>
</configsection>
</configsection>
<configsection name="SharedResource">
<!-- Specifies the directory, user and password of shared
filesystems used on Windows systems.
These parameters are only used on Windows systems.
-->
<configsection name="Directory">
<!-- 1st entry goes here -->
<!-- this is in comment, because it is only an example !!!
<configsection name="Directory_1">
<configentry name="RemoteNetwork">
<value
type="String">\\myserver\myuser</value>
</configentry>
<configentry name="User">
<value type="String">myuser</value>
</configentry>
<configentry name="Password">
<value
type="String">blabla1234567890</value>
</configentry>
</configsection>
-->
<!-- next entry goes here -->
</configsection>
</configsection>
</configsection>
</configuration>
Parameter Reference
311 FlexFrame Agents Guide
Parameter Reference
FlexFrame Agents Guide 312
7.5.13 FlexFrame FrameAgent
The FF FrameAgent uses the same type of config files as the FF AppAgent and the FF
CtrlAgent. The FF FrameAgent is a local installation on the Control Nodes. Therefore the
config files are located in a local pool structure on the Control Node.
Local file system: /opt/local/myAMC/vFF/vFF_<Pool>/config/*
Instead of shared file system: /opt/myAMC/vFF/vFF_<Pool>/config/*
The default parameter values e.g. timing parameters, or path values in the config files
may differ.
7.5.14 Adjustment of the FlexFrame Agents WebGUI
Some of the Icons shown in the FA WebGUI can be adapted with individual icons.
To change the standard icons with individual icons, the file
/opt/myAMC/FA_WebGui/FAWebGui.properties has to be changed.
application.title
Program title, when the FA WebGUI ist started as application
(/opt/myAMC/FA_WebGui/webgui.sh).
icons.logo-topleft
Left picture on the top left of the GUI.
Default is the „Fujitsu“ logo.
icons.logo-topright
Right picture on the top of the gui.
icons.heading
Icon in the middle of the top line of the GUI.
icons.node.<POOLNAME>.<AN>
icons.node.bygroup.<poolname>.<groupname>
icons.node.<AN>
Icon for the Application Node with the name <AN>.
icons.nodes.<POOLNAME>
Icon for the Application Node of the pool <POOLNAME>.
icons.node.bygroup.<groupname>
icons.node.byos.<operatingsystem>
linux or solaris
icons.nodes
node
Parameter Reference
313 FlexFrame Agents Guide
icons.system.<POOLNAME>.<SID>
icons.system.<SID>
icons.systems.<POOLNAME>
icons.systems
7.5.15 FlexFrame Webservices and Hypervisor Monitor con-figuration
The server part of the FlexFrame WebGui has some configuration parameters.
The configuration file is /opt/myAMC/config/FA_WebGui.conf
The following parameters are used for the collection of the virtual nodes of a hostsystem:
builder.enabled.vmhost
Enable or disable VMHost-Checkers: true or false
builder.update-cycletime.vmhost
Cycletime (in seconds) for the monitoring of available ESX servers.
Parameter Reference
FlexFrame Agents Guide 314
7.6 FlexFrame Domain Manager
7.6.1 Common
FlexFrame DomainManager is a component of FlexFrame Performance and Accounting
Solution.
FlexFrame DomainManager processing measuring data which are collected from Flex-
Frame Agents and Collectors as ASCII-Files (called Collet-Files).
Reading of Collet-Files
Profil control, generation of alerts
Storage in database
7.6.2 Configuration
7.6.2.1 Common Parameters
The DomainManager needs the config file DomainManager.xml in the directory
/opt/myAMC/FA_DomainManager/config/.
LogLevel
Possible values: -1=Off, 1=Critical, 2=Error, 3=Warning, 4=Normal, 5=Info,
6=Babble, 7=Debug.
ReaderRefreshTime
Cycletime in seconds to look for new collets (default: 60).
7.6.2.2 Database Connection
The parameters for the database connection are in the same config file:
DatabaseDSN
ODBC database name (DSN), the same as in the file /etc/odbc.ini
(default: AMC_PERF).
DatabaseUser
Database user name (default: myAMC).
DatabasePassword
(default: FlexFrame).
Parameter Reference
315 FlexFrame Agents Guide
7.6.3 Configuration of DomainManager with FA-ConfigGUI
For FlexFrame Accounting there is a separate manual, the "FlexFrame Agents Ac-
counting Guide" in which the configuration is described in detail.
Among other things, the following configurations are included in this document:
Accounting configuration see chapter 3.4:
o powervalue [SAPs equivalent] Value and Hyperthreading
o Release information for SAP systems
o Pricing information
Performance profiles see section 3.5:
o Profiles for File System Monitoring
o Profiles for directory monitoring
316 FlexFrame Agents Guide
8 BlackBoard
8.1 General
The FlexFrame Agents Suite offers a command interface via which a node, an instance
or a complete system can be started and stopped. Furthermore it enables a system to be
placed in the nowatch status, which results in monitoring being disabled for this period.
The command interface is used both for manual intervention and also for operation via
the WebInterface.
8.2 Implementation
The BlackBoard is implemented as an ASCII file which is secured against manipulation.
This file is also used as a log for the commands triggered via the BlackBoard.
Each BlackBoard command is valid for a particular period. After this the command is
discarded.
Each BlackBoard command has a mechanism which secures it against being modified
with an editor.
Command syntax:
Each command is represented by a line in the ASCII file blackboard.txt. This
line is formed of Variable=Value tuples. The following variables are available:
TimeStamp
Time the entry was made in the format “DD-MM-YY HH:MM:SS“
TimeLong
Timestamp of the entry (seconds since 1970)
SRC-ID
Source identifier (sender ID: "AppAgent", "CtrlAgent", "WebGui", "Extern" etc.)
SRC-Hostname
Host name of the sender
CMD-ID
Command identifier: String which identifies the command.
vFF
Name of the virtual FlexFrame for which this command applies.
Group
Name of the group for which this command applies.
Service
Name of the service for which this command applies.
BlackBoard
317 FlexFrame Agents Guide
SID
SID of the SAP system for which this command applies.
Inst-Nr
Instance number for which this command applies.
Node
Host name of the node for which this command applies.
Value
Value for specific commands.
Validity
Validity period for this command in seconds (examples: "0", "180", etc.).
Key
String which protects the command against manipulation and entries made with
an editor.
Example:
TimeStamp=31-12-2003 15:00:10;
TimeLong=1234567899;
SRC-ID=myAMC.FA_WebInterface;
SRC-Hostname=vader;
CMD-ID=Service_Start;
vFF=Bayer1;
Group=Produktiv;
Service=SRV_APP;
SID=P*;
Inst-Nr=;
Node=;
Value=;
Validity=240;
Key=65h57709845kjjhjkh465480
Individual command syntax fields can be empty. These are then not entered in the
BlackBoard file.
Individual command syntax fields can be filled with keywords (e.g. "*"). Wildcards
can also be used.
An empty field does not mean “*” !
If, for example, a command is to apply for all nodes, Node=* must be specified.
BlackBoard
FlexFrame Agents Guide 318
The following SRC-IDs (source identifiers) are currently permitted:
myAMC.FA_AppAgent
myAMC.FA_CtrlAgent
myAMC.FA_WebInterface
myAMC.FA_SuspendAutonomy
myAMC.FA_Stonith
myAMC.FA_BBTool
myAMC.FA_BBTool_1
myAMC.FA_BBTool_2
myAMC.FA_BBTool_3
myAMC.FA_BBTool_4
myAMC.FA_BBTool_5
No other SRC-IDs are accepted.
BlackBoard
319 FlexFrame Agents Guide
8.3 Generating BlackBoard Commands
8.3.1 WebInterface
Commands can be issued to the BlackBoard from the FlexFrame Agents WebInterface if
interaction is enabled or if the user’s rights permit this.
Here the variables required are requested by the users, if necessary, and the command
is written to the BlackBoard.
Over the Webinterface only the commands permitted for the relevant element (pool,
group, node, service) are offered.
In future enhancements it will also be possible to store security prompts and password
queries here to provide the greatest degree of security against user errors.
The actual entry in the BlackBoard file is made using the BBTool.sh script.
Hint: This interaction with WebInterface will be not supported by FlexFrame.
Example: Usage of WebInterface to generate BlackBoard commands:
With “right-click” to a node/service in the tree the context menu will be shown. There
services can be started, stopped or set to Watch/NoWatch.
Then the “Execute service action”-menu will appear, where depending on the selected
action different choices may be done.
BlackBoard
FlexFrame Agents Guide 320
Next press „Execute“ to write the action in the BlackBoard. The further execution will be
done by the FlexFrame Agents (after some delay).
8.3.2 Interactive
BlackBoard commands can be entered manually in the BlackBoard file using the BBTool.sh and BBT_dialog.sh scripts.
The scripts and programs only provide a limited plausibility check.
You should therefore prefereably use the FlexFrame Agents WebInterface.
BlackBoard
321 FlexFrame Agents Guide
8.4 Commands
Command Parameters
(wildcards permitted)
Parameters
(no wildcards)
Value ?
Service_Start Node, vFF Group, Service,
SID, Inst-Nr
(with APP)
Service_Start_as_SwGet
(for internal use only)
Node, vFF Group, Service,
SID, Inst-Nr
(with APP)
Service_Stop Node, vFF, Group,
Service, SID, Inst-Nr
Service_Restart Node, vFF, Group,
Service, SID, Inst-Nr
Service_SetPrio Node, vFF, Group,
Service, SID, Inst-Nr
Value Priority
Service_Watch Node, vFF, Group,
Service, SID, Inst-Nr
Service_Nowatch Node, vFF, Group,
Service, SID, Inst-Nr
Service_ReactionON Node, vFF, Group,
Service, SID, Inst-Nr
Service_ReactionOFF Node, vFF, Group,
Service, SID, Inst-Nr
Service_TrapsendON Node, vFF, Group,
Service, SID, Inst-Nr
Service_TrapsendOFF Node, vFF, Group,
Service, SID, Inst-Nr
Node_Reboot Node, vFF
Node_Shutdown Node, vFF
Node_Switchover Node, vFF
BlackBoard
FlexFrame Agents Guide 322
Command Parameters
(wildcards permitted)
Parameters
(no wildcards)
Value ?
Free command
(only autonomous test and
support)
Node, vFF Value Command
SuspendAutonomy
SuspendAutonomyReset
StonithRequest vFF, Node,
Value
<REQUES
T_NODE>:
<SHUTDO
WN_NODE
>:<ACTION
_ID>:<RES
PONCE_S
TATE>
StonithResponce vFF, Node,
Value
<REQUES
T_NODE>:
<SHUTDO
WN_NODE
>:<ACTION
_ID>:<RES
PONCE_S
TATE>
FlexFrame Agents Guide 323
9 FlexFrame Agents Traps
FlexFrame Agents can be easily linked into Enterprise Management scenarios. They
supply an SNMP trap for all major status changes and for all reactions which are imple-
mented.
9.1 General
Each trap supplies all the important information on the time, physical and virtual identifi-
cation of the trap sender, and the severity, through to a meaningful message text in short
or long form, enabling it to be used directly for display in Enterprise Event Management
Systems or as a brief info, mail or SMS.
The contents of the traps of the FlexFrame Agents V1.0 and V2.0 are in principle identi-
cal. Trap attributes which did not exist in Version 1.0, e.g. pool name and group name,
are not sent.
The format of the FlexFrame Agents traps is such that they can be analyzed and further
processed in Enterprise Management Systems using filter modules. All important attrib-
utes which concern FlexFrame Agents can be found in the variable bindings of a Flex-
Frame Agents trap.
Enterprise OID 1.3.6.1.4.1.231.694.27,
Major Trap ID 6
Minor Trap ID 2
All messages contain a severity code. The following section describes the different sever-
ity levels:
Normal
Messages which indicate the transition to a normal state are sent with severity nor-
mal.
Warning
Messages regarding problems which are detected by the FlexFrame Agents are sent
with severity warning.
Critical
Messages regarding problems which lead to a reaction (restart, reboot, switchover)
are sent with severity critical.
Emergency
Messages regarding severe problems or situations are sent with severity emergen-
cy. This e.g. includes messages regarding MultiNodeFailures.
FlexFrame Agents Guide 324
9.2 Structur of FlexFrame Agents Traps
All traps are sent with the Enterprise OID 1.3.6.1.4.1.231.694.27, plus the Major Trap ID 6 and the Minor Trap ID 2.
General trap format (all variable bindings have the prefix 1.3.6.1.4.1.231.694).
FlexFrame Agents Traps
FlexFrame Agents Guide 325
Example VB Description Comment
1 10 Trap version
27 20 Application ID (One of the two fields is always configurable)
myAMC.FA 21 Application name
1049117190 30 Timestamp (unixtime)
2003-04-01 31 Date %Y-%m-%d
12:15:00 32 Time %H:%M:%S
110 Symbolic group name (if configured)
111 Group name (if configured)
o11 121 System ID
belana 131 Physical device name
158.92.4.215 132 Physical IP address
1 140 Type: Alarm (1), Event (2), Log (3) ,
Heartbeat (4)
FA:ServiceUp
Log
152 Category (see VB 210) FA:<unique text as trap-id>
Log
00 161 Instance number
cio11_o11_00 162 Instance name <virtual_servername>_<sid>_<inst_no>
FlexFrame Agents Traps
326 FlexFrame Agents Guide
Example VB Description Comment
250 170 Severity (see table)
Cust_1 190 Pool name (vFF) Name of pool to which the host belongs
BSP_GR_1 191 Group name Group name of the service
2 192 Priority Priority of the service
SRV_DBORA 200 Service type name Unique service type name
The following names are possible: SRV_DBORA, SRV_DBSAP,
SRV_CI, SRV_APP, SRV_SCS, SRV_ASCS, SRV_JC, SRV_J
message server 201 Subservice type
Down 202 Status
cio11 203 Virtual host name (server LAN)
00 204 Service ID In the first version it corresponds to the instance number (VB 161)
myAMC_FA_AppAgent 205 Sender Process name : myAMC_FA_{App|Ctrl}Agent
db 206 Service type display name Display name of service type
123 210 Message ID (see VB 152) Unique trap number (internal ID)
message server is down 500 Short message Foramatted short message text
message server of service ci
(O11 00) on node belana is down
501 Long message Foramatted long message text
FlexFrame Agents Traps
FlexFrame Agents Guide 327
9.3 Overview of the FlexFrame Agents SNMP Traps
The following tables provide an overview of all defined traps of the FlexFrame Agents. Trap version V1 and V2.
These lists can be modified and extended at any time as a result of change requests.
A minus sign in a field means that this VB does not exist in the trap.
Table 1
VB 210
(as of
V2.0)
Internal ID see VB 210
(as of V2.0)
Short Message
VB 500
Long Message
VB 501
1 CtrlAgtUp has started <sender> on node <phys. server> has started
2 CtrlAgtDown has shut down <sender> on node <phys. server> has shut down
3 CtrlAgtSwitchOver switch over switching over service <service> <SID> <ID> node
<phys. server>
4 CtrlAgtSwitchOverFailed switch over failed switching over of service <service> <SID> <ID> from node
<phys. server> failed
5 CtrlAgtPoffHost_ok power off ok ‘<hostname>’ power off ok: <specific message>
6 CtrlAgtPoffHost_failed power off failed ‘<hostname>’ power off failed: <specific message>
7 CtrlAgtPoffHost_IF_ERR power off not done ‘<host-
name>’ power off failed: <specific message>
8 CtrlAgtPoffSwitchOffNetInterfacesOk switched off network interfaces switched off network interfaces
FlexFrame Agents Traps
328 FlexFrame Agents Guide
Table 1
VB 210
(as of
V2.0)
Internal ID see VB 210
(as of V2.0)
Short Message
VB 500
Long Message
VB 501
9 CtrlAgtPoffSwitchOffNetInterfacesFailed failed to switch off network
interfaces failed to switch off network interfaces
10 CtrlAgentPoffNotDone Power off not done
'<hostname>’
Power off not done '<hostname>’ , because node may be
available. Administrator, check availability of node.
11 CtrlAgentPoffHhk_PING_ok ext. SwitchOver-Check
'<hostname>’: ping ok
ext. SwitchOver-Check '<hostname>’: ping ok <specific
message>
12 CtrlAgentPoffHhk_PING_fail ext. SwitchOver-Check
'<hostname>’: ping failed
ext. SwitchOver-Check '<hostname>’: ping failed <specific
message>
13 CtrlAgentPoffHhk_SSH_ok ext. SwitchOver-Check
'<hostname>’: ssh ok
ext. SwitchOver-Check '<hostname>’: ssh ok <specific
message>
14 CtrlAgentPoffHhk_SSH_fail ext. SwitchOver-Check
'<hostname>’: ssh failed
ext. SwitchOver-Check '<hostname>’: ssh failed <specific
message>
15 CtrlAgentMultiFailure_ShortTime MultiNodeFailure: ShortTime. MultiNodeFailure: ShortTime. <specific message>
16 CtrlAgentMultiFailure_LongTime MultiNodeFailure: LongTime. MultiNodeFailure: LongTime. <specific message>
17 CtrlAgentMultiFailure_AllNodes MultiNodeFailure: AllNodes. MultiNodeFailure: AllNodes. <specific message>
18 CtrlAgentNodeLiveMessageFailed Node live message failed
‘<hostname>’. Node live message failed ‘<hostname>’. <specific message>
FlexFrame Agents Traps
FlexFrame Agents Guide 329
Table 1
VB 210
(as of
V2.0)
Internal ID see VB 210
(as of V2.0)
Short Message
VB 500
Long Message
VB 501
19 CtrlAgentNodeNotRechable_SwitchOver_decided Node not reachable, Switch-
Over decided ‘<hostname>’.
Node not reachable, SwitchOver decided ‘<hostname>’.
<specific message>
20 CtrlAgentNoSparePool_defined No spare pool defined. No spare pool defined. <specific message>
21 CtrlAgentNoSpare_to_TakeOver No spare for TakeOver. No spare for TakeOver. <specific message>
22 CtrlAgent_SPCT_SpareList_AddTo Add to list of spare nodes:
‘<hostname>’.
Add to list of spare nodes: ‘<hostname>’. <specific mes-
sage>
23 CtrlAgent_SPCT_SpareList_RemoveFrom Remove from list of spare
nodes: ‘<hostname>’.
Remove from list of spare nodes: ‘<hostname>’. <specific
message>
24 CtrlAgent_SPCT_RequestNotSuccessful Request not successful for
node: ‘<hostname>’.
Request not successful for node: ‘<hostname>’. <specific
message>
25 CtrlAgent_SPCT_MoveNodeSuccessful fa_move_node.sh successful
for node: ‘<hostname>’.
fa_move_node.sh successful for node: ‘<hostname>’. <spe-
cific message>
26 CtrlAgent_SPCT_MoveNodeNotSuccessful
fa_move_node.sh not suc-
cessful for node: ‘<host-
name>’.
fa_move_node.sh not successful for node: ‘<hostname>’.
<specific message>
27 CtrlAgent_VM_LicenseToKillServer Get the license to kill the VM-
Host. VM: ‘<hostname>’.
Get the license to kill the VM-Host. VM: ‘<hostname>’.
<specific message>
FlexFrame Agents Traps
330 FlexFrame Agents Guide
Table 1
VB 210
(as of
V2.0)
Internal ID see VB 210
(as of V2.0)
Short Message
VB 500
Long Message
VB 501
28 CtrlAgent_VM_NOLicenseToKillServer Get no license to kill the VM-
Host. VM: ‘<hostname>’.
Get no license to kill the VM-Host. VM: ‘<hostname>’. <spe-
cific message>
29 CtrlAgent_VM_Message VM-Host check gives a mes-
sage. VM: ‘<hostname>’.
VM-Host check gives a message. VM: ‘<hostname>’. <spe-
cific message>
30 CtrlAgtPoffHost_SD Valuid shutdown valid ‘<hostname>’ shutdown valid: <specific message>
31 CtrlAgtPoffHost_SD_Invalid shutdown invalid ‘<hostname>’ shutdown invalid: <specific message>
32 CtrlAgent_SPCT_PowerCycle_POff_Start power cycle: POff start <specific message>
33 CtrlAgent_SPCT_PowerCycle_POff_End power cycle: POff end <specific message>
34 CtrlAgent_SPCT_PowerCycle_POn_Start power cycle: POn start <specific message>
35 CtrlAgent_SPCT_PowerCycle_POn_End power cycle: POn end <specific message>
36 CtrlAgent_CrashDump_detected crash dump detected ‘<host-
name>’ crash dump detected: <specific message>
37 CtrlAgent_SuspendAutonomySet SuspendAutonomy set ‘<host-
name> SuspendAutonomy set <specific message>
38 CtrlAgent_SuspendAutonomyReset SuspendAutonomy reset
‘<hostname> SuspendAutonomy reset <specific message>
FlexFrame Agents Traps
FlexFrame Agents Guide 331
Table 1
VB 210
(as of
V2.0)
Internal ID see VB 210
(as of V2.0)
Short Message
VB 500
Long Message
VB 501
100 AgentUp has <state> <sender> on node <phys. server> has <state>
101 AgentDown has <state> <sender> on node <phys. server> has <state>
110 NodeShutDown (as of V2.0) node <state> node <phys. server> has <state>
111 NodeRebooting (as of V2.0) node <state> node <phys. server> has <state>
112 NodeRebootStart (as of V2.0) node <state> node <phys. server> has <state>
113 NodeSwitchOver (as of V2.0) node <state> node <phys. server> has <state>
115 NodeTakeOverFailed (as of V2.0) node <state> node <phys. server> has <state>
116 NodeGroupNameWrong node <state> <specific message>
117 NodeSuspendAutonomySet SuspendAutonomy set ‘<host-
name>’ SuspendAutonomy set <specific message>
118 NodeSuspendAutonomyReset SuspendAutonomy reset
‘<hostname>’ SuspendAutonomy reset <specific message>
119 NodeStonithRequest Stonith Request ‘<hostname>’ Stonith Request <specific message>
120 NodeStonithResponce Stonith Responce ‘<host-
name>’ Stonith Responce <specific message>
FlexFrame Agents Traps
332 FlexFrame Agents Guide
Table 1
VB 210
(as of
V2.0)
Internal ID see VB 210
(as of V2.0)
Short Message
VB 500
Long Message
VB 501
200 ServiceStarting is <state> service <service> on node <phys. server> is <state>
201 ServiceUp has <state> service <service> on node <phys. server> has <state>
202 ServiceStartFailed has <state> service <service> on node <phys. server> has <state>
203 ServiceStopping is <state> service <service> on node <phys. server> is <state>
204 ServiceDown has <state> service <service> on node <phys. server> has <state>
205 ServiceStopFailed has <state> service <service> on node <phys. server> has <state>
206 ServiceFailed <state> service <service> on node <phys. server> <state>
207 ServiceRestart (as of V2.0) is <state> service <service> on node <phys. server> is <state>
208 ServiceRestartFailed <state> service <service> on node <phys. server> <state>
209 ServiceWatch (as of V2.0) <state> watching service <service> on node <phys. server>
210 ServiceNowatch (as of V2.0) <state> No longer watching service <service> on node <phys. serv-
er>
211 ServiceReboot (as of V2.0) Is <state> service <service> on node <phys. server> is <state>
212 ServiceRebootStart (as of V2.0) Is <state> service <service> on node <phys. server> is <state>
FlexFrame Agents Traps
FlexFrame Agents Guide 333
Table 1
VB 210
(as of
V2.0)
Internal ID see VB 210
(as of V2.0)
Short Message
VB 500
Long Message
VB 501
213 ServiceSwitchOver (as of V2.0) Is <state> service <service> on node <phys. server> is <state>
214 ServiceSwitchOverStart (as of V2.0) Is <state> service <service> on node <phys. server> is <state>
215 ServiceFlagPhase (as of V8.0) Is <state> service <service> on node <phys. server> is <state>
300 (1)
SubServiceDown is <state> saposcol on node <phys. server> is <state>
300 (2)
<ServiceSubType> of service <service> on node
<phys. server> is <state>
301 SubServiceFailed <state> <ServiceSubType> of service <service> on node
<phys. server> <state>
302 SubServiceDeleted <state> <state> <ServiceSubType> for service <service> on node
<phys. server>
400 CHK_Filers Filer <state> <specific message>
401 CHK_Files File <state> <specific message>
402 CHK_Lock Lock <state> <specific message>
403 CHK_Mounts Mount <state> <specific message>
404 CHK_SRDF_NAS_FH_ErrCode SRDF Check ErrCode <specific message>
FlexFrame Agents Traps
334 FlexFrame Agents Guide
Table 1
VB 210
(as of
V2.0)
Internal ID see VB 210
(as of V2.0)
Short Message
VB 500
Long Message
VB 501
405 CHK_SRDF_NAS_FH_SRDFFailed SRDF failed <specific message>
406 CHK_SRDF_NAS_FH_AdminCall SRDF Admin intervention
necessary <specific message>
410 CHK_Time_Jump time jump detected <specific message>
411 CHK_Time_Gap time comparison gap detected <specific message>
450 PERF_SpaceUtilFS_VExc_MaxUtilization Perf Filesystem 'MaxUtiliza-
tion' exceeded <specific message>
451 PERF_SpaceUtilFS_VExc_MaxGrowthRate Perf Filesystem
'MaxGrowthRate' exceeded <specific message>
500 BB_get node <state> <specific message>
501 BB_DoIt node <state> <specific message>
502 BB_Error node <state> <specific message>
503 BB_NoMatch node <state> <specific message>
FlexFrame Agents Traps
FlexFrame Agents Guide 335
Table 2
VB 210
(as of V2.0) Service VB 200
Service Subtype
VB 201
ID
VB 204
Logical Server Name in
ServerLAN (SL) VB 203
SID
VB 121
Long Instance
No. VB 161
1 – 2 - - - - - -
3
SRV_DBORASRV_DBSAP, SRV_LC
-
- SL <SID> -
SRV_CI, SRV_JC, SRV_SCS,
SRV_ASCS, SRV_APP, SRV_J <ID> SL <SID> <Inst No>
SRV_ERS <ID> - <SID> <Inst No>
4
SRV_DBORASRV_DBSAP, SRC_LC
-
- SL <SID> -
SRV_CI, SRV_JC, SRV_SCS, RV_ASCS,
SRV_APP, SRV_J, SRV_ERS <ID> SL <SID> <Inst No>
SRV_ERS <ID> - <SID> <Inst No>
5 – 10,
18 – 19,
22 – 31,
36,
37 – 38
<hostname> - - - - -
FlexFrame Agents Traps
336 FlexFrame Agents Guide
Table 2
VB 210
(as of V2.0) Service VB 200
Service Subtype
VB 201
ID
VB 204
Logical Server Name in
ServerLAN (SL) VB 203
SID
VB 121
Long Instance
No. VB 161
1-2,
11-17, 20 -
21
32 – 35
100 – 101,
110 – 113,
115 - 116,
117 - 118
119 - 120
- - - - - -
Table 2
VB 210
(as of V2.0) Service VB 200
Service Subtype
VB 201
ID
VB 204
Logical Server Name in
ServerLAN (SL) VB 203
SID
VB 121
Long Instance
No. VB 161
200 – 215
SRV_DBORASRV_DBSAP, SRV_LC
-
- SL <SID> -
SRV_CI, SRV_JC, SRV_SCS,
SRV_ASCS, SRV_APP, SRV_J <ID> SL <SID> <Inst No>
SRV_ERS <ID> - <SID> <Inst No>
300 (1)
- saposcol - SL - -
FlexFrame Agents Traps
FlexFrame Agents Guide 337
300 (2)
SRV_DBSAP, SRV_LC vserver, kernel
- SL - - SRV_DBORA
ora_dbw, ora_lgwr,
ora_ckpt, ora_smon,
ora_pmon, tnslsnr,
ora_mman, ora_arc
SRV_ERS ers.sap <ID> - <SID> <Inst No>
SRV_CI ms.sap, dw.sap
<ID> SL <SID> <Inst No>
SRV_JC jc.sap
SRV_SCS, SRV_ASCS ms.sap, en.sap
SRV_APP dw.sap
SRV_J jc.sap
Table 2
VB 210
(as of V2.0) Service VB 200
Service Subtype
VB 201
ID
VB 204
Logical Server Name in
ServerLAN (SL) VB 203
SID
VB 121
Long Instance
No. VB 161
301
SRV_DBORASRV_DBSAP, SRV_LC
server-lan ping
- SL <SID> -
SRV_CI, SRV_JC, SRV_SCS,
SRV_ASCS, SRV_APP, SRV_J <ID> SL <SID> <Inst No>
SRV_CI, SRV_JC, SRV_SCS,
SRV_ASCS, SRV_APP, SRV_J client-lan ping <ID> SL <SID> <Inst No>
302 SRV_DBORASRV_DBSAP, SRV_LC control file - SL <SID> -
FlexFrame Agents Traps
338 FlexFrame Agents Guide
SRV_CI, SRV_JC, SRV_SCS,
SRV_ASCS, SRV_APP, SRV_J <ID> SL <SID> <Inst No>
SRV_ERS <ID> - <SID> <Inst No>
400 – 406,
410 – 411,
450 – 451,
500
- - - - - -
501 – 503
SRV_DBORASRV_DBSAP, SRV_CI,
SRV_JC, SRV_SCS, SRV_ASCS,
SRV_APP, SRV_J, SRV_ERS, SRV_LC
- <ID> SL <SID> <Inst No>
FlexFrame Agents Traps
FlexFrame Agents Guide 339
Table 3
VB 210
(as of V2.0) State VB 202 Sender VB 205 Long Severity VB 170
1 started myAMC_FA_[App]|[Frame]|[Ctrl]Agent 50
2 shut down myAMC_FA_[App]|[Frame]|[Ctrl]Agent 150
3 - myAMC_FA_[App]|[Frame]|[Ctrl]Agent 150
4 failed myAMC_FA_[App]|[Frame]|[Ctrl]Agent 250
5 poff ok myAMC_FA_[App]|[Frame]|[Ctrl]Agent 250
6 poff failed myAMC_FA_[App]|[Frame]|[Ctrl]Agent 250
7 poff not done myAMC_FA_[App]|[Frame]|[Ctrl]Agent 250
8 netoff ok myAMC_FA_[App]|[Frame]|[Ctrl]Agent 250
9 netoff failed myAMC_FA_[App]|[Frame]|[Ctrl]Agent 250
10 poff not done myAMC_FA_[App]|[Frame]|[Ctrl]Agent 150
11 ping ok myAMC_FA_[App]|[Frame]|[Ctrl]Agent 150
12 ping failed myAMC_FA_[App]|[Frame]|[Ctrl]Agent 50
13 ssh ok myAMC_FA_[App]|[Frame]|[Ctrl]Agent 150
14 ssh failed myAMC_FA_[App]|[Frame]|[Ctrl]Agent 50
15 MultiNodeFailure: ShortTime myAMC_FA_[App]|[Frame]|[Ctrl]Agent 275
FlexFrame Agents Traps
340 FlexFrame Agents Guide
Table 3
VB 210
(as of V2.0) State VB 202 Sender VB 205 Long Severity VB 170
16 MultiNodeFailure: LongTime myAMC_FA_[App]|[Frame]|[Ctrl]Agent 275
17 MultiNodeFailure: AllNodes myAMC_FA_[App]|[Frame]|[Ctrl]Agent 275
18 Node live message failed myAMC_FA_[App]|[Frame]|[Ctrl]Agent 150
19 Node not reachable myAMC_FA_[App]|[Frame]|[Ctrl]Agent 250
20 No spare pool defined myAMC_FA_[App]|[Frame]|[Ctrl]Agent 250
21 No spare for TakeOver myAMC_FA_[App]|[Frame]|[Ctrl]Agent 250
22 Add to list of spare nodes myAMC_FA_[App]|[Frame]|[Ctrl]Agent 50
23 Remove from list of spare nodes myAMC_FA_[App]|[Frame]|[Ctrl]Agent 50
24 Request not successful for node myAMC_FA_[App]|[Frame]|[Ctrl]Agent 250
25 fa_move_node.sh successful for node myAMC_FA_[App]|[Frame]|[Ctrl]Agent 50
26 fa_move_node.sh not successful for
node myAMC_FA_[App]|[Frame]|[Ctrl]Agent 250
27 Get the license to kill the VM-Host. VM myAMC_FA_[App]|[Frame]|[Ctrl]Agent 250
28 Get no license to kill the VM-Host. VM myAMC_FA_[App]|[Frame]|[Ctrl]Agent 250
29 VM-Host check gives a message. VM myAMC_FA_[App]|[Frame]|[Ctrl]Agent 250
FlexFrame Agents Traps
FlexFrame Agents Guide 341
Table 3
VB 210
(as of V2.0) State VB 202 Sender VB 205 Long Severity VB 170
30 shutdown valid myAMC_FA_[App]|[Frame]|[Ctrl]Agent 50
31 shutdown invalid myAMC_FA_[App]|[Frame]|[Ctrl]Agent 250
32 POff start myAMC_FA_[App]|[Frame]|[Ctrl]Agent 50
33 POff end myAMC_FA_[App]|[Frame]|[Ctrl]Agent 50
34 POn start myAMC_FA_[App]|[Frame]|[Ctrl]Agent 50
35 POn end myAMC_FA_[App]|[Frame]|[Ctrl]Agent 50
36 crash dump detected myAMC_FA_[App]|[Frame]|[Ctrl]Agent 150
37 SuspendAutonomy set myAMC_FA_[App]|[Frame]|[Ctrl]Agent 250
38 SuspendAutonomy reset myAMC_FA_[App]|[Frame]|[Ctrl]Agent 250
100 started myAMC_FA_[App]|[Frame]|[Ctrl]Agent 50
101 shut down myAMC_FA_[App]|[Frame]|[Ctrl]Agent 150
110 shut down myAMC_FA_[App]|[Frame]|[Ctrl]Agent 150
111 reboot (down) myAMC_FA_[App]|[Frame]|[Ctrl]Agent 150
112 reboot (up) myAMC_FA_[App]|[Frame]|[Ctrl]Agent 150
113 switch over myAMC_FA_[App]|[Frame]|[Ctrl]Agent 150
FlexFrame Agents Traps
342 FlexFrame Agents Guide
Table 3
VB 210
(as of V2.0) State VB 202 Sender VB 205 Long Severity VB 170
115 take over failed myAMC_FA_[App]|[Frame]|[Ctrl]Agent 250
116 groupname wrong myAMC_FA_[App]|[Frame]|[Ctrl]Agent 250
117 SuspendAutonomy set myAMC_FA_[App]|[Frame]|[Ctrl]Agent 250
118 SuspendAutonomy reset myAMC_FA_[App]|[Frame]|[Ctrl]Agent 250
119 Stonith Request myAMC_FA_[App]|[Frame]|[Ctrl]Agent 250
120 Stonith Responce myAMC_FA_[App]|[Frame]|[Ctrl]Agent 250
200 starting myAMC_FA_[App]|[Frame]|[Ctrl]Agent 150
201 started myAMC_FA_[App]|[Frame]|[Ctrl]Agent 50
202 failed to start myAMC_FA_[App]|[Frame]|[Ctrl]Agent 250
203 stopping myAMC_FA_[App]|[Frame]|[Ctrl]Agent 150
204 stopped myAMC_FA_[App]|[Frame]|[Ctrl]Agent 50
205 failed to stop myAMC_FA_[App]|[Frame]|[Ctrl]Agent 250
206 failed myAMC_FA_[App]|[Frame]|[Ctrl]Agent 250
207 restart myAMC_FA_[App]|[Frame]|[Ctrl]Agent 150
208 failed to restart myAMC_FA_[App]|[Frame]|[Ctrl]Agent 250
FlexFrame Agents Traps
FlexFrame Agents Guide 343
Table 3
VB 210
(as of V2.0) State VB 202 Sender VB 205 Long Severity VB 170
209 watch myAMC_FA_[App]|[Frame]|[Ctrl]Agent 50
210 no watch myAMC_FA_[App]|[Frame]|[Ctrl]Agent 50
211 reboot myAMC_FA_[App]|[Frame]|[Ctrl]Agent 150
212 reboot (start) myAMC_FA_[App]|[Frame]|[Ctrl]Agent 150
213 swich over myAMC_FA_[App]|[Frame]|[Ctrl]Agent 150
214 switch over (start) myAMC_FA_[App]|[Frame]|[Ctrl]Agent 150
215 PHASE-flag: <text> myAMC_FA_[App]|[Frame]|[Ctrl]Agent 150
300 (1)
down myAMC_FA_[App]|[Frame]|[Ctrl]Agent 150
300 (2)
down myAMC_FA_[App]|[Frame]|[Ctrl]Agent 250
301 failed (<nr>) myAMC_FA_[App]|[Frame]|[Ctrl]Agent 250
302 delete myAMC_FA_[App]|[Frame]|[Ctrl]Agent 150
400 failed myAMC_FA_[App]|[Frame]|[Ctrl]Agent 250
401 failed myAMC_FA_[App]|[Frame]|[Ctrl]Agent 250
402 failed myAMC_FA_[App]|[Frame]|[Ctrl]Agent 250
403 failed myAMC_FA_[App]|[Frame]|[Ctrl]Agent 250
FlexFrame Agents Traps
344 FlexFrame Agents Guide
Table 3
VB 210
(as of V2.0) State VB 202 Sender VB 205 Long Severity VB 170
404 SRDF Check ErrCode myAMC_FA_[App]|[Frame]|[Ctrl]Agent 250/275
405 SRDF failed myAMC_FA_[App]|[Frame]|[Ctrl]Agent 250/275
406 SRDF Admin intervention necessary myAMC_FA_[App]|[Frame]|[Ctrl]Agent 275
410 time jump myAMC_FA_[App]|[Frame]|[Ctrl]Agent 150/250
411 time comparison gap myAMC_FA_[App]|[Frame]|[Ctrl]Agent 150/250
450 Perf Filesystem 'MaxUtilization' exceed-
ed myAMC_FA_[App]|[Frame]|[Ctrl]Agent 250
451 Perf Filesystem 'MaxGrowthRate' ex-
ceeded myAMC_FA_[App]|[Frame]|[Ctrl]Agent 150
500 BB get cmd myAMC_FA_[App]|[Frame]|[Ctrl]Agent 50
501 BB do cmd myAMC_FA_[App]|[Frame]|[Ctrl]Agent 50
502 BB cmd error myAMC_FA_[App]|[Frame]|[Ctrl]Agent 50, 150, 250
503 BB cmd no match myAMC_FA_[App]|[Frame]|[Ctrl]Agent 50, 150, 250
FlexFrame Agents Guide 345
10 FlexFrame Troubleshooting
10.1 General
The FlexFrame Agents offer a large number of diagnostic options for detecting and diag-
nosing problems on the FlexFrame Agents themselves or other components.
Problems concerning FlexFrame Agents can be assigned to one of the following catego-
ries:
FlexFrame installation and configuration errors
Parameter errors
Configuration errors
Detection, reaction errors, start, stop, maintenance errors
Power-shutdown errors
Typical consequences of installation and configuration errors are:
FlexFrame Agents fail to start
Error messages during startup of FlexFrame Agents
10.2 Incident Diagnosis
Error: Mount points missing
Diagnosis:
In the case of mount points monitored by FlexFrame Agents, traps are sent to the
central trap consoles. With other mount points which are absolutely essential for the
operation of the node concerned it can happen that the agents cannot be started as
the directories required are not available.
Response:
Provide the mount points required with the appropriate mount options.
Error: Mount points without “File Locking”
Diagnosis:
The FlexFrame Agents log this situation both in the operating system’s Syslog and in
special files (/opt/myAMC/vFF/log/log_syslog*).
Response: Provide the mount points required with the appropriate mount options (lock).
Error: Rights for the directories/files are not sufficient
FlexFrame Troubleshooting
346 FlexFrame Agents Guide
Diagnosis:
In the case of files monitored by FlexFrame Agents, traps are sent to the central trap
consoles. With other directories/files which are absolutely essential for the operation
of the node concerned it can happen that the agents cannot be started as the directo-
ries/files required are not available.
Response:
Provide the directories/files with the required rights.
Error: Agents do not have the authorization to write to the
directories assigned
Diagnosis:
The FlexFrame Agents work and log files are not written.
Response:
Provide the directories/files with the required rights.
Error: Version incompatibility
FlexFrame installation and FlexFrame Agents Suite installation are not directly com-
patible. This can always be the case when older FlexFrame installations are updated
with new FlexFrame Agents Suite.
Diagnosis:
For diagnosis and troubleshooting, the mount points, the directory structure and the
access rights to the directories used by the agents must be checked.
Response:
Using the command line tool, check that the parameters used in the FlexFrame
Agents config files are compatible with the version and syntactically correct.
Error: Pool assignment not found
A node is assigned to the wrong pool or to the default pool.
Diagnosis:
Display on the FA WebGUI or in the agent’s start trap and display on an event con-
sole.
Response:
Check the LDAP configuration parameters, call the PGTool Pool.sh and check the
pool name returned.
Check the pool membership for each node.
Error: Group assignment is not correct
FlexFrame Troubleshooting
FlexFrame Agents Guide 347
Diagnosis:
Display on the FA WebGUI or in the agent’s start trap and display on an event con-
sole.
Response:
Check the group configuration in the group configuration file. Check the group mem-
bership for each node.
Error: Service priority not recognized
Diagnosis:
Display on the FA WebGUI or in the agent’s start trap and display on an event con-
sole.
Response:
Check the configuration of the service class and service priority in the group configu-
ration file. Check the values for each node.
Error: Availability problem not rectified by autonomous reaction
Diagnosis:
Services are discontinued (possibly due to hardware fault) and are not made avai-
lable again by FlexFrame Agents.
Response:
Check whether nodes are available for taking over the services (Spare Nodes).
Check whether the FlexFrame Agents on the nodes involved have been started.
Error: Services do not start
Constant reboot
Permanent switchover
Diagnosis:
SAP services which are started do not enter run mode but are repreatedly restarted
or, if the problem escalates, the node is rebooted or an internal SwitchOver takes
place. Possible causes:
The MaxRestart time for the service is too short. This parameter can be adjusted in
the FA configuration.
The virtual interfaces cannot be reached.
There is a permanent problem which prevents a service being started (e.g. neces-
sary database recovery).
Response:
Stop the FlexFrame Agents to interrupt escalation of the reaction and check whether
the service can be started manually.
If the service cannot be started manually, this problem must be corrected by the ad-
FlexFrame Troubleshooting
348 FlexFrame Agents Guide
ministrator.
If the service can be started manually, the time required for this must be matched to
the MaxStart time and MaxRestart time in the configuration and the configuration
must be adjusted, if necessary.
If the virtual interfaces cannot be reached from the Application Agent, the network
configuration must be checked.
Error: Service cannot be stopped
Diagnosis:
An active SAP service is repeatedly restarted after a manual stop command .
Response:
If the FlexFrame SAP scripts were not used for the manual stop command, this is the
cause. The FlexFrame SAP scripts must be used.
If the FlexFrame SAP scripts were used, the Monitor Alert Script might not be availa-
ble or does not have the required rights.
However, it is also possible that the Monitor Alert Time and CycleTime are configured
incorrectly. The agents’ CycleTime is too long in relation to the Monitor Alert Time.
The Monitor Alert Time must be at least 3 * the CycleTime.
Error: Maintenance activities are interrupted by autonomous
reactions
Diagnosis:
Unwanted autonomous reactions during maintenance.
Response:
Set NoWatch for the service concerned or stop the Application Agents for the node
concerned and restart them after maintenance has been completed.
Error: Incorrect display on the FlexFrame WebGUI
Diagnosis:
The state checked manually does not match the display.
Response:
Check whether the AppAgents concerned, the CtrlAgents concerned and the web
server are running properly for the WebInterface.
FlexFrame Troubleshooting
FlexFrame Agents Guide 349
Error: Group “GR_default”
Diagnosis:
There the group "GR_default" is displayed instead of the actual group.
Response:
The measure depends on the cause:
Cause "AN comes new in the group added, LDAP is temporarily unavailable":
Response:
Regenerate group information on the affected AN from LDAP with the following
command: /opt/myAMC/FA_AppAgent/PGTool_CreateGroupFileFromLDAP.sh
<pool> default_LDAP
Verify the group membership of the AN with the following command: /opt/myAMC/FA_AppAgent/PGTool_Group.sh
Restart App and CtrlAgenten of the affected pool (see below).
Cause "The configuration file "myAMC_FA_Groups.xml" is corrupt or empty:
/opt/myAMC/VFF/vFF_<PoolName>/config/myAMC_FA_Groups.xml
Response: Restore From Backup / Snapshot:
If a backup or snapshot is available, restore the file from the backup / snapshot.
Restart App and CtrlAgenten of the affected pool (see below).
Response: Restore from template:
If no backup or snapshot exists, take the file from the template and submit custom
changes manually.
This can be done by the following steps:
Copy file from the template in the pool:
cp /opt/myAMC/VFF/Common/.vFF_template/config/myAMC_FA_Groups.xml
/opt/myAMC/VFF/vFF_<PoolName>/config/myAMC_FA_Groups.xml
If service-priorities /-PowerValues were customized by the user, they must be
reconfigured (ConfigGUI).
Regenerate the group information on an affected AN from LDAP with the follow-
ing command:
FlexFrame Troubleshooting
350 FlexFrame Agents Guide
/opt/myAMC/FA_AppAgent/PGTool_CreateGroupFileFromLDAP.sh
<pool> default_LDAP
Verify the group membership of the AN with the following command: /opt/myAMC/FA_AppAgent/PGTool_Group.sh
Restart App and CtrlAgenten of the affected pool (see below).
Restart the App- and CtrlAgent of the affected pool:
Note: In the time where the FlexFrame Agents are restarted the SAP systems are not
monitored from FlexFrame Agents. When a SAP service will fail, there will be no re-
sponce.
ff_ha_cmd.sh unmanage ff_manage
/opt/myAMC/scripts/allnodes <PoolName>
"/opt/myAMC/FA_AppAgent/myAMC.FA_AppAgent stop"
/opt/myAMC/FA_CtrlAgent/myAMC.FA_CtrlAgent stop <PoolName>
/opt/myAMC/scripts/allnodes <PoolName>
"/opt/myAMC/FA_AppAgent/myAMC.FA_AppAgent start">
/opt/myAMC/FA_CtrlAgent/myAMC.FA_CtrlAgent start <PoolName>
ff_ha_cmd.sh manage ff_manage
FlexFrame Troubleshooting
FlexFrame Agents Guide 351
10.3 FlexFrame Agents Work and Log Files
10.3.1 General
The FlexFrame Agents write detailed log files.
The log files of the FlexFrame Agents can be used for the diagnosis.
The functions of the FlexFrame Agents are documented in their own files. These files
may not be modified manually as this can impair fault-free operation of the FlexFrame
Agents or lead to erroneous reactions.
These files are created dynamically during ongoing operation. Deleting these files leads
to a status in which the FlexFrame Agents reorganize themselves, and from this point
they reevaluate the situation from the current viewpoint without any previous knowledge.
10.3.2 Overview, Principal Directories, Files
Base directory: /opt/myAMC/
Base directory: /opt/local/myAMC/
Version numbers: In the table below, V<v>K<r> corresponds to
V<version number>K<revision_number>.
Subdirectories Content
./scripts Scripts for various tasks
./scripts/sap Link to the FlexFrame scripts
./scripts/PowerMng Scripts for the power managemnet
blades.
./scripts/ShutDown_Node Scripts to shutdown a node.
./scripts/fa_list_services.sh Script to list all FlexFrame service states.
./scripts/allnodes Script to execute a command at all nodes
of a pool.
./config General configuration data
FlexFrame Troubleshooting
352 FlexFrame Agents Guide
Subdirectories Content
./config/FA_WebGui.conf General settings for the FlexFrame
Agents WebGUI (directories, cycle times,
database settings)
./config/amc-users.xml User management
./FA_AppAgent Installation path of FlexFrame AppAgent
and of diverse scripts.
./FA_AppAgent/myAMC.FA_AppAgent Start/Stop scripts FlexFrame AppAgent
./FA_AppAgent/PGTool_Pool.sh Determination of pool membership
./FA_AppAgent/PGTool_Version.sh Determination of pool version
./FA_AppAgent/PVget.sh Determination of the powervalue [SAPs-
äquivalent] number of a node
./FA_ AppAgent/BBTool.sh BlackBoard control
./FA_ AppAgent/BBT_dialog.sh BlackBoard dialog mode control
./FA_AppAgent/bin_Linux_SLES9_<VxxKyy>.
/FA_AppAgent/lib_Linux_SLES9_<VxxKyy>
Binaries and libraries for each version.
./FA_AppAgent/config FlexFrame AppAgent-specific configura-
tion data
./FA_AppAgent/log empty
./FA_CtrlAgent Installation path of FlexFrame CtrlAgent
and of scripts
./FA_CtrlAgent/myAMC.FA_AppAgent Start/StopScripts FlexFrame CtrlAgent
./FA_CtrlAgent/PGTool_Pool.sh Determination of pool membership
./FA_CtrlAgent/PGTool_Version.sh Determination of pool version
./FA_CtrlAgent/PVget.sh Determination of the powervalue [SAPs-
äquivalent] number of a node
./FA_CtrlAgent/BBTool.sh BlackBoard control
./FA_CtrlAgent/BBT_dialog.sh BlackBoard dialog mode control
FlexFrame Troubleshooting
FlexFrame Agents Guide 353
Subdirectories Content
./FA_CtrlAgent/bin_Linux_SLES9_<VxxKyy>
./FA_CtrlAgent/lib_Linux_SLES9_<VxxKyy>
Binaries and libraries for each version.
./FA_CtrlAgent/config FlexFrame CtrlAgent-specific configura-
tion data
./FA_CtrlAgent/log empty
./FA_FrameAgent Installation path of FlexFrame
FrameAgent and of diverse scripts.
./FA_FrameAgent/myAMC.FA_FrameAgent Start/Stop scripts FrameAgent
./FA_FrameAgent/PGTool_Pool.sh Determination of pool membership
./FA_FrameAgent/PGTool_Version.sh Determination of pool version
./FA_FrameAgent/PVget.sh Determination of the powervalue [SAPs-
äquivalent] number of a node
./FA_ FrameAgent/BBTool.sh BlackBoard control
./FA_ FrameAgent/BBT_dialog.sh BlackBoard dialog mode control
./FA_FrameAgent/bin_Linux_SLES9_<VxxKyy
>./FA_FrameAgent/lib_Linux_SLES9_<VxxKy
y>
Binaries and libraries for each version.
./FA_FrameAgent/config FlexFrame FrameAgent-specific configu-
ration data
./FA_FrameAgent/log empty
./FA_ConfigGui Installation path of FlexFrame ConfigGui.
./FA_DomainManager Installation path of FlexFrame Domain-
Manager.
./FA_LogAgent Installation path of FlexFrame LogAgent.
./FA_Messenger Installation path of FlexFrame Messenger.
FlexFrame Troubleshooting
354 FlexFrame Agents Guide
Subdirectories Content
./FA_ Messenger/log/Robot.txt Protocolfile of received messages.
./FA_WebGui Installation path of FlexFrame WebGui.
./vFF Pool-specific (vFF) data
./vFF/SpareRequest Request directory for spares from spare
pool
./vFF/log Pool-specific log files
./vFF/log/myAMC_FA_Pools.xml
./vFF/log/myAMC_FA_Pools-default.xml
Pools configuration file and its default
version (this is used as LDAP cache)
./vFF/Common/.vFF_template.<VxxKxx> Template of pool-specific data for each
version
./vFF/Common/.vFF_template.<VxxKxx>/
config
Configuration
./vFF/Common/.vFF_template.<VxxKxx>/
config/TrapTargets.xml
Trap targets
./vFF/Common/.vFF_template.<VxxKxx>/
config/myAMC_FA.xml
./vFF/Common/.vFF_template.<VxxKxx>/
config/myAMC_FA-default.xml
FlexFrame Agents configuration and
delivery status (default).
./vFF/Common/.vFF_template.<VxxKxx>/
config/myAMC_FA_Checks.xml
./vFF/Common/.vFF_template.<VxxKxx>/
config/myAMC_FA_Checks-default.xml
Definition of „additional Checks“ and
delivery status (default).
./vFF/Common/.vFF_template.<VxxKxx>/
config/myAMC_FA_GUI.xml
./vFF/Common/.vFF_template.<VxxKxx>/
config/myAMC_FA_GUI-default.xml
FlexFrame GUI configuration and delivery
status (default).
FlexFrame Troubleshooting
FlexFrame Agents Guide 355
Subdirectories Content
./vFF/Common/.vFF_template.<VxxKxx>/
config/myAMC_FA_Groups.xml
./vFF/Common/.vFF_template.<VxxKxx>/
config/myAMC_FA_Groups-default.xml
FlexFrame groups configuration and
delivery status (default).
This is used as LDAP cache
./vFF/Common/.vFF_template.<VxxKxx>/
config/myAMC_FA_Hosts.xml
./vFF/Common/.vFF_template.<VxxKxx>/
config/myAMC_FA_Hosts-default.xml
Host configuration from LDAP and deliv-
ery status (default).
This is used as LDAP cache
./vFF/Common/.vFF_template.<VxxKxx>/
config/myAMC_FA_SD_Sec.xml
./vFF/Common/.vFF_template.<VxxKxx>/
config/myAMC_FA_SD_Sec-default.xml
FlexFrame shutdown security configura-
tion and delivery status (default).
./vFF/Common/.vFF_template.<VxxKxx>/
config/myAMC_FA_Rules.xml
./vFF/Common/.vFF_template.<VxxKxx>/
config/myAMC_FA_Rules-default.xml
FlexFrame service rules and delivery
status (default).
./vFF/Common/.vFF_template.<VxxKxx>/
config/myAMC_FA_SelectionRules.xml
./vFF/Common/.vFF_template.<VxxKxx>/
config/myAMC_FA_SelectionRules-
default.xml
Definition of „SelectionRules“ for „Pool
independent spare node“ and delivery
status (default).
./vFF/Common/.vFF_template.<VxxKxx>/log
./vFF/Common/.vFF_template.<VxxKxx>/log
/AppAgt
./vFF/Common/.vFF_template.<VxxKxx>/log
/CtlrAgt
./vFF/Common/.vFF_template.<VxxKxx>/log
/FrameAgt
Logfiles for each pool of:
FlexFrame AppAgent
FlexFrame CtrlAgent
FlexFrame FrameAgent
./vFF/Common/.vFF_template.<VxxKxx>/
data
./vFF/Common/.vFF_template.<VxxKxx>/
data/FA
Work files
./vFF/Common/.vFF_template.<VxxKxx>/
data/FA/livelist
Live list
livelist.log
./vFF/Common/.vFF_template.<VxxKxx>/
data/FA/xmlrepository
XML repository for the web interface
livelist.
xmlServices_<nodename>.xml
FlexFrame Troubleshooting
356 FlexFrame Agents Guide
Subdirectories Content
./vFF/Common/.vFF_template.<VxxKxx>/
data/FA/servicelists
Service lists
Services_<nodename>.lst
./vFF/Common/.vFF_template.<VxxKxx>/
data/FA/servicelogs
Service logs (history)
Services_<nodename>.log
./vFF/Common/.vFF_template.<VxxKxx>/
data/FA/reboot
Reboot files
Reboot_<nodename>.lst
Persistent FA-Info
Persist_FA_Info.txt
Persistent SuspendAutonomy log-file &
marker
Persist_SuspensAutonomy.log
Persist_SuspensAutonomy.marker
./vFF/Common/.vFF_template.<VxxKxx>/
data/FA/switchover
SwitchOver files
SwitchOver_<nodename>.lst
./vFF/Common/.vFF_template.<VxxKxx>/
data/FA/blackboard
BlackBoard
blackboard.txt
./vFF/Common/.vFF_template.<VxxKxx>/
data/FA/performance
Measured performance data
./vFF/Common/.vFF_Master Pool-Master with the same directories and
files as in template
./vFF/vFF_Cust_1
./vFF/vFF_Cust_1/config
./vFF/vFF_Cust_1/log/…
./vFF/vFF_Cust_1/data
./vFF/vFF_Cust_1/data/FA/….
Pool-specific data for pool “Cust_1”
(example).
See above for the description of the sub-
directories and files.
./vFF/vFF_Cust_2
./vFF/vFF_Cust_2/config
./vFF/vFF_Cust_2/log/…
./vFF/vFF_Cust_2/data
./vFF/vFF_Cust_2/data/FA/….
Pool-specific data for pool “Cust_2”
(example).
See above for the description of the sub-
directories and files.
FlexFrame Troubleshooting
FlexFrame Agents Guide 357
10.3.3 Collecting Diagnostic Information for Support Assis-tance
If support is needed, there is special data needed in the FlexFrame Support. This infor-
mation is required to analyze problems with FlexFrame and the FlexFrame Agents.
Error description, as precise as possible
What is the problem or error? On which pool/node/service does it occur?
Version of the FlexFrame Agents installed
Run rpm -qa | grep myAMC on the Control Node.
Configuration, work and log files of the FlexFrame Agents
The following script creates an archive with the desired information:
/opt/myAMC/FA_CtrlAgent/SAVE_FA_files_for_diag.sh
This script must be invoked on the control- node!
cd /opt/myAMC/FA_CtrlAgent
./SAVE_FA_files_for_diag.sh
Usage
cd /opt/myAMC/FA_CtrlAgent
./SAVE_FA_files_for_diag.sh –h usage: SAVE_FA_files_for_diag.sh <args>
-h (show this help)
-C (save config only)
-c "<customer name>" specify customer name
-i "<incident number>" specify incident number
-d "<problem description>" specify problem description
-P "<pools>" specify pools
-p <y Y n N> include performance files [Y/N]
-g <y Y n N> compress output file [Y/N]
Attention: The usage of the following '-D' option will delete log- or/and perfor-
mance-files from your system.
If you use accounting there may be lost performance-data.
-D LogFiles Delete log files
-D PerfFiles Delete performance files
-D LogAndPerfFiles Delete log and performance files
Please ensure, that this script will be started at the active Ctrl-Node (CtrlAgent
is active).
FlexFrame Troubleshooting
358 FlexFrame Agents Guide
The functions of the FlexFrame Agents are documented in various files. These files may
not be changed manually as this can impair error-free operation of the FlexFrame Agents
or result in incorrect reactions.
These files are created dynamically during ongoing operation. Deleting these files leads
to a status in which the FlexFrame Agents reorganize themselves, and from this point
they re-evaluate the situation from the current viewpoint without any previous knowledge.
FlexFrame Troubleshooting
FlexFrame Agents Guide 359
10.3.4 Selected Files
The write cycle for the entries (with the exception of reboot, switchover and BlackBoard)
and the storage location of the files described in the following are defined using a param-
eter in the configuration file myAMC_FA.xml.
10.3.4.1 Livelist
Each FlexFrame AppAgent regularly enters itself in this list. Through these entries the
FlexFrame CtrlAgent recognizes whether the various FlexFrame AppAgents are availa-
ble and functioning without error.
10.3.4.2 Services List
This file (testament) exists for each FlexFrame AppAgent on a node-specific basis. In it
the agent logs the services which it has detected using its detectors plus their current
status. A service-related status is logged in this file. The contents are updated after each
detector cycle.
10.3.4.3 Services Log
The contents of this file are identical to those of the Services-List file, with the differ-
ence that the history is contained in this file. This enables status changes and reaction
decisions to be detected and replicated.
10.3.4.4 Reboot
The contents of this file are identical to those of the Services-List file. The file serves
as an information storage when a reboot takes place. It is written only for the autonomous
reaction reboot and is deleted again after the reboot has been completed and the ser-
vices have been started up.
10.3.4.5 Switchover
The contents of this file are identical to those of the Services-List file. The file serves
as information storage (testament) when a switchover takes place. It is written only for the
autonomous reaction switchover and is deleted again after the services have been taken
over.
10.3.4.6 XML Repository
In terms of contents the files in the XML Repository are the same as those in the Livelist
and Services List. By contrast, the contents are written in XML notation and can thus be
visualized directly with the associated FlexFrame Agents WebInterface.
The write cycle for the entries and the storage location of the file are defined using a
parameter in the configuration file myAMC_FA.xml.
FlexFrame Troubleshooting
360 FlexFrame Agents Guide
10.3.4.7 BlackBoard
The BlackBoard is an input interface for the FlexFrame Agents. Commands can be en-
tered here which are executed by the FlexFrame AppAgents. The commands have a
specific validity period and are secured against manipulation. The file is written manually
using a tool which guarantees, among other things, protection against manipulation.
10.3.5 FlexFrame Diagnostic Tool
Manual diagnosis of the log files can be very time-consuming. The Fujitsu support organ-
ization works with specialized diagnostic tools which can analyze even large quantities of
data very quickly and efficiently. This service can be utilized when required and if a corre-
sponding service agreement exists.
To utilize this service, either individual log files or the entire virtual FlexFrame Agentsdi-
rectory of a pool can be sent to the support department, e.g. as a compressed and pro-
tected zip archive.
FlexFrame Troubleshooting
FlexFrame Agents Guide 361
10.3.6 Collecting Diagnostic Information for Support Assis-tance
If support is needed, there is special data needed in the FlexFrame Support. This infor-
mation is required to analyze problems with FlexFrame and the FlexFrame Agents.
Error description, as precise as possible
What is the problem or error? On which pool/node/service does it occur?
Version of the FlexFrame Agents installed
Run rpm -qa | grep myAMC on the Control Node.
Configuration, work and log files of the FlexFrame Agents
The following script creates an archive with the desired information:
/opt/myAMC/FA_CtrlAgent/SAVE_FA_files_for_diag.sh
This script must be invoked on the control- node!
cd /opt/myAMC/FA_CtrlAgent
./SAVE_FA_files_for_diag.sh
Usage
cd /opt/myAMC/FA_CtrlAgent
./SAVE_FA_files_for_diag.sh –h usage: SAVE_FA_files_for_diag.sh <args>
-h (show this help)
-C (save config only)
-c "<customer name>" specify customer name
-i "<incident number>" specify incident number
-d "<problem description>" specify problem description
-P "<pools>" specify pools
-p <y Y n N> include performance files [Y/N]
-g <y Y n N> compress output file [Y/N]
Attention: The usage of the following '-D' option will delete log- or/and perfor-
mance-files from your system.
If you use accounting there may be lost performance-data.
-D LogFiles Delete log files
-D PerfFiles Delete performance files
-D LogAndPerfFiles Delete log and performance files
Please ensure, that this script will be started at the active Ctrl-Node (CtrlAgent
is active).
FlexFrame Troubleshooting
362 FlexFrame Agents Guide
The functions of the FlexFrame Agents are documented in various files. These files may
not be changed manually as this can impair error-free operation of the FlexFrame Agents
or result in incorrect reactions.
These files are created dynamically during ongoing operation. Deleting these files leads
to a status in which the FlexFrame Agents reorganize themselves, and from this point
they re-evaluate the situation from the current viewpoint without any previous knowledge.
FlexFrame Agents Guide 363
11 Abbreviations
ABAP Advanced Business Application Programming
ACI Adaptive Computing Infrastructure
ACPI Advanced Configuration and Power Interface
APM Advanced Power Management
APOLC Advanced Planner & Optimizer Life Cache
CCU Console Connection Unit
cDOT Clustered Data ONTAP
CIFS Common Internet File System
DART Data Access in Real Time
DHCP Dynamic Host Configuration Protocol
DIT Domain Information Tree
ERP Enterprise Resource Planning
ESF Enhanced System Facility
EULA End User License Agreement
FAA FlexFrame Autonomous Agent
FC Fiber Channel
FTP File Transfer Protocol
IP Internet Protocol
IPMP IP Multipathing
KVM Kernel based virtual machine
LAN Local Area Network
LDAP Lightweight Directory Access Protocol
LUN Logical Unit Number
LVM Landscape Virtualization Management
SAP system for monitoring and controlling SAP landscapes.
MAC Media Access Control
MINRA Minimal Read Ahead
NAS Network Attached Storage
Abbreviations
364 FlexFrame Agents Guide
NDMP Network Data Management Protocol
NFS Network File System
NIC Network Interface Card
NVRAM Non-Volatile Random Access Memory
OBP Open Boot Prom
OLTP On-Line Transaction Processing
ONTAP Open Network Technology for Appliance Products
OSS Open Source Software
POST Power-On Self Test
PFS Production File System (on Celerra)
PW PRIMEPOWER
PXE Preboot Execution Environment
PY PRIMERGY
QA Quality Assurance
QS Quality of Service
RAID Redundant Array of Independent (or Inexpensive) Disks
RARP Reverse Address Resolution Protocol
RDBMS Relational Database Management System
RHEL Red Hat Enterprise Linux
RSB Remote Service Board
SCS System Console Software
SAN Storage Area Network
SAP BW SAP Business Warehouse
SAPGUI SAP Graphical User Interface
SAPOSS SAP Online System Service
SID System Identifier
SLD System Landscape Directory
SLES SUSE Linux Enterprise Server
SMB Server Message Block
Abbreviations
FlexFrame Agents Guide 365
SMC System Management Console
SNMP Simple Network Management Protocol
SPOC Single Point Of Control
TELNET Telecommunications Network
TFTP Trivial File Transfer Protocol
UDP User Datagram Protocol
UPS Uninterruptible Power Supply
VLAN Virtual Local Area Network
VTOC Virtual Table Of Contents
WAN Wide Area Network
WAS Web Application Server
WAFL Write Anywhere File Layout
XSCF Extended System Control Facility
FlexFrame Agents Guide 367
12 Glossary
Advanced Business Application Programming
Proprietary programming language of SAP.
Advanced Power Management
Advanced Power Management defines a layer between the hardware and the operat-
ing system that effectively shields the programmer from hardware details.
Application Agent
A software program for monitoring and managing applications.
Application Node
A host for applications (e.g. SAP instances db, ci, agate, wgate, app etc.). This
definition includes Application Servers as well as Database Servers.
Automounter
The automounter is an NFS utility that automatically mounts directories on an NFS
client as they are needed, and unmounts them when they are no longer needed.
Autonomous Agent
Central system management and high availability software component of FlexFrame.
Blade
A special form factor for computer nodes.
BladeRunner
The working title for the solution part of SAP for FlexFrame.
BOOTPARAM
Boot time parameters of the kernel.
BRBACKUP
SAP backup and restore tools.
Client LAN
Virtual network segment within FlexFrame, used for client-server traffic.
Common Internet File System
A protocol for the sharing of file systems (same as SMB).
Computing Node
From the SAP ACI perspective: A host that is used for applications.
Control Agent
A software program for monitoring and managing nodes within FlexFrame.
Control LAN
Virtual network segment within FlexFrame, used for system management traffic.
Glossary
368 FlexFrame Agents Guide
Control Node
A physical computer system, controlling and monitoring the entire FlexFrame land-
scape and running shared services in the rack (dhcp, tftp, ldap etc.).
Control Station
A Control Node in an SAP ACI environment.
DART
Operating system of Celerra data movers (Data Access in Real Time).
Dynamic Host Configuration Protocol
DHCP is a protocol for assigning dynamic IP addresses to devices on a network.
Dynamic Host Configuration Protocol server
A DHCP server provides configuration parameters specific to the DHCP client host,
required by the host to participate on the Internet.
Enterprise Resource Planning
Enterprise Resource Planning systems are management information systems that in-
tegrate and automate many of the business practices associated with the operations
or production aspects of a company.
Ethernet
A Local Area Network which supports data transfer rates of 10 megabits per second.
Fiber Channel
Fiber Channel is a serial computer bus intended for connecting high-speed storage
devices to computers.
Filer
Network attached storage for file systems of NetApp.
FlexFrame
FlexFrame is a Fujitsu solution and means a radically new architecture for SAP envi-
ronments. It exploits the latest business-critical computing technology deliver major
cost savings for SAP customers. FlexFrame is a joint project in which the main part-
ners are SAP, Network Appliance, Intel and Fujitsu.
FlexFrame® for SAP
®
The name of the Fujitsu solution FlexFrame® for the versions until FF4S V5.3A.
FlexFrame internal LAN Switch
Cisco network switches which are integral part of the FlexFrame for SAP hardware
configuration and which are automatically configured by the FlexFrame for SAP soft-
ware.
FlexFrame® Orchestrator
This is the advancement of the Fujitsu solution FlexFrame® and means a new ap-
proach to offer enhanced functionality and features step by step and become more
and more independent from certain hardware and software components.
Glossary
FlexFrame Agents Guide 369
Gigabit Ethernet
A Local Area Network which supports data transfer rates of 1 gigabit (1,000 mega-
bits) per second.
Host name
The name of a node (assigned to an interface) that is resolved to a unique IP ad-
dress. One node can have multiple host names (cf. node name).
In SAP environments host names are currently limited to 13 alphanumeric characters
including the hyphen (“ - “). The first character must be a letter. In the SAP environ-
ment host names are case-sensitive.
Image
In the FlexFrame documentation, “Image” is used as a synonym for “Hard Disk Im-
age”.
Internet Protocol Address
A unique number used by computers to refer to each other when sending information
through networks using the Internet Protocol.
Landscape Virtualization Management (LVM)
SAP system for monitoring and controlling SAP landscapes
Lightweight Directory Access Protocol
Protocol for accessing on-line directory services.
Local Area Network
A computer network that spans a relatively small area. Most LANs are confined to a
single building or group of buildings. However, one LAN can be connected to other
LANs over any distance via telephone lines and radio waves. A system of LANs con-
nected in this way is called a Wide Area Network (WAN).
Local host name
The name of the node (physical computer); it can be displayed and set using the
command /bin/hostname.
Logical Unit Number
An address for a single (SCSI) disk drive.
MAC address
Device identifier number of a Network Interface Card. In full: "media access control
address".
MaxDB
A relational database system from mySQL (formerly ADABAS and SAPDB).
Media Access Control address
An identifier for network devices, usually unique. The MAC address is stored physi-
cally on the device.
Glossary
370 FlexFrame Agents Guide
NAS system
Network Attached Storage of any vendor (in our context NetApp Filer).
NDMPcopy
NDMPcopy transfers data between Filers using the Network Data Management Pro-
tocol (NDMP).
Netboot
A boot procedure for computers where the operating system is provided via a net-
work instead of local disks.
Netweaver
SAP NetWeaver is the technical foundation of SAP solutions.
Network Appliance Filer
See “Filer”.
Network Attached Storage
A data storage device that is connected via a network to one or multiple computers.
Network File System
A network protocol for network-based storage access.
Network Interface Card
A hardware device that allows computer communication via networks.
Node
A physical computer system controlled by an OS.
Node name
The name of a physical node as returned by the command uname -n. Each node
name within a FlexFrame environment must be unique.
Non-Volatile Random Access Memory
A type of memory that retains its contents when the power is turned off.
On-Line Transaction Processing
Transaction processing via computer networks.
OpenLDAP
An Open Source LDAP Service Implementation.
Open Network Technology for Appliance Products
The operating system of Network Appliance Filers.
Open Source Software
Software that is distributed free of charge under an open source license, such as the
GNU Public License.
Oracle RAC
A cluster database by Oracle Corporation.
Glossary
FlexFrame Agents Guide 371
Physical host
Name of a physical computer system (node).
Power-On Self Test
Part of a computer's boot process; automatic testing of diverse hardware compo-
nents.
Preboot Execution Environment
An environment that allows a computer to boot from a network resource without hav-
ing a local operating system installed.
PRIMEPOWER
Fujitsu's SPARC-based server product line.
PRIMERGY
Fujitsu's i386-based server product line.
Red Hat Enterprise Linux
Linux distribution by Red Hat, Inc., targeting business customers.
Reverse Address Resolution Protocol
A protocol allowing resolution of an IP address corresponding to a MAC address.
SAP Service
In FlexFrame: SAP Service and DB Services.
SAP service script
An administration script for starting and stopping an SAP application on a virtual host.
SAP Solution Manager
Service portal for the implementation, operation and optimization of an SAP solution.
SAPLogon
Front-end software for SAPGUI.
SAPRouter
Router for SAP services like SAPGUI or SAPTELNET.
SavVol
A Celerra volume to which SnapSure copies original point-in-time data blocks from
the PFS before the blocks are altered by a PFS transaction.
Server
A physical host (hardware), same as node.
Service
A software program providing functions to clients.
Service type
The type of an application or service (db, ci, app, agate, wgate etc.).
Glossary
372 FlexFrame Agents Guide
Single Point of Control
In FlexFrame: One user interface to control a whole FlexFrame environment.
Storage LAN
A virtual LAN segment within a FlexFrame environment, carrying the traffic to NAS
systems.
SUSE Linux Enterprise Server
A Linux distribution by Novell, specializing in server installations.
Telecommunications Network
A terminal emulation program for TCP/IP networks such as the Internet.
Trivial File Transfer Protocol
A simple form of the File Transfer Protocol (FTP). TFTP uses the User Datagram
Protocol (UDP) and provides no security features. It is often used by servers to boot
diskless workstations, X-terminals, and routers.
TFTP server
A simple FTP implementation.
Virtual host
The name of the virtual host on which an application runs; it is assigned to a physical
node when an application is started.
Virtual Local Area Network
A VLAN is a logically segmented network mapped over physical hardware according
to the IEEE 802.1q standard.
Virtualization
Virtualization means the separation of hardware and processes. In a virtualized envi-
ronment (FlexFrame), a process can be moved between hardware nodes while stay-
ing transparent to the user and application.
FlexFrame Agents Guide 373
13 Index
Agents
application scenarios 89
user interactions 120
allnodes 161
autonomous functions 60
autonomous operation 119
BlackBoard commands
command file 316
command list 321
interactive 320
via WebInterface 319
class creation rules 67
Client LAN 20
directories, production and log files 152
Domain Manager 85
Data collection 85
Data retentiontime and compression
86
DomainManager 40
Database connection 40
mySQL 41
PostgreSQL 41
escalation with partner service 97
SCS/ASCS with ERS 97
service with SMD 98
FA pools 60
fa_list_services.sh 160
First Steps
General Information 19
Installation and Startup 20
Configuration 33
FlexFrame solution 20
generic services 35
Installation 21
EzSetup
Backup with EzBackup 24
General 23
Important 27
Standard Installation 25
Usage 31
EzSetup-Installation 23
Installation Packages 21
Requirements 21
rpm-Installation 22
start / stop 34
FlexFrame
accounting option 75
accounting Plug-in 203
Agents 89
collecting diagnostic information for
support assistance 361
diagnostic tool 360
performance 75, 203
reporting Plug-in 203
FlexFrame Agent
deactivating and activating 20
Index
374 FlexFrame Agents Guide
installing 20
system and directory 20
version 20
FlexFrame Agent WebInterface 19
FlexFrame Agents
collecting diagnostic information for
support assistance 357
directories, production and log files
351
FlexFrame Agents traps 323
FlexFrame Agents components
configuration and log files 68
service types 69
systems 69
FlexFrame Agents Suite 21
FlexFrame Agents Suite 14
FlexFrame Agents Suite architecture
55
FlexFrame Agents Suite rpm 19
FlexFrame Agents traps
format 323, 324
overview 327
FlexFrame Agents WebInterface
configuring 37
function 37
installing 37
starting and stopping 37
WebGui and database 38
mySQL 38
PostgreSQL 39
FlexFrame AppAgents 19
FlexFrame CtrlAgent 19
FlexFrame DomainManager 19
FlexFrame LogAgent 14, 19
FlexFrame Messenger 14, 19
Fujitsu FlexFrame scripts 119
generic services 73
group schema 62
grouping 60
grouping function 62
hardware resources 60
introduction 55
IP storage 20
live cache 70
maintenance scripts 94
Messenger
Database connection 42
Messenger 42
Messenger
Database connection
mySQL 42
Messenger
Database connection
PostgreSQL 42
migration 153
migration tool 157
Multi Node Failure 95
LongTime 96
ShortTime 96
Index
FlexFrame Agents Guide 375
Netapp Filer 20
node failure 95
NoWatch/Watch of SAP services 125
operating mode 117
event mode 117
reaction mode
central 117
local 117
parameter reference 225
parameterization
FlexFrame Agents
parameters for the performance and accounting option 257
parameterization
class creation rules 253
default parameter file 228, 234
FlexFrame Agents 254
FlexFrame Agents general
parameters 255
grouping 229
pool creation 229
service classes 253
service power value 253
service priority 253
traps 226
parameterization
FlexFrame Agents
node-related parameters 269
parameterization
FlexFrame Agents
service-related parameters 270
parameterization
FlexFrame Agents
definition of generic services parameters 272
parameterization
FlexFrame Agents
path configuration 277
parameterization
FlexFrame Agents
shutdown configuration 279
parameterization
FlexFrame Agents
default parameter file 280
parameterization
FlexFrame Agents
FA Frame Agent 312
pool creation 60
pool-indepenednt-spare 108
Productive Pool 108
SparePool 109
power shutdown 205
Application Nodes 217
architecture 206
Blade systems 207
configuration 209
default shutdown mode 222
Management Blades 215
other Server 208
PRIMERGY systems 207
switchover control parameters 210
Index
376 FlexFrame Agents Guide
user, password, community 212
virtual VM Server 208
Powershutdownkonzept
für virtuelle Server 208
reactions 90
Reboot 90
Restart 90
Special case 93
SwitchOver 90
read/write Root Image 20
read-only Root Image 20
reboot 60
related documents 18
replicated enqueue service (ERS) 69
restart 60
SAP HANA 71
SAP start scripts 20
self-repair strategies 94
Server LAN 20
service classes 66
service detection model 74
service failure 94
service instance 66
service ping
ServicePingDb.sh 87
ServicePingDef.sh 87
ServicePingErs.sh 87
ServicePingHDB.sh 87
service power value 66
service priotiy 66
service reaction model 74
service state model 73
SLES 20
Solution Manager Diagnostic (SMD) 69
spare nodes 95
start/stop of SAP services 123
start/stop/status of Agents 121
Storage LAN 20
Suspend Autonomy 131
switchover 60
TakeOver Rules 101
Add Rule 102
Dynamic Rule 104
Dynamic Rule Example 107
Replace Rule 102
SpareNode 101
Substitution Rule 103
TakeOver Startegy 100
testament types 67
Usage of Pool-Master 51
Automation of steps during pool
creation 54
Create Pool-Master from existing pool
52
Create/Update pool from Pool-Master
52
Usage of ‘fa_MasterPoolMgr.sh’ 54
use cases 136
Autonomy for Application Instances
137
Index
FlexFrame Agents Guide 377
File System Utilization 140
High Availability and Autonomy Rules
141
Performance, Capacity and
Accounting 140
semi-autonomous operation 137
version management 44
Lifecycle management files 45
version components 47
version dependency 45
version manager 48
Lifecycle process 44
WebInterface 163
BlackBoard settings 166
configuration 163
configuration of FlexFrame Agents
188
Hypervisor Monitor 167
info and help 202
interaction 200
login 169
message display 184
overview 170
paths and file names 167
pool / group tree 171, 199
reading out the FA data 167
starting 169
status display 177
updates 200
visualization 169