p Series 690 Availability

IBM ~ pSeries 690 Availability Best Practices

Document Number: p690 Availability Best PracticesLast Revised: 5/1/2002

Version: GA2

IBM ^ pSeries 690 Availability Best Practices White Paper

242.2.5.2 AIX Command Line Invocation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .232.2.5.1 Automatic Error Log Analysis (diagela) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .232.2.5 Periodic Diagnostics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .232.2.4 RAID Hot-plug Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .222.2.3 SCSI Hot-swap Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .212.2.2.1 PCI Hot-plug Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .202.2.2 Hot-plug Task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .192.2.1.4 Restarting an Aborted Processor Deallocation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .192.2.1.3 AIX Interface for Turning Processor Deallocation On and Off . . . . . . . . . . . . . . . .192.2.1.2 Processor Deallocation: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .182.2.1.1 Potential Impact to Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .172.2.1 Enable CPU Dynamic Deallocation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .172.2 AIX Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .162.1.8 SP Menus Which Should Not be Changed from Default Values . . . . . . . . . . . . . . . . . .16

2.1.7.2 SP Menu - System Information Menu - MemoryConfiguration/Deconfiguration Menu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

162.1.7.1 SP Menu - System Information Menu - ProcessorConfiguration/Deconfiguration Menu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

152.1.7 Processor/Memory Configuration/Deconfiguration Menu . . . . . . . . . . . . . . . . . . . . . . .152.1.6.2 Service Aid - Configure Reboot Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .132.1.6.1 SP Menu - Reboot/Restart Policy Setup Menu . . . . . . . . . . . . . . . . . . . . . . . . . . . . .122.1.6 Reboot/Restart on Failure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .122.1.5.3 Service Aid - Reboot/Restart Policy Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .122.1.5.2 SMS Menu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .112.1.5.1 SP Menu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .112.1.5 Unattended Start Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .112.1.4.1 Service Processor Menu - Scan Dump Log Policy . . . . . . . . . . . . . . . . . . . . . . . . . . .112.1.4 Scan Dump Log Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .102.1.3.1 Service Processor Menu - Serial Port Snoop Setup . . . . . . . . . . . . . . . . . . . . . . . . . .102.1.3 Serial Port Snoop Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10

2.1.2.2 Service Aid - Configure Service Processor - Configure SurveillancePolicy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

92.1.2.1 Service Processor Menu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .82.1.2 OS Surveillance Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .82.1.1 Boot Time Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .72.1 Boot Options Utilizing SP Menus and Alternatives . . . . . . . . . . . . . . . . . . .7

2.0 Configuring the p690 SMP for High Availability. . . . . . . . . . . . . .

61.3 Modifications From Last Release . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .51.2 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .51.1 Purpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5

1.0 About this White Paper. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .


p690 Availability Best Practices 043002 final.lwp Page 2 of 54

463.5 Service Enablement Through HMC Operations . . . . . . . . . . . . . . . . . . . . . .463.4.3 I/O Drawer Additions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .453.4.2 General Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .453.4.1 Adapter Placement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .453.4 I/O Drawer and Adapter Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .443.3 Disk Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .443.2.2 Update System or SP Flash . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .443.2.1 AIX Defaults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .433.2 AIX Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .423.1 Boot Options Utilizing SP Menus and Alternatives . . . . . . . . . . . . . . . . . .42

3.0 Configuring the LPAR Environment for HighAvailability

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

412.6 Internal Battery Feature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .402.5.7 Scheduling Critical Console Data Backups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .402.5.6 Redundant HMCs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .392.5.5 Service Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .382.5.4.3 Enabling Extended Error Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .382.5.4.2 Enabling The Automatic Call-Home Feature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .382.5.4.1 Enabling Error Reporting to SFP Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .372.5.4 Service Focal Point . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .372.5.3.4 Setting Routing Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .372.5.3.3 Setting Domain Names . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .372.5.3.2 Setting the IP Address . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .362.5.3.1 Customizing Network Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .36

2.5.3 Establish HMC to OS Partition Network Link for Error Reporting andManagement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

362.5.2 Creating Service Representative and hmcpe User IDs . . . . . . . . . . . . . . . . . . . . . . . . . . .352.5.1 IBM Hardware Management Console for pSeries (HMC) . . . . . . . . . . . . . . . . . . . . . . .352.5 Service Enablement Through HMC Operations . . . . . . . . . . . . . . . . . . . . . .332.4.2 I/O Drawer Additions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .332.4.1.1 Non Enhanced Error Handling (EEH) Adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . .322.4.1 Adapter Placement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .322.4 I/O Drawer and Adapter Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .302.3.2.1 RAID Levels and Their Performance Implications . . . . . . . . . . . . . . . . . . . . . . . . . .302.3.2 User Disks - Stand-Alone vs. RAID Disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .262.3.1 Boot Disk - Stand-Alone vs. Mirrored . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .262.3 Disk Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .252.2.8.2 Service Aid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .252.2.8.1 AIX Command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .252.2.8 Update System or SP Flash . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .242.2.7 Save/Restore Service Processor Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .242.2.6 Save or Restore Hardware Management Policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .



535.0 Conclusions

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

524.2.3 HACMP Clusters within One LPAR System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .524.2.2 Clusters of LPAR Images Across LPAR Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .524.2.1 Clusters of LPAR Images Coupled to Uni/SMP systems . . . . . . . . . . . . . . . . . . . . . . . . .524.2 LPAR Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .524.1 Uni/SMP Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .52

4.0 HACMP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

513.6.2 Standalone Diagnostics: NIM vs CD-ROM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .50

3.6.1 LPAR Considerations for Placement of Regatta System Unit CD-ROMDrive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

503.6 Standalone Diagnostic Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .493.5.1.4 Service Agent Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .493.5.1.3 Modem Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .493.5.1.2 Enabling Error Reporting to SFP Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .473.5.1.1 Creating a Service Authority Partition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .463.5.1 Hardware Management Console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .



1.0 About this White Paper

1.1 PurposeThe purpose of this paper is to describe in a level of detail useful to a system administrator or operatorhow to configure the pSeries 690 system to achieve higher levels of availability utilizing a Best Practicesapproach.

There are many factors which influence the overall availability of a system. Some factors take intoaccount the base design of the system including which components act as single points of failure andwhich components are replicated through N+1 design or some other form of redundancy. These areas ofthe pSeries 690 are controlled by development and are minimized through various design techniques(refer to the white paper entitled pSeries 690 RAS White Paper).

Other factors are directly influenced by how the administrator of the system chooses to configure theirvarious operating environments. This is the primary focus of this white paper. Other factors outside thescope of this white paper, but for which consulting services are available from IBM, deal with how thesystem is administered and the various availability management policies and practices established by theIT department.

1.2 ApproachThe first part of this white paper will address the pSeries 690 system running in symmetricmultiprocessing (SMP) mode and will form the basis on which the rest of the paper references.

The second part of the white paper will address the availability factors concerned with the LogicalPARtition (LPAR) mode of operation. Its recommended that the user interested in configuring an LPARsystem for high availability review the SMP sections prior to the LPAR section since most of the LPARmaterial references back to the SMP section.

The next section will address running HACMP in SMP and LPAR partitions which is followed by theconclusions regarding configuring the for availability.

This paper is best utilized by referencing the following documents or guides to actually perform thesuggested actions proposed in this paper: pSeries 690 Installation Guide - SA38-0587-00 Hardware Management Console for pSeries Operations Guide IBM ~ pSeries 690 Users Guide - SA38-0588-00 PCI Adapter Placement Reference - SA38-0538 HACMP for AIX Installation Guide Electronic Service Agent for pSeries Users Guide



1.3 Modifications From Last Release

Changes to default options to enable auto restart after failure.p690 GA2 release04/26/2002Initial p690 release10/10/2001

ModificationsReleaseLast

RevisedDate



2.0 Configuring the p690 SMP for High AvailabilityThere are many options left to the customer which ultimately can affect the overall system availability.This section of the white paper will address those factors that the customer has control over associatedwith configuring the pSeries 690 in an SMP mode of operation.

This section will begin by addressing base boot configuration options through use of Service Processor(SP) menus or through alternate methods such as AIX command line instructions, AIX DiagnosticService AIDs, Service Management System menus, or Web System Manager (Web SM) interfaces. Onceconfigured, these options remain in effect until specifically changed by the user.

Following this section, Boot and User disk configuration options will be reviewed. Disk configurationand I/O adapter configuration reflect two of the most significant areas that the customer has directcontrol over to affect overall system availability. Finally, service enablement and remote support topicswill be addressed which will help to automate the error report, service call, and dispatch of the servicerepresentative with the correct Field Replaceable Units (FRUs) to fix the system quickly and efficiently inthe event of a failure.

2.1 Boot Options Utilizing SP Menus and AlternativesListed below are a set of Boot options found in the SP menus along with the default and preferredsettings to enable for high availability. The setting of these options to the recommended values will have adirect affect on the time to recover from a failure by allowing the system to automatically attemptrecovery or will provide a simplified method to recover from the failure.

Each of the individual settings will be explained in additional sections following the table. The last columnin the table represents whether there is an alternative method from System Management Services (SMS)menus, the SMIT/SMITTY panels in the AIX operating system or through the AIX Diagnostic ServiceAids interfaces to change the setting. The user can choose whichever method they are most familiar withto effect the proposed changes. Each of the options are explained in the sections following the table.

System Power Control Menu

1 (As needed)1 (As needed)Scan Dump LogPolicy

Scan Log DumpPolicy Menu

1UnassignedSnoop Serial PortChoose reset stringUnassignedSystem Reset StringSerial Port

Snoop Setup

CustomerDetermined

2 minutesSurveillance Delay

CustomerDetermined

2 minutesSurveillance TimeInterval

Service AID Configure ServiceProcessor -ConfigureSurveillance Policy

OnOffOS SurveillanceOS SurveillanceSetup Menu

Service Processor Setup Menu

AlternateInvocation Method

RecommendedSetting

DefaultValue

OptionSP Menu



EnableEnableEnable MemoryRepeat Gard

MemoryConfig/DeconfigMenu

EnableEnableEnable CPU RepeatGard

ProcessorConfig/DeconfigMenu

System Information Menu

Yes

No - GA1Yes - Insystemsshipped withGA2 (April26th, 2002) &beyond

Enable supplementalrestart policy

NoYes - GA1No -Insystemsshipped withGA2 (April26th, 2002) &beyond

Use OS-Definedrestart policy

Service AID - Configure RebootPolicy SMIT -AutomaticallyReboot After Crash

11Number of rebootattempts

Reboot/RestartPolicy SetupMenu

Reboot/Restart Policy Setup Menu

SMS Menu-Unattended StartMenuService AID -Reboot/Restart PolicySetup

YesNo - GA1Yes - Insystemsshipped withGA2 (April26th,2002) &beyond

Enable/DisableUnattended StartMode

Enable/DisableUnattendedStart Mode

AlternateInvocation Method

RecommendedSetting

DefaultValue

OptionSP Menu

Refer to the IBM ^ pSeries Installation Guide for specific instructions on configuring thepreferred settings through use of the SP menus.

2.1.1 Boot Time OptionsOnce these options are set utilizing the Service Processor Setup menus, they are maintained in NVRAMand utilized on every system boot until they are specifically altered by the user.

2.1.2 OS Surveillance Setup



Surveillance is a function in which the service processor monitors the system, and the system monitorsthe service processor. This monitoring is accomplished by periodic samplings called heartbeats.Surveillance is available during two phases:

1. System firmware bringup (automatic)2. Operating system runtime (optional)

System Firmware SurveillanceSystem firmware surveillance is automatically enabled during system power-on. It cannot be disabled bythe user, and the surveillance interval and surveillance delay cannot be changed by the user. If the serviceprocessor detects no heartbeats during system IPL (for a set period of time), it cycles the system powerto attempt a reboot. The maximum number of retries is set from the service processor menus. If the failcondition persists, the service processor leaves the machine powered on, logs an error, and displaysmenus to the user. The service processor reports the failure to the IBM Hardware Management Consolefor pSeries (HMC) and displays the operating-system surveillance failure code on the operator panel. TheService Focal Point application running on the HMC will log the error in the Service Action Event (SAE)Log and will call for Service.

Operating System SurveillanceOperating system surveillance provides the service processor with a means to detect hang conditions, aswell as hardware or software failures, while the operating system is running. It also provides theoperating system with a means to detect a service processor failure caused by the lack of a returnheartbeat.

Operating system surveillance is not enabled by default, allowing you to run operating systems that donot support this service processor option. You can also use service processor menus and AIX service aidsto enable or disable operating system surveillance.For operating system surveillance to work correctly, you must set these parameters: Surveillance enable/disable Surveillance interval The maximum time the service processor should wait for a heartbeat from the operating system

before time out. Surveillance delay The length of time to wait from the time the operating system is started to when the first

heartbeat is expected. Surveillance does not take effect until the next time the operating systemis started after the parameters have been set.

If desired, you can initiate surveillance mode immediately from service aids. In addition to the threeoptions above, a fourth option allows you to select immediate surveillance, and rebooting of the system isnot necessarily required. If operating system surveillance is enabled (and system firmware has passedcontrol to the operating system), and the service processor does not detect any heartbeats from theoperating system, the service processor assumes the system is hung and takes action according to thereboot/restart policy settings. If surveillance is selected from the service processor menus which are onlyavailable at bootup, then surveillance is enabled by default as soon as the system boots. From service aids,the selection is optional.



2.1.2.1 Service Processor MenuOS Surveillance Setup Menu1. Surveillance:Currently Enabled2. Surveillance Time Interval:2 minutes3. Surveillance Delay:2 minutes98. Return to Previous Menu0> SurveillanceCan be set to Enabled or Disabled. Surveillance Time IntervalCan be set to any number from 2 through 255 (value is in minutes). Surveillance DelayCan be set to any number from 0 through 255 (value is in minutes).

2.1.2.2 Service Aid - Configure Service Processor - Configure Surveillance PolicyNote: This service aid runs on CHRP system units only.This service aid monitors the system for hang conditions; that is, hardware or software failures that causeoperating system inactivity. When enabled, and surveillance detects operating system inactivity, a call isplaced to report the failure. Use this service aid to display and change the following settings for theSurveillance Policy:Note: Because of system capability, some of the following settings might not bedisplayed by this service aid: Surveillance (on/off) Surveillance Time Interval This is the maximum time between heartbeats from the operating system. Surveillance Time Delay This is the time to delay between when the operating system is in control and when to begin

operating system surveillance. Changes are to Take Effect Immediately Set this to Yes if the changes made to the settings in this menu are to take place immediately.

Otherwise the changes take effect beginning with the next system boot.

You can access this service aid directly from the AIX command line, by typing:/usr/lpp/diagnostics/bin/uspchrp -s

2.1.3 Serial Port Snoop SetupThis menu can be used to set up serial port snooping, in which the user can configure serial port 1 as acatch-all...reset device.

2.1.3.1 Service Processor Menu - Serial Port Snoop SetupFrom the service processor main menu, select option 1, service processor setup menu, then select option 8 (Serial Port Snoop Setup menu).



SERIAL PORT SNOOP SETUP MENU1.System reset string:Currently Unassigned2.Snoop Serial Port:Currently Unassigned98.Return to Previous Menu1>Use the system reset string option to enter the system reset string, which resets the machine when it isdetected on the main console on Serial Port 1.

Use the Snoop Serial Port option to select the serial port to snoop.Note: Only serial port 1 is supported.

After serial port snooping is correctly configured, at any point after the system is booted to AIX,whenever the reset string is typed on the main console, the system uses the service processor rebootpolicy to restart. Pressing Enter after the reset string is not required, so make sure that the string is notcommon or trivial. A mixed-case string is recommended.

2.1.4 Scan Dump Log PolicyA scan dump is the collection of chip data that the service processor gathers after asystem malfunction, such as a checkstop or hang. The scan dump data may containchip scan rings, chip trace arrays, and SCOM contents.The scan dump data are stored in the system control store. The size of the scandump area is approximately 4MB.

2.1.4.1 Service Processor Menu - Scan Dump Log PolicyThe scan dump log policy can be set to:1 = As neededThis is the default value. In this case, the processor run-time diagnostics record the dump data based onthe error type.2 = Never3 = Always4 = ImmediatelyThis option can only be used when the system is in the standby state with power on. It is used to dumpthe system data after a checkstop or machine check occurs when the system firmware is running, or whenthe operating system is booting or running.

2.1.5 Unattended Start ModeUse this option to instruct the service processor to restore the power state of theserver after a temporary power failure. Unattended start mode can also be setthrough the System Management Services (SMS) menus. It is intended to be usedon servers that require automatic power-on after a power failure.

When ac power is restored, the system returns to the power state at thetime ac loss occurred. For example, if the system was powered-on when ac lossoccurred, it reboots/restarts when power is restored. If the system was powered-offwhen ac loss occurred, it remains off when power is restored.



2.1.5.1 SP MenuSYSTEM POWER CONTROL MENU1. Enable/Disable Unattended Start Mode:Currently Enabled

2.1.5.2 SMS MenuUnattended Start Mode : This selection is used to enable or disable unattendedstart mode. Use this option to instruct the service processor to restore the power-stateof the server after a temporary power failure, which is necessary on servers that requireautomatic power-on after a power failure. The default setting for GA1 was off. The default setting for systems shipped with GA2 (April 26th, 2002) and beyond is on.

2.1.5.3 Service Aid - Reboot/Restart Policy SetupEnable Unattended Start Mode (1=Yes, 0=No)

When enabled, Unattended Start Mode...allows the system to recover from the lossof ac power. If the system was powered-on when the ac loss occurred, the system reboots whenpower is restored. If the system was powered-off when the ac loss occurred, the system remains off whenpower is restored. You can access this service aid directly from the AIX command line, by typing:/usr/lpp/diagnostics/bin/uspchrp -b

2.1.6 Reboot/Restart on FailureThe operating systems automatic restart policy (see operating system documentation)indicates the operating system response to a system crash. The service processor canbe instructed to refer to that policy by the Use OS-Defined Restart Policy setup menu.If the operating system has no automatic restart policy, or if it is disabled, then theservice processor-restart policy can be controlled from the service processor menus.Use the Enable Supplemental Restart Policy selection.

The purpose of this option is to allow the system to attempt to restart after experiencing a fatal error.This option is defined by a combination of 2 parameters described in the table below. The objective is toallow the system to attempt to restart after failure. Choose the option that you feel most comfortable withto cause that affect.

Use OS-Defined restart policy - The default setting for GA1 was Yes. This causes the service processorto refer to the OS Automatic Restart Policy setting and take action (the same action the operating systemwould take if it could have responded to the problem causing the restart).

The default for systems shipped with GA2 (April 26th, 2002) & beyond is No. When this setting is No, orif the operating system did not set a policy, the service processor refers to enable supplemental restartpolicy for its action.

Enable supplemental restart policy - The default setting for GA1 was No. The default setting is Yesfor systems shipped with GA2 (April 26th, 2002) & beyond. If set to Yes, the service processor restartsthe server when the operating system loses control and either:



The Use OS-Defined restart policy is set to No.OR

The Use OS-Defined restart policy is set to Yes and the operating system has no automatic restartpolicy.

The following table describes the relationship among the operating system and service processor restartcontrols in SMP mode

yesControls reboot when AC power isrestored after a power failure. (truemeans restart will occur.)

"Enable Unattended Start Mode"

yesControls reboot on all crashes when"Use OS automatic Restart policy" is no

"Enable Supplemental Restart Policy"

noSelects whether OS automatic restart orsupplemental restart policy is used

"Use OS defined restart policy"

trueControls reboot on all crashes when "UseOS defined restart policy" is yes

OS automatic restart policy

RecommendedSetting

UseParameter

With the recommended settings, the system will automatically attempt restart, regardless of the setting ofthe OS automatic restart parameter.

The following table describes the interaction of the parameters in LPAR mode:

trueControls reboot on when AC power isrestored after a power failure. (truemeans reboot will occur.)

"Enable Unattended Start Mode"

trueControls reboot on platform crashes(requires "Use O/S defined Rebootpolicy" set to "no" on firmware releasesprior to April 26th, 2002 L3)

"Enable Supplemental Restart Policy"

Set to no, if fieldis available on theservice authoritypartition.

Undefined"Use OS defined restart policy"

true for eachLPAR partition

Controls reboot on partition crashesOS automatic restart policy

RecommendedSetting

UseParameter

With the recommended changes, the system will always automatically attempt reboot if the OS automaticrestart parameter is set to true in each LPAR partition.



2.1.6.1 SP Menu - Reboot/Restart Policy Setup Menu

Reboot/Restart Policy Setup Menu1. Number of reboot attempts:Currently 12. Use OS-Defined restart policy?Currently No3. Enable supplemental restart policy?Currently Yes4. Call-Out before restart:Currently Disabled98. Return to Previous Menu0>Reboot is the process of bringing up the system hardware; for example, from asystem reset or power on. Restart is activating the operating system after the systemhardware is reinitialized. Restart must follow a successful reboot. Number of reboot attempts - If the server fails to successfully complete the bootprocess, it attempts to reboot the number of times specified. Entry values equal toor greater than 0 are valid. Only successive failed reboot/restart attempts arecounted. Use OS-Defined restart policy - In SMP mode allows the service processor to react or notreact in the same way that the operating system does to major system faults byreading the setting of the operating system parameter AutomaticallyRestart/Reboot After a System Crash. This parameter may or may not bedefined, depending on the operating system or its version/level.. See your operating systemdocumentation for details on setting up operating system automatic restarts.

For proper operation in LPAR mode, this parameter should always be set to No. With the parameter setto No in LPAR mode, the OS defined restart policy is used for partition crashes and the SupplementalRestart policy is used for system crashes.

The default value for Use OS-Defined restart policy in GA1 was Yes. The default for systems shippedwith GA2 (April 26rh, 2002) & beyond is No.

Enable supplemental restart policy - The default setting for GA1 was No. The default for systemsshipped with GA2 (April 26th, 2002) & beyond is Yes. In SMP mode, if set to Yes, the service processor restarts the system when the system loses control asdetected by service processor surveillance, and either:The Use OS-Defined restart policy is set to NoORThe Use OS-Defined restart policy is set to Yes, and the operating systemhas no automatic restart policy.

In LPAR mode, if Use OS-Defined restart policy is No, then Enable supplemental restart policycontrols whether a the service processor restartsthe system when the system loses control as detected byservice processor.



Call-Out before restart (Enabled/Disabled) - This should be left in the default disabled state for thissystem. Call out is handled through the Service Focal Point application running on the IBM hardwaremanagement console for pseries. Refer to the Remote Support and Service Enablement sections laterin this white paper for more detail.

2.1.6.2 Service Aid - Configure Reboot PolicyThis service aid controls how the system tries to recover from a system crash.Use this service aid to display and change the following settings for the Reboot Policy.Note: Because of system capability, some of the following settings might not be displayed by this serviceaid. Maximum Number of Reboot Attempts Enter a number that is 0 or greater.

Note: A value of 0 indicates do not attempt to reboot to a crashed system. This number is the maximumnumber of consecutive attempts to reboot the system. The term reboot, in the context of this service aid,is used to describe bringing system hardware back up from scratch; for example, from a system reset orPower-on. When the reboot process completes successfully, the reboot attempts count is resetto 0, and a restart begins. The term restart, in the context of this service aid, is used to describe theoperating system activation process. Restart always follows a successful reboot. When a restart fails, anda restart policy is enabled, the system attempts to reboot for the maximum number of attempts. Use the O/S Defined Restart Policy (1=Yes, 0=No)

When Use the O/S Defined Restart Policy is set to Yes, the system attempts to reboot from a crash ifthe operating system has an enabled Defined Restart or Reboot Policy. When Use the O/S DefinedRestart Policy is set to No, or the operating system restart policy is undefined, then the restart policy isdetermined by the Supplemental Restart Policy. Enable Supplemental Restart Policy (1=Yes, 0=No)

The Supplemental Restart Policy, if enabled, is used when the O/S Defined Restart Policy is undefined,or is set to False. When surveillance detects operating system inactivity during restart, an enabledSupplemental Restart Policy causes a system reset and the reboot process begins. Call-Out Before Restart (on/off)

This option should remain disabled as stated above in the SP menu option.

2.1.7 Processor/Memory Configuration/Deconfiguration MenuAll failures that crash the system with a machine check or check stop, even if intermittent, are reported asa diagnostic call out for service repair. To prevent the recurrence of intermittent problems and improvethe availability of the system until a scheduled maintenance window, processors and memory books witha failure history are marked bad to prevent their being configured on subsequent boots. A processor ormemory book is marked bad under the following circumstances: A processor or memory book fails built-in self test (BIST) or power-on self test (POST) testing

during boot (as determined by the service processor). A processor or memory book causes a machine check or check stop during runtime, and the failure

can be isolated specifically to that processor or memory book (as determined by the processorruntime diagnostics in the service processor). A processor or memory book reaches a threshold of recovered failures that results in a predictive

call out (as determined by the processor runtime diagnostics in the service processor).



During boot time, the service processor does not configure processors or memory books that are markedbad. If a processor or memory book is deconfigured, the processor or memory book remains offline forsubsequent reboots until it is replaced or repeat gard is disabled.

The Repeat Gard function also provides the user with the option of manually deconfiguring a processoror memory book, or re-enabling a previously deconfigured processor or memory book. Both of thesemenus are submenus under the System Information Menu. You can enable or disable CPU Repeat Gardor Memory Repeat Gard using the Processor Configuration/Deconfiguration Menu, which is a submenuunder the System Information Menu.

2.1.7.1 SP Menu - System Information Menu - Processor Configuration/Deconfiguration MenuPROCESSOR CONFIGURATION/DECONFIGURATION MENU77. Enable/Disable CPU Repeat Gard: Currently Enabled1. 0 3.0 (00) Configured by system 2. 1 3.1 (00) Deconfigured by system3. 2 3.2 (00) Configured by system 4. 3 3.3 (00) Configured by system5. 4 3.4 (00) Configured by system 6. 5 3.5 (00) Deconfigured by system7. 6 3.6 (00) Configured by system 8. 7 3.7 (00) Configured by system98. Return to Previous Menu0>

To enable or disable CPU repeat gard, use menu option 77. CPU repeat gard is enabled by default. If CPU repeat gard is disabled, processors that are in thedeconfigured by system state will be reconfigured. These reconfigured processors are then testedduring the boot process, and if they pass, they remain online. If they fail the boot testing, they aredeconfigured even though CPU repeat gard is disabled. The failure history of each CPU is retained. If aprocessor with a history of failures is brought back online by disabling repeat gard, it remains online if itpasses testing during the boot process. However, if repeat gard is enabled, the processor is taken offlineagain because of its history of failures.

2.1.7.2 SP Menu - System Information Menu - Memory Configuration/Deconfiguration MenuMEMORY CONFIGURATION/DECONFIGURATION MENU77. Enable/Disable Memory Repeat Gard: Currently Enabled1. Memory card98. Return to Previous Menu

To enable or disable Memory Repeat Gard, use menu option 77 of the MemoryConfiguration/Deconfiguration Menu.The failure history of each book is retained. If a book with a history of failures is brought back online bydisabling Repeat Gard, it remains online if it passes testing during the boot process. However, if RepeatGard is enabled, the book is taken offline again because of its history of failures.

2.1.8 SP Menus Which Should Not be Changed from Default ValuesThe following SP menus should be left in their default state for proper operation of the p690 system withthe HMC:

Ring Indicate Power on Enable/Disable Modem Setup Modem Configuration Setup Dial-out Phone Numbers



Select Modem Line Speed

These functions are provided on the HMC and should be configured there since the modem will besupported from the HMC instead of attaching to the serial port on the CEC as on prior products.2.2 AIX Options

Service AID -Update System orService ProcessorFlash

Used to update System or ServiceProcessor Flash

Command lineentry

Update System orSP Flash

Service AID - Saveor Restore ServiceProcessor settings

Service Processorsettings

Save/RestoreService Processorsettings

Service AID - Saveor RestoreHardwareManagementPolicies

Used to save hardware and serviceprocessor settings

Surveillance Policyand Reboot Policy

Save or RestoreHardwareManagementPolicies

Service AID -PeriodicDiagnosticsAIX CommandLine

EnableEnableDisable or EnableAutomaic ErrorLog Analysis

PeriodicDiagnostics

RAID Hot-plugDevices

SCSI Hot-swapManager

Service AID -Hot-plug TaskUsed for Concurrent Maintenance

PCI Hot-plugManager

Hot-plug Task

AIX cmd lineWeb SMSMIT - CPU Gard

EnableDisableEnable/DisableCPU DynamicDeallocation

Enable CPUDynamicDeallocation

InvocationMethod

RecommendedSetting

DefaultValue

Option

2.2.1 Enable CPU Dynamic DeallocationL1 instruction cache recoverable errors, L1 data cache correctable errors, and L2 cache correctableerrors are monitored by the processor runtime diagnostics (PRD) code running in the service processor.When a predefined error threshold is met, an error log with warning severity and threshold exceededstatus is returned to AIX. At the same time, PRD marks the CPU for deconfiguration at the next boot.AIX will attempt to migrate all resources associated with that processor to another processor and thenstop the defective processor.

These errors are not fatal and, as long as they remain rare occurrences, can be safely ignored. However,when a pattern of failures seems to be developing on a specific processor, this pattern may indicate that



this component is likely to exhibit a fatal failure in the near future. This prediction is made by thefirmware based-on-failure rates and threshold analysis.

AIX, on these systems, implements continuous hardware surveillance and regularly polls the firmware forhardware errors. When the number of processor errors hits a threshold and the firmware recognizes thatthere is a distinct probability that this system component will fail, the firmware returns an error report toAIX. In all cases, AIX logs the error in the system error log. In addition, on multiprocessor systems,depending on the type of failure, AIX attempts to stop using the untrustworthy processor anddeallocate it. This feature is called Dynamic Processor Deallocation.

At this point, the processor is also flagged by the firmware for persistent deallocation for subsequentreboots, until maintenance personnel replaces the processor.

2.2.1.1 Potential Impact to ApplicationsThis processor decallocation is transparent for the vast majority of applications, including drivers andkernel extensions. However, you can use AIX published interfaces to determine whether an application orkernel extension is running on a multiprocessor machine, find out how many processors there are, andbind threads to specific processors.

The interface for binding processes or threads to processors uses logical CPU numbers. The logical CPUnumbers are in the range [0..N-1] where N is the total number of CPUs. To avoid breaking applicationsor kernel extensions that assume no "holes" in the CPU numbering, AIX always makes it appear forapplications as if it is the "last" (highest numbered) logical CPU to be deallocated. For instance, on an8-way SMP, the logical CPU numbers are [0..7]. If one processor is deallocated, the total number ofavailable CPUs becomes 7, and they are numbered [0..6]. Externally, it looks like CPU 7 hasdisappeared, regardless of which physical processor failed. In the rest of this description, the term CPU isused for the logical entity and the term processor for the physical entity.

Applications or kernel extensions using processes/threads binding could potentially be broken if AIXsilently terminated their bound threads or forcefully moved them to another CPU when one of theprocessors needs to be deallocated. Dynamic Processor Deallocation provides programming interfaces sothat those applications and kernel extensions can be notified that a processor deallocation is about tohappen. When these applications and kernel extensions get this notification, they are responsible formoving their bound threads and associated resources (such as timer request blocks) away formthe last logical CPU and adapt themselves to the new CPU configuration.

If, after notification of applications and kernel extensions, some of the threads are still bound to the lastlogical CPU, the deallocation is aborted. In this case AIX logs the fact that the deallocation has beenaborted in the error log and continues using the ailing processor. When the processor ultimately fails, itcreates a total system failure. Thus, it is important for applications or kernel extensions binding threads toCPUs to get the notification of an impending processor deallocation, and act on this notice.

Even in the rare cases where the deallocation cannot go through, Dynamic Processor Deallocation stillgives advanced warning to system administrators. By recording the error in the error log, it gives them achance to schedule a maintenance operation on the system to replace the ailing component before a globalsystem failure occurs.



2.2.1.2 Processor Deallocation:The typical flow of events for processor deallocation is as follows:

1. The firmware detects that a recoverable error threshold has been reached by one of the processors. 2. AIX logs the firmware error report in the system error log, and, when executing on a machine

supporting processor deallocation, start the deallocation process. 3. AIX notifies non-kernel processes and threads bound to the last logical CPU. 4. AIX waits for all the bound threads to move away from the last logical CPU. If threads remain

bound, AIX eventually times out (after ten minutes) and aborts the deallocation 5. Otherwise, AIX invokes the previously registered High Availability Event Handlers (HAEHs). An

HAEH may return an error that will abort the deallocation. 6. Otherwise, AIX goes on with the deallocation process and ultimately stops the failing processor.

In case of failure at any point of the deallocation, AIX logs the failure with the reason why thedeallocation was aborted. The system administrator can look at the error log, take corrective action(when possible) and restart the deallocation. For instance, if the deallocation was aborted because at leastone application did not unbind its bound threads, the system administrator could stop the application(s),restart the deallocation (which should go through this time) and restart the application.

2.2.1.3 AIX Interface for Turning Processor Deallocation On and OffDynamic Processor Deallocation can be enabled or disabled by changing the value of the cpugardattribute of the ODM object sys0. The possible values for the attribute are enable and disable.

The default, in this version of AIX, is that the dynamic processor deallocation is disabled (the attributecpugard has a value of disable). System administrators who want to take advantage of this feature mustenable it using either the Web-based System Manager system menus, the SMIT System Environmentsmenu, or the chdev command.

Note: If processor deallocation is turned off, AIX still reports the errors in the error log and you will see the error indicating that AIX was notified of the problem with a CPU (CPU_FAILURE_PREDICTED, see the following format).

2.2.1.4 Restarting an Aborted Processor DeallocationSometimes the processor deallocation fails because, for example, an application did not move its boundthreads away from the last logical CPU. Once this problem has been fixed, by either unbinding (when it issafe to do so) or stopping the application, the system administrator can restart the processor deallocationprocess using the ha_star command.

The syntax for this command is:

ha_star -C

where -C is for a CPU predictive failure event.



Processor State Considerations

Physical processors are represented in the ODM data base by objects named procn where n is the physicalprocessor number (n is a decimal number). Like any other "device" represented in the ODM database,processor objects have a state (Defined/Available) and attributes.

The state of a proc object is always Available as long as the corresponding processor is present,regardless of whether it is usable by AIX. The state attribute of a proc object indicates if the processor isused by AIX and, if not, the reason. This attribute can have three values:

enable The processor is used by AIX. disable The processor has been dynamically deallocated by AIX. faulty The processor was declared defective by the firmware at boot time.

In the case of CPU errors, if a processor for which the firmware reports a predictive failure is successfullydeallocated by AIX, its state goes from enable to disable. Independently of AIX, this processor is alsoflagged as defective in the firmware. Upon reboot, it will not be available to AIX and will have its stateset to faulty. But the ODM proc object is still marked Available. Only if the defective CPU was physicallyremoved from the system board or CPU board (if it were at all possible) would the proc object change toDefined.

2.2.2 Hot-plug TaskThe Hot-plug Task provides software function for those devices that support hot-plug or hot-swapcapability. This includes PCI adapters, SCSI devices, and some RAID devices. This task was previouslyknown as SCSI Device Identification and Removal or Identify and Remove Resource. The Hot-plugTask has a restriction when running in Standalone or Online Service mode; new devices may not beadded to the system unless there is already a device with the same FRU part number installed in thesystem. This restriction is in place because the device software package for the new device cannot beinstalled in Standalone or Online Service mode.Depending on the environment and the software packages installed, selecting this task displays thefollowing three subtasks: PCI Hot-plug Manager SCSI Hot-swap Manager RAID Hot-plug Devices

To run the Hot-plug Task directly from the command line, type the following: Diag -T"identifyRemove"If you are running the diagnostics in Online Concurrent mode, run the Missing Options ResolutionProcedure immediately after adding, removing or replacing any device. Start the Missing OptionsResolution Procedure is by running the diag -a command. If the Missing Options Resolution Procedureruns with no menus or prompts, then device configuration is complete. Otherwise, work through eachmenu to complete device configuration.



2.2.2.1 PCI Hot-plug ManagerThe PCI Hot-plug Manager task is a SMIT menu that allows you to identify, add, remove, or replace PCIadapters that are hot-pluggable. The following functions are available under this task: List PCI hot-plug slots Add a PCI hot-plug adapter Replace/Remove a PCI hot-plug adapter Identify a PCI hot-plug slot Unconfigure Devices Configure Devices Install/Configure Devices Added After IPL

The List PCI Hot-plug Slots function lists all PCI hot-plug slots. Empty slots and populated slots arelisted. Populated slot information includes the connected logical device. The slot name consists of thephysical location code and the description of the physical characteristics for the slot. The Add a PCIHot-plug Adapter function is used to prepare a slot for the addition of a new adapter. The function listsall the empty slots that support hot-plug. When a slot is selected, the visual indicator for the slot blinks atthe Identify rate. After the slot location is confirmed, the visual indicator for the specified PCI slot is setto the Action state. This means the power for the PCI slot is off and the new adapter can be plugged in.

The Replace/Remove a PCI Hot-plug Adapter function is used to prepare a slot for adapter exchange.The function lists all the PCI slots that support hot-plug and are occupied. The list includes the slotsphysical location code and the device name of the resource installed in the slot. The adapter must be inthe Defined state before it can be prepared for hot-plug removal. When a slot is selected, the visualindicator for the slot is set to the Identify state. After the slot location is confirmed, the visual indicatorfor the specified PCI slot is set to the Action state. This means the power for the PCI slot, is off and theadapter can be removed or replaced.

The Identify a PCI Hot-plug Slot function is used to help identify the location of a PCI hot-plugadapter. The function lists all the PCI slots that are occupied or empty and support hot-plug. When a slotis selected for identification, the visual indicator for the slot is set to the Identify state.

The Unconfigure Devices function attempts to put the selected device, in the PCI hot-plug slot, into theDefined state. This action must be done before any attempted hot-plug function. If the unconfigurefunction fails, it is possible that the device is still in use by another application. In this case, the customeror system administrator must be notified to quiesce the device.

The Configure Devices function allows a newly added adapter to be configured into the system for use.This function should also be done when a new adapter is added to the system.

The Install/Configure Devices Added After IPL function attempts to install the necessary softwarepackages for any newly added devices. The software installation media or packages are required for thisfunction.

Standalone Diagnostics has restrictions on using the PCI Hot-plug Manager. Forexample: Adapters that are replaced must be exactly the same FRU part number as the adapter being



replaced. New adapters cannot be added unless a device of the same FRU part number already exists in the

system, because the configuration information for the new adapter is not known after theStandalone Diagnostics are booted. The following functions are not available from the Standalone Diagnostics and will not display in

the list: Add a PCI Hot-plug Adapter Configure Device Install/Configure Devices Added After IPL

You can run this task directly from the command line by typing the following command:diag -d device -T"identifyRemove"However, note that some devices support both the PCI Hot-plug Task and the RAID Hot-plug Devicestask. If this is the case for the device specified, then the Hot-plug Task displays instead of the PCIHot-plug Manager menu.

2.2.3 SCSI Hot-swap ManagerThis task was known as SCSI Device Identification and Removal or Identify and Remove Resourcesin previous releases. This task allows the user to identify, add, remove, and replace a SCSI device in asystem unit that uses a SCSI Enclosure Services (SES) device. The following functions are available: List the SES Devices Identify a Device Attached to an SES Device Attach a Device to an SES Device Replace/Remove a Device Attached to an SES Device Configure Added/Replaced Devices

The List the SES Devices function lists all the SCSI hot-swap slots and their contents. Statusinformation about each slot is also available. The status information available includes the slot number,device name, whether the slot is populated and configured, and location.

The Identify a Device Attached to an SES Device function is used to help identify the location of adevice attached to a SES device. This function lists all the slots that support hot-swap that are occupiedor empty. When a slot is selected for identification, the visual indicator for the slot is set to the Identifystate.

The Attach a Device to an SES Device function lists all empty hot-swap slots that are available for theinsertion of a new device. After a slot is selected, the power is removed. If available, the visual indicatorfor the selected slot is set to the Remove state. After the device is added, the visual indicator for theselected slot is set to the Normal state, and power is restored.

The Replace/Remove a Device Attached to an SES Device function lists all populated hot-swap slotsthat are available for removal or replacement of the devices. After a slot is selected, the device populatingthat slot is Unconfigured; then the power is removed from that slot. If the Unconfigure operation fails, itis possible that the device is in use by another application. In this case, the customer or systemadministrator must be notified to quiesce the device. If the Unconfigure operation is successful, the visual



indicator for the selected slot is set to the Remove state. After the device is removed or replaced, thevisual indicator, if available for the selected slot, is set to the Normal state, and power is restored.Note: Be sure that no other host is using the device before you remove it.

The Configure Added/Replaced Devices function runs the configuration manager on the parentadapters that had child devices added or removed. This function ensures that the devices in theconfiguration database are configured correctly. Standalone Diagnostics has restrictions on using theSCSI Hot-plug Manager. For example: Devices being used as replacement devices must be exactly the same type of device as the device

being replaced. New devices may not be added unless a device of the same FRU part number already exists in the

system, because the configuration information for the new device is not known after the StandaloneDiagnostics are booted. You can run this task directly from the command line.

See the following command syntax: diag -d device -T"identifyRemove" ORdiag [-c ]-d device -T"identifyRemove -a [identify|remove ]"Flag Description-a Specifies the option under the task.-c Run the task without displaying menus. Only command line prompts are used. This flag is onlyapplicable when running an option such as identify or remove.-d Indicates the SCSI device.-T Specifies the task to run.

2.2.4 RAID Hot-plug DevicesPCI RAID Physical Disk IdentifyThis selection identifies physical disks connected to a PCI SCSI-2 F/W RAID adapter.You can run this task directly from the AIX command line. See the following commandsyntax: diag -c -d pci RAID adapter -T identify

2.2.5 Periodic DiagnosticsThis selection provides a tool for configuring periodic diagnostics and automatic error log analysis. Youcan select a hardware resource to be tested once a day, at a user-specified time. By default, the floatingpoint processor diagnostic is run at 4:00 a.m. each day Other devices can be added to the PeriodicDiagnostic Device list by setting PDiagAtt->attribute=test_mode. Error log analysis can be directed torun at different times.

Problems are reported by a message to the system console, and a mail message is sent to all members ofthe system group. The message contains the SRN.

2.2.5.1 Automatic Error Log Analysis (diagela)Automatic Error Log Analysis (diagela) provides the capability to do error log analysis whenever apermanent hardware error is logged. If the diagela program is enabled and a permanent hardwareresource error is logged, the diagela program is started. Automatic Error Log Analysis is enabled bydefault on all platforms.

The diagela program determines whether the error should be analyzed by the diagnostics. If the errorshould be analyzed, a diagnostic application will be invoked and the error will be analyzed. No testing is



done. If the diagnostics determines that the error requires a service action, it sends a message to yourconsole and to all system groups. The message contains the SRN, or a corrective action.

If the resource cannot be tested because it is busy, error log analysis is performed. Hardware errorslogged against a resource can also be monitored by enabling Automatic Error Log Analysis. This allowserror log analysis to be performed every time a hardware error is put into the error log. If a problem isdetected, a message is posted to the system console and a mail message sent to the users belonging to thesystem group containing information about the failure, such as the service request number.The service aid provides the following functions: Add or delete a resource to the periodic test list Modify the time to test a resource Display the periodic test list Modify the error notification mailing list Disable or Enable Automatic Error Log Analysis

2.2.5.2 AIX Command Line InvocationTo activate the Automatic Error Log Analysis feature, log in as root and type thefollowing command:

/usr/lpp/diagnostics/bin/diagela ENABLE

To disable the Automatic Error Log Analysis feature, log in as root and type thefollowing command:

/usr/lpp/diagnostics/bin/diagela DISABLE

2.2.6 Save or Restore Hardware Management PoliciesUse this service aid to save or restore the settings from Ring Indicate Power-On Policy, SurveillancePolicy, Remote Maintenance Policy and Reboot Policy. The following options are available: Save Hardware Management Policies

This selection writes all of the settings for the hardware-management policies to the following file:/etc/lpp/diagnostics/data/hmpolicies Restore Hardware Management Policies

This selection restores all of the settings for the hardware-management policies from the contents of thefollowing file: /etc/lpp/diagnostics/data/hmpoliciesYou can access this service aid directly from the AIX command line, by typing:/usr/lpp/diagnostics/bin/uspchrp -a

2.2.7 Save/Restore Service Processor ConfigurationUse this service aid to save or restore the Service Processor Configuration to or from a file. The ServiceProcessor Configuration includes the Ring Indicator Power-On Configuration. The following options areavailable: Save Service Processor Configuration

This selection writes all of the settings for the Ring Indicate Power On and the Service Processor to thefollowing file: /etc/lpp/diagnostics/data/spconfig Restore Service Processor Configuration



This selection restores all of the settings for the Ring Indicate Power On and the Service Processor fromthe following file: /etc/lpp/diagnostics/data/spconfig

2.2.8 Update System or SP Flash

2.2.8.1 AIX CommandYou can use the update_flash command to perform this function. The command is located in the/usr/lpp/diagnostics/bin directory. The command syntax is as follows:update_flash [-q ]-f file_name update_flash [-q ]-D device_name -f file_nameupdate_flash [-q ]-D device_name -lFlag Description-q Forces the update_flash command to update the flash EPROM and reboot the system without askingfor confirmation.-D Specifies that the flash update image file is on diskette. The device_name variable specifies thediskette drive. The default device_name is /dev/fd0 .-f Flash update image file source. The file_name variable specifies the fully qualified path of the flashupdate image file.-l Lists the files on a diskette for the user to choose a flash update image file.Attention: The update_flash command reboots the entire system. Do not use this command if more thanone user is logged on to the system.

2.2.8.2 Service AidYou must know the fully qualified path and file name of the flash update image file that was provided. Ifthe flash update image file is on a diskette, the service aid can list the files on the diskette for selection.The diskette must be a valid backup diskette. Refer to the update instructions, or the service guide for thesystem unit to determine the level of the system unit or service processor flash. When this service aid isrun from online diagnostics, the flash update image file is copied to the /var file system. If there is notenough space in the /var file system for the flash update image file, an error is reported. If this erroroccurs, exit the service aid, increase the size of the /var file system, and retry the service aid. After thefile is copied, a screen requests confirmation before continuing with the update flash. Continuing theupdate flash reboots the system using the shutdown -u command. The system does not return todiagnostics, and the current flash image is not saved. After the reboot, remove the/var/update_flash_image file.

When this service aid is run from standalone diagnostics, the flash update image file is copied to the filesystem from diskette. The user must provide the image on a backup diskette because the user does nothave access to remote file systems or any other files that are on the system. If not enough space isavailable, an error is reported, stating additional system memory is needed. After the file is copied, ascreen requests confirmation before continuing with the update flash. Continuing the update flashreboots the system using the reboot -u command. You may receive a Caution: some process(es) wouldn'tdie message during the reboot process. You can ignore this message. The current flash image is notsaved.



2.3 Disk Configuration2.3.1 Boot Disk - Stand-Alone vs. MirroredIn order to minimize single points of fail the system should be configured with mirrored Boot Disks. Inaddition to the redundant Boot Disk, the highest availability configuration will consist of two separate I/Odrawers, with two separate SCSI adapters in each drawer, each of the mirrored Boot Disks connected toone of these two SCSI adapters. Refer to section I/O Drawer Additions for specific placementsuggestions for the redundant adapters and disks.

BP BP BP BP

PLANAR BOARD 2PLANAR BOARD 1

POWERSUPPLY

1

POWERSUPPLY

2

PCI1

EADSEADSPCI

SW =Winnipeg

PCI2

PCI3

PCI4

PCI5

PCI7

PCI8

PCI9

PCI10

DASD2

DASD3

DASD4

DASD5

DASD6

DASD7

DASD8

DASD9

DASD10

DASD11

DASD12

DASD13

DASD14

DASD15

DASD16

RISER

PCI11

PCI12

PCI13

PCI14

PCI15

PCI16

PCI17

PCI18

PCI19

PCI20

RIO X2

MIDPLANE

SCSI SES SCSISES SCSI SES

PCI

RISER

EADS

PCI6

RIO X2

SW SW

VPD

SSSS SS SS SS SS SS SSSSSS SS SS SS SS SSSS

DASD1

SS SS SS SS SS SSSSSS SSSS

EADSEADSEADS

SS SS SS SS SSSSSSSS SSSS

SS =Soft Switch=PCI

MAINSS

TERM TERM TERM

SCSISES

VPD VPDVPD

TERM

PWRCNTL

PWRCNTL

PWRCNTL

PWRCNTL

=SCSI

SS SS SS SSMAIN

SS

The user will establish the configuration using the following procedure, described here at a high level.The user will need to use SMIT.

1. The user would have to install AIX.2. Then the user would have to add the appropriate other disk to the root volume group.3. Then the user would have to mirror the AIX boot logical volume to the other disk.4. The user would have to use the bootlist command to specify both disks as boot devices.Below are the commands you will need.

mirrorvg Command Purpose



Mirrors all the logical volumes that exist on a given volume group. This command only applies to AIXVersion 4.2.1 or later.

Syntax

mirrorvg [ -S | -s ] [ -Q ] [ -c Copies] [ -m ] VolumeGroup [ PhysicalVolume ... ] Description

The mirrorvg command takes all the logical volumes on a given volume group and mirrors those logicalvolumes. This same functionality may also be accomplished manually if you execute the mklvcopycommand for each individual logical volume in a volume group. As with mklvcopy, the target physicaldrives to be mirrored with data must already be members of the volume group. To add disks to a volumegroup, run the extendvg command.

By default, mirrorvg attempts to mirror the logical volumes onto any of the disks in a volume group. Ifyou wish to control which drives are used for mirroring, you must include the list of disks in the inputparameters, PhysicalVolume. Mirror strictness is enforced. Additionally, mirrorvg mirrors the logicalvolumes, using the default settings of the logical volume being mirrored. If you wish to violate mirrorstrictness or affect the policy by which the mirror is created, you must execute the mirroring of all logicalvolumes manually with the mklvcopy command.

When mirrorvg is executed, the default behavior of the command requires that the synchronization of themirrors must complete before the command returns to the user. If you wish to avoid the delay, use the -Sor -s option. Additionally, the default value of 2 copies is always used. To specify a value other than 2,use the -c option.

Notes: 1. This command ignores striped logical volumes. Mirroring striped logical volumesis not possible. 2. To use this command, you must either have root user authority or be a member ofthe system group.

Notes: 1. This command ignores striped logical volumes. 2. To use this command, you must either have root user authority or be a member ofthe system group.

Note: To use this command, you must either have root user authority or be a member of thesystem group.Attention: The mirrorvg command may take a significant amount of time before completingbecause of complex error checking, the amount of logical volumes to mirror in a volume group,and the time is takes to synchronize the new mirrored logical volumes.

You can use the Web-based System Manager Volumes application (wsm lvm fast path) to run thiscommand. You could also use the System Management Interface Tool (SMIT) smit mirrorvg fast pathto run this command.



Flags

Returns the mirrorvg command immediately without performing any type of mirrorsynchronization. If this option is used, the mirror may exist for a logical volume but isnot used by the operating system until it has been synchronized with the syncvgcommand.

-s DisableSync

Returns the mirrorvg command immediately and starts a background syncvg of thevolume group. With this option, it is not obvious when the mirrors have completelyfinished their synchronization. However, as portions of the mirrors becomesynchronized, they are immediately used by the operating system in mirror usage.

-SBackgroundSync

By default in mirrorvg, when a volume group's contents becomes mirrored, volumegroup quorum is disabled. If the user wishes to keep the volume group quorumrequirement after mirroring is complete, this option should be used in the command. Forlater quorum changes, refer to the chvg command.

-Q QuorumKeep

Allows mirroring of logical volumes in the exact physical partition order that the originalcopy is ordered. This option requires you to specify a PhysicalVolume(s) wherethe exact map copy should be placed. If the space is insufficient for an exact mapping,then the command will fail. You should add new drives or pick a different set of drivesthat will satisfy an exact logical volume mapping of the entire volume group. Thedesignated disks must be equal to or exceed the size of the drives which are to be exactlymirrored, regardless of if the entire disk is used. Also, if any logical volume to bemirrored is already mirrored, this command will fail.

-m exact map

Specifies the minimum number of copies that each logical volume must have after themirrorvg command has finished executing. It may be possible, through the independentuse of mklvcopy, that some logical volumes may have more than the minimum numberspecified after the mirrorvg command has executed. Minimum value is 2 and 3 is themaximum value. A value of 1 is ignored.

-c Copies

The following is a description of rootvg:

When this volume group has been mirrored, the default command causes Quorum todeactivated. The user must close all open logical volumes, execute varyoffvg andthen varyonvg on the volume group for the system to understand that quorum is oris not needed for the volume group. If you do not revaryon the volume group,

non-rootvgmirroring

When the rootvg mirroring has completed, you must perform three additional tasks:bosboot, bootlist, and reboot.

The bosboot command is required to customize the bootrec of the newly mirroreddrive. The bootlist command needs to be performed to instruct the system whichdisk and order you prefer the mirrored boot process to start.

Finally, the default of this command is for Quorum to be turned off. For this to takeeffect on a rootvg volume group, the system must be rebooted.

rootvg mirroring



The system dump devices, primary and secondary, should not be mirrored. In somesystems, the paging device and the dump device are the same device. However,most users want the paging device mirrored. When mirrorvg detects that a dumpdevice and the paging device are the same, the logical volume will be mirroredautomatically.

If mirrorvg detects that the dump and paging device are different logical volumes,the paging device is automatically mirrored, but the dump logical volume is not. Thedump device can be queried and modified with the sysdumpdev command.

rootvg andnon-rootvgmirroring

mirror will still work correctly. However, any quorum changes will not have takeneffect.

Examples1. To triply mirror a volume group, enter: mirrorvg -c 3 workvgThe logical partitions in the logical volumes held on workvg now have three copies. 2. To get default mirroring of rootvg, enter: mirrorvg rootvgrootvg now has two copies. 3. To replace a bad disk drive in a mirrored volume group, enter unmirrorvg workvg hdisk7reducevg workvg hdisk7rmdev -l hdisk7 -dreplace the disk drive, let the drive be renamed hdisk7extendvg workvg hdisk7mirrorvg workvg

Note: By default in this example, mirrorvg will try to create 2 copies for logical volumesin workvg. It will try to create the new mirrors onto the replaced disk drive. However, ifthe original system had been triply mirrored, there may be no new mirrors created ontohdisk7, as other copies may already exist for the logical volumes.

4. To sync the newly created mirrors in the background, enter: mirrorvg -S -c 3 workvg5. To create an exact mapped volume group, enter: mirrorvg -m datavg hdisk2 hdisk3

Implementation Specifics

NONEStandards Compliance:

Base Operating System/ AIX 3.2 to 4.1 CompatibilityLinks

Software Product/Option:

FilesDirectory where the mirrorvg commandresides.

/usr/sbin



2.3.2 User Disks - Stand-Alone vs. RAID DisksAvailability advantages can be leveraged for user disks by implementing RAID technology. There aremany types of RAID configurations which have varying degrees of availability, performance and costfactors. This section will describe the various RAID modes along with their benefits and detriments.

Redundant Array of Independent Disks (RAID) is a term used to describe the technique of improvingdata availability through the use of arrays of disks and various data-striping methodologies. Disk arraysare groups of disk drives that work together to achieve higher data-transfer and I/O rates than thoseprovided by single large drives. An array is a set of multiple disk drives plus a specialized controller (anarray controller) that keeps track of how data is distributed across the drives. Data for a particular file iswritten in segments to the different drives in the array rather than being written to a single drive.

Arrays can also provide data redundancy so that no data is lost if a single drive (physical disk) in the arrayshould fail. Depending on the RAID level, data is either mirrored or striped.

Subarrays are contained within an array subsystem. Depending on how you configure it, an arraysubsystem can contain one or more sub-arrays, also referred to as Logical Units (LUN). Each LUN hasits own characteristics (RAID level, logical block size and logical unit size, for example). From theoperating system, each subarray is seen as a single hdisk with its own unique name.RAID algorithms can be implemented as part of the operating system's file system software, or as part ofa disk device driver (common for RAID 0 and RAID 1). These algorithms can be performed by a locallyembedded processor on a hardware RAID adapter. Hardware RAID adapters generally provide betterperformance than software RAID because embedded processors offload the main system processor byperforming the complex algorithms, sometimes employing specialized circuitry for data transfer andManipulation.

2.3.2.1 RAID Levels and Their Performance ImplicationsEach of the RAID levels supported by disk arrays uses a different method of writing data and henceprovides different benefits.

RAID 0 is also known as data striping. RAID 0 is only designed to increase performance; there is noredundancy, so any disk failures require reloading from backups. It is not recommended to use this levelfor critical applications that require high availability.

RAID 1 is also known as disk mirroring. It is most suited to applications that require high dataavailability, good read response times, and where cost is a secondary issue. The response time for writescan be somewhat slower than for a single disk, depending on the write policy; the writes can either beexecuted in parallel for speed or serially for safety. Select RAID Level 1 for applications with a highpercentage of read operations and where the cost is not the major concern.

RAID 2 is rarely used. It implements the same process as RAID 3, but can utilize multiple disk drives forparity, while RAID 3 can use only one.

RAID 3 and RAID 2 are parallel process array mechanisms, where all drives in the array operate inunison. Similar to data striping, information to be written to disk is split into chunks (a fixed amount ofdata), and each chunk is written out to the same physical position on separate disks (in parallel). More



advanced versions of RAID 2 and 3 synchronize the disk spindles so that the reads and writes can trulyoccur simultaneously (minimizing rotational latency buildups between disks). This architecture requiresparity information to be written for each stripe of data; the difference between RAID 2 and RAID 3 isthat RAID 2 can utilize multiple disk drives for parity, while RAID 3 can use only one. The LVM doesnot support Raid 3; therefore, a RAID 3 array must be used as a raw device from the host system. RAID 3 provides redundancy without the high overhead incurred by mirroring in RAID 1.

RAID 4 addresses some of the disadvantages of RAID 3 by using larger chunks of data and striping thedata across all of the drives except the one reserved for parity. Write requests require aread/modify/update cycle that creates a bottleneck at the single parity drive. Therefore, RAID 4 is notused as often as RAID 5, which implements the same process, but without the parity volume bottleneck.

RAID 5, as has been mentioned, is very similar to RAID 4. The difference is that the parity information isdistributed across the same disks used for the data, thereby eliminating the bottleneck. Parity data is neverstored on the same drive as the chunks that it protects. This means that concurrent read and writeoperations can now be performed, and there are performance increases due to the availability of an extradisk (the disk previously used for parity). There are other enhancements possible to further increase datatransfer rates, such as caching simultaneous reads from the disks and transferring that information whilereading the next blocks. This can generate data transfer rates at up to the adapter speed.

RAID 6 is similar to RAID 5, but with additional parity information written that permits data recovery iftwo disk drives fail. Extra parity disk drives are required, and write performance is slower than a similarimplementation of RAID 5.

The RAID 7 architecture gives data and parity the same privileges. The level 7 implementation allowseach individual drive to access data as fast as possible. This is achieved by three features:

1. Independent control and data paths for each I/O device/interface. 2. Each device/interface is connected to a high-speed data bus that has a central cache capable of

supporting multiple host I/O paths. 3. A real time, process-oriented operating system is embedded into the disk drive array architecture.

The embedded operating system "frees" the drives by allowing each drive head to moveindependently of the other disk drives. Also, the RAID 7 embedded operating system is enabled tohandle a heterogeneous mix of disk drive types and sizes.

RAID 10 - RAID-0+1

RAID-0+1, also known in the industry as RAID 10, implements block interleave data striping andmirroring. RAID 10 is not formally recognized by the RAID Advisory Board (RAB), but, it is an industrystandard term. In RAID 10, data is striped across multiple disk drives, and then those drives are mirroredto another set of drives.

RAID 10 provides an enhanced feature for disk mirroring that stripes data and copies the data across allthe drives of the array. The first stripe is the data stripe; the second stripe is the mirror (copy) of the firstdata stripe, but it is shifted over one drive. Because the data is mirrored, the capacity of the logical driveis 50 percent of the physical capacity of the hard disk drives in the array.



2.4 I/O Drawer and Adapter Configuration2.4.1 Adapter PlacementThe following diagram (with reference to the diagram in the I/O drawer section which follows) will beused to help explain the placement of I/O adapters within and across I/O drawers in order to obtainincreasing levels of availability.The PCI adapter slots controlled by the various PCI Host Bridges are shown in the diagram below. For placement of redundant adapters or disks, refer to the following section on I/O Drawer Additions.

PHB6Slots 18,19,S4,20Loc codeUx.y-P2-I8Ux.y-P2-I9Ux.y-P2-Z2Ux.y-P2-I1064 bit 5V 33MHz

PHB4Slots 11,12,13,14Loc codeUx.y-P2-I1Ux.y-P2-I2Ux.y-P2-I3Ux.y-P2-I464 bit 3.3V 33/66MHz

PHB5Slots 15,16,S3,17Loc codeUx.y-P2-I5Ux.y-P2-I6Ux.y-P2-Z1Ux.y-P2-I764 bit 3.3V 33/66MHz

PHB3Slots 8,9,S2,10Loc codeUx.y-P1-I8Ux.y-P1-I9Ux.y-P1-Z2Ux.y-P1-I1064 bit 5V 33MHz

PHB1Slots 1,2,3,4Loc codeUx.y-P1-I1Ux.y-P1-I2Ux.y-P1-I3Ux.y-P1-I464 bit 3.3V 33/66MHz

PHB2Slots 5,6,S1,7Loc CodeUx.y-P1-I5Ux.y-P1-I6Ux.y-P1-Z1Ux.y-P1-I764 bit 3.3V 33/66MHz

x.y = location code of the drawer

RIO Bus

Speedwagon3.3

EADSPCI- PCIBridge (PHB4)

Ultra3SCSI

EADSPCI- PCIBridge(PHB6)

32

SES


SESUltra3SCSI

RIO Bus

Speedwagon3.3

EADSPCI- PCIBridge (PHB1)

Ultra3SCSI


32

SES


SESUltra3SCSI

I/O Drawer ConfigurationI/O Drawer Configuration

For more information about your device and its capabilities, see the documentationshipped with that device. For a list of supported adapters and a detailed discussionabout adapter placement, refer to the PCI Adapter Placement Reference, order numberSA38-0538.



2.4.1.1 Non Enhanced Error Handling (EEH) AdaptersBecause some devices do not have enhanced error handling (EEH) capabilities built into their devicedrivers, non-EEH I/O Adapters (IOA) on a PHB should be solely in one partition - do not split the PHBbetween partitions as a failure on a non-EEH adapter will affect all other adapters on that PHB.

This description uses the term EADS, which is IBM unique hardware in the I/O drawer controllingeach PCI slot. EADS, firmware, and device drivers act in concert to support EEH.The exploitation of EADS error handling occurs in three scenarios: During firmware probing of the PCI space during boot time configuration During PCI hot-plug when RTAS configures a PCI adapter after insertion/replacement During normal run-time operation or AIX diagnostic run.

At boot time, firmware can now deconfigure adapters which cause errors on the PCI bus, and continuewith the boot process, whereas in the past these types of failures would have caused machine checks andprevented the system from booting. The basic firmware process is as follows:

1. Detect and configure Speedwagon (PHB) bridges.2. Detect and configure EADS bridges. In each parent PHB, an ibm,eeh-implemented property is

added to indicate the number and addresses of all freeze mode capable slots under that PHB.3. Set all EADS bridges to freeze on error (Bridge Arbitration, Config space 0x0040, bit 16 =1)4. Probe for devices under each EADS bridge.5. If a return value of 0xFFFFFFFF is returned on any of the PCI config cycles or direct load requests,

check the freeze state of the bridge (Bridge Interrupt Status, BAR 0 + 0x1234, bit 25:26)6. If frozen, bypass the configuration of that adapter, leave in freeze mode, and continue with PCI

probing.7. Before turning control over to AIX, firmware must return all slots to a freeze mode disabled state. 8. Device drivers that support freeze mode detection will re-enable freeze mode for their respective

adapters on a slot-by-slot basis, using the ibm,set-eeh-option RTAS call. During PCI hot-plug, the firmware scenario would be similar to steps 3-8 above, but targeted to a singleEADS bridg

p Series 690 Availability

Documents

Transcript of p Series 690 Availability