How Does HBA-Fiberchannel Be Troble Shooted

8/9/2019 How Does HBA-Fiberchannel Be Troble Shooted

1/72

53-1000885-02 January 30, 2009

Fibre Channel HBATroubleshooting Guide

Supporting Models 415 425 815 825


2/72

Copyright © 2006-2009 Brocade Communications Systems, Inc. All Rights Reserved.

Brocade, Fabric OS, File Lifecycle Manager, MyView, and StorageX are registered trademarks and the Brocade B-wing symbol,DCX, and SAN Health are t rademarks of Brocade Communications Systems, Inc., in the United States and/or in other countries.All other brands, products, or service names are or may be trademarks or service marks of, and are used to identify, products orservices of their respective owners.

Notice: This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerningany equipment, equipment feature, or service offered or to be offered by Brocade. Brocade reserves the right to make changes tothis document at any time, without notice, and assumes no responsibility for its use. This informational document describesfeatures that may not be currently available. Contact a Brocade sales office for information on feature and product availability.Export of technical data contained in this document may require an export license from the United States government.

The authors and Brocade Communications Systems, Inc. shall have no liability or responsibility to any person or entity withrespect to any loss, cost, liability, or damages arising from the information contained in this book or the computer programs thataccompany it.

The product described by this document may contain “open source” software covered by the GNU General Public License or otheropen source license agreements. To find-out which open source software is included in Brocade products, view the licensingterms applicable to the open source software, and obtain a copy of the programming source code, please visithttp://www.brocade.com/support/oscd .

Brocade Communications Systems Incorporated

Document History

Corporate and Latin American Headquarters

Brocade Communications Systems, Inc.1745 Technology DriveSan Jose, CA 95110Tel: 1-408-333-8000Fax: 1-408-333-8101E-mail: [email protected]

Asia-Pacific Headquarters

Brocade Communications Singapore Pte. Ltd.30 Cecil Street#19-01 Prudential TowerSingapore 049712SingaporeTel: +65-6538-4700Fax: +65-6538-0302E-mail: [email protected]

European HeadquartersBrocade Communications Switzerland SàrlCentre SwissairTour B - 4ème étage29, Route de l'AéroportCase Postale 105CH-1215 Genève 15

SwitzerlandTel: +41 22 799 5640Fax: +41 22 799 5641E-mail: [email protected]

Title Publication number Summary of changes Date

Fibre Channel HBA TroubleshootingGuide

53-1000885-01 New document December 2008

Fibre Channel HBA TroubleshootingGuide

53-1000885-02 Revised with corrections. January 2009


3/72

Brocade Fibre Channel HBA Troubleshooting Guide iii53-1000885-02

Contents

About this Document

In this chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii

How this document is organized . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii

Supported hardware and software . . . . . . . . . . . . . . . . . . . . . . . . . . viii

What’s new in this document. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix

Document conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix

Notice to the reader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi

Additional information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiGetting technical help. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii

Document feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii

Chapter 1 Introduction to troubleshooting

In this chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

How to use this manual for troubleshooting . . . . . . . . . . . . . . . . . . . . 1

Gathering problem information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

Chapter 2 Isolating Problems

In this chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

General problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Resolving installation problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12Verifying installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12Errors when installing driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13Installer program does not autorun from CD (Windows only). . 13“Files needed for bfad.sys” message appears when removing driver13Cannot roll back driver on all HBA instances using Device Manager14Host not booting from remote LUN. . . . . . . . . . . . . . . . . . . . . . . 14

Confirming driver package installation. . . . . . . . . . . . . . . . . . . . 14Host system freezes or crashes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

HCM GUI fails to connect with HCM agent . . . . . . . . . . . . . . . . . . . . 17

Verifying Fibre Channel links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

Chapter 3 Tools for Collecting Data

In this chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23


4/72

iv Brocade Fibre Channel HBA Troubleshooting Guide

53-1000885-02

For detailed information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Data to provide support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

Collecting data using host system commands . . . . . . . . . . . . . . . . . 24

Collecting data using BCU commands and HCM . . . . . . . . . . . . . . . 25Using Support Save . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25HBA data collection using HCM . . . . . . . . . . . . . . . . . . . . . . . . . 28Collecting data using BCU commands . . . . . . . . . . . . . . . . . . . . 28

Collecting data using Fabric OS commands . . . . . . . . . . . . . . . . . . . 28

Event logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30Host system logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30HCM logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32Syslog support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34Windows Event Log support . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34Port statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

IOC statistics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35Fabric statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36Remote port statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36FCIP initiator mode statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . 37Logical port statistics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37Virtual port statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38Quality of service (QoS) statistics . . . . . . . . . . . . . . . . . . . . . . . . 39

Diagnostics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40Beaconing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40Loopback tests. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41PCI loopback test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42Memory test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42HBA temperature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43Ping end points . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43Trace route . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44Echo test. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44SCSI test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

Collecting SFP data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45SFP diagnostics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45Port power on management (POM) . . . . . . . . . . . . . . . . . . . . . . 46

Collecting port data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46Base port properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46Remote port properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46Logical port properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47Virtual port properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47Port log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47Port list . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47Port query . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47Port speed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

Authentication settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48Displaying settings through HCM . . . . . . . . . . . . . . . . . . . . . . . . 48Displaying settings through BCU. . . . . . . . . . . . . . . . . . . . . . . . . 48


5/72


6/72

vi Brocade Fibre Channel HBA Troubleshooting Guide

53-1000885-02


7/72

Brocade Fibre Channel HBA Troubleshooting Guide vii53-1000885-02

About this Document

In this chapter

• How this document is organized . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii• Supported hardware and software. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii• What’s new in this document . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix• Document conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix• Notice to the reader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi• Additional information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi• Getting technical help . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii• Document feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii

How this document is organized

This document is organized to help you find the information that you want as quickly and easily aspossible.

The document contains the following components:

• Chapter 1, “Introduction to troubleshooting” provides an introduction and approach totroubleshooting HBA problems, as well as tips for gathering problem information. A checklist isalso provided to verify that required procedures have been followed during installation.

• Chapter 2, “Isolating Problems” provides information on common HBA problems andprocedures to diagnose and recover from these problems

• Chapter 3, “Tools for Collecting Data” provides a summary of diagnostic and monitoring toolsavailable through the HCM, Brocade Command Line Utility (BCU), fabric switch operatingsystem and host system to help you isolate and resolve HBA-related problems.

• Chapter 4, “Performance optimization” contains guidelines for optimizing HBA performance onyour host system.

NOTE

This publication is a companion guide to be used with the Brocade Fibre Channel HBA Administrator’s Guide . That publication provides detailed information on HBA monitoring anddiagnostic tools in Host Connectivity Manager (HCM) and the BCU.


8/72

viii Brocade Fibre Channel HBA Troubleshooting Guide

53-1000885-02

Supported hardware and software

This section describes HBA hardware and software support.

HBA support The following Fibre Channel host bus adapters (HBAs) are supported in this release.

• Brocade 815. Single-port HBA with a per-port maximum of 8 Gbps using an 8 Gbps SFP+.• Brocade 825. Dual-port HBA with a per-port maximum of 8 Gbps using an 8 Gbps SFP+.• Brocade 415. Single-port HBA with a per-port maximum of 4 Gbps using a 4 Gbps SFP.• Brocade 425 Dual-port HBA with a per-port maximum of 4 Gbps using a 4 Gbps SFP.

Notes:

• This publication only supports the HBA models listed above and does not provide informationabout the Brocade 410 and 420 Fibre Channel HBAs, also known as the Brocade 400 FibreChannel HBAs.

• Although you can install an 8 Gbps SFP+ into a Brocade 415 or 425 HBA, only 4 Gbpsmaximum port speed is possible.

• Install only Brocade-branded SFPs in these HBAs.

Fabric OS and switch support For a current list of compatible servers, switches, storage, and applications, refer to the BrocadeHBA web site at www.brocade.com/hba .

Host operating system support The following operating systems support Brocade Host Connectivity Manager (HCM), BrocadeCommand Line Utility (BCU), and HBA drivers:

• Windows Server 2003, version R2 with SP2• Windows Server 2008• Windows NT (HCM support only)• Windows 2000 (HCM support only)• Linux RHEL4, RHEL5, SLES9, and SLES10• Solaris 10 (x86 and SPARC)• VMware ESX Server 3.5

NOTEDrivers, BCU, and HCM Agent are supported only on the VMware “console” Operating System.HCM is supported only on the guest operating system on VMware.

• Windows Vista (HCM only)• Windows XP (HCM only)


9/72

Brocade Fibre Channel HBA Troubleshooting Guide ix

53-1000885-02

NOTESpecific operating system service pack levels and other patch requirements are detailed in thecurrent HBA release notes.

What’s new in this document

This is a new document. For further information about new features not covered in this documentand documentation updates for this release, refer to the HBA release notes.

Document conventions

This section describes text formatting conventions and important notice formats used in thisdocument.

Text formatting The narrative-text formatting conventions that are used are as follows:

bold text Identifies command namesIdentifies the names of user-manipulated GUI elementsIdentifies keywords and operandsIdentifies text to enter at the GUI or CLI

italic text Provides emphasisIdentifies variablesIdentifies paths and Internet addressesIdentifies document titles

code text Identifies CLI outputIdentifies command syntax examples

For readability, command names in the narrative portions of this guide are presented in mixedlettercase: for example, switchShow. In actual examples, command lettercase is often alllowercase. Otherwise, this manual specifically notes those cases in which a command is casesensitive.


10/72


11/72

Brocade Fibre Channel HBA Troubleshooting Guide xi

53-1000885-02

For definitions of SAN-specific terms, visit the Storage Networking Industry Association onlinedictionary at:

http://www.snia.org/education/dictionary

Notice to the reader

This document may contain references to the trademarks of the following corporations. Thesetrademarks are the properties of their respective companies and corporations.

These references are made for informational purposes only.

Additional information

This section lists additional Brocade and industry-specific documentation that you might findhelpful.

Brocade resourcesFor HBA resources, such as product information, software, firmware, and documentation, visit theBrocade HBA web site at www.brocade.com/hba .

To get up-to-the-minute product information, join Brocade Connect. Go tohttp://www.brocadeconnect.com to register at no cost for a user ID and password.

For practical discussions about SAN design, implementation, and maintenance, you can obtainBuilding SANs with Brocade Fabric Switches through:

http://www.amazon.comFor additional Brocade documentation, visit the Brocade Web site:

http://www.brocade.com

Corporation Referenced Trademarks and Products

Microsoft Corporation Windows, Windows Server 2003, Windows Server 2008, Vista,XP, PE for Windows, Hyper V for Windows, Windows AutomatedInstallation Kit (WAIK)

Sun Microsystems, Inc. Solaris

Red Hat Inc. Red Hat Enterprise Linux (RHEL)

Novell, Inc SUSE Linux Enterprise Server (SLES)

VMware Inc. ESX Server

SPARC International, Inc SPARC

http://www.snia.org/education/dictionaryhttp://www.brocadeconnect.com/http://www.amazon.com/http://www.brocade.com/http://www.brocade.com/http://www.amazon.com/http://www.brocadeconnect.com/http://www.snia.org/education/dictionary


12/72

xii Brocade Fibre Channel HBA Troubleshooting Guide

53-1000885-02

Other industry resourcesFor additional resource information, visit the Technical Committee T11 Web site. This Web siteprovides interface standards for high-performance and mass storage applications for FibreChannel, storage management, and other applications:

http://www.t11.org

For information about the Fibre Channel industry, visit the Fibre Channel Industry Association Website:

http://www.fibrechannel.org

Getting technical help

Contact your Brocade HBA support supplier for hardware, firmware, and software support,including product repairs and part ordering. Provide the following information:

1. General information:

• Brocade HBA model number• Host operating system version• Software name and software version, if applicable• syslog message logs• bfa_supportsave output.

To expedite your support call, use the bfa_supportsave feature to collect debuginformation from the driver, internal libraries, and firmware. You can save valuableinformation to your local file system and send it to support personnel for furtherinvestigation. For details on using this feature, refer to “Using Support Save” on page 25 .

• Detailed description of the problem, including the switch or fabric behavior immediatelyfollowing the problem, and specific questions.

• Description of any troubleshooting steps already performed and the results.

2. HBA serial number:

The HBA serial number and corresponding bar code are provided on the serial number labelillustrated below. The serial number label is located on the end of the HBA opposite from theSFP receivers.

You can also display the serial number through the following HCM dialog boxes and BCUcommands:

• HBA Properties tab in HCM.Select an HBA in the device tree, then click the Properties tab in the right pane.

• BCU adapter --list command.

*FT00X0054E9*

FT00X0054E9

http://www.t11.org/http://www.fibrechannel.org/http://www.fibrechannel.org/http://www.t11.org/


13/72

Brocade Fibre Channel HBA Troubleshooting Guide xiii

53-1000885-02

NOTEFor details on using HCM and BCU commands, refer to the Brocade Fibre Channel HBA

Administrator’s Guide .

3. Port World-Wide Port Name (PWWN).

Determine this through the following resources:

• Label located on the end of the HBA opposite the SFP receiver slots. This label providesthe WWPN for each port.

• Brocade BIOS Configuration Utility.Select the appropriate HBA port from the initial configuration utility screen, then selectAdapter Settings to display the WWNN and PWWN for the port. For details, refer to theBrocade Fibre Channel HBA Installation and Reference Manual .

• HCM Port Properties dialog box.This dialog box displays the PWWN for each HBA port. To display this dialog box, click aport in the device tree, then click the Properties tab in the right pane.

• The following BCU commands:adapter --query

This command displays HBA information. The parameter is the HBA’s serialnumber.

port --list

This command lists all the physical ports on the HBA along with their basic attributes. The parameter is the HBA’s serial number.

Document feedback Quality is our first concern at Brocade and we have made every effort to ensure the accuracy andcompleteness of this document. However, if you find an error or an omission, or you think that atopic needs further development, we want to hear from you. Forward your feedback to:

[email protected]

Provide the title and version number of the document and as much detail as possible about yourcomment, including the topic heading and page number and your suggestions for improvement.


14/72

xiv Brocade Fibre Channel HBA Troubleshooting Guide

53-1000885-02


15/72

Brocade Fibre Channel HBA Troubleshooting Guide 153-1000885-02

Chapter

1Introduction to troubleshooting

In this chapter • How to use this manual for troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . 1• Gathering problem information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

How to use this manual for troubleshooting A Fibre Channel HBA is one component in a SAN consisting of Fibre Channel switches, storagedevices, and the fiber optic cabling and connections for these components. Although there may bea problem in the HBA or an HBA component, such as a small form factor pluggable (SFP) opticaltransceiver, the problem could also originate in another SAN component or connections betweenthese components. Before removing and replacing HBAs, launching HBA diagnostics, or evengathering statistics on HBA operation, it is important that you perform the following tasks:

1. Fully describe the problem and gather complete information about the symptoms that suggesta problem exists. Refer to “Gathering problem information” on page 2 .

2. Isolate or resolve the problem by first using information in Chapter 2, “Isolating Problems” .

• First study information in the general HBA problem symptoms, causes, and fixes or actionsin Table 2 on page 6 . This table should provide help on many general problems that youmay encounter with HBA operation. Fixes and actions often reference the BCU commands,

HCM features, and host operating system commands described in Chapter 3, “Tools forCollecting Data” to gather data for problem isolation or resolution.

• If you still require more information to isolate problems, use the following sections inChapter 3. Note that these sections are referenced from Table 2 (Troubleshooting GeneralProblems) when appropriate to further isolate problems. “Resolving installation problems” on page 12 . “Host system freezes or crashes” on page 16 . “Verifying Fibre Channel links” on page 20 .

3. Use the BCU commands, HCM features, and host operating system commands described inChapter 2, “Isolating Problems” to gather data to help you isolate problems. Although many ofthese tools are specifically referenced as actions for problems described in Table 2

(Troubleshooting General Problems) in Chapter 2, many more are included that can providehelpful data, such as event logs, operating statistics, and diagnostics. Note that Table 5 onpage 24 in Chapter 3 provides a list of useful host system commands for each supportedoperating system that you can use to gather data.

4. Consider these factors when isolating and resolving the problem:

• Can the issue be resolved by using the latest supported combination of host system BIOS,operating system, operating system updates, or HBA drivers?


16/72


17/72

Brocade Fibre Channel HBA Troubleshooting Guide 3

53-1000885-02

Gathering problem information 1

l

The bfa_supportSave and FOS supportsave commands can provide current information for thetopology map. Also, consider using the Brocade SAN Health products to provide information onyour SAN environment, including an inventory of devices, switches, firmware versions, and SANfabrics, historical performance data, zoning and switch configurations, and other data. Clickthe Support tab on www.brocade.com for more information on these products.

• Run appropriate diagnostic tools for storage targets.• Use additional HCM, BCU, host system, and Fabric OS commands summarized in Chapter 3,

“Tools for Collecting Data” to gather statistics and problem data on the HBA, host, FibreChannel links, and connected devices.

• Determine what has changed in the SAN. For example, if the SAN functioned without problemsbefore installing the HBA, then the problem is most likely in the HBA installation orconfiguration, HBA hardware, or HBA driver package. Other examples to investigate could bechanges in the switch or storage system firmware, an offline switch, or a disconnected or faultycable between the HBA, switch, or storage controller fiber optic ports.

• Record the time and frequency of symptoms and the period of time symptoms have beenobserved.

• Determine if unexpected behavior is intermittent or always present.

• List steps that have been taken to troubleshoot the problem, including changes attempted toisolate the problem.

TABLE 1 Topology map details

Component How to identify

HBA Model, World-Wide Name (WWN),and driver release level.

Fibre Channel switches. Model, WWN, and Fabric OSversion.

Fiber optic links between HBA,switches, and storage ports.

Port WWNs connected to all links.

Host hardware Model and hardware revision.


18/72

4 Brocade Fibre Channel HBA Troubleshooting Guide

53-1000885-02

Gathering problem information


19/72


Chapter

2Isolating Problems

In this chapter • General problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5• Resolving installation problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12• Host system freezes or crashes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16• HCM GUI fails to connect with HCM agent. . . . . . . . . . . . . . . . . . . . . . . . . . . 17• Verifying Fibre Channel links. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

General problemsTable 2 on page 6 describes general problems related to HBA operation, possible causes, andrecommended actions that may fix the problem. Recommended actions may refer you toinformation in the following locations as appropriate to gather information to further isolate andresolve the problem.

• Chapter 3, “Tools for Collecting Data” in this manual.Contains procedures to perform HBA diagnostics, display HBA statistics and event logs, andcollect data for troubleshooting using BCU commands, HCM options, Fabric OS commands,and your host system commands.

• Fabric OS Administrator’s GuideProvides detailed information on features available on Brocade storage area network (SAN)products, and how to configure and administer these products

• Fabric OS Command Reference Manual.Provides detailed descriptions of command line interface commands to help systemadministrators and technicians operate, maintain, and troubleshoot Brocade SAN products.

• Fabric OS Troubleshooting and Diagnostic Guide. Provides help with isolating problems in other Brocade SAN components.

• Your host’s operating system documentation and help system.Provides details on commands for gathering information and isolating problems.


20/72


53-1000885-02

General problems

TABLE 2 Troubleshooting general problems

Symptom Possible Cause Fix or Action

HBA not reported under server’s PCI subsystem. 1 Card not seated properly.2 Server slot issues.

3 HBA not compatible with hostoperating system or connectedstorage systems.

1 Execute host operating systemcommand to list PCI devices. Refer to

the “List PCI Devices” row in Table 5 on page 24 . If the HBA is not listed,perform the following steps.

2 Reseat the HBA.3 Replace the HBA with an HBA in

known working condition todetermine whether there is a slotmalfunction.

4 Verify compatibility by reviewing theBrocade Server ConnectivityCompatibility Matrix . To find thisdocument, log into Brocade Connecton www.brocade.com, then selectthe Compatibility Information quicklink under Documentation Library .

No HBAs reported when using the adapter --list BCU command:

1 HBA not reported under server’sPCI sub-system.

2 HBA driver not loaded.

1 Execute your host’s operating systemcommand to list PCI devices. Refer tothe “List PCI Devices” row in Table 5 on page 24 . If the HBA is not listed inthe output from this command, go onto the next step.

2 Refer to “HBA not reported underserver’s PCI sub-system” under the“Symptom” column in this table.

3 The HBA driver may not be loaded.Refer to “Confirming driver packageinstallation” on page 14 for methodsto verify driver installation.


21/72


53-1000885-02

General problems 2

Port link does not come up. 1 SFP or cable problems.2 Link between HBA and switch is

down.3 Switch port is disabled or switch isdisabled.

4 HBA port speed or topologymis-match with the switch port.

5 Non-Brocade branded SFPinstalled.

1 Ensure that the SFPs and cables areconnected properly on both HBA andswitch side. Check for any cabledamage.

2 Verify HBA side link status byexecuting the BCU port --list command. Check the FC Addr fieldfor an address and the State field forLinkup. For details on using thiscommand, refer to “Port list” onpage 47 .

3 Execute either the Fabric OSswitchShow or portShow commandson the attached switch to ensurethat the switch or individual port isnot disabled or offline.

4 Check the port topology setting onthe switch using the Fabric OSportCfgShow command to ensurethat Locked L_Port is OFF.Use the portCfgLport command tochange the setting to OFF if required.

Check the switch port speed usingthe Fabric OS portCfgShow command to verify that Speed iseither AUTO or matches the speed ofthe attached HBA por t (for example,the speed setting for both ports is 4Gbps).

Check port speed on the HBA withthe BCU port --list or port --query commands to display the currentand configured speed. Refer to “Portspeed” on page 48 and “Port query” on page 47 for details on using thesecommands.

5 If non-brocade branded SFPs areinserted on the HBA side or 8 Gbpsswitch or director, the port link willnot come up.

On the switch, execute the Fabric OSswitchShow command to verify that

“Mod_Inv” (invalid module) does notdisplay for the port state.




22/72


53-1000885-02

General problems

Port link does not come up (continued) On the HBA, execute the port --list and port --query BCU commands.Refer to “Port list” on page 47 and“Port query” on page 47 . If anunsupported SFP is detected, the“Sfp” field displays “us”(unsupported SFP) for port --list andthe “Media” field displays“Unsupported SFP” for port --query .

For additional actions and fixes for theport link not coming up, refer to “VerifyingFibre Channel links” on page 20 .

Loss of sync and loss of signal errors in portstatistics (refer to “Port statistics” on page 34 ).

Possible physical link problem. Ensure that the SFPs and cables areconnected properly on both HBA andswitch side. Check for any cable damage.

Refer to “Verifying Fibre Channel links” onpage 20

Fabric authentication failures Authenticating configuration is incorrect. • Check authentication settings on theswitch and HBA. For the switch,execute the authutil --show Fabric OScommand. For the HBA, execute theBCU auth --show command (refer to“Authentication settings” onpage 48 ).

• Use the BCU auth --show command on the HBA and Fabric OSauthutil --show command on switch.

• Check the shared secretconfiguration on the attached switchand HBA. For the switch, execute thesecAuthSecret Fabric OS command.For the HBA, execute the auth –-secret BCU command (refer to“Authentication settings” onpage 48 ) for details on using theauth-secret command.




23/72


53-1000885-02

General problems 2

Target not visible. 1 No fabric connectivity between HBAand target or target is not online.

2 The target and the HBA are not onthe same zone.3 The HBA driver is not loaded.4 There is a problem with remote

port.

1 Execute the Fabric OS nsAllShow command on the attached switch toverify that the target and the hostare online in the fabric andregistered in the name server.

2 Execute the Fabric OS cfgActvShow command on the attached switchand verify that the host and targetare in the same zone (either usingdomain area members, port areamembers, or port or node WWNs).


4 Verify that the remote target port(rport) is reporting itself online bycomparing rport online and rportoffline statistics (refer to “Remoteport statistics” on page 36 ). Therport online counter should be onegreater than the rport offlinecounter. If not, clear the countersand try connecting to the remoteport again. Verify the rport onlineand rport offline statistics again.

LUN not visible 1 Missing or improper storage arrayLUN masking setting.

2 HBA driver not loaded.

1 Check LUN mapping and maskingusing storage array configurationtools.


I/Os are not failing over immediately on a pathfailure in MPIO setup.

Improper driver mpiomode setting. Execute the port --query BCUcommand and ensure fcpim MPIO modeis enabled (which implies zero Path TOVvalues) or that fcpim MPIO mode isdisabled with the expected “Path TOV”settings (default is 10 seconds).

Unable to create more than 126 Virtual (NPIV)ports.

The switch’s maximum NPIV limit hasbeen reached.

Execute the Fabric OS configure command on the attached switch andchange the Maximum logins per port parameter under the F_Port loginparameters menu to increase the

maximum NPIV I/Ds allowed per port.

On Linux, the maximum IOPS numbers are verylow.

The amount of disk I/O requests arecausing low throughput and high latency.

Refer to “Linux tuning” on page 51 forsuggestions to optimize HBA performancein Linux systems.

On VMware, the maximum IOPS numbers are verylow.

The amount of disk I/O requests arecausing low throughput and high latency.

Refer to “VMware tuning” on page 53 forsuggestions to optimize HBA performancein Vmware systems.




24/72


53-1000885-02

General problems

QoS performance issues 1 QoS not enabled on both switchand HBA port.

2 QoS zones not created properly onswitch for high, medium, and lowpriority targets.

1 Verify if QoS is enabled for an HBAport using the qos -query BCU command. Verify if it is enabledon the switch using the islShow command.

2 Verify zones on the switch usingcfgActvShow command.

3 Verify that QoS is configured onswitch using instructions in theBrocade Fabric OS Administrator’sGuide.

The HBA is not showing in the fabric. There is a problem in the fabric or aprotocol issue between the HBA andfabric.

Check fabric statistics. Refer to “Fabricstatistics” on page 36 for methods todisplay fabric statistics for the HBA.• If counts for FLOGI sent and FLOLGI

accept fabric statistics do not match,

suspect fabric problem or protocolissue between HBA and fabric.

• If fabric offline counts increase andfabric maintenance is not occurring,this may indicate a serious fabricproblem. Refer to your switchtroubleshooting guide.

Errors when installingbfa_driver_linux--.noarch.rpm driverpackage.

Appropriate distribution kerneldevelopment packages are not installedon your host system for the currentlyrunning kernel.

Refer to “Errors when installing driver” onpage 13 for more information to isolatethis problem.

Installer program does not autorun (Windowsonly).

Autorun is not enabled on your system. Refer to “Installer program does notautorun from CD (Windows only)” onpage 13 for more information to isolatethis problem.

Host not booting from remote LUN. Boot from LUN not configured correctly. Refer to “Host not booting from remoteLUN” on page 14 for more information toisolate this problem.

Host system freezes or crashes. 1 Fault fiber optic cabling andconnections.

2 Faulty or unseated SFPs orunsupported SFPs.

3 Conflicts with port operating speedor topology of attached devices.

4 HBA not compatible with hostsystem.

Refer to “Verifying Fibre Channel links” onpage 20 for more information to isolatethis problem.

Virtual devices not listed in name server. • Problem in the fabric or a protocolissue between the HBA and fabric.

• NPIV is not supported or is disabledon the switch

Check virtual port statistics, such asFDISK sent, FDISK accept, and No NPIVsupport statistics. Refer to “Virtual portstatistics” on page 38 for methods todisplay virtual port statistics.

Operating system errors (blue screen). HBA driver not loaded. Refer to “Confirming driver packageinstallation” on page 14 for methods toverify driver installation.




25/72


53-1000885-02

General problems 2

“Failed to connect to agent on host...” error whenusing HCM.

The client application cannot connect tothe HCM Agent listening on theconfigured port for the following reasons:• The agent is not running.• The agent not accepting

connections on the expected port.• The agent is not listening on the

expected port.• Communication between the client

and agent is blocked by a firewallpreventing access to the port(usually only a consideration forremote HCM management).

Refer to “HCM GUI fails to connect withHCM agent” on page 17 to isolate andresolve the problem.

The HBA not registering with the name server orcannot access storage.

1 HBA cannot log in to the nameserver.

2 HBA has a problem registering with

the name server.3 HBA has a problem querying the

name server for available storage.

Display logical port statistics (refer to“Logical port statistics” on page 37 .Check for the following errors:

1 Increasing name server port login(NS PLOGI) error rejects andunknown name server port loginresponse (NS login unknown rsp)errors means that HBA most likelycannot log in to the name server.

2 Increasing errors of the followingtypes indicates that the HBA has aproblem registering with the nameserver.• Name server register symbolic

port name identifier (NSRSPN_ID) errors.

• Name server register symbolicport name identifier response(NS RFT_ID rsp) errors

• Name server register symbolicport name identifier responserejects (NS RFT_ID rejects).

3 Increasing name server get all portID response (NS GID_FT rsp), rejects(NS_GID FT rejects), or unknownresponses (NS_GID FT unknown rsp)means that the HBA has a problemquerying the name server foravailable storage.

Driver event messages appearing in host systemlog files.

Various causes and severity levels. • Follow recommended action inmessage.

• Resolve critical-level messages andmultiple major or minor-levelmessages relating to the same issueas soon as possible.

• For details on event messages, referto “Event logs” on page 30 .




26/72


53-1000885-02

Resolving installation problems

NOTEIf troubleshooting actions in Table 2 do not resolve problems, check the installed version of the HBA(chip revision) and driver (fw version) using the adapter --query BCU command. To use thiscommand, refer to “Collecting data using BCU commands” on page 28 . Refer to release notes

posted on the Brocade HBA web site (www.brocade.com/hba) for known problems relating to theHBA and driver versions.

Resolving installation problemsThis section provides a summary of common HBA problems that may occur during or immediatelyafter installing HBAs and describes methods to resolve these problems.

Verifying installationProblems with HBA operation may be due to improper hardware or software installation,incompatibility between the HBA and your host system, unsupported SFPs installed on the HBA,improper fiber optic cable connected to the fabric, or the HBA not operating within specifications.Determine if problems may exist because of these factors by reviewing your installation withinformation in the Brocade Fibre Channel HBA Installation and Reference Manual listed in Table 3 .

“Files needed for bfad.sys” message appearswhen removing driver

Installing the driver using the driverinstaller program (bfa_insta ller.exe) orBrocade Software Installer (GUI orcommand-based application), thensubsequently removing the driver usingthe Device Manager.

Do not uninstall the driver using theDevice Manager if you have used theBrocade installer programs to installdriver instances. Always use the Brocadeinstaller programs to remove the driver.Refer to ““Files needed for bfad.sys”message appears when removing driver” on page 13 for more information.

Cannot roll back driver on all HBA instances usingDevice Manager

Installing the driver using the Brocadedriver installer program(bfa_installer.exe) or Sof tware Installer(GUI or command-based application),then rolling back driver HBA instancesusing the Device Manager.

• Install the driver for each HBAinstances using the Device Manager,then roll back the driver using DeviceManager.

• Use the driver installer program(bfa_installer.exe) or BrocadeSoftware Installer (GUI orcommand-based application) to

install or upgrade the driver, thenuse the Brocade SoftwareUninstaller to roll back drivers on allHBA inst ances in one-step.

• Refer to “Cannot roll back driver onall HBA instances using DeviceManager” on page 14 for moreinformation.




27/72


28/72


53-1000885-02

Resolving installation problems

Cannot roll back driver on all HBA instances using Device ManagerWhen using the Windows Device Manager, you can only roll back the driver for the first HBAinstance.

This occurs if you perform the following sequence of steps:

1. Install the driver using the Brocade driver installer program (bfa_installer.exe) or SoftwareInstaller (GUI or command-based application).

2. Roll back driver HBA instances using Device Manager.

To avoid this problem, use one of the following methods:

• Install the driver for each HBA instances using the Device Manager, then roll back the driverusing Device Manager.

• Use the driver installer program (bfa_installer.exe) or Brocade Software Installer (GUI orcommand-based application) to install or upgrade the driver, then use the Brocade SoftwareUninstaller to roll back drivers on all HBA instances in one-step.

Host not booting from remote LUNIf booting the host from a remote boot device, verify whether “boot over SAN” configuration iscomplete and correct. For example, verify the following:

• A zone is created on the attached switch that contains only the PWWN of the storage systemport where the boot LUN is located and the PWWN of the HBA port.

• BIOS or EFI is enabled to support boot over SAN from a specific HBA port.• BIOS or EFI is configured to boot from a specific LUN.• The host’s operating system, HBA driver, and other necessary files are installed on the boot

LUN.

Confirming driver package installationHBA driver packages from Brocade contain the current driver, firmware, and HCM agent for specificoperating systems. Make sure that the correct package is installed for your operating system. Referto the installation chapter in the Brocade HBA Installation and Reference Manual .

An out-of-date driver may cause the following problems:

• Storage devices and targets not being discovered by the device manager or appearingincorrectly in the host’s device manager.

• Improper or erratic behavior of HCM (installed driver package may not support HCM version).• Host operating system not recognizing HBA installation.• Operating system errors (blue screen).

NOTEIf driver is not installed, try re-installing the driver or re-installing the HBA hardware and then thedriver.

You can use HCM and tools available through your host’s operating system to obtain informationsuch as driver name, driver version, and HBA port WWNs.


29/72


30/72


53-1000885-02

Host system freezes or crashes

• # modprobe -l bfaThis verifies that the module has loaded. If bfa displays, the module has been loaded to thesystem.

Solaris Verify if the HBA driver installed successfully using the following commands.

• pkgchk -nv bfaThis checks for and lists the installed HBA driver package files.

• pkginfo -l bfaThis displays details about installed Brocade HBA (bfa) drivers. Look for information as in thefollowing example. Note that the VERSION may be different, depending on the driver version youinstalled. The ARCH and DESC information may also be different, depending on your host systemplatform. If the HBA driver package is installed, bfa_pkg should display with a “completelyinstalled.” status.

PKGINST: bfa NAME: Brocade Fibre Channel Adapter Driver CATEGORY: system ARCH: sparc&i386 VERSION: alpha_bld31_20080502_1205 BASEDIR: / VENDOR: Brocade DESC: 32 bit & 64 bit Device driver for Brocade Fibre Channel adapters PSTAMP: 20080115150824 INSTDATE: May 02 2008 18:22 HOTLINE: Please contact your local service provider STATUS: completely installed

VMware

Verify if the HBA driver installed successfully using the following commands:

• vmkload_mod -lThis lists installed driver names, R/O and R/W addresses, and whether the ID is loaded. Verifythat an entry for bfa exists and that the ID loaded.

• cat /proc/vmware/versionThis displays the latest versions of installed drivers. Look for a bfa entry and related buildnumber.

Host system freezes or crashesIf the BIOS and the operating system recognize the HBA, but the host system freezes or crashesduring startup and does not boot, disconnect all devices from the HBA, then reboot the hostsystem.

• If the system does not freeze when rebooted and operates correctly, use the followinginformation to resolve the problem:

- Check for faulty fiber optic cable and cable connections.


31/72


53-1000885-02

HCM GUI fails to connect with HCM agent 2

- Try rebooting the system without any connectivity to the switch. This will help isolate anyhang caused by switch and device interactions.

- Reseat SFPs in the HBA. Determine whether the installed SFPs are faulty by observing LEDoperation by HBA ports. If all LEDs are flashing amber, the SFP is invalid and may not be arequired Brocade model. You can also verify SFP operation by replacing them with SFPs inknown operating condition. If the problem is resolved after replacement, original SFP isfaulty.

- Check for conflicts with attached devices. Verify that data speed (1-8 Gbps) andconnection topology (for example, point-to-point) for devices attached to the HBA arecompatible with settings on the HBA port. Although auto may be set, configuring settingsmanually on the HBA port and devices may allow connection. Also, note that the HBA onlysupports point-to-point connection topology. Refer to the Brocade Fibre Channel HBA

Administrator’s Guide for procedures to configure HBA ports.

NOTEObserve the LEDs by HBA ports. Illuminated LEDs indicate connection, link activity, andconnection speed negotiated with the attached device. Refer to “LED Operation” in the

Specifications chapter of the Brocade Fibre Channel HBA Installation and ReferenceManual .

• If the system freezes perform the following tasks:- Verify if the host system firmware supports PCIe specifications listed in the “Hardware and

Software Compatibility” section, “Introduction” chapter, of the Brocade Fibre Channel HBAInstallation and Reference Manual . If not, download a firmware update to support theHBA.

- On Windows systems, determine when the system freezes during the boot process. If itfreezes as the driver loads, uninstall and reinstall the driver. If it freezes during hardwarerecognition, uninstall both the driver and HBA, then reinstall both.

- Remove the HBA and reboot the system. If the system boots, reinstall the HBA.- Reseat the HBA.- Uninstall and reinstall the driver.- Try installing the HBA into another host system. If the problem does not occur, the HBA

may not be compatible with the original host system. If the problem occurs in the newsystem, replace the HBA.

HCM GUI fails to connect with HCM agent The HCM Agent installs with the HBA driver package on the host system. It is independent of theHCM user interface (client application), which is installed on the local or remote host.

If you receive an error such as “Failed to connect to agent on host...” when using HCM, thisindicates that the client application cannot connect to the HCM Agent listening on the configuredport - normally TCP port 34568. This occurs for one of the following reasons:

• The agent is not running.• The agent not accepting connections on the expected port.• The agent is not listening on the expected port.


32/72


53-1000885-02

HCM GUI fails to connect with HCM agent

• Communications between the client and agent is blocked by a firewall preventing access to theport (usually only a consideration for remote HCM management).

Linux, Solaris, and VMware systems

Perform the following tasks to isolate and resolve the problem.

1. Verify that the agent is running by executing the appropriate “status” command for youroperating system as described in the “Installation” chapter of the Brocade Fibre Channel HBAInstallation and Reference Manual . Refer to the section on modifying HCM agent operation.

2. If you receive a message that the hcmagent is stopped, restarting the agent should resolve theproblem. To restart, use the appropriate “start” command for your operating system which isalso described in the Brocade Fibre Channel HBA Installation and Reference Manual .

Note that one command described in the manual restarts the agent, but the agent will notrestart if the system reboots or the agent stops unexpectedly. Another command restarts theagent, but the agent will restart if the system reboots.

3. Confirm the HCM agent is responding to requests using the expected user password. Executethe following command to connect to the HCM agent and force it to collect the HBA driversupportsave data.

NOTEThis command is a single line. The localhost can be replaced with a different IP address.

wget --no-check-certificatehttps://admin:password@localhost:34568/JSONRPCServiceApp/SupportSaveController.do

If successful, the file SupportSaveController.do (actually a zip format file) will contain the datafrom the HCM agent.

4. If you are managing a VMware host system through HCM from a remote system, the host’sfirewall may be blocking TCP/IP port 34568, which allows agent communication with HCM.

Use the following command to open port 34568:

/usr/sbin/esxcfg-firewall-o 34568,tcp,out,https

Use Windows Firewall and Advanced Service (WFAS) to open port 34568.

NOTE You can change the default communication port (34568) for the agent using procedures in the“Installation” chapter of the Brocade Fibre Channel HBA Installation and Reference Manual.Refer to the section on modifying HCM agent operation.

5. If HCM is still unable to connect to the HCM agent after using the preceding steps, collect thefollowing data and send to your Support representative for analysis:

• Data collected from the previous step in SupportSaveController.do.• Data from the HCM application SupportSave feature. Select Tools > SupportSave to

generate a supportsave file. The data file name and location displays when theSupportSave feature runs.


33/72


53-1000885-02

HCM GUI fails to connect with HCM agent 2

• HBA agent files on the HBA host (where the HCM agent is installed). Collect these filesusing the following command:

tar cvfz hbaagentfiles.tgz /opt/hbaagent

Output collects to hbaagentfiles.tgz.

• Data collected on the HBA host from the bfa_supportsave feature using the followingcommand:

bfa_supportsave

Output collects to a file and location specified when the SupportSave feature runs.

Windows systems

Perform the following tasks to isolate and resolve the problem.

1. Verify that the agent is running by executing the appropriate “status” command for youroperating system described in the “Installation” chapter of the Brocade Fibre Channel HBA

Installation and Reference Manual . Refer to the section on modifying HCM agent operation.2. If you receive a message that the hcmagent is stopped, restarting the agent should resolve the

problem. To restart, use the appropriate “start” command for your operating system which isalso described in the Brocade Fibre Channel HBA Installation and Reference Manual .

Note that one command described in the manual restarts the agent, but the agent will notrestart if the system reboots or the agent stops unexpectedly. Another command restarts theagent, but the agent will restart if the system reboots.

3. If the HCM agent starts, verify which TCP port the agent is listening on by executing thefollowing command at the Windows command prompt:

netstat -nao | findstr 34568

Output similar to the following should display.TCP 0.0.0.0:34568 0.0.0.0:0 LISTENING 1960

1960 in the last column is the process identifier for the Windows process listening on the TCPport. Note that this identifier may be different on your system.

4. Enter the following command to confirm that the process identifier bound to TCP port 34568 isfor the hcmagent.exe process:

tasklist /svc | findstr 1960

The following should display if the identifier from step 3 is bound to TCP port 34568:

hcmagent.exe 1960 hcmagent

5. If you are managing a Windows 2008 host system through HCM from a remote system, thehost’s firewall may be blocking TCP/IP port 34568.

Use Windows Firewall and Advanced Service (WFAS) to open port 34568.

NOTE You can change the default communication port (34568) for the agent using procedures in the“Installation” chapter of the Brocade Fibre Channel HBA Installation and Reference Manual.Refer to the section on modifying HCM agent operation.


34/72


53-1000885-02

Verifying Fibre Channel links

6. If the hcmagent is running and listening on port 34568 and there are no firewall issues (asexplained in step 5 ), but you get the same “Failed to connect to agent on host..." error whenusing HCM, collect the following data. Send this data to your Support representative foranalysis:

• Copies of output from the commands in step 3 and step 4 .• Files from the output directory created after you execute the bfa_supportsave feature.

To collect these files, execute the following command:

bfa_supportsave

Data collected by this command saves to subdirectory named “bfa_ss_out.”

In Windows explorer, right-click this directory and select Send To > Compressed (zipped) .

This creates a zip file that you can send to your Support representative.

• Build information for the HCM application. Select Help > About in HCM to display theversion, build identification, and build date.

• Support data from the HCM application SupportSave feature.Select Tools > SupportSave to generate a supportsave file.

If HCM cannot connect to the agent, a message displays an error (Agent Support Savecould not be collected) and explains that only a basic collection is possible. Messages alsodisplay that provide the location of the zip file created.

By default, a zip file is created in the following location if installed the application inC:\Program Files\BROCADE\FCHBA:

C:\Program Files\BROCADE\FCHBA\client\data\localhost\supportsave

The zip file will have a name similar to the following:

SupportSave_Basic_2008723_0_50_57.zip

Verifying Fibre Channel linksCheck for link problems by observing LED operation for HBA ports. LED operation other thanexpected or LEDs may indicate link problems. For example, all LEDS flashing amber for a portindicates that an invalid non-Brocade SFP may be installed. For details on HBA LED operation, referto the specifications chapter in the Brocade HBA Installation and Reference Manual . If LEDs do notilluminate to indicate an active link, use appropriate Fabric OS and HBA diagnostic commands andHCM options in Table 4 on page 21 . For additional diagnostics commands, refer to Chapter 3,“Tools for Collecting Data” for HCM and BCU commands and the Fabric OS Administrators Guide forFabric OS commands.

NOTEAlso verify LED operation on switch port that is connected to an HBA port. Refer to the switchHardware Reference Manual to analyze LED meaning.

Common link problems can be caused by the following:

• Damaged fiber optic cables. (Note that damaged cables can also cause errors and invalid dataon links.)


35/72


53-1000885-02

Verifying Fibre Channel links 2

• Fiber optic cables may not be rated or compatible with HBA port speeds. Refer to Fibre OpticCable specifications in the Brocade Fibre Channel HBA Installation and Reference Manual .

• Faulty switch or HBA SFPs. Verify if an SFP is the problem by connecting a different link to theHBA port or, if convenient, replace the cable with a cable of known quality. If the errors orinvalid data on the link still indicate a cable problem, the SFP may be faulty. Try replacing theSFP.

• SCSI retries and timeouts determine communication between HBA and storage. Droppedpackets cause timeouts. Packets can drop because of SFP issues on HBA or switch - possiblythe SFP is not compatible with HBA, but is compatible with switch or vice versa. You can runthe BCU port --stats command to display port statistics, such as error and dropped frames.

Table 4 lists HCM options and BCU commands, as well as Fabric OS commands that you can use todetermine link status.

TABLE 4 Tools to determine link status

Application Tool References

HCM • Port Statistics

• Loopback and PCI loopback test• Fibre Channel ping, echo, and trace routetests

• Link Beaconing • Port Properties• SFP information

Chapter 3, “Tools for Collecting

Data”

BCU • BCU fcdiag and diag commands.• Port commands, such as port --stats .

Chapter 3, “Tools for CollectingData”

Switch Fabric OS • switchShow• portShow• portStatsShow• portErrShow• fcpProbeShow• fPortTest

• Chapter 3, “Tools for CollectingData”

• Fabric OS Administrator’sGuide

• Fabric OS Troubleshooting andDiagnostics Guide


36/72


53-1000885-02

Verifying Fibre Channel links


37/72


Chapter

3 Tools for Collecting Data

In this chapter • For detailed information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23• Data to provide support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24• Collecting data using host system commands . . . . . . . . . . . . . . . . . . . . . . . 24• Collecting data using BCU commands and HCM . . . . . . . . . . . . . . . . . . . . . 25• Collecting data using Fabric OS commands . . . . . . . . . . . . . . . . . . . . . . . . . 28• Event logs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

• Statistics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34• Diagnostics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40• Collecting SFP data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45• Collecting port data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46• Authentication settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48• QoS and target rate limiting settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48• Persistent binding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

For detailed informationThis chapter provides basic instruction on tools useful for gathering information to isolate HBAproblems. For more detailed information on using these tools, refer to the following publications:

• Brocade Fibre Channel HBA Administrator’s GuideThe following chapters in this guide cover HBA HCM and BCU monitoring and diagnostics tools:

- Monitoring - Diagnostics and Troubleshooting - Command Line Utility (BCU)

• Fabric OS Troubleshooting and Diagnostics GuideThis guide provides detailed information on collecting troubleshooting information and

isolating general SAN problems between the switch, host systems, and storage systems.• Fabric OS Command Reference Manual

Fabric OS diagnostic and monitoring commands.

• Your host system’s operating system user and administrator’s guides.Host system diagnostics, logs, and system monitoring tools.


38/72


53-1000885-02

Data to provide support

Data to provide support When problems occur requiring support assistance, provide a detailed description of the problem,as well as output collected from the following HCM and BCU tools:

• Support Save• Diagnostics• Port logs• Port statistics and properties• HBA properties• Host operating system commands

Collecting data using host system commandsThe following table describes commands common to each supported operating system that youcan use to gather information for troubleshooting problems. For details on these commands, referto your system’s online help and documentation.

NOTEOutput from all of these commands is captured using the Support Save feature.

TABLE 5 Host system data collection commands

Task Linux Windows VMware Solaris

List PCI devices lspci -vv In Windows registrylocationHKEY_LOCAL_MACHINE\SYSTEM\CurrentContro

lSet\Enum\PCI devconfind pci\*

lspci -vv, esxcfg-info -w prtdiag -v, prtconf -pv

List installed HW details lsdev msinfo32.exeClick the plus sign(+)next to Components toview hardware details.

esxcfg-info -a prtdiag -v, prtconf -pv

Process information ps -efl, top Windows Task Manager,tasklist.exe

ps -efl, top ps -efl, top

Memory usage top, vmstat -m Windows Task Manager,tasklist.exe

top, vmstat -m vmstat -s

Performance monitoring iostat, vmstat, sar Windows Task Manager,perfmon.exe

vmstat, VM Performance:esxtop [first type 'v', 'e'

then enter vm# in the listdown], Disk Performance:esxtop [type 'v' then 'd'].

iostat -nx 1 5, vmstat,mpstat, sar

List for driver modules lsmod driverquery vmkload_mod -l modinfo

To check for Brocade FibreChannel adapter (BFA) drivermodule

lsmod | grep bfa driverquery /v | findstrbfad

vmkload_mod -l modinfo | grep bfa


39/72


53-1000885-02

Collecting data using BCU commands and HCM 3

Collecting data using BCU commands and HCM You can collect a variety of information on installed Brocade HBAs, such as firmware versioninstalled, operational status, port speed, WWN, PCI data, configuration data, flash status, andother details for troubleshooting using BCU commands, HCM menu options, and host operatingsystem commands.

Using Support SaveThe Support Save feature is an important tool for collecting debug information from the driver,internal libraries, and firmware. You can save this information to the local file system and send tosupport personnel for further investigation. Use one of the following options to launch this feature:

• In HCM, launch Support Save through the Tools menu.• Through BCU, enter the bfa_supportsave command.

• Through your internet browser (Internet Explorer 6 or later or Firefox 2.0 or later), you cancollect bfa_supportsave output if you do not have root access, do not have access to filetransfer methods such as FTP and SCP, or do not have access to the Host ConfigurationManager (HCM).

NOTEA bfa_supportsave collection can also occur automatically for a port crash event.

The Support Save feature saves the following information:

System log message location

NOTE: For more information,

refer to “Host systemlogs” on page 30 .

dmesg, /var/log/message*

System Category inWindows Event Viewer(eventvwr.exe)

/var/log/message* , /var/log/vmkernel*, /var/log/vmkwarning*,/proc/vmware/log

dmesg, /var/adm/message*

Show OS distribution info (SuSE) cat /etc/SuSE-release,(RedHat) cat

/etc/redhat-release

systeminfo.exe cat /etc/vmware-release uname -a, cat /etc/release

BFA configuration file location /etc/bfa.conf Windows Registry(HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\bfad\Parameters\Device), HBAFlash

/etc/bfa.conf /kernel/drv/bfa.conf

BFA device file location /dev/bfa* Windows Registry

(HKEY_LOCAL_MACHINE\HARDWARE\DEVICEMAP\Scsi\Scsi Port x)

/dev/bfa* • (Release 1.0) - /devices/pci*/pci*/fibre-channel@0:devctl,

• (Release 1.1 andlater) -

/devices/pci*/pci*/bfa@0:devctl

TABLE 5 Host system data collection commands

Task Linux Windows VMware Solaris


40/72


53-1000885-02

Collecting data using BCU commands and HCM

• HBA model and serial number.• HBA firmware version.• Host model and hardware revision.• All support information.• HBA configuration data.• All operating system and HBA information needed to diagnose field issues• Information about all HBAs in the system.• Firmware and driver traces.• Syslog message logs.• Windows System Event log .evt file.• HCM GUI-related engineering logs• Events• HBA configuration data• Environment information

• Data.xml file• Vital CPU, memory, network resources• HCM Agent (logs, configuration)• Driver logs (bfa_supportsave output)• Core files

NOTEMaster and Application logs are saved when Support Save is initiated through HCM, but not throughBCU.

Initiating Support Save through HCM

Initiate Support Save by selecting Tools > Support Save .

Messages display during the Support Save operation that provide the location of the directorywhere data is saved. If you are initiating Support Save from a remote management station andreceive a warning message that support files and Agent logs could not be collected, the HCM Agentis unavailable on the remote host. Select Tools > Backup to backup data and configuration filesmanually.

For more information and additional options for using this feature, refer to the Brocade FibreChannel HBA Administrator’s Guide .

Initiating Support Save through BCU commands

Use the bfa_supportsave command to Initiate Support Save through the BCU:

bfa_supportsave {output_dir]

where:

[output_dir] An optional parameter that specifies the directory where you want outputsaved. If not specified, output is saved as a directory in the current workingdirectory as bfa_ss_out.


41/72


53-1000885-02

Collecting data using BCU commands and HCM 3

Messages display as the system gathers information. When complete, an output file and directorydisplay. The directory name specifies the date when the file was saved.

For more information on the bfa_supportsave feature, refer to the Host Connectivity Manager(HCM) Administrator’s Guide .

Initiating Support Save through the internet browser

Initiate bfa_supportsave through an internet browser.

1. Open an Internet browser and type the following URL:

https://localhost:34568/JSONRPCServiceApp/SupportSaveController.do

where localhost is the IP address of the server from which you want to collect thebfa_supportsave information.

2. Log in using the factory default user name (admin) and password (password). Use the currentuser name and password if they have changed from the default,

The File Download dialog box displays, prompting you to save the supportSaveController.do file.

3. Click Save and navigate to the location where you want to save the bfa_supportsave file.

Initiating Support Save through a port crash event

If the port crashes and triggers a port crash event, support save data is collected at a system-widelevel. An Application Log message is generated with the following message:

Port Crash Support Save Completed

Port crash events have a CRITICAL severity and you can view the details in the Master Log andApplication Log tables in HCM. For more information on these logs, refer to “HCM logs” on page 32 .

Support Save differences

Following are differences in data collection for the HCM, BCU, and browser applications ofbfa_supportsave:

• BCUCollects only driver-related logs and configuration files.

• BrowserCollects driver-related and HCM Agent logs and configuration files.

• HCMCollects, HCM, driver-related, and HCM Agent logs and configuration files.

bfa_supportsave collection on a port crash event

If the port crashes and triggers a port crash event, support save data is collected at a system-widelevel. An Application Log message is generated with the following message:

Port Crash Support Save Completed


42/72


53-1000885-02

Collecting data using Fabric OS commands

Port crash events have a CRITICAL severity and you can view the details in the Master Log andApplication Log tables in HCM.

HBA data collection using HCMUse the HBA Properties panel in HCM to display information about the HBA, such as its WWN,disabled or enabled status, temperature, installed driver name, driver version, firmware version,and BIOS version. This panel also displays PCI information for the HBA, such as vendor ID, deviceID, subsystem ID, current number of lanes, and PCI generation.

1. Launch HCM.

2. Select an HBA in the device tree

3. Click the Properties tab in the right pane.

Collecting data using BCU commands

Use the BCU adapter command to list and query available HBAs seen by the driver.The bcu adapter --list command lists all HBAs on the system with a brief summary of informationsuch as model number, serial number, and HBA number. Enter the following command:

adapter --list

where:

list Lists all adapters in the system. For each adapter in the system, a briefsummary line is displays.

The adapter --query command displays adapter information, such as the current version of the HBA(chip revision) and driver (fw version), maximum port speed, model information, serial number,number of ports, PCI information, pwwn, nwwn, hardware path, and flash information (such as

firmware version).adapter –-query

where:

ad_id ID of the adapter (HBA) for which you want to query.

Collecting data using Fabric OS commandsUse the following Fabric OS commands on attached Brocade switches to gather information andhelp isolate connectivity and other problems between the HBA, switch, and storage ports. Fordetails on using these commands, refer to the Fabric OS Command Reference Manual .

• authUtilUse this command to display and set local switch authentication parameters.

• cfgShowDisplays zone configuration information for the switch. You can use command output to verifytarget ports (by port WWN) and LUNs that are intended to be accessible from the HBA.


43/72


53-1000885-02

Collecting data using Fabric OS commands 3

• fcpProbeShowUse this command to display the Fibre Channel Protocol daemon (FCPd) device probinginformation for the devices attached to a specified F_Port or FL_Port. This information includesthe number of successful logins and SCSI INQUIRY commands sent over this port and a list ofthe attached devices.

• nsShowUse this command to display local NS information about all devices connected to a specificswitch. This includes information such as the device PID, device type, and the port and nodeWWN.

• zoneshowUse this command without parameters to display all zone configuration information (bothdefined and enabled).

• portErrShowUse this command to display an error summary for all switch ports.

• portLogShowUse this command to display the port log for ports on a switch.

• portLogShowPortUse this command to display the port log for a specified switch port.

• portPerfShowUse this command to display throughput information for all ports on the switch.

• portStatsShowUse this command to display hardware statistics counters for a specific switch port.

• portShowUse this command to display information and status of a specified switch port, including thespeed, ID, operating state, type, and WWN.

• SecAuthSecretUse this command to manage the DH-CHAP shared secret key database used forauthentication.This command displays, sets, and removes shared secret key information fromthe databases

• sfpShowUse this command to display detailed information about specific SFPs installed in a switch.

• switchShowUse this command to display switch and port information. Output may vary depending on theswitch model. Use this information to determine the fabric port WWN and PID connected to anHBA port. Also display topology, speed, and state of each port on the switch.


44/72


45/72


46/72


53-1000885-02

Event logs

Adjust the logging level, or the types of messages logged to your system log that relate to HBAdriver operation, using the following HCM options and BCU commands.

• Adjust logging level using this BCU command.log -–level []

• Adjust logging level using the HCM.

Select Configure > HCM Logging Levels . Select the a level on the Agent Communication Logand HCM Debug Log lists. Values are Trace, Debug, Info, Warning, Error, and Fatal.

NOTEFor details on adjusting logging levels, refer to the Brocade Fibre Channel HBA Administrator’sGuide .

HCM logs You can view data about HBA operation through the following HCM logs. Both of these logs displayon the bottom of the HCM main window. Click the Master Log or Application Log to toggle betweenlogs.

• The Master Log displays informational and error messages during HBA operation. This logcontains the severity level, event description, date and time of event, the function that reportedthe event (such as a specific HBA port or remote target port), WWN of device where eventoccurs, and other information.

• The Application Log displays informational and error messages related to HBA discovery orHCM application issues.

Master Log

The Master Log displays event information in seven fields:

• Sr No.

Sequence number that event occurred in ascending order.

bfal_aen_rport.html Remote port (RPORT) events, suchas a remote port is online, offline, ordisconnected, or the QoS priority orflow ID changed.

hba_error_codes.doc List of error codes and meanings foradapter, port, L_Port, R_Port, itnim,audit, and IOC events.

TABLE 9 Event message files

Event Catalog File Content


47/72


53-1000885-02

Event logs 3

• SeverityEvent severity level (informational, minor, major, or critical).

- Critical-level messages indicate that the software has detected serious problems that willeventually cause a partial or complete failure of a subsystem if not corrected immediately.

Examples of these could be a power supply failure or rise in temperature- Major messages represent conditions that do not impact overall system functionality

significantly. Examples of these could be timeouts on certain operations, failures ofcertain operations after retries, invalid parameters, or failure to perform a requestedoperation.

- Minor messages highlight a current operating condition that should be checked or it mightlead to a failure.

- Information-level messages report the current non-error status of the system components;for example, the online and offline status of a fabric port.

• WWNWorld Wide Name of HBA where event occurred.

• CategoryThe category or type of event. Categories define the component where events occur:

- Adapter - Events relating to the HBA (Adapter).- Port - Events relating to a specific port on the HBA.- LPORT - Events relating to a specific logical port.- RPORT - Events relating to a specific remote initiator or target port.- ItNIM - Events relating to an initiator-target nexus. Examples of these include end to end

target discovery, initiator target connectivity, and loss of connectivity.

- Audit - Audit events.- IOC - Driver and firmware events involving the I/O controller on the HBA.

• SubcategorySubcategory of main category.

• Description, Date, and TimeBrief description of event and date and time when event occurred.

You can block events from display in the Master Log by severity, category, and WWN of HBA. Todisplay the Master Log Filter dialog box, click the Filter button in the Master Log section of the mainHCM screen. Select areas that you want to filter and click OK.

Application Log

The Application Log displays all application-related informational and error messages, as well asthe following attributes.

• Date and time the message occurred.• Severity of the message.• Description of the message.• The agent IP address.


48/72


53-1000885-02

Statistics

Syslog support You can configure the HCM agent to forward events to a maximum of three syslog destinationsusing the Syslog option on the HCM Configure menu. These events will display in the operatingsystem logs for systems such as Solaris and Linux. For procedures to configure syslog destinations,

refer to the Brocade Fibre Channel HBA Administrator’s Guide .

Windows Event Log support HBA events are logged in the Windows Event Log by the HCM agent. Event messages appear withthe date and time the event occurred and the source as “hcmAgent.”

HBA events are logged under the HCM_Event category. Event types display with each event to noteseverity level. From most to least severity, NT events are error, warning, and information. Drivererrors are critical, error, warning, and info.

Statistics You can access a variety of statistics using BCU commands and HCM. Use these statistics tomonitor HBA performance and traffic between the HBA and LUNs and isolate areas that impactperformance and device login.

You can display statistics for the following:

• HBA ports• IO controller• Virtual ports (vport)• Logical ports (lport)• Remote ports (rport)• FCP initiator mode• Fabric (BCU only)• Targets• Security authenticationThis section provides an overview of these statistics and how to access them. For more detail, referto the Brocade Fibre Channel HBA Administrator’s Guide .

Port statisticsUse BCU and HCM to display a variety of port statistics, such as transmitted and received frames

and words, received loop initialization primitive (LIP) event counts, error frames received, loss ofsynchronization, link failure and invalid CRS counts, and end of frame (EOF) errors. Use thesestatistics to isolate link and frame errors. For example, loss of synch and loss of signal errorsindicate a physical link problem. To resolve these problems, check cables, SFPs on the HBA orswitch, and patch panel connections.


49/72


53-1000885-02

Statistics 3

Displaying statistics through BCU

Use the port --stats BCU command to display statistics for a specified HBA port.

port --stats

where:

port_id ID of the port for which you want to display statistics.

Displaying statistics through HCM

Use the Port Statistics dialog box to monitor a variety of port data.

1. Launch the HCM.

2. Select the base HBA port from the device tree window.

3. Click Monitor > Statistics > Port Statistics .

IOC statisticsUse BCU and HCM to display por t-level statistics for the I/O controller through the BCU and HCM.The I/O controller refers to the firmware entity controlling the port. The following types of IOCstatistics are displayed:

• IOC driver• IOC firmware• Firmware IO• Firmware port FPG• Firmware port PHYSM

• Firmware port LKSM• Firmware port SNSM

Displaying IOC statistics through BCU

Use the ioc -–stats command to display IOC statistics.

ioc -–stats

where:

ioc_id ID of the IOC controller for which you want to display statistics.

Displaying IOC statistics through HCM

Use the IOC Statistics dialog box to monitor a variety of port data.

1. Launch the HCM.

2. Select the base HBA port fro

How Does HBA-Fiberchannel Be Troble Shooted

Documents

Transcript of How Does HBA-Fiberchannel Be Troble Shooted