Cisco Networkers 2006 - SAN-3608 - Advanced SAN Troubleshooting
-
Upload
lino-vargas -
Category
Documents
-
view
50 -
download
0
Transcript of Cisco Networkers 2006 - SAN-3608 - Advanced SAN Troubleshooting
1© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Advanced SAN Troubleshooting
SAN-3608
2© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Agenda/First Half
• Overview of Data Gathering Capabilities of SAN/OS
• Network Design—Keep Troubleshooting in Mind
• Analyzer Uses
• Trace and Fabric Manager Tool Uses
• Tie Each Learned Skill to a Lab Exercise
3© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
MDS Switch Troubleshooting Levels
• Command line debugs and show commands
Used by network Level 2–3 engineers
• Analyzers and SPAN
Used by Level 3 engineers
• Fabric manager and device manager
Used by network operators and Level 1 engineers
• General statistics gathering from show commands and GUI managers
Network admins and operators Level 1 engineering
4© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Command Line Debugging
• Available debugs depend on features enabled in SAN/OS
• Many, many different options to select when turning on debugs
• Where is it output going?Logfile—Data file in switch memory
Capture to direct to screen via console, telnet or SSH
• Requires admin privileges to run debugs
• Debugs can only be run from CLI
• No debugging available from Fabric Manager or Device Manager
N
5© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Debug Logging
1. Display captured debug to screenTip: use show debug to see name of debug file
TOP-9216i# show debug logfile networkers_debug
2. Copy debug file off MDS to a serverTOP-9216i# copy log:networkers_debugs ftp:
Enter hostname for the ftp server: 10.91.42.166
Enter username: networkers
Password: networkers
To delete the debug logfile TOP-9216i# clear debug-logfile networkers_debugs
Or
TOP-9216i# undebug all
Or
debug logfile will be cleared and over written when next debug logfile is created, only one debug logfile is allowed by system
6© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Debugs to Direct Telnet Window
• Use a telnet/SSH or console application that will capture the expected output to buffer or file
• Undebug all or no debug of specific debug command is required to turn trace off
• The debugs are not persistent across reboots
• Most debugs are very readable and sensible to understand, some not
666N
7© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Debug Outputs
• Some debugs are self-explanatory, others not
• Drilling down to most single item you wish to view like the interface or VSAN helps in narrowing output
• Debugs are usually run as part of a recreate for an issue such as a bad Fibre Channel ID being assigned or a Host login issue
• Some debugs require expertise in state machine and good general knowledge of FC and SCSI standards
N
Debug Example
8© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Show Commands
• Hundreds of show commands
• Unlike debug commands, much information from show commands is available via the fabric manager and device manager
• Many show commands can be several screen pages. Use show terminal command to see information about your telnet terminal properties.
• Screen will pause at each screen unless terminal length is temporally configured different for current output
• Setting terminal length = 0 helpful in gathering long outputs, i.e. show tech-support detail
N
9© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Design for Troubleshooting
• Leverage VSAN design to support troubleshooting mythology
• Have a SPAN port allocated in port count needs
• Integrate a Cisco Port Analyzer Adapter (PAA) into SAN design
• Provision an Analyzer (Finisar, Agilent, other) in the network, keep it operational with FTP access to extract traces
• Design for Syslog servers and scheduled configuration saves (use CFS to simplify)
10© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
How Do We Troubleshoot the Network
mds-speed.server2
microsoft:server1
TOP-9216i
mds-speed.server3
Server1
Server2
Clarion Cx300
Seagate 92:7a:47-S
Seagate 92:7b:94-S
Seagate 92:7b:c5-S Seagate 67:67:87-S
BOT-9216i
Ethereal PC
Ethernet
Out-of-Band Management Network
PC
qlogic:networkers1
FC Analyzer Remote
SPAN (Port Monitor) on
Cisco L2 Switch
Debugs and Show Commands, Telnet
or Console
Fabric Manager and Device
Manager
Terminal
FC Analyzer Local
Run Ethereal on IP Host
FC Port Channel
FC ISL
iSCSI iS
CS
I
Server3
Span FC Ports, Port Channels,
iSCSI or FCIP Port to SD Port on MDS
PAA
11© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Gathering Protocol Traces for Analysis
• Using built-in FC Analyzer (CLI)
Linux version from Ethereal.com
• Using Ethereal on PC (local and remote)
• Using the MDS port analyzer adapter w/SPAN
• Using an external FC Analyzer tester in line or with SPAN
Non-Disruptive to Switch Operations and Traffic on the SAN
12© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Note: SPAN Is Used for FC Port to FC Port Monitoring
MDS FC Analyzer (SAN/OS Embedded)
• Output is displayed to the console in readable sniffer-like format
• Is only used to monitor Fibre Channel traffic to and from supervisor on the MDS9000
Traffic-like fabric logins, FSPF routing, switch-to-switch control traffic
• Output can go direct to your console screen or to a workstation running Ethereal program
13© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
FC Analyzer Options
• Local or remote—Where to send the trace. Can be to local devices or remote PAA attached to different MDS switch.
• Brief or detailed—Header information vs. full output of frame including hex. Detail is default.
• Limit-captured-frames—Number of frames to capture. Default is FC analyzer will trace 100 frames. Specifying zero is unlimited frame capture.
• Limit-capture-size—Allows to capture N number of bytes of frame. Useful for not capturing frame data when it is not relevant to troubleshooting.
14© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Display-Filter Options
• When not specified, FC analyzer will capture all traffic on VSAN 1Example: fcanalyzer local brief
• To specify a different VSAN, use a display-filter and specify the VSAN to be captured in hex or decimal
Example for 100 (note the two equal signs):
fcanalyzer local display-filter mdshdr.vsan==0x64 or mdshdr.vsan==100
• To capture for a specific address in VSAN 100:Example:
fcanalyzer local brief display-filter ((fc.d_id==64.01.00)or(fc.s_id==64.01.00))
• Recommend to use remote capture method for ease of filter capabilities on the Ethereal GUI
15© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Using Write Option for fcanalyzer
• Using write option sends output of the fcanalyzer to a file on the switch, directory called volatile. This trace file can be then copied off MDS switch and viewed with ethereal app on PC
Example: Capture 250 frames of all traffic on VSAN 200
fcanalyzer local display-filter mdshdr.vsan==0xc8 write volatile:capture.trc limit-captured-frames 250
• The file name on volatile: filesystem will have extra characters appended. Issue following command to see contents of filesystem and then copy the file via tftp/ftp
dir volatile:
copy volatile:capture_00001_20050321172628.trc tftp://<tftp server ip addr>/capture.trc Added Characters
When File Is Written
16© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Use of SPAN Feature
• Used for FC port to FC port analyzing
• Same type of tool as used on Cisco Catalyst® products. Cisco Catalyst uses port monitor.
• Can be left configured on switch
• Ingress and egress ports are sent to an FC-port setup as a SPAN Destination (SD-port type)
• No limits to where the ports are located on the MDS switch network
• Used to output to third-party test equipment or to Cisco Port Analyzer Adapter
N
17© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
FCAnalyzer Local Brief
• Display filters are a must for narrowing output on a busy network. See MDS Config Guide for other filter types.
• Capture is done by default to console screen, so make sure you are able to save output to large capture buffer or log with your telnet application
Capture Is Done in Configuration Mode
Brief Option Used to get Single Line Caption Along with a
Display Filter to Narrow Output to only VSAN 2
Control-C Stops Trace Capture
N
TOP-9216i# conf tEnter configuration commands, one per line. End with CNTL/Z.TOP-9216i(config)# fcanalyzer local brief display-filter mdshdr.vsan==2Warning: Couldn't obtain netmask info (eth2: no IPv4 address assigned).Capturing on eth2 2.829871 00.00.00 -> ff.ff.fe 0x2288 0xffff FC ELS FLOGI 2.853261 ff.ff.fe -> 7e.01.00 0x2288 0xc728 FC ELS ACC (FLOGI) 2.853422 7e.01.00 -> ff.ff.fc 0x22a0 0xffff FC ELS PLOGI 2.853592 ff.ff.fc -> 7e.01.00 0x22a0 0xc729 FC ELS ACC (PLOGI) 2.853565 7e.01.00 -> ff.ff.fd 0x22b8 0xffff FC ELS SCR 2.859648 ff.fc.7e -> 7e.01.00 0xc72c 0xffff FC ELS PLOGI 2.860885 7e.01.00 -> ff.fc.7e 0xc72c 0x22d0 FC ELS ACC (PLOGI) 2.861007 ff.fc.7e -> 7e.01.00 0xc72b 0xffff FC ELS PRLI 2.861175 7e.01.00 -> ff.fc.7e 0xc72b 0x22e8 FC ELS ACC (PRLI) 2.862053 7e.01.00 -> ff.ff.fc 0x2300 0xffff dNS RFT_ID 2.865904 ff.fc.7e -> ff.fc.0a 0xc72e 0xffff SW_ILS SW_RSCN 2.865981 ff.fc.7e -> ff.fc.cb 0xc72f 0xffff SW_ILS SW_RSCN 2.866153 ff.fc.0a -> ff.fc.7e 0xc72e 0x77f9 FC Link Ctl, ACK1 2.866297 ff.fc.cb -> ff.fc.7e 0xc72f 0x77fa FC Link Ctl, ACK1 2.866445 ff.fc.0a -> ff.fc.7e 0xc72e 0x77f9 SW_ILS SW_ACC (SW_RSCN) 2.866496 ff.fc.7e -> ff.fc.0a 0xc72e 0x77f9 FC Link Ctl, ACK1 2.866615 ff.ff.fd -> 7e.01.00 0x22b8 0xc72a FC ELS ACC (SCR) 2.868792 ff.fc.cb -> ff.fc.7e 0xc72f 0x77fa SW_ILS SW_ACC (SW_RSCN) 2.868857 ff.fc.7e -> ff.fc.cb 0xc72f 0x77fa FC Link Ctl, ACK1 2.871132 ff.fc.7e -> 7e.01.00 0xc730 0xffff FC ELS LOGO 2.872013 7e.01.00 -> ff.fc.7e 0xc730 0x2318 FC ELS ACC (LOGO) 2.872021 7e.01.00 -> ff.fc.7e 0x2318 0xffff FC ELS PLOGI 2.872139 ff.fc.7e -> 7e.01.00 0x2318 0xc731 FC ELS LS_RJT (PLOGI) 2.872163 ff.fc.cb -> ff.fc.7e 0x77fb 0xffff dNS GE_ID 2.872234 ff.fc.7e -> ff.fc.cb 0x77fb 0xc732 FC Link Ctl, ACK1 2.891239 ff.fc.7e -> ff.fc.cb 0x77fb 0xc732 dNS ACC (GE_ID) 2.891359 ff.ff.fc -> 7e.01.00 0x2300 0xc72d dNS ACC (RFT_ID) 2.891469 7e.01.00 -> ff.ff.fc 0x2330 0xffff dNS RFF_ID 2.891613 ff.fc.cb -> ff.fc.7e 0x77fb 0xc732 FC Link Ctl, ACK1 2.900160 ff.ff.fc -> 7e.01.00 0x2330 0xc733 dNS ACC (RFF_ID) 2.900394 7e.01.00 -> ff.ff.fc 0x2378 0xffff dNS RNN_ID 2.901916 ff.ff.fc -> 7e.01.00 0x2378 0xc734 dNS ACC (RNN_ID) 2.902151 7e.01.00 -> ff.ff.fc 0x23a8 0xffff dNS RSNN_NN 2.908296 ff.ff.fc -> 7e.01.00 0x23a8 0xc735 dNS ACC (RSNN_NN) 2.908444 7e.01.00 -> ff.ff.fc 0x23c0 0xffff dNS GNN_FT 2.919880 ff.ff.fc -> 7e.01.00 0x23c0 0xc736 dNS ACC (GNN_FT)
18© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
FCAnalyzer Local Detail(PLOGI)
FCID of HBA Device
N
Port Login
Fabric Domain Controller
Class 3 Device
VSAN 2
19© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
TOP-9216i
MDS FC Analyzer Remote
• For capture of internal MDS9000switch traffic, FC traffic to and from supervisor not FC port-to-FC port traffic
• Make sure MDS can ping workstation
Workstation Running EtherealIP Address 10.91.42.167
Management 10/100 Port on MDS Address 10.91.42.183
N
At DOS Prompt:ethereal -i rpcap://10.91.42.183/eth2
Or at Ethereal Interface Drop-Down:rpcap://10.91.52.183/eth2
SCSI
HBA
20© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Ethereal on PC
Same Output as FC Analyzer Local but in Nice Format with the Ethereal GUI
Filters applied after capture or before. Filters are stored in
application cache for later use.
21© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
MDS 9000 Port Analyzer Adapter
• MDS Port Analyzer Adapter is a protocol converter
Converts FC frames into Ethernet frames
FC-specific information is carried in encapsulation header
Enables use of Ethernet protocol analyzers to decode FC protocols
• Cisco has released decodes for all FC protocols, the encapsulation header and SCSI decoder to public domain
Part of the popular Ethereal protocol analyzer
22© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
FC
FC
TOP-9216i
Port Analyzer Connection Using Port Analyzer with Ethereal
• Configure Port Analyzer Adapter (dip-switch settings, mostly plug and play)
• Configure SPAN port on MDS9000 and what device ports you wish to sniff
• Set Ethereal on PC to capture on local Ethernet interface
Cisco PortAnalyzer Adapter
Workstation L2 Network Running Ethereal
SD Port FC 2/16
Raid Array
For Capture of FC Device to FC Device, Sends Fibre Channel Traffic to an IP-Attached Workstation Running Ethereal
PAA & EtherealCapture on
Ethernet Interface
Connected to PAA
FCPING from Domain Controller in Domain
6f to FCID 6f.02.00
ELS Echo
Boardwalk Is Cisco Code for PAA
24© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
TOP-9216i
SPAN Configuration with Single FC Port Analyzer Tool
SPAN DestinationPort (SD) FC 1/13
FC 1/2
FC Analyzer
TX TXRX RX
Dropped
• FC 1/2 RX input is sent to FC 1/13 SD port
• FC 1/2 TX input is sent to FC 1/13 SD port
TX
RX
span session 1
destination interface fc1/13
source interface fc1/2 rx
source interface fc1/2 tx
interface fc1/13
switchport description Span Destination
switchport mode SD
switchport speed 1000
no shutdown
25© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
RSPAN Requirements
• All switches must be running MDS code 1.2 or later
• No cable should be attached to the SFP in the ST (SPAN Tunnel) port
• Make sure FC tunnel is UP before starting frame collection
• FC Analyzer can be a PAA and PC running Ethereal, or a Finisar-like analyzer
26© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Fabric Manager/Device Manager
• The MDS Fabric Manager Server is your friend
• Used in combination with CLI interface helps tremendously in situations were you cannot remember all the configuration steps
• Use CLI to verify outputs to the managers but trust the manager displays to be proper
27© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Fabric Manager Views
• Show Tech Support output
• Zoning Checks
• Domain Manager ID across all VSANs
• FSPF Health
• RSCN Statistics and health
28© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Device Manager Views
• Span configuration
• Monitor traffic
• Monitor interface Errors and statistics
• View persistent logs
• Manage files in flash
For Troubleshooting:
29© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Lab 1
• Lab intro
• Debug output example
• Ethereal tool usage
• PAA and SPAN usage
• FM and DM tool navigation
Server2
iqn.1991-05.com.cisco.mds-speed.server3
iqn.1991-05.com.mds-speed.server2
fc1/5 iscsi1/1
fc1/14
ClarionCx300
Server3
fc1/6
Seagate92:7b:c5-S
Seagate92:7a:47-S
Seagate92:7b:94-S
Seagate67:67:87-S
Bot-9216i
Server1 iqn.1991-05.com.microsoft:server2
channel1,channel1
fc1/1
fc1/1 fc1/14
fc1/2
PAA
fc1/5
TOP-9216i
iscsi1/1
iscsi1/1
iscsi1/1
fc1/6
fc1/6
fc1/6
fc1/6
fc1/6
fc1/6
fc1/6
30© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Section Agenda/Second Half
• Fibre Channel Layer Review
• Determining Low-Level Interface Physical Issues
• Best Practices for Troubleshooting Device Connectivity
Lab
• Inter-Switch Link—Understanding and Troubleshooting
• Switch Domains and VSANs
• Zoning Details
• Inter-VSAN Routing—IVRLab
31© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Fibre Channel LayersReview
• FC-0—defines the physical interface characteristicsSignaling rates, cables, connectors, distance capabilities, etc.
• FC-1—defines how characters are encoded/decoded for transmissionTransmission characters are given desirable characters
• FC-2—defines how information is transported
Frames, sequences, exchanges, login sessions
• FC-3—place holder for future functions
• FC-4—defines how different protocols are mapped to use Fiber Channel
SCSI, IP, Virtual Interface Architecture, others
Structure Is Divided into Five Levels of Functionality
FC-PH
FC-0
FC-1
FC-2
FC-3
FC-4s
Common Services
Signaling Protocol
Transmission Code
Physical Interface
IPSCSI
32© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Fibre Channel Transmission Words: FC-1
• Ordered sets are transmitted continuously to indicate that specific conditions within the link are encountered
• Ordered sets transmitted while the condition exist
• Four primitive sequences which can determine where problem exist
Not Operational Sequence (NOS)
Offline Sequence (OLS)
Link Reset Sequence (LR)
Link Reset Response Sequence (LRR)
33© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Link Initialization FlowFibre Channel Layer 1 Protocol (FC-1)
These Are All Special-Ordered Sets of 8B/10B Coding:
Link Failure ConditionAC AC
LFLF
LR
OL
LR
AC
AC
NOS
OLS
LR
Idle
LRR
Idle
Port A Port B
NOS = Not Operational Sequence
OLS = Offline Sequence
LR = Link Reset
LRR = Link Reset Response
AC = Active State
LR = Link Recovery State
LF = Link Failure State
OL = Offline State
34© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Fibre Channel Port Issues
1. The switch port must first acquire bit and word synchronization with the N_Port
2. N_Port must issue a FLOGI to the MDS
Tip: Clear Counters and Monitor to Verify Active Issues, Use Device Manager Monitor Tool to Monitor Live; Set and Activate Threshold Manager to Alert You;
TOP-9216i# clear counters interface fc 1/1
In Order for an F_Port to Come Up on an MDS Switch:
Primitive Sequences Counters Can Determine
Layer 0–1 Problems
35© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Fabric Services Refresher
• FFFFFE—Well-known address of login server
Basic responsibility to assign FCID
• FFFFFC—Well-known address of fabric directory name server
Basic responsibility registration of devices and distribution of database
• FFFFFD—Well-known address of fabric controller
Basic duties responsible for managing fabric, initialization, FSPF routing, responsible for RSCNs and listens for SCNs
FFFcxx—Cisco Specific, Representing Name Server in a VSAN and Domain xx
FC Standards
36© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Line Card Basics
• Show module will display slot locations and type of card
• General information in interface and statistics can be gathered from switch main-level prompt
• More detailed gathering of ASIC counters may be required to troubleshoot difficult issues, attach would then be required
Attach to Module with “Attach” Command, All Modules Can Be Attached to, Including Standby Supervisor and IPS Blade
Exit to Detach
37© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
TOP-9216i
Using FC Test Equipment for Physical Issues
• Use MDS show commands and debugs to narrow scope of physical issues
• External analyzers will be required for detail diagnostics of intermittent physical layer issues and use of triggers to trap problem
• Testers must go inline, disruptive to network connection. Plan accordingly.
FC AnalyzerTX TXRX RX
N
FC
FC
38© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Errors on Application Server Logs, User Having Performance Problems
• Use logging capabilities on switch to piece together network issues
• Put together:
1. What interfaces are on the fabric (up, down, flapping)
2. What was health of each interface (virtual and physical interfaces)
3. Examine interface health (FC protocol errors, physical layer errors)
4. Are effects being seen on the complete network fabric or within the VSAN (RSCNs, Zones, ISLs, Errors on common controllers)
39© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Necessary Network Settings
• Switch Fabric clocks synchronized
Use NTP along with CFS to simplify job
Use fabric manager to set all clocks and time zones
• Setup system logs, syslogd server
Use CFS to simplify
• Save off switch configurations or even complete show tech-support details regularly
Use CLI scheduler in SAN/OS to simplify process
40© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Monitor Link Init State (Flow from Bottom to Top)
LINK: 052 022180c5 LR_RECEIVE(03) =>ACTIVE(01)
LINK: 051 022180c3 OLS_TRANSMIT(07) =>LR_RECEIVE(03)
LINK: 050 022180c1 SENTINAL(00) =>OLS_TRANSMIT(07)
LOOP: 049 022180c1 HW_ALPAS(0d) MASTER_LISA_WAIT(1f)=> OLD_PORT(3f)
LOOP: 048 022180c1 HW_ALPAS(0d) MASTER_LISA(1e)=> MASTER_LISA_WAIT(1f)
LOOP: 047 022180c1 HW_ALPAS(0d) MASTER_LIHA_WAIT(1d)=> MASTER_LISA(1e)
LOOP: 046 022180c1 HW_ALPAS(0d) MASTER_LIHA(1c)=> MASTER_LIHA_WAIT(1d)
LOOP: 045 022180c1 HW_ALPAS(0d) MASTER_LIPA_WAIT(1b)=> MASTER_LIHA(1c)
LOOP: 044 022180c1 HW_ALPAS(0d) MASTER_LIPA(1a)=> MASTER_LIPA_WAIT(1b)
LOOP: 043 022180c1 HW_ALPAS(0d) MASTER_LIFA_WAIT(19)=> MASTER_LIPA(1a)
LOOP: 042 022180c1 HW_ALPAS(0d) MASTER_LIFA(18)=> MASTER_LIFA_WAIT(19)
LOOP: 041 022180c1 HW_X_ARB(0c) MASTER_START(17)=> MASTER_LIFA(18)
LOOP: 040 022180c1 HW_LISM0(0a) OPEN_INIT_SELECT_MASTER(06)=> MASTER_START(17)
LOOP: 039 022180bf HW_R_LIP(09) OPEN_INIT_START(05)=> OPEN_INIT_SELECT_MASTER(06)
LOOP: 038 022180bf HW_X_LIP(08) NORMAL_INITIALIZE(04)=> OPEN_INIT_START(05)
LOOP: 037 022180bf HW_R_LIP(09) LPSM_STARTED(01)=> NORMAL_INITIALIZE(04)
LOOP: 036 022180b0 HW_OLDP(07) LPSM_DISABLED(00)=> LPSM_STARTED(01)
LINK: 035 022170e8 ACTIVE(01) =>SENTINAL(00)
Attached to Mod 1 FC Port 1/2
Interface Shut/No Shut via
Configuration
Port Tries Loop First
When Port Is Set to Auto
F-Port Point
to Point Negotiated
module-1# show hardware internal fc-mac port 2 stateinfo (older Line Cards)
module-1# show hardware internal fc-mac2 port 2 state-info-log (newer Line Cards
41© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Show flogi Database
• FLOGI stands for fabric login
• Shows which devices are logged into the fabric
• Only shows devices that have flogi’ed into this switch not complete multi switch network
N
TOP-9216i# show flogi database ---------------------------------------------------------------------------INTERFACE VSAN FCID PORT NAME NODE NAME ---------------------------------------------------------------------------fc1/1 2 0x7e00ef 50:06:01:60:30:20:54:f0 50:06:01:60:b0:20:54:f0fc1/2 2 0x7e0100 21:00:00:e0:8b:17:2f:50 20:00:00:e0:8b:17:2f:50fc1/6 5 0xa3026e 21:00:00:04:cf:92:7b:c5 20:00:00:04:cf:92:7b:c5fc1/6 5 0xa30271 21:00:00:04:cf:92:7b:94 20:00:00:04:cf:92:7b:94fc1/6 5 0xa30272 21:00:00:04:cf:92:7a:47 20:00:00:04:cf:92:7a:47fc1/6 5 0xa30273 21:00:00:04:cf:67:67:87 20:00:00:04:cf:67:67:87fc1/14 2 0x7e03ef 50:06:01:69:30:20:54:f0 50:06:01:60:b0:20:54:f0iscsi1/1 2 0x7e0004 21:07:00:0d:ec:20:97:c2 21:08:00:0d:ec:20:97:c2iscsi1/1 5 0xa30002 21:01:00:0d:ec:20:97:c2 21:12:00:0d:ec:20:97:c2iscsi1/1 5 0xa30004 21:07:00:0d:ec:1f:82:42 21:11:00:0d:ec:20:97:c2
42© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Show Fibre Channel Name Server Database
• Logging into fabric with FLOGI and registering with nameserver are separate events
• Shows all devices in entire fabric
• FCID shown is the FCID assigned by the switch that device FLOGI’ed into
• Initiator or target is attribute registered with switch from device during device login
• Vendor field is determined by OUI assigned by IEEE
43© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Show fcns databaseInformation output of show Command
Target or Initiator Passed to Switch via PROCESS ACCEPT and ACCEPT Frames During Nameserver Login
sh fcdomain domain-list vsan 2
OUI is Part of the PWWN and Is IEEE Assigned. 00:e0:8b is Assigned to Qlogic, 00:00:c9 is Assigned to Emulex, and 00:04:cf is Assigned to Seagate, Many Others
You will Require the Domain List from the Switches to Know Where the Devices Are Physically Connected on the SAN, Could Be IVR’ed from Other VSAN
44© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
----------------------------------------------------------------------------------VSAN:2 FCID:0x7e0100-----------------------------------------------------------------------------------port-wwn (vendor) :21:00:00:e0:8b:17:2f:50 (Qlogic) node-wwn :20:00:00:e0:8b:17:2f:50class :3node-ip-addr :0.0.0.0ipa :ff ff ff ff ff ff ff fffc4-types:fc4_features :scsi-fcp:init symbolic-port-name :symbolic-node-name :QLA2340 FW:v3.03.13 DVR:v9.0.2.17 (w32)port-type :N port-ip-addr :0.0.0.0fabric-port-wwn :20:02:00:0d:ec:20:97:c0hard-addr :0x000000permanent-port-wwn (vendor) :21:00:00:e0:8b:17:2f:50 (Qlogic)
Show fcns database detail
Gives Detailed Information About Each Device (Only One Shown)
FWWN This Device Attaches to the Fabric via
Registered to Talk SCSI over Fibre Channel
Device Registered and Thus Indicates It Can Converse with Class of Service 3
45© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Debugging Name Server Issues
1. Check if the host and the disk are in the same VSAN (use fcping or fctrace, show VSAN membership)
2. Debug fcns events (this will give the request and response from the fabric name server)
3. Check if the host and the disk are in the same zone (show zoneset active vsan x); look for the * next to each device FCID; missing * means device is not in the fabric
If a Host Is Not Receiving a Proper Response, or if Not Able to Discover a Target, Enable the Following Debug and Shut/No Shut the Host Interface
46© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
FCAnalyzer Brief Results During HBA LoginCLI View
From FC-GS-4 standard
Register FC-4 Type, Register Node Name, Register Symbolic Node Name, Get Node Names-FC4 Type, Get Platform Name
Device Logging into Fabric Initial Sends FCID with All 0’s
HBA Does Port Login to the Directory Server and Receives ACC Back
HBA Registers for State Change Registration (SCR) to be Notified by the Switch if There any Changes in Fabric in Same Zone and Receives ACCept from Switch
MDS Begins to Query the Device with RNN_ID, RSNN_NN, GNN_FT
Switch-to-switch communication noted by SW_ILS. Registered state change notifications going out to zone.
Switch Sends ACCept Back andAssigns the HBA a FCID
HBA Registers with Name Server, RFT
47© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
HBA LoginEthereal Application View
• HBA and FC port on switch finish physical level handshake and HBA logs into port server a FF.FF.FE
• Login Server assigns FCID from Domain 6f (VSAN 200) 6F.02.00
• HBA logs into Fabric Name Server
• HBA registers for State Change Notification, response comes back in frame 7
• HBA asks Name Server for FC-4 FC devices registered in the server. Response comes back in frame 24
• Name Server from VSAN 200 MDS9216-TOP quires HBA
• Name Server from VSAN 200 MDS9216-Bottom quires HBA
• HBA Port logs to FC-4 devices received from accept frame 24
• HBA talks to SCSI devices
N
48© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Link Events on a Switch
• Per module/per switch command
• Can be taken from Show tech-support detail
• Helps drill down as to what devices were leaving fabric at time of problem and general reason
• Shows both physical and virtual interfaces
49© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Gathering Internal Counter Information for Unknown Issues and Plaguing Connectivity Problems
• Determine what ports on which switches you need to examine, this is done by narrowing to physical switch, VSAN, zone
• Look at path through complete network, initiator and target-side along with ISLs
• One stop shopping on each switch for all the data:Attach to Line Card Module that has the interface you need information on
module-1# terminal length 0
Set scroll back buffer or log to file on telnet tool
module-1# show hardware internal debug-info interface fc1/2
Based on Line Card type proper information will be output
50© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
show hardware internal debug-info
Script Runs Giving Outputs Of:
show hardware internal errors
show port-config internal link-events
show port-config internal port-control
show port-config internal port-cfg-error
show hardware internal fc-mac2 port 2 link-status
show hardware internal fc-mac2 port 2 port-info
show hardware internal fc-mac2 port 2 misc-statistics
show hardware internal fc-mac2 port 2 status-reg
show hardware internal fc-mac2 port 2 state-info-log
show hardware internal fc-mac2 port 2 gbic-info
show hardware internal fc-mac2 port 2 statistics
show hardware internal fc-mac2 port 2 flow-control
show hardware internal fc-mac2 port 2 config-reg
show hardware internal fc-mac2 port 2 mask-reg
show hardware internal packet-flow interface fc1/2
We Have Already Looked at Details of
Theses 2 Outputs
51© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Lab 2
• HBA Login Lab
• Physical Layer Troubleshooting
• Debugging Fibre Channel services
52© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
ISL Connection
• ISL connections require proper standards operation
FSPF routing
Proper capabilities and parameter exchange
Proper domain ID assignment which could be disruptive or non-disruptive to network
Proper merging of fabric zoning
• ISL will not fully initialize unless FC-SW rules are complete
TOP-9216i BOT-9216i
53© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Switch-to-Switch, the ISL E-Port
• Fibre Channel Standard FC-SW addresses ISL operation
• FC incorporates distributive methods across the SAN Fabric E-Ports seen as SW_ILS in protocol field of traces
• Switches if determine connecting FC port is an ISL link
• Much can go wrong as part of the initial ISL handshake
• ISL needs to be analyzed for situations of domain ID assignment, routing, and zoning distribution issues
N
54© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Principal Switch SelectionReview
• Only one switch is designated principal switch in a fabric
• The switch with the lowest WWN becomes the principal switch at new install of SAN after that other rules apply
• Principal switch makes sure that no new switch is added to the fabric if it has a domain ID conflict with an existing switch in the fabric
• Only one upstream link per switch
N
Switch 1
Switch 3
Switch 4 Switch 5
Switch 6
Switch 2
55© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Fabric Configuration: PS SelectionReview
• A principal switch shall be selected when at least one inter-switch link (link between two E_Port) is established
• The selection process chooses a principal switch, which is then designated to assign domain identifiers to all the switches in the fabric, existing and new
56© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Fabric Configuration ProcessReview
• The fabric configuration process enables a switch port to determine its operating mode, exchange operating parameters, and provides for distribution of addresses
• The process is summarized in the following steps
Establish link parameters and switch port operating mode
Principal switch selection
Domain ID distribution
Path selection
N
57© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Fabric Configuration Disruptive/Non-Disruptive
• One of the following three conditions can trigger Build Fabric, BF (non-disruptive) or Reconfigure Fabric, RCF (disruptive)
Two disjointed fabric are combined together
The principal ISL link fails (upstream or downstream)
A switch with Domain_ID request for another Domain_ID
• Whenever a switch receives a BF/RCF, the switch starts F_S_TOV timer and enters the BF/RCF state; it forwards BF/RCF out of all E_ports except the incoming port (only once) and waits for the timer to expire
• When the timer expires, BF/RCF propagation state is over and principal switch selection begins
58© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Domain IDs
Fabric Manager View
Device Manager View
CLI View
59© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Cisco TE EISL TE Port SPAN to PAA
Domain ID Assign by Existing Principal Switch
Request Domain ID from New Switch
Exchange Fabric Parameters
Exchange Link Parameters
Exchange Switch Capabilities
Zone Merge Request
Exchange Peer Parameters (This Is Proprietary to MDS Only to Determine if Connecting Switch Is Another MDS)
Enhanced Zoning Merge Request Resource Allocation (New in FC-SW3 Standard)
ACK1 Filter Applied
Build Fabric
60© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Domain ID Problems
• Show interface
• Check fabric manager
• Check event logs
MDS9216-Bottom# sh int fc 1/9fc1/9 is down (Isolation due to domain overlap) Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN) Port WWN is 20:09:00:0d:ec:0d:0c:c0 Admin port mode is E, trunk mode is off Port vsan is 500 Receive data field Size is 2112 Beacon is turned off 5 minutes input rate 360 bits/sec, 45 bytes/sec, 0 frames/sec 5 minutes output rate 264 bits/sec, 33 bytes/sec, 0 frames/sec 2433 frames input, 198208 bytes 0 discards, 0 errors 0 CRC, 0 unknown class 0 too long, 0 too short 2423 frames output, 128924 bytes 0 discards, 0 errors 19 input OLS, 22 LRR, 1 NOS, 26 loop inits 310 output OLS, 107 LRR, 13 NOS, 10 loop inits 255 receive B2B credit remaining 255 transmit B2B credit remaining
61© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Troubleshooting Zoning
• Show zoning information from CLI on all switches in fabric
Show zone status
Show zoneset active
• Is zone configuration stored and controlled from one switch or any/all switches?
• Make sure you are working with active zone when in CLIzone copy active-zoneset full-zoneset vsan 5
*Key concern when working in mixed vendor interop VSANs
• Verify distribution and default values (permit-deny)
• See event logs for clues about issues
• Verify device in logged into fabric
62© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Zoneset Tools in FM
Backup Zone Configuration to File on ServerWhere Is Active ZoneSet in Each VSAN Applied?
Current Zoneset Status
63© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Show Zoneset Active
• Shows the zoneset activated in the fabric
• An asterisk to left of device indicates that device has registered with the nameserver
• Does not show if a zone has been configured but changes have not been activated
• Zoneset and zone names are case sensitive
• Inter-VSAN routing zone added to ZoneSet via IVR wizard if this VSAN has member being zoned to another VSAN
N
Domain a3(163) Is in VSAN 5
64© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Activate Zoneset Flow Across ISLs
ACC
ACA—Acquire Change Auth
Lock the Fabric
ACC
SFC—Stage Fabric ConfMove the Zone Data to the Switches
ACC
UFC—Update Fabric ConfTrigger Switches to
Activate the New Zoneset
ACC
RCA—Release Change Auth
Unlock the Fabric
Zoneset Distribution Would Go to Every Domain Within the VSAN
ACK1 Frames Are Not Shown Here
65© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Activate Zoneset with FC Analyzer and VSAN Filter
Only VSAN 100
FC interface 1/16
66© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
IVR Troubleshooting Tips
• Start out with valid well thought-out IVR design
• Pay close attention to zoning
• Show IVR Tech-Support
• Look at complete IVR path
• Use CFS (Cisco Fabric Services)
• Domain ID limit rules in Interop VSANs also apply to the IVR Domains (97–127 rule)
67© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Which Switch, What Zone, What Feature
TOP-9216i# sh zoneset brief active zoneset name ZoneSet1 vsan 2 zone Server1 zone Server3 zone iSCSIserver3 zone iSCSIserver2 zone IVRZ_Server1-IVR
TOP-9216i# sh ivr zoneset brief active
zoneset name IvrZoneSet1 zone name Server2-IVR
zone name Server1-IVR
CLI Displays Will Show Zones from IVR Feature While Displays on FM Only Display All None IVR Zones
68© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Lab 3
• TE port initialization
Detailed look
• Domain ID assignments
Adding a domain (switch) to the SAN
• Zone merge
Rules of zoning
69© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
MDS Documents and Whitepapers on Net
• http://cisco.com/en/US/products/hw/ps4159/ps4358/prod_white_papers_list.html
• http://www.cisco.com/univercd/cc/td/doc/product/sn5000/mds9000/interop/index.htm
• http://www.cisco.com/en/US/products/hw/ps4159/tsd_products_support_category_home.html
• http://www.emc.com/interoperability/index.jsp
• http://www.redbooks.ibm.com/abstracts/sg246397.html?Open
• ftp://service.boulder.ibm.com/storage/san/cisco/
• http://h18006.www1.hp.com/products/storageworks/cmds9509/index.html
70© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Q and A
71© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Recommended Reading
• Continue your Networkers learning experience with further reading from Cisco Press®
• Check the Recommended Reading flyer for suggested books
Available Onsite at the Cisco Company Store
72© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1
Complete Your Online Session Evaluation!
• Win fabulous prizes; Give us your feedback
• Receive ten Passport Points for each session evaluation you complete
• Go to the Internet stations located throughout the Convention Center to complete your session evaluation
• Drawings will be held in theWorld of Solutions
Tuesday, June 20 at 12:15 p.m.
Wednesday, June 21 at 12:15 p.m.
Thursday, June 22 at 12:15 p.m. and 2:00 p.m.
73© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1