Cisco Net Workers 2006 - SAN-1501 - Introduction to Storage Area Networking
-
Upload
sidneyelagib -
Category
Documents
-
view
1.069 -
download
5
Transcript of Cisco Net Workers 2006 - SAN-1501 - Introduction to Storage Area Networking
1© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2
Introduction to Storage Area Networks (SAN)
SAN-1501
2© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Session Agenda
• Storage BasicsThe What …
• Fibre Channel Basics… and the How.
• Where to go from here
Main Topics for Today
3© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Storage Basics
4© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Storage Basics
• Storage componentsHost bus adapter
Disks, JBODs, RAID, storage arrays
• Storage topologiesDAS, NAS, and SAN
• Storage protocolsSCSI
Storage transport protocols
Fibre Channel (FC) , FCIP, iSCSI
5© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Storage Components
6© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Host Storage Components
• Host bus adapter (HBA)
• Hardware resident on host server
• Connection interface to disk subsystem
• Connection methodCopper
Optical
7© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Data Storage Components
• Fundamental unit for data storage
• Disk drive typesParallel SCSI
Fibre channel
Advanced Technology Attachment ATA) or Integrated Drive Electronics (IDE)
Serial ATA (SATA)
Other
• Tape drives
I/O Devices—Disk Drives
8© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Storage Interconnects
Connecting Hosts to Storage
• Parallel SCSI copper interconnects
• Optical direct connect
• Fibre Channel switch
9© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
The SCSI I/O Channel—Starting Point
• SCSI I/O Channel provides half-duplex pipe for SCSI command, data, and status
• SCSI I/O channel can be internal or external to host
• Multiple SCSI I/O channels can exist within host
• A network approach can scale the I/O channel in many areas (length, devices, throughput)
NIC Adapter SCSI Adapter
NIC Driver
TCP/IP Stack
Adapter Driver
SCSI Generic
Block Device
File System
Applications Half-DuplexI/O Channel
SCSI
SCSI
SCSI
SCSIInitiator
SCSITarget
10© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
SCSISCSI
I/O C
hann
el
Target 2
ChannelController
Host (Initiator)
Target 1
The Parallel SCSI I/O Channel
• The SCSI Channel is used to transmit SCSI commands, data, and status
• Multiple devices can exist on the channel each playing the role of SCSI initiator or SCSI target
• Most common channel is the basic parallel SCSI bus which can be internal or external to a host
• Parallel SCSI Specifications:Up to 25m bus length
Shared channel bandwidth upto 320MBps
Up to 16 devices per SCSI bus
Half-duplex operation
11© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
SCSISCSI
Target
ChannelController
Host (Initiator)
Net
wor
ked
I/O C
hann
el
Initiator
Networking the I/O Channel
• Same SCSI protocol carried over a network transport via serial implementation
• Transport must not jeopardize SCSI payload (security, integrity, latency)
• Two primary transports to choose from today, namely IP and Fibre Channel
• A networked I/O channel allows for multiple improvements:
Distance limitations greatly increased
Dedicated bandwidth (not shared)
High # of addressable devices
Bandwidth increase (including link bundling)
Network
12© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Fibre ChannelFabric
SCSISCSI
Target
EthernetNIC
Host System
Initiator
Fibre Channel I/O Networking
• Very common method for networking SCSI
• Fibre Channel provides high speed transport for SCSI payload
• Fibre Channel overcomes many shortcomings of Parallel I/O including:
Addressing for up to 16 million nodes
Loop (shared) and fabric (switched) transport
Host speeds of 100 to 400 MBps (1-4 Gbps)
Segments of up to 10km (without extenders)
Support for multiple protocols
• Combines best attributes of a channel and a network together
13© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
IP I/O Channel Networking
• IP access to open systems iSCSI and Fibre Channel storage
• iSCSI driver is loaded onto hosts on Ethernet network
• Able to consolidate servers via iSCSI onto existing storage arrays
• Able to build Ethernet-based SANs using iSCSI arrays
• Storage can be mapped from iSCSI to Fibre Channel
iSCSI-EnabledHost (Initiator)
iSCSI Appliance
(Target)
StoragePool (Target)
iSCSIGateway
IPNetwork
FCFabric FC HBA
Attached Host
(Initiator)
iSCSI
iSCSIiSCSI
EthernetNIC
SCSISCSI
14© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Storage Subsystems
• Just a bunch of disks (JBOD)
• Storage arrays
• Redundant array of independent disks (RAID)
15© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Simple Storage SystemsI/O Devices—JBODs
ServerWith SCSI
ServerWith Fibre Channel
Arbitrated Loop(FC-AL)
TRGT 0 TRGT 2
TRGT 1 TRGT 3
TRGT 4
SCSI JBOD
FC-AL
TRGT 3TRGT 4TRGT 5
FC-AL JBOD
SCSI BUS
TRGT 0 TRGT 1 TRGT 2
SCSI CABLE
FC CABLE
16© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Large Storage Systems – Bus TypeI/O Devices—Intelligent Storage Arrays
X BUS
Y BUS
Disk DrivesDisk Drives
Host Bus Adapters Host Bus Adapters
Host ChannelConnections
Dual-ProcessorChannel DirectorCache
System Bus
Dual-ProcessorDisk Director
Disks
CDCDCDCDCDCDCDCDCacheCache
DDDDDDDDDDDDDDDD
17© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Large Storage Systems – Switched Type
Host ChannelConnections
Dual-ProcessorChannel Connectors
SwitchedBackplane
Dual-ProcessorDisk Connectors
Disks
Host Bus Adapters Host Bus Adapters
CacheCache
Disk DrivesDisk Drives
DDDDDDDDDDDDDDDD
CDCDCDCDCDCDCDCD
I/O Devices—Intelligent Storage Arrays
18© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Logical Arrays of Storage
• Redundant Array of Independent Disks (RAID)Word coined by researchers at University of California, Berkeley in 1987
A method to inexpensively put together a set of physical hard drives into an array
Provides fault tolerance by mirroring or parity operations
RAID can be performed using hardware or using host based software
RAID
19© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Types of RAID
• RAID 0: StrippingFor performance and size, two or more hard disks are concatenated together to form a larger volume
• RAID 1: MirroringFor reliability and availability
• RAID 3: Error DetectionFor reliability, availability of data using using a dedicated parity drive
• RAID 5: Error CorrectionFor reliability, availability of both data and parity
20© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
RAID 0: Striping
• Data is segmented and split across multiple spindles
• I/O benefitsShort reads and writes, easily handles multiple simultaneous reads
Long reads and writes, single operations split and processed in parallel
• Redundancy—none
• Cost—good (no extra hardware)
21© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
RAID 1: Mirroring
• Data is duplicated on multiple spindles
• I/O benefitsShort and long reads with shorter latency
Short and long writes slower due to multiple writes
• Redundancy—good
• Cost—need double the amount of disks
22© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
RAID 0+1:
• Data is striped and duplicated on multiple spindles
• I/O benefitsShort and long reads with shorter latency
Short and long writes faster as writes are spread across multiple spindles
• Redundancy—good
• Cost—need double the amount of disks
23© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
RAID 1+0:
• Data is duplicated and striped on multiple spindles
• I/O benefitsLess downtime than RAID 0+1
• Redundancy—good• Cost—need double
the amount of disks
24© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
RAID 5: Error Correcting
• Data protection using ECC (Error Correction Control) parity spread over all drives
• I/O benefitsShort and long reads at normal speeds
Short and long writes slower due to parity calculations
• Redundancy—better than RAID3
• Cost—needs only one extra disk for an entire logical disk
25© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Storage Topologies
26© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
IP Front-End Network SAN Back-End Network
SANIPNetwork
Storage Topologies
• Host to Host• Application to file system• Client to Server• NFS, SMB, CIFS, NCP• NAS, WAFS
• Host to Storage• File system to Device• Program to Device• SCSI, IDE, NTFS, FAT• SAN
27© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Direct Attached Storage (DAS)
• Storage is captive ‘behind’ the server, limited mobility
• Limited scalability due to limited devices
• No efficient storage sharing possible
• Costly to scale; complex to manage
FC
Clients
Direct Attached Storage
Application Servers
Win2k Linux Unix Unix
Tape
FC
LinuxWin2k
SCSI
LAN
28© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Network Attached Storage (NAS)
• Storage is accessed at a file level via NFS or CIFS
• Storage is accessed over an IP network
• Storage devices can be shared between servers
• Files can be shared between users
• TCP can be tuned to optimize for file transport Application
ServersNAS Appliances
orNAS Head Ends
Generic Generic
NAS = Optimized for File I/O
Win2k Linux Unix
LAN
29© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Storage Area Network (SAN)
• Storage is accessed at a block-level via SCSI
• High performance interconnect providing high I/O throughput
• Lower TCO relative to direct attached storage, storage can be shared
• Limited vendor interoperability
• Complex management
Separation of Storage from the Server
Servers
BlockStorageDevices
Fibre Channel
SAN
Clients
LAN
30© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
iSCSI Storage Network
• Storage is accessed at a block-level via iSCSI
• Good performance via standard Ethernet NIC
• Enhanced performance with TCP Offload Engine (TOE)
• Lower TCO relative to direct Fibre Channel HBA/Fabric
• Standards based
• Complex management
iSCSI-EnabledHosts (Initiators)
iSCSI Appliance
(Target)
StoragePool (Target)
iSCSIGateway
IPNetwork
FCFabric
FC HBA Attached
Host (Initiator)
iSCSIiSCSI
iSCSI
iSCSI
31© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
DAS, SAN, iSCSI, NAS Comparison
DAS
SCSI
Computer System
SCSI Bus Adapter
SCSI Device Driver
Volume Manager
File System
Application
iSCSI Appliance
IP
File System
Application
SCSI Device DriveriSCSI DriverTCP/IP stack
NIC
Volume Manager
NICTCP/IP StackiSCSI LayerBus Adapter
iSCSI Gateway
IP
FC
File System
Application
SCSI Device DriveriSCSI DriverTCP/IP stack
NIC
Volume Manager
NICTCP/IP StackiSCSI Layer
FC HBA
NASAppliance
IP
NICTCP/IP Stack
I/O Redirector
File System
Application
NFS/CIFS
NICTCP/IP StackFile System
Device Driver
Block I/O
NASGateway
IP
NICTCP/IP Stack
I/O Redirector
File System
Application
NFS/CIFS
FC
NICTCP/IP StackFile System
FC HBA
Host/Server
StorageTransport
StorageMedia
SAN
SAN
FC
FC HBASCSI Device Driver
Volume Manager
File System
Application
Computer System Computer System Computer System Computer System Computer System
Block I/O File I/O
32© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Wide Area File Services (WAFS)
IntelligentMessage Suppression
Protocol proxy to handle non-critical messages
Operation batching and message bundling
Decouple the user experience from the WAN
Data IntegrityMaintain data integrity at all costs
Never compromise security of the data
Guarantee support for all protocol semantics
Data Distributionand Caching
Validate and serve information locally
Prepopulate via preposition and on-demand
Asynchronous writes to mask large transfers
Link and ThroughputOptimization
Compress all messages
Pipeline multiple segments into a common window
Multiplex over many sockets if necessary
WAFS Provides File Services Between a Remote Client and a Centralized Server
33© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Storage Protocols
34© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Small Computer System Interface—SCSI
• SCSI is a standard that defines an interface between an initiator (usually a computer) and a target (usually a storage device such as a hard disk)
• Interface refers to connectors, cables, electrical signals, optical signals and the command protocol that allow initiators and targets to communicate
• Logical Unit Number (LUN): A 64-bit field within SCSI that identifies the Logically addressable Unit within a target SCSI device
35© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
SCSI Command Protocol
• SCSI command protocol is the de facto standard that is used extensively in high-performance storage applications
• The command part of SCSI can be: Transported over a Fibre Channel storage area network
Encapsulated in IP and carried across IP networks
• To understand the finer points involved with transporting SCSI across a network with FC or Ethernet, the basics of SCSI must be well understood
36© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Standards
• SCSI has evolved since it was introduced as SASI in 1979 by Shugart Associates – it was approved as a standard by ANSI in 1986 and is now referred to as SCSI-1
• SCSI-2 was approved by X3 in 1990 and by ANSI in 1994
• SCSI-3 refers to a collection of standards, each of which defines a very specific part of SCSI: physical interface, transport interface, command interface, architecture model, programming interface, etc.
Also known as SCSI Architecture Model – 3 (SAM-3)
37© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
SCSI Architecture Model
SAN
38© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
SCSI I/O Channel
The SCSI I/O Transaction
• The SCSI protocol forms the basis of an I/O transaction
• The channel provides connectivity between communicating devices in a SCSI transaction
The following shows two sample SCSI exchanges:
Host (Initiator) Disk (Target)
READDATA DATA DATA
WRITEDATA DATA DATA
STATUS
STATUS
SCSI I/O ChannelSTATUSSTATUS
Disk (Target)Host (Initiator)
SCSI READ OPERATION
SCSI WRITE OPERATION
39© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
SCSI Commands
• Data transferREAD, WRITE
• Commands used in boot/discovery:REPORT LUNS
INQUIRY
TEST UNIT READY
40© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
SCSI Read
1. Send SCSI Cmd issued by initiator – the command sent is READ;
2. SCSI Command Received by target;
Data transfers occur during the ‘working’ phase between initiator and target;
3. Send Command Complete is returned by the target;
4. Command Complete Receivedby target.
…
41© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
SCSI Write
1. Send SCSI Cmd issued by initiator—the command sent is WRITE;
2. SCSI command receivedby target;
3. Target returns TRGT-RDY;4. Initiator receives TRGT-RDY;
Data transfers occur during the ‘working’ phase between initiator and target;
5. Status complete is returned by the target;
6. Status complete received by initiator
42© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Storage Transport Protocols
• Protocols used to transport SCSI Command and DataParallel SCSI
Fibre Channel
FCIP
iSCSI
43© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Storage ProtocolsProtocol and Transport Stack
Parallel SCSIInterfaces Fibre Channel Ethernet
IP
TCP
Parallel SCSIInterfaces Fibre Channel iSCSI
SCSI Commands, Data, and Status
SCSI Block Commands
SCSI StreamCommands
Other SCSI Commands
SCSI Applications (File Systems, Databases)
FCIPFibre Channel
44© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
SCSISCSI
I/O C
hann
el
Target 2
ChannelController
Host (Initiator)
Target 1
The Parallel SCSI I/O Channel
• The SCSI Channel is used to transmit SCSI commands, data, and status
• Multiple devices can exist on the channel each playing the role of SCSI initiator or SCSI target
• Most common channel is the basic parallel SCSI bus which can be internal or external to a host
• Parallel SCSI Specifications:Up to 25m bus length
Shared channel bandwidth upto 320MBps
Up to 16 devices per SCSI bus
Half-duplex operation
45© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Storage Protocols
Parallel SCSIInterfaces Fibre Channel Ethernet
IP
TCP
Parallel SCSIInterfaces Fibre Channel iSCSI
SCSI Commands, Data, and Status
SCSI Block Commands
SCSI StreamCommands
Other SCSI Commands
SCSI Applications (File Systems, Databases)
FCIPFibre Channel
Fibre Channel Protocol and Transport Stack
46© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
What Is It? Fibre Channel
Channels• Connection service• Physical circuits• Reliable transfers• High speed• Low latency• Short distance• Hardware intense
Networks• Connectionless• Logical circuits• Unreliable transfers• High connectivity• Higher latency• Longer distance• Software intense
Fibre Channel• Circuit and packet
switched
• Reliable transfers
• High data integrity
• High data rates
• Low latency
• High connectivity
• Long distance
47© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
FICON
FC SingleByte Command
Sets
IP
Link Encapsulation
(FC-LE)
ULP (Upper Level Protocol)(sample – there are more) SCSI-3
SCSI-3 Mapping(SCSI-FCP)
FC-4 SNMP Mapping(FC-SNMP)
FC-3 Common Services
FC-0
FC-1
FC-2 Fibre Channel Physical & Signaling
Interface(FC-PH, FC-PH2,
FC-PH3)Physical Interface
Encode / Decode
Framing Protocol FC-AL FC-AL-2 FC-SW-2
Copper Optical
8B/10B Encoding
Fibre Channel Protocol Architecture
48© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Fibre Channel Functions
• FC-0• Defines the physical interface characteristics
Signaling rates, cables, connectors, distance capabilities, etc.• FC-1• Defines how characters are encoded/decoded for transmission
Transmission characters are given desirable characters• FC-2• Defines how information is transported
Frames, sequences, exchanges, login sessions• FC-3 • Place holder for future functions• FC-4• Defines how different protocols are mapped to use Fibre Channel
SCSI, IP, Virtual Interface architecture, FICON, others
Structure Is Divided into 5 Levels of Functionality
49© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
OX_ID & RX_ID
FrameFields
SEQ_ID
SEQ_CNT
ULP Information Unit
Exchange
Sequence Sequence Sequence
Frame Frame Frame
Fibre Channel FC-2 Hierarchy
• Multiple exchanges are initiated between initiators (hosts) and targets (disks)
• Each exchange consists of one or more bi-directional sequences
• Each sequence consists of one or more frames• For the SCSI3 ULP, each exchange maps to a SCSI command
50© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Idles* SOF Frame Header Data Field CRC EOF Idles*
General FC-2 Frame Format
Frame Content
0-528 Transmission Word
(4)(4)(0-2112)(24)(4)
Fibre Channel Frame Format
• All FC-2 frames follow the general frame format as shown below
• Idles are ‘Ordered Sets’ used for synchronization and basic signaling
• SOF – Start-of-Frame, EOF – End-of-Frame
* 6 Idle words (24 bytes) requires by TX2 Idle words (8 bytes) guaranteed to RX
51© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Fibre Channel Storage Area Network (SAN)
Switch ASwitch A
FC
EFFC
FCF
Switch BSwitch B
FC
FEFC
FCF
= Fibre Channel= F_Port
FCFHost
Storage
Storage
Host
52© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Storage Protocols
FCIP Protocol and Transport Stack
Parallel SCSIInterfaces Fibre Channel Ethernet
IP
TCP
Parallel SCSIInterfaces Fibre Channel iSCSI
SCSI Commands, Data, and Status
SCSI Block Commands
SCSI StreamCommands
Other SCSI Commands
SCSI Applications (File Systems, Databases)
FCIPFibre Channel
53© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
IP
Storage Protocols - FCIP
• Primary use is for Storage-to-Storage connectivity via an IP WAN/MAN
• The FCIP Link carries encapsulated fibre channel traffic between Link End Points over an IP network by using TCP on port 3225
• The result is a virtual Inter Switch Link (ISL) between FC Fabrics
• FC frame not changed
TCPFCIP
FCSCSI
Data
FCIP Frame Format
54© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
FCIP SAN Interconnect
Switch ASwitch A
FC
EFFC
FCF
Switch BSwitch B
FC
FEFC
GEVE
Switch CSwitch C
FC
EVEGE
GEF
= Fibre Channel= F_Port= E_Port= Virtual E_Port= Gigabit Ethernet
FCFEVEGE
IP
Virtual ISL
FCIP Link
StorageStorageHost
Host
55© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Storage Protocols
iSCSI Protocol and Transport Stack
Parallel SCSIInterfaces Fibre Channel Ethernet
IP
TCP
Parallel SCSIInterfaces Fibre Channel iSCSI
SCSI Commands, Data, and Status
SCSI Block Commands
SCSI StreamCommands
Other SCSI Commands
SCSI Applications (File Systems, Databases)
FCIPFibre Channel
56© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Storage Protocols
• Primary use is for Host-to-Storage connectivity via an IP LAN
• SCSI command and data are encapsulated into iSCSI by adding a special header
• iSCSI data is encapsulated into a TCP packet
• IP is the transport protocol
Storage Protocols—iSCSI
IPTCP
iSCSISCSI
Data
57© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
iSCSI Architecture
• SCSI deviceIP host
iSCSI node initiator
FC storage
iSCSI node target
• SCSI portiSCSI initiator port
iSCSI target port
• Network portalAny network interface with TCP/IP
Provides physical IP network access`
iSCSI Target Port
Network PortalIP AddressTCP Port
iSCSI Initiator Port
Logical Units
iSCSI Target Port
Network PortalIP AddressTCP Port
Network PortalIP AddressTCP Port
SCSI Device (iSCSI Node – Initiator)
iSCSI—SCSI Relationship
SCSI Device (iSCSI Node – Target)
Network PortalHBA/NIC
Network PortalHBA/NIC
IP Network
58© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
iSCSI Host Attachment
Switch ASwitch A
FC
EFFC
FCF
Switch BSwitch B
FC
FEFC
GEVT
= Fibre Channel= F_Port= E_Port= Virtual Target= Gigabit Ethernet
FCFEVTGE
IP
iSCSI
TCP/IP Link
StorageHost iSCSI
Host
59© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Storage Protocols Summary
• Protocols used to transport SCSIParallel SCSI – Legacy host to storage bus topology
Fibre Channel – Robust serial transport network for SCSI
Primary transport for Storage Area Networks
FCIP – FC-2 encapsulation in TCP/IP
SAN-to-SAN Connectivity
Typically used in the WAN/MAN
iSCSI – TCP/IP transport for SCSI protocol
Host-to-SAN Connectivity
Typically used in the LAN
Storage Transport Protocols
60© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Fibre Channel Basics
61© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Fibre Channel Basics
• Fibre Channel Physical AttributesPortsLinksNodesFabrics
• Fibre Channel Operational CharacteristicsNamingAddressingFabric ServicesRoutingZoning
62© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
FC Physical Attributes
63© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Physical
• The main physical objects in Fibre Channel are:Ports
Links
Nodes
Fabric
64© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Fabric Switch
FL_Port
G_Port
G_Port
G_Port
E_Port
F_Port
F_Port
F_Port
E_Port
NodeN_Port
NodeN_Port
NodeN_Port
Node NL_Port
Node NL_Port
Node NL_Port
Port Type Examples
65© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Fibre Channel Port Types
• ‘N’ port: Node ports used to connect devices to switched fabric or point to point configurations.
• ‘F’ port: Fabric ports residing on switches connecting ‘N’ port devices
• ‘L’ port: Loop ports are used in arbitrated loop configurations to build networks without FC switches. These ports often also have ‘N’ port capabilities and are called ‘NL’ ports.
• ‘E’ port: Expansion ports are essentially trunk ports used to connect two Fibre Channel switches
• ‘GL’ port: A generic port capable of operating as either an ‘E’ or ‘F’ port. Its also capable of acting in an ‘L’ port capacity. Auto Discovery.
N N
N F
NL FL
L L
E E
66© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Inter-Switch Link (ISL)
• The interconnection between switches is called the Inter-Switch Link (ISL)
E_Port to E_Port
• Supports all classes of serviceClass 1, 2, 3, and a special Class F
• FC-PH permits consecutive frames of a sequence to be routed over different ISL links for maximum throughput
E E
67© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Other standard and Cisco Specific Port Types
• TE Trunking E-Port that to carry Virtual SAN’s
• Fx Ports found on Over-subscription Port
• SD Span Destination port for Analyzer
• ST Span Tunnel port used in Remote Spanfunction
• TF Trunking F-Port
• TL Translation Loop (Private to Public addressTranslation)
68© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
N_Port
Device
N_Port
Host/ Device Interface
Serial Data Out Serial Data In
Host
FC
Can be more then one N_Port on the device
69© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Fibre Channel Link
• A link consists of2 unidirectional “fibers” transmitting in opposite directions
May be either:
Optical fiber or Copper
• Transmitters may be:Long wave laser
Short wave laser
LED
Electrical
70© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Node
• The equipment which contains one or more• N_Port or NL_Port (topology dependent)• May be:
Computer (HBA)Controller (Port on Disk Subsystem)Device (SCSI FC to Parallel converter)
• Is NOT a switch fabric device
N_Port
Link
N_Port
Link
N_Port
Link
N_Port
Link
Controller
71© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
FC Communications Model
• Point to point
• N_Port to N_Port
• Flow controlled
• Acknowledged
TX
RX
N_Port
Node Node
Link
N_Port
Host
FC
Host
FCRX
TX
Node Node
72© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Fibre Channel Fabric
• The entity which interconnects N_Ports
• Provides routing based on destination address
• Fabric may be:Point to point: No routing required
Arbitrated loop: Routing is distributed throughout attached L_Ports
Switched: Routing provided by switch
Fabric
73© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Point to Point
Communications Model• Source to destination
• Host Bus Adaptor (HBA) keeps in memory a Port Login Table of each N_Port known to it on the network
Node Node
Transmitter
Receiver
Receiver
Node Node
N_Port
Transmitter
N_Port
74© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Communications Model—Source to Destination Based on Address Routing Distributed in the NL_Ports on the Loop
Arbitrated Loop
NL_Node “A”
NL_Node “B”
Link
A B Still Pt. to Pt. Communication
75© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Switched Fabric
N_Port N_PortN_Port N_Port N_PortN_Port
Fabric
Switch
Communications Model—Source to Destination Based on Address Routing through the Fabric. Still a Pt. to Pt. connection. FSPF routing required when more then 2 switches make up the Fabric
A B
A B
76© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
FC Operational Characteristics
77© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Worldwide Names
• Each switch element is assigned a WWN at time of manufacture
• Each switch port is assigned a WWN at the time of manufacture
• During FLOGI the switch identifies the WWN in the service parameters of the accept frame and assigns a Fibre Channel ID (FCID)
• These address assignments can then correlate each fabric port with the switch element
78© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Fabric Addressing
• The 24 bit FCID address is partitioned into 3 fieldsDevice
Area
Domain
• This partitioning helps speed up routing
• Switch element assigns the address to N_Ports
• Address portioning is transparent to N_Ports
DeviceAreaSwitch Domain
8 bits 8 bits 8 bits
Switch Topology
79© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Directory Server
• Repository of information regarding the components that make up the Fibre Channel network
• Located at address ‘FF FF FC’ (Some readings call this the name server)
• Components can register their characteristics with the directory server
• An N_Port can query the directory server for specific information
Query can be the address identifier, WWN and volume names for all SCSI targets
80© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Fabric Controller
• Each switch has a fabric controller
• Assigned address ‘FF FF FD’Every fabric controller in the fabric has the same address
It is the N_Port within the switch
Responsible for managing fabric, initialization, routing, setup and teardown of Class-1 connections
• Responsible to receive request and generate responses for the switch fabric
Information must be consistent independent of which fabric controller responds to a request
81© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Extended Link Services
• Extended link services provide a set of protocol functions used by the port to specify a function or service at another port
Usually sent from N_Port to F_port to perform needed request
The R_CTL field of the first word will be set to 0x22 to indicate an extend link service request
Many ELS services will return a payload in response some have no reply
82© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Extended Link Services
Some of the more important and most used ELS commands are:
• FLOGI F_Port Login
• PLOGI N_Port Login
• FAN Fabric Address Notification
• PRLI Process Login
• PRLO Process Logout
• SCN State Change Notification
• SCR State Change Registration
• RSCN Registered State Change Notification
83© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Fabric Configuration: PS Selection
• A principal switch shall be selected whenever at least one inter-switch link (A link between two E_Port) is established
• The selection process chooses a principal switch, which is then designated to assign domain identifier to all the switches in the fabric, and any who join later the fabric later on
• The principal switch selection can be triggered by anyone of the following events
Switch boot and Exchange Fabric Parameters (EFP)
Build Fabric (BF)
Reconfigure Fabric (RCF)
84© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Fabric Configuration Disruptive/Non-Disruptive
• One of the following three conditions can trigger BF (non-disruptive) or RCF (disruptive)
Two disjoint fabrics are combined together
A principal ISL fails (upstream or downstream)
A switch with Domain_ID request for another Domain_ID
• Whenever a switch receives a BF/RCF, the switch starts F_S_TOV timer and enters the BF/RCF state; It forwards BF/RCF out of all E_ports except the incoming port (only once) and wait for the timer to expire
• When the timer expires, BF/RCF propagation state is left and principal switch selection begins
• BF is not a disruptive process
• RCF is a disruptive process
85© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Fabric Routing: FSPF
• For FSPF a domain ID identifies a single switchThis limits the max number of switches that can support in the Fabric to 239 when FSPF is supported
Treat each Cisco MDS VSAN as separate Fabric each potentially supporting 239 switches
• FSPF performs hop-by-hop routingEach Cisco MDS VSAN runs it’s only FSPF process
Routing between VSAN’s is done with Inter VSAN Routing (IVR)
• FSPF supports hierarchical path selectionProvides the scalable routing tables in large topologies
86© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Fabric Routing: FSPF
• Everyone says HELLO to their neighbor, on all initialized ISLs
• The neighbors say HELLO back, unless they are dead
• When the HELLO packet is received with both originator and recipient domain id, the two way communication is done and:
The ISL is active
The ISL may be available as a two-way path for frames
87© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
FSPF Characteristics
• Uses FSPF as the routing algorithm
• FSPF routes traffic based on destination domain ID
• FSPF uses total cost as the metric to determine most efficient path
• Static routes can be applied
88© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Fabric Switch
FL_Port E_Port E_Port
NodeN_Port
NodeN_Port
NodeN_Port
Node NL_Port
Node NL_Port
Node NL_Port
F_Port
Zone_A
F_Port
F_Port
Zone
_B
Fibre Channel Fabric Zoning
• Fibre Channel zoning is used to restrict access within the fabric to certain nodes/resources
• Zones may overlap devices
89© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Fibre Channel Fabric Zoning
• Zoning operationZone members “see” only other members of the zone
Zones can be configured dynamically based on WWN
Devices can be members of more than one zone
FC-AL zoning allows the creation of private loops on a single hub
Switched fabric zoning can take place at the port or device level
Based on physical switch port
Based on device WWN
Based on LUN ID
• BenefitsSecured device access
Allows operating system co-existence
SAN
Disk1
Host2Disk4
Host1
Disk2 Disk3
ZoneA
ZoneB
ZoneC
90© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Fibre Channel Basics Summary
• Fibre Channel is a very robust, hierarchical standard
• Fibre Channel utilizes a Point-to-Point communications model irrespective of the topology
• Fibre Channel includes a full set of services for naming, addressing, building, and managing fabrics
• Fibre Channel utilizes FSPF, an OSPF like routing protocol to route traffic
• Fibre Channel Zoning is a method of logically grouping devices within a given fabric
We have only scratched the surface of FC Protocol here!
91© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
√ SAN-2501: Fibre Channel Storage Area Network Design√ SAN-2602: SAN Migration and Interoperability√ SAN-2604: SAN Extension Design and Operation√ SAN-2605: iSCSI SAN Design and Operation √ SAN-2606: SAN Virtualization √ SAN-3607: Advanced SAN Design, Virtual Fabric and Fabric Routing√ SAN-3608: Advanced SAN Troubleshooting √ LABSAN-2251, 2252, 2253, 2254, and 2265: Cisco MDS 9000 Labs√ TECSAN-2601 SAN Fundamentals, Protocols, and Architecture
Where Do I Go from here ?
92© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Q and A
93© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Recommended Reading
• Continue your Cisco Networkers learning experience with further reading from Cisco Press
• Check the Recommended Reading flyer for suggested books
Available Onsite at the Cisco Company Store
94© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.
Complete Your Online Session Evaluation
• Win fabulous prizes; Give us your feedback
• Receive ten Passport Points for each session evaluation you complete
• Go to the Internet stations located throughout the Convention Center to complete your session evaluation
• Drawings will be held in theWorld of Solutions
Tuesday, June 20 at 12:15 p.m.
Wednesday, June 21 at 12:15 p.m.
Thursday, June 22 at 12:15 p.m. and 2:00 p.m.
95© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN-150112686_05_2006_c2 © 2005 Cisco Systems, Inc. All rights reserved.