TechBook: Extended Distance Technologies

Extended Distance Technologies

Version 1.2

• Distance Extension Technologies Overview

• Distance Extension Considerations

• Distance Extension Solutions

Eric PunVinay Jonnakuti

Extended Distance Technologies TechBook2

Copyright © 2011 - 2013 EMC Corporation. All rights reserved.

EMC believes the information in this publication is accurate as of its publication date. The information issubject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION MAKES NOREPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THISPUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY ORFITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any EMC software described in this publication requires an applicablesoftware license.

EMC2, EMC, and the EMC logo are registered trademarks or trademarks of EMC Corporation in the UnitedState and other countries. All other trademarks used herein are the property of their respective owners.

For the most up-to-date regulator document for your product line, go to EMC Online Support(https://support.emc.com).

Part number H8079.2

Contents

Preface.............................................................................................................................. 7

Chapter 1 Extended Distance OverviewEarly implementations of SAN environments.............................. 14DWDM ............................................................................................... 15CWDM................................................................................................ 19

Differences between DWDM and CWDM............................. 19SONET................................................................................................ 21GbE...................................................................................................... 23TCP/IP................................................................................................ 24

TCP terminology........................................................................ 24TCP error recovery .................................................................... 28Network congestion .................................................................. 31Internet Protocol security (IPsec) ............................................ 32

Chapter 2 Distance Extension ConsiderationsLink speed.......................................................................................... 36Data buffering and flow control ..................................................... 37

Fibre Channel ............................................................................. 37Maximum supported distance per Fibre Channel

BB_Credit guidelines.............................................................. 38Buffer-to-buffer credit information ......................................... 41

TCP/IP window................................................................................ 51Active and passive devices.............................................................. 52

Buffer-to-buffer local termination ........................................... 52SRDF with SiRT.......................................................................... 54Fast write/ write acceleration.................................................. 56SiRT with distance vendor write acceleration....................... 57

Extended Distance Technologies TechBook 3

Contents

Link initialization ...................................................................... 58FC SONET/GbE/IP ......................................................................... 59Network stability and error recovery ............................................ 60

Chapter 3 IP-Based Distance Extension SolutionsNetwork design best practices........................................................ 62

Network conditions impact on effective throughput .......... 62EMC-Brocade distance extension solutions.................................. 64

Brocade 7500............................................................................... 65Brocade 7800............................................................................... 67

Configuring IPsec ............................................................................. 76Fast Write and tape pipelining........................................................ 78

Supported configurations......................................................... 79EMC-Cisco MDS distance extension solution .............................. 82

Supported configurations......................................................... 82Symmetrix setup........................................................................ 83VNX setup .................................................................................. 83CLARiiON setup ....................................................................... 83References ................................................................................... 83

EMC-QLogic distance extension solution..................................... 84Supported configurations......................................................... 84Scalability.................................................................................... 85Best practices .............................................................................. 86SmartWrite.................................................................................. 86References ................................................................................... 87

Summary............................................................................................ 88

Index................................................................................................................................ 91


Title Page

Figures

1 DWDM example ............................................................................................. 152 Fibre Channel link extension ........................................................................ 173 STS-1 organization ......................................................................................... 224 Slow start and congestion avoidance .......................................................... 305 Fast retransmit ................................................................................................ 316 BB_Credit mechanism ................................................................................... 387 Flow control managed by Fibre Channel switch (without buffering

from distance extension devices) ...................................................................538 Flow control (with buffering from distance extension devices) .............. 549 Normal write command process .................................................................. 5510 SRDF SiRT ....................................................................................................... 5611 Write command with SiRT ............................................................................ 5712 All F_Ports will benefit .................................................................................. 5813 Link initialization (More than 100 ms R_T_TOV) ..................................... 5914 Brocade 7500 configuration example .......................................................... 6715 Basic overview of Trunking components ................................................... 6916 Single tunnel, Fastwrite and Tape Pipelining enabled ............................. 7217 Multiple tunnels to multiple ports, Fastwrite, and Tape Pipelining

enabled on a per-tunnel/per-port basis....................................................... 7218 Single tunnel, Fast Write and tape pipelining enabled ............................. 8019 Multiple tunnels to multiple ports ............................................................... 8120 Cisco MDS 9000 distance extension example ............................................. 8221 SANbox 6142 Intelligent Router ................................................................... 85


Figures


Preface

This EMC Engineering TechBook provides a basic understanding of distanceextension technologies and information to consider when working withextended distance. IP-based distance extension solutions are also included.

E-Lab would like to thank all the contributors to this document, includingEMC engineers, EMC field personnel, and partners. Your contributions areinvaluable.

As part of an effort to improve and enhance the performance and capabilitiesof its product lines, EMC periodically releases revisions of its hardware andsoftware. Therefore, some functions described in this document may not besupported by all versions of the software or hardware currently in use. Forthe most up-to-date information on product features, refer to your productrelease notes. If a product does not function properly or does not function asdescribed in this document, please contact your EMC representative.

Audience This TechBook is intended for EMC field personnel, includingtechnology consultants, and for the storage architect, administrator,and operator involved in acquiring, managing, operating, ordesigning a networked storage environment that contains EMC andhost devices.

EMC Support Matrixand E-Lab

InteroperabilityNavigator

For the most up-to-date information, always consult the EMC SupportMatrix (ESM), available through E-Lab Interoperability Navigator(ELN), at: http://elabnavigator.EMC.com, under the PDFs andGuides tab.

Under the PDFs and Guides tab resides a collection of printableresources for reference or download. All of the matrices, includingthe ESM (which does not include most software), are subsets of the


http://elabnavigator.EMC.com


https://elabnavigator.emc.com

8

Preface

E-Lab Interoperability Navigator database. Included under this tabare:

◆ The EMC Support Matrix, a complete guide to interoperable, andsupportable, configurations.

◆ Subset matrices for specific storage families, server families,operating systems or software products.

◆ Host connectivity guides for complete, authoritative informationon how to configure hosts effectively for various storageenvironments.

Under the PDFs and Guides tab, consult the Internet Protocol pdfunder the "Miscellaneous" heading for EMC's policies andrequirements for the EMC Support Matrix.

Relateddocumentation

Related documents include:

◆ The following documents, including this one, are availablethrough the E-Lab Interoperability Navigator, TopologyResource Center tab, at http://elabnavigator.EMC.com.

These documents are also available at the following location:

http://www.emc.com/products/interoperability/topology-resource-center.htm

• Backup and Recovery in a SAN TechBook• Building Secure SANs TechBook• Fibre Channel over Ethernet (FCoE): Data Center Bridging (DCB)

Concepts and Protocols TechBook• Fibre Channel over Ethernet (FCoE): Data Center Bridging (DCB)

Case Studies TechBook• Fibre Channel SAN Topologies TechBook• iSCSI SAN Topologies TechBook• Networked Storage Concepts and Protocols TechBook• Networking for Storage Virtualization and RecoverPoint TechBook• WAN Optimization Controller Technologies TechBook• EMC Connectrix SAN Products Data Reference Manual• Legacy SAN Technologies Reference Manual• Non-EMC SAN Products Data Reference Manual

◆ EMC Support Matrix, available through E-Lab InteroperabilityNavigator at http://elabnavigator.EMC.com >PDFs and Guides

◆ RSA security solutions documentation, which can be found athttp://RSA.com > Content Library

Extended Distance Technologies TechBook






http://RSA.com

Preface

All of the following documentation and release notes can be found atEMC Online Support at https://support.emc.com.

EMC hardware documents and release notes include those on:

◆ Connectrix B series◆ Connectrix MDS (release notes only)◆ VNX series◆ CLARiiON◆ Celerra◆ Symmetrix

EMC software documents include those on:

◆ RecoverPoint◆ Invista◆ TimeFinder◆ PowerPath

The following E-Lab documentation is also available:

◆ Host Connectivity Guides◆ HBA Guides

For Cisco and Brocade documentation, refer to the vendor’s website.

◆ http://cisco.com

◆ http://brocade.com

Authors of thisTechBook

This TechBook was authored by Eric Pun and Vinay Jonnakuti, withcontributions from the following EMC employees: Kieran Desmond,Ger Halligan, and Ron Stern, along with other EMC engineers, EMCfield personnel, and partners.

Eric Pun is a Senior Systems Integration Engineer and has been withEMC for over 12 years. For the past several years, Eric has worked inE-lab qualifying interoperability between Fibre Channel switchedhardware and distance extension products. The distance extensiontechnology includes DWDM, CWDM, OTN, FC-SONET, FC-GbE,FC-SCTP, and WAN Optimization products. Eric has been acontributor to various E-Lab documentation, including the SRDFConnectivity Guide.

Vinay Jonnakuti is a Sr. Corporate Systems Engineer in the UnifiedStorage division of EMC focusing on VNX and VNXe products,working on pre-sales deliverables including collateral, customerpresentations, customer beta testing and proof of concepts. Vinay hasbeen with EMC's for over 5 years. Prior to his current position, Vinay


10

Preface

worked in EMC E-Lab leading the qualification and architecting ofsolutions with WAN-Optimization appliances from various partnerswith various replication technologies, including SRDF (GigE/FCIP),SAN-Copy, MirrorView, VPLEX, and RecoverPoint. Vinay alsoworked on Fibre Channel and iSCSI qualification on the VMAXStorage arrays.

Conventions used inthis document

EMC uses the following conventions for special notices:

IMPORTANT

An important notice contains information essential to software orhardware operation.

Note: A note presents information that is important, but not hazard-related.

Typographical conventionsEMC uses the following type style conventions in this document.

Normal Used in running (nonprocedural) text for:• Names of interface elements, such as names of windows, dialog

boxes, buttons, fields, and menus• Names of resources, attributes, pools, Boolean expressions, buttons,

DQL statements, keywords, clauses, environment variables,functions, and utilities

• URLs, pathnames, filenames, directory names, computer names,links, groups, service keys, file systems, and notifications

Bold Used in running (nonprocedural) text for names of commands,daemons, options, programs, processes, services, applications, utilities,kernels, notifications, system calls, and man pages

Used in procedures for:• Names of interface elements, such as names of windows, dialog

boxes, buttons, fields, and menus• What the user specifically selects, clicks, presses, or types

Italic Used in all text (including procedures) for:• Full titles of publications referenced in text• Emphasis, for example, a new term• Variables

Courier Used for:• System output, such as an error message or script• URLs, complete paths, filenames, prompts, and syntax when shown

outside of running text

Courier bold Used for specific user input, such as commands


Preface

Where to get help EMC support, product, and licensing information can be obtained onthe EMC Online Support site as described next.

Note: To open a service request through the EMC Online Support site, youmust have a valid support agreement. Contact your EMC sales representativefor details about obtaining a valid support agreement or to answer anyquestions about your account.

Product informationFor documentation, release notes, software updates, or forinformation about EMC products, licensing, and service, go to theEMC Online Support site (registration required) at:

https://support.EMC.com

Technical supportEMC offers a variety of support options.

Support by Product — EMC offers consolidated, product-specificinformation on the Web at:

https://support.EMC.com/products

The Support by Product web pages offer quick links toDocumentation, White Papers, Advisories (such as frequently usedKnowledgebase articles), and Downloads, as well as more dynamiccontent, such as presentations, discussion, relevant CustomerSupport Forum entries, and a link to EMC Live Chat.

EMC Live Chat — Open a Chat or instant message session with anEMC Support Engineer.

Courier italic Used in procedures for:• Variables on the command line• User input variables

< > Angle brackets enclose parameter or variable values supplied by theuser

[ ] Square brackets enclose optional values

| Vertical bar indicates alternate selections — the bar means “or”

{ } Braces enclose content that the user must specify, such as x or y or z

... Ellipses indicate nonessential information omitted from the example


https://support.EMC.com

http://support.EMC.com

https://support.EMC.com/products

12

Preface

eLicensing supportTo activate your entitlements and obtain your Symmetrix license files,visit the Service Center on https://support.EMC.com, as directed onyour License Authorization Code (LAC) letter e-mailed to you.

For help with missing or incorrect entitlements after activation (thatis, expected functionality remains unavailable because it is notlicensed), contact your EMC Account Representative or AuthorizedReseller.

For help with any errors applying license files through SolutionsEnabler, contact the EMC Customer Support Center.

If you are missing a LAC letter, or require further instructions onactivating your licenses through the Online Support site, contactEMC's worldwide Licensing team at [email protected] or call:

◆ North America, Latin America, APJK, Australia, New Zealand:SVC4EMC (800-782-4362) and follow the voice prompts.

◆ EMEA: +353 (0) 21 4879862 and follow the voice prompts.

We'd like to hear from you!Your suggestions will help us continue to improve the accuracy,organization, and overall quality of the user publications. Send youropinions of this document to:

[email protected]

Your feedback on our TechBooks is important to us! We want ourbooks to be as helpful and relevant as possible. Send us yourcomments, opinions, and thoughts on this or any other TechBook to:

[email protected]




1

To comprehend the distance extension solutions for Storage AreaNetworks it is important to understand and recall the challengeswhen implementing SAN connectivity over remote distances. Thefollowing information is provided in this chapter:

◆ Early implementations of SAN environments............................... 14◆ DWDM................................................................................................. 15◆ CWDM................................................................................................. 19◆ SONET ................................................................................................. 21◆ GbE....................................................................................................... 23◆ TCP/IP................................................................................................. 24

Note: Refer to the “FCIP configuration” section in the WAN OptimizationController Technologies TechBook, located at http://elabnavigator.EMC.com,Topology Resource Center tab, for more details on Brocade and Cisco FCIPconfiguration information.

Note: Refer to the “FCIP configuration and setup” section in the WANOptimization Controller Technologies TechBook, located athttp://elabnavigator.EMC.com, Topology Resource Center tab, for adistance extension case study using FCIP.

Extended DistanceOverview

Extended Distance Overview 13



14

Extended Distance Overview

Early implementations of SAN environmentsTo increase a single port between two Fibre Channel switchesseparated by a large geographical distance, every two strands(transmit, receive) of optical fiber cable were required to be physicallyadded by the distance provider. The customer would generally incurexpensive construction, service, and maintenance costs when addinga bulk of fiber cables intended to satisfy current E_Port connectivityrequirements while allowing future growth potential andredundancy against accidental fiber breaks. Existing fibers that wereused for Ethernet implementations could not be shared and requiredseparate dedicated channels per protocol. The challenges involvedwith this process would stem anywhere from mandatory toextraneous costs associated with fiber cable maintenance.

In addition to costs, there were physical hardware limitations toachieving connectivity between (at least) two geographicallyseparated sites. Fibre Channel optics installed on the Fibre Channelswitch were at the mercy of the limited optical output transmissionpower. Even with repeater technology, distortion of the opticalwavelength transmitted by the optics can occur over several hops.

The Fibre Channel switches provided limitations as well. Linkinitialization and flow control were solely controlled by the FibreChannel switches. The Fibre Channel standard would actually dictatethe thresholds in regards to supporting large distances throughoptical connectivity and the obtainable bandwidth between two FibreChannel ports.

To finalize the list of challenges that SAN environments had toovercome, each Fibre Channel switch provider had its ownnon-standard and standard ways of implementing their nativeenvironments. This may deviate from the mass interpretation of theFibre Channel standards.



DWDMDense Wavelength Division Multiplexing (DWDM) is a process inwhich different channels of data are carried at different wavelengthsover one pair of fiber-optic links. This is in contrast with aconventional fiber-optic system in which just one channel is carriedover a single wavelength traveling through a single fiber.

Using DWDM, several separate wavelengths (or channels) of datacan be multiplexed into a multicolored light stream transmitted on asingle optical fiber (dark fiber). This technique to transmit severalindependent data streams over a single fiber link is an approach toopening up the conventional optical fiber bandwidth by breaking itup into many channels, each at a different optical wavelength (adifferent color of light). Each wavelength can carry a signal at any bitrate less than an upper limit defined by the electronics, typically up toseveral gigabits per second.

Different data formats being transmitted at different data rates can betransmitted together. Specifically, IP data, ESCON SRDF®, FibreChannel SRDF, SONET data, and ATM data can all be traveling at thesame time within the optical fiber.

DWDM systems are independent of protocol or format, and noperformance impacts are introduced by the system itself.

Figure 1 illustrates the DWDM technology concept:

Figure 1 DWDM example

DWDM 15

16


For EMC® customers it means that multiple SRDF® channels andFibre Channel Inter Switch Links (ISL) can be transferred over onepair of fiber links along with traditional network traffic. This isespecially important where fiber links are at a premium. For example,a customer may be leasing fiber, so the more traffic they can run overa single link, the more cost effective the solution.

With today's technology, the capacity of a single pair of fiber strandsis virtually unlimited. The limitation comes from the DWDM itself.Optical-to-electrical transfers for switching and channel protectionare required and limit the input traffic per channel.

Available DWDM topologies include point-to-point and ringconfigurations with protected and unprotected schemas. DWDMtechnology can also be used to tie two or more metro area datacenters together as one virtual data center.

DWDM systems can multiplex and de-multiplex a large amount ofchannel quantities. Each channel is allocated its own specificwavelength (lambda) band assignment. Each wavelength band isgenerally separated by 10 nm spacing(s). As optical technologiesimprove, separations between each channel may be further reducedenabling more channels to be packed (tighter) onto a single duplexdark fiber.

DWDM has a higher cost associated due to greater channelconsolidation, flexibility, utilization of higher quality hardwareprecision-cooling components (to prevent low frequency signal drift)and the capabilities of regenerating, re-amplifying and reshaping (3R)wavelengths assigned to channels to ensure optical connectivity overvast distances.

Varying circuits pack capabilities are also offered in a DWDMenvironment. DWDM circuit packs / blades can provide thefollowing protocol conversions:

◆ Fibre Channel to SONET

◆ Fibre Channel to Gigabit Ethernet

◆ Fibre Channel to IP

In addition, some circuit packs can enable features such as writeacceleration and buffer-to-buffer credit spoofing. To verify the latestsupported distance systems and features, refer to the EMC SupportMatrix.





Figure 2 shows a general concept of Fibre Channel link extensionusing DWDM.

Figure 2 Fibre Channel link extension

Note: All components are randomly selected and do not reflect a specificsetup or configuration.

Note: Distance limitation may also be affected by application responsetime-out values and should consider signal propagation delay over sitedistance.

The following list provides general envelope guidelines for usingDWDM systems:

◆ May be used for ESCON RDF distance extension, with directconnection between EMC Symmetrix® ESCON director ports andDWDM input ports.

◆ May be used for ISL extension of Fibre Channel switched fabrics.(E-Lab™ Navigator describes switch compatibility.)

◆ Fabric topology guidelines are provided per Fibre Channel switchtopology documentation.

LocalDWDM FC switch

Server

RemoteDWDM

d4 d2 d1 d3 d5

FC switch

d1 = DWDM signal over dark fiber medium.d2 and d3 = Local ISL connections between switches and DWDM input. Can be SM or MM depending on DWDM and switch interfaces or local distance requirements.d4 and d5 = Local storage or server connections into the fabric.

StorageStorage

DWDM 17


18


◆ Direct connections between host HBA or Symmetrix FibreChannel director to a DWDM port are not supported. E-LabNavigator contains specific DWDM distance and topologyguidelines.

◆ As a general approach, two distances need to be measured. Theshorter of the two is the maximum distance to be supported in thesite.

For differences between DWDM and CWDM, refer to “Differencesbetween DWDM and CWDM” on page 19.





CWDMCoarse Wave Division Multiplexing (CWDM), like DWDM, usessimilar processes of multiplexing and de-multiplexing differentchannels by assigning different wavelengths to each channel. CWDMis intended to consolidate environments containing a low number ofchannels at a reduced cost.

CWDM contains 20 nm separations between each assigned channelwavelength. CWDM technology generally uses cost-effectivehardware components that require a reduced amount ofprecision-cooling components usually dominant in DWDM solutionsdue to the wider separations. With CWDM technology the number ofchannel wavelengths to be packed onto a single fiber is greatlyreduced.

CWDM implementations, like DWDM, utilize anoptical-to-electrical-to-optical technology where all the channels aremultiplexed into a single CWDM device performing theoptical-to-electrical-to-optical conversion.

A CWDM connectivity solution can use optics generating a higherwavelength with increased output optical power. Each channel isdesignated its own specific wavelength by the specific hot-pluggableCWDM GBIC/SFP optic installed on the Fibre Channel Switches.With clean fibers, minimal patch panel connections, and ampleoptical power, CWDM optics alone can provide connectivitydistances of up to 100 km per channel. To complete this solution apassive MUX/DEMUX is required to consolidate multiplechannel-wavelengths into a single duplex 9-micron dark fiber.

Differences between DWDM and CWDMThe following are differences between DWDM and CWDM:

◆ Number of channels that are supported per solution.

DWDM systems can support channels ranging from 16 channelsor above while CWDM supports 16 channels or below.

◆ CWDM GBIC/SFP optics can be used to increase the wavelengthoutput of a channel (such as, FC-switch optics).

CWDM 19

20


The CWDM GBIC/SFP optics is usually installed in the FibreChannel switch or client device. The wavelength and opticalpower enhanced links are then multiplexed and de-multiplexedto and from a single-mode 9-micron dark fiber.

◆ Costs.

Hardware components included with DWDM units are higher incost due to precision-cooling techniques required to preventsignal drift. DWDM offers greater channel flexibility and capacity.

◆ Configurations can be complex with CWDM.

CWDM requires specific optics for each specific wavelength.Growth for a CWDM environment is limited and difficult tomanage when supporting environments growing to largerchannel support. More cabling would be required, therebyincreasing complexity.

◆ DWDM devices offer circuit packs with numerous features suchas, protocol conversions, buffer-to-buffer credit spoofing, writeacceleration).



SONETSynchronous Optical NETwork, (SONET), is a standard for opticaltelecommunications transport, developed by the Exchange CarriersStandards Association for ANSI. SONET defines a technology forcarrying different capacity signals through a synchronous opticalnetwork. The standard defines a byte-interleaved multiplexedtransport occupying the physical layer of the OSI model.

Synchronization is provided by one principal network element with avery stable clock (Stratum 3), which is sourced on its outgoing OC-Nsignal. This clock is then used by other network elements for theirclocks (loop timing).

SONET is useful in a SAN for consolidating multiple low-frequencychannels (Client ESCON and 1, 2 Gb Fibre Channel) into a singlehigher-speed connection. This can reduce DWDM wavelengthrequirements in an existing SAN infrastructure. It can also allow adistance solution to be provided from any SONET service carrier,saving the expense of running private optical cable over longdistances.

The basic SONET building block is an STS-1 (Synchronous TransportSignal), composed of the transport overhead plus a SynchronousPayload Envelope (SPE), totaling 810 bytes. The 27-byte transportoverhead is used for operations, administration, maintenance, andprovisioning. The remaining bytes make up the SPE, of which anadditional nine bytes are path overhead. It is arranged as depicted inFigure 3. Columns 1, 2, and 3 are the transport overhead.

SONET 21

22


Figure 3 STS-1 organization

An STS-1 operates at 51.84 Mb/s, so multiple STS-1s are required toprovide the necessary bandwidth for ESCON, Fibre Channel, andEthernet, as shown in Table 1. Multiply the rate by 95% to obtain theusable bandwidth in an STS-1 (reduction due to overhead bytes).

One OC-48 can carry approximately 2.5 channels of 1 Gb/s traffic, ssshown in Table 1. To achieve higher data rates for client connections,multiple STS-1s are byte-interleaved to create an STS-N. SONETdefines this as byte-interleaving three STS-1s into an STS-3, andsubsequently interleaving STS-3s.

By definition, each STS is still visible and available for ADD/DROPmultiplexing in SONET, although most SAN requirements can be metwith less complex point-to-point connections. The addition ofDWDM can even further consolidate multiple SONET connections(OC-48), while also providing distance extension.

Table 1 SONET/Synchronous Digital Hierarchy (SDH)

STS Optical carrier Optical carrier rate (Mb/s)

STS-1 OC-1 51.840

STS-3 OC-3 155.520

STS-12 OC-12 622.080

STS-48 OC-48 2488.320

STS-192 OC-192 9953.280



GbEGigabit Ethernet (GbE) is a terminology describing an array oftechnologies involved in the transmission of Ethernet packets at therate of 1024 megabits (Mb/s) or 1 gigabit per second. GigabitEthernet is specifically designed to surpass the traditional 10/100Mb/s link speeds. GbE is defined by the IEEE publication 802.3z,which was standardized in June, 1998. This is a physical layerstandard following elements of the ANSI Fibre Channel’s physicallayer. This standard is one of many additions to the original Ethernetstandard (802.3 - Ethernet Frame) published in 1985 by the IEEEorganization. The following are nomenclature and characteristics ofGbE.

◆ 1000Base-SX is defined as a fiber-optic Gigabit Ethernet standardencompassing the use of multi-mode (50 or 62.5 micron) fiberwith 850 nanometer wavelengths. Distances of over 500 meterscan be achieved.

◆ 1000Base-Lx is defined as a fiber-optic Gigabit Ethernet standardencompassing the use of single-mode (9 micron) fiber with 1310nanometer wavelengths. Distances of 10 km or more can beachieved.

◆ Copper coaxial cabling, multi-mode fiber-optic cabling (50 and62.5 micron) and single-mode (9 micron) cabling are availablechoices for the 802.3z standard.

◆ GbE is mainly used in distance extension products as thetransport layer for protocol such as TCP/IP. However, in somecases the product is based on a vendor-unique protocol.

◆ Distance products using GbE may offer features such ascompression, write acceleration, and buffer credit spoofing

GbE 23

24


TCP/IPThe Transmission Control Protocol (TCP) is a connection-orientedtransport protocol that guarantees reliable in-order delivery of astream of bytes between the endpoints of a connection. TCP achievesthis by assigning each byte of data a unique sequence number,maintaining timers, acknowledging received data through the use ofacknowledgements (ACKs), and retransmission of data if necessary.Once a connection is established between the endpoints data can betransferred. The data stream that passes across the connection isconsidered a single sequence of eight-bit bytes, each of which is givena sequence number.

This section contains information on the following:

◆ “TCP terminology” on page 24

◆ “TCP error recovery” on page 28

◆ “Network congestion” on page 31

◆ “Internet Protocol security (IPsec)” on page 32

TCP terminologyThis section provides information for TCP terminology.

Acknowledgements(ACKs)

The TCP acknowledgement scheme is cumulative as it acknowledgesall the data received up until the time the ACK was generated. AsTCP segments are not of uniform size and a TCP sender mayretransmit more data than what was in a missing segment, ACKs donot acknowledge the received segment, rather they mark the positionof the acknowledged data in the stream. The policy of cumulativeacknowledgement makes the generation of ACKs easy and any lossof ACKs do not force the sender to retransmit data. The disadvantageis the sender does not receive any detailed information about the datareceived except the position in the stream of the last byte that hasbeen received.

Delayed ACKs Delayed ACKs allow a TCP receiver to refrain from sending an ACKfor each incoming segment. However, a receiver should send an ACKfor every second full-sized segment that arrives. Furthermore, thestandard mandates a receiver must not withhold an ACK for morethan 500 ms. The receivers should not delay ACKs that acknowledgeout-of-order segments.



Maximum segmentsize (MSS)

The maximum segment size (MSS) is the maximum amount of data,specified in bytes, that can transmitted in a segment between the twoTCP endpoints. The MSS is decided by the endpoints, as they need toagree on the maximum segment they can handle. Deciding on a goodMSS is important in a general inter-networking environment becausethis decision greatly affects performance. It is difficult to choose agood MSS value since a very small MSS means an under-utilizednetwork, whereas a very large MSS means large IP datagrams thatmay lead to IP fragmentation, greatly hampering the performance.An ideal MSS size would be when the IP datagrams are as large aspossible without any fragmentation anywhere along the path fromthe source to the destination. When TCP sends a segment with theSYN bit set during connection establishment, it can send an optionalMSS value up to the outgoing interface’s MTU minus the size of thefixed TCP and IP headers. For example, if the MTU is 1500 (Ethernetstandard), the sender can advertise a MSS of 1460 (1500 minus 40).

Maximumtransmission unit

(MTU)

Each network interface has its own MTU that defines the largestpacket that it can transmit. The MTU of the media determines themaximum size of the packets that can be transmitted without IPfragmentation.

Retransmission A TCP sender starts a timer when it sends a segment and expects anacknowledgement for the data it sent. If the sender does not receivean acknowledgement for the data before the timer expires, it assumesthat the data was lost or corrupted and retransmits the segment. Sincethe time required for the data to reach the receiver and theacknowledgement to reach the sender is not constant (because of thevarying Internet delays), an adaptive retransmission algorithm isused to monitor performance of each connection and conclude areasonable value for timeout based on the round trip time.

SelectiveAcknowledgement

(SACK)

TCP may experience poor performance when multiple packets arelost from one window of data. With the limited information availablefrom cumulative acknowledgements, a TCP sender can only learnabout a single lost packet per round trip time. An aggressive sendercould choose to retransmit packets early, but such retransmittedsegments may have already been successfully received. The SelectiveAcknowledgement (SACK) mechanism, combined with a selectiverepeat retransmission policy, helps to overcome these limitations. Thereceiving TCP sends back SACK packets to the sender confirmingreceipt of data and specifies the holes in the data that has beenreceived. The sender can then retransmit only the missing datasegments. The selective acknowledgment extension uses two TCP

TCP/IP 25

26


options. The first is an enabling option, SACKpermitted, which maybe sent in a SYN segment to indicate that the SACK option can beused once the connection is established. The other is the SACKoption itself, which may be sent over an established connection oncepermission has been given by SACKpermitted.

TCP segment The TCP segments are units of transfer for TCP and used to establisha connection, transfer data, send ACKs, advertise window size andclose a connection. Each segment is divided into three parts:

◆ Fixed header of 20 bytes

◆ Optional variable length header, padded out to a multiple of 4bytes

◆ Data

The maximum possible header size is 60 bytes. The TCP headercarries the control information. SOURCE PORT andDESTINATION PORT contain TCP port numbers that identify theapplication programs at the endpoints. The SEQUENCE NUMBERfield identifies the position in the sender’s byte stream of the firstbyte of attached data, if any, and the ACKNOWLEDGEMENTNUMBER field identifies the number of the byte the source expectsto receive next. The ACKNOWLEDGEMENT NUMBER field isvalid only if the ACK bit in the CODE BITS field is set. The 6-bitCODE BITS field is used to determine the purpose and contents ofthe segment. The HLEN field specifies the total length of the fixedplus variable headers of the segment as a number of 32-bit words.TCP software advertises how much data it is willing to receive byspecifying its buffer size in the WINDOW field. The CHECKSUMfield contains a 16-bit integer checksum used to verify the integrity ofthe data as well as the TCP header and the header options. The TCPheader padding is used to ensure that the TCP header ends and databegins on a 32-bit boundary. The padding is composed of zeros.

TCP window A TCP window is the amount of data a sender can send withoutwaiting for an ACK from the receiver. The TCP window is a flowcontrol mechanism and ensures that no congestion occurs in thenetwork. For example, if a pair of hosts are talking over a TCPconnection that has a TCP window size of 64 KB, the sender can onlysend 64 KB of data and it must stop and wait for anacknowledgement from the receiver that some or all of the data hasbeen received. If the receiver acknowledges that all the data has beenreceived. The sender is free to send another 64 KB. If the sender getsback an acknowledgement from the receiver that it received the first



32 KB (which is likely if the second 32 KB was still in transit or it islost), then the sender could only send another 32 KB since it cannothave more than 64 KB of unacknowledged data outstanding (thesecond 32 KB of data plus the third).

The primary reason for the window is congestion control. The wholenetwork connection, which consists of the hosts at both ends, therouters in between, and the actual connections themselves, mighthave a bottleneck somewhere that can only handle so much data sofast. The TCP window throttles the transmission speed down to alevel where congestion and data loss do not occur.

The factors affecting the window size are as follows:

Receiver’s advertised windowThe time taken by the receiver to process the received data and sendACKs may be greater than the sender’s processing time, so it isnecessary to control the transmission rate of the sender to prevent itfrom sending more data than the receiver can handle, thus causingpacket loss. TCP introduces flow control by declaring a receivewindow in each segment header.

Sender’s congestion windowThe congestion window controls the number of packets a TCP flowhas in the network at any time. The congestion window is set usingan Additive-Increase, Multiplicative-Decrease (AIMD) mechanismthat probes for available bandwidth, dynamically adapting tochanging network conditions.

Usable windowThis is the minimum of the receiver’s advertised window and thesender’s congestion window. It is the actual amount of data thesender is able to transmit. The TCP header uses a 16 bit field to reportthe receive window size to the sender. Therefore, the largest windowthat can be used is 2**16 = 65K bytes.

Window scalingThe ordinary TCP header allocates only 16 bits for windowadvertisement. This limits the maximum window that can beadvertised to 64 KB, limiting the throughput. RFC 1323 provides thewindow scaling option, to be able to advertise windows greater than64 KB. Both the endpoints must agree to use window scaling duringconnection establishment.

The window scale extension expands the definition of the TCPwindow to 32 bits and then uses a scale factor to carry this 32- bit

TCP/IP 27

28


value in the 16-bit Window field of the TCP header (SEG.WND inRFC-793). The scale factor is carried in a new TCP option — WindowScale. This option is sent only in a SYN segment (a segment with theSYN bit on), hence the window scale is fixed in each direction when aconnection is opened.

TCP error recoveryIn TCP, each source determines how much capacity is available in thenetwork so it knows how many packets it can safely have in transit.Once a given source has this many packets in transit, it uses thearrival of an ACK as a signal that some of its packets have left thenetwork and it is therefore safe to insert new packets into the networkwithout adding to the level of congestion. TCP uses congestioncontrol algorithms to determine the network capacity. From thecongestion control point of view, a TCP connection is in one of thefollowing states.

◆ Slow start: After a connection is established and after a loss isdetected by a timeout or by duplicate ACKs.

◆ Fast recovery: After a loss is detected by fast retransmit.

◆ Congestion avoidance: In all other cases. Congestion avoidanceand slow start work hand-in-hand. The congestion avoidancealgorithm assumes that the chance of a packet being lost due todamage is very small. Therefore, the loss of a packet means thereis congestion somewhere in the network between the source anddestination. Occurrence of a timeout and the receipt of duplicateACKs indicates packet loss.

When congestion is detected in the network it is necessary to slowthings down, so the slow start algorithm is invoked. Two parameters,the congestion window (cwnd) and a slow start threshold (ssthresh),are maintained for each connection. When a connection isestablished, both of these parameters are initialized. The cwnd isinitialized to one MSS. The ssthresh is used to determine whether theslow start or congestion avoidance algorithm is to be used to controldata transmission. The initial value of ssthresh may be arbitrarilyhigh (usually ssthresh is initialized to 65535 bytes), but it may bereduced in response to congestion.

The slow start algorithm is used when cwnd is less than ssthresh,while the congestion avoidance algorithm is used when cwnd isgreater than ssthresh. When cwnd and ssthresh are equal, the sendermay use either slow start or congestion avoidance.



TCP never transmits more than the minimum of cwnd and thereceiver’s advertised window. When a connection is established, or ifcongestion is detected in the network, TCP is in slow start and thecongestion window is initialized to one MSS. Each time an ACK isreceived, the congestion window is increased by one MSS. The senderstarts by transmitting one segment and waiting for its ACK. Whenthat ACK is received, the congestion window is incremented fromone to two, and two segments can be sent. When each of those twosegments is acknowledged, the congestion window is increased tofour, and so on. The window size increases exponentially during slowstart as shown in Figure 4 on page 30. When a time-out occurs or aduplicate ACK is received, ssthresh is reset to one half of the currentwindow (that is, the minimum of cwnd and the receiver's advertisedwindow). If the congestion was detected by an occurrence of atimeout the cwnd is set to one MSS.

When an ACK is received for data transmitted the cwnd is increased,but the way it is increased depends on whether TCP is performingslow start or congestion avoidance. If the cwnd is less than or equalto the ssthresh, TCP is in slow start and slow start continues untilTCP is halfway to where it was when congestion occurred, thencongestion avoidance takes over. Congestion avoidance incrementsthe cwnd by MSS squared divided by cwnd (in bytes) each time anACK is received, increasing the cwnd linearly as shown in Figure 4.This provides a close approximation to increasing cwnd by, at most,one MSS per RTT.

TCP/IP 29

30


Figure 4 Slow start and congestion avoidance

A TCP receiver generates ACKs on receipt of data segments. TheACK contains the highest contiguous sequence number the receiverexpects to receive next. This informs the sender of the in-order datathat was received by the receiver. When the receiver receives asegment with a sequence number greater than the sequence numberit expected to receive, it detects the out-of-order segment andgenerates an immediate ACK with the last sequence number it hasreceived in-order (that is, a duplicate ACK). This duplicate ACK isnot delayed. Since the sender does not know if this duplicate ACK isa result of a lost packet or an out-of-order delivery, it waits for a smallnumber of duplicate ACKs, assuming that if the packets are onlyreordered there will be only one or two duplicate ACKs before thereordered segment is received and processed and a new ACK isgenerated. If three or more duplicate ACKs are received in a row, itimplies there has been a packet loss. At that point, the TCP senderretransmits this segment without waiting for the retransmission timerto expire. This is known as fast retransmit ( see Figure 5 on page 31).

cwnd

RTT

Slow start: Exponentialgrowth of cwnd

SYM-001457

ssthresh

Congestion avoidance: Lineargrowth of cwnd



After fast retransmit has sent the supposedly missing segment, thecongestion avoidance algorithm is invoked instead of the slow start;this is called fast recovery. Receipt of a duplicate ACK implies that notonly is a packet lost, but that there is data still flowing between thetwo ends of TCP, as the receiver will only generate a duplicate ACKon receipt of another segment. Hence, fast recovery allows highthroughput under moderate congestion.

Figure 5 Fast retransmit

Network congestionA network link is said to be congested if contention for it causesqueues to build up and packets start getting dropped. The TCPprotocol detects these dropped packets and starts retransmittingthem, but using aggressive retransmissions to compensate for packetloss tends to keep systems in a state of network congestion even afterthe initial load has been reduced to a level which would not normallyhave induced network congestion. In this situation, demand for linkbandwidth (and eventually queue space), outstrips what is available.When congestion occurs, all the flows that detect it must reduce theirtransmission rate. If they do not do so, the network will remain in anunstable state with queues continuing to build up.

Send segments 21 - 26

Receive ACK for 21and 22

Received 3 duplicateACKs expecting 23Retransmit 23

Received ACK for 26expecting 27

23 lost in the network

Received segment 21 and 22send ACK for 21 and 22expecting 23

Received 24 still expecting 23 senda duplicate ACK

Received 25 still expecting 23 senda duplecate ACK

Received 26 still expecting 23 senda duplicate ACK

GEN-000299

TCP/IP 31

32


Internet Protocol security (IPsec)Internet Protocol security (IPsec) is a set of protocols developed bythe IETF to support secure exchange of packets in the IP layer. IPSecurity has been deployed widely to implement Virtual PrivateNetworks (VPNs).

IP security supports two encryption modes:

◆ Transport

◆ Tunnel

Transport mode encrypts only the payload of each packet, but leavesthe header untouched. The more secure Tunnel mode encrypts boththe header and the payload.

On the receiving side, an IP Security compliant device decrypts eachpacket. For IP security to work, the sending and receiving devicesmust share a public key. This is accomplished through a protocolknown as Internet Security Association and Key ManagementProtocol/Oakley (ISAKMP/Oakley), which allows the receiver toobtain a public key and authenticate the sender using digitalcertificates.

Tunneling and IPsecInternet Protocol security (IPsec) uses cryptographic security toensure private, secure communications over Internet Protocolnetworks. IPsec supports network-level data integrity, dataconfidentiality, data origin authentication and replay protection. Ithelps secure your SAN against network-based attacks from untrustedcomputers, attacks that can result in the denial-of-service ofapplications, services, or the network, data corruption, and data anduser credential theft.

By default, when creating an FCIP tunnel, IPsec is disabled.

FCIP tunneling with IPsec enabled will support maximumthroughput as follows:

◆ Unidirectional: approximately 104 MB/s

◆ Bidirectional: approximately 90 MB/s

Used to provide greater security in tunneling on an FR4-18i blade or aBrocade SilkWorm 7500 switch, the IPsec feature does not require youto configure separate security for each application that uses TCP/IP.When configuring for IPsec, however, you must ensure that there is



an FR4-18i blade or a Brocade SilkWorm 7500 switch in each end ofthe FCIP tunnel. IPsec works on FCIP tunnels with or without IPcompression (IPComp).

IPsec requires an IPsec license in addition to the FCIP license.

IPsec terminology

AES Advanced Encryption Standard. FIPS 197 endorses the Rijndaelencryption algorithm as the approved AES for use by US governmentorganizations and others to protect sensitive information. It replacesDES as the encryption standard.

AES-XCBC Cipher Block Chaining. A key-dependent one-way hash function(MAC) used with AES in conjunction with theCipher-Block-Chaining mode of operation, suitable for securingmessages of varying lengths, such as IP datagrams.

AH Authentication Header. Like ESP, AH provides data integrity, datasource authentication, and protection against replay attacks but doesnot provide confidentiality.

DES Data Encryption Standard is the older encryption algorithm that usesa 56-bit key to encrypt blocks of 64-bit plain text. Because of therelatively shorter key length, it is not a secured algorithm and nolonger approved for Federal use.

3DES Triple DES is a more secure variant of DES. It uses three different56-bit keys to encrypt blocks of 64-bit plain text. The algorithm isFIPS-approved for use by Federal agencies.

ESP Encapsulating Security Payload is the IPsec protocol that providesconfidentiality, data integrity, and data source authentication of IPpackets, as well as protection against replay attacks.

MD5 Message Digest 5, like SHA-1, is a popular one-way hash functionused for authentication and data integrity.

SHA Secure Hash Algorithm, like MD5, is a popular one-way hashfunction used for authentication and data integrity.

MAC Message Authentication Code is a key-dependent, one-way hashfunction used for generating and verifying authentication data.

TCP/IP 33

34


HMAC A stronger MAC because it is a keyed hash inside a keyed hash. SASecurity association is the collection of security parameters andauthenticated keys that are negotiated between IPsec peers.


2

This chapter provides the following information to consider whenworking with extended distance.

◆ Link speed ........................................................................................... 36◆ Data buffering and flow control ...................................................... 37◆ TCP/IP window................................................................................. 51◆ Active and passive devices ............................................................... 52◆ FC SONET/GbE/IP........................................................................... 59◆ Network stability and error recovery.............................................. 60

Distance ExtensionConsiderations

Distance Extension Considerations 35

36

Distance Extension Considerations

Link speedLink speed is an important aspect of distance extensionconfigurations. Within the SAN networks link speeds equate to theamount of maximum bandwidth reachable on an E_Port and/or anF_Port. There are a variety of link speeds that are supported in a SANnetwork. Table 2 compares and contrasts the STS, optical carrier, andFibre Channel link speed rates.

Table 2 STS-1s and optical carrier rates

STS Optical carrier Optical carrier rate Fibre Channel link speeds

STS-1 OC-1 51.84 Mb/s

STS-3 OC-3 155.52 Mb/s

STS-12 OC-12 622.08 Mb/s

STS-24 OC-24 1244.16 Mb/s 1.0625 Gb/s or 100 MB/s



STS-192 OC-192 9953.28 Mb/s 10.51875 Gb/s or 12.75 Gb/s



Data buffering and flow controlThe following information is discussed in this section:

◆ “Fibre Channel,” next

◆ “Maximum supported distance per Fibre Channel BB_Creditguidelines” on page 38

◆ “Buffer-to-buffer credit information” on page 41

Fibre ChannelFibre Channel uses the BB_Credit (buffer-to-buffer credit) mechanismfor hardware-based flow control. This means that a port has theability to pace the frame flow into its processing buffers. Thismechanism eliminates the need of switching hardware to discardframes due to high congestion. EMC testing has shown thismechanism to be extremely effective in its speed and robustness.

BB_Credit management occurs between any two Fibre Channel portsthat are connected. For example:

◆ One N_Port and one F_Port

◆ Two E_Ports

◆ Two N_Ports in a point-to-point topology

◆ In Arbitrated Loop different modes

The standard provides a frame-acknowledgement mechanism inwhich an R_RDY (Receiver Ready) primitive is sent from thereceiving port to the transmitting port for every available buffer onthe receiving side. The transmitting port maintains a count of freereceiver buffers, and will continue to send frames if the count isgreater than zero.

The algorithm is as follows:

1. The transmitter's count initializes to the BB_Credit valueestablished when the ports exchange parameters at login.

In an Arbitrated Loop environment the credits are established bythe receiving port sending in advance R_RDY primitives after thelogin to establish the credit.

2. The transmitting port decrements the count per transmittedframe.

Data buffering and flow control 37

38


3. The transmitting port will stop sending frames when the creditreaches zero.

4. When a link reset occurs, the credit values are reestablished tovalues negotiated upon login.

5. The transmitting port increments the count per R_RDY it receivesfrom the receiving port.

Figure 6 provides a view of the BB_Credit mechanism.

Figure 6 BB_Credit mechanism

As viewed from Port A’s perspective, when a link is established withPort B, BB_Credit information is exchanged. In this case, Port Bprovided a BB_Credit count of 5 to Port A. For Port A, this means itcan transmit up to five Fibre Channel frames without receiving anR_RDY.

Maximum supported distance per Fibre Channel BB_Credit guidelinesIn order to achieve maximum utilization of the Fibre Channel link itis highly advisable that both ports, connected on either side of thelong haul setup provided by the DWDM, be capable of highBB_Credit counts. Use the following formula to calculate theapproximate BB_Credit(s) required for the specific long haulapplication. To calculate for BB_Credits, use the following formulafor calculating the required BB_Credit count:

Port BFrame

R_RDY

5 BB_Credits

Frame

Frame-

-

-

Port A5 BB_Credits

Speed Formula

1 Gb/s BB_Credit = ROUNDUP [2 * one-way distance in km/4] * 1



8 Gb/s BB_Credit=ROUNDUP [2 * one-way distance in km/4] * 8

10 Gb/s BB_Credit=ROUNDUP [2 * one-way distance in km/4] * 12



The factor of 2 in the formulas accounts for the time it takes the lightto travel the entire roundtrip distance: frame from transmitter toreceiver and R_RDY back to transmitter.

Maximum allowable distance is based on optical powermeasurements of the site. These measurements should be approvedby DWDM and fiber services provider(s). The distance between anISL ports on a Fibre Channel switch to a DWDM port should beincluded as part of the total distance (d1+d2+d3). Refer to Figure 2 onpage 17.

The following BB_Credit charts will aid in providing estimates inregards to the amount of credits that should be present on the linkwhen factoring Fibre Channel link speeds and link distances betweenthe E_Ports.

Assuming the following is true:

◆ Light propagation in glass is 5 microseconds/km, or 59

seconds/m.◆ Frame size is 2148 bytes/frame.◆ Fibre Channel bit rate depends on the Fibre Channel speed.

Maximum distances assume 100% utilization of the ISL. If the ISL isnot fully utilized, greater distances can be achieved since moreBB_Credits become available. For example, for a 2 Gb/s switch portwith 120 BB_Credits and with an ISL that is only 50% utilized, themaximum distance is 240 km.


40


Since Brocade’s credit information is provided by ASIC types, reviewTable 3 to correlate between switch ASIC and model numbers.

Table 4 provides information on Cisco Fibre Channel ASIC.

Table 3 Brocade switch ASIC and model numbers

Vendor ASIC/Family EMC name Vendor name

Brocade Condor Connectrix ED-48000B Brocade 48000

Condor Connectrix DS-4900B Brocade 4900

Condor Connectrix DS-5000B Brocade 5000

Condor 2 Connectrix DS-5100B Brocade 5100

Condor 2 Connectrix ED-DCX-B DCX

Condor 2 Connectrix ED-DCX-4S-B DCX-4S

Goldeneye Connectrix DS-220B SilkWorm 220E

Goldeneye 2 Connectrix DS-300B Brocade 300

Goldeneye 2 Connectrix DS-5300B Brocade 5300

Table 4 Cisco Fibre Channel ASIC information

Cisco MDS family Hardware (Similar Fibre Channel ASICs are listed in the same cell)

Generation 1 • 16, 32-port 2 G FC• 9216,9216A, 9216i• MPS-14/2• SSM

Generation 2 • 12, 24, 48-port 4 G FC• MSM18/4• 9222i

Generation 2 4-port 10 G FC (DS-X9704)

Generation 2 MDS 9124x

Generation 2 MDS 9134

Generation 3 24, 48, 4/44-port 8G FC

Generation 3 DS 9148



Buffer-to-buffer credit informationDetermining sufficient amount of buffer-to-buffer credits is crucialwhen provisioning Fibre Channel environments prior to utilization.Miscalculating the amount of credits may lead to less than desiredperformance (such as, buffer-to-buffer credit, starvation, orbackpressure).

Credit starvation occurs when the amount of available credits reachesa zero state preventing all forms of Fibre Channel I/O-transmissionfrom occurring. Once this condition is reached a timeout value will betriggered causing the link to reset.

Refer to the next sections for basic credit table for switches andstorage arrays for Brocade B Series and Cisco.

Brocade credit chart With regards to flow control, Brocade switches support at least twoforms of flow control options on the E_Port. VC_RDY and R_RDYflow control are both available options for all Brocade switch types.

For VC_RDY flow control, Brocade switches require an “ExtendedFabric Mode” which will require to be activated through license code.Table 5, next, Table 6 on page 42, and Table 7 on page 43, are providedto display the supported distances for an E_Port when activatingthese modes in a Fibre Channel point-to-point switched fabricenvironment. These tables are broken down by ASIC type.

Table 5 Bloom and Bloom II ASICs (page 1 of 2)

Mode Description Bufferallocation @ 1Gb/s

Bufferallocation @ 2Gb/s

Distance @1 Gb/s

Distance @2 Gb/s

EarliestFabric OSrelease

ExtendedFabric licenserequired?

L0 Level 0 staticmode; default

5 5 10 Km 5 Km All No

LE Level E StaticMode;

13 19 n/a 10 Km v3.x, v4.x No

L0.5 Level 0.5 staticmode

19 34 25 Km 25 Km v3.1.0,v4.1.0, 5.x

Yes

L1 Level 1 staticmode

27 54 50 Km 50 Km All Yes

L2 Level 2 staticmode

60 65 / 108 forBloom II

100 Km 60 Km100 Km forBloom II

All Yes


42


LD Dynamic mode;auto detectsdistance uponinitialization

Auto Auto Auto(Max is 200Km)

Auto(Max is 200Km)

v3.1.0,v4.1.0,v4.4.0, 5.x –dependingon model

Yes

LS Static longdistance mode(user specified)

User specified User specified Userspecified

Userspecified

v5.1.0 Yes

Table 5 Bloom and Bloom II ASICs (page 2 of 2)

Mode Description Bufferallocation @ 1Gb/s

Bufferallocation @ 2Gb/s

Distance @1 Gb/s

Distance @2 Gb/s



Table 6 Condor ASIC

Mode Bufferallocation@ 1 Gb/s

Bufferallocation@ 2 Gb/s

BufferAllocation@ 4 Gb/s

Distance@ 1 Gb/s

Distance@ 2 Gb/s

Distance@ 4 Gb/s



L0 5 5 5 10 Km 5 Km 2 Km All No

LE 11 16 26 10 Km 10 Km 10 Km 3.x, 4.x No

L0.5 18 31 56 25 Km 25 Km 25 Km 3.1.0,4.1.0, 4.x,5.x

Yes

L1 31 56 106 50 Km 25 Km 50 Km All Yes

L2 56 106 206 100 Km 100 Km 100 Km All Yes

LD Auto Auto Auto Auto (max500 Km)

Auto (max250 Km)

Auto (max100 Km

3.1.0,4.1.0, 4.x,5.x –dependingon model

Yes

LS Userspecified

Userspecified

Userspecified

Userspecified(max 500Km)



5.1.0 Yes



d

?

Table 7 Condor 2 ASIC





Distance@ 1 Gb/s

Distance@ 2 Gb/s

Distance@ 4 Gb/s

Distance@ 8 Gb/s

EarliestFabricOSrelease

ExtendeFabriclicenserequired

L0 8 8 8 8 10 Km 5 Km 2 Km 1 Km 6.0x Yes

LE 11 16 26 46 10 Km 10 Km 10 Km 10 Km 6.0x Yes

LD Auto Auto Auto Auto Auto Auto Auto Auto 6.0x Yes

LS Userspecified

Userspecified

Userspecified

Userspecified

Userspecified(Refer toTable 10onpage 45)




6.0x Yes

Table 8 Goldeneye ASIC




Distance @1 Gb/s

Distance @2 Gb/s

Distance @4 Gb/s



L0 3 3 3 6 Km 3 Km 1.5 Km All No

LE 11 16 31 10 Km 10 Km 10 Km 3.x, 4.x No

L0.5 18 31 56 25 Km 25 Km 25 Km 5.1.0 Yes

L1 31 56 106 50 Km 50 Km 50 Km 5.1.0 Yes

L2 56 106 n/a 100 Km 100 Km n/a 5.1.0 Yes

LD Auto Auto Auto Auto Auto Auto 5.1.0 Yes

LS UserSpecified

UserSpecified

UserSpecified

UserSpecified(max 293Km)



5.1.0 Yes


44


d

?

Keep in mind that each Brocade switch family, ASIC, and mode type(such as, L1, L2, LD,and so on) will have unique VC_RDY amountsand characteristics depending on specific fabric configurations.Please refer to the EMC Support Matrix for specific configurationinformation.

Brocade also supports R_RDY flow control (through Portcfgislmode).Brocade R_RDY mode can be activated when connecting to distanceextension devices providing additional Buffer-to-Buffer Credits.

Brocade Extended FabricsBrocade’s Extended Fabrics is a licensed feature that extends StorageArea Networks (SANs) across longer distances for disaster recoveryand business continuance operations by enabling a modifiedbuffering scheme in order to support long distance fibre channelextensions, such as MAN/WAN optical transport devices. Thisbulletin is suitable for external dissemination.

Table 9 Goldeneye 2 ASIC





Distance@ 1 Gb/s

Distance@ 2 Gb/s

Distance@ 4 Gb/s

Distance@ 8 Gb/s

EarliestFabricOSrelease

ExtendeFabriclicenserequired

L0 8 8 8 8 10 Km 5 Km 2 Km 1 Km 6.1x Yes

LE 11 16 26 46 10 Km 10 Km 10 Km 10 Km 6.1x Yes

LD Auto Auto Auto Auto Auto Auto Auto Auto 6.1x Yes

LS Userspecified

Userspecified

Userspecified

Userspecified





6.1x Yes




Configurable distances for Extended Fabrics

Table 10 shows the maximum supported extended distances (inkilometers) that can be configured for one port on a specific switch orblade at different speeds.

Table 10 Configurable distances for Extended Fabrics (page 1 of 2)

Switch/blade model

Maximum distances (km) that can be configured assuming 2112 Byte Frame Size

1 Gb/s 2 Gb/s 4 Gb/s 8 Gb/s

300 972 486 243 121

4100/5000 500 250 100 N/A

4900 500 250 100 N/A

5100 3388 1694 847 423

5300 588 294 147 73

5410 1164 582 291 145.5

5424 972 486 243 121.5

5450 940 470 235 117.5

5480 972 486 243 121.5

7500 500 250 100 N/A

7600 500 250 100 N/A

7800 822 410 205 102

VA-40FC 3388 1694 847 423

Brocade Encryption Switch 2784 1392 696 348

FA4-18 500 250 100 N/A

FC4-16 500 250 100 N/A

FC4-16IP 500 250 100 N/A

FC4-32 500 250 100 N/A

FC4-48 500 250 100 N/A

FC8-16 2589 / 2781 1294 / 1390 647 / 695 323 / 347

FC8-32 2589 / 3277 1294 / 1638 647 / 819 323 /409


46


Note: The 10 Gb/s FC10-6 blade has two port groups of three ports each. Forextended ISLs, all buffers available to a group are used to support one port atup to 100 km.

Refer to the Brocade Fabric OS switch documentation, located atEMC Online Support, for further details.

Flow control

The Fibre Channel standards specifications (for example, FC-PH andFC-SW) define a method of flow control called R_RDY to manage andcontrol traffic as it flows across data links. Although the standardsdefine how R_RDY flow control should be used, it does not prohibitthe use of other vendor unique methods. By default, Brocadeswitches use Virtual Channel (VC) flow control over E_Portconnections within a fabric.

VC flow control provides the following advantages over R_RDY:

◆ The ability to differentiate fabric internal traffic from end-to-enddevice traffic.

In this case, switches generate fabric internal traffic thatcommunicate state information to each other, such as link stateinformation for routing, and device information for NameService. This type of traffic is given a higher priority so thatswitches can distribute the most up-to-date information acrossthe fabric even under heavy device traffic.

◆ The ability to differentiate data flows of end-to-end device trafficto avoid head-of-line blocking.

FC8-48 2461 / 3149 1230 / 1574 615 / 787 307 / 393

FC10-6 See the note at the end of this table for information about this blade.

FR4-18i 500 250 100 N/A

FS8-18 3208 1604 802 401

FX8-24 2125 1062 531 265

Table 10 Configurable distances for Extended Fabrics (page 2 of 2)

Switch/blade model

Maximum distances (km) that can be configured assuming 2112 Byte Frame Size

1 Gb/s 2 Gb/s 4 Gb/s 8 Gb/s



In the case of (2), when there are multiple I/Os multiplexed overa single ISL, by assigning different VCs to different I/Os andgiving them the same priority, each I/O can have a fair share ofthe bandwidth so that a large-size I/O will not consume thewhole bandwidth and starve a small-size I/O, thus balance theperformance of the different devices communicating across theISL. To identify a VC between two end-points of a link, VC_RDYis used.

Buffer allocation

When a switch port is configured for Extended Fabrics, additionalcredit is given to virtual channels that carry class 2 or 3 data traffic.This allows distances between switches to be extended over greaterdistances while maintaining maximum performance over ISLs. TheBrocade Extended Fabrics license allows ISLs to be connected at up to60 km for 2 Gb/s links and up to 100 km for 1 Gb/s links withoutdegradation of performance.

When Extended Fabrics is enabled on Fabric OS v3.x and v4.xswitches, two changes occur:

◆ Additional buffer credits are allocated to certain Virtual Channelson the long distance E_Port, and

◆ ARB(vc) is used as inter-frame gap instead of idles.

The additional buffers allow the E_Port “pipe” to be fully utilizedover long distances and the ARB(vc) ordered set is used to notify thereceiving switch as to which VC queue the next incoming frameshould be placed on. There is a different ARB(vc) primitive for eachof the eight possible virtual channels.

MAN/WAN optical transport devices

Vendors of optical transport devices may not be aware of E_Portfunctionality on Brocade switches, which may cause interoperabilityissues under certain configurations. Although there are certainworkarounds, any vendor wishing to understand this functionalitycan contact Brocade. All devices tested in the Fabric Aware programare verified to operate under ideal switch configurations.

If the extension devices between the Brocade switches transparentlypropagate all traffic as is, these ARB(vc)s will not cause anyproblems. However, recently some transport devices have beenintroduced that do more than simply pass through the Fibre Channelframes. In some cases, and in some modes, these devices have been


48


shown to have problems processing the ARB(vc) frames resulting indisruption of traffic over the long distance connection.

In these cases there are at least three solutions to this issue:

◆ If the extension device is capable of being configured in a modewhich transparently passes Fibre Channel frames, there should beno disruption of traffic due to the ARB(vc) frames.

◆ If the 'fabric.ops.mode.longDistance' bit is set to '1' on all Brocadeswitches in the fabric, the ARB(vc) primitives will not be sent. Thedefault setting of this parameter is '0'. In order to set this bit theswitches will need to be disabled and the bit set using either theconfigure command in a telnet or serial console window orthrough a GUI management interface. In the Web Tools GUI thisbit can be set by selecting the Admin button from the main screenand then clicking the enable button under Extended FabricsMode on the Extended Fabric tab. Despite the label of this button,it does not actually enable/disable Extended Fabrics and, in fact,the only effect this button has to set or unset thefabric.ops.mode.longDistance bit.

Note: This parameter will need to be set on all switches in the fabric, notjust the switch that has the long distance connection. Also note that thisparameter affects all E_Ports on the switch (long distance or otherwise)by changing the amount of buffer credits allocated to the port.

◆ Since optical transport devices are designed to provideconnectivity over long distance, many vendors provide their ownmethod of managing flow control over long distance connections.This can allow FC performance to be maintained at up tohundreds or even thousands of kilometers without degradation.If the vendor supports this type of configuration, Brocadeswitches can be configured to use standards based R_RDY flowcontrol using the portCfgISLMode CLI command. ExtendedFabrics would not be necessary.

Note: The latest updated firmware levels and hardware levels support thecombination of both Extended Fabric Modes with R_RDY modeimplementation. This allows the customer to bypass the old challenges ofconfiguring the Brocade Fabric environment to its pure native mode.

Refer to the EMC Brocade switch documentation for further details.



Brocade M Seriescredit chart

Brocade M Series supports only R_RDY flow control. Each Brocade MSeries Family type switch will have unique credit amounts. Refer toTable 11 for details of the Brocade M Series credit chart.

Cisco MDS creditchart

Cisco MDS switches only utilizes R_RDY flow control. Table 12displays the number of BB-credits are available per E_Port.

Table 11 Brocade M Series credit chart

Switch type(EMC/Brocade M Series)

Module / Optic Link speed Number of credits Notes

ED-140M / ED-6140 Multi-mode,single-mode

1 Gb / 2 Gb 60

N/A / ES-4300 Multi-mode,single-mode

1 Gb / 2 Gb 12 / 7 12 on the first 4 and 7 on therest… Credit increasesapplies to specified quadareas.

ES-4500 Multi-mode,single-mode

1 Gb / 2 Gb 12 / 7 12 on the first 4 and 7 on therest… Credit increasesapplies to specified quadareas.

ED-10000M / Intepid 10000 Multi-mode,single-mode

1 Gb/2 Gb/10Gb 1373


1 Gb / 2 Gb/ 4 Gb


1 Gb / 2 Gb / 4 Gb

Table 12 Cisco MDS credit chart

Switch Type Blade/Optic Support Link Speed Number of Credits Notes

9509 Multi-mode, single-mode, CWDM 1 Gb / 2 Gb 255



9216A Multi-mode, single-mode, CWDM 1 Gb / 2 Gb 255

9216i Multi-mode, single-mode, CWDM 1 Gb / 2 Gb 255

9120 Multi-mode, single-mode, CWDM 1 Gb / 2 Gb 255 Based on the first quad



50


Symmetrix FibreAdapter credit chart

EMC Symmetrix boards uses R_RDY flow control. Table 13 displaysthe number of BB-credits available per Fibre Channel Adapter F_Port.

Table 13 Symmetrix Fibre Adapter credit chart

Symmetrix Family Board Type / Optic Link Speed Number of Credits

Symmetrix 5.0 Fibre Adapter / multi-mode 1 Gb / 2 Gb 7





TCP/IP windowA TCP window is the amount of data a sender can send withoutwaiting for an ACK from the receiver. The TCP window is a flowcontrol mechanism and ensures that no congestion occurs in thenetwork. For example, if a pair of hosts are talking over a TCPconnection that has a TCP window size of 64 KB (kilobytes), thesender can only send 64 KB of data and then it must stop and wait foran acknowledgment from the receiver that some or all of the data hasbeen received. If the receiver acknowledges that all the data has beenreceived then the sender is free to send another 64 KB. If the sendergets back an acknowledgment from the receiver that it received thefirst 32 KB (which could happen if the second 32 KB was still intransit or it could happen if the second 32 KB got lost), then thesender could only send another 32 KB since it cannot have more than64 KB of unacknowledged data outstanding (the second 32 KB of dataplus the third).

The primary reason for the window is congestion control. The wholenetwork connection, which consists of the hosts at both ends, therouters in between, and the actual connections themselves, will havea bottleneck somewhere that can only handle so much data so fast.The TCP window throttles the transmission speed down to a levelwhere congestion and data loss do not occur. The factors affecting thewindow size are as follows:

◆ Receiver’s advertised windowFor more information, refer to “Receiver’s advertised window”on page 27.

◆ Sender’s congestion windowFor more information, refer to “Sender’s congestion window” onpage 27.

◆ Usable windowFor more information, refer to “Usable window” on page 27.

◆ Window scalingFor more information, refer to “Window scaling” on page 27.

TCP/IP window 51

52


Active and passive devicesThis section contains the following information:

◆ “Buffer-to-buffer local termination,” next

◆ “SRDF with SiRT” on page 54

◆ “Fast write/ write acceleration” on page 56

◆ “SiRT with distance vendor write acceleration” on page 57

◆ “Link initialization” on page 58

Buffer-to-buffer local terminationIn Fibre Channel, BB_Credits are a method of maintaining the flowcontrol of transmitting Fibre Channel frames. BB_Credits helpmaintain a balanced flow of I/O transmissions while avoidingunderutilization or oversubscription of a Fibre Channel link.

Figure 7 on page 53 shows what the buffering flow control wouldnormally follow without the local termination. This places the burdenon the end nodes to maintain and track the BB_Credit flow control onthe Fibre Channel link. The flow control distance will be determinedby the amount of credits and the link speed that is supported by theend nodes. The end nodes can be an E_Port or F_Port.

BB_Credits are provided by the Fibre Channel switches. The distanceextension device is transparent and does not participate in BB_Creditflow control. Link speed, latency, and the amount of available creditswill determine the performance characteristics of theseconfigurations.



Figure 7 Flow control managed by Fibre Channel switch (without buffering fromdistance extension devices)

Determining sufficient amount of BB_Credits is crucial whenprovisioning Fibre Channel environments prior to utilization.Miscalculating the amount of credits may lead to performancedegradation due to credit starvation.

Note: EMC recommends adding 20% margin to calculated BB_Credit valuesto account for spikes in traffic.

Credit starvation occurs when the number of available credits reacheszero preventing all forms of Fibre Channel transmissions fromoccurring. Once this condition is reached a timeout value will betriggered causing the link to re-initialize. To avoid this condition,sufficient BB_credits must be available to meet the latency andperformance requirements for the particular SRDF deployment.

The standard Fibre Channel flow control and BB_Credit mechanismis adequate for most short-haul deployments. With longer distancedeployments however, the Fibre Channel flow control model is not aseffective. Additional buffering and WAN-optimized flow control areoften needed.

Figure 8 on page 54 shows a configuration where the distanceextension devices are providing additional buffering and flow controlmechanisms for the purpose of increasing distances betweenlocations. To accomplish this, the Fibre Channel end nodes are

SRDF RF SRDF RF

DISTANCE

NODE

DISTANCE

NODE

Switch

Local flowcontrol

Switch

Local flowcontrol

Local RemoteFlow control managed fromFibre Channel end nodes

Local flow control

Active and passive devices 53

54


provided with immediate R_RDY responses with every "sent"FC-frame. This occurs within the local flow control segments. Thedistance extension nodes, in turn, implement their own buffering andWAN-optimized flow control.

Figure 8 Flow control (with buffering from distance extension devices)

Refer to the distance extension vendor documentation for detailedinformation on each vendor’s buffering and flow controlimplementations.

SRDF with SiRTSingle RoundTrip (SiRT) for Fibre Channel SRDF directors (RFs) wasintroduced in EMC Enginuity™ 5772 for SRDF/S mode only. It isdynamically enabled for SRDF/S links > 12 Km for block sizes up to32K in Enginuity 5773 code. SiRT is compatible with Fast Write/WriteAcceleration switches and extenders, as it will measure link latencyand disable automatically if connected to these devices. As a bestpractice, it is recommended that either the EMC SiRT feature or thethird-party fast write feature should be used. Both should not beenabled simultaneously.

The Fibre Channel SiRT feature for the Fibre Channel director can beset to Off or Automatic. When set to Automatic, this feature will onlyaccelerate write I/Os using criteria based on latency and I/O size.

SRDF RF SRDF RF

DISTANCE

NODE

DISTANCE

NODELocal flow

controlLocal flow

controlDistance flow

control

Switch

Local flowcontrol

Local flowcontrol

Switch

Local flowcontrol

Local flowcontrol

Local Remote



Note: EMC recommends contacting your EMC Customer ServiceRepresentative to verify that the setting is enabled if required in yourenvironment.

Figure 9 shows the normal write process without the SiRT feature.

Figure 9 Normal write command process

The intended purpose of this feature is to maintain SRDF/Ssynchronicity while improving performance by localizing thetransfer-ready response to the local RF port, thereby reducing anunnecessary acknowledgement response (trip) over the dark fiberdistance (step 2 in Figure 9). Immediate benefits are apparent uponactivation in transparent SRDF synchronous distance extensionenvironments.


56


If applicable, multiple SRDF synchronous links can maximize theirI/O performance over the network (transparent WDM environment).

In the example shown in Figure 10, RF1 (R1 F_Port) and RF2 (R2F_Port) are managing the SiRT flow control.

Figure 10 SRDF SiRT

Legend:

Fast write/ write accelerationEMC Connectrix and other third-party products offer singleroundtrip for Fibre Channel capabilities (fast write/writeacceleration) that can also increase SRDF throughput for direct-attachor Fibre Channel switched fabric configurations over extendeddistances. It is transparent to SRDF FC links and is used for all SRDFmodes to decrease response time (SRDF/S) or improve performanceover long distance links (mostly for adaptive copy and SRDF/AR,but also for some SRDF/A configurations).

Red RF-ports with SiRT activated.

Blue A step-by-step of a single write command with SiRT enabled.



Figure 11 shows a write command with fast write features.

Figure 11 Write command with SiRT

For Connectrix or third-party products, refer to the EMC SupportMatrix available at http://elabnavigator.EMC.com to verify which ofthese products are supported for SRDF configurations.

IMPORTANT

Not all products offering this feature are supported with SRDF dueto unique write commands utilized by SRDF.

SiRT with distance vendor write accelerationWith this in mind SiRT usage, in combination with the distanceextension device-offered write acceleration mode, must be addressed.Essentially for environments where the distance extension device isalready servicing write commands on an E_Port level, it isrecommended to disable SiRT. Refer to Figure 12 on page 58.





58


Figure 12 All F_Ports will benefit

Legend:

In Figure 12, by enabling the write acceleration feature on thedistance extension device, potentially all F_Ports (RF ports, FA ports,tape, etc.) issuing writes traversing across the E_Port attached to thedistance extension client port can also take advantage of thethroughput benefits from the activated write acceleration feature.

Link initializationFor link initialization of a Fibre Channel port, Fibre Channelspecifications state that the maximum tolerable response time for aresponse is 100 milliseconds roundtrip time. This timeframecoincides with the limited timeframe of the Receiver-TransmitterTimeout Value (R_T_TOV), which is how long an FC port listens for alink response to a link service before an error is detected.

Red RF ports benefiting from distance extension device, write acceleration.

Blue Scope.



FC SONET/GbE/IPDistance devices or circuit packs/blades performing protocolconversions from Fibre Channel to and from an alternate backboneprotocol are required to maintain the lowest link initializationtimeout value. In contrast to Fibre Channel’s R_T_TOV, the SONET,GbE, and IP implementations can extend well beyond the 100millisecond roundtrip time. For these environments, the distanceextension devices should offer a setting enabling “local initialization”to occur between the “local” Fibre Channel port and the “local”distance extension client port rather than initializing the “local” FibreChannel port across the actual physical distance to its “remote” FibreChannel port (Figure 13).

Figure 13 Link initialization (More than 100 ms R_T_TOV)

FC SONET/GbE/IP 59

60


Network stability and error recoveryThis section explains how the following handle error recovery.

CWDM CDWM devices do not participate in error recovery at any level. Thedevice to handle the recovery depends on the level the erroroccurred. In case of link events, it will be handled by the FibreChannel ports (switch or storage) across the CWDM link. In case ofSCSI level errors, the application (SRDF or MirrorView™) will handlethe error recovery. Link bit errors will cause SCSI level errors.

DWDM Error recovery is based on the attach client circuit pack that the FibreChannel ports attached to. If the Fibre Channel ports attached to aBuffer-to-Buffer credit spoofing circuit then link events will behandled locally with the attached Fibre Channel port. SCSI levelerrors will be handled by the application. Link bit errors will causeSCSI level errors.

SONET Error recovery is based on the attach client circuit pack that the FibreChannel ports attached to. If the Fibre Channel ports attached to aBuffer-to-Buffer credit spoofing circuit, link events will be handledlocally with the attached Fibre Channel port. SCSI level errors will behandled by the application. Link bit errors will cause SCSI levelerrors.

GE Error recovery is based on the attach client circuit pack that the FibreChannel ports attached to. If the Fibre Channel ports attached to aBuffer-to-Buffer credit spoofing circuit then link events will behandled locally with the attached Fibre Channel port. SCSI levelerrors will be handled by the application. Link bit errors will causeSCSI level errors.

TCP/IP Error recovery will be handled by the TCP/IP distance device(review “TCP/IP” on page 24). If the errors persist and do notprovide sufficient quality for the link to recover, the errors will bepropagated to the attached Fibre Channel ports.


3

This chapter contains the following information on IP-based distanceextension solutions.

◆ Network design best practices ......................................................... 62◆ EMC-Brocade distance extension solutions ................................... 64◆ Configuring IPsec............................................................................... 76◆ Fast Write and tape pipelining ......................................................... 78◆ EMC-Cisco MDS distance extension solution................................ 82◆ EMC-QLogic distance extension solution ...................................... 84◆ Summary ............................................................................................. 88

IP-Based DistanceExtension Solutions

IP-Based Distance Extension Solutions 61

62

IP-Based Distance Extension Solutions

Network design best practicesThe network should be dedicated solely to the IP technology beingused and other traffic should not be carried over it.

The network must be well-engineered with no packet loss orduplication. This would lead to undesirable retransmission. Whileplanning the network, care must be taken to ensure that the utilizedthroughput will never exceed the available bandwidth.Oversubscribing available bandwidth will lead to networkcongestion, which causes dropped packets and leads to TCP slowstart. Network congestion must be considered between switches aswell as between the switch and the end device.

The MTU must be configured based on the maximum available MTUsupported by each component on the network.

Network conditions impact on effective throughputTable 14 on page 63 demonstrates the impact of network conditionson TCP/IP effective throughput (data provided to the distanceextension device by the Fibre Channel devices—the amount of dataon the link will be greater due to TCP retransmission).

The distance between the sites has a significant impact on thedistance system effective throughput. However, it is a fixed value.Packet loss, on the other hand is not a fixed value and can berelatively high due to TCP recovery mechanism and therefore has agreater impact. When designing the distance extension solution,network conditions must be taken into account to ensure that theeffective throughput is sufficient for the solution needs.Over-utilization of the effective throughput will result in errors at theapplication levels.

Review “TCP/IP” on page 24 for how to maximize effectivethroughput.



Table 14 Network impact on effective throughput example

Compression Network conditions Effective throughput

enabled 100 ms RTT with 1% packet loss 1 MB/s

100 ms RTT with no packet loss 6 MB/s

50 ms RTT with 1% packet loss 3 MB/s


200 ms RTT with 1% packet loss 800 KB/s

200 ms RTT with no packet loss 3.7 MB/s

disabled 100 ms RTT with 1% packet loss 360 KB/s






Network design best practices 63

64


EMC-Brocade distance extension solutionsThis section discusses:

◆ “Brocade 7500” on page 65

◆ “Brocade 7800” on page 67

The following Brocade terminology is used throughout this section.

Backbone Fabric Routers provide a backbone (BB) Fabric tointerconnect routers for more scalable andflexible routed SANs. Each router may havemany edge fabric connections, but only one BBfabric. Routers connect to the BB fabric throughE_Ports, and all N_Port and NL_ Portconnections on a router are part of the BB fabric.With 4 Gb routers, a number of hosts andstorage devices may be connected to the BBfabric.

Edge Fabric Fibre Channel fabric connected to a routerthrough an EX_Port (IFL). This is largely thesame as any standard Fibre Channel fabric. Thisis, for the most part, where the hosts and storageare attached.

E_Port A port on an FC switch or router, whichconnects to another switch or router, forming anISL. If the devices previously formed separatefabrics, these fabrics merge, putting all fabricservices into one distributed image.

EX_Port FC Routers use EX_Ports instead of E_Ports onrouted interfaces. To connect a router to aswitch, you connect its EX_Port to anotherswitch's E_Port using an appropriate cable.Routers still use E_ or VE_Ports to form abackbone fabric.

IFL The connection between an E_Port and anEX_Port is an "Inter-Fabric Link".

ISL The connection between two E_Ports is anInter-Switch Link.



Brocade 7500FCIP tunneling enables you to connect one central office to differentbranch offices using different VE_Ports or VEX_Ports, therebyenabling branch offices to connect with each other without having tomerge data center and branch office fabrics.

Fibre Channel frame encapsulation on one VE_Port and thereconstruction of Fibre Channel frames on the other VE_Port istransparent to the initiator and target, but the administration ofVE_Ports is different from other Fibre Channel port types.

Fabric OS supports FCIP ISLs between two Brocade switches(Brocade 7500 or 48000 with a FR4-18i blade) or routers.

FCIP also supports:

◆ Configuration and management of GbE ports

◆ Compression and decompression of Fibre Channel framesmoving through FCIP tunnels

◆ Statistics gathering on several layers

◆ Traffic shaping that adheres to a rate limit on a per tunnel basis

◆ FCIP tunnel/GbE port event notification

◆ Fibre Channel Router capabilities over VE_Ports

LSAN LSAN Logical SANs are zones which spanfabrics. They will traverse at least one EX_Portor VEX_Port. LSANs are how connectivity isconfigured across routers.

VE_Port An FCIP port on an FC switch will create a"Virtual E_Port". This is physically anIP/Ethernet interface, but each FCIP tunnel"looks" like an FC E_Port to the rest of the fabric.

VEX_Port In addition to supporting virtual E_Ports,Brocade platforms allow the FCIP and FCRouter features to be combined, creating aVirtual EX_Port. FC Router features to becombined, creating a Virtual EX_Port.

EMC-Brocade distance extension solutions 65

66


FCIP tunneling introduces the following concepts:

◆ Tunnel

An FCIP tunnel carries Fibre Channel traffic (frames) over IPnetworks such that the Fibre Channel fabric and all Fibre Channeldevices in the fabric are unaware of the IP network’s presence.Fibre Channel frames "tunnel" through IP networks by dividingframes, encapsulating the result in IP packets on entering thetunnel, and then reconstructing them as they leave the tunnel.

◆ VE_Port

Special types of ports, called VE_Ports (virtual E_Port), functionsomewhat like an E_Port. The link between an VE_Port and aVE_Port is called an interswitch link (ISL). You can configuremultiple ISLs from a Brocade 7500 or 48000 with an FR4-18i blade.After you configure the VE_Ports on either two Brocade 7500s or48000s with the FR4-18i blade, an FCIP connection is establishedbetween them. VE_Ports do not prevent fabric merging. Using aVEX_Port is one way to prevent fabrics from merging.

◆ VEX_Port

A VEX_Port enables routing functionality through an FCIPtunnel. VEX_Ports are virtual FC_Ports that are exposed by FCIPtunnels connecting to either the Brocade 7500 or 48000 with aFR4-18i blade; they run interfabric links (IFLs) as EX_Ports toenable Fibre Channel router capability. You can have up to eightVEX_Ports per GbE on the Brocade 48000 with a FR4-18i blade.

◆ GbE

Gigabit Ethernet ports are available on the Brocade 7500 and48000 with a FR4-18i blade. These ports support FCIP with linkspeeds up to 1 Gb/s. Each GbE port (ge0, ge1) supports up toeight FCIP tunnels.

Note: You cannot create more than one FCIP tunnel on a given pair of IPaddress interfaces (local and remote). However, you can create multiple FCIPtunnels on an IP interface so that, minimally, either the local or remote IPinterface will be unique and not have any other FCIP tunnel on it. When theGbE port has a valid SFP and is physically connected to any other GbE port,the status output from the switchShow command is online.



Supportedenvironment

Figure 14 shows an example of a Brocade 7500 configuration.

Figure 14 Brocade 7500 configuration example

References For more information, refer to www.brocade.com. For configurationhelp, refer to the Brocade FOS 5.1 Administration Guide.

Brocade 7800The FX8-24/7800 supports all features and functions associated withFCIP on the FR4-18i/7500 platforms. New FCIP functionalityassociated with the FX8-24 blade are:

◆ 10 x 1 GbE ports available

◆ 2 x 10 GbE ports available (note that both 10 GbE ports and 1 GbEports cannot be enabled simultaneously)

◆ 12 x 8 Gb FC ports

IP WANnetwork

Data centerFC SAN

OfficeFC SAN

OfficeFC SAN

OfficeFC SAN

FibreChannelinitiator

FibreChannelinitiator

SilkWorm 7500 SilkWorm 7500

VE_Port

VE_Port VE_Port

VE_Port

SilkWorm 48000with FR4-18i Blade

SilkWorm 48000with FR4-18i Blade

FibreChannel

target

FibreChanneltarget

GEN-000296


http://www.brocade.com

68


◆ FCIP Trunking

◆ IPV6

◆ IPV4

◆ DSCP marking

◆ VEX

New FCIP features supported on the 7800 platform are:

◆ 6 x 1 GbE ports

◆ 16 x 8 Gb FC ports

◆ FCIP Trunking

◆ IPV6

◆ IPV4

◆ DSCP marking

Note: Unlike the FR4-18i/7500, FCIP tunnels in FX8-24/7800 are no longerassociated with a specific GbE port.

FCIP TrunkingFCIP Trunking is a new feature which has been introduced with the7800 and FX 8-24 FOS Release v6.3.x. (Refer to the EMC SupportMatrix for the supported FOS v6.3.x versions.)

FCIP Trunking is a method for managing the use of WAN bandwidthand for providing redundant paths over the WAN that can protectagainst transmission due to WAN failure. Trunking is enabled bycreating logical circuits within an FCIP tunnel. A tunnel may havemultiple circuits. Each circuit is a connection between a pair of IPaddresses that are associated with source and destination end-pointsof an FCIP tunnel.

Figure 15 on page 69 shows the relationship of trunks and circuits toVE_Ports, FCIP tunnels, and the physical GbE interfaces. FC trafficenters and exits an FCIP tunnel on a VE_Port. Applications on the FCside have no awareness of the existence of the FCIP tunnel. FCIPTrunking routes the FC traffic over FCIP circuits. FCIP circuits routetraffic over a WAN using any of the GbE interfaces. An FCIP circuit isa logical connection between two peer switches or blades, so the sameconstruct exists in each peer switch or blade.





Figure 15 Basic overview of Trunking components

TCP Trunking provides the following features:

◆ Load balancing across multiple connections

◆ Failover to remaining connections if a link fails

◆ Lossless Failover

◆ Lossless Link Loss (LLL)—Data in-flight is not lost when a linkgoes down

◆ Data in-flight will be resen— Same as with TCP

◆ In-Order-Delivery (IOD) after a failover: Data in-flight will bedelivered in the correct order— Same as TCP

◆ Works with both FICON and FC: Supports FastWrite, OSTP andFICON Emulation over multiple links

CircuitEach circuit is a connection between a pair of IP addresses that areassociated with source and destination end-points of an FCIP tunnel.An Ethernet interface can have one or more FCIP tunnels and circuits.Circuits in a tunnel can use the same or different Ethernet interfaces.

MetricA circuit has a “cost metric”. Lower metric circuits are preferred overhigher metric circuits. When there are circuits with different metrics,all traffic goes through the circuits with lowest metric and no trafficgoes through circuits with higher metric. If all circuits with the lowest


70


metric fail, circuits with higher metric are used. If all circuits have thesame metric, traffic flows on all circuits. The remote end of a tunnelreorders frames to maintain in-order delivery. Load-leveling isautomatically done across circuits with the lowest metric.

If a circuit fails, FCIP Trunking tries first to retransmit any pendingsend traffic over another lowest metric circuit. If no lowest metriccircuits are available, then the pending send traffic is retransmittedover any available circuits with the higher metric.

TunnelFCIP tunnels are used to pass Fibre channel I/O through an IPnetwork. FCIP tunnels are built on a physical connection betweentwo peer switches or blades. An FCIP tunnel forms a single logicaltunnel from the circuits. A tunnel scales bandwidth with each addedcircuit, providing lossless recovery during path failures and ensuringin-order frame delivery.

FCIP Tunnels can be formed by using the VE_Ports or VEX_Ports.VE_Ports and VEX_Ports are virtual E_Ports. VE_Ports are used tocreate interswitch links (ISLs). If VE_Ports are used on both ends ofan FCIP tunnel, the fabrics connected by the tunnel are merged.

VEX_Ports enable interfabric links (IFLs). If a VEX_Port is on one endof an FCIP tunnel, the fabrics connected by the tunnel are not merged.The other end of the tunnel must be defined as a VE_Port. VEX_Portsare not used in pairs.

Adaptive Rate LimitingAdaptive Rate Limiting (ARL) is performed on FCIP tunnelconnections to change the rate in which the FCIP tunnel transmitsdata through the TCP connections. ARL uses information from theTCP connections to determine and adjust the rate limit for the FCIPtunnel dynamically. This allows FCIP connections to utilize themaximum available bandwidth while providing a minimumbandwidth guarantee.

ARL applies a minimum and maximum traffic rate and allows thetraffic demand and WAN connection quality to dynamicallydetermine the rate. As traffic increases, the rate grows towards themaximum rate. If traffic subsides, the rate reduces towards theminimum. If traffic is flowing error-free over the WAN, the rategrows towards the maximum rate. If TCP reports an increase inretransmissions, the rate reduces towards the minimum.



QoS prioritiesEach FCIP circuit is assigned four TCP connections for managing FCQuality of Service (QoS) priorities over an FCIP tunnel. The prioritiesare as follows:

◆ F class – F class is the highest priority, and is assigned bandwidthas needed, at the expense of lower priorities, if necessary.

◆ QoS high – The QoS high priority gets at least 50% of thebandwidth.

◆ QoS medium – The QoS medium priority gets at least 30% of thebandwidth.

◆ QoS low – The QoS low priority gets at least 20% of thebandwidth.

Open Systems Tape PipeliningOpen Systems Tape Pipelining (OSTP) can be used to enhance opensystems SCSI tape write I/O performance. When the FCIP link is theslowest part of the network, OSTP can provide accelerated speeds forread and write I/O over FCIP tunnels. To use OSTP, you need toenable FCIP Fastwrite and Tape Pipelining.

◆ FCIP Fastwrite accelerates the SCSI write I/Os over FCIP.

◆ Tape Pipelining accelerates SCSI read and write I/Os tosequential devices (such as tape drives) over FCIP, which reducesthe number of round-trip times needed to complete the I/O overthe IP network and speeds up the process. Each GbE portsupports up to 2048 simultaneous accelerated exchanges.

Both sides of an FCIP tunnel must have matching configurations forthese features to work. FCIP

Fastwrite and Tape Pipelining are enabled by turning them on duringthe tunnel configuration process. They are enabled on a per-FCIPtunnel basis.

FCIP Fastwrite and Tape Pipelining configurationsTo help understand the supported configurations, consider theconfigurations shown in the following two figures. In both cases,there are no multiple equal-cost paths. In Figure 16 on page 72, thereis a single tunnel with Fastwrite and Tape Pipelining enabled.


72


Figure 16 Single tunnel, Fastwrite and Tape Pipelining enabled

In Figure 17, there are multiple tunnels, but none of them create amultiple equal-cost path.

Figure 17 Multiple tunnels to multiple ports, Fastwrite, and Tape Pipeliningenabled on a per-tunnel/per-port basis



FCIP tunnels and VE_Ports on the 7800 switch

Note: A Brocade 7800 16/6 switch can support eight VE_Ports and Brocade7800 4/2 can support two FCIP tunnels, and therefore eight FCIP tunnels.

Each FCIP tunnel is associated with a VE port. VE_Ports arenumbered from 16 to 23. On the 7800 switch and on FX8-24 blades,VE_Ports do not have to be associated with a particular GbE port.

The full bandwidth provided by the six GbE ports is available to alltunnels. FCIP trunking provides load balancing. Failover capabilitiesare provided through the use of virtual FCIP circuits. Up to four FCIPcircuits may be defined per tunnel. A single circuit cannot exceed 1Gb/s capacity.

Note: The Open Systems Tape Pipelining is not supported with Brocade 78004/2.

FCIP tunnels and VE_Ports on the FX8-24 bladeAn FX8-24 blade can support 20 VE_Ports, and therefore 20 FCIPtunnels. Each FCIP tunnel is associated with a specific VE_Port. OnFX8-24 blades, and on the 7800 switch, VE_Ports do not have to beassociated with a particular GbE port.

VE_Ports 12 through 21 may use GbE ports ge0 through ge9, or theymay use XGE port 1. VE_Ports 22 through 31 can only be used byXGE port 0. The total bandwidth cannot exceed 20 Gb/s.

There are twelve FC ports, numbered 0 through 11. The FC ports canoperate at 1, 2, 4, or 8 Gb/s. There are ten GbE ports, number 0through 9. Ports XGE0 and XGE1 may be configured as 10 GbE ports.The FX8-24 blade provides a maximum of 20 Gb/s of bandwidth forEthernet connections, and can operate in one of three differentmodes:

◆ 1 Gb/s mode—You can use all the GbE ports (0 through 9).

◆ 10 Gb/s mode—You can use the XGE0 and XGE1 ports.

◆ Dual mode—You can use GbE ports 0 through 9, and port XGE0.

Note: VEX_Ports are not supported on the FX8-24 blade.

The full bandwidth provided by the ten GbE ports or two 10 GbEports is available to all tunnels.


74


FCIP trunking provides load balancing. Failover capabilities areprovided through the use of virtual FCIP circuits. FCIP tunnels usingGbE ports can have up to four FCIP circuits spread across four GbEports. FCIP tunnels using 10 GbE ports can have up to ten FCIPcircuits over one 10 GbE port. A single circuit cannot exceed 1 Gb/scapacity. To create an FCIP tunnel with a capacity of 10 Gb/s over a10GbE port, you must create an FCIP tunnel with ten FCIP circuits.

Virtual fabrics and the FX8-24 bladeThe FX8-24 FC ports can be part of any logical switch. The GE_Portsand VE_Ports on the FX8-24 blade can be part of any logical switch.GE_Ports and VE_Ports ports may be moved between any two logicalswitches. Ports do not need to be offline when they are moved.GE_Ports and VE_Ports are independent of each other, so both mustbe moved in independent steps, and you must clear the configurationon VE_Ports and GE_Ports before moving them between logicalswitches.

Note: This differs from the FR4-18i blade, where only GE_Ports need to bemoved and all the VE_Ports created on that GE_Port are automaticallymoved. You do not need to delete VE_Port and GbE port configurationinformation.

The total number of VE_Ports in all the logical switches is equal to themaximum number of VE_Ports on an FX8-24 blade (which is 20)multiplied by the maximum number of FX8-24 blades allowed on aDCX or DCX-4S chassis (which is 2). VEX_Ports are not supported onthe FX8-24 blade.

Table 15 compares the Brocade FX 8-24, Brocade 7800 16/6, andBrocade 7800 4/2.

Table 15 Product comparison (page 1 of 2)

Standard features Brocade FX8-24 Brocade 7800 16/6 Brocade 7800 4/2

Supported storage Open systems andmainframe

Open systems andmainframe

Open systems only

8 Gb/s Fibre Channel/FICON Ports 12 16 4

1 GbE ports 10 6 2

10 GbE ports (2) Optional N/A N/A

Maximum FCIP Bandwidth 20 Gb/s 6 Gb/s 2 Gb/s



Maximum number of FCIP tunnels 20 8 2

Maximum bandwidth per FCIP tunnel Up to 10 Gb/s withOptional FCIP Trunking

Up to 4 Gb/s withOptional FCIPTrunking

Up to 2 Gb/s with Optional FCIPTrunking

Integrated Routing Optional Optional Optional

High-performance compression Included Included Included

FCIP Fast Write Included Included Included

Open Systems Tape Pipelining Included Included Not Supported

Storage-Optimized TCP Included Included Included

Brocade DCFM FCIP management Included Included Included

FCIP Quality of Service Brocade DCX(Included) BrocadeDCX-4S (Optional)

Optional Optional

FCIP Trunking Optional Optional Optional

Adaptive Rate Limiting Optional Optional Optional

Advanced Accelerator for FICON Optional Optional Not Supported

FICON CUP Optional Optional Not Supported

Table 15 Product comparison (page 2 of 2)

Standard features Brocade FX8-24 Brocade 7800 16/6 Brocade 7800 4/2

Supported storage Open systems andmainframe

Open systems andmainframe

Open systems only


76


Configuring IPsecFor more information on IPsec, refer to the “Internet Protocol security(IPsec)” section in the iSCSI SAN Topologies TechBook, located athttp://elabnavigator.EMC.com, Topology Resource Center tab.

IPsec requires predefined configurations for IKE and IPsec. You canenable IPsec only when these configurations are well-defined andproperly created in advance.

The following steps provide an overview of the IPsec protocol. All ofthese steps require that the correct policies have been created.Because policy creation is an independent procedure from FCIPtunnel creation, you must know which IPsec configurations havebeen created. This ensures that you choose the correct configurationswhen you enable an IPsec tunnel.

1. Some traffic from an IPsec peer with the lower local IP address initiates the IKE negotiation process.

2. IKE negotiates SAs and authenticates IPsec peers during phase 1that sets up a secure channel for negotiation of phase 2 (IPsec)SAs.

IKE negotiates SA parameters, setting up matching SAs in thepeers. Some of the negotiated SA parameters include encryptionand authentication algorithms, Diffie-Hellman group and SAlifetimes.

3. Data is transferred between IPsec peers based on the IPsecparameters and keys stored in the SA database.

4. IPsec tunnel terminates. SA lifetimes terminate through deletionor by timing out.

The first step to configuring IPsec is to create a policy for IKE and apolicy for IPsec. Once the policies have been created, you assign thepolicies when creating the FCIP tunnel.

IKE negotiates SA parameters and authenticates the peer using thepreshared key authentication method. Once the two phases of thenegotiation are completed successfully, the actual encrypted datatransfer can begin.

IPsec policies are managed using the policy command.

You can configure up to 32 IKE and 32 IPsec policies. Policies cannotbe modified; they must be deleted and re-created in order to change




the parameters. You can delete and re-create any policy as long as thepolicy is not being used by an active FCIP tunnel.

Each FCIP tunnel is configured separately and may have the same ordifferent IKE and IPsec policies as any other tunnel. Only one IPsectunnel can be configured for each GbE port.

Limitations Be aware of the following limitations:

◆ IPv6, NAT, and AH are not supported.

◆ You can only create a single secure tunnel on a port; you cannotcreate a nonsecure tunnel on the same port as a secure tunnel.

◆ IPsec specific statistics are not supported.

◆ Fast Write and tape pipelining cannot be used in conjunction withsecure tunnels.

◆ To change the configuration of a secure tunnel, delete the tunneland re-create it with the desired options.

◆ Jumbo frames are not supported for IPsec.

◆ There is no RAS message support for IPsec.

◆ Only a single route is supported on an interface with a securetunnel.

Configuring IPsec 77

78


Fast Write and tape pipeliningIn cases where the FCIP link is the slowest part of the network, andwhere this affects speed, consider using Fast Write and tape writeacceleration (tape pipelining). Fast Write and tape pipelining are twoindividual features that provide accelerated speeds to FCIP tunnels insome configurations. Because of their similarities, they are bothdescribed in this section.

Supported only in Fabric OS 5.2.x andlater, Fast Write accelerates theSCSI write I/Os over FCIP.

Tape pipelining accelerates SCSI write I/Os to sequential devices(such as tape drives) over FCIP. This reduces the number of roundtriptimes needed to complete the I/O over the IP network and speeds upthe process. In order to use tape pipelining, you must enable FastWrite as well.

Both sides of an FCIP tunnel must have a matching configuration forthese features to work.

Compression, Fast Write, and tape pipelining features do not requireany predefined configurations like IPsec does. This makes it possibleto enable these features when you create the FCIP tunnels by addingoptional parameters such as –c, -f, or -t.

Table 16 on page 79 provides a comparison of Fast Write and tapepipelining.



a. Total of 2048 simultaneous exchanges combined for Fast Write and tape pipelining.

Supported configurationsTo help understand the supported configurations, review thesupported configurations shown in Figure 18 on page 80 andFigure 19 on page 81.

Table 16 Fast Write and tape pipelining comparison

Fast Write Tape pipelining

Does not support multiple equal-costpath configurations.

Does not support multiple equal-cost path configurations or multiple non-equal-costpath configurations. (Refer to “Supported configurations” on page 79.)

Class 3 traffic is accelerated with FastWrite.

Class 3 traffic is accelerated between host and sequential device.

With sequential devices (tape drives), there are 1024 initiator-type (IT) pairs per GbEPort, but 2048 initiator-tape-LUN (ITL) pairs per GbE Port. The ITL pairs are sharedamong the IT pairs.a

• Example 1:You can have two ITL pairs for each IT pair as long as the target has two LUNs.

• Example 2:If a target has 32 LUNs, you can have 32 ITL pairs for IT pairs. In this case, only 64IT pairs are associated with ITL pairs. The rest of the IT pairs are not associated toany ITP pairs, so no tape pipelining is performed for those pairs. By default, onlyFast Write-based acceleration is performed on the unassociated pairs.

Does not support multiple non-equal-cost path between host and sequential device.

Fast Write and tape pipelining 79

80


In Figure 18, there is a single tunnel with Fast Write and tapepipelining enabled.

Figure 18 Single tunnel, Fast Write and tape pipelining enabled

FC SAN FC SAN

This connectioncan be VE-VE or

VEX-VE

FCIP tunnelFW=1, TA=1

GE 0

GE 1

GE 0

GE 1

Tape2

Tape1

T1

T0

Hn Hn Tn

H2

H1



In Figure 19, there are multiple tunnels, but none of them create amultiple equal-cost path. Fast Write and tape pipelining are enabledon a per-tunnel, per-port basis.

Figure 19 Multiple tunnels to multiple ports

FC SAN

FC SAN

FCIP tunnel 0FW=0, TA=0

These connectionsmust all be VEX-VE

H1

H2

GE 0

GE 1

FC SAN

GE 0

GE 1

GE 0

GE 1

GE 0





H2

H1

H3

H9H8

H7

H6

H5

H4

Hn

H11

SYM-001461

H10

Fast Write and tape pipelining 81

82


EMC-Cisco MDS distance extension solutionThe Cisco MDS 9000 family of switches can be used to link EMCstorage devices (Symmetrix, VNX™ series, and CLARiiON®) acrossIP networks using the FCIP protocol for disaster recoveryapplications (SRDF and MirrorView) and for data migration (SANCopy™). The MDS 9000 family supports the Fibre Channel andGigabit Ethernet protocols.

Supported configurationsFigure 20 shows an example of Cisco MDS 9000 distance extension.

Figure 20 Cisco MDS 9000 distance extension example

Note these configuration rules:

◆ Cisco MDS switches can be used as part of a disaster recovery(DR) and/or data migration SAN only.

◆ SRDF, MirrorView, and SAN Copy are the only supportedconfigurations.

◆ Remote host I/O configurations are supported across the FCIPlink.

◆ Host I/O across the FCIP link can be supported if the applicationcan tolerate the latency incurred due to the FCIP link

VSAN Blocal SAN traffic

VSAN ASRDF/MV/SC

VSAN ASRDF/MV/SCVSAN A

FCIP

Local data center

VSAN Clocal SAN traffic

Remote data center

Allowed VSANs on FCIP = VSAN A

SRDF, MirrorView, SAN Copy



Note: E-Lab Navigator describes the latest supported configurations andminimum code requirements.

Symmetrix setupSymmetrix SRDF ports should be configured as standard FibreChannel SRDF ports. In a Fibre Channel environment, the Cisco MDSswitch provides all the services of a Fibre Channel switch, similar tothose provided by any other Fibre Channel switch.

VNX setupVNX MirrorView ports should be configured as standard FibreChannel MirrorView ports.

CLARiiON setupCLARiiON MirrorView ports should be configured as standard FibreChannel MirrorView ports.

ReferencesSearch for the additional documentation and the Cisco MDSConfiguration Guide at http://www.cisco.com and select thedocument relevant to the code running on your box.

EMC-Cisco MDS distance extension solution 83


http://www.cisco.com

84


EMC-QLogic distance extension solutionThe QLogic iSR-6142 Storage Router is a low cost FC/iSCSI solutiondesigned to enable users to replicate data between FC SANs over aLAN/WAN utilizing iSCSI/GigE as the transport over distance.

The router contains two 1/2 GB/s FC ports and dual 10/100/1000MB/s iSCSI/GigE ports. The routers interconnect through the dualGigE/iSCSI links allowing the replication data to be transmittedbetween two end devices. The two routers allow up to 4 FC SANs tobe connected as NL_ports (that is, 2 per router) and prevent the SANsfrom merging into one large SAN.

This router is intended for low to mid-range environments wheredistance extension and device replication, such as EMC's VNX seriesand CLARiiON MirrorView software, are essential.

Supported configurationsThe iSR-6142 Storage Router supports one distinct topology in anEMC environment:

WAN Topology — Interconnecting remote SAN Islands (also knownas Remote SAN Island Connectivity).

The SANbox 6142 Intelligent Router supports inter-connectingremote SAN islands. This does not result in the merging of the twoend fabrics but will allow communication to occur between two endnodes when correctly configured (Figure 21 on page 85).



Figure 21 SANbox 6142 Intelligent Router

As shown in Figure 21, CX1_SPA1 (CLARiiON MirrorView port) isattached to Fabric_1. CX2_SPA1 (CLARiiON MirrorView port) isattached to Fabric_2. Using the QLogic SANbox 6142 it is possible toestablish the communication between the MirrorView ports whilemaintaining two separate fabrics. the QLogic Sanbox 6142 will createvirtual entities on each fabric to represent the remote device. Themechanism to establish the connection is called remotemap. Theremotemap is created using the CLI/GUI from either of the routersand is communicated to the remote router over the WAN. Thisremotemap presents CX1_SPA1 to Fabric_2 and CX2_SPA1 toFabric_1 as an NL_Port. This NL_Port needs to be zoned local CXN_Port to allow communication between the two arrays overdistance.

ScalabilityThe following are scalability guidelines, restrictions, and limitations:

◆ Maximum number of connections = 1024.◆ Maximum number of virtual FC ports = 64 per unit (31 per FC

port with 1 additional dedicated to each FC port for discovery -VP0 and VP1).

◆ Maximum number of concurrent I/Os = 1024 per unit (typically32 per session).

◆ Maximum number of initiators/targets = 62 per unit (31 per port).

GEN-000288

CLARiiON

CLARiiON

CX1_SPA1

CX2_SPA1

FC SANFABRIC_1

FC SANFABRIC_2

TCP/IPiSCSI network

QLogicSANbox 6142

QLogicSANbox 6142

CX1_SPA2(virtual)CX1_SPA1(virtual)

EMC-QLogic distance extension solution 85

86


Best practicesRequirements for this configuration are as follows:

◆ At least one FC Port of the iSR-6142 should be connected to FCSAN.

◆ iSCSI/ GE Port IP addresses of remote router and iSCSI/GE portIP addresses of local routers must be accessible by each other.

◆ Remote iSR-6142 management port IP address and local SANbox6142 management port IP address must be accessible by eachother.

Recommendations for this configuration are as follows:

◆ Both GigE links are utilized with load balancing enabled.

◆ Compression is enabled over distance.

◆ Smart Writes is enabled.

◆ Windows Scaling is enabled with the recommended WindowsScaling Factor setting.

◆ Header and Data Digest is enabled.

◆ Zone each N_Port that will have a remotemap to both of therouter FC ports.

◆ Use WWPN zoning.

SmartWriteWhen connecting SAN over long distances, round trip delays createsignificant impact to the performance. Typically, data writes involvetwo or more round trip latencies that result in a significant barrier tothe data replication performance. SmartWrite technology is designedto minimize the round trip latency of any write I/O to a singleround-trip latency. Benefits realized with this feature key include:

◆ Minimizes round trip delays for any data write operation to asingle round trip latency.

◆ Allows load balancing over multiple IP links.

◆ Provides failover and failback between two gigabit ethernet links.



◆ Allows data compression. This is very useful when data roundtrip latencies between two routers exceed more than 25 ms orlong distance link rate is equal or less than 4500 Mb/s (DS-3 linerate).

ReferencesFor more information, refer to http://www.QLogic.com.

Please reference the QLogic SANbox 6142 Intelligent Storage RouterUser Guide for additional information regarding:

◆ Command Line Interface reference

◆ SANsurfer Router Manager GUI

◆ Recommended Windows Scaling Factor determined by latencybetween routers

◆ Hardware

Additional documentation regarding the QLogic SANbox 6142Intelligent Storage Router includes:

◆ QLogic SANbox 6142 Quick Start Guide

EMC-QLogic distance extension solution 87

http://www.QLogic.com

88


SummaryTable 17 compares the distance extension solutions features forTCP/IP products.

a. Only one FCIP tunnel can be configured per GigE port if TCP Byte Streaming is enabled.

Table 17 Distance extension comparison table for TCP/IP products

Feature Symmetrix(GigE)

Brocade Cisco MDS Brocade MSeries

QLogic

Fast Write n/a yes yes yes yes

Jumbo frames yes yes yes yes no

Encryption no no yes no no

Applications all families of srdf srdf, srdfa, mva,mvs, sancopy

srdf, srdfa, mva,mvs, sancopy, ors

srdf, srdfa, mva,mvs, sancopy

mva,mvs,sancopy

Host I/O n/a yes yes yes no

Protocols tcp fcip fcip ifcp iscsi

Authentication no yes yes no yes

Number of sessionsper port

64 8a 1 64 32

Load Balancing yes yes yes no yes

Compression yes yes yes yes yes



Table 18 compares the distance extension solution features for non-TCP/IP products.

Legend:

Table 18 Distance extension comparison table for non TCP/IP products

Distanceextensionchassis

Client/WDM/Protocol conversion Link speed Features Switch vendor support

Client Side WAN/ side/Line side 1Gb

2Gb

4Gb

10Gb

BBC CLB WA FEC COM Brocade Cisco BrocadeM Series

QLogic

FC-SW

FC-Direct

CWDM DWDM GbE SONET

ADVAFSP2000

X X X X X X X X X

ADVAFSP3000

X X X X X X

CienaCN2000

X X X X X X X X X X X X X

CienaCN4200

X X X X X X X X

CiscoONS15454

X X X X X X X X X X X X

CiscoONS15540

X X X X X X X

Nortel5200

X X X X X X X X X X

Nortel3500

X X X X X X X X X X

BBC: BBC spoofing

WA: Write Acceleration

CLB: Channel Load Balancing

WA: Write Acceleration

FEC: Forward Error Correction

COM Compression

Summary 89

90



Index

Aactive and passive devices 52

BBB_Credit

guidelines 38buffer-to-buffer

local termination 52

CCisco MDS 9000 82Congestion

network 31credit starvation 41CWDM 19

DData buffering and flow control 37devices

active and passive 52distance extension 35

technologies 35DWDM 15

FFast Write 78FCIP

with Cisco MDS 9000 family 82Fibre Channel

and BB-Credit 37BB_Credit guidelines 38

SiRT 54flow control and data buffering 37

GGbE (Gigabit Ethernet) 23

IInternet Protocol Security (IPsec) 32IPsec

and tunneling 32configuring 76terminology 33

IPsec (Internet Protocol security) 32iSCSI

technology 34

Llink initialization 58link speed 36

MMDS 9000 82

NNetwork congestion 31

Ppassive and active devices 52


92

Index

SSiRT

with SRDF 54SiRT (Single roundtrip) 54SmartWrite 86SONET 21

Ttape pipelining 78TCP

error recovery 28terminology 24

TCP/IP 24, 51


TechBook: Extended Distance Technologies

Technology

Transcript of TechBook: Extended Distance Technologies