Data Center Solution- Storage - docs.broadcom.com€¦ · 53-1003608-01 31 October 2014 Data Center...

53-1003608-0131 October 2014

Data Center Solution-StorageGen 5 Fibre Channel Distance ExtensionUsing ADVA FSP 3000 WDM PlatformDesign Guide

© 2014, Brocade Communications Systems, Inc. All Rights Reserved.

Brocade, the B-wing symbol, Brocade Assurance, ADX, AnyIO, DCX, Fabric OS, FastIron, HyperEdge, ICX, MLX, MyBrocade, NetIron,OpenScript, VCS, VDX, and Vyatta are registered trademarks, and The Effortless Network and the On-Demand Data Center are trademarksof Brocade Communications Systems, Inc., in the United States and in other countries. Other brands and product names mentioned may betrademarks of others.

Notice: This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerning anyequipment, equipment feature, or service offered or to be offered by Brocade. Brocade reserves the right to make changes to this documentat any time, without notice, and assumes no responsibility for its use. This informational document describes features that may not becurrently available. Contact a Brocade sales office for information on feature and product availability. Export of technical data contained inthis document may require an export license from the United States government.

The authors and Brocade Communications Systems, Inc. assume no liability or responsibility to any person or entity with respect to theaccuracy of this document or any loss, cost, liability, or damages arising from the information contained herein or the computer programs thataccompany it.

The product described by this document may contain open source software covered by the GNU General Public License or other opensource license agreements. To find out which open source software is included in Brocade products, view the licensing terms applicable tothe open source software, and obtain a copy of the programming source code, please visit http://www.brocade.com/support/oscd.

http://www.brocade.com/support/oscd

Contents

Data Center Solution-Storage Gen 5 Fibre Channel Distance Extension Using ADVA FSP3000 WDM Platform............................................................................................................5

Preface..............................................................................................................5Overview............................................................................................... 5

Reference Architecture..................................................................................... 7Business Requirements........................................................................ 8Special Considerations......................................................................... 9

Overview of Active WDM Systems..................................................................14WDM System Building Blocks.........................................................................14

The Control Unit.................................................................................. 15WDM Transponder and Muxponder Modules..................................... 15The WDM Optical Layer......................................................................17

ADVA FSP 3000 WDM Platform..................................................................... 18ADVA FSP 3000 Optical Layer........................................................... 19Protection Modules............................................................................. 21ADVA FSP 3000 Transponder and Muxponder Module Types ......... 22WDM Error Forwarding Settings ........................................................ 27SX and LX Optical Interfaces.............................................................. 31Buffer Credit Calculation and Settings................................................ 31Brocade Switch Port Settings .............................................................32Dual Fabric Over Distance ................................................................. 33Troubleshooting.................................................................................. 34

Appendix A......................................................................................................35Latency ...............................................................................................35Support of Brocade port-based Fibre Channel features..................... 36

Data Center Solution-Storage Gen 5 Fibre Channel Distance Extension Using ADVA FSP 3000 WDM Platform Design Guide 353-1003608-01

4 Data Center Solution-Storage Gen 5 Fibre Channel Distance Extension Using ADVA FSP 3000 WDM Platform Design Guide53-1003608-01

Data Center Solution-Storage Gen 5 Fibre Channel DistanceExtension Using ADVA FSP 3000 WDM Platform

● Preface..............................................................................................................................5● Reference Architecture..................................................................................................... 7● Overview of Active WDM Systems..................................................................................14● WDM System Building Blocks.........................................................................................14● ADVA FSP 3000 WDM Platform..................................................................................... 18● Appendix A......................................................................................................................35

Preface

OverviewThe most common reason for extending a Fibre Channel (FC) storage area network (SAN) overextended distances is to safeguard critical business data and provide near-continuous access toapplications and services in the event of a localized disaster. Designing a distance extension solutioninvolves a number of considerations, both business and technical.

From the business perspective, applications and their data need to be classified by how critical they arefor business operation, how often data must be backed up, and how quickly it needs to be recovered inthe event of failure. Two key metrics are the Recovery Point Objective (RPO) and the Recovery TimeObjective (RTO). The RPO is the time period between backup points and describes the acceptable lossof data after a failure has occurred. For example, if a remote backup occurs every day at midnight and asite failure occurs at 11 pm, changes to data made within the last 23 hours will be lost. RTO describesthe time to restore the data after the disaster. RTO determines the maximum outage that can occur withan acceptable impact to the business.

From a technology perspective, there are several choices for the optical transport network andconfiguration options for the FC SAN when it is extended over distance. Applications with strict RTOand RPO require high-speed synchronous or near-synchronous replication between sites withapplication clustering over distance for immediate service recovery. Less critical applications may onlyrequire high-speed replication that could be asynchronous to meet the RPO/RTO metrics. Lower priorityapplications that don't need immediate recovery after a failure can be restored from backup tapes fromremote vaults.

Brocade is a leader in Fibre Channel SAN switching providing a broad product portfolio with uniquefeatures the designer can leverage for cost-effective and efficient SAN distance extension. Inter-switchLinks (ISL) are used to connect two SAN switches together. By stretching ISLs over extended distances(a few kilometers to as much as 200 Km), data replication traffic can use Fibre Channel as the transportto a remote data center. For this reason, SAN distance extension over ISLs is a common method oftransporting replicated storage data for mission critical application disaster recovery.

Purpose of this DocumentThis guide contains design guidance for SAN distance extension using Brocade Gen-5 Fibre ChannelSAN products, Brocade Fabric Operating System (FOS) and wave division multiplexer (WDM) from


Adva, the Adva FSP-3000. Brocade Gen-5 products have a number of features designed to optimizeSAN extension using ISL connections.

Design best practices are included for SAN extension with ISL connections. The topology shown in thereference architecture has validated in Brocade's Strategic Solution Validation Lab.

This design can be used with array-based replication and/or tape backup systems due to its excellentscalability, high performance and very low latency at distances up to 200 KM.

AudienceThis document is intended for disaster recovery planners and SAN architects who are evaluating anddeploying DR solutions that use SAN distance extension for storage data.

ObjectivesThis design guide is intended to provide guidance and recommendations based on best practices for atwo-site data center disaster recovery solution using Fiber Channel ISL connections over extendeddistance.

Restrictions and LimitationsThis design guide only addresses SAN distance extension using ISL connections over WDM links onthe Adva FSP3000 WDM platform. Other WDM vendor specific design guides are available.

Related DocumentsThe following documents are valuable resources for the designer. This design is based on the DataCenter Infrastructure Base Reference Architecture which includes SAN building blocks and templates:

• Data Center Infrastructure Base Reference Architecture• SAN Blocks• Fibre Channel Core Blocks• Brocade Fabric OS Administrator Guide, v7.3.0• Brocade Fabric OS Command Reference, v7.3.0• Data Center Infrastructure, Storage-Design Guide: SAN Distance Extension Using ISLs

About BrocadeBrocade networking solutions help the world's leading organizations transition smoothly to a worldwhere applications and information reside anywhere. This vision is realized through the BrocadeOne™ strategy, which is designed to deliver key business benefits such as unmatched simplicity, non-stop networking, application optimization, and investment protection.

Innovative Ethernet and storage networking solutions for data center, campus, and service providernetworks help reduce complexity and cost while enabling virtualization and cloud computing toincrease business agility.

To help ensure a complete solution, Brocade partners with world-class IT companies and providescomprehensive education, support, and professional services offerings.

To learn more, visit www.brocade.com

Data Center Solution-Storage Gen 5 Fibre Channel Distance Extension Using ADVA FSP 3000 WDM Platform


http://community.brocade.com/docs/DOC-2758

http://community.brocade.com/t5/Design-Build/Data-Center-Infrastructure-Base-Reference-Architecture/ta-p/36885#_Toc384393782

http://community.brocade.com/t5/Design-Build/Data-Center-Infrastructure-Base-Reference-Architecture/ta-p/36885#_Toc384393784

http://www.brocade.com/forms/getFile?p=documents/product_manuals/B_SAN/FOS_AdminGuide.html

http://www.brocade.com/forms/getFile?p=documents/product_manuals/B_SAN/FOS_CommandRef.html

http://community.brocade.com/t5/Design-Build/Data-Center-Infrastructure-Storage-Design-Guide-SAN-Distance/ta-p/36627

http://www.brocade.com

About ADVAOur company began with a single vision: to transport data, storage, voice and video signals at nativespeeds and lowest latency. A lot's changed since that time, but our vision remains the same. Ourproducts are the building blocks for tomorrow's networks, enabling the transport of increasing amountsof data across the globe. From the access to the metro core to the long haul, we create intelligent,software-automated solutions that will provide future generations with networks that can scale to meetincreasing bandwidth demands.

To learn more, visit www.advaoptical.com

Document History

Date Version Description

2014-10-31 1.0 Initial release

Reference ArchitectureThis design guide is based on the Data Center Infrastructure Base Reference Architecture buildingblocks. Shown below is the SAN Template used for this design.

FIGURE 1 SAN Core Template with ADVA Optical Network Building Blocks

Reference Architecture


http://www.advaoptical.com

This template illustrates a common SAN topology, Core/Edge, with two edge blocks, Edge Switch andEdge Access Gateway and a Core Backbone block. Both edge blocks connect to the Core Backboneblock using ISL Trunks, providing automatic frame based flow balancing over multiple ISL links forhighest utilization with mixed traffic flows. The Edge Access Gateway block is commonly used withembedded FC switches found in blade servers. Access Gateway can also be used with rack mountswitches with Top-of-Rack SAN switches. When a switch is configured for Access Gateway mode, itdoes not consume a fabric Domain ID simplifying the fabric design. Similar to ISL Trunks, AccessGateway provides Access Gateway Trunks for excellent link utilization and automatic fail-over shoulda link in the trunk fail.

Fabric A and Fabric B are shown indicating the use of two physically independent SAN fabrics toconnect servers to storage arrays. This is a SAN best practice for high availability and resiliency. EachServer and storage use dual connections, each going to either Fabric A or Fabric B. Servers areconfigured with IO Adaptors and multipath IO device drivers for active/active IO from both adaptors toboth fabrics. Should a path in one fabric fail for any reason (HBA, cable, FC switch port, FC switch,array port, configuration error, power outage, etc.), then IO continues on the remaining path.

Fabric C and Fabric D are shown in the Core Backbone block. The Brocade DCS and DCX Backboneswitches support virtual fabrics allowing ports from the same switch to be allocated to logically isolatedfabrics. Again, dual physically independent fabrics are connected to the long distance optical transportnetwork for high availability and resiliency.

The Core Backbone block uses ISL links with a long distance optical network as shown by the cloudlabeled “San Distance Extension with ISLs”. Server IO at the edge blocks flows to the Core Backboneblock and then to storage arrays. The storage array(s) replicates changes to the data blocks to anarray(s) in a remote datacenter. The replication traffic flows over the ISL links in Fabric C and D thatare attached to the long distance optical network.

Note that “Backbone ICL” links connect the core switches in each fabric. This is an innovative featureavailable on Brocade DCX and DCX Backbone switches providing very high bandwidth trunksbetween core switches without consuming ports on port cards allowing all port card ports to be usedfor connecting arrays and for ISL Trunks to edge switches.

References

Data Center Infrastructure, Base Reference Architecture:

SAN Blocks

Fibre Channel Core Blocks

Brocade DCX Backbone Switch Data Sheet

Business RequirementsAs more applications drive business value, and the associated data becomes key to competitiveadvantage, cost-effective protection of the applications and data from site disasters and extendedoutages has become the norm. Modern storage arrays provide synchronous as well as asynchronousarray-to-array replication over extended distances. When the array provides block-level storage forapplications, Fibre Channel is the primary network technology used to connect the storage arrays toservers, both physical and virtual. For this reason, cost-effective disaster recovery designs leverageFibre Channel to transport replicated data between arrays in different data centers over distancesspanning a few to more than 100 kilometers. Therefore, SAN distance extension using Fibre Channelis an important part of a comprehensive, cost-effective and effective disaster recovery design.

Business Requirements




http://community.brocade.com/t5/Design-Build/Data-Center-Infrastructure-Base-Reference-Architecture/ta-p/36885

http://www.brocade.com/products/all/san-backbones/index.page?network=FIBRE_CHANNEL

Special ConsiderationsIt is helpful to review the following special considerations that apply to Fibre Channel SAN distanceextension. It is important to understand the Fibre Channel protocol and the optical transport technologyand how they interact.

Optical Fiber Cabling

There are two basic types of optical fiber, Multimode Fiber (MMF) and Single-Mode Fiber (SMF).Multimode fiber is generally used for short distance spans and is common for interconnecting SANequipment within the data center. Single-mode fiber has a smaller core diameter of 9 µm and carriesonly a single mode of light through the waveguide. It is better at retaining the fidelity of each light pulseover long distances and results in lower attenuation. Single mode fiber is always used for long-distanceextension over optical networks and often used even within the data center for FICON installations.

There are several types of single-mode fiber, each with different characteristics that should take intoconsideration when deploying a SAN extension solution. Non-Dispersion Shifted Fiber (NDSF) is theoldest type of fiber and was optimized for wavelengths operating at 1310 nm, but performed poorly inthe 1550 nm range, limiting maximum transmission rate and distance. To address this problem,Dispersion Shifted Fiber (DSF) was introduced. DSF was optimized for 1550 nm, but introducedadditional problems when deployed in Dense Wavelength Division Multiplexing (DWDM) environments.The most recent type of single-mode fiber, Non-Zero Dispersion Shifted Fiber (NZ-DSF) addresses theproblems associated with the previous types and is the fiber of choice in new deployments.

As light travels through fiber, the intensity of the signal degrades, called attenuation. The three maintransmission windows in which loss is minimal are in the 850, 1310, and 1550 nm ranges. The tablebelow lists common fiber types and the average optical loss incurred by distance for both multimode(MM) and single mode (SM) fiber.

Average attenuation of optical fiber due to distanceTABLE 1

Fiber Optical Loss (dB/km)

Size Type 850 nm 1310 nm 1550 nm

9/125 µ SM - 0.35 0.2

50/125 µ MM 3.0 - -

62.5/125 µ MM 3.0 - -

Optical Power Budget, Fiber Loss

A key part of designing SANs over long distance optical networks involves analyzing fiber loss andoptical power budgets. The decibel (dB) unit of measure for the signal power in a fiber link. The dB losscan be determined by comparing the launch power of a device to the receive power. Launch andreceive power are expressed in decibel-milliwatt (dBm) units, which is the ratio of measured signalpower in milliwatts (mW) to 1 mW.

References

Definition of dB, Wikipedia

Definition of dBm, Wikipedia

Definition of Optical Power Budget, Wikipedia

Special Considerations


http://en.wikipedia.org/wiki/Decibel

http://en.wikipedia.org/wiki/DBm

http://en.wikipedia.org/wiki/Optical_power_budget

The optical power budget identifies how much attenuation can occur across a fiber span while stillmaintaining sufficient output power for the receiver. It is determined by finding the difference between“worst-case” launch power and receiver sensitivity. Transceiver and other optical equipment vendorstypically provide these specifications for their equipment. A loss value of 0.5 dB can be used toapproximate attenuation caused by a connector/patch panel. It is useful to subtract an additional 2 dBfor safety margin.

Optical Power Budget = (Worst Case Launch Power) – (Worst Case Receiver Sensitivity) +(Connector Attenuation)

Signal loss is the total sum of all losses due to attenuation across the fiber span. This value should bewithin the power budget to maintain a valid connection between devices. To calculate the maximumsignal loss across an existing fiber segment, use the following equation:

Signal Loss = (Fiber Attenuation/km * Distance in km) + (Connector Attenuation) + (Safety Margin)

The previous table showed average optical loss characteristics of various fiber types that can be usedin this equation, although loss may vary depending on fiber type and quality. It is always better tomeasure the actual optical loss of the fiber with an optical power meter.

Some receivers may have a maximum receiver sensitivity that should not be exceed. If the opticalsignal is greater than the maximum receiver sensitivity, the receiver may become oversaturated andnot be able to decode the signal, causing link errors or even total failure of the connection. Fiberattenuators can be used to resolve the problem. This is often necessary when connecting FC switchesto DWDM equipment using single mode FC transceivers.

FC Transceivers for Extended Distances

Optical Small Form-factor Pluggable (SFP) transceivers are available in short- and long-wavelengthtypes. Short wavelength transceivers transmit at 850 nm and are used with 50 or 62.5 µm multimodefiber cabling. For fiber spans greater than several hundred meters without regeneration, use long-wavelength transceivers with 9 µm single-mode fiber. Long-wavelength SFP transceivers typicallyoperate in the 1310 or 1550 nm range.

Optical transceivers often provide monitoring capabilities that can be viewed through FC switchmanagement tools, allowing some level of diagnostics of the actual optical transceiver itself.

NOTEBrocade 8 and 16 Gbps products enforce the use of Brocade branded optics plus a restricted list ofspecialist third party options to meet requirements for extended distance or CWDM/DWDM optics.Other Brocade products do not enforce optics rules but qualified or certified optics only should be usedas shown in the latest Brocade Compatibility Matrix, Transceivers Quick Reference section (see theReferences below).

References

Brocade Compatibility Matrix: Network Solutions Section

FC Protocol over Extended Distance Considerations

Flow Control

Brocade switches can support two methods of flow control over an ISL

FC Transceivers for Extended Distances


http://www.brocade.com/downloads/documents/matrices/brocade-compatibility-matrix.pdf

• Virtual Channel (VC_RDY) – VC_RDY is the default method and uses multiple lanes or channels,each with different buffer credit allocations, to prioritize traffic types and prevent head-of-lineblocking. VC_RDY flow control differentiates traffic across an ISL. It serves two main purposes:

‐ To differentiate fabric internal traffic from end-to-end device traffic.‐ To differentiate different data flows of end-to-end device traffic to avoid head-of-line

blocking. Fabric internal traffic is generated by switches that communicate with each otherto exchange state information (such as link state information for routing and deviceinformation for Name Service). This type of traffic is given a higher priority so that switchescan distribute the most up-to-date information across the fabric even under heavy devicetraffic. Additionally, multiple IOs are multiplexed over a single ISL by assigning different VCsto different IOs and giving them the same priority (unless QoS is enabled). Each IO canhave a fair share of the bandwidth, so that a large-size IO will not consume the wholebandwidth and starve a small-size IO, thus balancing the performance of different devicescommunicating across the ISL.

• Receiver Ready (R_RDY) – R_RDY is defined in the ANSI T-11 standards and uses a single lane, orchannel, for all frame types.

NOTEWhen Brocade switches are configured to use R_RDY flow control, other mechanisms are used toenable QoS and prevent head-of-line blocking.

When connecting switches across dark fiber or wave division multiplexing (WDM) optical links, VC_RDYis the preferred method, but there are some distance extension devices that require the E_Port useR_RDY. To configure R_RDY flow control on Brocade switches, use the portCfgISLMode command.

References

Brocade FOS Administrator Guide, v7.

3.0

: Inter-switch Links (ISL)

Brocade FOS Command Reference, v7.0.1: portCfgISLMode command

Quality of Service

Starting with FOS release 6.0, Brocade Virtual Channel technology can be used to prioritize trafficbetween initiator/target pairs by mapping traffic flows to High, Medium, or Low priority queues. QoSsupport with Virtual Channels is enabled with the Adaptive Networking license. QoS is supported overlong-distance ISLs that utilize up to 255 buffers. When an E_Port is allocated more than 255 buffers, theremaining buffers are allocated to the medium priority queue.

References

Brocade FOS Administrator Guide, v7.3.0: Optimizing Fabric Behavior

Brocade FOS Administrator Guide, v7.3.0: QoS SID/DID Traffic Prioritization

Brocade FOS Administrator Guide, v7.3.0: QoS Zones

Brocade FOS Command Reference, v7.3.0: portCfgQoS command

Quality of Service


http://www.brocade.com/downloads/documents/html_product_manuals/FOS_AG_701/wwhelp/wwhimpl/js/html/wwhelp.htm



http://www.brocade.com/downloads/documents/html_product_manuals/FOS_730_CLI/wwhelp/wwhimpl/common/html/wwhelp.htm

HTTP://WWW.BROCADE.COM/DOWNLOADS/DOCUMENTS/HTML_PRODUCT_MANUALS/FOS_AG_701/WWHELP/WWHIMPL/JS/HTML/WWHELP.HTM#HREF=QOS.25.01.HTML

HTTP://WWW.BROCADE.COM/DOWNLOADS/DOCUMENTS/HTML_PRODUCT_MANUALS/FOS_AG_701/WWHELP/WWHIMPL/JS/HTML/WWHELP.HTM#HREF=QOS.25.04.HTML

HTTP://WWW.BROCADE.COM/DOWNLOADS/DOCUMENTS/HTML_PRODUCT_MANUALS/FOS_730_ADMIN_04/GUID-9C32AF4B-CD4F-45FD-8C79-612137F8F73D.HTML

HTTP://WWW.BROCADE.COM/DOWNLOADS/DOCUMENTS/HTML_PRODUCT_MANUALS/FOS_730_CLI/WWHELP/WWHIMPL/COMMON/HTML/WWHELP.HTM#CONTEXT=FOS_CMD_REF_HTML&FILE=COMMANDS_A_Z.PORTCFGQOS.HTML

Buffer Allocation

Before considering FC-level buffer allocation, note that the availability of sufficient FC-level buffering isnot itself sufficient to guarantee bandwidth utilization. Other limitations, particularly at the SCSI level ofthe storage initiator and/or target, are often the limiting factor. The I/O size, I/O per Second (IOPS)limit, and concurrent or outstanding I/O capability at the SCSI level of the initiators/targets can be andoften are gating factors.

While exact calculations are possible, a simple rule of thumb is often used to calculate the BB creditrequirement of a given link. Based on the speed of light in an optical cable, a full-size FC frame spansapproximately 4 km at 1 Gbps, 2 km at 2 Gbps, 1 km at 4 Gbps, 500m at 8 Gbps , 200m at 16 Gbps or400 m at 10 Gbps. The rule of thumb is this: 1 credit is required for every kilometer at 2 Gbps;therefore half a credit is required for every kilometer at 1 Gbps and 2 credits are required for everykilometer at 4 Gbps. With this simple set of guidelines it is easy to estimate the amount of requiredcredits per link to maintain line speed.

Having insufficient BB credits will not cause link failure, but it will reduce the maximum throughput.

In the example cited above, the 1-ms link running at 4 Gbps with only 100 BB credits can achieve amaximum throughput of approximately 2 Gbps.

Using the LS option, the portCfgLongDistance command can be used to allocate the required buffersfor the link distance.

References

Brocade FOS Administrator Guide, v7.3.0: Buffer Credit Management

Brocade FOS Administrator Guide, v7.3.0: Long Distance Link Modes

Brocade FOS Command Reference, v7.3.0: portCfgLongDistance command

Frame-Based Trunking

Long distance links using VC_RDY flow control can be part of an ISL trunk group if they are configuredfor the same speed and distance and the distances of all links are nearly equal. Within a Frame basedtrunk, the maximum allowed difference between shortest and longest links is approximately 400meters.

When R_RDY flow control is used, frame-based trunking is disabled. Exchanged-based routing policy,used to interleave FC exchanges across multiple ISLs, can be used with either type of flow control.

References

Brocade FOS Administrator Guide, v7.3.0: Manageing Trunking Connections, ISL Trunking over Long DistanceFabrics

Brocade FOS Administrator Guide, v7.3.0: Routing Traffic, Routing Policies, Exchange Based Routing

Dynamic Path Selection

Dynamic Path Selection (DPS), also called Exchange Based Routing, is a feature first available for 4Gbps products and later. DPS applies at a fabric level and has no restrictions on co-location of portson a given switch or even on their going the same route through the fabric. However, as with someother configuration options DPS is not supported for certain limited cases, specifically FICON and HP-EVA/CA. Where frame-based ISL Trunks cannot be used, DPS is a good alternative for highavailability with multiple ISL connections.

Buffer Allocation


HTTP://WWW.BROCADE.COM/DOWNLOADS/DOCUMENTS/HTML_PRODUCT_MANUALS/FOS_730_ADMIN_04/GUID-9FAD56E5-ED04-4777-900A-F9F207B7FC4A.HTML

HTTP://WWW.BROCADE.COM/DOWNLOADS/DOCUMENTS/HTML_PRODUCT_MANUALS/FOS_730_ADMIN_04/GUID-425DB881-8C24-4FC8-B2C5-654C34CEB50B.HTML

HTTP://WWW.BROCADE.COM/DOWNLOADS/DOCUMENTS/HTML_PRODUCT_MANUALS/FOS_730_CLI/WWHELP/WWHIMPL/COMMON/HTML/WWHELP.HTM#CONTEXT=FOS_CMD_REF_HTML&FILE=COMMANDS_A_Z.PORTCFGLONGDISTANCE.HTML

HTTP://WWW.BROCADE.COM/DOWNLOADS/DOCUMENTS/HTML_PRODUCT_MANUALS/FOS_730_ADMIN_04/GUID-B1AE3D63-C461-4E84-81DC-7ACF1EA9F54E.HTML

HTTP://WWW.BROCADE.COM/DOWNLOADS/DOCUMENTS/HTML_PRODUCT_MANUALS/FOS_730_ADMIN_04/GUID-B1AE3D63-C461-4E84-81DC-7ACF1EA9F54E.HTML

HTTP://WWW.BROCADE.COM/DOWNLOADS/DOCUMENTS/HTML_PRODUCT_MANUALS/FOS_730_ADMIN_04/GUID-3335EDAE-F9E2-4DD2-934C-E7316F656017.HTML

References

Brocade FOS Administrator Guide, v7.3.0: Routing Traffic, Routing Policies, Exchange Based Routing

D-port Advanced Diagnostics for Brocade 16G SFP+

A Brocade D-Port is used to diagnose optics and cables. It does not carry any FC control or data trafficand is supported on E_Ports and also F_Ports if a Brocade 1860 adaptor is used in the server. When aport is in D_Port mode, the following diagnostic tests can be conducted (refer to the diagram below. “C3ASIC” refers to 16 Gbps products).

• Performs Electrical loopback• Performs Optical loopback• Measures link distance• Performs link traffic test

FIGURE 2 D_Port Diagnostic Test Paths

References

Brocade FOS Command Reference, v7.3.0: portCfgDport command

Brocade FOS Command Reference, v7.3.0: portDPortTest command

In-flight Encryption and Compression over 16 Gbps ISLs

With the 16 Gbps products such as the DCX 8510 Backbone switch, in-flight encryption andcompression can be applied at an egress E_Port of an ISL between two Brocade switches. The E_Porton the receiving side of ISL will decrypted and decompressed the traffic. A maximum of two ports perASIC can be have in-flight encryption and compression enabled..

References

Brocade FOS Administrator Guide, v7.3.0: In-flight Encryption & Compression

Forward Error Correction (FEC) on 16Gbps Ports

FEC can recover bit errors for 10 Gbps and 16 Gbps ports for both FC frames and FC primitives. FECon 16 Gbps ports has the following capabilities.

• Can correct up to 11 error bits in every 2112-bit frame transmission• Enhances reliability of transmission and thus performance

D-port Advanced Diagnostics for Brocade 16G SFP+


http://www.brocade.com/downloads/documents/html_product_manuals/FOS_730_ADMIN_04/GUID-3335EDAE-F9E2-4DD2-934C-E7316F656017.html

http://www.brocade.com/downloads/documents/html_product_manuals/FOS_730_CLI/wwhelp/wwhimpl/common/html/wwhelp.htm

http://www.brocade.com/downloads/documents/html_product_manuals/FOS_730_CLI/commands_a_z.portDPortTest.html

HTTP://WWW.BROCADE.COM/DOWNLOADS/DOCUMENTS/HTML_PRODUCT_MANUALS/FOS_730_ADMIN_04/GUID-9E7170BF-02BC-44D3-9EB9-0C619E816154.HTML

• Enabled by default on backend links for 16 Gbps blades in 8510-8/8510-4 chassis• Supported on E/Ex_ports between 16Gbps ports at either 16Gbps or 10Gbps link speed.

References

Brocade FOS Administrator Guide, v7.3.0: Performing Advanced Configuration Tasks, Enabling Forward ErrorCorrection

Overview of Active WDM SystemsNetworks based on Wavelength Division Multiplexing (WDM) are an integral part of the picture when itcomes to connecting geographically-dispersed data centers. In today’s networks, WDM devices andoptical services based on WDM technology are becoming more and more of a commodity. As a result,most manufacturers and service providers are able to transport all kinds of signals over distance.

Modern WDM systems seem to be relatively exchangeable. In general, this statement is true, butlooking more closely at specific applications, there are small but important differences among theplatforms available.

In this chapter, a closer look into WDM systems technology is provided to better understand the ADVAFSP 3000 platform. For understanding this chapter, you should be familiar with the basics of WDMand adjacent technologies like TDM.

For this design guide it is not relevant to differ between CWDM and DWDM since the technology, froma transmission and data processing point of view is the same. Also the abbreviation ‘WDM’ and theterm ‘WDM system’ are use synonymical.

WDM System Building BlocksNearly all active WDM Systems consist of standard building blocks. Usually WDM systems use amodular approach, where modules will be plugged into one or more chassis. Typically, those modulesare:

• One or more control units for external communication, e.g. provisioning• Optical multiplexer (MUX) and de-multiplexer (DEMUX) modules• Transponders and/or Muxponder modules• Optical amplifiers, ROADM’s, dispersion compensation and other specialized modules• In the figure below, you can see the generic architecture of a WDM system showing its building

blocks.

Overview of Active WDM Systems


HTTP://WWW.BROCADE.COM/DOWNLOADS/DOCUMENTS/HTML_PRODUCT_MANUALS/FOS_730_ADMIN_04/GUID-5B207150-CE27-4785-93C8-48431A06C019.HTML

HTTP://WWW.BROCADE.COM/DOWNLOADS/DOCUMENTS/HTML_PRODUCT_MANUALS/FOS_730_ADMIN_04/GUID-5B207150-CE27-4785-93C8-48431A06C019.HTML

FIGURE 3 Architecture of a WDM System

The Control UnitThe control unit is a module that allows external and internal communication and is mainly used forprovisioning and OAM (operation and maintenance). Communication is established using Ethernetand/or serial interfaces. You can connect to your WDM equipment using protocols like HTTP, Telnet,SSH, SNMP, etc.

WDM Transponder and Muxponder ModulesThe main parts of active WDM systems are the transponder and muxponder modules. These are usedto connect end (client) devices with the optical layer of the WDM system. Client data-rates range fromseveral Mbit/Sec up to 100Gbit/Sec. Usually, WDM data-rates would be similar, but are typicallybetween 10 and 100Gbit/Sec for modern systems.

WDM Transponders

A transponder converts the incoming signal from the end or client device to a WDM wavelength orlambda. Transponders are available with single or multiple lanes per module. A quadruple transponder,for example, has four client and four WDM network ports per module; typically client and network portshave an equal number. The building blocks of a datacenter optimized transponder design are shown inthe figure below. The transponder takes the weak ‘grey’ signal coming from a client device (e.g. a FibreChannel switch), regenerates the signal and launches it towards the WDM optical stage using a highpower WDM interface. These interfaces can be built in or pluggable. Such transponders usually have amaximum reach of 200 kilometers.

The Control Unit


FIGURE 4 Datacenter optimized transponder design (quadruple transponder)

A typical transponder for Telco/ISP use is shown in below. The design is similar to the datacenteroptimized transponder, but the WDM output signal is standardized for use in a Telco operator’snetwork. Since the mapping procedure into standardized protocols is mandatory, such a device ismuch more complex. Therefore, latency is much higher and MTBF values are lower. Additionally,power consumption is much higher compared to a simple design. The maximum reach for this kind ofmodule is around 200 kilometers without optical-electrical-optical (OEO) conversion.

FIGURE 5 Telco/ISP transponder design (twin transponder)

WDM Muxponders

A muxponder is a hybrid between a TDM (Time Division Multiplexing) multiplexer and a WDMinterface. Thus a muxponder has several client interfaces (usually two to ten) and typically onenetwork interface as shown in the figure below.

FIGURE 6 Datacenter Optimized Muxponder Design (5x Muxponder with AES Encryption)

The TDM algorithm electrically aggregates the incoming client signals into a sum signal that is fed tothe WDM network interface. Like transponders, the muxponders are available in two different varieties- a Telco/ISP design and a more lightweight datacenter-optimized design.

WDM Muxponders


A datacenter optimized muxponder usually consists of a unique design where all parameters could becontrolled by the manufacturer. The use of proprietary techniques like high speed framing allowssupport of special protocols like Infiniband or Fibre Channel with the vendor’s various protocolextensions (Trunking, VSAN, etc).

Highly standardized techniques are used by Telco/ISP muxponders in order to fit seamlessly intocurrent telecommunications networks as shown in the figure below.

FIGURE 7 Telco/ISP Compliant Muxponder Design (10x Muxponder with Protected East/West NetworkInterface)

Using standardized mapping and framing procedures like GFP-T or ODU based mapping, anoptimization towards datacenter focused protocols are quite limited. Thus feeding entirely standardconform protocols might be accepted on the client ports. This could lead to feature loss or unexpectedissues, if you try to use e.g. feature-rich Fibre Channel ISL connections using such card types. Thebenefit of such a design is the support of a much wider range of client protocols like OC3/12/48. Thosemodules are also designed to span thousands of kilometers and are specifically built to interact with 3rd

party telco devices natively.

Recommendation for Transponder/Muxponders

Datacenter optimized transponders and muxponders are the first choice for connecting geographicallydispersed datacenters over distance, since they are low in latency and high in MTBF. This is especiallytrue for Fibre Channel and other latency sensitive protocols.

But if you have to use a Telco’s network, or if you need to have a full standard conform networkinterface like SDH, SONET or OTH, you should use a ISP compliant WDM design. Please keep in mindthat this could limit the features and capabilities of your Fibre Channel network.

The WDM Optical Layer

Optical Multiplexer/De-multiplexer

Optical multiplexer and de-multiplexers are passive optical modules used for combining (multiplexing)and separating (de-multiplexing) optical WDM signals into or out of an optical fiber. Modern port countsstandard for optical systems range from 40 to 120. Those filters are also known as Optical Add DropMultiplexers (OADM). Optical filters operate within a so called optical grid. This grid and thewavelengths used within are standardized by the ITU. Typically WDM systems use a 100GHz or a50GHz grid, where the WDM wavelengths are separated by the grid value. This is also known aschannel spacing.

Recommendation for Transponder/Muxponders


Amplifiers

Optical Amplifiers such as EDFA’s (Erbium Doped Fiber Amplifier) or Raman amplifiers are used toextend the optical WDM signal over longer spans. The maximum single span distance you canachieve is around 200 kilometers. With amplifier chains, you can go over several thousands ofkilometers.

Dispersion Compensation

Dispersion is a fiber based physical effect that spreads the optical pulse while it travels down theoptical fiber. This effect worsens the signal quality and thus limits the maximum distance you canachieve with your optical system. In order to counter this effect, dispersion compensation modules aretypically used. Those are available based on two different flavors:

• Dispersion compensating fiber (DCF) on a reel• Fiber Bragg gratings (FBG)

The differences between those technologies is that the FBG’s are doing the compensation withinsome tenths of a nanosecond whereas the fiber based modules add several 10’s of microseconds toall over link delay. For example, a dispersion compensation for a 100 kilometer link adds 60µs withDCF and 45ns with a FBG based module.

Therefore, FBG-based compensation is preferable for latency sensitive protocols. For example, IBMstipulates that DCG is required for dispersion compensation when implementing WDM connectivity inIBM mainframe environments.

ADVA FSP 3000 WDM PlatformThe ADVA FSP 3000 is a scalable and fully modular active Wavelength Division Multiplexing (WDM)platform specifically designed for large enterprises and service providers requiring a flexible and cost-effective solution that will multiplex, transport and protect high-speed data, storage, voice and videoapplications.

In combination with the unique optical layer design, the FSP 3000 offers complete deploymentflexibility. Up to 120 wavelengths per fiber pair and fully integrated transponder and muxponderoptions ranging from 8Mbit/s to 100Gbit/s optimize the spectral efficiency in the transmission fiber,eliminate fiber exhaust and reduce power and space consumption.

Amplifiers


FIGURE 8 FSP 3000 7HU shelf

For the ADVA FSP 3000, a special set of dedicated enterprise modules are available to enableBrocade’s customers to cost effectively transfer all kinds of storage and server-related traffic up tohundreds of kilometers between their data centers. Additionally, all datacenter optimized modules aredesigned and tested in order to fully support all ISL enhancements, implemented in Brocade’s SANswitch technology (e.g. frame- based trunking, credit buffer recovery, FEC …).

The transparent high-speed transmission technology of the ADVA FSP 3000 eliminates the need forcostly gateways or routers and provides the lowest latency in the industry. Customers benefit from amore reliable Disaster Recovery plan at much lower cost and higher data rates when using the ADVAFSP 3000 between data centers over fiber.

ADVA FSP 3000 Optical Layer

Optical Filter Modules

Optical Filters for WDM are usually fixed or variable. The variable filters are called ROADM(Reconfigurable Optical Add Drop Multiplexer) and are active components which require power tooperate. They are mainly used in rings and/or fully meshed optical networks which require a remoteconfiguration.

Fixed optical filters (also known as FOADM) are purely passive and are used mainly for Point-to-Point(PtP) networks. Usually datacenter interconnect are built this way. Today, FOADM’s come with WDMport counts ranging from 40 WDM channels up to 100 WDM Channels per entity.

In this chapter, only a few modules are shown and explained. The correct choice for the optical layerdesign depends on many parameters such as distance, fiber types, max bandwidth achievable, speedper WDM channel, etc. Therefore, careful and proper planning is mandatory. This is commonly done by

ADVA FSP 3000 Optical Layer


the WDM vendor itself or by specially trained consultants. Such planning will not be discussed withinthis guide.

A typical optical setup for an 80 WDM port system build with two 40 Channel FOADM, an opticalinterleaver and amplifier units is shown below.

FIGURE 9 Optical filter and amplifiers for an 80 WDM channel system using passive optical filters andamplifiers

Please note that all connections are bi-directional. Only one fiber is shown for a clearer scheme. Thefollowing FSP 3000 components are used for this design.

40CSM/2HU A 40 DWDM channel mux-demux. One with odd and one with even channels. Channel spacing is100Ghz

ILM50 A 2 port interleaver module, combining the two 40CSM’s. The ILM combines two 100Ghz gridsinto a 50Ghz Channel Grid sum signal.

EDVA-C-D20 A double stage erbium doped fiber amplifier with up to 20dBm output power

This optical layer configuration is only a small portion of the ADVA FSP 3000 optical layer portfolio. Atany rate, this would be a typical setup for a high bandwidth Point- to-Point network layout.

Optical Amplifiers

The ADVA FSP 3000 comes with a variety of optical amplifiers suited for different use cases within theoptical networking space. Relevant for datacenter networks are the following types:

Optical Amplifiers


EDFA-C-D20 A double stage erbium doped fiber amplifier with up to 20dBm output power. This module could beused for booster and/or preamplifier configuration. A dispersion compensating module could beinserted between the two stages.

EDFA-C-S20 A single stage erbium doped fiber amplifier with up to 20dBm output power. This module would beused for booster configurations.

RAMAN types Raman amplifiers could be used in addition to classic EDFA’s for long, single spans up to 200kilometers without the need of an amplification hut inbetween.

Dispersion Compensation Modules

There is different dispersion compensating modules available for the ADVA FSP 3000, which are eitherDCG or fiber based. We’ll focus on the DCG based modules due to the latency sensitivity of the SANprotocols. Mainly, there are 2 different types available compensating different fiber distances. Thedistance statement which comes with the modules’ name is only valid for G.652 standard single modefiber. For other fiber types like, for example G.655 or TrueWave®, a detailed dispersion calculation ofthe whole optical system is mandatory.

DCG-M DCG based modules with less than 50ns latency for a 100GHz optical grid. Available for 60,80,100kmdispersion load according to a G.652 fiber

DCG50-M DCG based modules with less than 50ns latency for a 50GHz optical grid. Available for20,40,60,80,100km dispersion load according to a G.652 fiber

Protection ModulesA fiber break can cause a complete outage for fiber optical networks. Since the fiber is subject toexternal events out of the control of the data center staff, it’s important to design the WDM system tosurvive fiber failures.

There are several different WDM protection options available. Almost all of these techniques are basedon a Telco operator’s needs and requirements rather than a data center operator. The requirements ofTelco/ISP’s are usually different from the requirements for protecting WDM used for datacenterinterconnect. Therefore, the protection options available might not be as good a fit.

Most of the time datacenter equipment such as Fibre Channel switches require “Client Layer Protection”(CLP) that follows the best practice of using dual fabrics for high availability SAN’s. Two independentWDM systems transporting data over two independent fiber ducts and routes is a proven design chociewhen interconnecting datacenters as shown in the figures below.

FIGURE 10 Dual WDM Optical Transport Network

Dispersion Compensation Modules


FIGURE 11 Remote Switch Module (RSM)

For additional security within these networks, fiber switch modules could be used on top of a CLP-based protection. Typically, the Remote Switch Module (RSM) or the Versatile Switch Module (VSM)would be used for the ADVA FSP 3000. With this combination being used, a double failure protectionis achieved and even if a fiber route has an outage, all services can still operate at one hundredpercent capacity. Those modules consist of an opto-mechanical switch (similar to a relay) whichswitches the light path from a defect long distance fiber to a backup fiber. This is done automaticallyand the client devices only see a short loss of sync or loss of light, depending on the settings of theWDM system. The drawback of such a configuration is that four long distance fiber paths between thetwo sites are mandatory.

Please note that other options are available as well, e.g. multisite protection or optical restorationmight be the option for your connectivity needs. Protecting optical networks and especially theinfluence of optical protection switching on the end devices (e.g., storage, servers) must be analyzedbefore choosing the optical protection scheme.

ADVA FSP 3000 Transponder and Muxponder Module TypesIn order to serve a wide range of applications and customer needs, there are a variety of modules withdifferent functionality and dedicated market segments available. To get a brief overview, our xPondersare divided into three classifications:

• Core modules — xPonders, dedicated to the Telco/ISP market• Access modulesx — Ponders, cost optimized and for metro distances• Enterprise modulesx — Ponders, optimized for datacenter connectivity solutions

The portfolio for datacenter optimized modules is the full set of ‘Enterprise’ modules and a subset ofthe ‘Access’ modules. A more detailed overview is shown in below.

ADVA FSP 3000 Transponder and Muxponder Module Types


FIGURE 12 FSP 3000 xPonder Module Types and Naming Convention

All ADVA FSP 3000 modules follow a similar naming convention. The first letter is an indicator thatseparates the modules into muxponders and transponders. A ‘W’ translates to transponder while a Tstands for muxponder.

The number in front of the triple-letter name show the number of client ports if higher than one. Forexample a 4WCE would translate to a 4 client port transponder for enterprise applications, a 2TCCwould be a 2 client port muxponder intended for Telco/ISP use.

Now let’s focus more on datacenter optimized modules, which are shown in the following table:

FIGURE 13 FSP 3000 Portfolio of Datacenter Optimized xPonder Modules

The technology of the six datacenter-optimized modules are explained more in detail in the nextchapter.

ADVA FSP 3000 – 4TCA Module

The 4TCA module (4TCA-PCN-4GU+4G), shown below, is an access type muxponder which has amaximum network data-rate of 4G. It is equipped with four client ports and two network ports, whichcould be used in parallel or for network protection options.

ADVA FSP 3000 – 4TCA Module


For datacenter applications, usually both network ports are used in parallel to achieve a maximumbandwidth of 8G per module. Thus 2 x 4G, 4x2G or any combination of 1G, 2G and 4G services arepossible.

FIGURE 14 FSP 3000 4TCA Muxponder Module

Specification of the 4TCE-PCN-4GU+4G:

• Services: GbE, 1,2,4G FC, ISC-3 peer (IBM Mainframe signal)• Network and client port SFP based• Built in protection option available• Module is one single slot width• 80 DWDM channels or 16 CWDM channels available

ADVA FSP 3000 – 2WCA Module

The 2WCA module (2WCA-PCN-10G), shown below, is an access type dual transponder with amaximum network datarate of 10G. It is equipped with two client and two network ports which could beused in parallel or for protection options.

For datacenter applications, usually both network ports are used in parallel to achieve a maximumbandwidth of 20G per module. Thus 2 x 4G, 2x8G, 2 x 10G or any combination from 4G, 8G and 10Gservices are possible.

FIGURE 15 ADVA FSP 3000 2WCA dual transponder module

Specification of the 2WCA-PCN-10G:

• Services: 4,8,10G FC; 10GbE; STM-64/OC-192;OTU2• Network and client port XFP based• Built in protection option available• Module is one single slot width• 80 DWDM channels or 8 CWDM channels available

ADVA FSP 3000 – 2WCA Module


ADVA FSP 3000 – 4WCE Module

The 4WCE module (4WCE-PCN-16G), shown below, is an enterprise type quadruple transponder witha maximum network datarate of 16G. It is equipped with four client and four network ports. Fordatacenter applications, usually all ports are used in parallel to achieve a maximum bandwidth of 64Gper module. Thus 4 x 8G, 4x10G, 4 x 16G or any combination from 8G, 10G and 16G services arepossible.

FIGURE 16 ADVA FSP 3000 4WCE quad transponder module

Specification of the 4WCE-PCN-16G:

• Services: 8,16G FC; 10GbE• Network port XFP based and client port SFP+ based• Module is one single slot width• 80 DWDM channels available

ADVA FSP 3000 – 5TCE Module Family

The 5TCE module, shown below, is available in some different HW variants. The two main variants are:

• Standard muxmonder with transponder functionality (5TCE-PCTN-10GU+10G)• AES 256 Encryption muxmonder with transponder functionality (5TCE-PCTN-10GU+AES10G)

This module is an enterprise type muxponder with a maximum network datarate of 10G. It is equippedwith five client ports and one network port. The module could be used as muxponder for client servicessmaller than 10G, like 3x 4G, 5x 2G, etc or any combination. A low-latency transponder model is alsoavailable for 10G based services.

FIGURE 17 ADVA FSP 3000 5TCE muxponder module with AES 256 encryption

Specification of the 5TCE-PCTN-10G+(AES)10G:

• Services: 1,2,4,8,10G FC; 10GbE, 5,10G InfiniBand; ISC-3 (IBM Mainframe)• Client port SFP(+) based, network port tunable, built in

ADVA FSP 3000 – 4WCE Module


• Module is one single slot width• Active roundtrip latency measurement• GFEC available• 120 DWDM channels available

ADVA FSP 3000 – 10TCE Module

The 10TCE module (10TCE-PCN-10G+100G), shown below, is an enterprise type muxponder with amaximum network datarate of 100G. It is equipped with ten client ports and four network ports. Thenetwork ports are combined and cannot be used independently. Each network port is running at 28Gspeed. For datacenter applications, usually all ports are used in order to achieve the maximumbandwidth of 100G per module. All client port combinations of 8G/10G services are possible.

FIGURE 18 ADVA FSP 3000 10TCE 100G Muxponder Module

Specification of the 10TCE-PCN-10G+100G:

• Services: 8G FC; 10GbE, STM+64/OC-192• Client port SFP(+) based, network port CFP based• Module is one double slot width• Enhanced FEC available

ADVA FSP 3000 – WCE-100G Module

The 10TCE module (WCE-PCN-100G), shown below, is an enterprise type transponder with amaximum network datarate of 100G. It is equipped with one client port and four network ports. Thenetwork ports are combined and cannot be used independently. Each network port is running at 28Gspeed.

FIGURE 19 ADVA FSP 3000 WCE 100G muxponder module

ADVA FSP 3000 – 10TCE Module


Specification of the 10TCE-PCN-10G+100G:

• Services: 100GbE LR4, LR10, SR10; OTU4• Client and network port CFP based• Module is one triple slot width• EFEC available

These six modules visualized more in detail are specifically designed to support datacenter applicationsand protocols. All modules capable of transporting Fibre Channel are tested and qualified by variousvendors (e.g., EMC, IBM, Brocade, HP) in order to support the necessary functionalities.

In general, all of these modules can cover unrepeated distances of 200 to 500 kilometers maximum.Unrepeated distances are links, where only pure optical amplification or regeneration is used.Compared to a repeated link, where a repeater takes the optical signal and converts it to the electricaldomain. There, the signal get regenerated and will be converted to an optical signal afterwards. This isalso known as OEO conversion.

Single span links are usually possible up to 200 kilometers on G.652 fiber, depending on the fibercharacteristics, the optical interface and the interface speed.

NOTEMore detailed capabilities of the various FSP 3000 modules can be found in Appendix A.

WDM Error Forwarding SettingsIf you have two devices directly connected via cables, it is easy to troubleshoot if any outage occurs.Trouble with the fiber connections usually causes a loss of sync or a loss of light. Those events arehandled quite differently by a switch, and could lead to data loss and/or fabric separation. Butnevertheless, the problem is quite easy to determine.

When a WDM system is between Fibre Channel switches, error determination will become morecomplex and more difficult. Thus error forwarding and its effect on the Fibre Channel fabric should betaken into account before planning a WDM network. The fabric behavior could also be influenced byseveral settings on the switch/director. For example the portCfgLossTov command could change thebehavior of the fabric in case of a loss of light event.

Error Forwarding Schemes

If a fiber outage or other transport related problem on the WDM network path occurs, as shown below,the client transmitter has to ‘tell’ the attached switch that an error has occurred on the light path. Thiscan be done by either switching the laser off or by sending a special error propagation signal towardsthe client device.

WDM Error Forwarding Settings


FIGURE 20 Error Forwarding for Datacenter-optimized Transponders

For TDM-based modules the implementation is slightly different, as shown below. Please note thatevents like loss of clock, WDM signal degradation or code violations could also trigger the errorforwarding. This functionality is available for all client ports on the FSP 3000 muxponder andtransponder modules.

FIGURE 21 Error forwarding for datacenter-optimized muxponders

There are basically two different settings for error forwarding on the various modules.

• LOS: If an error is detected by the WDM system, the client laser will be switched off to forward thiserror state to the FC switch. The switch will detect a LOL (loss of light) event on its port.

• EPC: If an error is detected by the WDM system , the client transmitter will send an errorpropagation signal to the FC switch. This signal is a special code, generated by the FSP 3000 card.For 8B10B coded signals, it is a 10Bit unrecognized code word with neutral disparity. The switchdetects a signal with errors which leads to a No_Sync state at the switch.



In order to overcome WDM based protection switching events, there is another setting which has to betaken into account:

• Laser off delay: This delays the LOS error forwarding by ~70ms in order to overcome protectionswitching at the WDM domain. Basically, it is a hold off time for the client laser(s). During the hold offtime, there is no valid signal transmitted towards the Fibre Channel switch, just the light staysswitched on.

FIGURE 22 Error Forwarding Settings and Its Effect on the FSP 3000 Client Port

WDM Failover Switching and Error Forwarding

Failover switching within the WDM domain is an approach mainly used by service providers in order toprotect a link in case of a fiber outage. This can be done using various techniques which will not bediscussed here due to their complexity. Only protection using the FSP 3000 RSM module and itspossible impacts will be shown within this guide. Please see Figure 8b.

All of these protection mechanisms have something in common: the switchover time is 50ms or less.This is the time from the event until the light path is fully recovered and a valid and stable signal istransmitted out of the FSP 3000 client ports.

There is also a technique called (optical) restoration, which takes significantly longer. Restoration cantake up to 1 second or more until a valid signal is re-established.

If you do not have WDM failover or WDM protection installed, the effect of the different error forwardingsettings are not relevant. The different client port and therefore fabric behavior is only detectable havinga sub 50ms switchover in place.

WDM Failover Switching and Error Forwarding


FIGURE 23 Error Forwarding Settings and Its effect on The FSP 3000 Client Port During WDMFailover Switching

Fibre Channel Fabric Behavior

The error forwarding settings at the FSP 3000 client ports could have different effects on the fabric.

If LOS is set at the FSP 3000 client port, the switch will detect a loss of light and immediately takedown all of the ports routed over the WDM system. This is also the case if a WDM failover option isinstalled since the sub 50ms switchover is reproduced at the FSP 3000 client laser. To mask thisswitch behavior, there are three options: Setting the port to ‘EPC’, using the FSP 3000 ‘laser off delay’option or using the portCfgLossTov feature on the switch.

Be aware that during a WDM switchover, you always lose data that could lead to higher layer eventslike link resets, buffer credit recovery trigger, etc.

If you have only one WDM system available, where a WDM protection is installed, and all your ISL arerouted this way, the loss of light at all ports simultaneously will lead to fabric separation. But this is onlythe case if these are the only connections between the two parts of a fabric. So masking this seems tobe a good choice.

But by masking this loss of light event, it could lead to buffer credit starvation, or to problems withtrunked ISL’s. Also this could lead to unexpected fabric behavior for example if the ‘normal’ link is10km and the backup path is 50km. So the switch has no way to determine that it is running on a50km ISL after the short loss of sync event.

On the other hand, masking leads to a shorter time between the event and the traffic flow establishedfabric wide.

Some customers prefer having the port taken down completely and starting over again; some like theshort switch event to traffic fully restored time by masking the switchover. Therefore, there is no realright or wrong when it comes to deciding on which error forwarding settings should be used.

Fibre Channel Fabric Behavior


SX and LX Optical InterfacesThere are two types of optical interfaces that connect switches and FSP 3000 WDM systems, which arecalled SX or LX interfaces.

Multi-mode Optical Cables and SX Interfaces

Multimode or SX interfaces are the first choice for most customers since the cost of the interface andthe optical cables are quite low. This is true for Brocade and ADVA Optical Networking components, ingeneral. From an optical signal quality point of view, those SX connections are less stable and have aspeed dependent length limitation. Please see the datasheets of both vendors for distance, speed andfiber. This is also described in detail in Brocade’s SAN Distance Extension Reference.

Single Mode Optical Cables and LX Interfaces

Singlemode or LX transceivers and cables are much more expensive and could range up to 10km or40km with special components. The optical signal has better quality and is less error prone compared toan SX-based installation.

Generally, both options are possible, but both have pro’s and con’s. For most people, a systemconnected via SX components is sufficient and offers a good balance between price and performance.

Nevertheless, for highest speeds like 16G, LX-based infrastructure could lead to a more stable system ifdone properly.

Buffer Credit Calculation and SettingsThe buffer credit technology used by the Fibre Channel protocol will not be discussed within this guide.Please refer to the Fibre Channel specifications or to several guides issued by Brocade.

Buffer Credit Usage and Calculations

The amount of buffer credits needed for a proper and preformat function for each Fibre Channel port isa function of port speed, average frame size and latency between the two switches. The latency is thesum of the latency introduced by the fiber plus the latency introduced by the WDM system. The latencyvalues of the FSP 3000 are shown in Table 1.

As a rule of thumb, you should calculate the minimum buffer credits for your E-ports in the followingway:

• 1G — 0.5BB per km• 2G — 1BB per km• 4G — 2BB per km• 8G — 4BB per km• 10G — 6.5BB per km• 16G — 8.5BB per km

NOTEThe values above are only valid for full sized frames (2112Bytes payload). If your average payload issmaller, you have to increase the amount of BB buffer credits according to your needs. As a bestpractice, you should double the amount of the rule of thumb. Also please be aware that a WDM couldadd some latency (virtual fiber distance equivalent) to your link. This virtual length must be taken intoaccount when calculating the buffer credits. Please see Table 2 below.

SX and LX Optical Interfaces


Buffer Credits for Protected Systems

For WDM systems equipped with protection options, the buffers should always be set to meet thecriteria of the link which has the higher latency. Dynamic buffer credit determination would be possiblebut is not recommended. Please use fixed buffer credit settings instead.

Brocade Switch Port SettingsFor connecting Fibre Channel switches over long distances using WDM, you should begin by buildingyour ports from scratch and adding features after you have a stable connection. The WDM systemmust be set correctly and all connections should be measured before the switch is attached to theWDM’s. Usually the company installing a WDM or the company that provides you with the serviceshould hand over a measurement protocol. From this protocol, you can see if you links are in goodcondition and are running error free.

General Settings

As a best practice, please use the step by step approach as described below:

• ·portCfgDefault; ensure all ports that will be connected to a WDM have a defined configurationbefore applying more port-based settings.

• ·portCfgSpeed; for connections over WDM, the speed should be fixed on all ports. Autonegotiationis not recommended.

• ·portCfgFillword; this command is optional and is not available within all FOS and HW versions ofthe switch. If available, you should set this to 1: -arbff -arbff.

• ·portCfgLongDistance; for this setting you should use the LS option, VC_link_init to 1 and thenspecify the desired distance or desired amount of buffer credits.

After applying those basic settings on both sides of the link, the WDM link should come upimmediately without any problems. If you have trouble, please refer to the troubleshooting sectionwithin this guide.

For more information on the syntax of the FOS commands please refer to the Fabric OS Commandreference for you FOS version loaded on your equipment.

Trunking

Trunking over distance works fine with the cards listed on the Brocade compatibility matrix. However,due to the different latency of the various card types, for trunking you must use only one type ofmodule with the exact same settings for FEC and transmission modes. Also the error forwardingsettings should be set to the same values for all client ports within one trunk.

Advanced Settings

After establishing the basic link, you have to decide whether you need or want to use other port/link/fabric based features.

Feel free to alter the port settings and enable features like FEC, Encryption, Compression and so on.

Buffer Credits for Protected Systems


Dual Fabric Over DistanceMost people use a so called dual fabric approach when they connect their SANs. This methodology isalso usually maintained in long distance connectivity via WDM systems. Depending on the overall SANdesign, there are different solutions possible on what you could be done at the WDM site.

Sample Configurations

The strict approach follows the same rules that you would probably follow if you connect the switchesvia optical cable directly. If the WDM link breaks, all ISL go down immediately and the whole fabriccollapses. All ISL could be trunked since they run down the same light path. The remaining fabric willstay fully intact and no error will occur there. This configuration is shown below.

FIGURE 24 Strict Dual Fabric Over WDM Approach

The mixed approach will give you a different characteristic and behavior. Both fabrics will stay up if oneof the WDM link breaks, but since the different WDM links are never the same length, features liketrunking will not work using both long distance paths. Also in case of a WDM or fiber outage, you willlose half of your connectivity bandwidth on both fabrics. Additionally, both fabrics are affected at oncewhat could lead to problems. However please note the two fabrics will not merge since the links arephysically separated. This configuration is shown below

Dual Fabric Over Distance


FIGURE 25 Figure 22 Mixed Dual Fabric Over WDM Approach

Both scenarios might have added availability through WDM-based protection. This will restore a fiber-based outage in a way that both fabrics will operate at full speed after the WDM failover.

TroubleshootingTroubleshooting a WDM system is quite difficult since it is an analogue technology. For in-depthtroubleshooting, a lot of knowledge, experience and special tools are needed. Hence, do not attemptto work on a WDM system if you are not a trained professional.

Therefore, the focus of this chapter is only related to problems you might experience with WDMsystems in conjunction with Fibre Channel switches.

Bit-Errors

In optical transport systems, a bit error is the most common error. For Fibre Channel datatransmission, the light is amplitude modulated (AM) and transported over optical fiber. Those links willnever be 100% error free. The amount of errors is given by the Bit Error Rate (BER). The minimumBER for Fibre Channel is 1012 as defined by the T11. This is one bit error in 1012 bits, or in otherwords 14 errors per hour at 4G Fibre Channel speed.

Usually WDM systems are much better, and if properly configured and set up, you usually won’t seean error for several days even running at 16G speed. However, if you see increasing error counts onyour ISL’s it could be the WDM long distance link, or the links between the switches and the WDM’s.As shown in the figure below, you have 3 links which could have problems, but they will be seen asone link from the switch perspective.

FIGURE 26 ISL links Between Two Switches via WDM

Troubleshooting


Checking the WDM link is the tricky part. You should check with your service provider (internal orexternal) on the health of the WDM link and the local legs. The WDM system might be helpful on errordetermination, as well.

Checking the ‘local’ links (link one and three) is easier and can be done without checking the WDM. Youshould carefully clean all fiber connectors as well as SFP’s with a fiber cleaning tool and check thatcorrect power levels are set. This can be done using the switch and or the WDM local port readings. Abetter way to check power levels is to use an optical power meter. The optical min and max values forthe interfaces used can be found in the corresponding Brocade and ADVA specification.

Throughput

Usually throughput problems via WDM links are nearly 100% based on wrong buffer credit settings.100% throughput of an ISL is possible only if the BB credits are set correctly. Since the correct setting isdependent on the average frame size, and thus not easy to determine, the buffer credits should becalculated as described above. The use of the FOS command: portStatShow provides a buffer creditzero counter: tim_txcrd_z; which will give you an indication of non-sufficient buffer credits.

If this counter is increasing, you should increase the amount of buffer credits on this particular port.

One additional problem might be buffer credit starvation. Buffer credit starvation occurs if R_RDY’s arelost during data transport. For example, if a bit error occurs, an R_RDY might be lost. This could alsohappen during a WDM failover.

To avoid buffer credit starvation, you might consider using the buffer credit recovery function. Pleaseuse the portCfgCreditRecovery command to enable or disable this feature.

Trunking

Trunking over distance is possible but you have to follow some rules:

• The cables between the switches and the WDMs shall have the same length to avoid deskew.• Use qualified and the same types of WDM modules with exactly the same settings for your trunk.• While using WDM failover, the WDM system should signal a Loss of light event towards the Fibre

Channel switch to bounce the ports. Do not use the LOS_TOV feature. (This is best practice only andmight not be your choice for your special requirements)

Appendix AThis appendix provides ADVA latency and virtual fiber distances for combinations of FSP 3000transponder and muxponder module types and link rates; Brocade Fibre Channel switch port settingsfor transponder and muxponder module types; and error forwarding settings for transponder andmuxponder module types.

LatencyThe overall latency between two switches is caused by the optical cable and the device(s) within thelink.

Since the speed of light is reduced to 200,000km/second in an optical fiber, one meter of fiber is equalto 5ns of latency; one kilometer of fiber is equal to 5µs. For all other devices, you should ask theappropriate vendor for specific latency figures. Please be aware that latency figures, like latency valuesfor the FSP 3000 below, might vary depending on the card settings. For example a forward errorcorrection, quite often used in WDM systems, could add quite a bit of latency if switched on.

Throughput


Table 1 Latency values for FSP 3000 modules per linkTABLE 2

FSP 3000 module / IF Speed 4TCA 2WCA 4WCE 5TCE 5TCE-AES 10TCE WCE-100G

1G FC 4µs N/A N/A 5µs* N/A N/A

2G FC 2.5µs N/A N/A 3.5µs* N/A N/A

4G FC 2µs 10ns N/A 2.5µs* N/A N/A

8G FC N/A 10ns 16ns 1.5µs* 9-15µs** N/A

10G FC N/A 10ns N/A 1.5µs* N/A N/A

16G FC N/A N/A 16ns N/A N/A N/A

10GbE N/A 10ns 16ns 1.5µs* 9-15µs** N/A

40/100GbE N/A N/A N/A N/A 9-15µs** 4.5-10µs**

Table 2 Virtual fiber distances for FSP 3000 modules per linkTABLE 3

FSP 3000 module / IF Speed 4TCA 2WCA 4WCE 5TCE 5TCE-AES 10TCE WCE-100G

1G FC 2km N/A N/A 2.5km* N/A N/A

2G FC 1.3km N/A N/A 1.8km* N/A N/A

4G FC 1km 0.05km N/A 1.3km* N/A N/A

8G FC N/A 0.05km 0.08km 0.8km* 5-8km** N/A

10G FC N/A 0.05km N/A 0.8km* N/A N/A

16G FC N/A N/A 0.08km N/A N/A N/A

10GbE N/A 0.05km 0.08km 0.8km* 5-8km ** N/A

40/100GbE N/A N/A N/A N/A 5-8km ** 2-5km**

Support of Brocade port-based Fibre Channel features

FSP 3000 module /Feature

2WCA 4G/8G

4WCE 8G

2WCA10G 4WCE 16G 5TCE (AES)4G/8G

5TCE(AES)10G

10TCE8G

VC-RDY ✓ ✓ ✓ ✓ ✓ ✓

R-RDY ✓ ✓ ✓ ✓ ✓ ✓

Trunking ✓ ✓ ✓ ✓ ✓ ✓



FEC ** N/A ✓ ✓ N/A ✓ N/A

In-Flight Encryption * ✓ ✓ ✓ ✓ ✓ ✓

In-Flight Compression * ✓ ✓ ✓ ✓ ✓ ✓

In-Flight Enc. + Comp * ✓ ✓ ✓ ✓ ✓ ✓

Table 3 Brocade port-features versus FSP 3000 modules

FSP 3000 module / Errorforwarding feature

4TCA 2WCA 4WCE 5TCE 5TCE-AEStransparentmux

5TCE 5TCE-AESframed mux

10TCE

LOS ✓ ✓ ✓ ✓ ✓ ✓

Laser off delay ✓ ✓ ✓ ✓ ✓ ✓

EPC ✓ N/A N/A N/A ✓ N/A

Table 4 Error forwarding settings available at the FSP 3000 modules



Data Center Solution- Storage - docs.broadcom.com€¦ · 53-1003608-01 31 October 2014 Data Center...

Documents

Transcript of Data Center Solution- Storage - docs.broadcom.com€¦ · 53-1003608-01 31 October 2014 Data Center...