DCB Reqs Design

Storage Infrastructure and Solutions Engineering Dell Product Group

January 2013

Data Center Bridging: Standards, Behavioral Requirements, and Configuration Guidelines with Dell EqualLogic iSCSI SANs A Dell EqualLogic Technical White Paper

2 BP1044 | Data Center Bridging with Dell EqualLogic iSCSI SANs

Revisions

Date Description

February 2013 Updated Figure 6 and added a note on page 27.

© 2013 Dell Inc. All rights reserved. Reproduction of this material in any manner whatsoever without the express

written permission of Dell Inc. is strictly forbidden. For more information, contact Dell.

Dell, the Dell logo, and the Dell badge, PowerConnect™, EqualLogic™, PowerEdge™, Force10™, and PowerVault™

are trademarks of Dell Inc. Broadcom® is a registered trademark of Broadcom Corporation. Intel® is a registered

trademark of Intel Corporation in the U.S. and other countries. Microsoft®, Windows®, Windows Server®, and Active

Directory® are either trademarks or registered trademarks of Microsoft Corporation in the United States and/or other

countries.


Table of contents Acknowledgements .......................................................................................................................................................................... 6

Feedback ............................................................................................................................................................................................ 6

Executive summary .......................................................................................................................................................................... 6

1 Introduction ................................................................................................................................................................................ 7

1.1 Audience ........................................................................................................................................................................... 7

1.2 Terminology ..................................................................................................................................................................... 7

1.3 Scope ................................................................................................................................................................................ 8

2 Converged network infrastructure ......................................................................................................................................... 9

3 DCB technologies, requirements, and configuration ....................................................................................................... 12

3.1 DCB technologies ......................................................................................................................................................... 12

3.1.1 Enhanced Transmission Selection ............................................................................................................................. 12

3.1.2 Priority-based Flow Control ........................................................................................................................................ 13

3.1.3 Application priority configuration .............................................................................................................................. 14

3.1.4 Data Center Bridging Exchange Protocol ................................................................................................................ 14

3.1.5 Congestion Notification .............................................................................................................................................. 15

3.2 DCB concepts ................................................................................................................................................................ 15

3.3 DCB parameter configuration .................................................................................................................................... 17

3.3.1 End-device auto configuration .................................................................................................................................. 17

3.3.2 Switch to switch configuration ................................................................................................................................... 18

3.4 Converged network example ..................................................................................................................................... 18

3.5 DCB requirements for EqualLogic iSCSI storage .................................................................................................... 20

3.5.1 Dell switch models ....................................................................................................................................................... 21

3.6 DCB configuration guidelines for EqualLogic Storage........................................................................................... 21

4 Network configuration and deployment topologies......................................................................................................... 24

4.1 General configuration process flow .......................................................................................................................... 24

4.2 Deployment topologies ............................................................................................................................................... 30

4.2.1 Single layer switching with blade servers ................................................................................................................. 30

4.2.2 Single layer switching with rack servers .................................................................................................................... 31

4.2.3 Dual layer switching ..................................................................................................................................................... 31

5 Component behavior with no DCB or partial DCB capability ......................................................................................... 34

5.1 Storage array behavior with no or partial DCB capability on the switch ............................................................ 34

5.2 Initiator behavior with no or partial DCB capability on switch ............................................................................. 35

5.2.1 Initiator with no DCB capability or partial DCB capability ..................................................................................... 35


6 Advanced DCB considerations, parameter resolution, and behavior ............................................................................ 36

6.1 Peer DCB parameter states ......................................................................................................................................... 36

6.2 VLAN for iSCSI ............................................................................................................................................................... 36

6.3 Switch requirements and behavior ............................................................................................................................ 37

6.3.1 DCBX version resolution .............................................................................................................................................. 37

6.3.2 ETS – Additional clarification ...................................................................................................................................... 37

6.3.3 PFC – Additional clarification ..................................................................................................................................... 38

6.4 NIC/CNA ......................................................................................................................................................................... 38

6.4.1 PFC and application priority parameter resolution ................................................................................................. 38

6.4.2 ETS parameter resolution ............................................................................................................................................ 38

6.4.3 iSCSI application priority tagging ............................................................................................................................... 39

6.5 EqualLogic iSCSI storage ............................................................................................................................................. 39

6.5.1 DCBX version resolution .............................................................................................................................................. 39

6.5.2 DCB enablement ........................................................................................................................................................... 39

7 Conclusion ............................................................................................................................................................................... 40

A Lossless queue implementation............................................................................................................................................ 41

A.1 Force10 MXL switch or Force10 S4810 switch ........................................................................................................ 41

A.2 PowerConnect 8132F switch ...................................................................................................................................... 41

B Switch DCB verification commands ..................................................................................................................................... 42

B.1 Force10 MXL switch or Force10 S4810 switch ........................................................................................................ 42

B.2 PowerConnect 8132F switch ...................................................................................................................................... 43

C Broadcom 57810S configuration and verification ............................................................................................................. 43

C.1 Broadcom 57810S DCBX willing mode configuration ........................................................................................... 43

C.2 Broadcom 57810S iSCSI VLAN configuration .......................................................................................................... 44

C.3 Broadcom 57810S DCB configuration verification ................................................................................................. 45

D EqualLogic storage .................................................................................................................................................................. 46

D.1 EqualLogic 10GbE storage DCB iSCSI VLAN configuration .................................................................................. 46

D.2 EqualLogic 10GbE storage DCB configuration verification .................................................................................. 47

D.3 Adding a new member to a DCB enabled group or create a group in which DCB will be used .................. 48

D.3.1 Specifying the DCB VLAN ID when creating or expanding a group .................................................................... 48

D.4 Warning message when switch ports are not fully DCB capable ........................................................................ 49

E DCB standards history ............................................................................................................................................................ 50

E.1 DCB history and versions............................................................................................................................................. 50

E.2 DCBX version differences ............................................................................................................................................ 51


Additional resources ....................................................................................................................................................................... 52


Acknowledgements

This whitepaper was produced by the PG Storage Infrastructure and Solutions of Dell Inc.

The team that created this whitepaper:

Ananda Sankaran and Margaret Boeneke

We would like to thank the following Dell team members for providing significant support during

development and review:

Tony Ansley, Steve Williamson, Kirt Gillum, and Mark Reinke

Feedback

We encourage readers of this publication to provide feedback on the quality and usefulness of this

information by sending an email to [email protected].

Executive summary

This white paper includes:

• An overview of the current Data Center Bridging (DCB) standards, purpose, terminology, and

guidelines for end-to-end operation

• DCB requirements and configuration guidelines for converged network deployment scenarios with

EqualLogic iSCSI storage

• Sample network configuration steps for a subset of component switches and topologies

• Discussion of interoperability behavior for implementation scenarios when certain components in

the infrastructure are not fully DCB capable

• Advanced considerations and clarification on DCB standards and behavior

[email protected]

mailto:[email protected]?subject=BP1044 - Data Center Bridging with Dell EqualLogic iSCSI SANs


mailto:[email protected]

mailto:[email protected]?subject=BP1044%20-%20Data%20Center%20Bridging%20with%20Dell%20EqualLogic%20iSCSI%20SANs



1 Introduction Data center infrastructures can be complex due to the hardware and software technologies involved in

hosting, managing, and enabling access to applications and data. Also challenging is the growing need to

host and scale more applications and data without significantly increasing the associated infrastructure.

Efficient usage of data center resources becomes critical to reduce costs and manage complexity.

A key data center resource is the network infrastructure that interconnects various devices. These devices

include server hardware systems that host enterprise applications and storage systems that host

application data. Data Center Bridging (DCB) enables sharing the same network infrastructure between

multiple traffic types such as server application traffic and storage data traffic. DCB provides network

resource sharing mechanisms including bandwidth allocation and lossless traffic. This allows converged

network implementations to carry multiple traffic types on the same network infrastructure with fair

resource sharing. This white paper covers I/O convergence using DCB for Dell™ EqualLogic™ iSCSI

storage.

1.1 Audience This white paper is intended for network administrators, storage/SAN administrators, SAN designers,

storage or network consultants, or anyone who is tasked with designing and implementing a DCB

converged I/O network with EqualLogic PS Series iSCSI storage. It is assumed that all readers have

experience in designing and/or administering a shared storage solution with a SAN. Also, there are

assumptions made in terms of familiarity with all current Ethernet standards as defined by the Institute of

Electrical and Electronic Engineers (IEEE) as well as TCP/IP and iSCSI standards as defined by the Internet

Engineering Task Force (IETF).

1.2 Terminology 10GbE or 10GE 10 Gigabit Ethernet

BCN Backward Congestion Notification

CEE Converged Enhanced Ethernet

Note: CEE is used in this document to refer to the DCB Capability Exchange Protocol Base Specification Rev 1.01 along with related PFC and priority grouping/bandwidth allocation specifications. The base specification was submitted to the IEEE DCB task group, which was later published as a set of IEEE specifications covering DCB technologies.

CNA Converged Network Adapter

CoS Class of Service, which is a 3-bit field in the VLAN tag (Also known as Priority Code

Point) as per IEEE 802.1Q standard

DCB Data Center Bridging

DCBX Data Center Bridging Exchange Protocol

ETS Enhanced Transmission Selection


FCoE Fibre Channel over Ethernet

FCoE TLV Refers to support of FCoE protocol in the application priority TLV

iSCSI Internet Small Computer System Interface

iSCSI TLV Refers to support of iSCSI protocol in the application priority TLV

LAN Local Area Network

LLDP Link Layer Discovery Protocol (IEEE 802.1AB)

NIC Network Interface Card

OS Operating System

PCP Priority Code Point, which is a 3-bit field in the VLAN tag (Also known as Class of

Service)

PFC Priority-based Flow Control

PG Priority Group (CEE version term; Also known as Traffic Class in IEEE version)

QCN Quantized Congestion Notification

SAN Storage Area Network (A dedicated Fibre Channel or iSCSI network for storage traffic)

TC Traffic Class (IEEE version term, Also known as Priority Group in CEE version)

TLV Type Length Value (Part of LLDP protocol frame)

1.3 Scope The scope of this white paper is limited to:

• DCB standards and how they apply to EqualLogic iSCSI storage

- DCB considerations for other storage protocols such as FCoE and server side protocols such as

IPC is not within the scope of this document

• Discussion of CEE and IEEE DCB versions

- CEE in this document refers to the DCB Capability Exchange Protocol Base Specification Rev

1.01 along with related PFC and priority grouping/bandwidth allocation specifications


2 Converged network infrastructure It is typical to dedicate a separate physical network infrastructure for SAN traffic to guarantee the

infrastructure for bandwidth and performance as shown in Figure 1. The dedicated SAN is comprised of

network equipment selected for the storage protocol chosen. For example, if Fibre Channel (FC) is the

storage protocol chosen, then a dedicated SAN comprised of FC switches is deployed to inter-connect

the server and storage systems. Also, the server and storage systems are configured with FC Host Bus

Adapters (HBAs) to communicate on the SAN. If iSCSI is the storage protocol chosen, then it is a best

practice to dedicate a SAN comprised of Ethernet switches for server and storage inter-connection. In this

case, the server and storage systems are configured with dedicated NICs to communicate on the iSCSI

SAN.

Figure 1 Dedicated SAN infrastructure


A converged network includes carrying both SAN and other network traffic such as server LAN on a single

network infrastructure as shown in Figure 2. Data Center Bridging (DCB) standards are enhancements to

Ethernet (IEEE 802.1 Bridge) specifications to support multiple protocols and applications in the data

center. The standards support converged network implementations to carry multiple protocols on a single

physical Ethernet network infrastructure. Storage protocols including iSCSI and FCoE can be converged

with non-storage based server LAN traffic allowing the network ports and the inter-connecting links to

carry multiple protocols. A DCB enabled converged Ethernet infrastructure includes the NIC/CNA on the

end-devices (servers and storage arrays) along with the switching infrastructure.

Figure 2 Converged network infrastructure


Sharing the same Ethernet infrastructure for multiple traffic types requires a fairness mechanism to provide

characteristics such as bandwidth allocation and flow control for each type of traffic. Additionally, FCoE

based storage traffic requires lossless behavior for operation on the network to avoid dropped frames.

Minimal loss or lossless behavior also becomes a necessity for iSCSI traffic under heavy load to avoid

retransmissions caused by dropped frames and to reduce network congestion. DCB enables this

bandwidth allocation and lossless behavior for storage traffic when the same physical network

infrastructure is shared between storage and other traffic. Without such a fairness mechanism, it was

typical to dedicate a separate SAN. DCB enables iSCSI SANs or FCoE SANs or both to converge with

regular server LAN traffic on the same physical infrastructure to increase operational efficiency, constrain

costs, and ease network management.


3 DCB technologies, requirements, and configuration This section provides an overview of the DCB technologies and explains DCB elements for an application

scenario. It also provides the DCB requirements for operation with EqualLogic iSCSI storage and highlights

configuration guidelines.

3.1 DCB technologies The key DCB technologies are discussed below with their overall purpose.

3.1.1 Enhanced Transmission Selection Enhanced Transmission Selection (ETS) allows a network port, which can be a NIC/CNA port or a switch

port, to allocate a percentage of available link bandwidth to different traffic types and schedule them for

transmission on the link by their allocation. ETS allows the link bandwidth to be shared among multiple

traffic types. The network traffic type is classified using the priority value in the VLAN tag of the Ethernet

frame. The priority value is the Priority Code Point (PCP), which is described in the IEEE 802.1Q

specification and has a 3-bit field in the VLAN tag that has eight possible values from 0 through 7.

A set of priorities form a traffic class or priority group. Traffic classes or priority groups are allocated

bandwidth as a percentage of available link bandwidth. Figure 3 describes a network port with three traffic

classes or priority groups – LAN, SAN, and IPC with 40%, 50%, and 10% of link bandwidth allocated

respectively. All Ethernet frames carrying LAN related traffic are marked with one of these priorities: 0, 1, 2,

5, 6, and 7 in the VLAN tag. Ethernet frames carrying SAN traffic are marked with priority 4, and IPC traffic

is marked with priority 3 in the VLAN tag. ETS allows unused bandwidth to be consumed by other classes

dynamically. All traffic priorities within a traffic class or priority group have the same characteristics in

terms of bandwidth and frame loss.


Figure 3 ETS Illustration of bandwidth allocation

3.1.2 Priority-based Flow Control Priority-based Flow Control (PFC) provides the ability to individually pause priorities of tagged traffic and

facilitates providing lossless or no drop behavior for a priority at the receiving port. Lossless behavior,

when implemented end to end on a network, avoids dropping frames during congestion by pausing traffic

types using PFC. Each frame transmitted by a sending port is tagged with a priority value (0 to 7) in the

VLAN tag. Traffic pause and resume functionality can be enabled between peer ports for each priority

independent of other priorities transmitted on a link. A receiving port can request a sending port to pause

traffic on a particular priority if PFC is enabled for that priority and continue traffic flow for other priorities.

In figure below, the receiver is pausing the traffic for priority 4 while allowing traffic for other priorities to

flow through the link. PFC is typically invoked during congestion scenarios by the receiving port when

traffic flows cause high buffer use. This avoids the necessity to drop frames.


Figure 4 PFC pausing traffic for priority 4

3.1.3 Application priority configuration Application priority is a DCB configuration construct which defines the mapping of an application or

protocol to a priority (PCP) value. All frames belonging to a particular application or protocol are tagged

with a priority value (0 to 7) which in turn gets mapped to a traffic class or priority group. This mapping

provides dedicated bandwidth for a protocol and also facilitates PFC with lossless behavior for a protocol

across the network. The application or protocol is identified by TCP/UDP port numbers or by an Ethertype

number. For example, iSCSI protocol is identified by TCP port 3260 and FCoE protocol is identified by

EtherType 0x8906. Protocols identified by this method can be mapped to a priority value using the

application priority feature. For example, iSCSI identified by TCP port number 3260 can be mapped to

priority 4. An end device which is the source of a frame carrying iSCSI payload marks the frame with this

priority 4 in the VLAN tag. All devices in the network forward the frame with this priority value and process

it with the same configured PFC lossless behavior and PG/TC bandwidth settings for this priority. The

default priority value is 0 and any network traffic not assigned a priority value using the application priority

feature will have priority 0 marked in the frame. This priority is mapped to a default priority group/traffic

class on the network.

3.1.4 Data Center Bridging Exchange Protocol Data Center Bridging Exchange Protocol (DCBX) enables two peer ports across a link to exchange and

discover ETS, PFC, and other DCB configuration parameters such as application priority and congestion

notification. ETS parameters include the number of traffic classes or priority groups, allocated bandwidth

percentage, and the priorities mapped to each of them. PFC parameters include which priorities are PFC

enabled and which are not. Application priority configuration parameters include the application priority

mapping indicating which application or protocol maps to a certain priority. DCBX leverages functionality

provided by Link Layer Discovery Protocol (LLDP) defined in the IEEE 802.1AB specification to transmit


DCB configuration parameters. The type-length-value (TLV) fields within the LLDP frame format are used

for conveying DCB configuration parameters between peer ports.

DCBX allows mismatch detection and remote configuration of DCB parameters. Each peer port on a link

initially advertises its locally configured DCB parameters. It then verifies the advertised configuration

parameters from the remote peer and detects mismatches. DCBX also allows remote configuration of a

peer port with the willing mode feature. A port may advertise that it is willing to apply DCB configuration

parameters from a peer port. The advertised configuration of the peer port is then accepted as the local

configuration for operation. For example an end device such as a server CNA or a storage port in willing

mode accepts and operates with the switch recommended DCB parameters. Typically most server CNAs

and storage ports are in willing mode by default. This enables the network switches to be the source of

DCB configuration. This model of DCB operation with end devices in willing mode is recommended and

prevalent due to ease of configuration and operation by minimizing mismatches.

3.1.5 Congestion Notification Congestion Notification (CN) provides end-to-end flow control in a Layer 2 network to eliminate

sustained, heavy congestion due to long-lived traffic flows. A congestion point (network switch port or

end-device port) can send information to a reaction point (end-device port) to throttle frame transmission.

This allows traffic flows to be throttled at the source to react or avoid congestion in the network. This

feature is not implemented today on most switches, NICs/CNAs, and storage devices. This feature has

been viewed by vendors in the industry as not required for small scale network deployments with fewer

hops.

3.2 DCB concepts This section discusses the DCB terminology used in standards. There are primarily three progressive

versions of DCB: CIN DCBX, CEE DCBX or baseline DCBX, and IEEE DCB. Appendix E provides detailed

information on the history, evolution of these versions, and their differences. The overall functionality is

similar across the versions for enabling a converged network that carries multiple traffic types. The table

below lists some of the key terminology in the DCB standards along with the meaning of the terms.

Note: CEE in this document refers to the DCB Capability Exchange Protocol Base Specification Rev 1.01

along with related PFC and priority grouping/bandwidth allocation specifications. This version is also

referred to CEE DCBX or baseline DCBX or base protocol since most implementations started with this as

a stable version. The IEEE DCB specifications were developed later. This document covers only CEE and

IEEE DCB functions with their associated terminology.

Table 1 Key terminology

Term Meaning

Discovery and Capability Exchange Protocol or Data Center Bridging Exchange Protocol (DCBX)

This specification provides discovery, parameter exchange, mis-configuration detection, and configuration of peers.


Term Meaning

Willing Bit Also referred to as Willing Mode.

A Boolean value in the DCBX feature TLV(s) that indicates that the device port is able to accept DCB configuration from a peer port.

Priority Legacy terminology: Class of Service (CoS)

Priority Code Point (PCP) field (3 bits) in the VLAN tag as described in IEEE 802.1Q. PCP has eight possible values of priority from 0 through 7. Value 7 has highest priority and value 0 has lowest priority. Also referred to as Class of Service (CoS) value based from legacy IEEE 802.1p task group.

Priority-based Flow Control (PFC)

Ability to pause traffic for each priority (PCP value). For example, frames with a particular priority value in the VLAN tag can be selectively paused allowing frames with other priority values to continue flow. PFC can be enabled for certain priority values and disabled for others. PFC facilitates lossless behavior by avoiding frame drops.

Priority Group (PG) or Traffic Class (TC) One or more priorities are mapped to a priority group or traffic class, and it has bandwidth allocated as a percentage of available link bandwidth. It can be viewed as a queue or virtual link with allocated bandwidth. All priorities within a priority group or traffic class share the same allocated bandwidth and are expected to have similar requirements with regards to bandwidth and frame loss. Some implementations refer to prioritized (PCP tagged) traffic as a ‘traffic class’ but this should be referred to as CoS.

Transmission Selection Algorithm (TSA) Present in IEEE DCB only to select a scheduling algorithm for a traffic class. Provides three types of TSA – strict priority, credit shaper, and Enhanced Transmission Selection (ETS) along with vendor specific algorithms. CEE DCB always uses ETS for transmission.

Enhanced Transmission Selection (ETS) Scheduling algorithm or mechanism for outgoing traffic on a port. Provides bandwidth allocation per PG/TC, and unused bandwidth distribution across PGs/TCs.

Application Protocol or Application Priority Defines application mapping to a particular priority value. All frames belonging to a particular application can be tagged with a priority value 0 to 7 which in turn gets mapped to a traffic class or priority group. The application is identified by TCP/UDP port numbers (for example, iSCSI ) or by Ethertype (for example, FCoE). “iSCSI TLV” is a term often used to refer to support of the iSCSI protocol in the application priority TLV.


3.3 DCB parameter configuration The key DCB technologies discussed in section 3.1 are applicable peer to peer on an Ethernet link

between two ports. A link comprised of peer ports may be between a NIC/CNA and a switch, between two

switches, or between a switch and a storage device. A successful converged network deployment based

on DCB includes proper configuration and operation of DCB parameters on all peer device ports that carry

converged traffic. DCB parameters include PFC, ETS, and application priority mapping as described in

section 3.1.4.

3.3.1 End-device auto configuration The DCBX willing mode on end devices like server NIC/CNA and storage enables automatic configuration

of end devices and minimizes mismatches on DCB parameters. In this model of configuration, network

switches are configured with the required DCB parameters and they advertise the configuration via DCBX

to attached end devices. The end devices operating in willing mode learn the DCB configuration

information from DCBX and operate with those parameters. As discussed in section 3.1.4 the TLV fields

within the LLDP protocol are used by DCBX for communicating the DCB parameters. The DCBX TLV

exchange between peer ports is illustrated in Figure 5, Figure 6, and Figure 7 below. It is a simple example

showing only one port per device to illustrate the DCBX exchange. The server CNA and storage ports are

in willing mode (willing = 1) to accept DCB parameters from switch port. The switch ports are configured

with DCB parameters as per deployment requirements and operate with the willing mode off (willing = 0)

as shown in Figure 5. In the figures, DCBX TLVs include PFC TLV, PG/ETS TLV, and application priority TLV.

Figure 5 Willing mode state of switch, CNA, and storage ports

Figure 6 DCBX TLVs exchange between peer ports on the operational link

Figure 7 CNA and storage port accept and confirm switch DCB parameters


In step 2 depicted in Figure 6, peer ports communicate their local DCB configuration via DCBX and their

willing mode. The server CNA and storage ports initially have a default local DCB configuration that they

communicate. In step 3 depicted in Figure 7, they apply the switch advertised configuration locally and

communicate the same parameters back to the switch.

Mismatches do not occur when the server NIC/CNAs and the storage devices support all the requisite

parameters configured on the switch. This can be ensured by deploying only components that meet the

converged network design requirements.

3.3.2 Switch to switch configuration A DCB enabled network typically consists of multiple switches for redundancy and scalability. There are

two possible ways to configure the network switches with the required DCB parameters. Both these

methods are discussed in detail in section 4.1. The methods are:

• Configuration propagation: DCB parameters are manually configured on two or more switches

(source switches) and they advertise the configuration via DCBX to other switches in the network.

Other switches have a certain number of ports facing the source switches called upstream ports.

These ports operate in DCBX willing mode, learn the DCB parameters from an upstream device, and

internally propagate the DCB configuration to other ports (downstream ports) on the switch. The

downstream ports apply the internally propagated configuration and advertise the same to peer

devices.

• Manual configuration: All switches on the network are manually configured with the same DCB

parameters. The switch ports are configured with willing mode turned off and they only advertise

DCBX configuration to peer devices.

3.4 Converged network example This section discusses a sample converged network implementation using DCB. The various DCB

parameters (PFC, PG/TC, and application priority) are illustrated in the context of a deployment scenario.

The scenario uses the hypothetical requirements given below:

• A virtualized server environment that has requirements to converge both LAN and SAN traffic on the

same physical Ethernet infrastructure (Server adapters and Ethernet switches).

• Server NICs/CNAs and Ethernet Switches are 10GbE.

• LAN traffic segregation requirements on virtualized servers:

- Separate logical network for hypervisor LAN traffic with bandwidth allocated for all LAN based

traffic.

- All LAN traffic can be lossy from a Layer 2 perspective. Lossy implies that the switches and

adapters can discard frames under congestion. Upper layer protocols such as TCP will perform

recovery of lost frames by re-sending data.

• SAN traffic requirements on virtualized servers:

- A separate logical network for iSCSI SAN traffic with dedicated bandwidth allocation.

- All SAN traffic must be lossless from a Layer 2 perspective (PFC enabled).

As discussed in section 3.3.1, the server CNAs are in DCBX willing mode and network switch ports are not

in willing mode per their default states.


Figure 8 illustrates the configuration values for various DCB parameters (PFC, PG/TC, and Application

Priority) for this deployment scenario and their relationship. The table columns in Figure 8 that are

highlighted in blue are descriptive text that is not part of the actual DCB parameters. The network switch

ports are configured with the DCB parameters to support the requirements listed above and advertised to

peer device ports, as follows:

• The first step is to assign the priority value for each traffic type by using the application priority mapping. This information is advertised as part of the DCBX Application TLV. The Application priority mapping table in the figure shows the priority mapping for iSCSI protocol. Priority 4 is the industry accepted value for iSCSI protocol and implies that all frames tagged with priority 4 carry iSCSI as payload. To ensure this, the CNA and storage must support iSCSI protocol in the application priority TLV and tag outgoing iSCSI frames with the configured priority. All other un-configured traffic types are mapped to default priority 0. In this example, LAN traffic is mapped to default priority 0 since it is not explicitly configured.

• The next step is to enable PFC (to enable lossless behavior) for iSCSI traffic on priority 4 and disable PFC for LAN traffic on priority 0 since it can be lossy. An example of priority values along with the PFC requirement is shown in the PFC table. The PFC enabled/disabled information for the priorities is advertised as part of the DCBX PFC TLV.

Figure 8 DCB parameters

• The priority groups or traffic classes are the final step. The PG/TC table in the figure shows the priority group or traffic classes and the respective bandwidth percentage assignment. The Priority to PG/TC mapping table shows the priorities that are mapped to priority groups or traffic classes. This information is advertised as part of the DCBX ETS or PG TLV. It must be noted that multiple priorities can be mapped to the same PG or TC. In that case, all those priorities will share the same bandwidth allocation and will not have dedicated bandwidth. In this example, we mapped iSCSI priority to its own PG/TC to provide dedicated bandwidth.


3.5 DCB requirements for EqualLogic iSCSI storage This section covers the minimum DCB requirements for use with EqualLogic iSCSI storage. The

operational model is that switches use non-willing DCBX mode while server NICs/CNAs and storage ports

operate in willing mode. This is the default behavior on most switches and NICs/CNAs. The DCB

parameters are then configured in the switch devices and learned by the end-devices. The minimum

application priority, PFC, and ETS requirements for EqualLogic iSCSI storage are given in table below.

Table 2 DCB requirements

DCB feature Requirement

Application priority

Support for iSCSI protocol in the application priority TLV is required. Typical industry accepted priority value for iSCSI is 4, but this can be set to other priorities Switch: Must support configuration of a priority value for iSCSI protocol and advertisement to peer ports Server NICs/CNAs: Must support iSCSI protocol in application priority (learned from the switch) and must support tagging outgoing iSCSI frames with this priority

ETS Requires a dedicated traffic class or priority group for iSCSI priority Switch: Configure dedicated TC/PG for iSCSI priority with allocated bandwidth Server NICs/CNAs: Adhere to TC/PG mapping for iSCSI priority (learned from the switch)

PFC Requires enabling PFC (no drop or lossless behavior) for iSCSI priority Switch: Configure PFC enable for iSCSI priority Server NICs/CNAs: Adhere to PFC enable for iSCSI priority (learned from the switch)

If the switch or NIC/CNA cannot support the above minimum requirements, then it should not be

considered for an end-to-end converged I/O deployment with DCB and EqualLogic storage. It is

important to remember that switches and NIC/CNAs must support lossless or no drop queue semantics

for supporting PFC. This enables iSCSI frames with PFC enabled to be lossless and prevents dropped

frames.

The EqualLogic Compatibility Matrix (ECM) lists the switches and NICs/CNAs that have been validated with

EqualLogic Storage. It also indicates DCB support along with version levels for switches and NICs/CNAs.

DCB standards are supported with NIC/CNA, switch, and storage hardware that are 10GbE capable or

above.

For more information, see the EqualLogic Compatibility Matrix (ECM) at:

http://en.community.dell.com/techcenter/storage/w/wiki/2661.equallogic-compatibility-matrix.aspx



3.5.1 Dell switch models Dell switch models that support the required DCB requirements are listed in Table 3 below. Refer to

appendix A.1 for more information on lossless queue implementation on Dell switches.

Table 3 Dell switch DCB support

Switch model DCB capability

Force10™ S4810 DCB versions: IEEE, CEE, CIN, and auto mode ETS classes or PGs: 4 Port egress queues: 4 Lossless queues: 2 (Enabled automatically with PFC) iSCSI application priority support: yes (default: iSCSI mapped to priority 4) DCBX propagation: Manual, Upstream/downstream propagation

Force10 MXL DCB versions: IEEE, CEE, CIN, and auto mode ETS classes or PGs: 4 Port egress queues: 4 Lossless queues: 2 (Enabled automatically with PFC) ISCSI application priority support: yes (default: iSCSI mapped to priority 4) DCBX propagation: Manual, Upstream/downstream propagation

PowerEdge™ M IO Aggregator

DCB versions: IEEE, CEE, CIN, and auto mode ETS classes or PGs: 4 Port egress queues: 4 Lossless queues: 2 (Enabled automatically with PFC) ISCSI application priority support: yes (default: iSCSI mapped to priority 4) DCBX propagation: Manual, Upstream/downstream propagation. Uplinks auto configured as auto-upstream ports Server ports auto configured as auto-downstream ports

PowerConnect™ 8100 Series

DCB versions: IEEE, CEE, CIN, and auto mode ETS classes or PGs: 3 (referred to as traffic class groups) Port egress queues: 7 Lossless queues: 2 (Enabled automatically with PFC) iSCSI application priority support: yes (default: iSCSI mapped to priority 4) DCBX propagation: Manual, Upstream/downstream propagation

3.6 DCB configuration guidelines for EqualLogic Storage The first step in a deployment is determining the converged network design requirements and mapping

them to DCB configuration parameters. These parameters encompass the minimum DCB requirements in

section 3.5 and additionally include other DCB requirements for a converged network deployment. Then

these DCB parameters must be configured on the network switches. These DCB parameters will be

discussed in the context of the deployment scenario laid out in section 3.4 containing two traffic types –

iSCSI (which is lossless) and all other server LAN traffic. The following considerations apply when

determining the DCB parameters to be configured on switches:

• Application priority

- Are there other protocols besides iSCSI for which application priority support is needed? For

example: FCoE. If so, application priority mapping must be configured and advertised for all

protocols including iSCSI. In the deployment scenario, only the iSCSI application priority needs


to be configured and advertised, which is typically 4. All other traffic, by default, will map to

priority 0.

• PFC

- How many priorities will have PFC (lossless behavior) enabled? Typically each application

priority configured may need PFC enabled. iSCSI priority must be enabled with PFC as detailed in

the requirements in section 3.5. In the deployment scenario, PFC (lossless behavior) needs to be

enabled for only iSCSI application priority 4 and PFC should be disabled for all other priorities (0,

1, 2, 3, 5, 6, 7).

• ETS

- How many traffics classes / priority groups are needed? Each application priority configured

may need to map to a dedicated TC/PG for dedicated bandwidth. All other priorities may map to

a separate TC/PG. iSCSI priority must be mapped to a dedicated TC/PG as detailed in the

requirements in section 3.5. In the deployment scenario, two TCs/PGs are needed: Map iSCSI

priority 4 to a dedicated PG (PG1) and all other priorities (0, 1, 2, 3, 5, 6, 7) map to another PG

(PG2).

- What is the bandwidth percentage allocation for each TC/PG? In the deployment scenario

PG0 will be allocated with 50% and PG 1 will also be allocated 50%. All other PGs/TCs available

will have 0% configured.

It must be ensured that the switches and server NICs/CNAs chosen for a deployment meet the overall

DCB parameters required. For the sample deployment scenario, they must support iSCSI application

priority, at least two TCs/PGs for ETS, and enabling PFC for at least one priority. If a server CNA does not

meet the parameter requirements, then parameter mismatches may happen between the CNA and switch

port. In this case you should consider a different CNA that supports these parameter requirements or re-

consider the converged network design goals. DCB parameter resolution process on mismatches is

discussed in section 6. However you should consider only deploying components that support all the

required parameters since resolved parameters may not meet the original objectives of the converged

network or may impact functional behavior. Also certain mismatches may require manual configuration on

the NICs/CNAs for resolution and may lead to errors.

Once determined, the DCB parameters should be configured on the switches and then they are

automatically advertised to end-devices (server NIC/CNA and storage ports) via DCBX. Server NIC/CNA

and EqualLogic array ports will inherit these parameters from peer switch ports via DCBX willing mode.

The Willing mode of operation is typically the default on most NICs/CNAs. EqualLogic arrays always

operate in willing mode, inheriting DCB parameters from attached switch ports.

Also for DCB to function properly, a VLAN needs to be configured for the iSCSI traffic that has been

prioritized and classified with a priority value. This is applicable for any application or protocol that needs

to be prioritized and classified using a priority value. Traffic that maps to the default priority 0 does not

need to have a VLAN ID configured. Do not use 0 or native VLAN (default 1) for the configured VLAN ID

since switches may not be configured to forward these VLANs with tags intact and priority information can

be lost. The chosen VLAN ID needs to be configured on all switch ports in the iSCSI path, server

NICs/CNAs, and the EqualLogic arrays. This also includes all switches in the Layer 2 DCB network domain

carrying iSCSI traffic to EqualLogic storage.

Once the links are up between ports, the operational state of DCB can be verified on the NICs/CNAs

through their respective OS management application or plug-ins. The operational state of DCB for


EqualLogic array ports can be verified through the Group Manager interface. To verify the DCB operational

state on switch ports, use the switch management application or CLI commands. The application priority

values, ETS settings, and PFC settings that are operational can be verified. Typically both local and peer

DCB states are reported.

Methods for DCB configuration and propagation in multi-layered switching environments are discussed in

section 4.


4 Network configuration and deployment topologies This section discusses detailed DCB parameter configuration for network deployments with EqualLogic

storage. It includes the configuration process for multi-layered switching environments and discusses

sample topologies.

Note: The term initiator in this document refers to an iSCSI initiator which is one of the following:

• A NIC in conjunction with an OS software initiator

• A dependent hardware initiator (CNA) that offloads iSCSI in conjunction with OS software initiator

support for configuration and session management

• An independent hardware initiator (CNA) that offloads iSCSI completely with no OS support

4.1 General configuration process flow The general configuration steps for setting up a DCB network infrastructure with EqualLogic storage are

given below. The assumptions for the discussion in this section are:

• iSCSI traffic is prioritized in its own traffic class or priority group

• All other traffic flow is in the default traffic class or priority group

• iSCSI traffic is PFC enabled and all other traffic is not

• iSCSI traffic is in a tagged VLAN ID and prioritized with priority 4

• All other traffic is untagged with default priority 0

The DCB configuration sequence involves the following steps:

1. Configure at least two top layer switches within the Layer 2 DCB network domain administratively

with the required DCB parameters and DCBX willing mode turned off on all ports. These become

the source switches. At least two switches are required since if one switch is shut down or fails,

another is available as a configuration source. Only the ports on these switches that are part of the

Layer 2 DCB domain need to be configured with DCB.

2. Configure intermediate layer switches and other peer switches:

a. Configuration propagation: If this mechanism is supported by switch model, then peer

switches, downstream switches, and I/O Aggregator modules should accept the DCBX

configuration from the source switches through certain Inter Switch Link (ISL) ports set as

upstream ports capable of receiving DCBX configuration from peers (enabled with willing

mode turned on the ports). Refer to Figure 9. In this scenario, a port is a single switch interface

or a LAG port-channel. These upstream ports internally propagate DCBX configuration to

other ports and ISLs on the same switch. Other ports and ISLs are set as downstream ports

capable of recommending DCBX configuration to their peer ports (enabled with willing mode

turned off). Figure 9 below also shows the downstream ports. The downstream ports apply

internally propagated DCBX configuration from upstream ports and recommend them to

connected edge devices and other switches. The upstream and downstream ports on a switch

are administratively identified based on network topology and are then configured manually

on each switch based on the topology. This identification and configuration is automatic for

certain switching modules such as I/O aggregators that are on the edge of the network

connecting to only edge devices on the downstream ports. In addition, when multiple


upstream ports are configured, an internal arbitration process is defined for one of them to

propagate DCBX configuration internally and the other ports accept that configuration.

b. Manual configuration: If this internal propagation method is not supported by your switch

model, then peer and intermediate layer switches must be manually configured with the same

DCB parameters as the source switches with willing mode off across all ports.

3. Configure and verify edge device DCBX parameters.

a. Configure edge device ports with the VLAN ID configured for iSCSI in the switches.

b. Verify operational DCBX parameters on the initiator/target storage ports and peer switch ports.

c. If mismatches occur, ensure edge device ports are in willing mode and they are capable of

supporting the DCB parameter requirements.

The configuration propagation model and DCBX exchanges using the willing bit is illustrated in Figure 9

below. The target storage is EqualLogic storage and the initiator hosts are Dell PowerEdge servers installed

with DCB capable NICs/CNAs. In the figure, SI links stands for Switch Interconnect links.

Figure 9 DCBX configuration propagation in intermediate switch layer


Note: The upstream port method can be used to configure other peer switches in the same top layer. A

pair of source switches can be configured with DCBX parameters and other switches in the same layer

can accept configuration from the source switches through the inter switch links (Willing mode on for

ISLs in the same layer). All switches in this layer then pass on configuration to downstream links and edge

devices like initiators and targets (willing mode off for these ports and downlinks).

The manual DCB configuration across all switches is illustrated in Figure 10 below.

Figure 10 Manual DCB configuration across all switch layers

Network configuration assumptions:

• All switches in the Layer 2 network carrying iSCSI traffic must be configured to be either DCB

enabled or DCB disabled. Mixed configuration with a partial set of switches in non-DCB mode

should not exist in the Layer 2 DCB network domain. The administrator must ensure that all switches

are either in DCB enabled state or DCB disabled state on the network during the configuration

process.

• For switches that support the DCB propagation mechanism: Switch inter-connectivity in the Layer 2

network must be designed so that there is no single point of failure that would cause a disjoint


network. When designed properly, the failure of a single switch device, link, or port interface should

not cause a disjoint network by design. The administrator must ensure that network redundancy

design best practices are followed to avoid disjoint networks.

The configuration steps summarized earlier in this section 4.1 are illustrated in the flow diagrams below.

Important: With all network configuration changes, there is a possibility of service interruptions. Network

changes required to properly configure DCB will result in a temporary loss of connectivity. Therefore,

Dell strongly recommends that all network environment changes are performed during a planned

maintenance window.

Further, when re-configuring an existing EqualLogic SAN to function in DCB mode, it is recommended to

first configure DCB VLAN ID for iSCSI on the arrays and then configure DCB along with VLAN ID on the

switch ports. This will ensure that inter-array connectivity is not lost during the process.


Figure 11 Source Switch Configuration


Figure 12 Peer and intermediate switch (non-source) configuration


4.2 Deployment topologies In this section, sample deployment topologies with EqualLogic iSCSI storage, Dell switches, and Dell

servers are demonstrated. For these topologies, it is assumed that there are no prior infrastructure

constraints from the network, initiators, or target storage to be considered. The assumption is that the

DCB infrastructure is a green field deployment. DCB enablement guidelines per the process flow

discussed in section 4.1 will be highlighted.

4.2.1 Single layer switching with blade servers Switch network configurations that are not hierarchical and include a single switching layer are covered in

this section. This section demonstrates a deployment configuration where all the components are within a

blade server chassis.

The sample configuration for this discussion includes PowerEdge M620 blade servers attached to

EqualLogic PS-M4110XV storage arrays via the Force10 MXL blade switches within the PowerEdge M1000e

chassis. The switches are inter-connected via a stack and the configuration is illustrated in Figure 13.

Figure 13 Single switching layer – Blade Servers


4.2.2 Single layer switching with rack servers This section demonstrates deployment configurations with rack servers and a single layer of top of rack

(ToR) switches. The sample configuration for this discussion includes PowerEdge R620 servers attached to

EqualLogic PS6110XV storage arrays via the PowerConnect™ 8132F ToR switches. The switches are inter-

connected via a LAG (Link Aggregation Group) and the configuration is illustrated in Figure 14.

Figure 14 Single switching layer – Rack severs

In both the Blade server and Rack server configurations, the first step is to configure the switches with

DCB parameters following the process defined in section 4.1. Appendix A provides information on lossless

queuing implementation with a Force10 MXL switch and a PowerConnect 8132F switch. The lossless

queuing implementation is the same with Force10 MXL and Force10 S4810 switches.

The second step is to setup the server initiators and targets with correct iSCSI VLAN ID. DCB is enabled in

willing mode by default. . Appendix C.1 illustrates this for a Broadcom 57810S NIC with Windows 2008 R2

software initiator. Appendix D.1 illustrates this for an EqualLogic 10GbE array. The last step is the

verification of DCB parameters and the operational state on the initiator, target, and corresponding switch

ports. Appendix C.2 and D.2 illustrate this for initiator and target. Appendix B illustrates the switch

commands for this verification.

4.2.3 Dual layer switching A hierarchical switching configuration with two layers of switching, one for servers and another for

storage, is covered in this section. An illustration of this configuration is given in Figure 15. The

configuration includes PowerEdge M620 blade servers connecting to Force10 M I/O Aggregator modules

within the PowerEdge M1000e chassis and connecting externally to Force10 S4810 switches. EqualLogic


PS6110XV arrays are connected to the external Force10 S4810 switches. The S4810 switches are

connected in VLT mode. The I/O Aggregator modules connect to the VLT switches via LACP.

Figure 15 Dual Layer Switching

In this scenario, the first step is to configure the external S4810 switches with DCB parameters following

the process defined in section 4.1.

The Force10 M I/O aggregator modules (FTOS version 8.3.17.0) are by default configured with DCB

enabled and all ports belonging to all VLANs. The external uplink ports (33 to 56) are auto configured to be

in a single LACP LAG (LAG 128). They are also auto configured to be in the DCB auto-upstream mode. The

auto-upstream ports are set with willing bit turned on. One of the auto-upstream uplink ports


(configuration source) will accept the recommended DCB configuration from the uplink peer and

internally propagate to remaining auto-upstream and all auto-downstream ports. The internal server

facing ports (1 to 32) are auto configured to operate in auto-downstream mode. This means that they

have the willing bit turned off and accept the internally propagated DCB configuration from the auto-

upstream port. These ports advertise the DCB configuration information to their server peers.

The second step is to setup the server initiators and targets with correct iSCSI VLAN ID. DCB is enabled in

willing mode by default. The last step is the verification of DCB parameters on the initiator, target, and

corresponding switch ports.


5 Component behavior with no DCB or partial DCB

capability This section discusses the infrastructure behavior when non-DCB capable or partially capable

components are present in a DCB enabled network infrastructure. It is recommended to deploy an end-

to-end DCB compatible configuration comprised of initiators, targets, and switches for a converged

network. Designing a converged network deployment with components that have no DCB support or

partial DCB support is not recommended for end-to-end converged I/O. Configurations with components

that have no DCB support or partial DCB support may result in loss of expected functionality or may be

functional without meeting the converged network objectives.

For operation with a DCB enabled EqualLogic iSCSI SAN, the requirements listed in section 3.5 must be

met by all components. For this discussion, these components are referred to as fully DCB capable for

EqualLogic iSCSI. Otherwise in this discussion they are referred to as partially DCB capable or not DCB

capable.

5.1 Storage array behavior with no or partial DCB capability on the

switch The array behavior can be summarized as follows:

• A DCB enabled array will operate in standard mode if there is no DCBX advertisement from the peer

switch port.

• If PFC is not supported and advertised then the array will issue a warning message indicating that

flow control cannot be enabled on array ports.

• If ETS/PG is not advertised then the array will place iSCSI traffic into the default priority group or

traffic class which may be lossy and shared with other traffic types on the switch.

• If iSCSI application priority is not supported, then the array will issue a warning message that flow

control cannot be enabled. This is because the default priority 0 to which iSCSI is mapped is not

typically PFC enabled on switches.

The iSCSI application priority TLV is informational and not a mandatory requirement from the standards,

but is required for proper operation with EqualLogic arrays. A partially DCB capable switch may support

PFC and ETS but not the iSCSI application priority. In this case, the array ports will send iSCSI frames with

default priority value 0 which is typically not PFC enabled on the switch. The switch port will place all iSCSI

frames from the array in a default priority group which may be lossy, resulting in potentially higher TCP

retransmissions on the network. The priority group may also include other traffic types and iSCSI may not

have dedicated bandwidth. Since PFC is not typically enabled for priority 0, the array will issue warning

messages (See appendix D.4) due to the inability to enable flow control on ports. Therefore, iSCSI

application priority needs to be supported/enabled on the switch ports along with PFC enabled for this

priority or the switch ports must be configured in non-DCB mode for the warning messages to be

terminated. Manual operation may be needed on the switch port to disable DCB and configure it in

standard non-DCB mode. A related recommendation would be to enable 802.3x PAUSE receive (Rx) mode

flow control on the switch port. The network design recommendation in this case would be to dedicate

the switches in non-DCB mode for iSCSI traffic alone and not converge other traffic types due to lack of

iSCSI application priority support.


5.2 Initiator behavior with no or partial DCB capability on switch An initiator in DCBX willing mode will not classify and prioritize network traffic if the switch port peer does

not advertise DCBX support. All network traffic will egress the port in standard non-DCB behavior. If iSCSI

application priority is not supported by the switch port peer, then an initiator will send iSCSI frames with

default priority 0. The switch port will place all iSCSI frames from the initiator in the default priority group

which may be lossy (no PFC) resulting in TCP retransmissions. The priority group may also include other

traffic types and iSCSI may not have dedicated bandwidth. If PFC is not advertised by the switch port peer,

then PFC will not be operational on the initiator. If ETS is not advertised by the switch port peer, then the

initiator will place all network traffic in the default priority group or traffic class.

5.2.1 Initiator with no DCB capability or partial DCB capability There may be scenarios where an initiator does not support DCB capability or supports only partial DCB

capabilities and needs to be part of the network infrastructure. Partial capability may be a scenario where

the initiator does not support iSCSI application priority. In these cases, the initiator will send iSCSI frames

with the default priority 0, when a VLAN ID has been configured for iSCSI traffic. When no VLAN ID is

configured, it will send the iSCSI frames as untagged. The switch port will place all iSCSI frames from the

initiator in the default priority group which may be lossy (no PFC) relying on TCP for recovery. The priority

group may also include other traffic types and iSCSI may not have dedicated bandwidth. It is

recommended not to converge iSCSI and other traffic on a NIC/CNA if it does not support iSCSI

application priority. In these cases, dedicated NIC/CNAs can be configured on server hosts for iSCSI traffic.

However iSCSI traffic from these dedicated NIC/CNA ports will still be lossy on the network since it gets

prioritized into the default TC/PG and is not a recommended practice.


6 Advanced DCB considerations, parameter resolution, and

behavior This section covers additional clarification on certain advanced aspects of the DCB standards. It also

includes discussion on switches and initiators, covering aspects such as behavior, parameter resolution,

and additional clarifications from the DCB standards.

6.1 Peer DCB parameter states The requirements for DCB parameter states on peer ports from the standards are:

• PFC – PFC enable/disable state for the eight priorities must be the same (symmetric) on peer ports

for PFC to be operational.

• ETS – The number of traffic classes/priority groups, their priority mapping and bandwidth allocation

percentage is not required to match between peer ports.

• Application priority – The application protocol mapping to a priority is not required to match

between peer ports.

Since ETS and application priority are not required to match between peer ports, mismatches can be

avoided by choosing components (switches and NICs/CNAs) that support the same set of overall

requirements discussed in section 3.5 and 3.6.

6.2 VLAN for iSCSI A non-default VLAN is typically required for operating prioritized lossless iSCSI traffic in a DCB enabled

Ethernet infrastructure. Typically the default VLAN ID is 1 on most switches. This is also referred to as the

native VLAN for switch ports. Switch ports that are based on the IEEE 802.1Q VLAN specification forward

frames in the default VLAN without tags (untagged frames). Normally these ports also receive frames in the

default VLAN as untagged. Such a port is typically termed a “tagged port” or “trunk port” and all non-

default VLAN frames are forwarded with the tags intact. Since the DCB priority information (PCP value) is

encoded in the VLAN tag, this information will be lost if an end-device sends iSCSI traffic in a default VLAN

and the switch receives it in a tagged or trunk port. As a result, DCB prioritization and traffic classification

will be lost. The same applies to VLAN 0 since that is mapped to the default VLAN within the switch by

default. This means that a non-default VLAN (typically not 0 and 1) is a requirement for lossless DCB

operations. If heterogeneous vendor switches exist in the network, then it must be ensured that the VLAN

for iSCSI is not the default VLAN across all of the switches. All devices in the iSCSI data path must have the

same VLAN ID configured on the respective ports participating in the iSCSI network to ensure proper

functioning. These devices include the server iSCSI NIC/CNA ports, EqualLogic arrays, and all switches on

the iSCSI SAN.

Note: The VLAN ID for iSCSI can be set in the EqualLogic Group Manager interface or the storage array

CLI. For more information, refer to Appendix D. Appendix D.3 lists information for adding new array

members to an existing DCB enabled group in a VLAN or creating a new DCB enabled group on a VLAN.


6.3 Switch requirements and behavior

6.3.1 DCBX version resolution Dell switches support both IEEE DCB standards and the CEE DCB base protocol specification. Dell

PowerConnect 8100 Series, Force10 S4810, and Force10 MXL switches include an “auto” mode where

during initial peer detection and configuration, IEEE DCBX will be automatically used by ports if supported

by the peer device (NICs/CNAs and target storage). If IEEE DCBX is not supported by the peer device, CEE

DCBX will be automatically used for configuration and operation. By default all ports in these switches are

in non-willing mode and use auto mode for DCBX version selection. When using these switches no

changes need to be done other than configuring the required ETS, PFC, and application priority

parameters. Similarly, certain NICs/CNAs support auto mode if supporting both CEE and IEEE DCBX.

From the configuration perspective, the version resolution should take place automatically. However, you

must verify that DCBX version resolution occurred correctly by verifying that DCB is operational through

the appropriate management interfaces for the NIC/CNA, storage, and switch ports. Sometimes due to

implementation differences, either version must be forced manually on the switch and/or NIC/CNA for

proper operation instead of using auto mode. Certain switch vendors support only the CEE base protocol

v1.01. In this case, CEE DCBX will be used for end-to-end operations as long as the storage arrays and

NICs/CNAs support CEE DCBX. The switch or NIC/CNA vendors may not support auto mode to detect the

peer version, and a version will need to be manually set for DCBX configuration on switch or NIC/CNA

ports matching the version supported on the peer. For example, you would set CEE DCBX on a switch port

when the attached NIC/CNA port only supports CEE DCBX.

6.3.2 ETS – Additional clarification IEEE standards support assigning traffic classes with any of these transmission algorithms: strict priority

(SP), credit shaper, and ETS. ETS supports bandwidth allocation as a percentage of available link bandwidth

to each traffic class and is required for DCB implementations. However, SP and credit shaper classes are

prioritized over ETS classes during frame transmission and do not support bandwidth sharing. Available

bandwidth for ETS on a link is defined as the bandwidth remaining after SP and credit shaper classes have

been transmitted. There is a potential to starve ETS traffic classes of bandwidth if SP and credit shaper

classes use the entire link bandwidth. At a minimum, ETS class traffic latencies may be impacted due to SP

and credit shaper classes being prioritized.

It is recommended to deploy converged I/O solutions with EqualLogic iSCSI storage only with ETS as the

transmission selection algorithm for all traffic classes and not to include traffic classes with SP or credit

shaper algorithms. Switches must be configured and verified so that all traffic classes have ETS as the

transmission selection algorithm. Certain switch implementations allow a second level scheduling

mechanism within a traffic class for scheduling priorities contained in it. This second level scheduling

mechanism can use strict priority for priorities with the traffic class. However the overall bandwidth for the

traffic class will still be limited by its ETS allocation percentage.

In CEE standards, a priority group with an ID of 15 has a special meaning and priorities mapped to this

priority group will not be subjected to bandwidth limits. A priority group with ID 15 is serviced before other

classes are serviced. It is recommended to deploy converged I/O solutions with EqualLogic iSCSI storage

using only priority group IDs 0 through 7 and not to use priority group ID 15. Switches must be configured

so that all PGs have IDs 0 to 7. You can verify that a PG with ID 15 is not configured on switches through


the appropriate management interfaces or CLI. It must also be noted that in CEE, priorities 8 through 14

are reserved and should not be used.

6.3.3 PFC – Additional clarification It is possible to have two or more priorities mapped into a single traffic class or priority group. If one of the

priorities is paused, it is possible that all priorities on the TC/PG will be paused even if they are not enabled

for PFC or do not require pausing at that time. This can be due to implementation behavior of traffic

classes as hardware queues. Mixing PFC and non-PFC priorities in the same traffic class or priority group

can result in non-PFC traffic being paused and therefore is not recommended. On switches, iSCSI traffic

priority must be enabled with PFC and be mapped into its own traffic class or priority group. It is also

recommended for the default priority 0 to have PFC disabled. Switches must have priority 0 mapped to a

default traffic class/priority group which is lossy and not PFC enabled.

6.4 NIC/CNA This section provides additional clarification for DCB requirements and DCB parameter resolution

recommendations for initiators.

6.4.1 PFC and application priority parameter resolution PFC and iSCSI application priority mapping from the peer switch port is inherited automatically on the

initiator operating in willing mode. Exceptions occur when the NIC/CNA does not support the iSCSI

protocol in the application TLV or does not support the recommended PFC enablement for priorities. In

these scenarios, you should consider choosing a different NIC/CNA model that meets these requirements.

You must verify PFC and application priority mappings that are operational at peer ports (switch, NIC/CNA)

using the appropriate management interfaces or tools. If iSCSI application priority is not supported, iSCSI

traffic will be mapped to default priority 0. Other non-prioritized network traffic will also use this default

priority and iSCSI traffic will not have dedicated bandwidth. Also this priority may not be lossless (PFC

enabled).

A NIC/CNA that does not meet the DCB requirements or is not DCB capable can still be dedicated for

iSCSI traffic alone on a server host. However the iSCSI traffic from this NIC/CNA to the DCB enabled

network could be lossy. This can impact performance or functional behavior of iSCSI traffic under heavy

load conditions. This is not a recommended approach.

6.4.2 ETS parameter resolution There may be scenarios where a NIC/CNA does not support the same number of traffic classes or priority

groups as the switch. This can be due to the implementation capabilities in the hardware or software

layers. In such scenarios, it is recommended to choose a different NIC/CNA model that meets these

requirements. Otherwise the NIC/CNA must be manually configured to map iSCSI priority within a single

traffic class or priority group to ensure dedicated iSCSI bandwidth. If not, the iSCSI traffic will not get

dedicated bandwidth due to contention with other traffic types. TC/PG mapping will happen automatically

on the NIC/CNA in most cases due to the default willing mode operation. You must verify that the TC/PG

mapping is operational at both peer ports (switch and NIC/CNA) using the appropriate management

interfaces or tools.


6.4.3 iSCSI application priority tagging The NIC/CNA must support tagging the Ethernet frames carrying iSCSI protocol packets with the set

priority value for iSCSI via the application priority feature. Certain operating systems provide driver support

for tagging the iSCSI frames at the OS level when using the iSCSI software initiator. In other cases, the

NIC/CNA driver provides the capability for tagging iSCSI frames when using the software initiator. When

using hardware initiators, the CNA driver provides the capability for tagging iSCSI frames to comply with

the application priority value set for iSCSI. The iSCSI tagging support available with the NIC/CNA for a

particular OS will influence the choice of hardware or software initiator for your deployment.

6.5 EqualLogic iSCSI storage The operational behavior for EqualLogic 10GbE iSCSI storage is listed below.

6.5.1 DCBX version resolution EqualLogic 10GbE arrays support both IEEE DCB and CEE DCB standards. The arrays function in auto

mode and automatically select IEEE DCBX if the switch port supports it, otherwise they select CEE DCBX

(base protocol version 1.01).

6.5.2 DCB enablement By default, DCB is enabled on the 10GbE EqualLogic arrays. A non-default VLAN ID must be provided for

proper functioning of DCB. If the switch supports DCB, then the switch ports must be configured with the

required PFC, TC/PG, and iSCSI application priority settings described in section 3.5. The array ports always

operate in DCBX willing mode and inherit DCBX settings from the switch port.


7 Conclusion Designing a fully functional DCB infrastructure for operation with an EqualLogic iSCSI SAN requires careful

consideration. The requirements discussed in this document from the DCB standards and the additional

DCB requirements for switches and initiators to operate with EqualLogic iSCSI SANs need to be followed

for proper operation. Appropriate planning and design of the infrastructure is critical for successful

deployment. The guidelines discussed in this document enable a seamless end-to-end configuration and

operational behavior for DCB enabled components. The paradigm of central configuration of DCB

parameters at the top most switching layer and its automatic propagation to other peer switches,

switching layers, and edge devices via the DCBX protocol provides a consistent and seamless

configuration experience. It is also important to understand the behavior of components with no DCB

capability or iSCSI application priority support because it will impact converged network design goals and

expected outcomes.


A Lossless queue implementation

A.1 Force10 MXL switch or Force10 S4810 switch

Refer to the latest user guide for Force10 MXL for information on switch commands:

http://www.force10networks.com/CSPortal20/KnowledgeBase/Documentation.aspx

Force10 MXL supports four egress queues (0 to 3) for switch port interfaces. The default mapping of

priority values in incoming frames to egress queues is given in table below.

Table 4 Queue mapping

Priority values Egress queue assignment

0 0

1 0

2 0

3 1

4 2

5 3

6 3

7 3

The mapping becomes effective when the switch is configured to trust incoming priorities on VLAN tags

using the command ‘service-class dynamic dot1p’. The above mapping can also be changed. Force10 MXL

supports up to two lossless queues among these four. When PFC is enabled for a priority, the queue that it

is mapped to becomes lossless by default. But only two queues can be lossless, so all PFC enabled

priorities can be mapped to only two egress queues on a port. When PFC is enabled for an interface, PFC

becomes operational for both ingress and egress traffic out of the interface. The number of priority groups

supported for ETS is equal to the number of queues, which is four. The number of priority groups can be

less than or equal to four. It is also recommended to map PFC enabled priorities and PFC disabled

priorities to separate queues and priority groups. All priorities mapped to the same queue in the queue

mapping table above should be in the same priority group. The queue mapping should be managed to

support the overall priority to priority group mapping configured in the network.

For a list of DCB related switch commands, go to Appendix B.1.

A.2 PowerConnect 8132F switch

PowerConnect 8100 series egress port queues are called Class of Service (CoS) queues. There are eight

queues (0 through 7). Queue 7 is reserved for system. Queues 0 to 3 are recommended for applications



and Queues 4 to 6 are recommended for system traffic. The default mapping of priorities to queues is

given in Table 5 below.

Table 5 Queue mapping

Priority values CoS queues

0, 3 1

1, 2 0

4, 5 2

6, 7 3

The default mapping can be changed. PowerConnect 8100 series switches support up to two lossless

queues. When PFC is enabled for a priority, a no drop criteria can be specified. If no drop is specified, then

the queue where the priority is mapped becomes lossless. Only two such queues can be lossless.

Therefore, all PFC enabled priorities with no drop criteria can be mapped to only two CoS queues on a

port. Lossless PFC priorities with no drop should be mapped one to one to a CoS queue. If there are two

lossless PFC priorities, each should be mapped to its own queue.

Priority groups in PowerConnect 8100 are called traffic class groups (TCGs). There are three TCGs

supported from 0 to 2. TCGs are serviced from the highest number to the lowest number. The CoS

queues should be mapped to traffic class groups. It is required to assign the TCGs in order from 0 to 2,

and it is recommended to use consecutive TCG numbers. A first level scheduler manages the bandwidth

scheduling of CoS queues and a second level scheduler manages the bandwidth scheduling of TCGs.

For a list of DCB related switch commands, go to Appendix B.2.

B Switch DCB verification commands

B.1 Force10 MXL switch or Force10 S4810 switch

Refer to the latest user guide for Force10 MXL for information on DCB configuration verification

commands: http://www.force10networks.com/CSPortal20/KnowledgeBase/Documentation.aspx.

Some important commands are:

show dot1p-queue mapping

show dcb [stack-unit unit-number]

show qos dcb-input [pfc-profile]

show qos dcb-output [ets-profile]

show qos priority-groups

show interface port-type slot/port pfc {summary | detail}



show interface port-type slot/port pfc statistics

show interface port-type slot/port ets {summary | detail}

show stack-unit all stack-ports all pfc details

show stack-unit all stack-ports all ets details

show interface dcbx detail

show lldp neighbors detail

B.2 PowerConnect 8132F switch

Refer to the latest user guide for PowerConnect 8132 for information on DCB configuration verification

commands at http://support.dell.com.

Some important commands are:

show lldp dcbx interface {all status |interface [detail]}

show interfaces datacenter-bridging [interface | port-channel port-channel-id]

show classofservice dot1pmapping

show classofservice trust

Show interface cos-queue

show classofservice traffic-class-group

show interfaces trafficclass-group

C Broadcom 57810S configuration and verification

C.1 Broadcom 57810S DCBX willing mode verification

Figure 16 below shows a screen shot from the Broadcom™ Advanced Control Suite application. This is

applicable to the Broadcom 57810S Dual-Port 10GbE SFP+ adapter and the Broadcom 57810S Dual-Port

10GbE KR Blade Converged Mezzanine Card on Windows Server 2008 R2 SP1. The parameter “local

machine willing” indicates the willing mode state and is enabled by default.

http://support.dell.com/


Figure 16 Initiator willing mode configuration

In this figure, the adapter is configured with willing mode enabled by default. Also the local DCBX settings

are shown in the default configuration. These settings include the priority and PFC values for FCoE and

iSCSI. This also includes the priority groups with their bandwidth allocation and priority mapping. Once the

adapter port establishes an operational link with a peer switch port and receives the switch DCBX settings,

it will use those settings for DCB operations. The local settings will not be operational at that point. The

default local DCBX setting can be changed by overriding them.

C.2 Broadcom 57810S iSCSI VLAN configuration

The figure below illustrates the VLAN configuration for iSCSI traffic on the adapter port.


Figure 17 Initiator VLAN ID configuration

C.3 Broadcom 57810S DCB configuration verification

The image below is the screen shot from the Broadcom Advanced Control Suite application. This is

applicable to the Broadcom 57810S Dual-Port 10GbE SFP+ adapter and the Broadcom 57810S Dual-Port

10GbE KR Blade Converged Mezzanine Card on Windows Server 2008 R2 SP1. The DCB parameters

shown are the values advertised by the switch peer and accepted by the NIC port. It also indicates the

DCB operational state for each feature.

Figure 18 DCBX operation state and configuration verification

The local and remote DCBX settings are illustrated in the advanced section illustrated in figure below.


Figure 19 Advanced DCB operation state and configuration verification

D EqualLogic storage

D.1 EqualLogic 10GbE storage DCB iSCSI VLAN configuration

The image below is a screen shot from the EqualLogic Group Manager. It shows the user interface option

to provide a VLAN ID for iSCSI in DCB mode. DCB is enabled by default on the array ports in willing mode.


Figure 20 EqualLogic Enable DCB option

The checkbox option to enable DCB is present in version 6.0.2 of the array firmware.

D.2 EqualLogic 10GbE storage DCB configuration verification

The image below is a screen shot from EqualLogic Group Manager. It shows the DCB parameters

operational on one of the array 10GbE ports.


Figure 21 EqualLogic DCB operational state and configuration verification

In the configuration illustrated above, iSCSI was configured on priority 4 on the switch with PFC enabled.

This is indicated by the phrase “(iSCSI)” next to priority 4 in the figure and the lossless column indicating

“on” for 4. If the switch was configured with another priority for iSCSI, then that priority will be marked

with the iSCSI indication. It must be ensured that the lossless property is shown as “on” for the iSCSI

priority indicating that PFC was enabled for the iSCSI priority at the peer switch port.

In the above illustration, the DCB state shows “on” when the array port receives the DCBX TLVs from

switch ports. The individual DCBX TLVs received are shown as “on” when they are received from the peer

switch port. Traffic classes implies ETS/PG, lossless priorities implies PFC, and iSCSI priority implies iSCSI

priority value on the application TLV.

D.3 Adding a new member to a DCB enabled group or create a group

in which DCB will be used

The following issues apply when working with an EqualLogic group in which DCB is enabled.

D.3.1 Specifying the DCB VLAN ID when creating or expanding a group

To add a new member to a group in which DCB has been enabled, or to create a new group in which DCB

will be used on a tagged VLAN, you must complete the following steps:


1. Connect to the member using a serial connection.

2. Press Control+C to cancel the setup program.

3. At the CLI prompt, enter setup def_dcb_vlan_id vlan_id

where vlan_id is the DCB VLAN ID number used by the group. Next, run the setup utility as

documented in the EqualLogic array configuration instructions.

More information can be found in the release notes of the EqualLogic array firmware.

The latest firmware updates and documentation can be found at: http://support.equallogic.com. This site

requires a login.

D.4 Warning message when switch ports are not fully DCB capable

The image below is a screen shot from EqualLogic Group Manager. It shows the warning message

displayed when DCB is enabled on the array but the attached switch ports are partially capable without

iSCSI application priority support. The array issues a warning message that the array ports cannot have

flow control turned on since PFC is not turned on for the default priority 0 to which iSCSI is mapped. If

iSCSI priority is not advertised by the switch then the array maps the iSCSI traffic to the default priority 0.

This warning message is also issued when PFC is not turned on for the priority value mapped for iSCSI

traffic, even when that is not the default priority 0.

Figure 22 Array warning message

In addition to this warning message, a more detailed message will appear in the event log and CLI as

shown below (firmware version 6.0.2):

http://support.equallogic.com/


Interface eth0 has negotiated DCB priority flow control with the switch. However, the switch did not configure iSCSI traffic for a lossless priority. No Ethernet flow control is enabled for iSCSI on this interface.

E DCB standards history

This section provides advanced information on the evolution of DCB standards and their differences.

E.1 DCB history and versions

The progression of DCB versions is summarized in table below.

Table 6 DCB versions

General naming reference

Year(s) Description

CIN DCBX 2006 - 2007 An initial specification called the DCB Capability Exchange Protocol Specification was developed by Cisco Systems, Intel Corporation, and Nuova Systems. This version is not in common use today.

CEE DCBX or baseline DCBX

2008 - 2009 A consortium of companies developed the DCB Capability Exchange Protocol Base Specification Rev 1.01. The version is also referred to CEE DCBX or baseline DCBX since most implementations started with this as a stable version. This version was proposed to the IEEE DCB task group.

IEEE DCB 2008 - 2011 The IEEE DCB task group defined a standardized set of protocol specifications to cover the key DCB technologies provided in links below.

The overall purpose of DCB technologies (DCBX, PFC, ETS, CN) is similar across the various DCB versions

developed, with some differences.

DB specification reference links:

• DCB Capability Exchange Protocol Specification Rev 1.0:

http://download.intel.com/technology/eedc/dcb_cep_spec.pdf

• DCB Capability Exchange Protocol Base Specification Rev 1.01:

http://www.ieee802.org/1/files/public/docs2008/az-wadekar-dcbx-capability-exchange-discovery-

protocol-1108-v1.01.pdf This specification was proposed to IEEE 802.1 work groups along with the

following:

- Proposal for Priority-based Flow Control: http://www.ieee802.org/1/files/public/docs2008/bb-

pelissier-pfc-proposal-0508.pdf

- Packet Scheduling with Priority Grouping and Bandwidth Allocation for DCB Networks:

http://www.ieee802.org/1/files/public/docs2008/az-wadekar-ets-proposal-0608-v1.01.pdf

• IEEE DCB task group: http://www.ieee802.org/1/pages/dcbridges.html


http://www.ieee802.org/1/files/public/docs2008/az-wadekar-dcbx-capability-exchange-discovery-protocol-1108-v1.01.pdf


http://www.ieee802.org/1/files/public/docs2008/bb-pelissier-pfc-proposal-0508.pdf

http://www.ieee802.org/1/files/public/docs2008/bb-pelissier-pfc-proposal-0508.pdf

http://www.ieee802.org/1/files/public/docs2008/az-wadekar-ets-proposal-0608-v1.01.pdf

http://www.ieee802.org/1/pages/dcbridges.html


- IEEE 802.1Qaz-2011 specification defines DCBX and ETS. The DCBX protocol leverages LLDP

specification defined in IEEE 802.1AB

- IEEE 802.1Qbb-2011 defines PFC

- IEEE 802.1Qau-2010 defines quantized congestion notification.

E.2 DCBX version differences

The key differences between the different DCBX versions are discussed in a document published by the

University of New Hampshire and is provided as a reference at the link below.

University of New Hampshire, Interoperability Laboratory – Comparison of DCBX versions:

https://www.iol.unh.edu/services/testing/dcb/training/DCBX_Comparison.pdf

The primary differences between the CIN, CEE, and IEEE DCBX versions are the DCBX TLV frame formats.

For example with CEE and IEEE versions, the LLDP frame formats for respective features are different and

state machines are different. In CEE, a single configuration TLV with sub-TLVs is used for protocol control

and all features (PFC, PG, application protocol). In IEEE, separate TLVs for each feature such as ETS TLV,

PFC TLV and application priority TLV are used.

The DCBX parameter exchange state machines are similar with CIN and CEE but different from the later

IEEE version. The CIN/CEE DCB versions include the definition for Backward Congestion Notification

(BCN). IEEE 802.1Qau defines Quantized Congestion Notification (QCN) for end-to-end Layer 2 flow

control and congestion management.

https://www.iol.unh.edu/services/testing/dcb/training/DCBX_Comparison.pdf


Additional resources

Support.dell.com is focused on meeting your needs with proven services and support.

DellTechCenter.com is an IT Community where you can connect with Dell Customers and Dell employees

for the purpose of sharing knowledge, best practices, and information about Dell products and your

installations.

Referenced or recommended Dell publications:

• EqualLogic Compatibility Matrix (ECM):

http://en.community.dell.com/techcenter/storage/w/wiki/2661.equallogic-compatibility-

matrix.aspx

• EqualLogic Switch Configuration Guides:

http://en.community.dell.com/techcenter/storage/w/wiki/4250.switch-configuration-guides-by-

sis.aspx

• The latest EqualLogic firmware updates and documentation (site requires a login):

http://support.equallogic.com

• Dell EqualLogic PS Series Network Performance Guidelines:

http://www.equallogic.com/resourcecenter/assetview.aspx?id=5229

• Force10 Switch documentation:


• PowerConnect Switch documentation: http://support.dell.com

• DCB Capability Exchange Protocol Specification Rev 1.0:


• DCB Capability Exchange Protocol Base Specification Rev 1.01:

http://www.ieee802.org/1/files/public/docs2008/az-wadekar-dcbx-capability-exchange-discovery-

protocol-1108-v1.01.pdf

• IEEE DCB task group: http://www.ieee802.org/1/pages/dcbridges.html

For EqualLogic best practices white papers, reference architectures, and sizing guidelines for enterprise

applications and SANs, refer to Storage Infrastructure and Solutions Team Publications at:

• http://dell.to/sM4hJT



http://en.community.dell.com/techcenter/storage/w/wiki/4250.switch-configuration-guides-by-sis.aspx

http://en.community.dell.com/techcenter/storage/w/wiki/4250.switch-configuration-guides-by-sis.aspx

http://support.equallogic.com/

http://www.equallogic.com/resourcecenter/assetview.aspx?id=5229


http://support.dell.com/




http://www.ieee802.org/1/pages/dcbridges.html

http://dell.to/sM4hJT

THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND.

DCB Reqs Design

Documents

Transcript of DCB Reqs Design