QOS Review:

This is a review of NCAR’s current campus network and the potential FRGP QOS implementation. Since NCAR’s initial QOS deployment, new hardware and IOS versions have been deployed in the network, CatOS has been converted to IOS, and the QOS toolset has changed. The intent of this document is to validate the existing QOS configuration and/or propose new QOS configurations using QOS best practices outlined in Cisco’s Enterprise QoS Solution Reference Network Design (Enterprise QoS SRND) as a guideline.

The intent of NCAR’s QoS deployment is to ensure VOIP traffic has priority over other traffic in the face of network congestion. QoS is not a “one size fits all” configuration – it is highly dependent on company policy decisions (i.e. prioritize application X) and on the overall composition of network traffic. NCAR’s QoS implementation should be reviewed on a regular and as-needed basis to ensure QoS is functioning as designed when new hardware or changes in traffic profiles occur.

Here is a summary of the recommendations that have been reviewed and approved by NETS engineers:

• Cisco’s Enterprise QoS and Medianet Campus Qos SRNDs will be used as the basis for QoS configurations

• Input queuing and policing will not be used

• Only voice traffic will be prioritized

• Transmit queue design will use the default settings applied by the ‘mls qos’ command except in a few cases where the default deviates from the SRND

• Cut and paste QoS files will be created for each queue type and be used as the default configuration for new cards

• Packet drops per queue and threshold will be monitored to ensure QoS policy is working properly

• Netflow will be enabled on internal NCAR traffic for additional QoS monitoring and statistics

This document is organized in the following manner:

1. QoS Overview – Review of QoS concepts and config examples 2. NCAR’s Existing QoS configuration - Review of NCAR’s current QoS configurations 3. NCAR’s queue types 4. NCAR’s traffic profile – VOIP traffic vs Other traffic 5. QoS Configuration Recommendations 6. Monitoring QoS 7. Telepresence and QoS at NCAR 8. Telepresence and QoS at the FRGP 9. References

1. QoS Overview

There are many different ways to implement QoS but the SRND recommends using DSCP markings to ensure consistent PHB (per hop behavior) of packets. A DSCP based QoS deployment consists of the following components:

1. Input Queuing 2. Classification and Marking 3. Policing 4. Transmit Queuing

The example commands presented below are for a 6500 with 1p3q8t queue. To complicate matters, the QoS implementation varies across the different hardware platforms and IOS versions. Commands may vary slightly depending on switch and/or queue type, however the basic procedure remains the same.

Input Queuing

Ingress queues (RX) are extremely difficult to congest. Ingress congestion implies that the combined ingress rates of traffic exceed the switch fabric channel speed, and thus would need to be queued simply to gain access to the switching fabric. On newer platforms, such as the Catalyst 6500 Sup720, this means that a combined ingress rate of up to 40 Gbps per slot would be required to create such an event*. The 6500 line cards do have ingress queues in the event that congestion does occur, but since this is highly unlikely, only the egress (TX) queues will be addressed in this document.

* End-to-end qos network design, By Tim Szigeti, Christina Hattingh

Classification and Marking

The Enterprise QoS SRND follows standards-based DSCP PHB markings to ensure interoperability and future expansion.

A general rule of thumb is to classify traffic as close to the source as possible and to establish “trust” boundaries. At the edge of our network, the Cisco IP phones appropriately mark their traffic as voice bearer (COS 5, DSCP 46|EF) and voice signaling traffic (COS 3, DSCP 24|CS3), so we’ll “trust” the DSCP markings from the phone. A PC attached to the phone’s PC port will not be trusted - should the PC try to send packets marked other than DSCP = 0, the switch will remark the packets to DSCP = 0. By default, the 6500s, with QoS enabled, will mark packets with DSCP 0 from any untrusted source.

The trust boundary can be established on a per port or per vlan basis. Cisco recommends per vlan when possible.

Once classification is performed at the edge for inbound traffic, the router/switch may need to map COS to DSCP markings (and vice-versa) before transmitting the packet. Use the IOS commands below to configure the mapping (global config):

mls qos map cos-dscp mls qos map dscp-cos

Here is the CatOS command (global):

set qos cos-dscp-map set qos dscp-cos-map

The DSCP/COS values are then trusted on the uplinks from Access to Distribution, and Distribution to Core.

Policing

Policing can be used to “markdown” excess traffic in a class or drop offending traffic. Policing will not be used in NCAR’s campus network at this time.

Transmit Queuing

Transmit queue types are hardware dependant. IOS command ‘show interface capabilities’ -> ‘QoS scheduling’ displays the type:

ml-16c-c1-gs#show interfaces capabilities mod 4 GigabitEthernet4/1 Model: WS-X6148A-GE-45AF Type: 10/100/1000BaseT Speed: 10,100,1000,auto Duplex: half,full Trunk encap. type: 802.1Q,ISL Trunk mode: on,off,desirable,nonegotiate Channel: yes Broadcast suppression: none Flowcontrol: rx-(off,on,desired),tx-(off,on,desired) Membership: static Fast Start: yes QOS scheduling: rx-(1q2t), tx-(1p3q8t)

The CatOS command is

‘show qos info config 1p3q8t tx’

In this example, the transmit queue is 1p3q8t.

1p = one priority queue. Traffic in the priority queue is serviced until the queue is empty before the other queues are serviced.

3q = there are three queues

8t = each queue has 8 drop thresholds that can be configured (does not apply to the priority queue)

For each queue type, the following needs to be configured:

• Map traffic to queue

• Set queue buffer size

• Set queue WRR weights

• Set queue WRED thresholds

Map traffic to queue

There are a number of RFC’s (2474, 2597, 3246, 4595) which describe the PHB for the different DSCP markings. Proper queue design (buffer size, WRR weights, and WRED thresholds) determines whether the queue behaves as Best Effort or CS3 or some other type. DSCP marked traffic must be mapped to a queue designed to handle that type of traffic. Cisco has taken all these factors into account and provided queuing models in the SRND for all the different types of queues found in their hardware:

http://www.cisco.com/en/US/docs/solutions/Enterprise/WAN_and_MAN/QoS_SRND/QoS-SRND-Book.html

Here are the IOS commands (per interface):

wrr-queue cos-map <queue> [<thresh>] <cos> … <cos> priority-queue cos-map <queue> <cos> … <cos>

Here is the CatOS command (global)

set qos map 1p2q2t tx 3 1 cos 5

Set queue buffer size

The size of the buffers must be set. Typically the lower weighted queues will have larger buffers since they are not serviced as often. Here is the IOS command to set the buffers (per interface):

wrr-queue queue-limit priority-queue queue-limit

Here is the CatOS command (global):

set qos txq-ratio 1p2q2t 10 90

Set queue WRR weights

With WRR, the switch uses a configured weight value for each egress queue. This weight value determines the implied bandwidth of each queue. The higher the weight value, the higher the priority that the switch applies to the egress queue. Priority queues do not have assigned weights and are serviced before other queues, until empty.

For example, consider the case of a Catalyst 3550 switch configured for QoS and WRR. The Catalyst 3550 uses four egress queues. If queues 1 through 4 are configured with weights 50, 10, 25, and 15, respectively, queue 1 can utilize 50 percent of the bandwidth when there is congestion. Here is the IOS command to set the queue weights (per interface):

wrr-queue bandwidth

Here is the CatOS command (global):

set qos wrr 1p2q2t 30 70

Set queue WRED thresholds

There is a minimum and maximum WRED (weighed random early detection) threshold configured. The minimum indicates the point at which packets will be randomly dropped for that queue/threshold and the MAX threshold determines when tail drop (i.e. all pkts in the queue) will begin. Here is a link to how WRED works: http://www.cisco.com/en/US/docs/ios/qos/configuration/guide/congestion_avoidance.html#wp1000959

IOS example (per interface):

! Sets Min WRED Threshold for Q1T1 to 80% and all others to 100% wrr-queue random-detect min-threshold 1 80 100 100 100 100 100 100 100 ! Sets Max WRED Threshold for Q1T1 to 100% and all others to 100% wrr-queue random-detect max-threshold 1 100 100 100 100 100 100 100 100

CatOS example (global):

set qos wred 1p3q8t tx queue 3 <[thr1Lo:]thr1Hi> … <[thr8Lo:]thr8Hi>

2. NCAR’s Existing QoS configurations

The existing QOS policy is, where possible, to use the default configuration when ‘mls qos’ global command is applied. Exceptions to the default configurations are COS to DSCP mappings, and queue/threshold mappings for COS 3 and COS 5 marked traffic.

Here is a typical configuration from one of our CATOS switches.

set qos enable set qos map 2q2t tx 2 1 cos 3 set qos map 2q2t tx 2 2 cos 5 set qos wrr 2q2t 100 255 set qos map 1p2q2t tx 2 1 cos 3 set qos wrr 1p2q2t 100 255 set qos wred 1p2q2t tx queue 1 40:80 70:100 set qos wred 1p2q2t tx queue 2 40:80 70:100

set qos map 1p2q1t tx 2 1 cos 3 set qos map 1p3q8t tx 3 3 cos 3 set qos cos-dscp-map 0 8 16 24 32 46 48 56 set qos ipprec-dscp-map 0 8 16 24 32 46 48 56 clear qos acl all #ACL-IP-PHONES set qos acl ip ACL-IP-PHONES trust-cos ip any any #ACL-TRUST-DSCP set qos acl ip ACL-TRUST-DSCP trust-dscp ip any any #ACL-TRUST-IPPREC set qos acl ip ACL-TRUST-IPPREC trust-ipprec ip any any # commit qos acl all set qos acl map ACL-IP-PHONES 700

A survey of all the CATOS switches showed that queue mapping is performed for all the queues, but buffer sizes, weights, and thresholds are not consistently applied to the different queues. A survey of IOS switches shows that qos is enabled globally, but are currently using the defaults for all other settings.

In CATOS, ‘show qos info config 1p3q8t tx’ shows the current queue settings (for IOS its ‘show queueing interface <interface name>’):

QoS setting in NVRAM for 1p3q8t transmit: QoS is enabled Queue and Threshold Mapping for 1p3q8t (tx): Queue Threshold CoS ----- --------- --------------- 1 1 0 2 1 1 2 3 3 3 3 5 4 3 7 6 7 4 1 5 Tx drop thresholds: Tx drop-thresholds feature is not supported for this port type. Tx WRED thresholds: Queue # Thresholds - percentage ------- ------------------------------------------ 1 70%:100% 100%:100% 100%:100% 100%:100% 100%:100% 100%:100% 100%:100% 100%:100% 2 70%:100% 100%:100% 100%:100% 100%:100% 100%:100% 100%:100% 100%:100% 100%:100% 3 40%:70% 40%:70% 50%:80% 50%:80% 60%:90% 60%:90% 70%:100% 70%:100% Tx queue size ratio: Queue # Sizes - percentage ------- ------------------------------------- 1 65%

2 15% 3 15% 4 5% Tx WRR Configuration of ports with 1p3q8t: Queue # Ratios ------- ------------------------------------- 1 20 2 100 3 200

3. NCAR Queue Types

Model Description Queue Type SchedulerVS-S720-10G Supervisor Mod 720 10G (enabled/disabled G ports) 1p3q4t/1p7q4t DWRR or SRRWS-X6K-S2-MSFC2 2 port 1000BaseX Supervisor Mod 2 (GBIC) 1p2q2tWS-X6K-S2U-MSFC2 2 port 1000BaseX Supervisor Mod 2 (GBIC) 1p2q2tWS-X6K-SUP2-2GE 2 port 1000BaseX Supervisor Mod 2 (GBIC) 1p2q2tWS-SUP32-10GE-3B 2 port 10GBaseX Supervisor module 1p3q8t DWRR or SRRWS-SUP720-3B Supervisor Mod 720 base board 1p2q2t WRRWS-SUP720-3BXL Supervisor Mod 720 base board 1p2q2t WRRWS-SUP720-BASE Supervisor Mod 720 base board 1p2q2t WRRWS-X6148-RJ45V 48 port 10/100BaseTX (RJ-45) 2q2t WRRWS-X6348-RJ-45 48 port 10/100BaseTX (RJ-45) 2q2t WRRWS-X6148A-GE-45AF 48 port 10/100/1000BaseTX (RJ-45) 1p3q8t WRRWS-X6408A-GBIC 8 port 1000BaseX (GBIC),Enhanced QoS 1p2q2t WRRWS-X6416-GBIC 16 port 1000BaseX (GBIC) 1p2q2t WRRWS-X6516A-GBIC 16 port 1000BaseX (GBIC) 1p2q2t WRRWS-X6704-10GE 4 port 10 GE 1p7q8t DWRRWS-X6716-10GE 16 port 10 GE 1p7q4t DWRR or SRRWS-X6748-GE-TX 48 port 10/100/1000 (RJ-45) 1p3q8t DWRRWS-X6748-SFP 48 port 1000Base FX (SFP GBIC) 1p3q8t DWRR3560E 1p3q3t SRRIE3000 Industrial Ethernet Switch 1p3q3t SRR3750 - NME-XD-48ES-2S-P NME-XD-48ES-2S-P: EtherSwitch SM 48 10/100T PoE + 2 SFP 1p3q3t SRR

Table 1: NCAR Queue Types

4. NCAR Traffic Profile

To get a rough sense of the percentage of VoIP traffic on the network vs. other traffic, in/out bps was pulled from all the VLANs at ML, FL, and CG for a 31 day period. For voice vlans, the max values were pulled along with average values to evaluate the worst case scenario. The table below shows percent of VoIP (all traffic on voice vlans) traffic per campus, which was typically less than 1%.

Inbound (%)

Outbound (%)

% CGRA VOIP traffic 0.085 0.055% CGRA MAX VOIP traffic 0.421 0.275

% FLRA VOIP traffic 0.065 0.032% FLRA MAX VOIP Traffic 0.926 0.183

% MLRA VOIP Traffic 0.043 0.042% MLRA MAX VOIP Traffic 0.246 0.655

5. QoS Configuration Recommendations

AutoQoS vs. Manual configuration – AutoQoS will automatically apply QoS configurations on a switch. Lab tests show that AutoQoS selected different buffer sizes, weights and thresholds than what is recommended in the SRND. Manual configuration is recommended.

Input Queuing and Policing – Input queuing and Policing will not be used at this time per the reasons discussed above.

Classification and Marking: The SRND classifications recommendations assume that all traffic will be classified and have configured transmit queues accordingly. To keep things simple, only voice bearer and call signaling traffic will be explicitly prioritized on NCAR’s campus. Queue configurations will therefore need to be adjusted accordingly.

The proposed classification of NCAR’s campus traffic is as follows:

1. Prioritize voice bearer traffic as COS 5, DSCP 46|EF 2. Prioritize call signaling traffic as COS 3, DSCP 24|CS3. 3. Router/switch control traffic (i.e. routing protocols, HSRP, etc) are marked as CS6 by the

router/switch. 4. All other traffic is marked with the default COS 0, DSCP 0.

All of NCAR’s campus traffic will be placed into a 4 level classification policy. Additional classification levels can easily be added should we need to prioritize traffic other than voice.

Application L3 Classification Voice EF, DSCP 46 Network Control CS6, DSCP 48 Call Signaling CS3, DSCP 24 All Other DSCP 0

Trust boundaries will be extended to the IP phones, all other *access* ports will be untrusted. All uplinks between access and distribution/core switches will be trusted. Here is a sample IOS configuration that extends the trust boundary to the IP phone:

Per Interface Configs:

interface gig <mod/port> mls qos vlan-based switchport voice vlan 700 <- Applied on interface Vlan700 no ip address service-policy input ACL-IP-PHONES-policy

Global Configs:

class-map match-all ACL-IP-PHONES-1-class match access-group name ACL-IP-PHONES-1 ! policy-map ACL-IP-PHONES-policy class ACL-IP-PHONES-1-class trust dscp

Here is a sample IOS configuration to enable trust on an uplink (per interface):

mls qos trust cos

COS to DSCP and DSCP to COS mappings will remain unchanged.

Transmit Queuing, Generic Proposal:

Transmit queuing is partially determined by the hardware (1p3q8t, 2q2t, etc) and IOS version being used on the switch/router. While different types of queuing and thresholds (WRR, SRR, WRED, WTD, etc) are implemented on the different platforms, the user has the ability to control buffer size, weights, and thresholds regardless of the type. The queues must behave in an appropriate manner as determined by the type of traffic (DSCP/COS) assigned to the queue. Based on the proposed classification of traffic, here are a couple of generic queue design proposals.

Option 1:

Use the SRND as a guide for designing queues, making adjustments as necessary based on our classification policy.

1. Mapping of DSCP/COS to queues will be performed per the SRND even though NCAR will only use 4 classification levels. In the future, should we decide to prioritize traffic other than voice, the mappings will already be in the standard configuration. I don’t see any real downside to doing this.

2. Buffer space will only be allocated to queues with DSCP/COS values for which traffic has been explicitly mapped. For example, 1p7q8t has a total of 8 queues available – 1 priority and 7 other queues. The proposed classification calls for at most 4 different queues, so 4 of the 1p7q8t queues will have a buffer size of ‘0’ assigned since no traffic would ever be assigned to these queues. The SRND will be used as a guide to determine buffer size. It will be necessary to deviate from the SRND because we will be marking all traffic except voice as DSCP/COS 0 and a larger buffer will be needed to accommodate this traffic.

3. Bandwidth servicing weights will only be allocated to queues that have buffers assigned. The SRND will be used as a guide to determine bandwidth servicing weights. For the most part, these should follow pretty closely with the SRND except where buffers have been set to 0.

4. Buffer thresholds will be explicitly defined to queue’s that have buffers assigned, for all DSCPs assigned to the queue regardless of whether or not any traffic will be marked with that DSCP value. The SRND will be used as a guide. For example, the SRND for 1p3q8t queues assigns CS7 to q3t7. The proposed classification does not include CS7, however, the q3t7 threshold will still be set. This will have no affect on traffic that is using the queue.

PROs:

• Follows Cisco’s recommendation for queue design which is standards based.

• Queue design is known and less likely to change due to IOS version changes.

CONs:

• Cisco’s queue design is based on marking/classifying ALL traffic. NCAR is only marking/classifying VOIP traffic, so adjustments will need to be made for buffers, weights, and thresholds.

• Queue design results in 14+ lines of config per port depending on queue type.

Option 2 (RECOMMENDED):

As a sanity check, we ran option1 past David and Option 2 came from that review. Option 2 primarily relies on the default values that result from enabling the ‘mls qos’ global command (Currently implemented in NCAR’s campus network CATOS devices).

1. Use the default mapping of DSCP/COS to queues that result from enabling the ‘mls qos’ global command except where it differs for DSCP 24 (call signaling) and DSCP 46 (voice traffic). In those cases, follow the SRND.

2. Use the default buffers, bandwidth servicing weights, and thresholds that result from enabling the ‘mls qos’ global command. For the most part, these settings are in line with SRND and where they deviate, it’s hard to quantify how it might affect QoS.

PROs:

• Easier admin, fewer config lines per port

• Still maps COS to queues per SRND

CONs:

• Queue design could change, if defaults change due to IOS upgrade

• Depending on queue type, some resources could go unused i.e. buffers, but it’s hard to quantify how this would affect QoS (see 1p3q8t example below).

Note that once a Tx queuing option has been agreed upon, it will need to be applied to all queue types described in Table 1.

Transmit Queuing – 1p2q2t:

Below are some examples of 1p2q2t (older 1G uplink queue type) queue design based on option 1 and option 2. Option 1 compares three queue designs: SRND – queue design if the SRND is exactly followed, Default – queue design created by enabling ‘mls qos’ global command (same for IOS and CATOS), and Proposed – the proposed design as described above. Option 2 compares two queue designs: Default – queue design created by enabling ‘mls qos’ global command, and Proposed – the proposed design as described above.

The grayed out COS/DSCP values represent unused COS/DSCP values. For example, no traffic will be marked as ‘AF21, CS2, COS2’. The blue and orange cell colors are used to visually separate the queues and have no other meaning.

OPTION 1 SRND

n/a EF, Cos5 n/a

70 CS7; Cos7CS6; Cos6AF31,CS3; Cos3AF21,CS2; Cos2AF41,CS4; Cos4

30 DF; Cos0 q1t2 80:100CS1; Cos1 q1t1 40:80

BW Servicing Weight1p2q2t - Buffer Size% & COS to Threshhold Map

Threshold Value

Queue 1 - 30%

Priority Queue - 30%

q2t2 80:100

q2t1 40:80

Queue 2 - 40%

Default

n/a EF, Cos5 n/a

255 CS7; Cos7 q2t2 70:100CS6; Cos6AF41,CS4; Cos4

100 (5 catos) AF21,CS2; Cos2AF31,CS3; Cos3CS1; Cos1DF; Cos0

BW Servicing Weight1p2q2t - Buffer Size% & COS to Threshhold Map

Threshold Value

Priority Queue - 15%

Queue 2 - 15%

Queue 1 - 70%

40:70q2t1

q1t2 70:100

q1t1 40:70

Proposed (based on SRND)

n/a EF, Cos5 n/a

70 CS7; Cos7CS6; Cos6AF31,CS3; Cos3AF21,CS2; Cos2AF41,CS4; Cos4

30 DF; Cos0 q1t2 80:100CS1; Cos1 q1t1 40:80

BW Servicing Weight1p2q2t - Buffer Size% & COS to Threshhold Map

Threshold Value

Priority Queue - 15%

q2t2 80:100

q2t1 40:80

Queue 2 - 15%

Queue 1 - 70%

Option 1: Proposed IOS Queue Configuration Commands (per interface):

wrr-queue cos-map 2 2 6 7 wrr-queue cos-map 2 1 2 3 4 wrr-queue cos-map 1 2 0 wrr-queue cos-map 1 1 1 wrr-queue bandwidth 30 70 wrr-queue random-detect 1 wrr-queue random-detect 2 wrr-queue random-detect min-threshold 1 40 80 wrr-queue random-detect max-threshold 1 80 100 Option 1: Proposed CATOS Queue Configuration Commands (per queue type): set qos map 1p2q2t tx 2 1 cos 2 3 4 set qos map 1p2q2t tx 2 2 cos 6 7 set qos map 1p2q2t tx 1 1 cos 1 set qos map 1p2q2t tx 1 2 cos 0 set qos wred 1p2q2t tx queue 2 40:80 80:100 set qos wred 1p2q2t tx queue 1 40:80 80:100 set qos wrr 1p2q2t 30 70

Option 2 Default

n/a EF, Cos5 n/a

255 CS7; Cos7 q2t2 70:100CS6; Cos6AF41,CS4; Cos4

100 (5 catos) AF21,CS2; Cos2AF31,CS3; Cos3CS1; Cos1DF; Cos0

BW Servicing Weight1p2q2t - Buffer Size% & COS to Threshhold Map

Threshold Value

Priority Queue - 15%

Queue 2 - 15%

Queue 1 - 70%

40:70q2t1

q1t2 70:100

q1t1 40:70

Proposed (based on default)

n/a EF, Cos5 n/a

255 CS7; Cos7 q2t2 70:100CS6; Cos6AF41,CS4; Cos4AF31,CS3; Cos3

100 AF21,CS2; Cos2 q1t2 70:100CS1; Cos1DF; Cos0

Queue 2 - 15%

Queue 1 - 70%

q2t1 40:70

BW Servicing Weight1p2q2t - Buffer Size% & COS to Threshhold Map

Threshold Value

Priority Queue - 15%

q1t1 40:80

Option 2: Proposed IOS Queue Configuration Commands (per interface):

wrr-queue cos-map 2 1 3

Option 2: Proposed CATOS Queue Configuration Commands (per queue type): set qos map 1p2q2t tx 2 1 cos 3 set qos wrr 1p2q2t 100 255

Transmit Queuing 1p3q8t:

Below are some examples of 1p3q8t queue design based on option 1 and option 2. Option 1 compares three queue designs: SRND – queue design if the SRND is exactly followed, Default – queue design created by enabling ‘mls qos’ global command (different defaults for IOS and CATOS), and Proposed – the proposed design as described above.

Option 2 compares two queue designs: Default – queue design created by enabling ‘mls qos’ global command (different defaults for IOS and CATOS), and Proposed – the proposed design as described above.

The grayed out COS/DSCP values represent unused COS/DSCP values. For example, no traffic will be marked as ‘AF21,CS2,COS2’

OPTION 1

SRND

EF, Cos5 n/a

CS7; Cos7 q3t5 90:100CS6; Cos6 q3t4 80:90AF31,CS3; Cos3 q3t3 70:80AF21,CS2; Cos2 q3t2 60:70AF41,CS4; Cos4 q3t1 50:60

DF; Cos0 q2t1 80:100

CS1; Cos1 q1t1 80:100

1p3q8t - Buffer Size% & COS to Threshhold Map

n/a

70

25

5

BW Servicing Weight

Queue 2 - 25%

Queue 3 - 40%

Priority Queue - 30%

Queue 1 - 5%

Threshold Value

Default CatOS

EF, Cos5 n/a

CS7; Cos7CS6; Cos6AF41,CS4; Cos4 q3t5 60:90AF31,CS3; Cos3 q3t3 50:80

CS1; Cos1AF21,CS2; Cos2

DF; Cos0 q1t1 70:100

70:100

Queue 2 - 15%

200

100

BW Servicing Weight

1p3q8t - Buffer Size% & COS to Threshhold Map

Threshold Value

Priority Queue - 5%n/a

20

q2t1 70:100

Queue 3 - 15%

Queue 1 - 65%

q3t7

Proposed (based on SRND)

EF, Cos5 n/a

CS7; Cos7 q3t5 100:100CS6; Cos6 q3t4 90:100AF31,CS3; Cos3 q3t3 80:90AF21,CS2; Cos2 q3t2 70:80AF41,CS4; Cos4 q3t1 70:80

DF; Cos0 q2t1 80:100

CS1; Cos1 q1t1 80:100

BW Servicing Weight

1p3q8t - Buffer Size% & COS to Threshhold Map

Threshold Value

Priority Queue - 10%n/a

Queue 3 - 15%

Queue 2 - 75%

0Queue 1 - 0%

60

40

Default IOS

n/a EF, Cos5 n/a

CS7; Cos7CS6; Cos6

AF31,CS3; Cos3AF41,CS4; Cos4CS2:Cos2 q2t1 40:70

40:70

CS1; Cos1 q1t2 70:100DF; Cos0 q1t1 40:70

BW Servicing Weight

1p3q8t - Buffer Size% & COS to Threshhold Map

Threshold Value

Priority Queue - 15%

Queue 1 - 50%

q3t1 70:100

Queue 3 - 15%

Queue 2 - 20%

200

150

100

q2t2 70:100

Option 1: Proposed IOS Queue Configuration Commands (per interface):

wrr-queue cos-map 3 5 7 wrr-queue cos-map 3 4 6 wrr-queue cos-map 3 3 3 wrr-queue cos-map 3 2 2 wrr-queue cos-map 3 1 4 wrr-queue cos-map 2 1 0 wrr-queue queue-limit 0 75 15 wrr-queue bandwidth 0 40 60 wrr-queue random-detect 1 wrr-queue random-detect 2 wrr-queue random-detect 3 wrr-queue random-detect min-threshold 1 80 70 70 70 70 70 70 70 wrr-queue random-detect max-threshold 1 100 100 100 100 100 100 100 100 wrr-queue random-detect min-threshold 2 80 70 70 70 70 70 70 70 wrr-queue random-detect max-threshold 2 100 100 100 100 100 100 100 100 wrr-queue random-detect min-threshold 3 70 70 80 90 70 70 70 70 wrr-queue random-detect max-threshold 3 80 80 90 100 100 100 100 100

Option 1: Proposed CATOS Queue Configuration Commands (per queue type): set qos map 1p3q8t tx 1 1 cos 1 set qos map 1p3q8t tx 2 1 cos 0 set qos map 1p3q8t tx 3 1 cos 4 set qos map 1p3q8t tx 3 2 cos 2 set qos map 1p3q8t tx 3 3 cos 3 set qos map 1p3q8t tx 3 4 cos 6 set qos map 1p3q8t tx 3 5 cos 7 set qos txq-ratio 1p3q8t 0 75 15 set qos wrr 1p3q8t 0 40 60 set qos wred 1p3q8t tx queue 3 70:80 40:70 70:100 100:100 100:100 100:100 100:100 100:100 set qos wred 1p3q8t tx queue 2 80:100 100:100 100:100 100:100 100:100 100:100 100:100 100:100 set qos wred 1p3q8t tx queue 1 80:100 100:100 100:100 100:100 100:100 100:100 100:100 100:100

Option 2 Default CatOS

EF, Cos5 n/a

CS7; Cos7CS6; Cos6AF41,CS4; Cos4 q3t5 60:90AF31,CS3; Cos3 q3t3 50:80

CS1; Cos1AF21,CS2; Cos2

DF; Cos0 q1t1 70:100

70:100

Queue 2 - 15%

200

100

BW Servicing Weight

1p3q8t - Buffer Size% & COS to Threshhold Map

Threshold Value

Priority Queue - 5%n/a

20

q2t1 70:100

Queue 3 - 15%

Queue 1 - 65%

q3t7

Proposed (based on Current/Default)

n/a EF, Cos5 n/a

AF31,CS3; Cos3 q3t3 70:100CS7; Cos7CS6; Cos6

AF41,CS4; Cos4 q2t2 70:100CS2:Cos2 q2t1 40:70

CS1; Cos1 q1t2 70:100DF; Cos0 q1t1 40:70

1p3q8t - Buffer Size% & COS to Threshhold Map

Threshold Value

Priority Queue - 15%

Queue 3 - 15%

BW Servicing Weight

Queue 2 - 20%

200

150

100

70:100q3t1

Queue 1 - 50%

Default IOS

n/a EF, Cos5 n/a

CS7; Cos7CS6; Cos6

AF31,CS3; Cos3AF41,CS4; Cos4CS2:Cos2 q2t1 40:70

40:70

CS1; Cos1 q1t2 70:100DF; Cos0 q1t1 40:70

BW Servicing Weight

1p3q8t - Buffer Size% & COS to Threshhold Map

Threshold Value

Priority Queue - 15%

Queue 1 - 50%

q3t1 70:100

Queue 3 - 15%

Queue 2 - 20%

200

150

100

q2t2 70:100

Option 2: Proposed IOS Queue Configuration Commands (per interface):

wrr-queue cos-map 3 3 3 wrr-queue cos-map 2 2 4 Option 2: Proposed CATOS Queue Configuration Commands (per queue type): set qos map 1p3q8t tx 1 2 cos 1 set qos map 1p3q8t tx 2 1 cos 2 set qos map 1p3q8t tx 2 2 cos 4 set qos map 1p3q8t tx 3 1 cos 6 7 set qos map 1p3q8t tx 3 3 cos 3 set qos txq-ratio 1p3q8t 50 20 15 set qos wrr 1p3q8t 100 150 200 set qos wred 1p3q8t tx queue 3 70:100 40:70 70:100 50:80 60:90 60:90 70:100 70:100 set qos wred 1p3q8t tx queue 2 40:70 70:100 100:100 100:100 100:100 100:100 100:100 100:100 set qos wred 1p3q8t tx queue 1 40:70 70:100 100:100 100:100 100:100 100:100 100:100 100:100 ****Recommend using Option 2****

Transmit Queuing, 3560E/3750/IE3000:

The 3560E uses SRR (shaped round-robin) queue’s with WTD (weighted tail drop). Weighted Tail Drop is a simpler congestion avoidance mechanism than WRED. The weight sets the % of the queue that, when full, will begin tail drop. The queue’s can be configured as 4qt3 or 1p3q3t. We’ll configure it to use 1p3q3t.

We will add this configuration information based upon the decisions for the other module types.

Transmit Queuing, Voice Gateways:

As of 12.2.11T and later, the voice gateways, by default will mark voice traffic with DSCP 46 | EF and call signaling with DSCP 24 | CS3.

Yet another queuing configuration format is used for voice gateways - HQF (Hierarchical queuing Framework). Mar-26a-c1-gw uses HQF on its T1 link.

The SRND’s for Enterprise QoS and Unified Communications don’t explicitly cover this. From a BW perspective, the links are lightly used and even if all PRI’s were in use, the max load on the uplinks would be : 3 pri * 24 calls/pri * 80kbps/call = 7.68 Mbps. For Gig uplinks, that would consume less than 1% of available BW.

Recommendation: No changes.

Applying QoS Configurations:

Smart port macros are available, and we could create one macro per queue configuration type. The other option is to follow our current setup of configuring the modules as we insert them. The settings are determined by the modules, so this would be a good way to keep us in sync as we change modules.

http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/configuration/guide/smrtport.html

When the macro is applied to the interface, the configuration shows all the commands and which macro was applied. Here is an example of a 1p3q8t port configured using a macro:

interface GigabitEthernet3/1 no ip address shutdown wrr-queue bandwidth 0 40 60 wrr-queue queue-limit 0 65 20 wrr-queue random-detect min-threshold 2 80 100 100 100 100 100 100 100 wrr-queue random-detect min-threshold 3 100 100 80 90 100 100 100 100 wrr-queue random-detect max-threshold 2 100 100 100 100 100 100 100 100 wrr-queue random-detect max-threshold 3 100 100 90 100 100 100 100 100 wrr-queue cos-map 1 1 1 wrr-queue cos-map 2 1 0 wrr-queue cos-map 3 1 4 wrr-queue cos-map 3 2 2 wrr-queue cos-map 3 3 3 wrr-queue cos-map 3 4 6 wrr-queue cos-map 3 5 7 macro description pd-test end

Note that MAX thresholds should be configured before MIN thresholds. This will avoid errors where the new MIN threshold might be greater than the existing MAX threshold.

6. Monitoring QoS

QoS configurations need to be validated to ensure packets are properly marked and monitored to ensure they are operating as designed.

To validate configurations, grab a packet trace (or use netflow) at the end points and check that in/out bound DSCP markings are set correctly (SCCP = CS3; RTP = EF routing protocols = CS6, and all other set to 0).

Locations to check

o Switch ports that connect to CMs and voicemail servers o Switch ports that connect to gateways o PC port on phone (random sampling).

Config checkers should be used to validate that configurations are applied properly.

Queue drops can be monitored per interface, per queue and per threshold. Excessively high drops in a particular queue could indicate the need to adjust queuing parameters. The CLI command 'show queuing interface <blah> detailed' provides per interface, per queue and threshold drops, however, the "detailed" portion of this command is only available in IOS ver x.x.x SXI.

These are the OIDs:

IOS - 1.3.6.1.4.1.9.9.580.1.5.1.1.4 - CISCO-SWITCH-QOS-MIB, csqIfStatsDropPkts

CATOS - .1.3.6.1.4.1.9.9.179.1.4.5.1.4 - CISCO-CATOS-ACL-QOS-MIB

Here is an IOS example: SNMPv2-SMI::enterprises.9.9.580.1.5.1.1.4.191.2.1.1 = Counter64: 9124503 SNMPv2-SMI::enterprises.9.9.580.1.5.1.1.4.191.2.1.2 = Counter64: 0 SNMPv2-SMI::enterprises.9.9.580.1.5.1.1.4.191.2.1.3 = Counter64: 0 SNMPv2-SMI::enterprises.9.9.580.1.5.1.1.4.191.2.1.4 = Counter64: 0 SNMPv2-SMI::enterprises.9.9.580.1.5.1.1.4.191.2.1.5 = Counter64: 0 SNMPv2-SMI::enterprises.9.9.580.1.5.1.1.4.191.2.1.6 = Counter64: 0 SNMPv2-SMI::enterprises.9.9.580.1.5.1.1.4.191.2.1.7 = Counter64: 0 SNMPv2-SMI::enterprises.9.9.580.1.5.1.1.4.191.2.1.8 = Counter64: 0 SNMPv2-SMI::enterprises.9.9.580.1.5.1.1.4.191.2.2.1 = Counter64: 0 SNMPv2-SMI::enterprises.9.9.580.1.5.1.1.4.191.2.2.2 = Counter64: 142487 SNMPv2-SMI::enterprises.9.9.580.1.5.1.1.4 = csqIfStatsDropPkts. The next number is the ifIndex of the interface, followed by the direction (1=inbound, 2=outbound), then by the queue, and finally the threshold. We currently lack the ability to determine amount of traffic per COS/DSCP which could be useful for planning and troubleshooting purposes. This would be remedied if we ran netflow internally.

7. Telepresence (TP) and QoS at NCAR

Here is a brief review of some key characteristics of Telepresence (TP):

• TP reserves 192.168.0.0/24 thru 192.168.3.0/24 for internal use but does not have a default gateway and therefore cannot route beyond the TP codec.

• TP appears as two SIP endpoints in CM, 1) the TP codec 2) 7975 IP phone

• Bandwidth requirements for TP: 1.628 Mbps to 12.756 Mbps per stream depending on configuration

• End to End Jitter target is < 10ms. Jitter greater than 20ms peak to peak will downgrade call quality. Jitter 40ms or greater call will terminate.

• Target packet loss is 0.5%

• Marks video traffic as CS4, voice as CS5, and signaling as CS3. CS4 marked traffic is to be assigned to the priority queue

• The SRND recommends a separate cluster for TP, unless there are no other video devices on the existing cluster

There are many different ways to deploy TP, i.e. point-to-point, point-to-multipoint, intra-enterprise, inter-enterprise, etc. If TP were to be deployed on NCAR’s campus, qos configurations would also have to be adjusted. Using the QoS configuration recommendations from above as a foundation, here is an overview of the changes that would be needed assuming an Intra-NCAR TP deployment:

• Input Queuing: No changes

• Classification and Marking: Extend trust boundary to TP equipment; configure CM to mark video traffic as CS4

• Policing: No changes

• Transmit Queuing: Re-map CS4 marked traffic to the priority queue; possibly re-allocate buffers

8. Telepresence (TP) and QoS at the FRGP

The FRGP uses DSCP markings to facilitate commodity policers at the FRGP. Inbound commodity traffic, destined for an FRGP/UPOP member not directly connected to the same router as the commodity provider, is marked as DSCP 1. To implement the policer strategy, QoS was globally enabled on the 6500s at the FRGP. QoS is NOT implemented at the FRGP to provide preference to specific traffic types i.e. voice, video, etc except for VOIP traffic destined to/from RAF. In this case, we trust the QoS markings along the path.

Telepresence (or other video conferencing) capabilities appear to be on the horizon for FRGP member institutions either via NLR, between FRGP members, or a single FRGP member with multiple connections

to the FRGP. Given the strict latency, jitter, and packet loss requirements for TP, it’s possible that member institutions will request preferential treatment of TP traffic transiting the FRGP routers.

A single FRGP member requesting preferential treatment for video will impact all FRGP members in all but the simplest case (ports are on the same router) because some QoS policy would need to be applied on ports connecting l3-gw-1 <-> frgp-gw-2 <-> frgp-gw-3. At that point, one member’s traffic is being preferred over all of the other member’s traffic.

In all the use cases mentioned above, the FRGP is the middleman transiting traffic from participant A to participant B. A generic path may look like this:

particpantA’s internal network <-> Tail circuit provider network<->FRGP<-> Tail Circuit provider network<->participantB’s internal network. In some cases, there may be a direct connection to a participant’s network, eliminating the tail circuit provider network.

Telepresence marks video CS4, voice CS5, and signaling as CS3 (this is configurable at the TP end). Both CS4 and CS5 are recommended to be placed in the priority queue. Ultimately we don’t really care how Telepresence is deployed in the participant’s networks, but we will need to know the COS/DSCP types and bandwidth requirements for each COS/DSCP marking.

Should it be deemed necessary to implement QoS at the FRGP, a QoS policy will need to be created that clearly defines the number and types of queues and COS/DSCP mappings to those queues. When deploying QoS, coordination with participants and tail circuit vendors will be necessary to determine a mapping between the different queue structures. If they don’t match, then traffic will need to be remarked either inbound and/or outbound to ensure similar treatment across different administrative boundaries. Participants will be responsible for re-marking.

Using the QoS configuration recommendations from above as a foundation, here is an overview of the changes that would be needed to create an FRGP QoS policy recommendation:

• Input Queuing: No changes

• Classification and Marking: o Inbound Option 1: Extend trust boundaries to participant/tail circuit vendor. Any DSCP

1 marked traffic from a participant/tail circuit vendor will need to be remarked to DSCP 0 to avoid conflict with the commodity policer. Pros: easier to config and maintain, participants responsible for correctly

marking traffic Cons: potential for abuse

o Inbound Option 2: explicitly classify and mark (example: a policy statement that identifies SIP traffic and marks it as CS3) Pros: clearly identifies what traffic will be marked and at what priority Cons: more complex policy statements, FRGP engineers responsible for

correctly marking traffic

Recommend Inbound Option 1 – Note that for either option, DSCP values might have to be re-marked on a per interface basis depending on how a participant’s/tail circuit provider’s queue strategy aligns with the FRGP’s. Participants should be responsible for remarking where possible.

o Outbound: Any packet marked DSCP 1 by the commodity policer will need to be remarked DSCP 0. If not there is a possibility that a participant/tail circuit vendor would deferentially treat this traffic as it is typically assigned to the Scavenger Class. .

• Policing: o Inbound Option 1: No policing of COS/DSCP marked traffic

Pro: easier to admin Con: Potential for abuse i.e. mark all traffic as priority

o Inbound Option 2: Place strict limits on COS/DSCP marked traffic Pro: Prevents potential abuse Con: more complicated policers

Recommend Inbound Option 2

o Outbound: No change

• Transmit Queuing: o Option 1: Come up with FRGP QoS templates for services and how they will be mapped.

Similar to QMOE QoS templates. Pro: limited set of configurations to maintain, easier to admin Con: less flexible

o Option 2: Configure queuing on a per participant/tail circuit vendor basis. Pro: Very flexible Con: More configs to maintain, but not that many members

o Its possible that neither options is viable. The QoS configuration on some cards is applied per ASIC (8 ports per ASIC on a 6148a) and not per port. More investigation is needed.

Open questions:

• Charge members $$ for priority traffic??

9. References

1) Enterprise QoS SRND

http://www.cisco.com/en/US/docs/solutions/Enterprise/WAN_and_MAN/QoS_SRND/QoS-SRND-Book.html

2) Medianet Campus Qos Design 4.0

http://www.cisco.com/en/US/docs/solutions/Enterprise/WAN_and_MAN/QoS_SRND_40/QoSCampus_40.html

3) Cisco Unified Communications SRND Based on Cisco Unified Communications Manager 6.x

http://www.cisco.com/en/US/docs/voice_ip_comm/cucm/srnd/6x/uc6_0.html

4) Comparison of the Cisco Catalyst and Cisco IOS Operating Systems for the Cisco Catalyst 6500 Series Switch

http://www.cisco.com/en/US/customer/prod/collateral/switches/ps5718/ps708/prod_white_paper09186a00800c8441.html

5) Implementing Quality of Service Policies with DSCP

http://www.cisco.com/en/US/tech/tk543/tk757/technologies_tech_note09186a00800949f2.shtml

6) Quality of Service - The Differentiated Services Model

http://www.cisco.com/en/US/prod/collateral/iosswrel/ps6537/ps6558/ps6610/product_data_sheet0900aecd8031b36d.html

7) Understanding How Routing Updates and Layer 2 Control Packets Are Queued on an Interface with a QoS Service Policy

http://www.cisco.com/en/US/customer/tech/tk543/tk544/technologies_tech_note09186a0080094612.shtml

8) Understanding Quality of Service on Catalyst 6000 Family Switches

http://www.cisco.com/en/US/customer/tech/tk543/tk762/technologies_white_paper09186a00800b0828.shtml#eigth

9) Cisco AutoQoS Data Sheet

http://www.cisco.com/en/US/customer/tech/tk543/tk759/tech_digest09186a00801348a6.html

10) Monitoring Voice over IP Quality of Service

http://www.cisco.com/en/US/customer/tech/tk543/tk759/technologies_tech_note09186a0080094bc3.shtml

11) Cisco TelePresence Network Systems 2.0 Design Guide

http://www.cisco.com/en/US/docs/solutions/Enterprise/Video/TP-Book.html