ASR9000 Troubleshooting architecture

ASR9000 Troubleshooting architectures

BRKSPG-2904

Xander Thuijs CCIE#6775 Principal Engineer

Highend Routing and Optical Group

© 2014 Cisco and/or its affiliates. All rights reserved. BRKSPG-2904 Cisco Public

Agenda • Introduction

– Architecture of SW and HW forwarding

• Troubleshooting Packet Forwarding

– Fabric

– NPU

– Local Packet Transport Services/LPTS

– Software forwarding/handling

• Troubleshooting L3 forwarding including Mcast

• Satellite

• QoS Architecture and verification

• Usability, XR strategy and SMU’s

3


Acknowledgements

• With contributions from

– Santosh Sharma (Satellite)

– Aleksandar Vidakovic (FIB)

– Sadananda Phadke (Multicast)

– Eddie Chami (Usability)

• YOU

– Thanks for all the feedback on the support forums and last years CL!

4

Introduction Hardware and Software Forwarding architectures


Getting started

• One common architecture between all platforms

• This applies between the different LC types also

– Trident (NP3)

– Typhoon (NP4)

– Tomahawk (NP5)

With packet forwarding troubleshooting

6

nV Satellite

IOS XRv

XR virtualization

9001 9001-S

9904 9006 9912 9010 9922 9000v,901,903


Forwarding paths

• Hardware based forwarding from Linecard to Linecard over the fabric

• Software based

– This is not truly forwarding as XR has no true software forwarding path

– This is for local packet consumption, eg “for me” packets

– These are either handled by • LC CPU

• RP CPU

Inside the CPU SW “forwarding” there are different switching techniques • SPP (software packet path) (somewhat similar to IOS “fast switching”).

• NETIO (aka “process switching)

In principal there are 2 forwarding handlers

7


Software forwarding on classic IOS

4. OSPF doesn’t return in time?

5. HW timer fires Calls function to spit CPU HOG Hw timer is defined with “scheduler max time”

Bit of History on CEF and Fast switching

8

CPU Complex IOS classic

CPU

IOS “main (void)”

“scheduler”

PA

BGP

OSPF

“process”

jump

return

HW timers Interupt

handler

jump

1 2

3

4

5

Software forwarding

Calls a receive interrupt (RX interrupt) to the CPU to handle the packet

Time in the ISR (interrupt service routine) is limited.

We need to turn around this packet quick during the RX interupt.

How? With all features and info? CEF! Prebuilt L2 headers, routing, features all in one entry!

CPU utilization for five seconds: 68% / 30%

Total CPU time used Time spent in ISR

(fast switching)

PA

PA

Interfaces

28% used

Jump procs

Packets not switched in ISR

Are handed over to IP Input (proc switching)


Hardware Forwarding

• Ingress PHY receives frame

• Ingress NPU handles input features (MAC is part of ingress NPU)

• Handed over to Fabric interface ASIC for ingress schedule over fabric

• If typhoon, pass over LC fabric when scheduled to

• Central fabric

– If RSP2/RSP440 this is on the RSP

– If 9922/9910 this is the separate fabric cards

• Egress LC fabric and FIA hand over packet to egress NPU for egress feature processing

• Egress NPU hands over to PHY and transmit

Transit

9


Software forwarding

• If ingress NPU detects this packet is for me.

– How does it determine it is for me?

– The iFIB (internal FIB) portion of the LPTS (local packet transport services)

• LPTS consists of 3 key portions

– Filtering of what can be punted and categorization of the flow • Holes are dynamically pierced in LPTS and categorized into flows

– Director of where it needs to go to • Is it for the LC or RP CPU (or DRP if you have CRS like devices)

– Policing of the flow categorizations. • This policing is done per “flow” and per NPU.

• For instance BGP-established is an LPTS flow. We can have multiple established BGP sessions that leverage this policer.

• After the CPU receives the packet after an RX interrupt the packet is handled by

– SPP, and potentially the

– NETIO chain

The “punt path”

10


“Punt” Packet Flow Overview

Switch Fabric

3x 10G 3x10GE

SFP +

3x10GE

SFP +

Typhoon

Typhoon

3x 10G

3x 10G 3x10GE

SFP +

3x10GE

SFP +

Typhoon

Typhoon

3x 10G

3x 10G 3x10GE

SFP +

3x10GE

SFP +

Typhoon

Typhoon

3x 10G

3x 10G 3x10GE

SFP +

3x10GE

SFP +

Typhoon

Typhoon

3x 10G FIA

FIA

FIA

FIA

Sw

itch

Fa

bric

AS

IC

11

CPU Complex

MEM

I/O FPGA

SDD

NVRAM Boot Flash

USB Disk0/1

Punt FPGA

FIA

CPU

CPU

Trace-points

LPTS

• iFIB

• Policing

Typhoon

Troubleshooting Packet Forwarding: Fabric


Packet Flow Overview

Switch Fabric

Switch Fabric

3x 10G 3x10GE

SFP +

3x10GE

SFP +

Typhoon

Typhoon

3x 10G

3x 10G 3x10GE

SFP +

3x10GE

SFP +

Typhoon

Typhoon

3x 10G

3x 10G 3x10GE

SFP +

3x10GE

SFP +

Typhoon

Typhoon

3x 10G

3x 10G 3x10GE

SFP +

3x10GE

SFP +

Typhoon

Typhoon

3x 10G FIA

FIA

FIA

FIA

Sw

itch

Fa

bric

AS

IC

Ingress

Typhoon

FIA

FIA

FIA

FIA

Sw

itch

Fab

ric

AS

IC

Egress

Typhoon

Ingress

Typhoon

Egress

Typhoon

100GE

MAC/PHY

100GE

MAC/PHY

100G

100G

100G

100G

• Two-stage IOS-XR packet forwarding

• Uniform packet flow: All packet go through central switch fabric

13


The Magic Of The Switch Fabric

• “0” packet loss during RSP failover and OIR

• Physically separated from LC. Resides on RSP or dedicated card (9912, 9922)

• Logically separated from LC and RSP

– All fabric ASICs run in active mode regardless of RSP Redundancy status

– Extra fabric bandwidth and instant fabric switch over

– If the FAB has been previously initiated then even with RP in rommon FABRIC IS ACTIVE!

• Access to fabric controlled using central arbitration.

– One Arbitration ASIC (Arbiter) per RSP

– Both Arbiters work in parallel – both answer to requests for fab access

– FIAs follow active Arbiter, and switch to backup if needed

– Arbiter switchover controlled by low level hardware signaling

Fabric Arbitration and Redundancy

14


RSP1

Fabric Arbitration

15

Arbitration

Crossbar

Fabric

ASIC

Crossbar

Fabric

ASIC

Crossbar

Fabric

ASIC

Crossbar

Fabric

ASIC

Arbitration

Fabric Interface

and VOQ

Fabric Interface

and VOQ

RSP0

1: Fabric Request

3: Fabric Grant

2: Arbitration

4: load-balanced transmission across fabric links

5: credit return


RSP1

Fabric Load Sharing – Unicast

16

Arbitration

Crossbar

Fabric

ASIC

Crossbar

Fabric

ASIC

Crossbar

Fabric

ASIC

Crossbar

Fabric

ASIC

Arbitration

Fabric Interface

and VOQ

Fabric Interface

and VOQ RSP0 4 3 2 1

Unicast traffic sent across first available fabric link to destination (maximizes efficiency)

Each frame (or superframe) contains sequencing information

All destination fabric interface ASIC have re-sequencing logic

Additional re-sequencing latency is measured in nanoseconds

VOQ (shaped at ~14G)

VQI (queue per priority)


RSP1

Fabric Load Sharing – Multicast

17

Arbitration

Crossbar

Fabric

ASIC

Crossbar

Fabric

ASIC

Crossbar

Fabric

ASIC

Crossbar

Fabric

ASIC

Arbitration

Fabric Interface

and VOQ

Fabric Interface

and VOQ RSP0

Multicast traffic hashed based on (S,G) info to maintain flow integrity

Very large set of multicast destinations preclude re-sequencing

Multicast traffic is non arbitrated – sent across a different fabric plane

A1 A2 B1 A3 B2 C1

Flows exit in-order


Fabric Super-framing Mechanism

• Multiple unicast frames from/to same destinations aggregated into one super frame

• Super frame is created if there are frames waiting in the queue, up to 32 frames or when min threshold met, can be aggregated into one super frame

• Super frame only apply to unicast, not multicast

• Super-framing significantly improves total fabric throughput

• Note that fabric counters are showing super frames not individual packets!! • (show controller fabric fia stats loc 0/X/CPU0)

0 (Empty) Max

Super-frame

Packet 1 Jumbo

Packet 1 No super-framing

Packet 1 Max reached Packet 2 Packet 3

Min

Super-frame

Packet 1 Min reached Packet 2

18


Troubleshooting Fabric Forwarding

• Check fabric counters on

– Ingress LC FIA

– Ingress LC crossbar (not applicable to Trident and SIP-700)

– Egress LC crossbar (not applicable to Trident and SIP-700)

– Egress LC FIA

• Check fabric drops

– Ingress LC FIA

– Egress LC FIA

19


RP/0/RSP0/CPU0:A9K-2#sh contr fabric fia stat loc 0/1/CPU0 | utility egrep "\] +[1-9]|^C|FIA"

Tue Jan 28 09:33:40.032 EST

********** FIA-0 **********

Category: count-0

From Unicast Xbar[0] 12684

From Unicast Xbar[1] 12938

To Unicast Xbar[0] 12870

To Unicast Xbar[1] 12752

To Line Interface[0] 25622

From Line Interface[0] 25622

********** FIA-1 **********

Category: count-1

<…>

********** FIA-2 **********

Category: count-2

<…>

********** FIA-3 **********

Category: count-3

<…>

Fabric packet counters

20

Presuming this is not an 9912 or 9922,

how many RSPs does this system

have?


Fabric Drop Counters

• There are four priority levels and four physical XBAR links. FIA egress drop stats are per priority, while FIA ingress drop stats are per XBAR link.

• The FIA egress drop stats, Tail, Hard, WRED, (offsets 0-3) represent fabric priority stats and correspond to: – 0 - high priority level 1

– 1 - high priority level 2

– 2 - low priority

– 3 - not used (asr9k)

• The FIA ingress drop stats offsets (0-3) represent XBAR link stats and correspond to: – 0-1 XBAR links to RSP0 (Trident+RSP2)

– 2-3 XBAR links to RSP1 (Trident+RSP2)

– On Typhoon cards the FIA links with 2 links to the local fabric.

– The local fabric connects with 8x55G links to the RSP fabric

sh controllers fabric fia drops [ingress|egress]

21


RP/0/RSP0/CPU0:A9K-2#sh controllers fabric fia drops ingress location 0/1/CPU0

Tue Jan 28 09:40:35.255 EST

********** FIA-0 ********** Category: in_drop-0

From Spaui Drop-0 0 Header parsing drp 0

accpt tbl-0 0 pw to ni drp 0

ctl len-0 0 ni from pw drp 0

short pkt-0 0 sp0 crc err 8

max pkt len-0 0 sp0 bad align 0

min pkt len-0 0 sp0 bad code 2

From Spaui Drop-1 0 sp0 align fail 3

accpt tbl-1 0 sp0 prot err 0

ctl len-1 0 sp1 crc err 3

short pkt-1 0 sp1 bad align 0

max pkt len-1 0 sp1 bad code 2

min pkt len-1 0 sp1 align fail 3

Tail drp 0 sp1 prot err 0

Vqi drp 0

********** FIA-0 **********

<…>

Fabric drop counters - ingress

22

https://supportforums.cisco.com/document/12135016/asr9000xr-

understanding-and-troubleshooting-fabric-issues-a9k

Ingress drops per link


RP/0/RSP0/CPU0:A9K-2#sh contr fabric fia drops egress location 0/1/CPU0

Tue Jan 28 09:51:03.746 EST

********** FIA-0 ********** Category: eg_drop-0

From Xbar Uc Crc-0 0 Uc dq pkt-len-crc/RO-seq/len error drp 0

From Xbar Uc Crc-1 0 Mc rf crc drp 0

From Xbar Uc Crc-2 0 Mc vl0 src0 buffer full drp 0

From Xbar Uc Crc-3 0 Mc vl1 src0 buffer full drp 0

From Xbar Uc Drp-0 0 Mc vl2 src0 buffer full drp 0




From Xbar Mc Crc-0 0 Mc vl2 src1 buffer full drp 0

From Xbar Mc Crc-1 0 Mc vl3 src1 buffer full drp 0

From Xbar Mc Crc-2 0

From Xbar Mc Crc-3 0

From Xbar Mc Drp-0 0




********** FIA-1 **********

Fabric drop counters - egress

23

Priority

Egress drops per priority


RP/0/RSP0/CPU0:A9K-2#sh controllers fabric crossbar statistics instance 0 location 0/1/CPU0

Tue Jan 28 10:00:38.306 EST

Port statistics for xbar:0 port:0

==============================

Hi priority stats (unicast)

===========================

Ingress Packet Count Since Last Read : 12821

Egress Packet Count Since Last Read : 9590

Low priority stats (multicast)

===========================

Port statistics for xbar:0 port:1

==============================

Hi priority stats (unicast)

===========================

Ingress Packet Count Since Last Read : 12782

Egress Packet Count Since Last Read : 9778

Low priority stats (multicast)

===========================

<…>

Fabric counters on Typhoon LC (3-stage fabric)

24

Troubleshooting Packet Forwarding: NP


NPU Packet Processing - Ingress

5 Stages:

Parse Search Resolve Modify Queueing

Scheduling

•L2/L3 header

packet parsing in

TCAM

•Builds keys for

ingress ACL,

QoS and

forwarding

lookups (uCode)

•Performs QoS

and ACL

lookups in

TCAM tables

•Performs L2

and L3 lookups

in RLDRAM

•Processes Search

results:

•ACL filtering

•Ingress QoS

classification and

policing

•Forwarding (egress

SFP determined)

• Performs L2 MAC

learning

•Adds internal

system headers

•Egress Control

Header (ECH)

•Switch Fabric

Header (SFH)

•Queuing,

Shaping and

Scheduling

functions

•All packets go

through this

stage

26


Troubleshooting NP Forwarding

27

1. Identify interface in question with problem.

2. Identify the mapping from interface to NPU.

3. Examine NP counters.

4. Look for rate counters that match lost traffic rate.

– If none of the counters match the expect traffic, check drops at interface controller

5. Lookup the counter description.

6. If required capture the packet hitting the counter (Typhoon only).

7. If packets are forwarded to the fabric, run fabric troubleshooting steps.

8. Identify egress NP and repeat steps 3 to 6.


Interface to NP mapping

28

RP/0/RSP0/CPU0:A9K-BNG#show controller np ports all loc 0/0/cpU0

Node: 0/0/CPU0:

----------------------------------------------------------------

NP Bridge Fia Ports

-- ------ --- ---------------------------------------------------

0 -- 0 GigabitEthernet0/0/0/0 - GigabitEthernet0/0/0/9

1 -- 1 GigabitEthernet0/0/0/10 - GigabitEthernet0/0/0/19

2 -- 2 TenGigE0/0/1/0

3 -- 3 TenGigE0/0/1/1


Examine NP Counters

29

RP/0/RSP0/CPU0:A9K-2#show controller np counters np0 loc 0/0/CPU0 Node: 0/0/CPU0: ---------------------------------------------------------------- Show global stats counters for NP0, revision v2 Read 57 non-zero NP counters: Offset Counter FrameValue Rate (pps) ------------------------------------------------------------------------------- 16 MDF_TX_LC_CPU 22755787 6 17 MDF_TX_WIRE 1614696 0 21 MDF_TX_FABRIC 1530106 0 29 PARSE_FAB_RECEIVE_CNT 1555034 0 33 PARSE_INTR_RECEIVE_CNT 22026578 6 37 PARSE_INJ_RECEIVE_CNT 335774 0 41 PARSE_ENET_RECEIVE_CNT 2115361 1 45 PARSE_TM_LOOP_RECEIVE_CNT 17539300 5

If delta between received from Fab to TX-wire is almost 0, then everything forwarded

and not punted. If not, we dropped packets, could be ACL, QOS, or for other

reasons (eg PUNT) or large portion of injected traffic.

MDF=Modify

TX transmit

WIRE to the wire

= egress

Packets received

from the fabric


NP Counter Description

RP/0/RSP0/CPU0:A9K-2#sh controllers np descriptions location 0/0/CPU0

Counter Drop Feature Description

------------------------------------------------------------------------------<...>

PARSE_DROP_IPV4_LENGTH_ERROR Drop L3 IPv4 packets dropped on receipt from Ethernet due to IP packet length field inconsistent with L2 frame length

<...>

Look up controller np counters via google OR they can be seen in

show controller np description

30


Interface Controller Stats

31

RP/0/RSP0/CPU0:A9K-2#sh controllers Gi0/0/1/8 stats | e = 0 Statistics for interface GigabitEthernet0/0/1/8 (cached values): Ingress: Input total bytes = 985886125 Input good bytes = 985886125 Input total packets = 8343919 Input pkts 64 bytes = 5536113 Input pkts 65-127 bytes = 2266459 Input pkts 128-255 bytes = 193779 Input pkts 256-511 bytes = 54126 Input pkts 1024-1518 bytes = 293442 Input good pkts = 8343919 Input unicast pkts = 6957281 Input multicast pkts = 1386636 Input broadcast pkts = 2

• Looks at the PHY. Phy would only drop L1 errors.

• MAC inside NPU will drop all identified other errors.

• “input drop other” is accumulate from all NP drops!

• Show in show interface | include drops

• Filter coming up to define what needs to be accounted for in the drop counters

• This pictured card layout is not accurate to 1G interfaces of course, illustrational only :)


Note on Aggregate NP Counters

• Some counters have an index to a port.

• For instance, there is an aggregate count per NPU showing the misses from vlan to subinterface mapping:

– UIDB_TCAM_MISS_AGG_DROP

• There is also a specific counter from which port index these drops came from:

– UIDB_TCAM_MISS_DROP_1

• This means that the second port (starting count from zero) on that NPU experienced that drop.

• So if your show controller np ports tells us that ports X Y and Z are connected to this NPU, and the drop index is _1, then port Y is the culprit.

32


Capturing lost packets in the NPU

• CLI:

– monitor np counter <COUNTER_NAME> <NPU> count <N>

• You can monitor any counter in the NPU on Typhoon generation line cards

• Captured packets are always dropped

• Exists automatically after capturing <N> packets or when timeout is reached

• NPU is reset upon exit (~50ms forwarding stop)

– This will be enhanced later

• Packets subject to punt cannot be captured by this methodology

• Use with care!

33


Capturing lost packets in the NPU - Example RP/0/RSP0/CPU0:A9K-2#monitor np counter PRS_HEALTH_MON np0 count 1 location 0/1/CPU0 Tue Jan 28 10:10:18.824 EST Warning: Every packet captured will be dropped! If you use the 'count' option to capture multiple protocol packets, this could disrupt protocol sessions (eg, OSPF session flap). So if capturing protocol packets, capture only 1 at a time. Warning: A mandatory NP reset will be done after monitor to clean up. This will cause ~50ms traffic outage. Links will stay Up. Proceed y/n [y] > Monitor PRS_HEALTH_MON on NP0 ... (Ctrl-C to quit) Tue Jan 28 10:10:22 2014 -- NP0 packet From OAM RTC: 64 byte packet, bytes[0-3] invalid! 0000: 40 00 00 48 80 00 00 00 00 00 00 00 00 00 10 00 @..H............ 0010: 00 00 00 00 00 00 00 00 04 55 aa 55 aa 55 aa 55 .........U*U*U*U 0020: aa 55 aa 55 aa 55 aa 55 aa 55 aa 55 aa 55 aa 55 *U*U*U*U*U*U*U*U <…> 0190: 00 00 00 00 00 00 00 00 ........ (count 1 of 1) Cleanup: Confirm NP reset now (~50ms traffic outage). Ready? [y] > RP/0/RSP0/CPU0:A9K-2#

34

STOP

Local Packet Transport Services


What is LPTS?

• Local packet transport services handling 4 key functions:

– Securing the control plane with dynamic ACL’s to filter

– Categorize the flow

– Policing the packets that get punted.

– Providing direction as to where a “for me” flow should go to

FILTER

CATEGORIZE POLICE

X

Bgp-known

bgp-cfg

Packet from established neighbor

Packet from configured neighbor

MPP denied packet to control

Plane, eg telnet from not

“known” station

“high” rate/critical priority

“medium” rate/medium priority

DIRECT

RP

LC

36


LPTS characteristics

• Classification and Policing inside NPU

• An LPTS “flow” is policed at a certain rate

• If you have eg multiple BGP neighbors on this NPU they share the same police rate

• For instance:

– All our established neighbors are categorized as “bgp-known”

– The “bgp-known” flow is policed at 10,000 pps

• Police rates can be adjusted on a per LC basis, or globally applicable to all LC’s.

• Each NPU has that configured police rate (rate not shared between all npu’s on LC)

37


LPTS What happens under the hood

38

3x 10G 3x10GE

SFP +

3x10GE

SFP +

Typhoon

Typhoon

3x 10G

3x 10G 3x10GE

SFP +

3x10GE

SFP +

Typhoon

Typhoon

3x 10G

3x 10G 3x10GE

SFP +

3x10GE

SFP +

Typhoon

Typhoon

3x 10G

3x 10G 3x10GE

SFP +

3x10GE

SFP +

Typhoon

Typhoon

3x 10G FIA

FIA

FIA

FIA

Sw

itch

Fa

bric

AS

IC

CPU Complex

CPU

CPU

TCAM

TCP

BGP

LPTS

CLI

Router bgp 100

Neighbor 1.2.3.4

Creates TCP

Socket

Esta

blis

he

d T

CP

so

cke

t ca

lls p

ts


IOS XR Control Plane Local Packet Transport Service

packets in

transit

packets out

for-us packets

App 1

App 2

Local

Stacks bad packets

LC

RP

RP

good packets

LPTS Internal FIB (IFIB) FIB

DCoPP

Dynamic Control

Plane Policing

LPTS

User T

raffic

Control Plane Traffic

LC

LPTS enables applications to reside on any or all RPs, DRPs, or LCs

Active/Standby, Distributed Applications, Local processing

IFIB forwarding is based on matching control plane flows

DCoPP is built in firewall for control plane traffic.

LPTS is transparent and automatic 39


IOS XR LPTS in action

Local port Remote port Rate Priority

Any ICMP ANY ANY 1000 low

any 179 any any 100 medium

Router bgp

neighbor 202.4.48.99

…

!

any 179 202.4.48.99 any 1000 medium

202.4.48.1 179 202.4.48.99 2223 10000 medium

200.200.0.2 13232 200.200.0.1 646 100 medium

LC 1 IFIB TCAM HW Entries

LP

TS

So

cket

BGP

LDP

SSH

LC 2 IFIB TCAM HW Entries …

mpls ldp

…

!

TCP Handshake

ttl_security

ttl

255

LPTS is an automatic, built in firewall for control

plane traffic.

Every Control and Management packet from the line

card is rate limited in hardware to protect RP and LC

CPU from attacks

40


Verifying LPTS policer values RP/0/RP0/CPU0:CRS1-4#show lpts pifib hardware police location 0/7/CPU0

-------------------------------------------------------------

Node 0/7/CPU0:

-------------------------------------------------------------

Burst = 100ms for all flow types

-------------------------------------------------------------

FlowType Policer Type Cur. Rate Def. Rate Accepted Dropped

---------------------- ------- ------- ---------- ---------- ---------- ----------

unconfigured-default 100 Static 500 500 0 0

Fragment 106 Global 0 1000 0 0

OSPF-mc-known 107 Static 20000 20000 0 0

OSPF-mc-default 111 Static 5000 5000 0 0

OSPF-uc-known 161 Static 5000 5000 0 0

OSPF-uc-default 162 Static 1000 1000 0 0

BGP-known 113 Static 25000 25000 18263 0

BGP-cfg-peer 114 Static 10000 10000 6 0

BGP-default 115 Global 0 10000 0 2

PIM-mcast 116 Static 23000 23000 19186 0

PIM-ucast 117 Static 10000 10000 0 0

IGMP 118 Static 3500 3500 9441 0

ICMP-local 119 Static 2500 2500 1020 0

ICMP-app 120 Static 2500 2500 0 0

na 164 Static 2500 2500 72 0

LDP-TCP-cfg-peer 152 Static 10000 10000 0 0

LDP-TCP-default 154 Static 10000 10000 0 0

……cut……

lpts pifib hardware police

flow fragment rate 0

flow bgp default rate 0

41

LPTS usage example the NTP DDOS


Verification of filtering

Example, config used:

ntp server 3.0.0.1 !

LPTS entries:

RP/0/RSP0/CPU0:A9K-BNG#show lpts pifib hardware entry brief location 0/0/cPU0 | i 123 Wed Apr 2 07:54:53.996 EDT 32 IPV4 default UDP any LU(30) any,123 3.0.0.1,any << yes we like that 204 IPV4 default UDP any LU(30) any,123 any,any << crap this we need to close of!

43


Verification of policing

RP/0/RSP0/CPU0:A9K-BNG#show lpts pifib hardware entry loc 0/0/cPU0 | be 3.0.0.1

Wed Apr 2 07:55:55.137 EDT

Source IP : 3.0.0.1

Is Fragment : 0

Interface : any

M/L/T/F : 0/IPv4_LISTENER/0/NTP-known <<<

DestNode : 48

DestAddr : 48 >> 48 in binary is 0000-11-0000

SID : 7 ^^ RP on 9010!

L4 Protocol : UDP

Source port : Port:any

Destination Port : 123

Ct : 0x613110

Accepted/Dropped : 0/0

Lp/Sp : 1/255

# of TCAM entries : 1

HPo/HAr/HBu/Cir : 14876884/200pps/200ms/200pps

State : Entry in TCAM

Rsp/Rtp : 26/40

44

M - Fabric Multicast; L - Listener Tag; T - Min TTL; F - Flow

Type;

DestNode - Destination Node;

DestAddr - Destination Fabric Address;

HPo - Policer; Ct - Stats Counter;

Lp - Lookup priority; Sp - Storage Priority;

HAr - Hardware Average rate limit; HBu - Hardware Burst;

Rsp - Relative sorting position;

Rtp - Relative TCAM position;


Understanding the outputs

• What we looked at is the flow categorization

• The policer of the flow:

RP/0/RSP0/CPU0:A9K-BNG#show lpts pifib hardware police location 0/0/cPU0 | i NTP

Wed Apr 2 07:57:29.351 EDT

FLOW NAME curr.rate accepted dropped

default rate applicable TOS values

NTP-

default 126 Static 200 200 0 0 01234567

NTP-known 180 Static 200 200 0 0 01234567

• Configuration

lpts pifib hardware police flow ntp default rate 1

This adjusts the policer rate to 1pps. (value of 0 is to be fixed up) Explanation of burst!

45

Troubleshooting Punt path (SW forwarding)


Inside the CPU

47

CPU

SPP

CpuCntrl Queues From high/medium/critical etc

BFD NetIO

raw UDP TCP

CDP

HDLC

Netflow

ICMP BGP

FTP

Each grey arrow is an “IPC” inter process call.

Because XR has protected or virtual mem space, each process can’t look in the other guys.

Unless shared mem is used which can be seen by all, this is limited in size however

Memory

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

0

1

0

1

0

1

2


SPP packet captures

48

RP/0/RSP0/CPU0:A9K-BNG#packet-trace spp platform

protocol arp start-capture count 5 location 0/0/cpu0

Wed Mar 12 16:28:30.176 EDT

Sending command: trace filter set 40 1 0x20

Trace filter set for protocol: ARP

Sending command: trace start 5

Started capture for 5 packets

Wrote ASCII trace to /tmp/spp_packet_trace_ascii

Sending command: trace print

Packet serial 3

client/inject:

length 110 phys_int_index -1 next_ctx 0xdeadbeef time

16:28:30.512

00: 00 65 7a 00 00 00 00 70 72 00 00 02 00 5e 00 00

10: 80 00 00 00 00 00 0f 8c 40 c1 0c c8 50 00 00 00

20: 00 00 0d 34 3f ff f2 90 20 04 fe 03 01 04 00 05

30: 00 00 00 00 5e 00 00 00 00 00 00 00 00 04 00 02

40: 40 00 10 34 ff ff ff ff ff ff 66 66 44 44 22 22

RP/0/RSP0/CPU0#run attach 0/0/CPU0

attach: Starting session 1 to node 0/0/CPU0

# spp_ui

spp-ui>

spp-ui> trace filter node client/punt

Node "client/punt" set for trace filtering. Index: 11

spp-ui> trace filter set 52 4 0xD4000001

Modified filter for offset 52 successfully

spp-ui> trace filter set 56 4 0xD4000002

Modified filter for offset 56 successfully

Decoder https://scripts.cisco.com/ui/use/xr_spp_ui_to_pcap

Troubleshooting Packet Forwarding: L3


LC NP

L3 Control Plane Architecture

50 50

LC CPU

RSP CPU

LDP RSVP-TE BGP

ISIS

OSPF

EIGRP

Static

HW FIB Adjacency

ARP/NDP

LSD RIB

AIB

SW FIB

AIB: Adjacency Information Base

RIB: Routing Information Base

FIB: Forwarding Information Base

LSD: Label Switch Database

RSP

LC


LC CPU

LC NP

L3 Control Plane Architecture – 2 Stage Lookup

LC CPU

HW FIB

Adjacency

SW FIB

Ingress LC

ARP/NDP

AIB

SW FIB

LC NP

HW FIB

Egress LC

Packet Packet Packet

Egress Lookup:

Find output interface

+ do L2 rewrite

Adjacency

stored only for

local and virtual

interfaces Recursions

resolved

Ingress Lookup:

Find egress NP +

egress interface ID

51


L3 NP FIB Architecture

Non-Recursive

Prefix Leaf

Recursive

Prefix Leaf

Adjacency

pointer(s) 1xLDI

NR

LDI

R

LDI

Adj OIF

LAG (64

members) Adj OIF

Protected TE

Adj LAG OIF

Backup TE

Adj OIF

Up to 32 ways

Up to 8 or 32 ways

… NR

LDI

NR

LDI

NR-LDI: Non-Recursive Load Distribution Information

R-LDI: Recursive Load Distribution Information

R

LDI

R

LDI

R

LDI

52


HASH

How does ECMP path or LAG member selection work?

• Every packet arriving on an NPU will under go a “HASH” computation. What fields are used is dependent on encap (see overview shortly)

Or even VQI selection, but what the heck is that… Stay focused (covered in QOS)

53

L2 L3 L4 payload

CRC32 32 bits

Path (ECMP) Member (LAG)

path1 path1 path1

path2 path2 path2

8 bits selected 8 bits selected

(3 drawn) 256 buckets

Buckets distributed over available members/paths


ECMP Load balancing A: IPv4 Unicast or IPv4 to MPLS (3)

– No or unknown Layer 4 protocol: IP SA, DA and Router ID

– UDP or TCP: IP SA, DA, Src Port, Dst Port and Router ID

B: IPv4 Multicast

– For (S,G): Source IP, Group IP, next-hop of RPF

– For (*,G): RP address, Group IP address, next-hop of RPF

C: MPLS to MPLS or MPLS to IPv4

– # of labels <= 4 : same as IPv4 unicast (if inner is IP based, EoMPLS, etherheader will follow: 4th label+RID)

– # of labels > 4 : 4th label and Router ID

- (3) L3 bundle uses 5 tuple as “1” (eg IP enabled routed bundle interface)

- (3) MPLS enabled bundle follows “C”

- (1) L2 access bundle uses access S/D-MAC + RID, OR L3 if configured (under l2vpn)

- (2) L2 access AC to PW over mpls enabled core facing bundle uses PW label (not FAT-PW label even if configured)

- FAT PW label only useful for P/core routers

IPv6 uses first 64 bits in 4.0

releases, full 128 in 42

releases

54


Load-balancing scenarios

55

MPLS/IP protocol stack

EoMPLS protocol stack

45 for ipv4


45… (ipv4)

0000 (CW)

41-22-33 (mac)

MPLS vs IP Based loadbalancing

56

L2 MPLS MPLS 4111.0000.

• When a labeled packet arrives on the interface.

• The ASR9000 advances a pointer for at max 4 labels.

• If the number of labels <=4 and the next nibble seen right after that label is

– 4: default to IPv4 based balancing

– 6: default to IPv6 based balancing

• This means that if you have a P router that has no knowledge about the MPLS service of the packet, that nibble can either mean the IP version (in MPLS/IP) or it can be the DMAC (in EoMPLS).

• RULE: If you have EoMPLS services AND macs are starting with a 4 or 6. You HAVE to use Control-Word

• Control Word inserts additional zeros after the inner label showing the P nodes to go for label based balancing.

• In EoMPLS, the inner label is VC label. So LB per VC then. More granular spread for EoMPLS can be achieved with FAT PW (label based on FLOW inserted by the PE device who owns the service).

– Take note of the knob to change the code: PW label code 0x11 (17 dec, as per draft specification). (IANA assignment is 0x17)


Loadbalancing ECMP vs UCMP and polarization

• Support for Equal cost and Unequal cost

• 32 ways for IGP paths

• 32 ways (Typhoon) for BGP (recursive paths) 8-way Trident

• 64 members per LAG

• Make sure you reduce recursiveness of routes as much as possible (static route misconfigurations…)

• All loadbalancing uses the same hash computation but looks at different bits from that hash.

• Use the hash shift knob to prevent polarization.

• Adj nodes compute the same hash, with little variety if the RID is close

– This can result in north bound or south bound routing.

– Hash shift makes the nodes look at complete different bits and provide more spread.

– Trial and error… (4 way shift trident, 32 way typhoon, values of >5 on trident result in modulo)

57


Hash shift, what does it do?

58

HASH

L2 L3 L4 payload

Hash shift 8

Path (ECMP) Member (LAG)

path1 path1 path1

path2 path2 path2

8 bits selected 8 bits selected

(3 drawn) 256 buckets

Buckets distributed over available members/paths

HASH X

Y

Y


FIB Architecture diffs: IOS XR vs IOS

59

• Adjacency vs RIB prefix preference

• Local vs remote ARP learning

https://supportforums.cisco.com/document/12098096/cscse46790-cef-prefers-arp-adjacency-over-rib-next-hop



















LC CPU

LC NP

Adjacency vs Prefix preference

LC CPU

HW FIB

Adjacency

SW FIB

ARP/NDP

AIB

SW FIB

LC NP

HW FIB

Local adjacency exists

adjacency overrides the /32 RIB entry

FIB creates /32 entry via Te0/0/0/0

TenGigE0/0/0/0

10.1.1.2/24

TenGigE0/0/0/1

TenGigE0/1/1/0

10.2.2.2/24

TenGigE0/1/1/1

router static

address-family ipv4 unicast

10.1.1.1/32 TenGigE0/1/1/1 10.2.2.1

RIB creates /32 prefix via Te0/1/1/0

router static


10.1.1.1/32 TenGigE0/1/1/0 10.2.2.1




FIB creates /32 entry via Te0/0/0/0 1

2

60


Switch Fabric

LC CPU

LC NP


LC CPU

HW FIB

Adjacency

SW FIB

ARP/NDP

AIB

SW FIB

LC NP

HW FIB



FIB creates /32 entry via Te0/0/0/0

TenGigE0/0/0/0

10.1.1.2/24

TenGigE0/0/0/1

TenGigE0/1/1/0

10.2.2.2/24

TenGigE0/1/1/1 Packet Packet

Packet w/ DestIP

10.1.1.1

Packet w/ DestIP

10.1.1.1

router static


10.1.1.1/32 TenGigE0/1/1/1 10.2.2.1


router static


10.1.1.1/32 TenGigE0/1/1/0 10.2.2.1


Local adjacency exists (arp learnt 10.1.1.1 on te 0/0/0/0


FIB creates 10.1.1.1/32 entry via Te0/0/0/0 1

2

61



• All IOS XR releases up to and including 5.1.1:

– Adjacency prefix overrides a RIB prefix

• IOS XR Release 5.1.2:

– New CLI introduced to prevent the override (global configuration mode) cef adjacency route override rib disable

• IOS XR Release 5.2.0:

– Default behavior change!

– Adjacency prefix will not override a RIB prefix

62


Local vs remote ARP learning

• IOS XR allows ARP learning when ARP reply/request is outside of the local subnet

• Starting with IOS XR Release 4.3.4:

– New CLI introduced to block out-of-subnet ARP learning: interface g0/0/0/0

arp learning local

63


Troubleshooting L3 Forwarding

• Identify the input interface, slot and NP

• Walk the SW+HW control plane to confirm the expected forwarding decision

– RIB

– SW FIB on ingress LC

– Ingress HW FIB on ingress LC

– Egress HW FIB on egress LC

• If packets are not forwarded as expected by ingress or egress NP, apply NP troubleshooting

• If packets are not crossing the fabric, apply fabric troubleshooting

64


L3 IPv4 Forwarding – Show Commands

RP/0/RSP0/CPU0:EAST-CORE-ASR9K-1#sh route 10.1.2.0/24 Routing entry for 10.1.2.0/24 Known via "static", distance 1, metric 0 Installed Jan 29 05:54:26.182 for 00:00:04 Routing Descriptor Blocks 40.0.3.1, via GigabitEthernet0/7/0/2 Route metric is 0 No advertising protos.

RIB

65



RP/0/RSP0/CPU0:EAST-CORE-ASR9K-1#sh cef 10.1.2.0/24 location 0/4/CPU0 10.1.2.0/24, version 256779, internal 0x4004001 (ptr 0xa3d59b84) [1], 0x0 (0xa3610aa0), 0x440 (0xa4fb4d50) Updated Jan 29 05:54:26.191 remote adjacency to GigabitEthernet0/7/0/2 Prefix Len 24, traffic index 0, precedence routine (0), priority 3 via 40.0.3.1, GigabitEthernet0/7/0/2, 8 dependencies, weight 0, class 0 [flags 0x0] path-idx 0 [0xa51a43b4 0xa5a5a5b8] next hop 40.0.3.1 remote adjacency local label 17012 labels imposed {ImplNull}

SW FIB on Ingress LC

66



RP/0/RSP0/CPU0:EAST-CORE-ASR9K-1#sh adjacency ipv4 g0/7/0/2 detail location 0/7/CPU0 Interface Address Version Refcount Protocol <. . .> Gi0/7/0/2 40.0.3.1 1562 2( 0) ipv4 001b53ff9a99001b53ff9c320800 mtu: 1500, flags 1 0 0 0 packets, 0 bytes RP/0/RSP0/CPU0:EAST-CORE-ASR9K-1#sh cef 10.1.2.0/24 location 0/7/CPU0 10.1.2.0/24, version 256779, internal 0x4004001 (ptr 0xa4d3ee6c) [1], 0x0 (0xa35bdc80), 0x440 (0xa5b1bd50) Updated Jan 29 05:54:26.192 local adjacency 40.0.3.1 Prefix Len 24, traffic index 0, precedence routine (0), priority 3 via 40.0.3.1, GigabitEthernet0/7/0/2, 9 dependencies, weight 0, class 0 [flags 0x0] path-idx 0 [0xa59683c8 0xa5a30268] next hop 40.0.3.1 local adjacency local label 17012 labels imposed {ImplNull}

Adjacency and SW FIB on Egress LC

67



RP/0/RSP0/CPU0:EAST-CORE-ASR9K-1#sh adjacency ipv4 g0/7/0/2 detail location 0/7/CPU0 Interface Address Version Refcount Protocol <. . .> Gi0/7/0/2 40.0.3.1 1562 2( 0) ipv4 001b53ff9a99001b53ff9c320800 mtu: 1500, flags 1 0 0 0 packets, 0 bytes RP/0/RSP0/CPU0:EAST-CORE-ASR9K-1#sh cef 10.1.2.0/24 location 0/7/CPU0 10.1.2.0/24, version 256779, internal 0x4004001 (ptr 0xa4d3ee6c) [1], 0x0 (0xa35bdc80), 0x440 (0xa5b1bd50) Updated Jan 29 05:54:26.192 local adjacency 40.0.3.1 Prefix Len 24, traffic index 0, precedence routine (0), priority 3 via 40.0.3.1, GigabitEthernet0/7/0/2, 9 dependencies, weight 0, class 0 [flags 0x0] path-idx 0 [0xa59683c8 0xa5a30268] next hop 40.0.3.1 local adjacency local label 17012 labels imposed {ImplNull}

Adjacency and SW FIB on Egress LC

68

Entry missing or not what you have expected?

sh adjacency trace all location …

sh cef trace location …


L3 IPv4 Forwarding – Show Commands HW FIB Show Command Structure

69

# Block • Description

1 SW FIB Entry • Same as in the SW FIB command output

2 HW Leaf • Leaf Context: Type of the Leaf entry

• Leaf HW result: features associated (e.g.: URPF, BGP policy

accounting, etc.)

3 NR-LDI • NR-LDI context description

• NR-LDI result (one section per path)

4 TX HW Entry for path #1 • Included only if Adjacency exists on location (i.e. if location is egress

LC)

5 RX HW Entry for path #1 • Always included

• Displays the egress NP and egress interface ID

• 4 & 5 are repeated for every available path



RP/0/RSP0/CPU0:EAST-CORE-ASR9K-1#sh cef 10.1.2.0/24 hardware ingress location 0/4/CPU0 <… SW FIB Entry …> LEAF - HAL pd context : sub-type : IPV4, ecd_marked:0, has_collapsed_ldi:0, collapse_bwalk_required:0, ecdv2_marked:0 Leaf H/W Result: <…> fast_switch_entry rx_adj_field: 0x5300 (LE) egress_ifhandle: 0x12000140 (LE) <…> nrLDI eng ctx: flags: 0x541, proto: 2, npaths: 0, nbuckets: 1 ldi_tbl_idx: 0x0, ecd_ref_cft: 0 RX H/W Result on NP:0 [Adj ptr:0x18 (BE)]: <…> rx_adj_field: 0x5300 UTurn_egress_ifhandle: 0x12000140 <…> RX H/W Result on NP:1 [Adj ptr:0x18 (BE)]: <…>

Ingress HW FIB

70

“Little Endian” vs “Big Endian”: 0x5300 LE == 0x53 BE

Egress interface identification

Egress NP identification


RP/0/RSP0/CPU0:EAST-CORE-ASR9K-1#show uidb index location 0/7/CPU0 gig0/7/0/2 ------------------------------------------------------------------------------------------------------- Location Interface-handle Interface-name Interface-Type Ingress-index Egress-index ------------------------------------------------------------------------------------------------------- 0/7/CPU0 0x12000140 GigabitEthernet0_7_0_2 Main interface 5 5 RP/0/RSP0/CPU0:EAST-CORE-ASR9K-1#sh controllers pm interface G0/7/0/2 location 0/7/CPU0 Ifname(1): GigabitEthernet0_7_0_2, ifh: 0x12000140 : iftype 0xf egress_uidb_index 0x5 ingress_uidb_index 0x5 port_num 0x2 subslot_num 0x0 phy_port_num 0x2 channel_id 0x3 channel_map 0x0 lag_id 0x0 virtual_port_id 0x0 switch_fabric_port 0x53 <…>

L3 IPv4 Forwarding – Show Commands Ingress HW FIB – Confirm egress NP and Interface ID on egress LC

71

Matches the egress_ifhandle from show cef hardware

Matches the rx_adj_field from show cef hardware



RP/0/RSP0/CPU0:EAST-CORE-ASR9K-1#sh cef 10.1.2.0/24 hardware egress location 0/7/CPU0 10.1.2.0/24, version 256779, internal 0x4004001 (ptr 0xa4d3ee6c) [1], 0x0 (0xa35bdc80), 0x440 (0xa5b1bd50) Updated Jan 29 05:54:26.189 local adjacency 40.0.3.1 Prefix Len 24, traffic index 0, precedence routine (0), priority 3 via 40.0.3.1, GigabitEthernet0/7/0/2, 9 dependencies, weight 0, class 0 [flags 0x0] path-idx 0 [0xa59683c8 0xa5a30268] next hop 40.0.3.1 local adjacency local label 17012 labels imposed {ImplNull} LEAF - HAL pd context : sub-type : IPV4, ecd_marked:0, has_collapsed_ldi:0, collapse_bwalk_required:0, ecdv2_marked:0 Leaf H/W Result: <…> fast_switch_entry rx_adj_field: 0x5300 (LE) egress_ifhandle: 0x12000140 (LE) <…> TX H/W Result for NP:0 (index: 0x82c (BE)): <…> uidb_index : 0x500 (LE) l3_mtu : 0xdc05 (LE) prefix_adj_cnt_index: 0x0 dest_mac : 0x001b.53ff.9a99 u1.reserved : 0123456789ab <…>

Egress HW FIB

72

“Little Endian” vs “Big Endian”:

0xdc05 LE == 0x5dc BE == 1500



RP/0/RSP0/CPU0:EAST-CORE-ASR9K-1#sh cef 220.0.0.52/32 hardware egress location 0/7/CPU0 220.0.0.52/32, version 256780, internal 0x4004001 (ptr 0xa4d1f4bc) [1], 0x0 (0xa644e520), 0x440 (0xa5b1bd80) Updated Jan 29 06:30:54.170 local adjacency point2point Prefix Len 32, traffic index 0, precedence routine (0), priority 3 via 41.52.0.2, tunnel-ip52, 5 dependencies, weight 0, class 0 [flags 0x0] path-idx 0 [0xa59cc25c 0x0] next hop 41.52.0.2 local adjacency local label 16005 labels imposed {None} <…> TX H/W Result for NP:0 (index: 0x1421 (BE)): <…> uidb_index : 0x6100 (LE) l3_mtu : 0xc405 (LE) prefix_adj_cnt_index: 0x27670300 GRE Adj: ip_sa : 110.0.0.41 ip_da : 110.0.0.52 tos : 0 ttl : 0xff df : 1 tos_reflect : 1 rsvd flag bits : 0 encap_checksum : 0xe471

Egress HW FIB – what would it show in case of more complex adjacencies?

73

What type of adjacency is this?

Multicast SSM/SM differences


Multicast forwarding considerations

• Difference in forwarding between SSM (source specific multicast) and SM (sparse mode)

• What results are when SSM is misconfigured

– Understanding the drop and punt counters/reasons

• Characteristics of SM forwarding with no receivers

• SSM range and MAP commands

– What they do and what the forwarding impact is

75


PIM-SM Sender Registration (IGMPv2)

Receiver

RP

(S, G) Joins

Source

Shared Tree

(S, G) Register (unicast)

Source Tree

(S, G) State created only

along the Source Tree.

76


Source Specific Multicast Example

Receiver 1

Source

Out-of-band

source directory,

example: web server

B C D

E IGMPv3 (S, G) Join

PIM (S, G) Join

Host learns of source, group/port Last-hop learns of source, group/port Last-hop send PIM (S,G) Join

RP

Default ssm:

50.0.0.10

232.x.x.x

Non-Default ssm:

50.0.0.10

239.x.x.x

CPU A

NPU

• SSM-range to define SSM groups on first hop router

• SSM-MAP to map v2 joins to v3 sources

77


Roadblock to Deploying PIM-SSM You Need to be Aware of • Dependent on IGMPv3

– Microsoft supports IGMPv3 in Windows XP natively

– Many IPTV STB’s are adding support.

• However, not every host supports IGMPv3 today.

– That is where the ssm group mapping comes into play!

• “Workaround”

– Source Mapping • Router maps IGMPv2 Joins in SSM range to well-known sources via DNS or static

configuration

– The ssm MAP links a multicast group v2 join to a multicast source

78


Switch

Fabric

NP0 PHY

NP2

NP3

NP1

FIA B0

B1

PI

M IGMP

MRIB

MFIB

2

1

3

4

CPU

CP

U 5

PHY

PHY

PHY Switch

Fabric

Crossbar

Fabric

ASIC

Crossbar

Fabric

ASIC • Multicast forwarding is based on FGID’s

(fabric group ID’s) and MGID’s (mcast group ID’s).

• The FGID/MGID gives instructions to the forwarding asics to which NPU’s (mgid) and LC’s (fgid) the mcast needs to be replicated

1. Incoming IGMP and PIM packets are punted to RP CPU

2. Protocols (PIM/IGMP) send their Route/OLIST Information to MRIB process to build multicast route/olist table

3. MRIB sends the multicast state information to MFIB process on all LCs

4. MFIB program HW forwarding tables in NP, Bridge FPGA and Fabric interface ASIC

5. Software switched multicast packet or data packet for protocol signaling is sent to local line card CPU

Multicast forwarding in ASR9000

79


IGMP joins

NP0

NP2

NP3

NP1 FIA

B0

B1

CPU PHY

PHY

PHY

PHY

NP0 PHY

NP2 PHY

NP3 PHY

NP1 PHY FIA

CPU

NP0 PHY

NP2 PHY

NP3 PHY

NP1 PHY

CPU

LC2

LC3

IGMP joins

Multicast Source

Switch Fabric

1

FGID/ FPOE

Fabric Replication replicate single copy to LCs which receive IGMP join, based on FGID table

in switch fabric

FIA Replication replicate single copy to Bridge which has IGMP join, based

on MGID table in FIA NP Replication replicate copy per receiver based on multicast FIB

table

1

2

4

FGID – Fabric Group ID

MGID – Multicast Group ID

MFIB – Multicast Forwarding Information Base

B0

B1

FIA B0

B1

IGMP joins

Bridge Replication similar as FIA replication, single copy to NP

3

2

2

3

3

4

MGID MGID

MFIB

Multicast Packet Replication (1) Switch Fabric and Egress LC Replication


MGIDs and FGIDs

• MGID - Multicast Group Identifier

– Unique ID assigned to a multicast group

– Used by Octopus/Bridge to determine replication requirements per multicast group

• FGID - Fabric Group Identifier

– Slotmask used by switch fabric to determine replication to line card/RSP slots

– Assigned to each group by multicast PD control plane


FGID (Slotmask)

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential EDCS:xxxx 5

LC 7

LC 6

LC 5

LC 4

RSP

0

RSP

1

LC 3

LC 2

LC 1

LC 0

Logical

Slot

9 8 7 6 5 4 3 2 1 0

Slot Slot Mask

Logical Physical Binary Hex

LC7 9 1000000000 0x0200

LC6 8 0100000000 0x0100

LC5 7 0010000000 0x0080

LC4 6 0001000000 0x0040

RSP0 5 0000100000 0x0020

RSP1 4 0000010000 0x0010

LC3 3 0000001000 0x0008

LC2 2 0000000100 0x0004

LC1 1 0000000010 0x0002

LC0 0 0000000001 0x0001

Target Linecards FGID Value (10 Slot Chassis)

LC6 0x0100

LC1 + LC5 0x0002 | 0x0080 = 0x0082

LC0 + LC3 + LC7 0x0001 | 0x0008 | 0x0200 = 0x0209

FGID Calculation Examples

FGIDs: 10 Slot Chassis

LC 3

LC 2

LC 1

LC 0

RSP 1

RSP 0

Logical

Slot

Phy

Slot

Number

5

4

3

2

1

0

Slot Slot Mask

Logical Physical Binary Hex

LC3 5 0000100000 0x0020

LC2 4 0000010000 0x0010

LC1 3 0000001000 0x0008

LC0 2 0000000100 0x0004

RSP1 1 0000000010 0x0002

RSP0 0 0000000001 0x0001

FGIDs: 6 Slot Chassis


MGID Tables MGID Bitmasks

83

MGID Bit 1 Bit 0

MGID Bit 1 Bit 0

MGID Bit 1 Bit 0

FIA

Bridge1

NP3

Bridge0

NP2 NP1 NP0


Source specific Multicast configurations

• SSM-range

– By default hardware assumes that the SSM multicast groups are starting with 232.x.x.x

– The SSM range gives the HW the instruction that there are other groups also “SSM enabled”.

• SSM-MAP

– Provides the mapping for non v3 capable receivers to map the mcast group to a specific source (whatever source address that is).

– DNS mapping: allow a name instead of an address for the “dynamic” mapping of groups to sources

84


L3 Multicast PIM SSM (SSM map) • Not all receivers have IGMP-V3 capability

• Use “SSM mapping” feature to translate V2 as V3

join

• Without SSM mapping, V2 joins to 239.x.x.x get

dropped. RP/0/RSP0/CPU0:ASR9K-1#show logging | in 239.

RP/0/RSP0/CPU0:Apr 23 10:33:45.688 : igmp[1154]: Received v2 Report on

GigabitEthernet0_1_0_1.1501 from 40.0.191.1 for 239.1.1.10

RP/0/RSP0/CPU0:Apr 23 10:33:45.688 : igmp[1154]: Discarding report as group

239.1.1.10 is in SSM range

• SSM map, adds source component to V2 join and

creates (S,G) towards source RP/0/RSP0/CPU0:ASR9K-1#show mrib route 239.1.1.10

(51.2.1.2,239.1.1.10) RPF nbr: 51.2.1.2 Flags:

Up: 00:13:54

Incoming Interface List

GigabitEthernet0/0/0/4.300 Flags: A, Up: 00:13:54

Outgoing Interface List

GigabitEthernet0/1/0/1.1501 Flags: F NS LI, Up: 00:13:54

RP/0/RSP0/CPU0:ASR9K-1#

ipv4 access-list PIM-SSM-groups Define additional SSM Groups 10 permit ipv4 239.0.0.0 0.255.255.255 any RP/0/RSP0/CPU0:ASR9K-1#show run multicast-routing multicast-routing address-family ipv4

ssm range PIM-SSM-groups add 239.x.x.x as SSM group RP/0/RSP0/CPU0:ASR9K-1#show running-config router igmp router igmp interface GigabitEthernet0/1/0/1.1501

static-group 239.1.1.10 51.2.1.2 SSM MAP ! RP/0/RSP0/CPU0:ASR9K-1#

Source Gig0/0/0/4.300

igmp v2 Receiver (51.2.1.2, 239.1.1.10)

0/1/0/1.1501

ASR9K-1


L3 Multicast PIM SSM

• Goal of this section is to explain the reasoning for enabling ssm-range

on first hop router if SSM group is different from default group

(232.x.x.x) • This is specifically important if first hop router has V2 and V3 sources If not configured mcast groups outside default range are perceived to be v2 and forwarded to the

Route Processor and might get dropped (if these are actually SSM mcast adds) or sent to the

Rendez-Vous point

• Below sample config, extends SSM group range ipv4 access-list PIM-SSM-groups 10 permit ipv4 <new group> <reverse mask> any ! multicast-routing address-family ipv4 ssm range PIM-SSM-groups ! !


ASR9k behavior on first hop router (closest to source)

• “SSM range” command is needed on first hop router unless:

• Each SSM multicast stream has at least one receiver: • In the presence of receivers, last hop router builds (S,G) towards source.

• Due to (S,G) from last-hop router, traffic get forwarded correctly.

• PIM-SM RP is not configured: • This scenario is applicable when first hop router is enabled for both PIM-SM and

PIM-SSM.

• If PIM-SM RP is not configured and if there are no receivers, packets gets dropped

in NP; Example output below: P/0/RSP0/CPU0:ASR9K-1#show controllers np counters np0 location 0/1/CPU0 41 PARSE_ENET_RECEIVE_CNT 81743 781 332 RSV_DROP_IPM4_ING_RTE_DROP 80396 777

Seen when there is no RendezVous Point configured and there is no registered v3 receiver


ASR9k behavior on first hop router • The PIM-SM RP is configured but “SSM range” is not

RP/0/RSP0/CPU0:ASR9K-1#show controllers np counters np0 location 0/1/C$ Offset Counter FrameValue Rate (pps) ------------------------------------------------------------------------------- 21 MDF_TX_FABRIC 75156 548 <== 41 PARSE_ENET_RECEIVE_CNT 439848 548 RP/0/RSP0/CPU0:EAST-PE-ASR9K-1#

• “show mrib route” shows packets get punted to RP. RP/0/RSP0/CPU0:EAST-PE-ASR9K-1#show mrib route (*,232.0.0.0/8) Flags: D P Default SSM group. No need for SSM range config Up: 02:56:41 (52.0.0.2,239.1.1.2) RPF nbr: 52.0.0.2 Flags: Group is not part of default SSM group Up: 00:00:15 Incoming Interface List GigabitEthernet0/1/0/15 Flags: A, Up: 00:00:15 RP/0/RSP0/CPU0:EAST-PE-ASR9K-1#

Seen when RP is configured, we are sending the traffic to the fabric; forwarding decision is made before we know the receivers in the NP. The fabric receives “erroneous” traffic in this case which will get dropped.


ASR9k behavior on first hop router (Continued… • The PIM-SM RP is configured but “SSM range” is not

• In the absence of SSM range config, packets get punted to RP with FGID Null.

• With FGID null, packets get dropped in Crossbar: [Port NP FIA Fabric Crossbar] RP/0/RSP0/CPU0:ASR9K-1#show mrib route detail (51.2.1.2,239.1.1.10) Ver: 0x659d RPF nbr: 51.2.1.2 Flags:,

PD: Slotmask: 0x0 FGID zero MGID: 16918 Up: 00:25:27 Incoming Interface List GigabitEthernet0/0/0/4.300 Flags: A NS, Up: 00:01:46 Source

RP/0/RSP0/CPU0:ASR9K-1#show controllers fabric crossbar statistics instance 0 loc 0/rsp0/cpu0 Port statistics for xbar:0 port:12 Low priority stats (multicast) =========================== NULL FPOE Drop Count : 71740 = Packets get dropped in RSP cross-bar ============================== RP/0/RSP0/CPU0:ASR9K-1#

• PIM periodically, sends “null registers” to PIM-RP to check if there are new receivers but data packets

are not really sent to RP. (few packets leaked from hw to CPU for this purpose)

RP/0/RSP0/CPU0:EAST-PE-ASR9K-1#show pim traffic PIM Traffic Counters Elapsed time since counters cleared: 01:03:25 Received Sent Null Register 0 67 Sent to RP Register Stop 69 0 Received from RP


ASR9k behavior on first hop router

• The “ssm range” command is configured and no receivers:

• Packets get dropped in ingress NP as there is no most specific (S,G), Example RP/0/RSP0/CPU0:ASR9K-1#show controllers np counters np0 location 0/1/CPU0 41 PARSE_ENET_RECEIVE_CNT 44156 651 333 RSV_DROP_IPM4_ING_RTE_DFLAG_DROP 43833 647 RP/0/RSP0/CPU0:ASR9K-1#show mfib hardware route statistics non-zero location 0/1/cpu0 LC Type: Typhoon A9K-MOD80-TR -------------------------------------------------------------------------- ID: Ingress Drop ED: Egress Drop Source: * Group: 239.0.0.0 Mask:8 ------------------------------------------------------------------------- NP R(packets:bytes)/F(packets:bytes)/P(packets)/ID(packets)/ED(packets) ------------------------------------------------------------------------- 0 0:0 / 0:0 / 0 / 19782 / 0 ------------------------------------------------------------------------- No OLIST interfaces found for this route RP/0/RSP0/CPU0:ASR9K-1#

This is the situation where you want to be, dropped in HW, no punting. If there is a receiver (either v2 via ssm-map or a direct v3 then hw gets programmed to forward)


Best-Practice • Configure SSM-Range on asr9k’s if SSM sources are directly connected

• Config sample if SSM group is 239.x.x.

RP/0/RSP0/CPU0:ASR9K-1#show run ipv4 access-list SSM-group ipv4 access-list SSM-groups 10 permit ipv4 239.0.0.0 0.255.255.255 any 20 permit ipv4 232.0.0.0 0.255.255.255 any RP/0/RSP0/CPU0:ASR9K-1#

RP/0/RSP0/CPU0:ASR9K-1#show run multicast-routing

multicast-routing address-family ipv4 ssm range SSM-groups interface all enable

RP/0/RSP0/CPU0:ASR9K-1#show run router pim router pim address-family ipv4 rp-address 110.0.0.24 Only if you need PIM-SM

Source Gig0/0/0/4.300

(51.2.1.2, 239.1.1.10) 0/1/0/1.1501 IGMPv2 join

RP/0/RSP0/CPU0:ASR9K-1#show mrib route 239.1.1.10 51.2.1.2 IP Multicast Routing Information Base (51.2.1.2,239.1.1.10) RPF nbr: 51.2.1.2 Flags: Up: 00:02:23 Incoming Interface List GigabitEthernet0/0/0/4.300 Flags: A NS, Up: 00:02:23 Outgoing Interface List GigabitEthernet0/1/0/9 Flags: F NS, Up: 00:02:23 <= V3 join GigabitEthernet0/1/0/1.1501 Flags: F NS, U V2 join (SSM map)

RP/0/RSP0/CPU0:ASR9K-1#show run router igmp Router igmp interface GigabitEthernet0/1/0/1.1501 static-group 239.1.1.10 51.2.1.2

Satellite


Satellite Basics

93

Host Fabric Port Satellite Fabric Port Satellite Access port

MAC DA MAC SA VLANs(opt) Payload/FCS MAC DA MAC SA VLANs(opt) Payload/FCS SAT-VLAN

Ingress: Satellite box adds a SAT-VLAN to the packet

and sends it to ASR9K host. SAT-VLAN id is 1:1

mapped to the incoming port on Satellite

Ingress: ASR9K uses the SAT-VLAN id and the

incoming uplink to determine the satellite port. Rest of

the packet processing is identical to packets received at

local ports

Egress: After all interface VLAN Ops are performed,

ASR9K adds the SAT-VLAN id of the outgoing port

Egress: Satellite box maps the incoming SAT-VLAN to

its front port, strips the SAT-VLAN and forwards the

packet.

Ingress

Egress

Satellite

ASR9k

(Host)


Satellite Bringup

• Discovery Protocol

– Operates at Layer 2.

– Provides the bootstrap mechanism by which a Satellite and Host begin communication.

– Enables Host to become aware of a reachable Satellite device, and exchange sufficient information to set up a full Control session.

– The Host device initiates discovery. Satellite devices are factory-shipped to listen for incoming Discovery probe packets.

• Control Protocol

– Makes use of the connectivity set up by the discovery protocol to provide a reliable and extensible mechanism for the Host device and each satellite device to exchange the required configuration, state, etc.

94


Discovery Protocol

95

• Discovery Protocol states

– Stopped – The link is down. Unshut the satellite Fabric link or the member links incase Bundle is used as ICL.

– Probing for satellites / Discovering – The Discovery protocol has sent probe packets to the satellite. It is waiting for a response from the satellite.

– Configuring satellite – The Discovery protocol has sent “configuring” packets to the satellite. It is waiting for a response from the satellite.

– Ready – Discovery protocol has completed. Control protocol bring-up can now start.

• “sh nv satellite status satellite <>” gives the state information

RP/0/RSP0/CPU0:nv-cluster-escalation#sh nv satellite status satellite 200

Wed Apr 23 22:27:20.927 UTC

Satellite 200

-------------

Status: Connected (New image transferred)


Satellite new topologies in XR51

96

Satellite

VLAN-B

VLAN-A Host A

Host B CFM

CFM

Satellite

Satellite

Satellite

Host A

Host B

Satellite Satellite Host

“fabric extension”

Cascading

Dual home ring

(rings to same host planned for

later)

ICCP


Discovery Protocol Troubleshooting Contd.

97

• Satellite Stuck in Discovering state and configured with 5.1.1 new topologies

– If the satellites are configured with 5.1.1 new topologies(Ring/Chain, L2 Fab, Dual head) and the satellite is not running with 5.1.1 compatible image, it is expected that satellite will not be discovered in the new topologies.

– Below is the mapping of Sat image and host image: • SVA --- 4.2.1

• SVB --- 4.2.3

• SVC --- 4.3.0

• SVD --- 4.3.1

• SVE --- 5.1.0

• SVF --- 4.3.2

• SVG --- 5.1.1

– Please use following steps to upgrade the image on satellite incase the image on satellite is not latest: (install nv satellite) • Connect the satellite in hub and spoke mode on the host.

• Transfer and activate the 9000v/901 image to the satellite.

• Check the output of show nv sat status to check the satellite displays “latest version”

• Move the satellites back to the new 511 topologies as required

RP/0/RSP0/CPU0:A9K-BNG#show nv sat stat

IPv4 address: 192.168.0.100 (VRF: default)

RP/0/RSP0/CPU0:A9K-BNG#telnet 192.168.0.100

Trying 192.168.0.100(192.168.0.100)...

Connected to 192.168.0.100.

LC:Satellite#show ver

Version 15.1(3)SVA, RELEASE SOFTWARE (fc1)



98

• Satellite Stuck in Discovering state(for all topologies)

– Satellite state shown as “Discovery halted; Conflict: no Identification received yet”

– “Status: Probing for Satellites; Conflict: no Identification received yet state” means that the Host is sending DPM probes to satellite and waiting for a response. The satellite is either not responding to the discovery probes from the host or the host drops the discovery replies from the satellite.



99

• If Host is not sending the discovery probes or if the host is dropping the discovery replies:

– Check “show int <ICL>” output on host, input/output counter would show SDAC discovery input/output packets.

– Check the NP drop counters of the ICL port.

– If IN_SATELLITE_DISCOVERY_DISCARD NP counter is increasing: • Check if icl bit is programmed in uidb for icl interface

show uidb data location 0/0/CPU0 tenGigE 0/0/1/0 ingress

Satellite IC interface 0x1 <<<

• If the ICL bit is not set in UIDB, we need to check the vlan ea db

sh ethernet infra internal ea trunks tenGigE 0/14/0/3 location 0/14/cPU0 --> main interfaces

sh ethernet infra internal ea subs tenGigE 0/14/0/3.1 location 0/14/cPU0 --> for icl which is a sub interface (incase of L2 Fab)

is_in_icl_mode: 1

• If the bit is not set in vlan ea then we need to check the vlan ma

sh ethernet infra internal ether-ma trunks tenGigE 0/14/0/3 location 0/14/cPU0 (use subs for sub interfaces)

is_in_icl_mode: 1



100

• If Host is sending the discovery probes and not receiving the discovery replies

– Check the ICL counter on the satellite(sh interface Tengige 1/45-48), if there are input counters but no output counters then the satellite is dropping the discovery packets. Check the BCM counters on the ICL link to check for drops: • login to bcm shell by the below command

LC:Satellite#test bcm shell ( for 9000v)

show c xe0 where xe0 -xe3 is tengige 1/45 to tengige 1/48

LC:Satellite#test platform bcm shell for (901)

Show c ge2 or ge3 ,where ge2 is gig 0/10 and ge3 is gi 0/11

BCM.0> show c xe0 RUC.xe0 : 87 +30 5/s RDBGC0.xe0 : 2 +1 -------------> RDBGC0 counter- this counter represents drops

RDBGC2.xe0 : 15 +4



101

• 511 L2 Fabric Satellite stuck in Discovery State:

– Only p2p l2 fab circuits are supported for l2 fab connections.

– If you have bridge domains/vpls/trunk ports in the l2fab cloud, convert them to p2p xconnects or VCs

– Trace the discovery probe packets from the host over the l2 cloud with each hop in the l2 cloud. Check the node where the packets are getting dropped and follow the usual triage process for packet drops.


Discovery Protocol Troubleshooting contd.

102

• Satellite stuck in configuring state for Dual Head/Ring Topology:

– Check if MPLS LDP is up between the 2 hosts

– Check if ICCP is up between the 2 hosts (show iccp group)

– Check if ORBIT is up between the 2 hosts (show nv satellite protocol redundancy)

– IF LDP or ICCP is not up, follow the usual triaging steps for ASR9K.

– If ICCP is up and ORBIT is down then “show tech satellite” will help to debug this issue further.

E-IC

CP

Satellite

Host A

Host B


Control Protocol Troubleshooting

103

• Control Protocol States

– Connecting: The Control protocol is trying to connect to the TCP Socket. “show tcp” on both the host and Satellite should give more info on this.

– Authenticating: The Control Protocol has sent an authentication message and is waiting for a response from the satellite.

– Checking ver: The control protocol has sent a version request to the satellite and is waiting for a response.

– Connected: The control protocol is up

• Satellite stuck in Connecting state:

• Check if ping works from the host to the satellite IP address.

– In case of Dual host/Ring topology, check if you have same ipv4 address or ipv4 unnumbered loopback address configured for the ICL on both hosts. The loopback address for the ICLs on both hosts should be different.

• Check if TCP session is UP. “show tcp brief” should show the state as “ESTAB”


Satellite image upgrade troubleshooting

104

• Image upgrade consists of following steps:

– Image transfer: The satellite binaries are copied from host to satellite. • “Install nv satellite <sat-id> transfer” can be used to initiate a transfer

• Image transfer generally takes 3-5 mins for 9000v and 4-6 mins for 901

• Following logs appear on the console once image transfer is done:

RP/0/RSP0/CPU0:dor1101eiuc202#install nv satellite 9002 transfer progress

Mon Dec 2 16:32:08.652 CET Install Op 6: transfer: 90021 configured satellite has been specified for transfer.

1 satellite has successfully initiated transfer.

Press Ctrl+C at any time to stop displaying the progress bar.

| Working...

RP/0/RSP0/CPU0:Dec 2 16:36:05.676 : icpe_satmgr[1164]: %PKT_INFRA-ICPE_GCO-6-TRANSFER_DONE : Image transfer completed on Satellite 9002 Completed.

– Image activation: The satellite is booted with the transferred binary. • For Host Upgrade from pre 511 image to 511 image, activation for 9000v is a 2 step process and would

cause the 9000v to reload twice(once with an intermediate image and second time with the 5.1.1 image)


Satellite image upgrade troubleshooting contd.

105

• image transfer to the 9000v or 901 satellite fails

– Check if the router has MPP configured. If yes, then add an exception for tftp under the ICL Links control-plane

management-plane

inband

interface TenGigE0/1/0/1

allow TFTP

– Check if the router has tftp homedir configured on the ASR9000 host. tftp vrf default ipv4 server homedir disk0:

If the tftp transfer requests from satellite comes on the default vrf [through manual IP configuration] and tftp home directory configured on the host is disk0: then image transfer request will fail as the tftp_fs will try to read the disk0:/ path.

Please remove the tftp homedir and retry the transfer.


Satellite image upgrade troubleshooting contd.

106

• Check whether the tftp service is configured or not using following cli.

– Show cinetd services

RP/0/RSP0/CPU0:R3#show cinetd services

Vrf Name Family Service Proto Port ACL max_cnt curr_cnt wait Program Client Option

default v4 telnet tcp 23 10 0 nowait telnet sysdb

default v4 tftp udp 69 unlimited 0 wait tftpd icpe-cpm /pkg/fpd-nv/ <<<<<<<<<<<<<<<< ICPE started the service

– Try “process restart “icpe_cpm” incase tftp service is not running for icpe-cpm

• ASR901 Specific image transfers issues/Checks

– If the ASR901 already have a standalone non nV image, image transfer may fail due to lack of space on the 901. Please delete the non-nV image from the 901 flash and execute "squeeze flash:" to free up space.

– For ASR901 the image transfer may take sufficiently more time sometimes than normal, if the flash does not have enough space, the squeeze will automatically kick in when image transfer is attempted, this operation may take close to 30 mins.


Ping/Packet drop troubleshooting with satellite

107

• Check the nv status(show nv satellite status). The satellite should be in connected state.

• Check the arp table of the destination IP Address. If ARP is not resolved:

– Check if cross-links are formed correctly on the satellite and interfaces are up on satellite. • LC:Satellite#sh satellite crosslink tenGigabitEthernet 1/45

Interconnect Link: TenGigabitEthernet1 /45 xos_if: 5 icl_id: 1 host_id: 1 cp_vlan: 0 Link State: Up SDAC State: Discovered Crosslink State: Up --------------------------------------------------------------- Access Port ICLID Link State ForcedDown ? TxDisabled ? --------------------------------------------------------------- Gi1/1 1 Up No No Gi1/2 1 Up No No Gi1/3 1 Up No No Gi1/4 1 Up No No

• If crosslink map is missed or not expected, please turn on “debug sdac control all feature-channel 3/ feature-channel 4(for bundle ICL)” for further debugging

• If interfaces are showing up on the satellite but down on the host, collect “show tech satellite” from host and “debug sdac control all feature-channel 1” from the satellite for further debugging


Ping/Packet drop troubleshooting with satellite contd.

108

• If ARP is resolved correctly then check for packet drops on host as well as satellite:

– Send ping packets from the host with packet size 1400. Check for ICL counters to confirm if the packet went out of the host.

– If ping packets are getting dropped on the host. Check the NP drop counter, which is increasing.

– Check ICL stats on Satellite to see if the ping packets are getting received on satellite

• If ARP is not resolved on the host check if the ARP is resolved on the other end connected to satellite. If ARP is resolved on the other end, send ping packets with size 1400 and trace the packet path same as above on the opposite direction.

• If ARP is not resolved on either end, then initiate ping from the host and host will send ARP requests, check the qos tm counters to see if packets go out of host.

• Configure static arp on the host and initiate ping, now the host will send normal ping packets(not arp packets), check the counters on the ICL link and the satellite as above.

Troubleshooting QoS


ASR9k QoS Architecture

110

Ingress (sub-)interface QoS Queues

Virtual Output Queues

Egress FIA Queues

End-to-End priority propagation Guarantee bandwidth, low latency for high priority traffic at any congestion point

Configure with Ingress MQC 4-layer hierarchy Two strict high priority + Normal priority

Egress (sub-)interface QoS Queues

Configure with Egress MQC 4-layer hierarchy

Two strict high priority + Normal priority

Implicit Configuration Two strict high priority +

Normal priority

Ingress side of LC Egress side of LC

NP0 PHY

NP2 PHY

NP3 PHY

NP1 PHY FIA

CPU NP0 PHY

NP2 PHY

NP3 PHY

NP1 PHY

FIA

CPU

Switch Fabric 1

2 3

4

1 2 3 4

One Queue set per each NP on the LC


ASR9k QoS Architecture – Fabric Bandwidth Access Overview

111

FIA FIA

RSP1

Crossbar

Fabric

ASIC

Crossbar

Fabric

ASIC

Crossbar

Fabric

ASIC

Crossbar

Fabric

ASIC

RSP0

1: Fabric Request

3: Fabric Grant

2: Arbitration

4: load-balanced transmission across fabric links

5: credit return

Ingress LC Egress LC

Arbiter

Arbiter

3 strict priority scheduling/queueing

Back pressure and virtual output queue

Multicast and Unicast separation (separated

queues and fabric plane)


Arbitration & Fabric QoS

• Arbitration is being performed by a central high speed arbitration ASIC on the RSP

• At any time a single arbiter is responsible for arbitration (active/active “APS like” protection)

• The Arbitration algorithm is QoS aware and will ensure that P1 classes have preference over P2 classes, both of which have preference over non-priority classes

• Fabric QoS is applied at Virtual Output Queue (VoQ) level

– Arbitration is performed relative to a given the egress VoQ

112


Virtual Queue Identifier (VQI) and Virtual Output Queue (VOQ) • On Trident LCs, VQI is assigned per NP basis.

– NP maps to a single 10GE port or a group of ten 1GE ports

• On Typhoon LCs, NP is designed for multiple 10G, 40G, and 100G ports.

– Each 10G port is 1:1 mapped to one VQI

– Each 40G port is mapped to 8 VQI

– Each 100G port is mapped to 16 VQI

– VOQ’s used to load balance across internal connections

• One VOQ comprises 4 VQI (Typhoon) or 3 VQI (Trident)

113

voq

vqi

vqi

…


ASR9k QoS Architecture – Backpressure and VoQ Mechanism

114

VOQ congestion on egress NP

backpressure propagated to ingress FIA

Packet is en-queued in the dedicated VOQ

No impact of the packet going to different egress VOQ

No head-of-line-block issue

Ingress side of LC1 Egress side of LC2

NP0 PHY

NP2 PHY

NP3 PHY

NP1 PHY FIA

CPU NP0 PHY

NP2 PHY

NP3 PHY

NP1 PHY

FIA

CPU 1

3

2

Backpressure: egress NP egress FIA fabric Arbiter ingress FIA VoQ

Switch Fabric

One VoQ set (4 queues) per each NP in the system

5Gbps

10Gbps

5Gbps


Default Interface Queues

• For every physical port in the system, the following queues get created:

115

Typhoon Trident

Ingress (NP->Fabric) Egress (NP->Line) Ingress (NP->Fabric) Egress (NP->Line)

High priority 1 queue

(routing control protocols)

High priority 1 queue (routing

control protocols and critical

traffic like BFD)

High priority queue (routing

control protocols)

High priority queue (routing

control protocols and critical

traffic like BFD)

Medium priority 2 queue

(Unused without policy-map)



Medium priority queue


Medium priority queue (Unused

without policy-map)





Low priority queue (Used by

all other traffic)

Low priority queue (Used by all

other traffic)

Low priority queue (Used by

all other traffic)

Low priority queue (Used by all

other traffic)

Default queue size is 100ms of the line rate


Default Interface Queues

RP/0/RSP0/CPU0:A9K#show qoshal default-queue subslot 1 port 8 location 0/0/CPU0 Thu Apr 3 06:00:08.931 UTC

Interface Default Queues : Subslot 1, Port 8

===============================================================

Port 72 NP 1 TM Port 20

Ingress: QID 0x20200 Entity: 1/0/2/4/64/0 Priority: Priority 1 Qdepth: 0 StatIDs: commit/fast_commit/drop: 0x690a00/0x678/0x690a01

Statistics(Pkts/Bytes):

Tx_To_TM 0/0 Fast TX: 58648/12629943

Total Xmt 58648/12629943 Dropped 0/0

Ingress: QID 0x20201 Entity: 1/0/2/4/64/1 Priority: Priority 2 Qdepth: 0

<...>

<...>

Egress: QID 0x20220 Entity: 1/0/2/4/68/0 Priority: Priority 1 Qdepth: 0 StatIDs: commit/fast_commit/drop: 0x690aa0/0x67b/0x690aa1

Statistics(Pkts/Bytes):

Tx_To_TM 0/0 Fast TX: 58702/18300724

Total Xmt 58702/18300724 Dropped 0/0

Egress: QID 0x20221 Entity: 1/0/2/4/68/1 Priority: Priority 2 Qdepth: 0

<...>

<...>

current number of packets

in the queue

TX statistics

interface Gig0/0/1/8

116


MQC to System QOS mapping • ASR 9000 supports traffic differentiation at all relevant points within the system

– P1/P2/LP differentiation or P1/LP differentiation support throughout the system

• Classification into these priorities is based on input MQC classification on the ingress linecard into P1, P2, Other

– Once a packet is classified into a P1 class on ingress it will get mapped to PQ1 queue along the system qos path

– Once a packet is classified into a P2 class on ingress it will get mapped to PQ2 queue along the system qos path, unless no MP is implemented. In this case HP would be used for P2.

– Once a packet is classified into a non-PQ1/2 class on ingress it will get mapped to LP queue along the system qos path

• Note: The marking is implicit once you assign a packet into a given queue on ingress; its sets the fabric header priority bits onto the packet.

– no specific “set” action is required

117


Feature order on ASR 9000 NP (simplified)

118

I/F classification

ACL classification Fwd lookup

QOS classification

IFIB lookup IFIB action QoS action ACL action L2 rewrite

QoS action

ACL action QOS

classification L2 rewrite ACL

classification Fwd lookup

From wire

To wire

Ingress linecard

egress linecard

To fabric From fabric

TCAM


QOS classification

Feature order on ASR 9000 NP: QoS Action Order

• WRED classifies on marked/remarked values (doesn’t switch class-maps!)

119

I/F classification

ACL classification Fwd lookup

IFIB lookup IFIB action ACL action L2 rewrite

ACL action QOS

classification L2 rewrite ACL

classification Fwd lookup

From wire

To wire

Ingress linecard

egress linecard

To fabric From fabric

Police Mark Queue/shape/WRED

QoS action

QoS action

QoS Action


Injected packets

• In general are injected “to-wire” (same as Pak Priority in IOS)

• Means that all features are bypassed.

• Including QOS

• Few exceptions

– ICMP

– Netflow

120


ASR 9000 QOS Implicit Trust

• For Bridged packets on ingress – outermost COS would be treated as trusted.

• For Routed packets on ingress – DSCP/Precedence/outermost EXP would be treated as trusted based on packet type.

• Default QOS will be gleaned from ingress interface before QOS marking is applied on the ingress policymap.

• By default ASR 9000 would never modify DSCP/IP precedence of a packet without a policy-map configured.

• Default QOS information would be used for impositioned fields only

121


Typhoon QoS Overview • Super-set of existing Trident linecard QoS functionality

– Dedicated TM for queuing

– Fabric/internal QoS mechanism

– Flexible 4-level H-qos ingress and egress

• Higher scale

– Higher queue and policer scale

– More granular bandwidth control for both policing and queuing

– Higher buffer size

• Additional new feature capability

– Conform-aware policer (a/k/a Coupled Policer)

– 4 strict priority: P1, P2, P3 (egress only) and normal priority

• P3 is egress only;

• Ingress TM for <=30G configs only

– No input shaping on high-NP loading configs (36x10G, 8x10 MPA, 40G MPA)

• By default all queue-limits are set to 100ms of “service rate” 122


Increased Priority Queues

• Trident –Max of 8 Child Queues per parent , with 1 Priority 1, 1 Priority 2, and 6 Normal-priority queues (including class-default)

• Typhoon – Max 8 Child Queues per Parent – Choices based on user config in policy.

– 1 Priority 1, 2 Priority 2 and 5 Normal-priority

– 1 Priority 1, 1 Priority 2, 1 Priority 3, 5 Normal-Priority (Egress only)

– 1 Priority 1, 1 Priority 2, and 6 Normal-priority

123


Typhoon QoS Overview • Super-set of existing Trident linecard QoS functionality

– Dedicated TM for queuing

– Fabric/internal QoS mechanism

– Flexible 4-level H-qos ingress and egress

• Higher scale – Higher queue and policer scale

– More granular bandwidth control for both policing and queuing

– Higher buffer size

• Additional new feature capability – Conform-aware policer (a/k/a Coupled Policer)

– 4 strict priority: P1, P2, P3 (egress only) and normal priority

• P3 is egress only; need special consideration in case of parent shaper

• Ingress TM for <=30G configs only – No input shaping on high-NP loading configs (36x10G, 8x10 MPA, 40G MPA)

• By default all queue-limits are set to 100ms of “service rate”

124


Typhoon vs Trident Priority Queues

• Trident –Max of 8 Child Queues per parent , with 1 Priority 1, 1 Priority 2, and 6 Normal-priority queues (including class-default)

• Typhoon – Max 8 Child Queues per Parent – Choices based on user config in policy.

– 1 Priority 1, 2 Priority 2 and 5 Normal-priority

– 1 Priority 1, 1 Priority 2, 1 Priority 3, 5 Normal-Priority (Egress only)

– 1 Priority 1, 1 Priority 2, and 6 Normal-priority

125


ASR9K QoS Classification Criteria • Very flexible L2/L3 field classification on L2 interfaces

• Inner/outer cos

• Inner/Outer vlan *

• DEI*

• Outer EXP

• Dscp/Tos

• TTL, TCP flags, source/destination L4 ports

• Protocol

• Source/Destination IPv4

• Source/Destination MAC address*

• Discard-class

• Qos-group

• match all/match any

• Note:

– Not all fields are supported on L3 interfaces*

– Some fields don’t make sense on ingress (e.g. discard-class, qos-group)

– MPLS classification is based on EXP only (note in 530 we will be able to apply QOS matching (tentative) and

ACL matching on MPLS labeled packets)

126


ASR9K QoS - Classification Formats • Per Policy-map a given classification format is chosen by SW, i.e a given policy-map can

only classify based on a single format

127

Format 0 Format 1 Format 2 Format 3

Fields

supported

• IPV4 source address

(Specific/Range)

• IPV4 Destination address

(Specific/Range)

• IPV4 protocol

• IP DSCP / TOS / Precedence

• IPV4 TTL

• IPV4 Source port

(Specific/Range)

• IPV4 Destination port

(Specific/Range)

• TCP Flags

• QOS-group (output policy only)

• Discard-class (output-policy

only)

• Outer

VLAN/COS/DEI

• Inner VLAN/COS

• IPV4 Source

address

(Specific/Range)

• IP DSCP / TOS /

Precedence

• QOS-group (output

policy only)

• Discard-class

(output policy only)

• Outer

VLAN/COS/DEI

• Inner VLAN/COS

• IPV4 Destination

address

(Specific/Range)

• IP DSCP / TOS /

Precedence


policy only)

• Discard-class


• Outer

VLAN/COS/DEI

• Inner VLAN/COS

• MAC Destination

address

• MAC source

address


policy only)

• Discard-class



ASR9K QoS - Packet marking details

• “settable” packet fields: • dscp/precedence

• EXP imposition

• EXP topmost

• cos inner/outer

• qos-group

• discard-class

• ASR9K supports maximum of 2 fields per class-map. The same 2 fields can be placed in any combination below

• - 2 sets per police-conform/exceed/violate

• - 2 sets without policing.

• Note: In MPLS context only EXP marking is supported

• Remember that mpls encapped packets can’t match on L3 criteria (same for ACL) – (note in 530 we will be able to apply QOS matching (tentative) and security ACL on MPLS labeled packets)

128


ASR9K QoS - Policing details

• RFC 2698 supported (2r3c) and 1r2c

• Ingress & egress policing supported

• General Rule: Policing required on priority queues.

– Priority level 2 classes can also accept shaping instead of policing.

• Granularity of 8Kbps supported (typhoon, 64k on trident)

• 2-level nested policy maps supported

– Note: policers at parent and child work independently

• 64k policers per NP (shared for ingress/egress) on extended linecards

• Policer actions supported:

– transmit

– drop

– set (implicitly behaves like set and transmit)

– each color can have two set actions:

129

Policy-map parent

Class class-default

Police rate 10 Mbps peak-rate 20 mbps

conform-action set dscp af12

conform-action set cos 2

exceed-action set dscp af13

exceed-action set cos 3


Conform Aware Policer

• Normal Hierarchical Policer

• At parent level, if it’s over the CIR, packet will be dropped randomly. There is no awareness which packet to be dropped

• Conform Aware Policer

• Parent CIR > aggregated child CIR

• Parent police only support 1R2C, child police support all: 1R2C, 2R3C, or 1R3C

• If drops happen at parent level, it will drop child out-of-profile packet, but guarantee the child in-profile packet

130

policy-map child

class class1

police rate 20 mbps peak-rate 50 mbps

class class2

police rate 30 mbps peak-rate 60 mbps

policy-map parent

class class-default

service-policy child

police rate 60 mbps

child-conform-aware


Common Policer problems

• Note that all L2 headers are included, added to the payload and that packet size is depleting the token bucket (applies to shaping also). Only IFG is not accounted for (crc is)

• Incorrect burst size configuration, allow for some excess burst to “catch up”.

• Mistake between 2 or 3 rate policers (exceed action drop)

• Trident’s policer can’t go negative, Typhoon can borrow

– This means that policer behavior is slightly different between the 2 hardware

131


ASR 9000 QoS - Queue scheduling

• Use “shape” for a shaped PIR for a graceful enforcement of a maximum bandwidth

– shaping at all configurable levels

– Min. granularity: 64kbps (L3, L4, 256kbps for L2)

• priority levels: priority level 1/2/3, minBw/CIR and Bw remaining

• Use “bandwidth” (minBw) for a CIR guarantee relative to the parent hierarchy level • Min. RATE: 64kbps (8k granularity)

• Use “bandwidth remaining ratio/percent” for the redistribution of excess bandwidth that is available after PQ classes have been scheduled

– configurable ratio values 1-1020

• Two parameter scheduler support at class level and subscriber group level (L4, L2):

– Shape & BwR (ratio / percent)

– Shape & MinBw (absolute / percent)

– Not supported: BwR & MinBw on the same class

132


ASR 9000 QoS - congestion management/buffering details • WRED based on: DSCP, IPP, EXP, COS, discard-class

• default queue-limit -to prevent buffer exhaustion- is 100ms of service rate (service rate is the sum of guaranteed bw/bwr assigned to a class)

• WRED configuration unit options are: bytes, kbytes, mbytes, us, ms, packets

– These values will be rounded up to a set of pre-defined profiles ranging from 8 kB to 262144 kB

– The actual implementation uses 512 byte buffer particles

• Novelty: ASR 9000 supports WRED on shaped PQ2 classes.

– Can be used for differentiation of two kinds of priority within the PQ2 class

133


Absolute vs Percentage

• All relevant policy actions support both, absolute and percentage based configuration:

– shape

– bandwidth

– Police

– bandwidth remaining*

• For tri-rate Copper SFPs (10/100/1000) percentage based QOS will be adjusted automatically based on the selected rate

134

*Note: Bandwidth remaining supports ratio/percent, not absolute bandwidth


Show/debug QOS commands show running-config

show running-config policy-map <policyname> Policy map configuration

show running-config class-map <classmap> Class map configuration

show running-config interface <interface> Interface running configuration

show policy-map interface <interface> [input | output] Policy-map statistics on a particular non-bundle interface

show policy-map interface <bundle-interface> [input|output]

member Policy-map statistics on a member of bundle interface

show qos interface <interface> <input|output> [member

<interface>]

Displays hardware and software configured values of each

class for a service-policy on an interface

show qos-ea interface <interface> <input|ouput> [member

<interface>] [detail]

Displays the detailed information of hardware and software

configured paramters in each class of a service-policy on an

interface

show qos summary <police|policy|queue> [interface <interface>]

[output|iNPt] [member <interface>] Lists the summary of all queues or policers or interfaces for a policy

show qoshal tm-config

<all|counters|fcu|general|priority|shape|topology|wfq|wred> np

<np> tm <tm>

Displays generic NP TM config

show qoshal <wfq|wred|wred-scale|shape|police|police-node> np

<np> tm <tm> level <level> profile <profile> <num-of-profiles>

[hw|sw]

Displays various profiles configured in sw and hw and the values of

each profile 135


Show/debug QOS commands - contd

show qoshal default-queue subslot <n> port <m> location

<location> Displays the default-queue information

show qoshal resource summary [np <np>] Displays the summary of all the resources used in hardware and

software for QoS such number of policy instances, queues, profiles

show qoshal fcu <limits|status|profile> Displays all Traffic Manager (TM) Flow control related info

show qoshal ha chkpt <all|<chkpt-tbl-name> {all|<recid>|info} Display HA related info for PRM QoS HAL

show qos-ea ha state Displays the HA State of process QoS EA whether it can accept the

service-policies

show qos-ea ha chkpt <all|<chkpt-tbl-name> {all|<recid>|info} Display HA Chkpt related info for all the chkpt tables for QoS EA

show qos-ea trace {all|errors|events|internal} Displays the trace of errors or events or internal events of QoS EA

process

show prm server trace hal Displays all the trace info of PRM QoS HAL thread

debug qos-ea all Debug commands for qos ea process

debug qoshal <level|module|events> <word> Debug commands for PRM qos HAL

debug prm server hal <all|error|events> Debug commands for PRM qos HAL API

136


policy-map p-map

class class-default

service-policy dummy

shape average 500 mbps

!

end-policy-map

!

policy-map dummy

class class-default

bandwidth percent 100

!

end-policy-map

!

Troubleshooting: What is programmed in HW?

RP/0/RSP0/CPU0:WEST-PE-ASR9K-2#sh qos interface g0/0/0/0 output

Fri Jan 10 15:05:40.347 EST

Interface: GigabitEthernet0_0_0_0 output

Bandwidth configured: 500000 kbps Bandwidth programed: 500000 kbps

ANCP user configured: 0 kbps ANCP programed in HW: 0 kbps

Port Shaper programed in HW: 500000 kbps

Policy: p-map Total number of classes: 2

----------------------------------------------------------------------

Level: 0 Policy: p-map Class: class-default

QueueID: N/A

Shape Profile: 1 CIR: 64 kbps CBS: 10240 bytes PIR: 499968 kbps PBS: 6291456 bytes

WFQ Profile: 4 Committed Weight: 1 Excess Weight: 1

Bandwidth: 0 kbps, BW sum for Level 0: 0 kbps, Excess Ratio: 1

----------------------------------------------------------------------

Level: 1 Policy: dummy Class: class-default

Parent Policy: p-map Class: class-default

QueueID: 642 (Priority Normal)

Queue Limit: 8388 kbytes Profile: 2 Scale Profile: 2



---------------------------------------------------------------------- 137


policy-map p-map

class class-default



!

end-policy-map

!

policy-map dummy

class class-default


!

end-policy-map

!

RP/0/RSP0/CPU0:WEST-PE-ASR9K-2#sh qos interface g0/0/0/0 output

Fri Jan 10 15:05:40.347 EST

Interface: GigabitEthernet0_0_0_0 output

Bandwidth configured: 500000 kbps Bandwidth programed: 500000 kbps

ANCP user configured: 0 kbps ANCP programed in HW: 0 kbps

Port Shaper programed in HW: 500000 kbps

Policy: p-map Total number of classes: 2

----------------------------------------------------------------------

Level: 0 Policy: p-map Class: class-default

QueueID: N/A

Shape Profile: 1 CIR: 64 kbps CBS: 10240 bytes PIR: 499968 kbps PBS: 6291456 bytes



----------------------------------------------------------------------

Level: 1 Policy: dummy Class: class-default

Parent Policy: p-map Class: class-default

QueueID: 642 (Priority Normal)

Queue Limit: 8388 kbytes Profile: 2 Scale Profile: 2



---------------------------------------------------------------------- 138

Rate is rounded to the nearest 8k or 64k value

Shape sets PIR

PBS is default rate of 100msec of configured shape rate

Parent BW is zero or 64k, only applicable in oversubscription at sum of parent levels

Troubleshooting: What is programmed in HW?


Troubleshooting: Policy-Map Statistics

139

RP/0/RSP0/CPU0:A9K#sh policy-map interface g0/0/1/8 output Thu Apr 3 06:47:56.728 UTC GigabitEthernet0/0/1/8 output: p-map-1 Class class-default Classification statistics (packets/bytes) (rate - kbps) Matched : 2/116 0 Transmitted : 4/232 0 Total Dropped : 0/0 0 Policy dummy Class class-default Classification statistics (packets/bytes) (rate - kbps) Matched : 2/116 0 Transmitted : 4/232 0 Total Dropped : 0/0 0 Queueing statistics Queue ID : 131626 High watermark : N/A Inst-queue-len (packets) : 0 Avg-queue-len : N/A Taildropped(packets/bytes) : 0/0 Queue(conform) : 4/232 0 Queue(exceed) : 0/0 0 RED random drops(packets/bytes) : 0/0

policy-map p-map

class class-default



!

end-policy-map

!

policy-map dummy

class class-default


!

end-policy-map

!


Troubleshooting Back-pressure Issues

140

• Look for FIA drops RP/0/RSP1/CPU0:ios#show drops Tue Jan 14 20:44:25.360 EST Node: 0/0/CPU0: <…> FIA 0 Drops: ---------------------------------------------------------------- Ingress Drops 287078960 Egress Drops 1 Total Drops 287078961 Ingress Generic Hard Drop-2 287078960 Egress Mcast RxFab Hdr-1 1 ----------------------------------------------------------------

• Check if any VQI is dropping packet RP/0/RSP1/CPU0:ios#show controller fabric fia q-depth location 0/0/CPU0 FIA 0 VoQ | ddr | pri | pkt_cnt ------+-----+-----+--------- 23 | 0 | 2 | 118

Total Pkt queue depth count = 118 Packets in the queue. Not good.


Troubleshooting Back-pressure Issues

141

• Look for FIA drops RP/0/RSP1/CPU0:ios#show controllers pm loc 0/5/CPU0 | I “^Ifname|switch_fabric_port”

Ifname(1): TenGigE0_5_0_0, ifh: 0xe000100 :

switch_fabric_port 0x17 VQI 23 is for interface ten0/5/0/0

• Look for egress NP TM Drops:

RP/0/RSP1/CPU0:ios#show controllers NP tm counters all location 0/5/CPU0

Node: 0/5/CPU0:

==== TM Counters (NP 3 TM 1) ====

TM Counters: commit_xmt_paks: 1509333316

excess_xmt_paks: 67641555690

Total Transmitted paks: 69150889006

wred_drop paks: 2441836834 timeout_drop 0 intf_drop 0

==== TM Counters (NP 3 TM 2) ====

TM Counters: commit_xmt_paks: 0

excess_xmt_paks: 0

Total Transmitted paks: 0

wred_drop paks: 0 timeout_drop 0 intf_drop 0


Shaping with PIR/PBS and CIR

142

• Shaper peaks to linerate for pbs time

• Should allow some burst to get to PIR faster

• CIR is ignored, will result in queue(exceed) counts, but they don’t mean drops!

linerate

PIR

CIR

PBS

RP/0/RSP0/CPU0:A9K-BNG#show policy-map int g 0/0/0/0 | i Queue

Queueing statistics

Queue ID : 136

Queue(conform) : 0/0 0

Queue(exceed) : 0/0 0


QOS summary

• All Ethernet linecards support Queuing, Marking and Policing.

• Some high speed linecards do not support ingress Queuing (but support policing and marking).

– Because their ingress TM (Traffic Manager) is disabled

• To guarantee priority end to end, make sure high priority traffic is marked on ingress (This will not burn a queue)

• https://supportforums.cisco.com/docs/DOC-15592

143

https://supportforums.cisco.com/docs/DOC-15592



Usability, XR strategy and SMU’s


Feedback on XR and actions

• SW quality improvements

• Install and manageability improvements in XR4.3

– Better install handling

– Better smu handling

– Disk mirror separation

– Auto FPD reducing number of reloads between upgrades

• SMU reduction effort and SMU management

– Better quality

– Better identification of what should be smu’d

– Cisco Software Manager (in PRIME now too). V2 just released

– Better DDTS info

• SW quality with EMR (extended Maintenance releases)

– XR 4.3.4

– XR 5.1.3

The XR drag coefficient, too many SMU’s, no maintenance releases

145


Smart Reload Minimizing Upgrade Time

34 Minutes

19 Minutes

21 Minutes

11 Minutes

4.2.1

4.3.0

4.3.1

Benefits

Faster image upgrades

Faster reload SMU installation and activation

Significantly reduced traffic loss due to

unplanned events (e.g. power outages)

Reduces maintenance window duration

Supports any-to-any image upgrade

Improved TFTP image download time

Improved traffic convergence time by

optimizing prefix download and programming

time.

Early availability of IOS-XR prompt, Line

Card services and resources

5.1.1

146


XR Software Manager (CSM) Highlights

Automatic analysis and optimization of router software

Periodically consult cisco.com for SMU info and updates

Secure, scalable, platforms: Win, Mac, VM

GUI; written in Java; runs as standalone tool

FREE and Public to ALL IOS XR customers

Machine to Machine

147


XR Software Manager (CSM) Benefits

Time saved: No more searching/querying Cisco.com for SMUs

Cost savings by avoiding incorrect and missing SMUs

New SMU Alerts: Similar to iPhone software upgrades

Analyze and recommend your device specific SMUs

Machine to Machine

148


XR Software Manager Next Enhancements

SMU ETA visibility

Auto SW download(BSD)

PDF format conformance report

SMU Size Calculation

Service Pack Support

Machine to Machine

149


Introducing ISSU SMUs! Patching got even better!

Diffe

ren

tiatio

n

• Industry unique ability to patch without a full

upgrade

• Non Reload SMUs – patching on the fly

• Reload SMUs – Need a router reload

• Previously 50 – 50 split between Non reload

and reload SMUs

SMU – Software Maintenance Unit

• Use ISSU infra to convert reload SMUs to

ISSU SMUs

• Non Reload :50%; ISSU SMU: 30%; Reload

SMU: 20%

ISSU SMU

Up to 60% of

reload SMUs can

be converted to

ISSU SMUs

Reduce packet

loss to <60 sec**

from 20-30

**Packet loss time will vary with LC and scale on LC


Reload SMU reduction (4.3.x)

• How Do we do this ?.

– “NP fast reset feature” to avoid LC reload for ucode code upgrade.

– Every effort is made to build ISSU SMU instead of reload SMU.

– Reload SMU’s will follow stringent rules before the decision is made.

• How do we Measure ?

– Current 4.3.2 reload SMU’s stands at 0%

– Goal: Reload SMU’s cannot be more then 20%

151


XR Service Packs Minimizing System Reloads

Service Packs on XR provide packaging of SMUs, and reduce the number of reboots for software updates between releases Benefits

• Lowers OpEx & Reduce TCO • Simplify Operations by tracking less SMUs • Reduce downtime & cost of system

upgrades

Highlights • Bundle multiple SMUs into a single package • Combine multiple reload SMUs to reduce

reboots • Critical SMUs are still available external to

Service Packs

152


IPV6-Etherconsole: • Customers don’t want to use serial (RS232) console. ✕

• RS232 to Ethernet conversion using External Dongle (e.g. Raspberry Pi) is not a viable solution. ✕

• Avoid the need to use the serial (RS232) console by ensuring that the router will always be accessible over IPV6 via Management Ethernet interfaces.

• Hardware Solution:

– Dedicated physical port in next-Gen RSP front Panel

– Goal is to achieve true emulation of console port

• Software Solution:

– Use front Panel Management Ethernet port for console emulation

– Requires ROMMON variable IPV6-ETHERCONSOLE

– All tasks that can be done in Rommon CLI can be done in XR CLI

– Not true Console emulation. • Router recovery, Password recovery, Rommon access, TFTP boot cannot be done with ether-console and still

requires console connection

153


FlexCli: config apply-groups

router isis green interface GigabitEthernet0/0/0/0 lsp-interval 20 hello-interval 40 address-family ipv4 unicast metric 10 ! interface GigabitEthernet0/0/0/1 lsp-interval 20 hello-interval 40 address-family ipv4 unicast metric 10 ! interface GigabitEthernet0/0/0/2 lsp-interval 20 hello-interval 40 address-family ipv4 unicast metric 10 ! interface GigabitEthernet0/0/0/3 lsp-interval 20 hello-interval 40 address-family ipv4 unicast metric 10 !

group G-ISIS-INTERFACE router isis “.*” interface “Gig.*” lsp-interval 20 hello-interval 40 address-family ipv4 unicast metric 10 ! config router isis green apply-group G-ISIS-INTERFACE

Apply-group command matches “router isis green”

config in running-config and applies configs

to specific interfaces

154


Accelerated Upgrade

• Python based Auto upgrade tool

• Architecture support Plugins to expand and customize to any platform

• Ordered list of plugins

– Pre upgrade plugins

– Upgrade plugins

– Post upgrade plugins

• Can set order of dependency between plugins

• On an upgrade failure box will be auto reverted & restored

155

Availability: NOW


Accelerated Upgrade cont..

Availability: NOW

Where & how can I use AU? • Operators that perform 10 upgrades a night can

now do 15 or 20

• Chances of upgrade failures due to human or

system error reduced

• Great tool for customers & labs

Feature Highlights • Supported on all XR platforms

• Can perform install upgrade or turboboot

• It’s open sourced and official, anywan can modify

script as needed

• Customer can share AU with Cisco or community

156


Where do I learn more? • Great video by Kashish Golani

https://www.youtube.com/watch?v=2DgXPi0Ink4

• Open source AU is now on SourceForge

http://sourceforge.net/projects/acceleratedupgrade/

• TechZone article

Questions? <[email protected]>

Roadmap highlights

• Integrate AU with CSM

• Add support for NG XR

• Simple script to support reading config file

• Support setting ROMMON variable from file for

turboboot Availability: NOW

157






Accelerated Upgrade DEMO

Pre-Upgrade Check

Upgrade

Post-Upgrade Check

Plugins

Plugins

Plugins

Syntax ./accelerated_upgrade -l cisco -p cisco -r

ftp://terastream:[email protected]://h

ome/terastream/echami -f pkg.txt -d

172.28.98.3

Eddie Chami in Dubai

Router & FTP in SJ

172.28.98.3 = ASR9001

172.20.168.195 = FTP Server

pkg.txt package list added

(mgbl, mpls being added)

158


Easy Upgrades

Avoid User Errors

Fewer Commands

Simplified Upgrade On RSP2

Flex Upgrades on RSP2 system for ease of upgrades and downgrades Enables install manager to use Disk0 and Disk1 Goal! Simplifying upgrades Benefits

• Reduce Maintenance window time • Reduce reloads during upgrades • Provides 3.8G of usable space to install process • Easy of backing out of an upgrade

Feature Highlights

• Run 5.1.1 from Disk0 • Run 5.1.3 from Disk1 • Run 5.2.0 from Disk0

Released: 5.1.1

See 5.1.1 install documentation for more info


What’s next?

• Continued focus on Support forums

– Read the blog for useful announcements

– Q&A

– Tech docs

• Increased attention to usability

– TCP transfer improvements (so necessary )

– Eg Bundle-Ether vs BE

– RPL functionality (such as test facility, variables)

– BGP show enhancements, tweaks and knobs

– TE show and debug improvements

– Long standings asks from IOS such as • Logging synchronous (auto ctrl-R after syslog message)

• interface-range

160


Participate in the “My Favorite Speaker” Contest

• Promote your favorite speaker through Twitter and you could win $200 of Cisco Press products (@CiscoPress)

• Send a tweet and include

– Your favorite speaker’s Twitter handle

– Two hashtags: #CLUS #MyFavoriteSpeaker

• You can submit an entry for more than one of your “favorite” speakers

• Don’t forget to follow @CiscoLive and @CiscoPress

• View the official rules at http://bit.ly/CLUSwin

Promote Your Favorite Speaker and You Could be a Winner

162

http://bit.ly/CLUSwin


Complete Your Online Session Evaluation

• Give us your feedback and you could win fabulous prizes. Winners announced daily.

• Complete your session evaluation through the Cisco Live mobile app or visit one of the interactive kiosks located throughout the convention center.

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online

163

https://www.ciscolive.com/online


Continue Your Education

• Demos in the Cisco Campus

• Walk-in Self-Paced Labs

• Table Topics

• Meet the Engineer 1:1 meetings

164

ASR9000 Troubleshooting architecture

Documents

Transcript of ASR9000 Troubleshooting architecture