Sample lld document v1.0

CORE INFRASTRUCTURE MIGRATION

LOW LEVEL DESIGN

v 0.1

RE-Solution Data Ltd

170 Greenford Road

Harrow

Middlesex

HA1 3QX

T +44 (0) 8450 031323

Reach | Recruit | Resolve | Refine

CLIENT LOGO HERE

Copyright © 2015 Re-Solution. All rights reserved. of 32

DOCUMENT CONTROL Purpose of Document

The purpose of this document is to provide A CUSTOMER with a low level design document proposing the design for the technology refresh taking place at their Headquarters based in Gloucestershire. This document will cover the core network and the DC environment only specifically the Nexus environment, all other components of the network infrastructure are out of scope. This low level design will be used for a definitive reference and a base as to how the fixed network will be developed and implemented. The information in this document is conveyed with the assumption that engineers possess CCNP or equivalent knowledge of network protocols and design fundamentals. It is not intended to explain every fine detail and every configuration line for both Catalyst and Nexus devices that will be used in the design, as it’s expected that engineers understand these devices and their configurations to a decent level. Record of Modifications

Version Date Revision Name

0.1 14/02/2015 First Draft Sean Draper

0.2 Initial Peer Review

Distribution List

Name Company Contact Details Role / Responsibility

Darren Smith Resolution [email protected] Sales Account Manager

Tom Giembicki Resolution [email protected] CTO

Contributors

Name Title Responsibility

Tom Giembicki CTO Reviewer

Referenced Documents

Document Title Description

Scope Of works Document Scope Of works

Sign Off

Name Title Signature

Darren Smith Sales Director


TABLE OF CONTENTS

DOCUMENT CONTROL ..........................................................................................................................................2

TABLE OF CONTENTS ............................................................................................................................................ 3

TABLE OF FIGURES ............................................................................................................................................... 4

TABLE OF TABLES ................................................................................................................................................ 4

RESTRICTION ON DISCLOSURE ............................................................................................................................ 5

INTRODUCTION .................................................................................................................................................... 6

1 SCOPE ............................................................................................................................................................. 7

1.1 Project Scope .......................................................................................................................................... 7 1.2 Design Scope ........................................................................................................................................... 7 1.3 Document Structure ............................................................................................................................... 7

2 NETWORK DESIGN ....................................................................................................................................... 8

2.1 Overview ................................................................................................................................................. 8 2.2 Core and Server Aggregation ................................................................................................................ 8 2.3 Server Access Layer (FlexPod) .............................................................................................................. 8 2.4 Device Abbreviations ............................................................................................................................. 9 2.5 Example Configurations ........................................................................................................................ 9 2.6 Overall Physical Diagram ...................................................................................................................... 10

3 CISCO 4500 HARDWARE .............................................................................................................................. 11

3.1 Overview ................................................................................................................................................. 11 3.2 Catalyst 4500- Physical Attributes ......................................................................................................... 11

3.2.1 Hardware Components .................................................................................................................. 11 3.2.2 Physical Specifications .................................................................................................................... 11 3.2.3 Physical Installation ....................................................................................................................... 12 3.2.4 Power Requirements ..................................................................................................................... 12 3.2.5 Cooing Requirements .................................................................................................................... 12

3.3 Catalyst 4500-X Software Attributes ................................................................................................... 12 3.3.1 Software Revision .......................................................................................................................... 12 3.3.2 Licensing ......................................................................................................................................... 12

4 NEXUS 5000 HARDWARE ............................................................................................................................ 13

4.1 Nexus 5672UP Hardware Attributes .................................................................................................... 13 4.1.1 Hardware Components ................................................................................................................. 13 4.1.2 Physical Specifications ................................................................................................................... 13 4.1.3 Power Requirements ..................................................................................................................... 13

4.2 Nexus 5548 Software Attributes .......................................................................................................... 13 4.2.1 Software Revision .......................................................................................................................... 13 4.2.2 Licensing ......................................................................................................................................... 13

5 NEW TECHNOLOGIES .................................................................................................................................. 14

5.1 Overview ................................................................................................................................................ 14 5.2 Virtual Switching System (VSS) ............................................................................................................ 14 5.3 VSS Components ................................................................................................................................... 14 5.4 VSS Domain ID & Switch Numbers ....................................................................................................... 14 5.5 Virtual Switch Link (VSL) Port-Channel and Ports ............................................................................... 15 5.6 VSS Dual Active Link Detection ............................................................................................................ 15 5.7 Converting the Switch to Virtual Switch Mode ................................................................................... 15

5.7.1 VSS - Single Homed Device Considerations .................................................................................. 15 5.8 Virtual Port Channels (vPC) .................................................................................................................. 16

5.8.1 Overview ........................................................................................................................................ 16 5.8.2 vPC Components ............................................................................................................................ 16


5.8.3 vPC Configuration .......................................................................................................................... 17 5.8.4 vPC Considerations ....................................................................................................................... 20 5.8.5 vPC Single Homed Devices (Orphan Ports) .................................................................................. 21 5.8.6 HSRP ............................................................................................................................................... 21

6 STANDARD PRACTICE ................................................................................................................................. 22

6.1 Overview ................................................................................................................................................ 22 6.2 Layer 2 .................................................................................................................................................... 22

6.2.1 VLAN naming and ID assignment .................................................................................................. 22 6.2.2 VLAN Configuration (Core) ........................................................................................................... 22 6.2.3 VTP .................................................................................................................................................. 22 6.2.4 CDP ................................................................................................................................................. 23 6.2.5 Spanning-Tree Protocol ................................................................................................................. 23 6.2.6 Link Aggregation........................................................................................................................... 26

6.3 Layer 3 .................................................................................................................................................... 27 6.3.1 HSRP and SVI interfaces ................................................................................................................ 27 6.3.2 VLSM addressing using /31 Subnets for Point-to-Point links ....................................................... 27 6.3.3 Bidirectional Forwarding Detection (BFD) .................................................................................. 28

6.4 IP Services ............................................................................................................................................. 30 6.4.1 DHCP Relay .................................................................................................................................... 30

6.5 Catalyst 4500-X Specifics ..................................................................................................................... 30 6.5.1 High Availability through SSO and NSF ....................................................................................... 30

6.6 Miscellaneous Diagrams ....................................................................................................................... 31 6.6.1 Spanning Tree & FHRP ................................................................................................................... 31

TABLE OF FIGURES Figure 1 – Overall Design Diagram ...................................................................................................................... 10 Figure 2 – VSS Physical vs Logical Topology ....................................................................................................... 14 Figure 3 – vPC ....................................................................................................................................................... 16 Figure 4 - Spanning Tree & FHRP Diagram ......................................................................................................... 31

TABLE OF TABLES Table 1 - Device Abbreviations ............................................................................................................................. 9 Table 2 – Catalyst 4500-X Weight ........................................................................................................................ 11 Table 3 – 4500-X Power & BTU Allocation........................................................................................................... 11 Table 4 - Nexus 5672UP Weight .......................................................................................................................... 13 Table 5 - Nexus 5672UP Dimensions ................................................................................................................... 13 Table 6 - Nexus 5548 Power Requirements ....................................................................................................... 13


RESTRICTION ON DISCLOSURE All information in this document shall not be published or disclosed wholly or in part to any third party without Resolution’s prior permission in writing, which will also be held securely. These obligations shall not apply to information that is published or becomes known legitimately from some source other than Resolution Data Ltd


INTRODUCTION Overview

A Customer has engaged Resolution to assist them with their core network infrastructure, which requires an urgent technology refresh. The existing Chassis are at full capacity, out of support and are creating bottlenecks on the corporate network. Resolution have been given the task to design and migrate to a new core network which will meet the growing demands of their user base and business requirements over the next 5 years. The Core and DC environment will undergo a technology refresh as per this document. The Access Layer, WAN and additional network services refresh such as load balancers and firewalls are out of scope in this design. The design will cover methods of connectivity for these services only. Approach

Due to the risk involved, the new infrastructure will be implemented in two phases: Phase 1: The initial phase will look to replace the existing core Catalyst 6509 switch as this is deemed the most critical device to the enterprise infrastructure, the new equipment being installed will be integrated with the existing server estate environment consisting of a Cisco UCS chassis. The new installed core switches will be deployed as a VSS pair Phase 2: The second phase will see the additions of two Nexus 5548-UP switches. These will be configured as a vPC pair and will make links to the newly installed core switches and the Fabric Interconnect (FI) switches for the server estate.


1 SCOPE

1.1 Project Scope

The project scope includes the following:

Production of a Low Level Design (This Document)

Migration Plan

Test Plan

Installation and migration of the existing Catalyst core to the new Nexus core network

1.2 Design Scope

The low level design will include the following:

Overview of the existing estate

Described methods of connectivity from the Core switches to the Nexus Switches in the FlexPod environment

Design of the Enterprise Core as well as the Nexus switches

Example of best practice configuration

Enterprise 4500 and Nexus device security

Enterprise 4500 and Nexus device management

The following are considered out of scope of this document:

Data centre physical layout

Fibre optic cabling design

Power provisioning for DC and communication rooms

UPS specification

1.3 Document Structure

The remaining structure of the low level will be divided up into a number of main sections which are titled as follows:

Network Design Overview

Cisco Catalyst 4500 Hardware

Nexus 5000 Hardware

New Technologies

Standard Practice

Core

Server Aggregation

External Access

OSPF Design

Quality Of Service

Multicast

Security

Network Management & Monitoring


2 NETWORK DESIGN

2.1 Overview

The new network design for A CUSTOMER uses a strategic selection of Cisco Catalyst and Nexus platforms to support IP and future converged FC/FCoE protocols for the business as well as the server infrastructure. These platforms will provide connectivity for devices that span from the core to the server aggregation layers within the network. The platform selection will offer CUSTOMER the scalability for the long term, superior functionality, and high levels of resilience on purpose built infrastructure that will exhibit extremely low convergence times around link and node failures, as well as higher throughput speeds of 10Gbps utilizing the VSS and vPC technologies.

A FlexPod will be deployed for the server estate. FlexPod components include Cisco Unified Computing System (Cisco UCS) servers, Cisco Nexus switches, and NetApp unified storage systems. The FlexPod architecture can scale up or out and it can be optimized for a variety of mixed workloads, in both virtualized and non-virtualized environments.

The A CUSTOMER core network will be contained to one server room only. This will be hosting the entire core and DC infrastructure. The network infrastructure will utilize a tiered model providing core, server aggregation and server access.

Note: The FlexPod implementation is considered as part of the overall design; however the UCS and Netapp unified storage components are considered out of scope.

The network hardware in the new design is as follows: 2.2 Core and Server Aggregation

Cisco Catalyst 4500-X Series Switches (VSS)

2.3 Server Access Layer (FlexPod)

Nexus N5K-5672UP Switches

http://www.cisco.com/c/en/us/products/servers-unified-computing/index.html

http://www.cisco.com/c/en/us/products/servers-unified-computing/index.html


Given the capabilities of the Cisco 4500 Series switches, the first main section will detail the initial setup of the 4500-X that allows the remainder of the design for A CUSTOMER to be implemented. The VSS section is a primer for the new core aggregation technologies that are leveraged heavily in the design which should be understood from a concept and configuration standpoint before moving further through the document. Common configurations are grouped to save duplication, and explain the best practice configurations for Layer 2 and Layer 3 protocols which should be kept standard across all areas throughout the design. The next sections will detail each sites physical and logical design for that particular network layer. This includes hostnames, IP addressing, device types etc. 2.4 Device Abbreviations

Throughout this document there are abbreviations used for different layers or devices within the design. As a summary these abbreviations are as follows:

Abbreviation Description

CAL Core Aggregation Layer

SAL Server Aggregation Layer

Table 1 - Device Abbreviations

2.5 Example Configurations

Example configurations throughout the remainder of this document will use both NX-OS and IOS syntax. Not every instance will show both examples as it’s expected that engineers know the configuration syntax differences between the two operating systems.


2.6 Overall Physical Diagram

The diagrams below depict the new infrastructure the project will deliver as part of the core network refresh:

SLOT

1

SLOT

5

SLOT

3

SLOT

7

SLOT

2

SLOT

6

SLOT

4

SLOT

8

!

UCS 5108

OK FAIL OK FAIL OK FAIL OK FAIL

! ResetConsole

UCS B200 M3

Dri

ve S

lot

Co

ver

DS4243

600GB

600GB

600GB

600GB

600GB

600GB

600GB

600GB

600GB

600GB

600GB

600GB

600GB

600GB

600GB

600GB

600GB

600GB

600GB

600GB

600GB

600GB

600GB

600GB

4 5 6 70 1 2 3 12 13 14 158 9 10 11 20 21 22 2316 17 18 19

DS2246

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

DS4243

600GB

600GB

600GB

600GB

600GB

600GB

600GB

600GB

600GB

600GB

600GB

600GB

600GB

600GB

600GB

600GB

600GB

600GB

600GB

600GB

600GB

600GB

600GB

600GB

4 5 6 70 1 2 3 12 13 14 158 9 10 11 20 21 22 2316 17 18 19

DS2246

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

900G

B

! ResetConsole

UCS B200 M3

A B

FAS8020FAS8020

PS

1P

S2

FAN

STAT

FAN

STATFAN1

FAN2

FAIL

OK

FAIL

OKID

STAT

MGMTOL1

L2 CONSOLE

PS

1P

S2

FAN

STAT

FAN

STATFAN1

FAN2

FAIL

OK

FAIL

OKID

STAT

MGMTOL1

L2 CONSOLE

ID

N5K-C56-72UP

STAT

2

5 6 7 8

1 3 4 10

13 14 15 16

9 11 12 18

21 22 23 24

17 19 201 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 4825 26 27 28 29 30 31 32

USBFANPS2PS1STATUSUID

Catalyst 4500X Series

1 2 3 4 5 6 7 8STATUS

OIR

OIR1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

SLOT

1

SLOT

5

SLOT

3

SLOT

7

SLOT

2

SLOT

6

SLOT

4

SLOT

8

!

UCS 5108

OK FAIL OK FAIL OK FAIL OK FAIL

! ResetConsole

UCS B200 M3

Dri

ve S

lot

Co

ver

! ResetConsole

UCS B200 M3

USBFANPS2PS1STATUSUID

Catalyst 4500X Series

1 2 3 4 5 6 7 8STATUS

OIR

OIR1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

ID

N5K-C56-72UP

STAT

2

5 6 7 8

1 3 4 10

13 14 15 16

9 11 12 18

21 22 23 24

17 19 201 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 4825 26 27 28 29 30 31 32

VSS

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

CO

NS

OL

E

UCS

C240 M3

!

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

CO

NS

OL

E

UCS

C240 M3

!

2x 10 GbE with FCoE per FI




2x 10 GbE FCoE

2x 10 GbE FCoE

Converged

10 GbE Only

FCoE Only

Mgmt

PO PO

vPC vPC

PO

vPC vPC

Legend

Figure 1 – Overall Design Diagram


3 Cisco 4500 Hardware

3.1 Overview

The Cisco Catalyst 4500-X switch is a fixed aggregation switching platform that is capable of virtualization and integrated network services for video, security and application visibility. With support for VSS the platform provides a single point of management whilst offering up to 1.6Tbps switching capacity. Using a VSS deployment enables cross-chassis high-availability capabilities such as In-Service Software Upgrade (ISSU), Nonstop Forwarding (NSF), and Stateful Switchover (SSO). Within the new design the two Catalyst 4500-X will provide the core infrastructure utilizing a VSS design. Note: For VSS capability, the minimum software requirement is Cisco IOS XE Software Release 3.4.0SG 3.2 Catalyst 4500- Physical Attributes

The Catalyst 4500-X comes in a one-rack unit (1RU) which lends to a low-power form factor deployment. For A CUSTOMER Deployment, each Catalyst 4500-X will consist of the following components:

3.2.1 Hardware Components

1 x Catalyst 4500-X Chassis No PSU (WS-C4500-X-40X-ES)

2 x 750E AC Power Support (C4KX-PWR-750AC-R)

1 x Catalyst 4500-X 8 Port 10G Network Module (C4KX-NM-8SFP+)

N+1 full power redundancy will be enabled for each device

3.2.2 Physical Specifications

The following physical specifications apply to each Catalyst 4500-X switches, below is the weight of each switch and its installed components along with dimensions of each chassis:

Model Additional Parts Height

(U) Width (cm)

Depth (cm)

Weight (Kg)

WS-C4500-X-40X-ES N/A 1 43.1 53.34 8.23

N/A C4KX-NM-8SFP+ N/A N/A N/A 0.84

N/A C4KX-PWR-750AC-R N/A N/A N/A 1.68

N/A C4KX-PWR-750AC-R N/A N/A N/A 1.68

Total 53.34 12.43

Table 2 – Catalyst 4500-X Weight

Model Output Power

(W) Switch Power Consumption

(W) Heat Dissipation

(BTU(W)/Hr)

WS-C4500-X-40X-ES 1500 330 1122 (329 W)

Table 3 – 4500-X Power & BTU Allocation


3.2.3 Physical Installation

The Catalyst 4500-X should ideally be installed in a standard four post 1200mm rack so that it can accommodate cable overhang while ensuring that the cabinet doors can be closed The airflow of the Catalyst 4500-X switches is front to back for the power supplies, as such special considerations need to be taken into account regarding the clearance on the front and back for the rack of the chassis, ensuring that all slots and openings on a chassis remain unobstructed, especially the fan assembly vent at the back of the chassis. To maintain proper air circulation through the switch chassis, it is recommended that you maintain a minimum 6-inch (15 cm) separation between a wall and the chassis hot air exhaust

3.2.4 Power Requirements

The power supplies for each Catalyst 4500-X are rated at 750W. Two power supplies will be used for each chassis for full redundancy. These power supplies will be configured to operate in ‘full-redundancy’ mode. Each active PSU requires two power feeds that should be rated not greater than 20A. Feed receptacles should be within 3.0m of each power supply.

Power Recommendations

Each power supply should be connected to separate input sources; otherwise the chassis could be susceptible to a power failure

3.2.5 Cooing Requirements

A single Catalyst 4500-X chassis in the data centre will dissipate approximately 1122 BTU’s per hour. It’s important that each chassis is placed in the appropriate position in racks/isles for optimal cooling. This will ensure that the system fans do not need to work excessively to cool the system. The air flow design of the 4500-X chassis is front-to-back for the power which suits hot isle cold isle data centres. 3.3 Catalyst 4500-X Software Attributes

The following sections describe the software version and licensing that will be used on each chassis.

3.3.1 Software Revision

The software revision to be used on the Nexus 7000 is version IOS XE 3.4.4SG and this is the recommended version for now. This will also allow support for VSS which is required as per the design

3.3.2 Licensing

Each Catalyst 4500-X will chassis have the “Enterprise Services” which will allow for every feature available in the 4500-X platform


4 NEXUS 5000 HARDWARE

4.1 Nexus 5672UP Hardware Attributes

The following sections describe the hardware specification for each Nexus 5600 switch.

4.1.1 Hardware Components

1 x Nexus 5672UP Switches

2 x 1100W Power supplies

N+1 full power redundancy will be enabled for all switches.

4.1.2 Physical Specifications

The following physical specifications apply to each of the Nexus 5672UP switch which will have an impact on the installation of each device. Shown below is the weight of each switch.

N5K

Quantity Part Weight

1 N5K-C5672UP 14.51Kg

Total 14.51Kg

Table 4 - Nexus 5672UP Weight

Displayed below are the physical dimensions of each Nexus 5672UP chassis:

N7K

Height Width Depth

4.4cm 43.9cm 74.9cm

Table 5 - Nexus 5672UP Dimensions

4.1.3 Power Requirements

The table below lists the power requirement for each of the Nexus 5672UP chassis that will be installed:

Ports Model Input

Current (A)

Power Supply Rating

(W)

Typical Power Used (W)

Maximum Power Used (W)

Heat Dissipation

(BTU/Hr)

48 N5K-

C5672UP 3.75 750 390 600 1331

Table 6 - Nexus 5548 Power Requirements

4.2 Nexus 5548 Software Attributes

The following sections describe the software version and licensing that will be used on each chassis.

4.2.1 Software Revision

The software revision to be used on the Nexus 5672UP is version 6.0(2)N1(2a) and this is the recommended version for now and is also in use at A CUSTOMER 72 Hammersmith Rd site.

4.2.2 Licensing

No special licensing is required for the Nexus 5000 series switches as these switches are acting as pure Layer 2 switches. The switches will be running the standard “LAN Base” feature set.


5 NEW TECHNOLOGIES

5.1 Overview

Given that some new technologies are being introduced into the A CUSTOMER network to support the new design, this section aims to cover new concepts and features that are later referenced in the site specific sections under layer 2 and layer 3 designs. These features are applicable to key components within each topology and it’s highly advisable that readers become familiar with this section prior to reading the remaining parts of this document. Each technology shown in the following sections includes configuration examples that are based on Cisco best practices which should be adhered to throughout the network design unless otherwise stated. This information will not be mentioned again so it should be known that where remaining parts of the document mentions a technology (which is mentioned in this section), the configuration will be more or less the same. 5.2 Virtual Switching System (VSS)

A VSS combines a pair of Catalyst 4500 or 4500-X series switches into a single network element. The VSS manages the redundant links, which externally act as a single port channel.

The VSS simplifies network configuration and operation by reducing the number of Layer 3 routing neighbours and by providing a loop-free Layer 2 topology.

Figure 2 – VSS Physical vs Logical Topology

5.3 VSS Components

5.4 VSS Domain ID & Switch Numbers

The VSS domain must match between both switches. The virtual switch domain is a number between 1 and 255, and must be unique for each VSS in your network (the domain number is incorporated into various identifiers to ensure that these identifiers are unique across the network).

Within the VSS, you must configure one switch to be switch number 1 and the other switch to be switch number 2. To configure the virtual switch domain and switch number on both switches, perform these tasks:

Config Example Switch-1(config)# switch virtual domain 100

Switch-1(config-vs-domain)# switch 1

Switch-2(config)# switch virtual domain 100

Switch-2(config-vs-domain)# switch 2


5.5 Virtual Switch Link (VSL) Port-Channel and Ports

The VSL link is formed between the two chassis with the VSS domain. This link is forms a port-channel that is used to pass data and control traffic between the two switches within the domain. To configure the virtual switch link on both switches, perform these tasks: . Config Example

Switch-1(config-interface)# interface range tengigabitethernet 4/1-2

Switch-1(config-if)# channel-group 10 mode on

Switch-1(config)# interface port-channel 20

Switch-1(config-interface)# switch virtual link 1

Switch-2(config-interface)# interface range tengigabitethernet 4/1-2

Switch-2(config-if)# channel-group 10 mode on

Switch-2(config)# interface port-channel 20

Switch-2(config-interface)# switch virtual link 2

5.6 VSS Dual Active Link Detection

The VSS switches require a dual active detection mechanism to protect against a split brain scenario in the event of a VSL link failure. Fast Hellos will be configured across the VSS pair to support the dual active failure protection. To configure Dual Active Link Detection on both switches, perform these tasks: Config Example Switch-1(config)# switch virtual domain 1

Switch-1(config-vs-domain)# dual-active detection fast-hello

Switch-1(config)# interface gigabitethernet 1/5/1

Switch-1(config-if)# dual-active fast-hello

Switch-2(config)# switch virtual domain 1

Switch-2(config-vs-domain)# dual-active detection fast-hello

Switch-2(config)# interface gigabitethernet 1/6/1

Switch-2(config-if)# dual-active fast-hello

5.7 Converting the Switch to Virtual Switch Mode

Once all of the above has been configured you can convert the single chassis into a VSS. To convert from a single switch to a VSS on both switches, perform these tasks:

Config Example

Switch-1# switch convert mode virtual

Switch-2# switch convert mode virtual

Note After you confirm the command (by entering yes at the prompt), the running configuration is automatically saved as the startup configuration and the switch reboots. After the reboot, the switch is in virtual switch mode, so you must specify interfaces with three identifiers (switch/module/port). When switches are being converted to VSS, you should not set them to ignore startup-config. If done, the switch can be enabled to parse the startup-config at the rommon prompt. Ignoring startup-config in VSS mode, causes a switch to boot in a semi-VSS mode, which can only be corrected by a reboot and by enabling the parsing of startup-config.

5.7.1 VSS - Single Homed Device Considerations

Devices that are single homed into a single VSS member switch are known as Orphan Ports. In the event of a VSL link failure, the VSS system will shut down all ports connected to the operation primary switch. If the switch that is deemed primary has orphan port connections, then these ports will be shutdown causing a loss of service to these devices. The VSL link failure is a rare scenario, but should be protected against and all devices should be dual homed across each of the VSS member switches. The desired design is to have no orphan ports connected to the VSS domain.


5.8 Virtual Port Channels (vPC)

5.8.1 Overview

Virtual PortChannels (vPCs) allow links that are physically connected to two different Cisco switches to appear to a third downstream device as coming from a single device and as part of a single port channel. The downstream device can be a switch, a server, or any other networking device that supports IEEE 802.3ad Port Channels. vPC allows the creation of Layer 2 PortChannels that span two switches and at present vPC is implemented on the Nexus 7000 and 5000 series platforms running NX-OS version 4.1(4) or higher. Benefits of vPC are device level redundancy with faster convergence than using spanning tree, elimination of blocked ports which promotes a loop-free topology, and much better bandwidth utilisation.

5.8.2 vPC Components

vPCs consist of two vPC peer switches, which are referred to as Switch 1 and Switch 2 in the diagram below. Of the vPC peers one is primary and the other is secondary. The system formed by Switch 1 and Switch 2 is referred to as a vPC domain.

Figure 3 – vPC

The vPC peer switches are connected through a link called peer link, also known as multi-chassis PortChannel trunk. This is a standard 802.1Q trunk that carries vPC and non vPC VLANs, Cisco Fabric Services messages (which are tagged as CoS 4), flooded traffic from the other vPC peer, and other protocols like STP BPDU’s, IGMP joins/updates, HSRP hellos etc. The ports that form the PortChannel are split between the vPC peers and are referred to as vPC member ports. An out-of-band link is used to resolve dual-active scenarios where the peer-link connectivity is lost. This link is referred to as vPC peer-keepalive or fault-tolerant link. Some devices can be single-attached to the topology, like server 1 and server 3. The ports connecting devices in a non-vPC mode to a vPC topology are called orphaned ports. Design considerations often surround these types of ports – as its extremely important for the design.


5.8.3 vPC Configuration

The following sections show how to correctly configure a vPC domain and attach downstream devices.

vPC Role and Priority

First step in configuring vPC is to enable the feature ‘vpc’ and then a domain needs to be defined (as indicated by a domain-id) as well as priorities to define primary and secondary roles in the vPC configuration. The lower number in a vPC priority is more preferred – Generally this is should be aligned with the spanning-tree priorities in the topology for ease of management. While mismatched Spanning-Tree and vPC priorities do not any impact on traffic forwarding under normal conditions, it is desirable to keep the priorities matched so as to have Spanning-Tree root and vPC primary on the same device and Spanning-Tree secondary root and vPC secondary on the same device. For the two switches (vPC peers) to form a vPC system, the domain-id of these switches need to match. The domain-id is used to generate the system-id which is used for the LAGID in the LACP negotiation and BPDU generation. Shown below is the configuration to create and set the vPC domain, and set the matching STP priorities: N5K-1(config)# feature vpc

N5K-1(config)# vpc domain 1

N5K-1(config-vpc-domain)# role priority 4096

N5K-2(config)# feature vpc


N5K-2(config-vpc-domain)# role priority 8192

It should also be noted that the vPC role is not preemptive, so a device may be operationally primary but secondary from a configuration perspective. This is not an issue.

vPC Peer Keepalive

Next step is to configure the mechanism for dual-active detection. The keepalive that resolves dual-active scenarios should never be carried as a VLAN on the peer link. It should be carried over separate routed infrastructure. The reason for this is if the peer-link fails and the keepalive fails at the same time, you will have split brain and a broken network topology. The recommendation is to use the Nexus out-of-band management interface for this task. However, you should not use the mgmt0 interface for a direct back-to-back connection between Nexus 7000 systems because the mgmt0 IP address is only ever active on one supervisor, and you cannot discern which supervisor is active at any given time. For the Nexus 5600 this is acceptable as only one mgmt interface exists, but not recommended as it means device management would have to be in-band. The keepalive itself is a UDP message on port 3200, is 96 bytes long (32 byte payload), and includes version, time stamp, local and remote IPs, and the domain ID. The following configuration shows how to use the management interface for peer-keepalive. This assumes that out-of-band management hasn’t currently been set up: N5K-1(config)# vrf context management

N5K-1(config-vrf)# ip route 0.0.0.0/0 192.168.0.1

N5K-1(config)# interface mgmt0

N5K-1(config-if)# ip address 192.168.0.2/24

N5K-1(config-if)# no shutdown

N5K-1(config-if)# exit


N5K-1(config-vpc-domain)# peer-keepalive destination 192.168.0.3 source 192.168.0.2 vrf management


vPC Peer-Link

A PortChannel is used to connect the two vPC devices together similar to a normal ISL, but instead its referred to as the peer-link. It still carries VLANs just like an ISL would. The peer-link link also carries additional traffic such as BPDUs and HSRP hellos, and MAC address synchronization between the vPC peers using CFS (Cisco Fabric Services). On the N7K Series, this PortChannel should be configured on 10 Gigabit Ethernet interfaces configured in dedicated-mode across two different 10 Gigabit Ethernet line cards. However there are exceptions to this which will be explained in later site specific sections. The peer-link is by far the most important component in the vPC system, in that its failure may impair the establishment of new flows and isolate orphan ports. Configuring the peer link in a redundant fashion ensures uninterrupted connectivity between the vPC peers in almost every likely failure scenario. The following configuration illustrates how to configure the peer-link across two 10Gbps interfaces which in this case is PortChannel1 using LACP active negotiation at both ends.

N5K-1(config)# feature lacp

N5K-1(config)# interface Port-Channel1

N5K-1(config-if)# switchport

N5K-1(config)# interface E1/1 , E2/1

N5K-1(config-if)# channel-group 1 mode active

N5K-1(config-if)# no shut

N5K-1(config)# interface Port-Channel1

N5K-1(config-if)# switchport mode trunk

N5K-1(config-if)# switchport trunk allowed vlan <vlans>

N5K-1(config-if)# vpc peer-link


Note: You should always create a Port-Channel interface first and specify if its Layer 2 or

Layer 3 using the ‘switchport’ or ‘no switchport’ command before bundling any links.

vPC Member Ports

PortChannels are configured by bundling ports on each Nexus switch through the command ‘vpc’, as shown in the following configuration example. The system will issue an error message if the PortChannel wasn’t previously configured as a switchport as it’s a requirement of vPC to operate only with Layer 2 interfaces. It is always good practice to use the Link Aggregation Control Protocol (LACP) for bundling of the ports in the vPC. This is because LACP determines that the ports being bundled are actually part of the same physical or virtual switch, preventing erroneous configurations.

The configuration shown below places a single 10Gbps interface on each vPC peer into a PortChannel and then applies vPC to the PortChannel on each peer. Following this is an example configuration of a PortChannel on a downstream device such as a Catalyst 2950:


N5K-1(config)# interface Port-channel10


N5K-1(config)# interface Ethernet1/7




N5K-1(config-if)# vpc 10






N5K-2(config)# interface Ethernet1/7




N5K-2(config-if)# vpc 10


Downstream device configuration:

C2950(config)# interface range GigabitEthernet1/0/49 , GigabitEthernet2/0/49

C2950(config-if)# switchport

C2950(config-if)# channel-group 10 mode passive

C2950(config)# interface Port-channel 10

C2950(config)# no shut

Note: The vpc ID doesn’t need to be the same as the PortChannel ID, however its recommended; also the recommended configuration of downstream devices is to use passive negotiation.

vPC Consistency Checks

Consistency checks refer to a number of configuration parameters on the PortChannels (both member and peer-links) that must be identical on both peers in the domain. If any of these values are mismatched it can cause the vPC formation to fail. Examples of these are allowed VLANs, speed, duplex, spanning-tree port types, and port mode (switchport / trunk). To view these you can use the ‘show vpc consistency-parameters’ command set.

vPC Reload Restore

In major outage scenarios, both Nexus 7000 devices that make up a vPC domain could be reloaded. During this scenario, occasionally only one of the peers can be restored. This is because the vPC port channels cannot come up if no handshake has ever been performed with its remote peer. A new feature in NX-OS 5.0(2) allows you to configure the Nexus 7000 to restore vPC services when its remote peer fails to come online. Its recommended to use the command ‘reload restore’ on both vPC peers to avoid this rare scenario.

vPC Peer Gateway

In NX-OS you can configure vPC peer devices to act as the gateway for packets that are destined to the vPC peer device's MAC address. Particularly with some network-attached storage (NAS) devices or load-balancers, they may have features which are aimed at optimising application performance. Behaviour of these devices will reply to traffic using the MAC address of the sender Cisco Nexus 7000 device rather than the common HSRP gateway. This behaviour is non-complaint with some basic Ethernet RFC standards. Packets reaching a vPC device for the non-local router MAC address are sent across the peer-link and could be dropped by the built in vPC loop avoidance mechanism if the final destination is behind another vPC. The vPC peer-gateway capability allows a vPC switch to act as the active gateway for packets that are addressed to the router MAC address of the vPC peer. This feature enables local forwarding of such packets without the need to cross the vPC peer-link. In this scenario, the feature optimizes use of the peer-link and avoids potential traffic loss.


Configuring the peer-gateway feature needs to be done on both primary and secondary vPC peers and is non-disruptive to the operations of the device or to the vPC traffic. The vPC peer-gateway feature can be configured globally under the vPC domain. The configuration is shown below: N5K-1(config)# vpc domain 1

N5K-1(config)# peer-gateway

Note: When enabling this feature it is also required to disable IP redirects on all interface VLANs mapped over a vPC VLAN to avoid generation of IP redirect messages.

vPC Peer-switch

The vPC peer switch feature addresses performance concerns around STP convergence. This feature allows a pair of Cisco Nexus devices to appear as a single STP root in the Layer 2 topology and it eliminates the need to pin the STP root to the vPC primary switch as well as improving vPC convergence if the vPC primary switch fails. To avoid loops the vPC peer link is excluded from the STP computation. In vPC peer switch mode, STP BPDUs are sent from both vPC peer devices to avoid issues related to STP BPDU timeout on the downstream switches, which can cause traffic disruption. Below is a configuration example: NX-OS

N5K(config)#vpc domain 1

N5K(config-vpc-domain)#peer-switch

Note: RSTP VLANs priority on both switches must be the same.

5.8.4 vPC Considerations

Attaching to a vPC domain

When attaching devices to a vPC domain you should always dual-home them to both vPC peers. There are cases where this cannot happen such as using load balancers or redundant firewalls, and if so appropriate measures should be taken to ensure that during certain failure scenarios that single homed devices are unaffected, and traffic flow is not black holed.

Double Sided vPC

Double sided vPC is the attachment of a pair vPC devices to one another pair and utilising the technology to create a single logical link between them. An example of this is a pair of Nexus 5600 switches attached to a pair of Nexus 7000, with a full mesh of links aggregated into a single PortChannel. As mentioned earlier domain-id defined for the vPC domain is used to generate the system-id of the system comprised of the vPC peers. Because of this it is important to ensure that each vPC “system” in a topology such as the one described utilizes a different domain-id number. This ensures uniqueness in the Bridge ID used for BPDUs and the LAGID used by LACP. Alternatively it’s possible to define the same domain-id as long as a different system-id (system-priority) has been configured under the vPC domain. It’s recommended to configure the system-priority manually the same as the domain ID.


5.8.5 vPC Single Homed Devices (Orphan Ports)

Overview

In a vPC environment the most important link in the technology is the Peer-Link. If the Peer-Link fails for whatever reason you have the potential for a split brain scenario where both vPC peers think everything is normal, and continue to advertise subnets to non-vpc peers and continue forwarding on downlinks (vPC member ports). This behavior is avoided by the use of the Peer Keepalive which determines if both peers are still responsive on the out-of-band link, and if so, the currently acting secondary role will shut all of its vPC member ports, and bring down all of its SVI interfaces for any VLANs that are configured across the Peer-Link. This behavior is fine for situations when all devices are dual homed to the vPC domain and Port-Channeled to both peers. If you have single homed devices attached to the domain, you need to think carefully about this failure scenario and determine if the failure of the peer-link would cause extended loss of service because its broken the path of traffic, and cannot be resolved without manual intervention. Within this design vPC is only configured on the server aggregation layer of the network, where devices are dual homed. When we consider the possibility of losing the vPC Peer-Link either by misconfiguration, or disconnected fibers etc. The secondary vPC peer will shut its member ports because the keep alive has identified that both are still reachable. For A CUSTOMER network devices are dual homed into both the core and server aggregation Nexus 7000 switches.

5.8.6 HSRP

The use of HSRP in the context of vPC does not require any special configuration. With vPC, only the active HSRP interface answers ARP requests, but both HSRP interfaces (active and standby) can forward traffic. If an ARP request coming from a server arrives on the secondary HSRP device, it is forwarded to the active HSRP device through the peer-link.


6 STANDARD PRACTICE

6.1 Overview

The following section goes through standard practices which should remain consistent across the design and on-going unless there is a good reason to change them. This is broken down into Physical, Layer 2, and Layer 3 technologies which are typically Layer 2 and Layer 3 protocols such as spanning-tree, HSRP etc. This section is mainly to avoid duplication of information throughout every section. If configurations are pertinent to certain network layers it will be specified otherwise the section is global to each network layer and/or site. Reason for this is that different areas of the network require different policy and security or trust levels. Devices covered will include:

NX-OS: Nexus 5600

IOS: Catalyst 4500

6.2 Layer 2

6.2.1 VLAN naming and ID assignment

When creating new VLANs they should have a naming convention to easily identify their function, and the actual VLAN ID should ideally be unique to the site. Exceptions to this are when Layer 2 extension is required between two locations using technologies such as OTV. OTV is not used within this phase of the design and may be looked at in future. Existing VLAN names will be used, unless absolutely required no additional VLANs will be created.

6.2.2 VLAN Configuration (Core)

A CUSTOMER have VLANs that span to many of the communication rooms scattered around the campus. There is no requirement to redesign the VLAN scheme and is out of scope, as such the VLAN design for A CUSTOMER will not change for the core/access layer and server aggregation/server access data networks, it is important to mention that these 2 network segments will be separated by the use of VDCs and connectivity between the two will be achieve with Layer 3 links.

6.2.3 VTP

VTP is a Layer 2 messaging protocol that maintains VLAN configuration consistency by managing the addition, deletion, and renaming of VLANs within a VTP domain. Nexus switches will support the standard VTP modes of client, server, and transparent, with an additional mode of ‘vtp off’. The main difference with VTP being turned off is that the device will not relay any VTP messages between devices. A CUSTOMER makes use of VTP to automatically configure VLANs on switches within the same VTP domain, this function ability will be enabled on the nexus switches which by default have it turned off. Within the design it is recommended that the core/access layer and the server aggregation/server access layer reside in separate VTP domains to prevent accident deletion of VLANs should Layer 2 links be added between the two domains. The Nexus 7000 switches will act as the VTP servers and all VLANs will be created in the appropriate VDCs, all other switches in the network will act as VTP clients. The VTP Server configuration on Nexus:

N5K-1(config)# feature vtp

N5K-1(config)# vtp mode server

N5K-1(config)# vtp domain Domain01

N5K-1(config)# vtp version 2

N5K-1(config)# vtp password Password123


6.2.4 CDP

CDP is a device discovery protocol that runs over Layer 2 (the data link layer) on all Cisco-manufactured devices (routers, bridges, access servers, and switches) and allows network management applications to discover Cisco devices that are neighbours of other already-known devices. CDP should be left enabled when facing TRUSTED devices as it assist with troubleshooting and is required for the implementation of Wireless AP’s and IP phones.

6.2.5 Spanning-Tree Protocol

Spanning Tree Protocol provides a mechanism for physically redundant network topologies to remain loop free. All devices in a Layer 2 domain run the spanning-tree calculations to discover the topology and calculate the best path to the root bridge. Through the spanning-tree process redundant network links are placed into a blocking state preventing loops from occurring at Layer 2. The recommended spanning-tree protocol across the board for A CUSTOMER is Rapid-PVST protocol.

Root Bridge Placement & STP Load Balancing (Core / Server Aggregation)

For every instance of spanning-tree (per-VLAN) there is one device elected Root Bridge. The root bridge is typically at the centre of the network topology and all of its Layer 2 links in forwarding state. In a typical network design, spanning-tree is also used to provide load balancing across Layer 2 paths where multiple VLAN’s exist. This normally achieved by configuring multiple devices to act as the root bridge, and then mapping the root bridge with other protocols such as HSRP. Root bridge placement in vPC deployments is a much more simplified configuration because one of the key characteristics of vPC is active/active forwarding on Layer 2 paths. In this scenario the root bridge placement is less of a concern since load balancing is an inherent part of vPC and the use of the peer-switch command. As such the root bridge placement in every Layer 2 domain of the A CUSTOMER design is to use the same priority for all VLAN’s to Nexus 7000 VDC’s, and make the root priorities the same as the vPC domain priorities where appropriate.

Config Example: vPC Primary N5K-1(config)# spanning-tree mode rapid-pvst

N5K-1(config)# spanning-tree vlan 1-4096 priority 4096

vPC Secondary N5K-2(config)# spanning-tree mode rapid-pvst

N5K-2(config)# spanning-tree vlan 1-4096 priority 4096

Spanning Tree behaviour with vPC

There are two key behavioural differences with spanning tree in a vPC environment:

vPC imposes the rule that the peer link should never be blocking because this link carries important traffic such as the Cisco

Fabric Services over Ethernet (CFSoE) Protocol. The peer link is always forwarding.

For vPC ports only, the operational primary switch generates and processes BPDUs. The operational secondary switch forwards

BPDUs to the primary switch.


Spanning Tree Path Cost

In a regular spanning-tree environment with no additional configurations, the cost of most links in a data centre is almost identical regardless of the available bandwidth along a given path:

1Gbps link = 4

2Gbps links = 3

3Gbps links = 3

4Gbps links = 3

Considering that spanning tree was designed before the availability of 10 GigabitEthernet or even GigabitEthernet, the default spanning-tree reference cost for a link is inadequate, as most links with 10 GigabitEthernet bandwidth or bandwidth with multiples of 10 Gigabit Ethernet will appear to have the same cost. With ‘spanning-tree pathcost method long’ enabled, spanning tree calculates the best forwarding path according to the link bandwidth to the root and not based on hop count. With this command, the cost of links is as follows (these numbers are weights so they don’t have a specific measurement unit):

One Gigabit Ethernet link = 20,000

Two Gigabit Ethernet links = 10,000

Three Gigabit Ethernet links = 6660

Four Gigabit Ethernet links = 5000

The cost with 10 Gigabit Ethernet links is as follows:

One 10 Gigabit Ethernet link = 2,000

Two 10 Gigabit Ethernet links = 1,000

Path cost method long should be enabled on every device participating in the spanning tree. This will be enabled for all devices being installed as part of the core LAN refresh, however all remaining switches such as the access layer switches remain the responsibility of the A CUSTOMER IT team to update.

Spanning-Tree Portfast

Portfast immediately brings an interface configured as an access or trunk port to the forwarding state from a blocking state, bypassing the listening and learning states. Portfast should be used on interfaces connected to a single workstation or server, to allow those devices to immediately connect to the network, rather than waiting for the spanning tree to converge (up to 50 seconds delay). NX-OS(config)# interface ethernet 2/1

NX-OS(config-if)# spanning-tree port type edge

OR

IOS(config)# interface GigabitEthernet 0/1

IOS(config-if)# spanning-tree portfast


NX-OS Spanning-Tree Port Types (Core /Aggregation)

In order to ease the administration of spanning-tree there are a number of port types that can be associated with network port based on their function in the design.

Normal ports - By default a switchport is considered a normal port for the purpose of spanning-tree. Normal ports remain

unmodified and operate as standard spanning-tree ports

Network ports - Network ports are used to define connections between two bridges. By default Bridge Assurance is enabled on

these ports, which is described in the next section.

Edge ports - Previously known as Portfast, a port configured as a spanning-tree edge should transition immediately into a

forwarding state, bypassing the listening and learning states. Only non-bridging L2 devices should be configured as edge ports.

This port type should be reserved for data centre hosts which are not capable of creating a L2 loop; this includes single

attached hosts, L3 routers and firewalls, or multi-homed devices that leverage some form of NIC teaming. Trunk ports to these

types of devices can also be configure as edge ports (this is called TrunkFast and generally used for virtualisation hosts).

It is recommended that all ports are left in the default configuration of ‘normal’ so that spanning-tree will protect the topology when needed, and explicit configuration is given for any ports that should be ‘network’ or ‘edge’. To configure port types the following configuration is used: N7K(config)# interface ethernet 2/11

N7K(config-if)# spanning-tree port type network

OR N7K(config)# interface ethernet 2/11

N7K(config-if)# spanning-tree port type edge

Warning: Edge port type (portfast) should only be enabled on ports connected to

a single host. Connecting hubs, concentrators, switches, bridges, etc...

to this interface when edge port type (portfast) is enabled, can cause temporary

bridging loops.

Use with CAUTION

OR N5K(config)# interface ethernet 2/11

N5K(config-if)# spanning-tree port type edge trunk

BPDU Guard

BPDU guard prevents a host port (port type edge or portfast) from participating in spanning tree. Under normal circumstances, Layer 2 access ports connected to a single workstation or server should not participate in spanning tree. When enabled on a port, BPDU guard shuts down the port as soon as a BPDU is received on that port. In this way, BPDU guard helps prevent unauthorized access and the illegal injection of forged BPDUs. BPDU guard can be configured per port, or globally. When configured globally, BPDU guard is effective only on ports in the operational port type ‘edge’ mode. Its recommended that global BPDU Guard be enabled across the A CUSTOMER infrastructure therefore affecting all ports that are placed in to ‘edge’ mode. Global BPDU Guard for edge or Portfast ports is enabled as follows: NX-OS(config)# spanning-tree port type edge bpduguard default

IOS(config)# spanning-tree portfast edge bpduguard default

IOS(config)# spanning-tree portfast bpduguard default

BPDU guard should also be enabled on individual interfaces that are in portfast or edge mode:

NX-OS(config)# interface E100/1/1


NX-OS(config-if)# spanning-tree bpduguard

STP Root Guard

STP Root Guard enforces the placement of the root bridge. STP root guard is a feature that is enabled on selected ports to prevent surrounding switches (usually downstream) from becoming the root switch. The root guard feature forces a port to become a designated port so that no switch on the other end of the link can become a root switch. If a port configured for root guard receives a superior BPDU, the port immediately goes into a root-inconsistent (blocked) state. In this way, STP root guard blocks other devices trying to become the root bridge by sending superior BPDUs. STP Root guard should be enabled on all downlinks away from the root bridge in each network layer, server access ports, and user access ports. Root guard can be enabled with the following command: switch(config)# interface ethernet 2/1

switch(config-if)# spanning-tree guard root

STP Loop Guard

STP Loop guard prevents alternate or root ports from becoming the designated port due to a failure that could lead to a unidirectional link. Loop guard operates only on ports that are considered point to point by spanning tree. Root guard is mutually exclusive with Loop guard, as Root guard is to be used on designated ports and does not allow the port to become non-designated. Loop guard works on non-designated ports and does not allow the port to become designated via max_age expiry. It’s recommend that Loop guard be enabled on root and alternate ports for all possible combinations of active topologies where Bridge Assurance is not supported. An example of this would be the 2960 switches. Configuration of loop guard shown below: switch(config)# interface GigabitEthernet1/0/1

switch(config-if)# spanning-tree guard loop

6.2.6 Link Aggregation

Where the aggregation of links is a requirement, PortChannel should be used to create a logical bundle. LACP should be used for the PortChannel management. A total of 8 physical ports maybe used as part of any single PortChannel configuration. LACP has the same characteristics as PAgP, which is Cisco proprietary. As PAgP and LACP are not interoperable, LACP will be the preferred negotiation protocol within A CUSTOMER. LACP will be set to active/passive mode and the medium type specified as point-to-point. The active device should always be closer towards the Core of the network with the other end as passive. Config Example: switch(conf-if)# channel-group 1 mode active

switch(conf-if)# channel-protocol lacp

Note: PortChannel active/active or active/passive will be preferred where it is supported When bundling links it’s important to ensure that they are all the same speed, their duplexing mode is full, and any configurations are mirrored across all ports. Additionally, PortChannel configurations should always be consist ports to the power of ‘2’ to provide the best load distribution.


6.3 Layer 3

6.3.1 HSRP and SVI interfaces

Hot-standby Routing Protocol (HSRP) provides high network availability and routes IP traffic from hosts on a particular LAN segment without relying on the availability of any single router. HSRP is configured across a number of routers (or routed interfaces) where the selection of an active and standby router takes place. The active and standby router is selected based upon the HSRP priority set on each device. When configuring priorities the best practice is to align the ‘active’ HSRP device to the primary root bridge or vPC primary switch for every HSRP instance. When specifying HSRP timers, In the A CUSTOMER design single supervisors will be used, however in the case that vPC dual supervisors are used in the future, the recommendation is to configure and extended hold timer on all devices to support NSF during ISSU and supervisor switchovers – sub second hellos should not be used. Values of 1s hello and 3s hold timers are ideal. Lastly ‘IP redirects’ should be disabled and MD5 authentication is recommended. Shown below is an example NX-OS configuration for HSRP on SVI interfaces:

Primary N5K-1(config)# feature hsrp

N5K-1(config)# feature interface-vlan

N5K-1(config)# hsrp timers extended-hold 15

N5K-1(config)# interface Vlan10



N5K-1(config-if)# no ip redirects

N5K-1(config-if)# hsrp version 2

N5K-1(config-if-hsrp)# hsrp 10

N5K-1(config-if-hsrp)# authentication md5 key-string <text string>

N5K-1(config-if-hsrp)# preempt delay minimum 30

N5K-1(config-if-hsrp)# priority 110

N5K-1(config-if-hsrp)# timers 1 3

N5K-1(config-if-hsrp)# ip <hsrp address>

Secondary N5K-2(config)# feature hsrp

N5K-2(config)# feature interface-vlan

N5K-2(config)# hsrp timers extended-hold 15

N5K-2(config)# interface Vlan10



N5K-2(config-if)# no ip redirects

N5K-2(config-if)# hsrp version 2

N5K-2(config-if-hsrp)# hsrp 10

N5K-2(config-if-hsrp)# authentication md5 key-string F1r$t_hop

N5K-2(config-if-hsrp)# preempt delay minimum 180

N5K-2(config-if-hsrp)# priority 105

N5K-2(config-if-hsrp)# timers 1 3

N5K-2(config-if-hsrp)# ip <hsrp address>

6.3.2 VLSM addressing using /31 Subnets for Point-to-Point links

31-bit prefixes as defined in RFC 3021 will provide twice as many subnets to be created out of the same block of addresses as using /30 addressing and is now considered best practice for all point-to-point link. In order to keep the new network design consistent with the existing network infrastructure, A CUSTOMER have requested that /30 subnets are kept in used for point-to-point links.


6.3.3 Bidirectional Forwarding Detection (BFD)

Overview

Bidirectional Forwarding Detection (BFD) is a network protocol used to detect faults between two forwarding engines connected by a link. It provides detection of faults for various physical media type, encapsulations, topologies, and routing protocols. BFD can enable sub-second failure detection between two adjacent devices without the need of configuring aggressive CPU intensive protocol hello messages, this is because some of the BFD load can be distributed onto the data plane with some I/O modules. The key to providing sub-second failover using BFD is that it talks directly to other network protocols and sends them a failure detection notice when network changes occur, this circumvents the delay usually experienced by the time it takes for line protocol to go down, or the dead intervals to expire. The current supported protocols and for BFD are:

BGP

EIGRP

OSPF

IS-IS

HSRP

PIM

Static Routes

Within the design, BFD should be used for all OSPF and PIM neighbours where Nexus and/or Catalyst 6500’s are directly connected together. HSRP is not included as there are number of instances that could cause the SVI to flap. In any case, the use of vPC reduces the need to enabled BFD on SVI interfaces as HSRP is active/active.

BFD Asynchronous Mode

NX-OS version 5.0 and higher supports BFD in Asynchronous mode. This sends BFD control packets between two adjacent devices to activate and maintain BFD neighbour sessions between them. You configure BFD on both devices (or BFD neighbours). Once BFD has been enabled on the interfaces and on the appropriate protocols, NX-OS creates a BFD session, negotiates BFD session parameters, and begins to send BFD control packets to each BFD neighbour at the negotiated interval.

BFD Echo Mode

BFD echo mode sends echo packets from the forwarding engine to the remote BFD neighbour. The neighbour then forwards the echo packet back along the same path in order to perform detection, the BFD neighbour does not participate in the actual forwarding of the echo packets. The echo function and the forwarding engine are responsible for the detection process.

You can configure BFD echo mode on one or both ends of a BFD monitored link. Echo mode slows down the required minimum receive interval, based on the configured slow timer. The RequiredMinEchoRx BFD session parameter is set to zero if echo mode is disabled. The slow timer becomes the required minimum receive interval if echo mode is enabled. Echo mode is enabled by default and is the recommended configuration. Note: Echo mode requires that the interface participating in BFD has the ‘no ip redirects’ command applied.


BFD Considerations

The following consideration need to be made when using BFD:

For Layer 3 PortChannels and Layer 2 PortChannels used by SVI interfaces, LACP needs to be enabled.

Initial BFD Configuration

Before you can enable the BFD you need to disable the ‘Address Identical’ IDS feature on NX-OS in the default VDC. You can then enable the BFD and set the global intervals per VDC (which can be overridden at the interface level). The recommended configuration values are shown below: Default VDC N7K(config)# no hardware ip verify address identical

Non-Default VDC N7K(config)# feature bfd

N7K(config)# bfd interval 100 min_rx 100 multiplier 4

N7K(config)# bfd slow-timer 3000

BFD OSPF Configuration

BFD can be enabled globally or you can configure the interaction between BFD and EIGRP per-interface for signalling. The configuration is shown below: N7K(config)# router ospf 1

N7K(config-router)# bfd

OR

N7K(config)# interface Ethernet 1/1

N7K(config-if)# no ip redirects

N7K(config-if)# ip ospf bfd

BFD PIM Configuration

Once BFD has been enabled globally you can then configure the interaction between BFD and PIM for signalling. The configuration is shown below: N7K(config)# ip pim bfd

N7K(config-router)# bfd

N7K(config)# interface Ethernet 1/1

N7K(config-if)# no ip redirects

N7K(config-if)# ip pim bfd-instance For other PIM configurations please refer to the Multicast Design section.


6.4 IP Services

6.4.1 DHCP Relay

DHCP relay enables and interface to forward broadcasted DHCP BOOTREQUEST packets off to a remote DHCP server as a unicast packet, the DHCP would then respond back to the interface and the DHCPOFFER would be broadcasted onto the segment for the client. In IOS this feature is referred to a helper address. The configuration in NX-OS is the much the same with slightly different syntax. An example is shown below: N7K(config)# service dhcp

N7K(config)# interface Vlan100

N7K(config-if)# ip dhcp relay address 192.168.1.100

IOS(config)# service dhcp

IOS(config)# interface Vlan100

IOS(config-if)# ip helper-address address 192.168.1.100

6.5 Catalyst 4500-X Specifics

6.5.1 High Availability through SSO and NSF

The access areas will experience rapid failover performance through the implement of Stateful Switchover (SSO) and Non Stop forwarding (NSF). Cisco Catalyst 6500 series switches support fault resistance by allowing a redundant supervisor engine to take over if the primary supervisor engine fails. Cisco NSF works with SSO to minimize the amount of time a network is unavailable to its users following a switchover while continuing to forward IP packets. The following events cause a switchover:

A hardware failure on the active supervisor engine

Clock synchronisation failure between supervisor engines

A manual switchover

SSO

SSO establishes one of the supervisor engines as active while the other supervisor engine is designated as standby, and then SSO synchronizes information between them. A switchover from the active to the redundant supervisor engine occurs when the active supervisor engine fails, or is removed from the switch, or is manually shut down for maintenance. This type of switchover ensures that Layer 2 traffic is not interrupted. In networking devices running SSO, both supervisor engines must be running the same configuration so that the redundant supervisor engine is always ready to assume control following a fault on the active supervisor engine. SSO switchover also preserves FIB and adjacency entries and can forward Layer 3 traffic after a switchover. Configuration information and data structures are synchronized from the active to the redundant supervisor engine at startup and whenever changes to the active supervisor engine configuration occur. Following an initial synchronization between the two supervisor engines, SSO maintains state information between them, including forwarding information. During switchover, system control and routing protocol execution is transferred from the active supervisor engine to the redundant supervisor engine. The switch requires between 0 and 3 seconds to switchover from the active to the redundant supervisor engine. The configuration is shown in the example below.

Switch(config)# power redundancy-mode redundant


NSF

Cisco NSF always runs in conjunction with SSO and provides redundancy for Layer 3 traffic. NSF works with SSO to minimize the amount of time that a network is unavailable to its users following a switchover. The main purpose of NSF is to continue forwarding IP packets following a supervisor engine switchover. Cisco NSF is supported by the OSPF protocol for routing and is supported by Cisco Express Forwarding (CEF) for forwarding. OSPF has been enhanced with NSF-capability and awareness, which means that routers running these protocols can detect a switchover and take the necessary actions to continue forwarding network traffic and to recover route information from the peer devices. It’s also known as graceful restart. A networking device is NSF-aware if it is running NSF-compatible software. A device is NSF-capable if it has been configured to support NSF; it will rebuild routing information from NSF-aware or NSF-capable neighbours. Each protocol depends on CEF to continue forwarding packets during switchover while the routing protocols rebuild the Routing Information Base (RIB) tables. After the routing protocols have converged, CEF updates the FIB table and removes stale route entries. CEF then updates the line cards with the new FIB information. Config Example: router(conf)# router ospf <process>

router(config-router)# nsf

Note: NSF is not supported on IP base switching software 6.6 Miscellaneous Diagrams

6.6.1 Spanning Tree & FHRP

Below is a diagram that shows the recommended configuration for STP tools, STP port types and L2/L3 alignment with respect to vPC, HSRP, and STP:

Figure 4 - Spanning Tree & FHRP Diagram

Prepared for Client on 27 April 2015


Sample lld document v1.0

Technology

Transcript of Sample lld document v1.0