1 © Copyright 2013 EMC Corporation. All rights reserved.
Twitter: @drjmetz
Designing High Availability in a Cisco Converged Networking Environment
J Metz, Ph.D Product Manager, Storage, Cisco Systems
Co-Sponsored by Intel®
2 © Copyright 2013 EMC Corporation. All rights reserved.
Agenda
• High Availability
– How we got here
– For FC, for Ethernet, and for both
– Isolation and Redundancy
• HA Topologies
– Access-Layer
– End-To-End
– Future HA Topologies?
3 © Copyright 2013 EMC Corporation. All rights reserved.
What is High-Availability (HA)?
4 © Copyright 2013 EMC Corporation. All rights reserved.
What is "HA"?
• Redundancy
5 © Copyright 2013 EMC Corporation. All rights reserved.
What is "HA"?
• Redundancy
• No Single Points of
Failure
– Control and
Switching Elements
have redundant
components
6 © Copyright 2013 EMC Corporation. All rights reserved.
Why HA?
• Switches were small (8-16
ports) and not very powerful
• Easily have an outage
• Switch can go down
FC
SAN
(not drawn to scale)
7 © Copyright 2013 EMC Corporation. All rights reserved.
Why HA?
• Meshes were created for adding
ports and redundancy
• Many of the ports (up to half)
were being used for ISLs
SAN
FC
(not drawn to scale)
8 © Copyright 2013 EMC Corporation. All rights reserved.
Why HA?
• Moved to high-density, highly-
available Director Class systems
• 80% of storage environments use
Director Class Fibre Channel
Switches
(not drawn to scale)
SAN
FC
9 © Copyright 2013 EMC Corporation. All rights reserved.
Why HA?
• Add separate Fabric
• SAN A/B introduced for redundancy
purposes
• Still have an outage if change control
procedures are not followed
(not drawn to scale)
SAN A SAN B
FC
10 © Copyright 2013 EMC Corporation. All rights reserved.
Best Practices for High-Availability
Ethernet and Fibre Channel: Common Elements
Fibre Channel HA
Ethernet Storage HA
11 © Copyright 2013 EMC Corporation. All rights reserved.
Best Practices for HA
• There are elements common to
both FC and Ethernet
– Director Class
• Redundant control plane,
switching, power, and cooling
– No single spoke network
topologies
• There are specific needs to
both Ethernet and FC
12 © Copyright 2013 EMC Corporation. All rights reserved.
Network and Fabric
• Ethernet/IP
– Goal: Provide any-to-any connectivity
• Unaware of packet loss (“lossy”) – relies on
Upper Layer Protocols (ULPs) for retransmission
and windowing
• Provides the transport without worrying about the
services –
– Services provided by upper layers
• East-west vs. north-south traffic ratios are
undefined
• Network design has been optimized for
– High Availability from a transport perspective
by connecting nodes in mesh architectures
– Service HA is implemented separately
– Takes into account control protocol interaction
(STP, OSPF, EIGRP, L2/L3 boundary, etc…)
?
?
?
?
? ? ?
?
?
?
? ?
Switch Switch
Switch
?
Client/Server
relationships are not
pre-defined
? ?
?
Fabric topology and traffic flows are highly flexible
13 © Copyright 2013 EMC Corporation. All rights reserved.
Network and Fabric
• Servers typically dual homed to two or more
access switches
• Redundant connections to the next layer
• Distribution and Core can be collapsed into
a single box
• L2/L3 boundary typically deployed in the
aggregation layer
– Spanning tree or advanced L2 technologies
(e.g., virtual link aggregation) used to prevent
loops within the L2 boundary
• Services deployed in the L2/L3 boundary of
the network (load-balancing, firewall, etc.)
L2
L3
Core
Aggregation
Access
Outside Data Center “cloud”
STP
STP
14 © Copyright 2013 EMC Corporation. All rights reserved.
Network and Fabric
• Fibre Channel SAN
– Transport and Services are on the same layer
in the same devices
– Well-defined end device relationships (initiators
and targets)
– Does not tolerate packet drop – requires
lossless transport
– Only north-south traffic, east-west traffic mostly
irrelevant
• Network designs optimized for scale and
availability
– High availability of network services provided
through dual fabric architecture
– Edge/Core vs Edge/Core/Edge
– Service deployment Client/Server
Relationships are pre-defined
I(c)
I(c) T(s)
Fabric topology, services, and traffic flows are structured
T2
I5
I4 I3 I2
I1
I0
T1 T0
Switch Switch
Switch
DNS FSPF
Zone RSCN DNS
FSPF Zone
RSCN
DNS
Zone
FSPF
RSCN
15 © Copyright 2013 EMC Corporation. All rights reserved.
Network and Fabric
• “Edge-Core” or “Edge-Core-Edge”
Topology
• Servers connect to the edge switches
• Storage devices connect to one or more core switches
• HA achieved in two physically separate, but identical,
redundant SAN fabric
• Very low oversubscription in the fabric (1:1 to 12:1)
• FLOGI scaling considerations
Fabric ‘A’ Fabric ‘B’
HBA
FC
Core
Edge
16 © Copyright 2013 EMC Corporation. All rights reserved.
Best Practices for HA
• FC Specific
– Air gap redundant fabrics
– Multipathing software resident on host
– Co-location of hosts and storage wherever possible
17 © Copyright 2013 EMC Corporation. All rights reserved.
Co-Location • What is it?
– Ideally, keeping hosts and storage on
same switch
– "Zero-hop" topologies
• Limits
– Scaling
– In reality, may not be possible
• Physical location can be a limitation
• ISLs for distance extension
• Keep in mind when you lose a link or a
switch, the other fabric has to be able to
pick up that slack
– Design that into the system
18 © Copyright 2013 EMC Corporation. All rights reserved.
Best Practices for HA
• FC specific - Scale – VE_Ports (ISLs) allows
for greater scalability
– Can still have mesh
fabric
– Can still maintain HA
19 © Copyright 2013 EMC Corporation. All rights reserved.
Best Practices for HA
• HA Elements common to
both FC and Ethernet
– Single Spoke (non-HA)
20 © Copyright 2013 EMC Corporation. All rights reserved.
Best Practices for HA
• HA Elements common to both
FC and Ethernet
– Redundant connectivity (HA)
21 © Copyright 2013 EMC Corporation. All rights reserved.
Best Practices for HA
• Ethernet-specific (not used for block I/O)
– NAS (NFS/CIFS)
– Dual attached to the same network segment
– Active/Active teaming
– Smaller L2 domains
22 © Copyright 2013 EMC Corporation. All rights reserved.
Best Practices for HA
• iSCSI specific (used for block I/O)
– Dual Attached to DIFFERENT Network
segments*
– No teaming
– Avoid routers if possible*
* Best Practice
23 © Copyright 2013 EMC Corporation. All rights reserved.
Oh, by the way... Server virtualization
Server virtualization considerations: How do you manage virtual switches?
Where is the switching performed Virtual switch, VEB, VEPA / VN_TAG Extending the L2 Domain over L3 constructs
Network virtualization Open vSwitch, VXLAN, etc. Bottom Line
If you have a redundant network, your virtual networks will also be redundant
Server and network virtualization will probably not have a big impact on basic HA network design
24 © Copyright 2013 EMC Corporation. All rights reserved.
Isolation and Redundancy
$
$
Less Sharing
More Sharing
Failure Tolerance
25 © Copyright 2013 EMC Corporation. All rights reserved.
Isolation and Redundancy
LAN SAN
26 © Copyright 2013 EMC Corporation. All rights reserved.
From Ethernet POV
• How can you maintain Ethernet best practices of HA . . .
– NIC teaming
– Multichassis trunking
Ethernet Fabric
FC
FC
27 © Copyright 2013 EMC Corporation. All rights reserved.
From a Storage POV
• . . . while still maintaining FC best practices
of HA?
– Isolated fabrics
– Multipathing
Ethernet Fabric
FC
FC
28 © Copyright 2013 EMC Corporation. All rights reserved.
High Availability Topologies
29 © Copyright 2013 EMC Corporation. All rights reserved.
Application Tiers
• How would you categorize your applications tiers today?
– By application type
– By server-to-server and/or server-storage bandwidth
• What is the growth you see for application bandwidth?
• Would you want different topologies / deployment models based on application type?
• What is the effect of server virtualization to the above questions?
Separate LAN / SAN SAN A / B w/ Converged
No SAN A / B
30 © Copyright 2013 EMC Corporation. All rights reserved.
Topology 1: No Convergence
• Same topologies as existing
networks, but using Ethernet
switches for SANs
• Physical and Logical separation of
LAN and SAN traffic
• Additional Physical and Logical
separation of SAN fabrics
Example Use Cases: - Outsourced networks - Compliance requirements
HBA/CNA
L2
L3
NIC or CNA
Fabric ‘A’ Fabric ‘B’
FCoE
Value
SAN can utilize higher performance, higher density, lower cost Ethernet switches
Native Ethernet LAN Fibre Channel/ Fibre Channel over Ethernet SAN
Core
Aggregation
Access
Core
Edge
Isolation Convergence
31 © Copyright 2013 EMC Corporation. All rights reserved.
Topology 2 (Converged at ToR)
• Consolidated the LAN Access and the SAN
Edge by using FCoE
• Physical and Logical separation
– LAN and SAN traffic at Aggregation Layer
– Additional Physical and Logical separation of
SAN fabrics
• Higher I/O, HA, fast re-convergence for host
LAN traffic
• The Unified Edge supports multiple LAN and
SAN topology options
– Virtualized Data Center LAN designs
– Fibre Channel edge with direct attached
initiators and targets
– Fibre Channel edge-core and edge-core-edge
designs
Converged
FCoE link
Dedicated
FCoE link
FC
Ethernet
Fabric ‘B’
L2
L3
CNA
Fabric ‘A’
FC FCoE
Isolation Convergence
FC/FCoE Switch
32 © Copyright 2013 EMC Corporation. All rights reserved.
Trunking and Channeling
• Switches operating in N_Port Virtualization mode(End-
Host mode for UCS) or FC-SW (Switch Mode for
UCS) Mode
• F-Port Trunking and Channeling on the links between
an NPV device and upstream switch (NP port ->
F_Port)
• F_Port Trunking: Better multiplexing of traffic using
shared links (multiple VSANs on a common link)
• F_Port Channeling: Better resiliency between NPV
edge and Director Core
– No host re-login needed per link failure
– No FSPF recalculation due to link failure
• Simplifies FC topology (single uplink from NPV device
to FC director
Fabric ‘A’ Supporting VSAN 20 & 40
VSAN 20, 40
Fabric ‘B’ Supporting VSAN 30 & 50
VF
VN
TF
TNP
Server ‘1’ VSAN 20 & 30
Server ‘2’ VSAN 40 & 50
NPV TOR Switch
VLAN 10,50 VLAN 10,30
Isolation Convergence
VSAN 30,50
VLAN 10,20
VLAN 10,40
FC/FCoE Switch
With Intel® Xeon® processor
33 © Copyright 2013 EMC Corporation. All rights reserved.
Trunking/Channeling with UCS • More flexibility in engineering FC traffic vs. 1 VSAN per uplink
– Aggregate Uplinks transparent to host Multi-path drivers
– Requires EMC Connectrix MDS or N5K to Work (both
features)
• Provide isolation to SAN traffic over the same physical link
– Help consolidate Infrastructure
– vHBAs can be on different VSANs
• All VSANs will be trunked on every uplink FC/FCoE port
– Selecting a subset of VSANs for individual uplink ports not
supported
• Scalability: Max of 32 VSANs per UCS system
• VSAN trunking supported in NPV and FC Switch mode FI
operation
• VSAN Trunking is not available for direct connect FC/FCoE
Storage Port types
vFCs VSAN 100
VSAN 300
VSAN 200
VSAN 400
SAN A SAN B
With Intel® Xeon® processor
34 © Copyright 2013 EMC Corporation. All rights reserved.
Topology 3 (Fabric/Completely Converged)
• LAN and SAN traffic share physical
switches and traffic uses dedicated
links between switches
• All Access and Aggregation
switches are FCoE FCF switches
• Improved HA, load sharing and
scale for LAN vs. traditional STP
topologies
VE
Fabric ‘B’
LAN/SAN
Converged
FCoE link Dedicated
FCoE link
FC
Ethernet
Fabric ‘A’
Isolation Convergence
36 © Copyright 2013 EMC Corporation. All rights reserved.
UCS iSCSI and Appliance Port Redundancy
• Host Multi-pathing drivers are used in lieu
of link aggregation network technology
• MS does not support using s/w iSCSI and
port channels for iSCSI failover
• Best practice is to use MPIO drivers
• Failure semantics look like FC in this
regard
UCS B-Series
UCS FI UCS FI
Storage
FCoE iSCSI NFS CIFS
Unified Appliance Port
With Intel® Xeon® processor
37 © Copyright 2013 EMC Corporation. All rights reserved.
Summary
• Highly Available network topologies
continue to be a requirement for today’s
data center environments
• Converging all protocols onto a single set
of physical links while maintaining isolation
can be managed
• Several converged topologies are
available that allow for various degrees of
isolation
Visit Cisco Booth 401
• Twitter: @ciscoDC
• Facebook.com/CiscoDC
• Video: http://www.youtubecisco.com/datacenter
• Cisco blog: http://blogs.cisco.com/datacenter
• Slideshare: http://slideshare.com/CiscoDataCenter
In Collaboration with Intel®
Intel, the Intel logo, Xeon and Xeon inside are trademarks of Intel Corporation in the U.S. and other countries.
Top Related