ITALIAN AND GERMAN UNIFICATION 1852-1871. ITALIAN UNIFICATION PROCESS.
EDUCATION Unification of Data Center Fabrics with Ethernet
Transcript of EDUCATION Unification of Data Center Fabrics with Ethernet
EDUCATION
Unification of Data Center Fabrics with Ethernet“A High Speed Peek into the Future!”
Matthew Brisse, Dell Inc.
EDUCATION
Unification of Data Center Fabrics with Ethernet: “A High Speed Peek into the Future!”© 2007 Storage Networking Industry Association. All Rights Reserved. 2
SNIA Legal Notice
• The material contained in this tutorial is copyrighted by the SNIA.
• Member companies and individuals may use this material in presentations and literature under the following conditions:– Any slide or slides used must be reproduced without
modification– The SNIA must be acknowledged as source of any
material used in the body of any document containing material from these presentations.
• This presentation is a project of the SNIA Education Committee.
EDUCATION
Unification of Data Center Fabrics with Ethernet: “A High Speed Peek into the Future!”© 2007 Storage Networking Industry Association. All Rights Reserved. 3
Abstract
Unification of Data Center Fabrics with Ethernet: A High Speed Peek into the Future!
– A key challenge that the network and storage industries are now facing is the abundance of new high speed interconnect protocols proposed for future data center applications. In this presentation we take a peek into what the future may hold for high speed fabrics and investigate the potential for their unification. We will provide a market and technical overview of the competitive landscape for next generation 10Gb technologies with particular focus on the operational characteristics and implementation aspects of Ethernet.
EDUCATION
Unification of Data Center Fabrics with Ethernet: “A High Speed Peek into the Future!”© 2007 Storage Networking Industry Association. All Rights Reserved. 4
10GE Technology
• 10GE is 10 Gbps Ethernet– It has four different types of physical layer interfaces
• 1m: Backplane (802.3ap), XAUI (KX4) – 2.65GB/lane, 4 lane = 10GBKR (Serial) – 10 GB 1 lane
• 100m to 10km: Optical (SR, LR, LRM, ER)• 15m: CX-4 (Infiniband Type Cable-XAUI)• 55-100m 10GBase-T Copper (802.3an), over CAT6, CAT6a
or CAT7– Offload MUST be a part of any 10GE solution
• A “dumb-only” 10GE NIC is of limited value– 10GE readily consumes compute and memory resources (unless
offloaded)– Less latency and greater bandwidth (both are important)
• 10GE is more than just another, interchangeable physical interface– Offload and system characteristics are key to plumbing 10GE properly
EDUCATION
Unification of Data Center Fabrics with Ethernet: “A High Speed Peek into the Future!”© 2007 Storage Networking Industry Association. All Rights Reserved. 5
Copper Cable Options
CAT52 twisted pairs
CAT5e4 twisted pairs
CAT6Longitudinal spacer
CAT6aLongitudinal spiral spacer
CAT7Longitudinal spacer
Wrapped pairs, Shielded
Alien Crosstalk
Category Type Spectral B/W
Length Applications
Notes
Cat5 UTP 100MHz 100m 100Base-T Common
Cat5e UTP 100MHz 100m 1000Base-T Common
Cat6 UTP 250MHz 55m 10GBase-T Emerging
Cat6a UTP 400MHz 100m 10GBase-T Emerging
Cat7 ScTP 600MHz 100m 10GBase-T Emerging
(55m)(100m)(100m)
(100m)
$0.10/ft$0.15/ft$0.65/ft $0.05/ft$1.25/ft
? RJ45 RJ45 RJ45 RJ45
1GE
10GE
SHIELDED UNSHIELDED
RJ45 Challenge!
EDUCATION
Unification of Data Center Fabrics with Ethernet: “A High Speed Peek into the Future!”© 2007 Storage Networking Industry Association. All Rights Reserved. 6
40 Gbps, bi-directional datarate
PCIe and 10GE
16 Gbps, effective bi-directional throughput
10GE
20Gbps bi-directional
PCIe, Gen1 x8
Server
80 Gbps, bi-directional datarate
32 Gbps, effective bi-directional throughput
10GE
20Gbps bi-directional
PCIe, Gen2 x8
Server
PCI Express Generation 2 will be required to support full 10GE with x8(Current 10GE NICs can’t realize full throughput due to PCIe Gen 1 bus speed limitations but PCIe Gen 2
will realize the maximum throughput)
Datarate: Bits on the wireThroughput: Information on the wire
EDUCATION
Unification of Data Center Fabrics with Ethernet: “A High Speed Peek into the Future!”© 2007 Storage Networking Industry Association. All Rights Reserved. 7
10 GE Requires Multi-Function Offload Capabilities
• Simple NICs offload nothing• Stateless offload accelerates packet processing (RSS, LSO, TSO, Checksum)• Stateful offload provides protocol and packet processing (TOE, Chimney, RDMA, iWARP)• RDMA abstracts the underlying transport and achieves direct memory to memory data placement • Virtualization mediates access to multiple instantiations of limited hardware resources
NIC
TOE
RDMA (iWARP)RDMA over TCP/IP
VIRTUALIZATION
ACCELERATION
Lower CPU usageGreater I/O efficienciesHigher network throughput
10 GE Viability
1 GE Viability
Now 2007
ProtocolAcceleration
Check SumOff Load
No
Off
Load
EDUCATION
Unification of Data Center Fabrics with Ethernet: “A High Speed Peek into the Future!”© 2007 Storage Networking Industry Association. All Rights Reserved. 8
• 10GE is an acceptable technology for fabric convergence• IB and 10GE bandwidth both have visibility to 100GE• FC will continue to have a place for a considerable time• 10GE latency will settle at < 10 microseconds end-to-end• Cost will come down – Economy of scale
10GE Overall Technology Comparison
EN
FC
IBSPEED LATENCY(Gbps)
(µsec)
108
15
10
6
400
300
400
COST
($)
16
POWER
5
4 4UNIFIED FABRIC
SWEET SPOT
(End-to-End)
Fastest, less latency, cluster + storage “was” the fabric convergence in the past
(W)(NIC/HBA) (SR MAC/PHY)
Fast, some latency good enough
Faster, less latency, storage
EDUCATION
Unification of Data Center Fabrics with Ethernet: “A High Speed Peek into the Future!”© 2007 Storage Networking Industry Association. All Rights Reserved. 9
10GE non-Technology Comparison• FC and IB will both perform “better” than 10GE in cluster and storage fabrics,
respectively• 10GE will, however, be “good enough” to potentially gain market share from IB
and FC• Non technical factors will drive 10 GE adoption and continued evolution
– Ethernet trust, ubiquity and robust investment climate are important driving factors• Ethernet always incorporates the best from competing technologies
– EtherNOT becomes EtherNET
TRUST
LO
HIHI
UBIQUITY
LO
MED
HI
INVESTMENT
LO
MED
HI
Well-known protocolsWell-known management
“Staying power”
Ethernet everywhere More $ to EthernetMore Ethernet R&D
EN
FC
IB
EDUCATION
Unification of Data Center Fabrics with Ethernet: “A High Speed Peek into the Future!”© 2007 Storage Networking Industry Association. All Rights Reserved. 10
Unified Fabric “A peek in the future”
• Reduce the number of disparate networks and Data Center complexity
• Provide a seamless model for networking, clustering and storage applications– Not to mention thin client, KVM and other usages
• Consolidate Management protocols over a common IP framework• Leverage economies of scale in terms of silicon integration,
interoperability and product development• Provide support for legacy network architectures in the transition to a
unified fabric architecture
EDUCATION
Unification of Data Center Fabrics with Ethernet: “A High Speed Peek into the Future!”© 2007 Storage Networking Industry Association. All Rights Reserved. 11
Unified Fabric VisionMaking Many Fabrics……………
IBEN
FC
COMPATIBLE SCALABLE
HIGH TRANSACTION RATE
HIGH
THROUGHPUTLOW
LATENCY
BLOCK
TRANSFERS
LOW CPU
UTILIZATION
HPCLAN
SAN
NAS
EDUCATION
Unification of Data Center Fabrics with Ethernet: “A High Speed Peek into the Future!”© 2007 Storage Networking Industry Association. All Rights Reserved. 12
10GE
Making Many Fabrics……………
Simplifying
COMPATIBLE SCALABLE
HIGH TRANSACTION RATE
HIGH
THROUGHPUT
LOW
LATENCY
BLOCK
TRANSFERS
LOW CPU
UTILIZATION
HPC
LAN
SAN
NAS
OneUnified Fabric Vision
EDUCATION
Unification of Data Center Fabrics with Ethernet: “A High Speed Peek into the Future!”© 2007 Storage Networking Industry Association. All Rights Reserved. 13
Emerging Trends• Clustering
– Improves cost-performance over large scale cache coherent systems– Brings flexibility to the data center– Provides better fault tolerance
• Grid– Academic processing Grids tried and tested– Commercial deployment in niche areas only– Storage Grids now in discussion
• Object Storage– Completing standardisation– New paradigm of intelligent storage– Requires converged compute & storage platform
• Systems Management– Keyboard Video & Mouse (KVM) already available and other tools are
coming• System Level Virtualization• Ethernet is rapidly becoming the key to enterprise systems convergence
EDUCATION
Unification of Data Center Fabrics with Ethernet: “A High Speed Peek into the Future!”© 2007 Storage Networking Industry Association. All Rights Reserved. 14
Industry Initiatives
• Blurring of compute and storage is now happening– Traditional view of totally discrete functions is getting old– New applications require dynamic configurations– Lightweight blades clustered with shared access to I/O resources are
enabling convergence
• Systems Virtualization – Protected virtual memory architectures return to give scale up
virtualisation– Scale out virtualization still requires major developments in the industry
but everyone expects it to happen
• Server re-purposing paradigm maturing – The 3 tier Data center model is now passing its sell by date
• Evolving Ethernet enables system wide convergence
EDUCATION
Unification of Data Center Fabrics with Ethernet: “A High Speed Peek into the Future!”© 2007 Storage Networking Industry Association. All Rights Reserved. 15
Target 10GE Applications
• Servers– HPC
• server-to-server interconnect– Virtualization
• increased bandwidth– High-Performance Transactions
• financial applications• IPTV streaming media
applications• Digital Cinema• web farms
• Switches– Edge Layer
• server-to-switch uplink• iSCSI storage array to switch
– Aggregation & Core Layer• switch-to-switch interconnect
• iSCSI Storage Arrays– General Purpose
• array-to-switch connectivity
• Single Adapter• Multiple Offloads• Common
Management• Simplicity
10GE provides the basis for a unified datacenter fabric
10GE UNIFIED FABRIC
LAN
HPC
SAN
EDUCATION
Unification of Data Center Fabrics with Ethernet: “A High Speed Peek into the Future!”© 2007 Storage Networking Industry Association. All Rights Reserved. 16
Enterprise Network Today
DatabaseClients Application ServersWeb Servers
SwitchesLoad
Balancers
IP/Ethernet Storage Cluster Management
StorageArray Units
SAN
Cluster
DAS
DAS
RAID Storage
RAID Storage
Enterprise Network Today
EDUCATION
Unification of Data Center Fabrics with Ethernet: “A High Speed Peek into the Future!”© 2007 Storage Networking Industry Association. All Rights Reserved. 17
Converged NetworkDatabaseClients Application ServersWeb Servers
SwitchesLoad
Balancers
IP/Ethernet
StorageArray Units
Storage Cluster Management
Cluster
SANDAS
DAS
RAID Storage
RAID Storage
SAN
Converged Network
EDUCATION
Unification of Data Center Fabrics with Ethernet: “A High Speed Peek into the Future!”© 2007 Storage Networking Industry Association. All Rights Reserved. 18
Backup Networking SwitchEthernetEthernet
Server Blade
NetworkSwitch
StorageSwitch
ClusterSwitch
ManagementSwitch
CPU
Compute Blades
Mid-plane
Backup Storage Switch
Backup Clustering Switch
Management Switch
Clustering Switch
Storage Switch
Local Hard Disk
Unified Fabric Example –Server Blades
E-net IB
FCMGT
Networking SwitchEthernetEthernet
EDUCATION
Unification of Data Center Fabrics with Ethernet: “A High Speed Peek into the Future!”© 2007 Storage Networking Industry Association. All Rights Reserved. 19
Ethernet Ethernet
CPU
Compute Blades
Server Blade
NetworkSwitch
StorageSwitch
ClusterSwitch
ManagementSwitch
Mid-plane
iSCSI
New Ethernet Technologies
Remote Direct Memory Access
(RDMA)
Integrated Management
Accelerated Networking (TOE)
Local Hard Disk
Unified Fabric Example –Server Blades
MGT
E-net
EthernetEthernet
IB
FC
EDUCATION
Unification of Data Center Fabrics with Ethernet: “A High Speed Peek into the Future!”© 2007 Storage Networking Industry Association. All Rights Reserved. 20
10GE Switch Market Growth
• 10GE growth (2008-09) driven by adoption of low-cost 10GBASE-T copper• 1GE will be pushed to the edge of LANs and must be aggregated with 10GE• 10GE will be initially positioned for inter-switch connections and clusters• Vendors drive low-cost 10GE, accelerating adoption resulting from customer demand
CY03 CY04 CY05 CY06 CY07 CY08 CY090
5
10
15
20
Rev
enue
($B
)
100 Mbps
1 Gbps
10 Gbps
Compound Annual Growth RateCY03 – CY09
-30%
10%
55%
Source: IDC 2005
EDUCATION
Unification of Data Center Fabrics with Ethernet: “A High Speed Peek into the Future!”© 2007 Storage Networking Industry Association. All Rights Reserved. 21
10GE Market
Available market data represents growth in the 10GE switch port market, not the NIC or other 10GE markets. The growth in 10GE infrastructure, however, implies growth in other 10GE markets, including server port shipments.
Source: Dell’Oro Group January 2006
000’s switch ports 2004 2005 2006 2007 2008 2009 2010
Copper Fiber Total % Copper
2556961
57 785
58
163
169
1% 4%
854
8%
1,616
1,871
14%
1,013
2,511
3,524
29%
2,509
4,008
6,517
39%
4,209
5,748
9,958
42%
5x port growth from 2005 to 2006; 10m ports, $4B market in 2010
0
1000
2000
3000
4000
5000
6000
2004 2005 2006 2007 2008 2009 2010
Fiber
Copper
2008 – Year of copper10W today, 5W power ’082-3W down on the Motherboard
EDUCATION
Unification of Data Center Fabrics with Ethernet: “A High Speed Peek into the Future!”© 2007 Storage Networking Industry Association. All Rights Reserved. 22
Cluster Fabric Trends
INFINIBAND
MYRINET
OTHER
ETHERNET1 GE 1 GE
I0GE
Top500.org, Nov 2005 Projection, 2008
EDUCATION
Unification of Data Center Fabrics with Ethernet: “A High Speed Peek into the Future!”© 2007 Storage Networking Industry Association. All Rights Reserved. 23
NIC 10GE Cost Trends
2005 2006 2007 20080
2x
4x
6x
8x
10x
Rel
ativ
e C
ost (
1GE
Cy0
4 =
1.0)
1GE vs. 10GE RELATIVE PRICE TARGET
10 GE 3-4x 1 GE for wide adoption
2004
10GE
1 GE
~$100/port 10GE NICs
EDUCATION
Unification of Data Center Fabrics with Ethernet: “A High Speed Peek into the Future!”© 2007 Storage Networking Industry Association. All Rights Reserved. 24
10GbE Standards
Media Access Controller (MAC) Full-Duplex
10 Gigabit Media Independent Interface (XGMII) or10 Gigabit Attachment Unit Interface (XAUI)
Serial LAN PHY(64B/66B)
Serial WAN PHY(64B/66B + WIS)
LAN PHY(8B/10B)
Various Optical Technologies
802.3ae
10GBaseT
10GBackplane
Cat6/7 Cable
802.3an 802.3ap
Layer 2
802.3an and 802.3ap make 10GbE even more compelling for data center applications
EDUCATION
Unification of Data Center Fabrics with Ethernet: “A High Speed Peek into the Future!”© 2007 Storage Networking Industry Association. All Rights Reserved. 25
10GE Copper/Backplane Standards and Opportunity Timeline
CY05 CY06 CY07 CY08 CY09 CY10
802.3an ratified
(10GBASE-T)
MATURE PRODUCTS
EMERGING SOLUTIONS
CONCEPT and DESIGN
BARRIERS TO OVERCOME
SWITCH
NIC
BLADEBLADE
LOM
802.3ap ratified
(Backplane)
EDUCATION
Unification of Data Center Fabrics with Ethernet: “A High Speed Peek into the Future!”© 2007 Storage Networking Industry Association. All Rights Reserved. 26
Changes with 10GbE
• 802.3ae defines 10GbE– 10Gbps Data Rates– Full-duplex only; no need for carrier-sensing multi-
access / collision detection (CSMA/CD)– Optical Physical Layer
• LAN PHY and WAN PHY options– WAN PHY compatible with SONET
• 802.3an adds twisted pair cabling– Cat 6 and Cat 7
• 802.3ap adds backplane specifications– Blade servers and communications equipment
EDUCATION
Unification of Data Center Fabrics with Ethernet: “A High Speed Peek into the Future!”© 2007 Storage Networking Industry Association. All Rights Reserved. 27
IEEE 10GE Standards
• IEEE P802.3ae, 10GE Working Group– Definition of the 10GE MAC architecture and components
• IEEE P802.3an, 10GBASE-T Task Force.– Definition of 10GE encoding over CAT 6/7 (Copper) cable
• IEEE P802.3ap, Backplane Ethernet Task Force.– Definition of 1/10GE encoding over backplanes (KX, KX4, KR)
• IEEE P802.3aq, 10GBASE-LRM Task Force.– Definition of short haul MMF/1310nm LR encoding
• IEEE P802.3ar, Congestion Management Task Force.– Definition of L2 congestion discovery and avoidance protocols
• IEEE P802.3as, Frame Expansion Task Force.– Not “Jumbo” frames; expansion of frame envelope, not data size
• IEEE P802.3at, DTE Power Enhancements Task Force.– Definition of how to deliver power at 40-60W over 1 and 10GE
EDUCATION
Unification of Data Center Fabrics with Ethernet: “A High Speed Peek into the Future!”© 2007 Storage Networking Industry Association. All Rights Reserved. 28
Unified Fabric End Goal
• STANDARDS• SIMPLICITY• PARTNERS• COORDINATION• CONSOLIDATION• SNIA Standards
LEADERSHIP
UNIFIED FABRIC10-Gigabit Ethernet
HardwareNIC/HBA
PHY
MACSWITCH
VIO
StandardsIEEE
IETF
DMTF
SoftwareOS
APPVOS
MGMT
SECURITY
InnovationSIMPLE
EFFICIENT
SECURE SNIA
EDUCATION
Unification of Data Center Fabrics with Ethernet: “A High Speed Peek into the Future!”© 2007 Storage Networking Industry Association. All Rights Reserved. 29
Call to Action
• Get involved!! – Participate in Standards– Drive your vendors– Leverage the power of the user
• Ethernet vendors are developing new protocol acceleration features and the speed is increasing from 1G to 10G, Understand the future impact
• 10Gb Ethernet with protocol acceleration could be used a 'Unified fabric' to handle SAN, HPC, management and client network needs in addition to traditional data networking requirements.
• Be prepared for the transition! Plan today!
EDUCATION
Unification of Data Center Fabrics with Ethernet: “A High Speed Peek into the Future!”© 2007 Storage Networking Industry Association. All Rights Reserved. 30
Q&A / Feedback• Please send any questions or comments on this presentation to
SNIA: [email protected]
Many thanks to the following individuals for their contributions to this tutorial.
SNIA Education Committee
Allen Light Robert WinterMatt Brisse Robert Peglar