25GbE Server to Switch Architectures - IEEE 802 SERVER TO SWITCH ARCHITECTURES Scott Kipp...
Transcript of 25GbE Server to Switch Architectures - IEEE 802 SERVER TO SWITCH ARCHITECTURES Scott Kipp...
![Page 1: 25GbE Server to Switch Architectures - IEEE 802 SERVER TO SWITCH ARCHITECTURES Scott Kipp skipp@brocade.com September 2014 kipp_25GE_01_0914a.pdf 1 . ... •File Server – NAS …](https://reader031.fdocuments.net/reader031/viewer/2022021504/5aa5b43a7f8b9a7c1a8dc4ae/html5/thumbnails/1.jpg)
25GBE SERVER TO SWITCH
ARCHITECTURES
Scott Kipp
September 2014
kipp_25GE_01_0914a.pdf
1
![Page 2: 25GbE Server to Switch Architectures - IEEE 802 SERVER TO SWITCH ARCHITECTURES Scott Kipp skipp@brocade.com September 2014 kipp_25GE_01_0914a.pdf 1 . ... •File Server – NAS …](https://reader031.fdocuments.net/reader031/viewer/2022021504/5aa5b43a7f8b9a7c1a8dc4ae/html5/thumbnails/2.jpg)
Supporters
• Jonathan King – Finisar
• Andy Moorwood – Infinera
• Steve Swanson – Corning
• John Abbott – Corning
• Gary J. Bernstein – Leviton
• Paul Kolesar – Commscope
• Jack Jewell – Green VCSEL
• Doug Coleman – Corning
• Rick Pimpinella – Panduit
• Nathan Tracy – TE Connectivity
9/10/2014 2
• Megha Shanbhag – TE Connectivity
• Robert Lingle Jr. – OFS
• Scott Sommers – Molex
• Dave Lewis – JDSU
• Chris Cole – Finisar
• Dave Chalupsky – Intel
• John D’Ambrosia - Dell
• Rick Rabinovich – Alcatel Lucent
• Tom Palkert - Molex
![Page 3: 25GbE Server to Switch Architectures - IEEE 802 SERVER TO SWITCH ARCHITECTURES Scott Kipp skipp@brocade.com September 2014 kipp_25GE_01_0914a.pdf 1 . ... •File Server – NAS …](https://reader031.fdocuments.net/reader031/viewer/2022021504/5aa5b43a7f8b9a7c1a8dc4ae/html5/thumbnails/3.jpg)
Server to Switch Architectures
• Server to switch architectures/deployments depend on 5
main characteristics
1. Application(s)
2. Server
3. Cabling
4. Switch
5. IT Departments
• With an amazing variety within each of these categories, an
amazing variety of solutions results – like the Cambrian
Period of life on Earth
• The 25GbE standard should support the widest variety of
deployments as possible so an optical PMD is needed
9/10/2014 3
![Page 4: 25GbE Server to Switch Architectures - IEEE 802 SERVER TO SWITCH ARCHITECTURES Scott Kipp skipp@brocade.com September 2014 kipp_25GE_01_0914a.pdf 1 . ... •File Server – NAS …](https://reader031.fdocuments.net/reader031/viewer/2022021504/5aa5b43a7f8b9a7c1a8dc4ae/html5/thumbnails/4.jpg)
Five Components of Switch To Server
Architectures
9/10/2014 4
Low End Mid Range High End
Application Web Server Enterprise Apps –
payroll, HR, CAD
Financial
Transactions
Server 1U Server Multi-U Server Mainframe
Cabling Simple
Patchcord
AOC or Simple
Cabling
Structured Cabling
Switch 1U or less
Switch
Multi-U Switch
Modular Switch
IT Staff 1 Guy or Gal Small Team Large Teams
![Page 5: 25GbE Server to Switch Architectures - IEEE 802 SERVER TO SWITCH ARCHITECTURES Scott Kipp skipp@brocade.com September 2014 kipp_25GE_01_0914a.pdf 1 . ... •File Server – NAS …](https://reader031.fdocuments.net/reader031/viewer/2022021504/5aa5b43a7f8b9a7c1a8dc4ae/html5/thumbnails/5.jpg)
Application Variety
• Single applications
• Low bandwidth – few users or small data rates
• High bandwidth – many users or high data rates
• Multiple applications via virtualization
• Massive web applications with millions or billions of
users
• File Server – NAS
• Financial transactions
Applications drive bandwidth
9/10/2014 5
![Page 6: 25GbE Server to Switch Architectures - IEEE 802 SERVER TO SWITCH ARCHITECTURES Scott Kipp skipp@brocade.com September 2014 kipp_25GE_01_0914a.pdf 1 . ... •File Server – NAS …](https://reader031.fdocuments.net/reader031/viewer/2022021504/5aa5b43a7f8b9a7c1a8dc4ae/html5/thumbnails/6.jpg)
Server Designs
• Microservers – ARM Servers
• Blade Servers
• 1/2U Servers
• 1U Servers
• 2U Servers
• 4-12U Servers
• Rack and multi-rack Servers
9/10/2014 6
Mainframe
Cray X-ES Microserver
8U
Storage
Server
![Page 7: 25GbE Server to Switch Architectures - IEEE 802 SERVER TO SWITCH ARCHITECTURES Scott Kipp skipp@brocade.com September 2014 kipp_25GE_01_0914a.pdf 1 . ... •File Server – NAS …](https://reader031.fdocuments.net/reader031/viewer/2022021504/5aa5b43a7f8b9a7c1a8dc4ae/html5/thumbnails/7.jpg)
Rack Space
7
Max Servers
/ 40RU
Micro-Server >100
Blade Server >100
1/2U Server 80
1U Server 40
2U Server 20
4U Server 10
8U Server 5
12U Server 3
Mainframe <1
42 RU Rack
2U for Switches
40U for everything else
Any cable management?
Any power supplies?
Any storage?
Any power or cooling
limiations?
![Page 8: 25GbE Server to Switch Architectures - IEEE 802 SERVER TO SWITCH ARCHITECTURES Scott Kipp skipp@brocade.com September 2014 kipp_25GE_01_0914a.pdf 1 . ... •File Server – NAS …](https://reader031.fdocuments.net/reader031/viewer/2022021504/5aa5b43a7f8b9a7c1a8dc4ae/html5/thumbnails/8.jpg)
Cabling Variety
• From lowest cost, complexity and scalability
• Patchcord
• Direct Attach Cabling
• AOC
• Structured Cabling
• Single-Link Channel
• Double-Link Channel
• Triple-Link Channel
• Inter-building Channel
9/10/2014 8
![Page 9: 25GbE Server to Switch Architectures - IEEE 802 SERVER TO SWITCH ARCHITECTURES Scott Kipp skipp@brocade.com September 2014 kipp_25GE_01_0914a.pdf 1 . ... •File Server – NAS …](https://reader031.fdocuments.net/reader031/viewer/2022021504/5aa5b43a7f8b9a7c1a8dc4ae/html5/thumbnails/9.jpg)
TIA-942 – Data Center Cabling and Design
• TIA-942 -
Telecommunications
Infrastructure for
Data Centers
defines:
• MDA (Main
Distribution Area)
that fans out to
• HDAs (Horizontal
Distribution Areas)
MDA
HDA HDA HDA
![Page 10: 25GbE Server to Switch Architectures - IEEE 802 SERVER TO SWITCH ARCHITECTURES Scott Kipp skipp@brocade.com September 2014 kipp_25GE_01_0914a.pdf 1 . ... •File Server – NAS …](https://reader031.fdocuments.net/reader031/viewer/2022021504/5aa5b43a7f8b9a7c1a8dc4ae/html5/thumbnails/10.jpg)
25GbE Switches
• Switch ASICs are increasing speed from 10GbE to
25GbE and more than doubling the port counts from
64 ports to 128+ ports
25GbE will be like 10GbE today
10
64 10GbE port ASIC enables
48 SFP+ and 4 QSFP+
640Gb/s of Throughput
128 25GbE port ASIC enables
32 QSFP+
3.2 Tb/s of Throughput
64 Ports
of 10G
128+ Ports
of 25G
![Page 11: 25GbE Server to Switch Architectures - IEEE 802 SERVER TO SWITCH ARCHITECTURES Scott Kipp skipp@brocade.com September 2014 kipp_25GE_01_0914a.pdf 1 . ... •File Server – NAS …](https://reader031.fdocuments.net/reader031/viewer/2022021504/5aa5b43a7f8b9a7c1a8dc4ae/html5/thumbnails/11.jpg)
10GbE Switch Designs
• Blade Switches
• 1/2U Switches
• 1U Switches
• 2U Switches
• 4-12U Modular Switches
9/10/2014 11
12 QSFP+ = 48 25GbE
64 SFP+
36 QSFP+
= 144 25GbE
4 SFP+
96 SFP+
216 QSFP+
= 864 25GbE
![Page 12: 25GbE Server to Switch Architectures - IEEE 802 SERVER TO SWITCH ARCHITECTURES Scott Kipp skipp@brocade.com September 2014 kipp_25GE_01_0914a.pdf 1 . ... •File Server – NAS …](https://reader031.fdocuments.net/reader031/viewer/2022021504/5aa5b43a7f8b9a7c1a8dc4ae/html5/thumbnails/12.jpg)
IT Departments
• One Person Shops
• Small teams
• Applications
• Servers
• Storage
• WAN
• Cabling
• Power
• Cooling
• Consultants
• Contractors
• …
• Large Teams/Departments 9/10/2014 12
From here
To there…
![Page 13: 25GbE Server to Switch Architectures - IEEE 802 SERVER TO SWITCH ARCHITECTURES Scott Kipp skipp@brocade.com September 2014 kipp_25GE_01_0914a.pdf 1 . ... •File Server – NAS …](https://reader031.fdocuments.net/reader031/viewer/2022021504/5aa5b43a7f8b9a7c1a8dc4ae/html5/thumbnails/13.jpg)
25GbE Deployments
• First deployments in the cloud data center, but they are still the smaller
segment of the market
• ToR done by many
• Second deployments in enterprises, and they will gladly do it for the
right applications if it saves money and time
• Wide variety of deployment models
9/10/2014 13
Source: Dell’Oro Controller and Adapter
Forecast Summary – July 2014
Enterprise and Premises
dominate server shipments
through forecast – Revenue is
even more lopsided
![Page 14: 25GbE Server to Switch Architectures - IEEE 802 SERVER TO SWITCH ARCHITECTURES Scott Kipp skipp@brocade.com September 2014 kipp_25GE_01_0914a.pdf 1 . ... •File Server – NAS …](https://reader031.fdocuments.net/reader031/viewer/2022021504/5aa5b43a7f8b9a7c1a8dc4ae/html5/thumbnails/14.jpg)
ToR, MoR, EoR
14
Not addressed by 25Gb/s CFI
ToR (3 to 5 m)
MoR (3 to 15 m)
EoR (3 to 50 m)
Addressed by
25Gb/s CFI
Home Run (50 to 100+ m)
No Switches
Here
Source of graphics: jimenez_3bq_01_0711
![Page 15: 25GbE Server to Switch Architectures - IEEE 802 SERVER TO SWITCH ARCHITECTURES Scott Kipp skipp@brocade.com September 2014 kipp_25GE_01_0914a.pdf 1 . ... •File Server – NAS …](https://reader031.fdocuments.net/reader031/viewer/2022021504/5aa5b43a7f8b9a7c1a8dc4ae/html5/thumbnails/15.jpg)
1U Server ToR Designs
5m is Fine
15
42 RU Racks
36 QSFP+
= 144 25GbE
42
Se
rvers
40
Se
rvers
42
Se
rvers
3m for Intra-rack
5m for Adjacent racks
20 ports (500G) left for
uplinks…
One ToR switch can support
multiple racks of 1U servers
If the servers are 2U and not
fully filled, the switch can
support many more racks
and 5m is not enough
124 Servers
![Page 16: 25GbE Server to Switch Architectures - IEEE 802 SERVER TO SWITCH ARCHITECTURES Scott Kipp skipp@brocade.com September 2014 kipp_25GE_01_0914a.pdf 1 . ... •File Server – NAS …](https://reader031.fdocuments.net/reader031/viewer/2022021504/5aa5b43a7f8b9a7c1a8dc4ae/html5/thumbnails/16.jpg)
1U Server EoR Designs
16
42 RU Rack
42
Se
rvers
42
Se
rvers
216 QSFP+
= 864 25GbE/Switch
Each switch could
support 800 servers
with 64 ports
(1.6Tb/s) of uplinks
This rack of modular
switches would
support 2,400 servers
42
Se
rvers
2,400 Servers
in 57 racks
![Page 17: 25GbE Server to Switch Architectures - IEEE 802 SERVER TO SWITCH ARCHITECTURES Scott Kipp skipp@brocade.com September 2014 kipp_25GE_01_0914a.pdf 1 . ... •File Server – NAS …](https://reader031.fdocuments.net/reader031/viewer/2022021504/5aa5b43a7f8b9a7c1a8dc4ae/html5/thumbnails/17.jpg)
Home Run
• A home run is a server connect architecture where a server is connected straight into the core of the network
• Common for storage servers or NAS that shares massive files – feedback to Ethernet Alliance Bird of Feather at HPC’13
• Mainframes and large enterprise servers may connect straight into the core
• These links need high speed
• Used when servers, storage and switches are consolidated into different areas
• Usually associated with structured cabling where every port is represented in the MDA
17
![Page 18: 25GbE Server to Switch Architectures - IEEE 802 SERVER TO SWITCH ARCHITECTURES Scott Kipp skipp@brocade.com September 2014 kipp_25GE_01_0914a.pdf 1 . ... •File Server – NAS …](https://reader031.fdocuments.net/reader031/viewer/2022021504/5aa5b43a7f8b9a7c1a8dc4ae/html5/thumbnails/18.jpg)
Large Data Center Design
• Most new data centers are being
designed with a Pod (CoLo or Cell)
Architecture
• Pods usually 15-20,000 sq ft
• HDA (Horizontal Distribution Area) is
where distribution switches are
located
• The Main Distribution Area (MDA)
interconnect PODs and connects to
the WAN and telecom networks
300,000 sq ft new data center
600’
500’
Modular HVAC Systems
MDA
POD
1
MDA
Admin
offices
Shipping,
receiving
and storage
75’
200’
Modular HVAC Systems
POD
2 POD
3
POD
4
POD
5 POD
6
POD
7 POD
8
HDA HDA HDA HDA
HDA HDA HDA HDA
![Page 19: 25GbE Server to Switch Architectures - IEEE 802 SERVER TO SWITCH ARCHITECTURES Scott Kipp skipp@brocade.com September 2014 kipp_25GE_01_0914a.pdf 1 . ... •File Server – NAS …](https://reader031.fdocuments.net/reader031/viewer/2022021504/5aa5b43a7f8b9a7c1a8dc4ae/html5/thumbnails/19.jpg)
POD Architecture
• 15,000 sq ft POD
• Up to 5,000 servers / POD
• 512 Racks possible
• 32 Rows of racks
• Each row has 16 racks
• Horizontal Distribution Area
(HDA) connects all of the racks
C
oo
l A
isle
Co
ol A
isle
Co
ol A
isle
Co
ol A
isle
Co
ol A
isle
Co
ol A
isle
Co
ol A
isle
Co
ol A
isle
HDA
75’
200’
1 2 3 4 5 6 7 8
9 10 11 12 13 14 15 16
17 18 19 20 21 22 23 24
25 26 27 28 29 30 31 32
![Page 20: 25GbE Server to Switch Architectures - IEEE 802 SERVER TO SWITCH ARCHITECTURES Scott Kipp skipp@brocade.com September 2014 kipp_25GE_01_0914a.pdf 1 . ... •File Server – NAS …](https://reader031.fdocuments.net/reader031/viewer/2022021504/5aa5b43a7f8b9a7c1a8dc4ae/html5/thumbnails/20.jpg)
POD Design with Consolidation
HDA
75’
200’
200 Server Racks
32 Switch Racks
50 Storage Racks
4 Router Racks
32 Tape Racks
8 Mainframe Racks
![Page 21: 25GbE Server to Switch Architectures - IEEE 802 SERVER TO SWITCH ARCHITECTURES Scott Kipp skipp@brocade.com September 2014 kipp_25GE_01_0914a.pdf 1 . ... •File Server – NAS …](https://reader031.fdocuments.net/reader031/viewer/2022021504/5aa5b43a7f8b9a7c1a8dc4ae/html5/thumbnails/21.jpg)
Home Runs
• Home Runs go
• From each server
• To HDA
• Patchcord within HDA
• To Switch
• 100 meters required
HDA
75’
200’
![Page 22: 25GbE Server to Switch Architectures - IEEE 802 SERVER TO SWITCH ARCHITECTURES Scott Kipp skipp@brocade.com September 2014 kipp_25GE_01_0914a.pdf 1 . ... •File Server – NAS …](https://reader031.fdocuments.net/reader031/viewer/2022021504/5aa5b43a7f8b9a7c1a8dc4ae/html5/thumbnails/22.jpg)
How 10GbE Servers Connect
• Dell’Oro tracks how servers connect to switches by
media type, but not topology
• Millions of 10GbE servers are currently connecting to
switches with optics
9/10/2014 22
Source: Dell’Oro Controller and Adapter
Forecast Summary – January 2014
10GbE Optical ports expected to be >1M
ports in 2016
![Page 23: 25GbE Server to Switch Architectures - IEEE 802 SERVER TO SWITCH ARCHITECTURES Scott Kipp skipp@brocade.com September 2014 kipp_25GE_01_0914a.pdf 1 . ... •File Server – NAS …](https://reader031.fdocuments.net/reader031/viewer/2022021504/5aa5b43a7f8b9a7c1a8dc4ae/html5/thumbnails/23.jpg)
Enable Multiple Solutions with 25GbE Optics
• With incredible variation in bandwidth requirements of
servers as shown in the 25GbE CFI, we should enable as
many applications as possible to increase broad market
potential
• Limiting IEEE standards to 3 to 5 meters will limit broad
market potential and restrict users to ToR configurations
• Let’s enable MoR, EoR and Home Runs with and optical
PMD to 100 meters
We are in the Cambrian Period of IT with many things possible
9/10/2014 23
![Page 24: 25GbE Server to Switch Architectures - IEEE 802 SERVER TO SWITCH ARCHITECTURES Scott Kipp skipp@brocade.com September 2014 kipp_25GE_01_0914a.pdf 1 . ... •File Server – NAS …](https://reader031.fdocuments.net/reader031/viewer/2022021504/5aa5b43a7f8b9a7c1a8dc4ae/html5/thumbnails/24.jpg)
THANK YOU
24