PLNOG 13: Piotr Szolkowski: 100G Ethernet – Case Study
description
Transcript of PLNOG 13: Piotr Szolkowski: 100G Ethernet – Case Study
![Page 1: PLNOG 13: Piotr Szolkowski: 100G Ethernet – Case Study](https://reader031.fdocuments.net/reader031/viewer/2022013118/55757f72d8b42adb7e8b501d/html5/thumbnails/1.jpg)
Piotr Szołkowski – Extreme Networks Paweł Nastachowski – TelecityGroup (ex. PLIX) PLNOG 2014 Kraków
![Page 2: PLNOG 13: Piotr Szolkowski: 100G Ethernet – Case Study](https://reader031.fdocuments.net/reader031/viewer/2022013118/55757f72d8b42adb7e8b501d/html5/thumbnails/2.jpg)
Connectivity & Consolidation • 14.5 RU - 1/3rd of a rack • 384 x 1G wire-speed • 768 x 10G wire-speed • 192 x 40G wire-speed • 32 x 100G wire-speed
The Flagship – BlackDiamond X8
Performance & Scale • Orthogonal architecture • 10 or 20 Tbps capacity/switch • 1.28 Tbps bandwidth/slot • 30.5 Bpps throughput capacity • 1 Million L2/L3 entries scale*
Latency • No chopping & reassembly • 2.3/3.5 uSec across fabric • <1.5/2.5 uSec within IO
Availability & Assurance • Designed for Tier 1-4 DC • 1+1 Control Plane • N+1 Data Plane • N+N Power Plane • EAPS, MLAG, VRRP, ISSU**
Virtualization & Segmentation • 128K/1M* Virtual Machines • VM lifecycle management • Tenant isolation • VEPA, VPP, XNV™ • VR, MPLS, TRILL, PSTAG*
Storage & Convergence • 10/40G storage • iSCSI, NFS, CIFS • DCBx (PFC, FIPS, ETS) • FCoE transit capability • VBLOCK certified
Efficiency & Conservation • Front-to-Back cooling • Variable fan speed • Intelligent power control • Efficient next-gen optics • Only 5.6W per 10GbE port
2
* Roadmap ** Patch level support
New
SW Defined Networking • OpenFlow 1.3* • Open Stack (Neutron) plugin • API for scripting/applications
![Page 3: PLNOG 13: Piotr Szolkowski: 100G Ethernet – Case Study](https://reader031.fdocuments.net/reader031/viewer/2022013118/55757f72d8b42adb7e8b501d/html5/thumbnails/3.jpg)
3
BlackDiamond X8 IO For Next-Gen Networks
Non
-XL
10G
40
G
100G
X
L 10
0G
40G
12-Port 40GbE QSFP+ Module
48-Port 10GbE SFP+ Module
24-Port 40GbE QSFP+ Module
12-Port 40GbE-XL QSFP+ Module
48-Port 100/1000/10000MbE RJ45 Module
4-Port 100GbE-XL CFP2 Module
PoR 4-Port 100GbE CFP2 Module
GA
PoR
* *
![Page 4: PLNOG 13: Piotr Szolkowski: 100G Ethernet – Case Study](https://reader031.fdocuments.net/reader031/viewer/2022013118/55757f72d8b42adb7e8b501d/html5/thumbnails/4.jpg)
Fabric Module
I/O module
Direct Mating
I/O M
odules
Mid-plane only for management path No mid-plane for data path!
Fabric Modules
4
• Direct Connect data path for ultimate performance
• Future-Proof chassis architecture: No mid-data plane design
Bandwidth to Performance Proportionality
![Page 5: PLNOG 13: Piotr Szolkowski: 100G Ethernet – Case Study](https://reader031.fdocuments.net/reader031/viewer/2022013118/55757f72d8b42adb7e8b501d/html5/thumbnails/5.jpg)
• 4-port wire-speed 100GbE (CFP2) or 40-port 10GbE (SFP+) • Choice of 100G-SR10 (100m) & LR4 (10Km) optics or 10x10 breakout • Supported with either 10Tbps or 20Tbps fabric module types • N+1 data and power plane support with fully populated chassis • Only 368 Watt un-terminated, 440 Watts terminated worst case • Availability: Now
Proud to Present – BlackDiamond X8 100GbE
Extreme’s first 100GbE product ever…
![Page 6: PLNOG 13: Piotr Szolkowski: 100G Ethernet – Case Study](https://reader031.fdocuments.net/reader031/viewer/2022013118/55757f72d8b42adb7e8b501d/html5/thumbnails/6.jpg)
100GbE Optics Comparison
6
CFP CFP2 CFP4
• Brocade MLXe • Cisco Nexus 7000 • Juniper MX • HP12900 • ALU 7750
• Extreme BDX8
• 2015
![Page 7: PLNOG 13: Piotr Szolkowski: 100G Ethernet – Case Study](https://reader031.fdocuments.net/reader031/viewer/2022013118/55757f72d8b42adb7e8b501d/html5/thumbnails/7.jpg)
C
FP2
(N
ot A
vaila
ble
Yet)
CFP4 / QSFP28
100GbE Transceiver (Optics) Comparison
C
FP
CFP2 / (CPAK)
Density
10Km
40Km
4 12 8 24 2
300m CXP / (MXP)
Distance
![Page 8: PLNOG 13: Piotr Szolkowski: 100G Ethernet – Case Study](https://reader031.fdocuments.net/reader031/viewer/2022013118/55757f72d8b42adb7e8b501d/html5/thumbnails/8.jpg)
• Industry leading CFP2 (IEEE 802.3ba) standard based efficient optics • Roughly half the form-factor and power compared to existing CFP • Lower-cost short range (< 100m) connectivity with SR10 • 10 x 10G breakout option using MPO ribbon cable • Long range (< 10Km) connectivity with LR4 over single-mode fiber • Availability: Now
Extreme’s first 100GbE product ever…
SR10 LR4
Proud to Present – BlackDiamond X8 100GbE
![Page 9: PLNOG 13: Piotr Szolkowski: 100G Ethernet – Case Study](https://reader031.fdocuments.net/reader031/viewer/2022013118/55757f72d8b42adb7e8b501d/html5/thumbnails/9.jpg)
The Rise of 100GbE
9 9
Source: Dell’Oro
• Initial adoption of 100GbE will remain to be in Service Provider and Internet Exchange Points (IXP) – and significantly on the Modular Products like BlackDiamond X8.
• In Data Centers, 100GbE adoption will depend on relative price per port. optics and 40GbE adoption on the server side.
• 40GbE server access will start gaining traction after 2014. Servers will quickly drive a significant portion of 40GbE ports.
• Customers will gravitate to 40GbE, not because they need all the bandwidth, but because 2 10GbE ports may not be sufficient.
• Vendors like Extreme Networks may prefer to push down the cost barriers of 100GbE instead of 40GbE for switch interconnects for better consolidation (such as in HPC Interconnect).
![Page 10: PLNOG 13: Piotr Szolkowski: 100G Ethernet – Case Study](https://reader031.fdocuments.net/reader031/viewer/2022013118/55757f72d8b42adb7e8b501d/html5/thumbnails/10.jpg)
10
100GbE Focus for Extreme Networks
Service Provider
Core Data Center
Campus
Edge
Agg.
Metro Border Core Edge
HPC Border Core
Increased Cost
HFT
Immediate Focus
IXP.
![Page 11: PLNOG 13: Piotr Szolkowski: 100G Ethernet – Case Study](https://reader031.fdocuments.net/reader031/viewer/2022013118/55757f72d8b42adb7e8b501d/html5/thumbnails/11.jpg)
100GbE Play Positioning
11
Density
Scale
Combination of Large Memory and MPLS / BGP drives
high scale
Combination of High Density and Lower
Cost drives BW consolidation
Data Center Downlink / HPC / IXP
Dat
a C
ente
r Bor
der /
C
loud
/ In
tern
et
![Page 12: PLNOG 13: Piotr Szolkowski: 100G Ethernet – Case Study](https://reader031.fdocuments.net/reader031/viewer/2022013118/55757f72d8b42adb7e8b501d/html5/thumbnails/12.jpg)
IXP Edge
Site 2
SP 4
SP 5
Cloud 2
SP 6
BDX8
10G
100G
IXP Core
ISP Edge
100G
DWDM
Opt
ical
Net
wor
k
Service Provider: Internet Exchange Point
IXP Edge
Site 1
SP 1
SP 2
SP 3
Cloud 1
BDX8
10G
100G
IXP Core
ISP Edge
100G
100G on customer side (PE) and in the core (P)
![Page 13: PLNOG 13: Piotr Szolkowski: 100G Ethernet – Case Study](https://reader031.fdocuments.net/reader031/viewer/2022013118/55757f72d8b42adb7e8b501d/html5/thumbnails/13.jpg)
Core
Service Provider: Aggregation/Core Offload
High density 10G aggregation through switches, fewer 100G into the core
100G
P Router
P Router
Edge/Agg.
10G
Customers
Customers
PE Router
P Router
Edge/Agg.
10G
Customers
Customers
PE Router B
DX
8 BD
X8
![Page 14: PLNOG 13: Piotr Szolkowski: 100G Ethernet – Case Study](https://reader031.fdocuments.net/reader031/viewer/2022013118/55757f72d8b42adb7e8b501d/html5/thumbnails/14.jpg)
14
48 x 10 GbE
12 x 40 GbE
4 x 100 GbE
24 x 40 GbE
12 x 100 GbE
24 x 100 GbE
10Tbps
Next-Gen
20Tbps
96 x 10 GbE
128 x 10 GbE
240 x 10 GbE
Fabrics Density Capacity
BlackDiamond X8 – Awesome Growth Capacity
32 x 40 GbE
Possibilities
![Page 15: PLNOG 13: Piotr Szolkowski: 100G Ethernet – Case Study](https://reader031.fdocuments.net/reader031/viewer/2022013118/55757f72d8b42adb7e8b501d/html5/thumbnails/15.jpg)
100G w TelecityGroup (ex. PLIX)
![Page 16: PLNOG 13: Piotr Szolkowski: 100G Ethernet – Case Study](https://reader031.fdocuments.net/reader031/viewer/2022013118/55757f72d8b42adb7e8b501d/html5/thumbnails/16.jpg)
Proof of Concept – 100G
• Podłączenie przełącznika BlackDiamond X8 do szkieletu sieci PLIX – 10 x 10G Ethernet
• Zestawienie trasy optycznej na długości 8.4 km • Przyłączenie interfesju 100G do urządzenia klienckiego (Cisco ASR) • Praktyczne testy współdziałania interfejsów 100G • Przeniesienie konfiguracji z istniejących w PLIX urządzeń • Konfiguracja BlackDiamond X8
– 3 x 10T Switch Fabric Module – 1 x 4 porty 100G Ethernet + 2 x CFP2 – 100GBASE-LR4 (10 km) – 2 x 48 portów 10G SFP+
©2014 Extreme Networks, Inc. All rights reserved. 16
![Page 17: PLNOG 13: Piotr Szolkowski: 100G Ethernet – Case Study](https://reader031.fdocuments.net/reader031/viewer/2022013118/55757f72d8b42adb7e8b501d/html5/thumbnails/17.jpg)
Topologia sieci PLIX - PoC
©2014 Extreme Networks, Inc. All rights reserved. 17
![Page 18: PLNOG 13: Piotr Szolkowski: 100G Ethernet – Case Study](https://reader031.fdocuments.net/reader031/viewer/2022013118/55757f72d8b42adb7e8b501d/html5/thumbnails/18.jpg)
Podsumowanie PoC
• Czas trwania testów 4 tygodnie • Udało się uruchomić produkcyjnie jeden kliencki port 100G na czas
MŚ w piłce nożnej • Jedna z optyk 100G CFP2 wykazywała błędy CRC – zamiana optyk
wyeliminowała problem. Dostarczone optyki pochodziły z wstępnych partii przedprodukcyjnych – wersje beta
• Przeprowadzone praktyczne testy w realnej infrastrukturze nie wykazały żadnych problemów
©2014 Extreme Networks, Inc. All rights reserved. 18
![Page 19: PLNOG 13: Piotr Szolkowski: 100G Ethernet – Case Study](https://reader031.fdocuments.net/reader031/viewer/2022013118/55757f72d8b42adb7e8b501d/html5/thumbnails/19.jpg)
Przyszłość 100G w PLIX
• Pojawiło się realne zapotrzebowanie klientów PLIX na przyłącza 100G
• Po testach produkcyjnych podjęta została decyzja o zakupie dwóch przełączników BlackDiamond X8 w następującej konfiguracji:
– 4 x 20T Switching Fabric – 3 x 48 portów 10G SFP+ – 2 x 24 porty 40G QSFP+ – 1 x 4 porty 100G CFP2
• Urządzenia zostaną wpięte do głównego ringu 4 x 40G (EAPS), a między sobą w pierwszym etapie 6 x 40G
©2014 Extreme Networks, Inc. All rights reserved. 19
![Page 20: PLNOG 13: Piotr Szolkowski: 100G Ethernet – Case Study](https://reader031.fdocuments.net/reader031/viewer/2022013118/55757f72d8b42adb7e8b501d/html5/thumbnails/20.jpg)
Nowa planowana topologia sieci PLIX
©2014 Extreme Networks, Inc. All rights reserved. 20
![Page 21: PLNOG 13: Piotr Szolkowski: 100G Ethernet – Case Study](https://reader031.fdocuments.net/reader031/viewer/2022013118/55757f72d8b42adb7e8b501d/html5/thumbnails/21.jpg)
• Przełom listopada i grudnia 2014 roku – uruchomienie pierwszego klienta z łączem 100G
©2014 Extreme Networks, Inc. All rights reserved. 21
![Page 22: PLNOG 13: Piotr Szolkowski: 100G Ethernet – Case Study](https://reader031.fdocuments.net/reader031/viewer/2022013118/55757f72d8b42adb7e8b501d/html5/thumbnails/22.jpg)
Thank You
22