Enabling(Topological(Flexibility(for( …Topological(Flexibility(for(DataCenters(Using(...
49
Enabling Topological Flexibility for Data Centers Using OmniSwitch Yi#ng Xia , Mike Schlansker , T. S. Eugene Ng , Jean Tourrilhes Rice University , HP Labs HotCloud 2015 § § * * * §
Transcript of Enabling(Topological(Flexibility(for( …Topological(Flexibility(for(DataCenters(Using(...
Enabling Topological Flexibility for Data Centers Using OmniSwitch
Yi#ng Xia , Mike Schlansker , T. S. Eugene Ng , Jean Tourrilhes
Rice University , HP Labs
HotCloud 2015
§ § * *
* §
TradiPonal data centers
Aggrega&on)switch))
Core)switch)
Top3of3Rack)(ToR))switch))
• Fixed network topology • MulP-‐rooted tree structure of Ethernet switches • Bandwidth oversubscripPon • OpPcal fibers and transceivers for fast transmission
1
Aggrega&on)switch))
Core)switch)
Top3of3Rack)(ToR))switch))
3 21
• Limited traffic opPmizaPon
2
Problems of fixed topology
Aggrega&on)switch))
Core)switch)
Top3of3Rack)(ToR))switch))
3 21
• Limited traffic opPmizaPon
Problems of fixed topology
2
Aggrega&on)switch))
Core)switch)
Top3of3Rack)(ToR))switch))
3 21
• Limited traffic opPmizaPon
Problems of fixed topology
2
• Limited traffic opPmizaPon • Complex wiring • Poor expansion
Problems of fixed topology
3
• Limited traffic opPmizaPon • Complex wiring • Poor expansion • High redundancy cost
Problems of fixed topology
4
Backup switch
Backup link
SoluPon: topological flexibility
• Dynamic cable rewiring • Can be achieved using circuit switch
5
• Dynamic cable rewiring • Can be achieved using circuit switch • Circuit switch -‐ dedicated links -‐ many possible topologies -‐ configured by so6ware
A B C D 5
SoluPon: topological flexibility
SoluPon: topological flexibility
• Dynamic cable rewiring • Can be achieved using circuit switch • Circuit switch -‐ dedicated links -‐ many possible topologies -‐ configured by so6ware
A B C D 5
SoluPon: topological flexibility
• Dynamic cable rewiring • Can be achieved using circuit switch • Circuit switch -‐ dedicated links -‐ many possible topologies -‐ configured by so6ware
A B C D 5
3 21
Topological flexibility example
6
3 21
3 21
Topological flexibility example
6
3 21
OmniSwitch architecture
• Modular design • Small opPcal circuit switches • Interleaving opPcal and Ethernet switches • 1 out of 5 Ethernet switch as backup • 16 fibers bundled into 1 mulPlink 7
128 port x 5Ethernet
Stack
128 Ports
...
4x5 port x 128Circuit Switch
Stack
SpareSwitch
Photonic Conversion
Front View of OmniSwitch Panel
8 M
ultil
ink
Con
nect
ors
Multilink16 x 25G
FrontPanel
128 Port x 5 Ethernet Stack
4 x 5 Port x 128 Circuit Switch Stack
Photonic Conversion
Spare Switch Front Panel
OmniSwitch architecture
• Modular design • Small opPcal circuit switches • Interleaving opPcal and Ethernet switches • 1 out of 5 Ethernet switch as backup • 16 fibers bundled into 1 mulPlink 7
128 Port x 5 Ethernet Stack
OmniSwitch architecture
• Modular design • Small opPcal circuit switches • Interleaving opPcal and Ethernet switches • 1 out of 5 Ethernet switch as backup • 16 fibers bundled into 1 mulPlink 7
128 Port x 5 Ethernet Stack
OmniSwitch architecture
• Modular design • Small opPcal circuit switches • Interleaving opPcal and Ethernet switches • 1 out of 5 Ethernet switch as backup • 16 fibers bundled into 1 mulPlink 7
128 Port x 5 Ethernet Stack
OmniSwitch architecture
• Modular design • Small opPcal circuit switches • Interleaving opPcal and Ethernet switches • 1 out of 5 Ethernet switch as backup • 16 fibers bundled into 1 mulPlink 7
128 Port x 5 Ethernet Stack
OmniSwitch architecture
• Modular design • Small opPcal circuit switches • Interleaving opPcal and Ethernet switches • 1 out of 5 Ethernet switch as backup • 16 fibers bundled into 1 mulPlink 7
128 Port x 5 Ethernet Stack
Spare Switch
OmniSwitch architecture
• Modular design • Small opPcal circuit switches • Interleaving opPcal and Ethernet switches • 1 out of 5 Ethernet switch as backup • 16 fibers bundled into 1 mulPlink 7
128 Port x 5 Ethernet Stack
Spare Switch
OmniSwitch architecture
• Modular design • Small opPcal circuit switches • Interleaving opPcal and Ethernet switches • 1 out of 5 Ethernet switch as backup • 16 fibers bundled into 1 mulPlink 7
128 Port x 5 Ethernet Stack
Spare Switch
OmniSwitch architecture
• Modular design • Small opPcal circuit switches • Interleaving opPcal and Ethernet switches • 1 out of 5 Ethernet switch as backup • 16 fibers bundled into 1 mulPlink 7
128 Port x 5 Ethernet Stack
Spare Switch
OmniSwitch architecture
• Modular design • Small opPcal circuit switches • Interleaving opPcal and Ethernet switches • 1 out of 5 Ethernet switch as backup • 16 fibers bundled into 1 mulPlink 7
128 Port x 5 Ethernet Stack
Spare Switch
OmniSwitch architecture
• Modular design • Small opPcal circuit switches • Interleaving opPcal and Ethernet switches • 1 out of 5 Ethernet switch as backup • 16 fibers bundled into 1 mulPlink 7
128 Port x 5 Ethernet Stack
4 x 5 Port x 128 Circuit Switch Stack
Spare Switch
OmniSwitch architecture
• Modular design • Small opPcal circuit switches • Interleaving opPcal and Ethernet switches • 1 out of 5 Ethernet switch as backup • 16 fibers bundled into 1 mulPlink 7
128 Port x 5 Ethernet Stack
4 x 5 Port x 128 Circuit Switch Stack
Photonic Conversion
Spare Switch
OmniSwitch architecture
• Modular design • Small opPcal circuit switches • Interleaving opPcal and Ethernet switches • 1 out of 5 Ethernet switch as backup • 16 fibers bundled into 1 mulPlink 7
128 Port x 5 Ethernet Stack
4 x 5 Port x 128 Circuit Switch Stack
Photonic Conversion
Spare Switch Front Panel
OmniSwitch architecture
• Modular design • Small opPcal circuit switches • Interleaving opPcal and Ethernet switches • 1 out of 5 Ethernet switch as backup • 16 fibers bundled into 1 mulPlink 7
128 port x 5Ethernet
Stack
128 Ports
...
4x5 port x 128Circuit Switch
Stack
SpareSwitch
Photonic Conversion
Front View of OmniSwitch Panel
8 M
ultil
ink
Con
nect
ors
Multilink16 x 25G
FrontPanel
128 Port x 5 Ethernet Stack
4 x 5 Port x 128 Circuit Switch Stack
Photonic Conversion
Spare Switch Front Panel
• Modular design • Small opPcal circuit switches • Interleaving opPcal and Ethernet switches • 1 out of 5 Ethernet switch as backup • 16 fibers bundled into 1 mulPlink 7
128 port x 5Ethernet
Stack
128 Ports
...
4x5 port x 128Circuit Switch
Stack
SpareSwitch
Photonic Conversion
Front View of OmniSwitch Panel
8 M
ultil
ink
Con
nect
ors
Multilink16 x 25G
FrontPanel
OmniSwitch architecture
• Modular design • Small opPcal circuit switches • Interleaving opPcal and Ethernet switches • 1 out of 5 Ethernet switch as backup • 16 fibers bundled into 1 mulPlink 7
128 port x 5Ethernet
Stack
128 Ports
...
4x5 port x 128Circuit Switch
Stack
SpareSwitch
Photonic Conversion
Front View of OmniSwitch Panel
8 M
ultil
ink
Con
nect
ors
Multilink16 x 25G
FrontPanel
OmniSwitch architecture
• Modular design • Small opPcal circuit switches • Interleaving opPcal and Ethernet switches • 1 out of 5 Ethernet switch as backup • 16 fibers bundled into 1 mulPlink 7
128 port x 5Ethernet
Stack
128 Ports
...
4x5 port x 128Circuit Switch
Stack
SpareSwitch
Photonic Conversion
Front View of OmniSwitch Panel
8 M
ultil
ink
Con
nect
ors
Multilink16 x 25G
FrontPanel
OmniSwitch architecture
• Modular design • Small opPcal circuit switches • Interleaving opPcal and Ethernet switches • 1 out of 5 Ethernet switch as backup • 16 fibers bundled into 1 mulPlink 7
128 port x 5Ethernet
Stack
128 Ports
...
4x5 port x 128Circuit Switch
Stack
SpareSwitch
Photonic Conversion
Front View of OmniSwitch Panel
8 M
ultil
ink
Con
nect
ors
Multilink16 x 25G
FrontPanel
OmniSwitch architecture
• Modular design • Small opPcal circuit switches • Interleaving opPcal and Ethernet switches • 1 out of 5 Ethernet switch as backup • 16 fibers bundled into 1 mulPlink 7
128 port x 5Ethernet
Stack
128 Ports
...
4x5 port x 128Circuit Switch
Stack
SpareSwitch
Photonic Conversion
Front View of OmniSwitch Panel
8 M
ultil
ink
Con
nect
ors
Multilink16 x 25G
FrontPanel
OmniSwitch architecture
• Modular design • Small opPcal circuit switches • Interleaving opPcal and Ethernet switches • 1 out of 5 Ethernet switch as backup • 16 fibers bundled into 1 mulPlink 7
128 port x 5Ethernet
Stack
128 Ports
...
4x5 port x 128Circuit Switch
Stack
SpareSwitch
Photonic Conversion
Front View of OmniSwitch Panel
8 M
ultil
ink
Con
nect
ors
Multilink16 x 25G
FrontPanel
OmniSwitch architecture
• Modular design • Small opPcal circuit switches • Interleaving opPcal and Ethernet switches • 1 out of 5 Ethernet switch as backup • 16 fibers bundled into 1 mulPlink 7
128 port x 5Ethernet
Stack
128 Ports
...
4x5 port x 128Circuit Switch
Stack
SpareSwitch
Photonic Conversion
Front View of OmniSwitch Panel
8 M
ultil
ink
Con
nect
ors
Multilink16 x 25G
FrontPanel
OmniSwitch architecture
• Modular design • Small opPcal circuit switches • Interleaving opPcal and Ethernet switches • 1 out of 5 Ethernet switch as backup • 16 fibers bundled into 1 mulPlink 7
128 port x 5Ethernet
Stack
128 Ports
...
4x5 port x 128Circuit Switch
Stack
SpareSwitch
Photonic Conversion
Front View of OmniSwitch Panel
8 M
ultil
ink
Con
nect
ors
Multilink16 x 25G
FrontPanel
OmniSwitch architecture
• Modular design • Small opPcal circuit switches • Interleaving opPcal and Ethernet switches • 1 out of 5 Ethernet switch as backup • 16 fibers bundled into 1 mulPlink 7
128 port x 5Ethernet
Stack
128 Ports
...
4x5 port x 128Circuit Switch
Stack
SpareSwitch
Photonic Conversion
Front View of OmniSwitch Panel
8 M
ultil
ink
Con
nect
ors
Multilink16 x 25G
FrontPanel
OmniSwitch architecture
OmniSwitch architecture
• Modular design • Small opPcal circuit switches • Interleaving opPcal and Ethernet switches • 1 out of 5 Ethernet switch as backup • 16 fibers bundled into 1 mulPlink 7
128 port x 5Ethernet
Stack
128 Ports
...
4x5 port x 128Circuit Switch
Stack
SpareSwitch
Photonic Conversion
Front View of OmniSwitch Panel
8 M
ultil
ink
Con
nect
ors
Multilink16 x 25G
FrontPanel
128 Port x 5 Ethernet Stack
4 x 5 Port x 128 Circuit Switch Stack
Photonic Conversion
Spare Switch Front Panel
OmniSwitch mesh network
1 2
6 5
3
4
8
7
3 4
2 1
768 Port384 Port
3072 Port
768 Port384 Port
3072 Port
3 4
2 1
(a)
(b) 8
OmniSwitch tree network 1 Spine
OmniSwitch
4 LeafOmniSwitches32 uplink cables
1536 Port 768 Port384 Port
3072 Port
(a)
(b)
1536 Port
3072 Port 9
Benefits of OmniSwitch • Easy wiring -‐ mul8links -‐ automa8c internal wiring
• Incremental expansion -‐ add equipments in cabinet -‐ purchase new cabinets
• Efficient backup -‐ 1:5 vs. 1:1 redundancy
• Traffic opPmizaPon -‐ traffic locality -‐ load balancing
• Cost effecPveness -‐ cheap small op8cal circuit switches -‐ constrained flexibility 10
128 Port x 5 Ethernet Stack
4 x 5 Port x 128 Circuit Switch Stack
Photonic Conversion
Spare Switch Front Panel
Control plane
• ExisPng rouPng schemes do not work -‐ need to configure topology & rou8ng • Data center administered by single enPty -‐ centralized network controller • OpPcal switches configured by so`ware • RouPng enforced by many possible mechanisms -‐ OpenFlow protocol -‐ VLAN -‐ source rou8ng • Control logic customized to use cases
11
VM clustering: a case study
• Tenant refers to a cluster of VMs • VM only talks to other VMs in the same tenant • Localize traffic within tenant • Tenant management, not flow management • Input: pipe model • Output: 1) opPcal switch secngs 2) placement of VLs on physical links
VM Virtual Link (VL)
1Gbps
12
Control algorithm
• Hard problem -‐ NP-‐complete for fixed topology -‐ search space larger for OmniSwitch • Tree-‐structured OmniSwitch cabinets • IteraPve from bodom up • VMs already placed à beder topology & rouPng • Search space small -‐ uplinks on switches -‐ permuta8ons on small op8cal switches • Configure topology and rouPng at the same Pme 13
Control algorithm
• Hard problem -‐ NP-‐complete for fixed topology -‐ search space larger for OmniSwitch • Tree-‐structured OmniSwitch cabinets • IteraPve from bodom up • VMs already placed à beder topology & rouPng • Search space small -‐ uplinks on switches -‐ permuta8ons on small op8cal switches • Configure topology and rouPng at the same Pme 13
768 Port384 Port
3072 Port
1 SpineOmniSwitch
4 LeafOmniSwitches32 uplink cables
1536 Port
EvaluaPons
• Tenant provisioning scenario • Smart VM placement vs. flexible topology • Systems compared ü dumb -‐ con8guous VM placement + ECMP rou8ng + fixed Clos network ü SecondNet -‐ SecondNet VM placement and rou8ng + fixed Clos network ü OmniSwitch -‐ con8guous VM placement + VM clustering algorithm + OmniSwitch network ü OmniSwitch (big OCS) -‐ same as above, but 1 big op8cal switch in each OmniSwitch cabinet
14
Bandwidth rejecPon rate
15
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.90
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
Load
Band
widt
h Re
ject
ion
Rate
dumb + fixed ClosSecondNet + fixed ClosOmniSwitchOmniSwitch (big OCS)
bw rejecPon rate = rejected bw / requested bw load = tenant size * arrival rate * duraPon / #VM
Transmission hop count
dumb SecondNet OmniSwitch OmniSwitch (big OCS)
4.422 4.164 3.217 3.048
16
Average hop count when load = 0.8
ComputaPon Pme
17
101 102 103102
103
104
105
106
107
108
109
1010
Tenant Size (#VM)
Com
puta
tion
Tim
e (u
s)
SecondNet − load = 0.2SecondNet − load = 0.5SecondNet − load = 0.8OmniSwitch − load = 0.2OmniSwitch − load = 0.5OmniSwitch − load = 0.8
1s
17min
Conclusion • OmniSwitch is a modular data center architecture • Integrated opPcal switches and Ethernet switches for
topological flexibility and large-‐scale connecPvity • Topological flexibility improves traffic opPmizaPon and
simplifies network management • VM clustering as a case study -‐ reduce transmission path length -‐ host more bandwidth -‐ low computa8on 8me
18
