Data Center Networking with Multipath TCPCostin RaiciuUniversity College London & Universitatea Politehnica Bucuresti

Christopher Pluntke, UCLAdam Greenhalgh, UCLSebastien Barre, Universite Catholique LouvainDamon Wischik. UCLMark Handley, UCL

UCL

Topology

Data Center Networking Today

Routing

Resource Allocation

FatTree, VL2, BCube, multi-rooted tree

Random load balancing

TCP

Path Selection

OSPF, VLANs, TRILL

Topology

Data Center Networking Tomorrow

Routing

Resource Allocation

FatTree, VL2, BCube, multi-rooted tree

Random load balancing

TCP

MultipathTCPPath

Selection

OSPF, VLANs, TRILL

Data Centers are Important

Cloud computing Economies of scale:

networks of tens of thousands of hosts

Cool apps Web search, GFS, BigTable,

DryadLINQ, MapReduce Dense traffic patterns

Flexibility is Important in Data Centers Apps distributed across thousands of

machines. Flexibility: want any machine to be able to

play any role.

But: Traditional data center topologies are tree

based. Don’t cope well with non-local traffic

patterns.

Many recent proposals for better topologies.

Traditional Data Center Topology

…Racks of servers

Top of Rack Switches

Aggregation Switches

Core Switch

1Gbps

10Gbps

10Gbps

Fat Tree Topology [Fares et al., 2008; Clos, 1953]

Aggregation Switches

K Pods with K Switches

each

K=4

Racks of servers

1Gbps

1Gbps

VL2 Topology [Greenberg et al, 2009, Clos topology]

10Gbps

20 hosts

10Gbps …

BCube Topology [Guo et al, 2009]

BCube (4,1)

How Do We Use this Capacity?

Need to distribute flows across paths. Basic solution: Random Load Balancing.

Use Equal-Cost Multipath (ECMP) routing.• Hash to a path at random.

Use many differently rooted VLANs.• End-host hashes to a VLAN; determines path.

Collisions

Racks of servers

1Gbps

1Gbps

Can MPTCP self-optimize data-center traffic?

With Multipath TCP we can explore many paths: Instead of using one random path, use

many random paths Don’t worry about collisions. Just don’t send (much) traffic on colliding

paths

Simulation Setup

~8000 hosts Long-lived flows Permutation traffic matrix

Each hosts sends and receives from a single other randomly chosen host

Smallest amount of traffic that can fill the network

Multipath TCP in the Fat Tree TopologyThroughput Allocation

Performance depends on topologyVL2 BCube

Overloaded Fat Tree: better fairness with Multipath TCP

Centralized Scheduling With RLB, it’s really hard to utilize FatTree.

Hedera [Fares et al.,2010] uses a centralized scheduler and flow switching. Start by using RLB Measure all flow throughput periodically. Any flow using more than 10% of its

interface rate is explicitly scheduled onto an unloaded link.

How does centralized scheduling compare with MPTCP?

MPTCP vs Centralized Dynamic Scheduling

Infinite

Centralized Scheduling MPTCP

Scheduling Interval

Can’t we just use many TCP connections?

Loss rate of MP-TCP (“linked”) vs multiple uncoupled TCP flows

Retransmit timeouts with MPTCP (“linked”) vs uncoupled TCP flows

MPTCP Linked Increases in DCs

Better fairness and less aggressive than uncoupled TCP

Improves throughput in dense traffic in BCube (25%)

The bigger picture

Topology

Routing

Resource Allocation

FatTree, VL2, Bcube, multi-rooted tree

MultipathTCPPath

Selection

OSPF, VLANs, etc.

?

Multipath TCP can utilize topologies TCP can’t

1Gb/s

1Gb/s

Requirement: a subset of hosts should be able to communicate at 10Gb/s

10Gb/s

Multipath TCP can utilize topologies TCP can’t [2]

Problem ToR switch failures wipe out tens of

hosts Repair time is on the order of days

Solution: use two ToRs/rack, multi-home servers

Single path TCP Single flows still get same max

throughput Which interface do I use?

With Multipath TCP Flows double their maximum

throughput Path selection automatic

Summary Data center networking offers many paths

between end-hosts. Yet: Random Load Balancing does a poor job

of utilizing them Centralized scheduling is laggy and has

inherently limited knowledge Multipath TCP naturally optimizes data

center networks: Improves throughput Improves fairness More robust than centralized scheduling

Question: what topologies does multipath TCP enable?

Backup Slides

Centralized Scheduling: Setting the Threshold

Throughput

1Gbps

100Mbps

Hope

App Limited

17% worse than

multipath TCP

Centralized Scheduling: Setting the Threshold

Throughput

1Gbps

100Mbps HopeApp Limited

21% worse than

multipath TCP

Centralized Scheduling: Setting the Threshold

Throughput

1Gbps

100Mbps17%

21%

500Mbps

45%

51%