GIAN Course on Distributed Network Algorithms Network ...stefan/netalg13-2-topology.pdf · The Many...

Stefan Schmid @ T-Labs, 2011

GIAN Course on Distributed Network Algorithms

Network Topologies

and Interconnects

The Many Faces and Flavors of Network Topologies

Social Networks.

Internet.

Nervous system.

Datacenter network

Smart grid.

Wireless network before topology control

Gnutella P2P.


Social Networks.

Internet.

Nervous system.

Datacenter network

Smart grid.

Wireless network before topology control

Gnutella P2P.

Also depends on the

abstraction: router level

topology vs Autonomous

System topology?


given

(social network)

Gnutella 2001

(unstructured p2p system)

Chord DHT

(structured p2p system)

semi – or unstructured

(according to simple

join protocol)

subject to

optimization

(datacenter or

structured p2p)


given

(social network)

Gnutella 2001

(unstructured p2p system)

Chord DHT

(structured p2p system)

semi – or unstructured

(according to simple

join protocol)

subject to

optimization

(datacenter or

structured p2p)

What makes a good network topology?

Properties of a good network topology? It depends…!

E.g., support simple and efficient routing!

e.g., low diameter and short routes (wrt #hops, latency, energy, ...?), no state

needed at „routers“ (destination address defines next hop), good expansion

(fast information dissemination and gossip), etc.

E.g., scalable!

e.g., small number of neighbors to store (and maintain?), low degree, large

bisection bandwidth / cutwidth, redundant paths / no bottleneck links, ...

E.g, robustness (random or worst-case failures?):

e.g., „symmetric“ structure, no single point of failure, redundant paths, good

expansion, large mincut, k-connectivity, ...

Etc.!

6

Properties of a good network topology? It depends…!

E.g., support simple and efficient routing!

e.g., low diameter and short routes (wrt #hops, latency, energy, ...?), no state

needed at „routers“ (destination address defines next hop), good expansion

(fast information dissemination and gossip), etc.

E.g., scalable!

e.g., small number of neighbors to store (and maintain?), low degree, large

bisection bandwidth / cutwidth, redundant paths / no bottleneck links, ...

E.g, robustness (random or worst-case failures?):

e.g., „symmetric“ structure, no single point of failure, redundant paths, good

expansion, large mincut, k-connectivity, ...

Etc.!

7

Small diameter does not necessarily imply short

routes! Especially in distributed setting. Need good

algorithm! Remember the maze…

Example: Is the Gnutella P2P network robust?


It depends…


It depends…

Measurement study 2001 with ~2000 peers: [Saroiu et al. 2002]

all connections 30% random peers

removed: still mostly

connected („giant

component“), robust

to random failures /

leaves

4% highest degree

peers removed:

many disconnected

components, not

robust

Network Topologies = Graphs

Graph G=(V,E): V = set of nodes/peers/..., E= set of edges/links/...

d(.,.): distance between two nodes = shortest path, e.g. d(A,D)=?

D(G): diameter (D(G)=maxu,v d(u,v)), e.g. D(G)=?

(U): neighbor set of nodes U (not including nodes in U)

(U) = |(U)| / |U| (size of neighbor set compared to size of U)

(G) = minU, |U| <V/2 (U): expansion of G (meaning?)

Expansion captures „bottlenecks“!

A

D

B

C

Network topologies are often described as graphs!

?

?

Network Topologies = Graphs

Graph G=(V,E): V = set of nodes/peers/..., E= set of edges/links/...

d(.,.): distance between two nodes = shortest path, e.g. d(A,D)=?

D(G): diameter (D(G)=maxu,v d(u,v)), e.g. D(G)=?

(U): neighbor set of nodes U (not including nodes in U)

(U) = |(U)| / |U| (size of neighbor set compared to size of U)

(G) = minU, |U| <V/2 (U): expansion of G (meaning?)

Expansion captures „bottlenecks“!

A

D

B

C

Network topologies are often described as graphs!

3

3

Example: Expansion

(U), (U)?

A

D

B

C

U

13

Example: Expansion

(U), (U)?

A

D

B

C

U

(U) = {C}

Therefore: (U)=1/3

14

bottleneck!

Example: Expansion

(U), (U)?

A

D

B

C

U

(U) = {C}

Therefore: (U)=1/3

15

bottleneck!

Worst possible: so

also expansion of

the whole graph!

The Clique

16

Complete network: pro and cons?

Pro: robust, simpleand fast routing, small diameter...

Cons: does not scale! (degree?, number of edges?, ...)

The Line

Line network: pro and cons?

Degree? Diameter? Expansion?

Pro: simple and fast routing (tree = unique paths!), small degree (2)...

Cons: does not scale! (diameter = n-1)

Expansion?

The Line




Cons: does not scale! (diameter = n-1, expansion = 2/n, ...)

Expansion?

U (n/2 nodes) (U) (= 1 node)

The Line




Cons: does not scale! (diameter = n-1, expansion = 2/n, ...)

Expansion?

U (n/2 nodes) (U) (= 1 node)

Can we reduce diameter without

increasing degree by much?

Good Topologies?

Binary tree network: pro and cons?


20

Good Topologies?



Pro: easy and fast routing (tree = unique paths!), small degree (3), log

diameter...

Cons: bad expansion?

21

Good Topologies?



Pro: easy and fast routing (tree = unique paths!), small degree (3), log

diameter...

Cons: bad expansion = 2/n, ...

Expansion:

U (~n/2 nodes)

G(U) (= 1 node)

All communication

from left to right tree

goes through root!

(no «bisection bandwidth»)22

Datacenter Interconnects

For this reason, datacenter inter

connects form fat-trees: more

bandwidth at higher layers which

interconnect more nodes.

23






24

More bandwidth for

constant bisection

bandwidth!

The Clos topology: one of the

most common datacenter inter-

connects today.

Note: more wires, not

thicker wires! (Due to ECMP

similar effect though?)






25



connects today.

Servers organized into

racks, racks organized

into pods. The core

routers usually connect

to the Internet.






26



connects today.

Servers organized into

racks, racks organized

into pods. The core

routers usually connect

to the Internet.

The Mesh

2d Mesh: pro and cons?


Pro: easy and fast routing (coordinates!), small degree (4), <2 √n

diameter...

Cons: diameter = ?, expansion = ?

The Mesh

2d Mesh: pro and cons?


Pro: easy and fast routing (coordinates!), small degree (4), <2 √n

diameter...

Cons: diameter = √n, expansion = ~2/√n, ...

U ~ n/2 nodes

(U) = √n nodes

The Hypercube

0 1

00 01

10 11

000

100

...

001

010

29

Use identifier manipulation to

describe topology and do routing!

Here: binary strings.

Connected iff Hamming distance

= 1: flip exactly one bit.

The Hypercube

d-dim Hypercube:

Nodes V = {(b1,...,bd), bi binary} (nodes are bitstrings!)

Edges E = for all i: (b1,..., bi, ..., bd)

connected to (b1, ..., 1-bi, ..., bd)

0 1

00 01

10 11

000

100

...

001

010

30

Formally:

The Hypercube

d-dim Hypercube:




0 1

00 01

10 11

000

100

...

001

010

31

Degree? Diameter? Expansion? How

to get from (100101) to (011110)?

The Hypercube

d-dim Hypercube:




2d = n nodes, hence d = log(n): degree

Diameter: fix one bit after another, so log(n) as well

0 1

00 01

10 11

000

100

...

001

010

32

The Hypercube

d-dim Hypercube:






0 1

00 01

10 11

000

100

...

001

010

33

Simply the maximal

Hamming distance!

The Hypercube

d-dim Hypercube:






0 1

00 01

10 11

000

100

...

001

010

34

Simply the maximal

Hamming distance!

What about expansion?

Expansion of Hypercube

d-dim Hypercube:

Nodes V = {(bd,...,b1), bi binary}

Edges E = for all i: (bd,..., bi, ..., b1)

connected to (bd, ..., 1-bi, ..., b1)

Expansion? Find small neighborhood!

Idea: nodes with i x`1’ are connected nodes with (i-1) x`1’ and (i+1) x`1’...:

...

all nodes

with 0x`1’

all nodes

with 1x`1’ all nodes

with 2x`1’

35


Idea:

...

all nodes

with 0x`1’

all nodes


with 2x`1’all nodes

with d/2 x`1’all nodes

with d/2+1 x`1’

U (~n/2 nodes) (U) = binomial(d,d/2+1)

How many nodes?

36


Idea:

...

all nodes

with 0x`1’

all nodes




with d/2+1 x`1’


How many nodes?

37

Number of possibilities to

place d/2+1 `1‘s at d

positions.


Idea:

...

all nodes

with 0x`1’

all nodes




with d/2+1 x`1’


How many nodes?

Expansion 1/√(log n) then follows from computing the ratio...

38



positions.


Idea:

...

all nodes

with 0x`1’

all nodes




with d/2+1 x`1’


How many nodes?

Expansion 1/√(log n) then follows from computing the ratio...

39



positions.

Can it even be lower?

Also for ball?

Many networks are hypercubic!

40

Many computer networks are variants or

generalizations of hypercubes!

E.g., peer-to-peer systems (Chord, Pastry, Kademlia, ...)

E.g., datacenter topologies (container-based datacenters,

BCube, MDCCube, ...)

E.g., parallel architectures (butterfly variants, etc.)

Many networks are hypercubic!

41

Many computer networks are variants or

generalizations of hypercubes!

E.g., peer-to-peer systems (Chord, Pastry, Kademlia, ...)

E.g., datacenter topologies (container-based datacenters,

BCube, MDCCube, ...)

E.g., parallel architectures (butterfly variants, etc.)

Butterfly: A rolled-out hypercube

0

1

0

1

2

0 100 10 01 11

d+1

(first index)

2d (other indices)

42

d=1: k={0,1} d=2:

k=0:

k=1:

Bitstrings like hypercube, connected if one

bit difference, but not at every position…

… but rolled out: just

at this position!


0

1

0

1

2

0 100 10 01 11

d+1

(first index)

2d (other indices)

43

d=1: k={0,1} d=2:

k=0:

k=1:




at this position!

Differ in first bit on this level!

Differ in second bit on this level!


0

1

0

1

2

0 100 10 01 11

d+1

(first index)

2d (other indices)

44

d=1: k={0,1} d=2:

k=0:

k=1:




at this position!

Accordingly: 2-dimensional

identifier: with the „rollout

dimension“.


Butterfly graph: (e.g., for parallel architectures)

Nodes V = {(k, b1...bd), k ϵ {0,...,d}, b ϵ {0,1}d} (2-dim: „number+bitstring“)

Undirected edges E = for all i: (k-1, b1...bk...bd)

connected to (k, b1...bk...bd) and (k, b1...1-bk...bd)

(i.e., to nodes on next level with same and opposite bit at only this position)

Essentially a rolled-out hypercube!

0

1

0

1

2

0 100 10 01 11

d+1

(first index)

2d (other indices)

45

d=1: k={0,1} d=2:

k=0:

k=1:

Formally…

Roll-out hypercube into

multiple levels: connect i-th

bit on i-th level!








0

1

0

1

2

0 100 10 01 11

d+1

(first index)

2d (other indices)

46

d=1: k={0,1} d=2:

k=0:

k=1:

Diameter, Degree, Expansion?

How many nodes in total?








0

1

0

1

2

0 100 10 01 11

d+1

(first index)

2d (other indices)

47

d=1: k={0,1} d=2:

k=0:

k=1:

Diameter, Degree, Expansion?

How many nodes in total? Degree 4, Diameter 2d (e.g., go to

corresponding „bottom“, then up)


Butterfly graph:

Nodes V = {(k, b1...bd), k ϵ {0,...,d}, b ϵ {0,1}d}

Edges E = for all i: (k-1, b1...bk...bd)


Expansion:

0

1

2

00 10 01 11

48

U has n/2 nodes.

Halfs only connected

where k=d: differ in

last bit. So n/d nodes

Expansion ~ 1/d.

CCC: Hypercube with cyclic corners

1,10 1,11

0,100,11

0,00

1,00 1,01

0,01

49


1,10 1,11

0,100,11

0,00

1,00 1,01

0,01

50

Hypercube „with round

corners“: cycles.


Cube-Connected Cycles: Hypercube with „replaced corners“ (split into d nodes!)

Nodes V = {(k, b1...bd) k ϵ {0,...,d-1}, b ϵ {0,1}d}

Edges E = for all i: (k, b1...bk...bd)

connected to (k-1, b1...bk...bd), (k+1, b1...bk...bd) and (k, b1...1-bk...bd)

(for each dimension one node!)

Example:

1,10 1,11

0,100,11

0,00

1,00 1,01

0,01

51

Formally

De Bruijn Graph: Pull-in bits from the back!

00

01

10

11 000

100 110

111

001

010 101

011

52

Like „rolled out hypercube“ but

now it is not bit difference at a

certain position which matters,

but how strings are shifted wrt

to each other!


00

01

10

11 000

100 110

111

001

010 101

011

53

Like „rolled out hypercube“ but

now it is not bit difference at a

certain position which matters,

but how strings are shifted wrt

to each other!

Shift in 0. Shift in 1.


De Bruijn Graph:

Nodes V = {(b1...bd) ϵ {0,1}d} (bitstrings...)

(Undirected) edges E = for all i: (b1...bk...bd)

connected to (b2...bd0) and (b2... bd1) („shift left and add 0 and 1“)

Example (undirected version):

00

01

10

11 000

100 110

111

001

010 101

011

54

Formally…


De Bruijn Graph:





00

01

10

11 000

100 110

111

001

010 101

011

55

How to do routing on

this graph?


De Bruijn Graph:





00

01

10

11 000

100 110

111

001

010 101

011

56

How to do routing on

this graph?Fill in destination bits

from the left!

We have seen graphs with diameter-degree pairs:

log(n)-log(n)

log(n)-O(1)

…

Is there a graph with:

O(1)-log(n)

max(diameter,degree)< O(log n)?

…

What is the degree-diameter tradeoff? Idea? Proof?

Theorem

Each network with n nodes and max degree d>2 must

have a diameter of at least log(n)/log(d-1)-1.

Implications: Constant diameter

networks need a linear degree! But

also: log(n)-log(n) is not the best

tradeoff for minimizing the maximum of

degree and diameter!


Theorem



Implications: Constant diameter

networks need a linear degree! But

also: log(n)-log(n) is not the best

tradeoff for minimizing the maximum of

degree and diameter!

How to prove this?


Theorem



1 d d-1 ...

...

Proof by simply

counting how

many nodes can

be reached given

a certain degree!


Theorem



1 d d-1 ...

...

In two steps, at most

d (d-1)

additional nodes can be reached!

So in k steps at most:

To ensure it is connected this must be at least n, so:

Reformulating this yields the claim... 61

Proof by simply

counting how

many nodes can

be reached given

a certain degree!

Formally: need to

reach n nodes (if

connected, i.e.,

finite diameter).

EXERCISE 1:

Understand the

Pancake Graph

62

O(log n / loglog n)

diamter and degree!

63

Exercise 1: Pancake Graphs Pn

Pancake Graphs

Graph which minimizes max(degree, diameter)!

Both in O( log n / log log n )

Nodes = permutations of {1,...,d}

Edges = prefix reversals

# nodes? degree?

d! many nodes and degree (d-1).

Routing?

E.g., from (3412) to (1243)?

Fix bits at the back, one after the other, in

two steps, so diameter also log n / log log n.

64

d! = n,

so by Stirling formula:

d = log(n)/loglog(n)(insert it to dd=n, resp. to

d log(d) = log(n)...)

65

Solution 1a: Pancake Graphs

66

Solution 1b: Degree Pancake Graphs

There are n-1 non-trivial prefix reversals for

an n-dimensional Pancake with n digits

67

Solution 1c: Diameter of Pancake Graphs

• How to get from node v=v1...vn to w=w1...wn?

• Idea: fix one digit after the other at the back!

– Two steps: Prefix reversal such that digit is at the

front, then full prefix reversal such that digit is at

the back

• Length of routing path: at most 2*(n-1)

• Papadimitriou and Bill Gates have shown that

this is asymptotically also optimal (i.e., close to

diameter)

GIAN Course on Distributed Network Algorithms Network ...stefan/netalg13-2-topology.pdf · The Many...

Documents

Transcript of GIAN Course on Distributed Network Algorithms Network ...stefan/netalg13-2-topology.pdf · The Many...