Post on 01-Apr-2015
Dynamic Evolution of Congestion Trees: Analysis and Impact on Switch Architecture
P. J. García1, J. Flich2, J. Duato2, I. Johnson3, F. J. Quiles1, F. Naven3
2Technical University of Valencia
Valencia, Spain
3Xyratex
Havant, UK
1University of Castilla-La Mancha
Albacete, Spain
HiPEAC 2005 17 November - 18 November Barcelona, Spain
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
2
Outline
• Introduction
• Congestion trees and HOL blocking
• HOL blocking elimination techniques
• Traditional view of congestion trees
• Different dynamics of congestion trees
• RECN improvements
• Performance evaluation
• Conclusions
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
3
Introduction
High-speed interconnection networks:• Myrinet, Infiniband, Quadrics, Advanced Switching…• Main features: High bandwidth, Low latencies• Additional features: Lossless networks, Flexible topology• Cost and power consumption considerations
recommend working close to the saturation point
Network performance may be affected by congestion
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
4
Contention:• Several packets request the same output port• One makes progress, the others wait
Congestion:• Persistent contention • It is quickly propagated by flow control (lossless nets),
forming congestion trees• Network performance degrades dramatically!!!
Congestion trees and HOL blocking
Head of line (HOL) blocking:• When the first packet in a queue is blocked, any other
packet in the same queue is also blocked, even if it will request available resources
WHY?
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
5
Congestion trees and HOL blocking
Networkcontention
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
6
Congestion trees and HOL blocking
Persistentnetworkcontention
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
7
Congestion trees and HOL blocking
Persistentnetworkcontention
Flow control
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
8
Congestion trees and HOL blocking
Persistentnetworkcontention
Congestionpropagates
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
9
Congestion trees and HOL blocking
Congestiontree root
Congestiontree leaf
Congestiontree leaf
Congestiontree branch
Congestiontree branch
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
10
Congestion trees and HOL blocking
Congestion trees introduce HOL blocking, and this may degrade network performance dramatically
33%
33%
HOL 33%
33%100%
33%
33%
33%
100%
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
11
HOL blocking elimination/reduction techniques
• DAMQs and Virtual Channels • Different buffers for different flows
• VOQ (Virtual Output Queues)• VOQ at switch level: A separate queue at every input port for every
output port• VOQ at network level: A separate queue at every input port for every
destination
• Credit Flow Controlled ATM• Handles congestion at network outputs only• A separate queue at every output port for every destination
In general, these techniques try to separate different flows of packets in order to avoid HOL blocking:
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
12
RECN: Regional Explicit Congestion Notification
• RECN is a new efficient and scalable congestion management technique
• Basic ideas:• The real problem is not the congestion, but its negative effects
(HOL blocking)• By eliminating HOL blocking, congestion becomes harmless• Non-congested flows do not introduce significant HOL blocking
• HOL blocking elimination: • Packets belonging to congested flows are stored in specific Set
Aside Queues (SAQs)• Packets belonging to non-congested flows are stored in a
“common” queue
• Implementation requirements:• Deterministic source routing• A reduced number of SAQs per port, controlled by a CAM
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
14
A congestion point forms
How RECN Works
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
15
How RECN Works
Cold queue fills over a threshold
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
17
How RECN Works
Internal notification to each input port
sending packets to the output port
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
19
How RECN Works
Input ports allocate a new SAQ for
packets addressed tothe congested output port
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
21
How RECN Works
Notification sent whenthe SAQ fills
over a threshold
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
23
How RECN Works
A new SAQ allocatedfor the congested port
at each output port
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
24
How RECN Works
Internal notification when the SAQ fills over
A threshold
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
25
How RECN Works
The input port allocatesA new SAQ
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
26
How RECN Works
At the end, the congestion tree builds and is mapped
entirely onto SAQs
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
27
Traditional view of congestion trees
Traditional ideas about congestion trees growth:• Congestion propagates from the root to the leaves• Congestion first appears at egress sides
This is not always true: Congestion trees may evolve in several ways
The effectiveness of HOL blocking elimination techniques may drop if they do not consider
congestion tree dynamics
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
28
Different dynamics of congestion trees
Effect of switch architecture (I):• Switch speedup may vary for different technologies• Depending on switch speedup, congestion may
appear at ingress or egress sides
No speedup switch
Full rateinjection Congestion
Switch speedup: 2
Congestion
Full rateinjection Congestion
Switch speedup: 2
Full rateinjection
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
29
• Switch speedup: 2• All the sources start
injection simultaneously
Different dynamics of congestion trees
Effect of switch architecture (II):• Several congested points may appear both at
ingress or egress sides along the branches of a congestion tree
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
30
Different dynamics of congestion trees
Impact of traffic patterns (I):• Depending on traffic patterns, the congestion tree
root may “move” downstream
• Switch speedup: 2• Solid flows appear first, dashed
ones later
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
31
Different dynamics of congestion trees
Impact of traffic patterns (II):• Different congestion trees may merge, even when
the involved packets have different destinations
• Switch speedup: 2• Solid flows appear first, dashed
ones later
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
32
Different dynamics of congestion trees
Impact of traffic patterns (III):• Different congestion trees may overlap without
merging
• Switch speedup: 2• Solid flows appear first, dashed
ones later
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
33
Different dynamics of congestion trees
Impact of traffic patterns (IV):• A congestion tree root may also move upstream
• Switch speedup: 2• dashed flow disappears first
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
36
RECN improvements
Modified (“Enhanced”) RECN:• Congestion is detected at ingress or egress ports
– Ingress cold queues are replaced by small “detection queues”, one per output port
– If a detection queue fills over a threshold, congestion is detected for the corresponding output port
• It is allowed the allocation of more-specific SAQs– In order to keep in-order delivery of packets, a new
allocated and more-specific SAQ is blocked until all the packets on the less-specific SAQ are forwarded
– A pointer to the new SAQ is placed on the less-specific SAQ in order to control the blocking
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
38
Performance Evaluation
•Objective: Evaluation of RECN improvements
•Comparative evaluation based on simulation results
•Evaluation metric:• Network throughput when using:
– Basic RECN– Enhanced RECN– VOQ at switch level (VOQsw)
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
39
Simulation Model
• Network configurations evaluated:• 64 hosts connected by a 64x64 BMIN• 512 hosts connected by a 512x512 BMIN• 2048 hosts connected by a 2048x2048 BMIN
• Simulation assumptions:• BMINs based on perfect shuffle scheme• Deterministic routing• 32 KB memories at ingress/egress ports• Multiplexed crossbar (BW=8 or12 Gbps)• Serial full-duplex pipelined links (BW=8 Gbps)• 64-byte packets• Credit-based and Xon-Xoff (for SAQs) flow control• Maximum of 8 SAQs at ingress/egress ports (RECN)
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
40
Traffic Load
• Six different synthetic traffic patterns:
•Traces:• From I/O activity at cello system disk interface• A compression factor applied
Normal traffic Congestion tree
Traffic case
Endnodes #Sources Dest.Injection
rate#Sources Dest.
Injection rate
Congestion type
#1 64x64 75% Rand. 50% 25%Single
hot-spot100% Incremental
#2 64x64 75% Rand. 100% 25%Single
hot-spot100% Incremental
#3 64x64 75% Rand. 50% 25%Single
hot-spot100% sudden
#4 64x64 75% Rand. 100% 25%Single
hot-spot100% sudden
#5 512x512 75% Rand. 100% 25%Four
hot-spot100% sudden
#6 2048x2048 75% Rand. 100% 25%Four
hot-spots100% sudden
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
41
Simulation Results
•Network throughput:• Traffic cases 1 and 2 (single hot-spot incremental traffic) • 64-endnodes networks• Speedup: 1.5
Traffic case 1
(Uniform traffic injection rate 50%)
Traffic case 2
(Uniform traffic injection rate 100%)
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
42
Simulation Results
•Network throughput:• SAN traffic (traces) • 64-endnodes networks• Traces compression factor: 40
Speedup 1.5No Speedup
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
43
Simulation Results
•Network throughput:• Traffic cases 3 and 4 (single hot-spot sudden traffic) • 64-endnodes networks• No Speedup
Traffic case 3
(Uniform traffic injection rate 50%)
Traffic case 4
(Uniform traffic injection rate 100%)
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
44
Simulation Results
•Network throughput:• Traffic cases 3 and 4 (single hot-spot sudden traffic) • 64-endnodes networks• Speedup: 1.5
Traffic case 3
(Uniform traffic injection rate 50%)
Traffic case 4
(Uniform traffic injection rate 100%)
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
45
Simulation Results
•Network throughput:• Traffic cases 5 and 6 (four hot-spots sudden traffic) • Uniform traffic injection rate 100%• Speedup: 1.5
Traffic case 5
(512-endnodes network)
Traffic case 6
(2048-endnodes network)
Tit
le:
Dynam
ic E
volu
tion o
f C
on
gest
ion T
rees:
Analy
sis
and Im
pact
on S
wit
ch A
rchit
ect
ure
Confe
rence
: H
iPEA
C 2
00
51
7-1
8 N
ovem
ber
Barc
elo
na, S
pain
46
Conclusions
• Congestion trees producing HOL blocking may affect network performance
• We have shown that congestion trees may form and evolve in different ways
• We have analyzed the importance of considering congestion trees dynamics on the design of HOL blocking elimination techniques
• We have proposed some improvements for RECN, in order to manage HOL blocking independently of the way congestion trees form
• From the results of our experiments, these improvements were necessary
Dynamic Evolution of Congestion Trees: Analysis and Impact on Switch Architecture
P. J. García1, J. Flich2, J. Duato2, I. Johnson3, F. J. Quiles1, F. Naven3
2Technical University of Valencia
Valencia, Spain
3Xyratex
Havant, UK
1University of Castilla-La Mancha
Albacete, Spain
HiPEAC 2005 17 November - 18 November Barcelona, Spain