collaborators: Mark Coates, Rui Castro, Ryan King, Mike Rabbat, Yolanda Tsang, Vinay Ribeiro, Shri Sarvotham,Rolf Reidi
Network Bandwidth Estimation and Tomography
Rob Nowak & Rich Baraniuk UW-Madison Rice University
spin.rice.edu
• too complex to measure everywhere, all the time• traffic measurements expensive (hardware, bandwidth)
1969 1993
Internet Boom
companies do not share data or performance information
Proprietary Concerns
• bits are bundled into packets• packets go through routers• queues absorb bursts of packets• congestion: queues fill up, large delays, packet drops
Networking 101
Network Measurement & Inference
Internet equivalent model
Path Modeling and Bandwidth Estimation
Network Tomography
Brain Tomography
unknown object
statistical model
measurements
Maximumlikelihood estimate
maximizelikelihood
physics
data
prior knowledge MRF model
counting &projection
Poisson
unknown object
statistical model
measurements
Maximumlikelihood estimate
maximizelikelihood
physics
data
prior knowledge
Network Tomography
queuing behavior
routing &counting
binomial /multinomial
Network Tomography
From link-level traffic measurements, infer end-to-end traffic flow rates
Vardi ’96, Tebaldi & West ’98
Cao, Davis, Vander Wiel, Yu ’00
y = packet losses or delays measured at the edge
A = routing matrix (graph)
= packet loss probabilities or queuing delays for each link
= randomness inherent traffic measurements
),|(),( AypAl likelihood function
Ay
Network Tomography (MINC Project, Towsley-Duffield)
Probe packets experience similar queuing effects and may interact with each other
Probing the Network
probe =packet stripe
cross-traffic(2)packet (1)packet
delay
Network Tomography: The Basic Ideasender
receivers
Network Tomography: The Basic Ideasender
receivers
Maximum Likelihood Estimation via EM
)()1( ny
)()2( ny
Suppose we were able to measure losses/delays on each link
)(1 nx
)(2 nx)(3 nx
)(3 nx
Expectation-Maximization (EM) alternates between computing expectation of unobserved internal measurements and the desired estimates of link-by-link loss/delay distributions
Problem: How to computemaximum likelihood estimates of link-by-link loss/delay distributions from end-to-end measurements ?
0
1
2
53 4
Topology ID via Probe Interactions
35
d
Topology ID via Probe Interactions
0
1
2
53 4
34
d
35 we can infer that receivers 3 & 4 have a longer shared path than 3 & 5
Finding the Maximum Likelihood Tree
Stochastic search through “forest” via Metropolis-Hastings
True topology
estimated topology
Internet measurement experiments
UNO
What have we learned?
• Clever probing and sampling schemes reveal “hidden” network structure and behavior
• Simple inference algorithms are effective, intuitive, easy to implement, scale nicely
• MLE criteria are easily modified to include prior information: Bayesian or regularized MLE methods are straightforward
Complex interplay between measurement/probing techniques, statistical modeling, and computational methods for optimization
Open Problems: Placement/Coverage
How should measurement devices be deployed ?
Logical graph coverage of physical topology ?
Can random graph models shed some light ?
Open Problems: Spatio-temporal Correlation
competing traffic
Can we detect correlations? Can we exploit them in measurement and mapping applications?
Fuse tomography and bandwidth estimation
Long-range dependence of network traffic
Correlations due to competing traffic flows
Open Problems: Detection and Localization
Detecting and locating anomalous behavior rather than estimating everything
Estimation Hypothesis Testing
How can we capitalize on conventional wisdom: most links are ‘good’ and only a few are ‘bad’ ?
Open Problems: Timing and Synchronization
• Hardware solutions (expensive)• Software solutions (more practical)
- sophisticated software clocks (Veitch ’02)
- crude software clocks (ICMP timestamping) and statistical averaging
sender network
sendermonitor
receivermonitor
receiver
How to accurately measure time ?
Open Problems: Network Security
How can measurement and monitoring across the Internet help detect and prevent malicious activities ?
Top Related