Netem -emulating real networks in the lab

Copyright 2005 OSDL, All rights reserved.

Netem emulating real networks in the labhttp://developer.osdl.org/shemminger/netem/

[email protected] April 2005

Stephen Hemminger

Copyright 2005 OSDL, All rights reserved. 2

Netem

Lots of new protocols and services are being researched– TCP congestion control, DCCP, ...

But:– lab environment is not same as real world– not everyone can access research networks– budget is limited

Need simple open source tool


Netem

Motivation – what is this good for? Implementation – how does it work? Usage – how do you use it? Examples – what did I find with it? Problems – what is still broken Q&A


Network delay

Speed of Light 3 x 108

Router Performance Business based routing

– Home to OSDL via Chicago?

Sydney to: km msCanberra 300 1Perth 4,100 14Los Angeles 12,050 40Frankfurt 21,376 71Moon 384,000 1,200


Design Goals

Start small with one thing Be a filter Generalize Robustness Get in mainline Reuse code


Other network emulators

NIST Net– patch for 2.4

Dummynet– part of FreeBSD

User space using tuntap (akpm)


Existing simulation tools

Simulation (ns2)– make model of protocol run in synthetic environment

User mode Linux (umlsim)– run mulitple kernels with synthetic network


Architecture

Queuing Disciplines– Linux Advanced Routing

and Traffic Control (LARTC)– Differentiated Services

Control flow of packets to network device

Protocol indendent


Advantages

Can plug into existing filtering Works with existing queuing and ratecontrol Can act as a bridge or router Code reuse allowed adding NIST Net functionality


Basic Usages

netem

TCP/IP

ethernet

TCP/IP

ethernet

iperf iperf


Useful for multi-flow and fairness testing

Receive1Source1

BridgeRouter

netem

Receive2Source2


Usage: command line

Add network delay

# tc qdisc add dev eth0 root netem \ delay 100ms 10ms 10% distribution experimental

100ms delay ± 10ms

statistical distribution from file

/usr/lib/tc/experimental.dist

tc → part of iproute2

root → base of tree

correlation


Bandwidth Delay Product (BDP)

BDP = amount of data in transit Examples

– DSL/Cable modem (international)1,000,000 bit/sec

* 1/8 byte/bit * .5 sec/bit = 62500 bytes

– Gigabit across US1,000,000,000 bit/sec

* 1/8 byte/bit * .07 sec/bit = 8.75 Mbytes


Bandwidth Delay Product

0.1 1 10 100 10000.1

1

10

100

1000

Delay (ms)

Ba

nd

wid

thM

bits

/se

c

8K1M64K

Broadband

ResearchLAN


Packet Loss

Typical Internet packet loss is 1 – 2%– Service Level Agreements typically allow .5%

Packets dropped at router– long drop tail queue

ISP's play games with traffic classification Wireless

– much higher packet loss due to signal and noise


Packet reordering

Closely spaced packets do get reordered:– Parallel path routers– Bad SMP implementations– Multiple paths (bonding)

Failover


Other options

Packet loss# tc qdisc add dev eth0 root netem loss .1%

Duplication# tc qdisc add dev eth0 root netem duplicate 2%

Reordering# tc qdisc add dev eth0 root netem gap 5 delay 10ms– Delays every fifth packet by 10ms


Nesting qdisc's

Using Token Bucket Filter to do rate control# tc qdisc add dev eth0 root handle 1:0 netem \

delay 100ms

# tc qdisc add dev eth0 parent 1:1 handle 10: tbf \rate 256kbit buffer 1600 limit 3000


Filtering qdisc structure

u32filter

Prio

Fifo

Netem

Fifo

TBF

eth0


• Using Filtering

# tc qdisc add dev eth0 root handle 1: prio

# tc qdisc add dev eth0 parent 1:3 handle 30: netem \

delay 200ms 10ms distribution normal

# tc qdisc add dev eth0 parent 30:1 tbf \rate 20kbit buffer 1600 limit 3000

# tc filter add dev eth0 protocol ip parent 1:0 prio 3 u32 \

match ip dst 65.172.181.4/32 flowid 10:3


•Usage – GUI

http://www.smyles.plus.com/phpnetemgui/


IANAS – I am not a statistician

Needed good source of random numbers for simulation– Replaced net_random() with Tausworthe from GNU scientific lib

Nonuniform random number– Use existing code from NIST net

• Supports normal (bell curve) and pareto (tail)• Convert experimental data into distribution

"Anyone attempting to produce random numbers by purely arithmetic means is, of course, in a state of sin." John Von Neumann.


Examples - Home DSL connection

0

10

20

30

40

50

60

70

80

90

100

0 20 40 60 80 100 120 140

Segm

ents

(cw

nd, s

sthr

esh)

time (seconds)

snd_cwndsnd_ssthresh


Examples – netem emulation of DSL

0

10

20

30

40

50

60

70

80

90

100

0 20 40 60 80 100 120 140

Segm

ents

(cw

nd, s

sthr

esh)

time (seconds)



Examples – unstable BIC (2.6.9)

0

5

10

15

20

25

30

35

40

45

50

0 5 10 15 20 25 30

Segm

ents

(cw

nd, s

sthr

esh)

time (seconds)



Examples – BIC still not right (2.6.10)

0

5

10

15

20

25

30

35

40

0 5 10 15 20 25 30

Segm

ents

(cw

nd, s

sthr

esh)

time (seconds)



Examples – correct (mostly) BIC (2.6.11.6)

0

200

400

600

800

1000

1200

1400

1600

1800

2000

0 5 10 15 20 25 30 35

Segm

ents

time (seconds)



TODO

Need more reordering options Should work with incoming as well (ingress) Better examples and website More documentation


Harder problems

Not all kinks worked out– Interaction of netem with assumptions of other qdisc

Timer resolution– limited to 1ms

Iproute2 (Netlink) interface awkward for GUI– plan to add /sysfs hooks to qdisc

Netem -emulating real networks in the lab

Technology

Transcript of Netem -emulating real networks in the lab