Experimental Assessment of Benchmark-orientedNetwork Traffic Generators

Vasileios PapadopoulosDepartment of Telematics Engineering

Universidad Carlos III de Madridvassilispapadop@gmail.com

Abstract—The burstiness of network traffic has a profoundimpact on the performance on many network protocols inareas such as, congestion control (eg. TCP), multiple-access (eg.CSMA/CA), routing (eg. BGP), switching and multiplexing ingeneral. Network measurement performance tools, have beenwidely used to exploit vulnerabilities, monitoring reports andtesting newly, under development protocols. Network TrafficGenerators (NTG), are capable of tracing burden from real-lifetraffic and replicate it offline, generating load under predefinedtraffic profiles such as, CBR, VBR and traffic according to wellknown probability distributions (Poisson process and Poissonianarrival). Though, it is not clear how accurately these tools areperforming, as little study has been done. A traffic generatoris required to satisfy hard real time requirements; time sensitiveapplications (eg. VoIP, Video), replication/emulation could lead todifferent than the expected experimental outcome, mainly becauseof the intrinsic limitations of the PC architecture and NTG’sdesign. Processes are competing each other for CPU attention,rising up uncertainties possibility and higher kernel overhead.The accuracy of timers , the network socket family and thepolicy of process scheduling should be consider as main sourcesof fluctuations and eventually bursty traffic generation. In thiswork, we use several well known statistical tools for capturinganomalies in frame generation process. Developers should takeinto account the rising limitations for better NTG design andimplementation.1

Keywords—Network traffic generators, iperf, mgen, c/rude, d-itg, burstiness, accuracy.

I. INTRODUCTION

The goal of this work is to present a framework capableof reporting NTG’s capabilities, performance, limitations anddesign strategies. We define traffic generation to result intimestamped series of packets arriving at and departing froma particular network interface with realistic values for at leastthe layer 3 (IP) and layer 4 (TCP/UDP) headers as well asthe whole ethernet frame (Preamble+Payload+Headers+IGP).This traffic should accurately reflect arrival rates and variancesacross a range of time scales, e.g capturing both averagebandwidth and burstiness. Most of traffic generators, reportaverage achievable bandwidth in relatively large time scaleshiding their inadequate to provide deterministic generationprocess or other user defined network traffic profiles. In thiswork we are focusing on possible bursty characteristics ofbirth/generation process originated from four widely used

1This project is considered as a M.Sc Thesis for the postgraduate studiesprogram of Universidad Carlos III de Madrid, Department of TelematicsEngineering. The work has been conducted under prof. Pablo Serrano (pabloat it.uc3m.es) supervision. It is fully funded by IMDEA Networks institute.July 2013, Madrid.

TABLE I. TRAFFIC GENERATORS CONSIDERED IN THE ASSESSMENT.

iperf mgen d-itg c/rude

Release date July, 2010 April, 2011 August, 2011 September, 2002Written in C++ C++ x CLicence BSD OpenSource GPL GPLv2Citations 4096 1024 512 512Platforms Cross Cross Cross Unix-like

traffic generators. Information regarding to NTGs are depictedto Table I. Our hypothesis is that realistic , responsive anddeterministic packet generation must be at least informed bythe hardware, software and protocols hosting the programs(NTG).

A. Why accurate generation is critical?

Protocol designers, network administrators and whom everbe concern, they all willing to generate network traffic underpre defined profiles for performance evaluation, maintenanceand protocol benchmarking etc. Testing tools have been pro-gressing over the years aiming to improve their functionalities,meeting quality guarantees at the highest level. From proto-col designers perspective such tools should be working justperfectly. Though, some factors could affect the performanceof tools such underneath platform, architecture design andeventually could lead to differently than the expected results.Software designers, aiming to produce applications havingalways extensibility in theirs user-requirements features. Allsoftwares nowadays are cross-platform. Although, for achiev-ing this, applications are designed to run at the applicationlayer increasing their interaction with underneath platform.That user-space to kernel-space interactions are needed forcritical operations, adding negligible overhead though. Themost widely used generators are running at user-space. Thus,not only their capability and design strategy but also systemkernel flexibility and performance could form a different thanthe expected experiment outcome.

Link burstiness can be considered as the operation regionoutside of the 95th percentile quality of service guarantee rule.But could greatly affect protocol performance. For example,MAC-layer immediate retransmissions are much less effectiveon links with bursty traffic and eventually bursts of losses thanon links with independent losses. This means that two networkscan have identical connectivity and packet reception ratio, yetexhibit completely different performance. Burstiness charac-terization will help to understand why some protocols behavedifferently on similar networks and will provide insights intotuning protocol parameters to improve performance. Though,all the analysis relies on the assumption that the generation

process doesn’t introduce by itself any bursty characteristics.As we will see later on this work, this is a weak assumption andmore than often someone would think, NTGs add backgroundnoise to frame generation process.

B. Related work

In work at [1], authors tried to address the similar questionas our work. They perform a comparative analysis in termsof achievable throughput and variance among several runs ofthe same experimental scenario. In [2], they perform packet-by-packet analysis and focusing on how the real-time kernelextensions could increase granularity of software timing re-quirements. They are also concerned of possible bursty trafficgeneration, although make use only coefficient-of-variancetechnique to capture such traffic. Additionally, authors in [3]explore any correlation in time interval between two consecu-tive packets. Though, they are focusing on the inter-departuretime as this observed at layer 3 (PF_FILTER).

C. Our Contribution

The question of how accurate your traffic generator tool hasbeen addressed before, the studies lacked from measurementaccuracy. Although, a fully test-bed description will followin next sections, it is worth mentioning that our work relieson dedicated hardware with nanosecond packet timestampingprocess accuracy. The sniffer is capable to introduce low, de-terministic processing delay when frames arriving to its ports.Additionally, we make use of several well know statistical toolsfor discriminating patterns in generation process and possiblebursty behaviour.

The rest of this work is organized as follows. At nextsection will discuss how and when a traffic flow could benot deterministic and we will present some techniques thatcould help us to capture the notion of bursty traffic flow.Then we present all the architectural limitations and concernsthat arising when traffic flow is generated by soft real timesystems instead of embedded systems. We present our test-bed and methodology and the main evaluation metrics that weuse to characterize flows. Finally, we replicate the laboratoryresults to a single server queueing system simulator, turningour attention on how the mean system residence time variesfrom the expected result.

II. NETWORK TRAFFIC

Understanding network traffic behavior is essential for allaspects of network engineering and operation. Componentand protocol design, management, modelling and simulationfeatures are all relying on deep understanding of either internetor local traffic. In this section, we will address the conceptof bursty behavior as this observed in ethernet single domainnetwork topology, as well as methods for identifying andcapturing such characteristics.

A. Notion of Burstiness

Traffic measurements and burden characterizations are vitalfor, the analysis and assessment of network performanceand for network design and cost optimization. Packets insome applications are expected to arrive at specified times

TABLE II. VARIANCE TO MEAN (VMR) FOR WELL KNOWNDISTRIBUTIONS.

VMR Description

Constant random variable VMR = 0 not dispersedBinomial distribution 0 ¡ VMR ¡ 1 under-dispersedPoisson Distribution VMR = 1 -Negative binomial distribution VMR ¿ 1 over-dispersed

(VoIP,Video). Although, the nature of NTGs and their un-derneath platform, could force packets to arrive in clusteredmanner. This phenomenon is known as packet burst. Burst, is a group of consecutive packets with shorter inter-packetgaps than packets arriving before or after the burst of packets.Burstinesss can be harmful to any traffic flow analysis ofnetwork assessment.

• Definition: A traffic process A is said to be morebursty than a traffic process B if a single server queue-ing system yields a longer system size tail distributionwhen the process A is used to drive the queueingsystem.

• Queue size normalized by service rate + service time= delay time.

• When buffer size is finite, system size tail distributionestimates packet loss probability.

B. Capture Techniques

There are a lot of techniques out there that are used todefine and measure burstiness. Some of them, provide limitedinformation about the flow and some are more complex.Peak-to-average ratio, is a simple measure to represent trafficburstiness. It provides a ratio of peak packet rate to the averagepacket rate over some specified averaging interval. It does notdescribe tendency in network traffic. Coefficient-of-variance(C.O.V), measures variability in relation to the mean. TheC.O.V is equal to zero for deterministic arrivals and one forPoisson arrivals, while bursty traffic has a C.O.V greater thanone. Table II depicts VMR for well known distributions. Al-though this approach is applicable for measuring the alterationin the data traffic it cannot capture the correlation betweenconsecutive packets. The correlation in the measured traffic canbe captured by calculating the variance of the time between narrivals and comparing it to n times the variance of successivearrivals. Indexes-of-dispersion also known as Fano Factor isa windowed analysis of variance normalized by the meanin particular interval. Finally, we could sense the notion ofburstiness of a flow based on its behavior through a networkelement, specifically on the queue size behaviour and averageresidence time.

III. ARCHITECTURAL LIMITATIONS

In general, time-stamping packets is tricky. For any trafficanalysis which requires high precision a way to accuratelymark packets is far from a simple requisite. The software-only implementation, timestamps packets in the applicationlayer which is furthest away from the physical layer. Thisimplementation yields the least accuracy due to the largestamount of delay and jitter that occurs in passing the time-stampthrough the various layers. Software timestamps typically giveerrors on the order of microseconds to milliseconds depending

on the operating system and platform. Such a timing mecha-nism provides no guarantees. Next we will discuss key factorsthat could yield to inaccurate time-stamp mechanism as wellas some architectural limitations that play significant role tosoftware timing requirements.

A. Scheduling

In multi-user environments, tasks and processes competingfor CPU attention. Catalytic role to resource assignment isplayed by an OS module, called scheduler. A scheduler, isliable to distribute CPU cycles among variety of tasks thatconcurrently have to be executed, according to a schedulingalgorithm. There are number of algorithms for scheduling taskson a single processor; some of them are used for schedulingtasks on multiprocessor system either under the partitioningscheme or under the global scheduling scheme. A partitioningscheme adds the constraint that all instances of a task mustexecute on the same processor. When scheduling hard real-timetasks on a multiprocessor system, it is important to ensure thatno task admitted in the system misses a deadline. This impliesthat a stringent admission control is required for hard real-timesystems [3]. A real-time system is one in which tasks that mustgenerate functionally results in a timely manner. This connotesthat the tasks submitted to the system have known timing ne-cessities. These tasks are called real-time tasks. In general, RT1 kernel extensions are used to integrate better to applicationswith strictly timing requirements. The OS scheduler is loadedas a process and could be preempted by other high prioritytask. This functionality could decrease system-kernel overhead.A NTG, is a software system consisting of several strictlyreal-time tasks. However, they rely on scheduling policies andlimitations of each particular platform. Kernel has to reactand make decisions immediately, without adding extra queuewaiting time to currently active/ready processes. The scheduleris concerned essentially with :

• Throughput - The total number of processes thatcomplete their execution per time unit.

• Turnaround time - Is the difference of time betweenthe time of arrival and time of dispatch/completion ofa process.

• Response time - amount of time it takes from whena request was submitted until the first response isproduced.

• Fairness - Equal CPU time to each process.

In practice, no matter which scheduling policy a kernel isdeploying, it is hard to meet service guarantees for strict real-time applications in soft real-time environments. Although,some policies can favourite processes which are having timingconstraints. In our concern, this translates into time offset offrame arrival from its theoretical point. A lot of experimentalanalysis has been performed relying on soft-time platforms,among several networking disciplines . Even though, the NTGis running as the process with the highest priority in the system,system has to take CPU cycles/time for its operations. NTGwill suffer by OS event scheduling accuracy and probablywould face de-scheduling; the testing tool makes an OS systemcall to some form of textttsleep or time-slice expiration. In fact,traffic generators has to give up by cpu contention process

every time the next sending packet that has to be addressedit would happen according to user-defined time between twoconsecutive packets.

B. Socket Family

The idea of a socket is to plug-in to the other processand send the message, with the operating system taking careof delivering it. The socket layer supports communicationbetween processes that are running, either on the same ma-chine or different machines. Sockets can be opened usinga number of different protocols including the Unix socketfor intra-machine communication and TCP sockets for inter-machine communication. The operating system provides aset of primitives that enables the computer to communicatewith other machines using standard protocols such as InternetProtocol (IP), Novell IPX, AppleTalk, and Microsoft SMB.These higher-level protocols are, in turn, built upon Ethernet(IEEE 802.3) or other Media Access Protocols. The operatingsystem primitives include device drivers for the communi-cation hardware, packet routing and filtering functions, andimplementation of the protocol stacks such as TransmissionControl Protocol/Internet Protocol (TCP/IP) . Typically, forapplications which have not tight timing requirements, thechoice of selecting socket family is based on functional criteria.Although, a NTG has to choose a family that guaranteesthe lowest time overhead and performance. The authors at[5], have analyzed two commonly used families of socketsavailable in Linux kernel 2.4: the INET and the PACKETsocket families. They have measured the cpu-cycle-latency andfound that, the PACKET family approach can reduce CPUcycles up to 50% compared to INET. PACKET family expecta complete Ethernet frame that is created at user-space layer.On the other hand, the INET socket family performs the setof operations required to construct UDP/IP and MAC headers.

C. Timers Accuracy

The system clock plays an important role in resourcemanagement, particularly in real-time applications. The clockallows applications and the kernel to query the current time,which is crucial for dealing with time constraints. It alsoprovides an alarm clock, which the operating system and appli-cations use to be notified at specified times. High performancetiming is tricky. Timing is one of those things where a littleknowledge can be problematic. A common way for doingtiming is to use functions like gettimeofday(). Such functionsreturn what is called a wall time, which is the time thatcorresponds to a calendar date/time. These clocks suffer fromsome limitations. They have low resolution. A NTG can notrely its high performance timing operations on low granularity.Furthermore, the clock can jump forwards and backwards intime causing a time drift. Some systems have NTP 2 enabledwhich periodically adjusts the system clock to keep it in syncwith actual time. A different approach is to use system clock.The application has to access the clock value and the rate atwhich it increments. The increment rate is the cpu frequency.Subtracting two successive clock reads and dividing the resultto clock frequency, will give as the floating point interval inseconds. A more complex approach to provide high accuracy

2Network Time Protocol (NTP) is a networking protocol for clock synchro-nization between computer systems over data networks.

timing guarantees to applications with high time sensitivity,is the use of Read Time-Stamp Counter (RDTSC). RDTSCprovides access to the current value of the processors time-stamp counter to upper layers. Although, several concerns arearising. The counter is processor core specific, meaning thatusing RDTSC will give different values on different processorcores. This causes non-monotonic clock behavior as a threadis migrated across cores during execution. It doesnt alwaystick at the same rate, as the clock frequency will vary as theCPU load changes. This due to power saving features in theprocessor that throttle down the clock speed when the loadis low. A convenient way when using RDTSC is to set CPUAffinity to NTG.

D. Is a modern OS able to keep up the rate ?

Network drivers are far from the simplest drivers in thekernel. They push the speed boundaries of modern multi-coreprocessors. Almost every new system comes with a 1Gigabitper second (Gbps) network card. Taking into account that thesmallest Ethernet frame size is 64 bytes (plus 8 bytes forsynchronization and a 12 byte inter-frame gap), a 1 Gbpsnetwork card should (by specification) be able to handle:

1Gbps/672bits/frames = 1.488Mframes/second.

This leaves us with a processing time of about 672 nanosec-onds per frame. A 2 Gigahertz processor can execute about1,400 cycles in that time and a single instruction can takemultiple cycles. Is a modern operating system able to keep upwith that rate? The simple answer is: no. The standard Linuxkernel cant keep that pace for an extended period of time as itrelies on buffering to handle short bursts of traffic beyond whatit can handle. It still does its best to get as many packets aspossible using modern network card resources and the multi-core processors available on the machine.

IV. TEST-BED AND METHODOLOGY

Even though our work is intended to be applied to everynetwork deployment, such as wired or wireless infrastructures,we have been following the most simple way. The goal of ourmethodology is to fulfill the following principle : Experimentsmust be repeatable and to guarantee reproducibility of theresults. For that reason, we decide to choose a single domainwired network (LAN) instead of an isolated wireless network(WLAN), mainly because a radio environment makes repeat-able experiments harder to achieve. The impact of stochastic

Network TapTraffic duplication

High precisiontraffic sniffer

Traffic generators poolSender

Traffic generators poolReceiver

Fig. 1. Test-bed overview.

Packets Network Card

Card Driver

Packet Filter

ProtocolStack

LibpcapSniffer

NetworkMonitor

WebBrowser

Web Appications

Hardware Kernel Space User Space

TransmittedPacket

ReceivedPacket

Fig. 2. Elements involved in capture process.

TABLE III. USE CASES AND EVALUATION METRICS.

Bandwidth levels Frame Size (Bytes) Evaluation Metric

25 100B, 1470B Payload Auto-correlation50 UDP + IP + Ethernet Headers Dynamic Allan Variance90 Preamble + IGP Link 1st Order Burst Metrics

100 Resource Management

factors is closely bound to the variance of the outcome.Medium contention parameters in typical 802.11 environment,are used for achieving network stability and fairness amongstations. Though, this some sort of randomness could harmthe main idea behind of this work. Furthermore, as we will seelater on this paper, our accuracy granularity reaches nanosec-onds resolution, which makes a wired network deployment themost suitable for our needs. The figure 1, depicts our laboratorytest-bed. Two identical PCs are powered up by Intels Core 2Quad CPU , clocking @ 2.66 GHz with 4 GB of RAM. Twowired cards, namely, one integrated Intel 82567 GigabitEthernet card, and one PCI RealTek, are interconnect thetwo machines.

The accuracy of the measurement tool is a critical factorwhen performing experimental work figure 2. The work in [4]has showed that any kind of network traffic sniffing tool shouldnot be trusted. Commonly used sniffing software, tsharkrelies on libpcap and PF_FILTER for capturing packets.Although, tshark provides plenty of features for network analy-sis, lacks of timestamp accuracy and precision. For the reasonsdescribed above, a good choice was to install to our test-beda network tap 3[11]. A network tap is able to capture trafficin high speed networks with extremely high time-stampingprocess accuracy.

Our measurement methodology, based on the analysis ofthe time interval between to consecutive frame arrivals, as thisobserved by the sniffer. The packet generator machine, injectstraffic to the link with fixed frame length and constant bit rates.The table III depicts the different use cases we used for ouranalysis and evaluation approaches. The same experimentalscenario was repeated 5 times and the duration of each run is30 seconds.

A. Considered Network Traffic Generators

1) IPERF: IPERF [6] is a bandwidth measurement toolwhich is used to measure the end-to-end achievable bandwidth

3NetOpitcs tap and the Endace DAG card

by using TCP/UDP streams. It allows tunings in parameterslike TCP window size and number of parallel streams. End-to-end achievable bandwidth is the bandwidth at which an ap-plication in one end-host can send data to an application in theother end-host. IPERF approximates the cumulative bandwidth(the total data transferred between the end-hosts over the totaltransfer period) to the end-to-end achievable bandwidth. IPERFis used widely. The popularity and widespread use can also bepartially attributed to its ease of installation and absence ofkernel or device driver modifications.

2) MGEN: MGEN [7] provides the ability to perform IPnetwork performance tests and measurements using UDP andTCP IP traffic. The application itself is an extension of ProteanProtocol Prototyping Library (Protolib), which provides aC++ framework for network protocols development. MGEN,implements traffic control and network management. It issuitable for low power capability systems with huge kerneloverhead.

3) C/RUDE: Next traffic generator considered in this as-sessment is C/RUDE [8]. C/RUDE originally designed anddeveloped to outperform some timing limitations that MGENtends to ignore. MGEN rely its operation on systems timer (see.Architectural Limitations : Timers ) which only provides timerresolution up to milliseconds magnitude.

4) D-ITG: D-ITG [9] is the most sophisticated traffictool. Is a platform capable to produce traffic at packet levelaccurately replicating appropriate stochastic processes for bothIDT (Inter Departure Time) and PS (Packet Size) randomvariables (exponential, uniform, cauchy, normal, pareto, etc.)

V. PERFORMANCE ASSESSMENT

In this section, we present the results of tests carried outin a test-bed running Linux kernel 2.6 [12]4. With these testswe intend to determine the maximal performance of all TGsand evaluate the stability and the reliability of the generatedpacket flows. Furthermore, we turn our focus on how far NTGsinject packets to the network from the optimal point of framegeneration. Section III, presents some common limitations toframe generations and shows, how a bad component choicecould turn against in such tools operations. Several well knownmodels are used to characterize as accurate as possible thegenerators.

The Figure 3 presents the CDF for two use cases. Thesaturation case for 100B and 1470B payload as well as 90 ofmaximum achievable bandwidth. The vertical line representsthe theoretical interpacket gap that frames should arrive at theother end of the communication link. An important observationis that all traffic generators fail to reach up to theoretical pointof packet generation. This it has to deal with Linux’s kernelnature and the overhead that it is introduced by its modulesfor operations such as scheduling etc. It is intuitively clearfrom Figures 3 and 4 that we could categorized the NTGs intotwo different generation policies software-based tools. IPERFand C/RUDE try as much as possible to inject traffic intoa more deterministic manner in contrast to MGEN and D-ITG which tend to deploy generation control mechanisms to

42.6 considered as one of the lowest in terms of latency/overhead Linuxkernel.

Left 100B, right 1470B. Top saturation, bottom 90%

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1

13.2 13.25 13.3 13.35 13.4 13.45 13.5

CD

F

all TGs

IPERF

MGEN

RUDE

D-ITG

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1

122.5 123 123.5 124 124.5 125

IPERF

MGEN

RUDE

D-ITG

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1

12 13 14 15 16 17 18 19

CD

F

inter-packet gap (us)

IPERF

MGEN

RUDE

D-ITG

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1

130 132 134 136 138 140

inter-packet gap (us)

IPERF

MGEN

RUDE

D-ITG

Fig. 3. Cumulative distribution for saturation and 90. The vertical line hintsthe optimal rate of packet reception.

Left 100B, right 1470B. Top 50%, bottom 25%

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1

15 20 25 30 35 40 45 50 55

CD

F

IPERF

MGEN

RUDE

D-ITG

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1

235 240 245 250 255 260

IPERF

MGEN

RUDE

D-ITG

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1

20 30 40 50 60 70 80 90 100

CD

F

inter-packet gap (us)

IPERF

MGEN

RUDE

D-ITG

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1

480 485 490 495 500 505 510 515 520

inter-packet gap (us)

IPERF

MGEN

RUDE

D-ITG

Fig. 4. Cumulative distribution for 50 and 25. The vertical line hints theoptimal rate of packet reception.

Instantaneous Inter-packet Gap for 1470B, 90%

100

1000

1 10 100 1000 10000

IAT

(us

)

IPERF

ObservedExpected

100

1000

1 10 100 1000 10000

MGEN

ObservedExpected

100

1000

1 10 100 1000 10000

IAT

(us

)

Inter-packet gap #

RUDE

ObservedExpected

100

1000

1 10 100 1000 10000

Inter-packet gap #

D-ITG

ObservedExpected

Fig. 5. Overview of observed inter-arrival time at sniffer versus expectedvalue. Use case : 1470B, 90 saturation.

Instantaneous Inter-packet Gap for 100B, 90%

10

100

1 10 100 1000 10000 100000

IAT

(us

)

IPERF

ObservedExpected

10

100

1 10 100 1000 10000 100000

MGEN

ObservedExpected

10

100

1 10 100 1000 10000 100000

IAT

(us

)

Inter-packet gap #

RUDE

ObservedExpected

10

100

1 10 100 1000 10000 100000

Inter-packet gap #

D-ITG

ObservedExpected

Fig. 6. Overview of observed inter-arrival time at sniffer versus expectedvalue. Use case : 100B, 90 saturation.

achieve on larger time scales more accurate injection. Theyintroduce scaling generation management to balance out anypossible drift to the birth process. Figures 5 and 6 hint outthe different approaches deployed by NTGs. Even thoughthe above analysis shows that all traffic generators losing itsaccuracy and there is a drift to observed inter-packet gaps atthe sniffer side, it is not clear whether there is a correlationor any persistence pattern to frame generation process. Nextsections try to tackle this.

A. Auto-correlation

Auto-correlation refer to the correlation of a time serieswith its own past and future values. Positive auto-correlationmight be considered a specific form of persistence, a tendencyfor a system to remain in the same state from one observationto the next.

R(k) =∑

i(Ti − Ttx)(Ti+k − Ttx)/(n− k)σ2

In contrast, negative correlation hints out a tendency foroscillation among different system states. We define the error eas the distance of an observed inter-packet gap from the meanand we calculate the auto-correlation with the above equation.We are interesting in capturing consecutive inter-packet gapswhose absolute distance of the probability distribution meanis equal.

Auto-correlation for 100B, 90%

-0.2

0

0.2

0.4

0.6

0.8

1

0 10 20 30 40 50 60

R(k

)

IPERF

-0.2

0

0.2

0.4

0.6

0.8

1

0 10 20 30 40 50 60

MGEN

-0.2

0

0.2

0.4

0.6

0.8

1

0 10 20 30 40 50 60

R(k

)

lags

RUDE

-0.2

0

0.2

0.4

0.6

0.8

1

0 10 20 30 40 50 60

lags

D-ITG

Fig. 7. Auto-correlation of the errors. Use case : 100B, 90 saturation.

B. Dynamic ALLAN variance

In this section we borrow basic concepts and principlesfrom clock stability assessment. We treat NTGs as a clock,whose clock tick rate is the frame generation rate. Ideally,every clock should keep its stationary properties throughoutits life period. Well tuned clocks, produce white frequency,white noise.

Non-stationary Clock Analysis: The mean and the varianceof the underlying process are not constant and the auto-covariances dont depend only on the time lag. A perfect tunedclock could be able to produce white frequency noise whosestandard deviation and mean remain constant over time. Ifonly white noise is present in signal, then the noise will bereduced by the square root of the number of points usedfor the averages. But, even if exists a low frequency drift indata, then after a certain number of averages the coefficientis shifted because of the drift of the data. The DAVAR is inpractice obtained by sliding the estimator of the Allan varianceon the data. We also define the dynamic Allan deviation,or DADEV, as the square root of the DAVAR (the DADEVestimator is defined in an identical way). There is a typical

Dynamic Allan variance for 100B, 90%

1e-11

1e-10

1e-09

1e-08

1e-07

1e-06

1e-05

1 10 100 1000 10000 100000 1e+06

sigm

a

IPERF

1e-22

1e-20

1e-18

1e-16

1e-14

1e-12

1e-10

1e-08

1e-06

0.0001

1 10 100 1000 10000 100000 1e+06

MGEN

1e-24 1e-22 1e-20 1e-18 1e-16 1e-14 1e-12 1e-10 1e-08 1e-06

1 10 100 1000 10000 100000 1e+06

sigm

a

Average time (tau)

RUDE

1e-24 1e-22 1e-20 1e-18 1e-16 1e-14 1e-12 1e-10 1e-08 1e-06

0.0001

1 10 100 1000 10000 100000 1e+06

Average time (tau)

D-ITG

Fig. 8. Dynamic Allan variance. Use case : 100B, 90 saturation.

tradeoff in the computation of the dynamic Allan variance. Ifthe window is long, the variance of the estimate is small, butthe localization of events in time is poor. Conversely, a shortwindow guarantees an excellent localization of events, but hasa poor variance reduction. It is better to choose the window ona case-by-case basis, depending on the type of data consider.The dynamic Allan variance can be applied in several differentways. In general one can think of two ways of using DAVAR:

• To prove that a clock is behaving in a non-stationaryway and to identify the types of non-stationarities thatare presented.

• To guarantee that a clock is behaving in a stationaryway. A clock may in fact fail the requirements locallybut meet the global requirements specified by the clas-sical Allan variance. This violation of the requirementscan be revealed only by showing the dynamic Allanvariance.

C. Burst Length

Our analysis in previous sections have shown that, noneof the four traffic generators are capable to produce traffic indeterministic manner. In fact we can conclude that all NTGscan be considered to behave in the same way, when we lookin higher time scales. Dynamic Allan variance method giveus the intuition that, NTGs meet global requirements but notlocally. But, whats the significance of this non deterministicbehavior? To address this, we developed an algorithm whichruns in Linuxs shell, for capturing any possible packet trainswithin the generated process. At this point it is convenientto introduce some of the basic tuning parameters of burstdetection algorithm.

• ta - inter-arrival time of packets in burst.• dta - inter-arrival time tolerance.• pi - a size of a packet.

It is intuitevely clear, tuning parameterization diversity whichis consisting of the above factors, could give a completely dif-ferent burst length distribution and characterization. By settingthe inter-arrival time at certain level we differentiate certainnumber of bursts. The inter-arrival time tolerance variable isused to form up a capture band, a zone in which the algorithmmarks packets.

Many burst detection algorithms are based on spike anal-ysis and use a window to integrate a spike train signal and athreshold to detect a burst occurrence. Thus, important tuningparameter for our algorithm, is the process trimming level.At first glance, threshold can be defined as the theoreticalarrival rate. Packets tend to arrive to the other end of thecommunication link in clustered manner. Every observed inter-packet gaps that is below a threshold is candidate for start oftrain. To indicate the end of a cluster, a frame has to arrive lateror equal to the considered threshold. An algorithmic overviewof code is presented above. It is worth to mention that ourwork is based on post-processing analysis. A real-time spikedetection software development is at the top of our future workqueue.gap← GAPS.firstwhile GAPS do

Cumulative Distribution Function for 100B, 90%

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1

1 10 100 1000 10000

CD

F

IPERF

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1

1 10 100 1000 10000

MGEN

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1

1 10 100 1000 10000

CD

F

Burst length

RUDE

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1

1 10 100 1000 10000

Burst length

D-ITG

Fig. 9. Burst length CDF for 5 measurements. Use case : 100B, 90.

Cumulative Distribution Function for 100B, 50%

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1

1 10 100 1000 10000

CD

F

IPERF

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1

1 10 100 1000 10000

MGEN

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1

1 10 100 1000 10000

CD

F

Burst length

RUDE

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1

1 10 100 1000 10000

Burst length

D-ITG

Fig. 10. Burst length CDF for 5 measurements. Use case : 100B, 50.

if gap 6 trim− level thenconsecutive− frames← ++

elseif consecutive− frames ≥ 2 then

burst− length[consecutive− frames]← ++end ifconsecutive− frames← 0

end ifgaps← GAPS.next

end while

As we can observe from figures 9,10,11,12 traffic genera-tors produce different burst lengths characteristics. Dependingthe bandwidth level and the size of frame, some generatorsoutperform others by injecting network traffic more accuratelyand in more deterministic manner. The slope of CDF lines,intuitively point out the distribution tail size of the burst lengthanalysis. An extension to previous approach, is to expand theclassic burst algorithm detection by adding functionality inwhich measure the depth of the burst. From queueing the-ory perspective, smaller inter-packet gaps leading to differentsystem states, as increasing the average residence time forincoming packets and delay. The packet loss probability de-

Cumulative Distribution Function for 1470B, 90%

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1

1 10 100 1000 10000

CD

F

IPERF

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1

1 10 100 1000 10000

MGEN

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1

1 10 100 1000 10000

CD

F

Burst length

RUDE

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1

1 10 100 1000 10000

Burst length

D-ITG

Fig. 11. Burst length CDF for 5 measurements. Use case : 1470B, 90.

Cumulative Distribution Function for 1470B, 50%

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1

1 10 100 1000 10000

CD

F

IPERF

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1

1 10 100 1000 10000

MGEN

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1

1 10 100 1000 10000

CD

F

Burst length

RUDE

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1

1 10 100 1000 10000

Burst length

D-ITG

Fig. 12. Burst length CDF for 5 measurements. Use case : 1470B, 50.

TABLE IV. 1ST ORDER BURST METRICS.

metric iperf mgen c/rude d-itg

100B, 90% Length (ms) 2.4178 0.1026 7.026 0.075Size (KB) 26.8836 1.11929 80.9527 0.78109

100B, 50% Length (ms) 0.8653 2.1554 5.5174 8.7964Size (KB) 0.78109 0.571852 34.3898 1.47691

100B, 25% Length (ms) 0.3743 46.3683 0.3464 45.484Size (KB) 0.81594 1.96481 0.837289 2.07256

1470B, 90% Length (ms) 1.1727 1.4201 1.2706 2.0391Size (KB) 9.97668 7.18811 12.9837 5.02086

1470B, 50% Length (ms) 1.8307 6.1344 2.7212 5.6685Size (KB) 8.96854 5.96565 15.8702 5.05238

1470B, 25% Length (ms) 1377.32 4.7424 7.4846 12.7318Size (KB) 4.50435 8.68084 22.1778 4.67261

pends on particular network devices capabilities. Small buffersizes offer in general low delay but are inadequate to absorbany packet train occurrence in the network increasing lossprobability as well. The table IV summarizes the results of thedefault configuration of burst detection algorithm. We presentthe average length of burst expressed in time (ms), the size inbytes during the experiment. The results are averaged out total5 repetitions of same use case scenario.

CPU cycles 100B, 100%

0

1e+10

2e+10

3e+10

4e+10

5e+10

6e+10

7e+10

8e+10

9e+10

IPERF MGEN RUDE D-ITG

CP

U C

ycle

s

UserKernel

Total

Fig. 13. CPU cycles as measured by PERF tool. Use case : 100B, saturation.

VI. RESOURCE MANAGEMENT

A. Consumption

Availability of system resources is critical for any eitheruser-space or kernel space process. In this section, we pre-liminarily present base resource consumption as this capturedby means of number of cpu cycles that are required for eachexperiment completion. According to authors knowledge aproper way to measure cpu consumption is to have privilegedaccess to source code, for manipulating it in way that couldreport consumption at specified part of the code. Though, thisis included as part of our future work.

We could distinguish NTGs, as tools that are self-orientedand rely most of the required frame generation and transmis-sion functionality in a selfish manner and system-orientedwhich are more platform dependent. The former perform asmuch as possible user-space calls, decreasing its dependencewith underneath platform while the latter triggers and relocatesfunctionality to kernel.

B. L2 Transmission Queue

Before presenting our last evaluation metric, it is conve-nient to preliminarily present how Linux handles outgoingframes at layer 2.

The main function of the kernel at the link layer isscheduling the packet to be sent out. For this purpose,Linux uses the queueing discipline (struct Qdisc) abstrac-tion. The dev_queue_xmit method puts the sk_buffon the device queue string using qdisc-¿enqueue virtualmethod. If it is necessary (when the device doesn’t sup-port scattered data) the data is linearised into the sk_buffbut this requires copying the data to different memorybuffer. Several Qdisc scheduling policies exist. The basicand most used on is pfifo_fast , which has threepriorities granularity. The device output queue is immedi-ately triggered with qdisc_run() method and it callsqdisc_restart(), which takes as sk_buff from thequeue using qdisc-¿dequeue virtual method. Specific queueingdisciplines may delay sending by not returning any sk_buffand setting up a qdisc_watchdog_timer() instead.

Backlog Packets at L2 Buffer in Linux Kernel. 100B Saturation

0

50

100

150

200

250

300

0 200 400 600 800 1000 1200 1400 1600

Pac

kets

in T

XQ

UE

UE

IPERF

-1

-0.5

0

0.5

1

0 200 400 600 800 1000 1200 1400 1600

MGEN

0

50

100

150

200

250

300

0 200 400 600 800 1000 1200 1400 1600

Pac

kets

in T

XQ

UE

UE

Relative observation time

RUDE

0

50

100

150

200

250

300

0 200 400 600 800 1000 1200 1400 1600

Relative observation time

D-ITG

Fig. 14. How TGs handling layer 2 transmission queue. Use case : 100B,saturation.

Backlog Packets at L2 Buffer in Linux Kernel. 1470B Saturation

0

5

10

15

20

0 500 1000 1500 2000 2500 3000 3500 4000

Pac

kets

in T

XQ

UE

UE IPERF

-1

-0.5

0

0.5

1

0 500 1000 1500 2000 2500 3000 3500 4000

MGEN

0

5

10

15

20

0 500 1000 1500 2000 2500 3000 3500 4000

Pac

kets

in T

XQ

UE

UE

Relative observation time

RUDE

0

5

10

15

20

0 500 1000 1500 2000 2500 3000 3500 4000

Relative observation time

D-ITG

Fig. 15. How TGs handling layer 2 transmission queue. Use case : 1470B,saturation.

When the timer expires, netif_schedule() is called tostart transmission. Eventually, the sk_buff is sent withdev_hard_start_xmit() and removed from the Qdisc.If sending fails, the skb is re-queued. netif_schedule()is called to schedule a retry. The later method raises asoftware interrupt, which causes net_tx_action() tobe called when the NET_TX_SOFTIRQ is ran by ksoftirq.net_tx_action() calls qdisc_run() for each devicewith an active queue. Since all traffic generators consideredin this work are user space applications, we argue that when apacket reaches at the outgoing transmission queue inside thekernel, the generator is not involving at this part of creationand packet transmission process. As we described above, thesystem is responsible to handle the interrupts and hardwareneeds for actual frame departure. Though, using tc (TrafficControl) [10] tool we have observed that each TG handlesdifferently the outgoing buffer. MGEN is more user spaceoriented software. It is not rely on underneath system and needto have more control of generation process. On the other hand,IPERF, RUDE and D-ITG are depend on system and generatebursty traffic inside the kernel, shifting responsibility towardsto operating system to meet timing restrictions.

VII. CONCLUSIONS

As underlined in the introduction, bursting activity playsan important role during to any under development MAC pro-tocol, and it is greatly influenced by network traffic generatorimplementation and design. A first step in the characterizationof the network behaviour is the capability to reliably detectand characterize such an activity. The choice of these toolsis motivated by the fact that they produce very specific burstpatterns, as shown by the spike trains represented above.

In this work, we analysed the behaviour of widely usedby research communities network traffic generators. We haveshown that for particular experimental scenario and differentnetwork tuning, one NTG could outperform the others interms of flow reliability and determinism by generating moreaccurately the traffic profile that has been defined by the user.

REFERENCES[1] Kolahi, S. S., Narayan, S., Nguyen, D. D., Sunarto, Y., Performance

monitoring of various network traffic generators. 2011 UkSim 13thInternational Conference on (pp. 501-506). IEEE.

[2] Paredes-Farrera, M., Fleury, M., Ghanbari, M., Precision and accuracyof network traffic generators for packet-by-packet traffic analysis. InTestbeds and Research Infrastructures for the Development of Networksand Communities, 2006. TRIDENTCOM 2006. 2nd International Con-ference on (pp. 6-pp). IEEE.

[3] Botta, A., Dainotti, A., Pescap, A., Do you trust your software-basedtraffic generator?. Communications Magazine, IEEE, 48(9), 158-165.

[4] Serrano, P., Zink, M., Kurose, J., Assessing the fidelity of COTS 802.11sniffers. In INFOCOM 2009, IEEE (pp. 1089-1097). IEEE.

[5] Bonelli, N., Giordano, S., Procissi, G., Secchi, R. BRUTE: A highperformance and extensible traffic generator. In Proc. of SPECTS (pp.839-845).

[6] Iperf, http://sourceforge.net/projects/iperf/. -.[7] Mgen, http://pf.itd.nrl.navy.mil/mgen/mgen.html. -.[8] Rude, http://rude.sourceforge.net/. -.[9] D-itg, http://traffic.comics.unina.it/software/ITG/. -.[10] Traffic control, http://linux.die.net/man/8/tc. -.[11] Network Tap, http://www.network-taps.eu/home/home.php -.[12] List of Linux kernel, https://www.kernel.org/ -.