A MEASUREMENT-BASED SIMULATION MODEL OF A WEB CLUSTER REPRESENTINGSYSTEM AND APPLICATION LEVEL DYNAMICS

Isabel Dietrich, Kai-Steffen Hielscher, Reinhard German

Department of Computer Science 7University of Erlangen-Nurnberg

D-91058 Erlangen, Germany{isabel.dietrich, ksjh, german}@informatik.uni-erlangen.de

Abstract— We present a simulation model of a Webcluster which includes important aspects of the hardware,operating and communication systems as well as theapplication layer for realistic performance evaluations. Weformulate the model in UML taking into account botharchitectural and behavioral aspects and apply detailedsystem-level measurements at low loads for determiningparameters. Measured one-way delays are used to charac-terize the load on the network. We apply advanced inputmodeling techniques including multi-modal distributions,multiple phases, and Bezier curves for unconventionalshapes to represent all quantities. We also propose a newmodel based on differences between successive delays formodeling the autocorrelations observed during the mea-surements. Several internal characteristics of the clusterwhich significantly affect the behavior at higher loads,including buffering, resource contention, and transportlayer mechanisms, are modeled in detail in our UMLsimulation. The model is able to predict the quantilesof the overall delay for HTTP responses of differentconfigurations beyond the laboratory setup. Replacing alldistributions with exponential ones with same means leadsto significant deviations, demonstrating the need for seriousinput modeling.

Index Terms— Measurement based simulation, discreteevent simulation, input modeling

I. INTRODUCTION

Clusters of commodity PC hardware are a popularmethod to cope with the performance requirements forweb servers due to the growing demand for rich mediain the web. A lab installation helps us to evaluate theperformance implications of this kind of server archi-tectures. We use a measurement infrastructure basedon GPS and kernel modifications which allows us tomonitor the system at a low level with high precision.This infrastructure is used to observe system parameterssuch as one-way delays at low loads. According tothe nature of the measurements, these delays implicitlyinclude some side-effects such as interrupt latencies andshow random behavior.

Emphasis is put to apply advanced input modelingtechniques in order to adequately represent these basicparameters. We consider multi-modal delays, as well asphased multi-modal delays, and the inclusion of bezierdistributions. We also present a new approach to modelauto-correlations observed in the measurements.

We developed a simulation model based on UMLconstructs which explicitly contains mechanisms such asqueuing in buffers, transport control mechanisms, andcontention for CPU power with other processes. The

model thus combines a precise stochastic representationof low-level system parameters with an explicit repre-sentation of system behavior at higher observable levelssuch as queuing and contention for the CPU.

Equipped with such a fine-tuned detailed model wecompare performance figures measured in the actualsystem with those predicted by the model. The match isgood for overall delay quantiles. Replacing the detailedinput distributions by exponential distributions with thesame mean shows significant differences, thus illustrat-ing the need for serious input modeling.

II. RELATED WORK

Performance analyses for web servers are included inthe work of numerous authors. Most of them modelthe system at higher levels of abstraction, e.g. [1].More detailed simulation models of web clusters are forexample included in [2]. Their simulation study is basedon a detailed model for the hardware of the system, butdoes not include fine-grained measurements of delaysinside the system. The authors of [3] investigate theeffect of different load balancing strategies on cluster-based web servers using both a laboratory cluster and asimulation model. However, they employ only high-levelmeasurements of the load balancer and server servicetimes. In [4] the performance of several load balancingschemes is evaluated. They use traces of the arrivalprocesses as input for their simulation model, but allother aspects of the overall system performance are notbased on measurements.

III. THE WEB CLUSTER

Our laboratory setup includes open-source softwarefor the operating system of the cluster nodes, the webserver daemon and the load balancing solution. Allsoftware is used on standard off-the-shelf PC hardware.

The Linux Virtual Server [5] system is used for loadbalancing in our web cluster. The whole distributed webserver carries one single IP address called Virtual IPAddress (VIP). Requests sent to this address are balancedamong the real servers carrying the Real IP Addresses(RIPi). Using the NAT strategy, incoming packets firstarrive at the load balancer node. This node then changesthe destination IP address (VIP) in the TCP segments tothe IP address of one of the real server nodes (RIPi) asdetermined by the scheduling algorithm. The load bal-ancer is configured as the standard gateway for the reply

Fig. 1. Illustration of the Delays

Delay 1

Delay [µs]

Den

sity

60 80 100 120 140 160

0.00

00.

006

Delay 2

Delay [µs]

Den

sity

0.7 0.8 0.9 1.0 1.1

05

1015

Delay 3

Delay [µs]

Den

sity

52 54 56 58 60 62 64 66

0.0

0.2

Delay 4

Delay [µs]

Den

sity

6 8 10 12 14

0.0

0.2

0.4

0.6 Delay 5

Delay [µs]

Den

sity

60 80 100 120 140 160

0.00

00.

010

Delay 6

Delay [µs]

Den

sity

1.05 1.10 1.15 1.20 1.25 1.30

05

1020

Delay 7

Delay [µs]

Den

sity

40 60 80 100 120 140 160

0.00

00.

008

Delay 8

Delay [µs]

Den

sity

20 25 30 35 40 45

0.00

0.10

0.20

Delay 9

Delay [µs]

Den

sity

60 80 100 120 140 160

0.00

00.

010

0.02

0

Delay 10

Delay [µs]

Den

sity

0.55 0.60 0.65 0.70

010

30

Delay 11

Delay [µs]

Den

sity

65 70 75 80

0.00

0.10

0.20 Delay 12

Delay [µs]

Den

sity

250 300 350 400 450 500 550

0.00

00.

010

Delay 13

Delay [µs]

Den

sity

260 280 300 320 340 360 380

0.00

00.

006

Delay 14

Delay [µs]

Den

sity

1.10 1.15 1.20 1.25

05

1525

Delay 15

Delay [µs]

Den

sity

60 80 100 120 140 160

0.00

00.

006

0.01

2

Delay 16

Delay [µs]

Den

sity

30 40 50 60

0.00

0.10

Delay 17

Delay [µs]

Den

sity

60 80 100 120 140 160

0.00

00.

010

0.02

0 Delay 18

Delay [µs]

Den

sity

0.60 0.65 0.70 0.75

04

812

Delay 19

Delay [µs]

Den

sity

50 55 60 65

0.00

0.10

Delay 20

Delay [µs]

Den

sity

12 14 16 18 20 22

0.00

0.15

0.30

Delay 21

Delay [µs]

Den

sity

60 80 100 120 140 160

0.00

00.

006

0.01

2

Delay 22

Delay [µs]

Den

sity

1.10 1.15 1.20 1.25 1.30

04

812

Delay 23

Delay [µs]

Den

sity

40 60 80 100 120 140 160

0.00

00.

006

0.01

2 Delay 24

Delay [µs]

Den

sity

5 6 7 8

0.0

0.4

0.8

1.2

Delay 25

Delay [µs]

Den

sity

60 80 100 120 140 160

0.00

00.

010

Delay 26

Delay [µs]

Den

sity

0.62 0.64 0.66 0.68 0.70 0.72

010

2030

Delay 27

Delay [µs]

Den

sity

52 54 56 58

0.0

0.2

Fig. 2. Histograms of Observed Delays

packets of the real servers. When packets belonging toreplies arrive, the source address is changed to the VIPand the TCP segment is forwarded to the client via theInternet. NAT involves rewriting the packets twice andis limited to geographically close nodes since the loadbalancer has to be the gateway for all real servers.

IV. MEASUREMENT INFRASTRUCTURE

Our measurement infrastructure is based on the so-lution presented in [6], where a software monitoringsolution with an extension of the Linux netfilter frame-work is implemented. Precise measurements of one-waydelays are generated using a GPS-based time synchro-nization solution in combination with NTP [7] and thePPS-API [8]. Some modifications and enhancements ofthe standard components help to improve timekeepingaccuracy. In addition to [6], an offline synchronizationsolution has been developed [9]. Time stamps of boththe events and the PPS pulses are recorded and used tosynchronize the clocks after the measurement is over.During the measurement, the system clocks that areused for generating the time stamps are left completelyunsynchronized. The offline synchronization after themeasurements uses a filtering algorithm to determine andcompensate for the variation of the clock frequencies thatoccurred during the measurement.

On the client side, an HTTP load generator is needed.The tool httperf [10] is able to generate load thatoverloads a standard web server. The number and the rateof the requests can be specified on the command line.This load generator is ideal for studying the behavior of

a web server in extreme situations to find the system’slimits. The request generation phase of httperf hasbeen instrumented so that each request on the HTTPlayer can be tagged with a time stamp.

V. MEASUREMENTS

For our sample measurements, five real servers andone load balancer were used in a NAT environmentwith round-robin scheduling. One test client generatedHTTP requests using the httperf load generator. Wegenerated 10000 HTTP/1.0 requests for a binary file witha size of 1024 bytes. This resulted in a request size of65 bytes. The web server added 244 bytes of headerinformation, so the resulting replies had a size of 1268bytes. Since this is smaller than the maximum segmentsize used (1500 bytes), all replies consisted of exactlyone TCP segment. Fig. 1 illustrates the 27 individualdelays occurring in the exchange of TCP segments. Timeadvances along the vertical axis from top to bottom. Thedelays, shown as vertical bars, contribute to the totalprocessing time of the HTTP request. The horizontalposition shows where the delays are caused: Either bythe load generator (LG), the network channel betweenthe load generator and the load balancer (C1), the loadbalancer (LB), the network between the load balancerand the real servers (C2) or by one of the real servers(RS).

The delays in the channel C1 and C2 include not onlythe physical propagation delay and the store-and-forwarddelay of the switches between the hosts, but also the timebetween the reception of the packet at the node of the

0 20 40 60 80 100

5258

64

Measurements: Delay 19

Time [s]D

elay

[µs]

Delay [µs]

Den

sity

50 55 60 65

0.00

0.10

0 100 200 300 400 500

0.0

0.4

0.8

Lag

Cor

rela

tion

Delay i

Del

ay i+

1

50 55 60 65

5055

6065

Fit: Delay 19

Time [s]

Del

ay [µ

s]

0 20 40 60 80 100

5258

64

Delay [µs]

Den

sity

50 55 60 65

0.00

0.10

0 100 200 300 400 500

0.0

0.4

0.8

Lag

Cor

rela

tion

Delay i

Del

ay i+

1

50 55 60 65

5060

Fig. 3. Delay 19: measured data vs. fitted distribution

cluster and the beginning of the packet processing in theTCP/IP stack of the operating system.

The segments sent during delay 11, 12 and 16 aredue to TCP protocol mechanisms (fast ACK) and do notmark state changes in the HTTP protocol.

The data gathered in this way enables us to builda detailed input model for a simulation model of ourdistributed web server and to calibrate and validate themodel.

VI. ADVANCED INPUT MODELING

Fig. 2 depicts the histograms of the 27 measureddelays. Each of these delays is used as input for thesimulation model. The next sections describe the differ-ent possibilities to represent the measured delays in thesimulation theoretically.

A. Standard theoretical distributions

A commonly used method to represent measurementsin a simulation are theoretical distribution functions. Thedistribution fitting tool ExpertFit [11] was used in anattempt to fit theoretical distributions to all 27 delays.

Good fits could be achieved by using this approachfor seven delays (1, 4, 6, 14, 22, 24, 27), using mostlythe lognormal or log-logistic distributions.

B. Multi-modal delays

Several of the measured distributions have two ormore peaks. This makes it impossible to achieve agood fit with ExpertFit, as the tool is unable to fitdistributions to multi-modal data. An easy workaroundfor this problem is to manually split the data at athreshold and then fit a separate distribution for eachpart. In the simulation, these separate distributions arecombined using the relative frequencies of the respectiveparts of the data. This method ensures that the weightsof the fitted partial distributions approach the weightsmeasured in the real system as the number of sampleddelays increases.

The delays 2, 3, 8 and 16 could be fitted in that way.

C. Phased multi-modal delays

There are some delays that can not be modeled withthe simple multi-modal distributions described abovebecause their peaks are not independent. Inspection of

the corresponding trace plots shows that these delaysfeature alternating phases during which values from theupper or lower parts occur almost exclusively. The topleft part of Fig. 3 shows how these phases appear in atrace plot.

The phases have been recreated in the model usinga state chart with two main states, one for the phaseswhich contain just values from the upper part, and onefor the phases which contain values from the lower part.In each state, delays are sampled exclusively from thedistribution associated with that state.

To determine when state changes should occur, ap-proximate minimum phase lengths in seconds for theupper and lower distributions have been identified. Onestate is active for the duration of the correspondingminimum phase length. Whenever this time has passed,a state change is possible. If the state change reallytakes place is determined randomly according to therelative frequencies of the measured data parts. Thispolicy ensures that the weights of the fitted distributionsapproach the measured weights even if the minimumphase lengths behave differently.

For data without a fixed minimum phase length itwould be required to fit a distribution for the phaselengths. As we observed a minimum phase length in ourmeasurements, this is not necessary in our model andthe presented approach is sufficient.

The delays 10, 11, 12, 18, 19, 20 and 26 are rep-resented in this way. Fig. 3 compares the original dataand the fit used in the model for delay 19. The traceplot shows that the phases in the simulation approxi-mately match the phases from the measurements. Thehistograms and also the lag-correlations plots show goodmatches. The only drawback is that the simulation fea-tures less transitions between the peaks than found inthe measurements. This is illustrated in the scatter ploton the right. It is probably caused by the fact that themeasured phases are not as ”clean” as the simulatedones, meaning that lower values sometimes occur inupper phases and upper values in lower phases, resultingin more transitions in the scatter plot.

D. Bezier distributions

An alternative to fitting theoretical distributions to datais to use bezier distributions.

0 20 40 60 80 100

0.60

0.75

Measurements: Delay 18

Time [s]D

elay

[µs]

Delay [µs]

Den

sity

0.60 0.70 0.80

04

8

0 100 200 300 400 500

0.0

0.4

0.8

Lag

Cor

rela

tion

0.60 0.70 0.80

0.60

0.75

Delay i

Del

ay i+

1

Fit: Delay 18

Time [s]

Del

ay [µ

s]

0 20 40 60 80 100

0.60

0.75

Delay [µs]

Den

sity

0.60 0.70 0.80

04

812

0 100 200 300 400 500

0.0

0.4

0.8

Lag

Cor

rela

tion

0.60 0.70 0.80

0.60

0.75

Delay i

Del

ay i+

1

Fig. 4. Delay 18: measured data vs. fitted distribution

0 2000 4000 6000 8000

6012

018

0

Measurement: Delay 5

Delay Number

Del

ay 5

[µs]

Delay [µs]

Den

sity

60 100 140 180

0.00

00.

006

0.01

2

0 100 200 300 400 500

0.0

0.4

0.8

Lag

Cor

rela

tion

60 100 140 180

6012

018

0

Delay i

Del

ay i+

1

0 2000 4000 6000 8000

6012

018

0

Fit: Delay 5

Delay Number

Del

ay 5

[µs]

Delay [µs]

Den

sity

60 100 140 180

0.00

00.

006

0.01

2

0 100 200 300 400 500

0.0

0.4

0.8

Lag

Cor

rela

tion

60 100 140 180

6012

018

0

Delay i

Del

ay i+

1

Fig. 5. Delay 5: measured data vs. fitted distribution

The classical bezier curve is a special case of aspline curve, and is often used in computer graph-ics to approximate smooth univariate functions on abounded interval. This is done by assigning a num-ber of control points pi = (xi, zi) through thevicinity of which the curve is bound to pass. Abezier curve with degree n and the control points{p0, p1, . . . , pn} is given by

P (t) =n∑

i=0

Bn,i(t)pi for t ∈ [0, 1] ,

where the blending function Bn,i(t) is the Bernsteinpolynomial

Bn,i(t) =

n!

i!(n−i)! ti(1− t)n−i for t ∈ [0, 1]

and i = 0, 1, . . . , n0 otherwise

Wagner and Wilson show in [12] how to create abezier distribution function and how to derive randomsamples from it with the method of inversion. To createa random sample given a uniformly distributed randomnumber U , the first step is to compute tU from thefollowing formula:

U =n∑

i=0

Bn,i(tU )zi

To solve this equation, an implementation of a numer-ical root-finding algorithm is needed. In our simulationmodel, a combination of two comparably simple algo-rithms, namely bisection and secant [13] is employed.

First, two approximate values are obtained using twobisection runs of relatively low order (4 and 5). The finalsolution is computed with the secant method using thebisection results as initial approximations. The randomvariate x(tU ) can then be delivered according to

x(tU ) =n∑

i=0

Bn,i(tU )xi

The graphical tool PRIME, also developed by Wagnerand Wilson [12] can be used to fit bezier distributions tosets of custom data. Several automated fitting methodsare available as well as the possibility to manually adjustthe control points. Result of the fitting process are thecontrol point coordinates.

Three delays, namely 10, 18 and 26, are representedby bezier distributions in the model. As the measureddistributions are multi-modal, each delay is modeledwith two bezier distributions which are combined usingthe phasing algorithm described above. Fig. 4 showsthe fit for delay 18 compared to the measured data.As with the phased multi-modal delays described in thelast section, the simulation has less transitions betweenthe phases compared to the measurements. In all otherrespects, the fitted distribution is a good approximationfor the measurements.

VII. A NEW MODEL FOR AUTOCORRELATED DATA

Nearly all delays in the Fast Ethernet channels (delays3, 5, 11, 13, 19, 21, 27) feature both a high coefficientof auto-correlation over relatively long lags and a clearupper and lower bound (dashed horizontal lines in the

60000 80000 100000 120000 140000 160000 180000

−1e+

05−5

e+04

0e+0

05e

+04

1e+0

5

Weighting Areas

Delay 5 [ns]

Del

ta 5

[ns]

AkC k

Fig. 6. Valid Sampling Area

top left graph in Fig. 5). The main idea used in oursimulation representation is not to sample the delays di

from a distribution directly, but to sample the differencesδi = di+1−di of successive delays. Due to the upper andlower bound dmax and dmin of the delays, there is onlya limited range dmin−di ≤ δi ≤ dmax−di to sample thedeltas from for a given delay di. This valid range can becalculated for all values dmin ≤ di ≤ dmax. This resultsin a parallelogram shaped dashed area in Fig. 6, whereδ is plotted over d. All points (di, δi) are located insidethe parallelogram.

Inside the valid area, the δi are to a large extentindependent of the di, as illustrated by the 3-dimensionalhistogram in Fig. 7.

This fact allows us to construct an overall histogramHo of all δi regardless of di. But since only values insidethe valid area are used to construct the histogram Ho, anadditional histogram Hw is built, where the bins of thehistogram Ho are weighted with a weighting factor. Thefactor wk for bin k is calculated as the ratio of the areaAk to the area Ck. Ak is a rectangle delimited on thevertical axis by the borders of bin k and on the horizontalaxis by dmin and dmax (see Fig. 6). Ck is the portion ofthis rectangle that is covered by the valid area, i.e. wk =Ak/Ck. For an equidistant histogram with bin width bthe area Ak is constant: Ak = b ·(dmax−dmin)∀k. If ok

denotes the number of observations in bin k of Ho, thenthe number of observations in bin k of Hw is calculatedas ok · wk = ok ·Ak/Ck.

The resulting histogram Hw is the histogram thatwould be generated, if delta values were observed forall delays dmin ≤ di ≤ dmax, i.e. even outside of thevalid area. The effect of the weighting process is shownin Fig. 8, where the overall histogram Ho is overlaidwith the weighted histogram Hw. The histogram Hw isused to sample values δi, which are used to calculatethe next delay from the previous one as di+1 = di + δi.For this purpose, we sample δi from a part of Hw thatresults in a δi ∈ [dmin − di; dmax − di] for the currentdelay di. In our example, the bounds for the delay aredmin =64955.5 ns and dmax = 192672.0 ns. When, e.g., the

Histogram for Delta of Delay 5

DeltaDelay

Frequency

0

50

100

150

200

250

Fig. 7. 3D Histogram of Delta 5

Histogram of Delta 5

Delta 5

Freq

uenc

y−1e+05 −5e+04 0e+00 5e+04 1e+05

050

015

0025

0035

00

HoHw

Fig. 8. Original and Modified Histogram of Delta 5

delay dj has reached 190000.0 ns, δj is sampled fromthe range [dmin − dj = −125044.5 ns; dmax − dj =2672.0 ns]. Due to the weighting, a considerable amountof probability mass is found on the extreme negative endof this range and there is a high probability that the delayjumps from a high dj to a low dj+1 = dj + δj by sam-pling a low negative δj . Fig. 5 compares the measureddata with values generated using this method. The graphsshow a close match of relevant characteristics.

VIII. SIMULATION MODEL

The modeling tool AnyLogic [14] was usedto build a simulation model of the web clus-ter. AnyLogic supports discrete event simulationin combination with UML state charts. Our sim-ulation model consists of five top-level buildingblocks, each of which represents another element ofthe laboratory cluster. The only means of commu-nication between the five blocks are simulated TCPpackets. Fig. 9 illustrates the message flow: simu-lated requests are generated in the client, then for-warded to the first network channel, processed bythe load balancer, and forwarded through the sec-ond network channel to the server. The server thenprocesses the received request, and sends one or morereply packets back through the second network channel,the load balancer and the first network channel to theclient. The stacked client and server graphics in Fig.9 indicate that multiple clients and servers are presentin the simulation: one client object for every requestgenerated and one server object for every real serverin the current simulation setup.

Client Channel1 LoadBalancer Channel2 Server

Fig. 9. Conceptual top-level view of the simulation model

Input parameters for the simulation are the distribu-tions for the 27 single delays, the arrival rate of clientrequests, the sizes of the requested objects, the numberof real servers and the load balancing strategy that shouldbe employed. Simulation output data include the 27simulated delays and the total delay, as well as theutilization, throughput and mean queue length for thenetwork channels, the load balancer and each of theserver processors.

A. Client

The client model evaluates the input parameters andsends HTTP requests to the servers. Its behavior duringthe communication is governed by a detailed model ofTCP. The processing times in the client are modeledaccording to the input distributions specified above.

Our TCP model supports all important aspects de-scribed in the RFCs 793 (basic TCP), 1122 (require-ments for Internet hosts), 2581 (congestion avoidance,slow start, fast retransmit) and 2988 (retransmissionmanagement). The model can thus deal with packetswith arbitrarily large sizes and faulty network channels,just as a real system would. As the measurements weredone in a loss-free environment without packet splitting,the mechanisms built into the TCP object provide thepossibility to estimate system behavior under differentconditions without repeating any measurements.

B. Network channels

Several aspects had to be paid regard to while model-ing the network channels. First, the physical channels arefull duplex. This is represented by an independent streamfor each direction inside the network objects (Channel1 and Channel 2 in Fig. 9).

Second, physical channels have a minimum delay thatdepends (among other factors) on the available band-width and the length of the channel. The largest factorcontributing to that minimum delay is the transmissiondelay, i.e. the time needed to feed the packet into thenetwork channel. The other components adding to thenetwork delay (such as propagation delay or queuingdelay) are comparably small.

Third, even if the queuing delay is not represented asan absolute value contributing to the minimum channeldelay, it still has to be modeled to account for queuingeffects at higher utilization levels.

Fourth, the measured delay values have to be ac-counted for. The measured delays are higher than thesum of transmission and propagation delays because themeasurements could not be taken at the exact start andend of the channel, but had to be placed inside thenetwork stacks at the sender and receiver. Thus, thechannel delays also represent the nodal processing delaywhich is caused by the network stack at both ends of

the communication and additional hardware inside thechannels, like the switch in channel 2. As illustrated in

Node BNode A injectspackets

infinite-server queuesingle server queue

t -tm t

ttt -tm t

tt

Fig. 10. Concept of the channel representations

Fig. 10, the network channels are assembled as follows:the transmission delay tt is calculated and a single serverqueue with the transmission delay as required serverbusy time processes the incoming packet. This also al-lows for queuing effects at higher utilization levels. Thenthe packets are processed by an infinite server queuewhich applies the delays drawn from the distributionsfitted above (tm), reduced by the transmission delay tt.The reason for this is that parts of the measured channeldelays are not caused by the network, but by the lowerparts of the TCP stack where packets are assumed to beprocessed in parallel.

Channel 1 is modeled as a plain channel with acapacity of one gigabit in each direction. Channel 2 startswith a gigabit channel which leads into a switch wherethe channel is divided into one 100-megabit channel foreach real server.

C. Load balancer

The load balancer object distributes the incomingload to the available real servers. The model currentlysupports three different load balancing schemes: ran-dom load distribution, least connection and round robin.Round Robin is the one used in the real system.

D. Servers

The server object is the model’s most complex build-ing block. It consists of three main parts, the process,system process and processor objects.

1) Process: There is one process object for everysingle connection to a client. The processes are respon-sible for processing all incoming packets and sendingappropriate reply packets.

2) System Process: The system process object hasthe sole purpose of occupying the processor from timeto time. There are two types of processor allocationdemands. The first one occurs quite often, at about thesame rate as client requests arrive. This type of systemprocess has to be introduced to model the CPU timeconsumed by the network stack in the kernel.

The second type of system process represents all theuser-mode processes that are running in a real system,require some CPU time, but don’t have anything to dowith the actual web server processes. They don’t occurvery often, but require a large amount of processor time,compared to the first type.

The intervals between system processes and theamount of processor time taken each time have been

calibrated with processor idle data measured in the realsystem, and with the amount of queuing occurring inthe server delays, which is measurable by the number ofresulting outlier values.

3) Processor: The reply packets generated by a serverprocess or a system process are handed to the processorobject where the actual timekeeping for the server takesplace. The processor is modeled as a single server queue.Incoming requests arrive from the process and systemprocess objects. The needed processor time is eitherspecified by the system request, or sampled from theappropriate distributions.

Incoming requests for processor time are divided intorequests from user-mode processes and system pro-cesses. User-mode processes include all TCP packetscarrying data and the second type of requests generatedby the system process object. System Processes are thefirst type of requests from the system process objectand TCP packets carrying the SYN or FIN flag. Userprocesses can be interrupted by system processes atany time. This is done to mimic the real system’sbehavior where longer running processes can and willbe interrupted by other processes.

That fact can also be seen while looking at the mea-surement data, where, at higher request rates, the mainserver delay has a lot of large outlier values whereas thesmaller delays for the SYN and FIN packets still staywithin their normal boundaries.

IX. RESULTS

Several experiments have been conducted with themodel. A sequence of five experiments with a requestrate increasing from 50 to 1000 requests per secondusing our detailed input model was used for comparisonwith the measurement data. Further experiments havebeen done to compare the performance of our inputmodel to standard exponential distributions. Simulationresults were also obtained for a setup with longer replypackets from the servers.

Fig. 11 shows traces, histograms, lag-correlation plotsand scatter plots (from left to right) for the differentsimulation setups and measurements at 100 requestsper second. The simulation was stopped as soon asa confidence level of 95% with the specified relativeerror was reached after a replication. Although moreexperiments have been conducted with similar results,a detailed graphical comparison is only presented forthe scenario with 100 requests per second due to spacelimitations.

An important evaluation of our detailed input modelconsisting of the distributions described in chapters VIand VII is the comparison of the total delays in thesimulation and in the measurements. The first and thirdrow of Fig. 11 are relevant for this comparison. Theyshow a good match for the range and magnitude of thedelays. Although the simulated mean is slightly largerthan in the measurement, the model is still a closerepresentation. The correlation present in the measureddata is also represented to some extent.

Det

aile

d In

put M

odel

1024

B

Expo

nent

ial M

odel

1024

B

Mea

sure

men

ts10

24B

Det

aile

d In

put M

odel

10kB

Expo

nent

ial M

odel

10kB

12

34

5

100 requests per second

tota

l del

ay [m

s]

Fig. 12. Quantile comparison for 100 requests per second

Also very important is the question how our inputmodel performs in comparison with a standard modelconsisting of exponential distributions with the samemean. This comparison is illustrated in the first andsecond rows of Fig. 11. The measurements, and alsoour input model, concentrate the total delays on a verynarrow range. This is in contrast to the exponentialdistributions which spread the delays over more thanfour times the measured range. Due to the nature ofexponential distributions, the autocorrelations are notrepresented. Although the mean total delay gives agood approximation, the overall representation of themeasured values is not appropriate.

Fig. 12 compares the quantiles of the total delay forrequest rate of 100 requests per second in the real clusterand the model in milliseconds. The box plots show theminimum, the 15%, 50% and 85% quantiles, and themaximum values for the total delay. The plots supportthe statements above. The median of the exponential dis-tributions seems to be a good approximation, but overallthe values are spread too far, which is illustrated by thesignificant deviations in the 15% and 85% quantiles. Incontrast, the detailed input model developed by us showsa very good approximation to the measured values.

Further simulation results concern the performance ofthe system if the requested file size is increased from thestandard 1024 bytes to 10 kilobytes. The experimentalresults are shown in rows four and five of Fig. 11.The corresponding quantiles are also shown in the tworightmost boxes in Fig. 12. Summary measurementstaken at the laboratory cluster at 100 requests per secondindicate that our detailed input model gives a betterapproximation for the measured mean and median thanthe exponential distributions.

X. CONCLUSIONS

The laboratory setup of the web cluster allows usto conduct fine-grained measurements of the internalbehavior in a cluster-based web server. The measurementinfrastructure uses GPS, NTP, the PPS-API and exten-sions like offline synchronization, a kernel buffer for theevent trace and an instrumentation of the netfilter codein the Linux kernel.

0 2 4 6 80.00

10.

003

0.00

5

Detailed Input Model, 1024 bytes

Time [s]

Del

ay [µ

s]

Delay [µs]

Den

sity

0.001 0.002 0.003 0.004 0.005

010

0020

00

0 100 200 300 400 500

0.0

0.4

0.8

Lag

Cor

rela

tion

0.001 0.003 0.005 0.0070.00

10.

004

0.00

7

Delay i

Del

ay i+

1

0 2 4 6 80.00

10.

003

0.00

5

Exponential, 1024 bytes

Time [s]

Del

ay [µ

s]

Delay [µs]

Den

sity

0.001 0.002 0.003 0.004 0.005

020

040

060

0

0 100 200 300 400 500

0.0

0.4

0.8

Lag

Cor

rela

tion

0.001 0.003 0.005 0.0070.00

10.

004

0.00

7

Delay i

Del

ay i+

1

0 2 4 6 8 100.00

10.

003

0.00

5

Measurements

Time [s]

Del

ay [µ

s]

Delay [µs]

Den

sity

0.001 0.002 0.003 0.004 0.005

010

0020

00

0 100 200 300 400 500

0.0

0.4

0.8

Lag

Cor

rela

tion

0.001 0.003 0.005 0.0070.00

10.

004

0.00

7

Delay i

Del

ay i+

1

0 2 4 6 80.00

10.

003

0.00

5

Detailed Input Model, 10k bytes

Time [s]

Del

ay [µ

s]

Delay [µs]

Den

sity

0.001 0.002 0.003 0.004 0.005

050

015

00

0 100 200 300 400 500

0.0

0.4

0.8

Lag

Cor

rela

tion

0.001 0.003 0.005 0.0070.00

10.

004

0.00

7

Delay i

Del

ay i+

1

0 2 4 6 80.00

10.

003

0.00

5

Exponential, 10k bytes

Time [s]

Del

ay [µ

s]

Delay [µs]

Den

sity

0.001 0.002 0.003 0.004 0.005

020

040

060

0

0 100 200 300 400 500

0.0

0.4

0.8

Lag

Cor

rela

tion

0.001 0.003 0.005 0.0070.00

10.

004

0.00

7

Delay iD

elay

i+1

Fig. 11. Traces, histograms and Scatter Plots for the total delays at 100 requests per second

The measured data were built into a detailedUML-based simulation model of the systemby combining several input representations liketheoretical distributions, multi-modal distributions,phased multi-modal distributions, bezier distributionsand a new approach for representing auto-correlated data using the differences between successivedelays.

Simulation experiments with a configuration similarto the measurement setup show that the measured delaysat different system loads are quite closely reproduced bythe model. This enables us to run simulation experimentswith various system configurations and system loads, anddo a detailed analysis of the estimated system behaviors.Due to the detailed model, the effect of applying evensmall changes to the system, like modifying the operat-ing system’s scheduling and buffering strategies, can beevaluated.

REFERENCES

[1] D. A. Menasce and V. A. F. Almeida, Capacity Planning for WebServices, 2nd ed. Prentice Hall, 2001.

[2] E. Casalicchio and M. Colajanni, “A client-aware dispatchingalgorithm for Web clusters providing multiple services,” in Proc.of 10th Int’l World Wide Web Conference, May 2001, pp. 535–544.

[3] Y. Teo and R. Ayani, “Comparison of Load Balancing Strategieson Cluster-based Web Servers,” The Journal of the Societyfor Modeling and Simulation, vol. 77, no. 5-6, pp. 185–195,November-December 2001.

[4] Q. Zhang, A. Riska, W. Sun, E. Smirni, and G. Ciardo,“Workload-aware load balancing for clustered Web servers,”Parallel and Distributed Systems, IEEE Transactions on, vol. 16,no. 3, pp. 219–233, March 2005.

[5] “Linux Virtual Server Project,”http://www.linuxvirtualserver.org/. [Online].Available: http://www.linuxvirtualserver.org/

[6] K. Hielscher and R. German, “A Low-Cost Infrastructure forHigh Precision High Volume Performance Measurements of WebClusters,” in TOOLS 2003, Computer Performance Evaluations,Modelling Techniques and Tools, 13th International Conference,Proceedings, P. Kemper and W. Sanders, Eds. Springer, Hei-delberg, 2003, pp. 11–28.

[7] D. Mills, “Internet time synchronization: the Network TimeProtocol,” IEEE Trans. Communications, vol. 39, no. 10, pp.1482–1493, October 1991.

[8] J. Mogul, D. Mills, J. Brittenson, J. Stone, and U. Windl, “Pulse-per-second API for Unix-like operating systems, Version 1,”Internet Engineering Task Force, Request for Comments RFC-2783, March 2000.

[9] K. Hielscher, “Measurement-Based Modeling of DistributedSystems,” Ph.D Thesis (Dissertation), University of Erlangen-Nuremberg, 2008.

[10] D. Mosberger and T. Jin, “httperf: A Tool for Measuring WebServer Performance,” in First Workshop on Internet ServerPerformance. ACM, June 1998, pp. 59–67. [Online]. Available:citeseer.nj.nec.com/mosberger98httperf.html

[11] A. M. Law, “Expertfit distribution fitting software,”http://www.averill-law.com/. [Online]. Available:http://www.averill-law.com/

[12] M. A. F. Wagner and J. R. Wilson, “Using univariate Bezierdistributions to model simulation input processes,” IEEE Trans-actions, vol. 28, pp. 699–711, 1996.

[13] R. L. Burden and J. D. Faires, Numerical Analysis, 7th ed.Wadsworth Group, 2001.

[14] “Anylogic,” http://www.xjtek.com. [Online]. Available:http://www.xjtek.com