Design and Implementation of a GenericResource Sharing Virtual Time Dispatcher

Tal Ben-Nun

School of Eng. & Comp. Sci.The Hebrew University91904 Jerusalem, Israel

talbn@cs.huji.ac.il

Yoav Etsion

Computer Sciences Dept.Barcelona Supercomput. Ctr.

08034 Barcelona, Spain

yoav.etsion@bsc.es

Dror G. Feitelson

School of Eng. & Comp. Sci.The Hebrew University91904 Jerusalem, Israel

feit@cs.huji.ac.il

Abstract

Virtual machine monitors, especially when used forserver consolidation, need to enforce a predefined shar-ing of resources among the running virtual machines.We propose a new mechanism for doing so that pro-vides improved pacing in the face of heterogeneousallocations and priorities. This mechanism lends fromtoken-bucket metering and from virtual-time schedul-ing, and prioritizes the different clients based on the di-vergence between their desired allocations and the ac-tual consumptions. The ideas are demonstrated by im-plementations for the CPU and networking subsystemsof the Linux kernel. Notably, both use exactly the samebasic module; future plans include using it for disk I/Oas well.

Categories and Subject Descriptors C.2.3 [COMPUTER-COMMUNICATION NETWORKS]: Network Operations—Network management; D.4.1 [OPERATING SYSTEMS]:Process Management—Scheduling; K.6.2 [MANAGE-MENT OF COMPUTING AND INFORMATION SYS-TEMS]: Installation Management—Pricing and re-source allocation

General Terms Design, management, performance

Keywords Virtual machine, fair share, resource allo-cation

[Copyright notice will appear here once ’preprint’ option is removed.]

1. Introduction

A hypervisor or virtual machine monitor (VMM) mayseem to have similar responsibilities to an operatingsystem: it abstracts the hardware and performs resourcemanagement, including scheduling the different virtualmachines. However, the context is actually quite differ-ent. Operating systems’ schedulers typically try to op-timize performance objectives, such as response time,based on knowledge about each process’s behavior. Ahypervisor, on the other hand, is concerned with virtualmachines that may each run a mixture of diverse pro-cesses that are unknown to the hypervisor. And its goal,especially in server consolidation scenarios, is moretypically to enforce a pre-defined allocation of the re-sources.

Controlling the relative allocation of resources tocontending processes (or virtual machines) is not new.Such “fair share” scheduling of the CPU is typicallydone based on variants of virtual time. Allocations ofnetwork bandwidth are typically done using variants of“leaky bucket” or “token bucket” approaches. Our ap-proach combines the two into “resource sharing virtualtime” scheduling (abbreviated RSVT). It includes me-tering of the allocation to each process on one hand,and scheduling so as to better pace the utilization of theallocated resources on the other.

The next section motivates the work by explainingthe need for supporting predefined allocations, espe-cially in virtualization scenarios. Section 3 then de-scribes previous work and leads up to our extensions,which are delineated in Section 4. This is followed bya description of the implementation in the Linux ker-nel in Section 5, and by an experimental evaluation inSection 6. Section 7 presents the conclusions.

1 2010/2/11

2. Motivation and Context

We are now witnessing the second wave of virtualiza-tion. The first wave occurred about forty years ago andled to the widespread use of virtualization as a meansto share large scale mainframe platforms [12]. The sec-ond wave, started about ten years ago, concerns data-center servers and desktop PCs, which are now pow-erful enough to support multiple virtual machines on asingle physical host. It is driven by the utility of virtu-alization for server consolidation, flexible provisioningof resources, and support for testing and developmentof new facilities.

Server consolidation is the practice of migratinglegacy servers from distinct physical machines thatpossibly use different operating systems to virtual ma-chines on a single more powerful platform. This re-duces operational expenses by saving the need to main-tain and support all those legacy systems, reducing thefloor footprint, and reducing cooling requirements. Itis especially beneficial when it increases server utiliza-tion, e.g. if the legacy servers are not highly utilized,but when consolidated they lead to a reasonably highutilization of the new server.

Another important benefit of consolidation is that itpromotes flexible provisioning of resources. With con-solidation, the resources provided to each server arenot fixed. Rather, the different servers compete for re-sources, which are provided by the underlying virtual-ization infrastructure. It is then possible to control theresources provided to each one, and assign them ac-cording to need or the relative importance of the differ-ent servers. Moreover, this partitioning of the resourcescan be changed easily to reflect changing conditions.

The virtualization infrastructure is therefore foundto assume many of the basic responsibilities of an op-erating system. However, the situation is actually some-what different. One difference is that hypervisors typ-ically operate with far less information than an oper-ating system. An operating system mediates all inter-actions with hardware devices for all processes, wherethe processes themselves are rather simple in structure.Therefore the operating system can use a pretty simplemodel of operation, e.g. blocking a process that has re-quested an I/O operation. But a hypervisor is one levellower down, and supports a virtual machine that in turnruns a full operating system which may support manyprocesses. When some process in the virtual machinerequests an I/O operation, this does not reflect the ac-

tivity of the virtual machine as a whole — only the ac-tivity of that process, which is not even directly knownby the hypervisor.

Another difference is that the goals are typically dif-ferent. Operating systems attempt to optimize metricssuch as response time of interactive processes, while atthe same time providing equitable service to all the pro-cesses. With hypervisors, it is more typical to try andcontrol the resource allocation, and ascertain that eachvirtual machine only gets the resources that it deserves.To complicate matters, this has to be done in multipledimensions, reflecting the different devices in the sys-tem: the CPU, the disks, and the network connectivity.The question is then what does it mean to provide acertain share of multiple resources, when the processesrunning on each virtual machine actually require differ-ent combinations of resources.

We are working on a global scheduling frameworkthat is designed to answer this question. It is based onthe combination of two basic ideas: the use of fair sharescheduling to control relative resource allocation, andthe identification of the bottleneck device as the locuswhere such control should be exercised [9]. The presentpaper reports our progress in the first component of thiswork, namely the RSVT scheduler.

3. Related Work

The requirement for control over the allocation of re-sources given to different users or groups of usershas been addressed in several contexts. It is usuallycalled “fair-share scheduling” in the scheduling litera-ture, where “fair” should be understood as accordingto each user’s due rather than as equitable. Early im-plementations were based on accounting, and simplygave priority to users who had not yet received theirdue share at the expense of those that had exceededtheir share [16, 17]. In Unix systems it has also beensuggested to manipulate each process’s nice value toachieve the desired effect [8, 15]. Simpler and more di-rect approaches include lottery scheduling [27] or usingan economic model [26], where each process’s priority(and hence relative share of the resource) is expressedby its share of lottery tickets or capital.

Another approach that has been used in several im-plementations is based on virtual time [2, 7, 22]. Theidea is that time is simply counted at a different ratefor different processes, based on their relative alloca-tions. Our RSVT scheduler falls in this category; it

2 2010/2/11

bases scheduling decisions on the difference betweenthe resources a process has actually received and whatit would have received if the ideal resource sharing dis-cipline had been used [11]. A similar approach wasused in [4].

Focusing on virtual machine monitors, Xen usesBorrowed Virtual Time (BVT). VMware ESX serveruses weighted fair queueing or lottery scheduling. TheVirtuoso system uses a scheduler called VSched thattreats virtual machines as real-time tasks that require acertain slice of CPU time per each period of real time[18, 19]. Controlling the slices and periods allows foradequate performance even when mixing interactiveand batch jobs.

Control over allocations of network bandwidth isusually combined with traffic shaping, i.e. the reductionof variability in bandwidth usage. One way to achievethis is the leaky bucket algorithm. The bucket figura-tively represents a buffer where packets are stored whenthe source creates them too quickly and they cannotbe transmitted immediately. Thus data flows into thebucket at a variable rate, but flows out at a steady rate.Additional packets that arrive when the bucket is al-ready full are called “nonconforming”, and are typi-cally discarded.

An alternative is to use a token bucket, which worksthe other way around: it stores tokens that allow pack-ets to be sent (similar to the economic framework men-tioned above). When a packet arrives, it will be sent if atoken is available; otherwise it is nonconforming (andthus needs to be queued in a buffer). Tokens are addedto the bucket at a steady rate, and removed wheneverpackets are sent. Thus a source may accumulate tokenswhen it is idle, and use them at a high rate (higher thantheir arrival rate) when it needs to transmit. As a resultthe momentary bandwidth of a source may surpass itsaverage allocation, but only for a limited time.

Leaky and token buckets limit the rate of individualsources, but do not specify how they are multiplexed.Using FCFS with these algorithms may still lead tooverload and is subject to manipulations. This was im-proved by Nagle’s “fair queueing” [21], in which the re-quests of each source are kept in a separate queue, andthese queues are served in round robin manner. How-ever, given that packets may have different sizes, thismay lead to deviations from the intended bandwidth al-location. Demers et al. therefore suggested an approx-imation of round-robin at thebyte level [6]. Concep-

tually this uses a counter of how many byte-by-byterounds have elapsed, which serves as a timepiece (it isactually identical to the idea of virtual time mentionedabove). Using this, one can find the round at which apacket will finish transmission: it is the sum of its starttime and length, where the start time is the max of itsarrival time and the end of the previous packet fromthe same source. As the counter is monotonically in-creasing, the order of finish times as counted in roundscorresponds to their order in real time. The algorithmthen is to calculate these finish times for all sources,and select to transmit the packet with the earliest finishtime. It is also possible to provide variable allocationsby modifying the count of rounds needed to transmit,leading to “weighted fair queueing”.

The main drawback of all the above approaches isthat they focus on one resource — either the CPUor the network. Similarly, there has been interestingwork on scheduling bottleneck devices other than theCPU, but this is then done to optimize performance ofthe said device and not to enforce a desired allocation[1, 14, 25]. This raises the question of the interactionbetween devices, e.g. the effect of CPU scheduling onI/O [23], or the prioritization of VMs that do I/O so asnot to cause delays and latency problems [13]. But suchinteractions may naturally interfere with the desiredallocations. The work presented here is part of a largerproject to achieve a global scheduling scheme basedon identifying the bottleneck device at each instant andusing it to dictate the allocations [9].

4. The RSVT Scheduler

In general, scheduling combines resource allocationand sequencing. RSVT, introduced in [11]1, tacklesthis combination. Like other virtual time schedulingschemes it controls allocations by making time pass ata different rate for different processes (or VMs), suchthat the rate reflects the allocation. The sequencingis done by selecting the process that is most “behindits time” to run next (e.g. [22]). RSVT is especiallyuseful for combining fair shares with pacing. The ideais that rather than just selecting the VM with the biggestlag in virtual time, we select the one with the biggestdifference from where it was supposed to be if it wasadvancing continuously. This helps spread out multipleVMs with the same profile [11].

1 It was originally called PSVT, as it was envisioned in the contextof a single resource: the processor.

3 2010/2/11

Unlike some other virtual time schemes, RSVT hasa concept of allocations. This raises the question ofwhat to do when a process becomes inactive: should itsallocation continue to grow? We handle this as follows.First, allocations continue to grow for a certain “graceperiod” that reflects expected continuity of operation.Then they are frozen. Finally, the relative allocation isreset to zero after a long time that reflects the system’smemory bound. This means that processes that havebeen inactive for a long time are simply treated as ifthey were new, and their previous history is forgotten.

4.1 Expressing Relative Priorities

There are two ways to express the desired allocationsto competing processes: absolute and relative. Usingthe allocation of network bandwidth as a concrete ex-ample, an absolute allocation would be something like“this process should transmit at 20MB/s”. Such an allo-cation is natural when using leaky bucket, token bucket,or economic models. A relative allocation, on the otherhand, is more like “this process should transmit at dou-ble the rate of that process”. This approach is naturalwith lottery scheduling.

RSVT uses the relative approach. Each process isgiven a priority, which is expressed as a rate. However,this is not an absolute rate, but rather a relative one.Thus if two processes exist and both have a rate of1, they will each get half of the bandwidth. If a thirdprocess is added also with a rate of 1, each of the threewill now get a third of the bandwidth. But if the thirdprocess has a rate of 2, it will get half of the bandwidth,and the original two processes will each get a quarter.

The reason for preferring the relative approach isthat it facilitates better utilization. With absolute rates,if the total rates of active processes exceed the capacitythen they cannot be satisfied, and if they fall bellowthe capacity then resources are wasted. Using relativerates solves these problems [5]. Other solutions, suchas using a large token bucket and allowing a sourceto send at a higher rate when there is no competition,lead to reduced control over the allocations and to thedanger of extended monopolization of the resource bya single client. Relative rates are also more portable, inthe sense that they remain equally relevant if a differentnetwork is used.

4.2 Interpretation of Virtual Time

As noted above, accounting for resource usage usingvirtual time is simpler than using real time [2]. Denote

acco

unte

d vi

rtua

l tim

e

elapsed real time

accounting rate = 1

allocation = 2accounting rate = 1/2

allocation = 1

Figure 1. Virtual time scheduling.

the consumption by jobi by ci, and its allocation rateby ri (these and other notations are summarized belowin Table 1). Then by definition

dci

dv= ri (1)

wherev denotes the virtual time. The relationship be-tween virtual time and real time is based on the avail-able physical rateR that is available (e.g. the band-width of the network), and the allocation rates to theset of active jobsA:

dv

dt=

R∑

j∈A rj(2)

Putting this together, we find that the rate of consump-tion by a specific job is proportional to its relative allo-cation:

dci

dt=

ri · R∑

j∈A rj(3)

Thus virtual time is indeed a weighted version of realtime, where the weight reflects the relative allocation.In other words, different jobs are accounted for theirresource usage at different rates.

This interpretation of virtual time leads to a verysimple scheduling algorithm [7]: dispatch the job withthe lowest accounted resource consumption. For exam-ple, consider two competing jobs, one with an alloca-tion r1 = 1 and the other with an allocationr2 = 2.When job 1 runs, it is accounted at full rate. But whenjob 2 runs, it is accounted at half rate, because its al-location is double. Therefore it will get to run twice asmuch. This is illustrated in Fig. 1.

The resource rateR is relevant for resources such asnetworks or disks, where it reflects the amount of datatransferred per unit time (that is, the bandwidth). In thecase of a CPU it can be taken as 1, since the CPU pro-vides one second of processing for each second of realtime. For simplicity, we will consider this situation in

4 2010/2/11

acco

unte

d an

d id

eal p

rogr

ess

elapsed real time

ideal rate = 1/3actual progress

ideal rate = 2/3actual progress

Figure 2. RSVT scheduling.

the sequel and dropR from the equations. In fact, thiscan also be done for any resource — it just means weschedule and account for “seconds of resource activity”rather than for “units of resource work done”.

4.3 Concept of RSVT

RSVT is a variation of the virtual time approach. Vir-tual time scheduling algorithms can be viewed as bas-ing the relative priority of each job on the differencebetween the global real time and its personal virtualtime, which reflects an accounting of its consumption.However, given that all jobs share the same real time,it is enough to compare their consumptions and selectthe lowest one.

RSVT likewise compares two values, but they areslightly different: we compare each job’s actual con-sumption with itsideal consumption. The actual con-sumption grows at a steady rate (equal to real time)when the job is running, and stays flat when it is not.The ideal consumption grows steadily at a rate that re-flects the job’s relative allocation. If the consumption isahead of the allocation, the job has a low priority andother jobs should run. But if a job’s consumption lagsits allocation, it has a higher priority. In particular, thejob for which the consumption lags the allocation bythe most is the job with the highest priority, and will beselected to run next, as illustrated in Fig. 2.

An examination of Figs. 1 and 2 shows that in thiscase the produced schedule is identical. The differencebetween the two approaches is only evident when sev-eral jobs have similar profiles and compete with otherjobs that have a different one. Let’s consider a concrete

elapsed real timeac

coun

ted

and

idea

l pro

gres

sac

coun

ted

virt

ual t

ime

elapsed real time

Figure 3. The advantage of RSVT scheduling inspreading low priority jobs that compete with a high-priority job.

example where two jobs have an allocationr1 = r2 =1, and a third hasr3 = 2. Virtual time scheduling willsettle into a pattern where the two low-allocation jobsare always executed one after the other, thus excludingthe high-allocation job for 2 time units in a row (topof Fig. 3). This happens because when the two low-allocation jobs have a lower virtual time than the high-allocation job, scheduling one of them leaves the otherwith its low virtual time, so it will be scheduled next(arrows). RSVT, in contradistinction to the above, willspread them out and give the high-allocation job better-paced access to the CPU [11] (bottom of Fig. 3). Thereason is that when one low-allocation job is sched-uled to run (arrows from below), the ideal allocationof the high-allocation job continues to rise. Thereforeby the next scheduling point the high-allocation job hasgained an advantage over the second low-allocation job(arrows from above).

4.4 RSVT for a Given Set of Clients

More formally, RSVT works as follows. Assume forthe moment that the set of active clients2 A is fixed —all clients arrived some timet0 in the past, they do not

2 We will use “client” in place of source, process, etc.

5 2010/2/11

terminate, and they are constantly active in using theresource. Each clienti is characterized by its ideal rela-tive rateri. Assuming constant activity, its ideal alloca-tion by timet (where obviously we are only interestedin t ≥ t0) is then

ai(t) =ri

∑j∈A rj

· (t − t0) (4)

which reflects its relative share of the capacity of theresource.

At time t, the client’s actual consumption so far willbe denoted byci(t). Its priority is then simply

prii(t) = ai(t) − ci(t) (5)

The scheduler will select the client with the highestpriority. Note that priority may be negative, if a clienthappens to consume more than its fair share at somepoint in time.

To implement this, we need to be able to calculateai(t) andci(t) for all active clients. In principle, thesevalues need to be updated upon each scheduling event,to reflect the changes since the last scheduling event.Note that there are two types of scheduling events:

• A client is selected to use the resource

• A client stops using the resource, either because itdoes not need it anymore at this time or becauseanother is selected as having higher priority

If the resource is constantly busy, pairs of such eventscoincide as one client stops and another is selected. Insuch a situation the stop events are redundant and canbe ignored. But if the resource becomes idle, the stopevent is important to note.

Assume that the time now ist, and that the timeof the previous event wastp. If the resource was busyduring this period, assume that clienti has just finishedusing it. As all the usage was consumed by clienti, weupdate

ci(t) = ci(tp) + (t − tp) (6)

All the other c values remain unchanged, as otherclients did not consume any of the resource in this in-terval. All cs remain unchanged also if the resourcehas been idle in this period; in this case the resourcescapacity during the period fromt to tp was wasted.

However, the ideal allocations ofall the clients havegrown by their respective shares. Therefore we shouldcompute

aℓ(t) = aℓ(tp) +rℓ

∑j∈A rj

(t − tp) (7)

A set of active clientsri relative rate of clientiai cumulative allocation of clientici cumulative consumption of clientiT tick period for allocations and timer eventsg grace period during which allocations con-

tinue when a client becomes inactivem rebirth period during which allocations are

retained after a client becomes inactiveb maximal allowed difference between alloca-

tion and consumption

Table 1. Parameters used in RSVT.

for all ℓ ∈ A (including clienti).Note that in both the above equations the total in-

crement is identical, and stands att − tp. This seemsto imply a nice invariant:

∑ai =

∑ci. However, this

is hard to maintain when clients have more dynamicbehavior instead of being able to use the full availableresources all the time.

4.5 Handling a Changing Set of Clients

Note that the above equations assume that the set ofactive clientsA does not change with time. In a realsystem, of course, new clients may arrive while oth-ers depart. In addition, clients may become inactive forextended periods, i.e. they may refrain from using theresource which we are scheduling (for example, whena client is blocked doing I/O it is not contending for theCPU). This affects the above equations because the rel-ative allocations depend on

∑j∈A rj . Another question

is whether to retain the client’s historical consumptiondata for when it will become active again.

One possible approach to handling this issue is todivide the timeline into segments with persistent activesets. Thus a new segment starts whenever any of thefollowing events occurs:

• A new client arrives

• A client terminates

• An active client becomes inactive

• An inactive client becomes active again

We could then do the calculations in each such segmentindividually, in a way that reflects the changing condi-tions.

The problem with this approach is that changes maybe very frequent, and it is not clear that trying to follow

6 2010/2/11

all their details is beneficial. In particular, a thorny is-sue is how to handle clients that temporarily becomeinactive. One option is to freeze their allocations, toavoid situations where a client gains a huge credit byvirtue of not using the resource, and then starves allother clients once it starts to use it again. But this vi-olates the notion of an ideal allocation in those caseswhere start/stop activity is natural, e.g. when sendingnetwork packets or performing disk I/O. For example,if a client should get an allocation of 1/3 of the band-width, we don’t want this to be reduced due to process-ing that occurs between send operations.

The suggested solution is to define a grace periodg which defines a time frame that reflects the naturalcontinuity in using the resource — typically on sub-second time scales (with a 2GHz CPU, even a mere1ms corresponds to 2 million cycles; with a 100Mb/snetwork, it corresponds to sending 100 Kb). When aclient becomes inactive, its ideal allocation will con-tinue to grow during this grace period. After the graceperiod it will be frozen, to avoid over-allocations as de-scribed above.

The grace period is measured in ticks of lengthT

from a timer. These ticks are also used to “smooth” thefluctuations in the active set in general. This is achievedby performing periodic allocations once everyT timeunits, instead of piecemeal allocations as described byEq. (7) above. Client arrivals and departures are alsosynchronized with these ticks.

To summarize, allocations are handled by a periodictimer that ticks everyT time units. Handling this timertick involves the following actions:

1. Clean up after clients who have terminated since thelast tick.

2. Note clients that have become active or inactive.

3. Make new allocations for all the active clients.

Allocations are made in advance: the allocation per-formed on a tick reflects expected usage from this timetill the next tick. However, it may happen that not all theallocated time will be used. In order to prevent the al-locations from outgrowing the consumptions, it is nec-essary to bound their growth. Thus after calculating theallocation according to Eq. 4, we perform

ai(t) = min( ai(t), ci(t) + b ) (8)

New allocations are only given to active clients, orthose in their grace period. Inactive clients retain their

ai andci values for a certain time (the “rebirth” period,m), but then we setai = ci. This is done for bothpositive and negative relative usage. On the positiveside, it avoids situations where a client may monopolizethe resource based on a credit gained long ago. On thenegative side, it avoids situations where a client staysat a disadvantage after using more than its fair shareof the resource when no other clients wanted to use it.Thus a client returning to activity after a long period ofnot being active will be effectively treated like a newclient. In the future we plan to decayai exponentiallytowardsci, rather than using an abrupt change.

Clients may also be added to the active set. Newclients are initialized withai(ti) = ci(ti) = 0. Notethat this is before the new allocation, which they willreceive together with all other active clients. Clientsthat have become active again since the last tick aftera period of inactivity are also added to the active set.

5. Implementation in the Linux Kernel

RSVT is a generic proportional share scheduler. Thusa system may have many instances of RSVT control-ling different resources. This is managed by the globalRSVT manager.

The implementation of RSVT was started in thecontext of the QoS facility of the Linux networkingsubsystem. This facility provides a hook that can beused to select the next packet that will be sent from thequeue of waiting packets. We use RSVT as the policy toguide this selection, and select the first queued packetbelonging to the socket whose transmissions so far arethe farthest behind what they should have been.

Rather than create a policy function that is specif-ically tied to the networking QoS facility, we build onthe fact that the interface separates the mechanism fromthe policy to create a generic implementation that canbe used as the policy guide for other subsystems aswell. This boils down to the definition of an interfacewhereby the policy receives the information it requiresin order to render its decisions. We then indeed usedthis policy module for CPU scheduling as well, and us-age for I/O scheduling is planned as future work.

5.1 The RSVT Module

The RSVT module is a Linux kernel module that canprovide RSVT scheduling of resources such as theCPU, disks, and networks. The scheduling algorithmis intended to provide predetermined shares of the re-

7 2010/2/11

sources to the different processes, which potentiallyembody different virtual machines.

A kernel running RSVT has one global RSVT man-ager and one or more instances of RSVT schedulers —one for each device. The global manager is responsi-ble for two main functions. First, it maintains a repos-itory of active RSVT instances; new RSVT schedulersare created (typically upon startup) using thersvt create

function, and should be removed before shutdown us-ing thersvt destroy function. Second, it activates the pe-riodic resource allocation on all such instances, as willbe described below. This is done by calling theallocate

function of each instance.Tasks (either processes or threads) are regarded as

clients of a resource. Therefore, a task is allocated aproxy client object for each resource it uses. Tasks arealso the basic entities whose priority can be set, andthe clients representing a task operate with the task’spriority. The relationship of tasks to resources is manyto many. There are many tasks in the system, severalindependent resources, and each task may use all theresources.

The semantics of resource operation depend on thenature of the resource, with some resources operat-ing on a request basis (network, disk), but some ona time basis (CPU). The design of the RSVT modulecan account for both. It is based on a core that main-tains a queue of pending clients, and two wrappers:one for direct access, and the other for request-basedaccess (called RRSVT). The difference is that RRSVTis based on requests. Therefore when thedispatch func-tion selects a client, it returns an abstract handle to astruct list head, which contains all that client’s requests.In addition to that, clients can enqueue and requeuerequests, as demonstrated in our network implementa-tion.

In either case, the queue of pending clients containsonly the clients that are actively waiting for the re-source at a given time. For example, if the RSVT mod-ule is managing a network device, those clients thathave network packets waiting to be sent will be placedon the queue of pending clients. When all the pendingpackets of a client are sent, it is automatically removedfrom the queue. In case the resource is not request-based, such as a CPU, the queue of pending clientsacts as the runnable tasks queue, and the module’s user(the kernel) has to explicitly mark clients as pending,thereby inserting them to the queue, or not, thereby re-

moving them from the queue. This is done by the func-tionsclient set pending andclient unset pending.

When the resource is free for processing, thedispatch

function is called to select the most deserving clientfrom the pending queue (using the RSVT algorithm,based on past resource usage relative to each client’spriority). A resource that is not request-based can thenstart operating on the selected client. A request-basedresource will simply pull the selected client’s requestsone by one. After processing a client, the resourceshould update the resource usage of the client, based onthe amount of processing performed. This provides theRSVT module with the information needed to make fu-ture scheduling decisions. As noted above, this is sim-ply done using the time that the resource was used. Thefunction used to note resource usage is calledcheckin.

5.2 Data Structures

The RSVT module completely separates the object rep-resentation of a task from that representing a client.Rather than storing each client’s information as partof the corresponding task’stask struct object, it isstored as a client object that is part of the correspond-ing resource’s RSVT management module. Client ob-jects are created withclient create, and destroyed withclient destroy.

The reason for this design, other than its generality,is that a task’s use of a resource may outlive the task— for example, a process may have network packetswaiting to be sent when it is killed. Therefore, a task’sproxy client for resources must be regarded as differententities than the task itself.

To access the client objects, a task’stask struct willcontain an array of handles to the client objects fordifferent resources. The array size limits the numberof resources that can be accommodated. In the otherdirection, each client object contains a pointer to thecorrespondingtask struct. If the task has been killedthis pointer is set to NULL, and when all requests havebeen serviced the client object is deleted.

To simplify the implementation (and especially thesearch for the next client to serve) only a discrete setof different rates (representing priorities) is supported.When the RSVT module is initialized, the number ofdistinct rates and their specific values should be spec-ified (these are arguments torsvt create). These arestored in an array that maps each priority level to itsassociated rate.

8 2010/2/11

The lowest priority is by convention set to 1. Thehighest priority should be set according to the desiredmaximal ratio of rates: should be highest rate by dou-ble the lowest rate? or 5 times higher? or 10 times? Ad-ditional intermediate levels should be set according tothe desired ratios that should be supported. This wouldtypically imply a logarithmic scale, as in 1, 2, 4, 8. Aspriorities should be integral numbers, it is possible todeviate from the convention of starting at 1 to accom-modate smaller ratios. For example, to achieve an ap-proximate ratio of

√2 between adjacent priorities one

can use 10, 14, 20, 28, 40, 57, 80.Clients may be linked to each other in two ways.

First, there is a linked list ofall clients. This is usedwhen all clients must be traversed, e.g. when makingnew allocations. In addition, active clients are linkedaccording to their priority. This creates a multi-queuestructure, with a separate queue for each discrete prior-ity level.

5.3 Operation

In principle each resource has its own units: CPU usageis measured in cycles, network transmission is mea-sured in bytes, and disk I/O is measured in blocks.Using these units requires us to calibrate the RSVTmodule so that it knows how many units to allocateor charge for a period of time. However, this is re-dundant if all allocations and consumptions are simplymeasured in time. We therefore use time as our basicunit for all resources, regardless of their nature. Specif-ically, our basic unit in the initial implementation is onemicrosecond. With 32 bits this limits us to usage of justover one hour. In a “real” implementation one wouldtherefore need to use 64 bits.

Note that in some resources, notably disk and net-work, it may be impossible to measure the time neededto serve a specific request. The problem is that thelower-level devices may handle multiple requests con-currently and transparently. The solution is to map therequest size to time using the known nominal resourcerate (e.g. effective bandwidth; for example, in a net-work setting this would account for overhead due tophysical layer headers). This is done by the “glue code”and is external to RSVT itself. In the network imple-mentation the resource rate is obtained from the con-figuration information. The implication of this decisionis that the rate of resource usage is constant at all times.In modern systems this is not necessarily the case, es-

pecially with regards to the CPU clock rate which maybe adjusted based on power and heating considerations.We currently ignore such difficulties.

The RSVT module does not generally manageallof a resource’s users, as it is designed to manageonly a subset of the existing processes — specifi-cally those executing a virtual machine. Other pro-cesses — for example system daemons — should bemanaged by another scheduling algorithm (probably aFIFO queue). Such clients should have priority overthe RSVT clients. For example, in our implementationfor networking, traffic generated by NFS clients fallsin this category. In CPU scheduling, this may refer tosome real-time processes.

However, we do need to track the times when theresource is being used by these other clients that areoutside our control, and reduce the allocations to ourclients accordingly. In general, a resource’s time can bedivided into three types of use:

• It is being used by RSVT’s clients based on itsscheduling decisions.

• It is idle.

• It is being used by some other clients we don’t knowabout (so called “dead time”).

The RSVT scheduler should be cognizant of the firsttwo, and use them for the purpose of calculating allo-cations. It should ignore the third, as if it were a gap inthe timeline that does not exist. To do so, it must knowwhen other clients take over the resource and when theyrelease it again.

5.3.1 Ticks and Allocations

The original concept of RSVT is based on resourcesharing, as if all clients consume their allowance ofthe resource continuously. In reality, of course, they aremultiplexed using time slicing. Allocations are likewisedone at discrete instances. That is why the experimentalresults below show stepwise progress rather than slopes(Fig. 4 as opposed to Fig. 2).

Allocations are done periodically based on the sys-tem’s clock interrupt (the same one that incrementsjiffies3). At each tick, an allocation reflecting the timesince the previous allocation is made, after subtractingknown dead time (that is, time when the resource wasused by entities outside our control). As part of this, thetime in jiffies is translated into microseconds. The total

3 On a 250Hz system this isT = 4ms.

9 2010/2/11

allocation is divided among allactive clients accordingto their relative priorities. To make this easier, an accu-mulator with the total rates of all active clients is main-tained at all times; it is updated when clients arrive,terminate, or change state (become active/inactive).

The total allocation reflects the total capacity avail-able to our clients after deducting dead time. But clientsmay not use their whole allocation, e.g. because theypace their activity based on wallclock time. To avoidwaste of resources, RSVT allows negative priorities, asmay happen if a client’s consumption is ahead of itsallocation. In fact, it may happen that one client’s allo-cation grows significantly more than its consumption,while another’s is significantly behind its consumption.This is undesirable as it may have detrimental effectson future usage. Specifically, if the allocation is wayahead the client has a high priority and may monopo-lize the resource for an extended time. If it is behind theclient may be blocked out for an extended time.

Our implementation solves this problem by bound-ing the divergence between the allocation and the con-sumption. This is always done by manipulating allo-cations, as the consumption reflects real usage by theclient. Thus if a client does not use its full allocation,future allocations are bounded and will not grow toomuch beyond the client’s consumption. Conversely, ifa client’s consumption grows more than its allocation,the missing allocation is made up so as to prevent toomuch lag. In both cases, the bound is set equal to thegrace period in the initial implementation (b = g).

5.3.2 Dispatch and Check-in

Dispatch decisions require finding the client with thelargest difference between its allocation and consump-tion. The problem is that allocations change at differ-ent rates for different clients. To reduce overhead wedo not want to scan all the active clients each time.Therefore clients are kept in separate queues accord-ing to their priorities (rates). Each queue is sorted suchthat the client with the largest difference is first; as allthe clients in the queue have the same rate, they cannotovertake each other [9]. Thus we just need to comparethe first client in each queue.

After a client is selected its resource usage must benoted. In the networking implementation this can bedone in advance. When a packet is sent, the lengthof the packet is translated into a time based on thenetwork’s bandwidth. This is then accounted to the

clients consumption. In the CPU implementation theaccounting is done when the client is de-scheduled,based on reading the processor’s cycle counter, becausethe length of time it will run cannot be known in ad-vance [10].

5.3.3 Handling Grace and Rebirth

Allocations should in principle be made at the finestgranularity possible, and as a compromise we do themat tick granularity. But we also need two coarser timers,in order to implement the grace period and rebirth. Bothof these also piggyback on the ticks used for allocation.

To implement the grace period, each client has atimestamp of when it was last active. As part of han-dling a tick we first scan all clients to check the differ-ence between their timestamps and the current time. Aclient that has been inactive for longer than the graceperiod then becomes inactive. The grace period in thecurrent implementation is set tog = 20ms for the net-work andg = 600ms for the CPU. This reflects thefact that the network operates at a much finer resolution— sending of individual packets vs. scheduling timequanta.

In addition, the ticks are also used for rebirth. Re-birth works using a second chance approach. It is ac-tivated periodically, with a period that is 60 times thegrace period. Each client has an activity flag, whichis set whenever it performs some activity. Upon in-vocation of a rebirth process, all clients are checked.Those whose flag is set are retained, but the flag is re-set. Those whose flags are unset have not been activesince the previous check, so they have been inactive forat least a full rebirth period. Their allocation is thenset to equal their consumption. When they become ac-tive again they will then be treated as if they were newclients (thus the name “rebirth”).

5.4 Networking Glue Code

The generic RSVT module is responsible for the schedul-ing policy only. It exists in parallel to the mechanismsused to actually manage the resource itself. The gluecode connects the policy and the mechanism.

The Linux kernel has long had a network Quality ofService (QoS) module in it, located innet/sched [24].It consists of a main control and various alternativepolicies (called “queueing disciplines”). It is relativelyeasy to add new policies to the QoS module, simplyby adding a file and implementing the basic functions(enqueue packet, dequeue packet, and so forth).

10 2010/2/11

Our “glue-code” constitutes such a QoS policy.It implements queueing packets as client requests toRSVT. It implements de-queueing by using the RSVTdispatch function to find the client with the highest pri-ority, then obtaining this client’s next request, and fi-nally checking-in with the approximate packet sendingtime in microseconds. Note that this is done on a packetbasis. Given that the QoS module is very low level, be-low the TCP/IP implementation, packets do not neces-sarily correspond to transmissions. Long transmissionsmay be fragmented into multiple packets due to MTUconsiderations, and all these packets have to be handledconsistently.

Importantly, the glue code also needs to distinguishbetween packets belonging to RSVT clients and pack-ets coming from other sources. The above procedure isapplied only for RSVT packets. Other packets are sentimmediately, thus implementing FIFO scheduling anda higher priority for them. This includes NFS traffic.The glue code keeps track of such packets, in order toaccount for the dead time that should not be reflectedin allocations to RSVT clients.

5.5 CPU Glue Code

A modular scheduling policy manager already existsin the Linux kernel (the Modular Scheduler Core [3]),containing different time-sharing techniques for pro-cesses. For example, the Completely Fair Schedul-ing (CFS) policy implements fair-sharing of the CPUamong all processes [20, Sect. 2.6]. One useful featureof the modular core is custom run “queues” for eachscheduling policy.

For our implementation, we created our own schedul-ing policy, which is essentially a wrapper for thenon-request based RSVT module. The run queue inthe policy is actually the RSVT object, and queue-ing/dequeueing processes to the run queue causes themto be set as pending or not pending correspondingly.“Taking out” the next process to the scheduler is im-plemented as dispatching processes, and “putting” theprocess back to the run queue is equivalent to checkingin the RSVT client.

The RSVT policy is lower than the CFS policy(which is the default) in the scheduler class hierarchy,in order to allow hypervisor processes to act as soon aspossible, if necessary. It is higher than the idle class,which includes the system idle loop. By hooking intothe dispatch loop, which searches for the highest pri-

ority runnable process, we keep track of time used byreal-time and CFS processes. From RSVT’s perspec-tive this is dead time that should not be allocated.

The initial implementation of RSVT CPU schedul-ing assumes a single processor, and is therefore unsuit-able for SMP machines. This is a relatively minor tech-nical issue concerned with keeping track of the cur-rently scheduled client. To support SMP machines allthat is needed is to allow multiple “current” clients.Also, care must be taken when calculating allocations.If there are less clients than processors, they shouldeach get an allocation that is equivalent to 100% of oneprocessor (possibly minus dead time as appropriate). Ifthere are more, the sum of the capacities of the pro-cessors should be divided among the clients, based onthe assumption that capacity can be allocated flexiblyin this case. This has the drawback of losing processoraffinity characteristics; we leave the possible integra-tion of affinity to future work.

6. Experimental Results

Using our implementation, we ran a number of experi-ments to demonstrate the capabilities and properties ofRSVT. For network allocations, the main applicationwe use is netperf (www.netperf.org), which is a bench-marking tool that continuously sends packets. For theCPU we use synthetic CPU-bound processes and ap-plications like MPlayer.

6.1 Basic Allocations

Fig. 4 shows the results for a simple case of rela-tive allocations with three competing netperf processeswith relative priorities of 1, 2, and 3. In this and sub-sequent graphs the X axis is real time, and the linesshow the growth of both allocations and consumptionswith time for all the processes. Hence the slope of aline reflects the rate (and priority) of the correspond-ing process. The lines showing consumption are on topof those showing the allocation, because in this simplescenario the consumption closely tracks the allocation.The smaller graph shows a zoom into one of these lines.At this fine detail one can see the periodic allocations(big steps of allocation line) and the sending of individ-ual packets (smaller steps of consumption line).

The actual bandwidths achieved by the competingprocesses are shown in the table. These are based on 10repeated runs of 10 seconds each. The results obviouslyclosely reflect the desired relative allocations, with ex-

11 2010/2/11

0

1e+06

2e+06

3e+06

4e+06

5e+06

6e+06

0 1e+06 2e+06 3e+06 4e+06 5e+06 6e+06 7e+06 8e+06

reso

urce

tim

e

real time [Kcycles]

alloc 1cons 1alloc 2cons 2alloc 3cons 3

rate BW1 15.69±0.112 30.81±0.033 46.10±0.03

910000

915000

920000

925000

930000

935000

1e+06 1.01e+06 1.02e+06

reso

urce

tim

e

real time [Kcycles]

alloccons

Figure 4. relative allocations and consumption ofcompeting netperf processes.

tremely small variability. During our measurements wesaw only a small number of results that deviated fromthe specified allocations. These were found to be dueto variation in startup time. When one process startsbefore the other, it gets the full bandwidth during thistime, leading to a higher average. Likewise, the laggingprocess gets the full bandwidth after the first one ter-minates, also leading to a higher average. This happensbecause they each run for 10 seconds.

6.2 Dynamic Scenarios

Fig. 5 shows more complicated scenarios, in which theset of clients changes with time. In the first there areinitially two netperf clients, with relative priorities 1and 3. After some time (about 2.3 million Kcycles) athird client is added, with relative priority 2. As a re-sult the allocations of the first two clients are reducedin a way that all three receive their appropriate shares.Then, at about 4.7 million Kcycles, the first client ter-minates. The allocations of the remaining two are thenincreased, but the change is very small because the ter-minated client was the one with the smallest allocation.The second graph shows two competing MPlayers withrelative allocations of 1 and 4 of the CPU. They start to-gether, and when the high-priority one terminates, theother picks up the slack.

0

1e+06

2e+06

3e+06

4e+06

5e+06

6e+06

0 1e+06 2e+06 3e+06 4e+06 5e+06 6e+06 7e+06 8e+06

reso

urce

tim

e

real time [Kcycles]

alloc 1cons 1alloc 2cons 2alloc 3cons 3

0

1e+07

2e+07

3e+07

4e+07

5e+07

6e+07

7e+07

8e+07

9e+07

1e+08

0 1e+08 2e+08 3e+08 4e+08 5e+08 6e+08 7e+08 8e+08

reso

urce

tim

e

real time [Kcycles]

alloc 4cons 4alloc 1cons 1

Figure 5. Allocations change dynamically as clientsare added and removed.

400000

600000

800000

1e+06

1.2e+06

1.4e+06

1.6e+06

1.8e+06

2e+06

2.2e+06

1.2e+06 1.4e+06 1.6e+06 1.8e+06 2e+06 2.2e+06 2.4e+06

reso

urce

tim

e

real time [Kcycles]

alloc 1cons 1alloc 2cons 2

Figure 6. Effect of the grace period.

Fig. 6 shows the effect of the grace period. The sce-nario is two competing netperf processes, with relativepriorities of 1 and 2. At two points the process with thehigher priority sleeps for some time, simulating a situa-tion that a process interleaves sending of network traffic

12 2010/2/11

0

1e+06

2e+06

3e+06

4e+06

5e+06

6e+06

0 1e+06 2e+06 3e+06 4e+06 5e+06 6e+06 7e+06 8e+06

reso

urce

tim

e

real time [Kcycles]

alloc 2cons 2alloc 1cons 1

Figure 7. Effect of the rebirth mechanism.

with other non-networking activities. The first occursat about 1.4 million Kcycles. This sleep is shorter thanthe grace period4 so the allocation continues to grow.Thus, when the process resumes its activity, it has arelatively high priority and blocks out the other pro-cess for a short time. The second sleep occurs at about2.1 million Kcycles. This one is much longer, so afterthe grace period ends the process is recognized as inac-tive, and it receives no more allocations. Therefore thefull bandwidth allocation is given to the other process,which uses it to increase its transmission rate. Then,when the high-priority process wakes up and resumessending, its allocations are renewed, and the other pro-cess drops down again to its reduced allocation.

If a process refrains from action for a longer time,the rebirth mechanism kicks in. This is demonstratedin Fig. 7 The scenario is the same as in the previouscase, except that when the high priority process remainsinactive for too long, its extra allocation is removed (ataround 2.2 million Kcycles).

Another interesting effect, using MPlayer and CPURSVT scheduling, is shown in Fig. 8. Two MPlayerprocesses are decoding videos, with relative prioritiesof 1 and 4 controlled by RSVT. In addition an X serveris displaying the resulting frames, running under CFSand thus with higher priority than the MPlayers (thissaves the need to propagate usage information as in[11]). Initially the allocations indeed reflect the relativepriorities. But the consumptions are similar, because itturns out that MPlayer does not really need so muchCPU power in order to decode such a video. Aftera short time RSVT therefore starts to curb the extra

4 For this experiment we used an extended grace period of 60 ms.

0

2e+06

4e+06

6e+06

8e+06

1e+07

1.2e+07

1.4e+07

1.6e+07

0 2e+07 4e+07 6e+07 8e+07 1e+08 1.2e+08 1.4e+08 1.6e+08

reso

urce

tim

e

real time [Kcycles]

alloc 1cons 1alloc 4cons 4

Figure 8. Effect of self-throttling by MPlayer.

allocation for the high-priority client, and it just followsthe consumption. The low-priority client gets its fairallocation of CPU capacity, which is somewhat lessthan it needs, and indeed it also prints out a warningthat the system is to slow to show the requested videoat its full rate.

7. Conclusions

RSVT is a flexible proportional share scheduler. It pro-vides adjustable allocations among competing clients,and improved pacing of allocations to high-priorityclients. The basic implementation is generic, and canbe used as the policy module for any schedulable re-source. This is done using “glue” code that interfacesRSVT to the desired subsystem. We presented our ini-tial implementation, which includes glue code for thenetworking QoS mechanism to controls network trans-missions, and glue code for the modular scheduler coreto control CPU scheduling. In future work we plan toalso implement control of disk I/O by integrating withthe Linux I/O scheduling framework.

The current implementation, while providing a proofof concept, is not complete. In particular, it is lackingsupport for groups of processes and for the inheritanceor partitioning of both allocations and consumptionsacross process forks. Implementing this will obviouslyincrease the usefulness of the system, but does notcontribute much at the conceptual level.

Another interesting avenue for future research is toextend RSVT by combining relative allocations withabsolute allocations. For example, it might be the casethat one client should receive half the allocation ofanother, but not less than 20Mb/s. This can be im-

13 2010/2/11

plemented by verifying that the allocations satisfy thespecification, but requires the addition of admissioncontrols to ensure that the specified rates do not surpassthe available capacity.

Finally, the whole RSVT development is part of a vi-sion of a global scheduling framework that controls rel-ative allocations of different resources in a coordinatedmanner [9]. This is to be integrated with the KVM ker-nel virtualization module. Realizing this vision is a ma-jor part of our future work in the coming years.

Acknowledgments

This research was supported by the Israel ScienceFoundation (grant no. 28/09).

References[1] N. Bansal and M. Harchol-Balter, “Analysis of SRPT

scheduling: Investigating unfairness”. In SIGMETRICSConf. Measurement & Modeling of Comput. Syst., pp.279–290, Jun 2001.

[2] A. Bavier and L. Peterson, “The power of virtual timefor multimedia scheduling”. In 10th Network & Op.Syst. Support for Digital Audio & Video, Jun 2000.

[3] D. P. Bovet and M. Cesati,Understanding the LinuxKernel. O’Reilly, 2001.

[4] A. Chandra, M. Adler, P. Goyal, and P. Shenoy,“Surplus fair scheduling: A proportional-share CPUscheduling algorithm for symmetric multiprocessors”.In 4thSymp. Operating Systems Design & Implementa-tion, pp. 45–58, Oct 2000.

[5] Y. Chen, C. Qiao, M. Hamdi, and D. H. K. Tsang, “Pro-portional differentiation: A scalable QoS approach”.IEEE Comm. Mag. 41(6), pp. 52–58, Jun 2003.

[6] A. Demers, S. Keshav, and S. Shenker, “Analysis andsimulation of a fair queueing algorithm”. In ACM SIG-COMM Conf., pp. 1–12, Sep 1989.

[7] K. J. Duda and D. R. Cheriton, “Borrowed-virtual-time(BVT) scheduling: supporting latency-sensitive threadsin a general-purpose scheduler”. In 17th Symp. Operat-ing Systems Principles, pp. 261–276, Dec 1999.

[8] D. H. J. Epema, “Decay-usage scheduling in multipro-cessors”. ACM Trans. Comput. Syst. 16(4), pp. 367–415, Nov 1998.

[9] Y. Etsion, T. Ben-Nun, and D. G. Feitelson, “Aglobal scheduling framework for virtualization envi-ronments”. In 5th Intl. Workshop System ManagementTechniques, Processes, and services, May 2009.

[10] Y. Etsion, D. Tsafrir, and D. G. Feitelson, “Effectsof clock resolution on the scheduling of interactiveand soft real-time processes”. In SIGMETRICS Conf.Measurement & Modeling of Comput. Syst., pp. 172–183, Jun 2003.

[11] Y. Etsion, D. Tsafrir, and D. G. Feitelson, “Process pri-oritization using output production: scheduling for mul-timedia”. ACM Trans. Multimedia Comput., Commun.& App. 2(4), pp. 318–342, Nov 2006.

[12] R. P. Goldberg, “Survey of virtual machine research”.Computer 7(6), pp. 34–45, Jun 1974.

[13] S. Govindan, A. R. Nath, A. Das, B. Urgaonkar, andA. Sivasubramaniam, “Xen and co.: Communication-aware CPU scheduling for consolidated Xen-basedhosting platforms”. In 3rd Intl. Conf. Virtual ExecutionEnvironments, pp. 126–136, Jun 2007.

[14] M. Harchol-Balter, B. Schroeder, N. Bansal, andM. Agrawal, “Size-based scheduling to improve webperformance”. ACM Trans. Comput. Syst. 21(2), pp.207–233, May 2003.

[15] J. L. Hellerstein, “Achieving service rate objectiveswith decay usage scheduling”. IEEE Trans. Softw. Eng.19(8), pp. 813–825, Aug 1993.

[16] G. J. Henry, “The fair share scheduler”. AT&T BellLabs Tech. J. 63(8, part 2), pp. 1845–1857, Oct 1984.

[17] J. Kay and P. Lauder, “A fair share scheduler”. Comm.ACM 31(1), pp. 44–55, Jan 1988.

[18] B. Lin and P. A. Dinda, “VSched: Mixing batch andinteractive virtual machines using periodic real-timescheduling”. In Supercomputing, Nov 2005.

[19] B. Lin and P. A. Dinda, “Towards scheduling virtualmachines based on direct user input”. In 2nd Intl. Work-shop Virt. Tech. in Distributed Comput., 2006.

[20] W. Mauerer,Professional Linux Kernel Architecture.Wiley Publishing Inc., 2008.

[21] J. B. Nagle, “On packet switches with infinite storage”.IEEE Trans. Commun. COM-35(4), pp. 435–438, Apr1987.

[22] J. Nieh, C. Vaill, and H. Zhong, “Virtual-Time RoundRobin: An O(1) proportional share scheduler”. InUSENIX Ann. Technical Conf., pp. 245–259, Jun 2001.

[23] D. Ongaro, A. L. Cox, and S. Rixner, “Scheduling I/Oin virtual machine monitors”. In 4th Intl. Conf. VirtualExecution Environments, pp. 1–10, Mar 2008.

[24] S. Radhakrishnan, “Linux - ad-vanced networking overview”. URLhttp://qos.ittc.ku.edu/howto/howto.html, 1999.

[25] B. Schroeder and M. Harchol-Balter, “Web servers un-der overload: How scheduling can help”. ACM Trans.Internet Technology 6(1), Feb 2006.

[26] I. Stoica, H. Abdel-Wahab, and A. Pothen, “A mi-croeconomic scheduler for parallel computers”. In JobScheduling Strategies for Parallel Processing, pp. 200–218, Springer-Verlag, 1995. LNCS vol. 949.

[27] C. A. Waldspurger and W. E. Weihl, “Lottery schedul-ing: Flexible proportional-share resource manage-ment”. In 1st Symp. Operating Systems Design & Im-plementation, pp. 1–11, USENIX, Nov 1994.

14 2010/2/11