Adaptive management ofepidemiological interventionsDaniel Merl ∗† and Marc Mangel ‡

†Department of Mathematics, Malott Hall, Cornell University, Ithaca NY 14853, and ‡Department of Applied Mathematics and Statistics, Baskin School of Engineering,

University of California, Santa Cruz, 1156 High St, Santa Cruz CA 95064

Submitted to Proceedings of the National Academy of Sciences of the United States of America

Epidemiological interventions such as vaccination, quarantine,and treatment aim to control the spread of infectious disease in dif-ferent ways, and carry with them different collateral costs. Here wepropose a stochastic dynamic programming framework for gen-erating optimal combined interventions designed to curtail thespread of disease while minimizing the total cost of the epidemic.The strategies produced through this process are adaptive; ad-justing the schedules of vaccination, quarantine, and treatment inanticipation of the trajectory of the epidemic. The adaptive inter-ventions are demonstrated to be more cost- and resource-efficientsolutions to the problem of epidemiological control than standardfixed policies.

keywords: SIR models | vaccination strategies | stochastic dynamic pro-

gramming

Epidemiological interventions generally remove susceptible in-dividuals or apply treatment to infected individuals in order to

prevent further spread of a disease, but these goals can be accom-plished in several ways. The susceptible population may be culled, asin the case of foot-and-mouth disease [21, 9], in which case the totalpopulation size is permanently reduced. The infected population maybe quarantined, as in the case of SARS [15], in which case total popu-lation size is not changed, but the fraction of susceptibles in potentialcontact with the disease is reduced. Under quarantine, infected indi-viduals may be isolated from the population of susceptible individualsuntil natural recovery occurs, or they may removed and the infectiontreated at some additional cost. Additionally, susceptibles may bevaccinated, as in the case with influenza or smallpox [12], in whichcase the total number of susceptibles but not the total population sizeis reduced.

Each of these actions incurs a quantifiable epidemiological cost.For culling, the cost is an additional number of deaths; for quarantinethe cost is likely to be measured in monetary units rather than lives; forvaccination the cost may be measured in both monetary units as wellthe number of additional vaccine-induced infections; for treatment thecost is once again monetary. Furthermore, the costs associated witheach of these actions can depend upon the state of the disease withinthe population of interest: for example, per dosage prices of vaccinecan increase as resources become scarce as a result of aggressive vac-cination efforts. Similarly, vaccine efficacy can decrease as a resultof selection for drug resistant strains, also as a result of aggressivevaccination efforts. These observations above beg the question ofhow to find optimal epidemiological interventions in a manner thatadaptively depends upon the state of the epidemic.

Much of the current literature on intervention strategy has soughtto answer a slightly different question; namely, where to apply vacci-nation efforts conditional upon some explicit characterization of theage and spatial structures of the host population. Ferguson et al [10]conducted large-scale computer simulations of an influenza outbreakon the Southeast Asian peninsula by incorporating the actual rural-and developed-population structures of the region into their model in

order to explore the benefits of antiviral prophylaxis, targeted vs massvaccination, and social-distancing policies such as quarantine, schooland business closings, and travel restrictions. Halloran et al [12] hadconducted a similar study three years earlier in the context of a small-pox outbreak in a hypothetical community. Riley and Ferguson [20]recently created an analogous simulation of a smallpox epidemic in apopulation designed to model that of the United Kingdom, and Ger-mann et al [11] have simulated an influenza outbreak in a populationdesigned to model that of the continental United States. Such studiesare comparative, but there is no real optimality criterion by which theoverall effectiveness of the intervention strategy is maximized throughvariation over policy space.

The difficulty of calculating optimal intervention strategies liesin devising ways to characterize and explore an infinite-dimensionpolicy space; thus, most existing work on optimal intervention hasrequired various limiting assumptions about the forms of such strate-gies. Ball and Lyne [5] considered optimal vaccination in terms ofthe allocation of vaccine doses to households of various size in anexplicitly structured population model. Patel et al [18] consideredoptimal vaccination in terms of the allocation of vaccine doses todifferent age classes in an explictly age- and geographically- struc-tured population model. Both of these methods are concerned withthe situation of pre-emptive vaccination; all vaccination is assumedto be completed before the onset of the epidemic. The necessarycomputation becomes more difficult in the situation of responding toan epidemic as it is unfolding, and thus assumptions about the formof the intervention strategy become more restrictive. Tildesley etal [21] describe optimal vaccination strategies for a foot-and-mouthepidemic in which the optimized parameter is the size of the radiussurrounding a point of infection within which all livestock are to bevaccinated. Recently, Clancy and Green [7] revisted some classic butoverlooked work by Abakuks [1, 2] in which optimal interventionsare considered to be functions of the disease state, in terms of num-bers of susceptibles and infectives. The goal of such work was tocalculate optimal thresholds in order to divide the disease state spaceinto regions corresponding to three different intervention strategies:isolation of infectives, non-intervention, and vaccination of suscepti-bles. Optimality was measured in terms of the expected cost of theepidemic under a given intervention strategy, using a cost functionthat incorporated the respective costs of isolation, non-intervention,and vaccination. In order to achieve analytical results, Abakuks hadto assume that under an isolation regime, a single infective is isolatedin concert with each naturally occuring transition of disease state (i.e.essentially constant isolation), and that under the vaccination regime,all remaining susceptibles are instantly immunised (i.e. mass vacci-

Conflict of interest footnote placeholder

Insert ’This paper was submitted directly to the PNAS office.’ when applicable.

∗To whom correspondence should be addressed. E-mail: merl@math.cornell.edu

c©2006 by The National Academy of Sciences of the USA

www.pnas.org — — PNAS Issue Date Volume Issue Number 1–6

nation).Under such assumptions, there was never reason to consider si-

multaneous isolation and immunisation; as soon as the immunisationpolicy was triggered, the epidemic would be eradicated by mass vac-cination. However in realistic epidemic scenarios, total mass vacci-nation of susceptibles and continuous removal of infectives will notbe implementable strategies, and therefore some synergistic blend ofthe two policies will be called for. Here we generalize and expand onAbakuks’ work on optimal intervention by considering combined in-terventions that can consist of simultaneous isolation and vaccinationof independent proportions of the infective and susceptible subpopu-lations, and by allowing a wider class of cost functions. Permittinggreater flexibility in the form of the possible interventions renderscalculation of optimal intervention strategies analytically intractable,but lends itself very nicely to evaluation by stochastic dynamic pro-gramming (SDP) [8, 6]. Stochastic dynamic programming is a gen-eralization of linear programming methods designed to perform op-timization of a function dependent upon the outcome of a stochasticprocess. In our case, the stochastic process is the underlying SIR(Susceptible-Infected-Recovered) model describing the dynamics ofdisease transmission and recovery, and the function to be optimizedis a measure of epidemiological cost. This cost function, also calledthe stochastic dynamic programming equation, is a recursive functionthat equates the total expected cost of the epidemic with the cost ofthe current action, plus the mean cost averaged over the distribution offuture states of the epidemic assuming the optimal actions, whateverthey may be, are carried out in all future states. The optimal actionassociated with each epidemiological state is calculated iteratively,starting with the “smallest” epidemiological state, to which all futurestates will eventually transition.

The policies produced by the SDP framework are optimal, in thatthey minimize the expected cost of the epidemic, and adaptive in thatthe optimal policy changes as a function of the state of the epidemic(in terms of numbers of suscpetibles and infectives). We demonstratethe advantages of the adaptive intervention strategies using simula-tions modeled after an influenza outbreak at a British boarding school[17]. We compare the distribution of costs arising from epidemio-logical intervention under the adaptive policies to those arising fromintervention under non-adpative policies (i.e. policies not dependenton the state of the epidemic), and find the adaptive policies to resultin lower total costs, and more efficient use of available resources.

Results and DiscussionMurray [17] describes a classic epidemiological data set arising froma 1978 outbreak of influenza at a British boarding school. Duringthe course of the epidemic, which was traced to the arrival of a sin-gle infectious student, all 763 students were eventually infected. Theepidemic conforms almost perfectly to the assumptions of standardSIR models: a population closed to immigration and emigration, re-covery with immunity, and near homogenous mixing of susceptiblesand infectives. Murray provides estimates of the transmission rateb = 0.00218 and recovery rate ν = 0.4. The transmission functionconsidered here includes an additional dispersion parameter, char-acterizing the degree of “clumpiness” of social interactions betweeninfectives and suceptibles, which we set equal to equal to 10 (seeMethods for more details on the underlying SIR model). We set theunit cost to be that of maintaining a single infected individual forone time step (ci = 1). Because we view the transmission dynamicsas a discrete time stochastic process, these initial conditions lead toa distribution of possible outcomes for the epidemic. However, theexpectation of these these outcomes is equal to the deterministic tra-

jectories obtained by solving the underlying SIR transmission modelexplicitly (Figure 1), and so provide a reasonable approximation. Un-der these assumptions, the average total cost of the epidemic with nointervention is approximately 1900 cost units.

It will be convenient for our purposes to assume that all costs canbe expressed in a common cost unit (i.e. a dollar value), though in thediscussion we address the issue of extending these methods to allownonconformable costs (i.e. lives vs dollars). We will consider twodifferent cost valuations. First, we assume the unit cost to be that ofmaintaining an infected individual for one time unit. We assume thecost per vaccination to be slightly less than the unit cost (cv = 0.8)and the cost of removal to be three times the unit cost (cr = 3).

The expected cost surface associated with non-adaptive interven-tion policies can be visualized through level curves in the plane de-scribing expected cost as a function of the fraction of individualsvaccinated and the fraction of individuals removed (Figure 2). Herewe have made the restriction that neither fraction can exceed 70% ofthe target population. In the scenario considered here, we see thatthe optimal fixed policy is no removal and 38% vaccination, leadingto an expected cost of approximately 620 cost units (Figure 3). Thisrepresents an enormous reduction in cost from that arising from theabsence of any intervention. The success of this policy stems fromthe massive quantity of vaccine administered over the course of theintervention, with the mean number of vaccinations being approxi-mately 760 individuals. This represents vaccination of virtually allsusceptibles.

Visualization of the optimal adaptive intervention is more in-volved, since here the schedules of vaccination and removal changewith the state of the epidemic. Figure 4 depicts the optimal vaccina-tion and removal schedules for the British boarding school epidemicusing a grayscale to represent the fraction of the appropriate subpopu-lation subject to each action. In this case, the optimal adaptive policyis comprised of declining schedules of vaccination and removal, withinitially strong vaccination supplemented by a short-term removal ef-fort. The effect of adaptive management of the intervention is a furtherreduction in the expected cost of the epidemic, to approximately 560cost units (Figure 5). We also notice a clear difference between thenon-adaptive and adaptive policies in terms of the way the way thedifferent policies make use of available vaccine resources. In partic-ular, the adaptive strategy is able to curb the spread of the epidemicin a similar amount of time, for less overall cost, while making use ofonly (on average) 460 doses of vaccine, compared to the 760 expecteddoses virtually guaranteed to be administered under the non-adaptivepolicy. There is greater variability in the expected cost of the epidemicunder the adaptive policy compared to the non-adaptive policy. In thisexample, there is positive probability that the adaptive policy will re-sult in greater overall cost than that resulting from the non-adaptivepolicy. In these simulations, the maximum cost accrued under theadaptive policy was approximately 870 cost units. The potential fora significant reduction in cost over the non-adaptive policy, however,is non-negligible.

Next, we consider the effects of increasing the cost per vaccina-tion to cv = 3, in order to explore the tradeoff between vaccinationand removal when both have the same cost. Figures 6 and 8 depict theoptimal non-adaptive and adaptive policies for this scenario. The op-timal non-adaptive policy now consists of maximum removal and novaccination, while the optimal adaptive policy consists of a decliningschedule of removal supplemented by a somewhat lessened vaccina-tion effort. We observe that the line R0 = 1 provides an approximatedemarcation between strong action and weak action in the adaptive in-tervention strategy. The region R0 < 1 describes the subspace where

2 www.pnas.org — — Footline Author

fewer than one new infection are produced per infected individual,and thus under the assumptions of a closed population and recoverywith immunity, in this region the number of infected individuals willdecline until the epidemic has ended. The adaptive intervention pol-icy recognizes that due to the natural tendency for the epidemic to dieout once this R0 = 1 threshold has been crossed, continued strongintervention is no longer necessary. Once again, the mean cost asso-ciated with the adaptive strategy is lower than that associated with thenon-adaptive strategy (1368 vs 1580 cost units). Additionally, fromfigures 7 and 9 we observe a clear difference in the way the costs arepaid out over the course of the epidemic. For the adaptive interven-tion the total cost plateaus rapidly, as a result of the initial period ofstrong action, followed by decreased efforts once the R0 = 1 thresh-old is crossed. The non-adaptive strategy takes longer to eradicatethe epidemic, incurring significant expenses by continuing to enact anaggressive intervention policy beyond the point where it is required.

Our method can be modified to include more complicated diseasedynamics such as latent states or vector-communicated diseases with-out fundamental changes in the approach. Also, in many cases whileit is reasonable to assume that we know the parameters characterizingthe rate of recovery and death (ν), we may be unable to specify exactvalues for the transmission rate (b) and social interaction parameter(k). In such cases, in addition to the disease state of the popula-tion, given by (S(t), I(t))=(s, i), the epidemic is characterized bythe informational state θ = (b, k). Here we have conditioned on theinformational state. If the parameter values were unknown, it wouldbe a simple matter to couple the methodology described here with aMonte Carlo-based inference framework (such as MCMC), therebycorrectly propagating uncertainty in parameter values through to theintervention strategy.

There is an important choice to be made in assigning values to cv ,cr , and ci. A monetary valuation scheme is the most straightforward,but it may be difficult to construct such a scheme that represents allaspects of the decision. One alternative would be a valuation in whicheach cost is chosen to represent a probability of mortality. In this way,the cost to be minimized would be the expected total loss of life forthe epidemic under a given intervention strategy. By assuming thatthe recovery rate can be expressed as ν = µ+ρ where µ is the rate ofdisease-induced mortality and ρ is the rate of natural recovery fromthe infected state, we can set ci = (1 − e−ν) µ

µ+ρ, so that the cost

associated with maintaining a given number of infected individualsfor a unit of time is the number of infected individuals that are ex-pected to die in a unit of time. Similarly, situations exist where it isreasonable to assign a probability of mortality to vaccination, as in thecase of smallpox, and to removal, as in the case of culling livestock.In general, any function that can be evaluated to give the cost of action(or inaction) based on the current disease state and current parametervalues can be applied.

A related extension to our adaptive policy model would involveapplying a monetary constraint to a loss-of-life cost function. If wewere to assume pi, pv , and pr to be the probabilities of mortality asso-ciated with untreated infected individuals, vaccination, and removalas just described, and define d to be the monetary resources availablefor the intervention, then within this framework is is possible to findthe intervention that minimizes the total loss-of-life subject to the to-tal spending constraint d. Similarly, it would be possible to optimizewith respect to some selective criterion in order to preserve vaccineefficacy rather than select immediately for drug-resistant pathogens.

The utility of adaptive interventions is especially evident in situa-tions of an emerging pathogen with which the host population has noprevious experience. In such a situation, vaccines will not be imme-

diately available at the onset of the epidemic, and so a methodologyfor combining currently available actions while anticipating the futureavailability of vaccines would be of great use. Effective epidemio-logical intervention requires swift decision in consideration for thevarious direct and indirect costs of intervention as described here.The methodological framework we describe provides a sound basisfor automating this decision process.

Materials and MethodsSIR Model. In order to illustrate our ideas with relatively simpledynamical complexity, we consider a standard Susceptible-Infected-Removed (SIR) model [13] without loss of immunity but with mor-tality. In this model, the dynamic variables at time t are the numberof susceptible individuals, S(t); the number of infected individuals,I(t); the number of recovered individuals, R(t); and the number ofremoved/dead individuals, D(t). We assume the population is closedto immigration such that S(t)+I(t)+R(t)+D(t) = N is constant,and any three of the dynamic variables define the fourth.

To characterize the transmission of the disease, we adopt the neg-ative binomial form for the transmission function [16], so that themodel parameters are the transmission rate b, the overdispersion pa-rameter k, the death rate m, and the rate of recovery to the immuneclass ν. Under these assumptions, the SIR model is described by thefollowing system of differential equations [13, 16]:

dS

dt= −kS log

„1 +

bI

k

«[1]

dI

dt= kS log

„1 +

bI

k

«− (ν + m)I [2]

dR

dt= νI [3]

dD

dt= mI. [4]

The negative binomial distribution can be interpreted as a compoundstochastic process in which encounters between infected and suscep-tible individuals occur randomly (i.e., according to a Poisson process)such that the encounter rate varies according to a gamma distribution

with coefficient of variationq

1k

. The source of variation in encounterrate can be social networks or individual variation in susceptibility tothe disease. Thus, negative binomial transmission accounts for socialinteractions and/or network factors in disease transmission, withoutrequiring explicit characterization of the population structure.

As with the classic SIR model, a host threshold density can befound. In particular, dI

dt> 0 if

S >(ν + m)I

k log`1 + bI

k

´ , [5]

which depends upon the combination of parameters: κ = ν+mk

andβ = b

k. Equation 5 generates a host threshold density that is a func-

tion of the number of infecteds and the four parameters; it also leadsto the basic reproductive rate R0 of the disease [3, 4].

The SIR model formulation also leads immediately to a naturaldiscrete time approximation for the numbers of infections (I), recov-eries (R) and deaths (D) arising in the unit time interval from t tot+1. Holding the total number of infected individuals I constant andintegrating Equation 1 over a unit time interval gives

S(t + 1) = S(t)

»k

k + bI(t)

–k

, [6 ]

so that the fraction of susceptible individuals surviving a unit time

interval ish

kk+bI(t)

ik

. Viewed as a discrete time stochastic process,

Footline Author PNAS Issue Date Volume Issue Number 3

the number of new infections occurring between time t and t+1 whenS(t) = s and I(t) = i can be described by

I|s, i ∼ B(s, pi(i)) [7]

where pi(i) = 1 −“

kk+bi

”k

, and B(n, π) is the standard binomialdistribution. Similarly, by integrating Equations 3 and 4, we have thatthe numbers of recoveries and deaths occurring between time t andt + 1 can be described by

R|i ∼ B(i, pr) [8]D|i, r ∼ B(i− r, pd) [9]

where pr = 1− e−ν and pd = 1− e−m. The forward dynamics forthe total numbers of susceptible and infected individuals are therefore

S(t + 1) = S(t)− I|s, i [10]I(t + 1) = I(t)− R|i− D|i, r + i. [11]

Here lower case denotes the realized value of the associated capitalletter random variable. In this discrete time approximation we haveassumed a particular ordering of events, namely that recoveries oc-cur first, followed by deaths from among those infected individualswho did not recover, followed by new infections. Simulation studiesindicated that these assumptions, as well as other possible orderings,resulted in system dynamics that were equal in expectation to thedeterministic solutions to the continuous time SIR model.

Epidemiological Cost Function. The management of any type ofepidemiological intervention requires an optimality criterion by whichwe can measure the success of the action. Here we formulate the totalexpected cost of the epidemic in terms of the underlying costs for eachintervention type and the parameters of the disease process. Havingdefined this cost function, we minimize the function over policy spaceusing stochastic dynamic programming in order to find the optimalstrategy.

We consider two general types of interventions. The first type ofintervention is that in which susceptible individuals are moved directlyfrom the susceptible class to an immune/recovered class, as for ex-ample in the case of perfect vaccination. Let c1(α, s) denote the costassociated with the epidemiological intervention when S(t) = s anda fraction α of the susceptibles are prevented from risk of infection inthis way. The second type of intervention involves removing infectedindividuals such that they are no longer capable of infecting suscep-tibles, as in the case of quarantine, culling, or successful treatment ofan infected individual. Let c2(γ, i) denote the cost associated withthe intervention when I(t) = i and a fraction γ of infected individ-uals are removed in this way. This second cost component may alsoinclude other costs associated with I(t), such as the costs associatedwith maintaining the non-removed portion of infected individuals andcosts associated deaths, as in

c2(γ, i) = crγi + ct(1− γ)i + cdd,

where cr is the cost per removal of an infected individual, ct is thecost per treatment/maintenance of a non-removed infected individual,and cd is the cost per death.

Assuming the initial epidemiological state is S(0) = s0, I(0) =i0, the expected total cost of the epidemic under intervention strategy(α, γ) can be expressed recursively as

E{C0} = c1(α, s0) + c2(γ, i0) + E{C1},

where E{Ct} denotes the expected cost accumulated from time tonwards. The optimal intervention strategy (α, γ) is the one thatminimizes the total accumulated cost over the course of the epidemic.

Non-adaptive Intervention Policies.If a single interventionstrategy is applied, unchanged, for the full course of the epidemic,the expected accumulated cost depends only on the initial epidemi-ological state (s0, i0). In this case, Monte Carlo simulation can beused to search over values of α and γ in order to find the combinationthat minimizes E{C0}. We call this the optimal non-adaptive policy.

Adaptive Intervention Policies. In order to find the optimal inter-vention as a function of epidemiological state, we introduce the valuefunction

V (s0, i0) = min(α,γ)

E{C0|(s0, i0)}. [12]

Here minα,γ indicates taking a minimum over all possible values ofthe parameters α and γ, which will be allowed to vary in time accord-ing to epidemiological state (s, i). Here E{•} denotes expectationover the dynamics of the disease, that is over the distributions of newinfections, new recoveries, and new deaths. Note that although time isimplicit in Equation 12, in the sense that the epidemiological state isS(t) = st, I(t) = it, it is not explicit; it depends only on the currentnumbers of susceptible and infected individuals. For this reason, thesolution of Equation 12 will be more complicated than the standardequations of stochastic dynamic programming [8, 6].

The Dynamic Programming Equation. In deriving the equationsatisfied by the value function for an arbitrary epidemiological state(s, i), we will refer to i, r and s, as the realized values of the randomvariables given by Equations 7-9. Given the epidemiological state(s, i), the cost in a unit interval of time is c1(α, s) + c2(γ, i) (recallthat the latter of which may also include costs associated with main-taining non-removed infectives, as well as deaths). At the end of thisinterval of time, the number of susceptibles will be s(1− α)− i andthe number of infecteds will be (1 − γ)i + i − r − d, so that theiteration equation characterizing the value function is

V (s, i) = minα,γ

E{c1(α, s) + c2(γ, i)+ [13]

V (s(1− α)− i, i + i− r − d)}.

Solution by Value Iteration. Equation 13 can be viewed as a selfconsistency condition for filling in the matrix V (s, i), but doing thisis not completely trivial. Along the boundaries s = 0 and i = 0, theentries are easily computed. For all other values, we employ valueiteration [6, 19] which is a means of converting a time independentconsistency condition such as Equation 13 into a convergent sequenceof consistency conditions, converging to the solution of Equation 13.

We begin by defining V0(s, i) that satisfies

V0(s, i) = E{c1(α, s) + c2(γ, i)+ [14]V0(s(1− α)− i, i + i− r − d)},

which is a series of linear equations for V0(s, i, θ). Subsequently, wedefine VN+1(s, i, θ) by

VN+1(s, i, θ) = minα,γ

E[c1(α, s) + c2(γ, i)+ [15]

VN (s(1− α)− i, i + i− r − d)],

and under very general conditions the sequence defined by Equations14 and 15 converges to the solution of Equation 13.

The stochastic nature of our epidemiological process model lendsitself to Monte Carlo-based techniques. Since Equation 14 does not

4 www.pnas.org — — Footline Author

involve a minimum over (α, γ), the solution to 14 can be obtained bychoosing an abitrary policy (we choose α = 0 and γ = 0) and fillingthe matrix V0[s, i] first along the boundaries s = 0 and i = 0, andthen for increasing values of i along increasing values of s (i.e. incolumn-major order).

In the boundary condition i = 0, V0[s, i] = 0. When s = 0,we fill V0[0, 1], V0[0, 2], . . . , V0[0, N ] (for increasing values of i), bycomputing the Monte Carlo mean

V0[0, i] = Er,d{c1(0, 0) + c2(0, i)+

V0[0, i− r − d]},

where the expectation is approximated by the sample mean of V0 com-puted for a large number of draws from the distributions of i, r, andd. Non-boundary cases can be computed in the same fashion, so longas the matrix is filled in the order of increasing i for each (increasing)s.

Computation of subsequent VN+1 is completely analogous, ex-cept that the values of VN computed during the previous iteration aresubstituted into the recurrence. For each entry in the matrix VN+1,

the expected cost is computed for each policy combination on a gridspanning policy space (in our case: [0, 1]× [0, 1]). For each entry westore the intervention policy resulting in the minimum VN+1[s, i], thevalue of that minimum cost, and the number of times that minimiz-ing policy was selected for the current value of (s, i). As the valueiteration proceeds and subsequent V matrices are filled, we use theaverage cost for the most frequently chosen cost-minimizing policyfor the recurrence in Equation 15.

After a burn-in period these averages and counts (but not the previ-ously filled VN ) are reset to 0, and we iterate for a sampling period. Atthe end of the sampling iterations, for each (s, i), we have the numberof times each intervention policy was chosen as the cost-minimizingpolicy, and the mean cost associated with each policy (averaged overthe times that policy minimized the cost). Thus, our method leadsto full distributions of optimal policies for each (s, i), though at thispoint we simply regard the mode of these distributions to be the opti-mal policies for each epidemiological state.

This research was partially supported by DARPA Contract DAAD19-03-0162and NSF Grant DMS 03-10542.

1. Abakuks, A. (1973) J. Appl. Prob. 10, 247-262.

2. Abakuks, A. (1974) Adv. Appl. Prob. 6, 494-511.

3. Anderson, R. M. (1991) Bulletin of Mathematical Biology 53, 3-32.

4. Anderson, R. M. and May, R. M. (1991) in Infectious Diseases of Humans: Dynamics and Control(Oxford University Press).

5. Ball, F., and Lyne, O. (2002) Mathematical Biosciences 177 & 178, 333-354.

6. Bertsekas, D. (2001) in Dynamic Programming and Optimal Control (Athena Scientific).

7. Clancy, D., and Green, N. (2007) Mathematical Biosciences 205, 297-314.

8. Clark, C. W. and Mangel, M. (2000) in Dynamic State Variable Models in Ecology: Methods andApplications (Oxford University Press).

9. Enserink, M. (2001) Science 294, 26-27.

10. Ferguson, N., Cummings, D., Cauchemez, S., Fraser, C., Riley, S., Meeyai, A., Iamsirithaworn, S.,and Burke, D. (2005) Nature 209-214.

11. Germann, T., Kadau, K., Longini Jr, I. M., and Macken, C. A. (2006) PNAS 103, 5935-5940.

12. Halloran, M. E., Longini, I. M., Nizam, A., and Yang, Y. (2002) Science 298, 1428-1432.

13. Hetchote, H. W. (2000) SIAM Review 42, 599-653.

14. Kaplan, E., Craft, D., and Wein, L. (2002) PNAS 99, 10935-10940.

15. Lloyd-Smith, J. O., Galvani, A. P., and Getz, W. M. (2003) Proceedings of the Royal Society ofLondon 270, 1979-1989.

16. McCallum, H., Barlow, N., and Hone, J. (2001) Trends in Ecology and Evolution 16, 295-300.

17. Murray, J. D. (2002) in Mathematical Biology I: An Introduction (Springer Verlag).

18. Patel, R., and Longini, I. M., and Halloran, M. E. (2005) Journal of Theoretical Biology 234,201-212.

19. Puterman, M. (1994) in Markov Decision Processes: Discrete Stochastic Dynamic Programming(Wiley Interscience).

20. Riley, S., and Ferguson, N. (2006) PNAS 103, 12637-12642.

21. Tildesley, M. J., Savill, N. J., and Shaw, D. J. et al (2006) Nature 440, 83-86.

Footline Author PNAS Issue Date Volume Issue Number 5

0 5 10 15 20 25 30

020

040

060

080

0

time (days)

susc

eptib

le

0 5 10 15 20 25 30

020

040

060

080

0

time (days)

infe

cted

0 5 10 15 20 25 30

020

040

060

080

0

time (days)

reco

vere

d

0 5 10 15 20 25 30

050

015

0025

00

time (days)

cost

Fig. 1. Mean and central 90% intervals for the numbers of susceptibles, infectives, recoveries, and associated costs under non-intervention over the course of 30days. The mean total cost accrued by day 30 is approximately 1900 cost units.

6 www.pnas.org — — Footline Author

0.0 0.2 0.4 0.6

0.0

0.2

0.4

0.6

αα

γγ

● (0.38,0)

Fig. 2. Level curves of the expected cost surface under non-adaptive intervention policies when (ci, cv , cr) = (1, 0.8, 3). Darker shades of grey indicate lowercosts. The expected cost is minimized at (α, γ) = (0.38, 0).

Footline Author PNAS Issue Date Volume Issue Number 7

0 5 10 15 20 25 30

020

040

060

080

0

time (days)

susc

eptib

le/v

acci

nate

d

SV

0 5 10 15 20 25 30

020

040

060

080

0

time (days)

infe

cted

/rem

oved

IR

0 5 10 15 20 25 30

020

040

060

080

0

time (days)

reco

vere

d

0 5 10 15 20 25 30

050

015

0025

00

time (days)

cost

Fig. 3. Mean and central 90% intervals for the numbers of susceptibles, infectives, recoveries, and associated costs under the optimal non-adaptive interventionpolicy (α, γ) = (0.38, 0). The mean total cost accrued by day 30 is approximately 620 cost units. The mean number of vaccinations dispensed is approximately760.

8 www.pnas.org — — Footline Author

0 200 400 600

020

040

060

0

susceptible

infe

cted

αα == 0αα == 0.7

0 200 400 600

020

040

060

0

susceptible

infe

cted

γγ == 0γγ == 0.7

Fig. 4. Visualization of the optimal adaptive vaccination (left) and removal (right) policies as a function of the epidemiological state when (ci, cv , cr) = (1, 0.8, 3).Darker shades indicate larger fractions of the target population to be subjected to the associated policy.

Footline Author PNAS Issue Date Volume Issue Number 9

0 5 10 15 20 25 30

020

040

060

080

0

time (days)

susc

eptib

le/v

acci

nate

d

SV

0 5 10 15 20 25 30

020

040

060

080

0

time (days)

infe

cted

/rem

oved

IR

0 5 10 15 20 25 30

020

040

060

080

0

time (days)

reco

vere

d

0 5 10 15 20 25 30

050

015

0025

00

time (days)

cost

Fig. 5. Mean and central 90% intervals for the numbers of susceptibles, infectives, recoveries, and associated costs under the optimal adaptive intervention policy.The mean total cost accrued by day 30 is approximately 560 cost units. The mean number of vaccinations dispensed is approximately 460.

10 www.pnas.org — — Footline Author

0.0 0.2 0.4 0.6

0.0

0.2

0.4

0.6

αα

γγ

● (0,0.7)

Fig. 6. Level curves of the expected cost surface under non-adaptive intervention policies when (ci, cv , cr) = (1, 3, 3). Darker shades of grey indicate lowercosts. The expected cost is minimized at (α, γ) = (0, 0.7).

Footline Author PNAS Issue Date Volume Issue Number 11

0 5 10 15 20 25 30

020

040

060

080

0

time (days)

susc

eptib

le/v

acci

nate

d

SV

0 5 10 15 20 25 30

020

040

060

080

0

time (days)

infe

cted

/rem

oved

IR

0 5 10 15 20 25 30

020

040

060

080

0

time (days)

reco

vere

d

0 5 10 15 20 25 30

050

015

0025

00

time (days)

cost

Fig. 7. Mean and central 90% intervals for the numbers of susceptibles, infectives, recoveries, and associated costs under the optimal non-adaptive interventionpolicy (α, γ) = (0, 0.7). The mean total cost accrued by day 30 is 1580 cost units. The mean number of removals is 456 individuals.

12 www.pnas.org — — Footline Author

0 200 400 600

020

040

060

0

susceptible

infe

cted

R0 == 1

αα == 0αα == 0.7

0 200 400 600

020

040

060

0

susceptible

infe

cted

R0 == 1

γγ == 0γγ == 0.7

Fig. 8. Visualization of the optimal adaptive vaccination (left) and removal (right) policies as a function of the epidemiological state when (ci, cv , cr) = (1, 3, 3).Darker shades indicate larger fractions of the target population to be subjected to the associated policy. Epidemiological states left of R0 = 1 correspond to the regionfor which dI

dt< 0.

Footline Author PNAS Issue Date Volume Issue Number 13

0 5 10 15 20 25 30

020

040

060

080

0

time (days)

susc

eptib

le/v

acci

nate

d

SV

0 5 10 15 20 25 30

020

040

060

080

0

time (days)

infe

cted

/rem

oved

IR

0 5 10 15 20 25 30

020

040

060

080

0

time (days)

reco

vere

d

0 5 10 15 20 25 30

050

015

0025

00

time (days)

cost

Fig. 9. Mean and central 90% intervals for the numbers of susceptibles, infectives, recoveries, and associated costs under the optimal adaptive intervention policy.The mean total cost accrued by day 30 is approximately 1368 cost units. The mean number of vaccinations dispensed is approximately 358 units, and the meannumber of removals is 63 individuals.

14 www.pnas.org — — Footline Author