Mixing it up: Discrete and Continuous Optimal Control for ... · Mixing it up: Discrete and...

Mixing it up: Discrete and Continuous Optimal Control for Biological Models

Mixing it up: Discrete and Continuous OptimalControl for Biological Models

Suzanne Lenhart

University of Tennessee, Mathematics DepartmentNational Institute for Mathematical and Biological

Synthesis


Dedicated to the memory of Sonia Kovalevsky


Outline

1. Background to Optimal Control of ODEs/Introduction

2. Cardiopulmonary Resuscitation Model (CPR)

3. Parabolic PDE Model for Rabies in Raccoons

4. Integro-difference Models for Dispersal, Growth and Harvest

5. Dynamic Fishery Problem

SHOW key mathematical features that make each of theseapplications different

Mathematical results can tell us something new about thebiological situations!


Work supported by the National Science Foundation and

Oak Ridge National Laboratory and

website www.nimbios.org


Deterministic Optimal Control- ODEs

time dependent control u(t)

Find piecewise continuous control u(t) and associated statevariable x(t) to maximize

max

∫ T

0

f (t, x(t), u(t)) dt

subject to

x ′(t) = g(t, x(t), u(t))

x(0) = x0 and x(T ) free


Optimal Control and Pontryagin’s Maximum Principle

Pontryagin and his collaborators developed optimal control theoryfor ordinary differential equations about 1950.

Pontryagin’s KEY idea was the introduction of the adjoint variableswhich link the differential equations to the objective functional.

Converted problem of finding an optimal control to maximize theobjective functional subject to dynamic equations to maximizingthe Hamiltonian pointwise.

The EXISTENCE of an optimal control must be proven first beforeusing this principle (using some type of compactness argument).

The four examples will show the use of PMP ideas in new waysand extensions for new applications.


Pontryagin Maximum Principle (PMP)If u∗(t) and x∗(t) are optimal for above problem, then there existsadjoint variable λ(t) s.t.

H(t, x∗(t), u(t), λ(t)) ≤ H(t, x∗(t), u∗(t), λ(t)),

for all u, at each time, where Hamiltonian H is defined by

H(t, x(t), u(t), λ(t)) = f (t, x(t), u(t)) + λg(t, x(t), u(t)).

and

λ′(t) = −∂H(t, x(t), u(t), λ(t))

∂x

λ(T ) = 0 transversality condition


Example 1 - Cardiopulmonary Resuscitation (CPR)

Each year, more than 250,000 people die from cardiac arrest in theUSA alone. Despite widespread use of CPR the survival of patientsrecovering from cardiac arrest remains poor.

The rate of survival for CPR performed out of the hospital is 3%,while for patients in the hospital, the rate of survival is 10-15%.

Here, we consider a model for CPR allowing chest and abdomencompression and decompression.

We apply the optimal control strategy for improving resuscitationrates to a validated circulation model developed by Babbs.Reference: Babbs, Circulation 1999.


Heart Diagram

Figure 23-11 Valvular structures of the heart. The atrioventricular valves are in an open position, and the semilunar

valves are closed. There are no valves to control the flow of blood at the inflow channels (i.e., vena cava and

pulmonary veins) to the heart.

Copyright © 2005 Lippincott Williams & Wilkins. Instructor's Resource CD-ROM to Accompany Porth's Pathophysiology: Concepts of Altered Health States, Seventh Edition.

Superior

vena cavaAortic valve

Pulmonary

veins

Inferior

vena cava

Mitral

valve

Pulmonic

valve

Tricuspid

valvePapillary

muscle


Diagram of Circulation Model


State Variables

As controls, we choose the the pattern of the external pressure onthe chest and on the abdomen. The pressure state variables are asfollows:P1 pressure in abdominal aortaP2 pressure in inferior vena aortaP3 pressure in carotid arteryP4 pressure in jugular veinP5 pressure in thoracic aortaP6 pressure in right heart, superior vena cavaP7 pressure in thoracic pump and left heart.


The chosen CPR model consists of seven difference equations, withtime as the discrete underlying variable.At the step n, when time is n∆t , the pressure vector is denoted by:

P(n) = (P1(n),P2(n), ...,P7(n)).

We assume that the initial pressure values are known, when n = 0.To make the chest pressure profiles medically reasonable, assumei.e., ui (0) = ui (N − 1).

u1 = (u1(0), u1(1), ..., u1(N − 2), u1(0)),

u2 = (u2(0), u2(1), ..., u2(N − 2), u2(0)),


Difference Equations Model

for n = 1, 2, ...,N − 1 (in vector notation)

P(1) = P(0) + T1(u1(0)) + T2(u2(0)) + ∆tF (P(0)), (1)

P(n + 1) = P(n) + T1(u1(n) − u1(n − 1)) (2)

+T2(u2(n) − u2(n − 1)) + ∆tF (P(n)), (3)

T1(u1(n)) = (0, 0, 0, 0, tpu1(n), tpu1(n), u1(n)),

T2(u2(n)) = (u2(n), u2(n), 0, 0, 0, 0, 0).

Interactions between compartments in function F


Show function F (P(n)) by some of its seven components:

1

cjug

[

1

Rh

(P3(n) − P4(n)) −1

RjV (P4(n) − P6(n))

]

1

cao

[

1

Ro

V (P7(n) − P5(n)) −1

Rc

(P5(n) − P3(n))

]

+1

Ra(P5(n) − P1(n)) −

1

Rht

V (P5(n) − P6(n))

]

where the valve function is defined byV (s) = s if s ≥ 0V (s) = 0 if s ≤ 0.

Three valves: between compartments 4 - 6, 5 - 7, and 5 - 6.


Goal

Choose the control set U ⊂ ℜ2N , defined as:

U = (u1, u2)|ui (0) = ui(N − 1)

−Ki ≤ ui (n) ≤ Li , i = 1, 2, n = 0, 1, . . . ,N − 2.

NEED positive and negative values due to compression anddecompression!

We define the objective functional J(u1, u2) to be maximized

N∑

n=1

[P5(n) − P6(n)] −

N−2∑

n=0

[B1

2u21(n) +

B2

2u22(n)] (4)


Key Feature

The calculation of the pressures at the next time step requires thevalues of the controls at the current and previous time steps. Weuse extension of the discrete version of PMP.

Use derivative of the map from controls-to-states to form thesensitivity equations. Use the sensitivity operator and the theobjective functional to find the adjoint system.

Use the adjoint system to simplify the quotient below and obtainOC characterizations

0 ≥ limǫ→0+

J(u∗ + ǫl)− J(u∗)

ǫ

Numerical method: iterative method with forward-backward sweeps


Lifestick


Optimal Controls for Lifestick

0 2 4 6 8 10 12

−20

0

20

40

60

Time (s)

Che

st C

ontr

ol (

mm

Hg)

0 2 4 6 8 10 12

−20

0

20

40

60

80

100

120

Time (s)

Abd

omin

al C

ontr

ol (

mm

Hg)


Concluding Remarks about CPR

We can increase the pressure difference across the thoracic aortaand the right heart by about 25 percent.

We received a US government patent for the idea of optimalcontrol of CPR models. (Vladimir Protopopescu and Eunok Jung)

This procedure with RAPID compression and decompression cycleshas recently been recommended by several medical groups. (use adevice)

Reference: IMA Journal Mathematical Medicine and Biology 25,2008, 157-170.


Optimal Control of PDEs

There is no complete generalization of Pontryagin’s MaximumPrinciple in the optimal control of PDEs.

After setting up a PDE with a control in a specifed set and anobjective functional, proving existence of an optimal control is afirst step.


Necessary Conditions

To derive the necessary conditions, we need to differentiate themap

control → objective functional

Note that the state contributes to the objective functional, so wealso must differentiate the map

control → state

The “sensitivity” is the derivative of the control-to-state map. Thesensitivity solves a PDE, which is linearized version of the statePDE.


How to Find and Use the Adjoint Function

The formal adjoint of the operator in the sensitivity PDE is found.

Transversality Condition: final time condition λ = 0 at t = T

Nonhomogeneous term

∂integrand of J

∂state

Differentiate the objective functional J(control) with respect tothe control.

Use the adjoint problem and the sensitivity problem to simplify andobtain the explicit characterization of an optimal control.


Example 2 - Rabies in Raccoons

Rabies is a common viral disease.Transmission occurs through the bite of an infected animal.Raccoons are the primary terrestrial vector for rabies in theeastern US.Vaccine is distributed through food baits.(preventative)Medical and Economic Problem -death to humans andlivestock and COSTS.


Epidemic Model - S susceptibles, I infecteds and R immune

L1S = b(S + R) − µ1S − βSI − avS ,

L2I = βSI − µ2I ,

L3R = −µ1R + avS ,

Lku ≡∂u

∂t−

2∑

i ,j=1

(akij(x)uxi

)xj+

2∑

i=1

(bki (x)(u))xi

With initial conditions and no-flux boundary conditions onQ = Ω × (0,T )

References: R. Miller Neilan and Lenhart, preprint, Ding et al. J.Bio. Dynamics 2007, Asano et al. Math Biosc. Eng. 2008


Optimal Control Problem

Define the class of admissible controls as

V = v ∈ L∞(Q) |v : Q → [0, vmax].

Optimal Control Problem: Characterize the optimal vaccinationv∗(x , t) ∈ V which minimizes the number of infected raccoons andthe costs of vaccination.

J(v∗) = minv∈V

∫

Q

(

I (x , t) + Cv(x , t)2)

dxdt

subject to state equations with initial and boundary conditions.

Key Feature

Inclusion of detailed spatial-temporal heterogeneous environmentfor the distribution of vaccine baits.


Numerical Results -Infecteds, Domain with River, Forestand Urban Areas

Figure: Constraints on movement due to environment, and control startsat t = 21


Numerical Results -Control Vaccine Baits

Figure: Put baits in the best location to reduce the spread of disease andminimize cost


Example 3 - Integro-difference Models for Dispersal,Growth and Harvest

Integro-difference equations are discrete-time models thatpossess many of the attributes of continuous-timereaction-diffusion equations. They arise naturally inpopulation biology as models for organisms with discretegenerations and well-def. growth and dispersal stages (Kot,JMB 1992, Neubert, Kot, Lewis TPB 1995).

Integro-difference equations do a better job of estimatinginvasion speeds, Mark Kot, Do Invading Organisms Do theWave? Canadian Applied Math Quarterly, 2003

For populations with continuous spatial dispersal and discrete

generations, like insects, birds, certain plants


Growth, Dispersal and Harvest Model

Integro-difference model for density of the population:

Nt+1(x) = (1 − αt(x))

∫

Ω

k(x , y)f (Nt(y), y) dy ,

where t = 0, 1, . . . ,T − 1. The state variable N and the harvestcontrol α:

N = N(α) = (N0(x),N1(x), . . . ,NT (x))

α = (α0(x), α1(x), . . . , αT−1(x)) ,

where N0(x) is known at start.

NOTICE the order... growth f , dispersal, and harvest!

Different order: growth, harvest and then dispersal

Nt+1(x) =

∫

Ω

k(x , y)(1 − αt(y))f (Nt(y), y)dy (5)


Various Stages

Space

Space

Space

GrowthSedentary Stage

Dispersal Stage

t (x) tN (y)

f(Nt (x)) f(Nt (y))

k(x,y)

N

k(x,x)

Nt+1(x) =∫

Ωk(x , y)f (Nt(y)) dy


Maximize the Profit, with order - growth, dispersal andharvest

The objective functional (revenue less cost) is

J(α) =

T−1∑

t=0

∫

Ω

[

Ate−δtαt(x)

∫

Ω

k(x , y)f (Nt(y), y)dy − BtV (αt(x))

]

dx

where N = N(α), At price factor

e−δt discount factor, f concave, V convex

Money now is worth more than money later.

Reference: Joshi, Gaff, and Lenhart, Optimal Control Applications andMethods, 2006, Zhong and Lenhart, Systems Theory Proceeding Volume,2009.


Hybrid Model

Optimal Control of this hybrid model combines discrete PontryaginMaximum Principle and techniques for continuous spatial variables(like control of PDE’s).

This combination of techniques is novel. Our results on thisharvesting system are the first results on optimal control for thistype of hybrid system.

Key Feature

Order of events is crucial due to discrete time growth, dispersal,and harvest. We use a mixture of discrete and continuous controlanalysis techiques

Used four types of kernels, different growth functions, and order ofevents


Harvest Rate for Finite Range Kernel

0.00 0.33

0.67 1.00

Space0

2

4

68

Time

0.00

0.05

0.10

0.15

0.20

0.25

0.30

Har

vest

Figure: Optimal spatial temporal harvest to maximize profit


Example 4 - Fishery Model

Neubert (Ecology Letters, 2003)

Examined steady state of this population PDE

Nt = DNxx + rN

(

1 −N

K

)

− qE (x)N

Population spreads out by diffusion and has logistic growth andharvest

Steady state

0 = DNxx + rN

(

1 −N

K

)

− qE (x)N, 0 < x < L.


Example 4 - Fishery Model

No-take marine reserves may be a part of optimal harveststrategy designed to maximize yield.

Marine reserves can protect habitat and defend endangeredstock from overexploitation.

The establishment of marine reserves is controversial in fisherymanagement.

Marine reserves = areas where harvesting is not allowed


Neubert’s work

rescaled equation u′′ + u(1 − u) − h(x)u = 0

u = 0 at the boundary x = 0 and x = L

control h

max yield∫ L

0h(x)u(x)dx

Depending on length of domain, marine reserves are part ofoptimal harvesting strategy.

For large length, there are many intervals of noharvest(reserve), leading to ‘chattering’ (rapid changes up anddown).

For small length, there is one reserve in the middle.


Convert to First Order System and Use PMP

u′′ + u(1 − u) − h(x)u = 0

Let w = u′ and then you obtain with control h

u′ = w

w ′ = u′′ = −u(1 − u) + hu

0 LHarvest Reserve

No harvestHarvest


Parabolic Case

This work extends the work of Neubert

includes both time and space

multi-dimensional spatial domain

non-steady state

Key Feature

The presence of marine reserves in harvesting strategy arising outof the maximizing yield problem.

We have a complete rigorous analysis of bang-bang control case inPDE model with a surprising solution!

We have completed the analysis for general semilinear parabolicPDE in a multidim.domain. Here we present a simpler case.


Parabolic Fishery Model

Our fishery model in domain Q = Ω × (0,T ) with Ω ⊂ Rn is :

ut = ∆u + u(1 − u) − hu in Q (6)

with initial and boundary conditions:

u(x , 0) = u0(x) for x ∈ Ω

u(x , t) = 0 on ∂Ω × (0,T )

u represents fish population

h represents CONTROL, the proportion to be harvested


Goal

We seek to maximize the objective functional over h ∈ U:

J(h) =

∫ T

0

∫

Ω

e−δthu dx dt (7)

where U = h ∈ L∞(Q) : 0 ≤ h(x , t) ≤ M ≤ 1 is class of controlsand e−δt represents a discount factor with interest rate δ.

This problem is linear in the control. Expect bang-bang or singularoptimal control.

References: Joshi, Herrera, Lenhart, Neubert, Natural ResourceModeling, 2009 and Ding and Lenhart, NRM 2009


Existence of an Optimal Control

Solution space u in V = L2(O,T ,H10 (Ω)) with

ut in L2(0,T ;H−1(Ω)) Note u > 0 in Q.

Theorem

There exists an optimal control h∗ maximizing the functional J(h)over U.

idea of proof

Choose a maximizing sequence hn in U.

Use apriori estimates.

Use weak convergence results.


Derivation of Optimality System

Theorem

The mapping h → u = u(h) is differentiable in the following sense:

u(h + ǫl) − u(h)

ǫ ψ

weakly in V as ǫ→ 0 for any h ∈ U and l ∈ L∞(Q) s.t.

(h + ǫl) ∈ U for ǫ small. The sensitivity ψ satisfies:

ψt = ∆ψ + ψ − 2uψ − hψ − lu in Q

ψ(x , 0) = 0 for x ∈ Ωψ(x , t) = 0 on ∂Ω × (0,T )

(8)


Characterization of Optimal Control

Theorem

Given an optimal control h∗ and corresponding solution u∗ = u(h∗)there exists a weak solution λ with satisfying the adjoint equation:

−λt − ∆λ− λ+ 2u∗λ+ h∗λ+ δλ = h∗ in Q

λ(x , t) = 0 on ∂Ω × (0,T ).(9)

and transversality condition λ(x ,T ) = 0 for x ∈ Ω.

And furthermore the characterization of an OC:

h∗(x , t) =

0 if λ(x , t) > 11+δ2

if λ(x , t) = 1M if λ(x , t) < 1

(10)


Optimal Control for Unexploited Stock Initial Condition

Figure: Final time 4, length of domain 4, discount .2


Optimal Control for Different Initial Conditions

Figure: Left IC -unexploited stock, Right IC -overexploited stock (needmore protection)


Optimal Control for Different Discount Rates with ICUnexploited Stock

Figure: Obtain a marine reserve in the interior of the domain , whilemaximizing yield!


Conclusions

Key Features

1. CPR - two previous time steps feeding into the state at next step

2. Rabies - spatial and temporal control in heterogeneousenvironment

3. Integro-difference - discrete and continuous features andORDER of events

4. Fishery PDE - bang-bang control with interesting outcome!


Collaborators

1.CPR: Vladimir Protopopescu (ORNL), Eunok Jung (Konkuk U.,S. Korea), Charlie Babbs (Purdue U.)

2. Rabies in Raccoons: Rachel Miller Neilan (LSU), Wandi Ding(Middle Tenn. State U.), Les Real (Emory), Erika Asano (U. of S.Florida, St. Petersburg), Lou Gross (UT)

3. Integro-difference Equations: Peng Zhong (UT), Hem Raj Joshi(Xavier U.), Holly Gaff (Old Dominion U.), Hongwei Lou (Fudan)

4. Fishery Models: Hem Raj Joshi (Xavier U.), Mike Neubert(Wood’s Hole), Ta Herrera (Bowdoin), Wandi Ding (UT)


Upcoming OC Talks and Poster

Mixing it up: Discrete and Continuous Optimal Control for ... · Mixing it up: Discrete and...

Documents

Transcript of Mixing it up: Discrete and Continuous Optimal Control for ... · Mixing it up: Discrete and...