PDE methods for Singular Perturbations in Deterministic and Stochastic Control...

PDE methods for Singular Perturbations inDeterministic and Stochastic Control Problems

João Meireles

University of Padua, Italy

Doctoral Days 2013, Palaiseau, 10-13 June 2013SADCO

João Meireles (Università di Padova) SP problems ENSTA, Paris, June 2013 1 / 38

Plan

Introduction and statement of the problem

The Hamilton-Jacobi approach to Singular Perturbations

Unbounded fast variables

Work in progress


Introduction

Consider the system with two groups of variables (xs, yτ ), τ = s/ε, andsmall parameter ε > 0, governed by ODEs

xs = f (xs, ys) xs ∈ Rn,

ys = 1εg(xs, ys) ys ∈ Rm,

or by SDEs

dxs = f (xs, ys) ds + σ(xs, ys) dWs

dys = 1εg(xs, ys) ds + 1√

εν(xs, ys) dWs

Are models whose state variables evolve on different time scales. Herethe variable x has the meaning of the slow variable in the dynamicalsystem, whereas y corresponds to the fast variable of the system.


Introduction

Passing to the limit as ε→ 0 amounts to reducing the dimension of thesystem by decoupling the behavior of the fast and slow variables.

A Singular Perturbation problem

Expect ys to disappear and the limit system to depend only on the longtime regime of ys .


Introduction

There is a large mathematical and engineering literature on singularperturbation problems. The theory is classical for Ordinary DifferentialEquations and was developed later also for systems with controls bothin the deterministic (σ ≡ 0, ν ≡ 0),

xs = f (xs, ys, αs)

ys = 1εg(xs, ys, αs)

and in the stochastic case (σ 6= 0 or ν 6= 0):

dxs = f (xs, ys, αs) ds + σ(xs, ys, αs) dWs,

dys = 1εg(xs, ys, αs) ds + 1√

εν(xs, ys, αs) dWs

αs a measurable control function taking values in a given set A


Example 1: Mechanical System with Large Damping

The large time behavior of

X = F (X )− Xε

is described by x(s) := X (s/ε) that solves

ε2x = F (x)− x .

This writesxs = ys

ys = F (xs)−ysε2 ,

Singular Perturbation problem

The limit is the Quasi-Static approximation x = F (x).


Example 2: A Financial model

The evolution of a stock S with stochastic volatility σ is given by theequation

d log Ss = γ ds + σ(ys) dWs

Here there is NO control and the classical Black-Scholes formula forthe option pricing problem is derived assuming that the parameters areconstants:

γ ≡ cte, σ(ys) ≡ cte.

However the volatility σ is not really a constant, it rather looks like anergodic mean-reverting stochastic process, so it is often model asσ(ys) with ys an Ornstein-Uhlenbeck diffusion process.


Example 2: A Financial model

It is argued in the book of Fouque - Papanicolaou - Sircar 2000, thatthe process ys also evolves on a faster time scale than the stockprices. The model with stochastic volatility σ proposed is

d log Ss = γ ds + σ(ys) dWs

dys = 1ε (m − ys) + ν√

εdWs.

where the Brownian motions W. and W. can be correlated.Problem: maximize the expected utility at time t , E[h(xt )], h increasingand concave.



Example 3: Control systems on thin structures

State constraint on the z variables

xs = f (xs, zs, αs)

zs = g(xs, zs, αs) |zs| ≤ ε,

by setting ys = zs/ε becomes

xs = f (xs, εys, αs)

ys = 1εg(xs, εys, αs) |ys| ≤ 1.



The main motivation for singular perturbation problems is reducing thedimension of the state space. The analysis as ε→ 0 gives informationabout the problem after a sufficiently large time, namely, when the fastvariable y reach its regime behaviour.

The first aproach is the order reduction method originated in the workof Levinson and Tichonov on ODEs and extended to deterministiccontrol systems by several authors.

In Example 1 the formal limit

xs = f (xs, ys), 0 = g(xs, ys)

is correct: it fits in the Reduced Order Method.


Reduced Order Method

The ROM gives in the limit the DIFFERENTIAL-ALGEBRAIC system

xs = f (xs, ys, αs), 0 = g(xs, ys, αs),

provided that (roughly speaking) the "fast subsystem" (with frozen xand ε = 1)

yτ = g(x , yτ , ατ )

has an equilibrium regime with an asymptotically stabilizing feedback.

There are examples in Engineering BUT many models do NOT havethis stability property!


The H-J approach to Singular Perturbations

Consider the controlled system

dxs = f (xs, ys, , αs) ds + σ(xs, ys, , αs) dWs, x0 = x ∈ Rn

dys = 1εg(xs, ys, , αs) ds + 1√

εν(xs, ys, , αs) dWs, y0 = y ∈ Rm

and the optimal control problem of minimizing the cost functional

Jε(t , x , y , , αs) := E[∫ t

0l(xs, ys, , αs)ds + h(xt , yt )

],

as αs varies in the set of admissible control functions A(t).



A different approach to the singular perturbation problem consists ofstudying the limit as ε→ 0+ of the value function,

uε(t , x , y) := infα∈A(t)

Jε(t , x , y , α),

and characterizing it as the unique solution of a limiting Hamilton-Jacobi-Bellman equation.

This approach starts from the HJB equation in Rn+m satisfied by uε

(via dynamic programming method), that in the deterministic case is offirst order

uεt + H(

x , y ,Dxuε,Dyuε

ε

)= 0

where

H(x , y ,p,q) = maxα∈A{−p · f (x , y , α, β)− q · g(x , y , α, β)− l(x , y , α, β)} .



In the stochastic case σ 6= 0 or ν 6= 0 the HJB equation is of secondorder

uεt + H(

x , y ,Dxuε,Dyuε

ε,Dxxuε,

Dyyuε

ε,Dxyuε√

ε

)= 0

whereH = max

α∈A[Lεαuε − l(x , y , α)]

and Lεα is the infinitesimal generator of the process with constantcontrol α and ε = 1.

This HJB PDE is (degenerate) parabolic.



With initial conditions

uε(0, x , y) = h(x , y)

we have the Cauchy Problem (CPε):{uεt + maxα∈A [Lεαuε − l(x , y , α)] = 0 in (0,+∞)× Rn × Rm,

uε(0, x , y) = h(x , y) in Rn × Rm,


Plan of the Method

One expects that the limit u(t , x) does not depend on y and solves aPDE in Rn governed by an effective Hamiltonian H. Once this is found,one tries to prove that the limit of uε solves the effective PDE

ut + H(x ,Dxu.D2xxu) = 0.

If this PDE, with suitable initial conditions, has at most one solution,then we have a characterization of the limit u(t , x) and a way tocompute it, at least in principle, by solving a lower dimensional PDE.

Next, one would like to associate to the effective Hamiltonian H aneffective control problem.


Was developed in the papers:O. ALVAREZ - M. B. : SIAM J. Cont. ’01, Arch. Rat. Mech. Anal. ’03,Memoir A.M.S. ’10;O. A. - M. B. - C. MARCHI : J. D. E. ’07, ’08 (more than 2 scales)under boundedness assumptions on the fast state variables, mostly inTm, and for unbounded but uncontrolled fast variables inM. B. - A. CESARONI - L. MANCA : SIAM J. Financial Math. 2010M. B. - A. CESARONI : European J. Control 2011Problems and techniques are related to HOMOGENIZATION of H- Jequations(see P.L.Lions- Papanicolaou- Varadhan 1986, L.C.Evans 1989, ... )and with the theory of VISCOSITY SOLUTIONS for first and secondorder, degenerate parabolic, fully nonlinear equations.


Plan of the Method

RESUME OF THE METHOD:

Search effective Hamiltonian H and effective initial data h s. t.

uε(t , x , y)→ u(t , x) as ε→ 0,

u solution of

(CP)

{ut + H

(x ,Dxu,D2

xxu)

= 0 in (0,+∞)× Rn,

u(0, x) = h(x) in Rn

If possible, interpret the effective Hamiltonian H as the BellmanHamiltonian for a new effective system

dxs = f (xs, ηs) + σ(xs, ηs)dWs xs ∈ Rn, ηs ∈ E(xs)

and effective cost functional

J(t , x , η) := E[∫ t

0l(xs, ηs) ds + h(xt )

],

This is a variational limit of the initial n + m-dimensional problem.João Meireles (Università di Padova) SP problems ENSTA, Paris, June 2013 18 / 38

How to define the effective Hamiltonian H and effective initial data h ?

Important Observation: It is an important feature of the problem thatthe limit should be independent of the fast variables y . This means thatwe have got a macroscopic model in n space dimensions byeliminating the microscopic oscillations taking places in Rm on a fasterscale.

Can be informally justified by observing that sending ε→ 0 in (HJB)ε

should force the singular terms Dy uε

ε , D2yy uε

ε and Dxy uε√ε

from blowing-up.


Definition of H:

Consider the fast subsystem with frozen τ = s/ε and xετ frozen at x

(FS) dyτ = g(x , yτ , ατ )dτ + ν(x , yτ , ατ )dWτ , y0 = y

and the family of value functions in Rm with parameters x ,p ∈ Rn,M ∈ Sn

w(t , y ; x ,p,M) := infα.∈A

E[∫ t

0L(yτ , ατ ; x ,p,M)dτ

]

L(y , α; x ,p,M) := tr(

MσσT (x , y , α)

2

)+ pf (x , y , α) + l(x , y , α)

Say (FS) is ERGODIC if, for all x ,p,M,

limt→+∞

w(t , y ; x ,p,M)

t= constant (in y ), uniformly in y

=: −H(x ,p,M)


Definition of H:

Can give an alternative definition of w(t , y ; x ,p,M), by a PDE insteadof stochastic control:it is the solution in (0,∞)× Rm for frozen x ,p,M, of the degeneratePARABOLIC PDE

wt + maxα∈A

{−tr(ννT (y , α)D2

yyw)− Dyw · g(y , α)− l(y , α)}

= 0

with initial condition w(0, y) = 0 in Rm.The effective operator H can also be characterized as the UNIQUEconstant such that the Cell Problem has a solution

maxα∈A

{−tr(ννT (y , α)D2

yyχ)− Dyχ · g(y , α)− l(y , α)}

= H

χ is called the corrector (as in homogenization).


Weak convergence theorem

From now on assume the fast variables y live in a compact manifold Y ,for simplicity the torus TM (i.e., all data are Zm- periodic in y ),or they are an uncontrolled non-degenerate diffusion.

TheoremFast subsystem (FS) ergodic =⇒

uε(t , x , y)→ u(t , x) as ε→ 0,

(in the sense of relaxed semilimits, or weak viscosity limits),and u solves (in viscosity sense)

ut + H(

x ,Dxu,D2xxu)

= 0


The effective initial data h

For the Fast Subsystem with frozen x

(FS) dyτ = g(x , yτ , ατ ) dτ + ν(x , yτ , ατ ) dWτ , y0 = y ,

consider now the value function

v(t , y ; x) := infα.∈A

E [h(x , yt )]

that solves a homogeneous parabolic PDE in Rm with initial conditionv(0, y ; x) = h(x , y).

(FS) is STABILIZING (to a constant) for the cost h if, ∀ x ,

limt→+∞

v(t , y ; x) = constant (in y ), uniformly in y =: h(x)

Convergence Theorem at t = 0: Fast subsystem (FS) stabilizing forthe cost h =⇒

limt→0

limε→0

uε(t , x , y) = h(x)

(in the sense of relaxed semilimits, or weak viscosity limits).João Meireles (Università di Padova) SP problems ENSTA, Paris, June 2013 23 / 38

Main Convergence Theorem

THEOREMIf (FS) is ERGODIC and STABILIZING, then the weak viscositysemilimits of uε satisfy

ut + H(x ,Dxu.D2xxu) = 0

and u(0, x) = h(x).

There are examples where the effective Cauchy problem abovesatisfies the comparison principle among viscosity sub- andsupersolutions, and therefore uε converge locally uniformly to itsunique solution.


Main Conclusion

The initial (n + m)-dimensional HJB equation is split into

two m-dimensional ergodic-type problems(one for H and one for h),

a n-dimensional "effective" PDE

=⇒ we got the desired SEPARATION OF SCALES for the HJBequation.


The next steps

Main remaining questions:1 when is (FS) ergodic and stabilizing?2 can we find effective dynamics, running cost, and control

constraints associated to H ?

Some answers:1 for bounded fast variables (related to homogenization):

I uniformly nondegenerate fast subsystem (FS),I deterministic fast subsystem (FS) controllable by one player from

each point to any other point in a uniformly bounded time,I under nonresonance conditions on the torus;

for unbounded fast variables: uncontrolled diffusion processeswith a unique invariant measure;

2 several examples but no general recipe.


Unbounded and uncontrolled fast variables

dxs = f (xs, ys, αs) ds + σ(xs, ys, αs)dWs

dys = 1εg(xs, ys) ds + 1√

εν(xs, ys) dWs,

Assumptions on data:1 h continuous, supy h(x , y) ≤ K (1 + |x |2)

2 f , σ,g, τ, l Lipschitz in (x , y) (unif. in α) with linear growth

Then the value function uε is the unique viscosity solution of theCauchy problem (CPε) s.t.

uε(t , x , y) ≤ C(1 + |x |2 + |y |2)

by a result of Da Lio-Ley 2006.



The HJB equation in (CPε) now is

uεt +H(

x , y ,Dxuε,D2xxuε,

1√ε

D2xyuε

)− 1εLuε = 0

H(x , y ,p,M,Z ) := maxa∈A

{−tr(σσT M)− f · p − l − tr(σνZ T )

}

L := tr(ννT D2yy ) + g · Dy



Now assume on the generator L of the fast subsystemEllipticity: ∃Λ(y) > 0 s.t. ∀x ν(x , y)νT (x , y) ≥ Λ(y)ILyapunov condition: ∃w ∈ C(Rm), k > 0,R0 > 0 s.t.

−Lw ≥ k for |y | > R0, ∀x

w(y)→∞ as |y | → ∞

Then for every x ∈ Rm frozen,1 the fast subsystem

dyτ = g(x , yτ )dτ + ν(x , yτ )dWτ

is uniquely ergodic, i.e., has a unique invariant measure µx .2 Liouville property:

any bounded subsolution of −Lv = 0 is constant.



A typical sufficient condition: ellipticity + recurrence condition

lim sup|y |→∞

supx

(g(x , y) · y + tr(ννT (x , y))

)< 0

Proof by choosing as Lyapunov function w(y) = |y |2.

Example: Ornstein-Uhlenbeck process

dyτ = (m(x)− yτ )dτ + ν(x)dWτ ,

where m, ν are bounded.



Under the previous assumptions, the (weak) limit u(t , x) as ε→ 0 ofthe value function uε(t , x , y) solves

ut +∫H(x , y ,Dxu,D2

xxu,0)dµx (y) = 0 in R+ × Rn

u(0, x) =∫

h(x , y)dµx (y)

If moreover, either g, ν do not depend on x , or

1 g(x , ·), ν(x , ·) ∈ C1 and g(·, y), ν(·, y) ∈ C1b

2 Dxg,Dyg,Dxν,Dyν are Holder in y uniformly w.r.t. x

then u is the unique viscosity solution (with quadratic growth) ofand uε(t , x , y)→ u(t , x) locally uniformly on (0,∞)× Rn.



Denote 〈φ〉(x) :=∫φ(x , y)dµx (y). Then effective H and h are

h(x) = 〈h〉(x)

H(x ,p,M) = 〈H(x , ·,p,M,0)〉

CorollaryFor split systems and cost, i.e.,

f = f0(x , y) + f1(x , α), l = l0(x , y) + l1(x , α),

the linear averaging of the data is the correct limit, i.e.,

limε→0

uε(t , x , y) = u(t , x) := infα∈A

E[∫ t

0〈l〉(xs, αs) ds + 〈h〉(xt )

],

dxs = 〈f 〉(xs, αS) ds + 〈σσT 〉1/2(xs) dWs



In general, however, for system or cost NOT split,

H(x ,p,M) = 〈maxA{...}〉 > max

A〈{...}〉

and the limit problem is not obvious.For some classical problems it was derived the explicit form of theeffective control problem:

Merton problem with stochastic volatility,Ramsey model of optimal economic growth with (fast) randomparameters,Vidale-Wolfe advertising model with random parameters,advertising game in a duopoly with Lanchester dynamics andrandom parameters.


Work in progress

the problem of unbounded and controlled fast variables is largelyopen;

an "abstract" representation of some effective control problemscan be obtained by the limit of occupational measures studied byArtstein, Gaitsgory, Borkar,...;

homogenization of deterministic differential games is wide open:there are easy examples of non-convergence of the valuefunctions and only a few known cases of convergence (see alsoCardaliaguet ’09)

a solution to an ergodic problem in Rn is proved in Ischihara’12.


Work in progress

dxs = f (xs, ys, αs) ds + σ(xs, ys, αs) dWs

dys = 1ε ξs ds + 1√

εdWs

Cost Functional:

Jε(x , y , t , α.ξ) = E(x ,y)

[∫ T

t(l(xs, ys, αs) +

1m∗|ξs|m

∗)ds + g(XT ,YT )

]

(m∗ > 1)

Value Function:V ε(x , y , t) = inf

α,ξJε(x , y , t , α.ξ)


Work in progress

Dynamic Programming method:

−V εt + H(x , y ,DxV ε,D2

xxV ε,D2

xyV ε

√ε

)− 1ε

∆yV ε +1

m′εm′ |DyV ε|m′ = 0

where 1m∗ + 1

m′ = 1 and

H(x , y ,p,X ,Z ) = maxα{−tr(

σσT

2(x , y , α)X )− F (x , y , α) · p

−l(x , y , α)− 2tr(σZ T )}

Cell Problem: for frozen x ,p,X

H(x , y ,p,X ,0)−∆yχ+1m′|Dyχ|m

′= H(x ,p,X )

which is in Ichihara form.João Meireles (Università di Padova) SP problems ENSTA, Paris, June 2013 36 / 38

H(x , y ,p,X ,0)−∆yχ+1m′|Dyχ|m

′= H(x ,p,X )

If we are in the same conditions of Ischihara’12 then we can concludethe existence and uniqueness of both the effective Hamiltonian and thecorrector

Goal: Prove regularity conditions for the effective Hamiltonian.

done in some cases;no general result


Thanks for your attention !


PDE methods for Singular Perturbations in Deterministic and Stochastic Control...

Documents

Transcript of PDE methods for Singular Perturbations in Deterministic and Stochastic Control...