Stochastic Mechanics

Random Media

Signal Processing

and Image Synthesis

Mathematical Economics

Stochastic Optimization

and Finance

Stochastic Control

Edited by

Applications of Mathematics Stochastic Modelling and Applied Probability

43 1. Karatzas M. Yor

Advisory Board P. Bremaud E. Carlen W. Fleming D. Geman G. Grimmett G. Papanicolaou J. Scheinkman

Springer-Science+ Business Media, LLC

Applications of Mathematics

1 FleminglRishel, Determioistic and Stochastic Optimal Control (1975) 2 Marchuk, Methods of Numerical Mathematics, Second Ed. (1982) 3 Balakrishnan, Applied Functional Analysis, Second Ed. (1981) 4 Borovkov, Stochastic Processes in Queueing Theory (1976) 5 Liptser/Shiryayev, Statistics ofRandom Processes 1: General Theory (1977) 6 LiptserlShiryayev, Statistics of Random Processes fi: Applications (1978) 7 Vorob'ev, Game Theory: Lectures for Economists and Systems Scientists

(1977) 8 Shiryayev, Optimal Stopping Rules (1978) 9 Ibragimov/Rozanov, Gaussian Random Processes (1978) 10 Wonham, Linear Multivariable Control: A Geometric Approacb, Third Ed.

(1985) 11 Hida, Drownian Motion (1980) 12 Hestenes, Conjugate Direction Methods in Optimization (1980) 13 Kallianpur, Stochastic Filtering Theory (1980) 14 Kry1ov, ControUed DitTusion Processes (1980) 15 Prabhu, Stochastic Storage Processes: Queues, Insurance Risk, Dams,

and Data Communication, Second Ed. (1998) 16 Ibragimov/Has'minskii, Statistical &tImation: Asymptotic Theory (1981) 17 Cesari, Optimization: Theory and Applications (1982) 18 EIliott, Stochastic CaIculus and AppHcations (1982) 19 MarchuklShaidourov, DllTerence Methods anei Their Extrapolations (1983) 20 Hijab, StabiHzation of Control Systems (1986) 21 Protter, Stochastic Integration and DitTerential Equations (1990) 22 Benveniste/M~tivierlPriouret, Adaptive Algorithms and Stochastic

Approximations (1990) 23 KloedenIP1aten, Numerical Solution of Stochastic DilTerential Equations

(1992) 24 KushnerlDupuis, Numerical Methods for Stochastic Control Problems

In Continuous Tlme (1992) 25 FleminglSoner, Controlled Markov Processes anei Viscosity Solutions

(1993) 26 BaccelliIBremaud, Elements of Queueing Theory (1994) 27 Winkler, Jmage Analysis, Random Fields, and Dynamic Monte Carlo

Methods: An Introduction to Mathematical Aspects (1994) 28 Kalpazidou, Cycle Representations of Markov Processes (1995) 29 EIliottl Aggoun/Moore, mdden Markov Models: Estimation and Control

(1995) 30 Heml1ndez-l.erma/Lasserre, Discrete-Tlme Markov Control Processes:

Daslc Optimality Crlteria (1996) 31 DevroyeJGytlrflfLugosi, A Probabilistic Theory of Pattern Recognition (1996) 32 Maitra/Sudderth, Discrete GambUng and Stochastic Games (1996)

Jiongmin Yong Xun Yu Zhou

Stochastic Controls Hamiltonian Systems and HJB Equations

i Springer

Jiongmin Yong Department of Mathematics Fudan University Shanghai 200433, China

Managing Editors

1. Karatzas Departments of Mathematics and Statistics Columbia University New York, NY 10027, USA

M. Vor CNRS, Laboratoire de Probabilites Universite Pierre et Marie Curie 4, Place Jussieu, Tour 56 F-75252 Paris Cedex 05, France

Xun Yu Zhou Department of Systems Engineering

and Engineering Management The Chinese University of Hong Kong Shatin, Hong Kong

Mathematics Subject Classification (1991): 93E20, 49K45, 49L20, 49L25, 6OH30, 90A09. 70H20, 35CI5

Library of Congress Cataloging-in-Publication Data Yong, J. (Jiongmin), 1958-

Stochastic controls : Hamiltonian systems and HJB equations I Jiongmin Yong, Xun Yu Zhou.

p. cm. - (Applications of mathematics ; 43) lncIudes bibliographical references (p. - ) and index. ISBN 978-1-4612-7154-3 ISBN 978-1-4612-1466-3 (eBook) DOI 10.1007/978-1-4612-1466-3

1. Stochastic control theory. 2. Mathematical optimization. 3. Hamiltonian systems. 4. Hamilton-Jacobi equations. 1. Zhou, Xun Yu. Il. Title. lll. Series. QA402.37.Y66 1999 629.8'312--dc21 98-55411

Printed on acid-free paper.

© 1999 Springer-Science+Business Media New York Originally published by Springer-V erlag New York Berlin Heidelberg in 1999 Softcover reprint of the hardcover 1 st edition 1999

AII rights reserved. This work may noi be translated or copied in whole or in part without the written pennission of the publisher (Springer-Science+Business Media. LLC), except for brief excerpts in connection with reviews or scholarly analysis. Use in connection with any form of information stomge and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed is forbidden. The use of general descriptive names, tmde names, tmdemarks, etc., in this publication, even if the former are not especially identified, is not to be taken as a sign that such names, as understood by the Tmde Marks and Merchandise Marks Act, may accordingly be used freely by anyone.

Production managed by Alian Abrams; manufacturing supervised by Thomas King. Photocomposed copy prepared from the authors' :r.t.'IEJX files.

98765 4 321

ISBN 978-1-4612-7154-3 SPIN 10707573

itIit~~ if] a~ Jt-fij: :

I*mJfttt*A

To Our Parents

Wenyao Yong and Xiangxia Chen

Qingji Zhou and Caizhen Shen

Preface

As is well known, Pontryagin's maximum principle and Bellman's dynamicprogramming are the two principal and most commonly used approaches insolving stochastic optimal control problems. * An interesting phenomenonone can observe from the literature is that these two approaches have beendeveloped separately and independently. Since both methods are used toinvestigate the same problems, a natural question one will ask is the fol­lowing:

(Q) What is the relationship betwccn the maximum principlc and dy­namic programming in stochastic optimal controls?

There did exist some researches (prior to the 1980s) on the relationshipbetween these two. Nevertheless, the results usually were stated in heuristicterms and proved under rather restrictive assumptions, which were notsatisfied in most cases.In the statement of a Pontryagin-type maximum principle there is an

adjoint equation, which is an ordinary differential equation (ODE) in the(finite-dimensional) deterministic case and a stochastic differential equation(SDE) in the stochastic case. The system consisting of the adjoint equa­tion, the original state equation, and the maximum condition is referredto as an (extended) Hamiltonian system. On the other hand, in Bellman'sdynamic programming, there is a partial differential equation (PDE), offirst order in the (finite-dimensional) deterministic case and of second or­der in the stochastic case. This is known as a Hamilton-Jacobi-Bellman(HJB) equation. This leads to the following question, which is essentiallya rephrase of Question (Q):

(Q') What is thc relationship bctwccn Hamiltonian systcms and HJBcquations?

Or, even more generally,

(Q") What is thc rclationship bctwccn ODEs/SDEs and PDEs?

Once the question is asked this way, one will immediately realize thatsimilar questions have already been or are being addressed and studied inother fields. Let us briefly recall them below.

Analytic Mechanics. Using Hamilton's principle and Legendre'stransformation, one can describe dynamics of a system of particles by a

* Here, by a stochastic optimal control problem we mean a completelyobserved control problem with a state equation of the Ito type and witha cost functional of the Bolza type; see Chapter 2 for details. When thediffusion coefficient is identically zero, and the controls are restricted todeterministic functions, the problem is reduced to a deterministic optimalcontrol problem.

viii Preface

family of ODEs called Hamilton's canonical system or the Hamiltonian sys­tem. On the other hand, by introducing Hamilton's principal function, onecan describe the particle system by a PDE called the Hamilton-Jacobi (HJ)equation. These two ways are in fact equivalent in the sense that the solu­tions of the canonical system can be represented by that of the HJ equation,and vice versa. One easily sees a strong analogy between optimal controland analytic mechanics. This is not surprising, however, since the classi­cal calculus of variations, which is the foundation of analytic mechanics, isindeed the origin of optimal control theory.

Partial Differential Equations. There is a classical method of char­acteristics in solving PDEs. More specifically, for a first-order PDE, thereis an associated family of ODEs for curves, called characteristic strips, bywhich the solutions to the PDE can be constructed. In the context of(deterministic) optimal controls, the Hamiltonian system involved in themaximum principle serves as the characteristics for the HJB equation in­volved in the dynamic programming.

Stochastic Analysis. The stochastic version of the method of char­acteristics is the Feynman-Kac formula, which represents the solutions toa linear second-order parabolic or elliptic PDE by thosc to some SDEs. Onthe other hand, a reversed representation has been recently developed viathe so-called four-step scheme, which represents the solutions to a coupledforward-backward SDE by those to a PDE. A deterministic version of thisis closely rclated to the so-called invariant embedding, which was studiedby Bellman-Kalaba-Wing.

Economics. The key to understanding the economic interpretation ofoptimal control theory is the shadow price of a resource under considera­tion. The very definition of the shadow price originates from one of therelationships between the maximum principle and dynamic programming,namely, the shadow price (adjoint variable) is the rate of change of theperformance measure (value function) with respect to the change of theresource (state variable).

Finance. The celebrated Black-Scholes formula indeed gives noth­ing but a way of representing the option price (which is the solution to abackward SDE) by the solution to the Black-Scholes equation (which is aparabolic PDE).

Interestingly enough, all the relationships described abovc can be cap­tured by the following simple, gcneric mathematical formula:

y(t) = O(t,x(t)),

where (x(t), y(t)) satisfies some ODE/SDE and 0 satisfies some PDE. Forexample, in the relationship betwecn the maximum principle and dynamicprogramming, (x(t), y(t)) is the solution to the Hamiltonian system and[-OJ is the gradient in the spatial variable of the value function (which isthe solution to the HJB equation). In the Black-Scholes model, y(t) is the

Preface ix

option price, x(t) is the underlying stock price, and () is the solution to theBlack-Scholes POE.

Before studying Question (Q), one has first to resolve the following twoproblems:

(PI) What is a general stochastic maximum principle if the diffusiondepends on the control and the control domain is not necessarily convex?

This problem has been investigated since the 1960s. However, almostall the results prior to 1980 assume that the diffusion term does not dependon the control variable and/or the diffusion depends on the control butthe control domain is convex. Under these assumptions, the statements ofthe maximum principle and their proofs are very much parallel to those ofthe deterministic case. One does not see much essential difference betweenstochastic and deterministic systems from those results. The stochasticmaximum principle for systems with control-dependent diffusion coefficientsand possibly nonconvex control domains had long been an outstanding openproblem until 1988.

(P2) How is one to deal with the inherent nonsmoothness when study­ing the relationship between the maximum principle and dynamic program­ming?

The relationship unavoidably involves the derivatives of the value func­tions, which as is well known could be nonsmooth in even very simple cases.

During 1987-1989, a group led by Xunjing Li at the Institute of Math­ematics, Fudan University, including Ying Hu, Jin Ma, Shige Peng, and thetwo authors of the present book, was studying those problems and relatedissues in their weekly seminars. They insisted on tackling the control­dependent diffusion cases, and this insistence was based on the followingbelief: Only when the controls/decisions could or would influence the scaleof uncertainty (as is indeed the case in many practical systems, especiallyin the area of finance) do the stochastic problems differ from the deter­ministic ones. In the stimulating environment of the seminars, Problems(PI) and (P2) were solved almost at the same time in late 1988, based onthe introduction of the so-called second-order adjoint equation. Specifically,Peng, then a postdoctoral fellow at Fudan, solved Problem (PI) by consid­ering the quadratic terms in the Taylor expansion of the spike variation,via which he established a new form of maximum principle for stochasticoptimal controls. On the other hand, Zhou (who was then a Ph.D. studentat Fudan) found a powerful way for solving Problem (P2). By utilizing vis­cosity solution theory, he managed to disclose the relationship between thefirst-order (respectively, second-order) adjoint equations and the first-order(respectively, second-order) derivatives of the value functions. After 1989,members of the Fudan group went to different places in the world, but theresearch they carried out at Fudan formed the foundation of their further re­search. In particular, studies on nonlinear backward and forward-backward

x Preface

SDEs by Pardoux-Peng, and Ma-Yong are natural extensions of those onthe (linear) adjoint equations in the stochastic maximum principle. Thisfashionable theory soon became a notable topic among probabilists and con­trol theorists, and found interesting applications in stochastic analysis, PDEtheory, and mathematical finance. The remarkable work on the nonlinearFeynman-Kac formula (representing solutions to nonlinear PDEs by thoseto backward SDEs) by Peng and the four-step scheme (representing solu­tions to forward-backward SDEs by those to PDEs) by Ma-Protter-Yongonce again remind us about their analogy in stochastic controls, namely, therelationship between stochastic Hamiltonian systems and HJB equations.On the other hand, (stochastic) verification theorems by Zhou and Zhou­Yong-Li by means of viscosity solutions are extensions of the relationshipbetween the maximum principle and dynamic programming from open-loopcontrols to feedback controls. These verification theorems lead to optimalfeedback synthesis without involving derivatives of the value functions. Fi­nally, the recent work by Chen-Li--Zhou, Chen-Yong, and Chen-Zhou onstochastic linear quadratic (LQ) controls with indefinite control weightingmatrices in costs demonstrates how fundamentally different it is when thecontrol enters into the diffusion term. The LQ case also provides an im­portant example where the maximum principle and dynamic programmingare equivalent via the stochastic Riccati equation.

The purpose of this book is to give a systematic and self-containedpresentation of the work done by the Fudan group and related work doneby others, with the core being the study on Question (Q) or (Q'). Inother words, the theme of the book is to unify the maximum principle anddynamic programming, and to demonstrate that viscosity solution theoryprovides a nice framework to unify them. While the main context is instochastic optimal controls, we try whenever possible to disclose some in­trinsic relationship among ODEs, SDEs, and PDEs that may go beyondcontrol theory. When writing the book, we paid every attention to the co­herence and consistency of the materials presented, so that all the chaptersare closely related to each other to support the central theme. In somesense, the idea of the whole book may be boiled down to the single for­mula y(t) = O(t,x(t)), which was mentioned earlier. That said, we do notmean to trivialize things; rather we want to emphasize the common groundof seemingly different theories in different areas. In this perspective, theBlack-Scholes formula, for instance, would not surprise a person who isfamiliar with mechanics or the Feynman-Kac formula.

Let us now sketch the main contents of each chapter of the book.

Chapter 1. Since the book is intended to be self-contained, some pre­liminary materials on stochastic calculus are presented. Specifically, thischapter collects notions and results in stochastic calculus scattered aroundin the literature that are related to stochastic controls. It also unifies termi­nology and notation (which may differ in different papers/books) that are

Preface xi

to be used in later chapters. These materials are mainly for beginners (say,graduate students). They also serve as a quick reference for knowledgeablereaders.

Chapter 2. The stochastic optimal control problem is formulated andsome examples of real applications are given. However, the chapter startswith the deterministic case. This practice of beginning with deterministicproblems is carried out in Chapters 3-6 as well. The reasons for doingso are not only that the deterministic case itself may contain importantand interesting results, but also that readers can see the essential differ­ence between the deterministic and stochastic systems. In the formulationof stochastic control problems we introduce strong and weak formulationsand emphasize the difference between the two, which is not usually spelledout explicitly in the literature. Stochastic control models other than the"standard" one studied in this book are also briefly discussed. This chap­ter finally provides a very extensive literature review ranging from the veryorigin of optimal control problems to all the models and applied examplespresented in this chapter.

Chapter 3. A stochastic Hamiltonian system is introduced that con­sists of two backward SDEs (adjoint equations) and one forward SDE (theoriginal state equation) along with a maximum condition. The generalstochastic maximum principle is then stated and proved. Cases with termi­nal state constraints and sufficiency of the maximum principle are discussed.

Chapter 4. First, a stochastic version of Bellman's principle of op­timality is proved by virtue of the weak formulation, based on which HJBequations are derived. The viscosity solution is introduced as the tool tohandle the inherent nonsmoothness of the value functions. Some propertiesof the value functions and viscosity solutions of the HJB equations are thenstudied. It is emphasized that the time variable here plays a special role dueto the nonanticipativeness of the underlying system. Finally, a simplifiedproof (compared with the existing ones) of the uniqueness of the viscositysolutions is presented. Notice that the verification technique involved inthe dynamic programming is deferred to Chapter 5.

Chapter 5. Classical Hamilton-Jacobi theory in mechanics is reviewedfirst to demonstrate the origin of the study of the relationship betweenthe maximum principle and dynamic programming. The relationship fordeterministic systems is investigated and is compared with the method ofcharacteristics in PDE theory, the Feynman-Kac formula in probabilitytheory, and the shadow price in economics. The relationship for stochasticsystems is then studied. It starts with the case where the value functionis smooth to give some insights, followed by a detailed analysis for thenonsmooth case. Finally, stochastic verification theorems workable for thenonsmooth situation are given, and the construction of optimal feedbackcontrols is discussed.

Chapter 6. This chapter investigates a special case of optimal controlproblems, namely, the linear quadratic optimal control problems (LQ prob-

xii Preface

lems). They constitute an extremely important class of optimal controlproblems, and the solutions of LQ problems exhibit elegant properties dueto their simple and nice structures. They also nicely exemplify the generaltheory developed in Chapters 3-5. In the chapter an LQ problem is firsttreated as an optimization problem in an infinite-dimensional space, andabstract results are obtained to give insights. Then linear optimal statefeedback is established via the so-called stochastic Riccati equation. It ispointed out that both the maximum principle and dynamic programmingcan lead to the stochastic Riccati equation, by which one can see moreclearly the relationship between the maximum principle and dynamic pro­gramming (actually, these two approaches are equivalent in the LQ case).We emphasize that the control weighting matrices in the cost are allowedto be indefinite in our formulation. Therefore, it is essentially differentfrom the deterministic case. Stochastic Riccati equations are extensivelystudied for various cases. Finally, as an example, a mean-variance portfolioselection is solved by the LQ method developed.

Chapter 7. This chapter presents the latest development on backwardand forward-backward SDEs, with an emphasis on the relationship betweennonlinear SDEs and nonlinear PDEs. Although the topics in this chaptergo beyond the scope of stochastic controls, they originate from stochasticcontrols as mentioned earlier. The chapter begins with the original argu­ment of Bismut for studying linear backward SDEs by using the martingalerepresentation theorem. Then the existence and uniqueness of solutions tononlinear backward SDEs are investigated for two types of time durations,finite deterministic horizon and random horizon, by virtue of two differentmethods. Feynman-Kac-type formulae with respect to both forward andbackward SDEs arc presented. Next, a kind of inverse of the Feynman-Kac­type formulae, the so-called four-step scheme, which represents solutions toforward--backward SDEs by those to PDEs, is discussed. Solvability andnonsolvability of forward-backward SDEs are also analyzed. Finally, theBlack-Scholes formula in option pricing is derived by the four-step scheme.

The idea of writing such a book was around in late 1994 when JY wasvisiting XYZ in Hong Kong. While discussing the stochastic verificationtheorems, they realized that the series of works done by the Fudan groupwere rich enough for a book, and there should be a book as a system­atic account of these results. The plan became firm with encouragementfrom Wendell Fleming (Brown), Ioannis Karatzas (Columbia), and Xun­jing Li (Fudan). The authors are greatly indebted to Robert Elliott (Al­berta), Wendell Fleming (Brown), Ulrich Haussmann (British Columbia),Ioannis Karatzas (Columbia), Thomas Kurtz (Wisconsin-Madison), MeteSoner (Princeton), and Michael Taksar (SUNY-Stony Brook), who sub­stantially reviewed some or all chapters, which led to a much improvedversion. Michael Kohlmann (Konstanz) and Andrew Lim (CUHK) readcarefully large portions of the manuscript and offered numerous helpfulsuggestions. During various stages in the prolonged, four-year course of

Preface xiii

the project, many experts and friends have shown their concern and en­couragement, conveyed their comments, or sent their research works forthe book. Among them the following deserve special mention: AlainBensoussan (CNES), Leonard Berkovitz (Purdue), Giuseppe Da Prato(Scuola Normale Superiore), Darrell Duffie (Stanford), Tyrone Duncan(Kansas), Fausto Gozzi (Scuola Normal Superiore), Suzanne Lenhart (Ten­nessee), Zhuangyi Liu (Minnesota-Duluth), John Moore (ANU), Makiko Ni­sio (Kobe), Bozenna Pasik-Duncan (Kansas), Thomas Seidman (Maryland­Baltimore County), Hiroshi Tanaka (Keio), Wing Wong (CUHK), Jia-anVan (Academia Sinica), George Yin (Wayne State), and Qing Zhang (Geor­gia). Especially, both authors would like to express appreciation to theirlong-time teachers and/or colleagues in the Fudan group: Shuping Chen(Zhejiang), Ying Hu (Rennes), Xunjing Li (Fudan), Jin Ma (Purdue), ShigePeng (Shandong), and Shanjian Tang (Fudan), whose elegant research workprovided a rich source for this book.JY would like to acknowledge the partial support from the Natural Sci­

ence Foundation of China, the Chinese Education Ministry Science Foun­dation, the National Outstanding Youth Foundation of China, and the LiFoundation at San Francisco, USA. In particular, with the financial supportof a Research Fellowship of the Chinese University of Hong Kong (CUHK),JY visited XYZ at CUHK in the Spring of 1998 for half a year, which madeit possible for the two authors to fully concentrate on finalizing the book.In the career of XYZ, he has been influenced enormously by three

scholars: Xunjing Li (Fudan), Makiko Nisio (Kobe), and Hiroshi Tanaka(Keio), whom he has the privilege of having worked with for a substantialperiod of time, and he would like to take this opportunity to pay them hishighest respect. Also, he would like to acknowledge the support from theResearch Grant Council and Additional Funding for Industry Support ofthe Hong Kong Government, and the Mainline Research Scheme of CUHK.It has been a truly enjoyable experience to work with the staff at

Springer-Verlag, especially the executive editor of statistics John Kimmeland the copyeditor David Kramer, whose helpful and professional servicesfor the book are gratefully acknowledged.Last, but not least, both authors would like to thank their families for

their long-lasting support and love.

JY, ShanghaiXYZ, Hong Kong

November 1998

Contents

Preface viiNotation xixAssumption Index xxiProblem Index xxii

Chapter 1. Basic Stochastic Calculus 1

1. Probability 11.1. Probability spaces 11.2. Random variables 41.3. Conditional expectation 81.4. Convergence of probabilities 13

2. Stochastic Processes 152.1. General considerations 152.2. Brownian motions 21

3. Stopping Times 234. Martingales 275. Ito's Integral 305.1. Nondifferentiability of Brownian motion 305.2. Definition of Ito's integral and basic properties 325.3. Ito's formula 365.4. Martingale representation theorems 38

6. Stochastic Differential Equations 406.1. Strong solutions 416.2. Weak solutions 446.3. Linear SDEs 476.4. Other types of SDEs 48

Chapter 2. Stochastic Optimal Control Problems 51

1. Introduction 512. Deterministic Cases Revisited 523. Examples of Stochastic Control Problems 553.1. Production planning 553.2. Investment vs. consumption 563.3. Reinsurance and dividend management 583.4. Technology diffusion 593.5. Queueing systems in heavy traffic 60

4. Formulations of Stochastic Optimal Control Problems 624.1. Strong formulation 624.2. Weak formulation 64

5. Existence of Optimal Controls 655.1. A deterministic result 655.2. Existence under strong formulation 67

xvi Contents

5.3. Existence under weak formulation 696. Reachable Sets of Stochastic Control Systems 756.1. Nonconvexity of the reachable sets 766.2. Noncloseness of the reachable sets 81

7. Other Stochastic Control Models 857.1. Random duration 857.2. Optimal stopping 867.3. Singular and impulse controls 867.4. Risk-sensitive controls 887.5. Ergodic controls 897.6. Partially observable systems 89

8. Historical Remarks 92

Chapter 3. Maximum Principle and StochasticHamiltonian Systems 101

1. Introduction 1012. The Deterministic Case Revisited 1023. Statement of the Stochastic Maximum Principle 1133.1. Adjoint equations 1153.2. The maximum principle and stochastic

Hamiltonian systems 1173.3. A worked-out example 120

4. A Proof of the Maximum Principle 1234.1. A moment estimate 1244.2. Taylor expansions 1264.3. Duality analysis and completion of the proof 134

5. Sufficient Conditions of Optimality 1376. Problems with State Constraints 1416.1. Formulation of the problem and the maximum principle 1416.2. Some preliminary lemmas 1456.3. A proof of Theorem 6.1 149

7. Historical Remarks 153

Chapter 4. Dynamic Programming and HJB Equations 157

1. Introduction 1572. The Deterministic Case Revisited 1583. The Stochastic Principle of Optimality and the HJB Equation. 1753.1. A stochastic framework for dynamic programming 1753.2. Principle of optimality 1803.3. The HJB equation 182

4. Other Properties of the Value Function 1844.1. Continuous dependence on parameters 1844.2. Semiconcavity 186

5. Viscosity Solutions 1895.1. Definitions 1895.2. Some properties 196

Contents xvii

6. Uniqueness of Viscosity Solutions , 1986.1. A uniqueness theorem 1986.2. Proofs of Lemmas 6.6 and 6.7 208

7. Historical Remarks 212

Chapter 5. The Relationship Between the MaximumPrinciple and Dynamic Programming 211

1. Introduction 2172. Classical Hamilton-Jacobi Theory 2193. Relationship for Deterministic Systems 2273.1. Adjoint variable and value function: Smooth case 2293.2. Economic interpretation 2313.3. Methods of characteristics and the Feynman-Kac formula. 2323.4. Adjoint variable and value function: Nonsmooth case 2353.5. Verification theorems 241

4. Relationship for Stochastic Systems 2474.1. Smooth case 2504.2. Nonsmooth case: Differentials in the spatial variable 2554.3. Nonsmooth case: Differentials in the time variable 263

5. Stochastic Verification Theorems 2685.1. Smooth case 2685.2. Nonsmooth case 269

6. Optimal Feedback Controls 2757. Historical Remarks 278

Chapter 6. Linear Quadratic Optimal Control Problems ..... 281

1. Introduction 2812. The Deterministic LQ Problems Revisited 2842.1. Formulation 2842.2. A minimization problem of a quadratic functional 2862.3. A linear Hamiltonian system 2892.4. The Riccati equation and feedback optimal control 293

3. Formulation of Stochastic LQ Problems 3003.1. Statement of the problems 3003.2. Examples 301

4. Finiteness and Solvability 3045. A Necessary Condition and a Hamiltonian System 3086. Stochastic Riccati Equations 3137. Global Solvability of Stochastic Riccati Equations 3197.1. Existence: The standard case 3207.2. Existence: The case C = 0, S = 0, and Q, G ~ O 3247.3. Existence: The one-dimensional case 329

8. A Mean-variance Portfolio Selection Problem 3359. Historical Remarks 342

xviii Contents

Chapter 7. Backward Stochastic Differential Equations 345

1. Introduction 3452. Linear Backward Stochastic Differential Equations 3473. Nonlinear Backward Stochastic Differential Equations 3543.1. BSDEs in finite deterministic durations: Method of

contraction mapping 3543.2. BSDEs in random durations: Method of continuation 360

4. Feynman-Kac-Type Formulae 3724.1. Representation via SDEs 3724.2. Representation via BSDEs 377

5. Forward-Backward Stochastic Differential Equations 3815.1. General formulation and nonsolvability 3825.2. The four-step scheme, a heuristic derivation 3835.3. Several solvable classes of FBSDEs 387

6. Option Pricing Problems 3926.1. European call options and the Black-Scholes formula 3926.2. Other options 396

7. Historical Remarks 398

References 401Index 433

xEA,

x~A.

Notation

The following notation is frequently used in the book.

IRn- n-dimensional real Euclidean space.

IRnxm - the set of all (n x m) real matrices.

sn -- the set of all (n x n) symmetric matrices.

S+ -- the set of all (n x n) nonnegative definite matrices.

5-::. - the set of all (n x n) positive definite matrices.

tr (A) - the trace of the square matrix A.

x T -- the transpose of the vector (or matrix) x.

( . , .) - inner product in some Hilbert space.

Q - the set of all rational numbers.

IN - the set of natural numbers.

INI - Lebesgue measure of the set N.~ - Defined to be (see below).

I A -- the indicator function of the set A: I A (x) ~ { 1,0,

<p+ ~ max{<p,O}, <p- ~ - min{<p,O}.

aVb~max{a,b}, al\b~min{a,b}.

2° --- the set of all subsets of D.

¢ -- the empty set.

C([O, T]; IRn) - the set of all continuous functions <p : [0, T] ~ IRn

.

C([O, 00); IRn) - the set of all continuous functions cp: [0, 00) ~ IRn

.

Cb(U) - the set of all uniformly bounded, continuous functions on U.

U(O, Tj IRn) - the set of Lebesgue measurable functions cp : [0, T] ~ IRn

such that J: 1<p(t)IPdt < 00 (p E [1,00)).£00(0, Tj IRn

) - the set of essentially bounded measurable functionscp: [O,T] ~ IRn

.

(D, F, P) - probability space.

{Fdt20 - filtration.

(D, F, {Fdt20, P) - filtered probability space.

B(U) - the Borel a-field generated by all the open sets in U.

a(~) ~~-l(F) - the a-field generated by the random variable~.

a(A)- the smallest a-field containing the class A,~ ~

Vol Fa = a(Ua F et ), /\et Fa = net Fa·

xx Notations

p{ = PO~-l - the probability measure induced by the randomvariable ~.

EX - the expectation of the random variable X.

Cov(X, Y) ~ E[(X - EX)(Y - EY)Tj, Var X ~ Cov(X, X).

E(XIQ) - conditional expectation of X given Q.L~(Dj Rn) ~ the set of Rn-valued Q-measurable random variables

X such that EIXIP < 00 (p E [1,00)).Lg(D; R n) - the set of bounded Rn-valued Q-measurable

random variables.

Lj.(O, Tj R n) - the set of all {Fh20-adapted Rn-valued processes

X(·) such that E J:: IX(t)[pdt < 00.L';(O, T; R") - the set of {Fh2o-adapted Rn-valued essentially

bounded processes.

Lj.(Dj C([O, T]j R n)) - the set of {Ftlt2o-adapted Rn-valuedcontinuous processes X (.) such thatESUPtE[O,TjIX(t)IP < 00 (p E [1,00)).

M 2 [0, T] - the set of square-integrable martingales.M~[O, TJ - the set of square-integrable continuous martingales.M 2 ,loc[0, T] --- the set of square-integrable local martingales.M~,loc[O,T] ~. the set of square-integrable continuous local

martingales.

W"[O, T] ~ C([O, T]; R"), W" ~ C([O, 00); R").C" -- the set of Borel cylinders in W"[O, s].C -- the set of Borcl cylinders in wn.

W~[O,T] ~{((-I\ t) 1((-) E wn[O,T]}, t E [O,T].

Bt(W"[O, T]) ~ B(W;'[O, T]), t E [0,T].

Bt+(W"[O, T]) ~ n..>t B.. (wn[o, T]), t E [0, T).

W~ ~{((-I\ t) 1((·) E wn}, t 2': 0.

Bt(W")~B(W~), t 2': O.

Bt+(wn) ~ n..>t Bs(wn), t 2': 0.

Ay(U) - the set of all {Bt+(W"[O, T])}t20-progressively measurableprocesses 'f/ : [0, T] x W" [0, TJ -+ U.

An(u) --- the set of all {Bt+(Wn}t20-progressivcly measurableprocesses 'f/: [0,00) x wn -+ U.

V[O, TJ - the set of all measurable functions u : [0, T] -+ U.Urn, T] - the set of all {Ftl t2o-adapted processes u : [0, TJ x D -+ U.Vad[O, TJ - the set of deterministic admissible controls.

U;d[O, T] - the set of (stochastic) strong admissible controls.U~[O,T] - the set of (stochastic) weak admissible controls.

Assumption Index

(B) 355(B)' 363(B)" 377(B)1I1 380(Dl) 102, 159, 228(D2)-(D3) 102(D2)' 159, 228(D3)' 228(D4) 106(DEl)-(DE4) 66(F) 373(F)' 376(FBl)-(FB3) 387(FB2)' 390(FB2)" 391(H) 42(H)' 44(Hl)-(H2) 68(Ll) 301(L2) 319(RC) 49(80)-(82) 114(83) 114, 249(81)'-(82)' 177,248(83)' 187(84) 138(85) 143(8El )-(8E4) 69(W) 50

Problem Index

Problem (D) 54, 103, 158Problem (D. y ) 159, 228Problem (8) 115, 176Problem (8. y ) 177, 248Problem (8C) 143Problem (8L) 68Problem (88) 63Problem (W8) 64Problem (DC) 76Problem (DT) 76Problem (DLQ) 285Problem (8LQ) 301