Nonlinear Optimal

Nonlinear Optimal Control Theory

CHAPMAN & HALL/CRC APPLIED MATHEMATICS AND NONLINEAR SCIENCE SERIES

Series Editor H. T. Banks

Published Titles

Advanced Differential Quadrature Methods, Zhi Zong and Yingyan Zhang Computing with hp-ADAPTIVE FINITE ELEMENTS, Volume 1, One and Two Dimensional Elliptic and Maxwell Problems, Leszek DemkowiczComputing with hp-ADAPTIVE FINITE ELEMENTS, Volume 2, Frontiers: Three Dimensional Elliptic and Maxwell Problems with Applications, Leszek Demkowicz, Jason Kurtz, David Pardo, Maciej Paszynski, Waldemar Rachowicz, and Adam ZdunekCRC Standard Curves and Surfaces with Mathematica®: Second Edition, David H. von SeggernDiscovering Evolution Equations with Applications: Volume 1-Deterministic Equations, Mark A. McKibbenDiscovering Evolution Equations with Applications: Volume 2-Stochastic Equations, Mark A. McKibbenExact Solutions and Invariant Subspaces of Nonlinear Partial Differential Equations in Mechanics and Physics, Victor A. Galaktionov and Sergey R. SvirshchevskiiFourier Series in Several Variables with Applications to Partial Differential Equations, Victor L. ShapiroGeometric Sturmian Theory of Nonlinear Parabolic Equations and Applications, Victor A. Galaktionov Green’s Functions and Linear Differential Equations: Theory, Applications, and Computation, Prem K. Kythe Introduction to Fuzzy Systems, Guanrong Chen and Trung Tat PhamIntroduction to non-Kerr Law Optical Solitons, Anjan Biswas and Swapan KonarIntroduction to Partial Differential Equations with MATLAB®, Matthew P. ColemanIntroduction to Quantum Control and Dynamics, Domenico D’AlessandroMathematical Methods in Physics and Engineering with Mathematica, Ferdinand F. CapMathematical Theory of Quantum Computation, Goong Chen and Zijian DiaoMathematics of Quantum Computation and Quantum Technology, Goong Chen, Louis Kauffman, and Samuel J. LomonacoMixed Boundary Value Problems, Dean G. DuffyModeling and Control in Vibrational and Structural Dynamics, Peng-Fei YaoMulti-Resolution Methods for Modeling and Control of Dynamical Systems, Puneet Singla and John L. Junkins Nonlinear Optimal Control Theory, Leonard D. Berkovitz and Negash G. Medhin Optimal Estimation of Dynamic Systems, Second Edition, John L. Crassidis and John L. JunkinsQuantum Computing Devices: Principles, Designs, and Analysis, Goong Chen, David A. Church, Berthold-Georg Englert, Carsten Henkel, Bernd Rohwedder, Marlan O. Scully, and M. Suhail ZubairyA Shock-Fitting Primer, Manuel D. Salas Stochastic Partial Differential Equations, Pao-Liu Chow

CHAPMAN & HALL/CRC APPLIED MATHEMATICS

AND NONLINEAR SCIENCE SERIES

Nonlinear Optimal Control Theory

Leonard D. Berkovitz Purdue University

Negash G. MedhinNorth Carolina State University

CRC PressTaylor & Francis Group6000 Broken Sound Parkway NW, Suite 300Boca Raton, FL 33487-2742

© 2013 by Taylor & Francis Group, LLCCRC Press is an imprint of Taylor & Francis Group, an Informa business

No claim to original U.S. Government works

Printed in the United States of America on acid-free paperVersion Date: 20120716

International Standard Book Number: 978-1-4665-6026-0 (Hardback)

This book contains information obtained from authentic and highly regarded sources. Reasonable efforts have been made to publish reliable data and information, but the author and publisher cannot assume responsibility for the validity of all materials or the consequences of their use. The authors and publishers have attempted to trace the copyright holders of all material reproduced in this publication and apologize to copyright holders if permission to publish in this form has not been obtained. If any copyright material has not been acknowledged please write and let us know so we may rectify in any future reprint.

Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying, microfilming, and recording, or in any information stor-age or retrieval system, without written permission from the publishers.

For permission to photocopy or use material electronically from this work, please access www.copy-right.com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that pro-vides licenses and registration for a variety of users. For organizations that have been granted a pho-tocopy license by the CCC, a separate system of payment has been arranged.

Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe.

Visit the Taylor & Francis Web site athttp://www.taylorandfrancis.com

and the CRC Press Web site athttp://www.crcpress.com

Contents

Foreword ix

Preface xi

1 Examples of Control Problems 1

1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 A Problem of Production Planning . . . . . . . . . . . . . . 11.3 Chemical Engineering . . . . . . . . . . . . . . . . . . . . . . 31.4 Flight Mechanics . . . . . . . . . . . . . . . . . . . . . . . . . 41.5 Electrical Engineering . . . . . . . . . . . . . . . . . . . . . . 71.6 The Brachistochrone Problem . . . . . . . . . . . . . . . . . 91.7 An Optimal Harvesting Problem . . . . . . . . . . . . . . . . 121.8 Vibration of a Nonlinear Beam . . . . . . . . . . . . . . . . . 13

2 Formulation of Control Problems 15

2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 152.2 Formulation of Problems Governed by Ordinary Differential

Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152.3 Mathematical Formulation . . . . . . . . . . . . . . . . . . . 182.4 Equivalent Formulations . . . . . . . . . . . . . . . . . . . . 222.5 Isoperimetric Problems and Parameter Optimization . . . . . 262.6 Relationship with the Calculus of Variations . . . . . . . . . 272.7 Hereditary Problems . . . . . . . . . . . . . . . . . . . . . . . 32

3 Relaxed Controls 35

3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 353.2 The Relaxed Problem; Compact Constraints . . . . . . . . . 383.3 Weak Compactness of Relaxed Controls . . . . . . . . . . . . 433.4 Filippov’s Lemma . . . . . . . . . . . . . . . . . . . . . . . . 563.5 The Relaxed Problem; Non-Compact Constraints . . . . . . 633.6 The Chattering Lemma; Approximation to Relaxed Controls 66

v

vi

4 Existence Theorems; Compact Constraints 79

4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 794.2 Non-Existence and Non-Uniqueness of Optimal Controls . . 804.3 Existence of Relaxed Optimal Controls . . . . . . . . . . . . 834.4 Existence of Ordinary Optimal Controls . . . . . . . . . . . . 924.5 Classes of Ordinary Problems Having Solutions . . . . . . . . 984.6 Inertial Controllers . . . . . . . . . . . . . . . . . . . . . . . 1014.7 Systems Linear in the State Variable . . . . . . . . . . . . . 103

5 Existence Theorems; Non-Compact Constraints 113

5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 1135.2 Properties of Set Valued Maps . . . . . . . . . . . . . . . . . 1145.3 Facts from Analysis . . . . . . . . . . . . . . . . . . . . . . . 1175.4 Existence via the Cesari Property . . . . . . . . . . . . . . . 1225.5 Existence Without the Cesari Property . . . . . . . . . . . . 1395.6 Compact Constraints Revisited . . . . . . . . . . . . . . . . . 145

6 The Maximum Principle and Some of Its Applications 149

6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 1496.2 A Dynamic Programming Derivation of the

Maximum Principle . . . . . . . . . . . . . . . . . . . . . . . 1506.3 Statement of Maximum Principle . . . . . . . . . . . . . . . 1596.4 An Example . . . . . . . . . . . . . . . . . . . . . . . . . . . 1736.5 Relationship with the Calculus of Variations . . . . . . . . . 1776.6 Systems Linear in the State Variable . . . . . . . . . . . . . 1826.7 Linear Systems . . . . . . . . . . . . . . . . . . . . . . . . . . 1866.8 The Linear Time Optimal Problem . . . . . . . . . . . . . . 1926.9 Linear Plant-Quadratic Criterion Problem . . . . . . . . . . 193

7 Proof of the Maximum Principle 205

7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 2057.2 Penalty Proof of Necessary Conditions in Finite

Dimensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2077.3 The Norm of a Relaxed Control; Compact Constraints . . . 2107.4 Necessary Conditions for an Unconstrained Problem . . . . . 2127.5 The ε-Problem . . . . . . . . . . . . . . . . . . . . . . . . . . 2187.6 The ε-Maximum Principle . . . . . . . . . . . . . . . . . . . 2237.7 The Maximum Principle; Compact Constraints . . . . . . . . 2287.8 Proof of Theorem 6.3.9 . . . . . . . . . . . . . . . . . . . . . 2347.9 Proof of Theorem 6.3.12 . . . . . . . . . . . . . . . . . . . . 2387.10 Proof of Theorem 6.3.17 and Corollary 6.3.19 . . . . . . . . . 240

vii

7.11 Proof of Theorem 6.3.22 . . . . . . . . . . . . . . . . . . . . 244

8 Examples 249

8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 2498.2 The Rocket Car . . . . . . . . . . . . . . . . . . . . . . . . . 2498.3 A Non-Linear Quadratic Example . . . . . . . . . . . . . . . 2558.4 A Linear Problem with Non-Convex Constraints . . . . . . . 2578.5 A Relaxed Problem . . . . . . . . . . . . . . . . . . . . . . . 2598.6 The Brachistochrone Problem . . . . . . . . . . . . . . . . . 2628.7 Flight Mechanics . . . . . . . . . . . . . . . . . . . . . . . . . 2678.8 An Optimal Harvesting Problem . . . . . . . . . . . . . . . . 2738.9 Rotating Antenna Example . . . . . . . . . . . . . . . . . . . 276

9 Systems Governed by Integrodifferential Systems 283

9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 2839.2 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . 2839.3 Systems Linear in the State Variable . . . . . . . . . . . . . 2859.4 Linear Systems/The Bang-Bang Principle . . . . . . . . . . . 2879.5 Systems Governed by Integrodifferential Systems . . . . . . . 2879.6 Linear Plant Quadratic Cost Criterion . . . . . . . . . . . . . 2889.7 A Minimum Principle . . . . . . . . . . . . . . . . . . . . . . 289

10 Hereditary Systems 295

10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 29510.2 Problem Statement and Assumptions . . . . . . . . . . . . . 29510.3 Minimum Principle . . . . . . . . . . . . . . . . . . . . . . . 29610.4 Some Linear Systems . . . . . . . . . . . . . . . . . . . . . . 29810.5 Linear Plant-Quadratic Cost . . . . . . . . . . . . . . . . . . 30010.6 Infinite Dimensional Setting . . . . . . . . . . . . . . . . . . 300

10.6.1 Approximate Optimality Conditions . . . . . . . . . . 30210.6.2 Optimality Conditions . . . . . . . . . . . . . . . . . . 304

11 Bounded State Problems 305

11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 30511.2 Statement of the Problem . . . . . . . . . . . . . . . . . . . . 30511.3 ǫ-Optimality Conditions . . . . . . . . . . . . . . . . . . . . . 30611.4 Limiting Operations . . . . . . . . . . . . . . . . . . . . . . . 31611.5 The Bounded State Problem for Integrodifferential Systems . 32011.6 The Bounded State Problem for Ordinary

Differential Systems . . . . . . . . . . . . . . . . . . . . . . . 32211.7 Further Discussion of the Bounded State Problem . . . . . . 32611.8 Sufficiency Conditions . . . . . . . . . . . . . . . . . . . . . . 329

viii

11.9 Nonlinear Beam Problem . . . . . . . . . . . . . . . . . . . . 332

12 Hamilton-Jacobi Theory 337

12.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 33712.2 Problem Formulation and Assumptions . . . . . . . . . . . . 33812.3 Continuity of the Value Function . . . . . . . . . . . . . . . . 34012.4 The Lower Dini Derivate Necessary Condition . . . . . . . . 34412.5 The Value as Viscosity Solution . . . . . . . . . . . . . . . . 34912.6 Uniqueness . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35312.7 The Value Function as Verification Function . . . . . . . . . 35912.8 Optimal Synthesis . . . . . . . . . . . . . . . . . . . . . . . . 36012.9 The Maximum Principle . . . . . . . . . . . . . . . . . . . . 366

Bibliography 371

Index 379

Foreword

This book provides a thorough introduction to optimal control theory for non-linear systems. It is a sequel to Berkovitz’s 1974 book entitled Optimal Con-trol Theory. In optimal control theory, the Pontryagin principle, Bellman’sdynamic programming method, and theorems about existence of optimal con-trols are central topics. Each of these topics is treated carefully. The bookis enhanced by the inclusion of many examples, which are analyzed in de-tail using Pontryagin’s principle. These examples are diverse. Some arise insuch applications as flight mechanics, and chemical and electrical engineer-ing. Other examples come from production planning models and the classicalcalculus of variations.

An important feature of the book is its systematic use of a relaxed controlformulation of optimal control problems. The concept of relaxed control isan extension of L. C. Young’s notion of generalized curves, and the relatedconcept of Young measures. Young’s pioneering work in the 1930s provided akind of “generalized solution” to calculus of variations problems with noncon-vex integrands. Such problems may have no solution among ordinary curves.A relaxed control, as defined in Chapter 3, assigns at each time a probabilitymeasure on the space of possible control actions. The approach to existencetheorems taken in Chapters 4 and 5 is to prove first that optimal relaxedcontrols exist. Under certain Cesari-type convexity assumptions, optimal con-trols in the ordinary sense are then shown to exist. The Pontryagin maximumprinciple (Chapters 6 and 7) provides necessary conditions that a relaxed orordinary control must satisfy. In the relaxed formulation, it turns out to besufficient to consider discrete relaxed controls (see Section 6.3). This is a note-worthy feature of the authors’ approach.

In the control system models considered in Chapters 2 through 8, thestate evolves according to ordinary differential equations. These models ne-glect possible time delays in state and control actions. Chapters 10, 11, and12 consider models that allow time delays. For “hereditary systems” as de-fined in Chapter 10, Pontryagin’s principle takes the form in Theorem 10.3.1.Hereditary control problems are effectively infinite dimensional. As explainedin Section 10.6, the true state is a function on a time interval [−r, 0] wherer represents the maximum time delay in the control system. Chapter 11 con-siders hereditary system models, with the additional feature that states areconstrained by given bounds. For readers interested only in control systems

ix

x Nonlinear Optimal Control Theory

without time delays, necessary conditions for optimality in bounded stateproblems are described in Section 11.6.

The dynamic programming method leads to first order nonlinear partialdifferential equations, which are called Hamilton-Jacobi-Bellman equations(or sometimes Bellman equations). Typically, the value function of an optimalcontrol problem is not smooth. Hence, it satisfies the Hamilton-Jacobi-Bellmanequation only in a suitable “generalized sense.” The Crandall-Lions Theoryof viscosity solutions provides one such notion of generalized solutions forHamilton-Jacobi-Bellman equations. Work of A. I. Subbotin and co-authorsprovides another interesting concept of generalized solutions. Chapter 12 pro-vides an introduction to Hamilton-Jacobi Theory. The results described theretie together in an elegant way the viscosity solution and Subbotin approaches.A crucial part of the analysis involves a lower Dini derivate necessary conditionderived in Section 12.4.

The manuscript for this book was not quite in final form when LeonardBerkovitz passed away unexpectedly. He is remembered for his many originalcontributions to optimal control theory and differential games, as well as forhis dedicated service to the mathematics profession and to the control com-munity in particular. During his long career at Purdue University, he was ahighly esteemed teacher and mentor for his PhD students. Colleagues warmlyremember his wisdom and good humor. During his six years as Purdue Math-ematics Department head, he was resolute in advocating the department’sinterests. An obituary article about Len Berkovitz, written by W. J. Brown-ing and myself, appeared in the January/February 2010 issue of SIAM News.

Wendell Fleming

Preface

This book is an introduction to the mathematical theory of optimal controlof processes governed by ordinary differential and certain types of differentialequations with memory and integral equations. The book is intended for stu-dents, mathematicians, and those who apply the techniques of optimal controlin their research. Our intention is to give a broad, yet relatively deep, conciseand coherent introduction to the subject. We have dedicated an entire chapterto examples. We have dealt with the examples pointing out the mathematicalissues that one needs to address.

The first six chapters can provide enough material for an introductorycourse in optimal control theory governed by differential equations. Chap-ters 3, 4, and 5 could be covered with more or less details in the mathematicalissues depending on the mathematical background of the students. For stu-dents with background in functional analysis and measure theory, Chapter 7can be added. Chapter 7 is a more mathematically rigorous version of thematerial in Chapter 6.

We have included material dealing with problems governed by integrodif-ferential and delay equations. We have given a unified treatment of boundedstate problems governed by ordinary, integrodifferential, and delay systems.We have also added material dealing with the Hamilton-Jacobi Theory. Thismaterial sheds light on the mathematical details that accompany the materialin Chapter 6.

The material in the text gives a sufficient and rigorous treatment of finitedimensional control problems. The reader should be equipped to deal withother types of control problems such as problems governed by stochastic dif-ferential equations and partial differential equations, and differential games.

I am very grateful to Mrs. Betty Gick of Purdue University and Mrs. An-nette Rohrs of Georgia Institute of Technology for typing the early and finalversions of the book. I am very grateful to Professor Wendell Fleming forreading the manuscript and making valuable suggestions and additions thatimproved and enhanced the quality of the book as well as avoided and re-moved errors. I also wish to thank Professor Boris Mordukovich for readingthe manuscript and making valuable suggestions. All or parts of the mate-rial up to the first seven chapters have been used for optimal control theorycourses in Purdue University and North Carolina State University.

This book is a sequel to the book Optimal Control Theory by Leonard

xi

xii Nonlinear Optimal Control Theory

D. Berkovitz. I learned control theory from this book taught by him. We de-cided to write the current book in 1994 and we went through various versions.

L. D. Berkovitz was my teacher and a second father. He passed away onOctober 13, 2009 unexpectedly. He was caring, humble, and loved mathemat-ics. He is missed greatly by all who were fortunate enough to have known him.This book was completed before his death.

Negash G. MedhinNorth Carolina State University

Chapter 1

Examples of Control Problems

1.1 Introduction

Control theory is a mathematical study of how to influence the behaviorof a dynamical system to achieve a desired goal. In optimal control, the goalis to maximize or minimize the numerical value of a specified quantity that isa function of the behavior of the system. Optimal control theory developed inthe latter half of the 20th century in response to diverse applied problems. Inthis chapter we present examples of optimal control problems to illustrate thediversity of applications, to raise some of the mathematical issues involved, andto motivate the mathematical formulation in subsequent chapters. It shouldnot be construed that this set of examples is complete, or that we chose themost significant problem in each area. Rather, we chose fairly simple problemsin an effort to illustrate without excessive complication.

Mathematically, optimal control problems are variants of problems in thecalculus of variations, which has a 300+ year history. Although optimal controltheory developed without explicit reference to the calculus of variations, eachimpacted the other in various ways.

1.2 A Problem of Production Planning

The first problem, taken from economics, is a resource allocation problem;the Ramsey model of economic growth. Let Q(t) denote the rate of productionof a commodity, say steel, at time t. Let I(t) denote the rate of investmentof the commodity at time t to produce capital; that is, productive capacity.In the case of steel, investment can be thought of as using steel to build newsteel mills, transport equipment, infrastructure, etc. Let C(t) denote the rateof consumption of the commodity at time t. In the case of steel, consumptioncan be thought of as the production of consumer goods such as automobiles.We assume that all of the commodity produced at time t must be allocated

1

2 Nonlinear Optimal Control Theory

to either investment or consumption. Then

Q(t) = I(t) + C(t) I(t) ≥ 0 C(t) ≥ 0.

We assume that the rate of production is a known function F of the capitalat time t. Thus, if K(t) denotes the capital at time t, then

Q(t) = F (K(t)),

where F is a given function. The rate of change of capital is given by thecapital accumulation equation

dK

dt= αI(t)− δK(t) K(0) = K0, K(t) ≥ 0,

where the positive constant α is the growth rate of capital and the positiveconstant δ is the depreciation rate of capital. Let 0 ≤ u(t) ≤ 1 denote thefraction of production allocated to investment at time t. The number u(t) iscalled the savings rate at time t. We can therefore write

I(t) = u(t)Q(t) = u(t)F (K(t))

C(t) = (1− u(t))Q(t) = (1− u(t))F (K(t)),

and

dK

dt= αu(t)F (K(t)) − δK(t)

K(t) ≥ 0 K(0) = K0.(1.2.1)

Let T > 0 be given and let a “social utility function” U , which depends onC, be given. At each time t, U(C(t)) is a measure of the satisfaction societyreceives from consuming the given commodity. Let

J =

∫ T

0

U(C(t))e−γt dt,

where γ is a positive constant. Our objective is to maximize J , which is ameasure of the total societal satisfaction over time. The discount factor e−γt

is a reflection of the phenomenon that the promise of future reward is usuallyless satisfactory than current reward.

We may rewrite the last integral as

J =

∫ T

0

U((1 − u(t))F (K(t)))e−γtdt. (1.2.2)

Note that by virtue of (1.2.1), the choice of a function u : [0, T ] → u(t), whereu is subject to the constraint 0 ≤ u(t) ≤ 1 determines the value of J . We havehere an example of a functional; that is, an assignment of a real number to

Examples of Control Problems 3

every function in a class of functions. If we relabel K as x, then the problemof maximizing J can be stated as follows:

Choose a savings program u over the time period [0, T ], that is, a functionu defined on [0, T ], such that 0 ≤ u(t) ≤ 1 and such that

J(u) = −∫ T

0

U((1− u(t))F (ϕ(t)))e−γtdt (1.2.3)

is minimized, where ϕ is a solution of the differential equation

dx

dt= αu(t)F (x) − δx ϕ(0) = x0,

and ϕ satisfies ϕ(t) ≥ 0 for all t in [0, T ]. The problem is sometimes stated asMinimize:

J(u) = −∫ T

0

U((1− u(t))F (x))e−γtdt

Subject to:

dx

dt= αu(t)F (x) − δx, x(0) = x0, x ≥ 0, 0 ≤ u(t) ≤ 1

1.3 Chemical Engineering

Let x1(t), . . . , xn(t) denote the concentrations at time t of n substances ina reactor in which n simultaneous chemical reactions are taking place. Let therates of the reactions be governed by a system of differential equations

dxi

dt= Gi

(x1, . . . , xn, θ(t), p(t)

)xi(0) = xi0 i = 1, . . . , n. (1.3.1)

where θ(t) is the temperature in the reactor at time t and p(t) is the pressurein the reactor at time t. We control the temperature and pressure at eachinstance of time, subject to the constraints

θb ≤ θ(t) ≤ θa (1.3.2)

pb ≤ p(t) ≤ pa

where θa, θb, pa, and pb are constants. These represent the minimum andmaximum attainable temperature and pressure.

We let the reaction proceed for a predetermined time T . The concentra-tions at this time are x1(T ), . . . , xn(T ). Associated with each product is aneconomic value, or price ci, i = 1, . . . , n. The price may be negative, as in the


case of hazardous wastes that must be disposed of at some expense. The valueof the end product is

V (p, θ) =

n∑

i=1

cixi(T ). (1.3.3)

Given a set of initial concentrations xi0, the value of the end product is com-pletely determined by the choice of functions p and θ if the functions Gi havecertain nice properties. Hence the notation V (p, θ). This is another exampleof a functional; in this case, we have an assignment of a real number to eachpair of functions in a certain collection.

The problem here is to choose piecewise continuous functions p and θ onthe interval [0, T ] so that (1.3.2) is satisfied and so that V (p, θ) is maximized.

A variant of the preceding problem is the following. Instead of allowing thereaction to proceed for a fixed time T , we stop the reaction when one of thereactants, say x1, reaches a preassigned concentration x1f . Now the final timetf is not fixed beforehand, but is the smallest positive root of the equationx1(t) = x1f . The problem now is to maximize

V (p, θ) =n∑

i=2

cixi(tf )− k2tf .

The term k2tf represents the cost of running the reactor.Still another variant of the problem is to stop the reaction when sev-

eral of the reactants reach preassigned concentrations, say x1 = x1f , x2 =

x2f , . . . , xj = xjf . The value of the end product is now

n∑

i=j+1

cixi(tf )− k2tf .

We remark that in the last two variants of the problem there is anotherquestion that must be considered before one takes up the problem of maxi-mization. Namely, can one achieve the desired final concentrations using pres-sure and temperature functions p and θ in the class of functions permitted?

1.4 Flight Mechanics

In this problem a rocket is taken to be a point of variable mass whosemoments of inertia are neglected. The motion of the rocket is assumed to takeplace in a plane relative to a fixed frame. Let y = (y1, y2) denote the positionvector of the rocket and let v = (v1, v2) denote the velocity vector of therocket. Then

dyi

dt= vi yi(0) = yi0 i = 1, 2, (1.4.1)


where y0 = (y10 , y20) denotes the initial position of the rocket.

Let β(t) denote the rate at which the rocket burns fuel at time t and letm(t) denote the mass of the rocket at time t. Thus,

dm

dt= −β. (1.4.2)

If a > 0 denotes the mass of the vehicle, then m(t) ≥ a.Let ω(t) denote the angle that the thrust vector makes with the positive

y1-axis at time t. The burning rate and the thrust angle will be at our disposalsubject to the constraints

0 ≤ β0 ≤ β(t) ≤ β1 ω0 ≤ ω(t) ≤ ω1, (1.4.3)

where β0, β1, ω0, and ω1 are fixed.To complete the equations of motion of the rocket we analyze the momen-

tum transfer in rectilinear rocket motion. At time t a rocket of mass m andvelocity v has momentummv. During an interval of time δt let the rocket burnan amount of fuel δµ > 0. At time t+ δt let the ejected combustion productshave velocity v′; their mass is clearly δµ. At time t+ δt let the velocity of therocket be v+ δv; its mass is clearly m− δµ. Let us consider the system whichat time t consisted of the rocket of mass m and velocity v. At time t + δtthis system consists of the rocket and the ejected combustion products. Thechange in momentum of the system in the time interval δt is therefore

(δµ)v′ + (m− δµ)(v + δv)−mv.

If we divide the last expression by δt > 0 and then let δt → 0, we obtainthe rate of change of momentum of the system, which must equal the sumof the external forces acting upon the system. Hence, if F is the resultantexternal force per unit mass acting upon the system we have

Fm− (v′ − v)dµ

dt= m

dv

dt.

If we assume that (v′ − v), the velocity of the combustion products relative tothe rocket is a constant c, and if we use dµ/dt = β, we get

F − cβ/m = dv/dt.

If we apply the preceding analysis to each component of the planar motionwe get the following equations, which together with (1.4.1), (1.4.2), and (1.4.3)govern the planar rocket motion

dv1

dt= F 1 − cβ

mcosω (1.4.4)

dv2

dt= F 2 − cβ

msinω vi(0) = vi0, i = 1, 2.


Here, the components of the force F can be functions of y and v. This wouldbe the case if the motion takes place in a non-constant gravitational field andif drag forces act on the rocket.

The control problems associated with the motion of the rocket are of thefollowing type. The burning rate control β and the thrust direction controlω are to be chosen from the class of piecewise continuous functions (or someother appropriate class) in such a way that certain of the variables t, y, v, mattain specific terminal values. From among the controls that achieve thesevalues, the control that maximizes (or minimizes) a given function of theremaining terminal values is to be determined. In other problems, an integralevaluated along the trajectory in the state space is to be extremized.

To be more specific, consider the “minimum fuel problem.” It is requiredthat the rocket go from a specified initial point y0 to a specified terminalpoint yf in such a way that the fuel consumed is minimized. This problem isimportant for the following reason. Since the total weight of rocket plus fuelplus payload that can be constructed and lifted is constrained by the state ofthe technology, it follows that the less fuel consumed, the larger the payloadthat can be carried by the rocket. From (1.4.2) we have

mf = m0 −∫ tf

t0

β(t)dt,

where t0 is the initial time, tf is the terminal time (time at which yf isreached), mf is the final mass, and m0 is the initial mass. The fuel consumedis therefore m0 −mf . Thus, the problem of minimizing the fuel consumed isthe problem of minimizing

P (β, ω) =

∫ tf

t0

β(t)dt (1.4.5)

subject to (1.4.1) to (1.4.4). This problem is equivalent to the problem ofmaximizing mf . In the minimum fuel problem the terminal velocity vector vfwill be unspecified if a “hard landing” is permitted; it will be specified if a“soft landing” is required. The terminal time tf may or may not be specified.

Another example is the problem of rendezvous with a moving object whoseposition vector at time t is z(t) = (z1(t), z2(t)) and whose velocity vector attime t is w(t) = (w1(t), w2(t)), where z1, z2, w1, and w2 are continuous func-tions. Let us suppose that there exist thrust programs β and ω satisfying(1.4.3) and such that rendezvous can be effected. Mathematically this is ex-pressed by the assumption that the solutions y, v of the equations of motioncorresponding to the given choice of β and ω have the property that theequations

y(t) = z(t) (1.4.6)

v(t) = w(t)

have positive solutions. Such controls (β, ω) will be called admissible. Since for


each admissible β and ω the corresponding solutions y and v are continuous,and since the functions z and w are continuous by hypothesis, it follows thatfor each admissible pair (β, ω) there is a smallest positive solution tf (β, ω)for which (1.4.6) holds. The number tf (β, ω) is the rendezvous time. Twoproblems are possible here. The first is to determine from among the admis-sible controls one that delivers the maximum payload; that is, to maximizemf = mf (tf (β, ω)). The second is to minimize the rendezvous time tf (β, ω).

1.5 Electrical Engineering

Example 1.5.1. A control surface on an airplane is to be kept at some arbi-trary position by means of a servo-mechanism. Outside disturbances such aswind gusts occur infrequently and are short with respect to the time constantof the servo-mechanism. A direct-current electric motor is used to apply atorque to bring the control surface to its desired position. Only the armaturevoltage v into the motor can be controlled. For simplicity we take the desiredposition to be the zero angle and we measure deviations in the angle θ fromthis desired position. Without the application of a torque the control surfacewould vibrate as a damped harmonic oscillator. Therefore, with a suitablenormalization the differential equation for θ can be written as

d2θ

dt2+ a

dθ

dt+ ω2θ = u θ(0) = θ0 θ′(0) = θ′0. (1.5.1)

Here u represents the restoring torque applied to the control surface, the termadθ/dt represents the damping effect, and ω2 is the spring constant. If nodamping occurs, then a = 0. Since the source of voltage cannot deliver avoltage larger in absolute value than some value v0, the restoring torque mustbe bounded in absolute value. Hence it follows that we must have

|u(t)| ≤ A, (1.5.2)

where A is some positive constant.If we set

x1 = θ x2 =dθ

dt

we can rewrite Eq. (1.5.1) as follows:

dx1

dt= x2 x1(0) = θ0 (1.5.3)

dx2

dt= −ax2 − ω2x1 + u x2(0) = θ′0.


FIGURE 1.1 [From: G. Stephens Jones and Aaron Strauss, An example ofoptimal control, SIAM Review, Vol. 10, 25–55 (1968).]

The problem is the following. A short disturbance has resulted in a de-viation θ = θ0 from the desired position and a deviation dθ/dt = θ′0 fromrest. How should the voltage be applied over time so that the control surfaceis brought back to the set position θ = 0, dθ/dt = 0 in the shortest possi-ble time? In terms of (1.5.3), the problem is to choose a function u from anappropriate class of functions, say piecewise continuous functions, such thatu satisfies (1.5.2) at each instant of time and such that the solution (x1, x2)of (1.5.3) corresponding to u reaches the origin in (x1, x2)-space in minimumtime.

Example 1.5.2. Figure 1.1 depicts an antenna free to rotate from any angularposition θ0 to any other angle θ1. The equation of motion under an appliedtorque T is given by

Id2θ

dt2+ β

dθ

dt= T θ(0) = θ0 θ′(0) = θ′0, (1.5.4)

where β is a damping factor and I is the moment of inertia of the systemabout the vertical axis.

The objective here is to move from the position and velocity (θ0, θ′0) at an

initial time t0 to the state and velocity (θ1, 0) at some later time t1 in a waythat the following criteria are met.

(a) The transfer of position must take place within a reasonable (but notspecified) period of time.

(b) The energy expended in making rotations must be kept within reason-


able (but not specified) bounds in order to avoid excessive wear on com-ponents.

(c) The fuel or power expended in carrying out the transfer must be keptwithin reasonable (but not specified) limits.

Since the energy expended is proportional to (dθ/dt)2 and the fuel orpower expended is proportional to the magnitude of the torque, a reasonableperformance criterion would be

J =

∫ t1

t0

(γ1 + γ2

(dθ

dt

)2

+ γ3|T |)dt,

where γ1 > 0, γ2 ≥ 0, γ3 ≥ 0, and t1 is free.The control torque T is constrained in magnitude by a quantity k > 0, that

is, |T | ≤ k, and (dθ/dt) is constrained in magnitude by 1, that is, |dθ/dt| ≤ 1.If as in Example 1.5.1 we set

x1 = θ x2 =dθ

dt,

we can write (1.5.4) as the system

dx1

dt= x2 x1(0) = θ0 (1.5.5)

dx2

dt= −β

Ix1 +

T

Ix2(0) = θ′0.

The problem then is to choose a torque program (function) T that minimizes

J(T ) =

∫ t1

t0

(γ1 + γ2(x2)2 + γ3|T |)dt

subject to (1.5.5), the terminal conditions x1(t1) = θ1, x2(t1) = 0, t1 free and

the constraints|T (t)| ≤ k |x2(t)| ≤ 1.

This example differs from the preceding examples in that we have a constraint|x2(t)| ≤ 1 on the state as well as a constraint on the control.

1.6 The Brachistochrone Problem

We now present a problem from the calculus of variations; the brachis-tochrone problem, posed by John Bernoulli in 1696. This problem can be re-garded as the starting point of the theory of the calculus of variations. Galileo


FIGURE 1.2

also seems to have considered this problem in 1630 and 1638, but was not asexplicit in his formulation.

Two points P0 and P1 that do not lie on the same vertical line are given ina vertical plane with P0 higher than P1. A particle, or point mass, acted uponsolely by gravity is to move along a curve C joining P0 and P1. Furthermore,at P0 the particle is to have an initial speed v0 along the curve C. The problemis to choose the curve C so that the time required for the particle to go fromP0 to P1 is a minimum.

To formulate the problem analytically, we set up a coordinate system inthe plane as shown in Fig. 1.2.

Let P0 have coordinates (x0, y0) with y0 > 0, let P1 have coordinates(x1, y1) with y1 > 0, and let C have y = y(x) as its equation. At time t, let(x(t), y(t)) denote the coordinates of the particle as it moves along the curveC, let v(t) denote the speed, and let s(t) denote the distance traveled. Weshall determine the time required to traverse C from P0 to P1.

From the principle of conservation of energy, we have that

1

2m(v2 − v20) = mg(y − y0). (1.6.1)

Also,

v =ds

dt=ds

dx

dx

dt= [1 + (y′)2]1/2

dx

dt. (1.6.2)

Hence, using (1.6.1) and (1.6.2), we get that

dt =[1 + (y′)2]1/2

vdx =

[1 + (y′)2

2g(y − α)

]1/2dx,

whereα = y0 − v20/2g.


Thus, the time of traverse T along C is given by

T =1

(2g)1/2

∫ x1

x0

[1 + (y′)2

y − α

]1/2dx.

The problem of finding a curve C that minimizes the time of traverse isthat of finding a function y = y(x) that minimizes the integral

∫ x1

x0

[1 + (y′)2

y − α

]1/2dx. (1.6.3)

Note that if v0 = 0, then the integral is improper.We put this problem in a format similar to the previous ones as follows.

Change the notation for the independent variable from x to t. Then set

y′ = u y(t0) = y0. (1.6.4)

A continuous function u will be called admissible if it is defined on [t0, t1],if the solution of (1.6.4) corresponding to u satisfies y(t1) = y1, if y(t) > y0on [t0, t1], and if the mapping t → [(1 + u2(t))/(y(t)− α)]1/2 is integrable on[t0, t1]. Our problem is to determine the admissible function u that minimizes

J(u) =

∫ t1

t0

(1 + u2

y − α

)1/2

dt (1.6.5)

in the class of all admissible u.The brachistochrone problem can be formulated as a control problem in

a different fashion. By (1.6.1) and (1.6.2), the speed of the particle along thecurve C is given by (2g(y − α))1/2. Hence, if θ is the angle that the tangentto C makes with the positive x-axis, then

dx

dt= (2g(y − α))1/2 cos θ

dy

dt= (2g(y − α))1/2 sin θ.

Let u = cos θ. Then the equations of motion become

dx

dt= (2g(y − α))1/2u x(t0) = x0 (1.6.6)

dy

dt= (2g(y − α))1/2(1− u2)1/2 y(t0) = y0.

The problem is to choose a control u satisfying |u| ≤ 1 such that the point(x, y), which at initial time t0 is at (x0, y0), reaches the prescribed point(x1, y1) in minimum time. If t1 is the time at which P1 is reached, then thisis equivalent to minimizing t1 − t0. This in turn is equivalent to minimizing

∫ t1

t0

dt (1.6.7)


subject to (1.6.6), the terminal condition (x1, y1), and the constraint |u(t)| ≤1.

The brachistochrone problem can be modified in the following fashion. Onecan replace the fixed point P1 by a curve Γ1 defined by y = y1(x) and seekthe curve C joining P0 to Γ1 along which the mass particle must travel if it isto go from P0 to Γ1 in minimum time. We can also replace P0 by a curve Γ0

where Γ0 is at positive distance from Γ1 and ask for the curve C joining Γ0

and Γ1 along which the particle must travel in order to minimize the time oftransit.

1.7 An Optimal Harvesting Problem

We present here a population model of McKendric type with crowdingeffect. The reformulation of the control problem coincides with the reformu-lation by Gurtin and Murphy [40], [68]. The age-dependent population modelis given by

∂p(r, t)

∂t+∂p(r, t)

∂t= −µ(r)p(r, t) − f(N(t))p(r, t) − u(t)p(r, t) (1.7.1)

p(r, 0) = p0(r)

p(0, t) = β

∫ ∞

0

k(r)p(r, t)dr, k(r) = k(r)h(r)

N(t) =

∫ ∞

0

p(r, t)dr

where p(r, t) denotes the age density distribution at time t and age r, µ(r) isthe mortality rate, k(r) is the female sex ratio at age r, h(r) is the fertilitypattern, and β is the specific fertility rate of females. The function f(N(·))indicates decline in the population due to environmental factors such as crowd-ing. The function u(·) ≥ 0 is the control or harvesting strategy.

We consider the problem of maximizing the harvest

J(u) =

∫ T

0

u(t)N(t)dt (1.7.2)

where 0 ≤ u(·) ≤M is piecewise continuous and (1.7.1) is satisfied. The upperbound M on u(·) is the maximum effort.


FIGURE 1.3 [From: H. Maurer and H. D. Mittelmann, Optimal ControlApplications and Methods, 12, 19–31 (1991).]

1.8 Vibration of a Nonlinear Beam

Consider the classical nonlinear Euler beam [56] with deflection limited byan obstacle parallel to the plane of the beam. The beam is axially compressedby a force P , which acts as a branching parameter α.

We assume that the energy of a beam that is compressed by a force P isgiven by

Iα =1

2

∫ 1

0

θ2dt+ α

∫ 1

0

cos θ(t)dt.

Here α = P/EJ , where EJ is the bending stiffness, t denotes the arc length,θ(t) is the angle between the tangential direction of the beam at t and thereference line (see Fig. 1.3), and the length of the beam is ℓ = 1.

For the deflection of the beam away from the reference line we have

x = sin θ, θ =x√

1− x2.

Hence, the energy can also be written as

Iα =1

2

∫ 1

0

x2

1− x2dt+ α

∫ 1

0

√1− x2 dt.

We assume that |x(t)| < 1, that is, −π/2 < θ(t) < π/2 holds on [0, 1].The variational problem for the simply supported beam consists of mini-

mizing the energy subject to the boundary conditions

x(0) = x(1) = 0

and the state constraints

−d ≤ x(t) ≤ d, 0 ≤ t ≤ 1, d > 0.


In the case of a clamped beam, one replaces the boundary conditions by

x(0) = 0, θ(0) = 0, x(1) = θ(1) = 0.

Chapter 2

Formulation of Control Problems

2.1 Introduction

In this chapter we discuss the mathematical structures of the examples inthe previous chapter.

We first discuss problems whose dynamics are given by ordinary differ-ential equations. We motivate and give precise mathematical formulationsand equivalent mathematical formulations of apparently different problems.We then point out the relationship between optimal control problems andthe calculus of variations. Last, we present various formulations of hereditaryproblems. These problems are also called delay or lag problems.

2.2 Formulation of Problems Governed by OrdinaryDifferential Equations

Many of the examples in the preceding chapter have the following form.The state of a system at time t is described by a point or vector

x(t) = (x1(t), . . . , xn(t))

in n-dimensional euclidean space, n ≥ 1. Initially, at time t0, the state of thesystem is

x(t0) = x0 = (x10, . . . , xn0 ).

More generally, we can require that at the initial time t0 the initial state x0 issuch that the point (t0, x0) belongs to some pre-assigned set T0 in (t, x)-space.The state of the system varies with time according to the system of differentialequations

dxi

dt= f i(t, x, z) xi(t0) = xi0 i = 1, . . . , n, (2.2.1)

where z = (z1, . . . , zm) is a vector in real euclidean space Rm and the functionsf i are real valued continuous functions of the variables (t, x, z).

15


By the “system varying according to (2.2.1)” we mean the following. Afunction u with values in m-dimensional euclidean space is chosen from someprescribed class of functions. In this section we shall take this class to be asubclass C of the class of piecewise continuous functions. When the substitu-tion z = u(t) is made in the right-hand side of (2.2.1), we obtain a system ofordinary differential equations:

dxi

dt= f i(t, x, u(t)) = F i

u(t, x) i = 1, . . . , n. (2.2.2)

The subscript u on the F iu emphasizes that the right-hand side of (2.2.2)

depends on the choice of function u. For each u in C it is assumed that thereexists a point (t0, x0) in T0 and a function φ = (φ1, . . . , φn) defined on aninterval [t0, t2] with values in R

n such that (2.2.2) is satisfied. That is, werequire that for every t in [t0, t2]

φ′i(t) =

dφi

dt= f i(t, φ(t), u(t)) φi(t0) = xi0 i = 1, . . . , n.

At points of discontinuity of u this equation is interpreted as holding for theone-sided limits. The function φ describes the evolution of the system withtime and will sometimes be called a trajectory.

The function u is further required to be such that at some time t1, wheret0 < t1, the point (t1, φ(t1)) belongs to a pre-assigned set T1 and for t0 ≤ t < t1the points (t, φ(t)) do not belong to T1. The set T1 is called the terminal setfor the problem. Examples of terminal sets, taken from Chapter 1, are givenin the next paragraph.

In the production planning problem, T1 is the line t = T in the (t, x)plane. In the first version of the chemical engineering problem, the set T1 isthe hyperplane t = T ; that is, those points in (t, x)-space with x = (x1, . . . , xn)free and t fixed at T . In the last version of the chemical engineering problem,T1 is the set of points in (t, x)-space whose coordinates xi are fixed at xif fori = 1, . . . , j and whose remaining coordinates are free. In some problems it isrequired that the solution hit a moving target set G(t). That is, at each time tof some interval [τ0, τ1] there is a setG(t) of points in x-space, and it is requiredthat the solution φ hit G(t) at some time t. Stated analytically, we require theexistence of a point t1 in [τ0, τ1] such that φ(t1) belongs to G(t1). An exampleof this type of problem is the rendezvous problem in Section 1.4. The set T1 inthe moving target set problem is the set of all points (t, x) = (t, z(t), w(t),m)with τ0 ≤ t ≤ τ1 and m > 0.

The discussion in the preceding paragraphs is sometimes summarized inless precise but somewhat more graphic language by the statement that thefunctions u are required to transfer the system from an initial state x0 at timet0 to a terminal state x1 at time t1, where (t0, x0) ∈ T0 and (t1, x1) ∈ T1.Note that to a given u in C there will generally correspond more than onetrajectory φ. This results from different choices of initial points (t0, x0) in T0

Formulation of Control Problems 17

or from non-uniqueness of solutions of (2.2.2) if no assumptions are made toguarantee the uniqueness of solutions of (2.2.2) with given initial data (t0, x0).

It is often further required that a function u in C and a correspondingsolution φ satisfy a system of inequality constraints

Ri(t, φ(t), u(t)) ≥ 0 i = 1, 2, . . . , r, (2.2.3)

for all t0 ≤ t ≤ t1, where the functions R1, . . . , Rr are given functionsof (t, x, z). For example, in the production planning problem discussed inSection 1.2 the constraints can be written as Ri ≥ 0, i = 1, 2, 3, whereR1(t, x, z) = x, R2(t, x, z) = z, and R3(t, x, z) = 1 − z. In Example 1.5.1,the constraints can be written as Ri ≥ 0, i = 1, 2, where R1(t, x, z) = z + Aand R2(t, x, z) = A− z.

In the examples of Chapter 1, the control u is to be chosen so that certainfunctionals are minimized or maximized. These functionals have the followingform. Let f0 be a real valued continuous function of (t, x, z), let g0 be a realvalued function defined on T0, and let g1 be a real valued function defined onT1. For each u in C and each corresponding solution φ of (2.2.2), define a costor payoff or performance index as follows:

J(φ, u) = g0(t0, φ(t0)) + g1(t1, φ(t1)) +

∫ t1

t0

f0(s, φ(s), u(s))ds.

If the function J is to be minimized, then a u∗ in C and a correspondingsolution φ∗ of (2.2.2) are to be found such that J(φ∗, u∗) ≤ J(φ, u) for all u inC and corresponding φ. In other problems, the functional J is to be maximized.Examples of J taken from Chapter 1 are given in the next paragraph.

In the examples of Chapter 1, the set T0 is always a point (t0, x0). Thedifferential equations in the examples, except in Section 1.3, are such that thesolutions are unique. In Section 1.3 let us assume that the functionsGi are suchthat the solutions are unique. Thus, in these examples the choice of u com-pletely determines the function φ. In the economics example, J(φ, u) is the to-tal cost J(u) given by (1.2.3). The function f0 is given by −U((1−z)F (x))e−γt

and the functions g0 and g1 are identically zero. In the first chemical engineer-ing example of Section 1.3, J(φ, u) = V (p, θ), where V (p, θ) is given by (1.3.3).The functions f0 and g0 are identically zero. In the minimum fuel problem ofSection 1.4, J(φ, u) = P (β, ω), where P is given by (1.4.5). Here f0 = β andg0 and g1 are identically zero. An equivalent formulation is obtained if onetakes J(φ, u) = −mf . Now f0 = 0, g0 = 0, and g1 = −mf .

We conclude this section with a discussion of two generalizations thatwill appear in the mathematical formulation to be given in the next section.The first deals with the initial and terminal data. The initial set T0 and theterminal set T1 determine a set B of points (t0, x0, t1, x1) in R

2n+2 as follows:

B = (t0, x0, t1, x1) : (t0, x0) ∈ T0, (t1, x1) ∈ T1. (2.2.4)

Thus, a simple generalization of the requirement that (t0, φ(t0)) ∈ T0 and


(t1, φ(t1)) ∈ T1 is the following. Let there be given a set B of points in R2n+2.

It is required of a trajectory φ that (t0, φ(t0), t1, φ(t1)) belong to B. That is,we now permit possible relationships between initial and terminal data. Weshall show later that in some sense this situation is really no more generalthan the situation in which the initial and terminal data are assumed to beunrelated.

The second generalization deals with the description of the constraintson u. For each (t, x), a system of inequalities Ri(t, x, z) ≥ 0, i = 1, . . . , rdetermines a set U(t, x) in the m-dimensional z-space; namely

U(t, x) = z : Ri(t, x, z) ≥ 0, i = 1, . . . , r.

The requirement that a function u and a corresponding trajectory satisfyconstraints of the form (2.2.3) can therefore be written as follows:

u(t) ∈ U(t, φ(t)) t0 ≤ t ≤ t1.

Thus, the constraint (2.2.3) is a special case of the following more generalconstraint condition.

Let Ω be a function that assigns to each point (t, x) of some suitable subsetof Rn+1 a subset of the z-space R

m. Thus,

Ω : (t, x) → Ω(t, x),

where Ω(t, x) is a subset of Rm. The constraint (2.2.3) is replaced by the moregeneral constraint

u(t) ∈ Ω(t, φ(t)).

2.3 Mathematical Formulation

The formulation will involve the Lebesgue integral. This is essential inthe study of solutions to the problem. The reader who wishes to keep theformulation on a more elementary level can replace “measurable controls” by“piecewise continuous controls,” replace “absolutely continuous functions” by“piecewise C(1) functions,” and interpret the solution of Eq. (2.3.1) as weinterpreted the solution of Eq. (2.2.2).

We establish some notation and terminology. Let t denote a real number,which will sometimes be called time. Let x denote a vector in real euclideanspace R

n, n ≥ 1; thus, x = (x1, . . . , xn). The vector x will be called the statevariable. We shall use superscripts to denote components of vectors and weshall use subscripts to distinguish among vectors. Let z denote a vector ineuclidean m-space R

m, m ≥ 1; thus, z = (z1, . . . , zm). The vector z will becalled the control variable. Let R be a region of (t, x)-space and let U be


a region of z-space, whereby a region we mean an open connected set. LetG = R × U , the cartesian product of R and U . Let f0, f1, . . . , fn be realvalued functions defined on G. We shall write

f = (f1, . . . , fn) f = (f0, f1, . . . , fn).

Let B be a set of points

(t0, x0, t1, x1) = (t0, x10, . . . , x

n0 , t1, x

11, . . . , x

n1 )

in R2n+2 such that (ti, xi), i = 0, 1 are in R and t1 ≥ t0 + δ, for some fixed

δ > 0. The set B will be said to define the end conditions for the problem.Let Ω be a mapping that assigns to each point (t, x) in R a subset Ω(t, x)

of the region U in z-space. The mapping Ω will be said to define the controlconstraints. If U(t, x) = U for all (t, x) in R, then we say that there are nocontrol constraints.

Henceforth we shall usually use vector-matrix notation. The system ofdifferential equations (2.2.2) will be written simply as

dx

dt= f(t, x, u(t)),

where we follow the usual convention in the theory of differential equationsand take dx/dt and f(t, x, u(t)) to be column vectors. We shall not distinguishbetween a vector and its transpose if it is clear whether a vector is a row vectoror a column vector or if it is immaterial whether the vector is a row vector ora column vector. The inner product of two vectors u and v will be written as〈u, v〉. We shall use the symbol |x| to denote the ordinary euclidean norm ofa vector. Thus,

|x| =(

n∑

i=1

|xi|2)1/2

= 〈x, x〉1/2.

If A and B are matrices, then we write their product as AB.If f = (f1, . . . , fn) is a vector valued function from a set ∆ in some

euclidean space to the euclidean space Rn such that each of the real value

functions f1, . . . , fn is continuous (or C(k), or measurable, etc.) then we shallsay that f is continuous (or C(k), or measurable, etc.) on the set ∆. Similarly,if a matrix A has entries that are continuous functions (or C(k), or measurablefunctions, etc.) defined on a set ∆ in some euclidean space, then we shall saythat A is continuous (or C(k), or measurable, etc.) on ∆.

Definition 2.3.1. A control is measurable function u defined on an interval[t0, t1] with range in U.

Definition 2.3.2. A trajectory corresponding to a control u is an absolutelycontinuous function φ defined on [t0, t1] with range in R

n such that:

(i) (t, φ(t)) ∈ R for all t in [t0, t1]


(ii) φ is a solution of the system of differential equations

dx

dt= f(t, x, u(t)); (2.3.1)

that is,φ′(t) = f(t, φ(t), u(t)) a.e. on [t0, t1].

The point (t0, φ(t0)) will be called the initial point of the trajectoryand the point (t1, φ(t1)) will be called the terminal point of the trajec-tory. The point (t0, φ(t0), t1, φ(t1)) will be called the end point of thetrajectory.

Note that since φ is absolutely continuous, it is the integral of its derivative.Hence (ii) contains the statement that the function t → f(t, φ(t), u(t)) isLebesgue integrable on [t0, t1].

The system of differential equations (2.3.1) will be called the state equa-tions.

We emphasize the following about our notation. We are using the letterz to denote a point of U; we are using the letter u to denote a function withrange in U.

Definition 2.3.3. A control u is said to be an admissible control if thereexists a trajectory φ corresponding to u such that

(i) t→ f0(t, φ(t), u(t)) is in L1[t0, t1].

(ii) u(t) ∈ Ω(t, φ(t)) a.e. on [t0, t1].

(iii) (t0, φ(t0), t1, φ(t1)) ∈ B.

A trajectory corresponding to an admissible control as in Definition 2.3.3will be called an admissible trajectory.

Definition 2.3.4. A pair of functions (φ, u) such that u is an admissiblecontrol and φ is an admissible trajectory corresponding to u will be called anadmissible pair.

Note that to a given admissible control there may correspond more thanone admissible trajectory as a result of different choices of permissible endpoints. Also, even if we fix the endpoint, there may be several trajectoriescorresponding to a given control because we do not require uniqueness ofsolutions of (2.3.1) for given initial conditions.

We now state the control problem.

Problem 2.3.1. Let A denote the set of all admissible pairs (φ, u) and let Abe non-empty. Let

J(φ, u) = g(t0, φ(t0), t1, φ(t1)) +

∫ t1

t0

f0(t, φ(t), u(t))dt, (2.3.2)


where (φ, u) is an admissible pair and g is a given real valued function definedon B. Let A1 be a non-empty subset of A. Find a pair (φ∗, u∗) in A1 thatminimizes (2.3.2) in the class A1. That is, find an element (φ∗, u∗) in A1 suchthat

J(φ∗, u∗) ≤ J(φ, u) for all (φ, u) in A1.

The precise formulation of Problem 2.3.1 is rather lengthy. Therefore, thefollowing statement, which gives the essential data of the problem, is oftenused to mean that we are considering Problem 2.3.1.

Minimize (2.3.2) in the class A1 subject to the state equation (2.3.1), the endconditions B, and the control constraints Ω.

We have stated Problem 2.3.1 as a minimization problem. In some appli-cations it is required that the functional J be maximized. There is, however,no need to consider maximum problems separately because the problem ofmaximizing J is equivalent to the problem of minimizing −J . Hence we shallconfine our attention to minimum problems.

Definition 2.3.5. A pair (φ∗, u∗) that solves Problem 2.3.1 is called an opti-mal pair. The trajectory φ∗ is called an optimal trajectory and the control u∗

is called an optimal control.

The first term on the right in (2.3.2) is the function g evaluated at theend points of an admissible trajectory. Thus, it assigns a real number to everyadmissible trajectory and so is a functional G1 defined on the admissibletrajectories. The functional G1 is defined by the formula

G1(φ) = g(t0, φ(t0), t1, φ(t1)).

Other examples of functionals defined on admissible trajectories are

G2(φ) = max|φ(t)| : t0 ≤ t ≤ t1and

G3(φ) = max|φ(t)− h(t)| : t0 ≤ t ≤ t1,where h is a given continuous function defined on an interval I containing allthe intervals [t0, t1] of definition of admissible trajectories. The functionals G2

and G3 arise in problems in which in addition to minimizing (2.3.2) it is alsodesired to keep the state of the system close to some preassigned state.

The preceding discussion justifies the consideration of the following gener-alization of Problem 2.3.1.

Problem 2.3.2. Let everything be as in Problem 2.3.1, except that (2.3.2)is replaced by

J(φ, u) = G(φ) +

∫ t1

t0

f0(t, φ(t), u(t))dt, (2.3.3)

where G is a functional defined on the admissible trajectories. Find a pair(φ∗, u∗) in A1 that minimizes (2.3.3) in the class A1.


2.4 Equivalent Formulations

Certain special cases of Problem 2.3.1 are actually equivalent to Prob-lem 2.3.1 in the sense that Problem 2.3.1 can be formally transformed intothe special case in question. This information is useful in certain investigationswhere it is more convenient to study one of the special cases than to studyProblem 2.3.1. The reader is warned that in making the transformation to thespecial case some of the properties of the original problem, such as linearity,continuity, convexity, etc. may be altered. In any particular investigation onemust check that the pertinent hypotheses made for the original problem arevalid for the transformed problem.

Special cases of Problem 2.3.1 are obtained by taking f0 = 0 or g = 0. Inkeeping with the terminology for related problems in the calculus of variations,we shall call a problem in which f0 = 0 a Mayer problem and we shall calla problem in which g = 0 a Lagrange problem. Problem 2.3.1 of Section 2.3is sometimes called a Bolza problem, also as in the calculus of variations. Weshall show that the Mayer formulation and the Lagrange formulation are asgeneral as the Bolza formulation by showing that Problem 2.3.1 can be writteneither as a Mayer problem or as a Lagrange problem.

We formulate Problem 2.3.1 as a Mayer problem in a higher dimensionaleuclidean space. Let x = (x0, x) = (x0, x1, . . . , xn). Let R = R

1 × R and

let G = R × U . The functions f0, f1, . . . , fn are defined on G since they aredefined on G and they are independent of x0. Let the mapping Ω be definedon R by the equation Ω(t, x) = Ω(t, x). Let

B = (t0, x0, t1, x1) : (t0, x0, t1, x1) ∈ B, x00 = 0.

Let (φ, u) be an admissible pair for Problem 2.3.1. Let φ = (φ0, φ), where φ0

is an absolutely continuous function such that

φ0′

(t) = f0(t, φ(t), u(t)) φ0(t0) = 0

for almost every t in [t0, t1]. By virtue of (i) of Definition 2.3.3 such a functionφ0 exists and is given by

φ0(t) =

∫ t

t0

f0(s, φ(s), u(s))ds.

Then (φ, u) is an admissible pair for a problem in which R,G,Ω,B, replacedby R, G, Ω, B, respectively, and in which the system of state equations (2.3.1)is replaced by

dx0

dt= f0(t, x, u(t)) (2.4.1)

dx

dt= f(t, x, u, (t)).


If we set f = (f0, f), then Eq. (2.4.1) can be written as

dx

dt= f(t, x, u(t)).

Conversely, to every admissible pair (φ, u) for a problem involving R, G, Ω, Band (2.4.1) there corresponds the admissible pair (φ, u) for Problem 2.3.1,

where φ consists of the last n-components of φ. Let

g(t0, x0, t1, x1) = g(t0, x0, t1, x1) + x01

and letJ(φ, u) = g(t0, φ(t0), t1, φ(t1)).

Then J(φ, u) = J(φ, u), where φ = (φ0, φ). Hence the Mayer problem of

minimizing J subject to state equations (2.4.1), control constraints Ω, and

end conditions B is equivalent to Problem 2.3.1.We now show that Problem 2.3.1 can be formulated as a Lagrange problem.

Let x, R, G, Ω be as in the previous paragraph. Let

B = (t0, x0, t1, x1) : (t0, x0, t1, x1) ∈ B, x00 = g(t0, x0, t1, x1)/(t1 − t0).(2.4.2)

(Recall that for all points in B we have t1 > t0.) Let (φ, u) be an admissible pair

for Problem 2.3.1 and let φ = (φ0, φ) where φ0(t) ≡ g(t0, x0, t1, x1)/(t1 − t0).

Then (φ, u) is an admissible pair for a problem in whichR,G,Ω,B are replaced

by roofed quantities with B as in (2.4.2) and with state equations

dx0

dt= 0 (2.4.3)

dx

dt= f(t, x, u(t)).

Conversely, to every admissible pair (φ, u) for the problem with roofed quan-tities there corresponds the admissible pair (φ, u) for Problem 2.3.1, where φ

consists of the last n components of φ. If we replace f0 of Problem 2.3.1 byf0 + x0 and let

J(φ, u) =

∫ t1

t0

(f0(t, φ(t), u(t)) + φ0(t))dt (2.4.4)

then J(φ, u) = J(φ, u). Hence the Lagrange problem of minimizing (2.4.4)

subject to state equations (2.4.3), control constraints Ω, and end conditions

B is equivalent to Problem 2.3.1.In Problem 2.3.1 the initial time t0 and the terminal time t1 need not be

fixed. We now show that Problem 2.3.1 can be written as a problem with fixed


initial time and fixed terminal time. We do so by changing the time parameterto s via the equation

t = t0 + s(t1 − t0) 0 ≤ s ≤ 1

and by introducing new state variables as follows.Let w be a scalar and consider the problem with state variables (t, x, w),

where x is an n-vector and t is a scalar. Let s denote the time variable. Letthe state equations be

dt

ds= w

dw

ds= 0 (2.4.5)

dx

ds= f(t, x, u(s))w

where u is the control and f is as in Problem 2.3.1. Let

B = (s0, t0, x0, w0, s1, t1, x1, w1) : s0 = 0, s1 = 1, (2.4.6)

(t0, x0, t1, x1) ∈ B, w0 = t1 − t0.

Note that the initial and terminal times are now fixed. Let Ω(s, t, x, w) =Ω(t, x). Let φ = (τ, ξ, ω) be a solution of (2.4.5) corresponding to a control u,where the Greek-Latin correspondence between (τ, ξ, ω) and (t, x, w) indicatesthe correspondence between components of φ and the system (2.4.5). Let

J(φ, u) = g(τ(0), ξ(0), τ(1), ξ(1)) +

∫ 1

0

f0(τ(s), ξ(s), u(s))ω(s)ds. (2.4.7)

Consider the fixed end-time problem of minimizing (2.4.7) subject to the stateequations (2.4.5), the control constraints Ω, and the end conditions B.

Since t1−t0 > 0, it follows that for any solution of (2.4.5) satisfying (2.4.6)we have ω(s) = t1 − t0, a positive constant, for 0 ≤ s ≤ 1. Let (φ, u) be anadmissible pair for Problem 2.3.1. It is readily verified that if

τ(s) = t0 + s(t1 − t0) ξ(s) = φ(t0 + s(t1 − t0))

u(s) = u(t0 + s(t1 − t0)) ω(s) = t1 − t0,

then (φ, u) = (τ, ξ, ω, u) is an admissible pair for the fixed end-time problemand J(φ, u) = J(φ, u). Conversely, let (φ, u) be an admissible pair for the fixedend-time problem. If we set

φ(t) = ξ

(t− t0t1 − t0

)u(t) = u (t− t0t1 − t0) , t0 ≤ t ≤ t1,

then since τ(s) = t0 + s(t1 − t0), we have t = τ(s) for 0 ≤ s ≤ 1. It is readilyverified that (φ, u) is admissible for Problem 2.3.1 and that J(φ, u) = J(φ, u).Hence Problem 2.3.1 is equivalent to a fixed end-time problem.


The following observation will be useful in the sequel. Since for any admis-sible solution of the fixed time problem we have ω(s) = t1 − t0 > 0, we cantake the set R for the fixed end-time problem to be [0, 1] × R × R

+, whereR

+ = w : w > 0.A special case of the end conditions occurs if the initial and terminal data

are separated. In this event, a set T0 of points (t0, x0) in Rn+1 and a set T1 of

points (t1, x1) in Rn+1 are given and an admissible trajectory is required to

satisfy the conditions

(ti, φ(ti)) ∈ Ti, i = 0, 1. (2.4.8)

The set B in this case is given by (2.2.4). We shall show that the apparentlymore general requirement (iii) of Definition 2.3.3 can be reduced to the form(2.4.8) by embedding the problem in a space of higher dimension as follows.

Let y = (y1, . . . , yn) and let y0 be a scalar. Let y = (y0, y). Let the sets Rand G of Problem 2.3.1 be replaced by sets R = R × R

n+1 and G = R × U .Then the vector function f = (f0, f) is defined on G since it is independent

of y. Let Ω(t, x, y) = Ω(t, x). Let the state equations be

dx

dt= f(t, x, u(t)) (2.4.9)

dy

dt= 0.

Let

T0 = (t0, x0, y00 , y0) : (t0, x0, y00 , y0) ∈ BT1 = (t1, x1, y01 , y1) : y01 = t1, y

i1 = xi1, i = 1, . . . , n.

Replace condition (iii) of Definition 2.3.2 by the condition

(ti, φ(ti)) ∈ Ti i = 0, 1, (2.4.10)

where φ is a solution of (2.4.9). Then it is easily seen that a function u is anadmissible control for Problem 2.3.1 if and only if it is an admissible control forthe system (2.4.9) subject to control constraints Ω and end-condition (2.4.10).

Moreover, the admissible trajectories φ are of the form φ = (φ, t1, x1). Hence

if we take the cost functional to be J , where

J(φ, u) = J(φ, u),

then Problem 2.3.1 is equivalent to a problem with end conditions of the form(2.4.8).


2.5 Isoperimetric Problems and ParameterOptimization

In some control problems, in addition to the usual constraints there existsconstraints of the form

∫ t1

t0

hi(t, φ(t), u(t))dt ≤ ci i = 1, . . . , q (2.5.1)

∫ t1

t0

hi(t, φ(t), u(t))dt = ci i = q + 1, . . . , p,

where the functions hi are defined on G and the constants ci are prescribed.Constraints of the form (2.5.1) are called isoperimetric constraints. A problemwith isoperimetric constraints can be reduced to a problem without isoperi-metric constraints as follows.

Introduce additional state variables xn+1, . . . , xn+p and let x = (x, x),where x = (xn+1, · · · , xn+p). Let the state equations be

dxi

dt= f i(t, x, u(t)) i = 1, . . . , n (2.5.2)

dxn+i

dt= hi(t, x, u(t)), i = 1, . . . , p

ordx

dt= f(t, x, u, (t)),

where f = (f, h). Let the control constraints be given by the mapping Ω

defined by the equation Ω(t, x) = Ω(t, x). Let the end conditions be given by

the set B consisting of all points (t0, x0, t1, x1) such that: (i) (t0, x0, t1, x1) ∈ B;(ii) xi0 = 0, i = n + 1, . . . , n + p; (iii) xi1 ≤ ci, i = n + 1, . . . , n + q; and (iv)xi1 = ci, i = n+ q + 1, . . . , n+ p. For the system with state variable x, let Rbe replaced by R = R× R

p and let G be replaced by G = R × U .Let (φ, u) be an admissible pair for Problem 2.3.1 such that the constraints

(2.5.1) are satisfied. Let φ = (φ, φ), where

φ(t) =

∫ t

0

h(s, φ(s), u(s))ds φ(0) = 0.

Then (φ, u) is an admissible pair for the system with state variable x. Con-

versely, if (φ, u) is admissible for the x system then (φ, u), where φ consists of

the first n components of φ, is admissible for Problem 2.3.1 and satisfies theisoperimetric constraints. Hence by taking the cost functional for the problemin x-space to be J , where J(φ, u) = J(φ, u), we can write the problem withconstraints (2.5.1) as an equivalent problem in the format of Problem 2.3.1.


In Problem 2.3.1, the functions f0, f1, . . . , fn defining the cost functionaland the system of differential equations (2.3.1) are regarded as being fixed.In some applications these functions are dependent upon a parameter vectorw = (w1, . . . , wk), which is at our disposal. For example, in the rocket problemof Section 1.4 we may be able to vary the effective exhaust velocity over somerange c0 ≤ c ≤ c1 by proper design changes. The system differential equations(2.3.1) will now read

dx

dt= f(t, x, w, u(t)) w ∈W,

where W is some preassigned set in Rk. For a given choice of control u a

corresponding trajectory φ will in general now depend on the choice of pa-rameter value w. Hence, so will the value J(φ, u, w) of the cost functional. Theproblem now is to choose a parameter value w∗ in W for which there existsan admissible pair (φ∗, u∗) such that J(φ∗, u∗, w∗) ≤ J(φ, u, w) for all w in Wand corresponding admissible pairs (φ, u).

The problem just posed can be reformulated in the format of Problem 2.3.1in (n + k + 1)-dimensional space as follows. Introduce new state variablesw = (w1, . . . , wk) and consider the system

dxi

dt= f i(t, x, w, u(t)) i = 1, . . . , n (2.5.3)

dwi

dt= 0 i = 1, . . . , k.

Let x = (x,w), let R = R× Rk, let G = R × U , and let Ω(t, x, w) = Ω(t, x).

Let the end conditions be given by

B = (t0, x0, w0, t1, x1, w1) : (t0, x0, t1, x1) ∈ B, w0 ∈W.

Let J(φ, u) = J(φ,w, u). It is readily verified that the problem of minimizing J

subject to (2.5.3), the control constraints Ω, and end conditions B is equivalentto the problem involving the optimization of parameters.

2.6 Relationship with the Calculus of Variations

The brachistochrone problem in Section 1.6 is an example of the simpleproblem in the calculus of variations, which can be stated as follows. Let tbe a scalar, let x be a vector in R

n, and let x′ be a vector in Rn. Let G be

a region in (t, x, x′)-space. Let f0 be a real valued function defined on G. LetB be a given set of points (t0, x0, t1, x1) in R

2n+2 and let g be a real valuedfunction defined in B. An admissible trajectory is defined to be an absolutelycontinuous function φ defined on an interval [t0, t1] such that:


(i) (t, φ(t), φ′(t)) ∈ G for t in [t0, t1]

(ii)(t0, φ(t0), t1, φ(t1)) ∈ B (2.6.1)

(iii) t→ f0(t, φ(t), φ′(t)) is integrable on [t0, t1].

The problem is to find an admissible trajectory that minimizes

g(t0, φ(t0), t1, φ(t1)) +

∫ t1

t0

f0(t, φ(t), φ′(t))dt.

As with the brachistochrone problem, the general simple problem in thecalculus of variations can be written as a control problem by relabeling x′ asz; that is, we set u = φ′. (Recall that z denotes the control variable and udenotes the control function.) The simple problem in the calculus of variationsbecomes the following control problem. Minimize

g(t0, φ(t0), t1, φ(t1)) +

∫ t1

t0

f0(t, φ(t), u(t))dt

subject to the state equations

dxi

dt= ui(t) i = 1, . . . , n,

end conditions (ii) of (2.6.1), and control constraints Ω, where

Ω(t, x) = z : (t, x, z) ∈ G.

Recall that the region G is an open set. Thus, Ω is also an open set in caseof any simple problem of calculus of variations. The problem of Bolza in thecalculus of variations differs from the simple problem in that in addition to(2.6.1) an admissible trajectory is required to satisfy a system of differentialequations

F i(t, φ(t), φ′(t)) = 0 i = 1, . . . , µ. (2.6.2)

The functions F 1, . . . , Fµ are defined and continuous on G and µ < n.In the development of the necessary conditions in the problem of Bolza,

the assumption is usually made that the functions f0 and F = (F 1, . . . , Fµ)are of class C(1) on the region G of (t, x, x′)-space and the matrix of partialderivatives Fx′ = (∂F i/∂x

′j), i =, 1 . . . , µ, j = 1, . . . , n, has rank µ everywhereon G. Hence in the neighborhood of any point (t2, x2, x

′2) at which

F i(t2, x2, x′2) = 0 i = 1, . . . , µ, (2.6.3)

we can solve for µ components of x′ in terms of t, x, and the remaining n− µcomponents of x′. Moreover, these µ components of x′ will be C(1) functionsof their arguments. Let us now suppose that we can solve (2.6.3) globally in


this fashion. Since we can relabel components we can assume that we solvethe first µ components in terms of the remaining n− µ, and get

x′i = Gi(t, x, x′) i = 1, . . . , µ,

where x′ = (x′µ+1, . . . , x

′n). Thus, Eq. (2.6.2) is equivalent to

φ′i(t) = Gi(t, φ(t), φ′(t)) i = 1, . . . , µ,

where φ′ = (dφµ+1/dt, . . . , dφn/dt). Let m = n−µ and let z = (z1, . . . , zm) =(x

′µ+1, . . . , x′n) = x′. It then follows that under the assumptions made here

that the Bolza problem is equivalent to the following control problem withcontrol variable z = (z1, . . . , zm).

The functional to be minimized is defined by the equation

J(φ, u) = g(t0, φ(t0), t1, φ(t1)) +

∫ t1

t0

f0(t, φ(t), u(t))dt,

where

f0(t, x, z) = f0(t, x,G1(t, x, z), . . . , Gµ(t, x, z), z1, . . . , zm).

The system equations are

dxi

dt= Gi(t, x, u(t)) i = 1, . . . , µ

dxµ+i

dt= ui(t) i = 1, . . . ,m.

The end conditions are defined by the set B of the Bolza problem and thecontrol constraints Ω are defined as follows:

Ω(t, x) = z : (t, x,G1(t, x, z), . . . , Gµ(t, x, z), z1, . . . , zm) ∈ G.

It is, of course, also required of an admissible (φ, u) that the mapping t →f0(t, φ(t), u(t)) be integrable.

Conversely, under certain conditions the control problem can be writtenas a problem of Bolza in the calculus of variations. Let us first suppose thatΩ(t, x) = R

m for all (t, x). That is, there are no constraints on the control.We introduce new coordinates y1, . . . , ym and let

dyi

dt= ui(t) i = 1, . . . ,m.

Then Eq. (2.3.1) can be written as

dxi

dt− f i

(t, x,

dy

dt

)= 0 i = 1, . . . , n.


If we setF i(t, x, y, x′, y′) = x

′i − f i(t, x, y′) i = 1, . . . , n, (2.6.4)

then the control problem can be written as the following problem of Bolzain (n +m+ 1)-dimensional (t, x, y)-space. The class of admissible arcs is the

set of absolutely continuous functions φ = (φ, η) = (φ1, . . . , φn, η1, . . . , ηm)defined on intervals [t0, t1] such that:

(i) (t, φ(t), η′(t)) is in the domain of definition of the function f = (f0, f);

(ii) (t0, φ(t0), t1, φ(t1)) is in B and η(t0) = 0;

(iii) the function t→ f0(t, φ(t), η′(t)) is integrable; and

(iv) F (t, φ(t), φ′(t)) = φ′(t)− f(t, φ(t), η′(t)) = 0 (2.6.5)

a.e. on [t0, t1]. The problem is to minimize the functional

g(t0, φ(t0), t1, φ(t1)) +

∫ t1

t0

f0(t, φ(t), η′(t))dt

in the class of admissible arcs.

It is clear from (2.6.4) that the ith row of the n× (n+m) matrix (Fx′Fy′)for the Bolza problem obtained from the control problem has the form

(0 . . . 0 1 0 . . . 0 ∂f i/∂y′1 . . . ∂f i/∂y

′m),

where the entry 1 occurs in the i-th column and all other entries in the first ncolumns are zero. Thus, the n× (n+m) matrix (Fx′Fy′) has rank n as usuallyrequired in the theory of the necessary conditions for the Bolza problem.

Let us now suppose that control constraints Ω are present and that thesets Ω(t, x) are defined by a system of inequalities. We suppose that there arer functions R1, . . . , Rr of class C(1) on G. The set Ω(t, x) is defined as follows:

Ω(t, x) = z : Ri(t, x, z) ≥ 0, i = 1, . . . , r.

We impose a further restriction, which we call the constraint qualification.

(i) If m, the number of components of z, is less than or equal to r, thenumber of constraints, then at any point (t, x, z) of G at most m of thefunctions R1, . . . , Rr can vanish at that point.

(ii) At the point (t, x, z) let i1, . . . , iρ denote the set of indices such thatRi(t, x, z) = 0. Let Rz,ρ(t, x, z) denote the matrix formed by taking therows i1, . . . , iρ of the matrix

Rz(t, x, z) =(∂Ri(t, x, z)∂zj

)i = 1, . . . , r j = 1, . . . ,m.

(2.6.6)Then Rz,ρ(t, x, z) has rank ρ.


To formulate the control problem as a Bolza problem we proceed as be-fore and let y′ = z. The constraints take the form Ri(t, φ(t), η′(t)) ≥ 0. Thisrestriction is not present in the classical Bolza formulation. We can, how-ever, write the variational problem with constraints as a Bolza problem byintroducing a new variable w = (w1, . . . , wr) and r additional state equations

Ri(t, x, y′)− (w′i)2 = 0 i = 1, . . . , r. (2.6.7)

The Bolza problem now is to minimize

g(t0, φ(t0), t1, φ(t1)) +

∫ t1

t0

f0(t, φ(t), η′(t))dt

subject to the differential equations (2.6.5) and (2.6.7) and the end conditions

(t0, φ(t0), t1, φ(t1)) ∈ B η(t0) = 0 ω(t0) = 0,

where the function ω is the component of the admissible arc corresponding tothe variable w.

Let

F i(t, x, y, w, x′, y′, w′) = x′i − f i(t, x, y′) i = 1, . . . , n, (2.6.8)

Fn+i(t, x, y, w, x′, y′, w′) = Ri(t, x, y′)− (w′i)2 i = 1, . . . , r.

We shall show that the (n+ r) × (n+m+ r) matrix

M =

(∂F q

∂x′j

∂F q

∂y′k

∂F q

∂w′s

)= (Fx′ Fy′ Fw′)

has rank n+ r as usually required in the theory of the Bolza problem. It is astraightforward calculation using (2.6.4) and (2.6.8) to see that

M =

(I −fy′ 0102 Ry′ −W

)

where I is the n×n identity matrix, fy′ is the n×m matrix with typical entry

∂f i/∂y′k, 01 is an n × r zero matrix, 02 is an r × n zero matrix, Ry′ is the

r×m matrix with typical entry ∂Ri/∂y′k, and W is an r× r diagonal matrix

with diagonal entries 2w′i.

From the form of the matrix M it is clear that to prove that it has rankn+r it suffices to show that the r×(m+r) matrix (Ry′−W ) has rank r. To dothis let us suppose that the indexing is such that the indices i1, . . . , iρ for whichRij (t, x, y′) = 0 are the indices 1, . . . , ρ. Let (Ry′)ρ denote the submatrix ofRy′ consisting of the first ρ rows of Ry′ and let (Ry′)r−ρ denote the submatrixof Ry′ consisting of the remaining rows. Thus if i > ρ, then Ri(t, x, y′) > 0; ifi ≤ ρ, then Ri(t, x, y′) = 0. Hence since

(w′i)2 = Ri(t, x, y′) i = 1, . . . , r,


it follows that2w

′i = 0 if i ≤ ρ 2w′i 6= 0 if i > ρ.

Hence

(Ry′ −W ) =

((Ry′)ρ 03 04

(Ry′)r−ρ 05 D

),

whereD is a diagonal matrix of dimension (r−ρ)×(r−ρ) with non-zero entries2w

′i, i > ρ, and where 03, 04, and 05 are zero matrices. By the constraintqualification (2.6.6) the matrix (Ry′)ρ has rank ρ. Since D has rank r − ρ itfollows that (Ry′ −W ) has rank r, as required.

2.7 Hereditary Problems

Hereditary problems, which are also called delay or lag problems, take thehistory of the system into account in their evolutions and in the measure oftheir performances. We give a general formulation that takes into account thehistory of the control as well as the history of the state and then discuss acommonly occuring special case.

The formulation of a hereditary problem requires the introduction of ad-ditional notation. Let Itα denote the interval [α, t], where α < t ≤ ∞, letX denote an open interval in R

n, and let U denote an open interval in Rm.

Let r > 0 and let (t, s, x, z) denote a generic point of I∞0 × I∞−r ×X × U . LetC(Iτ−r ,X ), 0 < τ ≤ ∞ denote the space of continuous functions from Iτ−r to Xwith supremum norm, and let AC(Iτ−r ,X ) denote the subspace of absolutelycontinuous functions. Let M denote the set of measurable functions on I∞0with range in U . We shall denote functions in M by the letter u. Let

(i) g0, . . . , gn from I∞0 × I∞−r ×X×U into Rn be continuous in all variables.

Further, ∂xgi(t, s, x, u), 0 ≤ i ≤ n, are continuous in all arguments.

Here, (t, s, x, u) is a generic point of I∞0 × I∞−r ×X×U. We assume thatu(t) ∈ Ω a.e., where Ω is a fixed compact subset of U.

(ii) h0, h1, . . . , hn be functions from I∞0 × C(I∞−r,X ) × U to R such that ifφ1 and φ2 are in C(I∞−r ,X ) and φ1 = φ2 on It−r, 0 < t <∞, then

hi(t, φ1, u(t)) = hi(t, φ2, u(t)) i = 0, 1, . . . , n.

The functions hi : C(I∞−r,X) × U → R are measurable in t ∈ I andcontinuous in u ∈ U. The functions hi(t, ·, u) are Frechet differentiable asa map from C(I∞−r ,X) into R. Further, the derivatives are continuous inthe second and third arguments. Denoting by dhi(t, φ(·), u) the Frechetderivative at φ, we assume that there exists Λ ∈ L1(R) such that

|hi(t, φ(·), u)| ≤ Λ(t), ∀ φ ∈ C(I∞−r,X),


|dhi(t, φ(·), u)(ψ)| ≤ Λ(t)‖ψ‖∞, ∀ ψ ∈ C(I∞−r).

(iii) w0, w1, . . . , wn be measurable functions on I∞0 × I∞−r to R such that foreach t in I∞0 , wi(t, ·) is of bounded variation on I∞−r, is continuous onthe right, and vanishes for s ≥ t. Let f0, f1, . . . , fn be functions definedon I∞0 × C(I∞−r,X )×M by the formula

f i(t, φ, u) = hi(t, φ, u(t)) +

∫ t

−r

gi(t, s, φ(s), u(s))dswi(t, s)

i = 0, 1, . . . , n. (2.7.1)

Recall that φ and u denote functions and φ(t), φ(s), u(t), u(s) denotethe values of functions at the indicated arguments.

A function u in M is said to be a control on the interval [0, t1] if thereexists a function φ in C(It1−r,X ) that is in AC(It10 ,X ) such that

φ′(t) = f(t, φ, u) a.e. on [0, t1] (2.7.2)

φ(t) = y(t) − r ≤ t ≤ 0 y ∈ C(I0−r,X ).

Here, f = (f1, . . . , fn). The function y is specified and is called the initialfunction. The function φ is called a trajectory corresponding to u.

Let t1 > 0 be fixed. Let Ω be a mapping from It10 ×X to subsets of U andlet B be a specified set in R

2n+1. A control u and a corresponding trajectoryφ are said to be admissible if

(i) t→ f0(t, φ, u) is in L1[0, t1]

(ii) u(t) ∈ Ω(t, φ(t)) a.e. on [0, t1]

(iii) (φ(0), t1, φ(t1)) ∈ B.The control problem is to choose an admissible pair that minimizes

g(φ(0), t1, φ(t1)) +

∫ t1

0

f0(t, φ, u)dt, (2.7.3)

where g is a given function defined on B.In a frequently encountered form of the hereditary problem the dependence

on the history of the control is absent and the dependence on the state hasa special form. Thus, the integrals in (2.7.1) are absent. For a given φ inC(I∞−r ,X ) we define for each t in I∞0 a function φt in C(I

0−r ,X ) by the formula

φt(θ) = φ(t+ θ), −r ≤ θ ≤ 0.

We now take the function h = (h0, h) = (h0, h1, . . . , hn) to be a mapping fromI∞0 × C(I0−r,X ) × U to R and take the state equations to be

φ′(t) = h(t, φt, u(t)).


The expression (2.7.3) becomes

g(φ(0), t1, φ(t1)) +

∫ t1

0

h0(t, φt, u(t))dt.

As a further specialization we take

h(t, φt, u(t)) = k(t, φ(t − r), u(t)),

where k is a function from I∞0 ×X×U to Rn+1. In this case, the state equationsare said to be retarded or delay equations.

Chapter 3

Relaxed Controls

3.1 Introduction

In this chapter we define relaxed controls and the relaxed control problemand determine some of the properties of relaxed controls. For problems withwell-behaved compact constraint sets, relaxed controls have a very useful com-pactness property. Also, at a given point in the subset R of (t, x) space, the setof directions that the state of a relaxed system may take is convex. This prop-erty is needed in existence theorems. We also shall prove an implicit functiontheorem for measurable functions that permits a definition of relaxed controlsalternative to the one given in the next section. This theorem will also be usedin our existence theorems. To motivate the definition of relaxed controls, wepresent two examples.

Example 3.1.1. Let the state equation be

dx1

dt= (x2)2 − u2

dx2

dt= u

and let Ω(t, x) = z : |z| ≤ 1. Note that the constraint sets are constant,compact, and convex. Let the initial set T0 be given by (t0, x

10, x

20) = (0, 1, 0).

Let the terminal set T1 be given by

(x11)2 + (x21)

2 = a2 0 < a < 1,

and t1 ≥ δ, where δ is a fixed number satisfying 0 < δ < 1 − a. The problemis to minimize the time t1 at which the terminal set T1 is attained.

From the state equations it is clear that t1 > 1 − a for all admissiblecontrols. To attain the terminal time of 1−a we would need to have φ2(t) ≡ 0and (u(t))2 ≡ 1 for a trajectory. This is clearly impossible in view of the secondstate equation. This equation suggests, however, that we can approximate thevalue 1− a by taking u(t) to be alternately +1 and −1 on small intervals. Tothis end, for each r = 1, 2, 3, . . . we define a control ur on the interval [0, 2r]as follows:

35


ur(t) =

1 (i− 1)/2r ≤ t < i/2r i = 1, 3, . . . , 4r2 − 1

−1 (i− 1)/2r ≤ t < i/2r i = 2, 4, . . . , 4r2.

For t > 2r, let ur(t) ≡ 0. Let ϕr = (ϕ1r , ϕ

2r) be the trajectory corresponding

to ur. Clearly,0 ≤ ϕ2

r(t) ≤ 1/2r (3.1.1)

on [0,∞), and so ϕ2r(t) → 0 uniformly on [0,∞) as r → ∞. It then follows

from the first state equation that for each r, dφ1r/dt > −1 except at a finiteset of points and that

limr→∞

dφ1r/dt = −1

uniformly on [0,∞]. Hence for r sufficiently large there exists a point tr > 1−asuch that (tr, φ

1r(tr), φ

2r(tr)) ∈ T1. It therefore follows from φ2r(tr) → 0 that

φ1r(tr) → a. Now

ϕ1r(tr)− 1 =

∫ tr

0

(ϕ1r)

′dt ≤∫ tr

0

(1

4r2− 1

)dt = tr

(1

4r2− 1

),

where the inequality follows from the first state equation and (3.1.1). Hence

tr

(1− 1

4r2r

)≤ 1− ϕ1

r(tr),

and solim supr→∞

tr ≤ 1− a.

Since tr > 1 − a, we get that limr→∞ tr = 1 − a. Recalling that the terminaltime for any admissible trajectory exceeds 1 − a, we have that the infimumof all terminal times is 1 − a. Thus, the problem has no solution, since as wealready noted, the terminal time 1 − a cannot be achieved by an admissibletrajectory.

The construction of the admissible sequence (ϕr , ur) suggests that wemight attain the terminal time 1 − a if we modified the problem to al-low controls that are an average in some sense of controls with values inΩ ≡ Ω(t, x) = z : |z| ≤ 1. To this end we define a problem with stateequations

dx1

dt= (x2)2 −

∫

Ω

z2dµt (3.1.2)

dx2

dt=

∫

Ω

zdµt,

where for each t, dµt is a regular probability measure on Ω. A control u ofthe original problem is also a control for the relaxed problem. To see this, foreach t let dµt be the measure concentrated at the point u(t). We take the

Relaxed Controls 37

initial and terminal sets to be as before and require that the terminal time beminimized. The problem just formulated is the relaxed version of the originalproblem. For this problem we also have t1 ≥ 1−a, with equality only possibleif for each t there exists a dµt such that

∫

Ω

zdµt = 0 and

∫

Ω

z2dµt = 1. (3.1.3)

If we take dµt to be the measure on Ω that assigns the measure 1/2 to eachof the points z = 1 and z = −1, then (3.1.3) holds. Thus, the relaxed problemhas a solution.

It is readily verified that in the relaxed problem at each point (t, x) inR the set of possible directions is convex. In the original problem the set ofadmissible directions (v1, v2) is the segment of the parabola

v1 = (x2)2 − (v2)2 − 1 ≤ v2 ≤ 1,

which is not convex. We shall see later that the set of admissible directionsfor the relaxed problem is the convex hull of this set.

Example 3.1.2. Let the state equations be

dx1

dt= u1(t)

dx2

dt= u2(t)

dx3

dt= 1

and let the constraint set be Ω = z = (z1, z2) : (z1)2 + (z2)2 = 1. Let theinitial set T0 be given by (t0, x

10, x

20, x

30) = (0, 0, 0, 0) and let the terminal set

T1 be given by (t1, x11, x

21, x

31) = (1, 0, 0, 1). Let

J(ϕ, u) =

∫ 1

0

[(φ1)2 + (φ2)2]dt,

where φ = (φ1, φ2) is an admissible trajectory corresponding to an admissiblecontrol u. For each k = 1, 2, 3, . . ., let

uk(t) = (u1k(t), u2k(t)) = (sin 2πkt, cos 2πkt),

and let φk = (φ1k, φ2k, φ

3k) be defined by

φ1k(t) = (1 − cos 2πkt)/2πk

ϕ2k(t) = sin 2πkt/2πk

ϕ3k(t) = t.

Then each (ϕk, uk) is admissible and 0 ≤ J(ϕk, uk) ≤ (πk)−2. Since J(ϕ, u) ≥


0 for all admissible (ϕ, u), it follows that infJ(ϕ, u) : (ϕ, u) admissible = 0.Therefore, if there exists an optimal pair (ϕ∗, u∗) we must have J(ϕ∗, u∗) = 0.But then we must have u∗(t) ≡ 0. This control, however, does not satisfy theconstraint. Hence an optimal control does not exist.

As in Example 3.1.1, we consider the relaxed problem with state equations

dx1

dt=

∫

Ω

z1dµt (3.1.4)

dx2

dt=

∫

Ω

z2dµt

dx3

dt= 1.

The sets T0, T1, and Ω are as before, and the functional to be minimized is

∫ 1

0

[(ψ1)2 + (ψ2)2] dt,

where ψ is a solution of (3.1.4). Let 0 ≤ θ < 2π be arbitrary. For each tin [0, 1] let dµt be the measure such that each of the points (cos θ, sin θ) and(− cos θ,− sin θ) have measure 1/2. Then the system (3.1.4) becomes dx1/dt =0, dx2/dt = 0, dx3/dt = 1, and the admissible relaxed trajectory ψ1(t) ≡ 0,ψ2(t) ≡ 0, ψ3(t) = t minimizes.

3.2 The Relaxed Problem; Compact Constraints

In this section we formulate the relaxed problem corresponding to a slightlyspecialized version of Problem 2.3.1, which we now restate for the reader’sconvenience.

Minimize

g(t0, x0, t1, x1) +

∫ t1

t0

f(t, x, u(t))dt

subject to

dx

dt= f(t, x, u(t)) (3.2.1)

(t0, x0, t1, x1) ∈ B u(t) ∈ Ω(t).

This problem differs from Problem 2.3.1 in that the constraint sets dependonly on t and not on (t, x).

The data of the problem are assumed to satisfy the following.

Relaxed Controls 39

Assumption 3.2.1. (i) The function f = (f0, f1, . . . , fn), where the f i

are real valued, is defined on a set R × U, where R = I × X , I is acompact interval in R

1, X is an interval in Rn, and U is an interval in

Rm.

(ii) The function f is continuous on X × U for a.e. t in I and measurableon I for each (x, z) in X × U.

(iii) For each t ∈ I, the set Ω(t) is compact and contained in U.

(iv) There exists at least one measurable function u defined on I such thatthe state equation with this u has a solution ϕ defined on I.

We now recall some definitions from measure theory.

(i) A probability measure µ on a compact set K is a positive measure onthe Borel sets of K such that µ(K) = 1.

(ii) A positive measure µ defined on the σ-algebra of all Borel sets in a locallycompact Hausdorff space X is said to be regular if for every Borel set E

µ(E) = supµ(K) : K ⊆ E, K compact= infµ(O) : E ⊆ O, O open.

(iii) If the measure µ only satisfies the first equality, then µ is said to beinner regular. Such measures are also called Radon measures.

We shall consider vector valued measures µ of the form µ = (µ1, . . . , µn),where each of the µi is a real valued measure. We shall say that µ is non-negative, or a probability measure, or is regular, if each of the µi has thatproperty.

Definition 3.2.2. A relaxed control on I is a function

µ : t→ µt a.e.

where µt is a regular probability measure on Ω(t) such that for every functiong defined on I × U with range in R

n that is continuous on U for a.e. t in Iand measurable on I for each z in U, the function h defined by

h(t) =

∫

Ω(t)

g(t, z)dµt

is Lebesgue measurable.

Remark 3.2.3. Since Ω(t) is a compact set in a euclidean space, if µt isa probability measure on Ω(t), then µt is regular [82, 2.18]. We keep theredundant word “regular” in Definition 3.2.2 to make the definition of relaxedcontrol applicable in situations more general than the one considered here.


Remark 3.2.4. The set of relaxed controls properly contains the set of ordi-nary controls. To see this, let u be a control defined on [t0, t1] with u(t) ∈ Ω(t)a.e. Let δu(t) be the Dirac measure on Ω(t) that is equal to one at the pointu(t) and equal to zero on any set that does not contain u(t). Then δu(t) is aprobability measure and for any g as in Definition 3.2.2 with I = [t0, t1], thefunction h defined by

h(t) = g(t, u(t)) =

∫

Ω(t)

g(t, z)dδu(t)

is measurable. Thus, the mapping t → δu(t) is a relaxed control. We cantherefore consider an ordinary control to be a special type of relaxed control.

We now exhibit relaxed controls that are not ordinary controls. Letp1, . . . , pk be nonnegative measurable functions defined on I whose sum is oneand let u1, . . . , uk be measurable functions defined on I such that ui(t) ∈ Ω(t).For any Borel set E in Ω(t) let

µt(E) =

k∑

i=1

pi(t)δui(t)(E). (3.2.2)

Then µt is a probability measure and µ : t→ µt is a relaxed control since

h(t) =

∫

Ω(t)

g(t, z)dµt =

k∑

i=1

pi(t)g(t, ui(t))

is Lebesgue measurable.Let µ be a relaxed control. Then for each x in X the function F defined

by

F (t, x) =

∫

Ω(t)

f(t, x, z)dµt

is a measurable function of t and for fixed t is a continuous function of x.Thus, F defines a direction field on I ×X and we can consider the differentialequation

x′ = F (t, x).

We shall call solutions of this differential equation relaxed trajectories.

Definition 3.2.5.

ψ′(t) =

∫

Ω(t)

f(t, ψ(t), z)dµt

for a.e. t. We call (ψ, µ) a relaxed control-trajectory pair.

Since ordinary controls are also relaxed controls, ordinary trajectories arealso relaxed trajectories.

Relaxed Controls 41

Definition 3.2.6. A relaxed trajectory is said to be admissible if (i) (t0, ψ(t0),t1, ψ(t1)) ∈ B and (ii) the function

t→∫

Ω(t)

f(t, ψ(t), z)dµt

is integrable. The pair (ψ, µ) is said to be a relaxed admissible pair.

We now state the relaxed problem corresponding to Problem (3.2.1).

Problem 3.2.1. Find a relaxed admissible pair (ψ∗, µ∗) that minimizes

J(ψ, µ) = g(t0, ψ(t0), t1, ψ(t1)) +

∫ t1

t0

∫

Ω(t)

f(t, ψ(t), z)dµtdt

over some subset A1 of the set A of relaxed admissible pairs. That is, find arelaxed admissible pair (ψ∗, µ∗) in A1 such that J(ψ∗, µ∗) ≤ J(ψ, µ) for alladmissible pairs (ψ, µ) in A1.

Definition 3.2.7. The pair (ψ∗, µ∗) is called a relaxed optimal pair. Thefunction ψ∗ is a relaxed optimal trajectory and the control µ∗ is a relaxedoptimal control.

The next lemma relates the direction sets of the ordinary problem andthe direction sets of the relaxed problem. Readers unfamiliar with the factsabout convex sets that we use are referred to [32]. We will, however, state atheorem due to Caratheodory concerning the representation of convey hulls.For a proof see [32]. We denote the convex hull of a set A by co(A).

Theorem 3.2.8 (Theorem (Caratheodory)). Let A be a set in Rn and let x

be a point in co(A). Then there exist (n + 1) points x1, . . . , xn+1 in A and(n+ 1) nonnegative real numbers p1, . . . , pn+1 such that

∑pi = 1 and

x =

n+1∑

i=1

pixi.

Lemma 3.2.9. For each (t, x) in R let

V (t, x) = y : y = f(t, x, z), z ∈ Ω(t)

Vr(t, x) = y : y =

∫

Ω(t)

f(t, x, z)dµt, µ a relaxed control.

Then Vr(t, x) = coV (t, x), where co denotes convex hull.

Proof. Since a convex combination of probability measures is again a prob-ability measure, the sets Vr(t, x) are convex. The sets Vr(t, x) contain thesets V (t, x) because the Dirac measure δz concentrated at z is a probabilitymeasure. Hence

co V (t, x) ⊆ Vr(t, x). (3.2.3)


We now show that equality holds in (3.2.3). Since Ω(t) is compact and f iscontinuous in z, each V (t, x) is compact. Therefore, so is co V (t, x). If equalitydid not occur in (3.2.3) there would exist a point

w =

∫

Ω(t)

f(t, x, z)dµt

in Vr(t, x) that is not in the compact, convex set co V (t, x). Hence there wouldexist a hyperplane 〈a, x〉 = α in R

n such that

〈a, w〉 > α and 〈a, y〉 < α for all y ∈ co V (t, x).

In particular,〈a, f(t, x, z)〉 < α for all z ∈ Ω(t).

But then, since µt is a probability measure,

α < 〈a, w〉 =∫

Ω(t)

〈a, f(t, x, z)〉dµt <

∫

Ω(t)

αdµt = α.

This contradiction shows that equality holds in (3.2.3).We now develop an equivalent formulation of the relaxed problem. Let ψ

be a relaxed trajectory on an interval I = [t0, t1]. Then ψ′(t) ∈ Vr(t, ψ(t)) for

a.e t in I, and by Lemma 3.2.9, ψ′(t) ∈ co V (t, ψ(t)). Hence by Caratheodory’stheorem there exist (n+ 1) points z1(t), . . . , zn+1(t) in Ω(t) and (n+ 1) non-negative numbers π1(t), . . . , πn+1(t) whose sum is one such that

ψ′(t) =

n+1∑

i=1

πi(t)f(t, ψ(t), zi(t)) (3.2.4)

for a.e. t in I. The functions π = (π1, . . . , πn+1) and z = (z1, . . . , zn+1) aredefined pointwise. We assert that there exist nonnegative measurable functionsp1, . . . , pn+1 with sum one and measurable functions u1, . . . , un+1 with ui(t) ∈Ω(t) for a.e. t in I, such that

ψ′(t) =

n+1∑

i=1

pi(t)f(t, ψ(t), ui(t)) (3.2.5)

for a.e. t in I.This assertion will follow from Lemma 3.2.10, whose proof will be given in

Section 3.4.

Lemma 3.2.10. Let I denote a real compact interval, U , an interval in Rk

and let h be a map from I ×U into Rn that is continuous on U for a.e. t in I

and is measurable on I for each z in U . Let W be a measurable function fromI to R

n and let V be a function defined on I with range in U such that

W (t) = h(t, V (t)) a.e.

Relaxed Controls 43

Then there exists a measurable function V defined on I with range in U suchthat

W (t) = h(t, V (t)) a.e. on I.

Moreover, the values V (t) satisfy the same constraints as the values V (t).

Let Z = (π, ζ1, . . . , ζn+1), where π ∈ Rn+1 and each ζi ∈ R

m, let W (t) =ψ′(t), and let

h(t, Z) =

n+1∑

i=1

πif(t, ψ(t), ζi).

The assertion that there exist measurable functions p1, . . . , pn+1 andu1, . . . , un+1 as in (3.2.5) follows from Lemma 3.2.10 and Eq. (3.2.4).

In summary, we have proved the following theorem.

Theorem 3.2.11. Every relaxed trajectory ψ is a solution of a differentialequation

x′ =

n+1∑

i=1

pi(t)f(t, x, ui(t)), (3.2.6)

where the real valued measurable functions p1, . . . , pn+1 are nonnegative andhave sum equal to one a.e. and where the functions u1, . . . , un+1 are measur-able and satisfy ui(t) ∈ Ω(t), a.e., i = 1, . . . , n+ 1.

Remark 3.2.12. For each t let µt be the measure on Ω(t) defined by (3.2.2).Then we may write (3.2.6) as

x′ =

∫

Ω(t)

f(t, x, z)dµt.

Since µt is a probability measure we see that every solution of (3.2.6) is arelaxed trajectory. Thus, in the case of compact constraints Ω(t), we couldhave defined a relaxed trajectory more simply, perhaps, to be a solution of(3.2.6). Thus, in Definition 3.2.2 we could have restricted ourselves to proba-bility measures that are convex combinations of Dirac measures. We did notdo so because the larger class of measures in Definition 3.2.2 has a usefulcompactness property that we shall develop in the next section.

3.3 Weak Compactness of Relaxed Controls

We review some concepts and theorems from functional analysis that wewill need. For full discussion and proofs see [82] and [89]. Let I denote acompact interval in R

1 and let Z denote a compact set in Rk. Let C(I × Z)

denote the space of Rn valued continuous functions on I × Z with sup norm,


and let C∗(I × Z) denote the space of continuous linear transformations Lfrom C(I ×Z) to R

n. The space C∗(I ×Z) is a Banach space, with the normof an element L given by

‖L‖ = sup|L(g)| : ‖g‖ ≤ 1,

where | | denotes the euclidean norm in Rn and ‖g‖ = max|g(t, z)| : (t, z)

in I × Z. Note that L = (L1, . . . , Ln), where each Li is a continuous linearfunctional on I × Z.

A sequence Ln of continuous linear transformations in C∗(I ×Z) is saidto converge weak-star (written weak-∗) to an element L in C∗(I × Z) if forevery g in C(I × Z)

limn→∞

Ln(g) = L(g).

A set Λ in C∗(I × Z) is said to be weak-∗ sequentially compact if foreach sequence Ln of elements in Λ, there exists an element L in Λ and asubsequence Lnk

such that Lnkconverges weak-∗ to L.

An important set of functions that is weak-∗ sequentially compact is thefollowing.

A closed ball in C∗(I × Z) is weak-∗ sequentially compact.

The Riesz Representation Theorem and its extensions [82] state:

Every continuous linear functional L in C∗(I ×Z) is represented uniquelyby a regular Borel measure ν on I × Z in the sense that

L(g) =

∫

I×Z

g(t, z)dν

and‖L‖ = ‖ν‖var ≡ |ν|(I × Z),

where |ν| denotes the total variation measure corresponding to ν. Moreover, ifL is positive, then so is ν, and ‖L‖ = ν(I × Z).

We now return to relaxed controls. Let Ω(t) = Z for each t in I and letµ be a relaxed control on I. From the definition of relaxed control, we havethat for g in C(I × Z), the function h defined by

h(t) =

∫

Z

g(t, z)dµt

is measurable. Also,

|h(t)| =∣∣∣∣∫

Z

g(t, z)dµt

∣∣∣∣ ≤ ‖g‖, a.e.

Therefore, the formula

Lµ(g) =

∫

I

∫

Z

g(t, z)dµtdt (3.3.1)

Relaxed Controls 45

defines a continuous linear transformation with

|Lµ(g)| ≤ ‖g‖|I|,

where |I| denotes the length of I. If we take g to be the function identicallyone on I × Z we get that

‖Lµ‖ = |I|. (3.3.2)

Henceforth, to simplify notation we take I to be a generic compact intervalwith the origin as left-hand end point.

We shall also use a theorem from real analysis, which was proved byUrysohn in a more general context than the one we need. See [82].

Lemma 3.3.1 (Urysohn’s Lemma). Let K be a compact set in Rk, let V be

an open set in Rk, and let K ⊂ V . Then there exists a continuous nonnegative

function f with support contained in V , with 0 ≤ f(x) ≤ 1 for all x and withf(x) = 1 for x in K.

Definition 3.3.2. A sequence µn of relaxed controls on I is said to convergeweakly to a relaxed control µ if for each τ in I and each g in C(I × Z)

limn→∞

∫ τ

0

∫

Z

g(t, z)dµntdt =

∫ τ

0

∫

Z

g(t, z)dµtdt. (3.3.3)

Remark 3.3.3. Let τ be a point in I, let Iτ = [0, τ ], and take I in (3.3.1) to beIτ . Then each µn defines a continuous linear transformation Lτ

n in C∗(Iτ ×Z),as does µ. Thus, Definition 3.3.2 states that for each τ in I the sequenceLτn converges weak-∗ to Lτ

µ, the continuous linear transformation defined by(3.3.1). Thus, the weak convergence of µn to µ is really a weak-∗ convergencefor each point τ in I. We are abusing the terminology and calling the con-vergence weak convergence. This weak convergence is distinct from the weakconvergence of a sequence of elements in a Banach space. The following ex-ample illustrates Definition 3.3.2 and Remark 3.3.3.

Example 3.3.4. Let the dimension of the state space be one, let I = [0, 1]and let Z = [−1, 1]. For each positive integer n subdivide I into 2n contiguoussubintervals, each of length 1/2n. Let un(t) be the function such that un(t) = 1for t in [0, 1/2n], un(t) = −1 for t in [1/2n, 2/2n], and un(t) then alternatesin value between +1 and −1 in successive subintervals. Let µnt = δun(t). Weassert that δun(t) converges weakly to (1/2)δ1 + (1/2)δ−1, where δ1 is theprobability measure concentrated at z = 1 and δ−1 is the probability measureconcentrated at z = −1. In other words, we assert that for every function gin C[I × Z] and τ in [0, 1],

limn→∞

∫ τ

0

∫

Z

g(t, z)dµntdt =1

2

∫ τ

0

g(t, 1)dt+1

2

∫ τ

0

g(t,−1)dt.

Since the set of polynomials in t and z is dense in C[I×Z] to prove the assertion


it suffices to show that for any τ in [0, 1] and any nonnegative integers p andq

limn→∞

∫ τ

0

∫

Z

tpzqdµntdt =1

2

∫ τ

0

tpdt+1

2

∫ τ

0

tp(−1)qdt

If q is even or zero,∫ τ

0

∫

Z

tpzqdµntdt =

∫ τ

0

tpdt = τp+1/(p+ 1) =1

2

∫ τ

0

tpdt+1

2

∫ τ

0

tp(−1)qdt.

This proves the assertion for q even or zero.If q is odd and p = 0, let ℓ be the largest positive integer such that

2ℓ/2n ≤ τ . Thus, τ lies in [2ℓ/n, (2ℓ + 1)/2n) or [(2ℓ + 1)/2n, (2ℓ + 2)/2n].Then

|∫ τ

0

∫

Z

zqdµntdt| = |∫ τ

2ℓ/2n

∫

Z

zqdµntdt| ≤∫ τ

2ℓ/2n

dt ≤ 2

2n→ 0.

If q is odd and p ≥ 1, let ℓ be as before. Then∫ τ

0

tp∫

Z

zqdµntdt =

∫ 2ℓ/2n

0

tp∫

Z

zqdµntdt+

∫ τ

2ℓ/2n

tp∫

Z

zqdµntdt.

The second integral in absolute value is less than or equal to 2/2n, and sotends to zero as n→ ∞.

We now estimate the first integral.

In ≡∫ 2ℓ/2n

0

tp∫

Z

zqdµntdt =

2ℓ−1∑

i=0

(−1)i∫ (i+1)/2n

i/2n

tpdt

=1

(2n)p+1(p+ 1)

2ℓ−1∑

i=0

(−1)i[(i+ 1)p+1 − ip+1].

If in the rightmost sum we group the first two terms, then the next two terms,and so on we can rewrite this sum as

1

(2n)p+1(p+ 1)

ℓ−1∑

j=0

(−1)2j+1[(2j + 2)p+1 − (2j + 1)p+1]

+ (−1)2j [(2j + 1)p+1 − (2j)p+1].

This in turn can be written as

1

(2n)p+1(p+ 1)

ℓ−1∑

j=0

[−(2j + 2)p+1 + 2(2j + 1)p+1 − (2j)p+1].

Using the binomial theorem, we find that each term in square brackets isO(jp−1). Using the integral

∫ ℓ

0

(x+ 1)p−1dx =(ℓ+ 1)p − 1

p,

Relaxed Controls 47

and using ℓ ≤ nτ , we get that

In =1

(2n)p+1O((2nτ)p),

and so In → 0 as n→ ∞. Combining this result with previous results gives

limn→∞

∫ τ

0

∫

Z

tpzqdµntdt = 0 =1

2

∫ τ

0

tpdt+1

2

∫ τ

0

tp(−1)qdt,

which proves the assertion for q even, and hence for all q.On the other hand, we assert that the sequence un as an element of

Lp[Iτ ], 1 < p < ∞, 0 < τ ≤ 1, converges weakly to zero. That is, we assertthat for any function v in Lq[Iτ ], where 1/p+ 1/q = 1, we have

limn→∞

∫ τ

0

v(t)un(t)dt = 0.

Since the set of polynomials in t is dense in Lq[Iτ ], it suffices to show that forany nonnegative integer j,

limn→∞

∫ τ

0

tjun(t)dt = 0.

Let ℓ again be the largest positive integer such that 2ℓ/2n ≤ τ . If j = 0, wehave ∫ τ

0

un(t)dt =

∫ τ

2ℓ/2n

un(t)dt → 0 as n→ ∞.

For j ≥ 1, we integrate by parts to get

∫ τ

0

tjun(t)dt = τ j∫ τ

0

un(t)dt−∫ τ

0

(jtj−1

∫ t

0

un(s)ds)dt → 0

Definition 3.3.5. A sequence µn of relaxed controls on I is weakly compactif there exists a subsequence µnk

and a relaxed control µ on I such that µnk

converges weakly to µ.

The next theorem is the principal result of this section.

Theorem 3.3.6. A sequence µn of relaxed controls on a compact intervalI is weakly compact.

Proof. Step I. Preliminary observations.The proof proceeds by induction on n, the dimension of f . Since the proof

of the general induction step and the proof for n = 1 are the same, we needonly present the proof for n = 1.

Let I ′ be an interval contained in I. The sequence µn defines a sequenceLn of continuous linear functionals in C∗(I ′ × Z) given by (3.3.1) withI = I ′. By (3.3.2) the sequence Ln lies in the closed ball of radius |I ′| and


hence is weak-∗ compact. That is, there is a continuous linear functional L inC∗(I ′ ×Z) with ‖L‖ ≤ |I ′| and a subsequence Lnk

such that for every g inC(I ′ × Z)

limk→∞

Lnk(g) = L(g).

Henceforth we shall relabel subsequences Lnk as Ln. It follows from (3.3.1)

that the linear functionals Ln are positive, and therefore so is L. From theRiesz Representation Theorem we get that there exists a positive regular Borelmeasure ν′ on I ′ × Z such that for each g in C(I ′ × Z)

L(g) =

∫

I′×Z

g(t, z)dν′.

Thus,

limn→∞

∫

I′

(∫

Z

g(t, z)dµnt

)dt =

∫

I′×Z

g(t, z)dν′.

Step II. Let τi be a countably dense set of points in I which includes theorigin and the right-hand end point of I, and let Ii = [0, τi]. Let µn be a se-quence of relaxed controls on I. Then there exists a subsequence (independentof i) that we relabel as µn, and for each i a regular positive Borel measureνi on Ii × Z such that for every g in C(Ii × Z)

limn→∞

∫ τi

0

(∫

Z

g(t, z)dµnt

)dt =

∫

Ii×Z

g(t, z)dνi.

Proof. From Step I it follows that for τ1 there exists a subsequence µn anda positive regular Borel measure ν1 on I1 × Z with the requisite properties.At τ2 we again apply Step I and obtain a subsequence of the subsequence ob-tained at τ1 as well as a positive Borel measure ν2 with the desired properties.Proceeding inductively in this manner we obtain a sequence of subsequencesand a sequence of measures. If we take the diagonal elements in the array ofsubsequences, we obtain the desired subsequence.

Step III. For each τ in I there exists a positive regular Borel measure ντ onIτ = [0, τ ] such that for every g in C(Iτ × Z)

limn→∞

∫ τ

0

(∫

Z

g(t, z)dµnt

)dt =

∫

Iτ×Z

g(t, z)dντ , (3.3.4)

where µn is the subsequence obtained in Step II.

Proof. For points τi in the dense set this was established in Step II. Nowlet τ be an arbitrary point in I not in the set τi. For g in C(Iτ × Z) let

Lτn(g) =

∫ τ

0

(∫

Z

g(t, z)dµnt

)dt n = 1, 2, . . .

Relaxed Controls 49

We shall show that for fixed g the sequence of real numbers Lτn(g) is Cauchy.

For every τ ′ < τ ′′ in I, since µn is a probability measure, and since C(Iτ ′′×Z) ⊆ C(Iτ ′ × Z), we have for g ∈ C(Iτ ′′ × Z)

|Lτ ′

n (g)− Lτ ′′

n (g)| = |∫ τ ′′

τ ′

(∫

Z

g(t, z)dµn,t

)dt| (3.3.5)

≤ ‖g‖(τ ′′ − τ ′).

Let ε > 0 be arbitrary. Then there exists a point τj < τ such that τ − τj < ε.Therefore, for arbitrary positive integers m,n

|Lτm(g)− Lτ

n(g)| ≤ |Lτm(g)− Lτj

m(g)|+ |Lτjm(g)− Lτj

n (g)|+ |Lτjn (g)− Lτ

n(g)|.

From (3.3.5) and the fact that Lτjn (g) is Cauchy we get that for m,n suffi-

ciently large|Lτ

m(g)− Lτn(g)| ≤ 2ε‖g‖+ ε.

Hence Lτn(g) is Cauchy and so converges to a number Lτ (g).

Since for fixed τ and each n the mapping g → Lτn(g) is linear, it follows

that g → Lτ (g) is linear. Also, since each Lτn is positive, so is Lτ . Also,

|Lτ (g)| = | limn→∞

∫ τ

0

(∫

Z

g(t, z)dµnt

)dt|

≤ limn→∞‖g‖∫ τ

0

(∫

Z

dµnt

)dt = ‖g‖τ.

If we take g to be the function identically one, we get that

‖Lτ‖ = τ.

Hence Lτ is a continuous linear functional on C(Iτ × Z). By the Riesz repre-sentation theorem there exists a positive regular Borel measure ντ on [Iτ ×Z]such that

Lτ (g) =

∫

Iτ×Z

g(t, z)dντ .

This establishes (3.3.4).Step IV. There exists a set T ⊆ I with |T | = |I| such that for each g inC(I ×Z) the mapping τ → Lτ (g) is differentiable at all points of T . Here |T |denotes the Lebesgue measure of T .

Proof. Since (3.3.5) holds for arbitrary g in C(I × Z), we get that for fixedg the mapping τ → Lτ (g) is Lipschitz, and hence is differentiable almosteverywhere. Let gi be a countable set of functions that is dense in C(I×Z).Corresponding to each gi there is a set Ti ⊂ I with |Ti| = I such that themapping τ → Lτ (gi) is differentiable at all points of Ti. Let T =

⋂∞i=1 Ti.


Then, |T | = |I| and for each i the mapping τ → Lτ (gi) is differentiable at allpoints of T .

We now show that for each g in C(I × Z) the mapping τ → Lτ (g) isdifferentiable at all points of T . Let τ0 be a point in T and for τ 6= τ0 define

∆τ,τ0(g) =Lτ (g)− Lτ0(g)

τ − τ0.

Then to show that τ → Lτ (g) is differentiable at τ0 it suffices to show thatfor τ1, τ2 → τ0 that

∆τ1,τ0(g)−∆τ2,τ0(g) → 0. (3.3.6)

For gi as in the preceding paragraph,

|∆τ1,τ0(g)−∆τ2,τ0(g)| ≤ |∆τ1,τ0(g − gi)|+ |∆τ2,τ0(g − gi)|+ |∆τ1,τ0(gi)−∆τ2,τ0(gi)|.

From the definition of ∆τ,τ0(g) and (3.3.5) we get that each of the first twoterms on the right do not exceed ‖g − gi‖. Hence by appropriate choice of githey can be made as small as we wish. Since τ → Lτ (gi) is differentiable atτ0, the last term tends to zero as τ1, τ2 → τ0, and (3.3.6) is established.

Step V. For each t in T there exists a positive regular Borel measure µt onZ such that for every g in C(I × Z) the mapping

t→∫

Z

g(t, z)dµt (3.3.7)

is Lebesgue measurable and for every τ in I

Lτ (g) =

∫ τ

0

(∫

Z

g(t, z)dµt

)dt. (3.3.8)

Proof. For each t in T we have a functional λt on C(I × Z) defined by

λt(g) = [dLτ (g)/dτ ]τ=t.

Clearly λt is linear. We next show that λt is bounded, and hence continuous.

|λt(g)| = limτ→t

∣∣∣∣Lτ (g)− Lt(g)

τ − t

∣∣∣∣ (3.3.9)

≤ limτ→t

max |g(s, z)| : t ≤ s ≤ τ, z ∈ Z

= max|g(t, z)| : z ∈ Z ≤ ‖g‖,

where the first inequality follows from (3.3.5).

Relaxed Controls 51

We now show that λt is positive. Let g in C(I×Z) be nonnegative. Then foreach n, τ → Lτ

n(g) is a nondecreasing function, and therefore so is τ → Lτ (g).Hence

λt(g) = [dLτ (g)/dτ ]τ=t ≥ 0.

From the Riesz Representation Theorem we get that there is a positiveregular Borel measure µt on I×Z such that ‖λt‖ = ‖µt‖ and for g in C(I×Z)

λt(g) =

∫

I×Z

g(s, z)dµt.

We assert that µt is concentrated on t×Z. If the assertion were false, thensince µt is regular and positive, there would exist a compact set K in I × Zdisjoint from t×Z such that µt(K) > 0. By Urysohn’s Lemma there existsa nonnegative function g0 in C(I × Z) that is one on K and zero on t × I.Hence, λt(g0) > 0. But from (3.3.9) we have

|λt(g0)| ≤ max|g0(t, z)| : z ∈ Z = 0.

Thus, µt is concentrated on t × Z and we may write

λt(g) =

∫

Z

g(t, z)dµt t ∈ T (3.3.10)

and‖λt‖ = ‖µt‖var = µt(Z). (3.3.11)

Since for fixed g, the mapping t → λt(g) is the derivative of a Lipschitzfunction, it is measurable. From this and from (3.3.10) the measurability of(3.3.7) follows.

It follows from (3.3.5) that the function τ → Lτ (g) is Lipschitz and henceabsolutely continuous. It is therefore the integral of its derivative, and so

Lτ (g) =

∫ τ

0

[dLt(g)/dt]dt =

∫ τ

0

λt(g)dt =

∫ τ

0

(

∫

Z

g(t, z)dµt)dt,

which establishes (3.3.8).The integrals are actually taken over the set [0, τ ]∩T , where T is the same

for all g.Step VI. For all t in T , µt is a regular probability measure.

Proof. Since µt is a positive regular measure, to prove this statement it suf-fices, in view of (3.3.11), to show that ‖λt‖ = 1.

Let t be an arbitrary point in T . Let g1 be a nonnegative function in I×Zwith range in [0, 1] such that g1 ≤ 1 and g1 ≡ 1 on a set (τ, z) : |τ − t| ≤ε, z ∈ Z for ε sufficiently small. Because Z is compact, such a function exists.Hence for τ sufficiently close to t we have by (3.3.8) that

Lτ (g1) =

∫ t

0

(∫

Z

g1(s, z)dµs

)ds+

∫ τ

t

(∫

Z

g1(s, z)dµs

)ds


= Lt(g1) + limn→∞

∫ τ

t

(∫

Z

g1(s, z)dµns

)dt = Lt(g1) + lim

n→∞(τ − t),

since for each n and s dµns is a probability measure. Thus,

Lτ (g1) = Lt(g1) + (τ − t),

and soλt(g1) = dLτ (g1)/dτ ]τ=t = 1 = ‖g1‖.

But from (3.3.9) we have that ‖λt(g)‖ ≤ ‖g‖ for all g in C(I × Z). Hence‖λt‖ = 1.Step VII. Completion of proof.

The mapping (3.3.7) is Lebesgue measurable and µt is a regular probabilitymeasure on Z for almost all t in I. Hence

µ : t→ µt

is a relaxed control. Since for every g in C(I × Z) and every τ in I

limn→∞

Lτn(g) = Lτ (g),

it follows from (3.3.8) that the subsequence µn converges weakly to µ.

Remark 3.3.7. The condition that Z is compact cannot be replaced by thecondition that Z is closed. To see this let Z be an unbounded closed set andlet I be a compact interval in R. Let zi be a sequence of points in Z whosenorms tend to infinity. Let g be a function in C(I ×Z) with compact supportand independent of t. Thus, g(t, z) = g(z). Let µn be a sequence of relaxedcontrols defined as follows:

µn(t) = δzn t ∈ I,

where δzn is the Dirac measure equal to one at the point zn and equal to zeroon any set that does not contain zn. Then

Ln(g) =

∫

I

(

∫

Z

g(z)dµn)dt = g(zn)|I|.

For n sufficiently large, g(zn) = 0, and so Ln(g) → 0.On the other hand, we cannot find a regular probability measure µ such

that for all g in C(Z) with compact support∫

Z

g(z)dµ = 0.

For if µ is a regular probability measure, there is a compact set K ⊂ Z suchthat µ(K) > 0. Let gK be a nonnegative continuous function with compactsupport such that gK(z) = 1 for z ∈ K. Then Ln(gK) → 0 as n→ ∞, but

∫

Z

gK(z)dµ ≥∫

K

gK(z)dµ = µ(K) > 0.

Relaxed Controls 53

The compactness of Z was used in the proof to show that for t ∈ T , ‖µt‖ = 1;it was not used elsewhere.

In establishing existence theorems and necessary conditions we shall beconsidering sequences of the form

∫

I

∫

Ω(t)

g(t, z)dµntdt,

where for each t the measure µn,t is concentrated on a compact set Ω(t). Thisis in contrast to the sequences in Theorem 3.3.6, where for each t the measureµn,t is concentrated on a fixed compact set Z. We shall impose a regularitycondition on the mapping Ω: t → Ω(t) which guarantees that all of the setsΩ(t) lie in some fixed compact set Z. We can then consider the measures µnt

to be measures on Z that are concentrated on Ω(t). Thus, we can write∫

I

∫

Ω(t)

g(t, z)dµntdt =

∫

I

∫

Z

g(t, z)dµntdt.

Theorem 3.3.6 gives the existence of a subsequence µn that convergesweakly to a relaxed control µ such that for almost every t, the probabilitymeasure µt is concentrated on Z. We would like to conclude, however, thatfor almost every t the measure µt is concentrated on Ω(t). We shall prove thatthis is indeed the case, and begin by introducing some notation and definitions.

Let X be a subset of Rp and let |ξ − η| denote the euclidean distance

between points ξ and η in Rp. Let Λ be a mapping from X to subsets of a

euclidean space Rq. For ξ0 in X let Nδ(ξ0) denote the delta neighborhood of

ξ0 relative to X ; that is,

Nδ(ξ0) = x : x ∈ X, |x− ξ0| < δ.

Let Λ(Nδ(ξ0)) denote the image of Nδ(ξ0) under Λ. If U is a set in Rq, let

dist(y, U) denote the euclidean distance between a point y in Rq and U . Thus,

dist(y, U) = inf|y − z| : z ∈ U.

Let[U ]ε = y : dist(y, U) ≤ ε.

We shall call [U ]ε a closed ε-neighborhood of U .

Definition 3.3.8. A mapping Λ is said to be upper semi-continuous withrespect to inclusion at a point x0, written u.s.c.i, if for each ε > 0, there existsa δ > 0 such that for all x in Nδ(x0) the inclusion Λ(x) ⊆ [Λ(x0)]ε holds. Themapping Λ is u.s.c.i. on X if it is u.s.c.i. at every point of X .

Remark 3.3.9. In Section 5.2 we shall define the notion of upper semi-continuity of a mapping. Some authors define upper semi-continuity to bethe notion that we call u.s.c.i. Readers of the literature should check whichdefinition of upper semi-continuity is being used.


Example 3.3.10. Let Λ1 and Λ2 be maps from [0, 1] to subsets of R1 definedby

Λ1(t) =

z |z| ≤ 1

t 0 < t ≤ 1

R1 t = 0

Λ2(t) =

z |z| ≤ 1

t 0 < t ≤ 1

0 t = 0.

Then Λ1 is u.s.c.i on [0, 1], while Λ2 is u.s.c.i on (0, 1] but is not u.s.c.i att = 0. Note that Λ1 is u.s.c.i on I but that GΛ1

= (ξ, λ) ξ ∈ [0, 1], λ ∈ Λ(ξ)is not compact. This does not contradict Lemma 3.3.11 because Λ1(0) is notcompact and one of the hypotheses of the lemma is that all sets Λ(ξ) arecompact.

Let Λ be a constant map; that is, a map such that for all ξ in X , Λ(ξ) = K,K a fixed set. Then Λ is u.s.c.i.

The next lemma shows that if the mapping Ω is u.s.c.i. on I, then all ofthe sets Ω(t) lie in a fixed compact set.

Lemma 3.3.11. Let Λ be a mapping from a compact set X in Rp to subsets

of Rq such that for each ξ in X, the set Λ(ξ) is compact. A necessary andsufficient condition that Λ be u.s.c.i. on X is that the set

GΛ = (ξ, λ) : ξ ∈ X, λ ∈ Λ(ξ)

is compact.

Proof. Suppose that Λ is u.s.c.i. on X . Let (ξn, λn) be a sequence of pointsin GΛ. Since X is compact, there exists a subsequence (ξn, λn) and a pointξ0 in X such that lim ξn = ξ0. Let ε > 0 be given. Since λn ∈ Λ(ξn) and Λis u.s.c.i. at ξ0, there exists a positive integer N such that for n > N, λn ∈[Λ(ξ0)]ε. Thus, λn is bounded and dist(λn,Λ(ξ0)) → 0. Hence there existsa subsequence that we again label as (ξn, λn) and a point λ0 such thatlim ξn = ξ0 and limλn = λ0. Letting n→ ∞ in the relation

dist (λ0,Λ(ξ0)) ≤ |λn − λ0|+ dist(λn,Λ(ξ0))

and recalling that Λ(ξ0) is closed gives λ0 ∈ Λ(ξ0). Hence (ξ0, λ0) ∈ GΛ, andso GΛ is compact.

Now suppose that GΛ is compact and that Λ is not u.s.c.i at some pointξ0 in X . Then there exists an ε0 > 0 and a sequence (ξn, λn) in GΛ suchthat lim ξn = ξ0 and dist(λn,Λ(ξ0)) > ε0. Since GΛ is compact there existsa subsequence that we relabel as (ξn, λn) and a point (ξ′0, λ0) in GΛ suchthat lim(ξn, λn) = (ξ′0, λ0). Hence ξ

′0 = ξ0, and so λ0 ∈ Λ(ξ0). This contradicts

dist(λn,Λ(ξ0)) > ε0, and so Λ is u.s.c.i at ξ0.

Relaxed Controls 55

Theorem 3.3.12. Let I be a compact interval in R1. Let Ω: t → Ω(t) be a

mapping from I to subsets of Rk that is upper semi-continuous with respectto inclusion on I and such that for each t in I the set Ω(t) is compact. Letµn be a sequence of relaxed controls such that for each n the measure µnt

is for almost every t concentrated on Ω(t). Then there exists a subsequence ofthe sequence µn that converges weakly to a relaxed control µ such that foralmost all t the measure µt is concentrated on Ω(t).

Proof. It follows from the upper semi-continuity with respect to inclusion ofΩ and Lemma 3.3.11 that there exists a compact set Z such that all the setsΩ(t) are contained in Z. It then follows from Theorem 3.3.6 that there existsa subsequence of µn that converges weakly to a relaxed control µ : t → µt,where for all t in a set T of full measure in I the measure µt is concentratedon t × Z. We shall show that for all t in T the measure µt is concentratedon t × Ω(t).

Suppose that at a point t in T , the set Ω(t) is a proper subset of Z andthat µt is not concentrated on t × Ω(t). Then there exists a compact setK in t × Z such that K is disjoint from Ω(t) and µt(K) > 0. Let ε0 > 0denote the distance between the compact sets K and Ω(t). Since Ω is u.s.c.i,there exists a δ > 0 such that for |τ − t| ≤ δ, the sets Ω(τ) are contained in[Ω(t)]ε0/2. Moreover, the set

Γδ = (τ, z) : |τ − t| ≤ δ, z ∈ Ω(τ)

is compact by virtue of Lemma 3.3.11. Thus, the compact sets K and Γδ aredisjoint. By Urysohn’s Lemma there exists a nonnegative continuous functiong0 on I × Z that takes on the value one on K and the value zero on Γδ.

Let λt(g0) be as in Step V in the proof of Theorem 3.3.6. Then from(3.3.10) we get

λt(g0) =

∫

Z

g0(t, z)dµt = µt(K) > 0. (3.3.12)

On the other hand, from the definition

λt(g) = [dLτ (g)/dτ |τ=t

in Step V of the proof of Theorem 3.3.6 we get

λt(g0) = limτ→t

Lτ (g0)− Lt(g0)

τ − t

= limτ→t

1

τ − t

[lim

n→∞

∫ τ

t

(∫

Z

g0(s, z)dµns

)dt

], (3.3.13)

where the notation is as in the proof of Theorem 3.3.6. Since µns is concen-trated on Ω(s), we have

∫ τ

t

∫

Z

g0(s, z)dµns =

∫ τ

t

∫

Ω(s)

g0(s, z)dµns. (3.3.14)


For |t − τ | ≤ δ, the integrand on the right in (3.3.14) has domain Γδ, and sois zero. From (3.3.13) we then get that λt(g0) = 0, which contradicts (3.3.12).Thus, µt is concentrated on Ω(t) for all t in T .

Remark 3.3.13. It follows from (3.3.1) and Definition 3.3.2 that if µn isa sequence of relaxed controls that converges weakly to a relaxed control µ,and if Ln and L are the corresponding continuous linear transformations,then Ln converges weak-∗ to L. In proving Theorems 3.3.6 and 3.3.12 we haveshown that the converse is also true. Let Ln be a sequence of continuouslinear transformations on C(I × Z) corresponding to a sequence of relaxedcontrols µn such that for each n and a.e. t in I, the measure µnt is concentratedon Ω(t), and let Ln converge weak-∗ to a continuous linear transformation Lfrom C(I × Z) to R

n. Then there exists a relaxed control µ such that µt isconcentrated on Ω(t) for a.e. t in I and L is given by (3.3.1). Moreover, µn

converges weakly to µ.

3.4 Filippov’s Lemma

In this section we shall prove a general implict function theorem for mea-surable functions. Originally, this theorem was given in a less general formby A. F. Filippov [33]. The form given here is due to McShane and Warfield[64]. This theorem has several important applications in optimal control the-ory. One was already given in proving Theorem 3.2.11, where we employed acorollary of the theorem.

Theorem 3.4.1. Let T be a measure space, let Z be a Hausdorff space, and letD be a topological space that is the countable union of compact metric spaces.Let Γ be a measurable map from T to Z and let ϕ be a continuous map fromD to Z such that Γ(T ) ⊆ ϕ(D). Then there exists a measurable map m fromT to D such that the composite map ϕ ∗ m from T to Z is equal to Γ.

Remark 3.4.2. In our applications, T = I, where I is a real interval withLebesgue measure, Z = R

p and D = Rq, real euclidean spaces. Recall that a

mapping Γ is measurable if for every compact set K in Z, the set Γ−1(K) inT is measurable.

Corollary 3.4.3 (Lemma 3.2.10). Let I denote a real compact interval, U aninterval in R

k, and let h be a map from I×U into Rn that is continuous on U

for a.e. t in I and is measurable on I for each z in U . Let W be a measurablefunction on I with range in R

n, and let V be a function from I to U such that

W (t) = h(t, V (t)) a.e. (3.4.1)

Then there exists a measurable function V : I → U such that

W (t) = h(t, V (t)) a.e. (3.4.2)

Relaxed Controls 57

Proof of Theorem. Let Ci∞i=1 denote the compact metric spaces whose unionequals D. By a theorem of Urysohn, every compact metric space is the con-tinuous image of a Cantor set. (For a proof of this theorem see [45]). LetLi, i = 1, 2, 3, . . . be the translate of the Cantor set in [0, 1] to the interval[2i− 1, 2i]. Let ψi be the continuous map of Li onto the compact metric spaceCi. Let L =

⋃∞i=1 Li. Since the complement of L is open, L is closed. Define a

mapping ψ from L to D as follows. If t ∈ Li, then ψ(t) = ψi(t). The mappingψ is clearly continuous. Let θ = ϕ ∗ ψ. Then θ is a continuous mapping fromL onto ϕ(D) in Z.

Let t ∈ T . Since Γ(T ) ⊆ ϕ(D) = θ(L), it follows that θ−1(Γ(t)) is a non-empty set in L. Since θ is continuous and Γ(t) is a point, the set θ−1(Γ(t))is closed and is bounded below. Hence infθ−1(Γ(t)) is finite and is in theset θ−1(Γ(t)), which is contained in L. Define a mapping γ from T to L asfollows:

γ(t) = infθ−1(Γ(t)).Hence

θ(γ(t)) = Γ(t). (3.4.3)

From the definition of L, it follows that γ(t) ≥ 1. From (3.4.3) and the defi-nition of θ, we get that

(ϕ ∗ ψ ∗ γ)(t) = Γ(t).

Let m = (ψ ∗γ). Then m is a mapping from T to D, and (ϕ∗m)(t) = Γ(t)for t in T . To complete the proof we must show that m is measurable. Sinceψ is continuous, it suffices to show that γ is measurable.

To show that γ is measurable, we need to show that for every real numberc, the set

Tc = t : γ(t) ≤ cis measurable. If c < 1, then Tc is empty, so we need only consider c ≥ 1. Ift ∈ Tc, then θ(γ(t)) ⊆ θ(L ∩ [0, c]). Thus, by (3.4.3), Γ(t) ∈ θ(L ∩ [0, c]). Nowsuppose that Γ(t) ∈ θ(L∩ [0, c]). Then by (3.4.3), θ(γ(t)) ∈ θ(L∩ [0, c]). Henceγ(t) ≤ c. Thus,

Tc = t : Γ(t) ∈ θ(L ∩ [0, c])or

Tc = Γ−1(θ(L ∩ [0, c])).

Now, L∩[0, c] is compact and therefore so is θ(L∩[0, c]). Since Γ is measurable,Γ−1(K) is measurable for any compact set. Hence, Tc is measurable for everyc ≥ 1.

Proof of Corollary. We first prove the corollary under the assumption that his continuous on I × U. In reading the proof, the reader should note that thecorollary will hold if we replace I by a measurable set T and assume that his continuous on T × U .

Let T = I, let Z = I × Rn, and let D = I × R

k. Let (τ, V ), where τ ∈ I


and V ∈ Rk, denote a generic point in D. Define a mapping ϕ : D → Z by the

formulaϕ(τ, V ) = (τ, h(τ, V )).

Since h is continuous, so is ϕ. Let Γ: I → Z be defined by

Γ(t) = (t,W (t)).

Thus, Γ is measurable. Equation (3.4.1) implies that Γ(I) ⊆ ϕ(D). Thus, allthe hypotheses of the theorem are fulfilled. Hence there exists a measurablemap m from I to D = I × R

k with

m(t) = (τ(t), V (t))

such that (ϕ ∗m)(t) = Γ(t). Thus, for all t in I

(ϕ ∗m)(t) = ϕ(τ(t), V (t)) = (τ(t), h(τ(t), V (t))) = (t,W (t)).

Since m is measurable, so is V . From the rightmost equality we first get thatτ(t) = t and then (3.4.2).

The completion of the proof of the corollary (Lemma 3.2.10) utilizes thefollowing result, whose proof we give at the end of this section. For manycontrol questions this result is used to validate one of Littlewood’s principlesof analysis, which states that generally what is true for continuous functionsis true for measurable functions.

Lemma 3.4.4. Let T be a compact interval in Rs and let U be an interval

in Rn. Let h be a function from T × U to R

n such that for almost all t in T ,h(t, ·) is continuous in U and for all z in U, h(·, z) is measurable on T . Thenfor each ρ > 0 there exists a closed set F with meas (T − F ) < ρ such that his continuous on F ×U . If U is closed, then there exists a continuous functionH from T ×U to R

n such that H(t, z) = h(t, z) for all t in F and all z in U .

By Lemma 3.4.4, for each positive integer j, there exists a closed set Fj ⊆ Isuch that meas (I − Fj) < 2−j and such that h is continuous on Fj × U . Bywhat was just proved there exists a function Vj defined and measurable onFj , with range in U such that

W (t) = h(t, Vj(t)) a.e. in Fj . (3.4.4)

Let Ej denote the set of measure zero on which (3.4.4) fails. Let E denote theunion of the sets Ej , j = 1, 2, . . .. Then meas E = 0.

We now define a sequence of closed sets F ′j and measurable functions V ′

j

inductively. Define

F ′1 = F1 V ′

1(t) = V1(t) t 6∈ E

and V ′1(t) = any point in U if t ∈ E. Then meas (I − F ′

1) < 2−j and W (t) =

Relaxed Controls 59

h(t, V ′1(t)) for t ∈ F ′

1, t 6∈ E. Suppose now that for each j = 1, . . . , k thereis defined a closed set F ′

j , and a measurable function V ′j defined on F ′

j withrange in U such that

meas (I − F ′j) < 2−j F ′

j ⊃ F ′j−1 (3.4.5)

W (t) = h(t, V ′j (t)) t ∈ F ′

j t 6∈ E.

V ′j (t) = V ′

j−1(t) t ∈ F ′j−1 t 6∈ E.

Define F ′k+1 = F ′

k ∪ Fk+1. Since meas(I − F ′k) < 2−k and meas (I − Fk+1) <

2−k+1 neither of the sets F ′k or Fk+1 is contained in the other. Thus, F ′

k+1 ⊃F ′k. For any sets A and B, c(A ∪ B) = (cA) ∩ (cB), where cA denotes the

complement of A. It therefore follows that meas (I − F ′k+1) < 2−k+1. Define,

for t 6∈ E,

V ′k+1(t) =

V ′k(t) if t ∈ F ′

k

Vk+1(t) if t ∈ Fk+1 − F ′k.

Then V ′k+1 is measurable on I, V ′

k+1(t) = V ′k(t), and W (t) = h(t, Vk+1(t)) for

t ∈ (F ′k+1 −E). We have thus defined a sequence of measurable sets F ′

j andmeasurable functions V ′

j such that V ′j is defined on F ′

j and (3.4.5) holds.We extend the definition of each V ′

j to I by setting V ′j (t) equal to an arbitrary

element of U if t 6∈ F ′j .

Let

G =

∞⋃

j=1

F ′j .

Then G ⊆ I and meas (I − G) = 0. For each t ∈ G, there exists a positiveinteger j0(t) such that for j ≥ j0(t), V

′j (t) = V ′

j0(t). Hence there exists a

measurable function V defined in G such that for t ∈ G, V (t) = limj→∞ V ′j (t).

If for t ∈ I −G we define V (t) = z, where z is an arbitrary element of U , wehave that

V (t) = limj→∞

V ′j (t) a.e. in I,

V is measurable andW (t) = h(t, V (t)) a.e.

We conclude this section with a proof of Lemma 3.4.4.

Proof of Lemma 3.4.4. It will be convenient for us to define the norm of avector z = (z1, . . . , zn) in R

n by

‖z‖ = max|zi| : i = 1, . . . , n.

Since all norms in Rn are equivalent there is no loss of generality by taking

this definition. The set

C(0, a) = z : |zi| < a, i = 1, . . . , n


will be called the open cube of radius a centered at the origin. If we denotethe closure of C(0, a) by C(0, a), then C(0, a) = z : ‖z‖ ≤ a and C(0, a) =z : ‖z‖ < a. Let

Tc = t : h(t, ·) is continuous on U,

and let A = T − Tc. Then by hypothesis, meas A = 0.For each positive integer a, let Ua denote the closure of the intersection

of U and C(0, a). We first consider h on the cartesian product T × Ua andassume that T is bounded.

Let ε > 0 be given. For each positive integer m define

Eεm = t : If z1, z2 in Ua, ‖z1 − z2‖≤ √

n/m, then |h(t, z1)− h(t, z2)| ≤ ε/4.

We assert that each Eεm is measurable. If Eεm is empty, there is nothing toprove. We shall show the set cEεm = T − Eεm is measurable, from which itfollows that Eεm is measurable.

A point t0 is in cEεm −A if and only if there exist points z1, z2 in Ua suchthat ‖z1 − z2‖ ≤ √

n/m and |h(t0, z1)− h(t0, z2)| > ε/4. We can assume thatz1 and z2 have rational coordinates. For each pair of points z1, z2 in Ua withrational coordinates and satisfying ‖z1 − z2‖ ≤ √

n/m let

Ez1z2 = t : |h(t, z1)− h(t, z2)| > ε/4.

The point t0 belongs to one of the sets Ez1z2 . Also, each set Ez1z2 is in cEεm.Since the functions h(·, z1) and h(·, z2) are measurable, the set Ez1z2 is mea-surable. Let D denote the union of the sets Ez1z2 . Then D is the countableunion of measurable sets and so is measurable. From t0 ∈ Ez1z2 for some z1, z2and Ez1z2 ⊆ cEεm, it follows that

cEεm −A ⊆ D ⊆ cEεm.

Since A has measure zero, it follows that cEεm is measurable.From the definition it is clear that Eε1 ⊆ Eε2 ⊆ . . . Let

Eε0 =

∞⋃

m=1

Eεm F0 = T − Eε0.

If t 6∈ A, then h(t, ·) is uniformly continuous on Ua, and t ∈ Eεm for sufficientlylarge m. Hence t ∈ Eε0. If t ∈ F0, than t 6∈ Eεm for all m. Hence t ∈ A, andso F0 ⊂ A. Since A has measure zero, F0 has measure zero. Thus, meas Eε0 =meas T .

From the definition of Eε0 it follows that given an η > 0, there exists apositive integer m0(ε, η) such that meas Eεm0

> meas Eε0 − (εη)/2. In thelast paragraph we showed that meas T = meas Eε0, so

meas Eεm0> meas T − (εη)/2. (3.4.6)

Relaxed Controls 61

By the definition of Eεm0, for every pair of points z1, z2 in Ua with ‖z1−z2‖ ≤√

n/m0 we have

|h(t, z1)− h(t, z2)| ≤ ε/4 for all t ∈ Eεm0. (3.4.7)

Set p = 1 + [a], where [a] is the largest integer ≤ a, and divide the cubeC(0, a) into q = pnmn

0 congruent closed cubes d1, . . . , dq. The side of eachcube dj will have length a/pm0 ≤ 1/m0. We shall say that cubes dj and dkare contiguous if dj and dk have non-empty intersection. Thus, a cube dj such

that no point of dj is a boundary point of C(0, a) will have 3n − 1 contiguouscubes. Let zj denote the center of the cube dj . For any pair of contiguouscubes dj and dk we either have ‖zj − zk‖ = m−1

0 or ‖zj − zk‖ =√n/m0. In

either case we have‖zj − zk‖ ≤ √

n/m0.

By hypothesis, for each j = 1, . . . , q the function h(·, zj) is measurable onT . Hence, by Lusin’s theorem, for each j there exists a closed set Fj ⊂ T with

meas Fj > meas T − (εη)/2q (3.4.8)

such that h(·, zj) is continuous on Fj . Since T is compact and Fj is closed,the continuity is uniform. Let

V =

q⋂

j=1

Fj .

Then by (3.4.8)meas V > meas T − (εη)/2. (3.4.9)

Each function h(·, zj) will be uniformly continuous on V . Hence there existsa δ1(ε) such that if t and t′ are in V and both lie in some cube in R

s of radiusδ1(ε)

|h(t, zj)− h(t′, zj)| < ε/4 (3.4.10)

for all j = 1, . . . , q. Let Eε = V ∩Eεm0. Then Eε is measurable and by (3.4.6)

and (3.4.9),meas Eε > meas T − εη. (3.4.11)

Note that Eε depends on η as well as ε.Let δ(ε) = min(δ1(ε),m

−10 ). Let (t, z) and (t′, z′) be two points in Eε ×Ua

such that ‖(t, z)− (t′, z′)‖ ≤ δ(ε). Then

‖t− t′‖ ≤ δ1(ε) and ‖z − z′‖ ≤ m−10 . (3.4.12)

It follows from (3.4.12) that z and z′ are in the same cube dj or in contiguouscubes dj and dk. If z ∈ dj and z′ ∈ dk where dj and dk are contiguous, then(recall that zi is the center of cube di, i = 1, . . . , q)

|h(t′, z′)− h(t, z)| ≤ |h(t′, z′)− h(t′, zk)|+ |h(t′, zk)− h(t′, zj)|


+|h(t′, zj)− h(t, zj)|+ |h(t, zj)− h(t, z)|.

Since t′ ∈ Eεm0and ‖z′ − zk‖ ≤ √

nm−10 , from (3.4.7) we see that the first

term on the right is ≤ ε/4. Since ‖zk − zj‖ =√nm−1

0 , ‖z − zj‖ <√nm−1

0 ,and t ∈ Eεm0

, it again follows from (3.4.7) that the second and last terms onthe right are each ≤ ε/4. It follows from (3.4.10) that the second term on theright is ≤ ε/4. Hence

|h(t′, z′)− h(t, z)| ≤ ε (3.4.13)

whenever ‖(t′, z′)− (t, z)‖ ≤ δ(ε).If z and z′ are in the same cube dj

|h(t′, z′)− h(t, z)| ≤ |h(t′, z′)− h(t′, zj)|+ |h(t′, zj)− h(t, zj)|+|h(t, zj)− h(t, z|.

Using the argument of the preceding paragraph, we again conclude that(3.4.13) holds whenever ‖(t′, z′)− (t, z)‖ ≤ δ(ε).

Let εi = 2−i, let Ei = Eεi and let

E =

∞⋂

i=1

Ei.

From (3.4.11) we see that

meas E > meas T − (

∞∑

i=1

εi)η = T − η. (3.4.14)

We assert that h is continuous on the cartesian product E × Ua. To provethis, let γ > 0 be given. Let i = i(γ) be such that εi = 2−i < γ. Let (t, z) and(t′, z′) be in E × Ua and satisfy ‖(t′, z′)− (t, z)‖ ≤ δ(εi). Since t and t

′ are inEi we have by (3.4.13) that |h(t′, z′)− h(t, z)| ≤ 2−i < γ.

The sequence Uk is an increasing sequence of compact sets whose unionis U . We have just proved that for each ρ > 0, there exists a measurable setEi ⊂ T such that meas Ei > meas T − ρ2−(i+1) and such that h is continuouson the cartesian product T × Ei. Let

G =

∞⋂

i=1

Ei.

Then h is continuous on U ×G, and

meas (G) > meas T − ρ

∞∑

i=1

2−(i+1) = meas T − ρ/2.

Since G has positive finite measure, there exists a closed set F ⊂ G such that

meas (F ) > meas G− ρ/2 > meas T − ρ.

Moreover h is continuous on F × U . If U is closed, the existence of h followsfrom Tietze’s Extension Theorem.

Relaxed Controls 63

In Assumption 3.2.1(iv) we postulated the existence of a measurable func-tion u defined on I with range in U such that the state equation with this uhas a solution φ defined on I. Lemma 3.4.5 shows that if Ω is u.s.c.i and eachΩ(t) is compact, then there exists a measurable function u with u(t) ∈ Ω(t)a.e.

Lemma 3.4.5. Let Ω be a mapping form I to compact sets Ω(t) in Rm that

is u.s.c.i. Then there exists a measurable function u such that u(t) ∈ Ω(t) forall t in I.Proof. Let

∆ = (t, z) : t ∈ I, z ∈ Ω(t).Then by Lemma 3.3.11, since Ω is u.s.c.i, the set ∆ is compact. Let

d(t) = inf|z| : z ∈ Ω(t).Then d(t) ≥ 0 and is finite; since Ω(t) is compact, there exists a z(t) ∈ Ω(t)such that d(t) = |z(t)|.

We assert that the function d is lower semicontinuous and hence measur-able. To show this we shall show that for each real α, the set Eα = t : d(t) ≤α is closed. Let t0 be a limit point of Eα. Then there exists a sequence tnin Eα such that tn → t0. The points (tn, z(tn)) are in the compact set ∆.Hence there exists a subsequence (tn, z(tn) and a point (t0, z(t0)) such that(tn, z(tn)) → (t0, z(t0)) ∈ ∆. Hence z(t0) ∈ Ω(t0). Now α ≥ d(tn) = |z|(tn),so α ≥ |z0| ≥ d(t0). Thus, t0 ∈ Eα, and so Eα is closed.

Since the norm function is continuous, it follows from the measurability ofd, Lemma 3.2.10, and the relation d(t) = |z(t)| that there exists a measurablefunction u with u(t) ∈ Ω(t) such that d(t) = |u(t)|.Remark 3.4.6. Let Ω be as above and let the state equations be

dx

dt= A(t)x + h(t, z),

where A is measurable in I and h is measurable in t for fixed z and continuousin z for fixed t. Then the system

dx

dt= A(t)x + h(t, u(t))

has a solution φ defined on all of I. This follows from standard theorems ondifferential equations.

3.5 The Relaxed Problem; Non-Compact Constraints

In Definition 3.2.2 we defined a relaxed control to be a mapping thatassigned a regular probability measure µt to each compact constraint set Ω(t).


We then showed that if we restrict the measures µt to be convex combinationsof Dirac measures, then we obtain the same set of relaxed trajectories as we dousing Definition 3.2.2. We introduced Definition 3.2.2 rather than the simplerone involving convex combinations of Dirac measures because the proof ofthe weak compactness of relaxed controls in the case of compact constraintsrequired the consideration of general probability measures.

In Remark 3.3.7 we showed that weak compactness of relaxed controls mayfail if the sets Ω(t) are not contained in some compact set. Thus, if the setsΩ(t) are not compact one of the advantages of using Definition 3.2.2 is lost.Moreover, if Ω(t) is not compact, for the integral

∫

Ω(t)

g(t, z)dµt

to exist, we must place conditions on the behavior of the function g(t, ·). Hencein the case of non-compact constraint sets we shall define a relaxed control tobe a convex combination of Dirac measures.

Assumption 3.5.1. Let f = (f0, f1, . . . , fn) be as in Assumption 3.2.1. LetΩ be a mapping from I to subsets of U ; that is, Ω: t→ Ω(t), where Ω(t) ⊆ U .

Definition 3.5.2. A relaxed control is a function v of the form v = (u1, . . .,un+2, p

1, . . . , pn+2), where each ui is a measurable function on I with rangein R

m satisfying the relation ui(t) ∈ Ω(t) and each pi is a measurable realvalued nonnegative function on I such that

n+2∑

i=1

pi(t) = 1 a.e.

Definition 3.5.3. A discrete measure control on I is a mapping µ on I toprobability measures

µt =

n+2∑

i=1

pi(t)δui(t), (3.5.1)

where each pi is a nonnegative measurable function, Σpi(t) = 1, and each uiis a measurable function with ui(t) ∈ Ω(t) a.e.

Let g be a mapping from I ×U to Rn that is continuous on U for a.e. t in

I and measurable on I for all z in U . Then for a discrete measure control if

h(t) =

∫

Ω(t)

g(t, z)dµt =

n+2∑

i=1

pi(t)g(t, ui(t)),

then h is measurable. Thus, a discrete measure control is a relaxed control.

Remark 3.5.4. If we define a relaxed control to be a discrete measure control,then this definition is equivalent to Definition 3.5.3.

Relaxed Controls 65

Definition 3.5.5. An absolutely continuous function ψ = (ψ1, . . . , ψn) de-fined in interval [t0, t1] ⊆ I is a relaxed trajectory corresponding to a relaxedcontrol v if

(i) (t, ψ(t)) ∈ R for all t ∈ [t0, t1].

(ii) ψ is a solution of the differential equation

dx

dt=

n+2∑

i=1

pi(t)f(t, x, ui(t)) =

∫

Ω(t)

f(t, x, z)dµt, (3.5.2)

where µt is as in (3.5.1).

The differential equation (3.5.2) is called the relaxed differential equation.

Definition 3.5.6. A relaxed trajectory ψ corresponding to a relaxed controlv is said to be admissible if

(i) (t0, ψ(t0), t1, ψ(t1)) ∈ B and the function

t→n+2∑

i=1

pi(t)f0(t, ψ(t), ui(t)) =

∫

Ω(t)

f0(t, ψ(t), z)dµt

is integrable. The pair (ψ, v) or (ψ, µ) is said to be an admissible pair.

The relaxed problem in the case of non-compact constraints is as follows:

Problem 3.5.1. Find a relaxed admissible pair (ψ, v) that minimizes

J(ψ, v) = g(t0, ψ(t0), t1, ψ(t1)) +

∫ t1

t0

[n+2∑

i=1

pi(t)f0(t, ψ(t), ui(t))

]dt. (3.5.3)

Remark 3.5.7. In view of Theorem 3.2.11, the relaxed problem in the caseof compact constraints can also be formulated as Problem 3.5.1.

Remark 3.5.8. The relaxed problem can also be viewed as an ordinary prob-lem with state variable x in R

n, with control variable

z ≡ (π, ζ) = (π1, . . . , πn+2, z1, . . . , zn+2) πi ∈ R, zi ∈ Rm,

with state equations

dx

dt≡ fr(t, x, z) =

n+2∑

i=1

πif(t, x, zi) πi ∈ R, zi ∈ Rm.

The integrand in (3.5.3) can also be written as

∫

Ω(t)

f0(t, ψ(t), z)dµt,


with dµt as in (3.5.1), or as

f0r (t, x, z) ≡

n+2∑

i=1

πif0(t, x, zi) πi ∈ R, zi ∈ Rm.

The constraints are now

zi ∈ Ω(t) πi ≥ 0

n+2∑

i=1

πi = 1. (3.5.4)

The end set B and the terminal payoff function g are as before.

Remark 3.5.9. Let

f = (f0, f) y = (y0, y) y0 ∈ R, y ∈ Rn

and let

Vr(t, x) =

y : y =

n+2∑

i=1

πif(t, x, zi), πi, zi as in (3.5.3)

V (t, x) =y : y = f(t, x, z) z ∈ Ω(t)

.

ThenVr(t, x) = co V (t, x).

This follows from Caratheodory’s theorem and the fact that for any set A, theset co(A) consists of all convex combinations of A.

3.6 The Chattering Lemma; Approximation to RelaxedControls

In this section we show that, under reasonable hypotheses, ordinary trajec-tories of a control system are dense in the relaxed trajectories of the system.The essence of this result is the following theorem, which is sometimes calledthe “Chattering Lemma” for reasons to be given in Remark 3.6.9.

Theorem 3.6.1. Let I be a compact real interval and let X be a compactset in R

n. Let f1, . . . , fq be functions defined on I × X with range in Rn and

possessing the following properties:

(i) Each fi is a measurable function on I for each x in X .

(ii) Each fi is continuous on X for each t in I.

Relaxed Controls 67

(iii) There exists an integrable function µ defined on I such that for all (t, x)and (t, x′) in I × X and i = 1, . . . , q:

|fi(t, x)| ≤ µ(t) (3.6.1)

|fi(t, x) − fi(t, x′)| ≤ µ(t)|x− x′|.

Let pi, i = 1, . . . , q, be real valued nonnegative measurable functions definedon I and satisfying

q∑

i=1

pi(t) = 1 a.e. (3.6.2)

Then for every ǫ > 0 there exists a subdivision of I into a finite collection ofnon-overlapping intervals Ej, j = 1, . . . , k and an assignment of one of thefunctions f1, . . . , fq to each Ej such that the following holds. If fEj

denotesthe function assigned to Ej and if f is a function that agrees with fEj

on theinterior E0

j of each Ej, that is,

f(t, x) = fEj(t, x) if t ∈ E0

j j = 1, . . . , k,

then for every t′, t′′ in I and all x in X∣∣∣∣∣

∫ t′′

t′

(q∑

i=1

pi(t)fi(t, x)− f(t, x)

)dt

∣∣∣∣∣ < ǫ. (3.6.3)

Remark 3.6.2. Let Ej = [τj , τj+1], j = 1, . . . , k. If we set f(τj , x) =fEj

(τj , x), j = 1, . . . , k, and set f(τk+1, x) = fEk(τk+1, x), then (3.6.3) still

holds. Moreover the function f satisfies (3.6.1), and the functions fi are ofclass C(r) on X for some values of t, then f is of class C(r) for the same valuesof t.

The first step in our proof is to establish the following lemma.

Lemma 3.6.3. Let I and X be as in the theorem and let f be a functionfrom I ×X to R

n having the same properties as the functions f1, . . . , fq of thetheorem. Then for every ǫ > 0 there exists a continuous function g, dependingon ǫ, from I × X to R

n such that for every x in X∫

I

|f(t, x)− g(t, x)|dt < ǫ. (3.6.4)

Proof. It follows from (3.6.1) that for x and x′ in X∫

I

|f(t, x)− f(t, x′)|dt ≤ |x− x′|∫

I

µ(t)dt.

Hence for arbitrary ǫ > 0, we have∫

I

|f(t, x)− f(t, x′)|dt < ǫ/2 (3.6.5)


whenever |x − x′| < ǫ/2∫Iµ(t)dt. Since X is compact, there exists a finite

open cover O1, . . . ,Ok of X such that if x and x′ are in the same Oi, then(3.6.5) holds.

Let x1, . . . , xk be a finite set of points such that xi ∈ Oi. For each i =1, . . . , k there exists a continuous function hi defined on I such that

∫

I

|f(t, xi)− hi(t)|dt < ǫ/2. (3.6.6)

Let γ1, . . . , γk be a partition of unity corresponding to the finite open coverO1, . . . ,Ok of X . That is, let γ1, . . . , γk be continuous real valued functions onX such that

(i) γi(x) ≥ 0 for all x ∈ X

(ii) γi(x) = 0 if x 6∈ Oi (3.6.7)

(iii)∑k

i=1 γi(x) = 1

For a proof of the existence of partitions of unity corresponding to finite opencovers of compact subsets of locally compact Hausdorff spaces, see Rudin [82,p. 40].

Define

g(t, x) =

k∑

i=1

γi(x)hi(t).

Then g is continuous on I × X . We now show that g satisfies (3.6.4) andtherefore is the desired function.

∫

I

|g(t, x)− f(t, x)|dt ≤∫

I

∣∣∣∣∣

k∑

i=1

γi(x)hi(t)−k∑

i=1

γi(x)f(t, xi)

∣∣∣∣∣ dt

+

∫

I

∣∣∣∣∣

k∑

i=1

γi(x)f(t, xi)−k∑

i=1

γi(x)f(t, x)

∣∣∣∣∣ dt

≤k∑

i=1

γi(x)

∫

I

|hi(t)− f(t, xi)|dt

+

k∑

i=1

γi(x)

∫

I

|f(t, xi)− f(t, x)|dt.

By virtue of (3.6.6) each of the integrals in the first sum on the right is lessthan ǫ/2. From this and from (3.6.7)(iii) it follows that the first sum on theright is less than ǫ/2. We now examine the i-th summand in the second sumon the right. If x 6∈ Oi then by (3.6.7)(ii), γi(x) = 0 and so the summandis zero. If x ∈ Oi, then by (3.6.5) the integral is less than ǫ/2 and thereforeby (3.6.7)(i) the summand is less than ǫγi(x)/2. Therefore, each summandin the second sum is less than ǫγi(x)/2. It now follows from (3.6.7)(iii) that

Relaxed Controls 69

the second sum is less than ǫ/2. Hence g satisfies (3.6.4) and the lemma isproved.

We now return to the proof of Theorem 3.6.1. Let ǫ > 0 be given and let

ǫ = ǫ/2(2 + q + |I|), (3.6.8)

where |I| denotes the length of I. Henceforth if A is a measurable set we shalluse |A| to denote the measure of A. From Lemma 3.6.3 we get that for eachi = 1, . . . , q there is a continuous function gi defined on I × X with range inR

n such that ∫

I

|fi(t, x)− gi(t, x)|dt < ǫ. (3.6.9)

Since each gi is continuous on I × X and I and X are compact, each gi isuniformly continuous on I × X . Therefore, there exists a δ > 0 such that forall i = 1, . . . , q if |t− t′| < δ then

|gi(t, x)− gi(t′, x)| < ǫ. (3.6.10)

Moreover, we may suppose that δ is such that if E is a measurable subset ofI with |E| < δ, then ∫

E

µ(t)dt < ǫ. (3.6.11)

Let Ik be a subdivision of I into a finite number of non-overlapping intervalswith |Ik| < δ for each interval Ik. Moreover, suppose that Ik = [tk, tk+1] andthat . . . < tk−1 < tk < tk+1 < tk+2 < . . . . For each Ik we can construct asubdivision of Ik into non-overlapping subintervals Ek1, . . . , Ekq such that

|Eki| =∫

Ik

pi(t)dt. (3.6.12)

This is possible since

q∑

i=1

|Eki| =q∑

i=1

∫

Ik

pi(t)dt =

∫

Ik

(q∑

i=1

pi(t)

)dt = |Ik|,

the last equality following from (3.6.2).Define

f(t, x) = fi(t, x) t ∈ E0ki, (3.6.13)

where E0ki denotes the interior of Eki. Thus, f is defined at all points of I

except the end points of the intervals Eki. At these points f can be definedas in Remark 3.6.2 or in any arbitrary manner. Let

λ(t, x) =

q∑

i=1

pi(t)fi(t, x)− f(t, x). (3.6.14)


The collection of intervals Eki where k ranges over the same index set as dothe intervals Ik and i ranges over the set 1, . . . , q, constitutes a subdivision ofI into a finite number of non-overlapping subintervals. This subdivision, rela-beled as Ej, is the subdivision whose existence is asserted in the theorem. Ifan interval Ej was originally the interval Eki, then the function fEj

assignedto Ej is fi. If we now compare the definition of λ in (3.6.14) with (3.6.3) andwe see that to prove the theorem we must show that for arbitrary t′ and t′′ inI and all x in X ∣∣∣∣∣

∫ t′′

t′λ(t, x)dt

∣∣∣∣∣ < ǫ. (3.6.15)

There is no loss of generality in assuming that t′ < t′′. The point t′ willbelong to some interval Iα of the subdivision Ik and the point t′′ will belongto some interval Iβ . If Iα 6= Iβ , let s1 denote the right-hand end point tα+1 ofIα and let s2 denote the left-hand end point tβ of Iβ . Then if J denotes theset of indices α+ 1, α+ 2, . . . , β − 1, we have

[s1, s2] ≡ [tα+1, tβ ] =⋃

j∈J

Ij .

See Figure 3.1.Hence we have

∣∣∣∣∣

∫ t′′

t′λdt

∣∣∣∣∣ ≤∣∣∣∣∫ s1

t′λdt

∣∣∣∣+∣∣∣∣∫ s2

s1

λdt

∣∣∣∣+∣∣∣∣∣

∫ t′′

s2

λdt

∣∣∣∣∣ ≡ A+B + C.

It follows from (3.6.14), (3.6.1), (3.6.2), (3.6.11), and the fact that t′ and s1are in an interval Iα with |Iα| < δ that:

A ≤∫ s1

t′

(q∑

i=1

|pifi|+ |f |)dt =

∫ s1

t′

q∑

i=1

pi|fi|dt+∫ s1

t′|f |dt

≤∫ s1

t′

(q∑

i=1

pi

)µdt+

∫ s1

t′µdt = 2

∫ s1

t′µdt < 2ǫ.

Note that if t′ and t′′ are in the same interval Iα, then the preceding estimateand (3.6.8) combine to give (3.6.15).

FIGURE 3.1

Relaxed Controls 71

An argument similar to the preceding one gives C < 2ǫ.We now estimate B. Recall that Ik = [tk, tk+1]. Then

B =

∣∣∣∣∫ s2

s1

λdt

∣∣∣∣ ≤∑

j∈J

∣∣∣∣∣

∫ tj+1

tj

λdt

∣∣∣∣∣ .

Let g(t, x) = gi(t, x) for t ∈ Eji, where i = 1, . . . , q and j ∈ J ; then we canestimate each summand on the right as follows

∣∣∣∣∣

∫ tj+1

tj

λdt

∣∣∣∣∣ ≤∣∣∣∣∣

∫ tj+1

tj

(q∑

i=1

pi(fi − gi)

)dt

∣∣∣∣∣

+

∣∣∣∣∣

∫ tj+1

tj

(q∑

i=1

pigi − g

)dt

∣∣∣∣∣+∣∣∣∣∣

∫ tj+1

tj

(g − f)dt

∣∣∣∣∣

≡ Aj +Bj + Cj .

HenceB ≤

∑

j∈J

(Aj +Bj + Cj). (3.6.16)

From pi ≥ 0 and (3.6.2) we get that

Aj ≤∫ tj+1

tj

(q∑

i=1

pi|fi − gi|)dt ≤

q∑

i=1

∫ tj+1

tj

|fi − gi|dt.

From the definitions of f and g we get that

Cj ≤∫ tj+1

tj

|g − f |dt ≤q∑

i=1

∫ tj+1

tj

|fi − gi|dt.

Therefore,

∑

j∈J

(Aj + Cj) ≤ 2

q∑

i=1

∫ s2

s1

|fi − gi|dt ≤ 2

q∑

i=1

∫

I

|fi − gi|dt < 2qǫ, (3.6.17)

where the last inequality follows from (3.6.9).We now consider Bj .

Bj =

∣∣∣∣∣

∫ tj+1

tj

(q∑

i=1

pigi − g

)dt

∣∣∣∣∣ =∣∣∣∣∣

q∑

i=1

∫ tj+1

tj

pigidt−q∑

i=1

∫

Eji

gidt

∣∣∣∣∣ .

In each set Eji select a point tji. Since Eji ⊂ Ij and |Ij | < δ it follows from(3.6.10) that for all t in Ij and all x in X and all i = 1, . . . , q

gi(t, x) = gi(tji, x) + ηi(t, x),


where |ηi(t, x)| < ǫ. Therefore, using (3.6.12), we get

Bj =∣∣∣

q∑

i=1

(∫ tj+1

tj

(pi(t)gi(tji, x) + pi(t)ηi(t, x))dt (3.6.18)

−∫

Eji

(gi(tji, x) + ηi(t, x))dt)∣∣∣

=∣∣∣

q∑

i=1

(gi(tji, x)|Eji| − gi(tji, x)|Eji|

+

∫ tj+1

tj

pi(t)ηi(t, x)dt −∫

Eji

ηi(t, x)dt)∣∣∣

<

q∑

i=1

(ǫ

∫ tj+1

tj

pidt+ ǫ|Eji|)

= 2ǫ|Ij |.

Hence ∑

j∈J

Bj < 2ǫ|s2 − s1| ≤ 2ǫ|I|.

Combining this with (3.6.17) gives B < 2ǫ(q + |I|). If we now combine thisestimate with the estimates on A and C and use (3.6.8) we get that

∣∣∣∣∣

∫ t′′

t′λdt

∣∣∣∣∣ < 2ǫ(2 + q + |I|) = ǫ,

which is (3.6.15), as required. This completes the proof of Theorem 3.6.1.

Remark 3.6.4. Recall that a family Ψ of functions ψ defined on a set X inR

n with range in Rm is said to be equicontinuous at a point x0 in X if for

each ǫ > 0 there exists a δ > 0 such that

|x− x0| < δ implies |ψ(x) − ψ(x0)| < ǫ (3.6.19)

for all ψ in Ψ. The family is equicontinuous on X if it is equicontinuous atall points of X . If X is compact, then each function is uniformly continuouson X . Thus, (3.6.19) holds for all x, x0 in X . Since Ψ is equicontinuous on X ,(3.6.19) holds for all x, x0 in X and all ψ in Ψ. Thus, the functions ψ in Ψare uniformly equicontinuous.

The next theorem states that (3.6.3) remains true if we replace the vectorsx in X by functions ψ from an equicontinuous family.

Theorem 3.6.5. Let f1, . . . , fq and p1, . . . , pq be as in Theorem 3.6.1. Let Ψbe a family of equicontinuous functions on I with range in X . Then for everyǫ > 0 there exists a subdivision of I into a finite number of disjoint intervalsEj and an assignment of one of the functions f1, . . . , fq to each interval Ej

Relaxed Controls 73

such that the following holds. If fEjdenotes the function assigned to Ej and

if f is a function that agrees with fEjon E0

j , the interior of Ej, that is,

f(t, x) = fEj(t, x) t ∈ E0

j ,

then for every t′ and t′′ in I and every function ψ in Ψ∣∣∣∣∣

∫ t′′

t′

(q∑

i=1

pi(t)fi(t, ψ(t)) − f(t, ψ(t))

)dt

∣∣∣∣∣ < ǫ. (3.6.20)

Proof. Let ǫ > 0 be given. Since the functions in Ψ are equicontinuous onI and I is compact, there exists a partition of I into a finite number ofnon-overlapping subintervals Ij = [tj, tj+1], j = 1, . . . , k such that · · · <tj−1 < tj < tj+1 < tj+2 < . . . and such that for all ψ in Ψ and all t in Ij ,j = 1, . . . , k.

|ψ(t)− ψ(tj)| < ǫ

(4

∫

I

µdt

)−1

≡ ǫ′. (3.6.21)

We now apply Theorem 3.6.1 to f1, . . . , fq and p1, . . . , pq with ǫ replacedby ǫ/2k. Then there exists a function f as described in Theorem 3.6.1 suchthat for all x in X and all t′, t′′ in I,

∣∣∣∣∣

∫ t′′

t′λ(t, x)dt

∣∣∣∣∣ < ǫ/2k, (3.6.22)

where λ is defined in (3.6.14). We must show that for all ψ in Ψ and t′, t′′ inI, ∣∣∣∣∣

∫ t′′

t′λ(t, ψ(t))dt

∣∣∣∣∣ < ǫ.

Define

λ(t) = λ(t, ψ(tj)) tj ≤ t < tj+1, j = 1, . . . , k

and let λ(tk+1) = λ(tk+1, ψ(tk)). Then∣∣∣∣∣

∫ t′′

t′λ(t, ψ(t))dt

∣∣∣∣∣ ≤∣∣∣∣∣

∫ t′′

t′(λ(t, ψ(t)) − λ(t))dt

∣∣∣∣∣ (3.6.23)

+

∣∣∣∣∣

∫ t′′

t′λ(t)dt

∣∣∣∣∣ ≡ A+B.

Let t′ < t′′, let t′ ∈ Iα = [tα, tα+1], and let t′′ ∈ Iβ = [tβ , tβ+1]. Let J nowdenote the index set α, α+ 1, α+ 2, . . . , β. Then

A ≤∫ t′′

t′

∣∣∣λ(t, ψ(t)) − λ(t)∣∣∣ ≤

∫ tβ+1

tα

|λ(t, ψ(t)) − λ(t)|dt


=∑

j∈J

∫ tj+1

tj

|λ(t, ψ(t)) − λ(t, ψ(tj))|dt

=∑

j∈J

∫ tj+1

tj

∣∣∣q∑

i=1

pi(t)fi(t, ψ(t))− fi(t, ψ(tj))

+ f(t, ψ(tj))− f(t, ψ(t))∣∣∣dt

≤∑

j∈J

∫ tj+1

tj

(q∑

i=1

pi(t)µ(t)|ψ(t) − ψ(tj)|+ µ(t)|ψ(t)− ψ(tj)|)dt,

where the last inequality follows from (3.6.1) and Remark 3.6.2. From (3.6.2)and from (3.6.21) we see that the preceding sum in turn is less than

∑

j∈J

ǫ′∫ tj+1

tj

2µdt ≤ 2ǫ′∫

I

µdt = ǫ/2.

We have thus shown that A < ǫ/2.To estimate B we write

B ≤∣∣∣∣∫ tα+1

t′λ(t, ψ(tα))dt

∣∣∣∣+β−1∑

j=α+1

∣∣∣∣∣

∫ tj+1

tj

λ(t, ψ(tj))dt

∣∣∣∣∣

+

∣∣∣∣∣

∫ t′′

tβ

λ(t, ψ(tβ))dt

∣∣∣∣∣ .

By (3.6.22) each summand on the right is < ǫ/2k. Since there are at most ksummands (the number of intervals Ij), it follows that B < ǫ/2. If we combinethis estimate with the estimate for A and substitute into (3.6.23), then we getthe desired result.

The proof of our next theorem requires an inequality that is very useful inthe study of differential equations, and is known as Gronwall’s Inequality.

Lemma 3.6.6. Let ρ and µ be nonnegative real valued functions continuouson [0,∞) such that

ρ(t) ≤ α+

∫ t

t0

µ(s)ρ(s)ds α ≥ 0 (3.6.24)

for all t0, t in [0,∞). Then

ρ(t) ≤ α exp

(∫ t

t0

µ(s)ds

). (3.6.25)

Proof. Suppose that α > 0. Then the right-hand side of (3.6.24) is strictlypositive and we get that

ρ(τ)µ(τ)

[α+

∫ τ

t0

µ(s)ρ(s)ds

]−1

≤ µ(τ).

Relaxed Controls 75

Integrating both sides of this inequality from t0 to t and using (3.6.24) gives

log ρ(t) ≤ log

[α+

∫ t

t0

µρds

]≤ logα+

∫ t

t0

µds.

From this we get (3.6.25).

If α = 0, then (3.6.24) holds for all α1 > 0. Hence (3.6.25) holds for allα1 > 0. Letting α1 → 0 now yields ρ(t) ≡ 0. Hence (3.6.25) is trivially true.

Remark 3.6.7. The proof shows that if α > 0 and strict inequality holds in(3.6.24), then strict inequality holds in (3.6.25).

Theorem 3.6.8. Let I be a compact interval in R1, let X be a compact

interval in Rn, and let R = I×X . Let G = R×U , where U is a region of Rm,

and let f be a continuous mapping from G to Rn. Let Ω be a mapping from R

to subsets of U that is independent of x; that is, Ω(t, x′) = Ω(t, x) ≡ Ω(t) forall x and x′ in X . Let there exist an integrable function µ defined on I suchthat for all (t, x, z) in G

|f(t, x, z)| ≤ µ(t)

and for all (t, x, z) and (t, x′, z) in G

|f(t, x, z)− f(t, x′, z)| ≤ µ(t)|x− x′|. (3.6.26)

Let I1 = [t0, t1] be a compact interval contained in the interior of I andX1 be a compact interval in the interior of X . Let R1 = I1 × X1. Let v =(u1, . . . , un+2, p

1, . . . , pn+2) be a relaxed control on I1 for the relaxed system

dx

dt=

n+2∑

i=1

pi(t)f(t, x, ui(t))

corresponding to the control system

dx

dt= f(t, x, u(t)).

Let both systems have initial point (t0, x0) in I1 × X1. Let ψ be a relaxedtrajectory corresponding to v on I1 and let ψ(t) ∈ X1 for all t in [t0, t1]. Thenthere exists an ǫ0 > 0 such that for each ǫ satisfying 0 < ǫ < ǫ0 there is acontrol uǫ defined on I1 with the following properties.

(i) The control uǫ(t) ∈ Ω(t) for a.e. t in I1,

(ii) the trajectory φǫ corresponding to uǫ lies in I1 ×X , and

(iii) |φǫ(t)− ψ(t)| < ǫ, for all t in I1.


Remark 3.6.9. Theorem 3.6.8 states that under appropriate hypotheses theordinary trajectories of a system are dense in the set of relaxed trajectoriesin the uniform topology on [t0, t1]. Thus, for any relaxed trajectory ψ on[t0, t1] there is a sequence of controls uk and a sequence of correspondingtrajectories φk such that uk(t) ∈ Ω(t) a.e. and φk → ψ uniformly on [t0, t1].We caution the reader that with reference to a specific control problem, if ψis an admissible relaxed trajectory the pairs (φk, uk) need not be admissiblefor the original problem in that either t → f0(t, φk(t), uk(t)) may not beintegrable or the end points of the φk may not satisfy the end condition. Recallthe distinction between a control (Definition 2.3.1) and an admissible control(Definition 2.3.2). We will return to this point in Chapter 4. Under a “localcontrollability” condition, the approximating relaxed trajectories φk can beslightly modified to satisfy the required end condition. See Theorem 4.4.6.

Note that no assumption is made concerning the nature of the constraintsets Ω(t).

Proof. Let ǫ0 denote the distance between ∂X and ∂X1, where for any set Athe symbol ∂A denotes the boundary of A. Then ǫ0 > 0. Let

K =

∫ t1

t0

µdt (3.6.27)

and let ǫ be any number satisfying 0 < ǫ < ǫ0. For (t, x) in I1 × X andi = 1, . . . , n+ 2 let

fi(t, x) = f(t, x, ui(t)). (3.6.28)

The hypotheses of the present theorem imply that the functions fi satisfythe hypotheses of Theorems 3.6.1 and 3.6.5. In particular note that since f iscontinuous on R and each ui is measurable, the functions fi are measurableon I1 for each fixed x in X .

Let ǫ′ = ǫe−K . We next apply Theorem 3.6.5 to the functions f1, . . . , fn+2

just defined, the functions p1, . . . , pn+2 in the relaxed control, the family Ψconsisting of one element — the relaxed trajectory ψ, and the value of epsilonequal to ǫ′. We obtain the existence of a function f such that for x ∈ X1 andt ∈ I1

f(t, x) = fEj(t, x) t ∈ E0

j (3.6.29)

and ∣∣∣∣∣

∫ t′′

t′

(n+2∑

i=1

pi(t)fi(t, ψ(t))− f(t, ψ(t))

)dt

∣∣∣∣∣ < ǫ′ (3.6.30)

for arbitrary t′ and t′′ in I1.It follows from the definition of fi and from (3.6.29) that

f(t, x) = fEj(t, x) = f(t, x, uEj

(t)) t ∈ E0j . (3.6.31)

Defineuǫ(t) = uEj

(t) if t ∈ E0j .

Relaxed Controls 77

Then since uEjis one of the u1, . . . , un+2 and each ui satisfies ui(t) ∈ Ω(t)

a.e. on I1 it follows that uǫ(t) ∈ Ω(t) on I1 a.e. From the definition of uǫ and(3.6.31) we get

f(t, x) = f(t, x, uǫ(t)).

Consider the system

dx

dt= f(t, x, uǫ(t)) = f(t, x) (3.6.32)

with initial point (t0, x0). Since f satisfies (3.6.26) it follows that through eachpoint (t2, x2) in the interior of I×X , there passes a unique solution of (3.6.32),provided we extend uǫ to be defined and measurable on I. In particular thereexists a unique solution φǫ of (3.6.32) with initial point (t0, x0). This solutionwill be defined on some open interval containing t0 in its interior. Let Imax =(a, b) denote the maximal interval on which φǫ is defined. If [a, b] ⊂ I1, thenlim supt→b φǫ(t) must be a boundary point of X ; otherwise we could extend thesolution φǫ to an interval containing Imax in its interior. This would contradictthe maximality of Imax. We shall show that for all t in Imax, the inequality|φǫ(t)− ψ(t)| < ǫ holds. Since ψ(t) ∈ X1 for all t in [t0, t1] and since ǫ < ǫ0 =dist (∂X1, ∂X ) it will follow that [a, b] ⊃ I1 and φǫ is defined in all of I1.Moreover, we shall have |φǫ(t)− ψ(t)| < ǫ for all of I1.

Since ψ is defined on all of I1 and ψ(t0) = φǫ(t0) = x0, we have for all tin [t0, b]

|ψ(t)− φǫ(t)| =∣∣∣∣∫ t

t0

(ψ′(s)− φ′ǫ(s))ds

∣∣∣∣

=

∣∣∣∣∣

∫ t

t0

(n+2∑

i=1

pi(s)fi(s, ψ(s)) − f(s, φǫ(s))

)ds

∣∣∣∣∣

≤∣∣∣∣∣

∫ t

t0

(n+2∑

i=1

pi(s)fi(s, ψ(s)) − f(s, ψ(s))

)ds

∣∣∣∣∣

+

∣∣∣∣∫ t

t0

(f(s, ψ(s)) − f(s, φǫ(s)))ds

∣∣∣∣

< ǫ′ +

∫ t

t0

|f(s, ψ(s))− f(s, φǫ(s))|ds,

where the last inequality follows from (3.6.30). It now follows from (3.6.31)and (3.6.26) that

∫ t

t0

|f(s, ψ(s))− f(s, φǫ(s))|ds ≤∫ t

t0

µ(s)|ψ(s) − φǫ(s)|ds.

Combining this with the preceding inequality gives

|ψ(t)− φǫ(t)| < ǫ′ +

∫ t

t0

µ(s)|ψ(s) − φǫ(s)|ds.


From Lemma 3.6.6, from (3.6.27), and the definition of ǫ′ we now concludethat

|ψ(t)− φǫ(t)| < ǫ′ exp

(∫ t

t0

µds

)≤ ǫ′eK = ǫ,

and the theorem is proved.

Remark 3.6.10. From the proof of Theorem 3.6.8 we see why we must as-sume that the functions fi of Theorem 3.6.1 are measurable in t and continuousin x, rather than continuous in (t, x). Since controls u are only assumed to bemeasurable, we can only guarantee that the functions fi defined in (3.6.28)will be measurable in t, no matter how regular we assume the behavior of fto be.

The reason for calling Theorem 3.6.1 the “Chattering Lemma” can now begiven. In most applications the functions f1, . . . , fq are obtained as in Theo-rem 3.6.8. That is, we have a system with state equations dx/dt = f(t, x, u(t)),we choose q controls u1, . . . , uq, and define functions f1, . . . , fq by means ofEq. (3.6.28). The function f of Theorem 3.6.1 is obtained in the same fash-

ion as the function f of the present theorem. That is, the basic interval Iis divided up into a large number of small intervals and on each subintervalwe choose one of the controls u1, . . . , uq to build the control uǫ. In a physicalsystem, the control uǫ corresponds to a rapid switching back and forth amongthe various controls u1, . . . , uq. In the engineering vernacular, the system issaid to “chatter.” The control uǫ is therefore sometimes called a chatteringcontrol. In Example 3.1.1, the controls ur are chattering controls.

From the proof of Theorem 3.6.8 we learn more than just the fact that arelaxed trajectory can be approximated as close as we please by an ordinarytrajectory. We learn that the approximation can be effected through the use ofa chattering control built from the controls used to define the relaxed controlin question.

Remark 3.6.11. The theorem remains valid if we take X = Rn and X1 any

compact set.

Chapter 4

Existence Theorems; Compact

Constraints

4.1 Introduction

Examples 3.1.1 and 3.1.2 in Section 3.1 of Chapter 3 showed that if the setof admissible directions is not convex, then existence may fail, even thoughthe dynamics and the constraints exhibit regular behavior. Relaxed controlswere introduced to provide convexity of the set of admissible directions. If theconstraint sets are compact and exhibit a certain regular behavior, then therelaxed controls were shown to have a compactness property that will be usedto prove the existence theorems in Section 4.3 and the necessary conditions oflater chapters. In the next section we shall present examples of non-existencethat illustrate the need for conditions on the behavior of the constraint sets,terminal sets, and dynamics. In Section 4.4 we introduce a convexity conditionimplying that an optimal relaxed trajectory is an ordinary trajectory, and thusis a solution of the ordinary problem. In Section 4.5 we give examples fromapplied areas that are covered by the existence theorems of Section 4.4.

In Section 4.6 we present an existence theorem for problems with inertialcontrollers. Section 4.7 is devoted to problems with system equations linear inthe state. For such problems we obtain the deep result that relaxed attainablesets and ordinary attainable sets are equal. One consequence of this is thatif the integrand in the payoff function is also linear in the state, then anordinary optimal solution exists without the requirement that the directionset be convex. Another consequence is the “bang-bang” principle, which holdsfor problems that are also linear in the control and have compact, convexconstraint sets. The “bang-bang” principle states that if the system can reacha point x0 in time t1, then this point can also be reached in time t1 using acontrol that assumes values at the extreme points of the constraint set.

79


4.2 Non-Existence and Non-Uniqueness of OptimalControls

Given a system of equations together with end conditions and controlconstraints there is no guarantee that admissible pairs exist. The followingsimple example emphasizes this point.

Example 4.2.1. Let x be one-dimensional. Let the state equation be

dx

dt= u(t). (4.2.1)

Let B consist of the single point (t0, x0, t1, x1) = (0, 0, 1, 2) and let

Ω(t, x) = z : |z| ≤ 1.

Thus, an admissible control satisfies the inequality |u(t)| ≤ 1 for almost everyt in [0, 1] and transfers the system from x0 = 0 at time t0 = 0 to the statex1 = 2 at time t1 = 1. From (4.2.1) it is clear that |φ(1)| ≤ 1. Thus, the setof admissible pairs is empty.

If the class of admissible controls is not void, it does not necessarily followthat an optimal control exists. The following examples illustrate this point.

Example 4.2.2. Let x be one-dimensional. Let the state equation be dx/dt =u(t). Let B consist of the single point (t0, x0, t1, x1) = (0, 1, 1, 0) and letΩ(t, x) = R. Let

J(φ, u) =

∫ 1

0

t2u2(t)dt. (4.2.2)

The set of controls is the set of functions in L1([0, 1]). To each control u therecorresponds a unique trajectory φ satisfying φ(0) = 1, namely the trajectorygiven by

φ(t) = 1 +

∫ t

0

u(s)ds.

For each 0 < ǫ < 1 define a control uǫ as follows:

uǫ(t) =

0, ǫ ≤ t ≤ 1

−ǫ−1, 0 ≤ t < ǫ.

Let φǫ denote the unique trajectory corresponding to uǫ and satisfying φǫ(0) =1. Clearly, (φǫ, uǫ) is an admissible pair. The class A of admissible pairs is notvoid. Moreover,

J(φǫ, uǫ) =

∫ ǫ

0

t2ǫ−2dt =1

3ǫ.

Existence Theorems; Compact Constraints 81

Since J(φ, u) ≥ 0 for all admissible pairs (φ, u), it follows that 0 = infJ(φ, u) |(φ, u) ∈ A. From (4.2.2) it is clear that J(φ, u) = 0 if and only if u(t) = 0 a.e.on [0, 1]. However, u∗ = 0 is not admissible because the corresponding trajec-tory φ∗ is identically one and thus does not satisfy the terminal constraint.

In this example, the set of admissible directions is convex, but the con-straint set, while constant, is not compact.

Example 4.2.3. Let everything be as in Example 4.2.2 except that the con-trol set is given as follows:

Ω(t, x) = z : |z| ≤ 1/t if 0 < t ≤ 1

Ω(0, x) = R.

The arguments of Example 4.2.2 are still valid and an optimal control fails toexist.

The set of admissible directions is convex. The constraint sets depend ont alone and fail to be compact at the single point t = 0.

Example 4.2.4. Let everything be as in Example 4.2.2 except that the con-trol set is given as follows:

Ω(t, x) = z : |z| ≤ 1/t if 0 < t ≤ 1

Ω(0, x) = 0.

If we now define

uǫ(t) =

0, ǫ ≤ t ≤ 1

−ǫ−1, 0 < t < ǫ

0, t = 0

and proceed as in Example 4.2.2, we again find that an optimal control failsto exist.

The set of admissible directions is convex, the constraint sets depend on talone, and for each t the set Ω(t) is compact. The mapping Ω fails to be u.s.c.iat the single point t = 0.

In the next example, the terminal set B is not compact and the timeinterval is unbounded.

Example 4.2.5. The state equation is dx/dt = u(t), and the control set isΩ(t, x) = z : |z| ≤ 1. Let B = (t0, x0, t1, x1) : t0 = 0, x0 = 0, x1 = 1/t1,t1 > 0. The functional J(φ, u) is given by J(φ, u) = φ(t1). Let 0 < t bearbitrary. Let u(t) = 0 for 0 ≤ t ≤ t and then let u(t) = 1 until time t1 definedby φ(t1) = 1/t1. Clearly 1/t1 ≤ 1/t. Since t can be taken arbitrarily large itfollows infJ(φ, u) : (φ, u) ∈ A = 0. However, for no admissible pair (φ∗, u∗)is J(φ∗, u∗) = 0.


In the next example an optimal pair fails to exist because the trajectoriesare not confined to a compact set in (t, x) space. The constraint set is constantand compact and the sets of admissible directions are convex. The dynamics,however, grow as x2, suggesting that some restriction on the rate of growthof the dynamics is needed.

Example 4.2.6. Let x be a scalar and let the state equation be

dx

dt= 2x2(1− t)− 1 + u(t). (4.2.3)

Let Ω(t, x) = z : |z| ≤ 1. Let the end conditions be given by T0 = (0, x0) :0 ≤ x0 ≤ 1 and T1 = (a, x1) : 0 ≤ x1 ≤ (1 − a)−2, with a > 1. LetJ(φ, u) = −φ(a). Hence, if u is an admissible control and φ is a correspondingtrajectory it is required to maximize φ(a) over all admissible pairs (φ, u).

The set of admissible controls for this problem is a subset of the measurablefunctions u on [0, a] such that |u(t)| ≤ 1 a.e. Since dx/dt is maximized whenu(t) = 1, we substitute u(t) = 1 into the right-hand side of (4.2.3) and we get

dx

dt= 2x2(1 − t). (4.2.4)

The solution of this differential equation satisfying the initial condition φ(0) =x0, x0 6= 0 is

φ(t) = [(1− t)2 + c]−1, (4.2.5)

where c = (1− x0)/x0. The solution of (4.2.4) satisfying the initial conditionφ(0) = 0 is φ(t) ≡ 0. The field of trajectories corresponding to u = 1 isindicated in Figure 4.1. Values of c ≥ 0 correspond to initial points x0 inthe interval 0 < x0 ≤ 1. Note that if x0 = 1, then c = 0 and u = 1 is notadmissible.

Let F denote the field of trajectories corresponding to u(t) = 1 and initialconditions 0 ≤ x0 < 1. Note that F does not include the trajectory startingfrom x0 = 1 at t0 = 0. It is clear from (4.2.3) and properties of the field oftrajectories F that if an optimal pair (φ∗, u∗) exists and if φ∗(0) = x0 < 1,then we must have u∗(t) = 1 a.e. It then follows from (4.2.5) (see Fig. 4.1)that u∗(t) = 1 and 0 ≤ x0 < 1 cannot be optimal. For if we take a new initialstate x′0, where x0 < x′0 < 1, then the solution φ of (4.2.4) corresponding tox′0 will give φ(a) > φ∗(a). On the other hand, an optimal trajectory cannothave x0 = 1 as initial point. For if x0 = 1, then u(t) ≡ 1 is not admissible.Moreover, once we take u(t) < 1 on a set E of positive measure the trajectorygoes into the interior of F. It is then possible to modify the control so as toincrease the value φ(a). We leave the rigorous formulation of this argumentto the reader.

The next example shows that an optimal pair need not be unique.

Example 4.2.7. Let x be one-dimensional. Let the state equation be dx/dt =


FIGURE 4.1

u. Let B consist of the single point (t0, x0, t1, x1) = (0, 0, 1, 0). Let Ω(t, x) =z : |z| ≤ 1, and let

J(φ, u) =

∫ 1

0

(1− u2(t))dt.

Clearly, J(φ, u) ≥ 0. Define a control u∗1 as follows: u∗1(t) = 1 if 0 ≤ t < 1/2and u∗1(t) = −1 if 1/2 ≤ t ≤ 1. Then, u∗1 is admissible and J(φ∗1, u

∗1) = 0,

where φ∗1 is the unique trajectory corresponding to u∗1. Hence u∗1 is optimal.

We now show that there are infinitely many optimal controls. For each integern = 1, 2, 3, . . . define a control u∗n as follows:

u∗n(t) = (−1)k ifk

2n≤ t ≤ k + 1

2n, k = 0, 1, 2, . . . , 2n − 1.

Then, J(φ∗n, u∗n) = 0, n = 1, 2, 3, . . . , where φ∗n is the unique trajectory corre-

sponding to u∗n. Thus, we have nonuniqueness.

4.3 Existence of Relaxed Optimal Controls

In Chapter 3, Definition 3.2.5, we defined a relaxed trajectory correspond-ing to a relaxed control µ to be an absolutely continuous solution of thedifferential equation

x′ =

∫

Ω(t)

f(t, x, z)dµt.


Henceforth we shall simplify our notation by writing the preceding integral asfollows:

f(t, x, µt) ≡∫

Ω(t)

f(t, x, z)dµt. (4.3.1)

Thus, a relaxed trajectory corresponding to a control µ will be a solution ofthe differential equation

x′ = f(t, x, µt). (4.3.2)

We shall use Greek letters to denote relaxed controls. The subscript t denotesthe probability measure µt on Ω(t), that is, the value of µ at t. The subscriptnotation is used to emphasize that f(t, x, µt) is defined by (4.3.1).

In this section we shall be concerned with functions f = (f1, . . . , fn),where each of the f i is real valued and defined on a set I ×X ×U , where I isa real compact interval, X is an open interval in R

n, and U is an open intervalin R

m. These functions are assumed to satisfy the following:

Assumption 4.3.1. (i) Each f i is measurable on I for each fixed (x, z) inX × U and is continuous on X × U for each fixed t in I.

(ii) For each compact set K in I × X × U there exists a function MK inL2[I] such that

|f(t, x, z)| ≤MK(t)

|f(t, x, z)− f(t, x′, z)| ≤MK(t)|x− x′|.for all (t, x, z) and (t, x′, z) in I × X × U .

Remark 4.3.2. If f is continuous on I × X × U , it follows that for eachcompact set K in I × X × U , there exists a constant AK > 0 such that

|f(t, x, z)| ≤ AK

for all (t, x, z) in K.

The weak compactness of relaxed controls will enable us to prove the ex-istence of a relaxed optimal control by following the pattern of the proof ofthe fact that a real valued continuous function defined on a compact set ina metric space attains a minimum. We begin with a result that will also beused in other contexts.

Lemma 4.3.3. Let f either (i) be continuous on I×X×U or (ii) be a functionas in Assumption 4.3.1. Let ϕn be a sequence of continuous functions definedon I and converging uniformly to a continuous function ϕ on I. Let µn bea sequence of relaxed controls with measures µnt all concentrated on a fixedcompact set Z and converging weakly to a relaxed control µ on I. Then forany function g in L2[I]

limn→∞

∫

I

g(t)f(t, ϕn(t), µnt)dt =

∫

I

g(t)f(t, ϕ(t), µt)dt.


Moreover, for any measurable set ∆ ⊆ I

limn→∞

∫

∆

g(t)f(t, ϕn(t), µnt)dt =

∫

∆

g(t)f(t, ϕ(t), µt)dt.

Proof. We first suppose that f is continuous on I×X ×U . Since the sequenceϕn converges uniformly to ϕ, all of the points (t, ϕn(t), z) and (t, ϕ(t), z),where t ∈ I and z ∈ Z, lie in a compact set K ⊆ I ×X × U . Hence since f isuniformly continuous on K, since ϕn converges uniformly to ϕ, and since eachµnt and µt are probability measures, there exists a constant AK such that

|f(t, ϕn(t), µnt)| ≤ AK and |f(t, ϕ(t), µt)| ≤ AK (4.3.3)

for all n and all t in I. Let

δn =

∫

I

g(t)[f(t, ϕn(t), µnt)− f(t, ϕ(t), µnt)] dt. (4.3.4)

Then∫

I

g(t)f(t, ϕn(t), µnt)dt−∫

I

g(t)f(t, ϕ(t), µt)dt (4.3.5)

= δn +

∫

I

g(t)[f(t, ϕ(t), µnt)− f(t, ϕ(t), µt)] dt

Since f is uniformly continuous onK and each µnt is a probability measureon Z, it follows that for each ε > 0 there exists a positive integer n(ε) suchthat for n > n(ε)

|δn| ≤ ε‖g‖,where ‖ ‖ denotes the L-norm. Thus, δn → 0 as n→ ∞. Also, for each ε > 0there is a continuous function hε on I such that ‖hε − g‖ < ε. The integralon the right in (4.3.5) is equal to

∫

I

[g(t)− hε(t)][f(t, ϕ(t), µnt)− f(t, ϕ(t), µt)]dt (4.3.6)

+

∫

I

hε(t)[f(t, ϕ(t), µnt)− f(t, ϕ(t), µt)]dt

The second integral in (4.3.6) tends to zero as n → ∞ because µnt convergesweakly to µt. It follows from (4.3.3) that the absolute value first integral in(4.3.6) does not exceed

2AK

∫

I

|g(t)− hε(t)|dt ≤ 2AK‖g − hε‖ < ε2AK .

The first conclusion now follows if (i) holds. We obtain the second conclusionby replacing g by gχ∆, where χ∆ is the characteristic function of ∆.


We now suppose that f satisfies Assumption 4.3.1. All of the points(t, φn(t), z) and (t, φ(t), z), where t ∈ I and z ∈ Z, lie in a compact setK ⊆ I × X × U . Hence in place of (4.3.3) we have

|f(t, x, z)| ≤MK(t) (4.3.7)

|(f(t, x, z)− f(t, x′, z)| ≤MK(t)|x− x′|for all (t, x, z) and (t, x′, z) in K. Let δn be as in (4.3.4). Then (4.3.5) holds.

Since µnt is a probability measure, we get from (4.3.7) that for each ε > 0there exists a positive integer n(ε) such that for n > n(ε)

|δn| ≤∫

I

|g(t)|MK(t)|φn(t)− φ(t)|dt ≤ ε‖g‖‖MK‖,

where ‖ ‖ denotes the L2 norm. Thus, again, δn → 0.Let In denote the integral on the right in (4.3.5). Let

h(t, z) = g(t)f(t, φ(t), z).

The function h is defined on an interval T × U , where T = I, a compactinterval, and U is a compact interval in R

m with Z ⊂ U ⊂ U . Then

In =

∫

I

∫

Z

h(t, z)dµnt −∫

Z

h(t, z)dµt

dt.

The function h is measurable on I for fixed z in U and is continuous on Ufor almost all t in I. By Lemma 3.4.4, for each ε > 0 there exists a closed setF ⊂ I with meas(I −F) < ε and a continuous function H on I ×U such thatH(t, z) = h(t, z) for all t in F and all z in U . Hence

In =

∫

I

[H(t, µn)−H(t, µ)]dt−∫

I−F

∫

Z

H(t, z)dµnt −∫

Z

H(t, z)dµt

dt

+

∫

I−F

∫

Z

h(t, z)dµnt −∫

Z

h(t, z)dµt

.

Since µn converges weakly to µ, the first integral on the right tends to zero asn → ∞. The function H is continuous on the compact set I × Z and µ andµnt are probability measures, so the second term on the right is bounded inabsolute value, independent of n, by A meas (I−F), for some constant A > 0.Since |h(t, z)| ≤ |g(t)|MK(t), the absolute value of the third integral on theright can be made arbitrarily small, independent of n, by taking ε sufficientlysmall. Hence for arbitrary η > 0, lim sup |IN | < η, and so lim In = 0.

The first conclusion now follows if (ii) holds. To obtain the second conclu-sion, replace g by χ∆g.

Remark 4.3.4. The definition of a sequence of relaxed controls µn converg-ing weakly to a relaxed control requires that for all t in I, and g in C(It ×Z),where It = [0, t] that

limn→∞

∫ t

0

∫

Z

g(s, z)dµsds =

∫ t

0

∫

Z

g(s, z)dµsds.


The proof of Lemma 4.3.3, however, only requires that the preceding holds att = 1. We shall make use of this observation in the sequel.

Theorem 4.3.5. Let f = (f0, f1, . . . , fn) be defined on I × X × U , whereI is a compact interval in R

1, X is an open interval in Rn, and U is an

open interval in Rm. Let f either satisfy Assumption 4.3.1 or be continuous

on I × X × U . Let B be a closed set of points (t0, x0, t1, x1) in Rn+2 with

t0 < t1 and with both (t0, x0) and (t1, x1) in R = I ×X . Let g be a real valuedlower semi-continuous function defined on B. Let Ω be a mapping from I tocompact sets Ω(t) contained in U that is u.s.c.i. on I. Let the set of relaxedadmissible pairs be non-empty and be such that all admissible trajectories aredefined on all of I and such that the graphs of these trajectories are containedin a compact subset R0 of R = I ×X . Then there exists an admissible relaxedpair (ψ∗, µ∗) that minimizes

J(ψ, µ) = g(t0, ψ(t0), t1, ψ(t1)) +

∫ t1

t0

f0(t, ψ(t), µt)dt (4.3.8)

over all relaxed admissible pairs.

Remark 4.3.6. In Lemma 4.3.14, which immediately follows the proof of thistheorem, we give a sufficient condition for trajectories to be defined on all ofI and to lie in a compact set. This condition is not necessary.

Remark 4.3.7. Henceforth, to simplify notation we define

e(ψ) ≡ (t0, ψ(t0), t1, ψ(t1))

and call e(ψ) the end point of ψ.

Remark 4.3.8. The assumptions that the graphs of admissible trajectorieslie in a compact set and that B is closed imply that the end points e(ψ) ofadmissible trajectories lie in a compact subset of B. Hence, we may assume Bto be compact to start with.

Proof of Theorem. By virtue of Remark 4.3.8 we take B to be compact. SinceB is compact and g is lower semi-continuous on B, there exists a constant Bsuch that

g(e(ψ)) ≥ B (4.3.9)

for all admissible ψ. Since Ω is u.s.c.i, it follows from Lemma 3.3.11 that allthe sets Ω(t), where t ∈ I, are contained in a compact set Z. By hypothesesall points (t, ψ(t)) where t ∈ I and ψ is an admissible relaxed trajectory liein a compact set R0 in I ×X . It then follows that there exists a nonnegativefunction M in L2[I] such that

|f(t, ψ(t), z)| ≤M(t) (4.3.10)

for all t ∈ I, ψ admissible and z ∈ Ω(t). If f is continuous, then M is aconstant.


Letm = infJ(ψ, µ) : (ψ, µ) an admissible relaxed pair,

where J(ψ, µ) is defined in (4.3.8). From (4.3.9) and (4.3.10) we get that m isfinite. Let (ψn, µn) be a sequence of admissible pairs such that

limn→∞

J(ψn, µn) = m. (4.3.11)

Each admissible pair (ψn, µn) is defined on I and has a restriction toIn = [t0n, t1n] such that e(ψn) = (t0n, ψn(t0n), t1n, ψn(t1n)) is in B. Since thepoints e(ψn) all lie in the compact set B, there is a subsequence of ψn, thatwe relabel as ψn, and a point (t0, x0, t1, x1) in B such that the e(ψn) →(t0, x0, t1, x1). In particular, t0n → t0 and t1n → t1.

Fromψ′n(t) = f(t, ψn(t), µnt) a.e. in I,

(4.3.10) and the fact that µnt is a probability measure on Ω(t) we get that forall n and a.e. t in I,

|ψ′n(t)| = |f(t, ψn(t), µnt)| ≤M(t) (4.3.12)

where M is as in (4.3.10). From this and the relation

ψn(t)− ψn(t′) =

∫ t

t′ψ′n(s)ds

it follows that the functions ψn are equi-continuous on the compact intervalI. Since all trajectories lie in a compact set in R, the functions ψn areuniformly bounded. Hence, by Ascoli’s theorem, there is a subsequence of theψn, that we relabel as ψn, and a continuous function ψ∗ on I such that

limn→∞

ψn(t) = ψ∗(t)

uniformly on I. By Theorem 3.3.12 there exists a subsequence of the relaxedcontrols corresponding to ψn that converges weakly to a relaxed control µ∗

on I such that µt is concentrated on Ω(t). Let µn denote the subsequence.Corresponding to µn there is a subsequence of the relaxed trajectories thatwe relabel as ψn. In summary, we have a subsequence of admissible pairs(ψn, µn) such that ψn converges uniformly on I to a continuous function ψ∗

and µn converges weakly to a relaxed control µ∗ such that µ∗t is concentrated

on Ω(t).From

|ψn(tin)− ψ∗(ti)| ≤ |ψn(tin)− ψ∗(tin)|+ |ψ∗(tin)− ψ∗(ti)| i = 0, 1,

from tin → ti, i = 0, 1, from the uniform convergence of ψn to ψ∗, and fromthe continuity of ψ∗ we get that

xi = limn→∞

ψ(tin) = ψ∗(ti) i = 0, 1. (4.3.13)


For each n, the admissible pair (ψn, µn) in the subsequence satisfies

ψn(t) = ψn(t0n) +

∫ t

t0n

f(s, ψn(s), µns)ds.

If we let n→ ∞ and use (4.3.12), (4.3.13), and Lemma 4.3.3, we get that

ψ∗(t) = ψ∗(t0) +

∫ t

t0

f(s, ψ∗(s), µ∗s)ds (4.3.14)

for t ∈ I. A similar argument gives

limn→∞

∫ t1n

t0n

f0(s, ψn(s), µn,s)ds =

∫ t1

t0

f0(s, ψ∗(s), µ∗s)ds. (4.3.15)

From (t0, x0, t1, x1) ∈ B, (4.3.13), (4.3.14), and the fact that µ∗t is concentrated

on Ω(t) we get that (ψ∗, µ∗) is an admissible pair.Since g is lower semicontinuous

lim infn→∞

g(e(ψn)) ≥ g(e(ψ∗)).

From this and (4.3.11) and (4.3.15) we get that

m = limn→∞

J(ψn, µn) ≥ J(ψ∗, µ∗) ≥ m.

Hence J(ψ∗, µ∗) = m, and the theorem is proved.

Remark 4.3.9. A sequence (ψn, µn) as in (4.3.11) is called a minimizingsequence. From the proof it is clear that we only need to assume that thereexists a minimizing sequence that lies in a compact set R0.

Remark 4.3.10. Theorem 4.3.5 holds if we replace the function g in (4.3.8)by a functional G, where G is a lower semicontinuous functional defined onC[t0, t1], the space of continuous functions on [t0, t1] with the uniform topol-ogy. Clearly the function g is a special case of G. We leave the minor modifi-cations in the proof in the case of G to the reader.

Remark 4.3.11. In some problems the admissible trajectories ψ are requiredto satisfy the additional constraint ψ(t) ∈ C(t) for each t, where each C(t) isa closed set. Since an optimal trajectory is the uniform limit of a sequence ofadmissible trajectories, it follows that an optimal trajectory will also satisfythis constraint.

Remark 4.3.12. In some problems it is possible to show that there is acompact set R0 such that those trajectories that do not lie in R0 give largervalues to the cost J than do those that lie in R0. In that event one can ignorethe trajectories that do not lie in R0. One simply redefines R to be R0 andredefines the set of admissible pairs to be those pairs whose trajectories lie inR0.


The proof of Theorem 4.3.5 shows that the following statement is true.

Corollary 4.3.13. Let the hypotheses of Theorem 4.3.5 hold with the as-sumption that all trajectories lie in a compact set R0 ⊂ R replaced by theassumption that there exists a minimizing sequence all of whose trajectorieslie in a compact set R0 ⊂ R. Then there exists a relaxed optimal pair (ψ∗, µ∗).

We next give a sufficient condition for the graphs of ordinary and relaxedtrajectories to lie in a compact set. Note that we allow the constraint sets todepend on (t, x) and do not assume that the sets Ω(t, x) are compact.

Lemma 4.3.14. Let R = I ×Rn. Let ∆ = (t, x, z) : (t, x) ∈ R, z ∈ Ω(t, x).

Let the function f = (f1, . . . , fn) satisfy

|〈x, f(t, x, z)〉| ≤ Λ(t)(|x|2 + 1), (4.3.16)

for all (t, x, z) in ∆, where Λ ∈ L1(I). Let each admissible trajectory φ containat least one point (t, φ(t)) that belongs to a given compact set C in R. Then,there exists a compact set R0 contained in R such that each admissible trajec-tory lies in R0. If we require that all initial points of admissible trajectorieslie in C, then we can omit the absolute value in the left-hand side of (4.3.16).

Proof. For any trajectory φ, let Φ(t) = |φ(t)|2 + 1. Then, Φ′(t) = 2〈φ(t),f(t, φ(t), u(t))〉, and by virtue of (4.3.16)

|Φ′(t)| ≤ 2Λ(t)(|φ(t)|2 + 1) = 2Λ(t)Φ(t).

Hence−2Λ(t)Φ(t) ≤ Φ′(t) ≤ 2Λ(t)Φ(t). (4.3.17)

If (t, φ(t)) is a point of the trajectory that belongs to C, then upon integrating(4.3.17) we get

Φ(t) ≤ Φ(t) exp

(2

∣∣∣∣∫ t

t

Λ(s)ds

∣∣∣∣)

≤ Φ(t) exp

(2

∫

I

Λ(s)ds

)

for all points of the trajectory. Since C is compact, there exists a constant Dsuch that if (t, x) is in C then |x| ≤ D. Hence

Φ(t) ≤ (D2 + 1) exp

(2

∫

I

Λ(s)ds

).

Since the right-hand side of this inequality is a constant independent of thetrajectory φ, it follows that all trajectories lie in some compact set R0.

If the initial points (t0, x0) all lie in a compact set, we need only utilize therightmost inequality (4.3.16) to obtain a bound on Φ(t) that is independentof φ.

Corollary 4.3.15. Under the hypotheses of Lemma 4.3.14 all admissible re-laxed trajectories lie in a compact set.


Example 4.3.16. Let x be one-dimensional with dx/dt = u(t). Let Ω(t) =z : |z| ≤ A where A > 1. Let B consist of a single point (t0, x0, t1, x2) =(0, 0, 1, 0) and let

J(φ, u) =

∫ 1

0

[φ2(t) + (1− u2(t))2]dt.

Then J(φ, u) > 0 for all φ, u with u(t) ∈ Ω(t) and φ(0) = φ(1) = 0. To showthat the infimum of J(φ, u) among all such (φ, u) equals 0, we define (φk, uk)for k = 1, 2, . . . as follows. For i = 0, 1, . . . , 2k − 1 let

uk(t) =

1 if t ∈ [i/2k, (i+ 1)/2k], i even,

−1 if t ∈ [i/2k, (i+ 1)/2k], i odd,

φk(t) =

∫ t

0

uk(s)ds.

Then φk(0) = φk(1) and J(φk, uk) =13 (2k)

−2, which tends to 0 as k → ∞.Although there is no admissible pair (φ, u) with J(φ, u) = 0, the followingpair (ψ, µ) is minimizing with J(ψ, µ) = 0. Let

µt =1

2δ1 +

1

2δ−1

ψ(t) =

∫ t

0

∫ A

−A

z dµtds =

∫ t

0

[1

2(1) +

1

2(−1)

]ds = 0.

In this example, the local controllability condition in Theorem 4.4.6 is satisfied.

The corollary follows from the observation in Remark 3.5.8 that a re-laxed problem can be considered as an ordinary problem with state equation(3.5.1) and controls (u1, . . . , un+2, p

1, . . . , pn+2) satisfying the constraints inEq. (3.5.2). If (4.3.16) holds for the ordinary problem, then (4.3.16) will holdfor the relaxed problem viewed as an ordinary problem.

Corollary 4.3.17. Either let Assumption 4.3.1 hold or let f be continuouson I × X × U and let the hypotheses of Lemma 4.3.14 hold. Let there exista control u defined on an interval I ⊂ I. Then a corresponding ordinaryor relaxed admissible trajectory defined on I can be extended to a trajectorydefined on all of I.Proof. As noted in the proof of Corollary 4.3.15, it suffices to consider or-dinary admissible trajectories. Let I = [0, a] and let (φ, u) be an admissibletrajectory-control pair defined on a maximal open interval Imax = (α, β). Ifwe set u(t) = u(t) for t in Imax and u(t) = u(t) for all other t in I, then wemay consider (φ, u) to be an admissible pair. Suppose that α > 0 and let tnbe a sequence of points in Imax such that tn → α. Since the graph of φ liesin a compact set, there is a subsequence of tn and a point x0 such thatφ(tn) → x0.


It follows from standard existence and uniqueness theorems for differentialequations that if f satisfies Assumption 4.3.1, then

x′ = f(t, x, u(t)) x(t0) = x0

has a unique solution on an open interval containing t0. If f is assumed to becontinuous, then a solution exists but need not be unique. This contradictsthe maximality of (α, β), so we must have α = 0. Similarly we have that β = aand the desired extension of (φ, u).

4.4 Existence of Ordinary Optimal Controls

Lemma 4.4.1. If (φ∗, u∗) is an ordinary admissible pair that is a solutionof the relaxed problem, then (φ∗, u∗) is a solution of the ordinary problem.Moreover, the minima of the relaxed problem and the ordinary problem areequal.

Proof.

J(ϕ∗, u∗) = infJ(ψ, µ) : (ψ, µ) relaxed admissible≤ infJ(φ, µ) : (φ, u) ordinary admissible≤ J(φ∗, u∗).

We introduce a convexity condition guaranteeing the existence of anoptimal relaxed control that is an optimal ordinary control. Let f =(f0, f1, . . . , fn) and let

Q(t, x) = y = (y0, y) : y = f(t, x, z), z ∈ Ω(t)Q+(t, x) = y = (y0, y) : y

0 ≥ f0(t, x, z), y = f(t, x, z), z ∈ Ω(t).

Theorem 4.4.2. Let the hypothesis of Theorem 4.3.5 hold and let the setsQ+(t, x) be convex. Then there exists an ordinary admissible pair that is op-timal for both the ordinary and relaxed problem.

Proof. We first prove the theorem under the assumption that f is continuous.By Theorem 4.3.5 there exists a relaxed optimal pair (ψ, µ) on an interval[t0, t1]. Define

ψ0(t) =

∫ t

t0

f0(s, ψ(s), µs)ds t0 ≤ t ≤ t1


and set ψ = (ψ0, ψ). Then

ψ′ = f(t, ψ(t), µt) a.e.

Hence by Theorem 3.2.11 there exist measurable functions u1, . . . , un+2 de-fined on [t0, t1] such that each ui(t) ∈ Ω(t) a.e. and real valued functionsp1, . . . , pn+2 defined on [t0, t1] such that

pi(t) ≥ 0

n+2∑

i=1

pi(t) = 1

with the property that

ψ′(t) =

n+2∑

i=1

pi(t)f (t, ψ(t), ui(t)) a.e. on [t0, t1].

Thus, ψ′(t) ∈ co Q(t, ψ(t)). Since Q(t, ψ(t)) ⊆ Q+(t, ψ(t)) and sinceQ+(t, ψ(t)) is convex, we have

ψ′(t) ∈ co Q(t, ψ(t)) ⊆ co Q+(t, ψ(t)) = Q+(t, ψ(t)).

Therefore, for a.e. t in [t0, t1] there exists a z(t) ∈ Ω(t) such that

ψ0′(t) ≥ f0(t, ψ(t), z(t)) (4.4.1)

ψ′(t) = f(t, ψ(t), z(t)).

We shall show, using Filippov’s Lemma, that (4.4.1) holds with z replacedby a measurable function v. Assume for the moment that (4.4.1) holds withz(t) replaced by v(t), where v is measurable and v(t) ∈ Ω(t). The secondequation in (4.4.1) shows that ψ is the trajectory corresponding to the controlv. The first equation shows that the function t→ f0(t, ψ(t), v(t)) is integrable.Since v(t) ∈ Ω(t) and e(ψ) ∈ B, it follows that (ψ, v) is an ordinary admissiblepair for the relaxed problem and is optimal for the relaxed problem. Thetheorem in the case of continuous f now follows from Lemma 4.4.1.

We use the theorem established under the assumption that f is continuousto prove the theorem under the more general assumption that f is measurableon I and continuous on X×U by the argument used in the analogous situationin the proof of Lemma 3.2.10.

To complete the proof we must establish the existence of the measur-able function v. With reference to Filippov’s Lemma, let T = t : ψ′(t) ∈Q+(t, ψ(t). Let Z = R× R

n × Rn × R. Let R0 denote the compact set con-

taining the graphs of all relaxed trajectories. Let ∆ = (t, x, z) : (t, x) ∈ R0,z ∈ Ω(t) and let

D = (t, x, z, η) : (t, x, z) ∈ ∆, η ≥ f0(t, x, z).


The set T is Lebesgue measurable and thus is a measure space. The set Z isclearly Hausdorff. Since Ω is u.s.c.i., it follows from Lemma 3.3.11 that ∆ iscompact. From this and the continuity of f0 we get that D is closed. If D isbounded, then D is compact. Otherwise D is the countable union of compactsets Di, where each Di is the intersection of D with the closed ball centeredat the origin with radius i.

Let Γ denote the mapping from T to Z defined by Γ(t) =

(t, ψ(t), ψ′(t), ψ0′(t)). Since each of the functions ψ and ψ′ is measurable,so is Γ. Let ϕ denote the mapping from D to Z defined by

ϕ(t, x, z, η) = (t, x, f(t, x, z), η).

Since f is continuous, so is ϕ. From (4.4.1) we get that Γ([t0, t1]) ⊆ ϕ(D).Hence all the hypotheses of Filippov’s Lemma hold, and so there exists ameasurable map m from T to D

m : t→ (τ(t), x(t), v(t), η(t))

such that

ϕ(m(t)) = (τ(t), x(t), f(τ(t), x(t), v(t)), η(t))

= Γ(t) = (t, ψ(t), ψ′(t), ψ0′(t)).

Henceψ′(t) = f(t, ψ(t), v(t)), ψ0′(t) ≥ f0(t, ψ(t), v(t)),

where v is measurable.In the proof of Theorem 4.4.2 we used the hypotheses of Theorem 4.3.5

to obtain the existence of an optimal relaxed control. The hypothesis that Ωis u.s.c.i. was used in Theorem 4.3.5 and in the proof of Theorem 4.4.2 toshow that the set D is closed. To show that D is closed, however, we canimpose a less restrictive assumption on the constraint map Ω, namely that itis upper semicontinuous. The definition of upper semicontinuity of a set valuedmapping is given in Definition 5.2.1. It follows from Lemma 5.2.4 that if Ω isupper semicontinuous, then the set ∆ is closed. From this and the continuityof f it follows that D is closed.

If we are considering a problem with non-compact constraints and takerelaxed controls and trajectories to be as given in Section 3.5, the proof ofTheorem 4.4.2 and the observations of the preceding paragraph yield the fol-lowing corollary.

Corollary 4.4.3. Let f be continuous and let the mapping Ω be upper semi-continuous. Let the relaxed problem have a solution (ψ, µ). If the sets Q+(t, x)are convex, then there exists an ordinary control that is optimal for both theordinary and the relaxed problem.


The situation in Lemma 4.4.1 and Theorem 4.4.2 does not always hold.Both the relaxed and ordinary problems can have solutions with the minimumof the relaxed problem strictly less than the minimum of the ordinary problem,as the next example shows.

Example 4.4.4. Let the state equations be

dx1/dt = (x2)2 − (u(t))2 (4.4.2)

dx2/dt = u(t)

dx3/dt = (x2)2.

Let the constraints be given by Ω(t) = z : |z| ≤ 1, and let

B = (t0, x0, t1, x1) : t0 = 0, x0 = 0, t1 = 1, x31 = 0. (4.4.3)

Let f0 = 0 and let g(t0, x0, t1, x1) = x11. Thus, the problem is to minimizeφ1(1) over all admissible pairs (ψ, u).

From the third equation in (4.4.2) we see that to satisfy the end conditionsφ3(0) = 0 and φ3(1) = 0 we must have φ2(t) ≡ 0. From the second equationin (4.4.2) we then get u(t) ≡ 0. Hence, from the first equation and the initialcondition x0 = 0 we get that φ1(t) ≡ 0. Thus, φ ≡ 0 and u ≡ 0 is the onlyadmissible pair, and so is optimal. Moreover, J(φ, u)) = 0.

The ordinary problem has a solution even though the sets Q+(t, x) =(η0, η1, η2, η3), where

η0 ≥ 0, η1 = (x2)2 − z2, η2 = z, η3 = (x2)2, |z| ≤ 1

are not convex. This follows from the fact that the sets

P (t, x) ≡ (η1, η2) : η1 = (x2)2 − z2, η2 = z, |z| ≤ 1

are those points on the parabola η1 = (x2)2 − (η2)2 in the (η1, η2) plane with|η2| ≤ 1.

The relaxed problem corresponding to (4.4.2) has state equations

dx1/dt = (x2)2 −4∑

i=1

pi(t)(ui(t))2 (4.4.4)

dx2/dt =

4∑

i=1

pi(t)ui(t)

dx3/dt = (x2)2.

From the first equation in (4.4.4) we see that inf J(ψ, u) = inf ψ1(1) ≥ −1. Ifwe take

u1(t) = 1 u2(t) = −1 u3(t) =0 u4(t) = 0


p1(t) = 1/2 p2(t) = 1/2 p3(t) =0 p4(t) =0,

then (4.4.4) becomes

dx1/dt = (x2)2 − 1 dx2/dt = 0 dx3/dt = (x2)2. (4.4.5)

The solution of the system (4.4.5) satisfying the end conditions (4.4.3) is

ψ1(t) = −t ψ2(t) = 0 ψ3(t) = 0. (4.4.6)

Since ψ1(1) = −1, we have a solution of the relaxed problem. Note that thesolution of the relaxed problem is not a solution of the ordinary problemand note that the minimum of the relaxed problem is strictly less than theminimum of the ordinary problem.

The Chattering Lemma asserts that there exists a sequence of ordinarytrajectories φn that converge uniformly to ψ. The trajectories φn neednot be admissible, as they may fail to satisfy the end conditions. We nowexhibit such a sequence.

For each positive integer n, partition [0, 1] into 2n equal subintervals andalternately take un(t) = 1 and un(t) = −1 on these subintervals. Then for anysolution of (4.4.2) with φ2n(0) = 0, we have φ2n(t) ≤ 2−n and 0 ≤ φ3n(t) ≤ 4−n

on [0, 1]. Thus, φ2n(t) → 0 and φ3n(t) → 0, uniformly on [0, 1]. From the firstequation of (4.4.2) and the initial condition φ1n(0) = 0 we have

φ1n(t) = −t+∫ t

0

[φ2n(s)]2ds.

From this and the estimate 0 ≤ φ2n(t) ≤ 2−n, we get that φ1n(t) → −t, uni-formly on [0, 1]. Thus, the sequence φn defined above is the desired sequence.Note that the sequence is not admissible because φ3n(1) > 0.

We conclude this section with a result that shows that the situation illus-trated by Example 4.4.4 does not occur if a “local controllability” conditionis assumed. For simplicity, we assume that a fixed initial point (t0, x0) and aterminal set T1 are given. Thus, B = (t0, x0, t1, x1) : (t1, x1) ∈ T1.

Definition 4.4.5. The system is locally controllable at T1 if, for each(t1, x1) ∈ T1 there exists a neighborhood N of (t1, x1) such that the followingproperty holds. If (τ, ξ) ∈ N with τ < t1, then there exists a control u(t)

on [τ, t1] and a corresponding solution φ(t) to (2.3.1) with u(t) ∈ Ω(t) and

φ(τ) = ξ, φ(t1) = x1.

Theorem 4.4.6. If the system is locally controllable at T1, then the infimumof J(φ, u) among admissible ordinary controls equals the minimum of J(ψ, µ)among admissible relaxed controls.

Proof. We sketch a proof, leaving certain details to the reader. Consider the


equivalent form of the control problem in Section 2.4, with augmented stateφ(t) = (φ0(t), φ(t)) where

φ0(t) =

∫ t

t0

f0(s, u(s), φ(s))ds.

Similarly, in the relaxed control problem (Definition 3.2.5) the augmented

state is ψ(t) = (ψ0(t), ψ(t)) where

ψ0(t) =

∫ t

t0

∫

Ω(t)

f0(s, ψ(s), dµsds.

The criterion to be minimized is

J(φ, u) = g(t1, φ(t1)) + φ0(t1)

and in the relaxed problem

J(ψ, µ) = g(t1, ψ(t1)) + ψ0(t1).

Let (ψ∗, µ∗) be an admissible relaxed pair, which minimizes J(ψ, µ) as in

Theorem 4.3.5. Let x1 = ψ∗(t1) and note that (t1, x1) ∈ T1. Since (ψ∗, µ∗) isminimizing,

J(ψ∗, µ∗) ≤ J(ψ, µ)

for all admissible pairs (ψ, µ) and in particular for all ordinary pairs (φ, u). Toprove the theorem, we must show that, for any a > 0, there is an admissiblepair (φ, u) such that

J(φ, u) < J(ψ∗, µ∗) + a.

We apply Theorem 3.6.8 to the augmented formulation of the problem. Foreach ǫ > 0 there exists an ordinary control pair (φǫ, uǫ) with φǫ(t0) = x0,φ0ǫ (t0) = 0, and

|φǫ(t)− ψ∗(t)| < ǫ for t0 ≤ t ≤ t1.

Let τ = t1−δ with δ > 0, ξ = φǫ(t1−δ), and x1 = ψ∗(t1) as in Definition 4.4.5.Let

u(t) = uǫ(t), φ(t) = φǫ(t), if t0 ≤ t < t1 − δ, and

u(t) = u(t), φ(t) = φ(t), if t1 − δ < t ≤ t1.

Note that φǫ(t) = φ(t) if t0 ≤ t ≤ t1 − δ. Then

J(φ, u)− J(ψ∗, µ∗) = φ0(t1)− ψ∗0(t1)

≤ |φ0(t1)− φ0ǫ (t− δ)|+ |φ0ǫ (t1 − δ)− ψ∗0(t1 − δ)|+ |ψ∗0(t1 − δ)− ψ∗0(t1)|.

Since Ω(t) is a subset of a fixed compact set Z, the right side is less than a ifǫ and δ are chosen small enough.


4.5 Classes of Ordinary Problems Having Solutions

We now point out important classes of problems where Theorem 4.4.2 isapplicable. We suppose that the hypotheses concerning R0,Ω, g, and B hold.

The first class of problems is the linear problems. The state equations are

dx

dt= A(t)x +B(t)u(t) + h(t), (4.5.1)

where A is an n × n matrix continuous on some interval I = [T0, T1], B isan n ×m matrix continuous on I, and h is an n-dimensional column vectorcontinuous on I. The cost functional J is given by

J(φ, u) = g(e(φ)) +

∫ t1

t0

〈a0(t), φ(t)〉 + 〈b0(t), u(t)〉+ h0(t)dt,

where a0, b0, and h0 are continuous functions on I. Here the set R is theslab T0 ≤ t ≤ T1, −∞ < xi < ∞ in (t, x) space and U is all of Rm. The

function f = (f0, f) is clearly continuous. The function f also satisfies (4.3.16),so that if all trajectories are required to pass through a fixed compact set,Lemma 4.3.14 guarantees that all trajectories lie in a fixed compact set R0.This would be the case if we assume B to be compact, or that the set of initialpoints (t0, x0) in B is compact. If we further assume that each of the setsΩ(t) is convex, then it is readily checked that the sets Q+(t, x) are convex,and hence an ordinary optimal pair exists. We shall see in Section 4.7 that anordinary optimal pair exists, even if the sets Ω(t) are not convex.

An important problem in the class of linear problems is the “time optimalproblem with linear plant.” In this problem, the state equations are of the form(4.5.1) and it is required to bring the system from a given initial position x0at a given initial time t0 to a given terminal position x1 in such a way asto minimize the time to carry this out. The regulator problem of Section 1.5,Chapter 1 is an example of such a problem. If t1 denotes the time at which thetrajectory reaches x1, then we wish to minimize t1−t0, and the cost functionalbecomes J(φ, u) = t1 − t0. Then, we can consider J as being obtained eitherby setting g(t0, x0, t1, x1) = t1− t0 and f0 ≡ 0 or by setting g ≡ 0 and f0 ≡ 1.

Another class of problems to which Theorem 4.4.2 can be applied is theso-called class of problems with “linear plant and convex integral cost crite-rion.” In these problems, the state equations are given by (4.5.1) and the costfunctional is given by

J(φ, u) = g(t0, φ(t0), t1, φ(t1)) +

∫ t1

t0

f0(t, φ(t), u(t))dt, (4.5.2)

f0 is continuous or satisfies Assumption 4.3.1 and is a convex function of z foreach (t, x) in R. If the constraint sets Ω(t) are convex, then the sets Q+(t, x)


will be convex. To see this, let y = (y01 , y1) and y2 = (y02 , y2) be two points inQ+(t, x). Then there exist points z1 and z2 in Ω(t), such that

y01 ≥ f0(t, x, z1) y1 = A(t)x+B(t)z1 + h(t) (4.5.3)

y02 ≥ f0(t, x, z2) y2 = A(t)x+B(t)z2 + h(t). (4.5.4)

Let α and β be two real numbers such that α ≥ 0, β ≥ 0, and α + β = 1.If we multiply the relations in (4.5.3) by α, the relations in (4.5.4) by β, andadd, we get

αy01 + βy02 ≥ αf0(t, x, z1) + βf0(t, x, z2)

αy1 + βy2 = A(t)x +B(t)(αz1 + βz2) + h(t).

Since Ω(t) is convex, there exists a point z3 in Ω(t) such that z3 = αz1 + βz2.From the convexity of f0 in z, we get

αf0(t, x, z1) + βf0(t, x, z2) ≥ f0(t, x, αz1 + βz2) = f0(t, x, z3).

Henceαy01 + βy02 ≥ f0(t, x, z3)

αy1 + βy2 = A(t)x +B(t)z3 + h(t),

and so Q+(t, x) is convex. Thus, by Theorem 4.4.2 the linear plant and convexintegral cost criterion problem has a solution in the class of ordinary controls.

An important problem in the class of linear problems with convex integralcost criterion is the minimum fuel problem for linear systems. In this problema linear system is to be brought from a given initial state x0 to any state x1 in aspecified set of terminal states in such a way as to minimize the fuel consumedduring the transfer. The terminal time can be either fixed or free. The controlu is required to satisfy constraints |ui(t)| ≤ 1, i = 1, . . . ,m. The rate of fuelflow at time t, which we denote by β(t), is assumed to be proportional to themagnitude of the control vector as follows:

β(t) =

m∑

i=1

ci|ui(t)|, ci > 0, constant.

Thus, the fuel consumed in transferring the system from x0 to x1 is

J(φ, u) =

∫ t1

t0

(m∑

i=1

ci|ui(t)|)dt.

The functional J is to be minimized. Here

f0(t, x, z) =

n∑

i=1

ci|zi|


and f0 is convex in z. The constraint sets Ω(t) are hypercubes and thus areconvex. Theorem 4.4.2 gives the existence of an ordinary optimal control.

Another important problem in the class of linear problems with convexintegral cost criterion is the “quadratic criterion” problem, which arises inthe following way. An absolutely continuous function ζ is specified on a fixedinterval [t0, t1]. This is usually a desired trajectory for the system. It is requiredto choose an admissible control u so that the mean square error over [t0, t1]between the trajectory φ and the given trajectory ζ be minimized and thatthis be accomplished with minimum energy consumption. If one takes theintegral

∫ t1t0

|u|2dt to be a measure of the energy consumption, one is led toconsider the cost functional

J(φ, u) = |φ(t1)− ξ(t1)|2 +∫ t1

t0

|φ(t) − ξ(t)|2dt+∫ t1

t0

|u(t)|2dt.

If we set φ(t) = φ(t)− ξ(t), then since φ is a solution of (4.5.1) φ will also bea solution of a linear system of the form (4.5.1). Hence we can suppose thatthe functional J has the form

J(φ, u) = |φ(t1)|2 +∫ t1

t0

|φ(t)|2dt+∫ t1

t0

|u(t)|2dt.

If one assigns nonnegative weights to the coordinates of the trajectory and tothe components of u, the functional becomes

J(φ, u) = 〈φ(t1), Rφ(t1)〉+∫ t1

t0

〈φ(t), X(t)φ(t)〉dt

+

∫ t1

t0

〈u(t), Q(t)u(t)〉dt,

where X and Q are continuous diagonal matrices with nonnegative diagonalentries and R is a constant diagonal matrix with nonnegative diagonal entries.If the constraint sets Ω(t) are convex, we again obtain the existence of anordinary optimal control.

More generally, we can take X and Q to be continuous positive semi-definite symmetric matrices on [t0, t1]. Later, when we consider noncompactconstraint sets, the matrix Q will be required to be positive definite. Thegenerality in assuming that Q is not diagonal is somewhat spurious, as thefollowing discussion shows. There exists a real orthogonal matrix P such thatD = P ∗QP where D is diagonal and P ∗ is the transpose of P . Under thechange of variable v = P ∗u the quadratic form 〈u,Qu〉 becomes 〈v,Dv〉 withD diagonal. The state equations (4.5.1) become

dx

dt= A(t)x + C(t)v(t) + h(t),

where C(t) = B(t)P (t). If X is a constant matrix, then there is a change of


variable y = Sx, where S is orthogonal and constant, such that the quadraticform 〈x,Xx〉 is replaced by 〈y, Y y〉, with Y diagonal, and the state equationsare transformed into equations that are linear in y and u.

The linear problems and the linear problems with convex integral costcriteria are special cases of the following problem, in which the existence ofan optimal ordinary control follows from Theorem 4.4.2.

Corollary 4.5.1. Let h be a continuous function from R to Rn, let B be an

n×m continuous matrix on R, and let f0 be a real valued continuous functionon R×U such that for each (t, x) in R, f0 is convex on U . Let all trajectoriesintersect a fixed compact set, and let there exist an integrable function L on Isuch that

〈x, h(t, x) +B(t, x)z〉 ≤ L(t)(|x|2 + 1)

for all (t, x, z) in R× U . Let the state equations be

dx

dt= h(t, x) +B(t, x)z

and let J(φ, u) be given by (4.5.2). Let each set Ω(t) be compact and convexand let Ω be u.s.c.i. on I. Then there exists an ordinary pair (φ, u) that isoptimal for both the ordinary and relaxed problems.

4.6 Inertial Controllers

An ordinary control is a measurable function and hence there are no re-strictions on the rate at which it can vary. That is, the control is assumed tohave no inertia. While this may be a reasonable model for electrical systems,it may not be reasonable for mechanical or economic systems. We thereforeconsider controls that do have inertia. Continuous functions u with piecewisecontinuous derivatives u′ such that |u′| ≤ K, where K is a constant, appearto be reasonable models of inertial controllers. At points of discontinuity ofu′ we interpret the inequality |u′(t)| ≤ K to hold for both the right- andleft-hand limits u′(t + 0) and u′(t − 0). We now have a bound on the rateat which a control can be changed. It turns out that the class of functions ujust described is too restrictive to enable us to prove an existence theorem forproblems with inertial controllers. For this purpose it is necessary to take ourmodel of an inertial controller to be an absolutely continuous function u suchthat |u′(t)| ≤ K a.e., where K is a constant independent of u.

We now state a minimization problem for inertial controllers.

Problem 4.6.1. Minimize the functional

J(φ, u) = g(e(φ)) +

∫ t1

t0



subject to

dφ/dt = f(t, φ(t), u(t))

u(t) ∈ Ω(t)

(t0, φ(t0), t1, φ(t1)) ∈ Bu is absolutely continuous on [t0, t1]

|u′(t)| ≤ K a.e. on [t0, t1],

whereK is a pre-assigned constant and g is a functional on the end conditions.

Theorem 4.6.1. Let the class of admissible pairs for Problem 4.6.1 be non-empty and let the following hold.

(i) There exists a compact set R0 ⊂ R such that for all admissible trajec-tories φ we have (t, φ(t)) ∈ R0 for all t in [t0, t1].

(ii) The set B is closed.

(iii) The mapping Ω is u.s.c.i. on R0.

(iv) For each t in [t0, t1] the set Ω(t) is compact.

(v) The function f = (f0, f) is continuous and g is lower semicontinuouson B.

Then Problem 4.6.1 has a solution that is also a solution of the relaxed versionof Problem 4.6.1.

Proof. We reformulate this problem by taking the control variable z to be astate variable and the derivative of the control to be the control. Thus, wedefine a new system by

dx

dt= f(t, x, z) (4.6.1)

dz

dt= v(t).

Let

R = (t, x, z) : (t, x) ∈ R, z ∈ Ω(t),B = (t0, x0, z0, t1, x1, z1) : (t0, x0, t1, x1) ∈ B, zi ∈ Ω(ti), i = 0, 1,

Ω(t) = w ∈ Rm : |w| ≤ K.

The functional to be minimized is

J(φ, ζ, v) = g(e(φ)) +

∫ t1

t0

f0(t, φ(t), ζ(t))dt,


where (φ, ζ) is a solution of (4.6.1), (t, φ(t), ζ(t)) ∈ R, v(t) ∈ Ω(t) a.e., and

(t0, φ(t0), ζ(t0), t1, φ(t1), ζ(t1)) ∈ B.By hypothesis, all points (t, φ(t)) of an admissible trajectory lie in a com-

pact set R0 ⊆ R. Since Ω(t) = w : |w| ≤ K for all t, all points (t, ζ(t)) liein a compact set of (t, z) space. Hence all points (t, φ(t), ζ(t)) of trajectoriesof the reformulated problem lie in a compact set in (t, x, z)-space.

The set B is closed. To see this let the sequence of points (t0n, x0n,z0n, t1n, x1n, z1n) tend to a point (t0, x0, z0, t1, x1, z1). Then since B is closed,(t0, x0, t1, x1) ∈ B. Let ε > 0 be given. Since the mapping Ω is u.s.c.i, thereexists a positive integer n0 such that for n > n0, z0n ∈ [Ω(t0)]ε. Hencez0 ∈ [Ω(t0)]ε. Since Ω(t0) is compact and ε is arbitrary, we get that z0 ∈ Ω(t0).

Similarly, z1 ∈ Ω(t1), and so B is closed.In the reformulated problem the control variable does not appear in

f = (f0, f). It appears linearly in the second equation in (4.6.1) and sat-isfies the constraint w : |w| ≤ K. Therefore, the sets Q+(t, x, z) of the re-formulated problem are convex. All the other hypotheses of Theorem 4.4.2are clearly fulfilled. It follows from Remark 4.3.10 and the fact that in thepresent situation an optimal trajectory is also a relaxed optimal trajectorythat ζ(t) ∈ Ω(t) for all t. Hence the reformulated problem has a solution, andtherefore so does Problem 4.6.1.

4.7 Systems Linear in the State Variable

A system that is linear in the state variable has state equations

dx

dt= A(t)x + h(t, u(t)). (4.7.1)

A control problem is linear in the state variable if the state equations are givenby (4.7.1) and the payoff is given by

J(φ, u) = g(e(φ)) +

∫ t1

t0

〈a0(t), φ(t)〉 + h0(t, u(t)) dt. (4.7.2)

In this section we shall assume the following.

Assumption 4.7.1. (i) The n× n matrix A and the n-vector a0 are con-tinuous on some compact interval I = [T0, T1].

(ii) The n-vector h and the scalar h0 are continuous on I × U , where U isan interval in R

m.

(iii) The terminal set B is closed and g is continuous on B.


(iv) For each t in I the set Ω(t) is a compact subset of U and the mappingΩ is u.s.c.i.

Lemma 4.7.2. Let the linear in the state system (4.7.1) satisfy (i), (ii), and(iv) of Assumption 4.7.1. Then there exists a control u defined on I such thatu(t) ∈ Ω(t) for all t.

Proof. For each t in I let

d(t) = min|z| z ∈ Ω(t).

Since the absolute value is a continuous function and Ω(t) is compact, we maywrite min and the minimum is achieved at some point z0(t) in Ω(t). We assertthat the function d is lower semicontinuous on I. To show this we shall showthat for each real α the set

Eα = t : d(t) ≤ α

is closed. Let tn be a sequence of points in Eα tending to a point t0. Letz0(tn) be a point in Ω(tn) such that d(tn) = |z0(tn)|. Since the mapping Ωis u.s.c.i. on the compact interval I and each Ω(t) is compact, it follows fromLemma 3.3.11 that the sequence z0(tn) lies in a compact set. Hence thereexists a subsequence z0(tn) and a point z0 in R

m such that z0(tn) → z0.We now show that z0 ∈ Ω(t). Let ε > 0 be given. Since Ω is u.s.c.i and

tn → t0, it follows that there is a positive integer n0 such that for n >n0, z0(tn) ∈ [Ω(t0)]ε. Hence z0 ∈ [Ω(t0)]ε. Since ε is arbitrary and Ω(t0) iscompact, z0 ∈ Ω(t0). Since |z0(tn)| = d(tn) ≤ α, we have |z0| ≤ α. Henced(t0) ≤ α, and so Eα is closed.

In summary, we have shown that for each t in I there exists a z0(t) ∈ Ω(t)such that d(t) = |z0(t)| for all t in I and that the function d is lower semi-continuous and hence measurable on I. Since the absolute value is a continuousfunction, it follows from Lemma 3.2.10 that there is a measurable function usuch that u(t) ∈ Ω(t) for t in I and d(t) = |u(t)|. From standard theorems onthe existence of solutions of linear differential equations, we get that

dx

dt= A(t)x + h(t, u(t))

has a solution defined on all of I. Hence u is a control.

To facilitate the study of systems linear in the state we introduce thenotion of attainable set.

Definition 4.7.3. Given a control system (not necessarily linear in the state)

dφ

dt= f(t, φ(t), u(t))

and initial condition (t0, x0), then the attainable set at time t > t0, written


K(t; t0, x0), is the set of points x such that there exists an admissible pair (φ, u)with φ(t) = x. The relaxed attainable set, at time t, written KR(t; t0, x0), isthe set of points x such that there exists an admissible relaxed pair (ψ, µ)with ψ(t) = x.

Since an ordinary trajectory is also a relaxed trajectory, we have that fort > t0,

K(t; t0, x0) ⊆ KR(t; t0, x0).

The principal result of this section is that for systems linear in the stateK(t; t0, x0) is not empty and K(t; t0, x0) = KR(t; t0, x0); from this severalimportant consequences will follow. We begin with a result that will be usedin our arguments.

Lemma 4.7.4. Let E be a measurable subset of the line with finite measure.Let y be a function defined on E with values in R

k and such that y ∈ L1(E).Let w be a real valued measurable function defined on E such that 0 ≤ w ≤ 1.Then there exists a measurable subset F ⊂ E such that

∫

E

y(t)w(t)dt =

∫

F

y(t)dt.

In the proof of Lemma 4.7.4 and elsewhere in this section we shall need theKrein-Milman theorem, which we state as Lemma 4.7.5. We refer the readerto Dunford-Schwartz [31, Theorem V 8.4, p. 440] for a proof of this theorem.We shall also need, here and elsewhere, a theorem of Mazur, which we state asLemma 4.7.6. Certain basic facts about the weak-* topology of a Banach spacewill also be used. For these topics we refer the reader to Dunford-Schwartz [31,Chapter V, pp. 420–439]. A short, readable treatment of some of the topicsused can also be found in Hermes and LaSalle [42, pp. 1–22].

Lemma 4.7.5 (Krein-Milman). Let C be a compact convex set in a locallyconvex topological vector space. Then C is the closed convex hull of its extremepoints. If C is a compact convex set in R

n, then C is the convex hull of itsextreme points.

Lemma 4.7.6 (Mazur). Every strongly closed convex subset of a Banachspace is weakly closed.

Proof of Lemma 4.7.4. Define a mapping T as follows. For each real valuedfunction ρ in L∞(E), let Tρ =

∫E y(t)ρ(t)dt. The mapping T so defined is a

continuous mapping from L∞(E) with the weak-* topology to Rk with the

euclidean topology. Let a =∫E y(t)w(t)dt. Then T

−1(a) is a convex, weak-*closed set of L∞(E). Let Σ denote the intersection of T−1(a) and the unitball in L∞(E). Since w ∈ Σ, the set Σ is not empty. The weak-* topology is aHausdorff topology. Therefore, since the unit ball in L∞(E) is weak-* compactand T−1(a) is weak-* closed, the set Σ is weak-* compact. It is also convex.Therefore, by the Krein-Milman theorem, Σ has extreme points. We show that


the extreme points of Σ are characteristic functions χF of measurable subsetsF of E. This will prove the lemma, for then

a =

∫

E

yχFdt =

∫

F

ydt.

The proof proceeds by induction on k, the dimension of the range of y. Weshall give the general induction step. The proof for the initial step, k = 1, isessentially the same as the proof for the general step and will be left to thereader.

Assume that the lemma is true for k− 1. We suppose that θ is an extremepoint of Σ and that θ is not a characteristic function of some set F . Thenthere exists an ǫ > 0 and a measurable set E1 ⊂ E with meas(E1) > 0 suchthat ǫ ≤ θ(t) ≤ 1− ǫ for a.e. t in E1. Let E2 and E3 be two subsets of E1 suchthat E2 and E3 have positive measure and E3 = E1 −E2. From the inductivehypothesis applied to E2 and E3 we obtain the existence of measurable setsF2 and F3 such that F2 ⊂ E2, F3 ⊂ E3 and

1

2

∫

Ej

yi(t)dt =

∫

Fj

yi(t)dt i = 1, . . . , k − 1, j = 2, 3.

Let h2 = 2χF2−χE2

and let h3 = 2χF3−χE3

. Then h2 and h3 are not identi-cally zero on E1, do not exceed one in absolute value, and for i = 1, . . . , k− 1satisfy ∫

E2

yih2dt =

∫

E3

yih3dt = 0.

Let h(t) = αh2(t) + βh3(t) where α and β are chosen so that |α| < ε/2,|β| < ε/2, α2 + β2 > 0, and

∫

E1

ykhdt = α

∫

E2

ykh2dt+ β

∫

E3

ykh3dt = 0.

This can always be done. The function h so defined also satisfies∫E1yihdt = 0

for i = 1, . . . , k − 1 and |h(t)| < ǫ. Hence 0 < θ ± h < 1 and

∫

E1

y(θ ± h)dt =

∫

E1

yθdt = a.

Hence θ + h and θ − h are in Σ. But then θ cannot be an extreme point ofΣ because it is the midpoint of the segment with end points θ + h and θ − h.Therefore, θ must be a characteristic function of some set F .

Theorem 4.7.7. Let I be a compact interval [T0, T1], let A be an n×n matrixcontinuous on I, and let h be a continuous mapping from I ×U to R

n, whereU is an interval in R

k. Let Ω be a mapping from I to compact subsets Ω(t) ofU that is u.s.c.i. on I. Then for t0 ∈ I and x0 ∈ R

n the sets KR(t; t0, x0) arenonempty, compact and convex, and KR(t; t0, x0) = K(t; t0, x0).


Proof. By Lemma 4.7.2 the sets K(t; t0, x0), and hence KR(t; t0, x0) are non-empty. By the variation of parameters formula, a relaxed trajectory ψ corre-sponding to a relaxed control µ is given by

ψ(t) = Ψ(t)[x0 +

∫ t

t0

Ψ−1(s)h(s, µs)ds], (4.7.3)

where Ψ is the fundamental matrix solution of the homogeneous system x′ =Ax satisfying Ψ(t0) = I, the n× n identity matrix, and

h(s, µs) =

∫

Ω(s)

h(s, z)dµs.

The convexity of KR(t, t0, x0) is an immediate consequence of (4.7.3). Thecompactness of KR(t, t0, x0) follows from (4.7.3) and Theorem 3.3.12.

From Theorem 3.2.11 and Remark 3.5.7 we have that the relaxed systemcorresponding to (4.7.1) can also be written as

dx

dt= A(t)x +

n+1∑

i=1

pi(t)h(t, ui(t)), (4.7.4)

where for almost all t in In+1∑

i=1

pi(t) = 1 0 ≤ pi(t) ≤ 1, i = 1, . . . , n+ 1 (4.7.5)

and ui(t) ∈ Ω(t). Thus, the solution of (4.7.4) that satisfies ψ(t0) = x0 isdefined for all t in I as is the solution of (4.7.1) that satisfies φ(t0) = x0.

Let Θ denote the set of measurable functions θ that satisfy

θ(s) ∈ co h(s,Ω(s)) a.e., (4.7.6)

where co A denotes the convex hull of A. We shall show that there exists aconstant A such that

|θ(t)| ≤ A a.e (4.7.7)

for all θ in Θ and that θ ∈ Θ if and only if

θ(s) =

n+1∑

i=1

pi(s)h(s, ui(s)) a.e., (4.7.8)

where p1, . . . , pn+1 are real valued measurable functions on [t0, t] satisfying therelations (4.7.5) and where u1, . . . , un+1 are measurable functions satisfyingui(s) ∈ Ω(s).

Any function θ satisfying (4.7.8) is measurable and satisfies (4.7.6). Con-versely, by Caratheodory’s Theorem (Sec. 3.2) any measurable function θ


that satisfies (4.7.6) can be written in the form (4.7.8) without any asser-tion about the measurability of the functions pi and ui, i = 1, . . . , n+1. FromLemma 3.2.10, however, we conclude that the pi and ui may be chosen to bemeasurable.

Since Ω is u.s.c.i. on I, it follows from Lemma 3.3.11 that the set(s, z) : s ∈ I, z ∈ Ω(s) is compact. The continuity of h implies that h isbounded on this set. Therefore, (4.7.7) holds. Thus, any measurable functionof the form (4.7.6) is integrable. Therefore, (4.7.6) and (4.7.7) give equivalentcharacterizations of Θ.

From (4.7.8) we see that Θ is contained in a closed ball B of finite ra-dius in L2[t0, t1]. We also note that Θ is a convex set in L2[t0, t]. Since Θ isconvex, if we can show that Θ is strongly closed, then by Mazur’s Theorem(Lemma 4.7.6), Θ will be weakly closed. The closed ball B is weakly compact,and since Θ ⊆ B, we can conclude that Θ is weakly compact. We now showthat Θ is indeed strongly closed.

In the next to the last paragraph we showed that the set (s,Ω(s)) =(s, z) : t0 ≤ s ≤ t1, z ∈ Ω(s) is compact. Since h is continuous, the seth(s,Ω(s)) is also compact. Therefore, so are the sets co h(s,Ω(s)). Let θk bea sequence of functions in Θ converging to a function θ0 in L2[t0, t1]. There ex-ists a subsequence, relabeled as θk, such that θk(s) → θ0(s), except possiblyon a set E0 of measure zero. Now θk(s) ∈ co h(s,Ω(s)) except possibly on a setEk of measure zero. Let E =

⋃∞k=0 Ek. Then for s 6∈ E, θk(s) ∈ co h(s,Ω(s))

for k = 1, 2, 3, . . . and θk(s) → θ0(s). Since for all s 6∈ E, the set co h(s,Ω(s))is closed, θ0(s) ∈ co h(s,Ω(s)). Thus, θ0 ∈ Θ and so Θ is strongly closed.

From (4.7.4), (4.7.6), and (4.7.8) it follows that a relaxed trajectory canalso be defined as an absolutely continuous function such that

ψ′(s) = A(s)ψ(s) + θ(s)

for some function θ in Θ. From this and from the variation of parametersformula we have that

ψ(t) = Ψ(t)[x0 +

∫ t

t0

Ψ−1(s)θ(s)ds]

for some θ ∈ Θ. Hence if x is in KR(t; t0, x0), then

x = Ψ(t)[x0 +

∫ t

t0

Ψ−1(s)θ(s)ds (4.7.9)

for some θ ∈ Θ.Since a trajectory of the system (4.7.1) is also a relaxed trajectory, we have

that K(t, t0, x0) ⊆ KR(t, t0, x0). Therefore, to complete the proof it suffices toshow that KR(t, t0, x0) ⊆ K(t, t0, x0).

Let x be an element of KR(t, t0, x0). Then x is given by (4.7.9). Let

a = Ψ−1(t)x− x0 =

∫ t

t0

Ψ−1(s)θ(s)ds.


Define a linear mapping T from L2[t0, t] to Rn by the formula

Tρ =

∫ t

t0

Ψ−1(s)ρ(s)ds.

The mapping T is a continuous map from L2[t0, t] to Rn.

The point a is in the set T (Θ). The set T−1(a) is non-empty, is closed, andis convex in L2[t0, t]. Let Σ denote the intersection of T−1(a) and Θ. ThenΣ is weakly closed and convex. Since Θ is bounded, so is Σ. Since Σ is alsostrongly closed and convex, by Mazur’s Theorem, Σ is also weakly closed. SinceΘ is weakly compact, Σ is weakly compact. By the Krein-Milman Theorem(Lemma 4.7.5), Σ has an extreme point θ0. Since θ0 ∈ Θ it follows that θ0has a representation given by (4.7.8). We now assert that on no measurablesubset E of [t0, t] with positive measure can we have 0 < ǫ ≤ pi(s) ≤ 1− ǫ forsome i in the set i, . . . , n + 1 and some ǫ > 0. This assertion implies that θ0can be written

θ0(s) = h(s, u(s)), (4.7.10)

where u is a measurable function on [t0, t] satisfying u(s) ∈ Ω(s). Once weestablish (4.7.10), the theorem will be proved, for then we will have that

a = Ψ−1(t)x− x0 =

∫ t

t0

Ψ−1(s)h(s, u(s))ds,

which says that x ∈ K(t, t0, x0).Let us suppose that θ0 has the form (4.7.8) and that there exists an ǫ > 0

and a measurable set E ⊆ [t0, t] such that meas(E) > 0 and ǫ ≤ pi(t) ≤ 1− ǫon E for some index i. Then for at least one other index i we must also haveǫ ≤ pi(t) ≤ 1 − ǫ on E. For definiteness let us suppose that i = 1 and i = 2are the indices. Since a belongs to T (Θ), we get from (4.7.8) that

a =

∫ t

t0

Ψ−1(s)

n+1∑

i=1

pi(s)h(s, ui(s))

ds

=

∫ t

t0

n+1∑

i=1

pi(s)(Ψ−1(s)h(s, ui(s))

ds.

We next apply Lemma 4.7.4 with w ≡ 1/2 and y the vector function inL1(E) with range in R

2n defined by

y(s) =

(Ψ−1(s)h(s, u1(s))Ψ−1(s)h(s, u2(s))

).

We obtain the existence of a set F ⊆ E such that

1

2

∫

E

Ψ−1(s)h(s, ui(s))ds =

∫

F

Ψ−1(s)h(s, ui(s))ds i = 1, 2.


Let χF denote the characteristic function of the set F and let χE denote thecharacteristic function of the set E. Let the function γ be defined as follows:

γ(s) = 2χF (s)− χE(s).

The function γ is equal to one in absolute value on the set E. Also, for i = 1, 2,∫ t

t0

Ψ−1(s)h(s, ui(s))γ(s)ds = 2

∫

F

Ψ−1(s)h(s, ui)ds−∫

E

Ψ−1(s)h(s, ui(s))ds = 0.

Let

π11 = p1 + ǫγ π2

1 = p2 − ǫγ πi1 = pi, i = 3, . . . , n+ 1

π12 = p1 − ǫγ π2

2 = p2 + ǫγ πi2 = pi, i = 3, . . . , n+ 1.

Then 0 ≤ πi1 ≤ 1 and 0 ≤ πi

2 ≤ 1 for i = 1, . . . , n + 1. Also, Σπi1 = 1 and

Σπi2 = 1.Let

θ1(s) =

n+1∑

i=1

πi1(s)h(s, ui(s))

θ2(s) =

n+1∑

i=1

πi2(s)h(s, ui(s)).

Then θ1 and θ2 are in Θ and

a =

∫ t

t0

Ψ−1(s)θ1(s)ds =

∫ t

t0

Ψ−1(s)θ2(s)ds.

Hence θ1 and θ2 are in Σ. But, θ0 = (θ1 + θ2)/2, which contradicts the factthat θ0 is an extreme point, and the theorem is proved.

An important consequence of Theorem 4.7.7 is that optimal control prob-lems that are linear in the state have ordinary solutions in the absence ofconvexity assumptions on the sets Q+(t, x).

Theorem 4.7.8. Let the optimal control problem have state equations (4.7.1)and payoff (4.7.2). Let Assumption 4.7.1 hold and let all trajectories of thesystem (4.7.1) have some point in a fixed compact set K. Then there exists anordinary control that is optimal for both the ordinary and relaxed problem.

Proof. By introducing an additional coordinate x0 and state equationdx0/dt = 〈a0(t), x〉 + h0(t, u(t)), we may assume that the state equations arelinear in the state and the payoff is a terminal payoff. Since the state equationsare linear in the state and h is bounded on the set (s, z); s ∈ I, z ∈ Ω(s)the growth condition (4.3.16) holds. The assumption that all trajectories in-tersect a fixed compact set imply (Lemma 4.3.14) that all trajectories lie in acompact set. It is readily checked that all other assumptions of Theorem 4.3.5hold. Hence a relaxed optimal pair (ψ∗, µ∗) exists on an interval [t0, t1].


Let m = infJ(ψ, µ) : (ψ, µ) admissible and let m0 = infJ(φ, u) : (φ, u)admissible. Then, m ≤ m0 and

m = J(ψ∗, µ∗) = g(e(ψ∗)).

The relation K(t1, t0, x0) = KR(t1, t0, x0) implies that there exists an ordinarypair (φ∗, u∗) with e(φ∗) = e(ψ∗). Hence

m = g(e(ψ∗)) = g(e(φ∗)) ≥ m0 ≥ m.

Thus g(e(φ∗)) = m and the pair (φ∗, u∗) is an ordinary optimal pair, which isoptimal for the relaxed problem.

Another corollary of Theorem 4.7.7 is the so-called “bang-bang principle,”which is contained in Theorem 4.7.9. The reason for the terminology and thesignificance of the principle in applications will be discussed after the proof ofTheorem 4.7.9 is given.

If C is a compact convex set in Rm, then we shall denote the set of extreme

points of C by Ce. By the Krein-Milman Theorem, Ce is non-void and C =co(Ce).

Theorem 4.7.9. Let I = [t0, t1] be a compact interval in R1, let A be an

n×n continuous matrix on I, and let B be an n×m continuous matrix on I.Let C be a compact convex set in R

m. Let K(t1, t0, x0) denote the attainableset at t1 for the system

dx

dt= A(t)x+ B(t)u(t) (4.7.11)

with initial point (t0, x0) and with the control constraint u(t) ∈ C. LetKe(t1, t0, x0) denote the attainable set for the system (4.7.11) with initial point(t0, x0) and with control constraint u(t) ∈ Ce. Then Ke(t1, t0, x0) is non-emptyand K(t1, t0, x0) = Ke(t1, t0, x0).

Proof. Since the function defined by u(t) = z0, where z0 is any point of Ceis admissible for the system (4.7.11) with initial point (t0, x0) and controlconstraint u(t) ∈ Ce, it follows that Ke(t1, t0, x0) is non-empty. Since C iscompact and convex, C = co(Ce). By Caratheodory’s theorem every point inco(Ce) can be written as a convex combination of at most (n + 1) points inCe. Therefore, any control u such that u(t) ∈ C can be written as

u(t) =n+1∑

i=1

pi(t)ui(t),

where 0 ≤ pi(t) ≤ 1, Σpi(t) = 1, and ui(t) ∈ Ce. By Lemma 3.2.10 the func-tions pi and ui can be chosen to be measurable. Hence the set K(t1, t0, x0)is contained in the relaxed attainable set KeR(t1, t0, x0) corresponding toKe(t1, t0, x0).


Conversely, every relaxed control for the system (4.7.11) with control con-straint u(t) ∈ Ce is a control for the system (4.7.11) with control constraintu(t) ∈ C. Hence

K(t1, t0, x0) = KeR(t1, t0, x0).

It is readily checked that the system (4.7.11) with initial point (t0, x0) andcontrol constraint u(t) ∈ Ce satisfies the hypotheses of Theorem 4.7.7. HenceKeR(t1, t0, x0) = Ke(t1, t0, x0) and the present theorem is established.

In many applications the constraint set C is a compact convex polyhedron,or even a cube, in R

m. The set of extreme points Ce is the set of vertices ofthe polyhedron, and is therefore closed. Theorem 4.7.9 in this situation statesthat if a control u with values u(t) ∈ C will transfer the system from a pointx0 at time t0 to a point x1 at time t1, then there exists a control ue with valuesue(t) in Ce that will do the same thing. Thus, in designing a control system thedesigner need only allow for a finite number of control positions correspondingto the vertices of C. The term “bang-bang” to describe controls with values onthe vertices of polyhedron derives from the case where C is a one-dimensionalinterval. In this case, controls ue with ue(t) ∈ Ce are controls that take onthe values +1 and −1. Such controls represent the extreme positions of thecontrol device and are therefore often referred to in the engineering vernacularas “bang-bang” controls. In the control literature, the terminology has beencarried over to theorems such as Theorem 4.7.9.

Chapter 5

Existence Theorems; Non-Compact

Constraints

5.1 Introduction

In this chapter we shall prove existence theorems for ordinary and relaxedversions of Problem 2.3.2, which we restate for the reader’s convenience.

Minimize

J(ϕ, u) = g(e(ϕ)) +

∫ t1

t0

f0(t, ϕ(t), u(t)) dt (5.1.1)

subject todϕ

dt= f(t, ϕ(t), u(t)) (5.1.2)

and(t0, ϕ(t0), t1, ϕ(t1)) ∈ B u(t) ∈ Ω(t, ϕ(t)). (5.1.3)

The constraint sets Ω(t, x) depend on t and x and are not assumed to beconvex.

In Chapter 4, when studying this problem, the constraint sets Ω(t) wereassumed to depend on t alone and to be compact. The weak compactness ofrelaxed controls in this case enabled us to pattern the proof of the existenceof an optimal relaxed pair after the proof of the theorem that a real valuedcontinuous function on a compact set attains a minimum. If the constraintsare not compact, then a set of relaxed controls need not be compact, as wasshown in Remark 3.3.7. To overcome this deficiency when the constraints arenot compact, we impose conditions on the data of the problem that guaranteethe following. Given a relaxed minimizing sequence (ψn, µn), there existsa subsequence, which we relabel as (ψn, µn), and an admissible pair (ψ, µ)such that ψn → ψ uniformly and

limn→∞

∫ t1

t0

f0(t, ψn(t), µnt)dt ≥∫ t1

t0

f0(t, ψ(t), µt)dt.

If we assume that g is a lower semi-continuous mapping from B to the reals,then we get that (ψ, µ) is optimal by the usual argument. Namely, let

m = infJ(ψ, µ) : (ψ, µ) admissible.

113


Then

m = limn→∞

J(ψn, µn) = lim infn→∞

J(ψn, µn) (5.1.4)

≥ lim infn→∞

g(e(ψn)) + lim infn→∞

∫ t1

t0

f0(t, ψn(t), µnt)dt

≥ g(e(ψ)) +

∫ t1

t0

f0(t, ψ(t), µ(t))dt = J(ψ, µ) ≥ m.

Hence J(ψ, µ) = m, and so (ψ, µ) is optimal.

5.2 Properties of Set Valued Maps

In this section we shall consider regularity properties of set valued mapsΛ from a subset X of a euclidean space R

p to subsets of a euclidean spaceR

q. The existence theorems of this chapter will involve the regularity of theconstraint mapping Ω. The reader may omit this section and return to it asvarious regularity conditions arise in the sequel.

One of the regularity conditions is upper semi-continuity with respect toinclusion (u.s.c.i.), which was introduced in Definition 3.3.8. In Lemma 3.3.11we gave a necessary and sufficient condition that a mapping Λ defined on acompact subset of Rp and whose values are compact sets be u.s.c.i.

We recall notation introduced in Chapter 3. Let ξ0 be a point in X . ThenNδ(ξ0) will denote the δ-neighborhood of ξ0 relative to X . That is,

Nδ(ξ0) = ξ : |ξ − ξ0| < δ, ξ ∈ X.By Λ(Nδ(ξ0)), the image of Nδ(ξ0) under Λ, we mean

Λ(Nδ(ξ0)) =⋃

[Λ(ξ) : ξ ∈ Nδ(ξ0)].

Definition 5.2.1. A mapping Λ is said to be upper semi-continuous at apoint ξ0 in X if ⋂

δ>0

cl Λ(Nδ(ξ0)) ⊆ Λ(ξ0), (5.2.1)

where cl denotes closure. A mapping Λ is upper semi-continuous on a set Xif it is upper semi-continuous at every point of X .

Since the inclusion opposite to that in (5.2.1) always holds, we may replacethe inclusion in (5.2.1) by equality and obtain an equivalent definition. Fromthis it follows that if Λ is upper semi-continuous at ξ0, then Λ(ξ0) must beclosed.

In Example 3.3.10 for both maps Λ1 and Λ2 we have Λi(Nδ(0)) = R1,

i = 1, 2. Hence⋃

δ>0 cl Nδ(Λi(0)) = R1, i = 1.2. Since Λ1(0) = R

1 andΛ2(0) = 0, we see that Λ1 is upper semi-continuous at 0, but Λ2 is not.

Existence Theorems; Non-Compact Constraints 115

Lemma 5.2.2. Let Λ be u.s.c.i. at ξ0 and let Λ(ξ0) be closed. Then Λ is uppersemi-continuous at ξ0.

Proof. Let ε > 0 be given. Then there exists a δ0 > 0 such that for all0 < δ < δ0

Λ(ξ0) ⊆ Λ(Nδ(ξ0)) ⊆ [Λ(ξ0)]ε.

Hence, since [Λ(ξ0)]ε is closed,

Λ(ξ0) ⊆ cl Λ(Nδ(ξ0)) ⊆ cl [Λ(ξ0)]ε = [Λ(ξ0)]ε.

Therefore, since ε > 0 is arbitrary and Λ(ξ0) is closed

Λ(ξ0) ⊆⋂

δ>0

cl Λ(Nδ(ξ0)) ⊆ Λ(ξ0).

Thus, equality holds throughout and the upper semi-continuity of Λ at ξ0 isproved.

We next give an example of a mapping Λ and a point ξ0 such that Λ(ξ0)is closed, Λ is upper semi-continuous at ξ0, but Λ is not u.s.c.i. at ξ0.

Example 5.2.3. Let t ∈ R and let

Λ(t) =

[0, 1] ∪ 1/t if t > 0

[0, 1] if t = 0.

Then for all t ≥ 0, the set Λ(t) is closed and Λ(Nδ(0)) = [0, 1]∪[1/δ,∞). HenceΛ is not u.s.c.i. at t = 0. On the other hand, Λ is upper semi-continuous att = 0 since

⋂δ>0 cl Λ(Nδ(0)) = [0, 1] = Λ(0).

The next lemma is the “closed graph theorem” for upper semi-continuousmappings, and is used in the proofs of existence theorems.

Lemma 5.2.4. Let X be closed. A necessary and sufficient condition that amapping Λ be upper semi-continuous on X is that the set GΛ = (ξ, λ) : ξ ∈ X,λ ∈ Λ(ξ) be closed.

Proof. We first suppose that Λ is upper semi-continuous on X , so that (5.2.1)holds at every point of X . Let (ξn, λn) be a sequence of points in GΛ con-verging to a point (ξ0, λ0). Since X is closed, ξ0 ∈ X . Moreover, for eachδ > 0 there exists a positive integer n(δ) such that for n > n(δ), ξn ∈ Nδ(ξ0).Since λn ∈ Λ(ξn), we have that for n > n(δ), λn ∈ Λ(Nδ(ξ0)). Henceλ0 ∈ cl Λ(Nδ(ξ0)), for each δ > 0. Thus, λ0 ∈ ⋂δ>0 cl Λ(Nδ(ξ0)), and from(2.1) we get that λ0 ∈ Λ(ξ0). Thus, (ξ0, λ0) ∈ GΛ, and GΛ is closed.

Conversely, let GΛ be closed and let λ0 ∈ cl Λ(Nδ(ξ0)) for each δ > 0.Then there exists a sequence of points ξn in X , a sequence of positivenumbers δn, and a sequence of points λn such that the following hold:(i) δn → 0; (ii) ξn ∈ Nδn(ξ0); (iii) λn ∈ Λ(ξn); and (iv) λn → λ0. Thus,(ξn, λn) → (ξ0, λ0). Since X and GΛ are closed, λ0 ∈ Λ(ξ0), and therefore(5.2.1) holds.


Definition 5.2.5. A mapping Λ is said to possess the Cesari property at apoint ξ0 if ⋂

δ>0

cl co Λ(Nδ(ξ0)) ⊆ Λ(ξ0). (5.2.2)

A mapping Λ possesses the Cesari property on a set X if it possesses theproperty at every point of X .

Since the inclusion opposite to that in (5.2.2) always holds, we may replacethe inclusion in (5.2.2) by an equality and obtain an equivalent definition.From this it follows that if Λ possesses the Cesari property at a point ξ, thenΛ(ξ0) must be closed and convex.

Remark 5.2.6. If Λ possesses the Cesari property at ξ0, then it is uppersemi-continuous at ξ0. To see this, note that

Λ(ξ0) ⊆⋂

δ>0

cl Λ(Nδ(ξ0)) ⊆⋂

δ>0

cl co Λ(Nδ(ξ0)) = Λ(ξ0).

The mapping Λ of Example 5.2.3 is upper semi-continuous at 0, but doesnot have the Cesari property at 0, even though Λ(0) is closed and convex. Atpoints ξ near 0 and different from 0, the sets Λ(ξ) are not convex.

Lemma 5.2.7. Let Λ be u.s.c.i. at ξ0 and let Λ(ξ0) be closed and convex.Then Λ has the Cesari property at ξ0.

Proof. From the definition of u.s.c.i. we have that for each ε > 0 there existsa δ0 such that for all 0 < δ < δ0

Λ(ξ0) ⊆ Λ(Nδ(ξ0)) ⊆ [Λ(ξ0)]ε.

This chain holds if we take convex hulls of all the sets. Since Λ(ξ0) and [Λ(ξ0]εare closed and convex, we have

Λ(ξ0) ⊆⋂

δ>0

cl co Λ(Nδ(ξ0)) ⊆ [Λ(ξ0)]ε.

Since ε > 0 is arbitrary and Λ(ξ0) is closed, we get that Λ(ξ0) =⋂

δ>0 cl coΛ(Nδ(ξ0)).

The Cesari property will be needed for mappings from (t, x)-space to var-ious euclidean spaces. We shall take the point ξ to be ξ = (t, x). We define anx-delta neighborhood of (t0, x0), denoted by Nδx(t0, x0) as follows,

Nδx(t0, x0) = (t0, x) in R : |x− x0| < δ.

Definition 5.2.8. A mapping Λ has the weak Cesari property at (t0, x0) if

⋂

δ>0

cl co Λ(Nδx(t0, x0)) ⊆ Λ(t0, x0).


We now give an example of a mapping that has the weak Cesari property,but not the Cesari property. Clearly, any map that has the Cesari propertyhas the weak Cesari property.

Example 5.2.9. Let

Λ(t, x) =

(y0, y) : y0 ≥ t2z2, y = z, |z| ≤ 1

t

if t 6= 0

Λ(0, x) = (y0, y) : y0 ≥ 0, y = 0 if t = 0.

All of the sets Λ(t, x) are independent of x. If t 6= 0, then Λ(t, x) is the setof points on or above the segment of the parabola y0 = t2y2, |y| ≤ 1/t in the(y0, y) plane. Since Λ(Nδ(0, 0)) =

⋃0≤t<δ Λ(t, x), we see that Λ(Nδ(0, 0)) is

the open upper half plane plus the origin. Hence⋂

δ>0 cl co Λ(Nδ(0, 0)) =closed upper half plane. But Λ(0, 0) = (y0, y) : y0 ≥ 0, y = 0, so the Cesariproperty fails. On the other hand, Nδx(0, 0) = (0, x) : |x| < δ. Since Λ isindependent of x,

Λ(Nδx(0, 0)) = (y0, y) : y0 ≥ 0, y = 0 = Λ(0, 0).

Hence ⋂

δ>0

cl co (Nδx(0, 0)) = Λ(0, 0),

and the weak Cesari property holds.

5.3 Facts from Analysis

Definition 5.3.1. A set F of functions f in L1[a, b], where [a, b] = t : a ≤t ≤ b, is said to have equi-absolutely continuous integrals if given an ε > 0there is a δ > 0 such that for any Lebesgue measurable set E ⊂ [a, b] withmeas (E) < δ and any f in F ,

|∫

E

fdt| < ε.

Since [a, b] is a finite interval and we are dealing with Lebesgue measure, itfollows that if the functions f in F have equi-absolutely continuous integrals,then there is a constant K > 0 such that for each f in F

∫ b

a

|f |dt < K. (5.3.1)

That is, the set F is bounded in L1[a, b].


Definition 5.3.2. A set F of absolutely continuous functions f defined on[a, b] is said to be equi-absolutely continuous if given an ε > 0 there is aδ > 0 such that for any finite collection of non-overlapping intervals (αi, βi)contained in [a, b], with

∑i |βi−αi| < δ, the inequality

∑i |f(βi)−f(αi)| < ε

holds for all f in F .

We leave it to the reader to verify that a set of absolutely continuousfunctions is equi-absolutely continuous if and only if the derivatives f ′ haveequi-absolutely continuous integrals.

Lemma 5.3.3. Let fn be a sequence of equi-absolutely continuous func-tions defined on an interval [a, b] and converging to a function f . Then f isabsolutely continuous.

Proof. Let (αi, βi), i = 1, . . . , k be a finite collection of non-overlapping in-tervals all contained in [a, b]. Then

k∑

i=1

|f(βi)− f(αi)| ≤k∑

i=1

|f(βi)− fn(βi)|+k∑

i=1

|fn(βi)− fn(αi)|

+

k∑

i=1

|f(αi)− fn(αi)|,

from which the lemma follows.

Remark 5.3.4. The inequality in the proof of Lemma 5.3.3 also shows thatif fn is a sequence of absolutely continuous functions converging uniformlyon [a, b] to a function f , then f is absolutely continuous. The next exampleshows that if the convergence is uniform, then the limit can be absolutelycontinuous even if the fn are not equi-absolutely continuous.

For each positive integer n let fn(x) = sinn2x/n for 0 ≤ x ≤ 1. Eachfunction is absolutely continuous and the sequence fn converges uniformlyto zero. To show that the functions fn are not equi-absolutely continuous wemust exhibit an ε > 0 such that for each δ > 0 there exists a positive integern = n(δ) with the following property. There exists a finite collection of pairwisedisjoint intervals [αi, βi] with Σ|βi −αi| < δ and with Σ|fn(βi)− fn(αi)| > ε.To this end, let ε = 1/2 and let δ > 0 be arbitrary, but less than one. Letn = n(δ) be the smallest positive integer such that π/2n < δ. Let

[0, π/2n2], [2π/2n2, 3π/2n2], . . . ,[2kπ/2n2, (2k + 1)π/2n2], . . . ,

[2(n− 1)π/2n2, (2n− 1)π/2n2]

be a collection of n pairwise disjoint intervals in [0, 1]. The sum of the lengthsof these intervals is π/2n, and

n−1∑

k=0

|fn((2k + 1)π/2n2)− fn(2kπ/2n2))| = n · 1

n= 1 > 1/2.


Thus, the functions fn are not equi-absolutely continuous.For us, the importance of the notion of equi-absolute continuity stems from

the following theorem.

Theorem 5.3.5. Let [a, b] be a finite interval and let fn be a sequenceof functions in L1[a, b]. The sequence of functions fn converges weakly toa function f in L1[a, b] if and only if the following conditions are satisfied:(i) The functions fn have equi-absolutely continuous integrals and (ii) forevery t in [a, b]

limn→∞

∫ t

a

fn(s)ds =

∫ t

a

f(s)ds.

We shall sketch a proof of the theorem, referring the reader to standardtexts for some of the arguments and leaving other parts to the reader.

We first consider the necessity of conditions (i) and (ii). Weak convergenceof fn to f means that for every bounded measurable function g defined on[a, b] ∫ b

a

gfndt→∫ b

a

gfdt. (5.3.2)

Hence by taking g to be the characteristic function of [a, t] we obtain (ii).By taking g to be the characteristic function of a measurable set E we getthat (5.3.2) holds when the integrals are taken over any measurable set E andg = 1.

From this fact, from the absolute continuity of the integral and from

|∫

E

fndt| ≤ |∫

E

(fn − f)dt|+ |∫

E

fdt|

it follows that for each ε > 0 there exists a δ > 0 and a positive integer Nsuch that for n > N and |E| < δ, |

∫E fndt| < ε. Since the integrals of the

finite set of functions f1, . . . , fN are equi-absolutely continuous, (i) follows.Now suppose that (i) and (ii) hold. Condition (i) implies that (5.3.1) holds

with f replaced by fn; that is, the sequence fn is bounded in L1[a, b].Condition (ii) implies that condition (ii) holds when the interval of integrationis taken to be [t′, t′′], where [t′, t′′] is any interval contained in [a, b]. From thisstatement and (i) it follows that (ii) holds when the integrals are taken overany measurable set E in [a, b]. It then follows that (5.3.2) holds for any stepfunction g. If g is an arbitrary measurable function, then g is the almosteverywhere limit of a sequence of step functions σk. By Egorov’s theorem,for every δ > 0 there is a set E of measure < δ such that on the complement ofE relative to [a, b], σk → g uniformly. From the last observation, from (5.3.2)with g replaced by a step function, the uniform L1 bound for the functionsfn, and the equi-absolute continuity of the fn there follows the validity of(5.3.2) for arbitrary bounded measurable g.

Lemma 5.3.6. Let fn be a sequence of functions in L1[a, b] that converges


weakly to a function f in L1[a, b]. Then for each positive integer j there existsa positive integer nj, a set of integers 1, . . . , k, where k depends nj, and a setof real numbers α1j , . . . , αkj satisfying

αij ≥ 0, i = 1, . . . , k

k∑

i=1

αij = 1 (5.3.3)

such that nj+1 > nj + k and such that the sequence

ψj =

k∑

i=1

αijfnj+i (5.3.4)

converges strongly in L1[a, b] to f .

Proof. The set cl co fn, where closure is taken in L1[a, b], is a stronglyclosed convex set in the Banach space L1[a, b]. Therefore, by Mazur’s theorem(Lemma 4.7.6) the set cl co fn is weakly closed. Hence f is in this setand can be approximated to any degree of accuracy in L1[a, b] by a convexcombination of functions in fn.

We now define the sequences nj and ψj inductively. Let j = 1. Then thereexists a convex combination of the functions in fn that approximates f inthe L1[a, b] norm to within 1/2. By choosing some of the coefficients in theconvex combination to be zero, we may suppose that we are taking a convexcombination of consecutive functions fn1+1, . . . , fn1+k, where k depends onn1. That is, there exist real numbers αi1, i = 1, . . . , k such that (5.3.3) holdsand the function ψ1 defined by (5.3.4) satisfies

∫ b

a

|ψ1 − f |dt < 1/2.

Now suppose that n1, . . . , nj and ψ1, . . . , ψj have been defined and that

for i = 1, . . . , j,∫ b

a |ψi − f |dt < 2−i. Let nj+1 be any integer greater thannj + k. Then cl co fnj+1, fnj+2, . . . is a weakly closed set in L1[a, b], andso contains f . We can then apply the argument used for j = 1 to this set toobtain a function ψj+1 defined by (5.3.4) such that

∫ b

a

|ψj+1 − f |dt < 1/2j+1.

Lemma 5.3.7. Let sn be a sequence of points in Rk converging to s. Let

nj be a subsequence of the positive integers and let

σj =

k∑

i=1

αijsnj+i,

where the αij and k are as in (5.3.3). Then σj → s.


We leave the proof as an exercise for the reader.We conclude this section with the following result.

Lemma 5.3.8. Let I be a compact real interval and let h : (t, ξ) → h(t, ξ) bea continuous mapping from I ×R

r to R1. Let vk and wk be sequences in

Lp[I,Rr], 1 ≤ p ≤ ∞ such that ‖vk‖p ≤M and ‖wk‖p ≤ M for some M > 0and such that (vk − wk) → 0 in measure on I. Then

h(t, vk(t)) − h(t, wk(t)) → 0

in measure on I.

Proof. We must show that for arbitrary η > 0 and ε > 0 there exists a positiveinteger K such that if k > K, then

meas t : |h(t, vk(t))− h(t, wk(t))| ≥ η < ε.

LetA =M(3/ε)1/p, (5.3.5)

where we interpret 1/∞ as zero. Let GA denote the set of points ξ in Rr such

that |ξ| ≤ A. Since h is uniformly continuous on I ×GA, there exists a δ > 0such that if ξ and ξ′ belong to GA and |ξ − ξ′| < δ, then

|h(t, ξ)− h(t, ξ′)| < η (5.3.6)

for all t in I.Let Ikv denote the set of points in I at which |vk(t)| > A and let Ikw

denote the set of points in I at which |wk(t)| > A. Let Ik = Ikv ∪ Ikw and let

Gk = t : |vk(t)− wk(t)| ≥ δ.

Then for t 6∈ Ik ∪Gk,

|h(t, vk(t))− h(t, wk(t)| < η.

Therefore, to establish the lemma we must show that for k sufficiently large,meas (Ik ∪Gk) < ε.

For 1 ≤ p <∞ we have

M ≥(∫

I

|vk(t)|pdt)1/p

≥(∫

Ikv

Apdt

)1/p

= A(meas Ikv)1/p.

From this and (5.3.5) we get that meas (Ikv) < ε/3. Similarly, meas (Ikw) <ε/3.

Since (vk − wk) → 0 in measure, for sufficiently large k, meas(Gk) < ε/3.Thus, meas(Ik ∪ Gk) < ε for k sufficiently large. For p = ∞, we have from(5.3.5) that A = M , so meas Ik = 0. Since (vk − wk) → 0 in measure,there exists a positive integer K such that for k > K, meas Gk < ε. Hencemeas(Ik ∪Gk) < ε, and the lemma is proved.


5.4 Existence via the Cesari Property

In this section we shall prove existence theorems for ordinary and relaxedproblems without assuming the constraint sets to be compact. Instead, weshall assume that certain set valued mappings possess the weak Cesari prop-erty. The assumptions about the data of the problem will be less restric-tive than in the case of problems with compact constraints. The functionsf = (f0, f1, . . . , fn) will not be required to be Lipschitz in the state variable.The constraint mappings Ω will be allowed to depend on (t, x), rather than ont alone, and will be assumed to be upper semicontinuous rather than u.s.c.i.

The definition of the relaxed problem for the case of non-compact con-straints given in Section 3.5 is consistent with the definition for compact con-straints given in Section 3.2 in the sense that the sets of relaxed trajectorieswill be the same under both definitions.

To facilitate our discussion of problems with constraints that are not as-sumed to be compact, we introduce some notation. Let p and v be measurablefunctions of the form

p = (p1, . . . , pn+2) v = (u1, . . . , un+2),

where the pi are real valued measurable functions and the ui are vector valuedfunctions with range in U . Let

v = (p, v) = (p1, . . . , pn+2, u1, . . . , un+2) (5.4.1)

π = (π1, . . . , πn+2) ζ = (z1, . . . , zn+2) πi ∈ R, zi ∈ Rm

z = (π, ζ)

Πn+2 =

π = (π1, . . . , πn+2) : πi ≥ 0,

n+2∑

i=1

πi = 1

f ir(t, x, z) ≡ f i

r(t, x, π, ζ) ≡n+2∑

j=1

πjf i(t, x, zj) zj ∈ Rm i = 0, 1, . . . , n+ 2,

and let fr = (f0r , fr) = (f0

r , f1r , . . . , f

nr ).

In terms of the notation just introduced, the relaxed problem correspond-ing to Eqs. (5.1.1) to (5.1.3) can be written as:

Minimize

J(ψ, v) = g(e(ψ)) +

∫ t1

t0

f0r (t, ψ(t), v(t))dt (5.4.2)

subject to the differential equation

dψ

dt= fr(t, ψ(t), v(t)), (5.4.3)


end condition(t0, ψ(t0), t1, ψ(t1)) ∈ B (5.4.4)

and control constraints on v(t) = (p(t), v(t))

p(t) ∈ Πn+2 ui(t) ∈ Ω(t, ψ(t)) i = 1, . . . , n+ 2. (5.4.5)

If we setΩ(t, x) = Ω(t, x) × . . .×

n+2 timesΩ(t, x)

then we may write (5.4.5) as

v(t) ∈ Ω(t, ψ(t)). (5.4.6)

We emphasize the observation made in Remark 3.5.8 that the relaxedproblem just formulated can be viewed as an ordinary problem with controlsv = (p, u1, . . . , un+2).

We assume that the f , g,B, and Ω satisfy the following:

Assumption 5.4.1. (i) The function f = (f0, f) = (f0, f1, . . . , fn) is de-fined on a set G = I × X × U , where I is a real compact interval, X isa closed interval in R

n and U is an open interval in Rm.

(ii) The function f is continuous on G.

(iii) The function f0 is lower semi-continuous on G and there exists an in-tegrable function β on I such that f0(t, x, z) ≥ β(t) for all (t, x, z) inG.

(iv) The terminal set B is a closed set of points (t0, x0, t1, x1) in Rn+2 with

t0 < t1 and with (t0, x0) and (t1, x1) in R = I × X .

(v) The function g is lower semi-continuous on B.

(vi) Ω is a mapping from R = I × X to subsets Ω(t, x) of U that is uppersemi-continuous on R.

We next recall sets introduced in Section 4.4 that will again play a crucialrole in our existence theorems.

Definition 5.4.2. For each (t, x) in R let

Q+(t, x) = (y0, y) : y0 ≥ f0(t, x, z) y = f(t, x, z), z ∈ Ω(t, x)

and let

Q+r (t, x) = (y0, y) : y0 ≥ f0

r (t, x, π, z) y = fr(t, x, π, z),

π ∈ Πn+2, ζ = (z1, . . . , zn+2) zi ∈ Ω(t, x), i = 1, . . . , n+ 2.


Lemma 5.4.3. If the mapping Q+r has the weak Cesari property (Cesari

property) at (t, x) and if Q+(t, x′) is convex for all x′ in a neighborhood of x,then Q+ has the weak Cesari property (Cesari property) at (t, x).

Proof. From the definition of Q+r (t, x) we have that Q+

r (t, x) = co Q+(t, x).Thus, if Q+(t, x) is convex, then Q+

r (t, x) = Q+(t, x). Hence for δx sufficientlysmall,

Q+(Nδx(t, x) =⋃

|x′−x|<δx

Q+(t, x′) =⋃

|x′−x|<δx

Q+r (t, x

′) = Q+r (Nδx(t, x)).

Thus, cl co Q+(Nδx(t, x)) = cl co Q+r (Nδx(t, x)). Therefore, since Q

+r has the

weak Cesari property at (t, x)

⋂

δx>0

cl co Q+(Nδx(t, x)) =⋂

δx>0

cl co Q+r (Nδx(t, x)) = Q+

r (t, x) = Q+(t, x).

The equality of the leftmost and rightmost sets shows that Q+ has the weakCesari property.

To prove the statements about the Cesari property one considers points(t′, x′) such that |(t′, x′)− (t, x)| < δ.

Theorem 5.4.4. Let Assumption 5.4.1 hold. Let the set of admissible relaxedpairs (ψ, µ) be non-empty. Let the relaxed problem have a minimizing sequence(ψn, µn) whose trajectories lie in a compact set R0 ⊆ R and are equi-absolutely continuous. Let the mapping Q+

r possess the weak Cesari property atall points of R0. Then the relaxed problem has an optimal solution. If Q+(t, x)is convex at all points (t, x) in R, then there exists an ordinary admissible pairthat is optimal for both the ordinary and relaxed problems.

To invoke Theorem 5.4.4 in a specific problem, we must find a minimizingsequence all of whose trajectories lie in a compact set and are equi-absolutelycontinuous, and that the mapping Q+ or Q+

r has the weak Cesari property.In a specific problem this may or not be easy to do. Lemma 4.3.14 gives asufficient condition for all trajectories of the ordinary or relaxed system tolie in a compact set. While this condition is useful in problems with compactconstraints, it is not as applicable to problems in which the constraint setsare not assumed to be compact. For such problems a sufficient condition fortrajectories (ordinary or relaxed) in a minimizing sequence to lie in a compactset will be given in Lemma 5.4.14. Before presenting the proof of Theorem 5.4.4we shall give a condition in terms of the data of the problem that is sufficientfor the mappings Q+ and Q+

r to have the weak Cesari property. Part of thiscondition is sufficient for the trajectories in a minimizing sequence to be equi-absolutely continuous.

Definition 5.4.5. Let G be a real valued nonnegative function defined onG = R×U . Let F be a function defined on G with range in R

n. Then F is said


to be of slower growth than G uniformly on G if for each ε > 0 there exists apositive number ν such that if |z| > ν, then

|F (t, x, z)| < εG(t, x, z).

Lemma 5.4.6. Let f = (f0, f) be as in Assumption 5.4.1 with β ≡ 0. Let fbe of slower growth than f0, uniformly on G, and let the function identicallyequal to one be of slower growth than f0, uniformly on G. Let the constraintmapping Ω be upper semi-continuous on R. Then:

(i) At each (t, x) the mapping Q+r has the weak Cesari property on R.

(ii) If at each (t, x) in R the set Q+(t, x) is convex, then the mapping Q+

has the weak Cesari property on R.

Remark 5.4.7. The functions f0r , fr need not satisfy the growth condition of

the lemma, even though the functions f0 and f do. To see this, let f0 = z2,f = z. Then f0

r = π1(z1)2 + π2(z2)

2 and fr = π1z1 + π2z2. For the relaxedproblem take the sequence of controls (πk, zk) = (0, 1, k, 0), k = 1, 2, . . ..Then |(πk, zk)| → ∞ and f0

r (πk, zk) = 0, fr(πk, zk) = 0, so the functionidentically one cannot be of slower growth than f0

r .

Proof of Lemma. By Lemma 5.4.3, if (i) holds, then so does (ii). Hence weneed only prove (i). We first note that as a consequence of the definition andof Lemma 3.2.10, that the upper semi-continuity of the mapping Ω impliesthat for each (t, x) in R, the sets Ω(t, x) and

D = (t, x, z) : (t, x) ∈ R, z ∈ Ω(t, x) (5.4.7)

are closed. To prove conclusion (i) we must show that if

y = (y0, y) ∈⋂

δx>0

cl co Q+r (Nδx(t, x)) (5.4.8)

then y ∈ Q+r (t, x).

Since for any collection Sα of sets we have⋃

α co Sα ⊆ co⋃

α Sα andsince Q+

r (t, x) = co Q+(t, x), the following holds:

Q+r (Nδx(t, x)) ≡

⋃

|x′−x|<δx

Q+r (t, x

′) =⋃

|x′−x|<δx

co Q+(t, x′)

⊆ co⋃

|x′−x|<δx

Q+(t, x′) = co Q+(Nδx(t, x)).

Thus, cl co Q+r (Nδx(t, x)) ⊆ cl co Q+(Nδx(t, x)), and so

⋂

δx>0

cl co Q+r (Nδx(t, x)) ⊆

⋂

δx>0

cl co Q+(Nδx(t, x)).


Hence, if y is as in (5.4.8), then

y = (y0, y) ∈⋂

δx>0

cl co Q+(Nδx(t, x)). (5.4.9)

Let y = (y0, y) satisfy (5.4.9). Then y0 ≥ 0. Let δk be a decreasingsequence of real numbers such that δk → 0. Then for each positive integer kthere exists a point yk = (y0k, yk) with

yk ∈ co Q+(Nδkx(t, x)) (5.4.10)

and |yk − y| < 1/k. In other words, the sequence yk satisfies (5.4.10) and

yk = (y0k, yk) → (y0, y) = y. (5.4.11)

From (5.4.10) and the Caratheodory theorem it follows that for each integerk there exist real numbers αk,1, . . . , αk,n+2 with

αki ≥ 0 and

n+2∑

i=1

αki = 1, (5.4.12)

points (t, xk1), . . . , (t, xk,n+2) in R, and points yk,1, . . . , yk,n+2 such that

|xki − x| < δk and yki ∈ Q+(t, xki) i = 1, . . . , n+ 2 (5.4.13)

and

yk =

n+2∑

i=1

αkiyki. (5.4.14)

From the second relation in (5.4.13) it follows that there exist points zk,1, . . .,zk,n+2 with zki ∈ Ω(t, xki) i = 1, . . . , n+ 2 such that

y0ki ≥ f0(t, xki, zki) yki = f(t, xki, zki) i = 1, . . . , n+ 2. (5.4.15)

It follows from (5.4.12) that for each i, the sequence αki is bounded.Hence there exists a subsequence of k, which we again label as k andnon-negative numbers α1, . . . , αn+2 such that

αki → αi αi ≥ 0

n+2∑

i=1

αi = 1. (5.4.16)

Since

y0k =

n+2∑

i=1

αkiy0ki and y

0k → y0,

the sequence y0k is bounded. Since αki ≥ 0 and y0ki ≥ 0, it follows that each


sequence αkiy0ki, i = 1, . . . , n + 2 is bounded. Hence for each i there exists

a subsequence αkiy0ki and a nonnegative number ηi such that

αkiy0ki → ηi ηi ≥ 0.

From the last relation in (5.4.16) it follows that the set of indices for whichαi > 0 is non-void. Let i = 1, 2, . . . , s denote this set. Then

y0ki →ηiαi

i = 1, . . . , s. (5.4.17)

We assert that this implies that the sequence (zk1, . . . , zks) is bounded. Ifthis assertion were false, there would exist a subsequence and an index i, suchthat |zki| → ∞. Since one is of slower growth than f0, this would imply thatfor each ε > 0 and all sufficiently large k

1

ε≤ f0(t, xki, zki) ≤ y0ki.

Thus, y0ki would be unbounded, contradicting (5.4.17). Hence (zk,1, . . . , zk,s)is bounded.

Since (zk,1, . . . , zk,s) is bounded there exists a subsequence k andpoints z1, . . . , zs such that zki → zi, i = 1, . . . , s. Also xki → x, so for eachi = 1, . . . , s, the sequence of points (t, xki, zki) in D converges to a point(t, x, zi). Since D is closed, (t, x, zi) ∈ D, and so zi ∈ Ω(t, x) for i = 1, . . . , s.Since f is continuous, for i = 1, . . . , s

f(t, xki, zki) → f(t, x, zi) zi ∈ Ω(t, x). (5.4.18)

We now consider indices i > s. If for an index i > s the sequence zkiis unbounded, we may select a subsequence such that |zki| → ∞. Then, since|f | is of slower growth than f0, for each ε > 0 and k sufficiently large

αki|f(t, xki, zki)| < εαkif0(t, xki, zki) ≤ εαkiy

0ki.

Since αkiy0ki → ηi and ε > 0 is arbitrary,

αkif(t, xki, zki) → 0. (5.4.19)

If for an index i > s the sequence zki is bounded, then since xki → x,the sequence of points (t, xki, zki) is bounded. Since f is continuous the setf(t, xki, zki) is bounded. Since αki → 0, the relation (5.4.19) holds in thiscase also.

It now follows from Eqs. (5.4.14) to (5.4.16), (5.4.18), (5.4.19), and αi = 0for i > s that

yk =

n+2∑

i=1

αkiyki →s∑

i=1

αif(t, x, zi),


where zi ∈ Ω(t, x), αi > 0 and∑αi = 1. But yk → y, so

y =

s∑

i=1

αif(t, x, zi). (5.4.20)

It follows from Eqs. (5.4.11), (5.4.14), (5.4.16), and (5.4.5) and the lowersemicontinuity of f0 that

y0 =

n+2∑

i=1

ηi ≥s∑

i=1

ηi =

s∑

i=1

limk→∞

(αkiy0ki) (5.4.21)

≥s∑

i=1

lim infk→∞

(αkif0(t, xki, zki))

≥s∑

i=1

αif0(t, x, zi),

where zi ∈ Ω(t, x) and αi ≥ 0 for i = 1, . . . , s and∑αi = 1. From (5.4.20) and

(5.4.21) we get that y = (y0, y) is in co Q+(t, x). But Q+r (t, x) = co Q+(t, x),

so y ∈ Q+r (t, x). Thus, Q

+r has the weak Cesari property at (t, x).

Remark 5.4.8. If we consider neighborhoods Nδ(t, x) and in (5.4.12) con-sider sequences of points (tk1, xk1), . . . , (tk,n+2, xk,n+2) such that |(tki, xki)−(t, x)| < δk, then the preceding argument gives the stronger result that themapping Q+ satisfies the Cesari property.

Lemma 5.4.9. Let Assumption 5.4.1 hold with β ≡ 0 in (iii). Let f be ofslower growth than f0, uniformly on G = R×U . Let there exist a minimizingsequence of either the ordinary or the relaxed problem whose trajectories lie ina compact subset of R. Then these trajectories are equi-absolutely continuous.

Proof. Since the trajectories of the minimizing sequence lie in a compact setand B is closed, the set of endpoints is compact. Since g is lower semicontinuouson B, the function g is bounded below on the set of endpoints. Hence the setof integrals in (5.1.1) or (5.4.2) evaluated along the pairs in a minimizingsequence is bounded above. We denote this bound by A.

We first consider the ordinary problem with a minimizing sequence(ϕn, un) of admissible pairs. We shall show that the integrals

∫ϕ′ndt are

equi-absolutely continuous. Let ε > 0 be given and let η = ε/2A. Then thereexists a positive number ν such that if |un(t)| > ν, then

|f(t, ϕn(t), un(t))| < ηf0(t, ϕn(t), un(t)).

Also, since all points (t, φn(t)) are in a compact set, there exists a positiveconstant Kη such that if |un(t)| ≤ ν, then

|f(t, ϕn(t), un(t))| ≤ Kη.


Thus, for each n and measurable set E ⊆ I,∫

E

|ϕ′n(t)|dt ≤ Kη meas(E) + η

∫

E

f0(t, ϕn(t), un(t))dt (5.4.22)

≤ Kη meas(E) + ηA.

If we take meas(E) < ε/2Kη and recall that η = ε/2A, we obtain the assertedequi-absolute continuity.

For the relaxed problem with ε as before and η = ε/2(n+ 2)A we have

∫

E

|ψ′n(t)|dt ≤

∫

E

n+2∑

i=1

pi(t)|f(t, ψn(t), uni(t))|dt

≤∫

E

n+2∑

i=1

pi(t)(Kη + ηf0(t, ψn(t), uni(t)))dt

≤ Kη meas(E) + η

∫

E

(n+2∑

i=1

pi(t)f0(t, ψn(t), uni(t))

)dt

≤ Kη meas(E) + η(n+ 2)A.

From the preceding we obtain the equi-absolute continuity, as from (5.4.22).If the control problem is of Mayer type, that is, f0 ≡ 0, then, in general,

the condition “f is of slower growth than f0” will not hold.

The proof of Lemma 5.4.9, however, shows that for the Mayer problem,the following is true. If there exists a nonnegative function F 0 defined on Gsuch that: (i) f is of slower growth than F 0, uniformly on G and (ii) thereexists a constant A > 0 such that for all (φk, uk) in u minimizing sequence

∫ t1

t0

F 0(t, ψk(t), uk(t)dt ≤ A,

then the trajectories φk are equi-absolutely continuous. A similar statementholds for relaxed trajectories ψk in a minimizing sequence.

The next lemma has a corollary that gives a sufficient condition that doesnot involve f0 for trajectories in a minimizing sequence to be absolutely con-tinuous.

Lemma 5.4.10. Let Φ be a positive, continuous non-decreasing function de-fined on [0,∞) such that Φ(ξ) → +∞ as ξ → +∞. Let θk be a sequenceof real valued functions such that θk is defined and integrable on [t0k, t1k] andsuch that θk(t) = 0 for t 6∈ [t0,k, t1k]. Let there exist a constant C such that

∫ t1k

t0k

|θk(t)|Φ(|θk(t)|)dt ≤ C. (5.4.23)

Then the functions θk have equi-absolutely continuous integrals.


Proof. Since Φ is continuous and each θk is measurable, it follows that thefunctions Φ(|θk| are measurable.

Let η > 0 be given. Then there exists a positive number Kη such that ifξ > Kη, then Φ(ξ) > 1/η. For each k let

E1k = t : |θk(t)| ≤ KηE2k = t : |θk(t)| > Kη.

If t ∈ E2k, then Φ(|θk(t)|) > 1/η. Therefore,

|θk(t)| =|θk(t)|Φ(|θk(t)|)

Φ(|θk(t)|)≤ Kη + η|θk(t)|Φ(|θk(t)|).

Hence, for any set E ⊆ I∫

E

|θk(t)|dt ≤ Kη meas E + η

∫

E

|θk(t)|Φ(|θk(t)|)dt

≤ Kη meas E + ηC.

From this and the argument following (5.4.22) we get the asserted equi-absolute continuity.

Corollary 5.4.11. Let ϕk be the trajectories in a minimizing sequence suchthat each component dϕi

k/dt, i = 1, . . . , n of the sequence of derivatives sat-isfies (5.4.23). Then the sequence ϕk is equi-absolutely continuous. Similarstatements hold for relaxed minimizing sequences ψk.Corollary 5.4.12. Let there exist a constant C and a real number 1 < p <∞such that the components of the trajectories of a minimizing sequence satisfy

∫ t1k

t0k

|dφik/dt|pdt ≤ C.

Then the trajectories φk of the minimizing sequence are equi-absolutely con-tinuous. A similar statement holds for the trajectories ψk of a relaxed min-imizing sequence.

Proof. Write |dφik/dt|p = |dφik/dt||dφik/dt|p−1.

Remark 5.4.13. The problem of minimizing (5.1.1) is the same as the prob-lem of minimizing (5.1.1) with integrand f0 − β. Similarly the problem ofminimizing (5.4.2) is the same as the problem of minimizing (5.4.2) with in-tegrand f0 − β. Hence in Assumption 5.4.1 there is no loss of generality inassuming β ≡ 0; that is, f0 ≥ 0.

The variational problem of finding the curve that minimizes the distancebetween the points (0, 0) and (1, 0) in the (t, x) plane has the following form

as a control problem. Minimize1∫0

(1 + u2)1/2dt subject to dx/dt = u, x(0) =

x(1) = 0 and u(t) in R1. Lemma 4.3.14 is not applicable because the dynamics

do not satisfy condition (4.3.16). The next lemma is applicable to this problem.


Lemma 5.4.14. Let Assumption 5.4.1 hold with the added condition that Bis compact. Let there exist a constant K > 0 such that for i = 1, . . . , n

|f i(t, x, z)| ≤ Kf0(t, x, z)

for all (t, x, z) in G. Let the notation be as in Section 3.5. Let (ψn, vn) be arelaxed minimizing sequence with (ψn, vn) defined on [t0n, t1n]. There exists acompact set R0 ⊆ R such that all trajectories ψn are contained in R0.

Proof. It follows from (5.4.1) and |f i| ≤ Kf0 that |f ir(t, x, z)| ≤ Kf0

r (t, x, z),and so

|fr(t, x, z)| ≤ K1f0r (t, x, z) (5.4.24)

for some constant K1 > 0. Since B is compact, there exists a closed interval[a, b] such that if (t0, x0, t1, x1) ∈ B, then [t0, t1] ⊆ [a, b]. Since g is lowersemicontinuous on B and f0

r ≥ 0, it follows that there exists a constant A > 0such that for all (ψn, vn) of the minimizing sequence

∫ t1n

t0n

f0r (t, ψn(t), vn(t))dt ≤ A. (5.4.25)

For any admissible trajectory ψ, let Ψ(t) = |ψ(t)|2 + 1. Then Ψ′(t) =2〈ψ(t), fr(t, ψ(t), v)〉. From the Cauchy-Schwarz inequality we get that

−2|ψ(t)||fr(t, ψ(t), v(t))| ≤ Ψ′(t) ≤ 2|ψ(t)||fr(t, ψ(t), v(t))|.

From this and (5.4.24) we get that

−2(|ψ(t)|2 + 1)f0r (t, ψ(t), v(t) ≤ Ψ′(t) ≤ 2(|ψ|2 + 1)f0

r (t, ψ(t), v(t)).

Hence

−2f0r (t, ψ(t), v(t)) ≤

Ψ′(t)

Ψ(t)≤ 2f0

r (t, ψ(t), v(t)).

Now let (ψ, v) be an element of the minimizing sequence. If we then integratethe preceding relation and use (5.4.25) and f0

r ≥ 0, we get that for t0n ≤ t ≤t1n,

−2A ≤ logΨ(t)

Ψ(t0n)≤ 2A.

Thus,e−2A ≤ (|ψn(t)|2 + 1)(|ψ(t0n)|2 + 1)−1 ≤ e2A.

Since all points (t0n, ψn(t0n)) lie in a compact set, it follows that all trajectoriesψn of the minimizing sequence lie in a compact set R0 ⊆ R.

Remark 5.4.15. If g ≡ 0, then (5.4.25) follows because (ψn, vn) is a min-imizing sequence. To show that all trajectories ψn lie in a compact set, theassumption that B is compact is not needed; we need only assume that allinitial points (t0, x0) lie in a compact set.


The next theorem is an immediate consequence of Theorem 5.4.4,Lemma 5.4.6, and Lemma 5.4.9. In a specific example, the hypotheses of The-orem 5.4.16 are usually easier to verify than those of Theorem 5.4.4.

Theorem 5.4.16. Let Assumption 5.4.1 hold. Let the set of admissible relaxedpairs (ψ, µ) be non-empty and let all the trajectories in a minimizing sequencelie in a compact set R0 ⊆ R. Let f and the function identically equal to onebe of slower growth than f0, uniformly on R0 × U. Then the relaxed problemhas a solution. If the sets Q+(t, x) are convex, then there exists an ordinarypair that is optimal for both the ordinary and relaxed problems.

Proof of Theorem 5.4.4. The last conclusion of the theorem follows fromCorollary 4.4.3, once the existence of an optimal relaxed solution is shown.We proceed to do this.

In Remark 3.5.8 and in the discussion preceding Assumption 5.4.1, it waspointed out that the relaxed problem Eqs. (5.4.1) to (5.4.5) can be viewed as anordinary problem. The notation henceforth will be as in Eqs. (5.4.1) to (5.4.5).

The function fr satisfies Assumption 5.4.1. Since Πn+2 is constant and since

Ω is upper semi-continuous, so is Ω. All other hypotheses of Assumption 5.4.1clearly hold for the relaxed problem treated as an ordinary problem.

In the proof we shall select subsequences of various sequences and shallrelabel the subsequence with the labeling of the original sequence. By Re-mark 5.4.13 we may assume that f0 ≥ 0.

Let (ψk, vk) be a minimizing sequence. The trajectory ψk is defined on an

interval [t0k, t1k]. We extend ψk to a function ψk defined on all of I by setting

ψk(t) = ψk(t0k) if t ≤ t0k , and ψk(t) = ψk(t1k) if t ≥ t1k. The sequenceof end points e(ψk) lies in the compact set R0 ∩ B. Hence there exists asubsequence ψk of the minimizing sequence and a point (t0, x0, t1, x1) in Bsuch that e(ψk) → (t0, x0, t1, x1), or

t0k → t0 ψk(t0k) → x0 t1k → t1 ψk(t1k) → x1. (5.4.26)

The functions ψk are equi-absolutely continuous by hypothesis. The

functions ψk are constant outside of the intervals [t0k, t1k], and so areequi-absolutely continuous on I. Since they all lie in R0, they are uniformlybounded. Hence by Ascoli’s theorem, there exists a subsequence ψk and a

continuous function ψ on I such that

ψk → ψ uniformly on I. (5.4.27)

Since ψ(tik) = ψ(tik), i = 0, 1, it follows from (5.4.26) and (5.4.27) that

ψ(t0k) → ψ(t0) = x0 and ψ(t1k) → ψ(t1) = x1.

If t < t0, then for k sufficiently large, t < t0k and ψk(t) = ψ(t0k). Hence

ψ(t) = ψ(t0) for t < t0. Similarly, ψ(t) = ψ(t1) for t > t1.


Since the functions ψk are equi-absolutely continuous and converge uni-

formly to ψ on I, the function ψ is absolutely continuous. Hence it is differ-entiable almost everywhere and

ψ(t) = ψ(t0) +

∫ t

t0

ψ′(s)ds.

Let I = [a, b]. Since ψ is constant on [a, t0] and on [t1, b], we have ψ′(t) = 0on (a, t0) and (t1, b). Hence we can write the preceding relation as

ψ(t) = ψ(a) +

∫ t

a

ψ′ds

for all t in [a, b]. We also have

ψk(t) = ψk(a) +

∫ t

a

ψ′kds.

It then follows from (5.4.27) that for all t in I,∫ t

a

ψ′kds→

∫ t

a

ψ′kds.

Since the functions ψk are equi-absolutely continuous, so are the integrals

of the functions ψ′k. Hence by Theorem 5.3.5

ψ′k → ψ′ weakly in L1[a, b].

We also note that ψ is in R0 and (t0, ψ(t0), t1, ψ(t1)) is in B.We summarize the preceding results as Step 1 of the proof.

Step 1. There exists a subsequence ψk of the extended functions of the min-

imizing sequence and an absolutely continuous function ψ such that (5.4.27)

holds and ψ′k → ψ′ weakly in L1[a, b]. The function ψ lies in R0 and is

the extension of a function defined on an interval [t0, t1] ⊆ [a, b]. Moreover,

(t0, ψ(t0), t1, ψ(t1)) is in B.Step 2. Let

I(ψ, v) ≡∫ t1

t0

f0r (s, ψ(s), v(s))ds.

In the first paragraph of the proof of Lemma 5.4.9 we showed that the sequenceI(ψk, vk) is bounded above. Since f0 ≥ 0 and all intervals [t0k, t1k] arecontained in the fixed compact interval I, the sequence I(ψk, vk) is boundedbelow. Hence there exists a subsequence (ψk, vk) and a real number γ suchthat I(ψk, vk) → γ. The subsequence (ψk, vk) is a subsequence of the sub-sequence in Step 1.

Henceforth we let ψ denote the restriction of ψ to the interval [t0, t1]. Thatis,

ψ(t) ≡ ψ(t) t ∈ [t0, t1].


Step 3. There exists a real valued function λ that is integrable on [t0, t1] suchthat (λ(t), ψ′(t)) ∈ Q+

r (t, ψ(t)) a.e. on [t0, t1] and such that∫ t1

t0

λ(s)ds ≤ γ, (5.4.28)

where γ is as in Step 2.Since ψ′

k → ψ′ weakly in L1[a, b], Lemma 5.3.6 gives the following state-ment. For each integer j there exists an integer nj , a set of integers 1, . . . , kwhere k depends on j and a set of real numbers α1j , . . . , αkj satisfying

αij ≥ 0, i = 1, . . . , k and

k∑

i=1

αij = 1

such that nj+1 > nj + k and such that the sequence

ωj =

k∑

i=1

αij ψ′nj+i

converges to ψ′ in L1[a, b]. Recall that for every positive integer q, if t 6∈[t0q, t1q] then ψ′

q(t) = 0 and that vq and ψq are defined on [t0q, t1q]. If fort 6∈ [t0q, t1q] we define fr(t, ψq(t), vq(t)) to be zero and recall that on [t0q, t1q],

ψ(t) = ψ(t), we can write ωj as follows:

ωj(t) =

k∑

i=1

αijfr(t, ψnj+i, vnj+i). (5.4.29)

Since ωj → ψ′ in L1[a, b], there is a subsequence ωj such that

ωj(t) → ψ′(t) a.e. in [a, b]. (5.4.30)

Corresponding to the sequence (5.4.30) we define a sequence λj as follows:

λj(t) =

k∑

i=1

αijf0r (t, ψnj+i(t), vnj+i(t)), (5.4.31)

where if t /∈ [t0q, t1q] we set f0(t, ψq(t), vq(t)) = 0 and where for each j thenumbers αij , the indices nj + i, and the functions ψnj+i and vnj+i are as in(5.4.29). Note that if t 6∈ [t0, t1], then there exists a positive integer j0 = j0(t)such that if j > j0, then λj(t) = 0.

Letλ(t) = lim inf λj(t). (5.4.32)

Since f0 ≥ 0 it follows that λ(t) ≥ 0. If t 6∈ [t0, t1], then λ(t) = 0. Therefore,if we set f0

rq(t) ≡ f0r (t, ψq(t), vq(t)) and use Fatou’s Lemma we get

∫ t1

t0

λdt =

∫ b

a

λdt ≤ lim infj→∞

[k∑

i=1

αij

∫ b

a

f0r,nj+idt

]


= lim infj→∞

[k∑

i=1

αij

∫ t1,nj+i

t0,nj+i

f0r,nj+idt

]

= lim infj→∞

[k∑

i=1

αijI(ψnj+i, vnj+i)

].

From Step 2 we have I(ϕnj+i, unj+i) → γ as j → ∞. It then follows fromLemma 5.3.7 that

limj→∞

k∑

i=1

αijI(ψnj+i, vnj+i) = γ,

which establishes (5.4.28). Since λ ≥ 0, it follows that λ is in L1[t0, t1], andso is finite a.e.

We now show that (λ(t), ψ′(t)) ∈ Q+r (t, ψ(t)) a.e. on [t0, t1]. Let T1 denote

the set of points in [t0, t1] at which λ(t) is finite and ωj(t) → ψ′(t). The setT1 has the full measure |t1 − t0|. For each positive integer k define a set Ek asfollows:

Ek = t : t ∈ [t0k, t1k], vk(t) 6∈ Ω(t, ϕk(t)).Then meas Ek = 0. Let E denote the union of the sets Ek. Then meas E = 0.Let T2 denote the set of points in [t0, t1] that do not belong to E. Let T ′ =T1 ∩ T2. Then meas T ′ = |t1 − t0|.

Let t be a fixed point in T ′, t 6= ti, i = 0, 1. There exists a subsequenceλj(t), which in general depends on t such that λj(t) → λ(t). For the corre-sponding sequence ωj(t) we have from (5.4.30) and the fact that t is interiorto (t0, t1) that ωj(t) → ψ′(t). Since t is interior to (t0, t1) and tik → ti, i = 0, 1it follows that there exists a positive integer j0 such that if j > j0, thent ∈ (t0,nj+i, t1,nj+i). For each δ > 0 there exists a positive integer k0 suchthat if k > k0, then |ψk(t)− ψ(t)| < δ. Thus, for k > k0

(t, ψk(t)) ∈ Nδx(t, ψ(t)).

Therefore, for j sufficiently large

fr(t, ψnj+i(t), vnj+i(t)) ∈ Q+r (Nδx(t, ψ(t))),

where fr = (f0r , fr). Therefore, by (5.4.29) and (5.4.31)

(λj(t), ωj(t)) ∈ co Q+r (Nδx(t, ψ(t))).

Since λj(t) → λ(t) and ωj(t) → ψ′(t) we have that

(λ(t), ψ′(t)) ∈ cl co Q+r (Nδx(t, ψ(t))).

Since δ is arbitrary, (λ(t), ψ′(t)) is in cl co Q+r (Nδx(t, ψ(t)) for each δ > 0,

and hence(λ(t), ψ′(t)) ∈

⋂

δx>0

cl coQ+r (Nδx(t, ϕ(t))).


Since the mappingQ+r has the weak Cesari property, we get that (λ(t), ψ′(t)) ∈

Q+r (t, ϕ(t)).

Step 4. There exists a measurable function u defined on [t0, t1] such that

for almost all t in [t0, t1]: (i) ψ′(t) = f(t, ψ(t), v(t)); (ii) v(t) ∈ Ω(t, ϕ(t));

(iii) λ(t) ≥ f0r (t, ψ(t), v(t)).

The existence of a function v satisfying the conclusion of Step 4 is a re-statement of (λ(t), ψ′(t)) ∈ Q+(t, ψ(t)). The problem is to show that there isa measurable function u with this property. This will be done using Filippov’sLemma, Theorem 3.4.1.

With reference to Theorem 3.4.1, let T = t : (λ(t), ψ′(t)) ∈ Q+r (t, ϕ(t)).

Step 3 shows that T is not empty. Let Z = R1 × R

n × Rn × R

1. LetD = (t, x, z) : (t, x) ∈ R, z ∈ Ω(t, x) and let D = (t, x, z, η) : (t, x, z) ∈ D,

η ≥ f0r (t, x, z). From the upper semi-continuity of the mapping Ω and

Lemma 5.2.4 it follows that D is closed. Since f0r is lower semi-continuous,

it follows that the set D is also closed. Moreover, D can be written as theunion of an at most countable number of compact sets Di, where Di is theintersection of D with the compact closed ball of radius i centered at theorigin.

Let Γ denote the mapping from T to Z defined by t→ (t, ψ(t), ψ′(t), λ(t)).Each of the functions ψ, ψ′, λ is measurable, so Γ is a measurable map. Let Φdenote the map from D to Z defined by (t, x, z, η) → (t, x, fr(t, x, z), η). Sincefr is continuous, so is Φ. By the definition of T , Γ(t) ⊆ Φ(D). Hence all thehypotheses of Filippov’s Lemma are fulfilled. Thus, there exists a measurablemapping m from T to D,

m : t→ (τ(t), x(t), v(t), η(t))

such that for t ∈ T

Φ(m(t)) = (τ(t), x(t), fr(τ(t), x(t), v(t)), η(t)) =

Γ(t) = (t, ψ(t), ψ′(t), λ(t)).

From this, the conclusion of Step 4 follows.Step 5. Completion of Proof.

The function ψ in Step 4 is the restriction to [t0, t1] of the function ψobtained in Step 1. Hence e(ψ) ∈ B and ψ lies in the compact set R0 ⊆ R.Thus, to show that (ψ, v) is admissible it remains to show that the mappingt→ f0

r (t, ψ(t), v(t)) is integrable. Since ψ and v are measurable and f0r is lower

semi-continuous t → f0(t, ψ(t), v(t)) is measurable. Since λ is integrable andf0r ≥ 0, it follows from (iii) of Step 4 that t→ f0

r (t, ψ(t), v(t)) is integrable.Finally, we show that (ψ, v) is optimal. From Step 3 we get that

∫ t1

t0

f0r (t, ψ(t), v(t))dt ≤

∫ t1

t0

λ(t)dt ≤ γ.

In Step 2 we obtained a subsequence (ψk, vk) of the minimizing sequence


such that

γ = limk→∞

∫ t1k

t0k

f0r (t, ψk(t), vk(t))dt,

and so

∫ t1

t0

f0r (t, ψ(t), v(t))dt ≤ lim

k→∞

∫ t1k

t0k

f0r (t, ψk(t), vk(t))dt. (5.4.33)

Let m = infJ |(ψ, v) : (ψ, v) admissible. Then from the uniform convergenceof ψk to ψ, the lower semi-continuity of g and (5.4.33) we get:

m = limk→∞

J(ψk, vk) = limk→∞

[g(e(ψk)) +

∫ t1k

t0k

f0r (t, ψk, vk)dt]

≥ lim infk→∞

g(e(ψk)) +

∫ t1

t0

f0r (t, ψ(t), v(t))dt

≥ g(e(ψ)) +

∫ t1

t0

f0r (t, ψ(t), v(t))dt ≥ m.

Hence (ψ, v) is optimal.

Remark 5.4.17. In Theorem 5.3.5 the optimal trajectory was obtained asthe uniform limit of trajectories in a minimizing sequence and the optimalrelaxed control was obtained as a weak limit of the corresponding relaxedcontrols. In contrast in Theorem 5.4.4, the optimal trajectory was obtainedas the uniform limit of trajectories in a minimizing sequence and it was thenshown that there exists a control that will yield this trajectory. No relationshipis established between the controls in the minimizing sequence and the controlthat gives the optimal trajectory.

Next, we shall obtain the classical Nagumo-Tonelli existence theorem forthe simple problem in the calculus of variations as an almost immediate con-sequence of Theorem 5.4.16.

Theorem 5.4.18. Let f0 be lower semicontinuous in G = R × Rn and let

f0(t, x, z) ≥ 0 for all (t, x, z) in G. For each (t, x) in R, let f0 be a convexfunction of z. Let B be closed and let g be lower semi-continuous on B. Letthe graphs of all trajectories lie in a compact subset R0 of R. Let there exista nonnegative function Φ defined on [0,∞) such that Φ(ξ)/ξ → ∞ as ξ → ∞and such that for all (t, x, z) in G, f0(t, x, z) ≥ Φ(|z|). Then there exists anabsolutely continuous function φ∗ that satisfies e(ϕ∗) ∈ B and that minimizes

J(φ) = g(t0, φ(t0), t1, φ(t1) +

∫ t1

t0

f0(t, φ(t), φ′(t))dt.

Moreover φ∗ is optimal for the relaxed version of this problem.


Proof. Write the variational problem as a control problem by writing theintegrand as f0(t, x, z), adding the state equation x′ = z and taking Ω(t, x) =R

n for all (t, x). It is readily checked that Assumption 5.4.1 holds for thecontrol problem. Since the trajectories of the control problem are the same asthose for the variational problem, the trajectories for the control problem liein a compact set.

In the control formulation,

Q+(t, x) = (y0, y) : y0 ≥ f0(t, x, z), y = z= y0 : y0 ≥ f0(t, x, y) y ∈ R

n.

Since for each (t, x), f0(t, x, y) is convex in y, it is readily checked that thesets Q+(t, x) are convex.

Let ε > 0 be given. Then since Φ(ξ)/ξ → ∞ as ξ → ∞, and f0(t, x, z) ≥Φ(|z|), we have that for |z| sufficiently large

f0(t, x, z)

|z| ≥ Φ(|z|)|z| ≥ 1

ε.

Thus, f(t, x, z) = z is of slower growth than f0. Similarly, the function iden-tically equal to one is of slower growth than f0. It now follows from Theo-rem 5.4.16 that there exists an absolutely continuous function φ∗ that mini-mizes J(ϕ).

Exercise 5.4.19. Show that if B is compact, then all trajectories in a min-imizing sequence (ordinary or relaxed) will lie in a compact subset R′ of R.Thus, we can replace the second and third sentences in the statement of The-orem 5.4.18 by the statement, “Let B be compact.”

Exercise 5.4.20. In this exercise we obtain an existence theorem for thelinear plant-quadratic criterion problem in which the state equations are

dx

dt= A(t)x +B(t)z + d(t)

and the function f0 is given by

f0(t, x, z) = 〈x,X(t)x〉 + 〈z,R(t)z〉.

The matrices A,B,X , and R are continuous on an interval [a, b], as is thevector d. For each t in [a, b], the matrix X(t) is symmetric, positive semi-definite and the matrix R(t) is symmetric positive definite. The set B is ann dimensional closed manifold consisting of points (t0, x0, t1, x1) with (t0, x0)fixed and (t1, x1) in a specified n-dimensional manifold J1. The controls u havevalues in the open set U

1. Show that ∫ b

a

〈u(t), R(t)u(t)〉dt < +∞


if and only if u ∈ L2[a, b]. (Hint: Recall that

ρ1(t)|u(t)|2 ≤ 〈u(t), R(t)u(t)〉 ≤ ρn(t)|u(t)|2

where ρ1(t) is the smallest eigenvalue of R(t) and ρn(t) is the largesteigenvalue of R(t). Show that ρ1 is continuous and ρ1(t) > 0 for all t in[a, b].

2. Use the result in (1) and the variation of parameters formula to showthat all trajectories (relaxed or ordinary) of a minimizing sequence liein a compact set R0 ⊆ R.

3. Use Theorem 5.4.16 to show that the linear-plant quadratic criterionproblem has a relaxed solution that is an ordinary solution (ϕ, u).

4. Obtain the same conclusion as in (3) using Theorems 5.5.3 and 5.5.7 ofthe next section.

5.5 Existence Without the Cesari Property

In this section we shall state and prove two existence theorems for relaxedand ordinary problems in which it is not assumed that the weak Cesari prop-erty holds. In both of these theorems it will be assumed that the constraintmapping Ω depends on t alone. In one of the theorems we assume that thefunction f = (f0, f) satisfies a generalized Lipschitz condition. In the otherwe assume that the controls in a minimizing sequence all lie in a closed ballof some Lp space, 1 ≤ p ≤ ∞.

We assume that f , g,B, and Ω, the data of both the ordinary and relaxedproblems satisfy the following:

Assumption 5.5.1. (i) The function f = (f0, f) = (f0, f1, . . . , fn) is de-fined on a set G = I × X × U , where I is a real compact interval, X isa closed interval in R

n, and U is an open interval in Rm.

(ii) The function f is continuous on G.

(iii) There exists an integrable function β on I such that f0(t, x, z) ≥ β forall (t, x, z) in G.

(iv) The set B is closed.

(v) The function g is lower semi-continuous on B.

(vi) Ω is a mapping from I to subsets Ω(t) of U that is upper semi-continuouson I.


Remark 5.5.2. Assumption 5.5.1 differs from Assumption 5.4.1 in that f0

is continuous rather than semi-continuous and Ω depends on t alone ratherthan on (t, x).

Theorem 5.5.3. Let Assumption 5.5.1 hold. Let there exist a non-decreasingfunction ρ defined on [0,∞) such that ρ(δ) → 0 as δ → 0 and a nonnegativefunction L defined on I × U such that

|f(t, x, z)− f(t, x′, z)| ≤ L(t, z)ρ(|x− x′|) (5.5.1)

for all (t, x, z) and (t, x′, z) in G.

(i) Let the sets Q+r (t, x) be closed and let there exist a minimizing sequence

ψk for the relaxed problem such that all the trajectories ψk lie in acompact set and are equi-absolutely continuous. Let there exist a constantA > 0 such that for all the functions uki, i = 1, . . . , n+2, k = 1, 2, . . .appearing in the sequence vk of relaxed controls

∫ t1

t0

L(t, uki(t))dt ≤ A. (5.5.2)

Then the relaxed problem has a solution.

(ii) If the sets Q+(t, x) are convex, then there exists an ordinary control thatis optimal for the relaxed and ordinary problem.

Remark 5.5.4. If f is Lipschitz continuous in x, uniformly for (t, z) in I×U ,then (5.5.1) holds with L(t) = K, the Lipschitz constant, and ρ(δ) = δ. If fis uniformly continuous on

D = (t, x, z) : (t, x) ∈ R, z ∈ Ω(t),

which occurs if D is compact, then (5.5.1) holds with L ≡ 1 and ρ the modulusof continuity.

Remark 5.5.5. A sufficient condition that the sets Q+(t, x) be closed isthat the function identically equal to one be of slower growth than f0. Thiscondition and the condition |f j | ≤ Kf0 for some constant K > 0, j = 1, . . . , nare sufficient for the sets Q+

r (t, x) to be closed.

To see this let yk = (y0k, yk) be a sequence of points in Q+(t, x) convergingto a point (y0, y). Then there exists a sequence of points zk in Ω(t) suchthat

y0k ≥ f0(t, x, zk) yk = f(t, x, zk). (5.5.3)

We assert that the sequence zk is bounded. For if not, there would exista subsequence such that |zk| → ∞. Let ε > 0 be arbitrary. Then for all ksufficiently large

1 ≤ εf0(t, x, zk) ≤ εy0k.


Since ε is arbitrary, the preceding would imply that the sequence y0k isunbounded. This contradicts y0k → y0. Hence zk is bounded and there existsa subsequence and a point z in R

m such that zk → z. Since the mapping Ω isupper semi-continuous, the sets Ω(t) are closed. Since the points zk are inΩ(t), so is the point z. If we let k → ∞ in (5.5.3), it follows from the continuity

of f that y = (y0, y) is in Q+(t, x), and so Q+(t, x) is closed.We now show that the sets Q+

r (t, x) are closed. Let yk = (y0k, yk) be asequence of points in Q+

r (t, x) converging to a point y = (y0, y). Then thereexists a sequence of points πk, ζk, where each πk is as in (5.4.1) and each ζkis as in (5.4.1) with zki ∈ Ω(t) such that

y0k =

n+2∑

i=1

πikf

0(t, x, zki) yk =

n+2∑

i=1

πikf(t, x, zki).

Since πik ≥ 0, i = 1, . . . , n+ 2 and f0 ≥ 0, it follows that

πikf

0(t, x, zki) ≤ y0k (5.5.4)

for i = 1, . . . , n+2. In what follows we shall be taking a series of subsequenceswith the implicit assumption that each subsequence is a subsequence of theprevious subsequence.

Since Πn+2 is compact there exists a subsequence of k such that πkconverges to a point π in Πn+2. Let π

1, . . . , πs be the positive componentsof π. Then

∑si=1 π

i = 1. We assert that each of the sequences zki, i =1, . . . , s is bounded. For if a sequence zki were unbounded there would exista subsequence such that |zki| → ∞. Hence for each ε > 0, we would haveεf0(t, x, zki) > 1 for all sufficiently large k. From this and from (5.5.4) wewould get

εy0k ≥ επikf

0(t, x, zki) > πik.

Since ε > 0 is arbitrary and πik → πi > 0, this would imply that y0k is un-

bounded, thus contradicting y0k → y0. Since each sequence zki, i = 1, . . . , sis bounded, there exists a subsequence of k and a point (z1, . . . , zs) suchthat zki → z. Since the points zki are in Ω(t) and Ω(t) is closed, each

zi ∈ Ω(t). From the continuity of f it follows that f(t, x, zki) → f(t, x, zi) fori = 1, . . . , s.

We now consider those components πi, if any, with πi = 0. If the sequencezki is bounded, then there exists a subsequence zki and a point zi ∈ Ω(t)

such that zki → zi. Hence f(t, x, zki) → f(t, x, zi) and so πikf(t, x, zki) → 0.

If the sequence zki is unbounded, then there exists a subsequence zkisuch that |zki| → ∞. Since one is of slower growth than f0, there is a subse-quence zki of the preceding subsequence such that 1 < (πi

k)1/2f0(t, x, zki)

for sufficiently large k. From this, from 0 ≤ πik ≤ 1, and from (5.5.4) we get

1 < (πik)

1/2f0(t, x, zki) ≤ πikf

0(t, x, zki) ≤ y0k.


Hence(πi

k)1/2 < πi

kf0(t, x, zki) ≤ (πi

k)1/2y0k.

If we now let k → ∞, we get that

πikf

0(t, x, zki) → 0.

From this and the condition |f j | ≤ Kf0, j = 1, . . . , n we get thatπikf

j(t, x, zki) → 0.In summary, we have shown that there exists a subsequence of yk =

y0k, yk and corresponding subsequences of πk and zki such that

yk →s∑

i=1

πif(t, x, zi)

with zi ∈ Ω(t) and (π1, . . . , πs) ∈ Πs where π1, . . . , πs are the positive com-ponents of π. Since yk → y, we get that y ∈ Q+

r (t, x), and so Q+r (t, x) is

closed.

Example 5.5.6. We give an example of a system in which the weak Cesariproperty fails to hold, but (5.5.1) and (5.5.2) do hold. This example alsoshows that the growth condition of Remark 5.5.5 is not a necessary conditionfor the sets Q+(t, x) to be closed. Let x = (x1, x2), let z be a real number, letΩ(t) = R

1, let f0 ≡ 0, and let f(t, x, z) = (z, x1z). Then (5.5.1) holds withL(t, z) = |z|. For each (t, x) in R

Q+(t, x) = (η, ξ) = (η, ξ1, ξ2) : η ≥ 0, ξ1 = z, ξ2 = x1z, z ∈ R. (5.5.5)

The sets Q+(t, x) are closed and convex, yet one is not of slower growth thanf0. Also, for each δ > 0

cl co Q+(Nδx(t, x)) = (η, ξ) : η ≥ 0, ξ ∈ R2.

This and (5.5.5) show that the weak Cesari property fails.

To show that Theorem 5.5.3 can be applied to this problem, let(ψk, p

1ku1k+p

2ku2k) be a relaxed minimizing sequence. Since Q+(t, x) is con-

vex we may replace the sequence of relaxed controls (p1ku1k + p2ku2k) by asequence of ordinary controls uk with corresponding trajectory ψk. Sincethe function ψk are equi-absolutely continuous, it follows from the equation(ψ1

k)′ = uk that the integrals

∫ t1t0

|uk|dt are bounded by a constant A. Hence

∫ t1

t0

L(t, uk(t), dt =

∫ t1

t0

|uk(t)|dt ≤ A,

and (5.5.2) holds.


Proof of Theorem 5.5.3. We first note that conclusion (ii) follows from (i) byvirtue of Corollary 4.4.3. Hence it suffices to prove (i).

If f satisfies (5.5.1), then

|fr(t, x, π, z)− fr(t, x′, π, z)| ≤

n+2∑

i=1

πi|f(t, x, zi)− f(t, x′, zi)|

≤n+2∑

i=1

πiL(t)|x− x′| = L(t)|x− x′|.

Hence fr satisfies (5.5.1) as well as the other hypotheses in Assumption 5.5.1.

Let Ω be as in the proof of Theorem 5.4.4. Then Ω is upper semi-continuous.Hence we may proceed as in the proof of Theorem 5.4.4 and take therelaxed problem to be an ordinary problem with control v = (p, v) =(p1, . . . , pn+2, u1, . . . , un+2).

The proof proceeds exactly as the proof of Theorem 5.4.4 up to and includ-ing the definition of λj in (5.4.31). The rest of the argument to prove Step 3proceeds differently. The reader is urged to keep in mind the order in whichvarious subsequences are chosen.

Define sequences of functions σj and θj corresponding to ψj and λj asfollows

σj(t) =

k∑

i=1

αijfr(t, ψ(t), vnj+i) (5.5.6)

θj(t) =

k∑

i=1

αijf0r (t, ψ(t), vnj+i), (5.5.7)

where if t 6∈ [t0q, t1q] we set f(t, ψq, vq(t)) = 0.

Let Mk = max|ψk(t) − ψ(t)| : t ∈ I. Since ψk converges uniformly to

ψ on I, Mk → 0 as k → ∞. Let f∗rq(t) ≡ fr(t, ψ(t), vq(t)) and let frq(t) ≡

fr(t, ψq(t), vq(t)). Note that f∗rq(t) = frq(t) = 0 for t 6∈ [t0q, t1q]. Using (5.5.1),

(5.5.2), and (5.5.6), we get

∫ b

a

|σj − ωj |dt ≤k∑

i=1

αij

∫ b

a

|f∗r,nj+i − fr,nj+i|dt

=

k∑

i=1

αij

∫ t1,nj+i

t0,nj+i

|f∗r,nj+i − fr,nj+i|dt

≤k∑

i=1

αijρ(Mnj+i)

∫ t1,nj+i

t0,nj+i

L(t, unj+i(t))dt

≤ Ak∑

i=1

αijρ(Mnj+i).


Since Mk → 0 and ρ(δ) → 0 as δ → 0 we get that σj − ωj → 0 in L1[a, b].A similar argument shows that θj − λj → 0 in L1[a, b]. Hence there exists asubsequence such that

σj(t)− ωj(t) → 0 and θj(t)− λj(t) → 0 a.e. (5.5.8)

We now define λ as in (5.4.30) and show as we did in the paragraphfollowing (5.4.30) that λ is in L1[a, b] and that (5.4.26) holds.

As in Step 3 of the proof of Theorem 5.4.4, let T ′ denote the set of pointsin [t0, t1] at which λ(t) is finite, ωj(t) → ψ′(t) and at which vk(t) ∈ Ω(t) forall k. This set has measure t1 − t0. Let T

′′ denote the set of points at which(5.5.8) holds. Let T = T ′ ∪ T ′′. Then meas T = t1 − t0.

Let t be a fixed but arbitrary point of T . Since ωj(t) → ψ′(t), it followsfrom (5.5.8) that σj(t) → ψ′(t). It follows from the definition of λ that there isa subsequence λj(t), which in general depends on t such that λj(t) → λ(t).From (5.5.8) we get that θj(t) → λ(t). By definition, for all t in T , and all jand i

fr(t, ψ(t), vnj+i(t)) ∈ Q+r (t, ψ(t)).

Since Q+r (t, ϕ(t)) is convex, the points (θj(t), σj(t)) belong to Q+

r (t, ϕ(t)).Since Q+

r (t, ϕ(t)) is also assumed to be closed and (θj(t), σj(t)) → (λ(t), ψ′(t)),we get that (λ(t), ψ′(t)) ∈ Q+

r (t, ϕ(t)). Since t is an arbitrary point of T , wehave that (λ(t), ψ′(t)) ∈ Q+

r (t, ϕ(t)) a.e. in [t0, t1].The rest of the proof is exactly the same as in the proof of Theorem 5.4.4.

Theorem 5.5.7. Let Assumption 5.5.1 hold.

(i) Let the sets Q+r (t, x) be closed. Let ψk be the relaxed trajectories in

a minimizing sequence for the relaxed problem such that the trajectoriesψk all lie in a compact set and are equi-absolutely continuous. Let all ofthe control functions uk,1, . . . , uk,n+2 corresponding to the trajectoriesψk all lie in a closed ball of some Lp space 1 ≤ p ≤ ∞. Then therelaxed problem has a solution.

(ii) Let the sets Q+(t, x) be closed and convex. Then there exists an ordinarycontrol that is a solution of both the relaxed and ordinary problems.

Proof. Since all the controls vk = uk,1, . . . , uk,n+2) lie in a closed ball some Lp

space and since p(t) = (p1(t), . . . , pn+1(t)) satisfies∑pi(t) = 1 and pi(t) ≥ 0,

it follows that all the controls vk = (pk, vk) lie in a closed ball of some Lp

space. The sets Q+r are convex. Hence we can view the relaxed problem as an

ordinary problem, with all the hypotheses of (i) satisfied. By Corollary 4.4.3we need only establish (i). We shall do so assuming that the relaxed problemis an ordinary problem, as was done in the proof of Theorem 5.4.4.

The proof proceeds as does the proof of Theorem 5.4.4 through Steps 1and 2. Step 3 is modified as follows. Since ψk → ψ uniformly on [a, b] andall trajectories lie in a compact set, it follows that there exists an M ′ > 0such that ‖ψk‖p ≤M ′ and ‖ψ‖p ≤M ′, where ‖ ‖p denotes the Lp[a, b] norm.


Let vk denote the extension of vk from [t0k, t1k] to [a, b] by setting vk(t) = 0if t 6∈ [t0k, t1k]. Since by hypothesis the functions vk lie in a closed ball of

radius M in Lp[t0k, t1k] we get that for all k, the functions vk = (ψk, vk) and

wk = (ψ, vk) lie in a ball in Lp[a, b]. Also note that vk(t) − wk(t) → 0 at allpoints of [a, b].

Let∆k(t) = fr(t, ψ(t), vk(t)) − fr(t, ψk(t), vk(t)), (5.5.9)

where we set ∆k(t) = 0 if t 6∈ [t0k, t1k]. It is then a consequence of Lemma 5.3.8

with ξ = (x, z) and h(t, ξ) = f(t, x, z) that ∆k → 0 in measure on [a, b]. Since

∆k → 0 in measure on [a, b] there exists a subsequence such that

∆k(t) → 0 a.e. (5.5.10)

in [a, b].The functions ωj, λj , and λ are next defined as in Step 3 of the proof of

Theorem 5.4.4 and it is shown that (5.4.28) holds. Sequences σj and θjare defined as in (5.5.6). If, as usual, we denote the first component of ∆k(t)by ∆0

k(t) and the remaining n components by ∆k(t) we get, using (5.5.6),(5.5.9), (5.4.27), and (5.4.29), that

θj(t)− λj(t) =

k∑

i=1

αij∆0nj+i(t)

σj(t)− ωj(t) =

k∑

i=1

αij∆nj+i(t).

It then follows from (5.5.10) and Lemma 5.3.7 that (5.5.8) holds.

The rest of the proof is a verbatim repetition of the last four paragraphsof the proof of Theorem 5.5.3.

5.6 Compact Constraints Revisited

In this section we shall use Theorem 5.4.4 to obtain an existence theoremfor problems with compact constraints.

Theorem 5.6.1. Let Assumption 5.4.1 hold, except for (vi), which we replaceby the following. The mapping Ω from R = I × X to compact subsets Ω(t, x)of U is u.s.c.i. on R.

(i) Let the set of admissible relaxed trajectories be non-empty and let alladmissible relaxed trajectories lie in a compact set R0 ⊆ R. Then therelaxed problem has a solution.


(ii) Let the sets Q+(t, x) be convex for all (t, x) in R. Then there exists anordinary control that is a solution of both the ordinary problem and therelaxed problem.

Remark 5.6.2. In proving (i) of Theorem 5.6.1 we take the formulation ofthe relaxed problem to be given by (5.4.1) and (5.4.2), rather than the onegiven in Section 3.2. In Section 3.5 we showed that these two formulations areequivalent. As already noted several times, the relaxed problem (5.4.1)–(5.4.2)can be viewed as an ordinary problem. Since the sets Q+

r (t, x) are convex, therelaxed problem, viewed as an ordinary problem satisfies the hypotheses ofTheorem 4.4.2, an existence theorem for the ordinary problem. As in previousarguments, we shall assume that the relaxed problem is an ordinary problem.We may also assume without loss of generality that β = 0, so f0 ≥ 0.

Remark 5.6.3. The hypotheses of Theorem 5.6.1 are less stringent thanthose of Theorem 4.3.5. In Theorem 5.6.1 we do not require f to be Lipschitzin x, whereas we do in Theorem 4.3.5. Also in Theorem 4.3.5 the sets Ω(t, x)are required to depend on t alone; that is, Ω(t, x) = Ω(t, x′) for all x, x′ in R.In Theorem 5.6.1 the sets Ω(t, x) can depend on t and x.

Proof of Theorem 5.6.1. By Corollary 4.4.3 we need only establish (i). Weprove Theorem 5.6.1 by showing that the hypotheses of Theorem 5.6.1 implythose of Theorem 5.4.4.

We first show that (vi) of Assumption 5.4.1 holds, namely that the mapping

Ω is upper semi-continuous on R. The set Ω is defined following (5.4.5). The

upper semi-continuity of Ω follows from Lemma 5.2.2 and the assumption thatΩ is u.s.c.i. and the sets Ω(t, x) are compact.

We next show that the trajectories ψk in a minimizing sequence areequi-absolutely continuous. Let

Dr = (t, x, z) : (t, x) ∈ R0, z ∈ Ω(t, x). (5.6.1)

By Lemma 3.3.11, the set Dr is compact. Since fr is continuous on Dr, thereexists a constant A > 0 such that |f(t, x, z)| ≤ A for all (t, x, z) in Dr. Foreach k and almost all t ∈ [t0k, t1k], we have (t, ψk(t), vk(t)) ∈ D. Since fora.e. t ∈ [t0k, t1k]

ψ′k(t) = f(t, ψk(t), vk(t)),

we have |ψ′k(t)| ≤ A, a.e. on [t0,k, t1k]. Hence the trajectories ψk are equi-

absolutely continuous.To complete the proof we show that the mapping Q+

r satisfies the weakCesari property at all points of R0. Let D be as in (5.4.7). The proof proceedsas does the proof of Lemma 5.4.6 from (5.4.8) to (5.4.16).

Since for each k, zki ∈ Ω(t, xki), the sequence (t, xki, zki) is in D fori = 1, . . . , n+2. From the compactness of D and from (5.4.13), it follows thatthere exists a subsequence

(t, xk1, zk1), . . . , (t, xk,n+2, zk,n+2)


and points z1, . . . , zn+2 such that (t, x, zi) ∈ D and

(t, xki, zki) → (t, x, zi) i = 1, . . . , n+ 2. (5.6.2)

Thus, zi ∈ Ω(t, x).From (5.4.10), (5.4.13), (5.4.14), (5.4.15), (5.6.2), and the continuity of f

we get that

y = limk→∞

yk = limk→n

n+2∑

i=1

αkiyki =

n+2∑

i=1

αif(t, x, zi) (5.6.3)

where zi ∈ Ω(t, x). From (5.4.11), (5.4.14), (5.4.16), (5.6.2), and the lowersemi-continuity of f0 we get that

y0 ≥n+2∑

i=1

limk→∞

inf αkiy0ki ≥

n+2∑

i=1

lim infk→∞

(αkif0(t, xki, zki)) (5.6.4)

=

n+2∑

i=1

αif0(t, x, zi)

with zi ∈ Ω(t, x). From (5.6.3) and (5.6.4) we get that y = (y0, y) is in coQ+(t, x). But co Q+(t, x) = Q+

r (t, x), so y ∈ Q+r (t, x), which shows that Q+

r

has the weak Cesari property at (t, x).

Chapter 6

The Maximum Principle and Some of

Its Applications

6.1 Introduction

In this chapter we state several versions of the maximum principle, corre-sponding to different hypotheses on the data of the problem. We shall collec-tively call each of these results, “the maximum principle”, and shall use themto characterize the optimal controls in several important classes of problems.The proof of the maximum principle will be given in the next chapter.

In Section 6.2 we use a dynamic programming argument to derive the max-imum principle for ordinary problems. Although the arguments are elementaryand mathematically correct, the assumptions made rule out most interestingproblems. The purpose of this section is to make the maximum principle plau-sible and to give a geometric interpretation of the theorem. From the viewpoint of logical development Section 6.2 can be omitted, except for the con-cepts of value function and optimal synthesis, or optimal feedback control,which are introduced in Section 6.2 and used again in Section 6.9.

In Section 6.3 we state the maximum principle for the relaxed problemin Bolza form. The statements of the maximum principle for other formula-tions of the problem, such as those discussed in Chapter 2, are taken up inthe exercises. In special cases of importance more precise characterizations ofthe optimal pair can often be given. Some of these are also taken up in theexercises. The exercises in this section are an important supplement to thegeneral theory.

In Section 6.4 we use the maximum principle and one of our existencetheorems to determine the optimal pair in a specific example. The purpose hereis to illustrate how the maximum principle is used and some of the difficultiesthat one can expect to encounter in large-scale problems.

The remaining sections of the chapter are devoted to applications of themaximum principle to special classes of problems. In Section 6.5 we obtain thefirst order necessary conditions of the calculus of variations from the maximumprinciple, both in the classical case and in the case where the functions are inthe class W 1,1. The functions in W 1,1 are absolutely continuous with squareintegrable derivatives. In the exercises we take up the relationship between the

149


classical Bolza problem in the calculus of variations and the control problem.In Section 6.6 we take up control problems that are linear in the state variable.We specialize this in Section 6.7 to linear problems, and further specialize inSection 6.8 to the linear time optimal problem. The standard results for theseproblems are obtained, whenever possible, as relatively simple consequences ofthe maximum principle. The power of the maximum principle will be apparentto the reader.

In Section 6.9 we take up the linear plant quadratic cost criterion prob-lem. Here again we obtain the standard characterization of the optimal pairfrom the maximum principle. We also show that the necessary conditions aresufficient and we obtain the standard synthesis of the optimal control.

6.2 A Dynamic Programming Derivation of theMaximum Principle

Let R1 be a region of (t, x)-space and let R be a subregion of R1 such thatthe closure of R is contained in R1. For each point (τ, ξ) in R we consider thefollowing problem. Minimize the functional

J(φ, u) = g(t1, φ(t1)) +

∫ t1

τ

f0(t, φ(t), u(t))dt (6.2.1)


dx

dt= f(t, x, u(t)), (6.2.2)

control constraints u(t) ∈ Ω(t), and end conditions

(t0, φ(t0)) = (τ, ξ) (t1, φ(t1)) ∈ T .

We assume that the terminal set T is a C(1) manifold of dimension r,where 0 < r ≤ n and that T is part of the boundary of R. See Figure 6.1. Forsimplicity we also assume that T can be represented by a single coordinatepatch. That is, we assume that T consists of all points of the form (t1, x1)with

t1 = T (σ) x1 = X(σ), (6.2.3)

where T and X are C(1) functions defined on an open parallelepiped Σ in Rr.

The Maximum Principle and Some of Its Applications 151

FIGURE 6.1

It is also assumed that the Jacobian matrix of the mapping (6.2.3),

∂(T,X)

∂σ=

∂T∂σ1 . . . ∂T

∂σr

∂X1

∂σ1 . . . ∂X1

∂σr

...∂Xn

∂σ1 . . . ∂Xn

∂σr

,

has rank r at all points of Σ. We assume that the function g in (6.2.1) isdefined and C(1) in a neighborhood of T and that f0 and f are C(1) functionson G1 = R1 × U. Note that the constraint mapping Ω is assumed to beindependent of x and to depend only on t.

We assume that for each (τ, ξ) in R the problem has a unique solution. Wedenote the unique optimal trajectory for the problem with initial point (τ, ξ)by φ(·, τ, ξ). The corresponding unique optimal control is denoted by u(·, τ, ξ).We assume that the function u(·, τ, ξ) is piecewise continuous and that at apoint of discontinuity td the value of u(·, τ, ξ) is its right-hand limit; thus,u(td, τ, ξ) = u(td + 0, τ, ξ). Points (t, x) on the trajectory satisfy the relationx = φ(t, τ, ξ). In particular, note that

ξ = φ(τ, τ, ξ).

The value of the optimal control at time t is u(t) = u(t, τ, ξ).For each point (τ, ξ) in R, let W (τ, ξ) denote the value given to the func-

tional (6.2.1) by the unique optimal pair (φ(·, τ, ξ), u(·, τ, ξ)). Thus, if A(τ, ξ)denotes the set of admissible pairs (φ, u) for the problem with initial point(τ, ξ) then

W (τ, ξ) = minJ(φ, u) : (φ, u) ∈ A(τ, ξ). (6.2.4)

The function W so defined is called the value function for the problem.


Let τ1 > τ and let (τ1, ξ1) be a point on the optimal trajectory φ(·, τ, ξ).Then ξ1 = φ(τ1, τ, ξ). We assert that the optimal pair for the problem startingat (τ1, ξ1) is given by (φ(·, τ, ξ), u(·, τ, ξ)). That is, for t ≥ τ1, t

′ = t− τ1,

φ(t′, τ1, ξ1) = φ(t, τ, ξ) (6.2.5)

u(t′, τ1, ξ1) = u(t, τ, ξ).

In other words, an optimal trajectory has the property that it is optimal forthe problem that starts at any point on the trajectory. To see this we write

W (τ, ξ) =

∫ τ1

τ

f0∗(t, τ, ξ)dt+

∫ t1

τ1

f0∗(t, τ, ξ)dt+ g(t1, φ(t1, τ, ξ)), (6.2.6)

wheref0∗(t, τ, ξ) = f0(t, φ(t, τ, ξ), u(t, τ, ξ)). (6.2.7)

If (φ(·, τ, ξ), u(·, τ, ξ)) were not optimal for the problem initiating at (τ1, ξ1),then by (6.2.4) with (τ, ξ) replaced by (τ1, ξ1) and by our assumption ofuniqueness of optimal pairs, we would have that W (τ1, ξ1) is strictly less thanthe sum of the last two terms in the right-hand side of (6.2.6). Recall thatu(·, τ1, ξ1) is the optimal control for the problem with initial point (τ1, ξ1).Hence for a control u defined by

u(t) =

u(t, τ, ξ) τ ≤ t < τ1

u(t, τ1, ξ1) τ1 ≤ t ≤ t1

the corresponding trajectory φ would be such that J(φ, u) < W (τ, ξ), thuscontradicting (6.2.4). Hence (6.2.5) holds.

We define a function U on R as follows

U(τ, ξ) = u(τ, τ, ξ).

If we set t = τ1 in the second equation in (6.2.5) and use the definition of Uwe get that for all τ1 ≥ τ

u(τ1, τ, ξ) = U(τ1, ξ1), (6.2.8)

where ξ1 = φ(τ1, τ, ξ). Thus, at each point (τ, ξ) in R the value U(τ, ξ) of Uis the value of the unique optimal control function associated with the uniqueoptimal trajectory through the point. The function U is called the synthesis ofthe optimal control or optimal synthesis function. It is also called the optimalfeedback control.

Recall that we are assuming here that given a point (τ, ξ) in R, there is aunique optimal pair for the problem with this initial point. In general, theremay exist initial points such that there is more than one optimal pair. In thiscase, the optimal feedback control is multi-valued at this point.

We now suppose that the function W is C(1) on R. We shall derive a


partial differential equation that W must satisfy. Consider a point (τ, ξ) in Rand an interval [τ, τ+∆t], where ∆t > 0. Let v be a continuous control definedon [τ, τ +∆t] satisfying v(t) ∈ Ω(t). We suppose that ∆t is so small that thestate equations (6.2.2) with u(t) replaced by v(t) have a solution ψ defined on[τ, τ +∆t] and satisfying the relation ψ(τ) = ξ. Let ∆x = ψ(τ +∆t)− ψ(τ).Thus, the control v transfers the system from ξ to ξ+∆x in the time interval[τ, τ +∆t]. For t ≥ τ +∆t let us use the optimal control for the problem withinitial point (τ +∆t, ξ+∆x). Let u denote the control obtained by using v on

[τ, τ +∆t] and then u(·, τ +∆t, ξ+∆x). Let φ denote the resulting trajectory.

Then (φ, u) ∈ A(τ, ξ) and

W (τ, ξ) ≤ J(φ, u) =

∫ τ+∆t

τ

f0(s, ψ(s), v(s))ds +

∫ t1

τ+∆t

f0∗(s, τ +∆t, ξ +∆x)ds

+ g(t1, φ(t1, τ +∆t, ξ +∆x)),

where f0∗ is defined in (6.2.7). The sum of the last two terms on the right isequal to W (τ +∆t, ξ +∆x). Hence

W (τ +∆t, ξ +∆x)−W (τ, ξ) ≥ −∫ τ+∆t

τ

f0(s, ψ(s), v(s))ds.

Since W is C(1) on R we can apply Taylor’s theorem to the left-hand side ofthe preceding inequality and get

Wτ (τ, ξ)∆t + 〈Wξ(τ, ξ),∆x〉 + o(|(∆t,∆x)|) ≥ −∫ τ+∆t

τ

f0(s, ψ(s), v(s))ds,

(6.2.9)where (Wτ ,Wξ) denotes the vector of partial derivatives of W ando(|(∆t,∆x)|)/ |(∆t,∆x)| → 0 as |(∆t,∆x)| → 0. From the relation

∆x/∆t =1

∆t

∫ τ+∆t

τ

f(s, ψ(s), v(s))ds

and the continuity of f, ψ, and v it follows that

lim∆t→0

∆x

∆t= f(τ, ψ(τ), v(τ)) = f(τ, ξ, v(τ)).

Therefore, if we divide through by ∆t > 0 in (6.2.9) and then let ∆t→ 0, weget that

Wτ (τ, ξ) + 〈Wξ(τ, ξ), f(τ, ξ, v(τ))〉 ≥ −f0(τ, ξ, v(τ)). (6.2.10)

If we carry out the preceding analysis with v(s) = u(s, τ, ξ) on [τ, τ +∆t],then equality holds at every step of the argument. Therefore, using (6.2.8),we obtain the relation

Wτ (τ, ξ) = −f0(τ, ξ, U(τ, ξ)) − 〈Wξ(τ, ξ), f(τ, ξ, U(τ, ξ))〉. (6.2.11)


We now make the further assumption that the constraint mapping Ω issufficiently smooth so that for every vector z ∈ Ω(τ) there exists a continuousfunction v defined on some interval [τ, τ + ∆t], ∆t > 0, with v(τ) = z andv(s) ∈ Ω(s) on [τ, τ + ∆t]. In particular, if Ω is a constant mapping, thatis, Ω(t) = C for all t, then we may take v(s) = z on [τ, τ + ∆t]. Under theassumption just made concerning Ω, we can combine (6.2.10) and (6.2.11) toget the relation

Wτ (τ, ξ) = maxzεΩ(τ)

[−f0(τ, ξ, z)− 〈Wξ(τ, ξ), f(τ, ξ, z)〉], (6.2.12)

with the maximum being attained at z = U(τ, ξ). Equation (6.2.12) is some-times called Bellman’s equation. Equation (6.2.11) is a Hamilton-Jacobi equa-tion.

Equations (6.2.11) and (6.2.12) can be written more compactly. First definea real valued function H on R

1 × Rn × R

m × R1 × R

n by the formula

H(t, x, z, q0, q) = q0f0(t, x, z) + 〈q, f(t, x, z)〉. (6.2.13)

If we now denote a generic point in R by (t, x) rather than by (τ, ξ) we canwrite (6.2.11) in terms of H as follows:

Wt(t, x) = H(t, x, U(t, x),−1,−Wx(t, x)). (6.2.14)

Equation (6.2.12) can be written in the form

Wt(t, x) = maxzεΩ(t)

H(t, x, z,−1,−Wx(t, x)). (6.2.15)

We now suppose that the functionW is of class C(2). Under this additionalhypothesis we shall derive the PontryaginMaximum Principle. Let (τ, ξ) againbe a fixed point in R. Consider the function F defined on R by the formula

F (x) =Wt(τ, x) + f0(τ, x, U(τ, ξ)) + 〈Wx(τ, x), f(τ, x, U(τ, ξ))〉. (6.2.16)

It follows from (6.2.11) that F (ξ) = 0. On the other hand, since U(τ, ξ) ∈ Ω(τ),we obtain the following inequality from (6.2.12) with (τ, ξ) replaced by (τ, x)

Wt(τ, x) ≥ −f0(τ, x, U(τ, ξ))− 〈Wx(τ, x), f(τ, x, U(τ, ξ))〉.

This says that F (x) ≥ 0. Hence the function F has a minimum at x = ξ. SinceW is C(2), F is C(1). Therefore, since ξ is an interior point of the domain ofdefinition of F and F attains its minimum at ξ, we have that Fx(ξ) = 0. Ifwe use (6.2.16) to compute the partial derivatives of F with respect to thestate variable and then set the partials equal to zero at x = ξ, we get that fori = 1, 2, . . . , n,

∂2W

∂t∂xi+ ∂f0∂xi +

n∑

j=1

∂2W∂xi∂xj f j +

n∑

j=1

∂W

∂xj∂f j

∂xi= 0, (6.2.17)


where the partial derivatives of W are evaluated at (τ, ξ) and the functionsf j and their partial derivatives are evaluated at (τ, ξ, U(τ, ξ)). Since (τ, ξ) isan arbitrary point in R, it follows that (6.2.17) holds for the arguments (t, x)and (t, x, U(t, x)), where (t, x) is any point in R.

Before proceeding with our analysis we introduce some useful terminology.

Definition 6.2.1. If h : (t, x, z) → h(t, x, z) is a function from G =R × U to Rk, k ≥ 1, then by the expression “the function h evaluatedalong the trajectory φ(·, τ, ξ)” we shall mean the composite function t →h(t, φ(t, τ, ξ), u(t, τ, ξ)). Similarly, if w is a function defined on R, by the ex-pression “the function w evaluated along the trajectory φ(·, τ, ξ)” we shallmean the composite function t→ w(t, φ(t, τ, ξ)).

We now let (τ, ξ) be a fixed point in R and consider the behavior of thepartial derivative Wx = (Wx1 , . . . ,Wxn) along the optimal trajectory startingat (τ, ξ). We define a function λ(·, τ, ξ) : t → λ(t, τ, ξ) from [τ, t1] to R

n asfollows:

λ(t, τ, ξ) = −Wx(t, φ(t, τ, ξ)). (6.2.18)

Since W is C(2), the function λ is differentiable with respect to t. Using therelation φ′(t, τ, ξ) = f(t, φ(t, τ, ξ), u(t, τ, ξ)) we get

dλi

dt= − ∂2W

∂t∂xi−

n∑

j=1

∂2W

∂xi∂xjf j i = 1, . . . , n, (6.2.19)

where the partial derivatives of W and the components of f are evaluatedalong the trajectory φ(·, τ, ξ). If we substitute (6.2.19) into (6.2.17) and use(6.2.18) we get

dλi

dt= −

−∂f

0

∂xi+

n∑

j=1

λj∂f j

∂xi

i = 1, . . . , n.

In vector-matrix notation this becomes

dλ

dt= −

[−∂f

0

∂x+

(∂f

∂x

)t

λ

], (6.2.20)

where dλ/dt, ∂f0/∂x and λ are column vectors and (∂f/∂x)t is the transposeof the matrix whose entry in the i-th row and j-th column is ∂f i/∂xj . Thepartials in (6.2.20) are evaluated along the trajectory φ(·, τ, ξ). To summarize,we have shown that associated with the optimal trajectory φ(·, τ, ξ) there isa function λ(·, τ, ξ) such that (6.2.20) holds. We point out that since ∂f0/∂xand ∂f/∂x are evaluated along φ(·, τ, ξ) they are functions of t on the interval[τ, t1]. Hence the system (6.2.20) is a system of linear differential equationswith time varying coefficients that the function λ(·, τ, ξ) must satisfy. Initialconditions for this system will be discussed below.


In terms of the function H introduced in (6.2.13), Eq. (6.2.20) becomes

λ′(t, τ, ξ) = −Hx(t, φ(t, τ, ξ), u(t, τ, ξ),−1, λ(t, τ, ξ)), (6.2.21)

where the prime denotes differentiation with respect to time. From the defi-nition of H in (6.2.13) it also follows that

φ′(t, τ, ξ) = Hq(t, φ(t, τ, ξ), u(t, τ, ξ),−1, λ(t, τ, ξ)). (6.2.22)

It follows from (6.2.21) and (6.2.22) that the functions φ(·, τ, ξ) and λ(·, τ, ξ)satisfy the system of differential equations

dx

dt= Hq(t, x, u(t, τ, ξ),−1, q) (6.2.23)

dq

dt= −Hx(t, x, u(t, τ, ξ),−1, q).

We can combine (6.2.14) and (6.2.15) and get

H(t, x, U(t, x),−1,−Wx(t, x)) = maxzεΩ(t)

H(t, x, z,−1,−Wx(t, x)). (6.2.24)

Since (6.2.24) holds for all (t, x) in R, it holds along the optimal trajectoryφ(·, τ, ξ). For points (t, x) on the optimal trajectory φ(·, τ, ξ) the relation

x = φ(t, τ, ξ)

holds. From this relation and from (6.2.8) it follows that for such points therelation

u(t, τ, ξ) = U(t, φ(t, τ, ξ)) (6.2.25)

holds. Relation (6.2.18) also holds along the optimal trajectory. Therefore, forall τ ≤ t ≤ t1, where t1 is the time at which φ(·, τ, ξ) hits T ,

H(t, φ(t, τ, ξ), u(t, τ, ξ),−1, λ(t, τ, ξ)) (6.2.26)

= maxzεΩ(t)

H(t, φ(t, τ, ξ), z,−1, λ(t, τ, ξ)).

Equation (6.2.23) with boundary conditions φ(τ, τ, ξ) = ξ and boundaryconditions on λ to be determined below together with (6.2.26) characterizethe optimal trajectory φ(·, τ, ξ) and optimal control u(·, τ, ξ) in the followingway. Associated with φ(·, τ, ξ) and u(·, τ, ξ) there is a function λ(·, τ, ξ) suchthat λ(·, τ, ξ) and φ(·, τ, ξ) are solutions of (6.2.23) with appropriate boundaryconditions and such that (6.2.26) holds for τ ≤ t ≤ t1. Equations (6.2.23) andtheir appropriate boundary conditions together with relation (6.2.26) consti-tute the Pontryagin Maximum Principle under the present hypotheses. Theyare a set of necessary conditions that an optimal pair must satisfy. A moreprecise and more general formulation will be given in Section 6.3.

Equations (6.2.23) and (6.2.26) involve the optimal control u(·, τ, ξ). We


rewrite these equations so as to involve the synthesis U . If we substitute(6.2.25) into (6.2.21) and (6.2.22) we see that φ(·, τ, ξ) and λ(·, τ, ξ) satisfythe equations

dx

dt= Hq(t, x, U(t, x),−1, q) (6.2.27)

dq

dt= −Hx(t, x, U(t, x),−1, q).

From (6.2.26) we see that φ(·, τ, ξ) and λ(·, τ, ξ) also satisfy

H(t, x, U(t, x),−1, q) = maxzεΩ(t)

H(t, x, z,−1, q). (6.2.28)

We now give a geometric interpretation of the maximum principle. Tosimplify the discussion we assume the problem to be in Mayer form, thatis, f0 ≡ 0. The problem in Bolza form can be written as a Mayer problem,as shown in Section 2.4. We assume that the value function W has level sur-faces W (t, x) = const., and that these level surfaces have gradients (Wt,Wx).Suppose we are at a point (t, x) = (t, φ(t)) of an optimal trajectory φ andwish to proceed to minimize the payoff. The best choice would be to go inthe direction of steepest descent; that is, we should choose a control z0 thatmaximizes −Wt(t, φ(t))−〈Wx(t, φ(t)), f(t, φ(t), z)〉. Thus, z0 = U(t, φ(t)) and

〈−Wx(t, φ(t)), f(t, φ(t), z0)〉 = maxz

〈−Wx(t, φ(t)), f(t, φ(t), z)〉.

If we now use λ(t) = −Wx(t, φ(t)), we have (6.2.26) for the Mayer problem,where f0 ≡ 0.

We return to the Bolza problem with f0 6= 0 and derive the “transversalityconditions.” These are boundary conditions that the value function W andits partial derivatives must satisfy. The transversality conditions are also theboundary conditions that the function λ(·, τ, ξ) must satisfy.

Let (τ1, ξ1) be the terminal point of the optimal trajectory for the problemwith initial point (τ, ξ). Then there is a point σ1 in

∑such that τ1 = T (σ1)

and ξ1 = X(σ1). Let Γi be the curve on T passing through (τ1, ξ1) definedparametrically by the equations

t1 = T (σ11 , . . . σ

i−11 , σi, σi+1

1 , . . . σq1)

x1 = X(σ11 , . . . σ

i−11 , σi, σi+1

1 , . . . σq1),

where σi ranges over some open interval (ai, bi). The curve Γi is obtainedby holding all components of the vector σ but the i-th component fixed andletting the i-th component vary over the interval (ai, bi). The curve Γi issometimes called the i-th coordinate curve on T .

We now assume that T is n-dimensional, that each point of T is the ter-minal point of a unique trajectory, and that W can be extended to a C(1)


function in a neighborhood of T . It follows from (6.2.1) and the definition ofW that for (t1, x1) in T

W (t1, x1) = g(t1, x1). (6.2.29)

It therefore follows that (6.2.29) holds along each Γi, i = 1, . . . , n. We maytherefore differentiate (6.2.29) along Γi with respect to σi and get that

Wt∂T

∂σi+ 〈Wx,

∂X

∂σi〉 = gt

∂T

∂σi+ 〈gx,

∂X

∂σi〉

holds along Γi. We rewrite this equation as

〈(Wt − gt, Wx − gx),

(∂T

∂σi,∂X

∂σi

)〉 = 0 i = 1, . . . , n. (6.2.30)

In particular, (6.2.30) holds at σi = σi1. We may therefore take the argument

of Wt,Wx, gt, and gx to be (τ1, ξ1) and the argument of ∂T/∂σi and ∂X/∂σi

to be σ1. Using (6.2.14) we can rewrite (6.2.30) as

⟨(H − gt,Wx − gx),

(∂T

∂σi,∂X

∂σi

)⟩= 0 i = 1, . . . , n, (6.2.31)

where H is evaluated at (τ1, ξ1, u(τ1, τ, ξ),−1,−Wx(τ1, ξ1)).If we use (6.2.18), Eq. (6.2.31) can be written

⟨(H − gt,−λ− gx),

(∂T

∂σi,∂X

∂σi

)⟩= 0 i = 1, . . . , n, (6.2.32)

where λ = λ(τ1, τ, ξ) and H is evaluated at (τ1, ξ1, u(τ1, τ, ξ1),−1, λ(τ1, τ, ξ)).Equations (6.2.32) when written out become

(H − ∂g

∂t

)∂T

∂σi−

n∑

j=1

(λj +

∂g

∂xj

)∂xj

∂σi= 0 i = 1, . . . , n.

Since H = −f0 + 〈λ, f〉, we can rewrite this system as follows:

−(f0 +

∂g

∂t

)∂T

∂σi−

n∑

j=1

∂g

∂xj∂xj

∂σi=

n∑

j=1

(∂xj

∂σi− ∂T

∂σif j

)λj . (6.2.33)

Here the functions f0, f j, ∂g/∂t, and ∂g/∂xj are evaluated at the end point(τ1, ξ1) of the trajectory φ(·, τ, ξ). The partial derivatives of T and X are eval-uated at σ1, the point in Σ corresponding to (τ1, ξ1). If T is a q-dimensionalmanifold, 0 ≤ q ≤ n, then (6.2.33) consists of q equations instead of n equa-tions. This does not follow from our arguments here, but will be shown tohold in the next chapter.

Since the unit tangent vector to Γi is the unit vector in the direction


of (∂T/∂σi, ∂X/∂σi), Eq. (6.2.30) states that at (τ1, ξ1) the vector (Wt −gt,Wx−gx) is either zero or is orthogonal to Γi. We assume that orthogonalityholds. From (6.2.31) and (6.2.32) we see that this statement is equivalent tothe statement that (H − gt,Wx − gx) is orthogonal to Γi at (τ1, ξ1) and tothe statement that (H − gt,−λ − gx) is orthogonal to Γi. Since this is truefor each coordinate curve Γi, i = 1, . . . , n and since the tangent vectors tothe Γi at (τ1, ξ1) generate the tangent plane to T at (τ1, ξ1), the followingstatement is true. The vector (Wt − gt,Wx − gx), or equivalently the vector(H−gt,Wx−gx), or equivalently the vector (H−gt,−λ−gx) evaluated at theend point of an optimal trajectory is orthogonal to T at that point. This is thegeometric statement of the transversality condition. The analytic statementconsists of Eq. (6.2.30), Eq. (6.2.31), or Eq. (6.2.32).

Equations (6.2.29) and (6.2.30) are the boundary conditions for the partialdifferential equation (6.2.14).

Equations (6.2.32), or equivalently, (6.2.33) specify the values of λ(·, τ, ξ)at t = τ1. They therefore furnish the heretofore missing boundary conditionsfor the systems (6.2.23) and (6.2.27). Note that Eqs. (6.2.32) and (6.2.33) arelinear in λ. Thus, λ satisfies a system of linear differential equations with linearboundary conditions. We point out that the system (6.2.23) with boundaryconditions φ(τ, τ, ξ) = ξ and (6.2.32) constitute a two-point boundary valueproblem in that the values of φ are specified at the initial time and the valuesof λ are specified at the terminal time.

Remark 6.2.2. If one solves the partial differential equation (6.2.14) subjectto the boundary conditions (6.2.29) and (6.2.30) by the method of character-istics, one finds that the characteristic equations for the problem are those inEq. (6.2.27). We leave the verification of this to the reader.

6.3 Statement of Maximum Principle

The theorems of this section are statements of the maximum principleunder various hypotheses on the data of the problem. Each statement is aset of necessary conditions satisfied by a relaxed pair (ψ∗, µ∗) that solvesthe relaxed optimal control problem, formulated as Problem 3.2.1, which werestate for the convenience of the reader.

Relaxed Optimal Control Problem: Minimize

J(ψ, µ) = g(e(ψ)) +

∫ t1

t0

f0(t, ψ(t), µt)dt

subject todψ

dt= f(t, ψ(t), µt) e(ψ) ∈ B µt ∈ Ω(t),


where µt ∈ Ω(t) means that µt is concentrated on Ω(t).If the constraint sets are not compact, then we can think of relaxed controls

as discrete measure controls. Since relaxed controls are also discrete measurecontrols, we may consider discrete measure controls even when the constraintsets Ω(t) are compact. We shall henceforth only consider discrete measurecontrols, unless explicitly stated otherwise. We continue to denote such con-trols by Greek letters such as µ and the corresponding discrete probabilitymeasure on Ω(t) by µt.

We now state the assumptions on the data of the problem in the case ofcompact constraints.

Assumption 6.3.1. (i) The function f = (f0, f1, . . . , fn) is defined on aset G0 ≡ I0 ×X0 ×U0, where I0 is an open interval in R, X0 is an openinterval in R

n, and U0 is an open interval in Rm.

(ii) For fixed (x, z) in X0 × U0 the function f(·, x, z) is measurable on I0.

(iii) For fixed z in U0 and almost all t in I0 the function f(t, ·, z) is of classC(1) on X0.

(iv) For almost all t in I0 and all x in X0 the function f(t, x, ·) is continuouson U0.

(v) The mapping Ω is a mapping from I0 to subsets Ω(t) of U0.

(vi) For each compact interval X ⊂ X0, each compact interval I = [t0, t1] ⊂I0, and each compact set Z ⊆ U0, there exists a function M(·) =M(·, I,X , Z) defined on [t0, t1] such that M is in L2[t0, t1] and for all(t, x) in [t0, t1]×X and z in Z,

|f(t, x, z)| ≤M(t) |fx(t, x, z| ≤M(t). (6.3.1)

Here fx(t, x, z) is the (n+ 1)× n matrix whose entry in row i column jis (∂f i/∂xj), i = 0, 1, . . . , n j = 1, . . . , n.

(vii) The set B that specifies the end conditions is a bounded C(1) manifoldof dimension r, where 0 ≤ r ≤ 2n+ 1.

Remark 6.3.2. It follows from (iii) of Assumption 6.3.1 that for each compactinterval I × X and compact Z ⊆ U0 there exists a real valued function Λ inL2[I] such that for each pair of points x and x′ in X

|f(t, x, z)− f(t, x′, z)| ≤ Λ(t)|x− x′|, (6.3.2)

for all t ∈ I and z ∈ Z. To see this note that for each i = 0, 1, . . . , n, the MeanValue Theorem gives

|f i(t, x, z)− f i(t, x′, z| = |〈f ix(t, x+ θ(x − x′), z), (x′ − x)〉|,


where 0 < θ < 1. From the Cauchy-Schwarz inequality, from (6.3.1), and thefact that all norms in a euclidean space are equivalent we get that the rightside of this equality is less than or equal to CM(t)|x − x′|, where M is asin (6.3.1). The inequality (6.3.2) now follows again from the equivalence ofnorms in a euclidean space.

Remark 6.3.3. If µ is a relaxed control defined on a compact interval I ⊂ I0and X is a compact interval contained in X0, then as in (6.3.1), there exists areal valued function M on L2[I] such that

|f(t, x, µt)| ≤M(t) |fx(t, x, µt)| ≤M(t). (6.3.3)

To see this note that

f(t, x, µt) =

∫

Ω(t)

f(t, x, z)dµt.

The first inequality in (6.3.3) follows from the first inequality in (6.3.1) andthe fact that µt is a probability measure. A similar argument establishes theinequality for fx. It follows from Remark 6.3.2 that f(t, x, µt) satisfies a Lip-schitz condition as in (6.3.1).

Remark 6.3.4. If instead of (ii) we require f to be continuous on G0, thenfor each compact interval X ⊆ X0, each compact interval I ⊆ I0, and eachcompact set Z ⊂ U0, the first inequality in (6.3.1) holds with M(t) replaced

by a constant M . If we further require f to be C(1) in X0 for fixed (t, z) in

I0 × U0, and require fx to be continuous on I × X × Z, then the secondinequality in (6.3.1) holds with M constant. Under these assumptions theLipschitz condition (6.3.2) holds with Λ a constant. Analogous statements

hold for f(t, x, µt) and fx(t, x, µt).

The maximum principle is stated most efficiently in terms of a Hamiltonianfunction H defined on I0 ×X0 × U0 × R× R

n by the formula

H(t, x, z, q0, q) = q0f0(t, x, z) + 〈q, f(t, x, z)〉 (6.3.4)

=n+1∑

j=0

qjf j(t, x, z) = 〈f(t, x, z), q〉,

where q = (q0, q).By Hr(t, x, µt, q

0, q) we mean

Hr(t, x, µt, q0, q) =

∫

Ω(t)

H(t, x, z, q0, q)dµt = 〈q,∫

Ω(t)

f(t, x, z)dµt〉. (6.3.5)

We use the subscript r to emphasize that we are considering relaxed controls.The statement of the maximum principle only involves a given relaxed

optimal pair. Therefore, to simplify the typography we shall drop the asterisknotation in designating a relaxed optimal pair. Thus, we write (ψ, µ) insteadof (ψ∗, µ∗) for a relaxed optimal pair.


Theorem 6.3.5 (Maximum Principle in Integrated Form; Compact Con-straints). Let Assumption 6.3.1 hold, let t0 = 0, and let t1 = 1. For each tin [0, 1] let Ω(t) be compact and let there exist a compact set Z ⊆ U0 suchthat each set Ω(t) is contained in Z. Let (ψ, µ) be a relaxed optimal pair withψ(t) ∈ X0 for all t in [0, 1]. Then there exist a constant λ0 ≤ 0 and an ab-solutely continuous function λ = (λ1, . . . , λn) defined on [0, 1] such that thefollowing hold.

(i) The vector λ(t) = (λ0, λ(t)) is never zero on [0, 1].

(ii) For a.e. t in [0, 1]

ψ′(t) = Hrq(t, ψ(t), µt, λ(t)) (6.3.6)

λ′(t) = −Hrx(t, ψ(t), µt, λ(t)).

(iii) For any relaxed admissible control ν defined on [0, 1]

∫ 1

0

Hr(t, ψ(t), µt, λ(t))dt ≥∫ 1

0

Hr(t, ψ(t), νt, λ(t))dt. (6.3.7)

(iv) The 2n vector

(−λ0gx0(e(ψ))− λ(0), −λ0gx1

(e(ψ)) + λ(1))

is orthogonal to B at e(ψ).

An equivalent formulation of conclusion (iv) is that

〈−λ0gx0(e(ψ)) − λ(0), dx0〉+ 〈−λ0gx1

(e(ψ)) + λ(1), dx1〉 = 0 (6.3.8)

for all tangent vectors (dx0, dx1) to B at e(ψ).

Remark 6.3.6. We emphasize that Theorems 6.3.5–6.3.22 give necessaryconditions that a relaxed optimal pair must satisfy. Thus, if we know a priorithat the relaxed optimal pair is an ordinary optimal pair then Theorems 6.3.5–6.3.22 are applicable, and in their statements replace ψ(t) by φ(t), replaceµt by u(t), and replace Hr by H . Theorem 4.4.2 and Corollary 4.4.3 give asufficient condition for a relaxed optimal pair to be an ordinary optimal pair.

If we know that (φ, u) is optimal for the ordinary problem, but do notknow that it is optimal for the relaxed problem, then we cannot apply The-orems 6.3.5–6.3.17. Example 4.4.4 shows that it is possible for the ordinarycontrol problem to have a solution (φ, u) and the relaxed problem to havea solution (ψ, u) with J(ψ, µ) strictly less than J(φ, u). Our theorems areapplicable to (ψ, µ), but not to (φ, u). Such cases are pathological.


Remark 6.3.7. From the definition of Hr it follows that the first equationin (6.3.6) can be written as

ψ′(t) = f(t, ψ(t), µt).

Thus, the first equation in (6.3.6) is a restatement of the fact that (ψ, µ) is anadmissible pair. The second equation in (6.3.6) written in component form isthe system

λi′

(t) = −λ0f0xi(t, ψ(t), µt)−

n∑

j=1

λj(t)f jxi(t, ψ(t), µt) i = 1, . . . , n. (6.3.9)

Since λ0 is a constant, dλ0/dt. If we introduce an additional coordinate x0

and set x = (x0, x), then since f does not depend on x0, we may write

λ′0 = −λ0f0x0(t, ψ, µt)−

n+1∑

j=1

λj(t)f jx0(t, ψ(t), µt).

If we adjoin this equation to (6.3.9) we get that λ = (λ0, λ(t)) satisfies

λ′(t) = −(fx(t, ψ(t), µt))tλ(t), (6.3.10)

where the superscript t denotes transpose and fx is the matrix whose entryin row i column j is (∂f i/∂xj), i = 0, 1, . . . , n and j = 0, 1, . . . , n. Thus, λsatisfies a system of linear homogeneous differential equations.

Remark 6.3.8. Since (λ0, λ(0)) 6= 0 never vanishes, we may divide throughby |(λ0(0), λ(0))| in (6.3.8)–(6.3.10) and relabel to obtain (λ0, λ(t)) such that|(λ0, λ(0))| = 1 and satisfies (6.3.6)–(6.3.8).

We now take up the statement of the necessary conditions for optimalitywhen the constraint sets are not assumed to be compact. We defined therelaxed problem for the case of non-compact constraints in Section 3.5 andelaborated on this definition in Section 4.4. We now repeat the essence of thesediscussions.

In Definition 3.5.3, a discrete measure control on an interval [t0, t1] wasdefined to be a control µ such that each µt has the form

µt =

n+2∑

i=1

pi(t)δui(t),

where each pi is a nonnegative measurable function,∑pi(t) = 1, and each ui

is a measurable function with range in U. Also, a discrete measure control is acontrol µ such that for each t in [t0, t1] the measure µt is a convex combinationof Dirac measures.

If the constraint sets Ω(t) are compact and (ψ, µ) is an admissible relaxed


pair, then by Theorem 3.2.11 there exists a discrete measure control µ suchthat

ψ′(t) = f(t, ψ(t), µt) = f(t, ψ(t), µt),

where f = (f0, f1, . . . , fn). Thus, for every admissible pair (ψ, µ), there existsa discrete measure control µ such that (ψ, µ) is admissible and J(ψ, µ) =J(ψ, µ). Therefore, in considering necessary conditions if the constraint setsΩ(t) are compact we need only consider discrete measure controls.

If µt is a Dirac measure concentrated at u(t), then Hr(t, x, µt, q0, q) =

H(t, x, u(t), q0, q). In general, if µt is a discrete measure control then

Hr(t, x, µt, q0, q) = 〈q, f(t, x, µt)〉 (6.3.11)

=n+1∑

j=0

(n+2∑

k=1

pk(t)f j(t, x, uk(t))qj

)

=n+2∑

k=1

pk(t)

n+1∑

j=0

qjf j(t, x, uk(t))

=

n+2∑

k=1

pk(t)H(t, x, uk(t), q0, q).

Thus, the relaxed Hamiltonian is a convex combination of ordinary Hamilto-nians.

From (6.3.11) we get that (6.3.10) can be written as

λ′(t) = −n+2∑

i=1

pi(t)Hx(t, ψ(t), ui(t), λ). (6.3.12)

Theorem 6.3.9 (Maximum Principle in Integrated Form). Let Assump-tion 6.3.1 hold. Let t0 = 0 and let t1 = 1. Let Y denote the union of the setsΩ(t) as t ranges over [0, 1], and let Y be unbounded. Let there exist a positiveinteger K such that for k > K and all t ∈ [0, 1], Ωk(t) = (cl Ω(t))∩(cl B(0, k))is not empty, where cl denotes closure and B(0, k) is the ball in R

m of radiusk centered at the origin. Let the mapping Ωk : t → Ωk(t) be u.s.c.i. on [0, 1].Let (ψ, µ) be a relaxed optimal pair, with ψ(t) ∈ X0 for all t ∈ [0, 1] and whereµ is a discrete measure control with

µt =

n+2∑

i=1

pi(t)δui(t).

Let there exist a function M in L1[0, 1] such that |f(t, ψ(t), z)| ≤M(t) a.e. forall z in Y. For each i = 1, . . . , n+ 2 let the function t → |fx(t, ψ(t), ui(t))| beintegrable on [0, 1]. For each compact set X ⊂ X0 let there exist a function Λin L1[0, 1] such that for a.e. t ∈ [0, 1] and all x, x′ in X

|f(t, x, ui(t))− f(t, x′, ui(t))| ≤ Λ(t)|x− x′|. (6.3.13)


Then the conclusion of Theorem 6.3.5 holds.

Definition 6.3.10. A triple (ψ, µ, λ) that satisfies the conclusions of Theo-rems 6.3.5 or Theorem 6.3.9 will be called an extremal. The pair (ψ, µ) will

be called an extremal pair and λ a multiplier.

Definition 6.3.11. A vector (t, ψ(t), µt, λ(t)), where (ψ, µ, λ) is an extremalwill be called an extremal element and we shall write

π(t) = (t, ψ(t), µt, λ(t)).

If we strengthen the hypotheses of Theorems 6.3.5 and 6.3.9 by requiringthe mapping Ω to be a constant mapping, we obtain a pointwise statementequivalent to (6.3.7) that is more useful than (6.3.7) in finding extremals.

Theorem 6.3.12 (Pointwise Maximum Principle). Let Assumption 6.3.1 holdwith Ω(t) = C, a fixed set in U0, for all t in I0. Let t0 = 0 and t1 = 1. Let(ψ, µ) be an optimal relaxed pair with ψ(t) ∈ X0 for all t ∈ [0, 1] and with

µt =n+2∑

i=1

pi(t)δui(t),

where u1, . . . , un+2 are measurable functions on [t0, t1] with ui(t) ∈ C, a.e. Fori = 1, . . . , n + 2 let Pi = t : pi(t) > 0. If C is unbounded, let the followingadditional hypotheses hold:

(i) There exists a function M in L[0, 1] such that for all z ∈ C,|f(t, ψ(t), z)| ≤M(t).

(ii) For each i = 1, . . . , n+2, the function t→ |fx(t, ψ(t), ui(t))| is integrableon [0, 1]

(iii) The Lipschitz condition (6.3.13) holds. Let

M(t, x, q) = supz ∈ C : H(t, x, z, q).

Then:

(i) There exists a constant λ0 ≤ 0 and an absolutely continuous vector

λ = (λ1, . . . , λn) defined on [t0, t1] such that λ(t) = (λ0, λ1(t), . . .,λn(t)) never vanishes, and

ψ′(t) = Hrq(t, ψ(t), µt, λ(t)) (6.3.14)

λ′(t) = −Hrx(t, ψ(t), µt, λ(t)).

(ii) Not all of the sets Pi have measure zero. If meas (Pk) 6= 0, then foralmost all t in Pk

M(t, ψ(t), λ(t)) = H(t, ψ(t), uk(t), λ(t)). (6.3.15)


(iii) The transversality condition (6.3.8) holds.

Remark 6.3.13. For each k such that meas Pk 6= 0,

pk(t)H(t, ψ(t), uk(t), λ(t)) = pk(t)M(t, ψ(t), λ(t)) (6.3.16)

for a.e. t in Pk. If t 6∈ Pk, then pk(t) = 0, so (6.3.16) holds in [t0, t1]Pk. If meas

Pk = 0, then pk(t) = 0 a.e. and (6.3.16) again holds. Hence

Hr(t, ψ(t), µt, λ(t)) =

n+2∑

k=1

pk(t)H(t, ψ(t), uk(t), λ(t)) =M(t, ψ(t), λ(t)).

(6.3.17)

Corollary 6.3.14. Let (ψ, µ) be an optimal relaxed pair defined on [0, 1] andlet ν be a discrete measure control on [0, 1] with

νt =

n+2∑

i=1

wi(t)δvi(t)

n+2∑

i=1

wi(t) = 1, wi ≥ 0 vi(t) ∈ C.

ThenHr(t, ψ(t), µt, λ(t)) ≥ Hr(t, ψ(t), νt, λ(t)) a.e. (6.3.18)

Proof. Since each vi(t) ∈ C, we have

M(t, ψ(t), λ(t)) ≥ H(t, ψ(t), vi(t), λ(t)).

Hence

M(t, ψ(t), λ(t)) ≥n+2∑

i=1

wi(t)H(t, ψ(t), vi(t), λ(t)) = Hr(t, ψ(t), νt, λ(t)).

The inequality (6.3.18) now follows from (6.3.17).

Remark 6.3.15. Inequality (6.3.18) is the pointwise version of the maximumprinciple for the relaxed problem. If (6.3.18) holds, then so does (6.3.7).

Remark 6.3.16. The proof of the corollary shows that for (6.3.18) to holdwe need not suppose that νt is the value at t of a discrete measure control ν.Inequality (6.3.18) will hold if we take νt to be a discrete measure on C givenby

νt =n+2∑

i=1

wi(t)δνi(t),

where νi(t) ∈ C.


Theorem 6.3.17. Let (ii), (iii), (iv), and (vi) of Assumption 6.3.1 be replaced

by the following assumptions. The functions f , ft, and fx are continuous onI0 × X0 × U0 and for fixed z in U0, the function f(·, ·, z) is of class C(1) onI0 × X0. Let t0 = 0 and let t1 = 1. Let (ψ, µ) be a relaxed optimal pair withψ(t) ∈ X0 for all t in [0, 1] and with µ a discrete measure control with

µt =n+2∑

i=1

pi(t)δui(t).

Let Ω(t) = C. If C is not compact, let the following hold.

(i) Each ui is bounded.

(ii) There exists a function M in L1[0, 1] such that

|f(t, ψ(t), z)| ≤M(t) a.e. for all z in C. (6.3.19)

(iii) For each compact set X ⊆ X0 there exists a function Λ in L1[0, 1] suchthat for all (t, x) and (t′, x′) in [0, 1]×X , and i = 1, . . . , n+ 2

|f(t, x, ui(t))− f(t′, x′, ui(t))| ≤ Λ(t)(|t− t′|+ |x− x′|). (6.3.20)

Then the conclusions of Theorem 6.3.12 and Corollary 6.3.14 hold. Fur-thermore, there exists an absolutely continuous function h defined on[0, 1] such that

h(t) = Hr(t, ψ(t), µt, λ(t)) a.e. (6.3.21)

andh′(t) = Hrt(t, ψ(t), µt, λ(t)) a.e. (6.3.22)

Definition 6.3.18. A discrete measure control µ is piecewise continuous ifeach of the functions p1, . . . , pn+2, u1, . . . , un+2 is piecewise continuous.

Corollary 6.3.19. If µ is piecewise continuous and C is closed, then themapping t → Hr(t, ψ(t), µt, λ(t)) is absolutely continuous and the derivative

of this mapping is t→ Hrt(t, ψ(t), µt, λ(t)), for all t in [0, 1].

Remark 6.3.20. Combining (6.3.21) and (6.3.22) gives

Hr(t, ψ(t), µt, λ(t)) =

∫ t

0

Hrt(s, ψ(s), µs, λ(s))ds + C a.e. (6.3.23)

If the control problem is autonomous, that is, f is independent of t, thenHr(t, ψ(t), µt, λ(t)) = c a.e.

In the corollary we identified the function t → Hr(t, ψ(t), µt, λ(t)) withh(t). We shall henceforth always make this identification.


Remark 6.3.21. If C is compact, then hypotheses (i), (ii), and (iii) are con-

sequences of the compactness of C and the assumption that f(·, ·, z) is of classC(1) on I0 × X0. Thus, if C is compact hypotheses (i), (ii), and (iii) are not

needed. Also, if C is compact, since f and fx are assumed to be continuous onI0 ×X0 × U0, assumptions (ii), (iii), (iv), and (vi) of Assumption 6.3.1 hold.

In Theorems 6.3.5–6.3.17 we assumed that the initial time t0 and terminaltime t1, were fixed. We now remove this assumption. This will only affect thetransversality condition.

Theorem 6.3.22. In each of Theorems 6.3.5–6.3.17 let the assumption thatt0 and t1 are fixed be removed. Let all the other assumptions of these theoremshold and in Theorems 6.3.5–6.3.12 let the following additional assumptions bemade. For each z in U0 let f(·, ·, z) be of class C(1) on I0 ×X0 and let ft, fxand f be continuous on I0 ×X0 × U0. For each compact set I ⊂ I0 and eachcompact set ⊆ X0 let there exist a function Λ in L1[I] such that for all (t, x)and (t′, x′) in I × X

|f(t, x, ui(t))− f(t′, x′, ui(t))| ≤ Λ(t)(|t− t′|+ |x− x′|).

Then in Theorems 6.3.5–6.3.17 the transversality condition is the following.The (2n+ 2) dimensional vector

(− λ0gt0(e(ψ)) +Hr(π(t0)),−λ0gx0(e(ψ))− λ(t0), (6.3.24)

− λ0gt1(e(ψ))−Hr(π(t1)),−λ0gx1(e(ψ)) + λ(t1)),

is orthogonal to B at e(ψ), where π(t) is as in Definition 6.3.11. The otherconclusions of Theorems 6.3.5–6.3.17 are unchanged.

Remark 6.3.23. The additional assumptions on f(·, ·, z) in Theorems 6.3.5–6.3.12 and the assumption on the Lipschitz condition in Theorems 6.3.9–6.3.12are both present in the assumptions of Theorem 6.3.17.

Remark 6.3.24. Since the transversality condition only involves the natureof B in a neighborhood of the end point of an optimal trajectory, there is noloss of generality in assuming that B can be represented by a single coordinatepatch. We thus assume that B is the image of an open parallelepiped Σ in R

q

under a mapping

t0 = T0(σ) x0 = X0(σ) t1 = T1(σ) x1 = X1(σ), (6.3.25)

where the functions Ti and Xi, i = 0, 1 are C(1) on Σ and the (2n + 2) × rJacobian matrix

(T0σX0σ T1σX1σ)t,

where the superscript t denotes transpose, has rank r everywhere on Σ.


Remark 6.3.25. If (dt0, dx0, dt1, dx1) denotes an arbitrary tangent vector toB at e(ψ), then the transversality condition says that the vector in (6.3.24) isorthogonal to (dt0, dx0, dt1, dx1). Hence the inner product of (6.3.24) with anarbitrary tangent vector (dt0, dx0, dt1, dx1) is zero. Thus,

[− λ0gt0 +Hr(π(t0)]dt0 + 〈−λ0gx0− λ(t0), dx0〉 (6.3.26)

+ [−λ0gt1 −Hr(π(t1)]dt1 + 〈−λ0gx1+ λ(t1), dx1〉 = 0,

for all tangent vectors (dt0, dx0, dt1, dx1), where the partials of g are evaluatedat e(ψ).

Since B has dimension r, the vector space of tangent vectors to B at e(ψ)is r-dimensional. We therefore need only require that (6.3.26) holds for the rtangent vectors in a basis for the tangent space.

In the parametric equations (6.3.25) defining B, let σ be the parametervalue corresponding to the point e(ψ) in B. Let Γi be the i-th coordinatecurve in B, obtained by fixing σj , j 6= i at σj and letting σi vary. The unittangent vector to this curve at e(ψ) is

ci

(∂T0∂σi

(σ),∂X0(σ)

∂σi,∂Ti(σ)

∂σi,∂Xi(σ)

∂σi

).

Therefore, we may replace (6.3.26) by a set of r homogeneous linear equationsfor λ0, λ(t0), and λ(t1)

[− λ0gt0 +Hr(π(t0))]∂T0(σ)

∂σi+ 〈−λ0gx0

− λ(t0),∂X0(σ)

∂σi〉

+ [−λ0gt1 −Hr(π(t1))]∂T1(σ)

∂σi+ 〈−λ0gx1

+ λ(t1),∂X1(σ)

∂σi〉 = 0,

where i = 1, . . . , r. Written out, this system of equations is:

− λ0[(

∂g

∂t0− f

0(t0)

)∂T0∂σi

+

(∂g

∂t1+ f

0(t1)

)∂T1∂σi

(6.3.27)

+n∑

j=1

(∂g

∂xj0

∂Xj0

∂σi+

∂g

∂xj1

∂Xj1

∂σi

)]

=n∑

j=1

λj(t0)

(∂Xj

0

∂σi− f

j(t0)

∂T0∂σi

)+

n∑

j=1

λj(t1)

(fj(t1)

∂T1∂σi

− ∂Xj1

∂σi

),

i = 1, . . . , r, where the partials of T0, T1, X0, and X1 are evaluated at σ, the

partials of g at e(ψ) and fj(t) = f j(t, ψ(t), µt), j = 1, . . . , n. Thus, the end

conditions of λ satisfy a system of linear homogeneous algebraic equations.

Remark 6.3.26. We generalize the transversality condition as follows. We donot assume that the end set B is a C(1) manifold of dimension r and that the


end point e(ψ) is interior to B. Instead we assume that the end point e(ψ) is inB and at e(ψ) there is a set V of (2n+2) dimensional vectors (dt0, dx0, dt1, dx1)with the following property. For each vector (dt0, dx0, dt1, dx1) there is a C

(1)

curvet0 = T0(σ) x0 = X0(σ) t1 = T1(σ) x1 = X1(r)

defined for 0 ≤ σ ≤ σ0 such that the points (T0(σ), X0(σ), T1(σ), X1(σ))are in B and e(ψ) = (T0(0), X0(0), T1(0), X1(0)). Under these assumptions(6.3.26) holds with the equality replaced by ≥ 0. To establish this one needsto slightly modify the arguments used to establish (6.3.26).

We leave it as an exercise to show that if B is a C(1) manifold and e(ψ)is interior to B, then (6.3.27) with = replaced by ≤ can be deduced from theassertion in the preceding paragraph.

In Remark 6.3.6 we indicated how the conclusions of Theorems 6.3.5–6.3.22should be modified if an ordinary pair (φ, u) is known a priori to be a relaxedoptimal pair. We now carry out this modification for Theorems 6.3.12–6.3.22.

Theorem 6.3.27. Let the hypotheses of Theorem 6.3.12 hold. Let the ordi-nary admissible pair (φ, u) be a solution of the relaxed problem. Then J(φ, u)is a solution of the ordinary problem, and the following hold.

(i) There exists an absolutely continuous function λ = (λ0, λ) = (λ0, λ1, . . .,λn) defined on [t0, t1] such that λ0 is either identically minus one or zero,and (λ0, λ(t)) 6= (0, 0) for all t.

(ii) The functions φ and λ satisfy

φ′(t) = Hq(t, φ(t), u(t), λ(t)) (6.3.28)

λ′(t) = −Hx(t, φ(t), u(t), λ(t)).

(iii) For all z in C

M(t, φ(t), λ(t)) = H(t, φ(t), u(t), λ(t)) ≥ H(t, φ(t), z, λ(t)). (6.3.29)

(iv) The 2n-vector

(−λ0gx0(e(φ))− λ(0), −λ0gx1

(e(φ)) + λ(1)) (6.3.30)

is orthogonal to B at e(φ). If the hypotheses of Theorem 6.3.22 hold,then:

(v) The (2n+ 2) vector

(− λ0gt0(e(φ)) +H(π(t0)),−λ0gx0(e(φ))− λ(t0), (6.3.31)

− λ0gt1(e(φ))−H(π(t1)),−λ0gx1(e(φ)) + λ(t1))

is orthogonal to B at e(φ)).

If the hypotheses of Theorem 6.3.17 hold, then:


(vi) There exists a constant c such that

H(t, φ(t), u(t), λ(t)) = c+

∫ t

0

Ht(s, φ(s), u(s), λ(s))ds a.e. (6.3.32)

Proof. That (φ, u) is a solution of the ordinary problem was shown in Theo-rem 4.4.2. The statement that λ0 is either −1 or 0 follows from λ0 ≤ 0 andExercise 6.3.30. All other statements follow from Theorems 6.3.12, 6.3.17, and6.3.22 and the fact that for an ordinary control v, we have v(t) = νt = δv(t)and

Hr(t, x, νt, q) = H(t, x, v(t), q).

The following example shows that, in general, the maximum principle isnot a sufficient condition for optimality.

Example 6.3.28. Let

f0(t, x, z) = az2 − 4bxz3 + 2btz4 a > 0 b > 0.

Let Ω(t, x) = R1, let the state equation be dx/dt = z and let B consist of a

single point (t0, x0, t1, x1) = (0, 0, 1, 0). The relaxed problem is to minimize

J(ψ, µ) =

∫ 1

0

f0(t, ψ(t), µt)dt.

We shall exhibit an ordinary admissible pair (φ, u) and a multiplier λ(t)that is extremal, is such that J(φ, u) = 0, and a sequence (φn, un) of or-dinary admissible pairs such that J(φn, un) → −∞. Since infJ(ψ, µ) : (ψ, µ)admissible ≤ infJ(φ, u) : (φ, u) admissible we will have shown that nei-

ther the relaxed nor ordinary problem has a solution, even though (φ, u, λ) isextremal.

We need to find a (φ, u, λ) that satisfies the conclusions of Theorem 6.3.27.The function H in this problem is given by

H(t, x, z, q) = q0(az2 − 4bxz3 + 2btz4) + qz.

Equations (6.3.28) become

φ′(t) = u(t) (6.3.33)

λ′(t) = λ04bu(t)3.

Since B is a point, the transversality conditions give no information aboutλ0, λ(0) or λ(1).

Let φ, u, and λ be identically zero on [0, 1] and let λ0 = −1. Then (6.3.28)is satisfied and

H(t, φ(t), z, λ(t)) = −z2(a+ 2btz2).


Since a > 0, b > 0 and 0 < t < 1, we have that M(t, φ(t), λ(t)) = 0 and

H(t, φ(t), u(t), λ(t)) = 0. Hence (6.3.29) holds, and so (φ, u, λ) is extremal.On the other hand, let 0 < k < 1 and for n = 2, 3, . . .

un(t) =

nt 0 ≤ t < 1/n

−nk/(n− 1) 1/n ≤ t ≤ 1

and let φn be the corresponding trajectory. It is a straightforward calculation,which we leave to the reader, to show that J(φn, un) → −∞ as n→ ∞.

Exercise 6.3.29. Show that J(φn, un) → −∞ as n→ ∞.

Exercise 6.3.30. (a) Show that if λ0 6= 0, then we may take λ0 = −1.

(b) Let J0 be a point and let J1 be an n-dimensional manifold of class C(1).Show that if an extremal trajectory is not tangent to J1, then λ

0 6= 0.

(c) Show that in this case if λ0 = −1, then λ(t1) is unique.

Exercise 6.3.31 (Isoperimetric Constraints). In the control problem supposethat we impose the additional constraints

∫ t1

t0

hi(t, φ(t), u(t)) = ci i = 1, . . . , p

on admissible pairs (φ, u), where c = (c1, . . . , cp) is a given constant and

h = (h1, . . . , hp) has the same properties as the function f in Theorem 6.3.9.For the relaxed version of the control problem, the additional constraints onrelaxed admissible pairs take the form

∫ t1

t0

hi(t, ψ(t), µt)dt = ci i = 1, . . . , p. (6.3.34)

In Section 2.5 we showed how to transform the problem with isoperimetricconstraints into a control problem without these constraints. Use this transfor-mation and Theorem 6.3.9 to determine a maximum principle for the relaxedproblem with the additional constraints in Eq. (6.3.34). If h is assumed to

have the properties of f in Theorem 6.3.12 and Ω(t) = C for t0 ≤ t ≤ t1, usethe transformation and Theorem 6.3.12 and Corollary 6.3.14 to determine amaximum principle for this relaxed problem.

Exercise 6.3.32 (Parameter Optimization). Consider the problem of control

and parameter optimization described in Section 2.5. Let f be defined andcontinuous on I0×X0×U0×W0, where I0,X0,U0 are as in Assumption 6.3.1and W0 is an open interval in R

k. For fixed z in U0, the function f is of classC(1) on I0 × X0 ×W0. For each compact interval X ⊆ X0, compact interval


W ⊆ W0 and all z in U0 there exists a function M(·) = M(·,X ,W) definedon [t0, t1] such that M is in L2[t0, t] and

|f(t, x, w, u(t)| ≤M(t) |ft(t, x, w, u(t)| ≤M(t)

|fx(t, x, w, u(t)| ≤M(t) |fw(t, x, w, u(t)| ≤M(t).

Let (iv) and (v) of Assumption 6.3.1 hold.

Use the transformation in Section 2.5 to obtain a maximum principle anal-ogous to Theorem 6.3.9 for the relaxed parameter optimization problem.

6.4 An Example

In this section we shall illustrate how the maximum principle and theexistence theorems are used to find an optimal control. The example is alsouseful in pointing out the difficulties to be encountered in dealing with morecomplicated systems. The reader will note that all of the information in themaximum principle is used to solve the problem. In this example, there isan optimal ordinary control pair (φ, u). In Chapter 8, several other examplesare analyzed in detail. For some of those examples, the optimal controls arerelaxed.

We consider a simplified version of the Production Planning Problem 1.2.We assume that the rate of production function F is the identity function, asis the social utility function U . Thus, F (x) = x and U(v) = v. We assumethat there is no depreciation of capital, so δ = 0, and we do not “discount thefuture,” so γ = 0. If we take α = 1 and denote the capital at time t as φ(t)rather than K(t), then the production planning problem can be written as:Minimize

J(φ, u) = −∫ T

0

(1− u(s))φ(s)ds (6.4.1)

subject to

dx

dt= u(t)x x0 = c (6.4.2)

0 ≤ u(t) ≤ 1 x ≥ 0, (6.4.3)

where c > 0, T is fixed, and the terminal state x1 is nonnegative, but otherwisearbitrary.

In the control formulation Ω(t) = [0, 1] for all t and

B = (t0, x0, t1, x1) : t0 = 0, x0 = c, t1 = T, x1 ≥ 0.

Hence B and Ω satisfy the hypothesis of Theorem 4.4.2. Also f0(t, x, z) =


−(1 − z)x and f(t, x, z) = zx, so that f0 and f satisfy the hypotheses ofTheorem 4.4.2. It follows from the state equations and the initial condition(6.4.2) that for any control u satisfying (6.4.3), the corresponding trajectorywill satisfy

c ≤ φ(t) ≤ cet. (6.4.4)

Hence the constraint x ≥ 0 is always satisfied and so can be omitted fromfurther consideration. Since t1 = T , (6.4.4) shows that all trajectories lie in acompact set. The sets Q+(t, x) in this problem are given by

Q+(t, x) = (y0, y) : y0 ≥ (z − 1)x, y = zx, 0 ≤ z ≤ 1= (y0, y) : y0 ≥ y − x, 0 ≤ y ≤ x.

For each (t, x) the set Q+(t, x) is closed and convex. Thus, by Theorem 4.4.2an optimal relaxed pair exists, and is an ordinary pair (φ, u). Hence we canuse Theorem 6.3.27 to determine (φ, u). In the process we shall show that(φ, u) is unique.

The function λ of Theorem 6.3.27 and the optimal pair (φ, u) satisfy

φ′(t) = u(t)φ(t) (6.4.5)

λ′(t) = λ0(1− u(t))− λ(t)u(t)

and the inequality

(λ0 + λ(t))φ(t)u(t) ≥ (λ0 + λ(t))φ(t)z

for all z in [0, 1] and almost all t in [0, T ].Since B consists of a fixed initial point and a terminal set J1 given by t = T ,

any tangent vector to B at any point of B is a scalar multiple of (0, 0, 0, 1).Therefore, the transversality condition reduces to the condition

λ(T ) = 0 (6.4.6)

in the present case. Since λ(t) = (λ0, λ(t)) is never zero, it follows that λ0 6= 0.It then follows from Exercise 6.3.30 that we may take λ0 = −1. Hence from(6.4.5), we get that φ, u and λ satisfy

φ′(t) = u(t)φ(t) (6.4.7)

λ′(t) = −(1− u(t))− λ(t)u(t)

and the inequality

(−1 + λ(t))φ(t)u(t) ≥ (−1 + λ(t))φ(t)z (6.4.8)

for all 0 ≤ z ≤ 1 and a.e. t in [0, T ].It follows from (6.4.7) that φ and λ are solutions of the system of equations

dx

dt= u(t)x (6.4.9)


dq

dt= −(1− u(t))− u(t)q.

The boundary conditions are given by (6.4.6) and φ(0) = c. Thus, we knowthe initial value φ and the terminal value of λ. Not knowing the values of bothφ and λ at the same point makes matters difficult, as we shall see.

Since c > 0, it follows from (6.4.4) that φ(t) > 0 for all t. Thus, althoughwe do not know φ(T ), the value of φ at T , we do know that φ(T ) > 0.

From (6.4.8) we see that for a.e. t, z = u(t) maximizes the expression

(−1 + λ(t))φ(t)z (6.4.10)

subject to 0 ≤ z ≤ 1. Therefore, the sign of the coefficient of z in (6.4.10) ismost important. If this coefficient is > 0, then z = 1 maximizes; if this coef-ficient is < 0, then z = 0 maximizes. Since φ(t) > 0 for all t, the determiningfactor is (−1 + λ(t)).

Let φ(T ) = ξ. Since ξ > 0 and since λ(T ) = 0 the coefficient of z at t = Tin (6.4.10) is negative. Moreover, by the continuity of φ and λ there exists amaximal interval of the form (T − δ, T ] such that the coefficient of z on thisinterval is negative. Hence z = 0 maximizes (6.4.10) for all t in this interval.Hence u(t) = 0 for all T − δ < t ≤ T .

At the initial point t = 0 no such analysis can be made since λ(0) andtherefore the sign of the coefficient of z is unknown. This suggests that weattempt to work backward from an arbitrary point ξ > 0 on the terminalmanifold t = T , as was done in the preceding paragraph.

We have already noted that there exists a maximal interval (T − δ, T ] onwhich u(t) = 0. This is the maximal interval with right-hand end point T onwhich −1 + λ(t) < 0. We now determine δ. From the first equation in (6.4.9)with u(t) = 0 and from the assumption φ(T ) = ξ we see that φ(t) = ξ on thisinterval. From the second equation in (6.4.9) with u(t) = 0 and from (6.4.6)we see that λ(t) = (T − t) on this interval. Therefore, if T > 1 it follows that−1+ λ(t) < 0 for T − 1 < t ≤ T and that −1+ λ(T − 1) = 0. Thus, δ = 1. Insummary, we have established that if T > 1 then on the interval (T − 1, T ],

u(t) = 0 φ(t) = ξ λ(t) = T − t.

If we define u(T − 1) = 0, then the preceding hold on the interval [T − 1, T ].If T ≤ 1, then u(t) = 0, φ(t) = ξ, where ξ = c is the optimal pair. It is clear

from the construction that the optimal pair is unique. Thus if the “planninghorizon” (the value of T ) is too short, the optimal policy is to consume. If youknow that you are going to die tomorrow, live it up today.

We now return to the case in which T > 1 and determine (φ, u) to theleft of T − 1. The reader is advised to graph the functions φ, u, and λ onthe interval [T − 1, T ] and to complete the graph as the functions are beingdetermined to the left of T − 1. We rewrite the second equation in (6.4.9) as

dq

dt= −1 + (1− q)u(t). (6.4.11)


We consider this differential equation for λ on the interval [0, T − 1]. Since λis continuous on [0, T ] we have the terminal condition λ(T − 1) = 1 from thediscussion in the next to the last paragraph.

Since λ(T − 1) = 1 and 0 ≤ u(t) ≤ 1, it follows from (6.4.11) and thecontinuity of λ that on an interval (T −1− δ1, T −1] we have λ′(t) < 0. Henceλ is increasing as we go backward in time on this interval. Hence λ(t) > 1 fort in some interval (T − 1 − δ1, T − 1). Since φ(t) > 0 for all t it follows thatz = 1 maximizes (6.4.10) on (T −1−δ1, T −1). Hence u(t) = 1 on this intervaland Eq. (6.4.9) becomes

dx

dt= x φ(T − 1) = ξ

dq

dt= −q λ(T − 1) = 1.

Hence

λ(t) = exp(T − 1− t) φ(t) = ξ exp(t− T + 1) (6.4.12)

on the interval (T − 1 − δ1, T − 1). But then λ(t) > 1 for all 0 ≤ t < T − 1so that (T − 1− δ1, T − 1) ≡ (0, T − 1). Therefore, on [0, T − 1), λ and φ aregiven by (6.4.12) and u(t) = 1.

At t = 0 we have φ(0) = ξ exp(−T + 1). We require that φ(0) = c. Henceξ = c exp(T − 1), and so

φ(t) = cet

on the interval 0 ≤ t ≤ T − 1.The pair (φ, u) that we have determined is an extremal pair. From the

procedure used to determine (φ, u) it is clear that (φ, u) is unique. Therefore,since we know that an optimal pair exists and must be extremal, it follows that(φ, u) is indeed optimal. Moreover, it is unique. We point out that although theexistence theorem guaranteed the existence of a measurable optimal control,the application of the maximum principle yielded a control that was piecewisecontinuous.

The procedure used in the preceding example, which is sometimes called“backing out from the target,” is one that can often be applied. It illustratesthe difficulties arising because the value of φ is specified at the initial pointand the value of λ is specified at the terminal point. In small-scale problemsthat can be attacked analytically, one can proceed backward from an arbitraryterminal point and adjust the constants of integration to obtain the desiredinitial point. In large-scale problems, or problems that must be solved with acomputer, this is not so easy. We shall not pursue these matters here.


6.5 Relationship with the Calculus of Variations

In this section we investigate the relationship between the maximum prin-ciple and the first order necessary conditions in the calculus of variations. Weshow, in detail, how the classical first order necessary conditions for the sim-ple problem in the calculus of variations can be obtained from the maximumprinciple. In Exercise 6.5.1 we ask the reader to derive the first order nec-essary conditions for the problem of Bolza from the maximum principle. InExercise 6.5.2 we ask the reader to derive the maximum principle for a certainclass of problems from the results stated in Exercise 6.5.1.

In Chapter 2, Section 2.6, we showed that the simple problem in the calcu-lus of variations can be formulated as a control problem as follows. Minimize

J(φ) =

∫ t1

t0


subject todφ

dt= u(t) e(φ) ∈ B Ω(t, x) = U,

where B is a given set in R2n+2, e(φ) denotes the end point (t0, φ(t0), t1, φ(t1))

of φ, and U is an open set in Rn. We shall assume that f0 is of class C(1)

on G = R × U, that B is an r-dimensional manifold of class C(1) in R2n+2,

0 ≤ r ≤ 2n + 1, and that g is identically zero. We also assume that thereexists a function M in L1[I] such that |f0(t, x, z)| ≤M(t) for all (t, x, z) in G.We assume that the relaxed problem has a solution ψ, which is an ordinaryfunction φ with φ′ bounded, and shall show that Theorem 6.3.27 reduces tothe usual first order necessary conditions that a minimizing curve must satisfy.

The function H in the present case is given by the formula

H(t, x, z, q) = q0f0(t, x, z) + 〈q, z〉.

Let φ be a solution of the variational problem and let φ be defined on aninterval [t0, t1]. Then (φ, u) = (φ, φ′) is a solution of the corresponding controlproblem. The pair (φ, u) therefore satisfies the conditions of Theorem 6.3.27.Thus, there exists a scalar λ0 ≤ 0 and an absolutely continuous vector functionλ = (λ1, . . . , λn) defined on [t0, t1] such that (λ0, λ(t)) 6= 0 for all t in [t0, t1]and such that for a.e. t in [t0, t1]

φ′(t) = −Hq(π(t)) = u(t) (6.5.1)

λ′(t) = −Hx(π(t)) = −λ0f0x(t, φ(t), u(t)), (6.5.2)

where π(t) is as in Definition 6.3.11, and

H(t, φ(t), z, λ(t)) = λ0f0(t, φ(t), z) + 〈λ(t), z〉 (6.5.3)


is maximized over U at z = u(t). Moreover, the vector

(H(π(t0)),−λ(t0),−H(π(t1)), λ(t1)) (6.5.4)

is orthogonal to B at e(φ). (Recall that g ≡ 0.)We assert that λ0 6= 0. For if λ0 = 0, then from (6.5.3) we get that for

a.e. t in [t0, t1]〈λ(t), u(t)〉 ≥ 〈λ(t), z〉

for all z in U. This says that for a.e. t the linear function z → 〈λ(t), z〉 ismaximized at some point u(t) of the open set U. This can only happen ifλ(t) = 0. But then (λ0, λ(t)) = 0, which cannot be.

Since λ0 6= 0 we have that λ0 = −1 in (6.5.1) to (6.5.4). From (6.5.2) weget that

λ(t) =

∫ t

t0

f0x(s, φ(s), u(s))ds + c (6.5.5)

for some constant vector c. From (6.5.3) we get that the mapping H(t, φ(t), z,−1, λ(t)) is maximized at z = u(t). Since U is open, u(t) is an interior point ofU. Since the mapping z → H(t, φ(t), z,−1, λ(t)) is differentiable, the derivativeis zero at z = u(t). Thus,

Hz(t, φ(t), u(t),−1, λ(t)) = 0,

and thereforeλ(t) = f0

z (t, φ(t), u(t)) (6.5.6)

for a.e. t in [t0, t1].From (6.5.1), (6.5.5), and (6.5.6) we get that

f0z (t, φ(t), φ

′(t)) =

∫ t

t0

f0x(s, φ(s), φ

′(s))ds+ c a.e. (6.5.7)

Equation (6.5.7) is sometimes called the Euler equation in integrated form orthe du-Bois Reymond equation. In the elementary theory it is assumed thatφ′ is piecewise continuous. Equation (6.5.7) then holds everywhere and thefunction f0∗

z defined by the formula

f0∗z (t) = f0

z (t, φ(t), φ′(t)) (6.5.8)

is continuous even at corners of φ. By a corner of φ we mean a point at whichφ′ has a jump discontinuity. This result is known as the Weierstrass-Erdmanncorner condition.

From (6.5.7) we get that

d

dt(f0∗

z ) = f0∗x a.e., (6.5.9)

where f0∗z is defined by (6.5.8) and f0∗

x denotes the mapping t → f0∗x (t, φ(t),


φ′(t)). Equation (6.5.9) is the Euler equation. If we assume that φ′ is piecewisecontinuous, then (6.5.9) holds between corners of φ.

We next discuss the transversality condition. If in (6.5.4) we take λ0 = −1and use (6.5.1) and (6.5.6), then (6.5.4) becomes

(−f0∗(t0) + 〈f0∗z (t0), φ

′(t0)〉,−f0∗z (t0), f

0∗(t1)− 〈f0∗z (t1), φ

′(t1)〉, f0∗z (t1)),(6.5.10)

where f0∗(t) = f0(t, φ(t), φ′(t)) and f0∗z is given by (6.5.8). The transversality

condition now states that (6.5.10) is orthogonal to B at e(φ).If we take λ0 = −1 and use (6.5.1) and (6.5.6), then the statement that

z = u(t) maximizes (6.5.3) over U becomes the following statement. For almostall t in [t0, t1] and all z in U

− f0(t, φ(t), φ′(t)) + 〈f0z (t, φ(t), φ

′(t)), φ′(t)〉 (6.5.11)

≥ −f0(t, φ(t), z) + 〈f0z (t, φ(t), φ

′(t)), z〉.

If we introduce the function E defined by

E(t, x, z, y) = f0(t, x, z)− f0(t, x, y)− 〈f0z (t, x, y), z − y〉

then (6.5.11) is equivalent to the statement that

E(t, φ(t), z, φ′(t)) ≥ 0 (6.5.12)

for almost all t and all z in U. The inequality (6.5.12) is known as the Weier-strass condition. Note that the left-hand side of (6.5.12) consists of the firstorder terms in the Taylor expansion of the mapping z → f0(t, φ(t), z) aboutthe point z = φ′(t).

If we assume that φ′ is piecewise continuous, then (6.5.12) will certainlyhold between corners. If t = τ is a corner, then by letting t → τ + 0 andt → τ − 0 we get that (6.5.12) holds for the one-sided limits obtained byletting t→ τ ± 0.

We now suppose that for fixed (t, x) in R the function f0 is C(2) on U.

Hence for each t the mapping z → H(t, φ(t), z, λ(t)) is C(2) on U. For a.e. tthis function is maximized at z = u(t), which since U is open must be aninterior point of U. Therefore, the quadratic form determined by the matrixHzz(t, φ(t), u(t), λ(t)) is negative semi-definite for almost all t. But

Hzz(t, φ(t), u(t), λ(t)) = −f0zz(t, φ(t), φ

′(t)).

Hence the quadratic form determined by f0zz(t, φ(t), φ

′(t)) is positive semi-definite for almost all t. Thus, for all η = (η1, . . . , ηn) 6= 0 and a.e. t

〈η, f0zz(t, φ(t), φ

′(t))η〉 ≥ 0. (6.5.13)

This is known as Legendre’s condition.We continue to assume that for fixed (t, x) in R the function f0 is C(2) on


U. Let φ′ be piecewise continuous. We say that φ is non-singular if for all t in[t0, t1] such that t is not a corner, the matrix f0

zz(t, φ(t), φ′(t)) is non-singular.

We shall show that if φ is non-singular, then φ is C(2) between corners. Thisresult is known as Hilbert’s differentiability theorem.

Let τ be a point in [t0, t1] that is not a corner. Consider the followingsystem of equations for the n-vector w:

f0z (t, φ(t), w) −

∫ t

t0

f0x(s, φ(s), φ

′(s))ds− c = 0, (6.5.14)

where c is as in (6.5.7). From (6.5.7) we see that at t = τ , w = φ′(τ) is a so-lution. Moreover, by hypothesis the matrix f0

zz(τ, φ(t), φ′(τ)) is non-singular.

This matrix is the Jacobian matrix for the system of equations (6.5.14). There-fore by the implicit function theorem there exists a unique C(1) solution, sayω, of the system (6.5.14) on an interval (τ −δ, τ+δ) that does not include anycorners of φ. On the other hand, from (6.5.7) we see that φ′ is a solution on(τ−δ, τ+δ). Therefore, φ′ = ω. Hence φ′ is C(1) and φ is C(2) on (τ−δ, τ+δ).Since τ is an arbitrary point between corners we get that φ is C(2) betweencorners.

If we now assume further that f0 is of class C(2) on G = R×U and that φis of class C(2) between corners, we may use the chain rule in (6.5.9) and getthat between corners

f0∗zzφ

′′ + f0∗zxφ

′ + f0∗zt − f0∗

x = 0,

where the asterisk indicates that the functions are evaluated at (t, φ(t), φ′(t))and the functions φ′ and φ′′ are evaluated at t.

Exercise 6.5.1. Consider the problem of Bolza as formulated in Chapter 2,Section 2.6. We now assume that the function F = (F 1, . . . , Fµ) that definesthe differential equation side conditions (2.6.2) is given by

F i(t, x, x′) = x′i −Gi(t, x, x′) i = 1, . . . , µ, (6.5.15)

where x′ = (x′µ+1, . . . , x′n). We assume the functions f0 and F are of class

C(1) on G = R× U. Note that because of (6.5.15) this amounts to assumingthat the function G = (G1, . . . , Gµ) is of class C(1) on an appropriate regionof (t, x, x′)-space. The set B is assumed to be a C(1) manifold of dimensionr, where 0 ≤ r ≤ 2n + 1 and the function g is assumed to be C(1) in aneighborhood of B. Let ρ = (ρ0, ρ) = (ρ0, ρ1, . . . , ρµ) and let

L(t, x, x′, ρ) = ρ0f0(t, x, x′) +

µ∑

i=1

ρiF i(t, x, x′)

= ρ0f0(t, x, x′) + 〈ρ, F (t, x, x′)〉.

Show that under these assumptions if φ is a piecewise C(1) minimizingfunction for the relaxed problem of Bolza, then the following conditions hold.


(a) (Lagrange Multiplier Rule) There exists a constant ψ0 that is either 0or −1 and an absolutely continuous function ψ = (ψ1, . . . , ψµ) definedon the interval [t0, t1] such that for all t, (ψ0, ψ(t)) 6= 0 and

Lx′(t, φ(t), φ′(t), ψ(t)) =

∫ t

t0

Lx(s, φ(s), φ′(s), ψ(s))ds+ c,

where c is an appropriate constant vector and ψ(t) = (ψ0, ψ(t)). More-over, if

P (t) = (t, φ(t), φ′(t), ψ(t)) and A = L− 〈x′, Lx′〉,

then the (2n+ 2)-vector

(−A(P (t0))− ψ0gt0(e(φ)), − Lx′(P (t0))− ψ0gx0(e(φ)),

A(P (t1))− ψ0gt1(e(φ)), Lx′(P (t1))− ψ0gx1(e(φ)))

is orthogonal to B at e(φ).

The last statement is the transversality condition.

(b) (Weierstrass Condition) If

E(t, x, x′, X ′, ρ) = L(t, x,X ′, ρ)− L(t, x, x′, ρ)− 〈Fx′(t, x, x′), X ′ − x′〉,

then for all X ′ and almost all t

E(t, φ(t), φ′(t), X ′, ψ(t)) ≥ 0,

where φ and ψ are as in (a).

(c) (Clebsch Condition) The inequality

〈η, Lx′x′(t, φ(t), φ′(t), ψ(t))η〉 ≥ 0

holds for almost all t and all η = (η1, . . . , ηn) 6= 0 such that

Lx′(t, φ(t), φ′(t), ψ(t))η = 0.

Exercise 6.5.2. Consider the control problem with control constraints asin Chapter 2, Section 2.6 and assume that the constraint qualification (2.6.6)holds. Assume that the terminal set B is a C(1) manifold of dimension r, where0 ≤ r ≤ 2n + 1. Assume that the functions g, f0, f , and R are of class C(1)

on their domains of definition. Assuming that the results of Exercise 6.5.1hold for the problem of Bolza in the calculus of variations, prove the followingtheorem.

Let (φ, u) be an optimal control defined on an interval [t0, t1]. Then thereexists a constant λ0 that is either zero or−1, an absolutely continuous functionλ = (λ1, . . . , λn) defined on [t0, t1], and a measurable function ν = (ν1, . . . , νr)defined on [t0, t1] such that the following hold.


(i) The vector λ(t) = (λ0, λ(t)) never vanishes and ν(t) ≤ 0 a.e.

(ii) For a.e. t in [t0, t1]

φ′(t) = Hq(t, φ(t), u(t), λ(t))

λ′(t) = −Hx(t, φ(t), u(t), λ(t)) −Rx(t, φ(t), u(t))ν(t)

Hz(t, φ(t), u(t), λ(t)) +Rz(t, φ(t), u(t))ν(t) = 0

νi(t)Ri(t, φ(t), u(t)) = 0 i = 1, . . . , r.

(iii) For almost all t in [t0, t1]

H(t, φ(t), u(t), λ(t)) ≥ H(t, φ(t), z, λ(t))

for all z satisfying R(t, φ(t), z) ≥ 0.

(iv) The transversality condition as given in Theorem 6.3.22 holds.

(v) At each t in [t0, t1] let R denote the vector formed from R by takingthose components of R that vanish at (t, φ(t), u(t)). Then for almost allt

〈e, (H(π(t)) + 〈ν(t), R(t, φ(t), u(t))〉)zze〉 ≥ 0

for all non-zero vectors e = (e1, . . . , em) such that Rz(t, φ(t), u(t))e = 0.


In this section we apply the maximum principle to the problem of mini-mizing

J(φ, u) = g(e(φ)) +

∫ t1

t0

(〈a0(s), φ(s)〉 + h0(s, u(s)))ds (6.6.1)

subject todx

dt= A(t)x + h(t, u(t)), (6.6.2)

control constraints Ω and terminal constraints B. The following assumptionwill be made in this section.

Assumption 6.6.1. (i) The constraint mapping Ω is a constant map; thatis, Ω(t) = C for all t, where C is a fixed compact set in R

m.

(ii) The set B is a compact C(1) manifold of dimension r, 0 ≤ r ≤ 2n+ 1.


(iii) The vector functions a0 and h = (h0, h1, . . . , hn) and the matrix functionA are continuous on an interval I0.

(iv) The function g is C(1) in a neighborhood of B in R2n+1.

As noted in the first sentence in the proof of Theorem 4.7.8, we can assumewithout loss of generality that f0 ≡ 0. The functional J is then given by

J(φ, u) = g(e(φ)). (6.6.3)

We henceforth assume that J is given by (6.6.3).By Theorem 4.7.8 an optimal relaxed control exists for the problem of

minimizing (6.6.1) subject to (6.6.2), control constraint Ω, and terminal setB, where the data of the problem satisfy Assumption 6.6.1. Moreover, theoptimal relaxed control is an ordinary control. Before we use Theorem 6.3.27to investigate the form of the optimal pair, we recall some properties of thesolutions of a system of linear homogeneous differential equations. We considerthe linear system

dx

dt= A(t)x (6.6.4)

and the systemdx

dt= −A(t)tx, (6.6.5)

where the superscript t denotes transpose. The system (6.6.5) is said to beadjoint to (6.6.4). The matrix A is assumed to be measurable on some intervalI0.

Lemma 6.6.2. Let Φ(t) denote the fundamental matrix of solutions of (6.6.4)such that Φ(t0) = I, where I is the identity matrix and t0 ∈ I0. Let Ψ(t) denotethe fundamental matrix of solutions of (6.6.5) such that Ψ(t0) = I. Then

Φ−1(t) = Ψ(t)t. (6.6.6)

Proof. By the rule for differentiating a product and (6.6.4) and (6.6.5) we getthat

(ΨtΦ)′ = (Ψt)′Φ+ΨtΦ′ = (−ΨtA)Φ + Ψt(AΦ) = 0.

Hence ΨtΦ = C, where C is a constant matrix. But Φt(t0)Φ(t0) = I · I = I,so C is the identity matrix, and the conclusion follows.

In this problem, the function H is given by

H(t, x, z, q) = 〈q, A(t)x〉 + 〈q, h(t, z)〉. (6.6.7)

Therefore, if aij denotes the element in the i-th row and j-th column of A, weget that

Hxj (t, x, z, q) =n∑

i=1

qiaij(t).


The second equation in (6.3.28) therefore becomes

dq

dt= −A(t)tq, (6.6.8)

which is the same as (6.6.5).Let λ be a solution of (6.6.8) satisfying the initial condition

λ(t0) = η. (6.6.9)

Let Ψ denote the fundamental matrix solution of (6.6.8), which satisfiesΨ(t0) = I. Thus,

Ψ′(t) = −A(t)tΨ(t) Ψ(t0) = I.

Thenλ(t) = Ψ(t)η. (6.6.10)

By (6.6.6), Eq. (6.6.10) can also be written as

λ(t) = (Φ−1)t(t)η.

Assumption 6.6.3. (i) The functions a0 and h and the matrix A are C(1)

on I0.

(ii) The end point e(φ) is interior to B and at e(φ) the vector

(gt0 , gx0, gt1 , gx1

) (6.6.11)

is neither zero nor orthogonal to B.

If we setf∗(t) = A(t)φ(ti) + h(t, u(t)),

then 〈λ, f∗〉 is absolutely continuous, and the transversality condition statesthat the vector

(〈λ(t0), f∗(t0)〉−λ0gt0 , −λ(t0)−λ0gx0, −〈λ(t1), f∗(t1)〉−λ0gt1 , λ(t1)−λ0gx1

),

where all the partial derivatives of g are evaluated at e(φ), is orthogonal to Bat e(φ). By virtue of (6.6.9) and (6.6.10) this vector can also be written as

(〈η, f∗(t0)〉−λ0gt0 , −η−λ0gx0, −〈Ψ(t1)η, f

∗(t1)〉−λ0gt1 , Ψ(t1)η−λ0gx1).

(6.6.12)By (6.6.6) this vector can also be written as

(〈η, f∗(t0)〉 − λ0gt0 ,−η − λ0gx0,−〈(Φ−1(t1))

tη, f∗(t1)〉−λ0gt1 , (Φ−1(t))tη − λ0gx1

). (6.6.13)

We now show that η 6= 0. For if η = 0, then by (6.6.10), λ(t) = 0 on [t0, t1].Hence by the maximum principle λ0 6= 0. Also, if η = 0, Eq. (6.6.12) becomes


−λ0(gt0 , gx0, gt1 , gx1

). Since λ0 6= 0, the transversality condition implies thatthe vector (6.6.11) is either zero or is orthogonal to B at e(φ). This, however,was ruled out.

From (6.6.7) and Theorem 6.3.27(iii), we have that

〈λ(t), h(t, u(t))〉 ≥ 〈λ(t), h(t, z)〉 (6.6.14)

for a.e. t in [t0, t1] and all z in C. From (6.6.10) and (6.6.6) we get that

〈λ(t), h(t, z)〉 = 〈Ψ(t)η, h(t, z)〉 = 〈η,Ψt(t)h(t, z)〉 = 〈η,Φ−1(t)h(t, z)〉.

Thus, (6.6.14) is equivalent to the inequality

〈η,Φ−1(t)h(t, u(t))〉 ≥ 〈η,Φ−1(t)h(t, z)〉. (6.6.15)

We summarize the preceding discussion in the following theorem, whichgives the maximum principle for systems linear in the state variable.

Theorem 6.6.4. (i) Let Assumption 6.6.1 hold. Then the relaxed versionof the problem “Minimize the functional (6.6.3) subject to (6.6.1), con-trol constraints Ω and terminal set B” has a solution that is an ordinaryadmissible pair (φ, u).

(ii) Let Φ denote the fundamental matrix solution of (6.6.4) such thatΦ(t0) = I, and let Assumption 6.6.3 hold. Then there exists a non-zero vector η in R

n and a scalar λ0 ≤ 0 such that the vector (6.6.13) isorthogonal to B at e(φ) and such that for a.e. t in [t0, t1]

max〈η,Φ−1(t)h(t, z)〉 : z ∈ C

occurs at z = u(t).

Remark 6.6.5. By virtue of (6.6.6) the quantity to be maximized can alsobe written as 〈η,Ψt(t)h(t, z)〉.

Remark 6.6.6. Note that, in principle, for systems linear in the state variablewe only need the initial value η of the function λ in order to determine anextremal trajectory. Of course, the terminal value λ(t1) will also do, since by(6.6.10), η = Ψ−1(t1)λ(t1). To determine η we use the transversality condition.This involves knowing e(φ) and u(ti), i = 0, 1. Once u is known the variationof parameters formula gives the extremal trajectory.

Exercise 6.6.7. Obtain an analytic formulation of the transversality con-dition as a system of r linear equations in the unknowns (λ0, η1, . . . , ηn) bysetting the inner product of (6.6.9) with each of r linearly independent tangentvectors to B at e(φ) equal to zero.


6.7 Linear Systems

A linear system is one in which the function h is given by

h(t, z) = B(t)z + d(t), (6.7.1)

where B is an n×m matrix and d is an n-vector. The system (6.6.2) becomes

dx

dt= A(t)x+B(t)z + d(t). (6.7.2)

For (iii) of Assumption 6.6.1 to hold, B and d must be continuous functions.The maximum principle for linear systems is an immediate consequence of

Theorem 6.6.4.

Theorem 6.7.1. Let Assumption 6.6.1 hold.

(i) Then the relaxed version of the problem “Minimize the functional (6.6.3)subject to (6.7.2), control constraints Ω and terminal constraint B” hasa solution that is an ordinary admissible pair (φ, u).

(ii) Let Φ be the fundamental solution matrix of (6.6.4) satisfying Φ(t0) = I,and let Assumption 6.6.3 hold. Then there exists a non-zero vector η inR

n and a scalar λ0 ≤ 0 such that the vector (6.6.13) is orthogonal to Bat e(φ) and such that for a.e. t in [t0, t1]

max〈η,Φ−1(t)B(t)z〉 : z ∈ C (6.7.3)

occurs at z = u(t).

Remark 6.7.2. By virtue of (6.6.6) we can write

max〈η,Ψt(t)B(t)z〉 : z ∈ Cin place of (6.7.3). In either case, we have that z = u(t) maximizes a linearform in z over a set C. To emphasize this we shall let

L(t, η, z) = 〈η,Φ−1(t)B(t)z〉 = 〈η,Ψt(t)B(t)z〉. (6.7.4)

Let ψi = (ψ1i . . . ψ

ni )

t denote the function comprising the i-th column ofΨ, let η = (η1, . . . , ηn), let bj(t) denote the j-th column of B(t), and let bkj

denote the entry in the k-th row and j-th column of B. Then the coefficientof zj in (6.7.4) is

n∑

i,k=1

ηiψki (t)b

kj(t) = ηΨt(t)bj(t).

In many problems the set C is the cube given by

C = z : |zi| ≤ 1, i = 1, . . . ,m. (6.7.5)

In this situation, an optimal control often can be characterized very simply.


Corollary 6.7.3. Let C be given by (6.7.5). For each j = 1, . . . ,m let

Ej(η) = t : ηΨt(t)bj(t) = 0

have measure zero. Then for almost all t in [t0, t1]

uj(t) = signum ηΨt(t)bj(t). (6.7.6)

Proof. Since for a.e. t, u(t) maximizes z → L(t, η, z) over C and sinceηΨt(t)Bj(t) is the coefficient of zj , the result is immediate.

Definition 6.7.4. A linear system (6.7.2) is said to be normal with respect toC on an interval [t0, t1] if for every non-zero n-vector µ and for a.e. t in [t0, t1],

maxL(t, µ, z) : z ∈ C

occurs at a unique point z∗(t) in C.

Note that whether a system is normal on a given interval is determined bythe matrices A and B and by the constraint set C. At the end of this sectionwe shall develop criteria for normality that involve conditions on A,B, and Cthat are relatively easy to verify.

If C is given by (6.7.5), a system is normal if and only if the set Ej(µ) hasmeasure zero for each µ in R

n and each j = 1, . . . ,m. Thus, Corollary 6.7.3states that if a system is normal with respect to C, where C is given by (6.7.5),then the optimal control is given by (6.7.6).

We now investigate the structure of an optimal control when C is a compactconvex set. If C is compact and convex, then by the Krein-Milman Theorem(Lemma 4.7.5) the set of extreme points Ce of C is non-empty. The followingcorollary of Theorem 6.7.1 holds.

Corollary 6.7.5. Let C be a compact and convex set and let the system benormal. If u is an optimal control, then u(t) ∈ Ce for almost all t.

Proof. If the conclusion were false, then u(t) 6∈ Ce for t in a set E of positivemeasure. Hence for t ∈ E, there exist points z1(t) and z2(t) in C and realnumbers α(t) > 0, β(t) > 0, with α(t) + β(t) = 1 such that u(t) = α(t)z1(t) +β(t)z2(t). Since the system is normal, the linear function L(t, η, ·) achieves itsmaximum at a unique point z∗(t) for a.e. t in E. By the maximum principle,the maximum is achieved at u(t), so that z∗(t) = u(t). Hence L(t, η, u(t)) >L(t, η, z1(t)) and L(t, η, u(t)) > L(t, η, z2(t)) a.e. in E. Therefore,

L(t, η, u(t)) = α(t)L(t, η, u(t)) + β(t)L(t, η, u(t))

> α(t)L(t, η, z1(t)) + β(t)L(t, η, z2(t))

= L(t, η, α(t)z1(t) + β(t)z2(t)) = L(t, η, u(t)),

which is a contradiction.


Definition 6.7.6. Let C be a compact polyhedron P with vertices e1, . . . , ek.A control u is said to be polyhedral bang-bang on an interval [t0, t1] if for a.e. tin [t0, t1], u(t) is equal to one of the vertices.

If C = P , Corollary 6.7.5 can be restated as follows.

Corollary 6.7.7. Let the system be normal and let the constraint set be acompact polyhedron P. Then any optimal control is polyhedral bang-bang.

Remark 6.7.8. The bang-bang principle (Theorem 4.7.9) tells us that if uis an optimal control, then there is another optimal control u∗ that is bang-bang. The system is not assumed to be normal. Corollary 6.7.5, on the otherhand, tells us that if a system is normal and the constraint set is compact andconvex, then any optimal control must be bang-bang.

The preceding results do not guarantee uniqueness of the optimal controlfor normal systems. The next theorem gives reasonable conditions under whichan optimal control is unique.

Theorem 6.7.9. Let C be compact and convex, let the system be normal, letB be a relatively open convex subset of a linear variety in R

2n+2, and let g begiven by

g(t0, x0, t1, x1) = g1(x0, x1) + g2(t0, t1), (6.7.7)

where g1 is convex. Let u1 and u2 be two optimal controls defined on the sameinterval [t0, t1]. Then u1 = u2 a.e. on [t0, t1].

Proof. Let φ1 be the trajectory corresponding to u1 and let φ2 be the trajec-tory corresponding to u2. Define u3 = (u1+u2)/2. Since C is convex, u3(t) ∈ C.Let φ3 be the trajectory corresponding to u3 that satisfies the initial conditionφ3(t0) = (φ1(t0) + φ2(t0))/2. Then

φ3(t) = Φ(t)φ3(t0) +1

2

∫ t

t0

Φ−1(s)[B(s)(u1(s) + u2(s)) + 2d(s)]ds

= (φ1(t) + φ2(t))/2

and e(φ3) = (e(φ1) + e(φ2))/2. Since B is a convex subset of a linear variety,it follows that e(φ3) ∈ B. Hence (φ3, u3) is an admissible pair.

Let µ = infJ(φ, u) : (φ, u) admissible. Then

µ = J(φ1, u1) = J(φ2, u2).

Recall that we are assuming that J(φ, u) is given by (6.6.3). From the def-inition of µ, from (6.6.3), from (6.7.7), from the convexity of g1, and theassumption that φ1 and φ2 have the same initial and terminal times we get

µ ≤ J(φ3, u3) = g(e(φ3)) = g((e(φ1)+e(φ2))/2) ≤1

2g(e(φ1))+

1

2g(e(φ2)) = µ.

Thus, J(φ3, u3) = µ, and the pair (φ3, u3) is optimal. By Corollary 6.7.5,u3(t) ∈ Ce a.e. This contradicts the definition of u3 unless u1 = u2 a.e.


Remark 6.7.10. For problems with t0 and t1 fixed, g automatically hasthe form (6.7.7) with g2 ≡ 0. If we assume that g is a convex function of(t0, x0, t1, x1), then the assumption that g has the form (6.7.7) can be dropped.

Definition 6.7.11. The linear system (6.7.2) is said to be strongly normalon an interval [t0, t1] with respect to a constraint set C if for every non-zerovector µ in R

n, maxL(t, µ, z) : z ∈ C is attained at a unique z∗(t) in C at allbut a finite set of points in [t0, t1].

Definition 6.7.12. A control u is said to be piecewise constant on an interval[t0, t1] if there exist a finite number of disjoint open subintervals (τj , τj+1) suchthat the union of the closed subintervals [τj , τj+1] is [t0, t1] and such that u isconstant on each of the open subintervals (τj , τj+1).

The next theorem gives a characterization of the optimal control instrongly normal systems that is of practical significance. Simple criteria forstrong normality will be given in Theorem 6.7.14 and its corollaries.

Theorem 6.7.13. Let (φ, u) be an optimal pair and let Assumptions 6.6.1and 6.6.3 hold. Let the matrix B be continuous and let the constraint set C bea compact polyhedron P. Let the system (6.7.2) be strongly normal on [t0, t1],the interval of definition of (φ, u). Then u is piecewise constant on [t0, t1] withvalues in the set of vertices of P.

Proof. If we remove the points t0, t1 and the finite set of points at whichthe maximum of L(t, η, z) is not achieved at a unique z∗(t), we obtain afinite collection of disjoint open intervals (τj , τj+1) such that the union ofthe closed intervals [τj , τj+1] is the interval [t0, t1]. Let J denote one of theintervals (τj , τj+1). From the assumption of strong normality and the proofof Corollary 6.7.5, it is seen that for each t in J , u(t) is equal to one of thevertices ei, i = 1, . . . , k, of P . Let Mi denote the set of points t in J at whichu(t) = ei. Then not all of the Mi, i = 1, . . . , k are empty, the sets Mi arepairwise disjoint and J = ∪Mi. We now show that if Mi is not empty, then itis open. Let τ ∈Mi. Then

L(τ, η, ei) > L(τ, η, ej) for all j 6= i. (6.7.8)

Since for fixed η, ei the mapping t → L(t, η, ei) is continuous, (6.7.8) holdsin a neighborhood of τ . Hence all points of this neighborhood are in Mi andhence Mi is open. Since J is connected and since J = ∪Mj , where the Mj areopen and pairwise disjoint, it follows that for j 6= i the setMj must be empty.Thus u(t) = ei in J , and the theorem is proved.

The conclusion of Theorem 6.7.13 is much stronger than that of Corol-lary 6.7.7. Here we assert that the optimal control is piecewise constant withvalues at the vertices e1, . . . , ek of P, while in Corollary 6.7.7 we merely assertthat the optimal control is measurable with values at the vertices of P. Ofcourse, the assumptions are more stringent here.


We conclude this section with a presentation of criteria for strong normal-ity.

Theorem 6.7.14. Let the state equations be given by (6.7.2). Let A be ofclass C(n−2) on a compact interval I and let B be of class C(n−1) on I. Letthe constraint set be a compact polyhedron P. Let

B1(t) = B(t) (6.7.9)

Bj(t) = −A(t)Bj−1(t) +B′j−1(t) j = 2, . . . , n.

If for every vector w in Rm that is parallel to an edge of P the vectors

B1(t)w, B2(t)w, . . . , Bn(t)w (6.7.10)

are linearly independent for all t in I, then the system (6.7.2) is stronglynormal with respect to P on I.

Proof. Suppose the conclusion is false. Then there exists a non-zero vector ηin R

n and an infinite set of points E in I such that for t in E, the maximumover P of L(t, η, z) is not achieved at a unique z∗(t) in P. Since for fixed (t, η)the mapping z → L(t, η, z) is linear and since P is a compact polyhedron, themaximum over P of L(t, η, z) is attained on some face of P. Since there areonly a finite number of faces on P, there exists an infinite set E1 ⊂ E and aface PF of P such that for t in E1, the maximum over P is attained on PF .Hence if e1 and e2 are two distinct vertices in PF , L(t, η, e1) = L(t, η, e2) forall t in E1. Hence if w = e1 − e2,

L(t, η, w) = 〈η,Ψt(t)B(t)w〉 = 0

for all t in E1. From the first equation in (6.7.9) we get

L(t, η, w) = 〈η,Ψt(t)B1(t)w〉 = 0 (6.7.11)

for all t in E1.Since E1 is an infinite set and I is compact, E1 has a limit point τ in I.

From (6.7.11) and the continuity of B1 and Ψt we get

L(τ, η, w) = 〈η,Ψt(τ)B1(τ)w〉 = 0. (6.7.12)

By hypothesis, the matrix A is of class C(n−2). Hence the fundamentalmatrix Ψ of the system adjoint to (6.6.4) is of class C(n−1). Since B1 = Band B is assumed to be of class C(n−1), it follows from the first equality in(6.7.9) and from (6.7.4) that the mapping t→ L(t, η, w) is of class C(n−1) onI. Also,

L′(t, η, w) = 〈η,Ψt′(t)B1(t)w〉 + 〈η,Ψt(t)B′1(t)w〉.

From (6.6.5) we get

Ψt′(t) = −Ψt(t)A(t).


If we substitute this into the preceding equation we get

L′(t, η, w) = 〈η,Ψt(t)(−A(t)B1(t) +B′1(t))w〉.

From the second equation in (6.7.9) we get

L′(t, η, w) = 〈η,Ψt(t)B2(t)w〉. (6.7.13)

The derivative of a function has a zero between any two zeros of the func-tion. Therefore, L′(t, η, w) = 0 for all t in an infinite set E2 having τ as a limitpoint. From (6.7.13) and the continuity of Ψ∗ and B2 it follows that

〈η,Ψt(τ)B2(τ)w〉 = 0.

We can proceed inductively in this manner and get

〈η,Ψt(τ)Bj(τ)w〉 = 0 j = 1, . . . , n.

Since the n vectors B1(τ)w, . . . , Bn(τ)w are assumed to be linearly indepen-dent, η 6= 0, and Ψt(τ) is non-singular, this is impossible. This contradictionproves the theorem.

Corollary 6.7.15. Let A and B be constant matrices. If for every vector win R

m that is parallel to an edge of P, the vectors

Bw,ABw,A2Bw, . . . , An−1Bw

are linearly independent, then the system (6.7.2) is strongly normal with re-spect to P on J.

The corollary follows from the observation that if A and B are constantmatrices, then

Bj = (−A)j−1B j = 1, . . . , n.

If the set P is a parallelepiped with axes parallel to the coordinate axis,then the only vectors w that we need consider are the standard basis vectorsw1, . . . , wm in R

m. Here, wi is the m-vector whose i-th component is equal toone and other components are all zero. Let bj denote the j-th column of thematrix B. Then bj = Bwj , and Corollary 6.7.15 yields the following:

Corollary 6.7.16. Let A and B be constant matrices and let P be a par-allelepiped with axes parallel to the coordinate axes. Let bj denote the j-thcolumn of B. For each j = 1, . . . ,m, let

bj , Abj , A2bj , . . . , An−1bj

be linearly independent. Then the system (6.7.2) is strongly normal with re-spect to P on I.


6.8 The Linear Time Optimal Problem

In the linear time optimal problem it is required to transfer a given pointx0 to another given point x1 in minimum time by means of a linear system.More precisely, in the linear time optimal problem it is required to minimize

J(φ, u) = t1

subject to the state equation (6.7.2), constraint condition Ω, and end conditionB, where

B = (t0, x0, t1, x1) : t0 = t′0, x0 = x′0, x1 = x′1,with t′0, x

′0, and x

′1 given. The function g is now g(t1) = t1.

If Ω(t) = C, where C is a fixed compact convex set, if the system (6.7.2)is normal with respect to C, and Assumptions 6.6.1 and 6.6.3 hold, then byCorollary 6.7.15 an optimal control u exists and has the form u(t) ∈ Ce a.e.If C is a compact polyhedron P, then u is polyhedral bang-bang. If u1 and u2are two optimal controls, then since the problem is one of minimizing t1, itfollows that u1 and u2 are both defined on the same interval [t0, t

∗1], where t

∗1

is the minimum time. Hence by Theorem 6.7.9 u1 = u2 a.e. We summarizethis discussion in the following theorem.

Theorem 6.8.1. In the linear time optimal problem if Assumptions 6.6.1 and6.6.3 hold and the system is normal with respect to C, then the optimal controlu is unique and u(t) ∈ Ce a.e.

There is another class of linear time optimal problems with the propertythat extremal controls are unique. For this class, the arguments used to showuniqueness of extremal controls prove directly, without reference to existencetheorems, that an extremal control is unique and is optimal.

Theorem 6.8.2. Let the system equations be given by (6.7.2) with d ≡ 0. LetC be a compact convex set with the origin of Rm an interior point of C. Letthe system be normal with respect to C. Let (φ1, u1) be an extremal pair forthe time optimal problem with terminal state x1 = 0. Let the terminal timeat which φ1 reaches the origin be t1. Let (φ2, u2) be an admissible pair whichtransfers x0 to the origin in time t2 − t0. Then t2 ≥ t1 with equality holdingif and only if u1(t) = u2(t) a.e.

Proof. Suppose there exists a pair (φ2, u2) for which t2 ≤ t1. From the varia-tion of parameters formula we get

0 = Φ(t1)

x0 +

∫ t1

t0

Φ−1(s)B(s)u1(s)ds

0 = Φ(t2)

x0 +

∫ t2

t0

Φ−1(s)B(s)u2(s)ds

,


where Φ is the fundamental matrix for the system (6.6.4) satisfying Φ(t0) = I.If we multiply the first equation by Φ(t1)

−1 on the left and multiply the secondequation by Φ(t2)

−1 on the left we get

−x0 =

∫ t1

t0

Φ−1(s)B(s)u1(s)ds =

∫ t2

t0

Φ−1(s)B(s)u2(s)ds. (6.8.1)

Since u1 is an extremal control there exists a non-zero vector η in Rn

such that for a.e. t in [t0, t1], u1(t) maximizes L(t, η, z) over C. If we compute〈η,−x0〉 in (6.8.1) we get

∫ t2

t0

+

∫ t1

t2

〈η,Φ−1(s)B(s)u1(s)〉ds =∫ t2

t0

〈η,Φ−1(s)B(s)u2(s)〉ds.

Therefore,

∫ t2

t0

L(s, η, u1(s)) − L(s, η, u2(s))ds = −∫ t1

t2

L(s, η, u1(s))ds. (6.8.2)

Since u1 is extremal and the system is normal with respect to C, u1(t) ∈ Cea.e. Since 0 is an interior point of C, u1(t) 6= 0 a.e. and

L(t, η(t), u1(t)) > L(t, η, 0) = 0.

Hence the right-hand side of (6.8.2) is ≤ 0, with equality holding if and onlyif t1 = t2. On the other hand, since the system is normal

L(t, η, u1(t)) ≥ L(t, η, u2(t))

for a.e. t, with equality holding if and only if u1(t) = u2(t) a.e. Hence theintegral on the left in (6.8.2) is ≥ 0 with equality holding if and only if u1(t) =u2(t) a.e. Therefore, each side of (6.8.2) is equal to zero. This implies thatt2 = t1 and u2 = u1 a.e., on [t0, t1] and the theorem is proved.

6.9 Linear Plant-Quadratic Criterion Problem

In the problems studied in this section the state equations are

dx

dt= A(t)x +B(t)z + d(t) (6.9.1)

and the function f0 is given by

f0(t, x, z) = 〈x,X(t)x〉 + 〈z,R(t)z〉. (6.9.2)

Unless stated otherwise, the following assumption will be in effect through-out this section.


Assumption 6.9.1. (i) The matrices A,B,X , and R in (6.9.1) and (6.9.2)are C(1) on an interval [a, b], as is the function d in (6.9.1).

(ii) For each t in [a, b] the matrix X(t) is symmetric, positive semi-definiteand the matrix R(t) is symmetric, positive definite.

(iii) For each t in [a, b],Ω(t) = O, where O is a fixed open set in Rm.

(iv) The set B is the n-dimensional manifold consisting of all points(t0, x0, t1, x1) with (t0, x0) fixed and (t1, x1) in a specified n-dimensionalmanifold J1.

(v) The function g : (t1, x1) → g(t1, x1) is C(1) on R

n+1.

(vi) The controls u are in L2[a, b].

The problem to be studied is that of minimizing

J(φ, u) = g(t1, φ(t1))+1

2

∫ t1

t0

〈φ(s), X(s)φ(s)〉+〈u(s), R(s)u(s)〉 ds (6.9.3)

subject to the state equation (6.9.1), the control constraints Ω, and the termi-nal (t0, x0, t1, x1) in B, where the data of the problem satisfy Assumption 6.9.1.

In Exercise 5.4.20, we showed that this problem has a solution (φ, u) whichis also a solution of the corresponding relaxed problem. Therefore, (φ, u) sat-isfies the conditions of Theorem 6.3.27. We assume the following:

Assumption 6.9.2. The trajectory φ is not tangent to T1 at (t1, φ(t1)).

We now characterize optimal pairs by means of the maximum principle.The function H is given by

H(t, x, z, q) =q0

2〈x,X(t)x〉+ 〈z,R(t)z〉+ 〈q, A(t)x〉 (6.9.4)

+ 〈q, B(t)z〉+ 〈q, d(t)〉.

Thus,Hx(t, x, z, q) = q0X(t)x+A(t)tq.

By (iv) of Assumption 6.9.1, the transversality condition given in Theo-rem 6.3.22 takes the following form. The vector

(−H(π(t1))− λ0gt1 , λ(t1)− λ0gx1),

where the partial derivatives of g are evaluated at (t1, φ(t1)), is orthogonalto T1 at (t1, φ(t1). It follows from (iv) of Assumption 6.9.1, from Assump-tion 6.9.2, and from Exercise 6.3.30 that λ0 6= 0, and that we may takeλ0 = −1. With λ0 = −1, the vector λ(t1) is unique. The transversality condi-tion now states that the vector

(gt1 + f01 − 〈λ(t1), f1〉, gx1

+ λ(t1)) (6.9.5)


is orthogonal to T1 at (t1, φ(t1)), where the partial derivatives of g are eval-uated at (t1, φ(t1)), f

01 denotes (6.9.2) evaluated at (t1, φ(t1), u(t1)), and f1

denotes the right-hand side of (6.9.1) evaluated at (t1, φ(t1), u(t1)).From Theorem 6.3.27 we have that

dφ

dt= A(t)φ(t) +B(t)u(t) + d(t) (6.9.6)

dλ

dt= X(t)φ(t)−At(t)λ(t)

and that (6.3.29) holds. From (6.9.4) we see that in the present context,(6.3.29) becomes

−1

2〈u(t), R(t)u(t)〉+ 〈λ(t), B(t)u(t)〉 ≥ −1

2〈z,R(t)z〉+ 〈λ(t), B(t)z〉

for all z in O and almost all t in [t0, t1]. Thus, for almost every t in [t0, t1] themapping

z → −1

2〈z,R(t)z〉+ 〈λ(t), B(t)z〉 (6.9.7)

attains its maximum over O at z = u(t). But O is open, so the derivative ofthe mapping (6.9.7) is zero at z = u(t). Hence

−R(t)u(t) +Bt(t)λ(t) = 0.

Since R(t) is non-singular for all t, we get that

u(t) = R−1(t)Bt(t)λ(t) a.e. (6.9.8)

Note that since B,R, and λ are continuous, the optimal control is alsocontinuous.

If we now substitute (6.9.8) into the first equation in (6.9.6), we get thefollowing theorem from the maximum principle.

Theorem 6.9.3. Let (φ, u) be an optimal pair with interval of definition[t0, t1]. Let Assumption 6.9.2 hold. Then there exists an absolutely continuousfunction λ = (λ1, . . . , λn) defined on [t0, t1] such that (φ, λ) is a solution ofthe linear system

dx

dt= A(t)x +B(t)R−1(t)Bt(t)q + d(t) (6.9.9)

dq

dt= X(t)x−At(t)q

and such that vector (6.9.5) is orthogonal to T1 at (t1, φ(t1)). The optimalcontrol is given by (6.9.8).

We now specialize the problem by taking T1 to the hyperplane t1 = T ;that is,

T1 = (t1, x1) : t1 = T, x1 free, (6.9.10)


and by taking g to be given by

g(x1) =1

2〈x1, Gx1〉, (6.9.11)

where G is a positive semi-definite symmetric matrix. We suppose that T < b.

Remark 6.9.4. If (6.9.10) holds, then every tangent vector (dt0, dx0, . . .,dxn) to T1 has its first component equal to zero. On the other hand, a tangentvector to the trajectory φ has its first component always different from zero.Moreover, it follows from (6.9.1) and the continuity of an optimal control uthat the trajectory has a tangent vector at all points. Hence if (6.9.10) holds,then Assumption 6.9.2 is automatically satisfied.

Corollary 6.9.5. If (6.9.10) and (6.9.11) hold, then φ and λ satisfy thesystem (6.9.16) subject to the boundary conditions

φ(t0) = x0 λ(T ) = −Gφ(T ). (6.9.12)

The first condition is a restatement of the initial condition already imposed.The second follows from the orthogonality of (6.9.5) to T1 at the terminal pointof the trajectory and from (6.9.11).

An admissible pair (φ, u) that satisfies the conditions of Theorem 6.9.3 willbe called an extremal pair. If (6.9.10) and (6.9.11) hold, then an extremal pairsatisfies (6.9.12).

In the next theorem we show that if (6.9.10) and (6.9.11) hold, then an ex-tremal pair is unique and must be optimal. This will be done without referenceto any existence theorems previously established.

Theorem 6.9.6. Let (6.9.10) and (6.9.11) hold. Let (φ, u) be an extremalpair and let (φ1, u1) be any other admissible pair. Then J(φ1, u1) ≥ J(φ, u),with equality holding if and only if u = u1. In that event, φ = φ1.

Proof. First note that because the system (6.9.1) is linear and (t0, x0) is fixed,if u = u1 then φ = φ1. Let

φf = φ(T ) φ1f = φ1(T ).

Since X(t) is positive semi-definite and R(t) is positive definite for all tand since G is positive semi-definite, we get

0 ≤ 〈(φ1f − φf ), G(φ1f − φf )〉

+

∫ T

t0

〈(φ1 − φ), X(φ1 − φ)〉 + 〈(u1 − u), R(u1 − u)〉dt,

with equality holding if and only if u1 = u. Hence

0 ≤ 2J(φ1, u1) + 2J(φ, u)− 2〈φ1f , Gφf )− 2

∫ T

t0

〈φ1, Xφ〉+ 〈u1, Ru〉dt,


which we rewrite as

J(φ1, u1) + J(φ, u) ≥ 〈φ1f , Gφf 〉+∫ T

t0

〈φ1, Xφ〉+ 〈u1, Ru〉dt. (6.9.13)

Since (φ, u) is an extremal pair, there is an absolutely continuous vector λsuch that λ and φ are solutions of (6.9.9) that satisfy (6.9.12) and such that(6.9.8) holds. We now substitute for Xφ in the right-hand side of (6.9.13) fromthe second equation in (6.9.9) and substitute for u in the right-hand side of(6.9.13) from (6.9.8). We get

J(φ1, u1)+J(φ, u) ≥ 〈φ1f , Gφf 〉+∫ T

t0

〈φ1, λ′+Atλ〉+〈u1, Btλ〉dt. (6.9.14)

The integral on the right in (6.9.14) can be written as

∫ T

t0

〈φ1, λ′〉+ 〈Aφ1 +Bu1, λ〉dt.

Since (φ1, u1) is admissible we have from (6.9.1) that

Aφ1 +Bu1 = φ′1 − d.

Substituting this into the last integral gives

∫ T

t0

〈φ1, λ′〉+ 〈φ′1, λ〉 − 〈d, λ〉dt.

Therefore, we can rewrite (6.9.14) as follows:

J(φ1, u1) + J(φ, u) ≥ 〈φ1f , Gφf 〉+ 〈φ1f , λ(T )〉 − 〈x0, λ(t0)〉 −∫ T

t0

〈d, λ〉dt.

If we now use (6.9.12) we get

J(φ1, u1) + J(φ, u) ≥ −〈x0, λ(t0)〉 −∫ T

t0

〈d, λ〉dt.

Recall that equality holds if and only if u1 = u, in which case φ1 = φ.Therefore if we take u1 = u in the preceding inequality we get

2J(φ, u) = −〈x0, λ(t0)〉 −∫ T

t0

〈d, λ〉dt. (6.9.15)

Substituting (6.9.15) into the preceding inequality gives

J(φ1, u1) ≥ J(φ, u),

with equality holding if and only if u1 = u.


Exercise 6.9.7. Consider the linear quadratic problem with t0 and t1 fixed,T1 a linear variety of dimension n, and g : x1 → g(x1) a convex function.Suppose also that the constraint set O is convex. Show, without appealing tothe maximum principle of Theorem 6.9.3, that if (φ, u) is an optimal pair then(φ, u) is unique.

The linear plant quadratic criterion problem posed in this section, with T1as in (6.9.10) and g as in (6.9.11), admits a very elegant and relatively simplesynthesis of the optimal control. The determination of this synthesis will takeup the remainder of this section.

For the problem with fixed initial point (τ, ξ), with a ≤ τ < T andξ ∈ R

n, it follows from Exercise 5.4.20 that there exists an ordinary opti-mal pair (φ(·, τ, ξ), u(·, τ, ξ)) that is a solution of the ordinary problem. ByTheorem 6.9.3 this pair is extremal and for τ ≤ t ≤ T

u(t, τ, ξ) = R−1(t)Bt(t)λ(t, τ, ξ).

It then follows from Theorem 6.9.6 that the optimal pair for the problem withinitial point (τ, ξ) is unique. Therefore, as in Section 6.2, we obtain a field F ofoptimal trajectories. We obtain a synthesis of the optimal control, or feedbackcontrol, U as follows:

U(τ, ξ) = u(τ, τ, ξ) = R−1(τ)Bt(τ)λ(τ, τ, ξ). (6.9.16)

This holds for all a ≤ τ < T and for all ξ, since we may choose any such (τ, ξ)to be the initial point for the problem.

The feedback law in (6.9.16) is not satisfactory since it requires knowingthe value of the adjoint variable, or multiplier, λ at the initial point. If theformalism of Section 6.2 is valid, then we have Wx(τ, ξ) = −λ(τ, τ, ξ) and wecan write

U(τ, ξ) = −R−1(τ)Bt(τ)Wx(τ, ξ). (6.9.17)

This leads us to investigate the value function W for the present problem. Weshall proceed formally, as in Section 6.2, assuming that all functions have therequired number of derivatives existing and continuous. In this way we shallobtain insights and conjectures as to the structure of the feedback control.We shall then show rigorously, by other methods, that these conjectures arevalid.

We henceforth suppose that d = 0 in (6.9.1).The function W satisfies the Hamilton-Jacobi equation (6.2.11), which in

the present case becomes

Wt = −1

2〈x,Xx〉 − 1

2〈U,RU〉 − 〈Wx, Ax〉 − 〈Wx, BU〉.

In this relation and in what follows we shall omit the arguments of the func-tions involved. Using (6.9.17) we can rewrite this equation as follows:

Wt = −1

2〈x,Xx〉 − 1

2〈R−1BtWx, B

tWx〉 − 〈Wx, Ax〉 + 〈Wx, BR−1BtWx〉.


Hence

Wt = −1

2〈x,Xx〉+ 1

2〈Wx, BR

−1BtWx〉 − 〈Wx, Ax〉. (6.9.18)

The form of Eq. (6.9.18) leads to the conjecture that there exists a solutionof the Hamilton-Jacobi equation (6.2.11) of the form

W (t, x) =1

2〈x, P (t)x〉, (6.9.19)

where for each t, P (t) is a symmetric matrix. For then

Wx = Px Wt =1

2〈x, P ′(t)x〉, (6.9.20)

and for proper choice of P (t) we would have a quadratic form in the left equalto a quadratic form on the right.

If we assume a solution of the form (6.9.19), substitute (6.9.20) into(6.9.18), and recall that P t = P , we get

1

2〈x, P ′x〉 = −1

2〈x,Xx〉+ 1

2〈x, PBR−1BtPx〉 − 〈x, PAx〉. (6.9.21)

For any matrix M , we can write

M =(M +M t)

2+

(M −M t)

2.

Hence

〈x,Mx〉 = 1

2〈x, (M +M t)x〉 for all x.

If we apply this observation to the matrix PA in (6.9.21) we get

1

2〈x, P ′x〉 = −1

2〈x,Xx〉+ 1

2〈x, PBR−1BtPx〉 − 1

2〈x, (PA +AtP )x〉.

Therefore, if a solution to the Hamilton-Jacobi equation of the form (6.9.19)exists, the matrix P must satisfy the following differential equation:

P ′ = −X + PBR−1BtP − (PA+AtP ). (6.9.22)

Moreover, since

W (T, x1) = g(x1) =1

2〈x1, Gx1〉,

it follows from (6.9.19) that the solution of (6.9.22) must satisfy the initialcondition

P (T ) = G. (6.9.23)

Equation (6.9.22) is sometimes called the matrix Riccati equation. If a


solution of (6.9.22) satisfying (6.9.23) exists, then from the first relation in(6.9.20) and from (6.9.17) we would expect the optimal synthesis or feedbackcontrol law to be given by

U(t, x) = −R−1(t)Bt(t)P (t)x. (6.9.24)

Note that the control law is linear in x, and its determination merely requiresthe solution of an ordinary differential equation.

We now show that the state of affairs suggested by the analysis in the lastfew paragraphs is indeed true.

Theorem 6.9.8. Let T1 be as in (6.9.10), let g be as in (6.9.11), let d = 0 in(6.9.1), and let the constraint set O contain the origin. Then the problem ofminimizing (6.9.3) subject to (6.9.1), control constraint Ω, and terminal setB, where the data of the problem satisfy Assumption 6.9.1, has an optimalsynthesis. This synthesis is given by (6.9.24) and holds for all a ≤ t < T andall x in Rn. The matrix P (t) is symmetric for each t and the function P isa solution, defined for all a ≤ t ≤ T of the matrix Riccati equation (6.9.22)with initial condition (6.9.23).

Proof. It follows from standard existence and uniqueness theorems for or-dinary differential equations that (6.9.22) has a unique solution satisfying(6.9.23) on some interval (T − δ, T ]. Note that if P is a solution of (6.9.22)satisfying (6.9.23), then so is P t. By the uniqueness of solutions we then getthat P = P t, so that P is symmetric.

Let τ be any point on (T − δ, T ) and let ξ be any point in Rn. We shall use

the solution P obtained in the previous paragraph to construct an extremalfor the problem with initial point (τ, ξ). By Theorem 6.9.6 this extremal willbe unique and will furnish the minimum for the problem with initial point(τ, ξ).

Consider the linear system

dx

dt= A(t)x −B(t)R−1(t)Bt(t)P (t)x (6.9.25)

subject to initial conditions x(τ) = ξ. We denote the solution of this systemby φ(·, τ, ξ). This solution is defined on the interval of definition of P and isunique.

Letλ(t, τ, ξ) = −P (t)φ(t, τ, ξ), τ ≤ t ≤ T. (6.9.26)

Note thatλ(T, τ, ξ) = −P (T )φ(T, τ, ξ) = −Gφ(T, τ, ξ),

where the last equality follows from (6.9.23). Thus, λ satisfies (6.9.12).If we differentiate (6.9.26) and then use (6.9.22) and (6.9.25), we get

dλ

dt= −dP

dtφ− P

dφ

dt


= (X − PBR−1BtP + PA+AtP )φ− P (A−BR−1BtP )φ

= Xφ+AtPφ.

If we now use (6.9.26) we get

dλ

dt= Xφ−Atλ.

Hence by Theorem 6.9.3 and Corollary 6.9.5, φ(·, τ, ξ), and λ(·, τ, ξ) determinean extremal pair (φ(·, τ, ξ), u(·, τ, ξ)) with

u(t, τ, ξ) = R−1(t)Bt(t)λ(t, τ, ξ).

It now follows from Theorem 6.9.6 that this extremal pair is the unique optimalpair for the problem.

From the definition of λ in (6.9.26), and the last equation it follows that

u(t, τ, ξ) = −R−1(t)Bt(t)P (t)φ(t, τ, ξ).

Therefore, since φ(τ, τ, ξ) = ξ,

u(τ, τ, ξ) = −R−1(τ)Bt(τ)P (τ)ξ.

Since (τ, ξ) is an arbitrary point in (T − δ, T )×Rn and since the optimal pair

from (τ, ξ) is unique we obtain a synthesis of the optimal control by setting

U(τ, ξ) = u(τ, τ, ξ).

Hence the optimal synthesis in (T − δ, T )× Rn can be written as

U(t, x) = −R−1(t)Bt(t)P (t)x,

where we have written a generic point as (t, x) instead of (τ, ξ). This, however,is precisely the relation (6.9.24).

We now show that the solution P of (6.9.22) with initial condition (6.9.23)is defined on the entire interval [a, T ]. It will then follow that (6.9.24) holdsfor all a ≤ t ≤ T and all x in R

n.Let us now suppose that δ > 0 is such that (T−δ, T ] is the maximal interval

with T as the right-hand end point on which the solution P is defined. Fromthe standard existence theorem in the theory of ordinary differential equationsand from the form of Eq. (6.9.22) it follows that P (t) must be unbounded ast→ T − δ from the right. We shall show that if T − δ ≥ a, then P is boundedas t → T − δ from the right. From this it will, of course, follow that P isdefined on [a, T ] and (6.9.25) holds for all a ≤ t ≤ T and x in R

n.From the existence theorem for linear quadratic problems (Exercise 5.4.20)

and from Theorem 6.9.6 it follows that for all (τ, ξ) in [a, T ]×Rn the function

W (τ, ξ) = J(φ(·, τ, ξ), u(·, τ, ξ))


is defined, where (φ(·, τ, ξ), u(·, τ, ξ)) is the unique optimal pair for the problemwith initial point (τ, ξ). The functionW so defined is called the value function

or value. Let φ(·, τ, ξ) denote the trajectory for the problem corresponding tothe control u, where u(t) = 0 on [τ, T ]. Then

0 ≤W (τ, ξ) ≤ J(φ(·, τ, ξ), u), (6.9.27)

where the leftmost inequality follows from (ii) of Assumption 6.9.1 and from(6.9.11). From (6.9.1) with d = 0 we see that

φ(t, τ, ξ) = Φ(t, τ)ξ,

where Φ(·, τ) is the fundamental matrix for the system dx/dt = A(t)x satis-fying Φ(τ, τ) = I. Therefore, from (6.9.3) and (6.9.11), we get

J(φ(·, τ, ξ), u) = 〈Φ(T, τ)ξ,GΦ(T, τ)ξ〉 +∫ T

τ

〈Φ(s, τ)ξ,X(s)Φ(s, τ)ξ〉ds.

From this it follows that given a compact set X in Rn, there exists a constant

M , depending on X such that for all a ≤ τ ≤ T and all ξ in XJ(φ(·, τ, ξ), u) ≤M.

Combining this inequality with (6.9.27) shows that given a compact set X inR

n, there exists a constant M , depending on X such that for all a ≤ τ ≤ Tand all ξ in X

0 ≤W (τ, ξ) ≤M. (6.9.28)

In (6.9.15), which was derived in the course of proving Theorem 6.9.6, wehave an expression for J(φ(τ, ξ), u(τ, ξ)), and hence for W (τ, ξ). Since d = 0in the present discussion, we have from (6.9.15)

W (τ, ξ) = −1

2〈ξ, λ(τ, τ, ξ)〉. (6.9.29)

Here λ is the adjoint function, or multiplier function, for the problem withinitial point (τ, ξ). It is not assumed here that λ is given by (6.9.26).

We now consider points (τ, ξ) such that T − δ < τ ≤ T . For such points(6.9.26) holds. Thus,

λ(τ, τ, ξ) = −P (τ)φ(τ, τ, ξ) = −P (τ)ξ.Substituting this into (6.9.29) gives

W (τ, ξ) =1

2〈ξ, P (τ)ξ〉, (6.9.30)

which is valid for T − δ < τ ≤ T and all ξ. From this and from (6.9.28) weget that for all T − δ ≤ τ ≤ T and all ξ in a compact set X ,

0 ≤ 1

2〈ξ, P (τ)ξ〉 ≤M.

Hence P (τ) must be bounded on T−δ ≤ τ ≤ T , and the theorem is proved.


In the course of proving Theorem 6.9.8 we also proved the following.

Corollary 6.9.9. The value functionW is given by (6.9.30) for all a ≤ τ ≤ Tand ξ in R

n.

Chapter 7

Proof of the Maximum Principle

7.1 Introduction

In this chapter we prove Theorems 6.3.5 through 6.3.22 and their corol-laries. Theorem 6.3.5 will be proved by a penalty function method, whichwe outline here. For simplicity, let f be a real valued differentiable functiondefined on an open set X in R

n. Consider the unconstrained problem of mini-mizing f on X . If f attains a minimum at a point x0 in X , then the necessarycondition df(x0) = 0 holds, where df is the differential of f . This condition isobtained by making a perturbation x0 + εδx, where δx is arbitrary but fixed,and ε is sufficiently small so that x0+εδx is in X . Then since f is differentiableand attains a minimum at x0,

f(x0 + εδx)− f(x0) = df(x0)εδx+ θ(εδx) ≥ 0,

where θ(εδx)/(εδx) → 0 as ε → 0. In the rightmost inequality if we firstdivide through by ε > 0 and then let ε → 0, we get df(x0)δx ≥ 0. If wedivide through by ε < 0 and then let ε→ 0, we get df(x0)δx ≤ 0. Since δx isarbitrary, we get df(x0) = 0.

Now consider the constrained problem of minimizing f over those points inX that satisfy the constraint g(x) = 0, where g is a differentiable function de-fined on X . Again let f attain its minimum at x0 for the constrained problem.The perturbation δx is now not arbitrary but must be such that for sufficientlysmall ε not only must x0 + εδx be in X but it must satisfy g(x0 + εδx) = 0.Thus, the argument used in the unconstrained problem, which must hold forarbitrary δx, fails and other arguments must be used.

In the penalty function method one considers a sequence of unconstrainedproblems: Minimize

F (x,Kn) = f(x) +Kn(g(x))2 x ∈ X , (7.1.1)

where Kn → ∞ as n → ∞. Conditions are placed on f and g such that thefollowing occur. For each n, the problem of minimizing F (x,Kn) has a solutionxn, at which point the unconstrained necessary condition dF (xn, Kn) = 0holds. As Kn → ∞, the ever increasing penalty forces the xn to converge to apoint x0 that satisfies the constraint g(x0) = 0 and satisfies f(x0) ≤ f(x) for

205


all x such that g(x) = 0. As n → ∞ the necessary condition dF (xn, Kn) = 0converges to the necessary condition for the constrained problem. We shallcarry out this program for the relaxed optimal control problem.

Our proof of Theorem 6.3.5 is suggested by E. J. McShane’s penaltymethod proof [59] of the necessary conditions for the following finite dimen-sional problem.

Problem 7.1.1. Let X0 be an open convex set in Rn. Let f,g, and h be C1

functions with domain X0 and ranges in R1, Rm, and R

k, respectively. Let

X = x : x ∈ X0, g(x) ≤ 0, h(x) = 0.

Minimize f over X .

The problem is often stated as:

Minimize f(x)

Subject to: g(x) ≤ 0 h(x) = 0.

If the constraints g(x) ≤ 0 and h(x) = 0 are absent, the problem becomesan unconstrained problem. From elementary calculus we have that a necessarycondition for a point x∗ to be a solution is that all first-order partial derivativesof f vanish at x∗.

The following notation will be used. If f is a real valued function andg = (g1, . . . , gm) is a vector valued function that is differentiable at x0, then

∇f(x0) =

(∂f

∂x1(x0),

∂f

∂x2(x0), . . . ,

∂f

∂xn(x0)

)

and

∇g(x0) =

∇g1(x0)∇g2(x0)

...∇gm(x0)

=

(∂gi(x0)

∂xj

).

By B(x, ε) we shall mean the open ball of radius ε centered at x. A point xsuch that g(x) ≤ 0 and h(x) = 0 will be called a feasible point. To keep theessentials of our proof of Theorem 6.3.5 from being obscured by the technicalrequirements imposed by the infinite dimensionality of our problem, we presentin the next section, McShane’s proof of the necessary conditions for a solutionto Problem 7.1.1.

Proof of the Maximum Principle 207

7.2 Penalty Proof of Necessary Conditions in FiniteDimensions

Theorem 7.2.1. Let x∗ be a solution of Problem 7.1.1. Then there exists areal number λ0 ≥ 0, a vector λ ≥ 0 in R

m, and a vector µ in Rk such that

(i) (λ0,λ,µ) 6= 0

(ii) 〈λ,g(x∗)〉 = 0 and

(iii) λ0∇f(x∗) + λt∇g(x∗) + µt∇h(x∗) = 0.

Vectors (λ0,λ,µ) having the properties stated in the theorem are calledmultipliers, as are the components of these vectors.

Remark 7.2.2. The necessary condition asserts that λ0 ≥ 0 and not thestronger statement λ0 > 0. If λ0 > 0, then we may divide through by λ0in (ii) and (iii) and then relabel λ/λ0 and µ/λ0 as λ and µ, respectively,and thus obtain statements (i) through (iii) with λ0 = 1. In the absence offurther conditions that would guarantee that λ0 > 0, we cannot assume thatλ0 = 1. Theorems with conditions guaranteeing λ0 > 0 are called Karush-Kuhn-Tucker or KKT theorems; those with λ0 ≥ 0 are called Fritz-John orF-J theorems.

Remark 7.2.3. If g ≡ 0, that is, the inequality constraints are absent, The-orem 7.2.1 becomes the Lagrange multiplier rule.

Remark 7.2.4. Since λ ≥ 0 and g(x∗) ≤ 0, condition (ii) is equivalent to

(i)′ λigi(x∗) = 0 i = 1, . . . ,m.

Condition (iii) is a system of n equations

(iii)′ λ0∂f∂xj

(x∗) +∑m

i=1 λi∂gi(x∗)∂xj

+∑k

i=1 µi∂hi(x∗)

∂xj= 0, j = 1, . . . , n.

Remark 7.2.5. The necessary conditions are also necessary conditions fora local minimum. For if x∗ is a local minimum, then there exists a δ > 0such that f(x∗) ≤ f(x) for all x that are in B(x∗, δ) ∩ X0 and that satisfythe constraints g(x) ≤ 0 and h(x) = 0. Thus, x∗ is a global solution to theproblem in which X0 is replaced by X0 ∩B(x∗, δ).

Proof of Theorem 7.2.1. Let E denote the set of indices such that gi(x∗) = 0and let I denote the set of indices such the gi(x∗) < 0. By gI we meanthe vector consisting of those components of g whose indices are in I. Thevector gE has similar meaning. To simplify the notation we assume that E =1, 2, . . . , r and that I = r+1, . . . ,m. Since E or I can be empty, we have0 ≤ r ≤ m.


By a translation of coordinates, we may assume that x∗ = 0 and thatf(x∗) = 0.

Let ω be a function from (−∞,∞) to R1 such that: (i) ω is strictly in-

creasing on (0,∞), (ii) ω(u) = 0 for u ≤ 0, (iii) ω is C1, and (iv) ω′(u) > 0for u > 0. We want ω > 0 for u > 0 as well as convex.

Since g is continuous and X0 is open, there exists an ε0 > 0 such thatB(0, ε0) ⊂ X0 and gI(x) < 0 for x ∈ B(0, ε0).

Define a penalty function F as follows:

F (x, p) = f(x) + ‖x‖2 + p

r∑

i=1

ω(gi(x)) +

k∑

i=1

[hi(x)]2

, (7.2.1)

where x ∈ X0 and p is a positive integer. We assert that for each ε satisfying0 < ε < ε0, there exists a positive integer p(ε) such that for x with ‖x‖ = ε,

F (x, p(ε)) > 0. (7.2.2)

We prove the assertion by assuming it to be false and arriving at a con-tradiction. If the assertion were false, then there would exist an ε′, with0 < ε′ < ε0, such that for each positive integer p there exists a point xp

with ‖xp‖ = ε′ and F (xp, p) ≤ 0. Hence, from (7.2.1)

f(xp) + ‖xp‖2 ≤ −p

r∑

i=1

ω(gi(xp)) +

k∑

i=1

[hi(xp)]2

. (7.2.3)

Since ‖xp‖ = ε′ and since S(0, ε′) = y : ‖y‖ = ε′ is compact, there existsubsequences, which we relabel as xp and a point x0 with ‖x0‖ = ε′ such thatxp → x0. Since f,g, and h are continuous,

f(xp) → f(x0) gE(xp) → gE(x0) h(xp) → h(x0).

Therefore, if in (7.2.3) we divide through by −p and then let p→ ∞, we get

0 ≥r∑

i=1

ω(gi(x0)) +k∑

i=1

[hi(x0)]2 ≥ 0.

Hence for each i = 1, . . . , r we have gi(x0) ≤ 0, and for each i = 1, . . . , k wehave hi(x0) = 0. Since ‖x0‖ = ε′ < ε0, we have gI(x0) < 0. Thus, x0 is afeasible point and so

f(x0) ≥ f(0) = 0. (7.2.4)

On the other hand, from (7.2.3) and from ‖xp‖ = ε′ we get that f(xp) ≤−(ε′)2, and so

f(x0) ≤ −(ε′)2 < 0,

which contradicts (7.2.4). This proves the assertion.For each ε in (0, ε0) the function F ( , p(ε)) is continuous on the closed ball


B(0, ε). Since B(0, ε) is compact, F ( , p(ε)) attains its minimum on B(0, ε)at some point xε with ‖xε‖ ≤ ε.

Since F (x, p(ε)) > 0 for x with ‖x‖ = ε, and since F (0, p(ε)) = f(0) = 0,it follows that F ( , p(ε)) attains its minimum on B(0, ε) at an interior pointxε of B(0, ε). Hence,

∂F

∂xj(xε, p(ε)) = 0 j = 1, . . . , n.

Calculating ∂F/∂xj from (7.2.1) gives:

∂f

∂xj(xε) + 2(xε)j +

r∑

i=1

p(ε)ω′(gi(xε))∂gi∂xj

(xε) (7.2.5)

+

k∑

i=1

2p(ε)hi(xε)∂hi∂xj

(xε) = 0

for j = 1, . . . , n.Define:

L(ε) = 1 +

r∑

i=1

[p(ε)ω′ (gi(xε))]2 +

k∑

i=1

[2p(ε)hi(xε)]2

λ0(ε) = 1/√L(ε)

λi(ε) = p(ε)ω′(gi(xε))/√L(ε) i = 1, . . . , r

λi(ε) = 0 i = r + 1, . . . ,m

µi(ε) = 2p(ε)hi(xε)/√L(ε) i = 1, . . . , k.

Note that

(i) λ0(ε) > 0 (ii) λi(ε) ≥ 0 i = 1, . . . , r (7.2.6)

(iii) λi(ε) = 0; i = r + 1, . . . ,m (iv) ‖(λ0(ε), λ(ε), µ(ε))‖ = 1,

where λ(ε) = (λ1(ε), . . . , λm(ε)) and µ(ε) = (µ1(ε), . . . , µk(ε)).If we divide through by

√L(ε) in (7.2.5), we get

λ0(ε)∂f

∂xj(xε)+

2(xε)j√L(ε)

+

r∑

i=1

λi(ε)∂gi(xε)

∂xj+

k∑

i=1

µi(ε)∂hi(xε)

∂xj= 0. (7.2.7)

Now let ε→ 0 through a sequence of values εk. Then since ‖xε‖ < ε, we havethat

xεk → 0. (7.2.8)

Since the vectors (λ0(ε), λ(ε), µ(ε)) are all unit vectors (see Eq. (7.2.6)),there exists a subsequence, that we again denote as εk, and a unit vector(λ0,λ,µ) such that

(λ0(εk), λ(εk), µ(εk)) → (λ0,λ,µ). (7.2.9)


Since (λ0,λ,µ) is a unit vector, it is different from zero.From Eqs. (7.2.7), (7.2.8), and (7.2.9) and the continuity of the partials of

f,g, and h, we get

λ0∂f

∂xj(0) +

r∑

i=1

λi∂gi(0)

∂xj+

k∑

i=1

µi∂hi(0)

∂xj= 0. (7.2.10)

From (7.2.6) and (7.2.9) we see that λi ≥ 0, i = 0, 1, . . . , r and that λi = 0 fori = r+1, . . . ,m. Thus, λ0 ≥ 0 and λ ≥ 0. Since gi(0) = 0 for i = 1, . . . , r andλi = 0 for i = r + 1, . . . ,m, we have that λigi(0) = 0 for i = 1, . . . ,m. Also,we can take the upper limit in the second term in (7.2.10) to be m and write

λ0∂f

∂xj(0) +

m∑

i=1

λi∂gi(0)

∂xj+

k∑

i=1

µi∂hi(0)

∂xj= 0.

This completes the proof of the theorem.

7.3 The Norm of a Relaxed Control; CompactConstraints

In Chapter 3, for problems with compact constraint sets Ω(t) containedin a fixed compact set Z, we defined the notion of a relaxed control µ on acompact interval I. In Eqs. (3.3.1) and (3.3.2) we pointed out that a relaxedcontrol µ determines a continuous linear transformation Lµ from C(I ×Z) toR

n by the formula

Lµ(g) =

∫

I

(∫

Z

g(t, z)dµt

)dt g ∈ C(I × Z). (7.3.1)

Moreover, ‖Lµ‖ = |I| where |I| denotes the length of I, and from the Rieszrepresentation theorem we have that

‖Lµ‖ = ‖ν‖var = |ν|(I × Z),

where ν is the measure dµtdt on I × Z and |ν| denotes the total variationmeasure of ν. We could therefore define the norm of µ, denoted by ‖µ‖ to bethe total variation measure of ν. For our purposes it is more useful to define‖µ‖ to be ‖Lµ‖.

Definition 7.3.1. The norm of a relaxed control µ, denoted by ‖µ‖ is thenorm of Lµ, the continuous linear transformation that µ determines by (7.3.1).Thus, ‖µ‖ = ‖Lµ‖ = |I|, where |I| denotes the length of I.


In the proof of Theorem 6.3.5 we shall consider certain linear combinationsof relaxed controls. Let I be a compact real interval, let Ω be a mapping fromI to compact subsets Ω(t) of a fixed compact set Z in R

m. Let µ1 and µ2

be relaxed controls such that for each t in I, µ1t and µ2t are concentrated onΩ(t). Then for any real numbers α and β by µ = αµ1 + βµ2 we mean themapping:

t→ µt ≡ αµ1t + βµ2t.

The mapping µ defines a continuous linear transformation Lµ from C(I ×Z)to R

n by the formula

Lµ(g) = α

∫

I

∫

Ω(t)

g(t, z)dµ1tdt+ β

∫

I

∫

Ω(t)

g(t, z)dµ2tdt (7.3.2)

for all g in C(I × Z).If αµ1 + βµ2 is a convex combination of µ1 and µ2 (i.e., α ≥ 0, β ≥ 0,

α + β = 1), then µ is again a relaxed control and ‖µ‖ is defined. Otherwise,we must define what is meant by the norm of αµ1 + βµ2.

Definition 7.3.2. If µ = αµ1 + βµ2, where µ1 and µ2 are relaxed controlson I such that for each t in I, µ1t and µ2t are concentrated on Ω(t), then

‖µ‖ ≡ ‖αµ1 + βµ2‖ ≡ ‖Lµ‖,

where Lµ is given by (7.3.2).

Note that this definition is consistent with Definition 7.3.1.We conclude with two known results from functional analysis involving

norms and their interpretation in our context.

Lemma 7.3.3. Let Ln be a sequence of continuous linear transformationsand L a continuous linear transformation from C(I × Z) to R

n such that‖Ln − L‖ → 0 as n→ ∞. Then Ln converges weak-* to L.

Proof. For arbitrary g in C(I × Z)

|Ln(g)− L(g)| ≤ ‖Ln − L‖‖g‖.

Since ‖Ln − L‖ → 0, the result follows.

Corollary 7.3.4. If µn is a sequence of relaxed controls and µ is a relaxedcontrol such that ‖µn − µ‖ → 0, then µn converges weakly to µ.

Proof. By Lemma 7.3.3 the sequence Ln of continuous linear transforma-tions corresponding to µn converges weak-* to the continuous linear trans-formation L corresponding to µ. The weak convergence of µn to µ follows fromRemark 3.3.13.

The converse of Lemma 7.3.3 is false. We have instead that the norm islower semi-continuous with respect to weak-* convergence.


Lemma 7.3.5. Let B be a Banach space and B∗ the space of continuouslinear transformations from B to R

n. Let Ln be a sequence of elements inB∗ converging weak-* to an element L in B∗. Then

lim infn→∞

‖Ln‖ ≥ ‖L‖, (7.3.3)

where ‖ ‖ denotes the norm in B∗.

Proof. From the identity L = (L− Ln) + Ln we get that for each g in B

|L(g)| ≤ |L(g)− Ln(g)|+ ‖Ln‖‖g‖.

Hence

|L(g)| ≤ limn→∞

|L(g)− Ln(g)|+ (lim infn→∞

‖Ln‖)‖g‖

= (lim infn→∞

‖Ln‖)‖g‖,

the first limit being equal to zero since Ln converges weak-* to L. From this(7.3.3) follows.

Corollary 7.3.6. Let µn be a sequence of relaxed controls converging weaklyto a relaxed control µ. Then lim infn→∞ ‖µn‖ ≥ ‖µ‖.

7.4 Necessary Conditions for an Unconstrained Problem

In this section we derive two necessary conditions satisfied by a local min-imizer for the unconstrained problem of minimizing

J(ψ) = γ(e(ψ)) +

∫ t1

t0

G(t, ψ(t), ψ′(t))dt (7.4.1)

over an appropriate set T of functions ψ, where G has a special form.Classical derivations of necessary conditions assume that a minimum exists

in the class of piecewise smooth functions. Tonelli [86], in consonance withresults of existence theorems, derived the necessary conditions for a minimumin the class of absolutely continuous functions. An account of these results inEnglish is given in Cesari [27]. For our purposes we need to consider minima inthe class of absolutely continuous functions with derivatives in L2 for a specialform of G. The following assumption will be adequate for our purposes andwill allow us to derive the necessary conditions using essentially the samearguments as those used in the classical case.


Assumption 7.4.1. (i) The function G is given by

G(t, x, x′) = A〈x′, x′〉+B〈x′, a(t, x)〉 + b(t, x), (7.4.2)

where t ∈ I, a compact real interval containing [t0, t1], x ∈ Rn and

x′ ∈ Rn.

(ii) The functions a and b are measurable on I for fixed x and are C(1) onR

n for fixed t ∈ I.

(iii) For each compact interval X in Rn there exists a function M in L2[I]

such that

|a(t, x)| ≤M(t) |b(t, x)| ≤M(t) (7.4.3)

|ax(t, x)| ≤M(t) |bx(t, x)| ≤M(t)

for a.e. t in I and all x in X .

(iv) The function γ is C(1) in a neighborhood of a manifold B, where Bis the closure of a C(1) manifold B of dimension 0 ≤ r < 2n + 2 inI × R

n × I × Rn. As usual, we denote points in B by (t0, x0, t1, x1).

(v) The set of functions T on which we are minimizing (7.4.1) are thosefunctions ψ whose graphs are in I ×X , where X is a compact set in R

n

and that have the following properties. The end points e(ψ) are in B,the functions ψ are absolutely continuous on [t0, t1] and have derivativesψ′ in L2[t0, t1].

Remark 7.4.2. As a consequence of (iii) and (v) of Assumption 7.4.1 theintegral in (7.4.1) with G as in (7.4.2) exists for all ψ in T .

Among the several established necessary conditions that a minimizingfunction ψ0 must satisfy are the Euler equations and the transversality condi-tions which the end points of ψ0 must satisfy. We now derive these necessaryconditions for the problem of minimizing (7.4.1) over a subset of T with G asin (7.4.2).

The next two lemmas are needed in the derivation of the Euler equations.

Lemma 7.4.3. Let G satisfy Assumption 7.4.1. Then for fixed functions ψand η, where ψ and η are absolutely continuous and have derivatives ψ′ andη′ in L2[t0, t1] the function

I(θ) =

∫ t1

t0

G(t, ψ(t) + θη(t), ψ′(t) + θη′(t))dt

is defined on an interval (−δ, δ) and is differentiable with respect to θ. More-over, if we define

G(t, θ) ≡ G(t, ψ(t) + θη(t), ψ′(t) + θη′(t))


and define Gx(t, θ) and Gx′(t, θ) similarly, then I is differentiable with deriva-tive I ′(θ) given by

I ′(θ) =

∫ t1

t0

[〈Gx(t, θ), η(t)〉 + 〈Gx′(t, θ), η′(t)〉]dt. (7.4.4)

Proof. That I(θ) exists follows from (i) through (iii) of Assumption 7.4.1. Toshow that I ′(θ) exists and is given by (7.4.4), we first write

σ−1[I(θ + σ)− I(θ)] =

∫ t1

t0

σ−1[G(t, θ + σ)−G(t, θ)]dt σ 6= 0.

From the Mean Value Theorem we get that there exists a σ, where 0 < σ < σ,such that

σ−1[I(θ + σ)− I(θ)] =

∫ t1

t0

[〈Gx(t, θ + σ), η(t)〉 + 〈Gx′(t, θ + σ), η′(t)〉]dt.

From (iii) of Assumption 7.4.1 we get that for all 0 < σ < σ the integrandis bounded by a fixed integrable function. If we now let σ → 0, then theexistence of I ′(θ) and Eq. (7.4.4) follow from (ii) of Assumption 7.4.1 and theDominated Convergence Theorem.

Lemma 7.4.4. Let h be a function in L2[t0, t1] with range in Rn such that for

every absolutely continuous function η from [t0, t1] to Rn with η′ in L2[t0, t1]

and η(t0) = η(t1) = 0 we have

∫ t1

t0

〈h(t), η′(t)〉dt = 0. (7.4.5)

Then h(t) = constant a.e. on [t0, t1].

Proof. Let

c = (t1 − t0)−1

∫ t1

t0

h(t)dt

and let

η(t) =

∫ t

t0

[h(s)− c]ds. (7.4.6)

Then η satisfies the hypotheses of the lemma. From (7.4.5), (7.4.6), and η(t0) =η(t1) = 0 we get that

0 =

∫ t1

t0

〈h(t), h(t) − c〉dt =∫ t1

t0

‖h(t)− c‖2dt,

where ‖ ‖ denotes the L2 norm. Hence h = c, a constant a.e.


Lemma 7.4.5 (Euler Equation). Let Assumption 7.4.1 hold and let ψ1 be afunction in the set T defined in (v) of Assumption 7.4.1. For fixed ε > 0 let

Tε(ψ1) = ψ ∈ T : ‖ψ′ − ψ′1‖2 < ε, |ψ(t0)− ψ1(t0)|2 < ε, (7.4.7)

where ‖ ‖ denotes the L2 norm. Let ψ minimize (7.4.1) on the set Tε(ψ1). Let

Gx(t) = Gx(t, ψ(t), ψ′(t)) and let Gx′(t) have similar meaning. Then there

exists a constant c such that for a.e. t in [t0, t1]

Gx′(t) =

∫ t

t0

Gx(s)ds+ c. (7.4.8)

Proof. From the Cauchy-Schwarz inequality and (7.4.7) we have that for allψ in Tε(ψ1)

|ψ(t)− ψ1(t)| ≤ |ψ(t0)− ψ(t1)|+∫ t

t0

|ψ′(s)− ψ′1(s)|ds

≤ ε1/2(1 + |t1 − t0|1/2).

Hence for ψ ∈ Tε(ψ1) all points ψ(t) with t0 ≤ t ≤ t1 are in a compact set.Consequently, for ψ ∈ Tε(ψ1) the integral

I(ψ) =

∫ t1

t0

G(t, ψ(t), ψ′(t))dt,

which is the integral in (7.4.1), exists. Since ψ minimizes (7.4.1) over all ψ inTε(ψ1), it follows that ψ minimizes (7.4.1) over all ψ in Tε(ψ1) with e(ψ) =e(ψ).

Let η be a function defined on [t0, t1] with properties as in Lemma 7.4.3and with η(t1) = η(t2) = 0. Then for real θ with |θ| sufficiently small, thefunction ψ = ψ + θη is in Tε(ψ1). Hence I(ψ + θη) has a minimum at θ = 0.By Lemma 7.4.3, the function θ → I(ψ + θη) is differentiable with derivativegiven by (7.4.4). Therefore, since I(ψ + θη) has a minimum at θ = 0

dI

dθ(ψ + θη)|θ=0 = 0.


∫ t1

t0

[〈Gx, η〉+ 〈Gx′ , η′〉]dt = 0.

Integrating by parts in the first term and using η(t0) = η(t1) = 0 gives

∫ t1

t0

〈−∫ t

t0

Gx(s)ds+Gx′(t), η′(t)〉dt = 0.

Equation (7.4.8) now follows from Lemma 7.4.4.


Remark 7.4.6. Equation (7.4.8) is the Euler equation in integrated form.From (7.4.8) it follows that Gx′ is absolutely continuous, is differentiablea.e., and that

d

dtGx′ |t=τ = Gx(τ) a.e.

This equation is usually called the Euler equation.

Remark 7.4.7. In Section 6.5 we derived the Euler equation for the uncon-strained problem from the maximum principle. The reasoning involved is notcircular, since we will not appeal to that result in our proof of the maximumprinciple. We have established the Euler equation for a restricted class of un-constrained problems and will use this result in our proof of the maximumprinciple.

We now take up the transversality condition under the assumption thatthe initial and terminal times are fixed. For typographic simplicity we takethe initial time t0 to be zero and the terminal time to be one. The set B thusconsists of points (0, x0, 1, x1).

Lemma 7.4.8. Let Assumption 7.4.1 hold. Let t0 = 0 and let t1 = 1. Let ψminimize (7.4.1) over the set Tε(ψ1) defined in (7.4.7) and let

Gx′(t) = Gx′(t, ψ(t), ψ′(t)).

Then for all unit tangent vectors (dx0, dx1) to B at e(ψ)

〈γx0(e(ψ))−Gx′(0), dx0〉+ 〈γx1

(e(ψ)) +Gx′(1), dx1〉 = 0, (7.4.9)

where Gx′ is identified with the absolutely continuous function on the right-hand side of (7.4.8) that equals Gx′ a.e.

Proof. The transversality condition is a consequence of the fact that if theend points of ψ are varied along a curve lying in B, then the functions thusobtained cannot give a smaller value to (7.4.1) than ψ does.

Because we are assuming that t0 = 0 and t1 = 1, the end point e(ψ) is(0, x0, 1, x1), where x0 = ψ(0) and x1 = ψ(1). Let ξ = (ξ0, ξ1) be a C(1)

function from an interval |s| ≤ δ to R2n that represents a curve lying in B

withξ(0) = e(ψ) = (0, x0, 1, x1), (7.4.10)

and with |ξ(s)− ψ1(0)|2 < ε. Let

P (t, s) = (ξ0(s)− x0)(1 − t) + (ξ1(s)− x1)t (7.4.11)

and letψ(t, s) = ψ(t) + P (t, s) (7.4.12)

for 0 ≤ t ≤ 1 and |s| ≤ δ. Then

ψ(0, s) = ξ0(s) ψ(1, s) = ξ1(s) ψ(t, 0) = ψ(t).


Thus, the end points of ψ lie in B.From (7.4.11) and (7.4.12) we get that

|ψ(0, s)− ψ1(0)| ≤ |ψ(0)− ψ1(0) + |P (0, s)|= |ψ(0)− ψ1(0)|+ |ξ0(s)− x0|.

Since ψ is in Tε(ψ1), the first term in the rightmost expression is <√ε. From

the continuity of ξ0 we get that the second term can be made arbitrarily smallby taking |s| sufficiently small. Hence for |s| sufficiently small

|ψ(0, s)− ψ1(0)|2 < ε. (7.4.13)

From (7.4.12) we get

‖ψ′(·, s)− ψ′1‖ ≤ ‖ψ′ − ψ′

1‖+ ‖Pt(·, s)‖≤ ‖ψ′ − ψ′

1‖+ ‖ξ0(s)− x0‖+ ‖ξ1(s)− x1‖.

By an argument similar to the one used to establish (7.4.13), we get that for|s| sufficiently small, all functions ψ(·, s) are in Tε(ψ1).

Since ψ minimizes (7.4.1) over Tε(ψ1), since for |s| sufficiently small allfunctions ψ(·, s) are in Tε(ψ1), and since ψ is in Tε(ψ1), the function ψ = ψ(·, 0)minimizes J over all ψ(·, s) for |s| sufficiently small.

Let

I(s) =

∫ 1

0

G(t, ψ(t, s), ψ′(t, s))dt,

where ′ indicates differentiation with respect to t. Then

J(ψ(·, s)) = γ(ξ(s)) + I(s) (7.4.14)

andJ(ψ(·, s)) ≥ J(ψ(·, 0)) = J(ψ). (7.4.15)

Since ξ0 and ξ1 areC(1) on |s| ≤ δ, it follows from Eqs. (7.4.10) through (7.4.12)

that the function t → ∂G(t, ψ(t, s), ψ′(t, s))/∂s is bounded on [0, 1] by an in-tegrable function. Hence, by the argument used in Lemma 7.4.3 we get thatI ′(s) exists. Also, since γ is of class C(1), the function s → γ(ξ(s)) is C(1).Hence s→ J(ψ(·, s)) is differentiable. It then follows from (7.4.15) that

dJ(ψ(·, s))ds

|s=0 = 0. (7.4.16)

A straightforward calculation of (7.4.16) gives

[dγ/ds]s=0 +

∫ 1

0

[〈Gx(t), Ps(t, 0)〉+ 〈Gx′(t), Pts(t, 0)〉]dt = 0, (7.4.17)


where Gx(t) = Gx(t, ψ(t), ψ′(t)) and Gx′(t) has similar meaning. The function

ψ satisfies the Euler equation, so there exists a constant c such that

Gx′(t) =

∫ t

0

Gx(σ)dσ + c a.e.

If we set

h(t) =

∫ t

0

Gx(σ)dσ + c,

then h is absolutely continuous and h′(t) = Gx(t) a.e. We may thereforerewrite (7.4.17) as

dγ/ds]s=0 +

∫ 1

0

[〈h′(t), Ps(t, 0)〉+ 〈h(t), Pts(t, 0)〉]dt = 0. (7.4.18)

From (7.4.11) we see that Pst(t, s) = Pts(t, s). Hence if we integrate the firstterm in (7.4.17) by parts we get that

dγ/ds]s=0 + h(t)Ps(t, 0]10 = 0.

If we take Gx′(t) = h(t) everywhere, and use (7.4.11), we can write the pre-ceding equality as

〈γx1(e(ψ)) +Gx′(1), ξ′1(0)〉+ 〈γx0

(e(ψ))−Gx′(0), ξ′0(0)〉 = 0.

Since the curve ξ(s) is arbitrary, (ξ′1(0), ξ′0(0)) is a scalar multiple of an arbi-

trary unit tangent vector (dx0, dx1), and the lemma follows.

Remark 7.4.9. If e(ψε) is a point of B\B, then curves (ξ0(s), ξ(s)) emanatingfrom e(ψε) and lying in B are only defined for s ≥ 0. Also the tangent vectorsdx0 = cξ′0(s) and dx1 = cξ′1(s) are such that their projections from the tangentplane onto B are directed into B. The proof is then modified to get I ′(0) ≥ 0.From this the conclusion follows that for all tangent vectors (dx0, dx1) pointinginto B, we have

〈γx0(e(ψ))−Gx′(0), dx0〉+ 〈γx1

(e(ψ)) +Gx′(1), dx1〉 ≥ 0.

7.5 The ε-Problem

The proof of Theorem 6.3.5 will be carried out in several steps in this andthe next two sections. First, we formulate a set of unconstrained penalizedproblems depending on a small parameter ε and show that these problemshave a solution. This constitutes the “ε-problem.” Second, we show that foreach ε the ε-problem has an interior minimum. Third, we apply the results


of Section 7.4 and other arguments to obtain necessary conditions satisfiedby the solution of the ε-problem. We call these necessary conditions the “ε-maximum principle.” This will be done in Section 7.6. Lastly, we let ε tendto zero and show that the solutions of the ε problem tend to the solutionof the constrained problem and that the ε-maximum principle tends to themaximum principle of Theorem 6.3.5.

Remark 7.5.1. In the sequel we shall be extracting subsequences of se-quences. If ψn is a sequence and ψnk

is a subsequence we shall relabelthe subsequence as ψn without explicitly saying so.

Theorem 6.3.5 involves a relaxed optimal pair. Therefore, for typographicsimplicity we designated an optimal relaxed pair by (ψ, µ), without any addi-tional symbols to denote optimality. The proof of Theorem 6.3.5 will involvepairs other than the optimal one. Therefore, we now denote a relaxed optimalpair by (ψ∗, µ∗). To simplify notation we take t0 = 0 and t1 = 1. We supposethat [0, 1], the interval on which (ψ∗, µ∗) is defined, is interior to I0.Step I Formulation of the ε-problem. Without loss of generality we may as-sume that J(ψ∗, µ∗) = 0. The graph of ψ∗, that is, (t, ψ∗(t)) : t ∈ [0, 1] iscompact and hence it is a positive distance, say ε′1, from the boundary ofI0 ×X0. Let ε1 = min(1, ε′1). It follows from (6.3.3) that ψ′

∗ is in L2[0, 1].Let D denote the set of absolutely continuous functions ψ and relaxed con-

trols µ defined on [0, 1] but not necessarily related by the differential equationψ′(t) = f(t, ψ(t), µt), such that the graph of ψ is in [0, 1]×X0, the derivative ψ

′

is in L2[0, 1], e(ψ) is in B, and µt is concentrated on Ω(t) for all t ∈ [0, 1]. Sinceall points x = ψ(t) : 0 ≤ t ≤ 1 are in a compact set, it follows from (6.3.3)that the function t → f(t, ψ(t), µt) is in L2[0, 1], as is t → f0(t, ψ(t), µt). Foreach 0 < ε < ε1 and (ψ, µ) in D let

F (ψ, µ,K, ε) = g(e(ψ)) +

∫ 1

0

f0(t, ψ(t), µt)dt+ |ψ(0)− ψ∗(0)|2 (7.5.1)

+ ‖ψ′ − ψ′∗‖2 + ε‖µ− µ∗‖L +K‖ψ′ − f(t, ψ(t), µt)‖2,

where ‖ ‖ denotes the L2[0, 1] norm and ‖ ‖L denotes the norm defined inSection 7.3. The function F is well defined on D × (0, ε1).

For each 0 < ε < ε1, let Dε denote the set of absolutely continuous func-tions ψ and relaxed controls µ defined on [0, 1] such that ψ′ is in L2[0, 1], µt

is concentrated on Ω(t) for all t ∈ [0, 1] and such that

|ψ(0)− ψ∗(0)| ≤ ε/2 ‖ψ′ − ψ′∗‖ ≤ ε/2 e(ψ) ∈ B (7.5.2)

‖µ− µ∗‖L ≤ ε.

From

ψ(t)− ψ∗(t) = ψ(0)− ψ∗(0) +

∫ t

0

[ψ′(s)− ψ′∗(s)]ds


and the Cauchy-Schwarz inequality we get that for ψ in Dε

|ψ(t)− ψ∗(t)| ≤ |ψ(0)− ψ∗(0)|+ ‖ψ′ − ψ′∗‖. (7.5.3)

Since the graph of ψ∗ is compact and is at distance ε′1 from the boundary ofI0×X0, it follows from (7.5.3) and the first two inequalities in (7.5.2) that allpoints ψ(t), 0 ≤ t ≤ 1 are in a fixed compact set X that is independent of ψand is contained in X0. Thus, the graph of ψ is contained in I0 × X0. HenceDε is contained in D and F is defined on Dε × (0, ε).

For each 0 < ε < ε1, we define the ε-problem to be: Minimize F (ψ, µ,K, ε)over the set Dε.

Step II consists of establishing the following lemma.

Lemma 7.5.2. For each 0 < ε < ε1, the ε-problem has a solution.

Proof. Let mε(K) = inf F (ψ, µ,K, ε), where the infimum is taken over all(ψ, µ) in Dε. In Step I we saw that if (ψ, µ) is in Dε, then all points ψ(t),0 ≤ t ≤ 1 are in a fixed compact set X ⊆ X0 and independent of ψ. Inparticular, the points e(ψ) are in a compact subset of B. Since g is continuous,g is bounded on Dε. Also, since Z is compact and dµt is a probability measure,it follows from (6.3.3) that the integrals

∫ 1

0

f0(t, ψ(t), µt)dt =

∫ 1

0

(

∫

Z

f0(t, ψ(t), z)dµt)dt

are bounded in Dε. Hence mε(K) is finite.Let (ψn, µn) be a sequence in Dε such that

limn→∞

F (ψn, µn, K, ε) = mε(K). (7.5.4)

From the identityψ′(t) = ψ′(t)− ψ′

∗(t) + ψ′∗(t)

and the Cauchy-Schwarz inequality, we get that for any measurable set E ⊆[0, 1]

|∫

E

ψ′(t)dt| ≤ ‖ψ′ − ψ′∗‖[meas E]1/2 + |

∫

E

ψ′∗(t)dt|.

Thus, the functions ψn are equi-absolutely continuous. In Step I we showedthat all ψ in Dε are uniformly bounded. It then follows from Lemma 5.3.3and Ascoli’s theorem that there exists a subsequence ψn and an absolutelycontinuous function ψε such that ψn converges uniformly to ψε. Since ψnis in Dε, |ψε(0) − ψ∗(0)| ≤ ε/2. Since e(ψn) ∈ B, it follows that e(ψε) ∈ B.Moreover, by Theorem 5.3.5, ψ′

n converges weakly in L1[0, 1] to ψ′ε.

Let µn be a subsequence corresponding to ψn. By the discussion inSection 7.3, the inequality ‖µn − µ∗‖L ≤ ε means that ‖Ln − L∗‖ ≤ ε whereLn and L∗ are the linear transformations from C(I×Z) to Rn determined byµn and µ∗. Since a closed ball in the dual space of a separable Banach space


is weak-* sequentially compact, there exists a subsequence Ln and a lineartransformation Lε such that Ln converges weak-* to Lε, where ‖Lε−L∗‖ ≤ ε.It then follows from Corollary 7.3.4 that the sequence µn of relaxed controlsconverges weakly to a relaxed control µε with ‖µε −µ∗‖L ≤ ε. We then selecta subsequence of ψn corresponding to µn so that (ψn, µn) satisfies allthe assertions of this paragraph.

Thus far we have shown that (ψε, µε) satisfies all the inequalities in (7.5.2)except ‖ψ′

ε − ψ′∗‖ ≤ ε/2. From (7.5.2) we have that ‖ψ′

n − ψ′∗‖ ≤ ε/2. Since

closed balls in L2 are weakly compact, there exists a subsequence (ψn, µn)and a function h in L2[0, 1] with ‖h− ψ∗‖ ≤ ε/2 such that ψ′

n → h weakly inL2. Therefore, for any bounded measurable function γ,

limn→∞

∫ 1

0

ψ′nγdt =

∫ 1

0

hγdt.

Thus, ψ′n → h weakly in L1. But, ψ

′n → ψ′

ε weakly in L1[0, 1]. Hence, by theuniqueness of weak limits, ψ′

ε = h, and so ‖ψ′ε − ψ′

∗‖ ≤ ε/2. In summary, wehave shown that (ψε, µε) is in Dε

It remains to show that (ψε, µε) is a minimizer. By the Cauchy-Schwarzinequality

∫ 1

0

[ψ′n(t)− f(t, ψn(t), µnt)][ψ

′ε(t)− f(t, ψε(t), µεt)]dt

≤ ‖ψ′n − f(t, ψn(t), µnt)‖‖ψ′

ε − f(t, ψε(t), µεt)‖.

From the uniform convergence of ψn to ψ, from the weak L2 convergence ofψ′n to ψ′

ε, from the weak convergence of µn to µε, and from Lemma 4.3.3, weget that as n → ∞, the integral on the left tends to ‖ψ′

ε − f(t, ψε(t), µεt)‖2.Hence

lim infn→∞

‖ψ′n − f(t, ψn(t), µnt)‖ ≥ ‖ψ′

ε − f(t, ψε(t), µεt)‖, (7.5.5)

and

limn→∞

∫ 1

0

f0(t, ψn(t), µnt)dt =

∫ 1

0

f0(t, ψε(t), µεt)dt. (7.5.6)

It follows from Lemma 7.3.5 and Corollary 7.3.6 that

lim infn→∞

‖ψ′n − ψ′

∗‖ ≥ ‖ψ′ε − ψ′

∗‖ lim infn→∞

‖µn − µ∗‖ ≥ ‖µε − µ∗‖. (7.5.7)

From (7.5.1), from e(ψn) → e(ψε), the continuity of g, from ψn(0) → ψε(0),and from (7.5.4) to (7.5.7) we get

mε(K) = limn→∞

F (ψn, µn, K, ε) = lim infn→∞

F (ψn, µn, K, ε)

≥ F (ψε, µε, K, ε) ≥ mε(K).

Hence (ψε, µε) minimizes, and the lemma is proved.


Step III (ψε, µε) is an interior minimum. The meaning of this statement isgiven in Lemma 7.5.4.

Step III is a consequence of the following result.

Lemma 7.5.3. For each 0 < ε < ε1, there exists a K(ε) > 0 such thatF (ψ, µ,K(ε), ε) > 0 for all (ψ, µ) in Dε that satisfy at least one of the equal-ities

|ψ(0)− ψ∗(0)| = ε/2 ‖ψ′ − ψ′∗‖ = ε/2 ‖µ− µ∗‖L = ε. (7.5.8)

Proof. If the conclusion of the lemma were false, then there would exist an0 < ε0 < ε1, a sequence Kn with Kn → +∞, and a sequence (ψn, µn) inDε0 whose elements satisfy at least one of the equalities in (7.5.8) with ε = ε0,and such that

g(e(ψn)) +

∫ 1

0

f0(t, ψn(t), µnt)dt+ |ψn(0)− ψ∗(0)|2 (7.5.9)

+ ‖ψ′n − ψ′

∗‖2 + ε‖µn − µ∗‖L ≤ −Kn‖ψ′n − f(t, ψn, µnt)‖2.

By the argument used in the proof of Lemma 7.5.2, there exists a subse-quence (ψn, µn), an absolutely continuous function ψ0 with derivative ψ′

0

in L2 and a relaxed control µ0 such that: (i) ψn converges uniformly to ψ0,(ii) e(ψ0) ∈ B, (iii) ψ′

n converges weakly in L2[0, 1] to ψ′0, (iv) µn converges

weakly to µ∗, (v) ‖ψ′0−ψ′

∗‖ ≤ ε0/2, and (vi) ‖µ0−µ∗‖L ≤ ε0. It follows fromthe preceding that (ψ0, µ0) is in Dε0 . From the preceding and Lemma 4.3.3we get that

limn→∞

∫ 1

0

f0(t, ψn(t), µnt)dt =

∫ 1

0

f0(t, ψ0(t), µ0t) dt (7.5.10)

limn→∞

∫

∆

[ψ′n(t)− f(t, ψn(t), µnt)]dt =

∫

∆

[ψ′0(t)− f(t, ψ0(t), µ0t)]dt,

where ∆ is any measurable subset of [0, 1].All terms on the left in (7.5.9) are bounded by a constant independent of

n. Hence if in (7.5.9) we divide both sides by −Kn and then let n → ∞ weget that

0 ≥ lim supn→∞

‖ψ′n − f(t, ψn(t), µnt)‖2 ≥ 0.

From (7.5.10) and the Cauchy-Schwarz inequality we get that∣∣∣∣∫

∆

[ψ′0(t)− f(t, ψ0(t), µ0t)]dt

∣∣∣∣ = limn→∞

∣∣∣∣∫

∆

[ψ′n(t)− f(t, ψn(t), µnt)]dt

∣∣∣∣≤ lim sup

n→∞‖ψ′

n − f(t, ψn(t), µnt)‖ = 0.

Since ∆ is any measurable subset of [0, 1] this implies that

ψ′0(t) = f(t, ψ0(t), µ0t) a.e.


Thus, (ψ0, µ0) is admissible for the non-penalized problem. Therefore,J(ψ0, µ0) ≥ J(ψ∗, µ∗) = 0, and so

g(e(ψ0)) +

∫ 1

0

f0(t, ψ0(t), µ0t)dt ≥ 0. (7.5.11)

From (7.5.9) and the assumption that at least one of the equalities in (7.5.8)holds, we get that

g(e(ψn)) +

∫ 1

0

f0(t, ψn(t), µnt)dt ≤ −ε2/4.

If we let n→ ∞ in this inequality and use (7.5.10), the convergence of ψn toψ, and the continuity of g, we get that

g(e(ψ0)) +

∫ 1

0

f0(t, ψ0(t), µ0t)dt ≤ −ε2/4,

which contradicts (7.5.11). This proves the lemma.

Lemma 7.5.4. For each 0 < ε < ε1, let K(ε) be as in Lemma 7.5.3 and let(ψε, µε) minimize F (ψ, µ,K(ε), ε) over Dε. Then

|ψε(0)− ψ∗(0)| < ε/2 ‖ψ′ε − ψ′

∗‖ < ε/2 ‖µε − µ∗‖L < ε. (7.5.12)

Proof. The existence of (ψε, µε) was shown in Lemma 7.5.2. Since (ψ∗, µ∗) isin Dε, and since

F (ψ∗, µ∗, K(ε), ε) = J(ψ∗, µ∗) = 0,

it follows that F (ψε, µε, K(ε), ε) ≤ 0.The inequalities in (7.5.12) now follow from Lemma 7.5.3.

Remark 7.5.5. In Lemma 7.5.3 if εn is a strictly decreasing sequence suchthat 0 < εn < ε1 and such that εn → 0, then we can take K(εn) to be astrictly increasing sequence such that K(εn) → +∞.

7.6 The ε-Maximum Principle

For each 0 < ε < ε1 the pair (ψε, µε) minimizes F (ψ, µ,K(ε), ε) over all(ψ, µ) in Dε. Hence, if we take µ = µε, then ψε minimizes F (ψ, µε, K(ε), ε)over the set

Tε/2(ψ∗) = ψ ∈ T : ‖ψ′ − ψ′∗‖ < ε/2, |ψ(0)− ψ∗(0)| < ε/2,

where the set T is defined in (v) of Assumption 7.4.1.


We can write

F (ψ, µε, K(ε), ε)− ‖µε − µ∗‖L = γ(e(ψ)) +

∫ 1

0

Gε(t, ψ(t), ψ′(t))dt, (7.6.1)

whereγ(e(ψ)) = g(e(ψ)) + 〈ψ(0)− ψ∗(0), ψ(0)− ψ∗(0)〉

and

Gε(t, x, x′) = f0(t, x, µεt) + 〈x′ − ψ′∗(t), x

′ − ψ′∗(t)〉

+K(ε)〈x′ − f(t, x, µεt), x′ − f(t, x, µεt)〉. (7.6.2)

The function Gε has the form of G in (7.4.2) and satisfies Assumption 7.4.1

by virtue of the assumptions on f and fx made in Assumption 6.3.1. Thus,ψε minimizes (7.6.1) with γ and Gε as in (7.6.2) over all ψ in Tε/2(ψ∗), andthe hypotheses of Lemma 7.4.5 hold with ψ∗ = ψ1. Let

Gε

x(t) = Gεx(t, ψε(t), ψ

′ε(t)) G

ε

x′(t) = Gεx′(t, ψε(t), ψ

′ε(t)). (7.6.3)

Then by Lemma 7.4.5 there exists a constant c = c(ε) such that for a.e. t in[0, 1]

Gε

x′(t) =

∫ t

0

Gε

x(s)ds+ c(ε). (7.6.4)

If we define λ(ε, t) to be equal to the right-hand side of (7.6.4), then λ(ε, ·)is absolutely continuous and

λ(ε, t) = Gε

x′(t) λ′(ε, t) = Gε

x(t) a.e. (7.6.5)

Calculating Gε

x′(t) and Gε

x(t) using (7.6.2) and (7.6.3) gives

λ(ε, t) = 2(ψ′ε(t)− ψ′

∗(t)) + 2K(ε) (ψ′ε(t)− f(t, ψε(t), µεt)) (7.6.6)

λ′(ε, t) = f0x(ε, t)− 2K(ε)fx(ε, t)

T (ψ′ε(t)− f(t, ψε(t), µεt)),

where

f0x(ε, t) = f0

x(t, ψε(t), µεt)

fx(ε, t) = fx(t, ψε(t), µεt) = (∂f i(t, ψε(t), µεt)/∂xj),

and the superscript T denotes transpose. We now combine the two equationsin (7.6.6) to get

λ′(ε, t) = f0x(ε, t)− fx(ε, t)

T λ(ε, t) + 2fx(ε, t)T (ψ′

ε(t)− ψ′∗(t)). (7.6.7)

LetM(ε) = 1 + |λ(ε, 0)| (7.6.8)


and letλ(ε, t) = λ(ε, t)/M(ε). (7.6.9)

We now divide through by M(ε) in (7.6.7) and use (7.6.9) to get

λ′(ε, t) =M(ε)−1f0x(ε, t)− fx(ε, t)

Tλ(ε, t) + 2M(ε)−1fx(ε, t)T (ψ′

ε(t)−ψ′x(t)).

(7.6.10)In summary, we have shown that since ψε minimizes (7.6.1) over Tε/2(ψ∗),there exists an absolutely continuous function λ(ε, ·) such that (7.6.10) holdsa.e. on [0, 1].

Since ψε minimizes (7.6.1) over Tε/2(ψ∗), the transversality condition(7.4.9) of Lemma 7.4.8 holds. We suppose the e(ψε) is an interior point ofB. From the first equation in (7.6.2) we get that

γx0(e(ψε)) = gx0

(e(ψε)) + 2(ψε(0)− ψ∗(0))

γx1(e(ψε)) = gx1

(e(ψε)).

From (7.6.5) we get that λ(ε, 0) = Gε

x′(0) and λ(ε, 1) = Gε

x′(1).Substituting these quantities into (7.4.9), dividing through by M(ε), and

using (7.6.9) gives

〈M(ε)−1(gx0(e(ψε)) + 2(ψε(0)− ψ∗(0)))− λ(ε, 0), dx0〉 (7.6.11)

+ 〈M(ε)−1gx1(e(ψε)) + λ(ε, 1), dx1〉 = 0

for all tangent vectors (dx0, dx1) to B at e(ψε). If e(ψε) is a boundary point ofB, then the equality in (7.6.11) is replaced by ≥ 0 and is required to hold forall tangent vectors (dx0, dx1) whose projection from the tangent plane ontoB is directed into B.

We now deduce another necessary condition, which is the analog of theWeierstrass condition in the calculus of variations, by considering changes inthe control µε. Let µ be an arbitrary relaxed control. For −1 ≤ θ ≤ 1 define

µθ = µε + θ(µ− µε) = (1− θ)µε + θµ.

Each µθ determines an element Lθ in C∗(I × Z) in the usual way. It is easilychecked that if 0 ≤ θ ≤ 1, then µθ is a relaxed control. Since ‖µε − µ∗‖L < ε,it follows that there exists a 0 < θ0 < 1 such that if 0 ≤ θ ≤ θ0, then‖µθ − µ∗‖L < ε. Thus,

ρε(θ) = F (ψε, µθ, K(ε), ε)

is defined for all 0 ≤ θ ≤ θ0 and has a minimum at θ = 0. Hence if ρε has aright-hand derivative ρ′ε(0+) at θ = 0, then we must have

ρ′ε(0+) ≥ 0.

We shall show that each of the three terms in the definition of ρε that


involve θ, namely the first integral, K‖ψ′ε − f(t, ψε(t), µθ)‖2 and ε‖µθ − µ∗‖L

has a right-hand derivative at θ = 0 and shall calculate their right-handderivatives at θ = 0.

We consider the first integral in the definition of ρε. Let

f0(t, θ) ≡ f0(t, ψε(t), µθt)

=

∫

Z

f0(t, ψε(t), z)dµεt + θ

∫

Z

f0(t, ψε(t), z)(dµt − dµεt).

Then f0 has a partial derivative with respect to θ given by

∂f0/∂θ =

∫

Z

f0(t, ψε(t), z)(dµt − dµεt).

Thus, ∂f0/∂θ is bounded by an integrable function, and the function

A(θ) =

∫ 1

0

f0(t, θ)dt,

which is the first term in the definition of ρε, is differentiable at θ = 0, withderivative given by

A′(0) =

∫ 1

0

[∂f0(t, θ)/∂θ]θ=0dt =

∫ 1

0

f0(t, ψε(t), µt − µεt)dt.

We next consider

B(θ) = K(ε)‖ψ′ε(t)− f(t, ψε(t), µθt)‖2.

Let

p(t, θ) = K(ε)〈(ψ′ε(t)− f(t, ψε(t), µθt)), (ψ

′ε(t)− f(t, ψε(t), µθt))〉.

Then

∂p

∂θ= −2K(ε)〈(ψ′

ε(t)− f(t, ψε(t), µθt)), f(t, ψε(t), µt − µεt)〉.

Thus, ∂p/∂θ is bounded by an integrable function, and so B(θ) is differentiableat θ = 0, with derivative given by

B′(0) = −∫ 1

0

〈2K(ε)(ψ′ε(t)− f(t, ψε(t), µεt)), f(t, ψε(t), µε − µεt)〉dt.

Using the first equation in (7.6.6), we rewrite this expression as

B′(0) =

∫ 1

0

〈−λ(ε, t), f(t, ψε(t), µt − µεt)〉dt


+ 2

∫ 1

0

〈ψ′ε(t)− ψ′

∗(t), f(t, ψε(t), µt − µεt)〉dt.

Lastly, we consider

γ(ε, θ) = ‖µθ − µ∗‖L = ‖µε − µ∗ + θ(µ− µε)‖L, −1 ≤ θ ≤ 1,

where µ is an arbitrary relaxed control with µε ∈ Ω(t). It follows from thetriangle inequality that γ(ε, ·) is a convex function on [−1, 1]. Hence γ has aright-hand derivative γ′(ε, 0+) at θ = 0.

We next obtain bounds for γ′(ε, 0+). From the three-chord property ofconvex functions we get that for 0 < θ < θ0,

γ(ε, 0)− γ(ε,−1) ≤ γ(ε, θ)− γ(ε, 0)

θ≤ γ(ε, 1)− γ(ε, 0).

Hence

‖µε−µ∗‖L−‖2µε−µ∗−µ‖L ≤ γ′(ε, 0+) ≤ ‖µ−µ∗‖L−‖µε−µ∗‖L ≤ ‖µ−µ∗‖L.Since ‖2µε − µ − µ∗‖ = ‖(µε − µ∗) + (µε − µ)‖, it follows from the triangleinequality that the left side of the inequality is greater than or equal to −‖µε−µ‖. From ‖µε − µ‖L = ‖(µε − µ∗) + (µ∗ − µ)‖L, the triangle inequality and‖µε − µ∗‖L < ε, we have that

−‖µε − µ‖L ≥ −ε− ‖µ∗ − µ‖L.Hence

−ε− ‖µ− µ∗‖L ≤ γ′(ε, 0+) ≤ ‖µ− µ∗‖L. (7.6.12)

The right-hand derivative ρ′ε(0+) is the sum of A′(0), B′(0), and εγ′(ε, 0+).Hence

A′(0) +B′(0) + εγ′ε(0+) ≥ 0. (7.6.13)

Since

f(t, ψ(t), µt − µεt) ≡∫

Ω(t)

f(t, ψ(t), z)dµt −∫

Ω(t)

f(t, ψ(t), z)dµεt

≡ f(t, ψ(t), µt)− f(t, ψ(t), µεt),

we can transpose those terms in (7.6.13) involving µt to the right-hand sideof the inequality and then divide through by M(ε) > 0 and use (7.6.9) to get

∫ 1

0

[−M(ε)−1f0(t, ψε(t), µεt) + 〈λ(ε, t), f(t, ψε(t), µεt)〉 (7.6.14)

− 2M(ε)−1〈ψ′ε(t)− ψ′

∗(t), f(t, ψε(t), µεt)〉]dt+ εγ′(ε, 0+)M(ε)−1

≥∫ 1

0

[−M(ε)−1f0(t, ψε(t), µt) + 〈λ(ε, t), f(t, ψε(t), µt)〉

− 2M(ε)−1〈ψ′ε(t)− ψ′

∗(t), f(t, ψε(t), µt)〉]dt.Equations (7.6.10), (7.6.11), and the inequality (7.6.14) constitute the ε-

maximum principle, which a solution (ψε, µε) of the ε-problem satisfies.


7.7 The Maximum Principle; Compact Constraints

In this section we complete the proof of Theorem 6.3.5 by letting ε→ 0.Let L = lim infε→0M(ε), where M(ε) is given by (7.6.8). Then 1 ≤ L ≤

+∞. If L is finite, there exists a sequence εn → 0 and a real number 0 < λ0 ≤ 1such that

limn→∞

1/M(εn) = λ0.

If L = +∞, then for every sequence εn → 0, limn→∞M(εn) = +∞. Thus, wemay always select a sequence εn → 0 such that

limn→∞

(1/M(εn)) = λ0 0 ≤ λ0 ≤ 1, (7.7.1)

where if possible we select a sequence εn such that λ0 > 0.It follows from (7.6.8) and (7.6.9) that |λ(ε, 0)| ≤ 1 for all 0 < ε < ε1.

Hence there exists a subsequence εn such that εn → 0,

λ(0) = limn→∞

λ(εn, 0) (7.7.2)

exists, and |λ(0)| ≤ 1.We rewrite the differential equation (7.6.10) for λ(ε, t) as

λ′(ε, t) = −fx(ε, t)Tλ(ε, t) +N(ε, t), (7.7.3)

where

N(ε, t) =M(ε)−1[f0x(ε, t) + 2fx(ε, t)

T (ψ′ε(t)− ψ′

∗(t))]. (7.7.4)

Let Λ(ε, t) denote the fundamental matrix of solutions with Λ(ε, 0) = I forthe homogeneous part of (7.7.3), namely the fundamental matrix of solutionsof

q′ = −fx(ε, t)T q. (7.7.5)

Then by the variation of parameters formula

λ(ε, t) = Λ(ε, t)[λ(ε, 0) +

∫ t

0

Λ(ε, s)−1N(ε, s)ds]. (7.7.6)

By Lemma 6.6.2, Eq. (7.7.6) can be written as

λ(ε, t) = Λ(ε, t)[λ(ε, 0) +

∫ t

0

P (ε, s)N(ε, s)ds], (7.7.7)

where P (ε, t) is the fundamental matrix with P (ε, 0) = I for the systemp′ = fx(ε, t)p, which is adjoint to (7.7.5).


Lemma 7.7.1. There exists a constant C such that for 0 ≤ t ≤ 1 and 0 <ε ≤ ε1,

|λ(ε, t)| ≤ C. (7.7.8)

Moreover, the integrals∫

E

λ′(ε, t)dt 0 < ε ≤ ε1 E ⊂ [0, 1]

are equi-absolutely continuous.

Proof. The lemma will follow from a sequence of bounds on the terms in theright-hand sides of (7.7.3) and (7.7.7).

By Lemma 7.5.2, (ψε, µε) is in Dε for 0 < ε ≤ ε1. Hence by (7.5.2) and(7.5.3), all points (t, ψε(t)) with 0 ≤ t ≤ 1 and 0 < ε ≤ ε1 are contained in acompact set [0, 1]×X ⊂ I0×X0. Hence by Assumption 6.3.1 and Remark 6.3.3there exists an L2 function M such that for 0 ≤ t ≤ 1 and 0 < ε ≤ ε1,

|fx(ε, t)| ≡ |fx(t, ψε(t), µεt)| ≤M(t) a.e., (7.7.9)

where f = (f0, f1, . . . , fn).From (7.7.9) we get that for 0 ≤ t ≤ 1 and 0 < ε ≤ ε0

|〈p, fx(ε, t)p〉| ≤ ‖p‖2M(t) ≤M(t)(|p|2 + 1) a.e.

Hence by Lemma 4.3.14 there exists a constant A > 0 such that for 0 ≤ t ≤ 1and 0 < ε ≤ ε1,

|P (ε, t)| ≤ A. (7.7.10)

Similarly, for 0 ≤ t ≤ 1 and 0 < ε ≤ ε1

|Λ(ε, t)| ≤ A. (7.7.11)

From (7.6.8) we get that 0 < M(ε)−1 ≤ 1. From this, and from (7.7.4), (7.7.9),and (7.7.10), we get that

∣∣∣∣∫ t

0

P (ε, t)N(ε, s)ds

∣∣∣∣ ≤ A

∫ t

0

M(s)ds+ 2A

∫ t

0

M(s)|ψ′ε(s)− ψ′

∗(s)|ds.

From the Cauchy-Schwarz inequality and from the fact that ψε ∈ Dε we getthat the second integral on the right is less than or equal to

(∫ 1

0

M2(s)ds

)1/2

‖ψ′ε − ψ′

∗‖ < (ε/2)

(∫ 1

0

M2(s)ds

)1/2

.

Thus,

∣∣∣∣∫ t

0

P (ε, s)N(ε, s)ds

∣∣∣∣ ≤ A

∫ 1

0

M(s)ds+Aε

(∫ 1

0

M2(s)ds

)1/2

.


Hence there exists a constant B > 0 such that for all 0 ≤ t ≤ 1 and 0 < ε ≤ ε0∣∣∣∣∫ t

0

P (ε, s)N(ε, s)ds

∣∣∣∣ ≤ B. (7.7.12)

By (7.7.2) the sequence λ(εn, 0) is bounded. From this and from (7.7.7),(7.7.11), and (7.7.12) the inequality (7.7.8) now follows.

We now show that the integrals∫E |λ′(ε, t)|dt are equi-absolutely continu-

ous. From (7.7.3), (7.7.4), (7.7.8), (7.7.9) and the inequality 0 ≤M(ε)−1 ≤ 1,we get the existence of a function M in L2[0, 1] such that for any measurableset E ⊂ [0, 1]

∫

E

|λ′(ε, t)|dt ≤ (C + 1)

∫

E

M(t)dt+ 2

∫

E

M(t)|ψ′ε(t)− ψ′

∗(t)|dt.

From the Cauchy-Schwarz inequality and ψε ∈ Dε, we get that the secondintegral on the right is less than or equal to

(∫

E

M2(t)dt

)1/2

‖ψ′ε − ψ′

∗‖ < (ε/2)

(∫

E

M2(t)dt

)1/2

.

Thus, ∫

E

|λ′(εt)|dt ≤ (C + 1)

∫

E

M(t)dt+ ε

(∫

E

M2(t)dt

)1/2

,

from which the equi-absolute continuity follows.Let εn be the sequence in (7.7.2) and let λ(εn, ·) be the correspond-

ing sequence of functions from the set λ(ε, ·) : 0 < ε ≤ ε1. It followsfrom Lemma 7.7.1 and Ascoli’s theorem that there exists a subsequenceλn ≡ λ(εn, ·) that converges uniformly on [0, 1] to a function λ. The equi-absolute continuity of the integrals

∫E |λ′(ε, t)|dt implies that the functions

λn are equi-absolutely continuous. Hence, by Lemma 5.3.3, the functionλ is absolutely continuous. Corresponding to the sequence λn there existsubsequences ψn ≡ ψεn and µn ≡ µεn. We assert that ψn → ψ∗

uniformly and that µn converges weakly to µ∗. We have

|ψn(t)− ψ∗(t)| ≤ |ψn(0)− ψ∗(0)|+∫ 1

0

|ψ′n(t)− ψ′

∗(t)|dt

≤ |ψn(0)− ψ∗(0)|+ ‖ψ′n − ψ′

∗‖.

Since (ψn, µn) ∈ Dεn and εn → 0, it follows from (7.5.2) that ψn convergesuniformly to ψ∗ on [0, 1]. It also follows from (7.5.2) that ‖µn − µ∗‖L → 0 asn→ ∞. Hence by Corollary 7.3.4, µn → µ∗ weakly.

Since ψn → ψ∗ uniformly and µn → µ∗ weakly, we get from Lemma 4.3.3that for any measurable set ∆ ⊂ [0, 1]

limn→∞

∫

∆

f(t, ψn(t), µnt)dt =

∫

∆

f(t, ψ∗(t), µ∗t)dt.


Since ∆ is arbitrary we get that

limn→∞

f(t, ψn(t), µnt) = f(t, ψ∗(t), µ∗t) a.e. (7.7.13)

A similar argument applied to fx gives

limn→∞

fx(t, ψn(t), µnt) = fx(t, ψ∗(t), µ∗t) a.e. (7.7.14)

By (vi) of Assumption 6.3.1, the convergence in (7.7.13) and (7.7.14) is dom-inated by an L2 function M .

Define:

λn(t) = λ(εn, t)

f0nx(t) ≡ f0

x(εn, t) = f0x(t, ψn(t), µnt)

fnx(t) ≡ fx(εn, t) = fx(t, ψn(t), µnt),

where fx is the matrix with i − j entry (∂f i/∂xj). Then from (7.7.3) and(7.7.4) we have that

λ′n(t) =M(εn)−1f0

nx(t)− fnx(t)Tλn(t) + 2M(εn)

−1fnx(t)(ψ′n(t)− ψ′

∗(t)),

where the superscript T denotes transpose. Hence

λn(t) = λn(0) +

∫ t

0

[M(εn)−1f0

nx(s)− fnx(s)Tλn(s)]ds (7.7.15)

+ 2M(εn)−1

∫ t

0

fnx(s)(ψ′n(s)− ψ′

∗(s))ds.

We now let n → ∞ in (7.7.15). We first consider the rightmost term in(7.7.15). From (7.6.8) we have that 0 < M(ε)−1 ≤ 1. From (7.7.9) we havethat |fnx(t)| ≤M(t), where M is in L2[0, 1]. Hence for all t in [0, 1]∣∣∣∣2M(ε)−1

∫ t

0

fnx(s)(ψ′n(s)− ψ′

∗(s))ds

∣∣∣∣ ≤ 2

∫ t

0

M(t)(|ψ′n(s)− ψ′

∗(s)|)ds(7.7.16)

≤ 2

(∫ 1

0

M2(s)ds

)1/2

‖ψ′n − ψ′

∗‖ < εn

(∫ 1

0

M2(t)dt

)1/2

,

where the last inequality follows from (7.5.2). From (7.7.16) we get that therightmost terms in (7.7.15) tend to zero uniformly on [0, 1] as n→ ∞.

Recall that λn → λ uniformly, that λ is absolutely continuous, that|λn(t)| ≤ C, that (7.7.1) holds, and that (7.7.13) and (7.7.14) hold, with|f0

n(t)| ≤ M(t) and |fnx(t)| ≤ M(t). Therefore, letting n → ∞ in (7.7.15)gives the existence of an absolutely continuous function λ and a constant0 ≤ λ0 ≤ 1 such that for 0 ≤ t ≤ 1

λ(t) = λ(0) +

∫ t

0

[λ0f0

x(s)− fx(s)Tλ(s)]ds, (7.7.17)


wheref0

x(t) = f0x(t, ψ∗(t), µ∗t) fx = fx(t, ψ∗(t), µ∗t).

Differentiating both sides of (7.7.17) gives

λ′(t) = λ0f0

x(t)− fx(t)Tλ(t). (7.7.18)

If we adjoin the equation λ0′

= 0 to the system (7.7.18) we see that (λ0, λ)is the solution of a system of linear homogeneous differential equations. Hence(λ0, λ(t)) never vanishes or is identically equal to zero. We shall show that(λ0, λ(0)) 6= 0, and therefore (λ0, λ(t)) never vanishes. If λ0 > 0, then thereis nothing to prove. By (7.7.1), if λ0 = 0, then M(εn) → ∞, where M(εn) isgiven by (7.6.8). From (7.6.8) we have that |λ(εn, 0)| → +∞. From (7.6.9) wehave that

|λn(0)| = |λ(εn, 0)| = |λ(εn, 0)|[1 + |λ(εn, 0)]−1 = [λ(εn, 0)−1 + 1]−1.

Letting n → ∞ gives |λ(0)| = 1. Thus, (λ0, λ(0)) 6= 0 and therefore (λ0, λ(t))never vanishes.

Let εn denote the subsequence such that λn = λ(εn) converges uniformlyto λ. We shall let εn → 0 in the transversality condition (7.6.11). Since ψn →ψ∗ uniformly and g is continuously differentiable, we have that gxi

(e(ψn)) →gxi

(e(ψ∗)) for i = 0, 1. Also, (ψn(0)−ψ∗(0)) → 0 and M(εn)−1 → λ0. Hence,

as εn → 0, the left-hand side of (6.11) tends to

〈λ0gx0(e(ψ∗))− λ(0), dx0〉+ 〈λ0gx1

(e(ψ∗)) + λ(1), dx1〉 = 0. (7.7.19)

We now let εn → 0 in (7.6.14). We first consider the third term on theleft. From 0 < M(ε)−1 ≤ 1, from (v) of Assumption 6.3.1, and from (7.5.2)we have that

∣∣∣∣∫ 1

0

2M(εn)−1〈ψ′

n(t)− ψ′∗(t), f(t, ψn(t), µnt)〉dt

∣∣∣∣

≤ 2

∫ 1

0

|〈ψ′n(t)− ψ′

∗(t), f(t, ψn(t), µnt)〉|dt

≤ 2

∫ 1

0

|ψ′n(t)− ψ′

∗(t)|M(t)dt

≤ 2‖M‖‖ψ′n − ψ′

∗‖ < εn‖M‖,

where ‖ ‖ denotes the L2 norm. A similar estimate holds for the third termon the right. Therefore, these terms tend to zero as εn → 0.

From (7.6.12) and 0 < M(εn)−1 ≤ 1, it follows that εnγ

′(εn, 0+)M(εn)−1 →

0 as εn → 0.To find the limit as εn → 0 of the first two terms on the left in (7.6.14),

letf0ε (t) = f0(t, ψε(t), µεt) fε(t) = f(t, ψε(t), µεt)


and write the sum of these terms as∫ 1

0

(−M(εn)−1 + λ0)f0

εn(t)dt +

∫ 1

0

(λ(εn, t)− λ(t))fεn (t)dt (7.7.20)

+

∫ 1

0

[−λ0f0εn(t) + 〈λ(t), fεn(t)〉]dt.

The sum of the first two terms in (7.7.20) is in absolute value less than

| −M(εn)−1 + λ0|

∫ 1

0

M(t)dt+

∫ 1

0

|λ(εn, t)− λ(t)|M(t)dt.

Since M(εn)−1 → λ0 and λ(εn, t) → λ(t) uniformly as εn → 0, the sum

of the first two terms in (7.7.20) tends to zero as εn → 0. Since ψn → ψ∗

uniformly and µεn → µ∗ weakly, it follows from Lemma 4.3.3 that as εn → 0,

the integrand in the last term in (7.7.20) converges to −λ0f0(t)+ 〈λ(t), f (t)〉,where f

0(t) = f0(t1ψ∗(t), µ∗t) and f(t) = f(t, ψ∗(t), µ∗(t)). Moreover, the

convergence is dominated by (1+A)M(t), where A is a bound for the sequence|λ(ε, 0)|. Thus, the left side of (7.6.14) converges to

∫ 1

0

[−λ0f0(t) + 〈λ(t), f (t)〉]dt.

A similar argument shows that the right side of (7.6.14) converges to

∫ 1

0

[−λ0f0(t, ψ∗(t), µt) + 〈λ(t), f(t, ψ∗(t), µt)〉]dt.

In summary, we get that as εn → 0, the inequality (7.6.14) tends to

∫ 1

0

[−λ0f0(t) + 〈λ(t), f (t)〉dt (7.7.21)

≥∫ 1

0

[−λ0f0(t, ψ∗(t), µt) + 〈λ(t), f(t, ψ∗(t), µt)〉]dt

for all relaxed controls µ with µt ∈ Ω(t).

If in (7.7.18), (7.7.19), and (7.7.20) we set λ0 = −λ0, and then relabel λ0

as λ0 we obtain the conclusion of Theorem 6.3.5.If (ψ∗, µ∗) is an admissible relaxed optimal pair, then there exists a constant

−1 ≤ λ0 ≤ 0 and an absolutely continuous function λ such that for all t ∈[0, 1], (λ0, λ(t)) 6= 0 and

λ′(t) = −λ0f0x(t)− fx(t)Tλ(t), (7.7.22)

where f0

x(t) and fx(t) are defined in (7.7.17). At the end point e(ψ∗), (λ0, λ(t))

satisfies

〈−λ0gx0(e(ψ∗))− λ(0), dx0〉+ 〈−λ0gx1

(e(ψ∗)) + λ(1), dx1〉 = 0 (7.7.23)


for all tangent vectors (dx0, dx1) to B at e(ψ∗). If e(ψ∗) is a boundary point ofB, then (7.7.23) holds with = replaced by ≥ 0, for all tangent vectors (dx0, dx1)whose projections point into B. For all relaxed controls µ with µt ∈ Ω(t),

∫ 1

0

[λ0f0(t) + 〈λ(t), f(t)〉]dt

≥∫ 1

0

[λ0f0(t, ψ∗(t), µt) + 〈λ(t), f(t, ψ∗(t), µt〉]dt.

We leave the statement of the above result in terms of the function Hr tothe reader.

7.8 Proof of Theorem 6.3.9

Since in the proof we will again compare the optimal pair with otheradmissible pairs, we denote the optimal pair by (ψ∗, µ∗), where

µ∗t =

n+2∑

i=1

p∗i(t)δu∗i (t)

.

If all the functions u∗i , i = 1, . . . , n + 2 are bounded, then their values u∗i (t)are all contained in a compact set Z in R

m. Hence, (ψ∗, µ∗) will be optimalfor the problem with constraint condition µt ∈ Ω′(t), where Ω′(t) = Ω(t) ∩Z.Since all the sets Ω′(t) are contained in the compact set Z, Theorem 6.3.5is applicable to this problem. We therefore assume that not all of the u∗i arebounded.

Since each u∗i is finite a.e. there exists a positive integer k0 such that fork > k0,

Gk = t : |u∗i (t)| ≤ k, i = 1, . . . , n+ 2is non-empty and measurable. For k > k0 and i = 1, . . . , n+ 2, set

Eki = t : |u∗i (t)| > k,

and set Ek = ∪n+2i=1 Eki. For each k the sets Eki are measurable and all are not

empty, so therefore Ek is non-empty and measurable. If I = [0, 1], then

I = Gk ∪ Ek meas Ek → 0 as k → ∞.

For each k > K define a mapping Ωk from I to subsets of Rm by the formula

Ωk(t) = (cl Ω(t)) ∩ cl B(0, k),


where cl denotes closure and B(0, k) is the ball in Rm of radius k, centered at

the origin. All the sets Ωk(t) are compact and are contained in the compactset cl B(0, k). By hypothesis, all the mappings Ωk : t→ Ωk(t) are u.s.c.i. Also,

Ωk(t) ⊆ Ωk+1(t) and Ω(t) =∞⋃

k=1

Ωk(t).

For each positive integer k > k0 we define Problem k to be:

Minimize: g(e(ψ)) +

∫ 1

0

[XEk(t)f0(t, x, µ∗

t ) + XGk(t)f0(t, x, µt)] dt

Subject to:dx

dt= XEk

(t)f(t, x, µ∗t ) + XGk

(t)f(t, x, µt)

µt ∈ Ωk(t) e(ψ) ∈ B,

where XEkis the characteristic function of Ek and XGk

is the characteristicfunction of Gk.

For each k > K define a relaxed control µk as follows:

µkt =

µ∗t if t ∈ Gk

an arbitrary discrete measure control, µt ∈ Ωk(t)

if t ∈ Ek

By Lemma 3.4.5 µt ∈ Ωk(t) exists. For t ∈ Ek, since µkt is a probabilitymeasure, ∫

Ωk(t)

f(t, x, µ∗t )dµkt = f(t, x, µ∗

t ),

where f = (f0, f). The trajectory ψk corresponding to µk satisfies

ψ′k(t) = XEk

(t)f(t, ψk(t), µ∗t ) + XGk

(t)f(t, ψk(t), µ∗t )

= f(t, ψk(t), µ∗t ).

The optimal trajectory ψ∗ also satisfies this differential equation. It followsfrom (6.3.13) in the hypothesis of Theorem 6.3.9 and from standard uniquenesstheorems for differential equations that if we set ψk(0) = ψ∗(0), then ψk(t) =ψ∗(t). From this it further follows that

∫ t

0

f0(s, ψk(s), µks)ds =

∫ t

0

f0(s, ψ∗(s), µ∗s)ds

and that e(ψk) = e(ψ∗) ∈ B. Hence

J(ψk, µk) = J(ψ∗, µk) = J(ψ∗, µ∗). (7.8.1)

We assert that (ψ∗, µk) is optimal for Problem k. To prove this assertion we


first note that Problem k satisfies the hypotheses of Theorem 4.3.5, and thusan optimal pair (ψ, µ) exists. This pair is admissible for the original problem.Thus, if (ψ∗, µk) were not optimal for Problem k we would have that

J(ψ, µ) < J(ψk, µk) = J(ψ∗, µ∗),

the equality coming from (7.8.1). This, however, would contradict the opti-mality of (ψ∗, µ∗), which proves the assertion.

Since (ψ∗, µk) is optimal for Problem k and, as can be readily checked, thedata of Problem k satisfy the hypotheses of Theorem 6.3.5, the pair (ψ∗, µk)satisfies the necessary conditions of Theorem 6.3.5. Thus, there exists a con-stant λ0k ≤ 0 and an absolutely continuous function λk = (λ1k, . . . , λ

nk ) on [0, 1]

such that (λ0k, λk(t)) 6= 0 for all t ∈ [0, 1], and such that:(i)

λ′k(t) = −λ0k[XEk(t)f0

x(t, ψ∗(t), µ∗

t ) + XGk(t)f0

x(t, ψ∗(t), µkt)] (7.8.2)

− [XEk(t)fT

x (t, ψ∗(t), µ∗t ) + XGk

(t)fTx (t, ψ∗(t), µkt)]λk(t)

= −λ0kf0x(t, ψ

∗(t), µ∗t )− fT

x (t, ψ∗(t), µ∗(t))λTk (t),

where fx is the matrix with entry in row i column j (∂f i/∂xj) and the super-script T indicates transpose,(ii)

〈−λ0kgx0(e(ψ∗))− λk(0), dx0〉+ 〈−λ0kgx(e(ψ∗)) + λk(1), dx1〉 = 0 (7.8.3)

for all tangent vectors (dx0, dx1) to B at e(ψ∗), and(iii)

∫ 1

0

[λ0kf0(t, ψ∗(t), µ∗

t ) + 〈λk(t), f(t, ψ∗(t), µ∗t )〉] dt (7.8.4)

≥∫ 1

0

XEk(t)[λ0f0(t, ψ∗(t), µ∗

t ) + 〈λk(t), f(t, ψ∗(t), µ∗t )〉]

+ XGk(t)[λ0kf

0(t, ψ∗(t), µt) + 〈λk(t), f(t, ψ∗(t), µt)〉]dt

for all µt ∈ Ωk(t).All (λ0k, λk) satisfy the differential equation (7.8.2), but may be different for

different k because of different initial conditions (λ0k, λk(0)). By Remark 6.3.8,we may assume that for each k, |(λ0k, λk(0))| = 1. Hence there exists a sub-sequence (λ0k, λk(0)) and a vector (λ0, λ(0)) such that (λ0k, λk(0)) convergesto (λ0, λ(0)). Since |(λ0k, λk(0))| = 1, we have |(λ0, λ(0))| = 1.

Let Λ(t) denote the fundamental matrix of solutions of

q′ = −fTx (t, ψ∗(t), µ∗

t )q (7.8.5)

with Λ(0) = I. Let P (t) denote the fundamental matrix of solutions ofp′ = fx(t, ψ

∗(t), µ∗t )p, the system adjoint to (7.8.5), with P (0) = I. Then


by arguments used to establish (7.7.10) and (7.7.11) we get that there existsa positive constant C such that |Λ(t)| ≤ C and |P (t)| ≤ C. By the variationof parameters formula and (7.8.2)

λk(t) = Λ(t)[λk(0)−∫ t

0

λ0kP (s)f0x(s, ψ

∗(s), µ∗s)ds].

Let

λ(t) = Λ(t)[λ(0) −∫ t

0

λ0P (s)f0x(s, ψ

∗(s), µ∗s)ds].

Thenλ′(t) = −λ0f0

x(t, ψ∗(t), µ∗

t )− fTx (t, ψ∗(t), µ∗

t )λ(t) (7.8.6)

and

|λk(t)− λ(t)| ≤ |Λ(t)|[|λk(0)− λ(0)|

+

∫ t

0

|P (s)||λ0k − λ0|f0x(s, ψ

∗(s), µ∗s)|ds]

≤ C[|λk(0)− λ(0)|+ |λ0k − λ0|∫ 1

0

|f0x(t, ψ

∗(t), µ∗t )|dt].

By (6.3.1), the last integral on the right is bounded. Since λ0k → λ0 andλk(0) → λ(0), it follows that λk → λ uniformly.

Also, |(λ0, λ(0))| = 1. By adjoining λ0′

= 0 to (7.8.6) we get that (λ0, λ) isa solution to a system of linear homogeneous differential equations. Therefore,(λ0, λ) either never vanishes or is identically zero. Since (λ0, λ(0)) 6= 0, we getthat (λ0, λ(t)) never vanishes.

If we let k → ∞ in (7.8.3) we get that

〈−λ0gx0(e(ψ∗))− λ(0), dx0〉+ 〈−λ0gx1

(e(ψ∗)) + λ(1), dx1〉 = 0 (7.8.7)

for all tangent vectors (dx0, dx1) to B at e(ψ∗).The inequality (7.8.4) can be written as

∫ 1

0

XGk(t)[λ0f0(t, ψ∗(t), µ∗

t ) + 〈λk(t), f(t, ψ∗(t), µ∗t )〉] dt (7.8.8)

≥∫ 1

0

XGk(t)[λ0f0(t, ψ∗(t), µt) + 〈λk(t), f(t, ψ∗(t), µt)〉] dt

for all µt ∈ Ωk(t). Let µ be an arbitrary discrete measure control with µt ∈Ω(t) and

µt =

n+2∑

i=1

pi(t)δui(t).

Then

f(t, ψ∗(t), µt) =n+2∑

i=1

pi(t)f(t, ψ∗(t), ui(t)).


By Lemma 3.4.5, for each positive integer k > K there exists a measurablefunction u such that uk(t) ∈ Ωk(t) a.e. in [0, 1]. For each k > K let

vki(t) =

ui(t) if |ui(t)| ≦ k

uk(t) if |ui(t)| > k

Let νk be the discrete measure control given by

νkt =

n+2∑

i=1

pi(t)δvki(t)

Then (7.8.8) holds with µt replaced by νkt. Since the controls ui are finitea.e., for a.e. t in [0, 1] there exists a positive integer k0(t) such that t ∈ Gk

and |ui(t)| ≦ k for all k > k0(t). Hence for a.e. fixed t and k > k0(t),

XGk(t)f (t, ψ∗(t), νkt) = f(t, ψ∗(t), µt). In other words,

limk→∞

XGk(t)f(t, ψ∗(t), µkt) = f(t, ψ∗(t), µt)

for a.e. t in [0, 1]. Also, (λ0, λk(t)) converges uniformly to (λ0, λ(t)). Thus, theintegrand on the right-hand side of (7.8.8) with µt replaced by νkt convergesa.e. to

λ0f0(t, ψ∗(t), µt) + 〈λ(t), f(t, ψ∗(t), µt)〉.Since for all z in the union of the ranges of u1, . . . , un+2,

|f(t, ψ∗(t), z)| ≤M(t),

where M is in L1[0, 1], the convergence is dominated. Also, limXGk(t) = 1

a.e. Using this result in the integral on the left in (7.8.8) and the precedingresult gives∫ 1

0

[λ0f0(t, ψ∗(t), µ∗t ) + 〈λ(t), f(t, ψ∗(t), µ∗

t )〉] dt (7.8.9)

≥∫ 1

0

[λ0f0(t, ψ∗(t), µt) + 〈λ(t), f(t, ψ∗(t)), µt)〉]dt

for all µ with µt ∈ Ω(t). Relations (7.8.6), (7.8.7), (7.8.9), and (λ0, λ(t)) 6= 0for all t establish Theorem 6.3.9.


Conclusions (i) and (iii) of Theorem 6.3.12 follow immediately from The-orems 6.3.5 and 6.3.9. The proof of (ii) requires the notion of point of densityof a measurable set and the notion of approximate continuity of a measurablefunction.


Definition 7.9.1. Let E be a measurable set on the line, let t0 be a point inE, let h > 0, and let I(h) denote the interval [t0 − h, t0 + h]. The point t0 isa point of density of E if

limh→0

meas (E ∩ I(h))/2h = 1.

If meas E > 0, then almost all points of E are points of density of E. See [74,pp. 260–261].

Definition 7.9.2. Let f be a measurable function defined on a closed interval[a, b]. If there exists a measurable subset of E of [a, b] having a point t0 as apoint of density such that f is continuous at t0 with respect to E, then f issaid to be approximately continuous at t0.

A measurable function f defined on a closed interval [a, b] is approximatelycontinuous at almost all points of [a, b]. See [74, p. 262].

We now take up the proof of (ii). Let Z1 denote the set of points of C withrational coordinates. Let Z2 denote the set of points of C that are isolatedpoints or limit points belonging to C of isolated points of C. Then Z2 is denu-merable (see [74, Theorem 2, p. 50]), and therefore so is the set Z = Z1 ∪Z2.

If (6.3.15) were not true, then there would exist a set Ek ⊂ Pk of positivemeasure such that for t ∈ Ek

H(t, ψ(t), uk(t), λ(t)) < M(t, ψ(t), λ(t)). (7.9.1)

For each zi in Z, the function

t→ H(t, ψ(t), uk(t), λ(t))−H(t, ψ(t), zi, λ(t)) (7.9.2)

is approximately continuous at all t ∈ [0, 1] except for those in a set Ti ⊂ [0, 1]of measure zero. Let T = [0, 1] − ⋃Ti. Then all of the functions defined in(7.9.2) are approximately continuous at all points of T .

Let τ ∈ Ek ∩ T . Then from (7.9.1) we get that there exists a z in Z, andtherefore in C, such that

H(τ, ψ(τ), uk(τ), λ(t))−H(τ, ψ(τ), z, λ(t)) < 0.

From the approximate continuity at t = τ of the function defined in (7.9.2)with zi replaced by z, we get that there exists a measurable set E ⊂ Ek ∩ Tof positive measure such that for t ∈ E

H(t, ψ(t), uk(t), λ(t)) < H(t, ψ(t), z, λ(t)). (7.9.3)

Let ν be the discrete measure control given by νt = µt if t 6∈ E and

νt = pk(t)δz +∑

i6=k

pi(t)δui(t)


if t ∈ E. Then

∫ 1

0

Hr(t, ψ(t), µt, λ(t)) −Hr(t, ψ(t), νt, λ(t))dt

=

∫

E

pk(t)H(t, ψ(t), uk(t), λ(t))−H(t, ψ(t), z, λ(t))dt < 0.

This contradicts (6.3.7), which establishes (ii).

Remark 7.9.3. If we assume that f is continuous on I0 ×X0×U0, the proofof (ii) is somewhat simpler. We again note that if (6.3.15) were not true, thenthere would exist a set Ek ⊂ Pk of positive measure such that (7.9.1) holds.

The function h defined by h(t) = H(t, ψ(t), uk(t), λ(t)) is measurable. Henceby Lusin’s theorem there exists a closed set E0 of positive measure such thatE0 ⊂ Ek and h is continuous on E0. Let τ be a point of density of E0. ThenH(τ, ψ(τ), uk(τ), λ(τ)) < M(τ, ψ(τ), λ(τ)), so there exists a z in C such that

H(τ, ψ(τ), uk(τ), λ(τ)) < H(τ, ψ(τ), z, λ(t)).

Since h is continuous on E0, the function t→ H(t, ψ(t), z, λ(t)) is continuousin E0 and since τ is a point of density of E0, there exists a measurable setE ⊂ E0 of positive measure such that (7.9.3) holds for t ∈ E.

7.10 Proof of Theorem 6.3.17 andCorollary 6.3.19

The only conclusions of the theorem that require proof are (6.3.21) and(6.3.22). We noted in (6.3.11) that if µ is a discrete measure control, then

Hr(t, x, µt, q0, q) = 〈q, f(t, x, µt)〉 =

n+2∑

k=1

pk(t)

( n+1∑

j=0

qjf j(t, x, uk(t)

)

=n+2∑

k=1

pk(t)H(t, x, uk(t), q0, q).

The functions (p1, . . . , pn+2, u1, . . . , un+2) are all measurable. Hence there ex-ists a measurable set E ⊆ [0, 1] of full measure such that all of these functionsare approximately continuous on E. Thus, if ω is any of the pi or ui, and t

′ isa point of E, then

limt→t′

t∈E

ω(t) = ω(t′).


We shall describe this succinctly by saying that µ is approximately continuouson E and that

limt→t′

t∈E

= µt′ .

We are assuming that all of the ui, i = 1, . . . , n + 2 are bounded; that isthere exists a constant K such that |ui(t)| ≤ K for all t ∈ [0, 1] and all ui,i = 1, . . . , n+2. The pi are by definition bounded. We shall summarize this bysaying that µ is bounded. Now let (ψ, µ) be an optimal relaxed pair and (λ0, λ)

a corresponding set of multipliers. Then, since f is assumed to be continuouson I0 ×X0 × U0, for t

′ ∈ E

limt→t′

t∈E

H(t, ψ(t), µt, λ0, λ(t)) = H(t′, ψ(t′), µt′ , λ

0, λ(t′)).

We now establish (6.3.21). Let

h(t) = Hr(t, ψ(t), µt, λ(t)). (7.10.1)

Let T1 denote the set of points t in [0, 1] at which (6.3.18) holds. Then measT1 = meas [0, 1]. Let t and t′ be in T1 with t > t′ and let

∆t = t− t′ ∆ψ = ψ(t)− ψ(t′) ∆λ = λ(t)− λ(t′).

LetP (s) = P (s, t′, t) = (t′ + s∆t, ψ(t′) + s∆ψ, µt, λ(t

′) + s∆λ).

Then, by (7.10.1) and (6.3.18),

h(t)− h(t′) ≤ Hr(t, ψ(t), µt, λ(t))−Hr(t′, ψ(t′), µt, λ(t

′)) (7.10.2)

= Hr(P (1))−Hr(P (0)).

The function s → Hr(P (s)) is continuously differentiable on [0, 1]. Hence, bythe Mean Value Theorem, there exists a real number θ in (0, 1) such that

Hr(P (1))−Hr(P (0)) = dHr/ds]s=θ. (7.10.3)

We have

dHr/ds]s=θ = Hrt(P (θ))∆t + 〈Hrx(P (θ)),∆ψ〉 + 〈Hrq(P (θ)),∆λ〉. (7.10.4)

Since ψ and λ are continuous on [0, 1] and λ0 is a constant, all of these functions

are bounded on [0, 1]. Hence so are the ∆ψ and ∆λ for all points t′, t in T1.By assumption µ is bounded on [0, 1]. Hence there exists a closed ball B offinite radius such that for all t′, t in T1 and all 0 ≤ s ≤ 1, the points P (s) are

in B. It then follows from the continuity of f, ft, and fx on I0 ×X0 ×U0 thatthere exists a constant K1 > 0 such that for all t, t′ in T1 and all 0 ≤ s ≤ 1,

|Ht(P (s; t′, t))| ≤ K1 |Hx(P (s; t

′, t))| ≤ K1 |Hq(P (s; t′, t))| ≤ K1.

(7.10.5)


From (6.3.14) we have that

∆ψ =

∫ t

t′Hq(s, ψ(s), µs; λ(s))ds ∆λ = −

∫ t

t′Hx(s, ψ(0), µs, λ(0))ds.

From these relations, the boundedness of ψ, µ, and λ on [0, 1], and the conti-nuity of Hx and Hq, there exists a constant K2 > 0 such that for all t′, t inT1

|∆ψ| ≤ K2∆t |∆λ| ≤ K2∆t.

From this and from (7.10.5), (7.10.4), (7.10.3), and (7.10.2), we get that thereexists a constant K > 0 such that for all t′, t in T1,

h(t)− h(t′) ≤ K(t− t′). (7.10.6)

Also,

h(t)− h(t′) ≥ Hr(t, ψ(t), µt′ , λ(t))−Hr(t′, ψ(t′), µt′ , λ(t

′)).

By arguments similar to those used to obtain (7.10.6) we get that h(t)−h(t′) ≥−K(t− t′). Combining this inequality with (7.10.6) gives

|h(t)− h(t′)| ≤ K|t− t′|

for all t′, t in T1.Thus, the function h is Lipschitz continuous on a dense set in [0, 1]. It is

an easy exercise in elementary analysis to show that h can be extended to afunction h that is Lipschitz on [0, 1] with the same Lipschitz constant as h.

Since h is Lipschitz continuous, it is absolutely continuous. If we now writeh as h, we have (6.3.21). Since h is absolutely continuous, it is differentiablealmost everywhere and

h(t) = c+

∫ t

0

h′(s)ds.

We now calculate h′. Recall that T1 denotes the set of points t in [0, 1] atwhich (6.3.18) holds. Let T2 = E, the set of points at which µ is approximatelycontinuous. Let T3 denote the set of points at which h is differentiable, let T4denote the set of points at which ψ is differentiable, and let T5 denote the setof points at which λ is differentiable. Let T6 denote the set of points at which(6.3.18) holds. Let T denote the intersection of the sets Ti, i = 1, . . . , 6. Theset has full measure. Let t be a point of T . Since h′(t) exists we have that

h′(t) = limtk→t

h(tk)− h(t)

tk − t,

where tk is a sequence of points in T . Let

∆tk = tk − t ∆ψk = ψ(tk)− ψ(t) ∆λk = λ(tk)− λ(t),


and let

P (s; ∆tk) = (t+ s∆tk, ψ(t) + s∆ψk, µtk , λ(t) + s∆λk) 0 ≤ s ≤ 1.

As in (7.10.2) we have

h(tk)− h(t) ≤ Hr(P (1;∆tk))−Hr(P (0;∆tk)).

The function s→ Hr(P (s; ∆tk)) is continuously differentiable on [0, 1]. Hence,by the Mean Value Theorem, there exists a real number θ in (0, 1) such that

Hr(P (1;∆tk))−Hr(P (0;∆tk)) = dHr/ds|s=θ.

Hence

h(tk)− h(t)

tk − t≤ Hrt(P (θ; ∆tk)) +

⟨Hrx(P (θ; ∆tk)),

∆ψk

∆tk

⟩

+

⟨Hrq(P (θ; ∆tk)),

∆λk∆tk

⟩.

If we now let tk → t, we get that

h′(t) ≤ Hrt(π(t)) + 〈Hrx(π(t)), ψ′(t)〉+ 〈Hrq(π(t)), λ

′(t)〉, (7.10.7)

where Π(t) = (t, ψ(t), µt, λ(t)), as in Definition 6.3.11 at t. From (6.3.14) wehave that

λ′(t) = −Hrx(π(t)) ψ′(t) = Hrq(π(t)).

Substituting these into (7.10.7) gives

h′(t) ≤ Hrt(π(t)). (7.10.8)

We also have that

h(tk)− h(t) ≥ H(tk, ψ(tk), µt, λ(tk))−H(t, ψ(t), µt, λ(t)).

An argument similar to the one in the preceding paragraphs gives h′(t) ≥Hrt(π(t)). Combining this with (7.10.8) gives

h′(t) = Hrt(π(t)), (7.10.9)

which is (6.3.22).We conclude this section with a proof of Corollary 6.3.19. It follows from

Definition 6.3.18 that since µ is piecewise continuous on [0, 1] there exist points0 = τ0 < τ1 < · · · < τk = 1 in [0, 1] such that each of the functions pi, ui iscontinuous on the open subintervals (τj , τj+1), j = 0, . . . , k − 1 and has one-sided limits at the points τj , j = 0, . . . , k. We summarize the last statementby saying that µ(τj+0) and µ(τj−0) exist for j = 1, . . . , k − 1 and that µ(τ0+0)

and µ(τk−0) exist.


By the theorem, the mapping

t→ Hr(t, ψ(t), µt, λ(t)) (7.10.10)

is almost everywhere equal to an absolutely continuous function. Since µ iscontinuous on (τj , τj+1), j = 0, . . . , k − 1, the function (7.10.10) is absolutelycontinuous on (τj , τj+1). Since C is closed, µτj−0 and µτj+0 are in C for eachj = 1, . . . , k− 1. From the continuity of µ on each (τj , τj+1) and from (6.3.18)we get that for each τj , j = 1, . . . , k − 1

Hr(τj , ψ(τj), µ(τj+0), λ(τj)) ≤ Hr(τj , ψ(τj), µ(τj−0), λ(τj))

Hr(τj , ψ(τj), µ(τj+0), λ(τj)) ≥ Hr(τj , ψ(τj), µ(τj−0), λ(τj)).

Hence the mapping (7.10.10) is continuous on (0, 1). We make it continu-ous on [0, 1] by taking the value of the mapping (7.10.10) at t = 0 to be

Hr(0, ψ(0), µ(0+0), λ(0)) and at t = 1 to be Hr(1, ψ(t1), µ(1−0), λ(1)).From (6.3.22) we have that

Hr(t, ψ(t), µt, λ(t)) =

∫ t

0

Hrt(s, ψ(s), µs, λ(s))ds+ C, (7.10.11)

with the relation now holding everywhere. From the continuity of ft on I0 ×X0 × U0, it follows that the integrand is continuous on [0, 1]. Hence

dHr

dt= Hrt(t, ψ(t), µt, λ(t))

at all t in [0, 1].


In Section 2.4 we transformed the ordinary problem with possibly variableinitial and terminal times into a problem with fixed initial time t0 = 0 andfixed terminal time t1 = 1. Henceforth we shall call the times (t0, t1), endtimes. For the relaxed problem we again make the change of variable

t = t0 + s(t1 − t0) 0 ≤ s ≤ 1, (7.11.1)

make time t state variable, make s the new independent variable, and intro-duce a new state variable w to transform the problem with possibly variableend times into a problem with fixed end times (0, 1). The state equations forthe transformed problem are

dt

ds= w

dw

ds= 0

dx

ds= f(t, x, µs)w (7.11.2)


where µ is a relaxed control defined on [0, 1]. The integrand in the transformed

problem is f0(t, x, µs)w and the terminal set B is given by

B = (s0, t0, x0, w0, s1, t1, x1, w1) : s0 = 0, s1 = 1 (7.11.3)

(t0, x0, t0, x1) ∈ B w0 = w1 = t1 − t0.

As in Section 2.4, it is readily checked that if (ψ, µ) = (τ, ξ, ω, µ) denotesa relaxed admissible pair for the fixed end time problem and (ψ, µ) a relaxedadmissible pair for the variable end time problem, then there is a one-onecorrespondence between the pairs (τ, ξ, ω, µ) and (ψ, µ) defined by

t = τ(s) ψ(t) = ξ(s) µt = µs ω(s) = t1 − t0, (7.11.4)

where s and t are related by the one to one mapping (7.11.1). Moreover, ifthe pairs (τ, ξ, ω, µ) and (ψ, µ) are in correspondence, then

J(ψ, µ) = J(τ, ξ, ω, µ), (7.11.5)

where

J(τ, ξ, ω, µ) = g(e(τ, ξ, ω)) +

∫ 1

0

f0(τ(s), ξ(s), µs)ω(s)ds.

In Theorems 6.3.5 through 6.3.17, f is assumed to be measurable in t andthe end times are assumed to be fixed at (0, 1). The state equations of thetransformed problem are given by (7.11.2) and the payoff by (7.11.5). Thus,

if f is assumed to be measurable in t, the functionf in the transformed prob-

lem is not of class C(1) in the state variables, as required in (iii) of Assump-

tion 6.3.1. In Theorem 6.3.22 we assume the f is C(1) in (t, x), so the functionf of the transformed problem does satisfy (iii) of Assumption 6.3.1. The otherhypotheses of Theorem 6.3.22 are such that the transformed problem satisfieshypotheses of Theorems 6.3.5 through 6.3.17. Hence the necessary conditionshold for a solution of the transformed problem. In particular the transversalitycondition holds for the transformed problem. Translating this back into thevariables of the original problem will give the conclusion of Theorem 6.3.5.We now proceed to carry out the proof just outlined.

Let (ψ, µ) be an optimal relaxed admissible pair for the variable end timeproblem. Let (τ, ξ, ω, µ) be the admissible pair for the fixed end time problemthat corresponds to (ψ, µ) via (7.11.4) and (7.11.1). Then by virtue of (7.11.5)the admissible pair (τ, ξ, ω, µ) is optimal for the fixed end time problem andsatisfies the conclusions of Theorems 6.3.5 to 6.3.17.

SetH(t, x, z, p0, p) = p0f0(t, x, z) + 〈p, f(t, x, z)〉,

set

H(t, x, w, z, p0, p, a, b) = p0f0(t, x, z)w + 〈p, wf(t, x, z)〉+ aw + b0,


and as in (6.3.5) set

Hr(t, x, w, µs, p0, p, a, b) =

∫

Ω(t)

H(t, x, w, z, p0, p, a, b)dµs,

where 0 ≤ s ≤ 1. Then there exists a constant λ0 ≤ 0 and absolutely contin-uous functions (λ, α, β) defined on [0, 1] such that if

Hr(s) = Hr(τ(s), ξ(s), ω(s), µs, λ0, λ(s), α(s), β(s))

thendλ

ds= −Hrx(s)

dα

ds= −Hrt(s)

dβ

ds= −Hrw(s). (7.11.6)

From the first equation in (7.11.6) we get that

dλ

ds= ω(s)[−λ0f0

rx(s)− fTrx(s)λ], (7.11.7)

where frx(s) = fx(τ(s), ξ(s), µs) and the superscript T denotes transpose.Using the first equation in (7.11.2), the equations in (7.11.4), and setting

λ0 = λ0 and λ(t) = λ(s), where s and t are related by (7.11.1) we can write(7.11.7) as

dλ

dt= −λ0f0

x(t, ψ(t), µt)− fTx (t, ψ(t), µ(t))λ(t)

= −Hrx(t, ψ(t), µt, λ0, λ(t)),

where Hr is given by (6.3.5). Thus, the first equation in (7.11.6) gives us nonew information.

For the fixed end time problem, the condition (6.3.7) takes the form

∫ 1

0

Hr(s)ds ≥∫ 1

0

Hr(τ(s), ξ(s), ω(s), µs, λ0, λ(s), α(s), β(s))ds. (7.11.8)

Using the first equation in (7.11.2), the relation (λ0, λ(s)) = (λ0, λ(t)) andthe fact that the terms involving α and β are independent of µ, we transform(7.11.8) into (6.3.7), so we again get no new information.

In summary, we have shown that the necessary conditions, other than thetransversality condition, for the problem with variable end times are the sameas those for the fixed end time problem. We next take up the transversalitycondition.

The end points of (τ, ξ, ω) are given by

τ(0) = t0 τ(1) = t1 ξ(0) = x0 ξ(1) = x1 ω(0) = ω(1) = t1 − t0,(7.11.9)


where (t0, x0, t1, x1) ∈ B. From Theorems 6.3.5 to 6.3.17 we get that thetransversality condition for the fixed end time problem is

−λ0dg + 〈λ(1), dx1〉+ α(1)dt1 + β(1)dw1 − 〈λ(0), dx0〉 (7.11.10)

− α(0)dt0 − β(0)dw0 = 0

for all tangent vectors (dt0, dx0, dw0, dt1, dx1, dw1) to B at the end point of(τ, ξ, ω), and where

dg = gt0dt0 + 〈gx0, dx0〉+ gt1dt1 + 〈gx1

, dx1〉

with the partials of g evaluated at the end point of (τ, ξ, ω). If we use (7.11.1)

and set (λ0, λ(t)) = (λ0, λ(s)), we can write (7.11.10) as

−λ0dg + 〈λ(t1), dx1〉+ α(t1)dt1 + β(t1)dw1 − 〈λ(t0), dx0〉 (7.11.11)

− α(t0)dt0 − β(t0)dw0 = 0.

Henceforth we shall always set (λ0, λ(t)) = (λ0, λ(t)).From the second equation in (7.11.6) we get that

dα

ds= −ω(s)[λ0f0

t (τ(s), ξ(s), µs) + 〈λ(s), ft(τ(s), ξ(s), µs].

Using the first equation in (7.11.2) and (7.11.1) gives

dα

dt= −[λ0f0

t (t, ψ(t), µt) + 〈λ(t), ft(t, ψ(t), µt)〉]

= −Hrt(t, ψ(t), λ0, λ(t)) t0 ≤ t ≤ t1.

Hence

α(t)− α(t0) = −∫ t

t0

Hrt(s, ψ(s), µs, λ0, λ(s))ds.

The hypotheses of Theorem 6.3.22 imply those of Theorem 6.3.17, so from(6.3.22) we get that

α(t)− α(t0) = −Hr(t) +Hr(t0), (7.11.12)

where Hr(t) = Hr(t, ψ(t), µt, λ0, λ(t)).

From the last equation in (7.11.6) we get that

dβ

ds= −λ0f0(τ(s), ξ(s), µs)− 〈λ(s), f(τ(s), ξ(s), µs)〉 − α(s)

and so

dβ

dt

dt

ds= −λ0f0(t, ψ(t), µt)− 〈λ(t), f(t, ψ(t), µt)〉 − α(t).


Using the first two equations in (7.11.2) and the relation w1 = (t1 − t0) in(7.11.3) gives

dβ

dt= −(t1 − t0)

−1[Hr(t) + α(t)].


dβ

dt= −(t1 − t0)

−1[Hr(t0) + α(t0)],

and so

β(t) = − (t− t0)

(t1 − t0)[Hr(t0) + α(t0)] + β(t0). (7.11.13)


α(t1)dt1 = [−Hr(t1) + α(t0) +Hr(t0)]dt1. (7.11.14)

From (7.11.13) and dw1 = (dt1 − dt0) we get that

β(t1)dw1 = [−Hr(t0)−α(t0)]dt1 + [Hr(t0) +α(t0)]dt0 + β(t0)dw1. (7.11.15)

Substituting (7.11.15) and (7.11.14) into (7.11.11) and recalling that w1 = w2

we obtain

−λ0dg −Hr(t1)dt1 + 〈λ(1), dx1〉+Hr(t0)dt0 − 〈λ(0), dx0〉 = 0

for all tangent vectors (dt0, dx0, dt1, dx1) to B at the end point e(ψ) and wherethe partials of g are evaluated at e(ψ).

Chapter 8

Examples

8.1 Introduction

In this chapter we illustrate the use of results presented in the precedingchapters to determine optimal controls and optimal trajectories.

8.2 The Rocket Car

A car, which we take to be a point mass, is propelled by rocket thrusts alonga linear track endowed with coordinates. Units are assumed to be normalizedso that the equation of motion is x = u, where u is the thrust force constrainedto satisfy −1 ≤ u ≤ 1. Initially the car is at a point x0 with velocity y0. Theproblem is to determine a thrust program u that brings the car to rest at theorigin in minimum time.

If we consider the state of the system to be x, the position of the car, andy, the velocity of the car, then we can write the equations governing the stateof the system by

x = y x(0) = x0 (8.2.1)

y = u y(0) = y.

The problem is to determine a control function u such that

|u(t)| ≤ 1 (8.2.2)

that minimizes

J = tf =

∫ tf

0

dt, (8.2.3)

where tf is the time at which the state (x, y) reaches the origin (0, 0). We havecast the problem as the control problem: Minimize (8.2.3) subject to the stateequations (8.2.1), control constraint (8.2.2), initial condition T0 = (0, x0, y0),and terminal condition T1 = (tf , 0, 0).

249


As an exercise, we ask the reader to show that for each initial point(0, x0, y0), the set of admissible trajectories is not empty. In this examplewe have that

|〈x, f(t, x, z)〉| = |xy + yz| ≤ |xy|+ |yz|≤ (|x|2 + |y|2)/2 + (|y|2/2) + 1/2 ≤ (|x|2 + |y|2 + 1).

It then follows from Lemma 4.3.14 that for a fixed initial point (x0, y0), the setof admissible trajectories lie in a compact set. It is readily verified that all theother hypothesis of Theorem 4.4.2 are satisfied. Thus, there exists an ordinaryadmissible pair that is the solution of both the relaxed and ordinary problems.Having shown the existence of a solution the next step is to determine theextremal trajectories and if there is more than one, determine which is thesolution.

Since the problem is a time optimal problem that is linear in the state andwith terminal state the origin, a preferable procedure is the following. Thestate equation can be written as

(xy

)=

(0 10 0

)(xy

)+

(01

)u = A

(xy

)+Bu.

The matrix B is an n× 1 column vector and the vectors B, AB are linearlyindependent. Hence by Corollary 6.7.16 the system is strongly normal. Theother hypotheses of Theorem 6.8.2 are clearly satisfied. Hence an extremaltrajectory is optimal, so we only need to determine extremal trajectories.From Theorem 6.8.1 we get that the optimal control only takes on the values+1 and −1.

The data of the problem satisfy the hypothesis of Theorem 6.3.27. There-fore an extremal trajectory and corresponding control satisfy the conclusionof Theorem 6.3.27.

The function H defined by (6.3.4) becomes

H = q0 + q1y + q2z.

The second set of equations (6.3.28) are

dλ1

dt= −Hx = 0 (8.2.4)

dλ2

dt= −Hy = −λ1.

Henceλ1(t) = c, λ2(t) = −c1t+ c2. (8.2.5)

The terminal set B = (0, x0, y0, tf , 0, 0) : tf free. Therefore, the transversal-ity condition (6.3.31) becomes

λ0 + λ1(tf )y(tf ) + λ2(tf )u(tf ) = 0. (8.2.6)

Examples 251

If c21 + c22 = 0, then λ1(t) = λ2(t) = 0 for all t. Then by (8.2.6) λ0 = 0,which contradicts the conclusion (λ0, λ1(t), λ2(t) 6= (0, 0, 0). Hence c21+c

22 6= 0.


H = λ0 + c1y + (c2 − c1t)z, (8.2.7)

and from (8.2.6) we get that

λ0 + (c2 − c1tf )u(tf ) = 0.

At time t the value u(t) of the optimal control maximizes (8.2.7) over theinterval |z| ≤ 1. Hence

u(t) = signum (c2 − c1t).

Thus, if c1 = 0, then c2 6= 0 and u(t) = signum c2 for all t. Let c2 > 0.Then u(t) = 1 for all t. We shall “back out of the target along the extremaltrajectory.” That is, we shall reverse time and determine all initial states(x0, y0) that can be reached using u(t) = 1 along the entire trajectory. Thus,setting t = −τ gives

dx

dτ= −y dy

dτ= −u x(0) = y(0) = 0,

and so with u(t) = 1

y(τ) = −τ x(τ) = τ2/2 τ ≥ 0. (8.2.8)

Equations (8.2.8) are the parametric equations of the parabolic segment OA(See Fig. 8.1) whose Cartesian equation is

y = −√2x x ≥ 0. (8.2.9)

Thus, we may take any point (x0, y0) on OA as an initial point. The optimalcontrol will be u(t) = 1 for 0 ≤ t ≤ tf and the optimal trajectory will be theportion of OA with 0 ≤ x ≤ x0.

Let W (x0, y0) denote the terminal time of an extremal, and hence optimaltrajectory with initial point (x0, y0) on OA. It follows from (8.2.8) that

W (t0, x0) = −y0. (8.2.10)

Recall that y0 < 0 for (x0, y0) ∈ OA.A similar analysis for c1 = 0 and c2 < 0 gives the existence of the parabolic

segment OB whose Cartesian equation is

y =√−2x, x ≤ 0 (8.2.11)

such that for any initial point (x0, y0) on OB, the optimal trajectory with


FIGURE 8.1

(x0, y0) is the portion of OB given by x0 ≤ x ≤ 0 and the optimal control isu(t) = −1. Also,

W (x0, y0) = y0. (8.2.12)

If c2 = 0 and c1 > 0 we again get the curve OA.If c2 = 0 and c1 < 0 we again get OB.We now consider the case in which c1c2 6= 0. We again reverse time by

setting t = −τ and “back out from target.” Then u(τ) = signum (c2 + c1τ)maximizes (8.2.7) over |z| ≤ 1. Thus if c1 > 0 and c2 > 0, we again get OA.If c1 < 0 and c2 < 0 we again get OB.

We now consider the case c1 < 0 and c2 > 0. Then u(τ) = 1 on someinterval [0, τs], where

τs = −c2/c1.From (8.2.8) we get that at τ = τs

ys ≡ y(τs) = −τs xs ≡ x(τs) = τ2s /2. (8.2.13)

For τ ≥ τs, the optimal control is u(τ) = signum (c2 + c1τ) = −1. Hence forτ ≥ τ optimal trajectory is the curve defined by

dy

dτ= −u(τ) = 1

dx

dτ= −y (8.2.14)

Examples 253

with initial conditions given by (8.2.13). It then follows that the optimal tra-jectory for τ ≥ τs is given parametrically by

y(τ) = τ − 2τs x(τ) = −(τ − 2τs)2/2 + τ2s . (8.2.15)

To summarize, in the case c1 < 0, c2 > 0 the optimal trajectory traversed“backwards in time” is the segment of OA corresponding to the time interval[0, τs], followed by the curve defined by (8.2.15) on [τs,∞). The time τs iscalled switching time. The parabolic segment defined by (8.2.15) lies on theparabola whose Cartesian equation is

y2 + 2x = 4x. (8.2.16)

Let Σ = OA ∪ OB. The curve Σ is called a switching curve. It follows from(8.2.9) and (8.2.11) that if we denote the region above the curve Σ by R−,then

R− ≡ (x, y) : y > −(signum x)√2|x|.

We assert that for each point (x0, y0) in R− there exists a solution to theproblem defined by Eqs. (8.2.1) to (8.2.3). The optimal trajectory consists ofa segment of the parabola given by (8.2.16), followed by a segment of OA. Theoptimal control is u(t) = −1 for the segment given by (8.2.16) and u(t) = 1for the segment in OA.

We shall prove this assertion by showing that given (x0, y0) in R− andreversing time by setting τ = −t, there is a trajectory consisting of a segmentof OA with initial point the origin corresponding to an interval [0, τs], followedby the extremal defined parametrically by (8.2.15) that reaches (x0, y0) atsome time τ0 > τs. Thus, we need to show that there exists a τ0 > τs suchthat

y0 = (τ0 − 2τs) x0 = −(τ0 − 2τs)2/2 + τ2s .

Hence the point (x0, y0) must lie on a parabola whose equation is (8.2.16).This parabola is a translate to the right of the parabola on which OB lies.It is geometrically obvious that the parabola (8.2.16) intersects OA at somepoint (xs, ys). This proves the assertion.

The point (x0, y0) is on the parabola (8.2.16) and therefore (x0, y0) satisfies(8.2.16). Combining this just with the fact that xs satisfies (8.2.9) gives:

xs = (y20 + 2x0)/4 ys = −(y20/2 + x0)1/2. (8.2.17)

We now determine the time τ0 required to traverse an optimal trajectoryfrom (x0, y0) in R− to the origin. That is, we determine the value functionW (x0, y0) in R−. We first determine the time required to go from y0 to ys.From (8.2.14) we get that

y0 − ys = y(τ0)− y(s0) = τ0 − τs.

From this and from (8.2.17) we get that

τ0 = y0 + (y20/2 + x0)1/2 + τs.


From (8.2.13) we get that τs, the time required to go from (xs, ys) to (0, 0),is given by τs = −ys. It then follows from (8.2.17) that

W (t0, x0) = y0 + 2(y20/2 + x0)1/2 (x0, y0) ∈ R−. (8.2.18)

For points (x0, y0) on OA, the right-hand side of (8.2.18) equals −y0. It thenfollows from (8.2.10) that (8.2.18) is valid for all (x0, y0) in R− ∪OA.

Let R+ denote the region below Σ. Then

R+ = (x, y) : y < −(signum x)(2|x|)1/2,−∞ < x <∞.If (x0, y0) ∈ R+, then the optimal control is u(t) = 1 on an interval [0, ts]and then u(t) = −1. The optimal trajectory is a segment of the parabolay2 − 2x = y20 − 2x0. Corresponding to u(t) = 1, until the parabola intersectsOB. The motion proceeds with u(t) = −1 along OB to the origin. For (x0, y0)in R+ ∪OB

W (x0, y0) = −y0 + 2(y20/2− x0)1/2. (8.2.19)

We leave the arguments justifying the statements in this paragraph to thereader.

Our analysis has also provided an optimal synthesis U(x, y). At each pointin R, U(x, y) is the value of the optimal control at (x, y) for an optimal tra-jectory starting at the point (x, y). We have that U(x, y) = −1 for (x, y) inR− ∪OB and U(x, y) = 1 for (x, y) in R+ ∪OA.

Since (8.2.18) holds on R−∪OA and (8.2.19) holds on R+∪OB, it followsthat W is continuous on all of R2. It further follows from (8.2.18) and (8.2.19)that W is continuously differentiable on R− ∪R+ and is Lipschitz continuouson compact subsets of R− ∪R+.

Exercise 8.2.1. (i) Let (ξ, η) ∈ OA.

(a) Find limWx(x, y) and limWy(x, y) as (x, y) → (ξ, η), where (x, y) ∈R−.

(b) Find limWx(x, y) and limWy(x, y) as (x, y) → (ξ, η), where (x, y) ∈R+.

(ii) Let (ξ, η) ∈ OB.

(a) Find limWx(x, y) and limWy(x, y) as (x, y) → (ξ, η), where (x, y) ∈R−.

(b) Find limWx(x, y) as (x, y) → (ξ, η), where (x, y) ∈ R+.

(iii) Determine and sketch the curves of constant W .

Exercise 8.2.2. Determine the optimal controls and optimal trajectoriesfor the time optimal problem with state equations (8.2.1), control constraint(8.2.2) and terminal set y = 0.

Exercise 8.2.3. Find the optimal controls and optimal trajectories for thetime optimal problem with state equations (8.2.1), control constraints (8.2.2),and terminal set x2 + y2 = ε2. What happens as ε→ 0?

Examples 255

8.3 A Non-Linear Quadratic Example

Consider the systemdx

dt= −xu, (8.3.1)

where x and u are scalars. Let the end conditions be t0 = 0, t1 = 1, x0 = 1,and x1 free. Let the constraint condition be 0 ≤ u(t) ≤ 1 and let the payoffbe

J(φ, u) = φ(1)2/2 +1

2

∫ 1

0

u2(t)dt. (8.3.2)

Show that an optimal control and trajectory exist and find them.In the notation used in this text, in this problem g(x1) = x21/2, f = z2/2,

Ω(t) = [0, 1] and B = (t0, x0, t1, x1) : t0 = 0, x0 = 1, t1 = 1, x1 = σ,−∞ < σ < ∞. For each (t, x), the set Q+(t, x) is the set of points (y0, y1)such that y0 ≥ 1

2 (z)2, and y1 = −xz, where 0 ≤ z ≤ 1. It follows from (8.3.1)

and Ω(t) = [0, 1] that all admissible trajectories are contained in the compactset bounded by x = 1, x(t) ≥ e−t, 0 ≤ t ≤ 1, and t = 1. The other hypothesesof Theorem 4.4.2 hold, so an ordinary solution that is also a solution of relaxedproblem exists.

The admissible control u(t) ≡ 0 results in the trajectory φ(t) ≡ 1 andJ(φ, u) = 1/2. The admissible control u(t) ≡ 1 results in the trajectoryφ(t) = e−t and J(φ, u) = [e−2 + 1]/2 > 1/2. Thus, the end point of theoptimal trajectory lies in the interval (e−1, 1]. We first assume that the endpoint of the optimal trajectory φ is in the open interval (e−1, 1), and will useTheorem 6.3.27 to determine φ and the optimal control u.

The function H is given by

H = q0z2/2− qxz. (8.3.3)

The differential equations (6.3.28) become

φ′(t) = Hλ = −φ(t)u(t) (8.3.4)

λ′(t) = −Hx = λ(t)u(t).


(λφ)′ = λ′φ+ λφ′ = λφu + λ(−φu) = 0.

Henceλ(t)φ(t) = const. = λ(1)φ(1). (8.3.5)

The differential equations in (8.3.4) are adjoint to each other. Thus, (8.3.5) isa special case of a general result established in Lemma 6.6.2.


Since we are assuming that φ(1) ∈ (e−1, 1), we may take the set B to begiven parametrically by

t0 = 0 x0 = 1 t1 = 1 x1 = σ, σ ∈ (e−1, 1).

The unit tangent vector to B at the point (0, 1, 1, φ(1)) is (0, 0, 0, 1). Thetransversality condition (6.3.31) becomes

0 = −λ0gx1(φ(1)) + λ(1) = −λ0φ(1) + λ(1).

Hence λ(1) 6= 0, for if λ(1) = 0, the λ0 = 0, which contradicts (λ0, λ(t)) 6=(0, 0) for all t ∈ [0, 1]. We may therefore take λ0 = −1. The transversalitycondition becomes φ(1)+ λ(1) = 0, and so λ(1) = −φ(1). From this and from(8.3.4) we get that for all t ∈ [0, 1]

λ(t)φ(t) = −[φ(1)]2 = −s2, (8.3.6)

where s is the value of the parameter σ such that φ(1) = s.Setting q0 = λ0 = −1, q = λ(t) and x = ϕ(t) in (8.3.3) and then using

(8.3.6), we get from (6.3.29) that the value u(t) of the optimal control at timet maximizes

F (z) = −z2/2 + s2z 0 ≤ z ≤ 1.

From F ′(z) = −z + s2 and F ′′(z) = −1 we get that F is concave on [0, 1]and is maximized at u(t) = s2. The trajectory corresponding to the controlu(t) ≡ s2 is φ(t) = exp(−s2t). Thus, the optimal pair (φ, u) is

φ(t) = e−s2t u(t) ≡ s2, (8.3.7)

and by (8.3.2)

J(φ, u) = (e−2s2 + s4)/2. (8.3.8)

From (8.3.7) we have φ(1) = exp(−s2). By the definition of s, φ(1) = s.Hence s must satisfy the equation

σ = e−σ2

. (8.3.9)

It is readily seen that this equation has a unique solution s, lying in the interval(0, 1). Moreover, it is also readily verified that

.6500 < s < .6551. (8.3.10)

Using (8.3.9) and (8.3.10) we get that

J(φ, u) = (s2 + s4)/2 < 0.3067. (8.3.11)

The preceding analysis was based on the assumption that φ(1) ∈ (e−1, 1).We already showed that the control u(t) ≡ 1, which yields a trajectory withend point e−1, cannot be optimal. To conclude that (φ, u) is indeed optimal wemust show that u(t) ≡ 0, which yields a trajectory with end point one, cannotbe optimal. We already showed that the payoff using the control u(t) ≡ 0equals 1/2. Comparing this with (8.3.11) shows that (φ, u) is indeed optimal.

Exercise 8.3.1. Investigate the problem with terminal time T > 1.

Examples 257

8.4 A Linear Problem with Non-Convex Constraints

A boat moves with velocity of constant magnitude one relative to a streamof constant speed s. It is required to transfer the boat to a point (ξ, η) inminimum time. The equations of motion of the boat are

dx

dt= s+ u1

dy

dt= u2 (u1)

2 + (u2)2 = 1. (8.4.1)

(i) Show that whenever the set of admissible pairs is not empty there existsan ordinary admissible pair (φ, u) that is a solution of the ordinary andrelaxed problems.

(ii) Determine the unique optimal pair in this case.

(iii) Find the minimum transfer time as a function of (ξ, η), that is, find thevalue function.

(iv) For each of the cases s < 1, s = 1, s > 1 find the set S of points (ξ, η)for which the problem has a solution.

(v) For s > 1 show that for points on the boundary of S, λ0 = 0 along theoptimal trajectory.

The sets Q+(x, y) in this problem are not convex. The problem is linear,however, and all of the other hypotheses of Theorem 4.7.8 are fulfilled, soif there exists a trajectory that reaches (ξ, η), then there exists an ordinaryoptimal pair that is also a solution of the relaxed problem.

Let (ξ, η) be a point that can be reached from the origin. Let (φ, u) denotethe optimal pair. The function H is given by

H = q0 + q1(s+ z1) + q2z2.

The end set B is given by

B = (t0, x0, y0, t1, x1, y1) : t0 = x0 = y0 = 0, t1 free, x1 = ξ, y1 = η.

The unit tangent vector to B is (0, 0, 0, 1, 0, 0). The transversality conditiongives

λ0 + λ1(t1)(s+ u1(t)) + λ2(t1)u2(t1) = 0.

From this we get that |λ1(t1)| + |λ2(t1)| 6= 0, for otherwise λ0 = 0, whichcannot be since (λ0, λ1(t), λ2(t)) 6= (0, 0, 0) for all t.

Let λ(t1) = c1 and λ(t2) = c2. The equations for the multipliers are

dλ1dt

= −Hx = 0dλ2dt

= −Hy = 0.


Hence for all tλ1(t) = c1 λ2(t) = c2 (8.4.2)

for some constants c1 and c2 with |c1| + |c2| 6= 0. We may therefore write Has

λ0 + c1(s+ z1) + c2z2. (8.4.3)

Hence the optimal control at time t is a unit vector (u1(t), u2(t)) that maxi-mizes

λ1(t)z1 + λ2(t)z2 = c1z1 + c2z2 = 〈(c1, c2), (z1, z2)〉.Thus, (u1(t), u2(t)) is a unit vector in the direction of (c1, c2), and so

u1(t) = c1/(c21 + c22)

1/2 u2(t) = c2/(c21 + c22)

1/2. (8.4.4)

Let α ≡ u1(t) and β ≡ u2(t). It then follows from (8.4.1) and the initialcondition x(0) = y(0) = 0 that the optimal trajectory is given by

x(t) = (s+ α)t y(t) = βt, (8.4.5)

which is a line from the origin to (ξ, η).Let τ = τ(ξ, η) be the time required to reach (ξ, η) from the origin. Then

from (8.4.5) we get

ξ2 = s2τ2 + α2τ2 + 2sατ2

η2 = β2τ2.

From this and from sατ = sξ−s2τ , which we get from (8.4.5), we get that

(1− s2)τ2 + 2sξτ − (ξ2 + η2) = 0. (8.4.6)

Therefore,

τ =[−sξ ± (ξ2 + (1− s2)η2)1/2]

(1− s2). (8.4.7)

If s < 1, since τ > 0 we must take the plus sign in (8.4.7) and get that

τ =[−sξ + (ξ2 + (1− s2)η2)1/2]

(1− s2). (8.4.8)

Since (ξ2 + (1− s2)η2)1/2 is greater than sξ for s < 1 and all ξ, it follows thatτ > 0. Thus, all points (ξ, η) can be reached. The boat moves faster than thestream, and even points upstream (ξ < 0) can be reached.

If s = 1, then (8.4.6) becomes 2ξτ − (ξ2 + η2) = 0. Hence, since ξ 6= 0

τ =(ξ2 + η2)

2ξ.

Thus, all points with ξ > 0, that is, all points downstream, can be reached.

Examples 259

If s > 1, we rewrite (8.4.8) as

τ =[sξ ± (ξ2 − (s2 − 1)η2)1/2]

(s2 − 1). (8.4.9)

From (8.4.5), since s > 1, |α| ≤ 1, and τ > 0, we get that ξ > 0. For τ to bereal we require that ξ2 − (s2 − 1)η2 ≥ 0, or equivalently

−(s2 − 1)−1/2 ≤ η

ξ≤ (s2 − 1)−1/2. (8.4.10)

Thus if, s > 1 any points (ξ, η) that lie in the region subtended by the angledetermined by the line segments from the origin with slope −(s2− 1)−1/2 and(s2 − 1)1/2 can be reached.

We assert that in (8.4.9) we take the minus sign. We have

(ξ2 − (s2 − 1)η2)1/2 =

(ξ2(1− (s2 − 1)η2

ξ2

))1/2

≤ (ξ2)1/2 < sξ,

where the next to the last inequality follows from (8.4.10). Therefore since wewant the smallest positive root in (8.4.9), we take the minus sign.

It follows from (8.4.2), z1 = u1(t), z2 = u2(t), and (8.4.3) that H evaluatedalong an optimal trajectory is given by

H = λ0 + (c21 + c22)1/2 + c1s.

From the transversality condition we concluded that H at the end time τ waszero. Therefore, again using (8.4.4), we get

− λ0(c21 + c2)1/2

= 1 + u1(t)s = 1 + αs.

From (8.4.5) we get that α = (ξ − sτ)/τ . From (8.4.9) and (8.4.10) we getthat along a boundary line of the reachable set, τ = sξ/(s2 − 1). Substitutingthis into the expression for α gives 1 + αs = 0. Hence λ0 = 0.

8.5 A Relaxed Problem

In Example 3.1.1 we used the following problem to motivate and introducethe concept of relaxed controls.

Problem 8.5.1. Minimize∫ t

0 1dt subject to the state equations

dx

dt= y2 − u2

dy

dt= u,


control constraints Ω(t) = z : |z| ≤ 1, initial set T0 = (t0, x0, y0) =(0, 1, 0), and terminal set T1 = (t1, x1, y1) : x21 + y21 = a2, t1 free, where0 < a < 1. We showed that this problem has no solution, but the correspondingrelaxed problem, whose state equations are

dx

dt= y2 −

∫ 1

−1

z2dµtdy

dt=

∫ 1

−1

zdµt, (8.5.1)

has a solution. Moreover, the solution is the discrete measure control

µt =

2∑

i=1

pi(t)δui(t),

where p1(t) = p2(t) = 1/2 and u1(t) = 1, u2(t) = −1. Here we shall use anexistence theorem and Theorem 6.3.22 to determine the solution.

In Example 3.1.1 we constructed a minimizing sequence, all of whose tra-jectories lie in a compact set. Hence, by Corollary 4.3.13, the relaxed problemhas a solution, (ψ, µ). We next use Theorem 6.3.22 to determine this solution.Although the direct method used in Example 3.1.1 is simpler, it is instructiveto see how Theorem 6.3.17 is used.

The functions H and Hr in the problem at hand are

H = q0 + q1(y2 − z2) + q2z (8.5.2)

Hr = q0 + q1y2 − q1

∫ 1

−1

z2dµt + q2

∫ 1

−1

zdµt

Equations (6.3.6) become

dλ1dt

= 0dλ2dt

= −2λ1y.

Hence

λ1(t) = cdλ2dt

= −2cy 0 ≤ t ≤ tf , (8.5.3)

where tf denotes the time at which the optimal trajectory hits T1.Let (tf , xf , yf ) be the point at which the optimal trajectory first hits T1.

Let −π/2 ≤ α ≤ π/2 be the angle such that xf = a cosα and yf = a sinα.Then in a neighborhood N (tf , xf , yf ) the terminal manifold can be givenparametrically by

x1 = a cos(θ + α) y1 = a sin(θ + α) t1 = τ

for θ and τ each in an interval (−δ, δ). The general tangent vector to T1 atpoints in the neighborhood N of (tf , xf , yf ) is (dτ,−a sin(θ+α)dθ, a cos(θ+α)dθ), which at the point (tf , xf , yf) is

(dτ, (−a sinα)dθ, a cosα)dθ).

Examples 261

The transversality condition (6.3.24) together with (8.5.1), (8.5.2), and (8.5.4)give

λ0+c[y2f −∫ 1

−1

z2dµt] + λ2(tf )

∫ 1

−1

zdµtf = 0 (8.5.4)

a[−c sinα+ λ2(tf ) cosα] = 0.

We next show that xf 6= 0 by assuming the contrary and reaching acontradiction. If xf = 0, then α = ±π/2 and the second equation in (8.5.4)gives λ1(t) = c = 0. It follows from (8.5.4), (6.3.21), and (6.3.22) that for0 ≤ t ≤ tf

λ0 + λ2(t)

∫ 1

−1

zdµt = 0.

Hence λ2(t) 6= 0 for all 0 ≤ t ≤ tf , for otherwise λ0 = 0 and there would exist

a t′ ∈ [0, tf ] such that (λ0, λ1(t′), λ2(t

′) = (0, 0, 0), which cannot be.The optimal control µ is a discrete measure control that is not an ordinary

control. Thus,

µt =

2∑

i=1

pi(t)δui(t) 0 < pi < 1 i = 1, 2 (8.5.5)

for t in a set P of positive measure contained in [0, tf ].Hence by (6.3.15), for t in P , maxH(t, ψ(t), z, λ(t)) : |z| ≤ 1 occurs at

two distinct points. Since c = 0, from (8.5.2) we get that

H(t, ψ(t), z, λ(t)) = λ0 + λ2(t)z,

whose maximum on the interval |z| ≤ 1 occurs at a unique point, since λ2(t) 6=0 for all t. Thus, xf 6= 0.

The preceding argument also shows that c 6= 0. Once we have establishedthat xf 6= 0, a simpler argument exists. Since xf 6= 0, cosα 6= 0. From thesecond equation in (8.5.4) we get that if c = 0, then λ2(tf ) = 0. From the firstequation in (8.5.4) we get that λ0 = 0, and so (λ01, λ1(tf ), λ2(tf )) = (0, 0, 0),which cannot be. Therefore, c 6= 0.

We now determine the optimal relaxed controls. From (8.5.2), (8.5.5), and(6.3.15) we get that

maxQ(z) ≡ −cz2 + λ2(t) : |z| ≤ 1

must occur at two distinct points of the interval −1 ≤ z ≤ 1. The function Qis a quadratic whose graph passes through the origin. It is easy to see that ifc > 0, then it is not possible for Q to have two maxima in the interval [−1, 1].It is also easy to see that if c < 0, for Q to have two maxima in the interval[−1, 1], we must have λ2(t) = 0. The maxima will then occur at z = 1 andz = −1. Thus,

u1(t) = 1 and u2(t) = −1. (8.5.6)


From λ2(t) ≡ 0 and (8.5.3) we get, since c 6= 0, that y(t) ≡ 0. From (8.5.1),(8.5.5), and (8.5.6) we get that 0 = p1(t)−p2(t). Also, p1(t)+p2(t) = 1. Hencep1(t) = p2(t) = 1/2, and the optimal relaxed control is

µ1 =1

2δ1 +

1

2δ−1.

From (8.5.1) and (t0, x0, y0) = (0, 1, 0) we get that the optimal trajectory is

x(t) = 1− t y(t) = 0

and the optimal time is tf = 1− a.

8.6 The Brachistochrone Problem

In Section 1.6 we formulated the brachistochrone problem, first as a simpleproblem in the calculus of variations and then as two versions of a controlproblem. The classical existence theorem, Theorem 5.4.18, that would ensurethat the calculus of variations version of the brachistochrone problem has asolution requires that the integral in (1.6.3) be a convex function of y′ andthat [1+ (y′)2]1/2/|y′| → ∞ as |y′| → ∞. A straightforward calculation showsthat d2(1 + (y′)2)/d2y′ > 0, so the integral is convex. The growth conditionfails, since [1 + (y′)2]1/2/|y′| → 1 as |y′| → ∞. In the absence of an existencetheorem, the problem is solved as follows. The Euler equation is solved todetermine the extremals for the problem, which are found to be cycloids. Itis then shown that there is a unique cycloid C passing through P0 and P1. Afield, in the sense of the calculus of variations, containing C is constructed,and an argument using the properties of such fields is used to show that Cminimizes. See [15], [27], [87].

The formulation of the brachistochrone problem as in (1.6.4) and (1.6.5)also suffers from the inapplicability of the existence theorem, Theorem 5.4.16,to guarantee even the existence of a relaxed optimal control. The theoremrequires |u| to be of slower growth than [(1+u2)/(y−α)]1/2, which is not thecase.

We shall show that the problem formulated in (1.6.7) subject to (1.6.6) hasa solution in relaxed controls. The maximum principle, Theorem 6.3.12, willthen be used to show that the relaxed optimal control is an ordinary controland that the optimal trajectory is a cycloid.

We change the notation in (1.6.6) and let ξ, η be the state variables and zthe control variable. We then have the following:

Problem 8.6.1. Minimize∫ tft0dt subject to:

dξ

dt= [2g(η − α)]1/2 cos z, ξ(t0) = x0 (8.6.1)

Examples 263

dη

dt= [2g(η − α)]1/2 sin z, η(t0) = y0,

π ≥ z ≥ −π and the terminal conditions ξ1 = x1, η1 = y1, tf free, wherex1 > x0, y1 > α and α = y0 − v20/2g. Thus, Ω(t) = [−π, π] for all t andτ1 = (tf , x0, y1) : tf free. We further assume that v0 6= 0; that is, theparticle has an initial velocity. Without loss of generality we may take theorigin of coordinates to be at the initial point and take the initial time to bezero. Thus, (t0, x0, y0) = (0, 0, 0). The constant α then becomes α = −v20/2g,and y − α = y + v20/2g.

We now show that the hypotheses of Theorem 4.3.5 are satisfied for Prob-lem 8.6.1, and therefore there exists an optimal relaxed pair ((x∗, y∗);µ∗) thatminimizes the transit time from (x0, y0) to (x1, y1) over all admissible pairs.

Let f denote the right-hand side of (8.6.1) and let f = (f0, f) = (1, f). Then,

f is continuous on R×R2. The set B = (0, 0, 0, tf , x1, y1) : tf free is closed.

The mapping Ω is a constant map, and so is u.s.c.i. It remains to show thatwe can restrict our attention to a compact interval I of the time variable, thatthe set of relaxed pairs ((x, y), µ) is not empty, and that the graphs of thesetrajectories are contained in a compact subset of I ×R2.

If we take z(t) = θ, a constant, then the motion determined by (8.6.1)satisfies dη/dξ = tan θ, and thus is motion along the line η = ξ tan θ. For themotion to hit the point (x1, y1), then we must have

tan θ = y1/x1. (8.6.2)

Let (x, y) denote the trajectory obtained from (8.6.1) with θ determinedby (8.6.2). For (x, y) to be admissible we must show that there exists a timet1 such that x(t1) = x1 and y(t1) = y1. We now show that this is so.

From the second equation in (8.6.1) we get that

dy/dt = [2g(y + v20/2g)]1/2 sin θ.

A straightforward calculation gives

[2g(y(t) + v20/2g)]1/2 = gt sin θ + v0.

To find x(t), the corresponding value of x at time t, we substitute this equationinto the first equation in (8.6.1) and

dx = [gt sin θ cos θ + v0 cos θ]dt.

Hencex(t) = gt2 sin θ cos θ/2 + v0 cos θt.

To find a value of t such that x(t) = x1, in the preceding equation set x(t) = x1and use (8.6.2) to get cos θ = x1/d and sin θ = y1/d, where d = [x21+y

21 ]

1/2 6= 0.We get that

(gy1/2d2)t21 +

v0dt1 − 1 = 0.


Hencet1 = −v0/d+ [(v0/d)

2 + 4(gy1/2d2)]1/2/(gy1/d2).

We have just shown that the set of ordinary admissible pairs is not emptyand hence that the set of relaxed admissible pairs is not empty. Since inftf : tfterminal time of an admissible trajectory is less than or equal to t1, it followsthat we can restrict our attention to admissible pairs defined on compactintervals [0, tf ] ⊆ [0, t1]. If we set I = [0, t1], we need only consider f onI × R

2.It is easy to show that there exists a constant K such that

|〈(ξ, η), f(τ, ξ, η)〉| ≤ K(|ξ|2 + |η|2 + 1)

for all t in I, and all (ξ, η) in R2 and all −π ≤ z ≤ π. Hence by Corollary 4.3.15

all admissible relaxed trajectories defined on intervals contained in I lie in acompact set in I ×R

2. Thus, all the hypotheses of Theorem 4.3.5 are satisfiedin Problem 8.6.1.

Hence we shall only consider optimal relaxed pairs and simply write((x, y), µ). Such a pair satisfies the Maximum Principle, Theorem 6.3.12, whichwe now use to determine the unique optimal pair.

The functions H and Hr, defined in (6.3.4) and (6.3.5), are in our problemgiven by

H = q0 + [2gη + v20 ]1/2[q1 cos z + q2 sin z]

Hr = q0 + [2gη + v20 ]1/2

[q1

∫ π

−π

cos zdµt + q2

∫ π

−π

sin zdµt

],

where µ is a discrete measure control on [−π, π]. Thus,

Hrξ = 0, Hrη = g[2gη + v20 ]−1/2

[q1

∫ π

−π

cos zdµt + q2

∫ π

−π

sin zdµt

].

Let ((x, y), µ) be a relaxed optimal pair defined on the interval [0, t1]. ByTheorem 6.3.12, there exists a constant λ0 ≥ 0 and absolutely continuousfunctions λ1 and λ2 defined on [0, t1] such that (λ0, λ1(t), λ2(t)) 6= (0, 0, 0) forall t in [0, t1] and such that

λ′1(t) = 0, λ′2(t) =− g[2gy(t) + v20 ]− 1

2 (8.6.3)

×[λ1(t)

∫ π

−π

cos zdµt + λ2(t)

∫ π

−π

sin zdµt

]

for almost all t in [0, t1]. Hence

λ1(t) = c, (8.6.4)

for some constant c. The transversality condition (6.3.24) gives

λ0 + [2gy(t1) + v20 ]1/2

[λ1(t1)

∫ π

−π

cos zdµt1 + λ2(t1)

∫ π

−π

sin zdµt1

]= 0.

Examples 265

The problem is autonomous, so by (6.3.21) and (6.3.22)

λ0 + [2gy(t) + v20 ]1/2

[λ1(t)

∫ π

−π

cos zdµt + λ2(t)

∫ π

−π

sin zdµt

]= 0 (8.6.5)

for all t in [0, t1]. Hence, for all t in [0, t1],

λ21(t) + λ22(t) 6= 0. (8.6.6)

Otherwise, there would exist a t′ in [0, t1] such that λ1(t′) = λ2(t

′) = 0. Thisin turn would imply that λ0 = 0, and so (λ0, λ1(t

′), λ2(t′)) = (0, 0, 0) which

cannot be.We next use (6.3.15) to determine the discrete measure control µ. We have,

setting λ(t) = (λ0, λ1(t), λ2(t)),

M(t, x(t), y(t), λ(t)) = sup[H(t, x(t), y(t), z, λ(t)) : |z| ≤ π]

= λ0 + [2gy(t) + v20 ]1/2[supλ1(t) cos z + λ2(t) sin z : |z| ≤ π].

The term involving the sup can be written, using (8.6.4) as

sup〈(cos z, sin z), (c, λ2(t))〉 : |z| ≤ π.

By (8.6.6), (c, λ2(t)) 6= (0, 0). Hence, the supremum is attained when z =θ(t), where θ(t) is such that (cos θ, sin θ) is the unit vector in the direction of(c, λ2(t)).

Thus,

cos θ(t) = c/(c2 + λ22(t))1/2, sin θ(t) = λ2(t)/(c

2 + λ22(t))1/2. (8.6.7)

We have shown that at each t in [0, t1] the supremum is achieved at a uniquevalue θ(t) in [−π, π]. Hence for each t in [0, t1] the discrete measure control issuch that µt is concentrated at θ(t). Thus, the relaxed optimal control µ is anordinary control θ. The relaxed optimal trajectory is an ordinary trajectory.

We assert that c > 0. If c = 0, then by (8.6.7), cos θ(t) = 0 for all t in [0, t1].Hence θ(t) = ±π/2 and sin θ(t) = ±1 for all t in [0, t1]. This would imply thatthe optimal trajectory lies on a vertical line. If c ≤ 0, then cos θ(t) < 0, for allt in [0, t1], and so dx/dt < 0 for all t. Since x1 > 0, this is impossible. Hence

cos θ(t) > 0 and − π/2 < θ(t) < π/2 a.e. (8.6.8)

It follows from (8.6.8) and the first equation in (8.6.1) that dx/dt > 0. Hencethe optimal trajectory can be given by y = y(x). This is not assumed a priorihere, in contrast to the calculus of variations treatments, where a minimizingcurve that has the form y = f(x) is sought.

Since c and cos θ(t) are greater than zero, we get from (8.6.7) that

λ2(t) = c tan θ. (8.6.9)


If we substitute (8.6.4) and (8.6.9) into (8.6.3) and (8.6.5) and use the factthat µt is concentrated at θ(t) we get that

λ′2(t) = −gc[2gy(t) + v20 ]−1/2 sec θ(t), (8.6.10)

andλ0 + c[2gy(t) + v20 ]

1/2 sec θ(t) = 0. (8.6.11)

From (8.6.11) we conclude that λ0 6= 0. Hence we may take λ0 = −1, and(8.6.11) becomes

−1 + c[2gy(t) + v20 ]1/2 sec θ(t) = 0.

Thus,[2gy(t) + v20 ]

1/2 = cos θ(t)/c, (8.6.12)

and soy(t) + v20/2g = cos2 θ(t)/2gc2.

Setting θ = u/2 and using the half angle formula gives

y + v20/2g = b(1 + cosu), b = (4gc2)−1. (8.6.13)

From (8.6.9) and the fact that θ(t) ∈ (−π/2, π/2) we get that θ =arctan(λ2/c). Since λ2 is absolutely continuous, so is θ and θ′(t) exists foralmost all t in (0, t1). From (8.6.9) we get that

λ′2(t) = [c sec2 θ(t)]dθ/dt.

Substituting (8.6.12) into (8.6.10) gives

λ′2(t) = −gc2 sec2 θ.

Hencedθ/dt = −gc.

Therefore, θ is a strictly decreasing differentiable function of t. Hence t is astrictly decreasing differentiable function of θ and

dt/dθ == (dθ/dt)−1 = −(gc)−1. (8.6.14)

From the first equation in (8.6.1) and from (8.6.12) we get that

dx/dt = cos2 θ(t)/c.

From this and from (8.6.14) we get that

dx/dθ = (dx/dt)(dt/dθ) = −(gc2)−1 cos2 θ(t)

= −(2gc2)−1(1 + cos 2θ).

Therefore,x = (−4gc2)−1(u+ sinu) + a,

Examples 267

where a is an arbitrary constant. If we now set u = −w in (8.6.13), we getthat

y + v20/2g = b(1 + cosw) (8.6.15)

x− a = b(w + sinw)

where b = (4gc2)−1 > 0. Equations (8.6.15) use the parametric equations of aninverted cycloid. They represent the locus of a fixed point on the circumferenceof a circle of radius b as the circle rolls on the lower side of the line y = v20/2g.

8.7 Flight Mechanics

In this section we consider the problem formulated in Section 1.4. Wechange the notation from that in Section 1.4 and denote the position coor-dinates by (ξ, η), the velocity components by (π1, π2), the mass variable byν, the angle coordinate by z1, and the thrust coordinate by z2. We assumethat the motion takes place in the earth’s gravitational field and that the onlyexternal force is gravity.

The equations governing planar rocket flight become:

dξ

dt= π1 dη

dt= π2 (8.7.1)

dπ1

dt= −(cz2/ν) cos z1

dπ2

dt= −g − (cβ/ν) sin z1

dν

dt= −z2

where

0 ≤ B ≤ z2 ≤ A, −π ≤ z1 ≤ π, ν > M > 0 (8.7.2)

Ω(t) = (z1, z2) : −π ≤ z1 ≤ π, 0 < B ≤ z2 ≤ A.

Thus, the constraint sets Ω(t) are constant and given as in (8.7.2) for givenconstants A,B,M . We take the initial time t0 = 0 and the other initial values(ξ0, η0, π

10 , π

20 , ν0) also to be fixed.

We first consider the hard landing problem. The terminal position (ξ1, η1)is fixed and the terminal time, velocity, and mass, (t1, π

11 , π

21 , ν1), are free with

π11 > 0, π2

1 > 0.Thus, J1 is given by (ξ1, η1) fixed and

t1 = σ0, π11 = σ3, π2

1 = σ4, ν1 = σ5; σi > 0, i = 3, 4, σ5 > M.(8.7.3)


The problem is to minimize

ν0 − ν1 = −∫ t1

0

(dν/dt)dt =

∫ t1

0

z2dt

We leave it as an exercise for the reader to show that the set of admissibletrajectories is not empty. For example, if we assume a constant directionz1 ≡ ω0 and a constant thrust of magnitude z2 ≡ T , B ≤ T ≤ A so that

π20/π

10 > (β/T ) tanω0,

(β/T )(tanω0)(ξ1 − ξ0)− (η1 − η0) ≥ 0,

(π10/g)[π

20/π

10 − (β/T ) tanω0] < ν0,

then the resulting trajectory will be admissible. The attainment of these in-equalities depends on the initial and target points, the magnitude of the maxi-mum initial velocity, and the allowable thrust limits A and B. The first and lastrequirements above depend on the state of rocket technology. It is a straight-forward calculation to show that the condition |〈x, f(t, x, z)〉| ≤ Λ(t)[|x|2 + 1]of Lemma 4.3.14 holds. Since all trajectories have a fixed initial point, it fol-lows that for any compact interval I, all the trajectories restricted to I lie ina compact set. The sets Q+(t, x) = Q+(ξ, η, π1, π2, ν) in this problem are notconvex, so we cannot use Theorem 4.4.2 to obtain a solution to our problem.

The hypotheses of Theorem 4.3.5 are fulfilled, however, so we get that therelaxed problem has an optimal relaxed admissible pair, which we denote by((x, y, p, q,m);µ). Theorem 6.3.27 is applicable to this solution. The functionsH and Hr are given by

H = ρ0z2 + ρ1π1 + ρ2π

2 − ρ3(cz2/ν) cos z1 + ρ4(−g − (cz2/ν) sin z1)− ρ5z2(8.7.4)

and

Hr = ρ0∫

Ω

z2dµt + ρ1π1 + ρ2π

2 − ρ3

∫

Ω

(cz2/ν) cos z1dµt

− ρ4g − ρ4

∫

Ω

(cz2/ν) sin z1dµt − ρ5

∫

Ω

z2dµt

where µt is a discrete control measure on Ω.There exist a constant λ0 ≤ 0 and absolutely continuous functions

λ1, . . . , λ5 defined on [0, t1] such that for t in [0, t1]

(λ0, λ1(t), λ2(t), λ3(t), λ4(t), λ5(t)) 6= 0

and

dλ1dt

= −Hrξ = 0dλ2dt

= Hrη = 0 (8.7.5)

dλ3dt

= −Hrπ1 = −λ1(t)dλ4dt

= −Hrπ2 = −λ2(t)

Examples 269

dλ5dt

= −Hrν = λ3(t)

∫

Ω

cz2ν2

cos z1dµt + λ4(t)

∫

Ω

cz2ν2

sin z1dµt.

Hence

λ1(t) = a1 λ2(t) = a2 for all t in [0, t1] (8.7.6)

λ3(t) = −a1t+ a3 λ4(t) = −a2t+ a4

for constants a1, a2, a3, a4.Since the initial point is fixed and the terminal set J∞ is given by (8.7.3),

the transversality condition (6.3.24) gives that the n+ 1 vector (−Hr(π(t1)),λ(t1)) is orthogonal to J∞ at (t1, x(t1), y(t1), p(t1), q(t1),m(t1)). From (8.7.3)we get that the vectors (dσ0, dσ3, dσ4, dσ5) of the form (1, 0, 0, 0), (0, 1, 0, 0),(0, 0, 1, 0), and (0, 0, 0, 1) are a basis to the tangent space to J1 at (t1, x(t1),y(t1), p(t1), q(t1),m(t1)). Thus,

H(π(t1)) = 0 λ3(t1) = λ4(t1) = λ5(t1) = 0. (8.7.7)

From (8.7.7) and (8.7.6) we get that

λ3(t) = a1(t1 − t) λ4(t) = a2(t1 − t). (8.7.8)

From (8.7.7), with ρ = λ(t), and (8.7.4) we get that

λ0∫

Ω

z2dµt1 + a1p(t1) + a2q(t1) = 0.

This implies that a21+a22 6= 0. For otherwise, since p(t1) > 0 and q(t1) > 0, we

would have λ0 = 0, and (λ0, λ1(t1), λ2(t1), λ3(t1), λ4(t1), λ5(t1)) = 0, whichcannot be.

We now use (6.3.15) to show that the optimal discrete measure control isan ordinary control. We let ψ = (x, y, p, q,m) denote the optimal trajectory.Then, as in Theorem 6.3.12, we get

M(t, ψ(t), λ(t) = supH(t, ψ(t), z, λ(t)) : z = (z1, z2) ∈ Ω

From (8.7.4) we see that in calculating the sup we need only calculate

sup[λ0 − λ5(t)]z2 − (cz2m

)[λ3(t) cos z1 + λ4(t) sin z1] : (z1, z2) ∈ Ω. (8.7.9)

Since c > 0,m > 0 and z2 > 0, we can first maximize with respect to z1. Thequantity λ3(t) cos z1 + λ4(t) sin z1 can be interpreted as the inner product of(λ3(t), λ4(t)) with (cos z1, sin z1). Hence it is maximized at the unique anglez1 = ω(t), where

cosω(t) = −λ3(t)/(λ23(t) + λ24(t))1/2,

sinω(t) = −λ4(t)/(λ23(t) + λ24(t))1/2.


From (8.7.8) we get

cosω(t) = −a1/(a21 + a22)1/2 sinω(t) = −a2/(a21 + a22)

1/2. (8.7.10)

Thus, the optimal direction is constant throughout the flight.From (8.7.8) and (8.7.10) we get that

λ3(t) cosω(t) + λ4(t) sinω(t) = (a21 + a22)1/2(t1 − t) (8.7.11)

substituting (8.7.11) into (8.7.9), and using the fact that ω(t) maximizes(8.7.9) over all z1 in Ω, we get that to complete the determination of thesup in (8.7.9) we need to determine

sup[λ0 − λ5(t) + (c/m(t))(a21 + a22)1/2(t1 − t)]z2 : B ≤ z2 ≤ A. (8.7.12)

Let F (t) equal the expression in square brackets in (8.7.12). The function Fis absolutely continuous and

F ′(t) = −λ′5(t)− α((cm′/m2(t))(t1 − t) + c/m(t)),

where α = (a21 + a22)1/2. From (8.7.5) with z1 = ω(t) and (8.7.11) we get

that λ′5(t) > 0. Hence F ′(t) < 0, and so F is strictly decreasing. Therefore,the supremum in (8.7.12) is attained at a unique value z2 = v(t) in [B,A].Therefore, the relaxed optimal control µ is the ordinary control (v(t), ω(t)).

Since λ0 ≤ 0 and λ5(t1) = 0, we get that F (t1) = λ0 ≤ 0. If λ0 < 0, thenF (t1) < 0. If F (0) > 0, then the supremum is attained at the unique valuez2 = v(t) on some interval [0, ts] and v(t) = B on [ts, t1]. If F (0) ≤ 0, thenz2 = v(t) = B uniquely on [t0, t1]. If λ

0 = 0, then F (t1) = 0, and so F (0) > 0for t ∈ [0, t1). Therefore, z2 = v(t) = A, uniquely on [0, t1]. To summarizewe have shown that the supremum in (8.7.9) is achieved at a unique point(z1, z2) = (ω(t), v(t)). Therefore, the relaxed optimal control µ is the ordinarycontrol (ω(t), v(t)).

We now consider the soft landing problem. The state equations are givenby (8.7.1) and the constraint set by (8.7.2). The initial conditions are fixedas in the hard landing case. The terminal conditions are now (ξ1, η1, π

11 , π

12)

fixed, with π11 ≥ 0, π1

2 ≥ 0, and t1,m1 free. Thus,

J2 = (t1, ν1) : t1 = σ1 > 0, ν1 = σ2 > u1 ∪ ξ1, η1, π11 , π

21.

We again leave the verification of the assumption that the set of admissiblerelaxed pairs is non-empty to the reader. The argument in the hard landingcase that all trajectories defined on a given compact interval lie in a compactset is independent of the terminal conditions, and so is valid here also. Thus,an optimal relaxed pair ((x, y, p, q,m);µ) exists and satisfies Theorem 6.3.27.

The functionsH andHr are as in (8.7.4), and there exists a constant λ0 andabsolutely continuous functions λ1, . . . , λ5 defined on [0, t1] such that for t in[0, t1] (λ

0, λ1(t), . . . , λ5(t)) 6= 0. The functions λi, i = 1, . . . , 5 satisfy (8.7.5).

Examples 271

Hence λ1, λ2, λ3, and λ4 are given by (8.7.6). The transversality condition(6.3.24) now gives that the two-dimensional vector (−H(π(t1)), λ5(t1)) is or-thogonal to J2 at the end point of the optimal trajectory. Thus,−H(π(t1))dt+λ5(t1)dm = 0 for all tangent vectors (dt, dm) to J2 at (t1, ψ(t1)). Hence

H(π(t1)) = 0 λ5(t1) = 0. (8.7.13)

We assert that λ3(t) and λ4(t) cannot both be identically equal to zero. Ifit were the case that λ3(t) ≡ 0 and λ4(t) ≡ 0, then a1 = 0 and a2 = 0. It wouldthen follow from (8.7.6) and (8.7.13) that λ0 = 0 and λi(t) = 0, i = 1, . . . , 5,which cannot be.

We now proceed to calculate (8.7.9) in the present case. Since λ3 and λ4are linear functions of t, they can vanish simultaneously at most at one point.We denote such a point, if it exists, as t = τ . As in the hard landing case, wefirst maximize with respect to z1. With the possible exception of a point τ atwhich λ3(τ) = λ4(τ) = 0, the maximum with respect to z1 occurs at a uniqueangle ω, where

cosω(t) = −λ3(t)/(λ3(t)2 + λ4(t)2)1/2 (8.7.14)

sinω(t) = −λ4(t)/(λ3(t)2 + λ4(t)2)1/2.

We consider two cases.

Case 1. λ3 and λ4 never vanish simultaneously. Then (λ3(t))2 + (λ4(t))

2 6= 0for all t, and thus ω is given by (8.7.14) for all t.

Having maximized (8.7.9) with respect to z1, to complete the calculationof (8.7.9) we need to calculate

sup[λ0 − λ5(t) + (c/m(t))(λ3(t)2 + λ4(t)

2)1/2]z2 : B ≤ z2 ≤ A,

where we have substituted (8.7.14) into (8.7.9). To determine the supremum,we must again determine the sign of the term in the square brackets.

Case 2. There exists τ such that λ3(τ) = λ4(τ). Then,

λ3(t) = α(t− τ) λ4(t) = β(t− τ)

for appropriate α, β. Then, for t 6= τ

cosω(t) = − α(t − τ)√α2 + β2|t− τ |

sinω(t) = − β(t− τ)√α2 + β2|t− τ |

.

Thus, as t “passes through τ” cosω(t) and sinω(t) both change sign, that is,

limt→τ±

cosω(t) = ∓ α√α2 + β2

limt→τ±

sinω(t) = ∓ β√α2 + β2

.


Thus, ω(t) jumps by ±π as we pass through τ . This is a thrust reversal.Now maximize with respect to z2. Again let F (t) denote the coefficient of

z2F (t) = λ0 − λ5(t) +

c

m(t)[λ3(t)

2 + λ4(t)2]1/2

LetQ(t) = λ3(t)

2 + λ4(t)2.

Note that Q(t) is quadratic in t.

Case 2.1. Q(t) 6= 0 for all t. That is, λ3 and λ4 do not have simultaneouszeros, that is, no thrust reversal.

dF

dt= −dλ5

dt− cQ(t)1/2

m2(t)

dm

dt+

1

2

c

m(t)Q(t)−1/2 dQ

dt.

Now,dλ5dt

=cv(t)

m2(t)Q1/2(t),

dm

dt= −v(t).

SodF

dt=

1

2

c

m(t)Q(t)−1/2 dQ

dt

Note that dQ/dt is linear. In fact,

1

2

dQ

dt= λ3(t)(−a1) + λ4(t)(−a2)

= (a21 + a22)t− a1a3 − a2a4.

Hence dQ/dt has a positive slope. Since cm(t) Q

−1/2(t) > 0 we have

signdF

dt= sign

dQ

dt.

Case 2.2. Now consider the case in which thrust reversal occurs. Again let

Q(t) = λ3(t)2 + λ4(t)

2.

Then,

Q(t) = (α2 + β2)(t− τ)2

Q(t)1/2 = (α2 + β2)1/2|t− τ |.

And we now have

F (t) = λ0 − λ5(t) +c

m(t)Q(t)1/2

= λ0 − λ5(t) + (α2 + β2)1/2|t− τ |(

c

m(t)

).

Examples 273

Therefore,

dF

dt= −(α2 + β2)1/2

c

m(t), 0 ≤ t < τ

dF

dt= (α2 + β2)1/2

c

m(t), τ < t ≤ t1.

We are now in a position to completely characterize the control (ω(t), v(t)) aswe did in the soft landing case. We note that F is decreasing for t < τ andincreasing for t > τ .

8.8 An Optimal Harvesting Problem

In this section we will apply the maximum principle to the example pre-sented in Section 1.7. In dealing with this example and the one in the nextsection we employ Corollary 4.5.1 and directly proceed to employ the maxi-mum principle. A population model of McKendric type [40], [68] with crowdingeffect is given by

∂p(r, t)

∂t+∂p(r, t)

∂r= −µ(r)p(r, t)− f(N(t))p(r, t) − u(t)p(r, t) (8.8.1)

p(r, 0) = p0(r)

p(0, t) = β

∫ ∞

0

k(r)p(r, t)dr

N(t) =

∫ ∞

0

p(r, t)dr

We consider the problem of maximizing the harvest

J(u) =

∫ T

0

u(t)N(t)dt, (8.8.2)

0 ≤ u(·) ≤M

Assumption 8.8.1. We need some technical assumptions.

(i) The functions k, p0 ≥ 0 are continuous. Further, k(r) = p0(r) = 0 ifr ≥ rm.

(ii) µmax ≥ µ ≥ ǫ0 > 0,∫∞

0 exp(−∫ r

0 µ(s)ds)ds <∞.

(iii) f ∈ C1((0,∞)), f ′ ≥ 0, f(0) = 0, f 6≡ 0.


Using the method of characteristics

p(r, t) =

p0(r − t) exp(−∫ r

r−tµ(ρ)dρ

)

× exp(−∫ t

0 [f(N(s)) + u(s)]ds), r > t

p(0, t− r) exp(−∫ r

0µ(ρ)dρ

)

× exp(−∫ t

t−r[f(N(s)) + u(s)]ds), r < t

(8.8.3)We have

p(0, t) = β

∫ t

0

k(r)p(0, t− r) exp

(−∫ r

0

µ(ρ)dρ

)(8.8.4)

× exp

(−∫ t

t−r

[f(N(s)) + u(s)]ds

)dr

+ β

∫ ∞

t

k(r)p0(r − t, 0) exp

(−∫ r

r−t

µ(ρ)dρ

)

× exp

(−∫ t

0

[f(N(s)) + u(s)]ds

)dr

Let

w(t) = p(0, t) exp

(∫ t

0

[f(N(s)) + u(s)]ds

)(8.8.5)

Then,

w(t) = β

∫ t

0

k(r)w(t − r) exp

(−∫ r

0

µ(ρ)dρ

)dr (8.8.6)

+ β

∫ ∞

t

k(r)p0(r − t) exp

(−∫ r

r−t

µ(ρ)dρ

)dr,

N(t) =

∫ ∞

0

p(r, t)dr (8.8.7)

= exp

(−∫ t

0

[f(N(s)) + u(s)]ds

)[∫ t

0

w(t− r) exp

(−∫ r

0

µ(ρ)dρ

)dr

+

∫ ∞

t

p0(r − t, 0) exp

(−∫ r

r−t

µ(ρ)dρ

)dr

]

Let

η(t) =

∫ t

0

w(t − r) exp

(−∫ t

0

µ(ρ)dρ

)dr (8.8.8)

+

∫ ∞

t

p0(r − t) exp

(−∫ r

r−t

µ(ρ)dρ

)dr

Examples 275

Then

N(t) = η(t) exp

(−∫ t

0

[f(N(s)) + u(s)]ds

)

N ′(t) =

(η′(t)

η(t)− [f(N(t)) + u(t)]

)N(t) (8.8.9)

N(0) =

∫ ∞

0

p0(r)dr ≡ α0. (8.8.10)

Integrating by parts we observe that

∫ T

0

uNdt = α0 −N(T ) +

∫ T

0

[η′(t)

η(t)− f(N(t))

]N(t)dt (8.8.11)

Thus, we have to choose u to maximize the harvest given in (8.8.11) underthe conditions (8.8.9) and (8.8.10). Suppose (N , u) is optimal. The adjointvariable λ satisfies

−dλdt

=

[η′(t)

η(t)− f ′(N(t))N(t)− f(N(t))− u(t)

]λ (8.8.12)

+η′(t)

η(t)− f ′(N(t)N(t)− f(N(t))

λ(T ) = −1

The maximum principle reads

−λ(t)N(t)u(t) ≥ −λ(t)N(t)v ∀ v ∈ [0,M ]

Thus,

u(t) =

M, λ(t) < 0

0, λ(t) > 0

Letting

Φ(t) =η′(t)

η(t)− f ′(N(t))N (t)− f(N(t)) (8.8.13)

we have

λ(t) = −1 +

∫ T

t

u(s) exp

(∫ s

t

[Φ(ρ)− u(ρ)]dρ

)ds (8.8.14)

Note that λ(T ) = −1. Thus, we solve for t1 such that

0 = λ(t1) = −1 +

∫ T

t1

M exp

(∫ s

t1

[Φ(ρ)−M ]dρ

)ds

or ∫ T

t1

exp

(∫ s

t1

[Φ(ρ)−M ]dρ

)ds =

1

M(8.8.15)


Thus, for t ≤ t1 we set u ≡ 0 and u ≡ M for t ≥ t1. Note that we can solvefor N(t) explicitly from (8.8.9) setting u ≡ 0 for t ≤ t1 and u ≡M for t ≥ t1.Thus, Φ is known explicitly using (8.8.13). Thus, (8.8.15) is an equation fort1.

In conclusion we see that the population should be allowed to build up untiltime t1, and then harvest afterward with maximal effort, that is, u(t) ≡ M .(Recall 0 ≤ u(t) ≤M .)

8.9 Rotating Antenna Example

In this section we follow [47] and discuss in a sketchy manner the designof a control policy for the antenna problem presented in Example 1.5.2. Thisproblem is discussed in complete detail in [47].

For the antenna problem in question we recall the control torque T isconstrained in magnitude by requiring

|T | ≤ k, k > 0.

Our initial state at time t0 is (θ0, θ0) and our desired terminal state at t1 > t0is (θ1, 0). It is convenient to define the following quantities:

x1 = Ik−1(θ − θ1), (8.9.1)

x2 = Ik−1θ, (8.9.2)

ξ = Ik−1(θ0 − θ1), (8.9.3)

σ = Ik−1θ0, (8.9.4)

u = Tk−1. (8.9.5)

The equation of motion under an applied torque T is given by

Iθ + βθ = T,

where β is a damping factor and I is the moment of inertia of the systemabout the vertical axis. We make the simplifying assumption that friction isnegligible and the equation of motion is then

Iθ = T (8.9.6)

In terms of the x1, x2 variables, Eq. (8.9.6) becomes

x1 = x2, (8.9.7)

x2 = u,

Examples 277

and we have x1(t0) = ξ, x2(t0) = σ. In the plane determined by the coordi-nates (x1, x2) our objective will be achieved by passing to any point in theset

Ω =

(±2π

kn, 0

): n = 0, 1, 2, . . .

,

where|u| ≤ 1

and the performance index

J =

∫ t1

t0

(λ1 + λ2x22 + λ3|u|)dt,

where λ1 = γ1, λ2 = γ2k2I−2, and λ3 = kγ3. We can, without loss of general-

ity, assume that λ1 + λ2 + λ3 = 1, since multiplying J by a positive constantdoes not affect our problem.

We can apply Theorem 6.3.12 to deal with this problem. Let (ψ1, ψ2) bethe adjoint variable. Then, consider the Hamiltonian given by

H = λ0(λ1 + λ2x22(t) + λ3|u(t)|)− ψ1(t)x2(t)− ψ2(t)u(t), λ0 ≥ 0.

The adjoint equations are given by

ψ′1 = 0

ψ′2 = 2λ0λ2x2 − ψ1.

If λ0 = 0, thenH = −ψ1(t0)x2(t)− ψ2(t)u(t) a.e.

Thus,u(t) = sign ψ2(t), a.e. t ∈ [t0, t1) ∩ t : ψ2(t) 6= 0.

If λ0 = 0,ψ2(t) = ψ2(t0)− ψ1(t0)(t− t0).

If ψ1(t0) = 0 also, ψ2(t) ≡ ψ2(t0) must be nonzero and u is either identically 1or −1. If ψ1(t0) 6= 0, ψ2 can vanish in at most one point, and u has to changesign across this point, that is, from −1 to 1 or 1 to −1. However, this can beeliminated.

Next, consider λ0 > 0, in which case λ0 may be taken to be 1. In this case

ψ′2 = 2λ2x2 − ψ1(t0),

x1(t) = ξ1 +

∫ t

t0

x2(τ)dτ.

Thus,ψ2(t) = ψ2(t0) + 2λ2(x1(t)− ξ)− ψ1(t0)(t− t0),


and the Hamiltonian becomes

H(t) = λ1 + λ2x22 + λ3|u| − ψ1(t0)x2(t)− (ψ2(t0)

+ 2λ2(x1(t)− ξ)− ψ(t0)(t− t0))u(t),

and H(t) is minimized ifλ3|u| − ψ2(t)u(t)

is minimized. Thus, we have

u(t) =

1 if ψ2(t) > λ3,

0 if |ψ2(t)| < λ3,

−1 if ψ2(t) < −λ3.(8.9.8)

If ψ2(t) = λ3, then 0 ≤ u(t) ≤ 1, and if ψ2(t) = −λ3, then −1 ≤ u(t) ≤ 0.We note that

ψ2(t) = 2λ2x2 − ψ1(t0).

Thus,ψ2(t) = 2λ2x2(t) = 2λ2u(t). (8.9.9)

Thus,

u(t) = 1 and ψ2(t) > 0 if ψ2(t) > λ3 (8.9.10)

u(t) = 0 and ψ2(t) = 0 if |ψ2(t)| < λ3

u(t) = −1 and ψ2(t) < 0 if ψ2(t) < −λ3

If ψ2(t) ≡ λ3 on an interval, then ψ2 = ψ2 = 0 on the same interval. Likewiseif ψ2(t) ≡ −λ3 on an interval. Hence, we have

ψ(t) = 0 if |ψ2(t)| ≤ λ3, (8.9.11)

and from (8.9.9) and (8.9.11) it follows that u ≡ 0 on any interval on which|ψ2(t)| = λ3. We observe from (8.9.10) and (8.9.11) that ψ2 is linear on anyinterval [a, b] on which |ψ2(t)| ≤ λ3. Since ψ2 is continuous it follows thateither ψ2 is constant on [a, b] or ψ2 is constant and nonzero on [a, b]. The fourgeneric possibilities are illustrated in Fig. 8.2.

If ψ2(t0) > λ3 the possible ψ2-trajectories are as in Fig. 8.3, and thepossible sequence of values assumed by u are

(a) +1 (c) +1, 0,+1

(b) +1, 0 (d) +1, 0,−1

If ψ2(t0) < −λ3 the sequence of values assumed by u are

(a) −1 (c) −1, 0,−1

(b) −1, 0 (d) −1, 0, 1

Examples 279

FIGURE 8.2

FIGURE 8.3


FIGURE 8.4

If |ψ2(t0)| ≤ λ3, the possible sequence of values assumed by u are

(a) 0 (c) 0,−1

(b) 0, 1If λ3 = 0, we immediately see from (8.9.8) that u takes values 1 or −1, exceptat a single point which we can neglect. That is, the sequence of control valuesassumed are

(a) −1, 1

(b) 1,−1When λ3 = 0 the trajectory pattern and control synthesis could be deduced

from Fig. 8.4. Note that as λ2 → 0 the lines x2 =√λ1/λ2 and x2 = −

√λ1/λ2

pass to infinity and the trajectory pattern gets simple.If λ2 and λ3 are positive the trajectory pattern and control synthesis are

as in Fig. 8.5.Finally, we analyze the cost calculation relevant to the target (0, 0) as a

function of the present state of the system. That is, we shall determine afunction C defined on the (x1, x2)-plane such that C(x1, x2) is the value ofJ computed for an optimal trajectory starting at (x1, x2) and passing to theorigin.

Examples 281

FIGURE 8.5

Let η(0) denote the curve in the (x1, x2)-plane defined by the formula

x1 = −1

2x22, x2 ≥ 0,

x1 =1

2x22, x2 ≤ 0.

For (µ, ν) on η(0), we have

C(µ, ν) =

∫ t1

0

(λ1+λ2x2(τ)2+λ3|u(τ)|)dτ = λ1+λ3t1+λ2

∫ t1

0

(ν−sign(ν)τ)2dτ,

and t1 = |ν|. Hence,

C(µ, ν) = (λ1 + λ3)|ν|+λ23

|ν|3.

Let S1 be the region in the (x1, x2)-plane to the left of η(0), on or above thecurve ψ, and on or below the line x2 =

√λ1/λ2. Then,

C(µ, ν) = (λ1 + λ3)ν +λ23ν3 +

∫ τ2

0

(λ1 + λ2ν2)dτ

= (λ1 + λ3)ν +λ23ν3 + (λ1 + λ2ν

2)τ2,


where τ2 is the time when the trajectory reached η(0). τ2 is easily computedto be given by the formula

τ2 =|µ|ν

− 1

2ν.

We refer the reader to [47] for cost calculation relevant to all the terminalstates Ω =

(± 2π

k n, 0): n = 0, 1, 2, . . .

.

Chapter 9

Systems Governed by

Integrodifferential Systems

9.1 Introduction

In this chapter we treat problems governed by integrodifferential systems[65], [67]. In Section 9.2 we present a version of the problems of interest anddiscuss existence. In Section 9.3 we specialize to systems linear in the state.In the next section we discuss linear systems and the bang-bang principle. InSection 9.5 we present a second version of the problem of interest. In Sec-tion 9.6 we discuss so-called linear plant quadratic cost criterion. We close thechapter by presenting minimum principle for a constrained problem governedby an integral equation.

9.2 Problem Statement

We consider a process governed by the equation

φ(t) = F (t) +

∫ t

0

L(t, φ(s), u(s), s)ds (9.2.1)

under the conditions

W (φ(t1)) +

∫ t1

0

M(φ(t), u(t), t)dt = 0 (9.2.2)

u(t) ∈ Ω(t, φ(t)). (9.2.3)

It is required that the cost

∫ t1

0

F 0(φ(t), u(t), t)dt (9.2.4)

be minimum.

283


Assumption 9.2.1. (i) Let X,U be open subsets of Rn and Rm, respec-

tively, and I0 an open interval of R.

(ii) The function L : I0×X×U×I0 → R is such that L(t, x, u, s) is continuousin (t, x, u) for each fixed s, and measurable in s for fixed (t, x, u).

(iii) The function M(x, u, t) from X×U× I0 into R is differentiable in x andcontinuous in (x, u) for fixed t, and measurable in t for fixed (x, u). Weassume the same for the function (x, u, t) 7→ F 0(x, u, t).

(iv) For each compact subset Γ ⊂ X × U ∃ Λ ∈ L2(R) such that|∂xL(t, x, u, s)| + |L(t, x, u, s)| ≤ Λ(s) for almost every s ∈ I0 and|M1(x, u, t)| + |M(x, u, t)| + |F 0

1 (x, u, t)| + |F 0(x, u, t)| ≤ Λ(t), for al-most every t, where ∂xL, M1, F

01 denote matrices of partial derivatives

with respect to x. Here and in what follows | · | denotes the euclideannorm of the vector or matrix in question. For existence we only needLipschitz condition in the x-variable for L, M , and F 0.

(v) The function F is absolutely continuous with F (0) ∈ X. We have im-posed this condition for the purpose of obtaining necessary conditions.For existence we only need F to be continuous and F (0) ∈ X.

(vi) The function W : Rn → R is continuously differentiable. For existencewe only need it to be lower semi-continuous.

(vii) The set valued map Ω(t, x), (t, x) ∈ I0 × X is u.s.c.i. on I × X, and foreach (t, x) ∈ I0 × X, Ω(t, x) is compact.

Notation 9.2.2. We denote by AR the set of all admissible pairs (φ, ν). Thatis, the set of all pairs (φ, ν) that satisfy (9.2.1), (9.2.2), and (9.2.3). Here, ν isa relaxed control.

Theorem 9.2.3. Assume AR 6= ∅. Then, with Assumption 9.2.1 in forcethere exists a relaxed pair (φ, ν) satisfying Eqs. (9.2.1) to (9.2.3) and givingthe cost in (9.2.4) its infimum value.

Exercise 9.2.4. Prove Theorem 9.2.3.

Following Theorem 5.4.18 we also have the following:

Theorem 9.2.5. In Assumption 9.2.1 we replace (vii) by the assumption thatΩ is u.s.c. on I0 × X and

⋂

δ>0

clΩ(Nxδ(t0, x0)) ⊆ Ω(t0, x0) ∀ (t0, x0) ∈ I0 × X,

F 0(t, x, u) ≥ Φ(u), where Φ is continuous and |u|−1Φ(u) → ∞ as |u| → ∞,u ∈ U. Then, problem (9.2.1)–(9.2.4) has a relaxed solution.

Systems Governed by Integrodifferential Systems 285


Consider a plant governed by the system

φ(t) = F (t) +

∫ t

0

A(t, s)φ(s)ds +

∫ t

0

B(t, s)h(s, u(s))ds, (9.3.1)

with the restrictions

W (φ(t1)) +

∫ t1

0

[M(s)φ(s) + k(s, u(s))]ds = 0, (9.3.2)

andu(t) ∈ Ω(t). (9.3.3)

The cost to be minimized is given by

∫ t1

0

[A0(s)φ(s) + h0(s, u(s))]ds. (9.3.4)

We assume that Assumption 9.2.1 remains in force. To apply Assump-tion 9.2.1 to the problem (9.3.1)–(9.3.4) we identify L(t, φ(s), u(s), s) of (9.2.1)with A(t, s)φ(s) + B(t, s)h(s, u(s)) of (9.3.1). Similarly M(φ(t), u(t), t) of(9.2.2) is identified with M(s)φ(s) + k(s, u(s)) of (9.3.2). We also assumethat A(t, s) and B(t, s), M(t) satisfy (ii), (iii), and (iv) of Assumption 9.2.1.We also remark that A0(s)+h0(s, u) in (9.3.4) satisfies the same condition asF 0 that is stated in (iv) of Assumption 9.2.1. We enforce Assumption 9.2.1in Sections 9.4 and 9.7. In (9.3.1)–(9.3.4), in addition to Assumption 9.2.1 weassume that

|A0(s)|+ |h0(s, u(s))| ≤ Λ(s)

where Λ is as in (iv) of Assumption 9.2.1. Note that, here, the set-valued mapΩ(t) is independent of x.

Theorem 9.3.1. Assume Assumption 9.2.1 is in force. We also assume thath(·, ·), h0(·, ·), and k(·, ·) are continuous in both of their arguments. Suppose(φ, ν) is a relaxed admissible pair for (9.3.1)–(9.3.4). Then, there exist anordinary control u such that (φ, u) is also admissible giving the same value tothe cost.

Proof. From (9.3.4) we can define

ψ(t) =

∫ t

0

[A0(s)φ(s) + h0(s, u(s))]ds,

and replace the cost by ψ(t1). Thus, it is no loss of generality to assume thatthe cost is in the form g(φ(t1)) in the proof, and we do.


Set

ζ(t) =W (φ(t1)) +

∫ t

0

[M(s)φ(s) + k(s, νs)]ds,

ψ(t) =

∫ t

0

[A0(s)φ(s) + h0(s, νs)]ds.

Now, corresponding to the relaxed admissible pair (φ, ν) we set

φ(t)− F (t)−∫ t

0

A(t, s)φ(s)ds =

n+1∑

i=1

∫ t

0

B(t, s)λi(s)h(s, ui(s))ds

ψ(t)−∫ t

0

A0(s)φ(s)ds =

n+1∑

i=1

∫ t

0

λi(s)h0(s, ui(s))ds (9.3.5)

ζ(t)−W (φ(t1))−∫ t

0

M(s)φ(s)ds =

n+1∑

i=1

∫ t

0

λi(s)k(s, ui(s))ds

where λi ≥ 0,∑n+1

i=1 λi = 1. Our objective is to show that in (9.3.5), an

ordinary control u exists which can be used in place of νs =∑n+1

i=1 λi(s)δui(s).

Define a mapping T from L2([0, t]) to Rn+2 as follows

Tρ =

∫ t

0

B(t, s) 0 0

0 1 00 0 1

ρ(s)ds, ρ ∈ L2([0, t]). (9.3.6)

The mapping T is continuous from the strong topology of L2([0, t]) toR

n+2. Letting a ∈ Rn+2 be the left-hand side of (9.3.6) T−1(a) is closed and

convex in L2([0, t]). Let Σ denote the intersection of T−1(a) and Θ = ξ |ξ(s) ∈ co h(s,Ω(s)), h = (h, h0, k)

T a.e.. Let ξ0 be an extreme point of Σ.

Then, ξ0 has the form ξ0(s) =∑n+1

i=1 λi(s)h(s, ui(s)), ui(s) ∈ Ω(s). We canthen prove that on no measurable subset E of [0, t] with positive measure canwe have 0 < ǫ ≤ λi(s) ≤ 1− ǫ for some i in the set i = 1, . . . , n+1 and someǫ > 0. The argument for this has been presented in the proof of Theorem 4.7.7,and the proof is complete.

Corollary 9.3.2. With Assumption 9.2.1 in force, the problem of minimizingthe cost (9.3.4) under the conditions (9.3.1) to (9.3.3) has a solution wherethe optimal control is ordinary.


9.4 Linear Systems/The Bang-Bang Principle

For linear systems we modify (9.3.1) to (9.3.4) and consider

φ(t) = F (t) +

∫ t

0

A(t, s)φ(s)ds +

∫ t

0

B(t, s)u(s)ds, (9.4.1)

W (φ(t1)) +

∫ t1

0

M(s)φ(s)ds+

∫ t1

0

k(s)u(s)ds = 0, (9.4.2)

u(t) ∈ Ω(t). (9.4.3)

The cost to be minimized is given by

∫ t1

0

[A0(s)φ(s) + h0(s)u(s)]ds. (9.4.4)

The control constraint set is independent of x and compact for each t. Wemake the additional assumptions

|A(t+ h, s)−A(t, s)| ≤ Λ(s)|h|,|B(t+ h, s)−B(t, s)| ≤ Λ(s)|h|,

where Λ is as in (iv) of Assumption 9.2.1. We also assume that the set ofadmissible relaxed pairs is not empty. We have the following.

Remark 9.4.1. The problem of minimizing the cost in (9.4.4) under theconditions (9.4.1) to (9.4.3) with Ω(t) compact for each t has a solution withthe control being ordinary. Note that a minimizing sequence of admissibletrajectories for this problem constitutes an equi-continuous family.

Exercise 9.4.2. Verify Remark 9.4.1.

In (9.4.3) we replace Ω by C, a compact convex polyhedron in Rm. Then

the above problem has an optimal control taking values in the vertices of C;that is, the control is bang-bang.

9.5 Systems Governed by Integrodifferential Systems

Consider now the problem

min

∫ t0

0

f0(φ(t), u(t), t)dt (9.5.1)


subject to:

d

dtφi(t) = f i(t, φ(t), u(t)) +

∫ t

0

gi(t, s, φ(s), u(s))ds (9.5.2)

u(t) ∈ Ω(t, φ(t)) (9.5.3)

T (φ(0), φ(t1)) = 0. (9.5.4)

For assumptions on f0, f1, . . . , fn refer to Assumption 6.3.1. For assumptionson gi, i = 1, . . . , n refer to Section 2.7. T is continuously differentiable.

Remark 9.5.1. Under Assumption 9.2.1 and additional assumptions onthe kernel functions, we can rewrite (9.2.1) to (9.2.4) to fit the form of(9.5.1) to (9.5.4). We may introduce a new state variable ψ such thatψ′(t) = M(φ(t), u(t), t), and consider the constraint [W (φ(t1)) + ψ(t1)]

2 +[φ(0)− F (0)]2 = 0. Finally we differentiate both sides of Eq. (9.2.1).

Necessary conditions for systems governed by integrodifferential systemsare presented in Section 11.5. The above system is a special version of theproblem (11.2.1) to (11.2.8).

9.6 Linear Plant Quadratic Cost Criterion

Consider a process where the cost is given by

∫ t1

0

〈φ(t), X(t)φ(t)〉dt +∫ t1

0

〈u(t), R(t)u(t)〉dt, (9.6.1)

where the matrices X and R are symmetric, X is nonnegative, R is positivedefinite, and

φ′(t) = A0(t)φ(t) +B0u(t) +

∫ t

0

[A1(t, s)φ(s) +B1(t, s)u(s)]ds, (9.6.2)

u(t) ∈ Rm, (9.6.3)

T (φ(t1)) = 0. (9.6.4)

The data of the problem obey the same assumptions as in (9.5.1) to (9.5.4).Suppose (φ0, u0) is optimal. Then from Theorem 11.5.3 and Remark 11.5.4

we must have

− λ0〈u0(t), R(t)u0(t)〉+Φ(t) ·B0(t)u0(t) (9.6.5)

+

∫ t1

t

Φ(s) ·B1(t, s)u0(s)ds ≥ −λ0〈u(t), R(t)u(t)〉 +Φ(t) · B0(t)u(t)


+

∫ t1

t

Φ(s) ·B1(t, s)u(s)ds,

where

Φ(t)−∫ t1

t

Φ(s)A0(s)ds+

∫ t1

t

∫ t1

s

Φ(τ)A1(τ, s)dτ ds (9.6.6)

+ 2λ0

∫ t1

t

φ0(s)X(s)ds = Φ(t−1 )

Integrating both sides of the inequality from t = 0 to t = t1 and switchingorder of integration we obtain the inequality

− λ0〈u0(t), R(t)u0(t)〉 +Φ(t)

[B0(t) +

∫ t

0

B1(s, t)ds

]u0(t) (9.6.7)

≥ −λ0〈u(t), R(t)u(t)〉+Φ(t)

[B0(t) +

∫ t

0

B1(s, t)ds

]u(t)

From (9.6.7) we can eliminate the possibility λ0 = 0. Thus, we may takeλ0 = 1 in what follows, and

uT0 (t) =1

2Φ(t)

[B0(t) +

∫ t

0

B1(s, t)ds

]R−1(t) (9.6.8)

From (9.6.6) we see the dependence of Φ on Φ(t−1 ), and hence of u0(t) on Φ(t−1 ).We have to determine Φ(t−1 ) consistent with (iii) and (iv) of Theorem 11.5.3.In this case Φ(t−1 ) = c∇T (φ0(t1)), c a constant. If we still have undeterminedparameters we determine φ0(t) using u0 as given by (9.6.8) and insist onT (φ0(t1)) = 0.

Exercise 9.6.1. Under Assumption 9.2.1, and additional regularity assump-tions as needed, use Theorem 11.5.3 to obtain a set of necessary conditions ofoptimality for (9.2.1) to (9.2.4).

9.7 A Minimum Principle

In Exercise 9.6.1 the reader was asked to get necessary optimality condi-tions for (9.2.1) to (9.2.4) using Theorem 11.5.3 under Assumption 9.2.1 andadditional regularity assumptions. In this section we will relax the assump-tions on the data and obtain necessary optimality conditions.

We consider the system governed by

φ(t) = F (t) +

∫ t

0

L(t, φ(s), u(s), s)ds (9.7.1)


W (φ(t1)) +

∫ t1

0

M(φ(t), u(t), t)dt = 0, (9.7.2)

u(t) ∈ Ω(t), (9.7.3)

and the cost to be minimized is given by

g(φ(t1)). (9.7.4)

To obtain necessary conditions at any optimal control ν0 we define thefunctional

FK(ν) = g(φ(t1))+ǫ‖ν−ν0‖L+K[W (φ(t1)) +

∫ t1

0

M(φ(t), νt, t)dt

]2(9.7.5)

under the restriction

φ(t) = F (t) +

∫ t

0

L(t, φ(s), νs, s)ds, (9.7.6)

and proceed as in Section 7.6.

Assumption 9.7.1. (i) F is continuous.

(ii) W and g are continuously differentiable.

(iii) The functions L and M obey the assumptions in (i) through (iv), inAssumption 9.2.1.

(iv) |L(t+ h, x, u, s)− L(t, x, u, s)| ≤ Λ(s)|h|,

|Lx(t, x, u, s)− Lx(t, x′, u, s)| ≤ Λ(s)|x− x′|,

where Λ is as in Assumption 9.2.1.

(v) Ω(t) is contained in a fixed compact set Ω.

Suppose the problem of minimizing g(φ(t1)) subject to (9.7.1) to (9.7.3)has a solution (φ0, ν0), where ν0 is a relaxed control. We assume that givenany relaxed control ν (9.7.1) has a unique solution. Without loss of generalitywe assume that g(φ0(t1)) = 0.

Given 0 < ǫ < 1 we can verify that there existsK(ǫ) such that FK(ǫ)(ν) > 0for all ν such that ‖ν − ν0‖L = ǫ. Let

Ar = ν : ‖ν − ν0‖L ≤ ǫ.

Suppose νǫ minimizes FK(ǫ)(ν) in Ar. We note that ‖νǫ − ν0‖L < ǫ. Corre-sponding to νǫ we have the trajectory φǫ such that

φǫ(t) = F (t) +

∫ t

0

L(t, φǫ(s), νǫs, s)ds.


We note that φǫ → φ0 uniformly as ǫ→ 0. For 0 < θ < 1 let φǫθ be such that

φǫθ(t) = F (t) +

∫ t

0

L(t, φǫθ(s), νǫs + θ(ν − νǫ)s, s)ds.

We note that φǫθ0<ǫ<ǫ00<θ<1

is a uniformly bounded equicontinuous family. We

can also verify that φǫθ → φǫ as θ → 0 and that (φǫθ(t)− φǫ(t))/θ0<θ<1 isuniformly bounded and converges to δφǫ(t) pointwise as θ → 0+, where δφǫsatisfies the equation

δφǫ(t) =

∫ t

0

Lx(t, φǫ(s), νǫs, s)δφǫ(s)ds+

∫ t

0

L(t, φǫ(s), (ν − νǫ)s, s)ds.

We can also verify that δφǫ0<ǫ<ǫ0 is uniformly bounded and that δφǫ →δφ0 uniformly, where δφ0 satisfies the integral equation

δφ0(t) =

∫ t

0

Lx(t, φ0(s), ν0s, s)δφ0(s)ds+

∫ t

0

L(t, φ0(s), (ν − ν0)s, s)ds.

We can also verify that

limθ→0+

[FK(ǫ)(νǫ + θ(ν − νǫ))− FK(ǫ)(νǫ)]/θ ≥ 0

leads to the inequality

g′(φǫ(t1))δφǫ(t1) + ǫρ′(0+) + 2K(ǫ)cǫ

W ′(φǫ(t1))δφǫ(t1)

+

∫ t1

0

Mx(φǫ(t), νǫt, t)δφǫ(t)dt+

∫ t1

0

M(φǫ(t), (ν − νǫ)t, t)dt

≥ 0

where

cǫ =W (φǫ(t1)) +

∫ t1

0

M(φǫ(t), νǫt, t)dt

and ρ′(0+) is the right derivative of ρ(θ) = ‖νǫ + θ(ν − νǫ)− ν0‖L.Dividing the above inequality M(ǫ) = 1 + 2K(ǫ)cǫ and letting ǫ → 0+

through an appropriate subsequence we obtain

λ0g′(φ0(t1))δφ0(t1) + λ

W ′(φ0(t1))δφ0(t1) +

∫ t1

0

Mx(φ0(t), ν0t, t)δφ0(t)dt

+

∫ t1

0

M(φ0(t), (ν − ν0)t, t)dt

≥ 0 (9.7.7)

where λ0 ≥ 0 and λ0 + |λ| = 1, and δφ0(t) satisfies the integral equation

δφ0(t) =

∫ t

0

Lx(t, φ0(s), ν0s, s)δφ0(s)ds


+

∫ t

0

L(t, φ0(s), (ν − ν0)s, s)ds (9.7.8)

Let R(t, s) be the resolvent kernel for the integral equation (9.7.8). Let

q(t) =

∫ t

0

L(t, φ0(s), (ν − ν0)s, s)ds. (9.7.9)

Then,

δφ0(t) = q(t) +

∫ t

0

R(t, s)q(s)ds.

Now we substitute for δφ0 in (9.7.7) to obtain

λ0g′(φ0(t1))q(t1) + λ0∫ t1

0

g′(φ0(t1))R(t1, s)q(s)ds

+ λW ′(φ0(t1))q(t1) + λ

∫ t1

0

W ′(φ0(t1))R(t1, s)q(s)ds

+ λ

∫ t1

0

Mx(φ0(t), ν0t, t)

[q(t) +

∫ t

0

R(t, s)q(s)ds

]dt

+ λ

∫ t1

0

M(φ0(t), (ν − ν0)t, t)dt

≥ 0.

We rewrite this inequality as

λ0g′(φ0(t1))q(t1) +

∫ t1

0

λ0g′(φ0(t1))R(t1, t)

+ λW ′(φ0(t1))R(t1, t) + λMx(φ0(t), ν0t, t)

+ λ

∫ t1

t

Mx(φ0(s), ν0s, s)R(s, t)

q(t)dt

+ λW ′(φ0(t1))q(t1) + λ

∫ t1

0

M(φ0(t), (ν − ν0)t, t)dt ≥ 0.

We rewrite this last inequality. However, we first set

ψ(t) = λ0g′(φ0(t1))R(t1, t)

+

W ′(φ0(t1))R(t1, t) +Mx(φ0(t), ν0t, t)

+

∫ t1

t

Mx(φ0(s), ν0s, s)R(s, t)ds

,


a row vector. Now the last inequality can be rewritten as

λ0g′(φ0(t1))q(t1) + λW ′(φ0(t1))q(t1)

+ λ

∫ t1

0

M(φ0(t), (ν − ν0)t, t)dt+

∫ t1

0

ψ(t)q(t)dt ≥ 0. (9.7.10)

Recalling the definition of q(t) (9.7.9) we can rerite (9.7.10) as

∫ t1

0

[λ0g′(φ0(t1)) + λW ′(φ0(t1))]L(t1, φ0(t), νt, t)dt

+

∫ t1

0

[∫ t1

t

ψ(s)L(s, φ0(t), νt, t)ds

]dt+ λ

∫ t1

0

M(φ0(t), νt, t)dt

≥∫ t1

0

[λ0g′(φ0(t1)) + λW ′(φ0(t1))

]L(t1, φ0(t), ν0t, t)dt

+

∫ t1

0

[∫ t1

t

ψ(s)L(s, φ0(t), ν0t, t)ds

]dt+ λ

∫ t1

0

M(φ0(t), ν0t, t)dt.

(9.7.11)

We use (9.7.11) to state our theorem next.

Theorem 9.7.2. Under Assumption 9.7.1 let (φ0, ν0) be optimal for the prob-lem of minimizing the cost (9.7.4) under the conditions (9.7.1) to (9.7.3). Thenthe following conditions are met: There exist multipliers λ0 ≥ 0, λ, and ψ suchthat

(i) λ0 + |λ| = 1

(ii) ψ(t) = λ0g′(φ0(t1))+R(t1, t)+λ

W ′(φ0(t1))R(t1, t)+Mx(φ0(t), ν0t, t)+

∫ t1t Mx(φ0(s), ν0s, s)R(s, t)ds

(iii) [λ0g′(φ0(t1)) + λW ′(φ0(t1))]L(t1, φ0(t), νt, t) + λM(φ0(t), νt, t)

+∫ t1tψ(s)L(s, φ0(t), νt, t)ds ≥ [λ0g′(φ0(t1))+λW

′(φ0(t1))]L(t1, φ0(t), ν0t, t)

+λM(φ0(t), ν0t, t) +∫ t1t ψ(s)L(s, φ0(t), ν0t, t)ds a.e.

Remark 9.7.3. Condition (iii) of Theorem 9.7.2 follows from (9.7.11).

Chapter 10

Hereditary Systems

10.1 Introduction

In this chapter we consider processes where the evolution of the statedepends upon the present and past history of the state and control.

In the formulation considered, the right-hand side of the dynamics is splitinto two parts. The first part contains the history of the state, and the secondpart contains the history of the state as well as the control. Many familiardelay problems are included in the first part of the formulation.

In Section 10.2 we will state the problem and assumptions, and addressexistence issues briefly. In the next section we state a corresponding mini-mum principle. In Section 10.4 we will consider linear systems and systemsthat are linear in the state variable. These systems are concrete versions ofthe general problem considered in Chapter 11, and the constituents of theminimum principle, Theorem 11.4.4 can be spelled out in these cases easily.In Section 10.5 we consider an example of a linear plant with quadratic costcriterion. In these examples we note that the relaxed optimal controls can bereplaced by ordinary controls.

As is well known, hereditary problems are infinite dimensional problems.Our approach is to essentially employ finite dimensional techniques. This willbe seen in Chapter 11. In Section 10.6, we deal with a hereditary problem inan infinite dimensional setting.

We remark that systems of the form given in Section 10.4 can be handledby the method presented in Section 10.6.

10.2 Problem Statement and Assumptions

Consider a process governed by the system

φ′(t) = f(φ(·), u(·), t) (10.2.1)

φ(t) = y(t), −r ≤ t ≤ 0, y ∈ M (10.2.2)

295


u(t) ∈ Ω(t, φ(t)), (10.2.3)

T (φ(0), φ(t1)) = 0. (10.2.4)

It is desired that the cost

∫ t1

0

f0(φ(·), u(·), t)dt (10.2.5)

be infimum.

Assumption 10.2.1. For the definition of f(φ(·), u(·), t), and f0(φ(·), u(·), t)see Section 2.7. We assume that the set-valued map Ω(t, x), (t, x) ∈ It10 ×X isu.s.c.i. on It10 ×X, and for each (t, x) ∈ It10 ×X, Ω(t, x) is compact. Finally, weassume T is continuously differentiable, although for existence we only needlower semi-continuity. The set M is a closed convex subset of H1(−r, 0).

Theorem 10.2.2. Assume the set of admissible pairs for (10.2.1) to (10.2.4)is not empty. We assume that all admissible trajectories are such that(φ(0), φ(t1)) lie in a fixed compact subset of Rn ×R

n. To guarantee existencewe also assume that the set M in (10.2.2) is bounded in H1(−r, 0). Then, thereexists a relaxed admissible pair giving the cost (10.2.5) its infimum value. How-ever, the derivation of necessary conditions at an optimal pair known to existdoes not require that the set M be bounded in H1(−r, 0).

Exercise 10.2.3. Verify Theorem 10.2.2 regarding existence.

For more on existence refer to Chapter 4 and Chapter 5.

10.3 Minimum Principle

The problem stated in Section 10.2 is a special case of what is presented inChapter 11, Section 11.2. Thus, from Theorem 11.4.4 we obtain the followingminimum principle for it. This problem has been considered in [69].

We suggest to the reader to take a quick look at Theorem 11.4.4 and scanits proof to see where Theorem 10.3.1 came from.

Theorem 10.3.1. For the problem stated in Section 10.2, under Assump-tion 10.2.1, we have the following necessary conditions met at any optimalrelaxed pair (φ0, ν0): There exist a function of bounded variation Φ on It10 ,scalars λ0 ≥ 0, β; Γ(t, s) = (Γ1(t, s), . . . ,Γn(t, s)), and Γ0(t, s) such thats 7→ Γi(t, s) is of bounded variation, continuous from the right and vanishingfor s > t.

(i) λ0 + |β|+ |Φ(t−1 )| = 1;

Hereditary Systems 297

(ii) Φ(t)− λ0∫ t1t

Γ0(s, t)ds+∫ t1t

Φ(s)Γ(s, t)ds = Φ(t−1 )

Φ(s)Γ(s, t) = Φ1Γ1(s, t) + · · ·+ΦnΓn(s, t)

(iii) for any relaxed control ν,

λ0f0(t, φ0(·), ν0t)− Φ(t) · f(t, φ0(·), ν0t)≥ λ0f0(t, φ0(·), νt)− Φ(t) · f(t, φ0(·), νt) a.e.

(iv) Φ(0) = β∂1T (φ0(0), φ0(t1)) +∫ t10λ0Γ0(t, 0)− Φ(t) · Γ(t, 0)dt

+∫ t10−λ0Γ0(t,−r) + Φ(t) · Γ(t,−r)dt+B(0),

Φ(t−1 ) = −β∂2T (φ(0), φ0(t1)).

(v) B′′(t)−B(t) +∫ t10[−λ0Γ0(s, t) + Φ(s)Γ(s, t)]ds

−∫ t10 [−λ0Γ0(t,−r) + Φ(t)Γ(t,−r)]dt = 0,

B(−r) = 0, (B′, B) ∈ ∂IM(y).

To see how (iv) is obtained consult Theorem 11.4.4(iii) and (ii). Note thatsince we do not have state constraint here, we remove quantities involving themultiplier λ in Theorem 11.4.4.

Remark 10.3.2. To see how Γ0 and Γ in the previous theorem arise, see(11.3.12) to (11.3.14). In the following example and the one in Section 10.5Γi(t, s), i = 0, 1, . . . , n are explicitly given in terms of the data of the problem.

Now, consider the process (10.2.1) to (10.2.5) where now

f i(φ(·), u(·), t) =∫ t

−r

φ(s)dsηi(t, s) + ki(t, u(t)) (10.3.1)

+

∫ t

−r

〈φ(s), Li(t, s)〉dsωi(t, s) +

∫ t

−r

M i(t, s, u(s))dsωi(t, s),

i = 0, 1 . . . , n.

In this case Γi(t, s), ith entry of Γ(t, s) = (Γ1(t, s), . . . ,Γn(t, s)) in Theo-rem 10.3.1, is given by

Γi(t, s) = ηi(t, s)−∫ t

s

Li(t, τ)dτωi(t, τ), (10.3.2)

and (iii) of the same theorem is

− λ0k0(t, ν0t) +

∫ t

−r

M0(t, s, ν0s)dsω0(t, s)

(10.3.3)


+Φ(t) ·(k(t, ν0t) +

∫ t

−r

M(t, s, ν0s)dsω(t, s)

)≥

− λ0k0(t, νt) +

∫ t

−r

M0(t, s, νt)dsω0(t, s)

+Φ(t) ·(k(t, νt) +

∫ t

−r

M(t, s, νs)dsω(t, s)

)a.e. t ∈ [0, t1]

10.4 Some Linear Systems

We specialize (10.2.1) to (10.2.5) as follows.We consider

d

dtφi(t) = f i(φ(·), u(·), t), i = 1, . . . , n, (10.4.1)

where

f i(φ(·), u(·), t) = ki(t, u(t)) +

∫ 0

−r

φ(t+ s)dη(s) (10.4.2)

+

∫ t

−r

M i(t, s, u(s))ds, i = 1, 2, . . . , n

and it is required tominimize g(φ0(t1)). (10.4.3)

We assume that the set-valued map Ω(t, x) is independent of x. We also assumethat η is of bounded variation and continuous from the right.

Writing

L(t, φt) =

∫ 0

−r

φ(t+ s)dη(s), (10.4.4)

we define Φ(t, s) by the equation [41, page 144–146]

Φ(t, s) =

∫ t

s L(ξ,Φξ(·, s))dξ + I, a.e. in s, t ≥ s,

0, s− r ≤ t < s(10.4.5)

and T by the formulaΦt(·, s) = T (t, s)X0 (10.4.6)

X0(θ) =

0, −r ≤ θ < 0

I, θ = 0

Now

φ(t) = (T (t, 0)y)(0) +

∫ t

0

Φ(t, s)

k(s, u(s)) +

∫ s

−r

M(s, ξ, u(ξ))dξ

ds

(10.4.7)


That is,

φ(t) = (T (t, 0)y)(0) +

∫ t

0

Φ(t, s)k(s, u(s))ds+

∫ 0

−r

∫ t

0

Φ(t, ξ)M(ξ, s, u(s))dξ ds

(10.4.8)

+

∫ t

0

∫ t

s

Φ(t, ξ)M(ξ, s, u(s))dξ ds

Assumption 10.4.1. In (10.4.8) assumeM(ξ, s, u) is of the formM(ξ)h(s, u)where both M and h are continuous.

Suppose (φ∗, ν∗) is a relaxed optimal pair for (10.4.1) to (10.4.3) with re-strictions (10.2.2) to (10.2.4), where as stated previously Ω(t, x) is independentof x.

Rewriting (10.4.8) as

φ∗(t)− (T (t, 0)y∗)(0)−∫ 0

−r

∫ t

0

Φ(t, ξ)M(ξ, s, ν∗s )dξ ds

=

∫ t

0

[Φ(t, s)k(s, ν∗s ) +

∫ t

s

Φ(t, ξ)M(ξ, s, ν∗s )dξ

]ds

We can argue, as in the proof of Theorem 9.3.1, that there exists an ordinarycontrol u∗0 such that

k(s, ν∗s ) +

∫ t

s

M(ξ, s, ν∗s )dξ = k(x, u∗0(s)) +

∫ t

s

M(ξ, s, u∗0(s))ds. (10.4.9)

Similarly, there is an ordinary control u∗−r such that

∫ 0

−r

∫ t

0

Φ(t, ξ)M(ξ, s, ν∗s )dξ ds =

∫ 0

−r

∫ t

0

Φ(t, ξ)M(ξ, s, u∗−r(s)dξ ds.

(10.4.10)Thus, we have the following:

Theorem 10.4.2. The problem (10.4.1) to (10.4.3) under Assumption 10.4.1and the restrictions (10.2.2) to (10.2.4), where Ω(t, x) is independent of x, hasan optimal pair where the control is ordinary.

Necessary conditions for the above problem can be derived using the infi-nite dimensional approach of Section 10.6. We leave this to the reader.


10.5 Linear Plant-Quadratic Cost

Consider the process governed by the system

d

dtφi(t) = 〈ki(t), u(t)〉+

∫ t

−r

φ(s)dsωi(t, s)

+

∫ t

−r

〈M i(t, s), u(s)〉ds, i = 1, . . . , n (10.5.1)

φ(t) = y(t), −r ≤ t ≤ 0, y ∈M (10.5.2)

u(t) ∈ Rm, u(t) = u(t), −r ≤ t ≤ 0 (10.5.3)

T (φ(0), φ(t1)) = 0 (10.5.4)

It is required to

minimize

∫ t1

0

〈φ(t), X(t)φ(t)〉dt +∫ t1

0

〈u(t), R(t)u(t)〉dt

(10.5.5)

where X is a symmetric positive definite matrix, and R is a symmetric strictlypositive definite matrix.

Let (φ0, u0) be optimal for (10.5.1) to (10.5.5). Then, from (10.3.3)

− λ0〈u0(t), R(t)u0(t)〉+ 〈Φ(t), k(t)u0(t) +∫ t

−r

M(t, s)u0(s)ds〉 ≥ (10.5.6)

− λ0〈u(t), R(t)u(t)〉 + 〈Φ(t), k(t)u(t) +∫ t

−r

M(t, s)u(s)ds〉

If λ0 > 0 we can take it to be 1 and

uT0 (t) =1

2

Φ(t)k(t) +

∫ t1

t

Φ(s)M(s, t)ds

R−1(t) (10.5.7)

Thus, to resolve this problem one must seek Φ consistent with Theorem 10.3.1.In this example, Γi(t, s) of Theorem 10.3.1 is ωi(t, s).

10.6 Infinite Dimensional Setting

In this section we consider hereditary systems, and a slightly differentapproach to obtain necessary conditions. This approach was initially followedin [70].


For 0 < a < b, let C([a, b],Rn) be the Banach space of continuous functionsmapping the interval [a, b] into R

n with the topology of uniform convergence.Let r > 0 and C = C([−r, 0],Rn). For φ ∈ C, let

|φ| = max|φ(θ)| : −r ≤ θ ≤ 0.

Let σ ∈ R and A > 0. If

x ∈ C([σ − r, σ +A],Rn),

then for any t ∈ [σ, σ +A], let xt ∈ C be defined by

xt(θ) = x(t + θ), −r ≤ θ ≤ 0

For t1 > 0 we consider the problem of minimizing the functional

J(u, z) =

∫ t1

0

f0(xt, u(t))dt (10.6.1)

subject to the autonomous functional differential equation

dx(t)/dt = f(xt, u(t)), 0 < t < t1, (10.6.2)

x0 = z,

and the conditionsP (xt1 , x0) ≤ 0 (10.6.3)

Q(xt1 , x0) = 0 (10.6.4)

Assumption 10.6.1. In (10.6.1) and (10.6.2) we assume that f0 : C×Rm →

R, and f : C×Rm → R are continuous and that the Frechet derivatives f0

1 , f1are continuous. In (10.6.3) and (10.6.4) the functions P and Q are continuouslydifferentiable from C × C into R.

LetH1(−r, 0) = z : [−r, 0] → R

n | z, z′ ∈ L2(−r, 0).Corresponding to ζ = (z, ν), z ∈ H1(−r, 0), ν a relaxed control, denote byy[ζ](·) the unique solution of the equation

dy/dt = f(yt, νt), t0 < t < t1 (10.6.5)

y0 = z

Definition 10.6.2. We say that ζ = (z, ν) is admissible if y[ζ](·) satisfies(10.6.2) to (10.6.4).

Assumption 10.6.3. There exists a bounded continuous function ℓ(u) suchthat the function f appearing in (10.6.2) satisfies

(i) |f(0, u)| ≤ ℓ(u)

(ii) |f(x, u)− f(x, u)| ≤ ℓ(u)|x− x|.


10.6.1 Approximate Optimality Conditions

Assume that ζ0 = (z0, ν0) is a relaxed solution for (10.6.1) to (10.6.4). Let

Uad = (z, ν) | z ∈ H1(−r, 0), ν relaxed control (10.6.6)

Let

FK(z, ν) =

∫ t1

t0

f0(yt, νt)dt+ ‖z′ − z′0‖2 + |z(0)− z0(0)|2 (10.6.7)

+ ǫ‖ν − ν0‖L +Kγ(P (yt1, yt0)) +KQ2(yt1 , yt0),

where γ is a smooth convex function such that γ(t) = 0, t ≤ 0, γ(t) > 0 ift > 0, and

‖ · ‖ = L2-norm on [−r, 0],‖ν − ν0‖L = ess sup|ν(t)− ν0(t)|(Ω) : 0 ≤ t ≤ t1,

Ω a fixed compact subset of Rm, and yt is obtained from (10.6.2) and theinitial condition. As mentioned earlier we will indicate this by writing y[ζ](·),ζ = (z, ν), and y[ζ]t will stand for the function θ 7→ y[ζ](t+ θ), −r ≤ θ ≤ 0.

Assume that ∫ t1

0

f0(y[ζ0]t, ν0t)dt = 0.

We state, without proof, the following two lemmas.

Lemma 10.6.4. For any 0 < ǫ ≤ 1 there exists K(ǫ) > 0 such thatFK(ǫ)(z, ν) > 0 if any of the following inequalities is an equality:

|z(0)− z0(0)| ≤ ǫ, ‖z′ − z′0‖ ≤ ǫ, ‖ν − ν0‖L ≤ ǫ.

Lemma 10.6.5. Let K(ǫ) be as in Lemma 10.6.4, 0 < ǫ ≤ 1. Then, thefunctional (z, ν) → FK(ǫ)(z, ν) attains its minimum on B(ǫ). For 0 < ǫ ≤ 1,let (zǫ, νǫ) ∈ B(ǫ) be such that

FK(ǫ)(zǫ, νǫ) = infFK(ǫ)(z, ν) | (z, ν) ∈ B(ǫ). (10.6.8)

Then,

‖zǫ′ − z′0‖ < ǫ, |zǫ(0)− z0(0)| < ǫ, ‖νǫ − ν0||L < ǫ. (10.6.9)

From (10.6.8) and (10.6.9), for (z, ν) ∈ Uad we have

limh→0

dFK(ǫ)(zǫ + hz, νǫ)/dh = 0, (10.6.10)

limh→0+

dFK(ǫ)(zǫ, νǫ + h(ν − νǫ))/dh ≥ 0. (10.6.11)

Now, let Tǫ(s, σ) be the solution operator of the homogeneous equation

dW/dt = f1(y[ζǫ]t, νǫ)Wt, (10.6.12)


whereζǫ = (zǫ, νǫ) (10.6.13)

Also, we define

X0(θ) =

0, −r ≤ θ < 0

I, θ = 0(10.6.14)

where I is the identity operator. Then, if we set

δy[ζǫ](t) = limǫ→0

y[ζǫ,h](t)− y[ζǫ](t)

ǫ, (10.6.15)

whereζǫ,h = (zǫ + h(z − zǫ), νǫ), 0 < h≪ 1, (10.6.16)

we may writeδy[ζǫ]t = Tǫ(t, 0)X0(z − zǫ). (10.6.17)

Thus, from (10.6.10), writing yǫ = y[ζǫ](·), we obtain∫ t1

0

f01 (y

ǫt , ν

ǫt )Tǫ(t, 0)X0(z − zǫ)dt (10.6.18)

+K(ǫ)[γ′(P (yǫt1 , yǫt0))P1(y

ǫt1 , y

ǫt0)

+ 2Q(yǫt1 , yǫ0)Q1(y

ǫt1 , y

ǫ0)]Tǫ(t1, 0)X0(z − zǫ)

+K(ǫ)[γ′(P (yǫt1 , yǫ0))P2(y

ǫt1 , y

ǫ0) + 2Q(yǫt1 , y

ǫ0)Q2(y

ǫt1 , y

ǫ0)](z − zǫ)

+

∫ 0

−r

2(z′ǫ − z′0)(z − zǫ)dt+ 2(zǫ(0)− z0(0))(z(0)− zǫ(0)) = 0

If, now, we set

ζǫ,h = (zǫ, νǫ + h(ν − νǫ)), 0 ≤ h ≤ 1, (10.6.19)

and compute δy[ζǫ] as in (10.6.15), then by the variation of constants formulawe obtain

δy[ζǫ]t =

∫ t

0

Tǫ(t, s)X0f(yǫs, νs − νǫs)ds (10.6.20)

and so from (10.6.11) we obtain∫ t1

0

f0(yǫt , νt − νǫt ) +

∫ t1

t

f01 (y

ǫs, ν

ǫs)Tǫ(s, t)X0f(y

ǫt , νt − νǫt )ds

dt

(10.6.21)

+

∫ t1

0

K(ǫ)[γ′(P (yǫt1 , y

ǫ0))P1(y

ǫt1 , y

ǫ0) + 2Q(yǫt1 , y

ǫt0)Q1(y

ǫt1 , y

ǫ0)]·

· Tǫ(t1, s)X0f(yǫs, νs − νǫs)

ds+ ǫρ′ǫ(0

+) ≥ 0,

whereρǫ(h) = ‖νǫ + h(ν − νǫ)− ν0‖L, 0 ≤ h ≤ 1.


10.6.2 Optimality Conditions

In this section we obtain optimality conditions.

Assumption 10.6.6. We assume that |P1(y01 , y

00)|+ |P2(y

01 , y

00)| 6= 0. Let

λ0 =1

limǫ→0M(ǫ)(10.6.22)

From the assumptions on the data one can immediately prove that ∃ǫ1, ǫ2, . . . , ǫn, · · · → 0, λ1 ≥ 0, ℓ1, ℓ2 ∈ C∗([−r, 0]) such that

λ0 + λ1 + |ℓ1|+ |ℓ2| 6= 0 (10.6.23)

and∫ t1

0

λ0f01 (y

0t , ν0t)T (t, 0)dt+ (λ1P1(y

0t1 , y

00) + ℓ1)T (t1, 0) (10.6.24)

+ (λ1P2(y0t1 , y

00) + ℓ2)I = 0,

∫ t1

0

λ0f0(y0t , νt − ν0t) +

∫ t1

t

f01 (y

0s , ν

0s )T (s, t)X0f(y

0t , νt − ν0t)ds

dt

(10.6.25)

+

∫ t1

0

(λ1(P1(y0t1 , y

00) + ℓ1)T (t1, s)X0f(y

0s , νs − ν0s)ds ≥ 0,

wherey0(t) ≡ y0[ζ0](t), (10.6.26)

ζ0 = (z0, ν0), (10.6.27)

and T (s, σ) be the solution operator of the homogeneous system

dw/dt = f1(y0t , ν0t)wt. (10.6.28)

Define

ψt = λ0∫ t1

t

f01 (y

0s , ν0s)T (s, t)ds+ (λ1P1(y

0t1 , y

00) + ℓ1)T (t1, t). (10.6.29)

Now, we have the following theorem.

Theorem 10.6.7. Suppose (z0, ν0) provides an optimal control for (10.6.1)to (10.6.4). With Assumptions 10.6.1 to 10.6.6 in force there exist multipliersλ0, λ1 ≥ 0, ℓi ∈ C∗([−r, 0]), i = 1, 2 and an adjoint variable ψ such that

(i) λ0 + λ1 + |ℓ1|+ |ℓ2| = 1, λ1P (y01 , y00) = 0.

(ii) ψt = λ0∫ t1tf01 (y

0s , ν0s)T (s, t)ds+ (λ1P1(y

0t1 , y

00) + ℓ1)T (t1, t)

(iii) ψ0 = −(λ1P2(y0t1 , y

00) + ℓ2)I

(iv) ∀ ν relaxed

λ0f0(y0t , νt) + ψtX0f(y0t , νt) ≥ λ0f0(y0t , ν0t) + ψtX0f(y

0t , ν0t) a.e.

Chapter 11

Bounded State Problems

11.1 Introduction

In this chapter we deal with bounded state problems. We give in detailthe derivation of necessary conditions for problems governed by hereditarysystems. Then, we specialize to problems governed by other systems. We followthe approaches in [66, 69, 70].

In Sections 11.2 to 11.4 we proceed to present the bounded state prob-lems for problems governed by hereditary systems. In Section 11.5 we coverproblems governed by integrodifferential systems, and in Section 11.6 prob-lems governed by ordinary differential systems. Since problems governed byordinary differential systems are more common we have treated them inde-pendently.

11.2 Statement of the Problem

The problem considered is

min

∫ t1

0

f0(φ(·), u(·), t)dt (11.2.1)

such thatd

dtφ(t) = f(φ(·), u(·), t), (11.2.2)

φ(t) = y(t), −r ≤ t ≤ 0, y ∈ M, (11.2.3)

M closed convex subset of H1(−r, 0), (11.2.4)

u(t) ∈ Ω(t, φ(t)), a.e. t ∈ [0, t1], (11.2.5)

T (φ(0), φ(t1)) = 0, (11.2.6)

G(t, φ(t)) ≤ 0, 0 ≤ t ≤ t1, (11.2.7)

wheref(φ(·), u(·), t) = 〈f1(φ(·), u(·), t), . . . , fn(φ(·), u(·), t)〉,

305


f(φ(·), u(·), t) = 〈f0(φ(·), u(·), t), f(φ(·), u(·), t)〉,and

f i(φ(·), u(·), t) = hi(t, φ(·), u(·)) (11.2.8)

+

∫ t

−r

gi(t, s, φ(s), u(s))dsωi(t, s), 0 ≤ i ≤ n

Remark 11.2.1. For the assumptions on the data of the problem we go backto Section 2.7.

Assumption 11.2.2. In (11.2.7) the function G is continuous in both ar-guments, and twice continuously differentiable in the second argument. In(11.2.6) T is assumed to be continuously differentiable.

11.3 ǫ-Optimality Conditions

In what follows we remind the reader that Remark 11.2.1 is in force.

Remark 11.3.1. We assume that all admissible trajectories are such that(φ(0), φ(t1)) lie in a fixed compact subset of Rn ×R

n. To guarantee existencewe also assume that the set M in (11.2.4) is bounded in H1(−r, 0). Then, weleave it to the reader to verify that optimal pairs exist provided the set ofadmissible pairs is not empty. However, the derivation of necessary conditionsat an optimal pair known to exist does not require that the set M be boundedin H1(−r, 0).

Let (φ0, ν0) be optimal for the relaxed version of (11.2.1) to (11.2.7). Inwhat follows, we write y0 for φ0 on [−r, 0]. We fix (φ0, ν0) in the rest of ourdiscussion. Without loss of generality, we assume that

J(φ0, ν0) ≡∫ t1

0

f0(φ0(.), ν0, t)dt = 0.

Let

Ad = (φ, y, ν) | φ ∈ AC(It10 ,X), φ′ ∈ L2(I

t10 ), (11.3.1)

y ∈ H1(−r, 0), φ(0) = y(0), ν a relaxed control

B(ǫ) = (φ, y, ν) ∈ Ad | ‖φ′ − φ′0‖ ≤ ǫ, ‖ν − ν0‖L ≤ ǫ, |φ(0)− φ0(0)| ≤ ǫ,(11.3.2)

‖y′ − y′0‖∗ ≤ ǫ,G(t, φ(t)) ≤ 0, 0 ≤ t ≤ t1,

Bounded State Problems 307

where‖ν − ν0‖L = ess-sup|ν(t)− ν0(t)|(Ω) : −r ≤ t ≤ t1

‖ · ‖ = L2-norm on It10

‖ · ‖∗ = L2-norm on I0−r

Let

IM(y) =

0, if y ∈ M

∞, if y 6∈ M

On H1(−r, 0), and for K ≥ 0 a scalar, we define

PK(z) = inf2−1K(‖z′ − x′‖2∗ + ‖z− x‖2∗) + IM(x) : x ∈ H1(−r, 0) (11.3.3)

Note that z 7→ PK(z) is lower semi-continuous, convex, and Gateaux differ-entiable.

For (φ, y, ν) ∈ Ad, set

φ = φ on It10 , (11.3.4)

φ = y on I0−r. (11.3.5)

Now, define a functional on Ad by

FK(φ, y, ν) = J(φ, ν) + ‖φ′ − φ′0‖2 + |φ(0)− φ0(0)|2 (11.3.6)

+ ǫ‖ν − ν0‖L +K‖d/dt(φ(t))− f(φ(·), ν, t)‖2

+ ‖y′ − y′0‖2∗ + PK(y) +KT 2(φ(0), φ(t1))

Remark 11.3.2. In B(ǫ), we choose 0 < ǫ < ǫ1 to ensure that all admissibletrajectories lie in a fixed compact set X′ ⊂ X.

Lemma 11.3.3. For any 0 < ǫ ≤ ǫ1, ∃ K(ǫ) > 0 such that FK(ǫ)(φ, y, ν) > 0,(φ, y, ν) ∈ B(ǫ), if any of the following inequalities is an equality:

|φ(0)− φ0(0)| ≤ ǫ, ‖φ′ − φ′0‖ ≤ ǫ, ‖y′ − y′0‖∗ ≤ ǫ, ‖ν − ν0‖L ≤ ǫ.

Proof. Suppose the lemma were false. Then, for any 0 < ǫ ≤ ǫ1 ∃ 〈φj , yj , νj〉⊂ B(ǫ), Kj, Kj → ∞, where φj , yj, νj are such that at least one of theinequalities is an equality and FKj

(φj , yj , νj) ≤ 0. Then, it is easy to conclude

that there exists j1 < j2 < · · · such that φ′jk → φ∗′

, y′jk → y∗′

weaklyin L2, and φjk → φ∗, yjk → y∗ uniformly, and νjk → ν∗ weak-∗. Further,(φ∗, ν∗) is admissible where φ∗ = φ∗ on It10 and φ∗ = y∗ on I0−r. Note thatJ(φ∗, ν∗) + ǫ2 ≤ 0. Since (φ0, ν0) is optimal, we have a contradiction and thelemma is proved.

Lemma 11.3.4. Let K(ǫ) be as in Lemma 11.3.3, 0 < ǫ < ǫ1. Then, thefunctional (φ, y, ν) 7→ FK(ǫ)(φ, y, ν) attains its minimum in B(ǫ). For 0 < ǫ <ǫ1, let (φǫ, yǫ, ν

ǫ) ∈ B(ǫ) be such that

FK(ǫ)(φǫ, yǫ, νǫ) = infFK(ǫ)(φ, y, ν) | (φ, y, ν) ∈ B(ǫ).


Then‖φ′ǫ − φ′0‖ < ǫ, |φǫ(0)− φ0(0)| < ǫ,

‖y′ − y′0‖∗ < ǫ, ‖νǫ − ν0‖L < ǫ.

Proof. The proof follows immediately from Lemma 11.3.3.

For 0 < ǫ ≤ ǫ1, let

V (ǫ) = (φ, y) | φ ∈ AC(It10 ,X), y ∈ H1(−r, 0), φ′ ∈ L2(It10 ), (11.3.7)

φ(0) = y(0), ‖φ′ − φ′0‖ ≤ ǫ, ‖y′ − y′0‖∗ ≤ ǫ, |φ(0)− φ0(0)| ≤ ǫ

Consider the functional HjK(ǫ)(φ, y) on V (ǫ) defined by

HjK(ǫ)(φ, y) = j

∫ t1

0

ω(G(t, φ(t)))dt + FK(ǫ)(φ, y, νǫ) (11.3.8)

where νǫ is as in Lemma 11.3.4 and ω is a smooth convex function such that

ω(t) = 0, t ≤ 0,

ω(t) > 0, t > 0.

One can verify that, for each j = 1, 2, 3, . . . there exist (φj , yj) ∈ V (ǫ) suchthat

HjK(ǫ)(φj , yj) = infHj

K(ǫ)(φ, y) | (φ, y) ∈ V (ǫ) (11.3.9)

Lemma 11.3.5. Let (φj , yj) be as in (11.3.9). Then, there exists a subse-quence jkℓ

such that

‖φ′jkℓ − φ′0‖ < ǫ, ‖y′jkℓ − y′0‖∗ < ǫ, |φjkℓ (0)− φ0(0)| < ǫ.

Proof. For each j = 1, 2, 3 . . . , we have

HjK(ǫ)(φj , yj) ≤ Hj

K(ǫ)(φǫ, yǫ) = FK(ǫ)(φǫ, yǫ, νǫ).

Thus, we can infer that

limj→∞

j

∫ t1

0

ω(G(t, φj(t)))dt + limj→∞

∫ t1

0

f0(φj(·), νǫ, t)dt+ limj→∞

‖φ′j − φ′0‖2

(11.3.10)

+ limj→∞

‖y′j − y′0‖2∗ + limj→∞

K(ǫ)‖d/dt (φj)− f(φj(·), νǫ, t)‖2

+ limj→∞

|φj(0)− φ0(0)|2 + ǫ‖νǫ − ν0‖L ≤ FK(ǫ)(φǫ, yǫ, νǫ),

whereφj = φj , on It10 ,


φj = yj, on I0−r.

There exists a subsequence of j = 1, 2, 3, . . . , which we denote by jk,φǫ, yǫ such that φ′jk → φ′ǫ, y

′jk

→ y′ǫ weakly in L2 in It10 (I0−r), respectively, and

φjk → φǫ, yjk → yǫ uniformly on It10 (I0−r), respectively. From (11.3.8) we seethat

G(t, φǫ(t)) ≤ 0, 0 ≤ t ≤ t1.

Also, (φǫ, yǫ) ∈ V (ǫ), and we conclude that

FK(ǫ)(φǫ, yǫ, νǫ) = FK(ǫ)(φǫ, yǫ, ν

ǫ).

Now, using Lemma 11.3.4, we have

‖φ′ǫ − φ′0‖ < ǫ, ‖y′ǫ − y′0‖∗ < ǫ, |φǫ(0)− φ0(0)| < ǫ.

Now, from (11.3.8) and the fact


ǫ)

the lemma follows.

Remark 11.3.6. In the course of proving Lemma 11.3.5 we have come across(φǫ, yǫ) satisfying

‖φ′ǫ − φ′0‖ < ǫ, ‖y′ǫ − y′0‖∗ < ǫ, |φǫ(0)− φ0(0)‖ < ǫ,

φǫ(0) = yǫ(0), G(t, φǫ(t)) ≤ 0, 0 ≤ t ≤ t1,


ǫ).

In subsequent discussions, we keep this in mind. Again, for convenience we set

φǫ(t) =

φǫ(t), 0 ≤ t ≤ t1

yǫ(t), −r ≤ t ≤ 0.

Lemma 11.3.7. Let p, q ∈ L1(0, 1). Let the inequality

∫ t1

0

pw + qw′dt ≥ 0

hold for all w satisfying w(0) = w(1) = 0, w ≥ 0, with w piecewise continu-ously differentiable. Then, the function

T (t) = q(t)−∫ t

0

p(s)ds

is nonincreasing outside of a set of measure zero.


Proof. Let t′′ > t′ ≥ 0 be Lebesgue points of q. Taking

w(t) =

(t− t′)/ǫ, t′ ≤ t ≤ t′ + ǫ

1, t′ + ǫ ≤ t ≤ t′′ − ǫ

(t′′ − t)/ǫ, t′′ − ǫ ≤ t ≤ t′′

0, 0 ≤ t ≤ t′, t′′ ≤ t ≤ t1

gives the result.

By the Riesz representation theorem, there exists a function s 7→Γi(t, φǫ(·), νǫt , s) defined for s ∈ It1−r, of bounded variation and continuous

from the right such that, for each ζ ∈ C(It1−r), we have

dhi(t, φǫ(·), νǫt )(ζ) =∫ t1

−r

ζ(s) · dsΓi(t, φǫ(·), νǫt , s) (11.3.11)

Here, dhi(t, φǫ(·), νǫt ) denotes the Frechet derivative of hi(t, ·, νǫt ) at φǫ. Since hidoes not depend on φǫ(s) for s > t, s 7→ Γi(t, φǫ(·), νǫt , s) is constant for s > t.

We take this constant to be zero and uniquely determine Γi(t, φǫ(·), νǫt , s).Thus, we may write (11.3.11) as

dhi(t, φǫ(·), νǫt )(ζ) =∫ t

−r

ζ(s)·dsΓi(t, φǫ(·), νǫt , s) =∫ t

−r

ζ(s) · dsΓi(t, ǫ; s)

(11.3.12)Let

Γi(t, φǫ(·), νǫt , s) = −∫ t

s

gix(t, τ, φǫ(·), νǫt )dτωi(t, τ), (11.3.13)

for −r ≤ s ≤ t, and equal to zero for s ≥ t. Let

Γi = Γi + Γi, i = 0, 1, 2, . . . , n. (11.3.14)

Denote the Frechet derivative of f i(·, νǫt , t) at φǫ by df i(φǫ(·), νǫt , t). Then, forζ ∈ C(It1−r), we have

df i(φǫ(·), νǫt , t)(ζ) =∫ t

−r

ζ(s) · dsΓi(t, φǫ(·), νǫt , s) =∫ t

−r

ζ(s) · dsΓi(t, ǫ; s)

(11.3.15)At this point, let us note that for a.e. t ∈ It10 , i = 0, . . . , n,

|Γi(t, ǫ; s)| = |Γi(t, ǫ; s)− Γi(t, ǫ; t)| ≤ V (Γi(t, ǫ; ·)) ≤ Λ(t),

where V stands for total variation.Let

Γ(t, ǫ; s) = (Γ1(t, ǫ; s), . . . ,Γn(t, ǫ; s)) (11.3.16)

where Γ1, . . . ,Γn are row vectors. If v = (v1, . . . , vn), then by v · Γ we mean


v1Γ1 + · · · + vnΓ

n. Thus, v1Γ1 + · · · + vnΓn is an n-dimensional row vector.Let

ψ(ǫ; t) = 2(φ′ǫ(t)− φ′0(t)) + 2K(ǫ)

[d

dtφǫ(t)− f(ǫ; t)

], (11.3.17)

where we have written f(ǫ; t) for f(φǫ(·), νǫt , t). We will use this sort of notationin the sequel.

We next let

Ψ(ǫ; t) = ψ(ǫ; t)−∫ t1

t

Γ0(s, ǫ; t)ds+

∫ t1

t

[ψ(ǫ; s)−2(φ′ǫ(s)−φ′0(s))]·Γ(s, ǫ; t)ds.(11.3.18)

Assumption 11.3.8. We assume that for 0 ≤ t ≤ t1 we have∇xG(t, x) 6= 0 ∀x such that |φ0(t)−x| < ǫ2, ǫ2 > ǫ1.∇xG(t, x) = (∇x1

G(t, x), . . . ,∇xnG(t, x)).

Let ζ be an absolutely continuous scalar function with square integrablederivative on It10 such that

ζ(t) ≥ 0, 0 ≤ t ≤ t1

ζ(0) = ζ(t1) = 0.

Then, there exists θ0 > 0 such that for 0 ≤ θ ≤ θ0,

G(t, φǫ(t) + θζ(t)ξǫ(t)) ≤ 0,

whereξǫ(t) = −∇xG(t, φǫ(t))/|∇xG(t, φǫ(t))|2.

From the fact that

dFK(ǫ)(φǫ + θζξǫ, yǫ, νǫ)/dθ|θ=0+ ≥ 0 (11.3.19)

we obtain ∫ t1

0

((Ψ(ǫ; t) · ξǫ)ζ′ + (Ψ(ǫ; t) · ξ′ǫ)ζ)dt ≥ 0. (11.3.20)

Thus, by Lemma 11.3.7, we obtain a nonincreasing function,

λ(ǫ; t) = Ψ(ǫ; t) · ξǫ(t)−∫ t

0

Ψ(ǫ; s) · ξ′ǫ(s)ds. (11.3.21)

Leth(t) = ∇xG(t, φǫ(t))/|∇xG(t, φǫ(t))|2.

In Lemma 11.3.5, for simplicity of notation, we assume that the conclusion ofthe lemma is true for jk instead of jkℓ

. Let h(t) be defined by

h(t) = h(t), 0 ≤ t ≤ 1,


h(t) = ∇xG(0, φǫ(0))/|∇xG(0, φǫ(0))|2, −r ≤ t ≤ 0

For θ > 0 small,

(φjk + θh, yjk + θ∇xG(0, φǫ(0))/|∇xG(0, φǫ(0))|2) ∈ V (ǫ).

Then, using (11.3.8),

dHjkK(ǫ)(φjk + θh, yjk + θ∇xG(0, φǫ(0))/|∇G(0, φǫ(0))|2)|θ=0 = 0. (11.3.22)

From (11.3.22), we can conclude that jk∫ t10ω′(G(t, φjk (t)))dt is bounded as

k → ∞.Let ζ be a smooth vector function defined on It10 and vanishing for t ≤ 0

and t ≥ t1. Then let

w(t) = ζ(t) + (∇xG(t, φǫ(t)) · ζ)ξǫ, (11.3.23)

where ξǫ is defined as before. For θ > 0 small enough, (φjk + θw, yjk ) ∈ V (ǫ).Thus, again from (11.3.8) we have

dHjkK(ǫ)(φjk + θw, yjk)/dθ|θ=0 = 0. (11.3.24)

In (11.3.24), we let k → ∞ after differentiating with respect to θ at θ = 0 andobtain

Φ(ǫ; t)−∫ t1

t

Γ0(s, ǫ; t)ds (11.3.25)

+

∫ t1

t

[Φ(ǫ; s)− 2(φ′ǫ(s)− φ′0(s))] · Γ(s, ǫ; t)ds

−∫ t1

t

λ(ǫ; s)∇xG(s, φǫ(s)) · Γ(s, ǫ; t)ds

+

∫ t1

t

λ(ǫ; s)[dGx(s, φǫ(s))/ds]ds = Cǫ,

whereΦ(ǫ; s) = ψ(ǫ; s) + λ(ǫ; s)∇xG(s, φǫ(s)). (11.3.26)

Note that from (11.3.25) we have

Cǫ = Φ(ǫ, t−1 ). (11.3.27)

Lemma 11.3.9. The nonincreasing function λ(ǫ, ·) defined in (11.3.21) is

constant at points of It10 where G(t, φǫ(t)) < 0.

Proof. Suppose that t0 ∈ (0, t1) is such that G(t0, φǫ(t0)) < 0. Then, thereexist 0 < α < β < t1 such that

G(t, φǫ(t)) < 0, α ≤ t ≤ β.


Let η ∈ C∞0 (α, β). Using Remark 11.3.6, we have

dFK(ǫ)(φǫ + θη, yǫ, νǫ)/dθ|θ=0 = 0,

from which we infer that Ψ(ǫ; ·) defined by (11.3.18) is constant over theinterval (α, β). Then, from (11.3.21), we immediately see that

λ(ǫ; t) = λ(ǫ;α), t ∈ (α, β).

Let ζ be an absolutely continuous n-dimensional vector function on It1−r

such that ζ(−r) = 0 and ζ = 0 on [0, t1]. From Remark 11.3.6 we have

dFK(ǫ)(φǫ, yǫ + θζ, νǫ)/dθ|θ=0 = 0. (11.3.28)

Then, from (11.3.28) we obtain

∫ 0

−r

ζ′(t) ·∫ t1

0

[−Γ0(s, ǫ; t) + (ψ(ǫ; s)− 2(φ′ǫ(s)− φ′0(s))) · Γ(s, ǫ; t)]ds

+ 2(y′ǫ(t)− y′0(t))

dt+ ∂PK(ǫ)(yǫ)(ζ) = 0 (11.3.29)

Writing

∂PK(ǫ)(yǫ)(ζ) =

∫ 0

−r

(b′ǫ · ζ′ −Bǫ · ζ′)ds,

where B′ǫ = bǫ, Bǫ(−r) = 0, we conclude from (11.3.28) that, for a.e. t,

∫ t1

0

[· · · ]ds+ 2(y′ǫ(t)− y′0(t)) + b′ǫ(t)−Bǫ(t) = Dǫ, (11.3.30)

where Dǫ is a constant.Let ζ be a smooth scalar function such that ζ ≥ 0, ζ(t) = 1, 0 ≤ t ≤ δ/2

and ζ(t) = 0, t ≥ 3δ/4.Let

ηN (t) = (η1(t), . . . , ηn(t))

such thatηi(t) = e−Ntζ(t), ηj(t) = 0 if j 6= i.

Let

ηN+(t) = ηN (t), 0 ≤ t ≤ t1

ηN−(t) = ηN (0), −r ≤ t ≤ 0

For the next results dealing with boundary conditions we assume that As-sumption 11.4.1 is in force. Then, in

dHjkK(ǫ)(φjk + θηN+, yjk + θηN−)

dθ

∣∣∣θ=0

= 0,


we let N → ∞. We carry out this operation for i = 1, 2, . . . , n and obtain

0 =− ψ(ǫ; 0) +

∫ t1

0

Γ0(t, ǫ; 0)−

[ψ(ǫ; t)− 2(φ′ǫ(t)− φ′0(t))

]· Γ(t, ǫ; 0)

dt

+

∫ t1

0

−Γ0(t, ǫ;−r) +

[ψ(ǫ; t)− 2(φ′ǫ(t)− φ′0(t))

]· Γ(t, ǫ;−r)

dt

+ 2K(ǫ)T (φǫ(0), φǫ(t1))∂1T (φǫ(0), φǫ(t1)) + 2(φǫ(0)− φ0(0)) +Bǫ(0)(11.3.31)

Next, let ζ be a smooth scalar function such that ζ ≥ 0

ζ(t) = 1, t1 − δ/2 ≤ t ≤ t1

ζ(t) = 0, t ≤ t1 − 3δ/4

Now letηN (t) = (η1(t), . . . , ηn(t))

such thatηi(t) = e−N(t1−t)ζ(t), ηj(t) = 0 if j 6= i.

ηN+(t) = ηN (t), 0 ≤ t ≤ t1

ηN−(t) = 0, −r ≤ t ≤ 0

Then, indHjk

K(ǫ)(φjk + θηN+, yjk + θηN−)

dθ

∣∣∣θ=0

= 0,

we let N → ∞. We carry out this operation for i = 1, 2, . . . , n and obtain

ψ(t1; ǫ) + 2K(ǫ)T (φǫ(0)φǫ(t1))∂2T (φǫ(0), φǫ(t1)) = 0 (11.3.32)

According to Lemma 11.3.9,

λ(ǫ; ·) = λ(ǫ; 0+) in (0, δ)

λ(ǫ; ·) = λ(ǫ; t−1 ) in (t1 − δ, t1)

Now, we can rewrite (11.3.31) and (11.3.32)

Φ(ǫ; 0) = λ(ǫ; 0+)∇G(0, φǫ(0)) + 2K(ǫ)T (φǫ(0), φǫ(t1))∂1T (φǫ(0), φǫ(t1))

+

∫ t1

0

Γ0(t, ǫ; 0)−

[Φ(ǫ; t)− 2(φ′ǫ(t)− φ′0(t))

]· Γ(t, ǫ; 0)

+ λ(ǫ; t)∇G(t, φǫ(t)) · Γ(t, ǫ; 0)]dt

+

∫ t1

0

−Γ0(t, ǫ;−r) +

[Φ(ǫ; t)− 2(φ′ǫ(t)− φ′0(t))

]· Γ(t, ǫ;−r)


− λ(ǫ; t)∇G(t, φǫ(0)) · Γ(t, ǫ;−r)dt

+ 2(φǫ(0)− φ0(0)) +Bǫ(0) (11.3.33)

Φ(ǫ; t1) = λ(ǫ; t−1 )∇G(t, φǫ(t1))− 2K(ǫ)T (φǫ(0), φǫ(t1))∂2T (φǫ(0), φǫ(t1)) (11.3.34)

Next let ζ be an absolutely continuous vector function such that ζ ∈ L2(I0−r)

and ζ(t) = 0, 0 ≤ t ≤ t1. Using (11.3.28) we obtain

ζ(−r) ·∫ t1

0

[−Γ0(t, ǫ;−r) + ψ(ǫ; t) · Γ(t, ǫ;−r)]dt

+

∫ 0

−r

ζ′(t) ·∫ t1

0

[−Γ0(s, ǫ; t) + ψ(ǫ; s) · Γ(s, ǫ; t)]ds

+ 2(y′ǫ − y′0) + b′ǫ −Bǫ(t)

dt = 0,

whereψ(ǫ; t) = ψ(ǫ; t)− 2(φ′ǫ(t)− φ′0(t)).

We have seen in (11.3.30) that the vector in braces is constant, and thus canbe pulled out of the integral. Thus, we have

ζ(−r)·∫ t1

0

[−Γ0(t, ǫ;−r) + ψ(ǫ; t) · Γ(t, ǫ;−r)]dt

−∫ t1

0

[−Γ0(s, ǫ; t) + ψ(ǫ; s)Γ(s, ǫ; t)]ds

− 2(y′ǫ(t)− y′0(t)) − (b′ǫ(t)− Bǫ(t))

= 0

Thus,

b′ǫ(t)−Bǫ(t)−∫ t1

0

[−Γ0(t, ǫ;−r) + ψ(ǫ; t) · Γ(t, ǫ;−r)]dt

+

∫ t1

0

[−Γ0(s, ǫ; t) + ψ(ǫ; s)Γ(s, ǫ; t)]ds+ 2(y′ǫ(t)− y′0(t)) = 0

Since B′ǫ(t) = bǫ and Bǫ(−r) = 0 we have

B′′ǫ (t)−Bǫ(t) +

∫ t1

0

[−Γ0(s, ǫ; t) + ψ(ǫ; s) · Γ(s, ǫ; t)]ds

−∫ t1

0

[−Γ0(t, ǫ;−r) + ψ(ǫ; t)Γ(t, ǫ;−r)]dt+ 2(y′ǫ(t)− y′0(t)) = 0


Bǫ(−r) = 0, (11.3.35)

(B′ǫ, Bǫ) ∈ ∂PK(ǫ)(yǫ).Next, for 0 < ǫ < ǫ1 and 0 ≤ θ ≤ 1, let

ν(θ) = νǫ + θ(ν − νǫ).

Since‖νǫ − ν0‖L < ǫ,

there exists θ0 > 0 such that 0 ≤ θ ≤ θ0 implies

‖ν(θ)− ν0‖L < ǫ.

Now, from Remark 11.3.6 and Lemma 11.3.4, we have

dFK(ǫ)(φǫ, yǫ, ν(θ))/dθ|θ=0 ≥ 0. (11.3.36)

Letρǫ(θ) = ‖ν(θ)− ν0‖L.

The function θ 7→ ρǫ(θ) has a right derivative at θ = 0. Thus, from (11.3.36)we obtain

H(t, φǫ(·), νǫt ) ≥ H(t, φǫ(·), νt)− ǫρ′ǫ(0+), a.e. t, (11.3.37)

where

H(t, φǫ(·), σt) = −f0(t, φǫ(·), σt) (11.3.38)

+ [Φ(ǫ; t)− λ(ǫ; t)∇G(t, φǫ(t)) − 2(φ′ǫ(t)− φ′0(t))] · f(t, φǫ(·), σt),

and

φǫ(t) =

φǫ(t), 0 ≤ t ≤ 1

yǫ(t), −r ≤ t ≤ 0(11.3.39)

11.4 Limiting Operations

Before we proceed in this section we make Assumption 11.4.1.

Assumption 11.4.1. We make the assumption that there exists δ > 0 suchthat for t ∈ (0, δ) ∪ (t1 − δ, t1), 0 < δ < t1, we have G(t, φ0(t)) < 0.


Note that if ǫ, 0 < ǫ ≤ ǫ1, is sufficiently small,

G(t, φǫ(t)) < 0, t ∈ (0, δ) ∪ (t1 − δ, t1).

According to Lemma 11.3.9,

λ(ǫ; ·) = λ(ǫ; 0+), in (0, δ),

λ(ǫ; ·) = λ(ǫ; t−1 ), in (t1 − δ, t1).

Remark 11.4.2. Wemodify λ by subtracting d ≡ λ(ǫ; t−1 ) from it. Henceforthwe assume that this has been done. Thus, λ ≥ 0.

Remark 11.4.3. The effect of Remark 11.4.2 is as follows. Modifying λ asindicated will force us to modify Φ(ǫ; t) by taking off d∇xG(t, φǫ(t)) from it,

and d∇xG(t1, φǫ(t1)) from Cǫ to maintain (11.3.25). We assume that thesemodifications have already been done in what follows. The modifications in(11.3.33) and (11.3.34) are similarly done. For example, (11.3.34) is modified

by removing d∇xG(t1, φǫ(t1)) from the right-hand side.

From (11.3.25) we can verify that

|Φ(ǫ; t)| ≤ C(d1 + λ(ǫ, 0+)d2 + |Φ(ǫ; 1−)|), (11.4.1)

where C > 0, d1 > 0, d2 > 0 are constants independent of ǫ, 0 < ǫ ≤ ǫ1. Wemay also verify that

VΦ(ǫ; ·) ≤ C1(d1 + λ(ǫ, 0+)d2), (11.4.2)

where C1, d1, d2 are positive constants independent of ǫ, 0 < ǫ ≤ ǫ1, andV Φ(ǫ; ·) is the total variation of Φ(ǫ; ·). Motivated by (11.4.1) and (11.4.2) wedefine a constant

M(ǫ) = 1 + |Φ(ǫ; 1−)|+ λ(ǫ, 0+) + |βǫ|. (11.4.3)

Using the Helley compactness theorem, there exists a subsequence 0 < ǫ1 <ǫ2 < · · · < ǫn → 0 such that

λ(ǫi; ·)/M(ǫi) → λ(·), a.e. t ∈ [0, t1] (11.4.4)

Φ(ǫi; ·)/M(ǫi) → Φ(·), a.e. t ∈ [0, t1]. (11.4.5)

Clearly, we have ∫ t1

0

∫ t1

0

|Γi(s, ǫ; t)|2ds dt ≤ ‖Λ‖2, (11.4.6)

and thus we can extract a subsequence ǫ′1, ǫ′2, . . . , ǫ

′n, . . . such that

Γi(·, ǫ′j , ·) → Γi(·, ·) (11.4.7)


weakly in L2((0, t1)×(0, t1)). By Mazur’s theorem, there exist αn1, αn2, . . . , αnkn,

n = 1, 2, 3, . . . , αnm ≥ 0,∑kn

m=1 αnm = 1, such that

vin =

kn∑

m=1

αnmΓi(·, ǫm, ·) → Γi(·, ·), in L2

We can extract a subsequence n1 < n2 < · · · such that

vinj→ Γi(·, ·), a.e. (11.4.8)

Suppose that (s∗, t∗) ∈ [0, t1]× [0, t1] such that

limj→∞

vinj(s∗, t∗) = Γi(s∗, t∗) (11.4.9)

Note that we haveV vinj

(s∗, ·) ≤ Λ(s∗), (11.4.10)

where V denotes the total variation. By the Helley compactness theorem,there exists j1 < j2 < · · · such that, pointwise in the t-variable,

limk→∞

vinjk(s∗, t) = γs∗(t), (11.4.11)

where γs∗(·) is of bounded variation and

V γs∗(·) ≤ Λ(s∗) (11.4.12)

Using (11.4.11), we see that we can modify t 7→ Γi(s∗, t) to agree with γs∗(·).Having done the modification, we can say Γi(s∗, ·) is of bounded variation and

V Γi(s∗, ·) ≤ Λ(s∗) (11.4.13)

Letλ0 = 1/ lim

ǫ→0+M(ǫ) (11.4.14)

We can extract a subsequence ǫ′′1 , ǫ′′2 , . . . tending to zero such that

1/M(ǫ′′i ) → λ0, (11.4.15)

β′′ǫi/M(ǫ′′i ) → β. (11.4.16)

Clearly, we can take an appropriate subsequence of ǫǫ>0 tending to zeroso that all limiting operations involving ǫ tending to zero are simultaneouslyvalid.

In (11.3.25), (11.3.33), (11.3.34), and (11.3.37), keeping in mind Re-marks 11.4.2 and 11.4.3, we divide both sides of the equation/inequality byM(ǫ) and let ǫ → 0+ through an appropriate subsequence to obtain the fol-lowing theorem.


Theorem 11.4.4. Suppose Assumption 11.4.1 is in force. Then, at any re-laxed optimal pair (φ0, ν0), the following conditions are met: There exist afunction of bounded variation Φ on It10 , a bounded nonincreasing function λ,λ(·) ≥ 0, λ(t−1 ) = 0, Γ(t, s) = (Γ1(t, s), . . . ,Γn(t, s)), and a function Γ0 suchthat s 7→ Γi(t, s), s ∈ It1−r, i = 0, 1, . . . , n, is of bounded variation satisfying(11.4.13), continuous from the right vanishing for s > t, and scalars β, λ0,λ0 ≥ 0, γ0 such that

(i) |Φ(t−1 )|+ λ(0+) + |β|+ λ0 = 1,

(ii) Φ(t)−λ0∫ t1t

Γ0(s, t)ds+∫ t1t

Φ(s)Γ(s, t)ds−∫ t1tλ(s)∇G(s, φ0(s))Γ(s, t)ds

+∫ t1tλ(s)[d∇G(s, φ0(s))/ds]ds = Φ(t−1 )

(iii) Φ(0) = λ(0+)∇G(0, φ0(0))+β∂1T (φ0(0), φ0(t1))+∫ t10 λ0Γ0(t, 0)−Φ(t)·

Γ(t, 0)+λ(t)∇G(t, φ0(t))·Γ(t, 0)dt+∫ t10 −λ0Γ0(t,−r)+Φ(t)·Γ(t,−r)−

λ(t)∇G(t, φ0(t)) · Γ(t,−r)dt+B(0)

B′′(t)−B(t) +∫ t10 [−λ0Γ0(s, t) + Φ(s) · Γ(s, t)− λ(s)∇G(s, φ0(s))]ds

−∫ t10[−λ0Γ0(t,−r) + Φ(t) · Γ(t,−r)− λ(t)∇G(t, φ0(t)) · Γ(t,−r)]dt = 0

B(−r) = 0

(B′, B) ∈ ∂IM(y0)

(iv) Φ(t−1 ) = −β∂2T (φ(0), φ0(t1)).

(v) H(t, φ(·), ν0t) ≤ H(t, φ(·), σt), a.e. t, where

H(t, φ(·), σt) = −[Φ(t)−λ(t)∇G(t, φ0(t))]·f(t, φ(·), σt)+λ0f0(t, φ(·), σt),

φ(t) =

φ0(t), 0 ≤ t ≤ t1

y0(t), −r ≤ t ≤ 0

Remark 11.4.5. We have the condition

‖Φ(·)− λ(·)∇G(·, φ0(·))‖∞ + |β|+ λ0 6= 0.

Otherwise,Φ(·) = λ(·)∇G(·, φ0(·))

Thus, Φ(t−1 ) = 0, and (ii) of Theorem 11.4.4 says

Φ(t) +

∫ t1

t

λ(s)[d∇G(s, φ0(s))/ds] = 0,

and Φ is absolutely continuous. Thus,

Φ′(t) = λ(t)[d∇G(t, φ0(t))/dt].


Thus,λ′ = 0.

Since λ(t−1 ) = 0, it follows that λ ≡ 0. In particular λ(0+) = 0 and (i) ofTheorem 11.4.4 is contradicted.

Exercise 11.4.6. Verify that λ(0+) + |β|+ λ0 6= 0.

11.5 The Bounded State Problem for IntegrodifferentialSystems

In this section we specialize the problem in Section 11.2. Consider theproblem

min

∫ t1

0

f0(φ(t), u(t), t)dt (11.5.1)

subject to

d

dtφi(t) = f i(t, φ(t)) +

∫ t

0

gi(t, s, φ(s), u(s))ds, 1 ≤ i ≤ n (11.5.2)

u(t) ∈ Ω, 0 ≤ t ≤ t1 (11.5.3)

T (φ(0), φ(t1)) = 0 (11.5.4)

G(t, φ(t)) ≤ 0, 0 ≤ t ≤ t1 (11.5.5)

The assumptions on T and G in (11.5.4) and (11.5.5) remain the same asin Section 11.2 and Ω is a fixed compact set in R

m.

Assumption 11.5.1. For each compact set X ⊂ X× U, we have

|f0(x, u, t)|+ |f0x(x, u, t)| ≤ Λ(t), (x, u, t) ∈ X× [0, t1].

|gi(t, s, x, u)|+ |gix(t, s, x, u)| ≤ Λ(t),

(t, s, x, u) ∈ [0, t1]× [0, t1]× X, 1 ≤ i ≤ n.

We also assume that f0 and gi, 1 ≤ i ≤ n are continuous in the u variable.

Assumption 11.5.2. In (11.5.2), f i : X × [0, t1] → R is such that f i(·, x) ismeasurable, f i(t, ·) is continuously differentiable satisfying

|f i(t, x)|+ |f ix(t, x)| ≤ Λ(t), Λ ∈ L2(0, t1).


Let fx = (f1x , . . . , f

nx ) where f i

x = (f i1, . . . , f

in). Similarly, let gx =

(g1x, . . . , gnx ). If v = (v1, . . . , vn) by v ·gx we mean v1g

1x+ · · ·+vngnx where v ·gix

is inner-product. Thus, v1g1x + · · ·+ vng

nx is an n-dimensional row vector.

The following theorem is a special case of Theorem 11.4.4.

Theorem 11.5.3. Suppose Assumptions 11.4.1, 11.5.1, and 11.5.2 hold. Atany relaxed pair (φ0, ν0) optimal for (11.5.1) to (11.5.5) the following condi-tions are met: There exist a function of bounded variation Φ on It10 , a boundednonincreasing function λ, λ ≥ 0, λ(t−1 ) = 0, scalars β, γ0, λ

0 (λ0 ≥ 0) suchthat

(i) |Φ(t−1 )|+ λ(0+) + |β|+ λ0 = 1,

(ii) Φ(t) +∫ t1tλ0f0

x(φ0(s), ν0s, s)ds−∫ t1t

Φ(s)fx(s, φ0(s))ds

+∫ t1tλ(s)∇G(s, φ0(s))fx(s, φ0(s))ds

−∫ t1t

∫ t1s

Φ(τ)gx(τ, s, φ0(s), ν0s)dτ ds

+∫ t1t

∫ t1s λ(τ)∇G(τ, φ0(τ))gx(τ, s, φ0(s), ν0s)dτ ds

+∫ t1t λ(s)[dGx(s, φ0(s))/ds]ds = Φ(t−1 ).

(iii) Φ(0) = λ(0+)∇G(0, φ0(0)) + β∂1T (φ0(0), φ0(1))

(iv) Φ(t−1 ) = −β∂2T (φ0(0), φ0(1))

(v) H(t, φ0(t), ν0t) ≥ H(t, φ0(t), νt) a.e., where

H(t, φ0(t), σt) = [Φ(t)− λ(t)∇xG(t, φ0(t))] · (f(t, φ0(t))

+

∫ t

0

g(t, s, φ0(s), σs)ds− λ0f0(φ0(t), σt, t)

Remark 11.5.4. An equivalent condition to (v) of Theorem 11.5.3 is

(v′) H(t, φ0(t), ν0t) ≥ H(t, φ0(t), νt) a.e., where

H(t, φ0(t), σt) = (Φ(t)− λ(t)∇xG(t, φ0(t))) · f(t, φ0(t))

+

∫ t1

t

[Φ(s)− λ(s)∇xG(s, φ0(s))] · g(s, t, φ0(t), σt)ds

− λ0f0(φ0(t), σt, t)

Remark 11.5.5. Let us remind the reader that Remark 11.4.5 is still valid.

Remark 11.5.6. In (11.5.2) we can replace f i(t, φ(t)) by f i(φ(t), u(t), t). Weonly need to make appropriate assumptions on the functions f i(·, ·, ·) (seeRemark 11.2.1).


11.6 The Bounded State Problem for OrdinaryDifferential Systems

Again we specialize the problem in Section 11.2. In this case we considerthe problem

min

∫ t1

0

f0(φ(t), u(t), t)dt (11.6.1)

subject toφ′(t) = f(φ(t), u(t), t), (11.6.2)

u(t) ∈ Ω, 0 ≤ t ≤ t1, (11.6.3)

T (φ(0), φ(1)) = 0, (11.6.4)

G(t, φ(t)) ≤ 0, 0 ≤ t ≤ t1 (11.6.5)

Assumption 11.6.1. Again, Ω is a fixed compact subset of Rm, and the

assumptions on T and G remain the same as in Section 11.2, and f in (11.6.2)has the same properties as f0 (see Assumption 11.5.1).

Before we state the maximum principle in this situation, we make thefollowing remark:

Remark 11.6.2. Repeating the procedure that was used to obtain (11.3.31)we obtain

Φ(0) = λ(0+)∇G(0, φ0(0)) + β∂1T (φ0(0), φ(t1)) (11.6.6)

Theorem 11.6.3. Suppose Assumptions 11.4.1 and 11.6.1 hold. At any re-laxed pair (φ0, ν0) optimal for (11.6.1) to (11.6.5) the following conditions aremet: There exists an absolutely continuous function Φ, a bounded nonincreas-ing function λ, λ ≥ 0, λ(t−1 ) = 0, scalars β, γ0, λ

0 (λ0 ≥ 0), such that

(i) |Φ(0)|+ |λ(0+)|+ |β|+ λ0 = 1,

(ii) Φ′(t) = λ0f0x(φ0(t), ν0t, t)− Φ(t) · fx(φ0(t), ν0t, t)

+λ(t)∇xG(t, φ0(t)) · fx(φ0(t), ν0t, t) + λ(t)(d∇xG(t, φ0(t))/dt)

(iii) Φ(0) = λ(0+)∇G(0, φ0(0)) + β∂1T (φ0(0), φ0(t1))

(iv) Φ(t1) = −β∂2T (φ0(0), φ0(t1))

(v) [Φ(t)− λ(t)∇G(t, φ0(t))] · f(φ0(t), ν0t, t)− λ0f0(φ0(t), ν0t, t)

≥ [Φ(t)− λ(t)∇G(t, φ0(t))] · f(φ0(t), νt, t)− λ0f0(φ0(t), νt, t) a.e. t.

Remark 11.6.4. Remark 11.4.5 remains valid. Also, we can replace (i) by(i′) |Φ(t1)|+ λ(0+) + |β|+ λ0 6= 0.


Exercise 11.6.5. Verify that λ(0+) + λ0 + |β| 6= 0.

The proof of Theorem 11.6.3 can be obtained from Theorem 11.4.4. Thebounded state problem (11.6.1) to (11.6.5) is frequently encountered in appli-cations. Thus, we will briefly present direct proof of Theorem 11.6.3 below.We follow the approach in [66].

Corresponding to the functional (11.3.6) we have

Fk(φ, ν) =

∫ t1

0

f0(φ(t), νt, t)dt+ ‖φ′ − φ′0‖2 + |φ(0)− φ0(0)|2

+ ǫ‖ν − ν0‖L +KT 2(φ(0), φ(t1)) +K‖φ′(t)− f(φ(t), νt, t‖2

where (φ0, ν0) is an optimal pair, and we assume FK(φ0, ν0) = 0.Lemma 11.3.3 remains valid. That is, for any 0 < ǫ ≤ ǫ1 ∃ K(ǫ) such thatFK(ǫ)(φ, ν) > 0, (φ, ν) ∈ B(ǫ) if any of the inequalities is an equality:

|φ(0)− φ0(0)| ≤ ǫ, ‖φ′ − φ′0‖ ≤ ǫ, ‖ν − ν0‖L ≤ ǫ

where

B(ǫ) = (φ, ν) ∈ B | ‖φ′ − φ′0‖ ≤ ǫ, |φ(0)− φ0(0)| ≤ ǫ, ‖ν − ν0‖L ≤ ǫ

and

B = (φ, ν) | φ ∈ L2(0, t1)n, φ absolutely continuous,

G(t, φ(t)) ≤ 0, 0 ≤ t ≤ t1, ν a relaxed control.

Lemma 11.3.4 also remains valid in this case.For 0 < ǫ ≤ ǫ1, let

V (ǫ) = φ | φ absolutely continuous, ‖φ′ − φ′0‖ ≤ ǫ, |φ(0)− φ0(0)| ≤ ǫ.

As in (11.3.8) we consider the functional φ 7→ HjK(ǫ)(φ) on V (ǫ), where

HjK(ǫ)(φ) = j

∫ t1

0

ω(G(t, φ(t))dt + FK(ǫ)(φ, νǫ)

Lemma 11.3.5 is also valid in this case. That is, there exists a subsequencejk such that

‖φ′jk − φ′0‖ < ǫ, |φjk(0)− φ0(0)| < ǫ.

Remark 11.6.6. The proof of Lemma 11.3.5 also gives an absolutely contin-uous function φǫ such that

‖φ′ǫ − φ′0‖ < ǫ, |φǫ(0)− φ0(0)| < ǫ,

G(t, φǫ(t)) ≤ 0, 0 ≤ t ≤ t1,

FK(ǫ)(φǫ, νǫ) = FK(ǫ)(φǫ, ν

ǫ).

That is, (φǫ, νǫ) minimizes FK(ǫ)(φ, ν) over B(ǫ).


Let ζ be an absolutely continuous scalar function on [0, t1] such that ζ ≥ 0on [0, t1], ζ(0) = ζ(t1) = 0. Let us set

ξǫ(t) =−∇G(t, φǫ(t))|∇G(t, φǫ(t))|2

.

We can easily verify that ∃ θ0 > 0 such that 0 < θ ≤ θ0 implies

G(t, φǫ(t) + θζ(t)ξǫ(t)) ≤ 0, 0 ≤ t ≤ t1

‖[φǫ(t) + θζ(t)ξǫ(t)]′ − φ′0(t)‖ < ǫ

Then, using Remark 11.6.6 we have

dFK(ǫ)(φǫ + θζξǫ, νǫ)

dθ

∣∣∣θ=θ+

≥ 0

Now using Lemma 11.3.7 we obtain a nonincreasing function λ(ǫ; t) definedby

λ(ǫ; t) = ψ(ǫ; t) · ξǫ(t)−∫ t

0

f01 (φǫ(s), ν

ǫs, s) · ξǫ(s) + ψ(ǫ; s) · ξ′ǫ(s)

− [ψ(ǫ; s)− 2(φǫ(s)− φ0(s))] · f1(φǫ(s), νǫs, s) · ξǫ(s)ds (11.6.7)

where

ψǫ(ǫ; t) = 2(φ′ǫ(t)− φ′0(t)) + 2K(ǫ)(φ′ǫ(t)− f(φǫ(t), νǫt , t)) (11.6.8)

In what follows we sometimes write h(ǫ; t) for h(φǫ(t), νǫt , t).

When we write out dFK(ǫ)(φǫ + θη, νǫ)/dθ we get

J(φǫ, η) ≡∫ t1

0

f01 (ǫ; t) · η(t)dt +

∫ t1

0

ψ(ǫ; t) · η′(t)dt

−∫ t1

0

(ψ(ǫ; t)− 2(φ′ǫ − φ′0)) · f1(ǫ; t) · η(t)dt (11.6.9)

Integration by parts shows that

J(φǫ, ζξǫ) =

∫ t1

0

λ(ǫ; t)ζ′(t)dt (11.6.10)

Let h(t) be an absolutely continuous vector function such that h ∈L2([0, t1])

n. Then, we have

dHjkK(ǫ)(φjk + θh)

dθ

∣∣∣θ=0

= 0

If we take h(t) = ∇G(t, φǫ(t))/|∇G(t, φǫ(t))|2 we see that jk∫ t10ω′(G(t,


φjk(t))dt remain bounded as k → ∞. If we next replace h by ζ(t) +

[∇G(t, φǫ(t)) · ζ]ξǫ, ζ(0) = ζ(t1) = 0 we obtain

J(φǫ, ζ) + J(φǫ, [∇G(·, φǫ(·)) · ζ]ζǫ) = 0

Now, using (11.6.9) and (11.6.10) we obtain

ψ(ǫ; t) + λ(ǫ; t)∇G(t, φǫ(t)) (11.6.11)

=

∫ t

0

−[ψ(ǫ; s) + λ(ǫ; s)∇G(s, φǫ(s))] · f1(ǫ; s) + f0

1 (ǫ; s)

+ λ(ǫ; s)∇G(s, φǫ(s)) · f1(ǫ; s)

+ λ(ǫ; s)

[d∇G(s, φǫ(s))

ds

]+ 2(φ′ǫ(s)− φ′0(s)) · f1(ǫ; s)

ds+ cǫ

We next obtain an ǫ-minimum principle. From Remark 11.6.6 we knowthat FK(ǫ)(φǫ, ν

ǫ) minimizes FK(ǫ)(φ, ν) over B(ǫ). For 0 < ǫ < ǫ′ < ǫ and0 ≤ θ ≤ 1, let

ν(θ) = νǫ + θ(ν − νǫ)

Since ‖νǫ−ν0‖ < ǫ ∃ θ0 > 0 such that 0 ≤ θ ≤ θ0 implies that ‖ν(θ)−ν0‖ < ǫ.Thus,

dFK(ǫ)(φǫ, ν(θ))

dθ

∣∣∣θ=0+

≥ 0 (11.6.12)

Letρǫ(θ) = ‖ν(θ)− ν0‖L

Thus, from (11.6.12) it follows that

−∫ t1

0

ψ(ǫ; t) · f(φǫ(t), νt, t)dt+∫ t1

0

f0(φǫ(t), νt, t)dt

+

∫ t1

0

2(φ′ǫ − φ′0) · f(φǫ(t), νt, t)dt

≥ −∫ t1

0

ψ(ǫ; t) · f(φǫ(t), νǫt , t)dt+∫ t1

0

f0(φǫ(t), νǫt , t)dt

+

∫ t1

0

2(φ′ǫ − φ′0) · f(φǫ(t), νǫt , t)dt− ǫρ′ǫ(0+) (11.6.13)

where ψ(ǫ; ·) is as in (11.6.8).Finally we deal with end conditions. We enforce Assumption 11.4.1 here,

too. We remark that Lemma 11.3.9 continues to hold here, too. Accordingto Assumption 11.4.1 ∃ δ > 0 such that for t ∈ (0, δ) ∪ (t1 − δ, t1) we have

G(t, φ0(t)) < 0. Note that if ǫ, 0 < ǫ ≤ ǫ1 is sufficiently small G(t, φǫ(t)) < 0,t ∈ (0, δ) ∪ (t1 − δ, t1). According to Lemma 11.3.9,

λ(ǫ; ·) = λ(ǫ; 0+) in (0, δ)


λ(ǫ; ·) = λ(ǫ; t−1 ) in (t1 − δ, t1)

Let ζ be a smooth scalar function such that ζ ≥ 0,

ζ(t) = 1 0 ≤ t ≤ δ/2

ζ(t) = 0 t ≥ 3δ/4

For absolutely continuous vector function η such that η′ ∈ L2([0, t1])n, we

havedFK(ǫ)(φǫ + θζη, νǫ)

dθ

∣∣∣θ=0

= 0 (11.6.14)

In (11.6.14), let η(t) = (η1(t), . . . , ηn(t)) such that ηi(t) = e−Nt, ηj(t) = 0 ifj 6= i and then let N → ∞. Repeat this for i = 1, . . . , n. We obtain

−ψ(ǫ; 0) + φǫ(0)− φ0(0) + 2K(ǫ)∂1T (φǫ(0), φǫ(t1)) = 0 (11.6.15)

Similarly we obtain

ψ(ǫ; t1) + 2K(ǫ)∂2T (φǫ(0), φǫ(t1)) = 0 (11.6.16)

Writing Φ(ǫ; t) for ψ(ǫ; t)+λ(ǫ; t)∇G(t, φǫ(t)) where we have replaced λ(ǫ; ·)by λ(ǫ; ·)−λ(ǫ; t−1 ), we can rewrite (11.6.11), (11.6.13), (11.6.15), and (11.6.16).Then, we can proceed with limiting operations.

Let

M(ǫ) = 1 + |Φ(ǫ; 0)|+ λ(ǫ; 0+) + 2K(ǫ)|T φǫ(0), φǫ(t1))|

Φ(ǫ; ·) = Φ(ǫ; ·)M(ǫ)

λ(ǫ; ·) = λ(ǫ; ·)M(ǫ)

βǫ = 2K(ǫ)|T (φǫ(0), φǫ(t1))|/M(ǫ)

We take appropriate subsequences in Φǫ0<ǫ≤ǫ′, λ(ǫ; 0)0<ǫ≤ǫ′,

βǫ0<ǫ≤ǫ′ and verify that each of the assertions in Theorem 11.6.3 is true.

11.7 Further Discussion of the Bounded State Problem

This section should be considered an extension of Section 11.6. The pur-pose is to attempt to do away with Assumption 11.4.1.

Consider the problem

min

∫ t1

0

f0(φ(t), u(t), t)dt (11.7.1)



T (φ(0)) = 0, φ(t1) = A (11.7.3)

G(t, φ(t)) = 0, (11.7.4)

u(t) ∈ Ω, (11.7.5)

which is the same problem as (11.6.1) to (11.6.5) except for two changes. First,the state here is completely on the boundary, and the end conditions here areseparated, and the state at t1 is fixed. We could have fixed the state at t = 0and require that T (φ(t1)) = 0.

The following theorem is useful here in obtaining necessary conditions.

Theorem 11.7.1 (Helley’s Convergence Theorem). Let gn be a sequence offunctions of bounded variations on [a, b] converging to g everywhere on [a, b].Suppose V (gn) ≤ C ∀ n. Then, g is also of bounded variation and

∫

Λ

f(x)dgn →∫

Λ

f(x)dg

for any continuous function f .

Let (φ0, ν0) be optimal for (11.7.1) to (11.7.5). Define a functional

FK(φ, ν) =

∫ t1

0

f0(φ(t), νt, t)dt+ ‖φ′ − φ′0‖2 + |φ(0)− φ0(0)|2

+ ǫ‖ν − ν0‖L +K(ǫ)

[T 2(φ(0)) +

∫ t1

0

G2(t, φ(t))dt

]

+K‖φ′(t)− f(φ(t), νt, t)‖2.

We proceed as in Section 11.3 or Section 11.6 and immediately obtain thefollowing:

Theorem 11.7.2. Let (φ0, ν0) be a relaxed optimal pair for (11.7.1) to(11.7.5). Then, there exist scalars λ0 ≥ 0, β, an absolutely continuous functionψ, a function of bounded variation σ such that

(i) λ0 + |ψ(0)|+ |β|+ |dσ| = 1, |dσ| = total variation of the measure dσ.

(ii) ψ(t) = λ0∫ t

0f01 (φ0(s), ν0s, s)ds−

∫ t

0ψ(s) · f1(φ0(s), ν0s, s)ds

+∫ t

0 ∇G(s, φ0(s))dσ(s) + β∇T (φ0(0))

(iii) −λ0f0(φ0(t), ν0t, t) + ψ(t) · f(φ0(t), ν0t, t) ≥ −λ0f0(φ0(t), νt, t) + ψ(t) ·f(φ0(t), νt, t) a.e. t.


Now let us consider the problem

min

∫ t1

0

f0(φ(t), u(t), t)dt (11.7.6)


T (φ(0)) = 0, (11.7.8)

T (φ(t1)) = 0, (11.7.9)

G(t, φ(t)) ≤ 0, (11.7.10)

u(t) ∈ Ω. (11.7.11)

Now, suppose (φ0, ν0) is a relaxed optimal pair for (11.7.6) to (11.7.11).Suppose G(t, φ0(t)) = 0 for t ∈ [0, τ1]∪ [τ2, t1], 0 < τ1 < τ2 < t1. Suppose also∃ δ > 0 such τ1 + δ < τ2 − δ and G(t, φ0(t)) < 0 on (τ1, τ1 + δ) ∪ (τ2 − δ, τ2).On [0, τ1] we consider the problem

min

∫ τ

0

f0(φ(t), u(t), t)dt (11.7.12)


T (φ(0)) = 0, (11.7.14)

φ(τ1) = φ0(τ1), (11.7.15)

G(t, φ(t)) = 0, (11.7.16)

u(t) ∈ Ω,

and apply Theorem 11.7.1. Similarly on [τ2, t1]. On [τ1, τ2] we consider theproblem

min

∫ τ2

τ1

f0(φ(t), u(t), t)dt (11.7.17)


φ(τ1) = φ0(τ1), (11.7.19)

φ(τ2) = φ0(τ2), (11.7.20)

G(t, φ(t)) ≤ 0, (11.7.21)

u(t) ∈ Ω, (11.7.22)

and apply Theorem 11.6.3.


11.8 Sufficiency Conditions

Consider the following problem

min

∫ t1

0

f0(φ(t), u(t), t)dt (11.8.1)

subject toφ′(t) = f(φ(t), u(t), t) (11.8.2)

G(φ(t)) ≤ 0 (11.8.3)

T1(φ(0)) = T2(φ(t1)) = 0 (11.8.4)

u(t) ∈ Ω (11.8.5)

This is the same problem as (11.6.1) to (11.6.5) except that (11.6.4) is replacedby (11.8.4), and (11.6.5) by (11.8.3). We also make assumptions additional tothose stated in Theorem 11.6.3.

Assumption 11.8.1. Let the assumptions stated in Theorem 11.6.3 remainin force. In addition assume that f0 and f are continuously differentiablein u. We also assume that G, T1, and T2 in (11.8.3) and (11.8.4) are twicecontinuously differentiable.

Using Theorem 11.6.3 and Assumption 11.8.1 we state the following:

Theorem 11.8.2. Let Assumption 11.8.1 be in force. Suppose (φ0, ν0) isa relaxed optimal pair for (11.8.1) to (11.8.5). Then, there exist an adjointvariable Φ, a nonincreasing nonnegative function λ, multipliers λ0 ≥ 0, β,λ(t1) = 0 such that

(i) Φ′(t) = λ0f0x(φ0(t), ν0t, t)− Φ(t) · fx(φ0(t), ν0t, t)

+λ(t)∇xG(φ0(t)) · fx(φ0(t), ν0t, t) + λ(t)d∇xG(φ0(t))/dt,

λ is constant where G(φ0(t)) < 0.

(ii) Φ(0) = λ(0+)∇G(φ0(0)) + β1∇T1(φ0(0)),

Φ(t1) = −β1∇T2(φ0(t1)).

(iii) [Φ(t)− λ(t)∇G(φ0(t))] · f(φ0(t), ν0t, t)− λ0f0(φ0(t), ν0t, t)

≥ [Φ(t)− λ(t)∇G(φ0(t))] · f(φ0(t), νt, t)− λ0f0(φ0(t), νt, t) a.e.


We may write

ν0t =n+1∑

i=1

Π0i (t)δu0

i (t), Π0

i ≥ 0,∑

Π0i = 1,

and thus,

F 0(φ0(t), ν0t, t) =∑

Π0i (t)f

0(φ0(t), u0i (t), t).

Similarly,

F (φ0(t), ν0t, t) =∑

Π0i (t)f(φ0(t), u

0i (t), t).

Then, we write

u0(t) ≡ (Π01(t), . . . ,Π

0n+1(t), u

01(t), . . . , u

0n+1(t)),

Π0i ≥ 0,

∑Π0

i = 1, u0i ∈ Ω,

Remark 11.8.3. In general, in this section, when we write u(t), we think ofit in the form

u(t) ≡ (Π1(t), . . . ,Πn+1(t), u1(t), . . . , un+1(t)),

Πi ≥ 0,∑

Πi = 1, ui ∈ Ω.

Then,

F 0(φ(t), u(t), t) ≡∑

Πi(t)f0(φ(t), ui(t), t),

F (φ(t), u(t), t) ≡∑

Πi(t)f(φ(t), ui(t), t).

Now, in (i) above we assume λ0 = 1, and let

H(φ, u, t) = F 0(φ(t), u(t), t) − (Φ(t) − λ(t)∇G(φ(t))) · F (φ(t), u(t), t)− Φ · φ + λ(0+)G(φ(0))/t1

− (Φ(0) · φ(0)− Φ(t1) · φ(t1))/t1+ [β1T (φ(0)) + β1T (φ(t1))]/t1

Set

J(φ, u) =

∫ t1

0

H(φ, u, t)dt.

Then, using (i) and (ii) of Theorem 11.8.2,

J(φ0, u0) =

∫ t1

0

F 0(φ0(t), u0(t), t)dt,

and for (φ, u) admissible,

J(φ, u) =

∫ t1

0

F 0(φ(t), u(t), t)dt −∫ t1

0

G(φ(t))dλ(t)


≤∫ t1

0

F 0(φ(t), u(t), t)dt

Using (i) and (ii) of Theorem 11.8.2 we can verify that Hφ(φ0, u0, t) = 0.

Thus,J(φ0 + δφ, u0)− J(φ0, u

0) = O(δφ2).

Further,HΦ(φ0, u

0, t) = φ0(t)− F (φ0(t), u0(t), t) = 0

and

Hu(φ0, u0, t) = F 0

u (φ0(t), u0(t), t) − (Φ− λ∇G(t, φ0(t))) · Fu(φ0(t), u

0(t), t).

Note that Hu(φ0, u0, t) is the same as the derivative of the Hamiltonian with

respect to u. Using (iii) of Theorem 11.8.2, for u admissible, we should insistthat

Hu(φ0, u0, t)(u − u0) ≥ 0 a.e. t.

Thus, we may state the following:

Theorem 11.8.4. Let Assumption 11.8.1 be in force. For problem (11.8.1)–(11.8.5) a sufficient condition for (φ0, u0) to be minimum is that

Hu(φ0, u0, t)(u− u0) ≥ 0 a.e. t.

and the matrix (Hφφ(φ0, u

0, t) Hφu(φ0, u0, t)

Hφu(φ0, u0, t) Huu(φ0, u

0, t)

)

is positive definite a.e. t.

Next, we modify (11.8.1) to (11.8.5) as follows:

min

∫ t1

0

F 0(φ(t), u(t), t)dt (11.8.6)

subject toφ′(t) = F (φ(t), u(t), t), (11.8.7)

G(φ(t)) ≤ 0, (11.8.8)

T1(φ(0)) = T2(φ(t1)) = 0, (11.8.9)

u(t) ∈ Ω. (11.8.10)

Let (φ0, ν0) be a relaxed optimal pair for (11.8.6) to (11.8.10). Again we applyTheorem 11.8.2. We write

u0(t) = (Π01(t), . . . ,Π

0n+1(t), u

01(t), . . . , u

0n+1(t)),


∑Π0

i = 1, Πi ≥ 0, u0i (t) ∈ Ω

if

ν0t =

n+1∑

i=1

Π0i (t)δu0

i (t)

For (φ, u) admissible we have

∫ t1

0

F 0(φ(t), u(t), t)dt ≥ J(φ, u).

Here, u is, in general, of the form

u(t) =

n+1∑

i=1

Πi(t)δui(t), Πi ≥ 0,∑

Πi = 1, ui(t) ∈ Ω.

Then, we have

∫ t1

0

F 0(φ(t), u(t), t)dt −∫ t1

0

F 0(φ0(t), u0(t), t)dt ≥ J(φ, u)− J(φ0, u0)

=

∫ t1

0

H(φ, u, t)− H(φ0, u0, t)− Hφ(φ− φ0)

dt

≥∫ t1

0

H(φ, u, t)− H(φ0, u0, t)− Hφ(φ− φ0)−Hu(u− u0)

dt,

where we have used (i) and (ii) of Theorem 11.8.2 and the fact

Hu(φ0, u0, t)(u(t)− u0(t)) ≥ 0 a.e.,

for u admissible.

11.9 Nonlinear Beam Problem

We now discuss the example presented in Section 1.8. We have to minimizethe cost

Iα(u) =1

2

∫ 1

0

u2dt+ α

∫ 1

0

cos θ dt

subject tod

dt

(x

θ

)=

(sin θ

u

),

x(0) = x(1) = 0

x2(t) ≤ (0.05)2.


We use Theorem 11.6.3 to analyze this problem. Corollary 4.5.1 can also beused to simply look for ordinary controls. Let (ψ1, ψ2) be the adjoint variable.Then,

ψ1 = 2λ sin θ

ψ2 = −αλ0 sin θ − ψ1 cos θ + 2λx cos θ

ψ2(0) = ψ2(1) = 0.

The Hamiltonian H is given by

H = −1

2λ0u2 + ψ2u+ (ψ1 − 2λx) sin θ − αλ0 cos θ.

On physical ground we eliminate λ0 = 0 and thus we may take λ0 = 1. Thus,we get

ψ1 = 2λ sin θ

ψ2 = −α sin θ + (2λx− ψ1) cos θ

ψ2(0) = ψ2(1) = 0

The Hamiltonian H is given by

H = −1

2u2 + ψ2u+ (ψ1 − 2λx) sin θ − α cos θ

andH(ψ, λ, x, θ, u0) ≥ H(ψ, λ, x, θ, u).

Thus,−u+ ψ2 = 0

∴ u = ψ2.

Thus, we must solve the system of equations

x = sin θ

θ = ψ2

ψ1 = 2λ sin θ (11.9.1)

ψ2 = −α sin θ + (2λx− ψ1) cos θ

x(0) = x(1) = ψ2(0) = ψ2(1) = 0.

Off the state constraint λ is constant. On the state constraint itself we caneasily verify, using (11.9.1), that λ is constant. Thus,

ψ1 − 2λ sin θ = ψ1 − 2λx = (ψ1 − 2λx). = 0

∴ ψ1 − 2λx = ψ1(0) (11.9.2)


Since x(0) = x(1) = 0 it follows from (11.9.2) that

ψ1(0) = ψ1(1).

Thus, using (11.9.1) and (11.9.2) we have

ψ2 = −α sin θ − ψ1(0) cos θ, (11.9.3)

θ = ψ2.

Also, using (11.9.1) and (11.9.3)

0 =

∫ 1

0

ψ2ds = −α∫ 1

0

sin θ ds− ψ1(0)

∫ 1

0

cos θ ds

= −α∫ 1

0

xds− ψ1(0)

∫ 1

0

cos θ ds

= −ψ1(0)

∫ 1

0

cos θ ds

∴ ψ1(0) = 0.

Thus, we rewrite (11.9.3) as

θ = ψ2 (11.9.4)

ψ2 = −α sin θ

ψ2(0) = ψ2(1) = 0

We have

x(t) =

∫ t

0

sin θ ds = − 1

α

∫ t

0

θ ds = − 1

α[θ(t)− θ(0)],

and from the fact θ(0) = ψ2(0) = 0, we have

x(t) = − 1

αθ(t). (11.9.5)

Since we require x2(t) ≤ 0.0025 it follows that

|θ(t)| ≤ α

20.

Now, we go back to the cost

J =1

2

∫ 1

0

u2dt+ α

∫ 1

0

cos θ dt.

Since θ = u and u = ψ2, it follows that

J =1

2

∫ 1

0

ψ22dt+ α

∫ 1

0

cos θ dt.


Note that

ψ2 = −α sin θ

ψ2θ = −α(sin θ)θ∴ ψ2ψ2 = −α(sin θ)θ

∴d

dt

(1

2ψ22 − α cos θ

)= 0

∴1

2ψ22 = α cos θ − α cos θ(0).

Thus, the cost becomes

J = 2α

∫ 1

0

cos θ dt− α cos θ(0)

where

θ + α sin θ = 0 (11.9.6)

θ(0) = θ(1) = 0, |θ| ≤ α

20

Corresponding to the equation,

θ + α sin θ = 0.

We have the phase portrait shown in Fig. 11.1.

FIGURE 11.1

In (11.9.6) what we have is the familiar pendulum equation. For the undampedpendulum any periodic orbit must intersect the θ-axis of the phase portraitat two points, say (−b, 0) and (b, 0) and the period T is given by

T =4√α

∫ π/2

0

dξ√1− sin2

(b2

)sin2 ξ

(11.9.7)


In the problem at hand, θ(0) = θ(1) = 0. Thus,

T =2

k, k = 1, 2, 3, . . . (11.9.8)

We verified previously that |θ(t)| ≤ α/20. Thus, it takes at least 20/α unitsof time to traverse a closed orbit. Thus,

T > 20/α (11.9.9)

From (11.9.8) we see that (11.9.9) is impossible if 0 < α ≤ π2. From (11.9.6),if 0 < α ≤ π2, then θ ≡ 0. Thus, in this case the control policy u ≡ 0and the corresponding trajectory x ≡ 0. From (11.9.8) and (11.9.9) the sameconclusion is reached if α ≤ 10. Thus, for a nontrivial control policy we shouldinsist that α > 10.

From (11.9.7) and (11.9.8) we have

2

k=

4√α

∫ π/2

0

dξ√1− sin2 b

2 sin2 ξ(11.9.10)

Since4√α

∫ π/2

0

dξ√1− sin2 b

2 sin2 ξ≥ 4√

α

π

2=

2π√α,

from (11.9.10) we have2

k≥ 2π√

α

α ≥ k2π2 (11.9.11)

Thus, given α and k = 0, 1, 2, . . . such that k2π2 ≤ α we use (11.9.10) toobtain a corresponding b. This procedure picks θ and hence a control policyu = θ.

Chapter 12

Hamilton-Jacobi Theory

12.1 Introduction

In Section 6.2 we used dynamic programming to derive the nonlinear par-tial differential equation (12.1.1) for the value function associated with an op-timal control problem. This partial differential equation is called a Hamilton-Jacobi-Bellman (HJB) equation, also Bellman’s equation. Typically, the valuefunction W is not smooth, and (12.1.1) must be understood to hold in someweaker sense. In particular, under suitable assumptionsW satisfies (12.1.1) inthe Crandall-Lions viscosity solution sense (Section 12.5). Section 12.6 givesan alternate characterization (12.6.2) of the value function using lower Diniderivatives. This provides a control theoretic proof of uniqueness of viscositysolutions to the HJB equation with given boundary conditions.

From Section 6.2, we recall the optimal control problem formulation.


J(φ, u) = g(t1, φ(t1)) +

∫ t1

τ



dx

dt= f(t, x, u(t)),

control constraints u(t) ∈ Ω(t), and end conditions

(t0, φ(t0)) = (τ, ξ) (t1, φ(t1)) ∈ J .

In Section 6.2 we assumed that at each point (τ, ξ) in a regionR of (t, x)-spacethe above control problem has a unique solution. Under certain assumptionson the data of the problem and under the assumption that the value functionW is C(1) we showed that W satisfies

Wτ (τ, ξ) = maxz∈Ω(τ)

[−f0(τ, ξ, z)− 〈Wξ(τ, ξ), f(τ, ξ, z)〉]. (12.1.1)

In this chapter we assume that (τ, ξ) ∈ R0, where R0 is a region with proper-ties described in part (vii) of Assumption 12.2.1. If we now denote a generic

337


point in R by (t, x) rather than by (τ, ξ), set

H(t, x, z, q0, q) = q0f0(t, x, z) + 〈q, f(t, x, z)〉

and setH(t, x, q) = sup

z∈Ω(t)

H(t, x, z,−1, q),

then we may write (12.1.1) as

−Wt(t, x) +H(t, x,−Wx(t, x)) = 0. (12.1.2)

In this chapter we shall give conditions under which W is continuous andLipschitz continuous. We shall then show that if W is Lipschitz continuous,then W satisfies (12.1.2) for a generalized notion of solution. We shall alsoconsider the problem of determining an optimal synthesis and shall obtainthe maximum principle in some cases.

12.2 Problem Formulation and Assumptions

We consider Problem 12.1.1 and assume the following.

Assumption 12.2.1. (i) The function f = (f0, f) = (f0, f1, . . . , fn) iscontinuous on I ×R

n ×U , where I is a real compact interval [0, T ] andU is an interval in R

m.

(ii) There exists a function β in L2[I] such that f0(t, x, z) ≥ β(t) for a.e.t ∈ I, all x ∈ R

n, and all z ∈ U .

(iii) For each compact set X0 ⊂ Rn there exists a function K in L2[I] such

that for all x and x′ in X0, all t in I, and all z in U

|f(t, x, z)− f(t, x′, z)| ≤ K(t)|x− x′|. (12.2.1)

(iv) There exists a function M in L2[I] such that for all x in Rn, all t in I,

and all z in ∪Ω(t) : t ∈ I

|f(t, x, z)| ≤M(t)(|x|+ 1). (12.2.2)

(v) The terminal set J is an r-dimensional manifold, 0 ≤ r ≤ n of classC(1) in R× R

n with supt1 : (t1, x1) ∈ J ≤ T .

(vi) The function g is real valued and defined on an open set in R×Rn that

contains J .

Hamilton-Jacobi Theory 339

(vii) There exists an open connected set R0 ⊂ I × Rn with the following

properties.

(a) The terminal set J is a subset of the boundary of R0.

(b) At each point (t, x) in R0, let A(t, x) denote the set of admissiblecontrols u for Problem 12.1.1 with initial point (t, x) such that alltrajectories corresponding to u are admissible. Then A(t, x) is notempty.

(c) Problem 12.1.1 with initial point (t, x) in R0 has a solution (φ, u),not necessarily unique, such that (r, φ(r)) ∈ R0 for t ≤ r < t1 and(t1, φ(t1)) ∈ J . In the rest of this chapter, when we say solution toProblem 12.1.1 we mean a solution with property (c).

Later we will specialize to consider terminal sets J , which are relativelyopen subsets of a hyperplane t1 = T (Section 12.3), including the casewhen J is the entire hyperplane (Section 12.6).

Remark 12.2.2. In Chapter 4 and 5 existence theorems were proved thatensured the existence of a solution (φ, u). These theorems required the setsQ+(t, x) defined in Section 5.4 to be convex. To overcome this limitation, thenotion of relaxed controls and the relaxed problem were introduced. As inearlier chapters, we denote a relaxed control by µ and a relaxed trajectory byψ.

The relaxed problem corresponding to Problem 12.1.1 is the following:


J(ψ, µ) = g(t1, ψ(t1)) +

∫ t1

τ

f0(t, ψ(t), µt)dt

subject to the state equation

dψ

dt= f(t, ψ(t), µt),

control constraint µt ∈ Ω(t) and end conditions

(t0, ψ(t0)) = (τ, ξ) (t1, ψ(t1)) ∈ J .

Remark 12.2.3. Statements (i) to (vii-a) in Assumption 12.2.1 hold for therelaxed problem, and therefore can be taken as assumptions for the relaxedproblem. Assumptions for the relaxed problem corresponding to (vii-b) and(vii-c) are obtained by replacing A(t, x) by Ar(t, x), by replacing u by µ, andreplacing φ by ψ. Assumptions (i) to (vii-c) for the relaxed problem will becalled Assumption 12.2.1-r.


Remark 12.2.4. In some of the existence theorems in Chapter 5 for problemswith non-compact constraints, the mapping Q+

r is required to have the weakCesari property. We showed that if f is of slower growth than f0 and if thefunction identically one is of slower growth than f0, then the weak Cesariproperty holds. If we assume that these growth conditions hold, then thereexists a positive constant A such that |f(t, x, z)| ≤ A+ |f0(t, x, z)|. Hence we

need only require (12.2.2) to hold for f0 for it to hold for f .

Remark 12.2.5. If all the constraint sets Ω(t) are compact and the mappingΩ is u.s.c.i, then by Lemma 3.3.11 for any compact set X0 ⊆ X , the set

∆ = (t, x, z) : t ∈ [0, T ], x ∈ X0, z ∈ Ω(t) (12.2.3)

is compact. It then follows from the continuity of f that (12.2.2) holds withM a constant, for all (t, x, z) in ∆.

Remark 12.2.6. In the HJB equation for the relaxed problem, H in (12.1.2)is replaced by Hr where

Hr(t, x, q) = supµ[〈−f0(t, x, ·), µ〉+ 〈q · f(t, x, ·), µ〉] (12.2.4)

with µ any probability measure on Ω(t). It is easy to show that H = Hr. Thus,the ordinary and relaxed control problems have the same HJB equation.

12.3 Continuity of the Value Function

We define the value function W on R0 by

W (τ, ξ) = minJ(ψ, µ) : (ψ, µ) ∈ Ar(τ, ξ).

We are justified in writing “min” by virtue of (vii-c) of Assumption 12.2.1-r.We first establish Bellman’s “Principle of Optimality.”

Theorem 12.3.1 (Principle of Optimality). Let the relaxed problem withinitial point (τ, ξ) have a solution (ψ, µ) with terminal time t1. If (t, x) is apoint on the trajectory ψ, then (ψ, µ) restricted to [t, t1] is optimal for therelaxed problem with initial point (t, x). If (ψ, µ) is a control trajectory pairwith µ in Ar(τ, ξ) and with terminal time t1, then for any τ ≤ t ≤ t1

W (τ, ξ) ≤∫ t

τ

f0(s, ψ(s), µs)ds+W (t, ψ(t)). (12.3.1)

Equality holds if and only if (ψ, µ) is optimal.


Proof. We have that

W (τ, ξ) =

∫ t

τ

f0(s, ψ(s), µs)ds+

∫ t1

t

f0(s, ψ(s), µs)ds+g(t1, ψ(t1)). (12.3.2)

If (ψ, µ) were not optimal for the problem with initial point (t, ψ(t)), then

W (t, ψ(t) <

∫ t1

t

f0(s, ψ(s), µs)ds+ g(t1, ψ(t1)). (12.3.3)

Let (ψ1, µ1) be optimal for the problem with initial point (t, ψ(t)) and let(t1, ψ1(t1)) be the terminal point of (ψ1, µ1). Define a control µ on [τ, t1] by

µs = µs on [τ, t] µs = µ1s on [t, t1].

Let ψ be the trajectory corresponding to µ. Then by virtue of (12.2.1) andthe uniqueness theorem for solutions of ordinary differential equations, ψ(s) =ψ(s) on [τ, t]. Then µ ∈ Ar(τ, ξ) and

J(ψ, µ) =

∫ t

τ

f0(s, ψ(s), µs)ds+W (t, ψ(t)).

Combining this with (12.3.3) and (12.3.2) gives J(ψ, µ) < W (τ, ξ), whichcontradicts the definition of W and establishes the first conclusion. Inequality(12.3.1) and the statement about equality in (12.3.1) are obvious.

Theorem 12.3.2. Let Assumption 12.2.1-r hold. Let g be continuous. Let Jbe a relatively open set in the hyperplane t1 = T . Let the sets Ar(t, x) dependonly on t; that is, Ar(t, x) = Ar(t). Then the value function W is continu-ous on R0. If g is locally Lipschitz continuous, then W is locally Lipschitzcontinuous.

Remark 12.3.3. The value function is continuous or Lipschitz continuouswhen the terminal set has a different structure from the one assumed here.See [5], Section 2.10 of [35], [94] and the references therein.

Proof. To simplify the proof we transform the problem to the Mayer form,as in Section 2.4. Assumption 12.2.1-r is valid for the Mayer form. The valueW (t, x) of the Mayer problem is related to the value W (t, x) of the Bolzaproblem by the relation W (t, x) = W (t, x) + x0, where x = (x0, x). Hence if acontinuity property holds for the value in Mayer form, it holds for the value inBolza form and vice versa. We henceforth assume the problem to be in Mayerform.

We shall need the following result.

Lemma 12.3.4. Let µ ∈ Ar(τ) be an admissible relaxed control and letψ(·, τ, x) and ψ(·, τ, x′) be corresponding trajectories with initial conditions(τ, x) and (τ, x′). Then there exists a positive constant A such that for allτ ≤ t ≤ T

|ψ(t, τ, x)− ψ(t, τ, x′)| ≤ A|x− x′|. (12.3.4)


Proof. To simplify notation we suppress the dependence on τ and write ψ(·, x)for ψ(·, τ, x) and ψ(·, x′) for ψ(·, τ, x′). From (12.2.1) and Gronwalls’ Lemmawe get that for τ ≤ t ≤ T

|ψ(t, x)− ψ(t, x′)| ≤ |x− x′|+∫ t

τ

|(f(s, ψ(s, x), µs)− f(s, ψ(s, x′), µs)|ds

≤ |x− x′|+∫ t

τ

K(s)|ψ(s, x) − ψ(s, x′)|ds

Thus,

|ψ(t, x)− ψ(t, x′)| ≤ |x− x′| exp∫ t

τ

K(s)ds ≤ |x− x′|A,

where A = exp∫IK(s)ds.

We return to the proof of the theorem. Let (τ, ξ) be a point in R0 and let(ψ, µ) = (ψ(·, τ, ξ, µ), µ(τ, ξ)) be an optimal relaxed pair for Problem 12.2.1.Since the problem is in Mayer form

W (τ, ξ) = g(T, ψ(T, τ, ξ)) = g(T, ψ(T )). (12.3.5)

Now let (τ ′, ξ′) be another point in R0 with τ ′ > τ . If τ ′ < τ , interchange theroles of (τ, ξ) and (τ ′, ξ′). Then

|W (τ, ξ)−W (τ ′, ξ′)| ≤ |W (τ, ξ)−W (τ ′, ψ(τ ′, τ, ξ))|+ |W (τ ′, ψ(τ ′, τ, ξ))−W (τ ′, ξ′)|.

It follows from (12.3.5) and the Principle of Optimality (Theorem 12.3.1) thatthe first term on the right is zero. Hence

|W (τ, ξ)−W (τ ′, ξ′)| ≤ |W (τ ′, ψ(τ ′, τ, ξ))−W (τ ′, ξ′)|. (12.3.6)

We now estimate the term on the right in (12.3.6). To simplify notation,let x′ = ψ(τ ′, τ, ξ). Let ψ = ψ(·, τ ′, ξ′, µ) denote the trajectory on the interval[τ ′, T ] with initial point (τ ′, ξ′) corresponding to the optimal µ = µ(·, τ ′, x′)for the problem with initial point (τ ′, x′). Then since µ ∈ Ar(τ

′), the pair(ψ, µ) is admissible for the problem with initial point (τ ′, ξ′). Then

W (τ ′, ξ′)−W (τ ′, x′) ≤ J(ψ, µ)−W (τ ′, x′)

= g(T, ψ(T ))− g(T, ψ(T )),

the equality following from (12.3.5) and Theorem 12.3.1. Let ε > 0 be given.It then follows from (12.3.4) with x = ξ′ and the continuity of g that thereexists a δ > 0 such that if |ξ′ − x′| < δ, then

|g(T, ψ(T ))− g(T, ψ(T )| < ε,


and soW (τ ′, ξ′)−W (τ ′, x′) < ε. (12.3.7)

If g is Lipschitz continuous with Lipschitz constant L, we get that

W (τ ′, ξ′)−W (τ ′, x′) ≤ LA|ξ′ − x′|. (12.3.8)

A similar argument applied to W (τ ′, x′)−W (τ ′, ξ) gives (12.3.7) and (12.3.8)with the left side replaced byW (τ ′, x′)−W (τ ′, ξ′). Hence we have shown that

limx′→ξ′

W (τ ′, x′) = W (τ ′, ξ′), (12.3.9)

and that if g is Lipschitz continuous

|W (τ ′, ξ′)−W (τ ′, x′)| ≤ LA|ξ′ − x′|. (12.3.10)

We conclude the proof by relating x′ − ξ′ to ξ − ξ′. Recall that x′ =ψ(τ ′, τ, ξ). Hence

|x′ − ξ′| ≤ |ξ − ξ′|+∫ τ ′

τ

|f(t, ψ(t, τ, ξ), µt)|dt.

It follows from (12.2.2) and Lemma 4.3.14 that for all ξ, ξ′ in a compactset, |ψ(t)| is bounded by a positive constant B, which depends on the com-pact set. Since f is continuous there exists a positive constant C such that|f(t, ψ(t, τ, ξ), µt)| ≤ C. Therefore,

|x′ − ξ′| ≤ |ξ − ξ′|+ C|t− t′|. (12.3.11)


x′ → ξ′ as (τ, ξ) → (τ ′, ξ′). (12.3.12)

The continuity of W now follows from (12.3.9) and (12.3.12); the Lipschitzcontinuity follows from (12.3.10) and (12.3.11).

Corollary 12.3.5. Let every point of J be the terminal point of an optimaltrajectory with initial point in R0. For each such initial point, let the optimaltrajectory be unique. For (T, x1) ∈ J let

W (T, x1) = g(T, x1) = g(T, ψ(T, t′, x′)), (12.3.13)

where ψ( , t′, x′) is the unique optimal trajectory with initial point (t′, x′) inR0 terminating at x1. Then W is continuous on R0 ∪ J .

Proof. Let (t, x) in R0 tend to (T, x1) in J . Then to prove the corollary wemust show that

W (t, x) →W (T, x1) = g(T, x1) = g(T, ψ(T )), (12.3.14)


where ψ( ) = ψ( , t′, x′) and x1 = ψ(T ). Let ψ( , t, x) denote the optimaltrajectory with initial point (t, x) that terminates at J and let µ be theassociated optimal control. Then

W (t, x) = g(T, ψ(T, t, x)), (12.3.15)

and

ψ(T, t, x) = x+

∫ T

t

f(s, ψ(s, t, x), µs)ds.

From (12.2.2) we get that the absolute value of the integral does not exceed∫ T

t

M(s)(|ψ(s, t, x)|+ 1)ds. (12.3.16)

Since we are considering points (t, x) approaching a point (T, x1) in J , wemay suppose that all points (t, x) are in a compact set K containing (T, x1).It then follows from Lemma 4.3.14 that there exists a positive constant B suchthat |ψ(s, t, x)| ≤ B for all (t, x) ∈ K and t ≤ s ≤ T . Therefore, the integralin (12.3.16) tends to zero as t→ T . By hypothesis x→ x1. Hence

ψ(T, t, x) → x1 as (t, x) → (T, x1).

Since g is continuous we have shown that g(T, ψ(T, t, x)) → g(T, x1) as(t, x) → (T, x1). Hence, by (12.3.13) and (12.3.15) we have that W (t, x) →W (T, x1), and the corollary is established.

Remark 12.3.6. The argument also shows that if KT is a compact set con-tained in J , then the convergence W (t, x) →W (T, x1) is uniform on KT .

12.4 The Lower Dini Derivate Necessary Condition

In this section we shall develop a necessary condition that a Lipschitzcontinuous value function satisfies. This necessary condition involves the lowerDini directional derivate, which we now define.

Definition 12.4.1. Let L be a real valued function defined on an open set inR×R

n. The lower Dini derivate of L at the point (t, x) in the direction (1, h),where h ∈ R

n, is denoted by D−L(t, x; 1, h) and is defined by

D−L(t, x; 1, h) = lim infδ↓0

[L(t+ δ, x+ δh)− L(t, x)]δ−1. (12.4.1)

The upper Dini derivate of L at (t, x) in the direction (1, h) is denoted byD+L(t, x; 1, h) and is defined as in (12.4.1) with lim inf replaced by lim sup.The function L is said to have a directional derivative at (t, x) in the direction(1, h) if D+L(t, x; 1, h) = D−L(t, x; 1, h). We denote the directional derivativeby DL(t, x; 1, h).


Remark 12.4.2. If L is differentiable at (t, x), then DL(t, x; 1, h) exists forevery h ∈ R

n and is given by

DL(t, x; 1, h) = Lt(t, x) + 〈Lx(t, x), h〉.

We shall be concerned with the value function W for the relaxed problem,Problem 12.2.1. We shall take the formulation of the relaxed problem to bethat given in Section 5.4. Recall that in this formulation the control variableis z = (ζ, π), where ζ = (z1, . . . , zn+1), with zi ∈ R

m and π = (π1, . . . , πn+1),

with πi ∈ R. The constraint set Ω is given by Ω = Γn+1 × (∏

n+1 Ω), whereΩ is a constraint set in R

m,∏

n+1 denotes the (n+1)-fold Cartesian productand

Γn+1 =

π : π = (π1, . . . , πn+1), πi ≥ 0,

n+1∑

i=1

πi = 1

.

To simplify notation we shall omit the subscript r and write (f0, f) insteadof (f0

r , fr). We shall, as in (5.4.1), write the control function as v, and shallwrite Q and Q+ for Qr and Q+

r , respectively. In this context,

Q(t, x) = y = (y0, y) : y = f(t, x, z) : z ∈ Ω(t, x)Q+(t, x) = y = (y0, y) : y0 ≥ f0(t, x, z), y = f(t, x, z), z ∈ Ω(t, x).

Since the problem is a relaxed problem, both Q(t, x) and Q+(t, x) are convex.For problems in Mayer form with initial point (τ, ξ) we shall consider

D−W (τ, ξ; 1, h) = lim infδ↓0

[W (τ + δ, ξ + δh)−W (τ, ξ)]δ−1 (12.4.2)

for h ∈ Q(τ, ξ). If we transform a problem in Lagrange or Bolza form to Mayer

form, then an initial point (τ, ξ) = (τ, ξ0, ξ) for the transformed problem

always has ξ0 = 0. Therefore, W (τ, ξ) = W (τ, ξ). Thus, for the problem inBolza or Lagrange form we shall consider

D−W (τ, ξ; 1, h) = lim infδ↓0

[δh0 +W (τ + δ, ξ + δh)−W (τ, ξ)]δ−1,

for h = (h0, h) ∈ Q(τ, ξ).The principal result of this section is the following theorem:

Theorem 12.4.3. Let Assumption 12.2.1-r hold, with the function M in(12.2.2) in L∞[I]. Let the value function W , defined on R0, be Lipschitzcontinuous on compact subsets of R0. Let the mapping Q+ possess the Cesariproperty at all points of R0. For each τ in [0, T ] and each z in Ω(τ) let thereexist a δ0 and a control v defined on [τ, τ + δ0] such that limt→τ+0 v(t) = z.

Let h = (h0, h), with h0 ∈ R and h ∈ Rn. Then for each (τ, ξ) in R0

min[D−W (τ, ξ; 1, h) : h ∈ Q(τ, ξ)] = 0. (12.4.3)


Remark 12.4.4. If Ω is a constant map, then for z ∈ Ω, the control v canbe v(t) = z, for τ ≤ t ≤ τ + δ.

Proof. As in the proof of Theorem 12.3.2 we shall assume that the problemis in Mayer form. Then (12.4.3) becomes

min[D−W (τ, ξ; 1, h) : h ∈ Q(τ, ξ)] = 0.

The assumption that Q+ has the Cesari property at each point of R0 becomesthe assumption that Q does.

We first show that

inf[D−W (τ, ξ; 1, h) : h ∈ Q(τ, ξ)] ≥ 0. (12.4.4)

Let h ∈ Q(τ, ξ). Then there exists a z in Ω(τ) such that h = f(τ, ξ, z).Also, there exists a δ0 > 0 and a control v defined on [τ, τ + δ0] with v(τ) = zand that is continuous from the right at t = τ . The control can be extendedto [τ, T ] and we denote the extended control also by v. Let ψ denote thetrajectory corresponding to v and having initial point (τ, ξ). Then for δ > 0

ψ(τ + δ) = ξ +

∫ τ+δ

τ

f(s, ψ(s), v(s))ds (12.4.5)

= ξ +

∫ τ+δ

τ

[f(τ, ξ, z) + o(1)]ds

= ξ + δf(τ, ξ, z) + o(δ),

where o(δ) is as δ → 0.By the Principle of Optimality ((12.3.1) with f0 = 0),

[W (τ + δ, ψ(τ + δ))−W (τ, ξ)]δ−1 ≥ 0.

If we substitute the rightmost member of (12.4.5) into this inequality and usethe Lipschitz continuity of W , we get that

[W (τ + δ, ξ + δh)−W (τ, ξ)]δ−1 + o(1) ≥ 0,

where h = f(τ, ξ, z). From this, (12.4.4) follows.

We next show that there exists an h∗ ∈ Q(τ, ξ) such thatD−W (τ, ξ; 1, h∗) ≤ 0. This in conjunction with (12.4.4) will establish thetheorem.

Let ψ now denote an optimal trajectory for Problem 12.2.1 with initialpoint (τ, ξ). Then

ψ(τ + δ) = ξ +

∫ τ+δ

τ

ψ′(s)ds,

with ψ′(s) ∈ Q(s, ψ(s)) for almost all t ∈ [τ, T ]. Let Nε(τ, ξ) = (t, x) : (t, x) ∈


R0, |(t, x) − (τ, ξ)| < ε. Since ψ is continuous, given an ε′ > 0, there existsa δ(ε′) with 0 < δ(ε′) < ε′ such that if τ ≤ s ≤ τ + δ, then

ψ′(s) ∈ Q(Nε(τ, ξ)) a.e., (12.4.6)

where ε = δ(ε′). Let Kε denote the set of points in [τ, τ + δ] at which theinclusion (12.4.6) holds. Then the Lebesgue measure of Kε equals δ. Thus,

∫ τ+δ

τ

ψ′(s)ds = δ

∫ τ+δ

τ

ψ′(s)

(ds

δ

)= δ

∫

Kε

ψ′(s)

(ds

δ

).


cl co ψ′(s) : s ∈ Kε ⊆ cl co Q(Nε(τ, ξ)).

From Lemma 3.2.9 we get that

cl co ψ′(s) : s ∈ Kε = cl

∫

Kε

ψ′(s)dµ : µ ∈ P(Kδ)

,

where P(Kδ) is the set of probability measures on Kδ. Hence

∫

Kε

ψ′(s)

(ds

δ

)∈ cl co Q(Nε(τ, ξ)).

Let

hδ ≡∫ τ+δ

τ

ψ′(s)

(ds

δ

).

We have shown that for each ε′ > 0 there exists a δ(ε′), with 0 < δ(ε′) < ε′,and a point hδ such that

ψ(τ + δ) = ξ + δhδ hδ ∈ cl co (Q(Nε(τ, ξ))), (12.4.7)

where ε = δ(ε′). We also have that

|hδ| ≤ δ−1

∫ τ+δ

τ

|ψ′(s)|ds ≤ δ−1

∫ τ+δ

τ

|(1 + ψ(s))|M(s)ds,

the last inequality following from (12.2.2). Since M is in L∞[0, T ] and |ψ(s)|is bounded on [τ, T ], we get that there exists a positive constant A such that|hδ| ≤ A for all δ = δ(ε).

Let ε′k be a decreasing sequence of positive terms with ε′k → 0. We cantake the corresponding sequence δ(ε′k) also to be decreasing. Then the sequenceεk is also decreasing, and εk → 0. Let hk = hδ(εk) be the correspondingsequence given by (12.4.7). Thus, hk ∈ cl co (Q(Nεk(τ, ξ))). Since |hk| ≤ Afor all k, there exists a subsequence, that we relabel as hk, that converges toan element h∗ in R

n. Since for 0 < ρ < ρ′

Q(Nρ(τ, ξ)) ⊂ Q(Nρ′(τ, ξ)), (12.4.8)


it follows that for all k,

cl co (Q(Nεk+1(τ, ξ))) ⊆ cl co (Q(Nεk(τ, ξ))).

Thus, for fixed k0 we have that for all k ≥ k0

hk ∈ cl co (Q(Nεk0(τ, ξ))).

Hence h∗ ∈ cl co (Q(Nεk0(τ, ξ))). But k0 is an arbitrary positive integer, so

for all kh∗ ∈ cl co (Q(Nεk(τ, ξ))). (12.4.9)

It follows from (12.4.8) that

⋂

k

cl co (Q(Nεk(τ, ξ))) =⋂

δ>0

cl co (Q(Nδ(τ, ξ))).

From this, from (12.4.9), and from the fact that Q has the Cesari property weget that h∗ ∈ Q(τ, ξ).

We return to our sequences δk and hk. From the definition of h∗ andfrom (12.4.7) we get that

ψ(τ + δk) = ξ + δkh∗ + o(δk). (12.4.10)

From the Principle of Optimality we have that

[W (τ + δk, ψ(τ + δk))−W (τ, ξ)]δ−1k = 0.

Substituting (12.4.10) into this equation and using the Lipschitz continuity ofW , we get that

limk→∞

[W (τ + δk, ξ + δkh∗)−W (τ, ξ)]δ−1

k = 0.

Hencelim infδ→0+

[W (τ + δ, ξ + δh∗)−W (τ, ξ)]δ−1 ≤ 0.

Thus,D−W (τ, ξ; 1, h∗) ≤ 0. This proves the theorem for the problem in Mayerform. Note that our argument shows that D−W (τ, ξ; 1, h∗) = 0, so we arejustified in writing min in (12.4.2).

Remark 12.4.5. In Theorem 12.4.3 we do not assume that the sets Ω(t) arecompact. Instead we assume that the sets Q(t, x) possess the Cesari property.

Growth conditions and a regularity condition on the constraint mapping Ωguaranteeing that the Cesari property holds are given in Lemma 5.4.6. If themapping Ω is u.s.c.i. and the sets Ω(t) are compact, then the sets Q(t, x)possess the Cesari property. (See the proof of Theorem 5.6.1.)


12.5 The Value as Viscosity Solution

The definition of viscosity solution of a nonlinear partial differential equa-tion was first given by Crandall and Lions in [28]. For subsequent develop-ments in the theory of viscosity solutions the reader is referred to Crandall,Ishii, and Lions [29], Bardi and Capuzzo-Dolcetta [5], and Fleming and Soner[35]. We shall confine our attention to Eq. (12.1.2), which for the reader’sconvenience we present again here, rather than consider the general nonlinearpartial differential equation.

LetH(t, x, z, q0, q) = q0f0(t, x, z) + 〈q, f(t, x, z)〉

and letH(t, x, q) = sup

z∈Ω(t)

H(t, x, z,−1, q).

We consider the nonlinear partial differential equation

−Vt(t, x) +H(t, x,−Vx(t, x)) = 0. (12.5.1)

This equation is a Hamilton-Jacobi-Bellman equation. We consider this equa-tion on R0.

Definition 12.5.1. A continuous function V on R0 is a viscosity subsolutionof (12.5.1) on R0 if for each function ω in C(1)(R0)

−ωt(t, x) +H(t, x,−ωx(t, x)) ≤ 0

at each point (t, x) in R0 at which V − ω has a local maximum.

A continuous function V on R0 is a viscosity supersolution of (12.5.1) onR0 if for each function ω in C(1)(R0)

−ωt(t, x) +H(t, x,−ωx(t, x)) ≥ 0

at each point (t, x) in R0 at which V − ω has a local minimum.A continuous function V on R0 is a viscosity solution on R0 if it is both a

viscosity subsolution and a viscosity supersolution.The functions ω are called test functions.

Remark 12.5.2. If V is a viscosity solution of (12.5.1) and the partial deriva-tives of V exist at a point (t, x) in R0, then V satisfies (12.5.1) in the usualsense at (t, x). To see this, let ω be a test function such that V − ω has alocal maximum at (t, x). Then Vt(t, x) = ωt(t, x) and Vx(t, x) = ωx(t, x).Since W is a viscosity solution of (12.5.1), it is a subsolution, and so−ωt(t, x) + H(t, x,−ωx(t, x)) ≤ 0. Hence −Vt(t, x) + H(t, x,−Vx(t, x)) ≤ 0.By considering a test function ω such the V −ω has a local minimum at (t, x),we get the reverse inequality. Thus, V satisfies (12.5.1) at (t, x).


If V is a solution of (12.5.1) in the ordinary sense, then V is also a viscositysolution of (12.5.1) on R0. To see this, let (t, x) be a point in R0 and let ω bea test function such that V − ω has a relative maximum (minimum at (t, x)).Then again Vt = ωt and Vx = ωx. Since V is an ordinary C(1) solution of(12.5.1), then

0 = −Vt(t, x) +H(t, x,−Vx(t, x)) = −ωt(t, x) +H(t, x, ωx(t, x)).

Thus, V is a viscosity solution.

Theorem 12.5.3. Let the hypotheses of Theorem 12.4.3 hold. Then the valuefunction W is a viscosity solution on R0 of the Hamilton-Jacobi equation(12.5.1).

Theorem 12.5.3 is an immediate consequence of Theorem 12.4.3 and thefollowing lemma.

Lemma 12.5.4. Let V be a continuous function such that at each point (τ, ξ)in R0

inf[D−(V (τ, ξ; 1, h) : h ∈ Q(τ, ξ)] = 0. (12.5.2)

Then V is a viscosity solution of (12.5.1).

Proof. Assume that the problem is in Mayer form.We first show that (12.5.2) implies that V is a subsolution of (12.5.1). Let

ω be a test function such that V −ω has a local maximum at (τ, ξ). Then forall (t, x) sufficiently close to (τ, ξ)

V (t, x) − ω(t, x) ≤ V (τ, ξ)− ω(τ, ξ).

Hence for fixed h in Q(τ, ξ) and sufficiently small δ > 0,

[V (τ + δ, ξ + δh)− V (τ, ξ)]δ−1 ≤ [ω(τ + δ, ξ + δh)− ω(τ, ξ)]δ−1.

Letting δ → 0 gives, for fixed h ∈ Q(τ, ξ),

D−V (τ, ξ; 1, h) ≤ ωt(τ, ξ) + 〈ωx(τ, ξ), h〉= −[−ωt(τ, ξ) + 〈−ωx(τ, ξ), h〉].

If we now take the infimum over h ∈ Q(τ, ξ) and use (12.5.2) we get that

0 ≤ inf−[−ωt(τ, ξ) + 〈−ωx(τ, ξ), h〉] : h ∈ Q(τ, ξ)= − sup[−ωt(τ, ξ) + 〈−ωx(τ, ξ), h〉] : h ∈ Q(τ, ξ)= ωt(τ, ξ)−H(τ, ξ,−ωx(τ, ξ)),

the last equality following from the definition of H. Hence

−ωt(τ, ξ) +H(τ, ξ,−ωx(τ, ξ)) ≤ 0,


and so V is a subsolution.We now show that V is a supersolution. Let ω be a test function such

that V − ω has a local minimum at (τ, ξ). Then for fixed h ∈ Q(τ, ξ) and δsufficiently small,

V (τ + δ, ξ + δh)− ω(τ + δ, ξ + δh) ≥ V (τ, ξ) − ω(τ, ξ).

HenceD−V (τ, ξ; 1, h) ≥ ωt(τ, ξ) + 〈ωx(τ, ξ), h〉,

and so−ωt(τ, ξ) + 〈−ωx(τ, ξ), h〉 ≥ −D−V (τ, ξ; 1, h).

If we now take the supremum over h ∈ Q(t, x), we get that

−ωt(τ, ξ) +H(τ, ξ,−ωx(τ, ξ)) ≥ suph[−D−V (τ, ξ; 1, h)]

= − infh[D−V (τ, ξ; 1, h)] = 0.

Hence V is a supersolution, and the lemma is proved.

Lemma 12.5.4 and Theorem 12.4.3 justify calling a continuous function Vthat satisfies (12.5.2) a generalized solution of the Hamilton-Jacobi equation.

Definition 12.5.5. A continuous function V defined on R0 that satisfies(12.5.2) will be called a lower Dini solution of the Hamilton-Jacobi equation.

Lemma 12.5.4 has a partial converse.

Lemma 12.5.6. Let V be a Lipschitz continuous viscosity solution of(12.5.1). Let f be continuous, and let the constraint mapping Ω be u.s.c.i

and such that each set Ω(t) is compact. Then V is a lower Dini solution.

Proof. We again take the problem to be in Mayer form. Let (t, x) be an arbi-trary point in R0. We first show that for all h ∈ Q(t, x)

D−V (t, x; 1, h) ≥ 0. (12.5.3)

Fix h ∈ Q(t, x). There exists a decreasing sequence of positive numbers δksuch that δk → 0 and

D−V (t, x; 1, h) = limk→∞

[V (t+ δk, x+ δkh)− V (t, x)]δ−1k . (12.5.4)

Since V is Lipschitz continuous, by Rademacher’s theorem, V is differentiablealmost everywhere. Hence there exists a sequence (tk, xk) such that V isdifferentiable at each point (tk, xk) and

|tk − t| ≤ δ2k |x− xk| < δ2k. (12.5.5)


Then

V (t+ δk, x+ δkh)− V (t, x) = V (t+ δk, x+ δkh)− V (tk + δk, xk + δkh)+ V (tk + δk, xk + δkh)− V (tk, xk)+ V (tk, xk)− V (t, x)≡ Ak +Bk + Ck

From the Lipschitz continuity of V and (12.5.5) we have that

(Ak + Ck) = O(|t− tk|) +O(|x − xk|) = O(δ2k)

Hencelimk→∞

(Ak + Ck)δ−1k = 0. (12.5.6)

Since V is differentiable at (tk, xk),

Bk = [Vt(tk, xk) + 〈Vx(tk, xk), h〉]δk + o(δk(1 + h))

≥ [Vt(tk, xk) + infh∈Q(t,x)

〈Vx(tk, xk), h〉]δk + o(δk).

Hence

Bk ≥ −[−Vt(tk, xk) + suph∈Q(t,x)

〈−Vx(tk, xk), h〉]δk + o(δk).

At points of differentiability a viscosity solution satisfies (12.5.1) in the ordi-nary sense. Hence

lim infk→∞

Bkδ−1k ≥ 0. (12.5.7)

From (12.5.4), (12.5.6), and (12.5.7) we get that

D−V (t, x; 1, h) = limk→∞

(Ak + Bk + Ck)δ−1k

≥ limk→∞

(Ak + Ck)δ−1k + lim inf

k→∞Bkδ

−1k ≥ 0,

which establishes (12.5.3).To complete the proof it suffices to show that there exists an h0 ∈ Q(t, x)

such thatD−V (t, x; 1, h0) ≤ 0, (12.5.8)

for this in conjunction with (12.5.3) will establish (12.5.2).As above, let (tk, xk) be a sequence of points in R0 satisfying (12.5.5)

and such that V is differentiable at each point (tk, xk). Since V is differentiableat (tk, xk), for each h in Q(t, x),

D−V (tk, xk, 1, h) = Vt(tk, xk) + 〈Vx(tk, xk), h〉.

Hence

infh∈Q(tk,xk)

D−V (tk, xk; 1, h) = −[−Vt(tk, xk) (12.5.9)


+ suph∈Q(tk,xk)

〈−Vx(tk, xk), h〉] = 0,

the last equality being valid because V is a viscosity solution and is differ-entiable at (tk, xk). Since f is continuous, Ω(tk) is compact and Q(tk, xk) =

h : h = f(tk, xk, z), z ∈ Ω(t), it follows that there is a point zk in Ω(tk) atwhich the supremum in (12.5.9) is attained. Let hk ∈ Q(tk, xk) be defined byhk = f(tk, xk, zk). Then

−Vt(tk, xk) + 〈−Vx(tk, xk), hk〉 = 0. (12.5.10)

The sequence of points (tk, xk, hk) are in the set ∆ defined in (12.2.3) which,as noted in Remark 12.2.5, is compact in the present case. It then followsfrom (12.5.5) that there is a subsequence, which we relabel as (tk, xk, hk) thatconverges to the point (t, x, h0) in ∆. Thus, h0 ∈ Q(t, x).

With the sequence δk related to (tk, xk) as in (12.5.5), we have that

D−V (t, x; 1, h0) ≤ lim infk→∞

[V (t+ δk, xk + δkh0)− V (t, x)]δ−1k

= lim infk→∞

[V (t+ δk, x+ δkh0)− V (tk + δk, xk + δkhk)

+ V (tk + δk, xk + δkhk)− V (tk, xk) + V (tk, xk)− V (t, x)]δ−1k

≡ lim infk→∞

[Ak +Bk + Ck]δ−1k .

From (12.5.5), the Lipschitz continuity of V and hk → h0 we get that

(Ak + Ck)δ−1k = [O(|t− tk|) +O(|x − xk|+ δk|h0 − hk|)]δ−1

k (12.5.11)

as k → ∞. From the differentiability of V at (tk, xk) we get that

Bk = [Vt(tk, xk) + 〈Vx(tk, xk), hk〉]δk + o(δk|1 + hk|).

By (12.5.10), the expression in square brackets is zero, and thus Bkδ−1k = o(1).

Therefore, from (12.5.11), we get that

lim infk→∞

[(Ak + Ck) +Bk]δ−1k ≤ lim sup

k→∞(Ak + Ck)δ

−1k

+ lim infk→∞

Bkδ−1k = 0.

Hence (12.5.8) holds and the lemma is proved.

12.6 Uniqueness

After showing that under appropriate hypotheses the value function is aviscosity solution of the Hamilton-Jacobi equation, the question arises whether


this is the only solution. This question is answered in the affirmative in the vis-cosity solution literature. See Fleming and Soner [35] and Bardi and Capuzzo-Dolcetti [5] and the references given therein. In this section we shall also an-swer the uniqueness question affirmatively, but without using the analyticaltechniques or results of the viscosity solution literature. Rather, we shall usethe techniques of control theory to give a self-contained presentation.

The following modification of Assumption 12.2.1-r will hold in this section.

Assumption 12.6.1. The function K in (12.2.1) is the constant functionK. In conditions (v) to (vii) the set R0 = [0, T )× R

n, where we have takenI = [0, T ]. The terminal set J = T × R

n.

Theorem 12.6.2. Let Assumption 12.6.1 hold. Let the sets Ω(t), 0 ≤ t ≤ Tbe a fixed compact set C. Let g be locally Lipschitz continuous. Then the valuefunction W is the unique locally Lipschitz continuous viscosity solution of theboundary value problems

−Vt(t, x) +H(t, x,−Vx) = 0 (t, x) ∈ R0 (12.6.1)

V (T, x) = g(x).

The value function W is the unique locally Lipschitz continuous solution of

minz∈C

D−V (t, x; 1, f(t, x, z)) = 0 (t, x) ∈ R0 (12.6.2)

V (T, x) = g(x).

Proof. In view of Lemma 12.5.6 to prove the theorem it suffices to show thatW is the unique solution of (12.6.2). That is, we must show that if (t0, x0)is an arbitrary point of R0 and V is a locally Lipschitz solution of (12.6.2),then V (t0, x0) = W (t0, x0). To simplify the argument, we shall assume thatthe problem is in Mayer form. We showed in Section 2.4 that there is no lossof generality in making this assumption.

We first show that

V (t0, x0) ≤W (t0, x0) (t0, x0) ∈ R0. (12.6.3)

Let (ψ, v) be an arbitrary admissible pair for the control problem with initialpoint (t0, x0). Since V is locally Lipschitz continuous and ψ is absolutelycontinuous, the function t→ V (t, ψ(t)) is absolutely continuous. Hence

V (T, ψ(T ))− V (t0, x0) =

∫ T

t0

(d

dtV (t, ψ(t)

)dt. (12.6.4)

Let s be a point in [t0, T ] at which t → V (t, x(t)) is differentiable, v(s) ∈ C,and s is a Lebesgue point for t → f(t, ψ(t), v(t)). The set of such points hasmeasure T − t0, and

d

dtV (t, ψ(t))|t=s = lim

δ→0δ−1[V (s+ δ, ψ(s+ δ))− V (s, ψ(s))]


= limδ→0

δ−1[V (s+ δ, ψ(s) +

∫ s+δ

s

f(σ, ψ(σ), v(σ))dσ) − V (s, ψ(s))]

= limδ→0

δ−1[V (s+ δ, ψ(s) + δf(s, ψ(s), v(s)) + o(δ)) − V (s, ψ(s))]

≥ D−V (s, ψ(s); 1, f(s, ψ(s), v(s)) ≥ 0,

where the next to the last inequality follows from the Lipschitz continuity ofV and the definition of lower Dini derivate. The last inequality follows from(12.6.2). Substituting this into (12.6.4) gives

V (t0, x0) ≤ V (T, ψ(T )) = g(ψ(T )).

Since (ψ, v) is an arbitrary admissible pair and J(ψ, v) = g(ψ(T )), we get that(12.6.3) holds.

To prove the theorem we need to show that

V (t0, x0) ≥W (t0, x0). (12.6.5)

The calculation in the preceding of dV (t, ψ(t))/dt and (12.6.2) suggest thatwe attempt to find an admissible trajectory ψ such that at points (t, ψ(t)) ofthe trajectory

D−V (t, ψ(t); 1, f(t, ψ(t), v(t)) = 0.

For then,

V (t0, x0) = V (T, ψ(T )) = g(ψ(T )) = J(ψ, v) ≥W (t0, x0).

This in turn suggests taking ψ to be a solution of the differential inclusionψ′ = F (t, x), with initial condition ψ(t0) = x0, where

F (t, x) = w : w = argmin D−V (t, x; 1, f(t, x, z)), z ∈ C. (12.6.6)

Unfortunately, we cannot guarantee the existence of a solution on [t0, T ] ofthe differential inclusion x′ ∈ F (t, x) with ψ(t0) = x0 for F defined in (12.6.6).We do, however, have the following result.

Lemma 12.6.3. Let V be a locally Lipschitz continuous function that satisfies(12.6.2). Then for each ε > 0 there exists an admissible pair (ψ, v) such that

V (t0, x0) ≥ g(ψ(T ))− ε. (12.6.7)

Remark 12.6.4. Since g(ψ(T )) = J(ψ) ≥ W (t0, x0), and since ε > 0 isarbitrary, the inequality (12.6.7) implies (12.6.5).

The proof of Lemma 12.6.3 requires two preparatory lemmas, whose state-ments and proofs are facilitated by the introduction of additional notation.Let ψ(·; τ, ξ, v) denote an admissible trajectory with initial point (τ, ξ) cor-responding to the admissible control v. Let R(τ, ξ) denote the set of pointsreachable by admissible trajectories with initial point (τ, ξ). Thus,

R(τ, ξ) = (t, x) : t > τ x = ψ(t, τ, ξ, v), v admissible


Let vz denote the control on an interval with vz(t) = z ∈ C for all t in theinterval. Let B denote the closed unit ball in R

n and let L denote the Lipschitzconstant for V.

Lemma 12.6.5. There exists a positive number M and a positive integer i0with T − t0 > i−1

0 such that for all (τ, ξ) in [t0, T ]×MB, all z in C and all[τ, σ] ⊂ [t0, T ] with |σ − τ | < i−1 for all i ≥ i0:

(i) |ψ(σ; τ, ξ, vz)| ≤ 2M (ii) |ξ + (σ − τ)f(τ, ξ, z)| ≤ 2M (12.6.8)

(iii) |ξ + (σ − τ)f(τ, ξ, z)− ψ(σ; τ, ξ, vz)| ≤ |σ − τ |/2L.

Proof. It follows from (12.2.2) and Lemma 4.3.14 that all admissible trajec-tories with initial point (t0, x0) lie in a compact set [t0, T ] ×MB for someM > 0. Thus, R(t0, x0) ⊆ [t0, T ]×MB.

The set Σ ≡ [t0, T ] ×MB × C is compact, and since f is continuous itis bounded on Σ. Therefore, there exists a positive integer i2 such that fori > i2, (τ, ξ, z) in Σ and σ in [t0, T ] with |σ − τ | < i−1, (ii) holds.

The set [0, T ]×MB is compact, so again by (12.2.2) and Lemma 4.3.14,there exists a constant M1 > 0 such that for any point (τ, ξ) in [t0, T ]×MB,an admissible trajectory starting at (τ, ξ) lies in [t0, T ]×M1B. Thus, for anyz in C,

|ψ(σ, τ, ξ, vz)| = |ξ +∫ σ

τ

f(s, ψ(s, τ, ξ, vz), z)ds|

≤M +

∫ σ

τ

M1ds.

Therefore, there exists an i1 such that for i > i1 and |σ − τ | < i−1, (i) holdsfor i > i1.

The functions f and ψ are uniformly continuous on compact sets. Hencefor τ ≤ s ≤ σ,

f(s, ψ(s; τ, ξ, vz), z) = f(τ, ξ, z) + η(s; τ, ξ, z),

where η(s) → 0 as σ → τ , uniformly in (τ, ξ, z) on [t0, T ] ×MB × C. Con-sequently, there exists an i3 such that |η(s)| ≤ 1/2L whenever i > i3 and|σ − τ | < i−1. The left-hand side of (iii) is

|(σ − τ)f(τ, ξ, z)−∫ σ

τ

f(s, ψ(s; τ, ξ, vz), z)ds|,

and so is less than or equal to∫ σ

τ

|η(s)|ds ≤ |σ − τ |/2L,

which establishes (iii) for i > i3. If we set i0 = max(i1, i2, i3), then (i), (ii),and (iii) all hold for i > i0.


Lemma 12.6.6. Let V be a locally Lipschitz continuous function that satisfies(12.6.2). Then there exists a positive number M , an integer i0 > 0, and asequence δi∞i=1, of positive numbers such that for all i ≥ i0

0 < δi < i−1

the following holds. Corresponding to each integer i ≥ i0 and point (τ, ξ) ∈[0, T − i−1]×MB there exists a number δ ∈ (δi, i

−1) and a point z in C suchthat

δ−1(V (τ + δ, ψ(τ + δ; τ, ξ, vz))− V (τ, ξ)) < i−1. (12.6.9)

Proof. Let i0 be as in Lemma 12.6.5. Choose any i ≥ i0 and let (τ, ξ) ∈[0, T ]×MB. The i chosen will be fixed for the remainder of this proof. Thensince V satisfies (12.6.2), there exists a δτ,ξ ∈ (0, i−1) and a zτ,ξ ∈ C suchthat

δ−1τ,ξ [V (τ + δτ,ξ, ξ + δτ,ξf(τ, ξ, zτ,ξ))− V (τ, ξ)] < i−1/2.

From this, from (12.6.8), and from the Lipschitz continuity of V we get that

δ−1τ,ξ [V (τ + δτ,ξ, ψ(τ + δτ,ξ; τ, ξ, vzτ,ξ

))− V (τ, ξ)] < i−1.

For each point (τ, ξ) in [0, T − i−1]×MB consider the set

Oτ,ξ = (σ, η) : δ−1τ,ξ [V (σ + δτ,ξ, ψ(σ + δτ,ξ;σ, η, vzτ,ξ

))− V (σ, η)] < i−1.

Each set Oτ,ξ contains (τ, ξ) and is open by the continuity of the functionsinvolved. Thus, the family of sets Oτ,ξ : (τ, ξ) ∈ [0, T − i−1] × MB is anopen cover of the compact set [0, T − i−1] ×MB. Hence there exists a finitesubcover. That is, there exists a positive integer N , numbers δj ∈ (0, i−1),points zj ∈ C, and sets Oj , j = 1, . . . , N , where

Oj = (σ, η) : δ−1j [V (σ + δj, ψ(σ + δj;σ, η, vzj

))− V (σ, η)] < i−1,

such that

[0, T − i−1]×MB ⊆N⋃

j=1

Oj . (12.6.10)

Let δi = minδ1, . . . , δj. Then δi ∈ (0, i−1), since this is true for each δj .Now let (t, x) be an arbitrary point in [0, T − i−1]×MB. Then by (12.6.10),(t, x) ∈ Oj for some j = 1, . . . , N . But then by the definition of Oj , (12.6.9)is true with δ = δj and z = zj . Since δi ≤ δj ≤ i−1, we have that δ ∈ [δi, i

−1)as required.

We now take up the proof of Lemma 12.6.3.

We noted in the proof of Lemma 12.6.5 that there exists an M >0 such that for any admissible trajectory ψ( ; t0, x0, v), the inequality|ψ(s; t0, x0, v)| ≤M holds for all t0 ≤ s ≤ T . It then follows from the continu-ity of f that all admissible trajectories ψ(·; t0, x0, v) satisfy a uniform Lipschitzcondition on [t0, T ].


We shall now obtain the admissible pair (ψ, v) whose existence is as-serted in (12.6.7). For an arbitrary positive integer i ≥ i0, where i0 is asin Lemma 12.6.6, a set of mesh points t0, t1, . . . , tNi+1

= T will be chosentogether with a control v with v(t) = zj ∈ C for tj−1 ≤ t < tj, j = 1, . . . , Ni+1

so that the corresponding trajectory ψ( ; t0, x0, v) satisfies (12.6.7).We define the sequences tj and zj recursively. Suppose that t0, t1, . . .,

tk and z1, . . . , zk have been defined, where tj−1 < tj , where tk < T − i−1,and where zj ∈ C, j = 1, . . . , k. Let v(t) = zj for t ∈ [tj−1, tj ], j = 1, . . . , kand let ψ( ; t0, x0, v) be the corresponding admissible trajectory defined on[0, tk]. Let ξk = ψ(tk; t0, x0, v).

By Lemma 12.6.6 we can choose a zk+1 ∈ C and a mesh point tk+1 so that

tk + δk < tk+1 < tk + i−1 (12.6.11)

and(tk+1 − tk)

−1[V (tk+1, ξk+1)− V (tk, ξk)] < i−1, (12.6.12)

whereξk+1 = ψ(tk+1; tk, ξk, vzk+1

) = ψ(tk+1; t0, x0, v).

Note that |ξk+1| ≤M . The recursion is terminated when we arrive at the firstindex value k = Ni for which

T − i−1 ≤ tNi≤ T. (12.6.13)

This will occur by virtue of (12.6.11). On the interval [TNi, T ] take v(t) = zT ,

an arbitrary element of C.From the Lipschitz continuity of V and from (12.6.12) we have that

V (T, ψ(T ))− V (t0, x0) = V (T, ψ(T ))− V (tNi, ψ(tNi

))

+

Ni−1∑

k=0

[V (tk+1, ψ(tk+1))− V (tk, ψk(t))]

≤ K|(T, ψ(T ))− (TNi, ψ(TNi

)|+Ni−1∑

k=0

(tk+1 − tk)i−1.

From (12.6.13) we have that T−TNi< i−1. Let A denote the uniform Lipschitz

constant for all admissible trajectories with initial point (t0, x0). Then

V (T, ψ(T ))− V (t0, x0) ≤ i−1(K(1 +A) + T ).

Given an ε > 0, we may choose i so that i−1(K(1 + A) + T ) < ε. SinceV (T, ψ(T )) = g(ψ(T )), we get that (12.6.7) holds.


12.7 The Value Function as Verification Function

In Problem 12.2.1 let ψ be a trajectory with control v that satisfies theMaximum Principle. In the absence of other information such as, for example,the existence of a solution and the uniqueness of a solution that satisfiesthe Maximum Principle, we cannot conclude that ψ solves Problem 12.2.1.Thus, it is desirable to have a condition that verifies the optimality of asuspect solution. Theorem 12.7.1 will furnish such a test, which involves thevalue function. Hence the value function can be considered to be a verificationfunction. We take Problem 12.2.1 to be in Mayer form.

Theorem 12.7.1. Let Assumption 12.2.1-r hold. Let the value function Wbe Lipschitz continuous on compact subsets of R0 and continuous on R0 ∪J .For (t1, x1) ∈ J let

W (t1, x1) = g(t1, x1). (12.7.1)

Let ψ( ; τ, ξ) be an admissible trajectory for the problem with initial point (τ, ξ)in R0, and let t1 be the terminal time of ψ( ; τ, ξ). Let ω(t;ψ) = W (t, ψ(t))on [τ, t1). If

D−W (t, ψ(t); 1, ψ′(t)) = 0 a.e. on [τ, t1). (12.7.2)

Then ψ is optimal for Problem 12.2.1.

Proof. Since W is Lipschitz continuous, the function ω( ;ψ) is absolutelycontinuous. Thus, almost all points of (τ, t1) are simultaneously Lebesguepoints of ψ( ; τ, ξ) and points of differentiability of ω( ;ψ).

At such points we have

dω

dt= lim

δ→0[W (t+ δ, ψ(t+ δ))−W (t, ψ(t))]δ−1 (12.7.3)

= limδ→0

[W (t+ δ, ψ(t) +

∫ t+δ

t

ψ′(s)ds)−W (t, ψ(t))]δ−1

= limδ→0

[W (t+ δ, ψ(t) + δψ′(t) + o(δ)) −W (t, ψ(t))]δ−1

= DW (t, ψ(t); 1, ψ′(t)),

where in passing to the last line we used the Lipschitz continuity of W andthe fact that we know that the limit in the next to the last line exists.

From the definition of ω( ;ψ) we get that

W (t1, ψ(t1))−W (τ, ξ) = ω(t1;ψ)− ω(τ, ξ) =

∫ t1

τ

dω

dt(t;ψ)ds.

It then follows from (12.7.1) and (12.7.3) that if (12.7.2) holds, then

g(t1, ψ(t1)) =W (t1, ψ(t1)) =W (τ, ξ). (12.7.4)


Now let ψ0( ; τ, ξ) be an arbitrary admissible trajectory for Problem 12.2.1with initial point (τ, ξ). The chain of equalities in (12.7.3) is valid for thefunction ω( ;ψ0) in place of ω( ;ψ). From Theorem 12.4.3 we get that foralmost all t in [τ, t01), where t01 is the terminal time of ψ0,

DW (t, ψ0(t); 1, ψ′0(t)) ≥ 0.

From this and from (12.7.4) we see that

g(t10, ψ0(t01)) = W (t10, ψ0(t01)) ≥W (τ, ξ) = g(t1, ψ(t1)).

Since for any admissible pair (ψ0, v0), J(ψ0, v0) = g(t01, ψ0(t01)), the optimal-ity of (ψ, v) follows.

Remark 12.7.2. Clearly, the assumption (12.7.2) can be replaced bydω(t, ψ(t))/dt = 0 a.e.

The next result states that if ψ is optimal, then (12.7.2) holds. Thus,(12.7.2) can be said to characterize an optimal control.

Corollary 12.7.3. Let ψ( ) = ψ( ; τ, ξ) be an optimal trajectory with initialpoint (τ, ξ) and terminal time t1. Then for almost all t in [τ, t1] and all h inQ(t, ψ(t))

0 =dω

dt= D−W (t, ψ(t); 1, ψ

′(t)) ≤ D−W (t, ψ(t); 1, h).

If ψ( ) = ψ(·, τ, ξ) is an optimal trajectory for the problem with initialpoint (τ, ξ), then by the Principle of Optimality,

W (τ + δ, ψ(τ + δ))−W (t, ψ(t)) = 0

for all τ ≤ t ≤ T . Hence (12.7.3) with ψ = ψ gives

DW (t, ψ(t); 1, ψ′(t)) = 0.

Combining this with (12.4.3) gives the result.

Remark 12.7.4. We emphasize that the conclusion (12.4.3) of Theo-rem 12.4.3 holds at all points of R0, while the conclusion of the corollaryabove holds at almost all t along an optimal trajectory.

12.8 Optimal Synthesis

We continue to consider the control problem in Mayer form. Taken to-gether, Theorem 12.7.1 and Corollary 12.7.3 state that under appropriate


hypotheses, if ψ( ; τ, ξ) is an admissible trajectory then ψ( ; τ, ξ) is optimalfor the problem with initial point (τ, ξ) if and only if

0 = D−W (t, ψ(t); 1, ψ′(t)) ≤ D−W (t, ψ(t); 1, h)

for almost all t in [τ, T ] and all h in Q(t, ψ(t)). Thus, if we set

U(t, x) = argmin [D−W (t, x; 1, h) : h ∈ Q(t, x)],

then the vectors in f(t, x, U(t, x)) would contain the tangent vectors of alloptimal trajectories passing through (t, x). The control vectors v in U(t, x)would be the set of optimal controls at (t, x) for optimal trajectories startingat (t, x). We would expect to obtain an optimal trajectory for the problemwith initial condition (τ, ξ) by solving the differential inclusion

x′ ∈ f(t, x, U(t, x)) x(τ) = ξ.

We noted earlier, in connection with a related question, that to carry out thisprogram, regularity conditions must be imposed on the possibly set valuedfunction (t, x) → U(t, x). The behavior of U , however, cannot be determineda priori from the data of the problem. Instead, to obtain an optimal trajectory,we shall construct a sequence of pairs ψk, vk such that ψk converges to anoptimal trajectory.

Assumption 12.8.1. (i) Statements (i) to (iv) of Assumption 12.2.1-rhold with K(t) = K and M(t) =M .

(ii) The set R0 = [0, T ]× Rn.

(iii) The terminal set J = T × Rn.

(iv) Ω(t) = C, a fixed compact set.

(v) The sets Ar(t) are non-empty.

(vi) The function g is locally Lipschitz continuous.

It follows from (iv) of Assumption 12.8.1 and from the definition of Ω, that

the constraint set Ω is a constant compact set. Let C = Ω.

The next theorem will be used to show that our sequence of trajectoriesψk converges to an optimal trajectory ψ. The theorem states that for points(t, x) in a compact set, the minimum over h in Q(t, x) of the difference quotientin (12.4.1) is uniformly small in absolute value for δ sufficiently small. Thisallows for the interchange of order min(lim inf) = lim inf(min) in (12.4.3).

Theorem 12.8.2. Let Assumption 12.8.1 hold. Let BL denote the closed ballin R

n centered at the origin with radius L. Let ROL = [0, T ]×BL. Then foreach ε > 0 there exists a δ0 > 0 such that for all 0 < δ < δ0, all (τ, ξ) in ROL

and all τ ≤ t ≤ τ + δ

minz∈Ω

|δ−1(W (τ + δ, ξ + δf(τ, ξ, z))−W (τ, ξ))| < ε. (12.8.1)


Proof. Under Assumption 12.8.1 the relaxed control problem has a solutionfor each initial point (t, x) in R0 and the value function is Lipschitz continuouson compact subsets of R0. (See Theorem 4.3.5 and Theorem 12.3.2.)

Let z be an arbitrary element of C. Let ψ( ; τ, ξ, z) denote the trajectorywith initial point (τ, ξ) and v(t) = z. It follows from (12.2.2) and Corol-lary 4.3.15 that there exists a constant A > 0 such that for all (τ, ξ) in ROL

and all z in C, |ψ(t; τ, ξ, z)| ≤ A for all τ ≤ t ≤ T . The function f is uniformly

continuous on [0, T ] × BA × C where BA is the closed ball in Rn of radius

A with center at the origin. From the solution, which holds for all admissiblepairs (ψ, v),

ψ(t; τ, ξ) = ξ +

∫ t

τ

f(s, ψ(s; τ, ξ), v(s))ds (12.8.2)

it follows that ψ(t, τ, ξ) → ξ as t → τ , uniformly for (τ, ξ) in ROL. It furtherfollows that if we take v(t) = z for τ ≤ t ≤ T in (12.8.2), then given an

ε > 0 there exists a δ0(ε), independent of (τ, ξ) in ROL and z in C such thatwhenever 0 < δ < δ0(ε), we have

ψ(τ + δ; τ, ξ, z) = ξ +

∫ τ+δ

τ

[f(τ, ξ, z) + ε(s)]ds,

with |ε(s)| < ε, for all (τ, ξ) in ROL and z in C. Hence

ψ(τ + δ; τ, ξ, z) = ξ + δf(τ, ξ, z) + o(δ), (12.8.3)

where o(δ) is uniform for (τ, ξ) in ROL and z in C.By the Principle of Optimality we have

W (τ + δ, ψ(τ + δ; τ, ξ, x))−W (τ, ξ) ≥ 0.

From (12.8.3) and the Lipschitz continuity of W on compact sets we get that

W (τ + δ, ξ + δf(τ, ξ, z)) −W (τ, ξ) ≥ o(δ),

where o(δ) is uniform with respect to (τ, ξ) in ROL and z in C. Since z in Cis arbitrary we get that

minz∈C

W (τ + δ, ξ + δf(τ, ξ, z))−W (τ, ξ) ≥ o(δ), (12.8.4)

where o(δ) is independent of (τ, ξ) in ROL.Now let (ψ, v) be optimal for the problem with initial point (τ, ξ). Then

|ψ(t, τ, ξ)| ≤ A for all (τ, ξ) inROL and τ ≤ t ≤ T . The function f is uniformly

continuous on [0, T ]×BA × C. It therefore follows from (12.8.2) that for eachε > 0 there exists a δ0 = δ0(ε) such that for 0 < δ < δ0 and all (τ, ξ) in ROL

ψ(τ + δ; τ, ξ) = ξ +

∫ τ+δ

τ

[f(τ, ξ, v(s)) + ε(s; τ, ξ)]ds, (12.8.5)


where |ε(s; τ, ξ)| < ε for all 0 ≤ s < δ and all (τ, ξ) in ROL.We have

∫ τ+δ

τ

f(τ, ξ, v(s))ds = δ

∫ τ+δ

τ

f(τ, ξ, v(s))ds

δ, (12.8.6)

where v(s) ∈ C for a.e. s in [τ, τ + δ]. Since C is a fixed compact set,

f(τ, ξ, v(s)) ∈ Q(τ, ξ), a.e.,

where Q(τ, ξ) is a closed convex set. Let Kδ denote the set of points in [τ, τ+δ]at which the inclusion holds. Then

cl cof(τ, ξ, v(s)) : s ∈ Kδ ∈ cl co Q(τ, ξ) = Q(τ, ξ).

From Lemma 3.2.9 we get that

cl co f(τ, ξ, v(s)) : s ∈ Kδ = cl ∫

Kδ

f(τ, ξ, v(s))dµs : µ ∈ P (Kδ),

where P (Kδ) is the set of probability measures in Kδ. Thus,

∫ τ+δ

τ

f(τ, ξ, v(s))ds

δ∈ Q(τ, ξ).

Hence there exists a zδ in C such that

∫ τ+δ

τ

f(τ, ξ, v(s))ds

δ= f(τ, ξ, zδ).

From this and from (12.8.6) and (12.8.5) we get that

ψ(τ + δ, τ, ξ) = ξ + δf(τ, ξ, zδ) + o(δ), (12.8.7)

where o(δ) is uniform with respect to z in C and (τ, ξ) in ROL.Since ψ is optimal, by the Principle of Optimality

W (τ + δ, ψ(τ + δ))−W (τ, ξ) = 0.


W (τ + δ, ξ + δf(τ, ξ, zδ) + o(δ)) −W (τ, ξ) = 0.

From this and from the Lipschitz continuity of W we get that

W (τ + δ, ξ + δf(τ, ξ, zδ))−W (τ, ξ) = o(δ),

where o(δ) is uniform with respect to (τ, ξ) in ROL. Hence

minz∈C

[W (τ + δ, ξ + f(τ, ξ, z))−W (τ, ξ)] ≤ o(δ), (12.8.8)


where o(δ) is uniform with respect to (τ, ξ) in ROL.The theorem now follows from (12.8.8) and (12.8.4).We next present an algorithm for generating a sequence (ψk, vk) of ad-

missible pairs that will furnish the desired approximation to an optimal pair(ψ, v).

Algorithm 12.8.3. Consider Problem 12.2.1 in Mayer form with initial point(τ, ξ). Let Assumption 12.8.1 hold. For each positive integer k let tk,0 =τ, tk,1, . . . , tk,k−1, tk,k = T be a uniform partition of [τ, T ]. Let δk = (T−τ)/k.

We define (ψk, vk) on [tk,0, tk,1]. Let xk,0 = ξ and let

vk,0 = argminz∈C

[W (tk,0, xk,0 + δf(tk,0, xk,0, z))−W (tk,0, xk,0)].

For t ∈ [tk,0, tk,1) let vk,0(t) = vk,0. Define ψk,0 on [tk,0, tk,1] to be the solutionof

ψ′k,0(t) = f(t, ψk,0(t), vk,0(t)) ψk,0(tk,0) = xk,0.

Now suppose that (ψk, vk) has been defined on [tk,0, tk,i], for i in the range1 ≤ i ≤ k − 1. We shall define an admissible pair (ψk,i, vk,i) on [tk,i, tk,i+1] insuch a way that if we extend (ψk, vk) to [tk,i, tk,i+1] by setting ψk(t) = ψk,i(t)and vk(t) = vk,i+1(t) for tk,i ≤ t ≤ tk,i+1, we shall have an admissible pairdefined on [tk,0, tk,i+1). Let xk,i = ψ(tk,i). Let

vk,i = argminz∈C

[W (tk,i, xk,i + δf(tk,i, xk,i, z))−W (tk,i, xk,i)]. (12.8.9)

For t ∈ [tk,i, tk,i+1) let vk,i(t) = vk,i. Define ψk,i on [tk,i, tk,i+1) to be thesolution of

ψ′k,i(t) = f(t, ψk,i(t), vk,i(t)) ψk,i(tk,i) = xk,i.

Define ψk,i(tk+1) = limt→(tk+1−0) ψk(t). That this limit exists follows from theuniform boundedness of all admissible trajectories, the continuity of f , andthe Cauchy criterion.

Since f is continuous and since vk,i(t) = vk,i on [tk,i, tk,i+1) it follows thatfor t ∈ [tk,i, tk,i+1)

ψk,i(t) = ψk,i(tk,i) + (t− tk,i)f(tk,i, ψ(tk,i), vk,i) + o(t− tk,i).

Hence

ψ(tk,i+1) = ψ(tk,i) + δkf(tk,i, ψ(tk,i), vk,i) + o(δk). (12.8.10)

Theorem 12.8.4. Let Assumption 12.8.1 hold. Then the sequence ψk gen-erated by Algorithm 12.8.3 has subsequences ψkj

that converge uniformlyto absolutely continuous functions ψ. Corresponding to each such ψ there ex-ists an admissible control v such that (ψ, v) is admissible and is optimal forProblem 12.2.1.


Proof. It again follows from (12.2.2) and Corollary 4.3.15 that the functionsψk are uniformly bounded. It then follows from the continuity of f that|f(t, ψk(t), vk(t)| is uniformly bounded for τ ≤ t ≤ T . From this and from

ψk(t) = ξ +

∫ t

τ

f(s, ψk(s), vk(s))ds (12.8.11)

it follows that the functions ψk are equi-absolutely continuous. Hence thereexists a subsequence, again denoted by ψk, that converges uniformly to anabsolutely continuous function ψ.

We may write (12.8.11) as

ψk(t) = ξ +

∫ t

τ

f(s, ψk(s), µks)ds, (12.8.12)

where µk is a relaxed control given by

µks =n+1∑

i=1

pi(s)δvki (s)

concentrated on the compact set C and defined on the compact interval [τ, T ].By Theorem 3.3.6 the sequence µk is weakly compact. Hence there exists asubsequence µk that converges weakly to a relaxed control µ concentratedin C. Corresponding to µk is the subsequence ψk that converges uniformlyto ψ. Relation (12.8.12) holds for the sequence (ψk, µk). From Lemma 4.3.3and the relation ψ(t) = limk→∞ ψk(t) uniformly we conclude that for all t in[T, τ ]

ψ(t) = limk→∞

ψk(t) = limk→∞

[ξ +

∫ t

τ

f(s, ψk(s), µks)ds]

= ξ +

∫ t

τ

f(s, ψ(s), µs)ds.

Thus, (ψ, µ) is an admissible pair.It remains to show that ψ is optimal. Since g(ψk(T )) = J(ψk, µk) and

limk→∞

g(ψk(T )) = g(ψ(T )) = J(ψ, µ), to show that (ψ, µ) is optimal it suffices

to show thatlimk→∞

g(ψk(T )) =W (τ, ξ). (12.8.13)

We noted previously that there exists a compact set ROC ⊆ R0 such thatfor all admissible trajectories ψ with initial point (τ, ξ), the points (t, ψ(t)) lie

in ROC . The function f is bounded by some constant A on ROC × C and isuniformly continuous there. The value function W is Lipschitz continuous onROC with Lipschitz constant B.


We have

lim supk→∞

|g(ψk(T ))−W (τ, ξ)| (12.8.14)

= lim supk→∞

|W (tk,k, ψ(tk,k))−W (τ, ξ)|

≤ lim supk→∞

k∑

j=1

|W (tk,j , ψk(tk,j))−W (tk,j−1, ψk(tk,j−1))|

= lim supk→∞

k∑

j=1

|W (tk,j−1 + δk, ψ(tk,j−1)

+ δkf(tk,j−1, ψ(tk,j−1), vk,j−1))

−W (tk,j−1, ψk(tk,j−1))|+ o(δk)

where the last equality follows from (12.8.10) and δk = (tj − tj−1). It then fol-lows from the Lipschitz continuity ofW , the definition of δk as δk = (T−τ)/k,the definition of vk,j−1 in (12.8.9), and Theorem 12.8.2 that each summandin the rightmost side of (12.8.14) is o(1/k) as k → ∞. Hence the rightmostside of (12.8.14) is o(1) as k → ∞, and thus (12.8.13) and Theorem 12.8.4 areestablished.

12.9 The Maximum Principle

In Section 6.2 we derived the maximum principle for a class of problemsunder the assumption that the value function is of class C(2). In this section weshall derive the maximum principle for a certain class of problems under theassumption that the value function is a Lipschitz continuous viscosity solutionof the Hamilton-Jacobi equation (12.5.1). We assume that the problem at handis the relaxed problem, Problem 12.2.1. We consider this problem to be anordinary problem as in Section 5.4.

Assumption 12.9.1. Assumption 12.2.1-r is in force with the followingchanges.

(i) For fixed (t, z) the function f(t, ·, z) is of class C(1) on Rn.

(ii) The set R0 = [0, T ]× Rn.

(iii) The terminal set T is T ×Rn. Thus, the function g is a function of x

alone.

(iv) For all (t, x) in R0, the sets Ω(t, x) are a fixed compact set, C.


Remark 12.9.2. Assumption 12.9.1(i) implies Assumption 12.2.1-r (iii). As-sumption 12.9.1 and Theorems 12.5.3 and 12.6.2 imply that the value functionW is the unique viscosity solution of the Hamilton-Jacobi equation (12.5.1)with boundary condition V (T, x) = g(x).

Remark 12.9.3. In the definition of viscosity solution the test functions weretaken to be of class C(1), as in the definition given by Crandall and Lions.An examination of the proofs in Sections 12.5 and 12.6 shows that we did notuse the continuity of the partial derivatives of the test functions. All we usedwas the existence of partial derivatives. Therefore, for our purposes we couldhave restricted ourselves to test functions that possess partial derivatives. Therequirement of continuous differentiability is needed for the general nonlinearpartial differential equation, but is not needed for the Hamilton-Jacobi equa-tion (12.5.1).

Let (τ0, ξ0) be a point in [0, T )×Rn. Let ψ∗( ) = ψ∗( ; τ0, ξ0) be an optimal

trajectory for the problem with initial point (τ0, ξ0) and let v∗( ) = v∗( ; τ0, ξ0)

be the corresponding optimal control. Let f = (f0, f) and for t ≥ τ and x ∈ Rn

letF (t, x) = f(t, x, v∗(t)). (12.9.1)

Then for each τ0 ≤ t ≤ T the function F (t, ·) is of class C(1) on Rn and the

function F (·, x) is measurable on [τs, T ] for each fixed x in Rn.

We consider the differential equation

x′ = F (t, x) x(τ) = ξ (12.9.2)

with τ ≥ T0 and ξ in Rn. It follows from Assumption 12.2.1-r that Eq. (12.9.2)

has a unique solution ψ( ) = ψ( ; τ, ξ) defined on [τ, T ]. Note that ψ∗( ) =ψ∗( ; τ0, ξ0) = ψ( ; τ0, ξ0).

For τ ≥ τ0 and ξ ∈ Rn we define a function ω as follows.

ω(τ, ξ) = g(ψ(T ; τ, ξ)) +

∫ T

τ

F 0(t, ψ(t; τ, ξ))dt, (12.9.3)

where ψ is the unique solution of (12.9.2). It follows from the definitions of

W, F , ψ∗, and v∗ that

ω(τ0, ξ0) = W (τ0, ξ0) = J(ψ∗( ; τ0, ξ0), v∗( ; τ0, ξ0)). (12.9.4)

From the Principle of Optimality it further follows that

ω(t, ψ∗(t; τ0, ξ0)) =W (t, ψ∗(t; τ0, ξ0)) (12.9.5)

for all τ0 ≤ t ≤ T . For Problem 12.2.1 with initial point (τ, ξ), τ0 ≤ τ < T ,the control v∗ need not be optimal. Therefore, since ω(τ, ξ) is the payoff forthis choice of control, we have that

ω(τ, ξ) ≥W (τ, ξ), τ0 ≤ τ < T, ξ ∈ Rn. (12.9.6)


From (12.9.5) and (12.9.6) we get that at all points (t, ψ∗(t; t0, x0)), τ0 ≤ t ≤T , on the optimal trajectory W − ω have a maximum. Lemma 12.9.4 statesthat at almost all points (t, ψ∗(t)) of the optimal trajectory the function ω isa test function as defined in Remark 12.9.3. Assume this to be true. Since Wis a viscosity solution, it is also a subsolution. Hence for almost all t in [τ0, T ]

−ωt(t, ψ∗(t)) +H(t, ψ∗(t),−ωx(t, ψ

∗(t))) ≤ 0. (12.9.7)

Lemma 12.9.4. At almost all points (t, ψ∗(t)), τ0 ≤ t ≤ T the partial deriva-tives ωτ and ωξ exist.

Proof. From (12.9.3) we get that

ωξ(τ, ξ) = gx(ψ(T ; τ, ξ))ψξ(T ; τ, ξ)+

∫ T

τ

F 0x (t, ψ(t; τ, ξ))ψξ(t; τ, ξ)dt. (12.9.8)

The matrix ψξ( ; τ, ξ) is a solution of the system

γ′ = Fx(t, ψ(t; τ, ξ))γ γ(τ) = I. (12.9.9)

Thus, ψξ( ; τ, ξ) is an absolutely continuous matrix function defined on [τ, T ].Hence standard theorems on differentiation under the integral justify the for-mula in Eq. (12.9.8).

We now consider the function ω evaluated along the optimal trajectoryψ∗( ) = ψ∗( ; τ0, ξ0). Designate the trajectory with initial point (t, ψ∗(t))resulting from the control v∗( ) on [t, T ] by ψ( ) = ψ( ; t, ψ∗(t)). Then by theuniqueness theorem for solutions of ordinary differential equations and by thePrinciple of Optimality we get that for all t ≤ s ≤ T

ψ(s; t, ψ∗(t)) = ψ∗(s; t, ψ∗(t)) = ψ∗(s; τ0, ξ0). (12.9.10)

Thus, from (12.9.8) we get that

ωξ(t, ψ∗(t)) = gx(ψ

∗(T ))ψ∗ξ (T ; t, ψ

∗(t)) (12.9.11)

+

∫ T

t

F 0x (s, ψ

∗(s))ψ∗ξ (s; t, ψ

∗(t))ds.

It follows from (12.9.11) that for fixed t in (τ, T ), the function ωξ is con-tinuous in a neighborhood of (t, ψ∗(t)). We shall not be able to conclude thatωt, whose existence at almost all t we show next, is continuous.

From (12.9.3) and (12.9.10) we get that

ω(t, ψ∗(t)) = g(ψ∗(T )) +

∫ T

t

F 0(s, ψ∗(s; τ0, ξ0))ds.

Thus, t→ ω(t, ψ∗(t)) is absolutely continuous and

dω(t, ψ∗(t))/dt = −F 0(t, ψ∗(t; τ0, ξ0)) (12.9.12)


for almost all t in [τ0, T ].Let t ∈ [τ0, T ] be a Lebesgue point of ψ∗ and a point of differentiability of

t→ ω(t, ψ∗(t)). The set of such points has full measure. We have for δ > 0

[ω(t+ δ, ψ∗(t))− ω(t, ψ∗(t))]δ−1 = Aδ−1 +Bδ−1,

where

A = ω(t+ δ, ψ∗(t))−ω(t+ δ, ψ∗(t+ δ)) B = ω(t+ δ, ψ∗(t+ δ))−ω(t, ψ∗(t)).

From (12.9.12) we have that limδ→0 Bδ−1 = −F 0(t, ψ∗(t)).

Since t is a Lebesgue point of ψ∗ we have that

A = ω(t+ δ, ψ∗(t)) − ω(t+ δ, ψ∗(t) + δψ∗′

(t) + o(δ)).

Since ωξ(t+ δ, ·) is continuous in a neighborhood of ψ∗(t),

A = −〈ωξ(t+ δ, ψ∗(t)), ψ∗′

(t) + o(δ)〉δ + o(δ).

Hence

limδ→0

Aδ−1 = −〈ωξ(t, ψ∗(t)), ψ∗′

(t)〉 = −〈ωξ(t, ψ∗(t)), F (t, ψ∗(t))〉.

Thus, ωτ (t, ψ∗(t)) exists and

ωτ (t, ψ∗(t)) = −F 0(t, ψ∗(t))− 〈ωξ(τ, ψ

∗(t)), F (t, ψ∗(t))〉 (12.9.13)

for almost all t in [τ0, T ].Having established Lemma 12.9.4, we have established (12.9.7). We rewrite

(12.9.7) using the definitions in Section 12.5 to get that for almost all t in[τ0, T ]

−ωτ (t, ψ∗(t)) + max

z∈C[−f0(t, ψ∗(t), z)− 〈ωξ(t, ψ

∗(t)), f(t, ψ∗(t), z)〉 ≤ 0.

(12.9.14)

We rewrite (12.9.13) using the definition of F in (12.9.1) to get that for almostall t in [τ0, T ]

−ωτ(t, ψ∗(t))− f0(t, ψ∗(t), v∗(t))− 〈ωξ(t, ψ

∗(t)), f(t, ψ∗(t), v∗(t))〉 = 0.(12.9.15)

From (12.9.14) and (12.9.15) we get that for almost all t in [τ0, T ]

maxz∈C

[−f0(t, ψ∗(t), z)− 〈ωξ(t, ψ∗(t)), f(t, ψ∗(t), z)〉 (12.9.16)

= −f0(t, ψ∗(t), v∗(t)) − 〈ωξ[t, ψ∗(t)), f(t, ψ∗(t), v∗(t))〉.

We next introduce the multipliers λ( ) ≡ λ( ; τ0, ξ0) via the adjoint equa-tions and show that ωξ(t, ψ

∗(t)) = −λ(t) a.e. on [τ0, T ]. Let λ( ; τ0, ξ0) be theunique solution on [τ0, T ] of the linear system

dp

dt= f0

x(t, ψ∗(t), v∗(t)) − pfx(t, ψ

∗(t), v∗(t)) (12.9.17)


p(T ) = −gx(ψ∗(T )).

Then for fixed t in [τ0, T ) and all almost all t ≤ s ≤ T

λ′(s; τ0, ξ0)ψ∗ξ (s; t, ψ

∗(t)) = f0x(s, x(s), v

∗(s))ψ∗ξ (s; t, ψ

∗(t))

− λ(s; τ0, ξ0)fx(s, ψ∗(s), v∗(s))ψ∗

ξ (s; t, ψ∗(t)).

Also,λ(T ; τ0, ξ0)ψ

∗ξ (T ; t, ψ

∗(t)) = −gx(ψ∗(T ))ψ∗ξ (T ; t, ψ

∗(t)).

Recalling the definition of F in (12.9.1) and substituting into (12.9.11) andusing (12.9.9) gives

ωξ(t, ψ∗(t)) = −λ(T )ψ∗

ξ (T ; t, ψ∗(t)) (12.9.18)

+

∫ T

t

[λ′(s)ψ∗ξ (s; t, ψ

∗(t))

+ λ(s)fx(s, ψ∗(s), v∗(s))ψ∗

ξ (s; t, ψ∗(t))]ds

= −λ(T )ψ∗ξ (T ; t, ψ

∗(t)) +

∫ T

t

d(λ(s)ψ∗ξ (s; t, ψ

∗(t)))

= −λ(t)ψ∗ξ (t; t, ψ

∗(t)) = −λ(t)I.

Combining (12.9.16), (12.9.17), and (12.9.18) gives the following theorem,which is the maximum principle in this case.

Theorem 12.9.5. Let (ψ∗, v∗) be an optimal pair for Problem 12.2.1 withinitial point (τ, ξ). Then there exists an absolutely continuous function λ( ) =λ( ; τ, ξ) such that for almost all t in [τ, T ]

λ′(t) = f0x(t, ψ

∗(t), v∗(t))− λ(t)fx(t, ψ∗(t), v∗(t))

λ(T ) = −gx(ψ∗(T )).

Moreover, for almost all t in [τ, T ]

maxz∈C

[−f0(t, ψ∗(t), z) + 〈λ(t), f(t, ψ∗(t), z)〉]

= [−f0(t, ψ∗(t), v∗(t)) + 〈λ(t), f(t, ψ∗(t), v∗(t))〉].

Bibliography

[1] N. U. Ahmed, Properties of relaxed trajectories for a class of nonlinearevolution equations on Banach space, SIAM J. Control and Optimiza-tion, 21 (6), (1983), 953–967.

[2] K. J. Arrow, S. Karlin, and H. Scarf, Studies in the Mathematical Theoryof Inventory and Production, Stanford University Press, Stanford, CA,1958.

[3] M. Athans and P. Falb, Optimal Control, McGraw-Hill, New York, 1966.

[4] Jean-Pierre Aubin, Optima and Equilibria, 2nd ed., Springer, Berlin,Heidelberg, New York, 1998, Corrected second printing 2003.

[5] M. Bardi and I. Capuzzo-Dolcetta, Optimal Control and Viscosity Solu-tions of Hamilton-Jacobi-Bellman Equations, Birkhauser, Boston, 1997.

[6] R. Bellman, I. Glicksberg, and O. Gross, On the “bang-bang” controlproblem, Quart. Appl. Math., 14 (1956), 11–18.

[7] L. D. Berkovitz, Variational methods in problems of control and pro-gramming, J. Math. Anal. Appl., 3 (1961), 145–169.

[8] L. D. Berkovitz, Necessary conditions for optimal strategies in a classof differential games and control problems, SIAM J. Control 5 (1967),1–24.

[9] L. D. Berkovitz, An existence theorem for optimal controls, J. Optimiza-tion Theory Appl., 6 (1969), 77–86.

[10] L. D. Berkovitz, Existence theorems in problems of optimal control,Studia Math., 44 (1972), 275–285.

[11] L. D. Berkovitz, Existence theorems in problems of optimal control with-out property (Q), in Techniques of Optimization, A. V. BalakrishnanEd., Academic Press, New York and London, 1972, 197–209.

[12] L. D. Berkovitz, Optimal Control Theory, Springer-Verlag, New York,1974.

[13] L. D. Berkovitz, A penalty function proof of the maximum principle,Applied Mathematics and Optimization, Vol. 2, (1976), 291–303.

371


[14] L. D. Berkovitz, Optimal feedback controls, SIAM J. Control and Op-timization, 27 (5), (1989), 991–1006.

[15] G. A. Bliss, The problem of Lagrange in the calculus of variations. Am.J. Math., 52 (1930), 673–741.

[16] G. A. Bliss, Lectures on the Calculus of Variations, The University ofChicago Press, Chicago, 1946.

[17] V. G. Boltyanskii, R. V. Gamkrelidze, and L. S. Pontryagin, The theoryof optimal processes I, the maximum principle. Izv. Akad. Nauk SSSR.Ser Mat., 24 (1960), 3–42. English translation in Am. Math. Soc. Transl.Ser., 2, 18 (1961), 341–382.

[18] O. Bolza, Vorlesungen uber Variationsrechnung, Reprint of 1909 edition,Chelsea Publishing Co., New York.

[19] A. E. Bryson and Y. C. Ho, Applied Optimal Control, Waltham, Toronto,London, 1969.

[20] Z. Brzezniak and R. Serrano, Optimal relaxed control of dissipativestochastic partial differential equations in Banach spaces, arXiv: 1001.3165v2 (2010).

[21] D. Bushaw, Optimal discontinuous forcing terms, Contributions to theTheory of Nonlinear Oscillations IV, Annals of Math Study 41, S. Lef-schetz, Ed., Princeton University Press, Princeton, 1958, 29–52.

[22] C. Casting, Sur les multi-applications measurables, Rev. Francaise Au-tomat. Informat. Recherche Operationnelle 1 (1967), 91–126.

[23] L. Cesari, Existence theorems for optimal solutions in Pontryagin andLagrange problems, SIAM J. Control, 3 (1966), 475–498

[24] L. Cesari, Existence theorems for weak and usual optimal solutions inLagrange problems with unilateral constraints I, Trans. Am. Math. Soc.,124 (1966), 369–412.

[25] L. Cesari, Existence theorems for optimal controls of the Mayer type,SIAM J. Control, 9(1968), 517–552.

[26] L. Cesari, Closure, lower closure, and semicontinuity theorems in opti-mal control, SIAM J. Control, 9(1971), 287–315.

[27] L. Cesari, Optimization—Theory and Applications, Vol. 17 of Applica-tions of Mathematics, Springer-Verlag, New York, 1983.

[28] M. G. Crandall and P. L. Lions, Viscosity solutions of Hamilton-Jacobiequations, Trans. Ams. 277 (1983), 1–42.

Bibliography 373

[29] M. G. Crandall, H. Ishii, and P.-L. Lions, Users guide to viscosity so-lutions of second order partial differential equations, Bull. Am. Math.Soc., 279, 1–67.

[30] B. Dacorogna, Direct Methods in the Calculus of Variations, Springer-Verlag, Berlin, New York, 1989.

[31] N. Dunford and J. T. Schwartz, Linear Operators Part I: General The-ory, Interscience, New York, 1958.

[32] H. G. Eggleston, Convexity, Cambridge University Press, Cambridge,1958.

[33] A. F. Filippov, On certain questions in the theory of optimal control,SIAM J. Control, 1 (1962), 76-89. Orig. Russ. Article in Vestnik Moskov.Univ. Ser. Mat. Mech. Astr., 2 (1959), 25–32.

[34] W. H. Fleming and R. W. Rishel, Deterministic and Stochastic Optimal,Springer-Verlag, New York, 1975.

[35] W. H. Fleming and H. Mete Soner, Controlled Markov Processes andViscosity Solutions, Springer-Verlag, New York, 1993, Second Edition2006.

[36] R. V. Gamkrelidze, Theory of time-optimal process for linear systems,Izv. Akad. Nauk. SSSR. Ser Mat., 22 (1958), 449–474 (Russian).

[37] R. V. Gamkrelidze, On sliding optimal regimes, Dokl. Akad. Nauk SSSR.143 (1962), 1243–1245. Translated as Soviet Math. Dokl., 3 (1962), 1243–1245. 3 (1962), 390–395.

[38] R. V. Gamkrelidze, On some extremal problems in the theory of dif-ferential equations with applications to the theory of optimal control,SIAM J. Control, 3 (1965), 106–128.

[39] R. V. Gamkrelidze, Principles of Optimal Control Theory, Plenum, NewYork, 1978.

[40] M. E. Gurtin and L. F. Murphy, On the optimal harvesting of persistentage-structured populations, J. Math. Biol., 13 (2) (1981), 131–148.

[41] J. Hale, Theory of Functional Differential Equations, Springer-Verlag,New York, 1977.

[42] H. Hermes and J. P. LaSalle, Functional Analysis and Time OptimalControl, Academic Press, New York, 1969.

[43] M. R. Hestenes, Calculus of Variations and Optimal Control Theory,John Wiley, New York, 1966.


[44] E. Hille and R. S. Phillips, Functional Analysis and Semi-Groups, Re-vised Ed., American Mathematical Society, Providence, RI, 1957.

[45] J. G. Hocking and G. S. Young, Topology, Addison-Wesley, Reading,MA, 1961.

[46] M. Q. Jacobs, Attainable sets in systems with unbounded controls, J.Differential Equations, 4 (1968), 408–423.

[47] G. S. Jones and A. Strauss, An example of optimal control, SIAM Re-view, 10 (1) (1968), 25–55.

[48] J. P. LaSalle, Study of the Basic Principle Underlying the Bang-BangServo, Goodyear Aircrafts Corp. Report GER-5518 (July 1953). Ab-stract 247t. Bull. Am. Math. Soc., 60 (1954), 154.

[49] J. P. LaSalle, The time optimal control problem, Contributions to theTheory of Nonlinear Oscillations, Vol. 5, Annals of Math Study No. 45Princeton University Press, Princeton, 1960, 1–24.

[50] E. B. Lee and L. Markus, Foundations of Optimal Control Theory, JohnWiley, New York, 1967.

[51] G. Leitmann, On a class of variational problems in rocket flight, J.Aero/Space Sci., 26 (1959), 586–591.

[52] G. Leitman, An Introduction to Optimal Control, McGraw-Hill, NewYork, 1966.

[53] J. Lindenstrauss, A short proof of Liapounoff’s convexity theorem, J.Math. Mech., 15 (1966), 971–972.

[54] H. Lou, Analysis of the optimal relaxed control to an optimal controlproblem, Appl. Math. Optim., 59 (2009), 75–97.

[55] H. Lou, Existence and non-existence results of an optimal control prob-lem by using relaxed control, SIAM J. Control Optim., 46 (2007) 923–1941.

[56] H. Maurer and M. D. Mittelmann, The non-linear beam via optimalcontrol with bounded state vriables, Optimal Control Applications andMethods, 12 (1991), 19–31.

[57] D. McDonald, Nonlinear techniques for improving servo performance,National Electronics Conference, 6 (1950), 400–421.

[58] E. J. McShane, Integration, Princeton University Press, Princeton, 1944.

[59] E. J. McShane, On multipliers for Lagrange problems, Am. J. Math., 61(1939), 809–819.

Bibliography 375

[60] E. J. McShane, Necessary conditions in generalized-curve problems inthe calculus of variations, Duke Math. J., 6 (1940), 1–27.

[61] E. J. McShane, Existence theorems for Bolza problems in the calculusof variations, Duke Math. J., 7 (1940), 28–61.

[62] E. J. McShane, Generalized curves, Duke Math. J., 6 (1940), 513–536.

[63] E. J. McShane, Relaxed controls and variational problems, SIAM J.Control, 5 (1967), 438–485.

[64] E. J. McShane and R. B. Warfield, Jr., On Filippov’s implicit functionslemma, Proc. Am. Math. Soc., 18 (1967), 41–47.

[65] N. G. Medhin, Optimal processes governed by integral equations, J.Math. Anal. App., 120 (1) (1986), 1–12.

[66] N. G. Medhin, Necessary conditions for optimal control problems withbounded state by a penalty method, JOTA, 52 (1) (1987), 97–110.

[67] N. G. Medhin, Optimal processes governed by integral equations withunilateral constraint, J. Math. Anal. App., 129 (1) (1988), 269–283.

[68] N. G. Medhin, Optimal harvesting in age-structured populations, JOTA,74 (3) (1992), 413–423.

[69] N. G. Medhin, Characterization of optimal pairs for hereditary controlproblems, JOTA, 75 (2) (1992), 355–367.

[70] N. G. Medhin, On optimal control of functional differential systems,JOTA, 85 (2) (1995), 363–376.

[71] N. G. Medhin, Bounded state problem for hereditary control problems,JOTA, 79 (1) (1993), 87–103.

[72] S. Mirica, On the admissible synthesis in optimal control theory anddifferential games, SIAM J. Control, 7 (1969), 292–316.

[73] B. Mordukhovich, Variational analysis of evolution inclusions, SIAM J.Optim., 18 (3) (2007), 752–777.

[74] I. P. Natanson, Theory of Functions of a Real Variable, Eng. trans. byLeo Boron, revised ed., F. Ungar, New York, 1961.

[75] L. W. Neustadt, The existence of optimal controls in the absence ofconvexity conditions, J. Math. Anal. Appl., 7 (1963), 110–117.

[76] L. W. Neustadt, Optimization: A Theory of Necessary Conditions,Princeton University Press, Princeton, NJ, 1977.

[77] C. Olech, Extremal solutions of a control system, J. Differential Equa-tions, 2 (1966), 74–101.


[78] L. A. Pontryagin, V. G. Boltyanskii, R.V. Gamkrelidze, and E. F. Mis-chenko, The Mathematical Theory of Optimal Processes (Translated byK. N. Trirogoff, L. W. Neustadt), John Wiley, Ed., New York, 1962.

[79] L. S. Pontryagin, Optimal regulation processes,Uspehi Mat. Nauk (N.S.)14 (1959), 3–20. English translation in Am. Math. Soc. Transl. Ser., 218(1961), 321–339.

[80] R. T. Rockafellar, Convex Analysis, Princeton University Press, Prince-ton, NJ, 1970.

[81] E. Roxin, The existence of optimal controls, Mich. Math. J., 9 (1962),109–119.

[82] W. Rudin, Real and Complex Analysis, McGraw-Hill, New York, 1966.

[83] P. Sattayatham, Relaxed control for a class of strongly nonlinear impul-sive evolution equations, Comp. Math. Appl., 52 (2006), 779–790.

[84] G. Scorza-Dragoni, Un teorema sulle funzioni continue rispetto ad una amisurabile rispetto ad un’altra variable, Rend. Sem. Mat. Univ. Padova,17 (1948), 102–106.

[85] L. M. Sonneborn and F. S. Van Vleck, The bang-bang principle for linearcontrol systems, SIAM J. Control, 2 (1964), 151–159.

[86] L. Tonelli, Fondamenti del calcolo delle variazioni I, II, Zanichelli,Bologna, 1921, 1923.

[87] John L. Troutman, Variational Calculus and Optimal Control, Springer,New York, 1996.

[88] J. Warga, Relaxed variational problems, J. Math. Anal. Appl., 4 (1962),111–128.

[89] J. Warga, Optimal Control of Differential and Functional Equations,Academic Press, New York, 1972.

[90] X. Xiang, W. Wei, and H. Liu, Relaxed trajectories of integrodifferentialequations and optimal control on Banach Space, Comp. Math. Appl., 52(2006), 735–748.

[91] X. Xiang, P. Sattayatham, and W. Wei, Relaxed controls for a classof strongly nonlinear delay evolution equations, Nonlinear Analysis, 52(2003), 703–723.

[92] L. C. Young, Generalize curves and the existence of an attained abso-lute minimum in the calculus of variations, Compt. Rend. Soc. Sci. etLettres., Varsovie, Cl III 30 (1937), 212–234.

Bibliography 377

[93] L. C. Young, Lectures on the Calculus of Variations and Optimal ControlTheory, W. B. Saunders Co., Philadelphia, 1969.

[94] R. Yue, Lipschitz Continuity of the Value Function in a Class of Op-timal Control Problems Governed by Orindary Differential Equations,Control Theory, Stochastic Analysis, and Applications, World ScientificPublishing, New Jersey, 1991, 125–136.

Index

ε-neighborhood, 53

admissible control, 7, 20, 25, 35, 37,76, 80, 82, 100

admissible pair, 7, 20, 22, 33, 41, 65,80–82, 87–89, 91–93, 95,102, 105, 113, 124, 128

admissible relaxed control, 341admissible relaxed trajectory, 38, 76,

87, 90admissible trajectories, 21admissible trajectory, 20, 21, 25, 27,

28, 36, 37, 90, 91, 103approximate continuity, 238, 239Ascoli, 88, 132, 220, 230attainable set, 79, 104, 111

bang-bang, 112bang-bang principle, 79, 111Bernoulli, 9Bolza problem, 22, 29–31brachistochrone, 9, 11, 12, 27

calculus of variations, 9, 15, 22Bolza problem, 27, 28simple problem, 28

Caratheodory, 42, 111Caratheodory’s Theorem, 107Cesari property, 116, 117, 122, 124,

125, 128, 136, 139, 142, 146,147

chattering control, 78Chattering Lemma, 96chattering lemma, 66, 78Clebsch condition, 181constraint qualification, 30, 32, 181control, 19

control constraints, 19, 21, 23, 123control variable, 18, 28, 29, 102convex function, 137, 189convex hull, 41, 105, 107convex integral, 100cost functional, 25–27, 98, 100

convex integral, 98, 99, 101

du-Bois Reymond equation, 178dynamic programming, 149, 150

Egorov, 119end conditions, 19, 21, 23equi-absolutely continuous, 124, 132equivalent formulation, 17, 42equivalent formulations, 22Euler equations, 213extremal controls, 192extremal element, 165extremal trajectories, 250extremal trajectory, 172, 250extreme points, 79, 105, 111, 112

feedback control, 149, 152, 198, 200Filippov, 93Filippov’s lemma, 56

Galileo, 9Gronwall, 74, 342

Hamilton-Jacobi equation, 154, 198,199

Hausdorff space, 39, 56, 94Hausdorff spaces, 68Hilbert’s differentiability theorem,

180hyperplane, 16, 42

379

380 Index

inequality constraints, 17, 207

Krein-Milman, 105

Lagrange multiplier, 207Lagrange problem, 22, 23Legendre’s condition, 179linear systems, 186linear variety, 188, 198lower closure, 372lower semi-continuous, 87lower semicontinuous, 63

maximum principle, 172maximum principle in integrated

form, 161, 164Mayer problem, 22, 23Mazur, 105Mazur’s Theorem, 108, 109McShane and Warfield, 56, 375minimizing sequence, 89, 90, 113minimum fuel, 6, 17, 99minimum fuel problem, 6multiplier rule, 180

Nagumo-Tonelli, 137

optimal control, 1optimal pair, 21optimal trajectory, 21

parameter optimization, 26, 172partition of unity, 68payoff, 17, 66pointwise maximum principle, 165production planning, 1, 16, 17, 173

quadratic cost criterion, 150, 288

relaxed admissible pair, 41relaxed attainable set, 105, 111relaxed control, 64, 83relaxed controls, 35, 40relaxed problem, 36–38, 92relaxed trajectories, 40relaxed trajectory, 43, 65, 83rendezvous, 6

Riccati equation, 199, 200rocket problem, 27

Scorza-Dragoni, 376servo-mechanism, 7simple problem, 27, 28state equations, 20, 22, 23, 25, 28state variable, 18, 24, 26strongly normal, 189synthesis, 149, 152

terminal set, 16terminal state, 16, 173time optimal, 192trajectory, 19transversality condition, 166, 194transversality conditions, 157two-point boundary value problem,

159

upper semi-continuous with respectto inclusion, 53

value function, 149, 151

weak compactness, 43, 84weak convergence, 45, 119Weierstrass condition, 179, 181Weierstrass-Erdmann, 178

Nonlinear Optimal

Documents

Transcript of Nonlinear Optimal