Economics 2010c: Lecture 2Iterative Methods in Dynamic Programming

David Laibson

9/04/2014

Outline:

1. Functional operators

2. Iterative solutions for the Bellman Equation

3. Contraction Mapping Theorem

4. Blackwell’s Theorem (Blackwell: 1919-2010, see obituary)

5. Application: Search and stopping problem

1 Functional operators:

Sequence Problem: Find () such that

(0) = sup{+1}∞=0

∞X=0

( +1)

subject to +1 ∈ Γ() with 0 given.

Bellman Equation: Find () such that

() = sup+1∈Γ()

{ ( +1) + (+1)} ∀

Bellman operator operating on function is defined

()() ≡ sup+1∈Γ()

{ ( +1) + (+1)} ∀

• Definition is expressed pointwise — for one value of — but applies to allvalues of

• Operator maps function to a new function .

• Operator maps functions; it is referred to as a functional operator.

• The argument of — the function — may or may not be a solution tothe Bellman Equation.

• Let be a solution to the Bellman Equation.

• If = then =

()() = sup+1∈Γ() { ( +1) + (+1)} ∀= () ∀

• Function is a fixed point of ( maps to ).

2 Iterative Solutions for the Bellman Equation

1. (Guess a solution — from last lecture. This is not iterative.)

2. Pick some 0 and analytically iterate 0 until convergence.(This is really just for illustration.)

3. Pick some 0 and numerically iterate 0 until convergence.(This is the way that iterative methods are used in most cases.)

What does it mean to “iterate ”?

()() = sup+1∈Γ() { ( +1) + (+1)}(())() = sup+1∈Γ() { ( +1) + ()(+1)}((2))() = sup+1∈Γ()

n ( +1) + (2)(+1)

o... = ...(())() = sup+1∈Γ() { ( +1) + ()(+1)}

• What does it mean for functions to converge? Notation:

lim→∞0 =

• Informally, as gets large, the set of remaining elements of the sequenceof functions n

0 +10

+20 +30

oare getting closer and closer together.

• What does it mean for one function to be “close” to another? Themaximum distance between the functions is bounded by

• We will not worry about these technical issues in this mini-course.

3 Contraction Mapping Theorem

Why does converge as →∞?

Answer: is a “contraction mapping.”

Definition 3.1 Let ( ) be a metric space and : → be a functionmapping into itself. is a contraction mapping if for some ∈ (0 1)

() ≤ ( ) for any two functions and .

Intuition: is a contraction mapping if operating on any two functionsmoves them strictly closer together: and are strictly closer togetherthan and

Remark 3.1 A metric (distance function) is just a way of representing thedistance between functions (e.g., the supremum pointwise gap between thetwo functions).

Theorem 3.1 If ( ) is a complete metric space and : → is acontraction mapping, then:1) has exactly one fixed point ∈

2) for any 0 ∈ lim0 =

3) 0 has an exponential convergence rate at least as great as − ln()

Consider the contraction mapping ()() ≡ ()+() with ∈ (0 1)

()() = () + ()

(())() = 2() + () + ()

((2))() = 3() + 2() + () + ()

()() = () + −1() + + ()

lim()() = ()(1− )

So fixed point, () is

() = ()(1− )

Note that ()() = ()

Proof of Contraction Mapping Theorem

Broad strategy:

1. Show that {0}∞=0 is a Cauchy sequence.

2. Show that the limit point is a fixed point of .

3. Show that only one fixed point exists.

4. Bound the rate of convergence.

Choose some 0 ∈ Let = 0 Since is a contraction

(2 1) = (1 0) ≤ (1 0)

Continuing by induction,

(+1 ) ≤ (1 0) ∀

We can now bound the distance between and when

( ) ≤ ( −1) + + (+2 +1) + (+1 )

≤h−1 + + +1 +

i(1 0)

= h−−1 + + 1 + 1

i(1 0)

≤

1− (1 0)

So {}∞=0 is Cauchy. X

Since is complete → ∈

We now have a candidate fixed point ∈

To show that = note

( ) ≤ (0) + (0 )≤ (−10) + (0 )

These inequalities must hold for all

And both terms on the RHS converge to zero as →∞

So ( ) = 0 implying that = X

Now we show that our fixed point is unique.

Suppose there were two fixed points: 6= ∗

Then = and ∗ = ∗ (since fixed points).

Also have (∗) ( ∗) (since is contraction)

So, ( ∗) = (∗) ( ∗)

Contradiction.

So the fixed point is unique. X

Finally, note that

(0 ) = ((−10) )≤ (−10 )

So (0 ) ≤ (0 ) follows by induction. X

This completes the proof of the Contraction Mapping Theorem.

4 Blackwell’s Theorem

These are sufficient conditions for an operator to be contraction mapping.

Theorem 4.1 (Blackwell’s sufficient conditions) Let ⊆ < and let ()be a space of bounded functions : → <, with the sup-metric. Let : ()→ () be an operator satisfying two conditions:1) (monotonicity) if ∈ () and () ≤ () ∀ ∈ ,then ()() ≤ ()() ∀ ∈

2) (discounting) there exists some ∈ (0 1) such that

[( + )]() ≤ ()() + ∀ ∈ () ≥ 0 ∈

Then is a contraction with modulus

Remark 4.1 Note that is a constant and ( + ) is the function generatedby adding to the function

Remark 4.2 Blackwell’s conditions are sufficient but not necessary for tobe a contraction.

Proof of Blackwell’s Theorem:

For any ∈ ()

() ≤ () + ( ) ∀

We’ll write this as

≤ + ( )

So property 1) and 2) imply

≤ ( + ( )) ≤ + ( )

≤ ( + ( )) ≤ + ( )

Combining the last two lines, we have

− ≤ ( )

− ≤ ( )

|()()− ()()| ≤ ( ) ∀

sup|()()− ()()| ≤ ( )

() ≤ ( ) X

Example 4.1 Check Blackwell conditions for Bellman operator in a consump-tion problem (with stochastic asset returns, stochastic labor income, and aliquidity constraint).

()() = sup∈[0]

n() + (̃+1(− ) + ̃+1)

o∀

Monotonicity. Assume () ≤ () ∀. Suppose ∗ is the optimal policywhen the continuation value function is

()() = sup∈[0]

n() + (̃+1(− ) + ̃+1)

o= (∗) + (̃+1(− ∗) + ̃+1)

≤ (∗) + (̃+1(− ∗) + ̃+1)

≤ sup∈[0]

n() + (̃+1(− ) + ̃+1)

o= ()()

Discounting. Adding a constant (∆) to an optimization problem does notaffect optimal choice, so

[( +∆)]() = sup∈[0]

n() +

h(̃+1(− ) + ̃+1) +∆

io= sup

∈[0]

n() + (̃+1(− ) + ̃+1)

o+ ∆

= ()() + ∆

5 Application: Search/ stopping.

An agent draws an offer, from a uniform distribution with support in theunit interval. The agent can either accept the offer and realize net presentvalue (ending the game) or the agent can reject the offer and draw againa period later. All draws are independent. Rejections are costly because theagent discounts the future with discount factor . This game continues untilthe agent receives an offer that she is willing to accept.

The Bellman equation for this problem is:

() = max{ h(0)

i}

• In the first lecture, we showed that the solution would be a stationarythreshold:

REJECT if ≤ ∗

ACCEPT if ≥ ∗

• This implies that

() =

(∗ if ≤ ∗

if ≥ ∗

)

• We showed that if

∗ = −1µ1−

q1− 2

¶

then, () solves the Bellman Equation.

Iteration of Bellman Operator starting with 0() = 0 ∀

• This is the Bellman Operator for our problem:

()() ≡ max{ h(0)

i}

• Iterating the Bellman Operator once on 0() yields

1() = (0)()

= max{ h0(

0)i}

= max{ 0}=

• So associated threshold cutoff is 0 In other words, accept all offers greaterthan or equal to 0 since the continuation payoff function is 0 = 0

General notation.

• Iteration of the Bellman Operator, .

() = (0)()

= (−10)()= max{

h(−10)(

0)i}

• Let ≡ h(−10)(0)

i, which is the continuation payoff for ()

• is also the cutoff threshold associated with () = (0)()

() = (0)() =

( if ≤ if ≥

)

is a sufficient statistic for (0)()

Another illustrative example. Iterate the functional operator on 0

2() = (20)()

= (0)()

= max{ (0)(0)}= max{ 1(0)}= max{ 0}

2 = 0 =

2

2() = (20)() =

(2 ≤ 2 ≥ 2

)

The proof itself:

(−10)() =

(−1 if ≤ −1 if ≥ −1

)

= (−10)(0)

=

"Z −1

=0−1()+

Z =1

=−1()

#

=

2

³2−1 + 1

´

Set = −1 to confirm that this sequence converges to

lim→∞ = −1

µ1−

q1− 2

¶

1 2 3 4 5 6 7 8 9 100

0.1

0.2

0.3

0.4

0.5

0.6

0.7

iteration number

x n v

alue

Iterating functional for search/stopping problem with discount rate =.1 = -ln(delta)

x1

x2

Review of Today’s Lecture:

1. Functional operators

2. Iterative solutions for the Bellman Equation

3. Contraction Mapping Theorem

4. Blackwell’s Theorem

5. Application: Search and stopping problem (analytic iteration)