1

Solving Infinite Horizon Stochastic Optimization Problems

John R. BirgeNorthwestern University

(joint work with Chris Donohue, Xiaodong Xu, and

Gongyun Zhao)

2

Motivating Problem

• (Very) long-term investor (example: university endowment)

• Payout from portfolio over time (want to keep payout from declining)

• Invest in various asset categories• Decisions:

• How much to payout (consume)?• How to invest in asset categories?

• Complication: restrictions on asset trades

3

Problem Formulation• Notation:

x – current state (x 2 X)

u (or ux) – current action given x (u (or ux) 2 U(x)) – single period discount factor

Px,u – probability measure on next period state y depending on x and u

c(x,u) – objective value for current period given x and uV(x) – value function of optimal expected future rewards given

current state x

• Problem: Find V such that V(x) = maxu2 U(x){c(x,u) + EPx,u[V(y)] }for all x 2 X.

4

Approach

• Define an upper bound on the value functionV0(x) ¸ V(x) 8 x 2 X

• Iteration k: upper bound Vk

Solve for some xk

TVk(xk) = maxu c(xk,u) + EPxk,u[Vk(y)]

Update to a better upper bound Vk+1

• Update uses an outer linear approximation on Uk

5

Successive Outer Approximation

V*

V0

TV0

x0

V1

6

Properties of Approximation

• V*· TVk · Vk+1· Vk

• Contraction|| TVk – V* ||1 · ||Vk – V*||1

• Unique Fixed PointTV*=V*

) if TVk ¸ Vk, then Vk=V*.

7

Convergence• Value Iteration

Tk V0 ! V*

• Distributed Value IterationIf you choose every x2 X infinitely often, then Vk ! V*.(Here, random choice of x, use concavity.)

• Deepest Cut Pick xk to maximize Vk(x)-TVk(x) DC problem to solveConvergence again with continuity (caution on boundary of

domain of V*)

8

Details for Random Choice

• Consider any x

• Choose i and xi s.t. ||xi – x|| < i

• Suppose || r Vi || · K 8 i|| Vk(x) – V*(x)|| · ||Vk(x)-Vk(xk)+Vk(xk)-V*(xk)

+V*(xk)-V*(x)||

· 2 k K + ||Vk-1(xk)-V*(xk)||

· 2i i K + k ||V0(x0) – V*(x0)||

9

Cutting Plane AlgorithmInitialization: Construct V0(x)=maxu c0(x,u) + EPx,u[V0(y)], where c0¸ c and c0 concave.

V0 is assumed piecewise linear and equivalent to

V0(x)=max { | · E0 x + e0}. k=0.

Iteration: Sample xk2 X (in any way such that the probability of xk2 A is positive for any A½ X of positive measure) and solve

TVk(xk) = maxu c(xk,u) + EPxk,u[Vk(y)] where

Vk(y) =max{ | · El y + el,l=0,…,k.}

Find supporting hyperplanes defined by Ek+1 and ek+1such that Ek+1 x + ek+1 ¸ TVk (x). kà k+1.

Repeat.

10

Specifying AlgorithmFeasibility:

Ax + Bu · bTransition:

y=Fi u for some realization i with probability pi

Iteration k Problem:

TVk(xk) = maxu, c(xk,u) + i pi i

s.t. A xk + B u · b, - El(Fi u) - el + i · 0, 8 i,l.From duality:

TVk(xk) = inf,l,i maxu, c(xk,u) -(Axk+Bu-b)

+ i (pi i+l i,l(El(Fi u) + el - i))

· maxu, c(xk,u) -k(Axk+Bu-b) + i (pi i+l i,l,k(El(Fi u) +el - i)) for optimal k, i,l,k for xk

· c(xk,uk) + r c(xk,uk)T (x-xk,-uk) -k Ax +k b +i ( l i,l,kel)

Cuts:Ek+1= rx c(xk,uk)T -k A ek+1 equal to the constant terms.

11

Results

• Sometimes convergence is fast

12

Results

• Sometimes convergence is slow

13

Challenges

• Constrain feasible region to obtain convergence results

• Accelerate the DC search problem to find the deep cut

• Accelerate overall algorithm using:• multiple simultaneous cuts?• nonlinear cuts?• bundles approach?

14

Conclusions

• Can formulate infinite-horizon investment problem in stochastic programming framework

• Solution with cutting plane method• Convergence with some conditions• Results for trade-restricted assets

significantly different from market assets with same risk characteristics

15

Investment Problem• Determine asset allocation and consumption policy to

maximize the expected discounted utility of spending• State and Action

x=(cons, risky, wealth) u=(cons_new,risky_new)

• Two asset classes • Risky asset, with lognormal return distribution• Riskfree asset, with given return rf

• Power utility function

• Consumption rate constrained to be non-decreasingcons_new ¸ cons

1

__

1newconsnewconsc

16

Existing Research• Dybvig ’95*

• Continuous-time approach• Solution Analysis

• Consumption rate remains constant until wealth reaches a new maximum

• The risky asset allocation is proportional to w-c/rf, which is the excess of wealth over the perpetuity value of current consumption

• decreases as wealth decreases, approaching 0 as wealth approaches c/rf (which is in absence of risky investment sufficient to maintain consumption indefinitely).

• Dybvig ’01• Considered similar problem in which consumption rate can

decrease but is penalized (soft constrained problem)

* “Duesenberry's Ratcheting of Consumption: Optimal Dynamic Consumption and Investment Given Intolerance for any Decline in Standard of Living” Review of Economic Studies 62, 1995, 287-313.

17

Objectives

• Replicate Dybvig continuous time results using discrete time approach

• Evaluate the effect of trading restrictions for certain asset classes (e.g., private equity)

• Consider additional problem features• Transaction Costs• Multiple risky assets

18

Results – Non-decreasing Consumption

0

10

20

30

40

50

60

70

80

90

100

2 2.25 2.5 2.75 3 3.25 3.5 3.75 4 4.25 4.5 4.75 5

Consumption Rate

Allo

cati

on

to

Sto

ck

Dybvig

N = 1

N = 2

N=3

N=4

N = 6

N=12

As number of time periods per year increases, solution converges to continuous time solution

19

Results – Non-Decreasing Consumption with Transaction Costs

0

10

20

30

40

50

60

3.5 3.7 3.9 4.1 4.3 4.5 4.7 4.9

Consumption Rate

Sto

ck A

llocati

on

No Transaction Cost

Transaction Cost (initial stock allocation = 0%)

Transaction Cost (initial stock allocation = 100%)

20

Observations

• Effect of Trading Restrictions• Continuously traded risky asset: 70% of

portfolio for 4.2% payout rate• Quarterly traded risky asset: 32% of portfolio

for same payout rate

• Transaction Cost Effect• Small differences in overall portfolio

allocations• Optimal mix depends on initial conditions

21

Extensions

• Soft constraint on decreasing consumption• Allow some decreases with some penalty

• Lag on sales• Waiting period on sale of risky assets (e.g., 60-

day period)

• Multiple assets• Allocation bounds

22

Approach

• Application of typical stochastic programming approach complicated by infinite horizon

• Initialization.• Define a valid constraint on Q(x)

xTbAxts

xQexcpxQ

iii

iiiii

t

..

max

00 exExQi

Requires problem knowledge. For optimal consumption problem, assume extremely high rate of consumption forever

23

Approach (cont.)• Iteration k

wherexVxUFind kkx

k ,min Expensive search over x, possible for the optimal consumption problem because of small number of variables

andexExV jj

kj

k ,min0

1,,0,

..

max

kjexE

xTbAxts

excpxU

jj

i

tk

ii

iii

iiii

i

ki

k

k

ebpe

TpE

cutnewadefineElse

If

iiii

iii

,

terminate.,