Estimating Discrete Choice Dynamic Programming...

Discrete Choice Models Dynamic Programming Models Discrete-Choice Dynamic-Programming Models References

Estimating Discrete Choice DynamicProgramming Models

Katsumi Shimotsu1 Ken Yamada2

1Department of Economics, Hitotsubashi University

2School of Economics, Singapore Management University

Japanese Economic Association Spring MeetingTutorial Session (May, 2011)


Introduction to Part II

• The second part focuses on econometric implementation ofdiscrete choice dynamic programming models.

• A simple machine replacement model is estimated using thenested fixed point algorithm (Rust, 1987).

• For more applications, see Aguirregabiria and Mira (2010) andKeane, Todd, and Wolpin (2010).

• For audience with different backgrounds, I will briefly review

1. Discrete choice models and2. Numerical methods for (i) maximum likelihood estimation and

(ii) dynamic programming.


Outline

Discrete Choice ModelsThe Random Utility ModelMaximum Likelihood Estimation

Dynamic Programming ModelsMachine ReplacementNumerical Dynamic Programming

Discrete-Choice Dynamic-Programming ModelsThe Nested Fixed Point AlgorithmThe Nested Pseudo Likelihood Algorithm

References


The Random Utility Model• Consider a static problem in which the agent chooses among J

alternatives, such as transportation mode and brand choice.• Assume that the choice-specific utility can be expressed as

Vij = uij + εij i = 1, . . . ,N, j = 0 . . . ,J ,

where uij is a deterministic component while εij is a stochasticcomponent.

• A simple example is uij = xijθ , where xij is a vector of observedcharacteristics such as price and income.

• Let A = {1, . . . ,J} denote the choice set. The choice probabilityis

P (a) =∫

ε

1{

a = argmaxj∈A

(uij + εij)

}f (εi )dεi ,

where f (εi ) is a joint density of εi = (εi1, . . . ,εiJ)′.


The Conditional Logit Model• Assume that each εj is independently, identically distributed

extreme value. The density is f (εj) = e−εj exp(−e−εj ). Thechoice probability follows the logit model (McFadden, 1981).

pij = P (a = j) =exp(uij)

∑Jj=1 exp(uij)

=exp(uij −ui1)

1 + ∑Jj=2 exp(uij −ui1)

• Note that this is a system of J−1 equations. For J = 2,

pi2 =exp(ui2−ui1)

1 + exp(ui2−ui1)= Λ(ui2−ui1) .

There exists a mapping between the utility differences andchoice probabilities.

ui2−ui1 = Λ−1 (pi2) or ui2−ui1 = ln(

pi21−pi2

)See Hotz and Miller (1993) for the details on invertibility.


Outline




References


MLE

• The choice probability is

P (a = j) =exp(uij)

∑Jj=1 exp(uij)

• Suppose Vij = xijθ + εij . The log-likelihood function is

l (θ) =N

∑i=1

li (θ) =N

∑i=1

J

∑j=1

1(ai = j) lnP (ai = j |xij) .

• If the model is correctly specified,

θd→N

(θ ,E

(− ∂ 2l

∂θ∂θ ′

)−1).


Numerical Maximization (Train, 2009)


Numerical Methods for MLE

• The Newton-Raphson procedure uses the following formula.

θk+1 = θk +(−H−1

k)

gk ,

where gk = ∂ l/∂θ |θk

and Hk = ∂ 2l/

∂θ∂θ ′∣∣θk

.

• Quasi-Newton methods:[fval,theta,g,H] = csminwel(′fun′,theta0,H0,g,crit,nit) byChristopher Sims

• Non-gradient based algorithm (simplex methods):[theta,fval] = fminsearch(′fun′,theta0,options)


Extension to Dynamic Models• We may be concerned with dynamics in some applications such

as firm’s adjustment of labor and capital, educational andoccupational choices.

• The value function V is the unique solution to the Bellmanequation:

V (xt ,εt) = maxj∈A{Vjt = V (xt ,εt ,at = j)} ,

where the choice-specific value function (including ε) is

V (xt ,εt ,at) = u (xt ,at) + εt (at) + βE [V (xt+1,εt+1)|xt ,εt ,at ]

= {u (xt ,at) + βE [V (xt+1,εt+1)|xt ,εt ,at ]}+ εt (at)

= v (xt ,at) + εt (at)

for β ∈ (0,1). The model considered here is a natural extensionof the conditional logit model (or the random utility model) to adynamic setting.


Outline




References


Plant Investment Decisions• Here we consider the plant’s investment decision problem set out

by Kasahara at UBC. The tradeoff is that investment (machinereplacement), a ∈ {0,1},

1. requires large fixed costs for now but2. increases profits and lowers replacement costs in the future.

• The profit function is

u (zt ,at−1,at) = R (zt ,at)−FC (at ,at−1) ,

where R is revenues net of variable input costs, zt is plant’sproductivity, and FC is fixed costs.

• Assume that zt = ρzt−1 + ηt , where ηt ∼N(0,σ2

η

). Specify

the profit function as

u (zt ,at−1,at) = u (zt−1,at−1,at)

= exp(θ0 + θ1zt−1 + θ2at)− [θ3at + θ4 (1−at−1)at ] .


Transition Probability

• Let Z = {z1, . . . ,zg , . . . ,zG}.• Denote the transition probability by

qgh = q (zt = zh|zt−1 = zg ) = Pr (z ∈ [zh,zh+1]|z ∈ [zg ,zg+1]) .

By the Tauchen method, this can be approximated by

qgh =

Φ((z1−ρzg )/ση ) if h = 1,

Φ((zh−ρxg )/ση )−Φ((zh−1−ρzg )/ση ) if 1 < h < G ,1−Φ((zG −ρzg )/ση ) if h = G ,

where zg = (zg + zg+1)/2 and Φ is the standard normal cdf.


The Choice-Specific Value Function

• Define the integrated value function as

V (zt−1,at−1) =∫

εtV (zt−1,at−1,εt)dF (εt) .

• Assume that each εt (at) is iid extreme value.

V (zt−1,at−1) = γ + ln(

J

∑j=1

exp(v (zt−1,at−1,at = j))

),

where γ is the Euler constant.• The choice-specific value function is

v (zt−1,at−1,at) = u (zt−1,at−1,at)+β ∑xt

q (zt |zt−1)V (zt−1,at−1) .


Outline




References


Infinite Horizon Problem• In the infinite-horizon problem with a finite number of states, the

Bellman equation is a system of nonlinear equations.

Vg = maxa∈A

[u (xg ,a) + β

G

∑h=1

qgh (a)Vh

],

where a ∈ {1, . . . ,J} is a choice variable,x ∈ {x1, . . . ,xg , , . . . ,xG} is a state variable, and qgh (a) is thetransition probability from state g to state h.

• Let a = (a1, . . . ,aG)′ denote the policy function andua = (u (x1,a1) , . . . ,u (xG ,aG))′ the return. Then, in vectornotation, Va = ua + βQaVa, which leads to the solution:

Va = (I−βQa)−1 ua,

where the ghth element of Qa is qgh (a). For details, see Judd(1998) and Adda and Cooper (2003).


Value Iteration Algorithm

1. Choose a grid X = {x1, . . . ,xg , , . . . ,xG}, where xg < xh forg < h, and specify u (x ,a) and qgh (a).

2. Make an initial guess V 0 and choose stopping criterion c > 0.

3. For g = 1, . . . ,G , compute

V k+1g = max

a∈A

{u (xg ,a) + β

G

∑h=1

qgh (a)V kh

}.

4. If∥∥V k −V k+1∥∥< c , stop; else go to step 3.


Policy Iteration Algorithm

1. Choose a grid X = {x1, . . . ,xg , , . . . ,xG}, where xg < xh forg < h, and specify u (x ,a) and qgh (a).

2. Make an initial guess V 0 and choose stopping criterion c > 0.

3. Compute ak+1 =(

ak+11 , . . . ,ak+1

G

)′, where

ak+1g = argmax

a∈A

{u (xg ,a) + β

G

∑h=1

qgh (a)V kh

}.

4. Compute uk+1 =(

u(

x1,ak+11

), . . . ,u

(xG ,ak+1

G

))′.

5. Compute

Vk+1 =(

I−βQak+1)−1

uk+1.

6. If∥∥V k −V k+1∥∥< c , stop; else go to step 3.


Outline




References


The Nested Fixed Point Algorithm (Rust, 1987)

• The conditional choice probability takes the form:

P (a|x) =exp(u (x ,a,θ) + βEV (x ′,ε ′))

∑Jj=1 exp(u (x , j ,θ) + βEV (x ′,ε ′))

.

• The transition probability can be obtained separately.

1. Inner loop: Given θ , compute the value function V using thevalue iteration algorithm or the policy iteration algorithm, andobtain the conditional choice probability P .

2. Outer loop: Given P , obtain a pseudo-likelihood estimate of θ

using the Newton method.

3. Repeat steps 1 and 2 until convergence.


Outline




References


Hotz and Miller (1993) and Aguirregabiria and Mira (2002)• Denote the optimal decision rule by do

jt = 1{at = j |xt ,εt}.

V (xt) = E

{T

∑τ=t

J

∑j=1

βτ−tdo

jt [u (xt ,at) + εt (at)]

}The Bellman equation can be rewritten as

V (x) = ∑a

P (a|x)

[u (x ,a) +E [ε|x ,a] + β ∑

x ′q(

x ′∣∣x ,a)V

(x ′)]

.

• The system of equations can be solved for the value function. Inmatrix notation,

V =

(I−β ∑

aP (a)◦Q (a)

)−1

∑a

P (a)◦[u (a) + (γ− lnP (a))] ,

where ◦ is the element-by-element product.


The Conditional Logit Form• Suppose u (x ,a) = z (x ,a)α . The policy iteration operator is

Ψ(P) =exp(z (x ,a)α + e (x ,a))

∑Jj=1 exp(z (x ,a)α + e (x ,a))

,

where

z (x ,a) = z (x ,a) + β ∑x ′

q(

x ′∣∣x ,a)Wz

(x ′,P

),

e (x ,a) = β ∑x ′

q(

x ′∣∣x ,a)We

(x ′,P

),

Wz(x ′,P

)=

[I−β ∑

jP ◦Q

]−1

∑a

P ◦ z ,

We(x ′,P

)=

[I−β ∑

aP ◦Q

]−1

∑a

P ◦ (γ− ln(P)) .


The Nested Pseudo Likelihood Algorithm

• Make an initial guess P0 and choose the number of iterations K .

1. Given z (x ,a) = z (x ,a) + β ∑x ′ q (x ′|x ,a)Wz(x ′,Pk−1) and

e (x ,a) = β ∑x ′ q (x ′|x ,a)We(x ′,Pk−1), obtain a

pseudo-likelihood estimate of αk as

ak = argmaxα

N

∑i=1

Ψ(

Pk−1).

2. Given ak , update P using

Pk = Ψ(

Pk−1).

3. Iterate steps 1 and 2 until k = K .


References1. Adda, J. and R. Cooper, (2003): Dynamic Economics: Quantitative Methods and

Applications, MIT Press.

2. Aguirregabiria, V. and P. Mira, (2002): “Swapping the nested fixed point algorithm: Aclass of estimators for discrete Markov decision models,” Econometrica.

3. Aguirregabiria, V. and P. Mira, (2010): “Dynamic discrete choice structural models: Asurvey,” Journal of Econometrics.

4. Hotz, J. and R. A. Miller, (1993). “Conditional choice probabilities and the estimation ofdynamic models,” Review of Economic Studies.

5. Judd, K. L. (1998): Numerical Methods in Economics, MIT Press.

6. Keane, M. P., P. E. Todd, and K. I. Wolpin, (2010). “The structural estimation ofbehavioral models: discrete choice dynamic programming methods and applications,”Handbook of Labor Economics, Vol. 4A.

7. McFadden, D. L. (1981): “Structural discrete probability models derived from theories ofchoice,” in C. F. Manski and D. L. McFadden (eds.) Structural Analysis of Discrete Dataand Economic Applications, MIT Press.

8. Rust, J. (1987): “Optimal replacement of GMC bus engines: An empirical model ofHarold Zurcher” Econometrica.

9. Train, K. (2009): Discrete Choice Methods with Simulation, Cambridge UniversityPress.

Estimating Discrete Choice Dynamic Programming...

Documents

Transcript of Estimating Discrete Choice Dynamic Programming...