An Introduction toDiscrete–Event Simulation

Peter W. Glynn

Peter J. Haas

IMA Workshop, May 12, 2008

Markov Jump Processes

Goal: Compute u∗(t) = (u∗(t, x) : x ∈ S),

where u∗(t, x) = Exf(X(t))

Method: Solve

u′(t) = Qu(t)

s/t u(0) = f

IMA Workshop 2 of 22

Classical method for establishing

existence/uniqueness:

u∗n(t, x) = Exf(X(t))I(J(t) ≤ n)

u∗n(t, x) = f(x)e−λ(x)t

+∑

y 6=x

∫ t

0

λ(x)e−λ(x)s · Q(x, y)

λ(x)u∗

n−1(t− s, y)ds

Now, let n→∞.

Note: Jump time out of x determined by single

exponential rv

IMA Workshop 3 of 22

Classical method for simulating Markov jump

processes:

1. n← 0, T ← 0

2. Generate η ∼ Exp(λ(X(T )))

3. Generate Y from p.m.f.Q(X(T ), ·)λ(X(T ))

4. T ← T + η

5. X(T )← Y and go to 2.

( aka Gillespie algorithm... but goes back to early days

of Markov jump processes)

IMA Workshop 4 of 22

Competing Exponentials

e.g. N independently evolving individuals

����

A-µ

�λ ��

��B

X(t) = # of type A individuals at time t

X = (X(t) : t ≥ 0) is a birth–death process with rates

λ(x) = λx and µ(x) = (N − x)µ.

Simulation: One exponential

vs

N competing exponentials

IMA Workshop 5 of 22

Competing exponentials can be more efficient when :

� one uses an efficient data structure to store FES

� “Classical method” is slow because:(

Q(X(Tn), y)

λ(X(Tn)): y ∈ S

)

must be dynamically computed and/or

(Q(X(Tn), y)

λ(X(Tn)): y ∈ S

)

is non–sparse0 p1 p2

@@IU

pn

IMA Workshop 6 of 22

Thinning

� X = (X(t) : t ≥ 0) Markov jump process with

time–dependent rate matrix Q(t)

� Assume Λ = sup {λ(t, x) : x ∈ S, t ≥ 0} <∞

� Generate points S1, S2, . . . according to a Poisson

process with rate Λ � Accept with prob. λ(Si, X(T ))/Λ

� At transition epoch, generate next state according to

p.m.f. (Q(Si, X(T ), y)/λ(Si, X(T )) : y ∈ S)

IMA Workshop 7 of 22

Variance Reduction Methods

for Markov Jump Processes

I. Discrete–Time Conversion

Y = (Yn : n ≥ 0) embedded DTMC

To estimate α = Eg(X(t) : t ≥ 0), note that

α = EW

where W = E[g(X(t) : t ≥ 0)|Y ].

e.g. g(X(t) : t ≥ 0) = inf{t ≥ 0 : X(t) ∈ A}

W =

τA−1∑

i=0

1/λ(Yi)

var W ≤ var g(X)

IMA Workshop 8 of 22

II. Antithetics

X = (X(t) : t ≥ 0) birth–death process

X1(t) = X(0) + N+

(∫ t

0

λ(X1(s))ds

)−N−

(∫ t

0

µ(X1(s))ds

)

X2(t) = X(0) + N−

(∫ t

0

λ(X2(s))ds

)−N+

(∫ t

0

µ(X2(s))ds

)

IMA Workshop 9 of 22

If g(X) (, g(X(t) : t ≥ 0)) is monotone in N+, N−, then

var

(g(X1) + g(X2)

2

)< var g(X).

IMA Workshop 10 of 22

III. Control Variates

Suppose we can compute Eβ. Then α = Eg(X) can

be estimated via

g(X)− λ(β − Eβ).

Optimal λ∗ (to reduce variance maximally) is

λ∗ =cov(g(X), β)

varβ

IMA Workshop 11 of 22

For any Markov jump process and function

f : [0,∞)× S → R,

M(t) , f(t,X(t))−∫ t

0

[∂f(s,X(s))

∂s+ (Q(s)fs)(X(s))

]ds

is a martingale, so

ExM(t) = f(0, x).

IMA Workshop 12 of 22

If we wish to compute Exg(X(t)), note that if we choose

f as the solution to

∂f

∂s= Qf

s/t f(0) = g,

then∂f(t− s,X(s))

∂s+ (Qft−s)(X(s)) = 0 and we

conclude that

E[f(0, X(t))− f(t, x)] = 0

i.e. g(X(t))− f(t, x) is a control

IMA Workshop 13 of 22

Schlogl Model

R0

-1

�3

R R2 + 2R-3

�1

3R

Xn(t) = # molecules at time t of a substance R in a

volume n

Xn is a birth–death process with birth–death rates:

λn(x) = n(1 + 3x(x− 1/n))

µn(x) = n(3x + x(x− 1/n)(x− 2/n))

IMA Workshop 14 of 22

n−3/4Xn(n1/2t)− n1/4 ⇒ Z(t)

where

dZ(t) = −Z(t)3dt + 2√

2dB(t)

In place of f(s, x) = Exg(X(s)), use

fapprox(s, x) = En−3/4x−n1/4g(n1/4 + n3/4Z(n−1/2t))

IMA Workshop 15 of 22

IV. Sensitivity Analysis

Goal: Compute α = Eg(X1)− Eg(X2)

Xi: birth–death with birth rates (λi(x) : x ≥ 0) and

death rates (µi(x) : x ≥ 1).

e.g. λ1(x) = λ(θ, x)

λ2(x) = λ(θ + h, x)

Xi(t) = Xi(0)+N+

(∫ t

0

λi(Xi(s))ds

)−N−

(∫ t

0

µ(Xi(s))ds

)

same N+, N−

IMA Workshop 16 of 22

Use of change–of–measure

Pλ: probability under which N evolves as a Poisson

process with rate λ

Pλ2(τ1 ∈ dt1, . . . , τn ∈ dtn) =

n∏

i=1

λ2e−λ2tidti

=n∏

i=1

(λ2

λ1

)e−(λ2−λ1)tiλ1e

−λ1tidti

= Eλ1I(τ1 ∈ dt1, . . . , τn ∈ dtn)Ln

Ln = exp

(−(λ2 − λ1)Tn +

∫

[0,Tn]

(log λ2 − log λ1)N(ds)

)

IMA Workshop 17 of 22

More generally:

Pi: X evolves according to

X(t) = X(0)+N+

(∫ t

0

λi(X(s))ds

)−N−

(∫ t

0

µ(X(s))ds

)

P2(A) = E1I(A)L(t)

L(t) = exp

(−

∫ t

0

(λ2(X(s))− λ1(X(s))) ds

+

∫

[0,t]

(log(λ2(X(s−)))− log(λ1(X(s−))))N+(ds)

)

IMA Workshop 18 of 22

For sensitivity analysis:

λ2(x) = λ(h, x)

d

dhEhg(X) = E0g(X)

d

dhL(h, t)

∣∣∣∣h=0

=E0g(X)

[∫

[0,t]

λ′(0, X(s−))

λ(0, X(s−))N+(ds)−

∫ t

0

λ′(0, X(s))ds

]

IMA Workshop 19 of 22

For rare–event simulation,

P2(A) = E1I(A)L(t)

Choose λ1 to simulate more rare reactions.

IMA Workshop 20 of 22

Massively Parallel Simulations

� Start same computation independently on many

processors

p

...i

...21 -

-

-

-t

� If one terminates computation at t, slowly running

simulations are excluded from final estimate

Bias!

IMA Workshop 21 of 22

Conclusions

� Review of discrete–event simulation and modeling

� Simulation of Markov jump processes

� Basic variance reduction techniques

IMA Workshop 22 of 22