Trondheim, LGM2012

Vanilla Rao–Blackwellisation of

Metropolis–Hastings algorithms

Christian P. Robert

Universite Paris-Dauphine, IuF, and CRESTJoint works with Randal Douc, Pierre Jacob and Murray Smith

LGM2012, Trondheim, May 30, 2012

1 / 32

Main themes

1 Rao–Blackwellisation on MCMC

2 Can be performed in any Hastings Metropolis algorithm

3 Asymptotically more efficient than usual MCMC with acontrolled additional computing

4 Takes advantage of parallel capacities at a very basic level(GPUs)

2 / 32

Metropolis Hastings revisitedRao–Blackwellisation

Rao-Blackwellisation (2)

Metropolis Hastings algorithm

1 We wish to approximate

I =

∫h(x)π(x)dx∫π(x)dx

=

∫

h(x)π(x)dx

2 π(x) is known but not∫π(x)dx.

3 Approximate I with δ = 1n

∑nt=1 h(x

(t)) where (x(t)) is a Markov

chain with limiting distribution π.

4 Convergence obtained from Law of Large Numbers or CLT forMarkov chains.

3 / 32



Metropolis Hasting Algorithm

Suppose that x(t) is drawn.

1 Simulate yt ∼ q(·|x(t)).

2 Set x(t+1) = yt with probability

α(x(t), yt) = min

{

1,π(yt)

π(x(t))

q(x(t)|yt)q(yt|x(t))

}

Otherwise, set x(t+1) = x(t) .

3 α is such that the detailed balance equation is satisfied: ⊲ π is thestationary distribution of (x(t)).

◮ The accepted candidates are simulated with the rejection algorithm.

4 / 32



Metropolis Hasting Algorithm

Suppose that x(t) is drawn.

1 Simulate yt ∼ q(·|x(t)).

2 Set x(t+1) = yt with probability

α(x(t), yt) = min

{

1,π(yt)

π(x(t))


}

Otherwise, set x(t+1) = x(t) .

3 α is such that the detailed balance equation is satisfied:

π(x)q(y|x)α(x, y) = π(y)q(x|y)α(y, x).

⊲ π is the stationary distribution of (x(t)).

◮ The accepted candidates are simulated with the rejection algorithm.

4 / 32



Some properties of the HM algorithm

1 Alternative representation of the estimator δ is

δ =1

n

n∑

t=1

h(x(t)) =1

n

Mn∑

i=1

nih(zi) ,

where

zi’s are the accepted yj ’s,Mn is the number of accepted yj ’s till time n,ni is the number of times zi appears in the sequence (x(t))t.

5 / 32



Lemma (Douc & X., AoS, 2011)

The sequence (zi, ni) satisfies

1 (zi, ni)i is a Markov chain;

2 zi+1 and ni are independent given zi;

3 ni is distributed as a geometric random variable with probabilityparameter

p(zi) :=

∫

α(zi, y) q(y|zi) dy ; (1)

4 (zi)i is a Markov chain with transition kernel Q(z, dy) = q(y|z)dyand stationary distribution π such that

q(·|z) ∝ α(z, ·) q(·|z) and π(·) ∝ π(·)p(·) .

7 / 32



Old bottle, new wine [or vice-versa]

zi−1

8 / 32




zi−1 zi

ni−1

indep

indep

8 / 32




zi−1 zi zi+1

ni−1 ni

indep

indep

indep

indep

8 / 32




zi−1 zi zi+1

ni−1 ni

indep

indep

indep

indep

δ =1

n

n∑

t=1

h(x(t)) =1

n

Mn∑

i=1

nih(zi) .

8 / 32



Formal importance samplingVariance reductionAsymptotic resultsIllustrations

Importance sampling perspective

1 A natural idea:

δ∗ =1

n

Mn∑

i=1

h(zi)

p(zi),

9 / 32





1 A natural idea:

δ∗ ≃

∑Mn

i=1

h(zi)

p(zi)∑Mn

i=1

1

p(zi)

=

∑Mn

i=1

π(zi)

π(zi)h(zi)

∑Mn

i=1

π(zi)

π(zi)

.

9 / 32





1 A natural idea:

δ∗ ≃

∑Mn

i=1

h(zi)

p(zi)∑Mn

i=1

1

p(zi)

=

∑Mn

i=1

π(zi)

π(zi)h(zi)

∑Mn

i=1

π(zi)

π(zi)

.

2 But p not available in closed form.

9 / 32





1 A natural idea:

δ∗ ≃

∑Mn

i=1

h(zi)

p(zi)∑Mn

i=1

1

p(zi)

=

∑Mn

i=1

π(zi)

π(zi)h(zi)

∑Mn

i=1

π(zi)

π(zi)

.

2 But p not available in closed form.

3 The geometric ni is the replacement, an obvious solution that isused in the original Metropolis–Hastings estimate sinceE[ni] = 1/p(zi).

9 / 32




The Bernoulli factory

The crude estimate of 1/p(zi),

ni = 1 +∞∑

j=1

∏

ℓ≤j

I {uℓ ≥ α(zi, yℓ)} ,

can be improved:

Lemma (Douc & X., AoS, 2011)

If (yj)j is an iid sequence with distribution q(y|zi), the quantity

ξi = 1 +

∞∑

j=1

∏

ℓ≤j

{1− α(zi, yℓ)}

is an unbiased estimator of 1/p(zi) which variance, conditional on zi, is

lower than the conditional variance of ni, {1− p(zi)}/p2(zi).

10 / 32




Rao-Blackwellised, for sure?

ξi = 1 +∞∑

j=1

∏

ℓ≤j

{1− α(zi, yℓ)}

1 Infinite sum but finite with at least positive probability:

α(x(t), yt) = min

{

1,π(yt)

π(x(t))


}

For example: take a symmetric random walk as a proposal.

2 What if we wish to be sure that the sum is finite?

Finite horizon k version:

ξki = 1 +

∞∑

j=1

∏

1≤ℓ≤k∧j

{1− α(zi, yj)}∏

k+1≤ℓ≤j

I {uℓ ≥ α(zi, yℓ)}

11 / 32




Variance improvement

Proposition (Douc & X., AoS, 2011)

If (yj)j is an iid sequence with distribution q(y|zi) and (uj)j is an iiduniform sequence, for any k ≥ 0, the quantity

ξki = 1 +

∞∑

j=1

∏

1≤ℓ≤k∧j

{1− α(zi, yj)}∏

k+1≤ℓ≤j


is an unbiased estimator of 1/p(zi) with an almost sure finite number ofterms.

12 / 32







ξki = 1 +

∞∑

j=1

∏

1≤ℓ≤k∧j

{1− α(zi, yj)}∏

k+1≤ℓ≤j


is an unbiased estimator of 1/p(zi) with an almost sure finite number of

terms. Moreover, for k ≥ 1,

V

[

ξki

∣

∣

∣zi

]

=1− p(zi)

p2(zi)− 1− (1− 2p(zi) + r(zi))

k

2p(zi)− r(zi)

(

2− p(zi)

p2(zi)

)

(p(zi)− r(zi)) ,

where p(zi) :=∫

α(zi, y) q(y|zi) dy. and r(zi) :=∫

α2(zi, y) q(y|zi) dy.

12 / 32







ξki = 1 +

∞∑

j=1

∏

1≤ℓ≤k∧j

{1− α(zi, yj)}∏

k+1≤ℓ≤j


is an unbiased estimator of 1/p(zi) with an almost sure finite number of

terms. Therefore, we have

V

[

ξi

∣∣∣ zi

]

≤ V

[

ξki

∣∣∣ zi

]

≤ V

[

ξ0i

∣∣∣ zi

]

= V [ni| zi] .

12 / 32




zi−1

ξki = 1 +∞∑

j=1

∏

1≤ℓ≤k∧j

{1− α(zi, yj)}∏

k+1≤ℓ≤j


13 / 32




zi−1 zi

ξki−1

not indep

not indep

ξki = 1 +∞∑

j=1

∏

1≤ℓ≤k∧j

{1− α(zi, yj)}∏

k+1≤ℓ≤j


13 / 32




zi−1 zi zi+1

ξki−1 ξki

not indep

not indep

not indep

not indep

ξki = 1 +∞∑

j=1

∏

1≤ℓ≤k∧j

{1− α(zi, yj)}∏

k+1≤ℓ≤j


13 / 32




zi−1 zi zi+1

ξki−1 ξki

not indep

not indep

not indep

not indep

δkM =

∑Mi=1 ξ

ki h(zi)

∑Mi=1 ξ

ki

.

13 / 32




Let

δkM =

∑Mi=1 ξ

ki h(zi)

∑Mi=1 ξ

ki

.

For any positive function ϕ, we denote Cϕ = {h; |h/ϕ|∞ <∞}.

14 / 32




Let

δkM =

∑Mi=1 ξ

ki h(zi)

∑Mi=1 ξ

ki

.

For any positive function ϕ, we denote Cϕ = {h; |h/ϕ|∞ <∞}. Assumethat there exists a positive function ϕ ≥ 1 such that

∀h ∈ Cϕ,∑Mi=1

h(zi)/p(zi)∑Mi=1

1/p(zi)

P−→ π(h)

Theorem (Douc & X., AoS, 2011)

Under the assumption that π(p) > 0, the following convergence property holds:

i) If h is in Cϕ, then

δkMP−→M→∞ π(h) (◮Consistency)

14 / 32




Let

δkM =

∑Mi=1 ξ

ki h(zi)

∑Mi=1 ξ

ki

.

For any positive function ϕ, we denote Cϕ = {h; |h/ϕ|∞ <∞}.Assume that there exists a positive function ψ such that

∀h ∈ Cψ ,√M

(

∑Mi=1

h(zi)/p(zi)∑Mi=1

1/p(zi)− π(h)

)

L−→ N (0,Γ(h))


Under the assumption that π(p) > 0, the following convergence property holds:

ii) If, in addition, h2/p ∈ Cϕ and h ∈ Cψ , then

√M(δkM − π(h))

L−→M→∞ N (0, Vk[h− π(h)]) , (◮Clt)

where Vk(h) := π(p)∫

π(dz)V[

ξki

∣

∣

∣z

]

h2(z)p(z) + Γ(h) .

14 / 32




We will need some additional assumptions. Assume a maximal inequalityfor the Markov chain (zi)i: there exists a measurable function ζ such thatfor any starting point x,

∀h ∈ Cζ , Px

∣∣∣∣∣∣

sup0≤i≤n

i∑

j=0

[h(zi)− π(h)]

∣∣∣∣∣∣

> ǫ

≤ nCh(x)

ǫ2


Assume that h is such that h/p ∈ Cζ and {Ch/p, h2/p2} ⊂ Cφ. Assume

moreover that

√M

(δ0M − π(h)

) L−→ N (0, V0[h− π(h)]) .

Then, for any starting point x,

√

Mn

(∑nt=1 h(x

(t))

n− π(h)

)

n→+∞−→ N (0, V0[h− π(h)]) ,

where Mn is defined by15 / 32




We will need some additional assumptions. Assume a maximal inequalityfor the Markov chain (zi)i: there exists a measurable function ζ such thatfor any starting point x,

∀h ∈ Cζ , Px

∣∣∣∣∣∣

sup0≤i≤n

i∑

j=0

[h(zi)− π(h)]

∣∣∣∣∣∣

> ǫ

≤ nCh(x)

ǫ2

Moreover, assume that ∃φ ≥ 1 such that for any starting point x,

∀h ∈ Cφ, Qn(x, h)P−→ π(h) = π(ph)/π(p) ,



moreover that

√M

(δ0M − π(h)

) L−→ N (0, V0[h− π(h)]) .


√(∑n

h(x(t)))

n→+∞15 / 32




∀h ∈ Cζ , Px

∣∣∣∣∣∣

sup0≤i≤n

i∑

j=0

[h(zi)− π(h)]

∣∣∣∣∣∣

> ǫ

≤ nCh(x)

ǫ2

∀h ∈ Cφ, Qn(x, h)P−→ π(h) = π(ph)/π(p) ,



moreover that

√M

(δ0M − π(h)

) L−→ N (0, V0[h− π(h)]) .


√

Mn

(∑nt=1 h(x

(t))

n− π(h)

)

n→+∞−→ N (0, V0[h− π(h)]) ,

where Mn is defined by15 / 32






moreover that

√M

(δ0M − π(h)

) L−→ N (0, V0[h− π(h)]) .


√

Mn

(∑nt=1 h(x

(t))

n− π(h)

)

n→+∞−→ N (0, V0[h− π(h)]) ,

where Mn is defined by

Mn∑

i=1

ξ0i ≤ n <

Mn+1∑

i=1

ξ0i .

15 / 32




Variance gain (1)

h(x) x x2 IX>0 p(x)τ = .1 0.971 0.953 0.957 0.207τ = 2 0.965 0.942 0.875 0.861τ = 5 0.913 0.982 0.785 0.826τ = 7 0.899 0.982 0.768 0.820

Ratios of the empirical variances of δ∞ and δ estimating E[h(X)]:100 MCMC iterations over 103 replications of a random walk Gaussianproposal with scale τ .

16 / 32




Illustration (1)

Figure: Overlay of the variations of 250 iid realisations of the estimatesδ (gold) and δ∞ (grey) of E[X] = 0 for 1000 iterations, along with the90% interquantile range for the estimates δ (brown) and δ∞ (pink), inthe setting of a random walk Gaussian proposal with scale τ = 10.

17 / 32




Extra computational effort

median mean q.8 q.9 timeτ = .25 0.0 8.85 4.9 13 4.2τ = .50 0.0 6.76 4 11 2.25τ = 1.0 0.25 6.15 4 10 2.5τ = 2.0 0.20 5.90 3.5 8.5 4.5

Additional computing effort due: median and mean numbers of additionaliterations, 80% and 90% quantiles for the additional iterations, and ratioof the average R computing times obtained over 105 simulations

18 / 32




Illustration (2)

Figure: Overlay of the variations of 500 iid realisations of the estimatesδ (deep grey), δ∞ (medium grey) and of the importance sampling version(light grey) of E[X] = 10 when X ∼ Exp(.1) for 100 iterations, alongwith the 90% interquantile ranges (same colour code), in the setting ofan independent exponential proposal with scale µ = 0.02. 19 / 32



Independent caseGeneral MH algorithms

Integrating out white noise [C+X, 96]

In Casella+X. (1996), averaging over possible past and future histories(by integrating out uniforms) to improve weights of accepted values

20 / 32





In Casella+X. (1996), averaging over possible past and future histories(by integrating out uniforms) to improve weights of accepted valuesGiven the whole sequence of proposed values yt ∼ µ(yt), averaging overuniforms is possible: starting with y1, we can compute

1

T

T∑

t=1

E[h(Xt)|y1, . . . , yT ] =1

T

T∑

t=1

ϕth(yt)

through a recurrence relation:

20 / 32





In Casella+X. (1996), averaging over possible past and future histories(by integrating out uniforms) to improve weights of accepted values

ϕ(i)t = δt

p∑

j=t

ξtj

with δ0 = 1 , δt =

t−1∑

j=0

δjξj(t−1)ρjt

and ξtt = 1 , ξtj =

j∏

u=t+1

(1− ρtu)

occurence survivals of the yt’s, associated with Metropolis–Hastings ratio

ωt = π(yt)/µ(yt) , ρtu = ωu/ωt ∧ 1 .

20 / 32




Integrating out white noise (2)

Extension to generic M-H feasible (C+X, 96)Potentialy large variance improvement but cost of O(T 2)...

Possible recovery of efficiency thanks to parallelisation:Moving from (ǫ1, . . . , ǫp) towards...

(ǫ(1), . . . , ǫ(p))

by averaging over ”all” possible orders

21 / 32




Case of the independent Metropolis–Hastings algorithm

Starting at time t with p processors and a pool of p proposed values,

(y1, . . . , yp)

use processors to examine in parallel p different “histories”

22 / 32




Improvement

The standard estimator τ1 of Eπ [h(X)]

τ1(xt, y1:p) =1

p

p∑

k=1

h(xt+k)

is necessarily dominated by the average

τ2(xt, y1:p) =1

p2

p∑

k=0

nkh(yk)

where y0 = xt and n0 is the number of times xt is repeated.

23 / 32




Further Rao-Blackwellisation

E.g., use of the Metropolis–Hastings weights wj : j being the index suchthat xt+i−1 = yj , update of the weights at each time t+ i:

wj = wj + 1− ρ(xt+i−1, yi)

wi = wi + ρ(xt+i−1, yi)

resulting into a more stable estimator

τ3(xt, y1:p) =1

p2

p∑

k=0

wkh(yk)

E.g., Casella+X. (1996)

τ4(xt, y1:p) =1

p2

p∑

k=0

ϕkh(yk)

24 / 32




Markovian continuity

The Markov validity of the chain is not jeopardised! The chain continues

by picking one sequence at random and taking the corresponding x(j)t+p as

starting point of the next parallel block.

25 / 32




Impact of Rao-Blackwellisations

Comparison of

τ1 basic IMH estimator of Eπ [h(X)],

τ2 improving by averaging over permutations of proposed values andusing p times more uniforms

τ3 improving upon τ2 by basic Rao-Blackwell argument,

τ4 improving upon τ2 by integrating out ancillary uniforms, at a costof O(p2).

26 / 32




Illustration

Variations of estimates based on RB and standard versions of parallelchains and on a standard MCMC chain for the mean and variance of thetarget N (0, 1) distribution (based on 10, 000 independent replicas).

27 / 32




Impact of the order

Parallelisation allows for the partial integration of the uniforms

What about the permutation order?Comparison of

τ2N with no permutation,

τ2C with circular permutations,

τ2R with random permutations,

τ2H with half-random permutations,

τ2S with stratified permutations,

28 / 32




Impact of the order

Parallelisation allows for the partial integration of the uniforms

What about the permutation order?

28 / 32




Importance target

Comparison with the ultimate importance sampling

29 / 32




Extension to the general case

Same principle can be applied to any Markov update: if

xt+1 = Ψ(xt, ǫt)

then generate(ǫ1, . . . , ǫp)

in advance and distribute to the p processors in different permutationordersPlus use of Douc & X’s (2011) Rao–Blackwellisation ξki

30 / 32




Implementation

Similar run of p parallel chains (x(j)t+i), use of averages

τ2(x(1:p)1:p ) =

1

p2

p∑

k=1

p∑

j=1

nkh(x(j)t+k)

and selection of new starting value at random at time t+ p:

31 / 32




Illustration

Variations of estimates based on RB and standard versions of parallelchains and on a standard MCMC chain for the mean and variance of thetarget distribution (based on p = 64 parallel processors, 50 blocs of pMCMC steps and 500 independent replicas).

RB par org

−0.

10−

0.05

0.00

0.05

0.10

RB par org

0.9

1.0

1.1

1.2

1.3

32 / 32

Trondheim, LGM2012

Education

Transcript of Trondheim, LGM2012