A Simple Gibbs Sampler - University of Michigan School of
Transcript of A Simple Gibbs Sampler - University of Michigan School of
![Page 1: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/1.jpg)
A Simple Gibbs Sampler
Biostatistics 615/815Lecture 20
![Page 2: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/2.jpg)
Scheduling
No Class• November 30
Review session• Thursday , December 7
Final Assessment• Tuesday, December 12
![Page 3: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/3.jpg)
Optimization Strategies
Single Variable• Golden Search• Quadratic Approximations
Multiple Variables• Simplex Method• E-M Algorithm• Simulated Annealing
![Page 4: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/4.jpg)
Simulated Annealing
Stochastic Method
Sometimes takes up-hill steps• Avoids local minima
Solution is gradually frozen• Values of parameters with largest impact on
function values are fixed earlier
![Page 5: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/5.jpg)
Gibbs Sampler
Another MCMC Method
Update a single parameter at a time
Sample from conditional distribution when other parameters are fixed
![Page 6: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/6.jpg)
Gibbs Sampler Algorithm
steps. previousrepeat and Increment
),...,,...,,|( from for valueSample b. say update, tocomponent Selecting a.
:by valuesparameter ofset next theDefine
valuesparameter of choice particular aConsider
1121)1(
)(
t
θθθθθxθpθi
kiiit
i
t
+−+
θ
![Page 7: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/7.jpg)
Alternative Algorithm
steps. previousrepeat and Increment
),...,,|( from for valueSample z. ...
),...,,|( from for valueSample c.
),...,,|( from for valueSample b. in turn ,1 component,each Updatea.
:by valuesparameter ofset next theDefine
valuesparameter of choice particular aConsider
131)1(
312)1(
2
321)1(
1
)(
t
θθθxθpθ
θθθxθpθ
θθθxθpθ .. k
kkt
k
kt
kt
t
−+
+
+
θ
![Page 8: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/8.jpg)
Key Property:Stationary Distribution
ondistributiposterior their fromvaluesparameter sample sampler to Gibbs expect the we,Eventually
)|(~)|(~
fact...In
)|,...,,()|,...,(),,...,|(
as ddistribute is ),...,,(Then
)|,...,,(~),...,,( that Suppose
)1()(
21221
)()(2
)1(1
21)()(
2)(
1
xpxp
xpxpxp
xp
tt
kkk
tk
tt
kt
ktt
θθθθ +
+
⇒
= θθθθθθθθ
θθθ
θθθθθθ
![Page 9: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/9.jpg)
Gibbs Sampling for Mixture Distributions
Sample each of the mixture parameters from conditional distribution• Dirichlet, Normal and Gamma distributions are
typical
Simple alternative is to sample the sourceof each observation• Assign observation to specific component
![Page 10: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/10.jpg)
Sampling A Component
Calculate the probability that the observation originated from a specific component…
… can you recall how random numbers can be used to sample from one of a few discrete categories?
∑==
lljl
ijijj xf
xf,xiZ
),|(),|(
),,|Pr(ηφπ
ηφπηφπ
![Page 11: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/11.jpg)
C Code: Sampling A Componentint sample_group(double x, int k,
double * probs, double * mean, double * sigma){int group; double p = Random();double lk = dmix(x, k, probs, mean, sigma);
for (group = 0; group < k - 1; group++){double pgroup = probs[group] *
dnorm(x, mean[group], sigma[group])/lk;
if (p < pgroup) return group;
p -= pgroup;}
return k - 1;}
![Page 12: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/12.jpg)
Calculating Mixture Parameters
iiZj
iiji
iZjiji
ii
iZji
nxnxs
nxxnnp
n
j
j
j
⎟⎟⎠
⎞⎜⎜⎝
⎛−=
=
=
=
∑
∑
∑
=
=
=
:
22
:
:1 Before sampling a
new origin for an observation…
… update mixture parameters given current assignments
Could be expensive!
![Page 13: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/13.jpg)
C Code:Updating Parametersvoid update_estimates(int k, int n,
double * prob, double * mean, double * sigma,double * counts, double * sum, double * sumsq)
{int i;
for (i = 0; i < k; i++){prob[i] = counts[i]/ n;mean[i] = sum[i] / counts[i];sigma[i] = sqrt((sumsq[i] - mean[i]*mean[i]*counts[i])
/ counts[i] + 1e-7);}
}
![Page 14: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/14.jpg)
C Code:Updating Mixture Parameters IIvoid remove_observation(double x, int group,
double * counts, double * sum, double * sumsq){counts[group] --;sum[group] -= x;sumsq[group] -= x * x;}
void add_observation(double x, int group,double * counts, double * sum, double * sumsq)
{counts[group] ++;sum[group] += x;sumsq[group] += x * x;}
![Page 15: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/15.jpg)
Selecting a Starting State
Must start with an assignment of observations to groupings
Many alternatives are possible, I chose to perform random assignments with equal probabilities…
![Page 16: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/16.jpg)
C Code:Starting Statevoid initial_state(int k, int * group,
double * counts, double * sum, double * sumsq){int i;
for (i = 0; i < k; i++)counts[i] = sum[i] = sumsq[i] = 0.0;
for (i = 0; i < n; i++){group[i] = Random() * k;
counts[group[i]] ++;sum[group[i]] += data[i];sumsq[group[i]] += data[i] * data[i];}
}
![Page 17: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/17.jpg)
The Gibbs Sampler
Select initial state
Repeat a large number of times:• Select an element• Update conditional on other elements
If appropriate, output summary for each run…
![Page 18: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/18.jpg)
C Code:Core of The Gibbs Samplerinitial_state(k, probs, mean, sigma, group, counts, sum, sumsq);for (i = 0; i < 10000000; i++)
{int id = Random() * n;
if (counts[group[id]] < MIN_GROUP) continue;
remove_observation(data[id], group[id], counts, sum, sumsq);update_estimates(k, n - 1, probs, mean, sigma,
counts, sum, sumsq);group[id] = sample_group(data[id], k, probs, mean, sigma);add_observation(data[id], group[id], counts, sum, sumsq);
if ((i > BURN_IN) && (i % THIN_INTERVAL == 0))/* Collect statistics */
}
![Page 19: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/19.jpg)
Gibbs Sampler:Memory Allocation and Freeingvoid gibbs(int k, double * probs, double * mean, double * sigma)
{int i, * group = (int *) malloc(sizeof(int) * n);double * sum = alloc_vector(k);double * sumsq = alloc_vector(k);double * counts = alloc_vector(k);
/* Core of the Gibbs Sampler goes here */
free_vector(sum, k);free_vector(sumsq, k);free_vector(counts, k);free(group);}
![Page 20: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/20.jpg)
Example ApplicationOld Faithful Eruptions (n = 272)
Old Faithful Eruptions
Duration (mins)
Freq
uenc
y
1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0
05
1015
20
![Page 21: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/21.jpg)
Notes …
My first few runs found excellent solutions by fitting components accounting for very few observations but with variance near 0
Why? • Repeated values due to rounding• To avoid this, I set MIN_GROUP to 13 (5%)
![Page 22: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/22.jpg)
Gibbs Sampler Burn-InLogLikelihood
-450
-400
-350
-300
-250
0 2500 5000 7500 10000
Iteration
Log-
Like
lihoo
d
![Page 23: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/23.jpg)
Gibbs Sampler Burn-InMixture Means
1
2
3
4
5
0 2500 5000 7500 10000
Iteration
Mea
n
![Page 24: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/24.jpg)
Gibbs Sampler After Burn-InLikelihood
0 20 40 60 80 100
-300
-290
-280
-270
-260
-250
Iteration (Millions)
Like
lihoo
d
![Page 25: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/25.jpg)
Gibbs Sampler After Burn-InMean for First Component
0 20 40 60 80 100
1.80
1.85
1.90
1.95
Iteration (Millions)
Com
pone
nt M
ean
![Page 26: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/26.jpg)
Gibbs Sampler After Burn-In
0 20 40 60 80 100
23
45
6
Iteration (Millions)
Com
pone
nt M
ean
![Page 27: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/27.jpg)
Notes on Gibbs Sampler
Previous optimizers settled on a minimum eventually
The Gibbs sampler continues wandering through the stationary distribution…
Forever!
![Page 28: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/28.jpg)
Drawing Inferences…
To draw inferences, summarize parameter values from stationary distribution
For example, might calculate the mean, median, etc.
![Page 29: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/29.jpg)
Component Means
Mean
Den
sity
1 2 3 4 5 6
02
46
810
Mean
Den
sity
1 2 3 4 5 6
02
46
810
Mean
Den
sity
1 2 3 4 5 60
24
68
![Page 30: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/30.jpg)
Component Probabilities
Probability
Den
sity
0.0 0.2 0.4 0.6 0.8 1.0
02
46
8
Probability
Den
sity
0.0 0.2 0.4 0.6 0.8 1.0
02
46
810
Probability
Den
sity
0.0 0.2 0.4 0.6 0.8 1.0
01
23
45
6
![Page 31: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/31.jpg)
Overall Parameter Estimates
The means of the posterior distributions for the three components were:• Frequencies of 0.073, 0.278 and 0.648• Means of 1.85, 2.08 and 4.28 • Variances of 0.001, 0.065 and 0.182
Our previous estimates were:• Components contributing .160, 0.195 and 0.644• Component means are 1.856, 2.182 and 4.289• Variances are 0.00766, 0.0709 and 0.172
![Page 32: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/32.jpg)
Joint Distributions
Gibbs Sampler provides other interesting information and insights
For example, we can evaluate joint distribution of two parameters…
![Page 33: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/33.jpg)
Component Probabilites
0.05 0.10 0.15 0.20 0.25 0.30
0.05
0.10
0.15
0.20
0.25
0.30
Group 1 Probability
Gro
up 3
Pro
babi
lity
![Page 34: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/34.jpg)
So far today …
Introduction to Gibbs sampling
Generating posterior distributions of parameters conditional on data
Providing insight into joint distributions
![Page 35: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/35.jpg)
A little bit of theory
Highlight connection between Simulated Annealing and the Gibbs sampler …
Fill in some details of the Metropolis algorithm
![Page 36: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/36.jpg)
Both Methods Are Markov Chains
The probability of any state being chosen depends only on the previous state
States are updated according to transition matrix with elements pij. This matrix defines important properties, including periodicity and irreducibility.
)|Pr(),...,|Pr( 110011 −−−− ====== nnnnnnnn iSiSiSiSiS
![Page 37: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/37.jpg)
Metropolis-HastingsAcceptance Probability
πππ
qqππ
aqπqπ
a
ππ
iSjSqq
i
j
jiiji
jij
iji
jijij
ji
nnij
of valuesactual not the known, bemust
ratio Only the
if ,1minor ,1min
:isy probabilit acceptance Hastings-Metropolis The
stateeach of iesprobabilit relative thebe and Let
)| propose(Let 1
=⎟⎟⎠
⎞⎜⎜⎝
⎛=⎟
⎟⎠
⎞⎜⎜⎝
⎛=
=== +
![Page 38: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/38.jpg)
Metropolis-Hastings Equilibrium
e.communicat tostatesall allowmust density proposal thehappen, toFor this
Pr whereondistributi mequilibriuan reach it will Chain, Markov a
update toalgorithm Hastings-Metropolis theuse weIf
ii)(S π==
![Page 39: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/39.jpg)
Gibbs Sampler
1,1min e,consequenc a As
that ensuressampler Gibbs The
=⎟⎟⎠
⎞⎜⎜⎝
⎛=
=
iji
jijij
jijiji
a
ππ
ππ
![Page 40: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/40.jpg)
Simulated Annealing
statesy probabilithigh given to are ghtslarger wei res, temperatulowAt flattened ison distributiy probabilit theratures,high tempeAt
with replace
,parameter re temperatuaGiven
1
1
)(
∑=
jj
iiiπ
τ
ττ
π
ππ
τ
![Page 41: A Simple Gibbs Sampler - University of Michigan School of](https://reader031.fdocuments.net/reader031/viewer/2022020705/61fb7a5f2e268c58cd5ea42f/html5/thumbnails/41.jpg)
Additional Reading
If you need a refresher on Gibbs sampling• Bayesian Methods for Mixture Distributions
M. Stephens (1997)http://www.stat.washington.edu/stephens/
Numerical Analysis for Statisticians• Kenneth Lange (1999)• Chapter 24