Expectation-Maximization Algorithm

EM Algorithm

Karl Gregory and Poulami Barman

Department of StatisticsTexas A&M University

STAT 613 Presentation, 2011 Spring

Expectation-Maximization Algorithm

Outline

Review of EM AlgorithmDefinitionBasic Model SetupComplete Log-Likelihood

The AlgorithmExpectation StepMaximization Step

Graphical Output

Expectation-Maximization Algorithm

Review of EM Algorithm

Definition

Definition

EM algorithm is an iterative process of finding the maximum likelihood estimators bytreating the observed data as an incomplete version of an ideal dataset whose analysiswould have been easy, i.e. the model depends on unobserved missing data.

Our objective is to estimate the unknown parameter θ.

Expectation-Maximization Algorithm

Review of EM Algorithm

Basic Model Setup

The Form of The Observed Likelihood

I Let us suppose we observe Y = {Yi}ni=1.

I The joint density of Y is f (Y ; θ).

I The log likelihood of Y is

L(Y ; θ) = log f (Y ; θ)

I It is often difficult to maximize the above likelihood.

Expectation-Maximization Algorithm

Review of EM Algorithm

Basic Model Setup

The Form of Complete Likelihood

Often there may exist latent variables or unobserved data{U = {Ui}mi=1

}.

It is easier to evaluate the joint likelihood of the observed and unobserved data.

L(Y ,U; θ)

The above likelihood is called the Complete Likelihood.

Expectation-Maximization Algorithm

Review of EM Algorithm

Basic Model Setup

Application in Mixture Distributions

Let Y come from a mixture of p distributions. Then the density of Y is expressed

f(y;θ) =∑p

r=1 πr fr (y ; θ)∑pr=1 πr = 1

0 ≤ πr ≤ 1

where θ = (π1, . . . πp , θ1, . . . , θp) are unknown parameters.

Expectation-Maximization Algorithm

Review of EM Algorithm

Complete Log-Likelihood

Complete Log-likelihood

If the value u of U were known the complete likelihood (y;u) would be

log f (y , u; θ) =

p∑r=1

I (u = r){log πr + log fr (y ; θ)}

Expectation-Maximization Algorithm

The Algorithm

Expectation Step

Expectation Step

In order to apply EM, we compute the expectation of log f (y , u; θ) over theconditional(observed) distribution

P(U = r |Y = y ; θ′) =π′r fr (y ; θ′)∑ps=1 π

′s fs(y ; θ′)

, r = 1, . . . , p

Let the above quantity be represented by wr (yj ; θ′). Then the expected value of the

log-likelihood becomes

Q(θ; θ′) =n∑

j=1

p∑r=1

wr (yj ; θ′){log πr + log fr (yj ; θ)})

Expectation-Maximization Algorithm

The Algorithm

Maximization Step

Maximization Step

When we maximize Q(θ; θ′), we obtain the following expressions for µ†, σ2†, π†, interms of previous values of µ, σ, π, and the data:

π†r = n−1∑n

j=1 wr (yj ; θ′)

µ†r =∑n

j=1 wr (yj ;θ′)yj∑n

j=1 wr (yj ;θ′)

σ2r† =

∑nj=1 wr (yj ;θ

′)(yj−µ†r )2∑n

j=1 wr (yj ;θ′)

Expectation-Maximization Algorithm

Graphical Output

Output

Expectation-Maximization Algorithm

Graphical Output

Output

Expectation-Maximization Algorithm

Graphical Output

Output

Expectation-Maximization Algorithm

Graphical Output

Output

Expectation-Maximization Algorithm

Graphical Output

Output

Expectation-Maximization Algorithm

Graphical Output

Output

Expectation-Maximization Algorithm

Graphical Output

Output