Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm

Phisical Fluctuomatics (Tohoku University) 1

Physical Fluctuomatics4th Maximum likelihood estimation and EM algorithm

Kazuyuki TanakaGraduate School of Information Sciences, Tohoku University

[email protected]://www.smapip.is.tohoku.ac.jp/~kazu/


Textbooks

Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese) , Chapter 4.


Statistical Machine Learning and Model Selection

Inference of Probabilistic Model by using Data

Model Selection

Statistical Machine Learning

Maximum Likelihood

EM algorithmImplement by Belief Propagation and Markov Chain Monte Carlo method

Akaike Information Criteria, Akaike Bayes Information Criteria, etc.

Extension

Complete Data and Incomplete data


Maximum Likelihood Estimation

,

Parameter



1

0

222 2

1exp

2

1,

N

iiggP

1

1

0

Ng

g

g

g

,

Parameter

1,,1,0 NV

0 1 2

3 4 5

6 7 8



1

0

222 2

1exp

2

1,

N

iiggP

Data

1

1

0

Ng

g

g

g

,

Parameter

1,,1,0 NV

0 1 2

3 4 5

6 7 8

Data



1

0

222 2

1exp

2

1,

N

iiggP

Data

1

1

0

Ng

g

g

g

,

Parameter

1,,1,0 NV

0 1 2

3 4 5

6 7 8

Data Histogram



1

0

222 2

1exp

2

1,

N

iiggP

Data

1

1

0

Ng

g

g

g

,

Parameter

1,,1,0 NV

0 1 2

3 4 5

6 7 8

Data Histogram

,maxargˆ,ˆ,

gP



1

0

222 2

1exp

2

1,

N

iiggP

,maxargˆ,ˆ,

gP

Data

1

1

0

Ng

g

g

g

,

Parameter

Probability density function for data with average μ and variance σ2 is regarded as likelihood function for average μ and variance σ2 when data is given.g



1

0

222 2

1exp

2

1,

N

iiggP

,maxargˆ,ˆ,

gP

Data

0

,

0,

ˆ,ˆ

ˆ,ˆ

gP

gPExtremum Condition

1

1

0

Ng

g

g

g

,

Parameter




1

0

222 2

1exp

2

1,

N

iiggP

,maxargˆ,ˆ,

gP

Data

0

,

0,

ˆ,ˆ

ˆ,ˆ

gP

gP

1

0

1ˆ

N

iig

N

1

0

22 ˆ1

ˆN

iig

N

Extremum Condition

Sample AverageSample Deviation

1

1

0

Ng

g

g

g

,

Parameter




1

0

222 2

1exp

2

1,

N

iiggP

,maxargˆ,ˆ,

gP

Data

0

,

0,

ˆ,ˆ

ˆ,ˆ

gP

gP

1

0

1ˆ

N

iig

N

1

0

22 ˆ1

ˆN

iig

N

Extremum Condition

Sample AverageSample Deviation

1

1

0

Ng

g

g

g

,

Parameter


Histogram


1

1

0

Nf

f

f

f

Maximum Likelihood

1

0

2

22 2

1exp

2

1,

N

iii fgfgP

gP

maxargˆ Data

0

,1

ˆ

gP

1

1

22 11ˆ

N

iig

N

Extremum Condition

1

0

2

2

1exp

2

1N

iiffP

ff

fPfgPgfPgP

,,

1

1

0

Ng

g

g

g

Hyperparameter

Parameter

fdgfPff

,ˆ

gP

fPfgPgfP

,,

Bayes Formula

f

is unknown

Marginal Likelihood


Probabilistic Model for Signal Processing

Source Signal Observable Data

Transmission

Noise

Likelihood Marginal

yProbabilitPrior LikelihoodyProbabilitPosterior

Signal Observable

Signal SourceData ObservableData ObservableSignal Source

Pr

PrSignal Source|PrPr

NoiseGaussian WhiteSignal SourceData Observable

i

fi

i

gi

Bayes Formula


Prior Probability for Source Signal

Image DataOne dimensional Data

1 2 3 4 5

1 2 2 3X

3 4 4 5XX

=

Ejiji

EjiEjiji

ffZ

ffZ

fP

},{

2

Prior

},{},{

2

Prior

2

1exp

1

2

1exp

1

E： Set of all the links

i j

1 2 3 4

6 7 8 9

21 22 23 24

5

10

25

11 12 13 14

16 17 18 19

15

20


Data Generating Process

Additive White Gaussian Noise

2,0~ Nfg ii

Viii gffgP 2

22 2

1exp

2

1,

V ： Set of all the nodes


Probabilistic Model for Signal Processing

Viii fgfgP 2

22 2

1exp

2

1,

Ejiji ff

ZfP

},{

2

prior 2

1exp

1

1

1

0

Nf

f

f

f

データ

1

1

0

Ng

g

g

g

Hyperparameter

i

fi

i

gi

Parameter

fdfPfgP

fPfgPgfP

,

,,,

fdgfPff ii

,,ˆ

Posterior Probability


1

1

0

Nf

f

f

f

Maximum Likelihood in Signal Processing

,maxargˆ,ˆ,

gP

Data

0

,,0

,

ˆ,ˆˆ,ˆ

gPgP

Extremum Condition

fdfPfgPgP ,,

1

1

0

Ng

g

g

g

Hyperparameter

Parameter

Incomplete Data

Marginal Likelihood


1

1

0

Nf

f

f

f

Maximum Likelihood and EM algorithm

Data

0

,,0

,

ˆ,ˆˆ,ˆ

gPgP

Extemum Condition

fdfPfgPgP ,,

1

1

0

Ng

g

g

g

Hyperparameter

Parameter

fdgfPgfP

Q

,,ln,,

,,

)(),(,maxarg

)1()1(

Update:Step M

)(),(, Calculate :Step E

),(ttQ

t,σtα

ttQ

0

,,

0,,

,

,

Q

Q

Expectation Maximization (EM) algorithm provide us one of Extremum Points of Marginal Likelihood.

Q function

Marginal Likelihood

Incomplete Data


Model Selection in One Dimensional Signal

Expectation Maximization (EM) Algorithm

i

i

i

0 127 255

0 127 255

0 127 255

100

0

200

100

0

200

100

0

200

if

ig

if

Original Signal

Degraded Signal

Estimated Signal

40

0.04

0.03

0.02

0.01

α(t)

0

α(0)=0.0001, σ(0)=100


Model Selection in Noise Reduction

Original Image

Degraded Image EM algorithm and

Belief Propagation

α(0)=0.0001σ(0)=100

Estimate of Original Image

MSE

327 0.000611 36.30

MSE

260 0.000574 34.00

2ˆ

||

1MSE

i

ii ff

40


Summary

Maximum Likelihood and EM algorithm

Statistical Inference by Gaussian Graphical Model


Let us suppose that data {gi |i=0,1,...,N-1} are

generated by according to the following probability density function:

, 　

Prove that estimates for average m and variance s2 of the maximum likelihood

are given as

,

Practice 4-1

1

0

222 2

1exp

2

1,

N

iiggP

1

1

0

Ng

g

g

g

, 　

,maxargˆ,ˆ,

gP

1

0

1ˆ

N

iig

N

1

0

22 ˆ1

ˆN

iig

N

,

, where

Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm

Documents

Transcript of Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm