Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3...
Transcript of Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3...
![Page 1: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/1.jpg)
Presented at SSIP 2011, Szeged, Hungary
Markov Random Fields in
Image Segmentation
Zoltan Kato
Image Processing & Computer Graphics Dept.University of SzegedHungary
![Page 2: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/2.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 2
Overview
Segmentation as pixel labelingP b bili ti hProbabilistic approach
Segmentation as MAP estimationM k R d Fi ld (MRF)Markov Random Field (MRF)Gibbs distribution & Energy function
Cl i l i i i tiClassical energy minimizationSimulated AnnealingMarkov Chain Monte Carlo (MCMC) samplingMarkov Chain Monte Carlo (MCMC) sampling
Example MRF model & DemoP t ti ti (EM)Parameter estimation (EM)
![Page 3: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/3.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3
Segmentation as a Pixel Labelling Task
1. Extract features from the input imageEach pixel s in the image has a feature vector p gFor the whole image, we have
sfr
}:{ Ssff ∈=r
2. Define the set of labels ΛE h i l i i d l b l
}:{ Ssff s ∈=
ΛEach pixel s is assigned a label For the whole image, we have
Λ∈sω
}{ Ss ∈ωωFor an N×M image, there are |Λ|NM possible labelings.
Whi h i th i ht t ti ?Whi h i th i ht t ti ?
},{ Sss ∈= ωω
Which one is the right segmentation?Which one is the right segmentation?
![Page 4: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/4.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 4
Probabilistic Approach, MAP
Define a probability measure on the set of all possible labelings and select the most likely onepossible labelings and select the most likely one.
measures the probability of a labelling, given the observed feature
)|( fP ωfgiven the observed feature
Our goal is to find an optimal labeling which maximizes
ω̂f
)|( fPmaximizesThis is called the Maximum a Posteriori (MAP) estimate:
)|( fP ω
estimate:
)|(maxargˆ fPMAP ωω = )|(maxarg fP ωωω Ω∈
=
![Page 5: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/5.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 5
Bayesian Frameworklikelihood
By Bayes Theorem, we haveprior
)()|()(
)()|()|( ωωωωω PfPfPPfPfP ∝=
is constant
)()|()(
)|( ffP
f
)( fP is constant We need to define and in our
d l
)( fP)(ωP )|( ωfP
modelWe will use Markov Random Fields
![Page 6: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/6.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 6
Why MRF Modelization?
In real images, regions are often homogenous; neighboring pi els s all ha e similar propertiesneighboring pixels usually have similar properties (intensity, color, texture, …)M k R d Fi ld (MRF) i b bili ti d lMarkov Random Field (MRF) is a probabilistic model which captures such contextual constraintsW ll t di d t th ti l b k dWell studied, strong theoretical backgroundAllows MCMC sampling of the (hidden) underlying t t Si l t d A listructure Simulated Annealing
Fast and exact solution for certain type of models G h t [K l ]Graph cut [Kolmogorov]
![Page 7: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/7.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 7
What is MRF?
To give a formal definition for Markov Random Fi ld d b i b ildi bl kFields, we need some basic building blocks
Observation Field and (hidden) Labeling Field Pixels and their NeighborsCliques and Clique PotentialsEnergy functionGibbs Distribution
![Page 8: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/8.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 8
Definition – Neighbors
For each pixel, we can define some surrounding i l it i hbpixels as its neighbors.
Example : 1st order neighbors and 2nd order neighbors
![Page 9: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/9.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 9
Definition – MRF
The labeling field X can be modeled as a M k R d Fi ld (MRF) ifMarkov Random Field (MRF) if
1. For all 0)(: >=ΧΩ∈ ωω P2. For every and :Ss ∈ Ω∈ω
),|(),|( srsrs NrPsrP ∈=≠ ωωωω
denotes the neighbors of pixel s),|(),|( srsrs
sN
![Page 10: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/10.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 10
Hammersley-Clifford Theorem
The Hammersley-Clifford Theorem states that a random field is a MRF if and only if follows a Gibbs )(ωPydistribution.
)(
11 ∑ ))(exp(1))(exp(1)( ∑∈
−=−=Cc
cVZ
UZ
P ωωω
where is a normalization constant∑Ω∈
−=ω
ω))(exp( UZ
This theorem provides us an easy way of defining MRF models via clique potentialsclique potentials.
![Page 11: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/11.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 11
Definition – Clique
A subset is called a clique if every pair of i l i thi b t i hb
SC ⊆pixels in this subset are neighbors.A clique containing n pixels is called nth orderclique, denoted by .The set of cliques in an image is denoted by
nCq g y
kCCCC UUU ...21=
singleton doubleton
![Page 12: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/12.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 12
Definition – Clique Potential
For each clique c in the image, we can assign a l hi h i ll d li t ti l f)(Vvalue which is called clique potential of c,
where is the configuration of the labeling field)(ωcV
ωThe sum of potentials of all cliques gives us the energy of the configuration)(ωU ωgy g)(
...),(V)(V)(V)(U21 jiCiCc ∑∑∑ +ωω+ω=ω=ω
2
2
1
1C)j,i(
jCiCc
∑∑∑∈∈∈
![Page 13: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/13.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 13
Segmentation of grayscale images:Segmentation of grayscale images:A simple MRF modelpConstruct a segmentation model where regions are formed by spatial clusters of pixels with similar y p pintensity:
MRF segmentation model+
find MAP estimate ω̂
Model parameters
find MAP estimate ω
segmentationω̂
Input imageω
![Page 14: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/14.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 14
MRF segmentation modelPixel labels (or classes) are represented by Gaussian distributions:
⎟⎟⎠
⎞⎜⎜⎝
⎛ −−= 2
2
2)(
exp2
1)|(s
s
s
sss
ffP
ω
ω
ω σμ
σπω
Clique potentials:Singleton: proportional to the likelihood of f t i l (P(f | ))features given ω: log(P(f | ω)).Doubleton: favours similar labels at neighbouring pixels – smoothness priorp p
⎩⎨⎧
≠+=−
==ji
jijic if
ifjiV
ωωβωωβ
ωωβδ ),(),(2
As β increases, regions become more homogenous
⎩ ≠+ jiif ωωβ
![Page 15: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/15.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 15
Model parameters
Doubleton potential βless dependent on the input
can be fixed a priori
Number of labels (|Λ|)Problem dependent
usually given by the user or inferred from some higher level knowledge
E h l b l λ Λ i t d b G iEach label λ∈Λ is represented by a Gaussian distribution N(µλ,σλ):
estimated from the input image
![Page 16: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/16.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 16
Model parameters
The class statistics (mean and variance) can be estimated via the empirical meancan be estimated via the empirical mean and variance:
where Sλ denotes the set of pixels in thewhere Sλ denotes the set of pixels in the training set of class λa training set consists in a representative region selected by the userregion selected by the user
![Page 17: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/17.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 17
Energy function
Now we can define the energy function of our MRF model:
∑ ∑⎟⎞
⎜⎛ −f 2)( μ
∑ ∑+⎟⎟⎠
⎞⎜⎜⎝
⎛+=
s rsrs
s
s
s
s
fU
,2 ),(
2)(
)2log()( ωωβδσ
μσπω
ω
ωω
Recall: ))(exp(1))(exp(1)|( ∑−=−=
CcV
ZU
ZfP ωωω
Hence∈CcZZ
)(minarg)|(maxargˆ ωωω UfPMAP == )(minarg)|(maxarg ωωωωω
UfPΩ∈Ω∈
![Page 18: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/18.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 18
Optimization
Problem reduced to the i i i ti fminimization of a non-convex
energy functionMany local minima
Gradient descent?Works only if we have a goodinitial segmentation
Simulated AnnealingAlways works (at least in theory)y ( y)
![Page 19: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/19.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 19
ICM (~Gradient descent) [Besag86]
![Page 20: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/20.jpg)
ZoltanZoltan Kato: Kato: Markov Random Fields in Markov Random Fields in Image SegmentationImage Segmentation 20
ICM (iterated conditional mode)
x2means observed
x xx x1 x4x3
x5
ICM Global min
Simulated Annealing: accept a move even if energy increases (with certain probability)
Can get stuck in local minima!Slide adopted from C. Rother ICCV’09 tutorial: Slide adopted from C. Rother ICCV’09 tutorial: http://research.microsoft.com/enhttp://research.microsoft.com/en--us/um/cambridge/projects/tutorial/us/um/cambridge/projects/tutorial/
![Page 21: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/21.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 21
Simulated Annealing (Metropolis)
![Page 22: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/22.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 22
Temperature Schedule
![Page 23: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/23.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 23
Temperature Schedule
Initial temperature: set it to a relatively low value (~4)faster execution
must be high enough to allow random jumps at the beginning!
Schedule:Stopping criteria:
Fixed number of iterationsEnergy change is less than a threshols
![Page 24: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/24.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 24
DemoDownload from:Download from:http://www.inf.u-szeged.hu/~kato/software/
![Page 25: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/25.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 25
Summary
Design your model carefullyOptimization is just a tool, do not expect a good segmentation from a wrong modelg g g
What about other than graylevel features?Extension to color is relatively straightforwardExtension to color is relatively straightforward
![Page 26: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/26.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 26
What color features?RGB histogram
![Page 27: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/27.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 27
Extract Color Feature
We adopt the CIE-L*u*v* color space because it is perceptually uniformis perceptually uniform.
Color difference can be measured by Euclidean distance of two color vectors.
We convert each pixel from RGB space to CIE-L*u*v* space
We have 3 color feature imagesWe have 3 color feature images
L* u* v*
![Page 28: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/28.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 28
Color MRF segmentation modelPixel labels (or classes) are represented by three-variate Gaussian distributions:
))()(21exp(
||)2(1)|( 1 T
ssnss sss
s
ufuffP ωωω
ωπω rrrr
−Σ−−Σ
= −
Clique potentials:Singleton: proportional to the likelihood of f t i l (P(f | ))features given ω: log(P(f | ω)).Doubleton: favours similar labels at neighbouring pixels – smoothness priorp p
⎩⎨⎧
≠+=−
==ji
jijic if
ifjiV
ωωβωωβ
ωωβδ ),(),(2
As β increases, regions become more homogenous
⎩ ≠+ jiif ωωβ
![Page 29: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/29.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 29
Summary
Design your model carefullyOptimization is just a tool, do not expect a good segmentation from a wrong model
What about other than graylevel features?Extension to color is relatively straightforwardy g
Can we segment images without user interaction?interaction?
Yes, but you need to estimate model parameters automatically (EM algorithm)parameters automatically (EM algorithm)
![Page 30: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/30.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 30
Incomplete data problem
Supervised parameter estimationwe are given a labelled data set to learn from
e.g. somebody manually assigned labels to pixels
How to proceed without labelled data? Learning from incomplete datag pStandard solution is an iterative procedurecalled Expectation-Maximizationp
Assigns labels and estimates parameterssimultaneouslyChicken-Egg problem
![Page 31: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/31.jpg)
31
EM principles : The two steps
E Step : For each pixel, use parameters to compute probability distributionuse parameters to compute probability distribution
Parameters :P(pixel/label)P(label)
Weighted labeling :P(label/pixel)
M Step : Update the estimates of parametersbased on weighted (or ”soft”) labeling
![Page 32: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/32.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 32
The basic idea of EM
Each of the E and M steps is t i htf d i th th istraightforward assuming the other is
solvedKnowing the label of each pixel, we can estimate the parameters
Similar to supervised learning (hard vs. softlabeling)
Knowing the parameters of the distributionsKnowing the parameters of the distributions, we can assign a label to each pixel
by Maximum Likelihood i e using the singletonby Maximum Likelihood – i.e. using the singletonenergies only without pairwise interactions
![Page 33: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/33.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 33
Parameter estimation via EM
Basically, we will fit a mixture of Gaussian to the image histogram
We know the number of labels |Λ| ≡ number ofWe know the number of labels |Λ| number of mixture components
At each pixel the complete data includesAt each pixel, the complete data includesThe observed feature fs
Hidden pixel labels ls (a vector of size |Λ|) specifies the contribution of the pixel feature to each of the labels – i.e. a soft labeling
![Page 34: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/34.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 34
Parameter estimation via EM
E step: recompute lsi at each pixel s:
∑==
λλλλλ
)()|()()|()|(
PPPPP s
sis f
ffl∑
Λ∈λ
λλ )()|()|(
PP sss f
M step: update Gaussian parameters for each label λ: )|()|( ∑∑ λλeach label λ:
,...)|(
)|(,
||
)|()(
∑∑∑∈∈ == SsSs
P
P
S
PP
sss
f
fff
λ
λμ
λλ λ )|(|| ∑
∈SsPS sfλ
![Page 35: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/35.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 35
SummaryDesign your model carefully
Optimization is just a tool, do not expect a goodOptimization is just a tool, do not expect a good segmentation from a wrong model
What about other than graylevel featuresExtension to color is relatively
Can we segment images without user interaction?Y b t d t ti t d l tYes, but you need to estimate model parameters automatically (EM algorithm)
Can we segment more complex images?Can we segment more complex images?Yes, but then you need a more complex MRF model
![Page 36: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/36.jpg)
ZoltanZoltan Kato: Markov Random Fields in Image SegmentationKato: Markov Random Fields in Image Segmentation 36
Color Textured Segmentation
segmentation
segmentation
![Page 37: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/37.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 37
Color & Motion Segmentation
![Page 38: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/38.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 38
SummaryDesign your model carefully
Optimization is just a tool, do not expect a good segmentation from a rong modelsegmentation from a wrong model
What about other than graylevel featuresExtension to color is relativelyy
Can we segment images without user interaction?Yes, but you need to estimate model parameters automatically (EM algorithm)y ( g )
Can we segment more complex images?Yes, but then you need a more complex MRF model
What if we do not know |Λ|? Fully automatic segmentation requires
Modeling of the parameters ANDModeling of the parameters ANDa more sophisticated sampling algorithm (Reversible jump MCMC)
![Page 39: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/39.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 39
MRF+RJMCMC vs. JSEGR
JMMC
MC
(1730 X
500
7 min)
73JS
E
JSEG (Y. Deng, B.S.Manjunath: PAMI’01):1. color quantization: colors are
quantized to several representing
EG
(1.5 m
quantized to several representing classes that can be used to differentiate regions in the image.
2. spatial segmentation: A region min)
2. spatial segmentation: A region growing method is then used to segment the image.
![Page 40: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/40.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 40
Benchmark results using theBenchmark results using the Berkeley Segmentation Datasety g
RJMCMCJSEG
![Page 41: Markov Random Fields Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 3 Segmentation as a Pixel Labelling Task 1. Extract features from the input image Each](https://reader030.fdocuments.net/reader030/viewer/2022040817/5e61f9a123f58a10d0217755/html5/thumbnails/41.jpg)
Zoltan Kato: Markov Random Fields in Image SegmentationZoltan Kato: Markov Random Fields in Image Segmentation 41
References
Visit http://www.inf.u-szeged.hu/~kato/Forthcoming book:g