Image segmentation combining Markov Random Fields and ... · Inference : Swendsen-Wang algorithm...

ANR meeting

Image segmentation combining MarkovRandom Fields and Dirichlet Processes

Jessica SODJO

IMS, Groupe Signal Image, TalenceEncadrants : A. Giremus, J.-F. Giovannelli, F. Caron, N. Dobigeon

Jessica SODJO ANR meeting 1 / 28

ANR meeting

Plan

1 Introduction

2 Segmentation using DP modelsMixed MRF / DP modelInference : Swendsen-Wang algorithm

3 Hierarchical segmentation with shared classesPrincipleHDP theory

4 Conclusion and perspective


ANR meeting

Introduction

Segmentation

– partition of an image in K homogeneous regions calledclasses

– label the pixels : pixel i ↔ zi ∈ 1, . . . ,K

Bayesian approach

– prior on the distribution of the pixels– all the pixels in a class have the same distribution

characterized by a parameter vector Uk

– Markov Random Fields (MRF) : exploit the similarity ofpixels in the same neighbourhood

Constraint : K must be fixed a priori

Idea : use the BNP models to directly estimate K


ANR meeting

Segmentation using DP models

Plan

1 Introduction





ANR meeting


Notations

– N is the number of pixels– Y is the observed image– Z = z1, . . . , zN– Π = A1, . . . ,AK is a partition and m = m1, . . . ,mK with

mk = |Ak |

A1

A3

A2

AK

m1 = 1m2 = 5m3 = 6mK = 4

FIGURE: Example of partitionJessica SODJO ANR meeting 5 / 28

ANR meeting


Mixed MRF / DP model

Markov Random Fields (MRF)

– Description of the image by a neighbouring system

4-neighbours 8-neighbours

ConsideredpixelNeighbours

FIGURE: Examples of neighbouring system

– A clique c is either a singleton either a set of pixels in thesame neighbourhood


ANR meeting



Markov Random Fields

Let θi ∈ U1, . . . ,UK be the parameter vector associated to thei-th pixel

MRF⇔ p(θi | θ−i) = p(θi | θV(i))where V(i) is the set of neighbours of pixel i

Hammersley-Clifford theorem⇒ Gibbs field

p(θ) =1

ZΦexp (−Φ(θ)) =

1ZΦ

exp

(−∑

c

Φc(θc)

)(1)

with Φc(θc) the local potential and Φ(θ) the global one

Limitation : K is assumed to be known


ANR meeting



Potts model

The Potts model is a special MRF defined by :

M(Π) ∝ exp

∑i↔j

βij1zi =zj

(2)

where– i ↔ j means that the pixels i and j are neighbours– βij > 0 if i and j are neighbours and βij = 0 otherwise


ANR meeting



The DP model

τ ′k | γ,H ∼ Beta(1, γ) τk = τ ′k

k−1∏l=1

(1− τ ′l ) (3)

where Beta(.) is the Beta distribution

Let us write τ ∼ Stick(γ), τ = τ1, τ2, . . . and∑∞

k=1 τk = 1

G | γ,H ∼ DP(γ,H) G =∞∑

k=1

τkδUk (4)

withUk | H

iid∼ H (5)

The distribution of the observations is f , defined as :

yi | θi ∼ f (. | θi) and θi | G ∼ G (6)


ANR meeting



The DP model

The Chinese Restaurant Process says,

θi | θ−i ∼K−i∑k=1

m−ik

N − 1 + γδUk +

γ

N − 1 + γH

– m−ik is the size of cluster k if we remove pixel i from the

partition– K−i is the number of clusters in the image with the i-th

pixel removed– Uk is the parameter vector associated to the k -th cluster

Limitation : the spatial interactions are not taken into account


ANR meeting



Principle of the segmentation using DP models

Define a distribution on the partitions using :– a model that allows that pixels in the same neighbourhood

are likely to be in the same cluster (MRF)– DP model to deduce automatically the number of clusters

(and if needed their parameters)


ANR meeting



Prior distribution mixing DP and MRF

p(θ) ∝ 1ZG

exp(−∑

i

Φi(θi))︸︷︷︸Ψ(θ) DP model

1ZM

exp(−∑c∈C2

Φc(θc))

︸︷︷︸M(θ) MRF model

where– C2 means |c| > 2 and |.| is the size.– Φi(.) is defined as :

Φi(θi) = − logG(θi) and ZG =

∫ N∏i=1

exp(− logG(θi))dθ1 . . . dθN

⇒ Ψ(θ) =N∏

i=1

G(θi)

P. Orbanz & J. M. BuhmannNonparametric Bayesian image segmentation, International Journal of Computer Vision, 2007


ANR meeting



Prior distribution mixing DP and MRF

We can deduce :

P(θi | θ−i) ∝K∑

k=1

M(θi | θ−i)m−ik δUk +

γ

ZΦH (7)

Probability of assignment to a new cluster :

qi0 ∝∫

Ωθ

f (yi | θ)H(θ)dθ (8)

Probability of assignment to an existing cluster :

qik ∝ m−ik exp(−Φ(Uk | θ−i))f (yi | Uk ) (9)

Parameter update :

Uk ∼ G0(Uk )∏

i|i∈Ak

f (yi | Uk ) (10)


ANR meeting


Inference : Swendsen-Wang algorithm

Swendsen-Wang algorithm : principle

* Estimation based on the joint posterior p(θ,Z | Y )

* Intractable⇒ Markov Chain Monte Carlo (MCMC)

Problem : very slow convergence

Goal : Sample faster the partition of the image– Introduction of a new set of latent variables r such that :

p(Π, r) = p(Π)p(r | Π)

p(r | Π) =∏

1<i<j<Np(rij | Π)

p(rij = 1 | Π) = 1− exp(βijδij1zi =zj )

The marginal posterior p(θ,Z | Y ) is unchanged– The links define the "so-called" spin-clusters


ANR meeting




– Update the labels of the spin-clustersThis operation update simultaneously the labels of all thepixels in a spin-cluster

FIGURE: Example of label update for spin-clusters


ANR meeting




– rij ∼ Ber(1− exp(βijδij1zi =zj ))with Ber(.) is the Bernouilli distribution

Let S = S1, . . . ,Sp be the set of spin-clusters.

– While removing the spin-cluster Sl ,

Π−l = A−l1 , . . . ,A−l

K−l is the partition obtained while

removing all pixels in spin-cluster Sl

m−lk = |A−l

k |


ANR meeting




For l = 1 : p* The probability to assign pixels in spin-cluster Sl to cluster

k is :

qlk ∝ Ψ(m−l1 , . . . ,m−l

k + |Sl |, . . . ,m−lK−l

)p(ySl | yA−lk

)∏(i,j)|i∈Sl ,rij =0

exp(βij (1− δij )1zi =zj )

* The probability to assign pixels in spin-cluster Sl to a newcluster is :

ql0 = Ψ(m−l1 , . . . ,m−l

K−l, |Sl |)p(ySl )

with p(yAk ) =∫ ∏

i∈Ak

f (yi | Uk )H(Uk )dUk


ANR meeting

Hierarchical segmentation with shared classes

Plan

1 Introduction





ANR meeting


Principle

Proposed idea

– Different levels of classification can be considered– Coarse categories : urban, sub-urban, forest, etc.– Sub-classes shared between the categories : trees, roads,

buildings

Taking into account the fact that the classes are sharedbetween different categories can help estimating theirparameters and thereby improve the segmentation


ANR meeting


HDP theory

Solution : Hierarchical DP

Let J be the number of categories

G0 | γ,H ∼ DP(γ,H)

Gj | α0,G0 ∼ DP(α0,G0) for j = 1, . . . , J

α0 ∈ R∗+G0 is a discrete distribution

Discreteness of G0 ⇒ clusters shared among categories


ANR meeting


HDP theory

G0 =∞∑

k=1

τkδUk (11)

where τ |γ ∼ Stick(γ), τ = τ1, τ2, . . . and Uk | H ∼ H

Gj =∞∑

k=1

πjkδUk (12)

with πj | α0, τ ∼ DP(α0, τ ) and πj = πj1, πj2, . . .

ϕji | Gj ∼ Gj (13)

So, samples of the processes G0 and Gj can be seen as infinitecountable mixtures of Dirac measures with respectivecoefficients τ and πj .


ANR meeting


HDP theory

Principle - Chinese Restaurant Franchise

NOTATIONS

– J restaurants– Same menu for all restaurants - U1,U2, . . .

– Tj is the number of tables in restaurant j– θjt is the t-th table of restaurant j– ϕji is the i-th client in restaurant j– njt is the number of clients at a table t– ηjk is the number of tables in restaurant j

which have chosen dish Uk and ηk =∑

k ηjk


ANR meeting


HDP theory

Principle - Chinese Restaurant FranchiseRestaurant 1

. . .ϕ11ϕ13

ϕ12 ϕ14

Restaurant 2

. . .ϕ21ϕ23

ϕ22ϕ25

Restaurant 3

. . .ϕ31 ϕ32 ϕ33

MenuU1U2U3...

θ 11=

U 1

θ 12=

U 2

θ 13=

U 2

θ 21=

U 2

θ 22=

U 1

θ 31=

U 1

θ 32=

U 2

θ 33=

U 3


ANR meeting


HDP theory

Principle - Chinese Restaurant Franchise

Exemple : Restaurant 1 ϕ13ϕ11

ϕ12

ϕ14

θ 11=

U 1

θ 12=

U 2

θ 13=

U 2

ϕ15

n11 =

2

n12 = 1

n13=

1

α 0

θ14

MenuU1

U2

U3

η1 = 3η2 = 4

η3 = 1

γU4Jessica SODJO ANR meeting 24 / 28

ANR meeting


HDP theory

Chinese Restaurant Franchise

ϕji | ϕj1, . . . , ϕji−1, α0,G0 ∼Tj∑

t=1

njt

i − 1 + α0δθjt +

α0

i − 1 + α0G0 (14)

θjt | θj1, . . . , θ21, . . . , θjt−1, γ,H ∼K∑

k=1

ηk∑k ηk + γ

δUk +γ∑

k ηk + γH (15)

Y. W. Teh, M. I. Jordan, M. J. Beal & D. M. BleiHierarchical Dirichlet Processes, JASA, 2006


ANR meeting

Conclusion and perspective

Plan

1 Introduction





ANR meeting

Conclusion and perspective

Conclusion– Spatial constraints : Potts model– Flexibility : DP model– Rapidity : Swendsen-Wang algorithm– Sharing : HDP

Perspective– Efficient sampling algorithm


ANR meeting

Thank

Thank you for your attention


Image segmentation combining Markov Random Fields and ... · Inference : Swendsen-Wang algorithm...

Documents

Transcript of Image segmentation combining Markov Random Fields and ... · Inference : Swendsen-Wang algorithm...