Post on 28-Mar-2015
Image Segmentationwith Level Sets
Group reading22-04-05
Outline
1. Active Contours
2. Level sets in Malladi 1995
3. Prior knowledge in Leventon 2000
1. Active Contours
1. Active Contours
1.1 Introduction to AC1.2 Types of curves1.3 Types of influences1.4 Summary
1.1 Introduction to AC
• AC = curve fitted iteratively to an image– based on its shape and the image values– until it stabilises (ideally on an object’s boundary).
• curve: polygon = parametric AC continuous = geometric AC
parametric geometric
1.1 Introduction to AC
• Example: cell nucleus• Initialisation: external• Fitting: iterative
– Using the current geometry (“Contour influence”)and the image contents (“Image influence”)
– “Influence”:scalar = energyvector = force
– Until the energy is minimum, or the forces are null
Image: Ian Diblik 2004
1.2 Types of curves
• Parametric AC:– Stored as vertices– Each vertex is moved
iteratively
• Geometric AC:– Stored as coefficients– Sampled before each
iteration– Each sample is moved– New coefficients are
computed (interpolation)
1.3 Types of influences
• Each vertex /sample vi is moved:
in a direction that minimises its energy:E(vi) = Econtour(vi) + Eimage(vi)
or, along the direction of the force:F(vi) = Fcontour(vi) + Fimage(vi)
( Linear combinations of influences )
1.3 Types of influences
• Contour influence:– meant to control the geometry locally around vi
– corresponds to the prior knowledge on shape
• Energy:example: Econtour(vi) = . ||v’i||2 + . ||v’’i||2
• Force:example: Fcontour(vi) = v’i + v’’i + ni
where ni is the normal of the contour at vi
and is a constant pressure (“balloon”)
1.3 Types of influences
• Image influence:– meant to drive the contour locally– corresponds to the prior knowledge on the object’
boundaries
• Energy:example: Eimage(vi) = ( I (vi) - Itarget)2
Eimage(vi) = . 1 / ( 1 + || I (vi) || ) Malladi 1995
Eimage(vi) = - . || I (vi) || Leventon 2000
• Force:example: Fimage(vi) = Eimage(vi) ni
Fimage(vi) = - I (vi)
1.3 Types of influences
• Exotic influences:Econtour(vi) proportional to the deformations of a template:
Eimage(vi) based on region statistics:
Ioutside
Iinside
Bredno et al. 2000
Hamarneh et al. 2001
1.4 Summary
• Active Contours:– function v : N P ( [ 0 , Iwidth ] x [ 0 , Iheight ] )
t { (x,y) }– v(0) is provided externally
– v(t+1) is computed by moving sample points vi of v(t)
– moved using a linear combination of influences
– on convergence, v(tend) is the border of the object
• Issues:– fixed topology– possible self-crossing
2. Level sets in Malladi 1995
2. Level sets in Malladi 1995
2.1 Introduction to LS2.2 Equation of motion2.3 Algorithm 12.3 Narrow-band extension2.4 Summary
2.1 Introduction to LS
• Transition from Active Contours:– contour v(t) front (t)– contour energy forces FA FC
– image energy speed function kI
• Level set:– The level set c0 at time t of a function (x,y,t)
is the set of arguments { (x,y) , (x,y,t) = c0 }
– Idea: define a function (x,y,t) so that at any time,(t) = { (x,y) , (x,y,t) = 0 }
• there are many such • has many other level sets, more or less parallel to • only has a meaning for segmentation, not any other level set of
2.1 Introduction to LSUsual choice for : signed distance to the front (0)
- d(x,y, ) if (x,y) inside the front (x,y,0) = 0 “ on “
d(x,y, ) “ outside “
0
0
0
0 0 0 0
0 0
0
0
0
0
0
0
0
0
000
0
0
000
0
-1-1
-1
-1
-1
-1
-1
-1
-1-1-1
-1
-1
-1-1-1
-1
-1
-1 -1 -1 -1
-2 -2 -2 -2
-2
-2
-2
-2-2-2
-2
-2
-2 -2 -3 -3
-3
-3
-3
-3
1
1
1
1
1
1
1
1
111
1
1
111
1
1
1
1
1
1
1 1 1 1
1 1
1
1
2
2
2
2
2
2
2
22
2
222
2
2
2
2
2
2
2
2
2 2 2 2 2
2
2
2
3
3
3
3
3
3
3
3
3
33
333
3
3
3
3
3
3
3 4
4
4
4
4
4
4
4444
4
4
4
4
4
4
7
5
5
5
5
55
5
5
5
5
6
6
6 6
(t)(x,y,t)
(x,y,t)
0
-2
5
2.1 Introduction to LS
0
0
0
0 0 0 0
0 0
0
0
0
0
0
0
0
0
000
0
0
000
0
-1-1
-1
-1
-1
-1
-1
-1
-1-1-1
-1
-1
-1-1-1
-1
-1
-1 -1 -1 -1
-2 -2 -2 -2
-2
-2
-2
-2-2-2
-2
-2
-2 -2 -3 -3
-3
-3
-3
-3
1
1
1
1
1
1
1
1
111
1
1
111
1
1
1
1
1
1
1 1 1 1
1 1
1
1
2
2
2
2
2
2
2
22
2
222
2
2
2
2
2
2
2
2
2 2 2 2 2
2
2
2
3
3
3
3
3
3
3
3
3
33
333
3
3
3
3
3
3
3 4
4
4
4
4
4
4
4444
4
4
4
4
4
4
7
5
5
5
5
55
5
5
5
5
6
6
6 6
(x,y,t)
(x,y,t+1) = (x,y,t) + ∆(x,y,t)
1
-1
-1
-1
-1
-1
-1
-1
-1-1
-1
-1
-1-1-1
-1
-1 -1
1
1
1
-1
1
1
1
1
1
1
1
1
1
111
1
111
1
1
1
1
1
1 1
1 1
1
-2
-2
-3
-2 -2 -2 -2
-2
-2
-2
-2-2-2
-2
-2
-2 -2 -3 -3
-3
-3
-3
-3
2
2
2
2
2
2
2
2
22
2
222
2
2
2
2
2
2
2
2
2 2 2
0
0
0
0
0
0
0
0
0
0
0
0
00
0
000
0
0
0
0
0
0
0 0
0 0
0
2
2
2
2
3
3
3
3
3
3
3
33
333
3
3
3
3
3
3
3 4
4
4
4
4
4
4
4444
4
4
4
4
4
4
7
5
5
5
5
55
5
5
5
5
6
6
6 6
• no movement, only change of values
• the front may change its topology
• the front location may be between samples
2.1 Introduction to LS
Segmentation with LS:• Initialise the front (0)• Compute (x,y,0)• Iterate:(x,y,t+1) = (x,y,t) +
∆(x,y,t)until convergence
• Mark the front (tend)
2.2 Equation of motion
• Equation 17 p162:
t ˆ k I FA FG () 0
link between spatial and temporal derivatives, but not the same type of motion as contours!
div
constant “force”
(balloon pressure)
(x,y,t+1) - (x,y,t)
extension of the speed function kI (image influence)
smoothing “force” depending on the local curvature (contour influence)
spatial derivative of
product of influences
2.2 Equation of motion
• Speed function:– kI is meant to stop the front on the object’s boundaries
– similar to image energy: kI(x,y) = 1 / ( 1 + || I (x,y) || )
– only makes sense for the front (level set 0)– yet, same equation for all level sets extend kI to all level sets, defining
– possible extension:
k̂I
k̂I (x,y) = kI(x’,y’)where (x’,y’) is the point in the front closest to (x,y)
^( such a kI (x,y) depends on the front location )
2.3 Algorithm 1 (p163)
0
0
0
0 0 0 0
0 0
0
0
0
0
0
0
0
0
000
0
0
000
0
-1-1
-1
-1
-1
-1
-1
-1
-1-1-1
-1
-1
-1-1-1
-1
-1
-1 -1 -1 -1
-2 -2 -2 -2
-2
-2
-2
-2-2-2
-2
-2
-2 -2 -3 -3
-3
-3
-3
-3
1
1
1
1
1
1
1
1
111
1
1
111
1
1
1
1
1
1
1 1 1 1
1 1
1
1
2
2
2
2
2
2
2
22
2
222
2
2
2
2
2
2
2
2
2 2 2 2 2
2
2
2
3
3
3
3
3
3
3
3
3
33
333
3
3
3
3
3
3
3 4
4
4
4
4
4
4
4444
4
4
4
4
4
4
7
5
5
5
5
55
5
5
5
5
6
6
6 6
1
-1
-1
-1
-1
-1
-1
-1
-1-1
-1
-1
-1-1-1
-1
-1 -1
1
1
1
-1
1
1
1
1
1
1
1
1
1
111
1
111
1
1
1
1
1
1 1
1 1
1
-2
-2
-3
-2 -2 -2 -2
-2
-2
-2
-2-2-2
-2
-2
-2 -2 -3 -3
-3
-3
-3
-3
2
2
2
2
2
2
2
2
22
2
222
2
2
2
2
2
2
2
2
2 2 2
0
0
0
0
0
0
0
0
0
0
0
0
00
0
000
0
0
0
0
0
0
0 0
0 0
0
2
2
2
2
3
3
3
3
3
3
3
33
333
3
3
3
3
3
3
3 4
4
4
4
4
4
4
4444
4
4
4
4
4
4
7
5
5
5
5
55
5
5
5
5
6
6
6 6
1. compute the speed kI on the front extend it to all other level sets
2. compute (x,y,t+1) = (x,y,t) + ∆(x,y,t)
3. find the front location (for next iteration) modify (x,y,t+1) by linear interpolation
0
0
0
0 0 0 0
0 0
0
0
0
0
0
0
0
0
000
0
0
000
0
-1-1
-1
-1
-1
-1
-1
-1
-1-1-1
-1
-1
-1-1-1
-1
-1
-1 -1 -1 -1
-2 -2 -2 -2
-2
-2
-2
-2-2-2
-2
-2
-2 -2 -3 -3
-3
-3
-3
-3
1
1
1
1
1
1
1
1
111
1
1
111
1
1
1
1
1
1
1 1 1 1
1 1
1
1
2
2
2
2
2
2
2
22
2
222
2
2
2
2
2
2
2
2
2 2 2 2 2
2
2
2
3
3
3
3
3
3
3
3
3
33
333
3
3
3
3
3
3
3 4
4
4
4
4
4
4
4444
4
4
4
4
4
4
7
5
5
5
5
55
5
5
5
5
6
6
6 6
0
0
0
0
0
0 0 0 0
0 0
0
0
0
0
0
0
0
0
000
0
0
000
0
(x,y,t)
2.4 Narrow-band extension
• Weaknesses of algorithm 1– update of all (x,y,t): inefficient, only care about the
front– speed extension: computationally expensive
• Improvement:– narrow band: only update a few level sets around – other extended speed: kI(x,y) = 1 / ( 1 + || I(x,y)|| )^
0
0
0
0 0 0 0
0 0
0
0
0
0
0
0
0
0
000
0
0
000
0
-1-1
-1
-1
-1
-1
-1
-1
-1-1-1
-1
-1
-1-1-1
-1
-1
-1 -1 -1 -1
-2 -2 -2 -2
-2
-2
-2
-2-2-2
-2
-2
-2 -2
1
1
1
1
1
1
1
1
111
1
1
111
1
1
1
1
1
1
1 1 1 1
1 1
1
1
2
2
2
2
2
2
2
22
2
222
2
2
2
2
2
2
2
2
2 2 2 2 2
2
2
2
2.4 Narrow-band extension
• Caution:– extrapolate the
curvature at the edges
– re-select the narrow band regularly:an empty pixel cannot get a value may restrict the evolution of the front
0
0
0
0 0 0 0
0 0
0
0
0
0
0
0
0
0
000
0
0
000
0
-1-1
-1
-1
-1
-1
-1
-1
-1-1-1
-1
-1
-1-1-1
-1
-1
-1 -1 -1 -1
-2 -2 -2 -2
-2
-2
-2
-2-2-2
-2
-2
-2 -2
1
1
1
1
1
1
1
1
111
1
1
111
1
1
1
1
1
1
1 1 1 1
1 1
1
1
2
2
2
2
2
2
2
22
2
222
2
2
2
2
2
2
2
2
2 2 2 2 2
2
2
2
2.5 Summary
• Level sets:– function : [ 0 , Iwidth ] x [ 0 , Iheight ] x N R
( x , y , t ) (x,y,t)– embed a curve : (t) = { (x,y) , (x,y,t) = 0 } (0) is provided externally, (x,y,0) is computed (x,y,t+1) is computed by changing the values of (x,y,t)
– changes using a product of influences
– on convergence, (tend) is the border of the object
• Issue:– computation time (improved with narrow band)
3. Prior knowledge in Leventon 2000
3. Prior knowledge in Leventon 2000
3.1 Introduction3.2 Level set model3.3 Learning prior shapes3.4 Using prior knowledge3.5 Summary
3.1 Introduction
• New notations:function (x,y,t) shape u(t) ( still a function of (x,y,t) )
front (t) parametric curve C(q) (= level set 0 of u(t) )
extended speed kI stopping function g(I)
Malladi’s model:
choose: FA = -c (constant value)
FG() = - (penalises high curvatures),
^
utg(I)u div
u
u
c u
t ˆ k I (FA FG ()) 0
ut g(I)(FA FG ()) u 0
divu
u
3.1 Introduction
u(t) use a level set model (based on current geometry and local image values):
u(t+1) = u(t) + ∆u(t)
use prior knowledge andglobal image values to
findthe prior shape which isclosest to u(t): call it u*(t)
combine linearly these two newshapes to find the next iteration: u(t+1) = u(t)
+ 1 ∆u(t)
+ 2 ( u*(t) - u(t) )
3.2 Level set model
u(t) use a level set model (based on current geometry and local image values):
u(t+1) = u(t) + ∆u(t)
use prior knowledge andglobal image values to
findthe prior shape which isclosest to u(t): call it u*(t)
combine linearly these two newshapes to find the next iteration: u(t+1) = u(t)
+ 1 ∆u(t)
+ 2 ( u*(t) - u(t) )
3.2 Level set model
• not Malladi’s model:
• it’s Caselles’ model:
utg(I)u div
u
u
c u
utu div g(I)
u
u
c u
utg(I) c u u g
use it in: u(t+1) = u(t) + 1 ∆u(t) + 2 ( u*(t) - u(t))
That’s all about level sets. Now it’s all about u*.
( implicit geometric active contour )
( implicit geodesic active contour )
3.3 Learning prior shapes
From functions to vectors:
u(t) (continuous)
00
00 0 0 0
0 00
00
00
00
0000
00
0000
-1-1-1-1
-1-1
-1-1
-1-1-1-1
-1-1-1-1
-1-1
-1 -1 -1 -1-2 -2 -2 -2
-2-2
-2-2-2-2
-2-2
-2 -2 -3 -3-3-3
-3-3
11
11
11
11
1111
1111
1111
11
1 1 1 11 1
11
22
222
22
222
2222
2222
22
22 2 2 2 2
22
2
33
333
33
33
33333
33
3
33
33 4
44
44
4
44444
44
4
44
4
7
55
55
555
5
55
66
6 6
sample it:111
2
222
2
.
3
3
3
3
4
.
4
4
4
4
7
5
55
.
5
5
5
5
6
6
6
6
put the values in one column = vector
u(t)
(discrete)
3.3 Learning prior shapes
• Storing the training set:– set of n images manually segmented: { Ci , 1 i n }
– compute the embedding functions : T = { ui , 1 i n }
– compute the mean and offsets of T: = 1/n i ui
ûi = ui -
– build the matrix M of offsets:M = ( û1 û2 ûn )
– build the covariance matrix 1/n MMT , decompose as:1/n MMT = U UT
U orthogonal, columns: orthogonal modes of shape variations
diagonal, values: corresponding singular values
3.3 Learning prior shapes
• Reducing the number of dimensions:– u: dimension Nd (number of samples, e.g. Iwidth x Iheight)
– keep only first k columns of U: Uk (most significant modes)
– represent any shape u as: = Uk
T ( u - )
= “shape parameter” of u, dimension k only– restore a shape from :
u = Uk + – other parameter defining a shape u: pose p (position)
– prior shape: u* ( , p)
~
3.3 Learning prior shapes
• At the end of the training:
2
0
0.1
k 10 0 0
0 5 0
0 0 2
k 1
0.1 0 0
0 0.2 0
0 0 0.5
most common variation
least common variation
0.1
1
5
Tk 1 0.405
P()1
2 k k
exp 1
2Tk
1
big
P()1
2 k k
exp 1
2Tk
1
small
Tk 1 7.901
The most common shapes have a higher probability.
3.4 Using prior knowledge
u(t) use a level set model (based on current geometry and local image values):
u(t+1) = u(t) + ∆u(t)
use prior knowledge andglobal image values to
findthe prior shape which isclosest to u(t): call it u*(t)
combine linearly these two newshapes to find the next iteration: u(t+1) = u(t)
+ 1 ∆u(t)
+ 2 ( u*(t) - u(t) )
3.4 Using prior knowledge
• Finding the prior shape u* closest to u(t)– Given the current shape u(t) and the image I, we want:
u*map = (map , pmap) = arg max P( , p | u(t) , I)
– Bayes rule:P(,p | u,I) = P(u | ,p) . P(I |u,,p) . P() . P(p) / P(u,I)
has to be max
= P (u | u*) = exp( - Voutside)compares u and u*
= P (I | u, u*) ≈ P ( I | u*)= exp ( - | h(u*) - |I| |2)compares u* with the image features
measures how likely u* is as a prior shape
1
2 k k
exp 1
2Tk
1
= U(–∞,∞)uniform distribution of poses (no prior knowledge)
constant
3.4 Using prior knowledge
• P( u | u* ) :P(u | u*) = exp ( - Voutside) , Voutside : volume of u outside of u*
u* uVoutside = Card { (x,y) , u(x,y) < u*(x,y) }
– meant to choose the prior shape u* most similar to u– only useful when C(q) gets close to the final boundaries
3.4 Using prior knowledge
P(u(t) | u*1) =1
P(u(t) | u*2) =1
P(u(t) | u*3) =1
u*1
u*2
u*3
P(u(t) | u*2) =0.6P(u(t) | u*2) =0
P(u(t) | u*3) =0.6
P(u(t) | u*1) =1P(u(t) | u*1) =1
P(u(t) | u*3) =1
u(t)
I(x,y)I(x,y)
3.4 Using prior knowledge
• P( I | u , u* ) :– meant to choose the prior shape u* closest to the image– depends on the boundary features (image influence)
P( I | u*) = exp ( - | h(u*) - |I| |2)
u*=0
u*
I
0-2 2
u*=2
u*=-2h(u*)
3.4 Using prior knowledge
u*1
u*2
P(I|u*1) = 0.9
P(I|u*2) = 0.6
u(t)
u(t) + 2 (u*1 - u(t))
3.5 Summary
• Training:– compute mean and covariance of training set– reduce dimension
• Segmentation:– initialise C, compute u(0) with signed distance– to find u(t+1) from u(t), compute:
a variation ∆u(t) using current shape and local image valuesa prior shape u* so that:
• u* is a likely prior shape• u* encloses u(t)• u* is close to the object boundaries
– combine ∆u(t) and u* to get u(t+1)
Conclusions
• Active contours:– use image values and a target geometry– topology: fixed but little robust
• Level sets:– also use image values and a target geometry– but in higher dimension, with a different motion– topology: flexible and robust
• Prior knowledge:– prior shapes influence the segmentation– not specific to level sets– ideas can be adapted to other features (boundaries,
pose)
That’s all folks!