Fast and exact L1+TV Minimization and some links with ......tu-logo ur-logo Fast and exact L1 +TV...
Transcript of Fast and exact L1+TV Minimization and some links with ......tu-logo ur-logo Fast and exact L1 +TV...
tu-logo
ur-logo
Fast and exact L1 + TV Minimization andsome links with Mathematical Morphology
Jerome Darbon1
1EPITA Research and Development Laboratory (LRDE)14-16, rue Voltaire F-94276 Le Kremlin Bicetre, France
Medical Team SeminarJune 9 2006
Medical Team Seminar 1
tu-logo
ur-logo
Context
Many computer vision/image proccessing problem can beexpressed as an energy minimization problem
restorationsegmentation
E(u|v) =
∫Ω
D(u, v)︸ ︷︷ ︸Data fidelity
+β
∫Ω
R(u)︸ ︷︷ ︸Regularisation
High dimensional problemGenerally non-convexFast algorithmsExact solution
Medical Team Seminar 2
tu-logo
ur-logo
ContextOptimization: continuous approaches
Gradient descent, Euler-Lagrange, [Rudin et al. Physica D. 92]Duality [Chambolle JMIV 04]”Graduated Non Convexity” (GNC) [Blake and Zisserman 87]
Optimization: Discrete approaches, Markov Random Fields (MRFs)Dynamic programming (global minimizer) [Amini et al PAMI 90]Simulated Annealing (global optimizer) [Geman et al. PAMI 84]Iterated Conditional Mode (local minimizer) [Besag JRSC 86]Message Passing [Wainright et al. ITIT 05]Maximum Flow:
global optimum for some binary MRFs [Barahona J. Phys. A 85][Boykov et al., PAMI 01] [Greig et al. JRSC 89][Kolmogorov et al., PAMI 04] [Picard et al. Networks 75]Approximate solution [Boykov et al. PAMI 01]Exact solution [Ishikiwa PAMI 03]Exact solution for Convex MRFs [Bioucas 05][Darbon 05] [Kolmogorov 05]
Our approach:Reformulate energies as one (or many) binary MRF(s)Get a global minimizer
Medical Team Seminar 3
tu-logo
ur-logo
ContextOptimization: continuous approaches
Gradient descent, Euler-Lagrange, [Rudin et al. Physica D. 92]Duality [Chambolle JMIV 04]”Graduated Non Convexity” (GNC) [Blake and Zisserman 87]
Optimization: Discrete approaches, Markov Random Fields (MRFs)Dynamic programming (global minimizer) [Amini et al PAMI 90]Simulated Annealing (global optimizer) [Geman et al. PAMI 84]Iterated Conditional Mode (local minimizer) [Besag JRSC 86]Message Passing [Wainright et al. ITIT 05]Maximum Flow:
global optimum for some binary MRFs [Barahona J. Phys. A 85][Boykov et al., PAMI 01] [Greig et al. JRSC 89][Kolmogorov et al., PAMI 04] [Picard et al. Networks 75]Approximate solution [Boykov et al. PAMI 01]Exact solution [Ishikiwa PAMI 03]Exact solution for Convex MRFs [Bioucas 05][Darbon 05] [Kolmogorov 05]
Our approach:Reformulate energies as one (or many) binary MRF(s)Get a global minimizer
Medical Team Seminar 4
tu-logo
ur-logo
Outline
α) NotationsA) Total Variation minimization with convex fidelityB) L1 + TV as a constrast invariant filterC) L1 + TV on the FLST-treeD) Vectorial Mathematical Morphology via constrained L1 + TVE) Conclusions
Medical Team Seminar 5
tu-logo
ur-logo
Notations
Discretization
s ∈ S finite discrete grid
us ∈ [0, L− 1] finite number of gray-levels
s ∼ t → (s, t) neighbors → cliques (C-connectivity)
Level setsu λ = s ∈ S|1lus≤λ
We consider level sets as variables
Medical Team Seminar 6
tu-logo
ur-logo
Total variation minimization
A) Total Variation minimisation with convex fidelityConvex problemImage restorationReformulation through level setsPolynomial algorithmResults
Medical Team Seminar 7
tu-logo
ur-logo
TV: Reformulation through level sets
Total Variation∫Ω|∇u| =
∫IR
P(uλ)︸ ︷︷ ︸Co-area formula
=L−2∑λ=0
P(uλ) =L−2∑λ=0
∑(s,t)
wst |uλs − uλ
t |︸ ︷︷ ︸Rst (uλ
s ,uλt )
Data fidelity
D (us, vs) =L−2∑λ=0
(D(λ + 1, vs)− D(λ, vs)) (1− uλs )︸ ︷︷ ︸
Dλ(uλs ,vs)
+D(0, vs)
ÔE(u|v) =L−2∑λ=0
(Rst(uλ
s , uλt ) + Dλ(uλ
s , vs))
︸ ︷︷ ︸binary MRF
+C =L−2∑λ=0
Eλ(uλ, v)
Medical Team Seminar 8
tu-logo
ur-logo
TV: Independent minimization and reconstruction
Minimize (MAP) independently each binary MRFE(u|v) → E(uλ, v)Family of minimizers: uλλ=0...(L−2)
Reconstruction: us = infλ|1luλs
= 1 provided that
uλs ≤ uµ
s ∀λ < µ ∀s (monotony)
monotone lemmaIf all conditional energies E(us | Ns, vs) are convex functions ofgrey level us ∈]0, L− 1[, for any neighborhood configurationand local observed data, then the monotone property holds.
⇒ ”convex+TV” models satisfies lemma’s conditions
Medical Team Seminar 9
tu-logo
ur-logo
TV : MAP of a binary MRF
How to minimize a binary Markovian energy ?Build a graph such that its maximum flow yields an optimal labelling
construction of the graph: [Kolmogorov and Zabih, PAMI 2004]maximum flow algorithm: [Boykov and Kolmogorov, PAMI 2004]in practice quasi-linear (w.r.t number of pixels)in theory O(n2√m) (n=# pixels, m=# arcs)
picture taken from Boykov et al. PAMI 2001
Medical Team Seminar 10
tu-logo
ur-logo
TV: Graph construction conditions
Regularity conditions described in [Picard et al. Networks 1975][Barahona J. Physics A 1985] [Kolmogorov et al., PAMI 2004]Binary Markovian energy with pairwise interaction
E(x1, . . . , xn) =∑
i
E i(xi) +∑i<j
E i,j(xi , xj)
E i(xi) : always regularE i,j(xi , xj) : regular iff submodular, i.e.
E i,j(0, 0) + E i,j(1, 1) ≤ E i,j(1, 0) + E i,j(0, 1)
TV case:∑
st wst |us − ut |0 ≤ wst ok
Medical Team Seminar 11
tu-logo
ur-logo
TV: Minimization algorithm
Decomposition through level sets (recall)
E(u|v) =L−2∑λ=0
Eλ(uλ|v)
Direct approach =⇒ (L− 1) minimum cost cuts per pixelA divide-and-conquer algorithm with dichotomy
decompose into independent subproblemssolve each subproblemrecompose the solution
Medical Team Seminar 12
tu-logo
ur-logo
TV: Minimization algorithm
Decomposition: Solve for a level λ ; connected components
Independance: wst |uλs − uλ
t |Solving sub-problems and recomposition
Thresholding: dichotomy on [0, L− 1] =⇒ log2(L) maximum flowsMedical Team Seminar 13
tu-logo
ur-logo
TV: Results
Medical Team Seminar 14
tu-logo
ur-logo
TV: Results, additive Gaussian noise
µ = 0, σ = 12 restored image (β = 23, 5)
Medical Team Seminar 15
tu-logo
ur-logo
TV: Results, additive Gaussian noise
µ = 0, σ = 20 restored image (β = 44, 5)
Medical Team Seminar 16
tu-logo
ur-logo
TV: Results, time
Medical Team Seminar 17
tu-logo
ur-logo
TV: Results, time
size (512× 512) ; L2; time in seconds
Image β = 2 β = 5 β = 10 β = 20 β = 30 β = 40Lena 2,07 2,24 2,53 3,04 3,40 3,75Aerial 2.13 2.24 2.45 2.75 3.06 3.28Barbara 2.07 2.26 2.51 2.87 3.22 3.50
size (256× 256) ; L2; time in seconds
Image β = 2 β = 5 β = 10 β = 20 β = 30 β = 40Lena 0,51 0,54 0,60 0,72 0,80 0,87Aerial 0,55 0,57 0,61 0,67 0,74 0,78Barbara 0,53 0,55 0,60 0,69 0,75 0,80Girl 0,52 0,55 0,64 0,75 0,85 0,92
Experiments performed on a Pentium 4 3 GHz
Medical Team Seminar 18
tu-logo
ur-logo
TV: Partial conclusion
Exact solution for ”convex+TV” modelsReformulation through level setsPolynomial algorithm. Complexity log2(L) · T (n, (C + 2)n)
Similar algorithm proposed byDirect approach: Zalesky [Zalesky JAM 02]Dichotomy: Chambolle [Chambolle EMMCVPR 05]Parametric max-flow: Hochbaum [Hochbaum ACM 01]
More details and generalization to convex priors and”levelable” priors in
J. Darbon and M. Sigelle. Image Restoration with DiscreteConstrained Total VariationPart I: Fast and Exact OptimizationPart II: Levelable Functions, Convex Priors andNon-Convex CasesAccepted to Journal of Mathematical Imaging and Vision
where T (n, m) is the time required to perform a maximum flow on agraph of n nodes and m edges.
Medical Team Seminar 19
tu-logo
ur-logo
L1 + TV is a contrast invariant filter
α) NotationsA) Total Variation minimization with convex fidelityB) L1 + TV as a constrast invariant filter
Definition and TheoremExperiments
Medical Team Seminar 20
tu-logo
ur-logo
Definitions
Level sets and threshold decomposition principleLλ(u) = x ∈ Ω|u(x) ≤ λ , Uλ(u) = x ∈ Ω|u(x) > λNotation Lλ(u) = uλ
Reconstruction from level setsu(x) = supλ|1lLλ(u)(x) = 1 = infµ|1lUµ(u)(x) = 1Any continuous non-decreasing function g is called acontinuous change of contrast.Lemma [Guichard 02 ISMM]:
∀λ ∃µ Lλ(g(u)) = Lµ(u) .
F is a contrast invariant filter iff
g(F (u)) = F (g(u))
Medical Team Seminar 21
tu-logo
ur-logo
L1 + TV : Theorem
L1 + TV based filter
min Ev (u) =
∫Ω|u(x)− v(x)|dx + β
∫Ω|∇u| ,
Theorem: Assumeobserved image vcontinuous change of contrast: ga global minimizer of Ev (·): u
Then g(u) is a global minimizer of Eg(v)(·)It is enough to show that for all λ a minimizer for g(v)λ isg(u)λ
Medical Team Seminar 22
tu-logo
ur-logo
L1 + TV : Idea of the proof
Reformulation of L1 + TV through level setsCoarea Formula:∫
Ω
|∇u| =∫
IR
∫Ω
|∇χuλ |dλ =
∫IR
P(uλ)dλ
Fidelity terms:
|u(x)− v(x)| =∫
IR
∣∣uλ(x)− vλ(x)∣∣ dλ
Finally
Ev (u) =
∫IR
Eλv (uλ, vλ)dλ
where
Eλv (uλ, vλ) =
∫Ω
(β |∇χuλ |+
∣∣∣uλ(x)− vλ(x)∣∣∣ dx
)Medical Team Seminar 23
tu-logo
ur-logo
L1 + TV : Experiments
original β = 1, 5
Medical Team Seminar 24
tu-logo
ur-logo
L1 + TV : Results
β = 2, 5 β = 3, 0
Medical Team Seminar 25
tu-logo
ur-logo
L1 + TV : Results
Original imageMedical Team Seminar 26
tu-logo
ur-logo
L1 + TV : Results
[Yin et al. 05 CAM report] Result with β = 2 (left) and difference with the originalimage (right)Medical Team Seminar 27
tu-logo
ur-logo
L1 + TV : Partial Conclusion
Original proof in J. Darbon. Total Variation Minimization with L1 DataFidelity as a Contrast Invariant Filter. 4th International Symposium onImage and Signal Processing and Analysis (ISPA 2005).Proof in the discrete case in J. Darbon and M. Sigelle. ImageRestoration with Discrete Constrained Total Variation: Part I: Fast andExact Optimization.
Medical Team Seminar 28
tu-logo
ur-logo
L1 + TV on the FLST-tree
A) Total Variation minimization with convex fidelityB) L1 + TV as a constrast invariant filterC) L1 + TV on the FLST-tree
Medical Team Seminar 29
tu-logo
ur-logo
L2 + TV, contrast and contours
Loss of contrast → use L1
How to preserve contours ?
Medical Team Seminar 30
tu-logo
ur-logo
Fast Level Set Transform Tree
Level SetsLλ(u) = x ∈ Ω|u(x) ≤ λ , Uλ(u) = x ∈ Ω|u(x) > λInclusion property
Uλ(u) ⊂ Uµ(u) ∀λ ≥ µ
Lλ(u) ⊂ Lµ(u) ∀λ ≤ µ
Induce a tree [Salembier et al. ITIP 98] :conected components of lower/upper setsLλ → ”Min-Tree” → dark objects on light backgroundUλ → ”Max-Tree” → light objects on dark background
”Fast Level Set Transform” (FLST) [Monasse et al ITIP 2000]Merge the 2 trees into a single oneNeed of a criteria : Holes
Medical Team Seminar 31
tu-logo
ur-logo
Fast Level Set Transform Tree
Shapes = connected components of level setswhose holes have been filed.Definition of the tree
1 node = 1 shapeParent = smallest form which contains itchildren = included forms
Decomposition of the image into forms S1 ...Sn
Medical Team Seminar 32
tu-logo
ur-logo
Fast Level Set Transform Tree
L1 + TV on the FLST treeequivalent toL1 + TV + edge preservationAttributes associated to each node
gray level uiarea for data fidelity → |Di |perimeter for TV (co-aire formula) → Pi
Data fidelity:N∑
i=1
|Di ||ui − vi |
Total Variation (Dibos etal 2000 SIAM NA):
N−1∑i=1
Pi |ui − upi |
Medical Team Seminar 33
tu-logo
ur-logo
L1 + TV on the FLST-tree
Finally
EL1+TV (u|v) =N∑
i=1
|Di ||ui − vi |+ β
N−1∑i=1
Pi |ui − upi |
Sites: nodes of the treeNeighborhoods → parents et childrenpairwise interactionsMAP of the Markovian energyL1 + TV→ L1 + TV algorithm of the first part
Medical Team Seminar 34
tu-logo
ur-logo
Results
Original image β = 3
Medical Team Seminar 35
tu-logo
ur-logo
Results
β = 15 borders of the result (β = 30)
5 regions
Medical Team Seminar 36
tu-logo
ur-logo
Results
Original image
Pertinent contours ?
Medical Team Seminar 37
tu-logo
ur-logo
Results
β = 1
Medical Team Seminar 38
tu-logo
ur-logo
Results
β = 2
Medical Team Seminar 39
tu-logo
ur-logo
Results
borders of the result (β = 10) superimposed on the original image
2 regions
Medical Team Seminar 40
tu-logo
ur-logo
Time result
Image FLST MinimizationLena (256x256) 0.18 0.11Lena (512x512) 1.09 1.04Woman (522x232) 0.39 0.06Squirrel (209x288) 0.24 0.19
FLST tree computed with the implementation available in Megawave(ENS de Cachan)
Medical Team Seminar 41
tu-logo
ur-logo
Partial conclusion
Contrast preservation (since it is morphological)Edge preservationFastGood simplification for future segmentaionRestatement of morphological connected filters as aconstrained L1 + TV minimization
Medical Team Seminar 42
tu-logo
ur-logo
L1 + TV on the FLST-tree
α) NotationsA) Total Variation minimization with convex fidelityB) L1 + TV as a constrast invariant filterC) L1 + TV on the FLST-treeD) Vectorial Mathematical Morphology via L1 + TV
Medical Team Seminar 43
tu-logo
ur-logo
Vectorial Mathematical Morphology: Related works
Lot of work for color images→ specific order in design color spaces
Vector Median Filter:Replace the pixel value by the median of the pixels contained in awindow around itMedian ≡ value that minimizes the L1-norm between all pixels in awindow
Variational ApproachL1+ Total Variation yields a self dual and contrast invariantinvariant filter
E(u) =
∫Ω|u(x)− v(x)|dx + β
∫Ω|∇u|
Medical Team Seminar 44
tu-logo
ur-logo
Total Variation based Approach
Idea: Use the fact that L1 + TV -scalar is morphologicvectorial image : u = (u1, . . . , uN) (N channels)L1 + TV on each chanel
‖u‖L1 =N∑
i=1
∫Ω|ui(x)|dx .
−→TV (u) =
N∑i=1
∫Ω|∇ui |.
Invariance w.r.t. a vectorial change of contrast
⇔
Invariance w.r.t. a change of contrast on each channel
Medical Team Seminar 45
tu-logo
ur-logo
Total Variation based Approach
Enforce the morphological behavior argminu
‖u − v‖1 + β−→TV (u) contrast invariance
s.t. no new vectorial values
Difficult problemnon convex problem !
Medical Team Seminar 46
tu-logo
ur-logo
Minimization Algorithm
Definition of α−expansion
Let u be an image and α a new label.An image uα is at one α-expansion of u iff either uα
s = us or uαs = α.
Theorem
Let u be an image such that
E(u) > infu′
E(u′) .
Then, there exists uα which is within one α−expansion move of u,such that
E(u) > E(uα) .
Medical Team Seminar 47
tu-logo
ur-logo
Minimization Algorithm
Initialize with a labelling u such that ∀s us ∈ CDo
∀α ∈ C u′ = argminu
E(u) where u is an α−expansion from u
if E(u′|v) < E(u|v)
u ← u′
success ← true
while success 6= false
Medical Team Seminar 48
tu-logo
ur-logo
Results
Original β = 1, 1
Medical Team Seminar 49
tu-logo
ur-logo
Results
β = 1, 7 β = 2, 5
Medical Team Seminar 50
tu-logo
ur-logo
Results
β = 3, 5 Difference original and β = 3, 5
Medical Team Seminar 51
tu-logo
ur-logo
Partial conclusion
ContributionA vectorial morphological vectorNo need of any order relation on vectors
CommentsVery slow !Generalization to non-separable case−→ seems to be very difficult
More details in Jerome Darbon and Sylvain Peyronnet. AVectorial Self-Dual Morphological Filter based on TotalVariation Minimization. in ISVC 2005 (LNCS 3804).
Medical Team Seminar 52
tu-logo
ur-logo
Conclusion
Exact optimization forConvex + TV (polynomial)Any + Levelable (pseudo-polynomial)Any + Convex (pseudo-polynomial)
We have sucessfully adapted these approaches toSolve Convex + Convex model (pseudo-polynomial)Exact solution of the Chan and Vese model
L1 + TV is invariant with respect to changes of contrastRestatement of morphological connected filters as a L1 + TVminimizationExtension to the vectorial case
Medical Team Seminar 53