Carsten Rother - Vladimir Kolmogorov - Andrew Blake ... · 2. Adapt GMM parameters 3. Estimate...
Transcript of Carsten Rother - Vladimir Kolmogorov - Andrew Blake ... · 2. Adapt GMM parameters 3. Estimate...
Ursina Caluori, [email protected] 2.2.2006
Interactive Foreground Extraction using Iterated Graph Cuts
Carsten Rother - Vladimir Kolmogorov - Andrew Blake
Siggraph '04
Outline
Introduction & MotivationPrevious Approaches• Magic Wand• Intelligent Scissors• Graph CutThe Algorithm• Novelties• Behind the ScenesResultsConclusions
Introduction | Previous Approaches | The Algorithm | Results | Conclusions 2
MotivationForeground Extraction: Say what?
Introduction | Previous Approaches | The Algorithm | Results | Conclusions 3
We have We want
MotivationThe GrabCut Approach
Introduction | Previous Approaches | The Algorithm | Results | Conclusions 4
User input Output
GrabCut
IntroductionOverview
Introduction | Previous Approaches | The Algorithm | Results | Conclusions 5
Iterative Minimization
1. User marks object 2. Intermediate Result
3. User adjusts selection 4. Final Result
"Grab" "Cut"
The Goals...• interactive foreground extraction• high performance & quality - minimal user input• usability in non-trivial images
...can only be achieved with• good user interface ( rectangle / lasso)• accurate segmentation ( iterative graph cuts)• convincing alpha values ( border matting)
Introduction | Previous Approaches | The Algorithm | Results | Conclusions 6
IntroductionAspirations
Previous ApproachesMagic Wand
Introduction | Previous Approaches | The Algorithm | Results | Conclusions 7
Input: Seed Pixel(s) + Tolerance Output: Pixels within tolerance of seed pixels' colors statistics
Thresholding
Introduction | Previous Approaches | The Algorithm | Results | Conclusions 8
Input: Points on segmentation boundary
Output: Pixels within minimum cost contour
Energy Minimization&
Interpolation
Previous ApproachesIntelligent Scissors
Previous Approaches
Graph Cut• foundation of GrabCut pay attention • uses boundary and region information• segmentation: min-cut by energy minimization
Introduction | Previous Approaches | The Algorithm | Results | Conclusions 9
Input: Clue-mark inside and outside region
Output: Pixels within "best" inside region
Min-Cut
Graph Cut
Previous Approaches
A little bit more detailed1. Represent image as graph (weights ~ 1/energy)2. Energy minimization3. Find minimum cost path & cut it
Introduction | Previous Approaches | The Algorithm | Results | Conclusions 10
Graph Cut
Source (Foreground)
Sink (Background)
∑= )( weightscutCost
The Algorithm
GrabCut: extension of Graph Cut• iterative optimization• incomplete labelling• simplified user interaction• border matting
Introduction | Previous Approaches | The Algorithm | Results | Conclusions 11
Novelties
Behind the Scenes
Given
Introduction | Previous Approaches | The Algorithm | Results | Conclusions 12
Model
GMM array
GMM componentsinitial alpha matte
initial trimapcolor array
N pixels),...,( 1 Nzzz =
K}{1,...,),,...,,...,( n1 ∈= kkkkk Nn
},,{ FUB TTTT =
}1,0{),,...,( 1 ∈= αααα N
Kff ,...,1 Kbb ,...,1
fromstart
afterinit
Behind the Scenes
Wanted
Energy functionU: fit of to given GMM (color/region information) V: smoothness term (boundary/edge information)
Introduction | Previous Approaches | The Algorithm | Results | Conclusions 13
Model
final alpha matte
GMM parameters KkkkkΘ ,...,1;1,0)},,(),,(),,({ ==Σ= αααµαπ
]1,0[),,...,( 1 ∈= αααα N
αVUE +=
z
Behind the Scenes
Introduction | Previous Approaches | The Algorithm | Results | Conclusions 14
Outline
• Trimap• Alpha matte• GMMs
Initialization
1. Assign GMMs to pixels2. Adapt GMM parameters3. Estimate Segmentation4. Apply border matting
Iterative Minimization
until convergence
User: fg / bg brushGrabCut : • update trimap• step 3+4 once
User editing
Introduction | Previous Approaches | The Algorithm | Results | Conclusions 15
Trimap & Alpha Matte
• Trimap• Alpha matte• GMMs
Initialization
Trimap(background, unknown and foreground region)
},,{ FUB TTTT =
?BT
UT
Behind the Scenes
Alpha Matte(alpha value per pixel)
α
U
Bn Tn
Tnif α
∈∈
=10
1. user indicates background region(hard constraints!)
2.
BT
BUF TTT == ,ø
Introduction | Previous Approaches | The Algorithm | Results | Conclusions 16
Trimap & Alpha Matte
• Trimap• Alpha matte• GMMs
Initialization
Trimap(background, unknown and foreground region)
},,{ FUB TTTT =
Behind the Scenes
Alpha Matte(alpha value per pixel)
α
U
Bn Tn
Tnif α
∈∈
=10
1. user indicates background region(hard constraints!)
2.
BT
BUF TTT == ,ø
Introduction | Previous Approaches | The Algorithm | Results | Conclusions
Gaussian Mixture Models
• Trimap• Alpha matte• GMMs
Initialization
Behind the Scenes
Gaussian Mixture Model (GMM)• approximation of color distribution• weighted sum of K gaussians• parameters: weights , means , covariances• in color space
µ Σπ1D
1=α1. fit bg GMM to colors of pixels with 2. fit fg GMM to colors of pixels with
0=α
In GrabCut• K = 5• 1 foreground GMM, 1 background GMM
17
Introduction | Previous Approaches | The Algorithm | Results | Conclusions 18
Energy function revisitedBehind the Scenes
1. Assign GMMs to pixels2. Adapt GMM parameters3. Estimate Segmentation4. Apply border matting
Iterative Minimization Energy function
U: fit of to given GMM (region information) V: smoothness term (boundary/edge information)
α z
00 1
GMM
U
),(),,,(),,,( zVzΘkUzΘkE ααα +=
∑∈
−−≠=
Cnm
zz
mn
nm
eV),( differencecolor low penalize
)2
||||( 2
2
][ σααγ
)log(GMMU −=
the better GMMs, the lower U neighboring pixels at segmentation boundary with similar color get punished
Introduction | Previous Approaches | The Algorithm | Results | Conclusions 19
ComputationBehind the Scenes
Iterative Minimizationwhile (!converged) {
1. Fill : for each pixel, find best GMM component2. Find best GMM parameters3. Perform Graph Cut
1. Minimize energy
2. Cut adjust }
1. Assign GMMs to pixels2. Adapt GMM parameters3. Estimate Segmentation4. Apply border matting
Iterative Minimization
k
α),,,(minmin
}:{zΘkE
kTn Un
αα ∈
),( kΘ α
Introduction | Previous Approaches | The Algorithm | Results | Conclusions 20
DemoBehind the Scenes
1. Assign GMMs to pixels2. Adapt GMM parameters3. Estimate Segmentation4. Apply border matting
Iterative Minimization
1 2 3 4
Result Energy
Introduction | Previous Approaches | The Algorithm | Results | Conclusions 21
DemoBehind the Scenes
1. Assign GMMs to pixels2. Adapt GMM parameters3. Estimate Segmentation4. Apply border matting
Iterative Minimization
1 2 3 4
Result Energy
Introduction | Previous Approaches | The Algorithm | Results | Conclusions 22
DemoBehind the Scenes
1. Assign GMMs to pixels2. Adapt GMM parameters3. Estimate Segmentation4. Apply border matting
Iterative Minimization
1 2 3 4
Result Energy
Introduction | Previous Approaches | The Algorithm | Results | Conclusions 23
DemoBehind the Scenes
1. Assign GMMs to pixels2. Adapt GMM parameters3. Estimate Segmentation4. Apply border matting
Iterative Minimization
1 2 3 4
Result Energy
Introduction | Previous Approaches | The Algorithm | Results | Conclusions 24
DemoBehind the Scenes
1. Assign GMMs to pixels2. Adapt GMM parameters3. Estimate Segmentation4. Apply border matting
Iterative Minimization
1 2 3 4
Result Energy
Introduction | Previous Approaches | The Algorithm | Results | Conclusions 25
Border Matting
1. Assign GMMs to pixels2. Adapt GMM parameters3. Estimate Segmentation4. Apply border matting
Iterative Minimization
Behind the Scenesα
1
bg fg0
-w w
hard segmentation soft segmentation
Border Matting
α1
0bg fg-w w
Introduction | Previous Approaches | The Algorithm | Results | Conclusions 26
Border Matting
1. Assign GMMs to pixels2. Adapt GMM parameters3. Estimate Segmentation4. Apply border matting
Iterative Minimization
:pixel UT∈∀
∆ σ
α1
bg fg0
∆
σ
-w w
Behind the Scenes
Border Matting1. obtain C from hard segmentation, define w (here w=4)2. assign t(n)3. go along C, find best transition for every t
(i.e. find and by energy minimization)
Avoid color bleeding1. estimate foreground color2. get closest "matching" color from foreground neighborhood
Introduction | Previous Approaches | The Algorithm | Results | Conclusions 27
Border Matting
1. Assign GMMs to pixels2. Adapt GMM parameters3. Estimate Segmentation4. Apply border matting
Iterative Minimization
:pixel UT∈∀
∆ σ
Behind the Scenes
Border Matting1. obtain C from hard segmentation, define w (here w=4)2. assign t(n)3. go along C, find best transition for every t
(i.e. find and by energy minimization)
Avoid color bleeding1. estimate foreground color2. get closest "matching" color from foreground neighborhood
Introduction | Previous Approaches | The Algorithm | Results | Conclusions 28
User EditingBehind the Scenes
User (fore- / background brush)adjust hard constraints GrabCut• update trimap• estimate new segmentation (& apply border matting)
User: fg / bg brushGrabCut : • update trimap• step 3+4 once
User editing
Results
GrabCut• Target rectangle: 450 x 300 pixels• 2.5 GHz CPU, 512 MB RAM• Initial segmentation: 0.9 sec • after each brush stroke: 0.12 sec
GrabCut vs. Graph Cut• quality: perform comparably• time: Graph Cut probably faster• GrabCut fewer user interactions
Introduction | Previous Approaches | The Algorithm | Results | Conclusions 29
Performance
Results
Problems• camouflage & low contrast• thin structures• inadequate bg representation• several objects
Introduction | Previous Approaches | The Algorithm | Results | Conclusions 30
Problems
Results
Problems• camouflage & low contrast• thin structures• inadequate bg representation• several objects
Introduction | Previous Approaches | The Algorithm | Results | Conclusions 31
Problems
Results
Problems• camouflage & low contrast• thin structures• inadequate bg representation• several objects
Introduction | Previous Approaches | The Algorithm | Results | Conclusions 32
Problems
Results
Problems• camouflage & low contrast• thin structures• inadequate bg representation• several objects
Introduction | Previous Approaches | The Algorithm | Results | Conclusions 33
Problems
Conclusions
The Algorithm• fast• good results in many cases• nice ideas
The Paper• claims proven (Gimp-plugin by Matthew Marsh, MS Expression)• rather minimalist explanations
Introduction | Previous Approaches | The Algorithm | Results | Conclusions 34
My 2 Cents
Future Work• 3D support• video support • combination of GrabCut with
image completion
35
Future WorkConclusions
GCGC
IC
Introduction | Previous Approaches | The Algorithm | Results | Conclusions
Discussion
Introduction | Previous Approaches | The Algorithm | Results | Conclusions 36
Questions? Ideas?