Image Segmentation: beyond Graph Cuts

Image Segmentation:beyond Graph Cuts

Victor LempitskyGraph cut segmentationGraph cut segmentation [Boykov&Jolly 01]:

s.t. the shape is sufficiently tight

1......

pqst[Hammer 64] [Greig, Porteous, Seheult 89]Alternative notation:Image Segmentation beyond Graph CutsExample: Interactive segmentation

s.t. the shape is sufficiently tight

......pqst Integrating local cues ...but getting global solutions Many application scenarios...What if some global cues are also available?Image Segmentation beyond Graph Cuts

objectbackgroundImage segmentation: the problem Prior knowledge (compactness) Low-level cues (e.g. edge cues) High-level knowledge (e.g. Penguin on a rock) User input

Image Segmentation beyond Graph CutsWhy image segmentation?

Image Segmentation beyond Graph CutsImage segmentation: the storySince long Ago: rule-based methods, such as thresholding or region growing (magic wand)

Since 1988 [Kass, Witkin, Terzopoulos]: energy optimization via local curve evolution

Since 2001[Boykov, Jolly]:global energy optimization via graph cuts (st-mincut) Image Segmentation beyond Graph CutsGraph cut segmentation

Pairwise terms: Ising prior Edge cuesUnary terms: Color models User brushes[Boykov and Jolly, 2001] S (object, xp=1)T (background, xp=0)

objectxp=1backgroundxp=0Exponential number of segmentationsIn polynomial time!

Alternative notation:Image Segmentation beyond Graph CutsWhy go beyond?We can express:Edge cues & Ising prior (via Ppq)Brushes (set Fp or Bp to )Color likelihoods (via Fp and Bp)

but

How to segment a car in the image?

How to ensure tightness of the bounding box?Still want non-local and efficient optimization!Part ILempitsky, Blake, Rother, Image Segmentation by Branch-and-Mincut, ECCV 2008Part IILempitsky, Kohli, Rother, Image Segmentation with A Bounding Box Prior, ICCV 2009Image Segmentation beyond Graph CutsImage Segmentation by Branch-and-MincutVictor Lempitsky Andrew Blake Carsten Rother

An example10Fp=0, Bp=1Fp=1, Bp=0Ppqimage from UIUC car dataset

Standard graph cut segmentation energy [Boykov, Jolly 01]:......

pqst[Freedman, Zhang 05], [Ali, Farag, El-Baz 07],... 10[Hammer 64] [Greig, Porteous, Seheult 89]Image Segmentation beyond Graph CutsWe will use the unary terms to express our shape prior knowledge. Since we want to segment a car located in the center of the image, we will take a car mask template, and set the foreground costs Fp to 0 inside the white region and to 1 inside the black region and vice versa for the background terms Bp. Thus, we penalize the background pixels inside the white region and the foreground pixels inside the black region.

Now, if we consider some 0-1 segmentation, and superimpose it on top of the mask, we will see that its unary cost will be the sum of the darkred and lightblue, i.e. The Hamming distance between our segmentation and our car template.

On top of that, we add the pairwise terms inversely proportional to some measure of the local contrast, so that the pairwise cost of our segmentation will be the 10

10

A harder exampleimage from UIUC car dataset

Optimal Ppq

Optimal xminx, Image Segmentation beyond Graph Cuts11Energy optimization

Optimization options:Choose reasonable , solve for x [Freedman, Zhang 05], [Pawan Kumar, Torr, ZissermanObjCut 05], [Ali, Farag, El-Baz 07] ....Alternate between x and (EM) [Rother, Kolmogorov, Blake GrabCut 04], [Bray, Kohli, TorrPoseCut 06], [Kim, Zabih 03]....Optimize continuously [Chan, Vese 01], [Leventon, Grimson, Faugeras 00], [Cremers, Osher, Soatto 06], [Wang, Staib 98]...Exhaustive searchconstantunary potentials (costs)pairwise potentials (costs) 0 global parameter ( 0) ....shape priors, color distribution/intensity priors (Chan-Vese, GrabCut)...Image Segmentation beyond Graph CutsOur approach

[Gavrila, Philomin 99], [Lampert, Blaschko, Hofman 08], [Cremers, Schmidt, Barthel 08]along x dimensionalong dimension Extremely large, structured domain Specific graph cut function Low-dimensional (discretized) domain Function of the general formBranch-and-boundMincutBranch-and-MincutImage Segmentation beyond Graph Cuts0Search tree

0Image Segmentation beyond Graph CutsBounding the energy: an example

Fp=1, Bp=0Fp=0, Bp=1

1

2

Fp=0, Bp=0min

Image Segmentation beyond Graph Cuts

The lower bound

pqstComputable with mincut!

Image Segmentation beyond Graph CutsLower bound Monotonic increase towards leaves Tightness at leaf nodes:

Image Segmentation beyond Graph CutsLower bound: example

precomputedcomputed at runtimeImage Segmentation beyond Graph CutsBranch-and-BoundStandard best-first branch-and-bound search:additional speed-up from reusing maxflow computations [Kohli,Torr 05]Small fraction of nodes is visitedlowest lower boundABC

Image Segmentation beyond Graph CutsResults: shape prior

30,000,000 shapes

Exhaustive search: 30,000,000 mincutsBranch-and-Mincut: 12,000 mincutsSpeed-up: 2500 times(30 seconds per 312x272 image)Image Segmentation beyond Graph CutsResults: shape priorLeft ventricle epicardium tracking (work in progress)Our segmentationShape prior from other sequences5,200,000 templates20 seconds per frameSpeed-up 1150Data courtesy: Dr Harald Becher, Department of Cardiovascular Medicine, University of Oxford

Original sequence

No shape priorImage Segmentation beyond Graph CutsResult: shape prior

Can add feature-based detector hereUIUC car datasetImage Segmentation beyond Graph CutsResults: Discrete Chan-Vese functional

Global minima of the discrete Chan-Vese functional:Chan-Vese functional [Chan, Vese 01]: [0;255]x[0;255]: quad-tree clusteringSpeed-up 28-58 timesc_pcfcb

Image Segmentation beyond Graph CutsPerformance

Sample Chan-Vese problem:

Image Segmentation beyond Graph CutsResults: GrabCut

E = -618E = -624 (speed-up 481)E = -628E = -593E = -584 (speed-up 141)E = -607 corresponds to color mixtures [Rother, Kolmogorov, Blake GrabCut 04] uses EM-like search Branch-and-Mincut searches over 65,536 starting pointsImage Segmentation beyond Graph CutsConclusion

good energy to integrate low-level and high-level knowledge in segmentation.

Branch-and-Mincut framework can find its global optimum efficiently in many cases

Ongoing work: Branch-and-X algorithms

C++ code at http://research.microsoft.com/~victlem/Branch-and-boundMincutDynamic ProgrammingImage Segmentation beyond Graph CutsMulti-label Augmented MRFs

Image Segmentation beyond Graph CutsAn experiment

22 pointsBioID dataset: 1520 faces, 800 for training, 720 for testing

FGNet annotations byDavid Cristinacce and Kola BabalolaImage Segmentation beyond Graph Cuts

Pictorial structure MRF

Tree-structured MRF 22 nodes Label space all image locations[Felzenszwalb Huttenlocker 05]

Unaries for eye cornerUnaries for the chin

Image Segmentation beyond Graph CutsBranch-and-DP0

10,000 configurationsScales/rotations/deformationsMessages are still cheap - O(n)(distance transforms + van Herk-Gil-Werman algorithm)1 configurationMessages are cheap - O(n)(distance transforms)Branch-and-DP[Felzenszwalb Huttenlocker 05]

Image Segmentation beyond Graph CutsResults Space size 10,000, average speed-up 11.5* (3 minutes for fitting)10,000 templates (mean error 2.8)1 template (mean error 4.2)

Image Segmentation beyond Graph CutsImage Segmentation with A Bounding Box PriorVictor Lempitsky Pushmeet Kohli Carsten RotherToby Sharp

ICCV 2009

Magnetic LassoMortensen & Barret 95GrabCutRother, Kolmogorov & Blake 04* Globally optimal* user intensiveMotivation

Interactive graph cutBoykov & Jolly 01* Globally optimal* user friendly* NP hard (global color model)* very user friendly

Image Segmentation beyond Graph Cuts33Basic idea is to combine the information which was used in the 2 well known tools: MW and IS.MS select a region of similar colour according to your input. Fat pen and the boundary snapps to high contrast edges. Grabcut combines it. And this allows us to simplify the user interface considerably by Graph cut systemsGraph cut segmentation [Boykov&Jolly 01] integrates cues and input via:

s.t. the shape is sufficiently tight

......pqstGMMRF&GrabCut [Blake et al. 04, Rother et al.05] adds the Gaussian mixture fitting idea + reiterationImage Segmentation beyond Graph CutsWhen it is not right straight away

Solutions:More interactionHigh-level semantic knowledgeOr just look at the user input more attentively

Tightness constraint would help!Image Segmentation beyond Graph CutsProblem FormulationGraph cut segmentation [Boykov&Jolly 01] integrates cues and input via:

s.t. the shape is sufficiently tight

Minimize subject to shape is sufficiently tight?Image Segmentation beyond Graph CutsCrossing pathsUpper marginLower marginMiddle boxcrossing pathcrossing pathLeft marginRight marginImage Segmentation beyond Graph CutsShape tightnessDefinition: the shape is tight if it intersects all crossing pathsbounding boxmiddle boxImage Segmentation beyond Graph Cuts

Shape tightnessCorollary: the shape is tight if and only if the shape has a connected component touching all 4 sides of the middle box

Tight:Not tight:Image Segmentation beyond Graph CutsIncorporating Shape Tightness

C1C2all 4-connected crossing pathsbounding boxmiddle boxImage Segmentation beyond Graph CutsIncorporating Shape Tightness

Problems:Its integer (hence non-convex)It has combinatorial number of constraintsRelax![]Related work:K. Kolev, D. Cremers: Integration of Multiview Stereo and Silhouettes via Convex Functionals on Convex Domains. ECCV 2008Trivial to convert to an LP now!Image Segmentation beyond Graph Cuts\\\\Solving the Linear Relaxation

100.25 Cannot enforce all constraints Dijkstra can check if all constraints are satisfied Dijkstra can find the most violated constraint Can switch the constraints ongraduallySee also:S. Nowozin and C. H. Lampert: Global Connectivity Potentials for Random Field Models. CVPR 2009

bounding boxmiddle boxImage Segmentation beyond Graph Cuts42Solving the Linear Relaxation\\\\\\\Start with no constraintsIterate:Pick a set of violated pathsActivate respective constraintsRerun linear optimizationUntil all constraints satisfiedImage Segmentation beyond Graph Cuts43Solving the Linear Relaxation

Image (in the bounding box)unary termsLP solutionImage Segmentation beyond Graph CutsHow to Round?

[]Previous works e.g. [Kolev&Cremers08]: just threshold at low enough valueOur work: use the problem structure to perform provably better rounding

Image Segmentation beyond Graph Cuts

Pinpointing Algorithm

pinpointing setPinpointing algorithm idea: use the fractional solution to the initial problem to guide the construction of the pinpointing set Image Segmentation beyond Graph CutsPinpointing Algorithm\\\\\\\\ Use dynamic graph cut [Boykov&Jolly01],[Kohli&Torr04]LP solutionRounded solutionbounding boxbounding boxmiddle boxmiddle boxImage Segmentation beyond Graph Cuts

Pinpointing AlgorithmCorollary: pinpointing always gets a lower (or same) energy solution compared to thresholding.

input bounding boxLP solutionthresholding, E = -0.46pinpointing, E = -0.54Image Segmentation beyond Graph CutsPinpointing AlgorithmPinpointing can be used as a fast, standalone heuristics.

Pinpointing unaries,E= -16.5Pinpointing LP solution,E= -17.0LP solutionUnary termsImage Segmentation beyond Graph CutsQuantitative resultsmethodErrorThresholding unaries12.7Just do graph cut (no bounding box prior)6.7Solve LP and threshold5.4Solve LP and do pinpointing5.0Pinpointing based on unaries5.2Relative ordering in terms of the obtained energy is the sameGMMRF [Blake et al. 04]:Fit Gaussian mixtures to get unary termsOptimize the graph cut energy+ a bounding box priorfastfastfastslowslowImage Segmentation beyond Graph CutsRefining color models

GrabCut [Rother et al. 04]: iterateFitting gaussian mixturesOptimizing the graph cut energy+ a bounding box priorThe error rate goes down from 5.1% to 3.7%Image Segmentation beyond Graph CutsConclusion

Global constraints are powerful Approximate optimization is possibleThank you for your attention!Image Segmentation beyond Graph Cuts