Stephen J. Guy

1

PhotomontagePhotomontage PhotomontagePhotomontage

GrabCut – Interactive Foreground Extraction 1

Background Segmentation Intro Graph Cut for Segmentation (Boykov & Jolly

‘01) Copying

Lazy Snapping (Li et al. ’04) GrabCut (Rother et al. ’04)

Other Texture & Video Synthesis (Kwatra et al. ’04)

3

Goal Splitting Image into similar, meaningful(?)

regions E.g. Clustering, Edge Detection, *Graph Cuts*

X. Ren and J. Malik. Learning a classification model for segmentation.ICCV 2003.

“SuperPixels” GrabCutRother et al. 2004Rother et al. 2004

4

A – Proposed Segmentation

E(A) – Overall Energy

R(A) – Degree to which pixels fits model

B(A) – Degree to which the cuts breaks up similar pixels

- Balance A() and B()

• Goal: Find Segmentation, A, which minimizes E(A) 5

Image As Graph Each pixel is a node Create links between each pixel More nodes/links possible (e.g. foreground node)

Graph Cuts Assign energy for each link based on similarity Break the links with least energy, but Require each pixel is still connected to exactly one

foreground or background node Benefits

No Topological constraints on regions (can be concave, unconnected, etc.)

Fast (polynomial time) solution – Ford-Fulkerson Algorithm

6

7

Pixel links based on color/intensity similarities

Source/Target links based on histogram models of fore/background

8

Effect of Recall:

9

10

Main Steps:1. Mark strokes as foreground & background2. Perform Boykov & Jolly style graphcut3. *Edit Boundary*

Familiar Formulation Minimize Gibbs Energy:

E1(X) – Cluster (K-means) known foreground/background colors to find representative colors

E2(X) – Similar to Boykov & Jolly 11

Too Slow to allow for real-time refinement Solution?

Superpixels!

12

13

Graph Cut is only step 1 Allow user to edit boundary directly Formulate as Graph Cut Allow user to guide boundary regions w/

brush

14

15

Compared Lazy Snapping to Photoshop

16

Results Easy of Use – Lazy Snapping 20% less mistakes Speed – Lazy Snapping 60% less time Accuracy – Lazy Snapping 60% less pixels

wrong Feedback

“Almost Magic” “Much Easier”

Suggestions Get rid of 2 step process Make it easy to switch between graph cut

strokes and boundary editing17

18

GrabCutGrabCut Interactive Foreground Extraction Interactive Foreground Extraction

using Iterated Graph Cutsusing Iterated Graph Cuts

Carsten RotherCarsten RotherVladimir Kolmogorov Vladimir Kolmogorov

Andrew BlakeAndrew Blake

Microsoft Research Cambridge-UKMicrosoft Research Cambridge-UK

GrabCutGrabCut Interactive Foreground Extraction Interactive Foreground Extraction

using Iterated Graph Cutsusing Iterated Graph Cuts

Carsten RotherCarsten RotherVladimir Kolmogorov Vladimir Kolmogorov

Andrew BlakeAndrew Blake

Microsoft Research Cambridge-UKMicrosoft Research Cambridge-UK

New model of foreground & background pixels:GMM vs. K-mean colors vs.

Intensity Histogram More automation:

No need to specify foregroundIterative Graph Cuts

20

C:\Documents and Settings\sjguy\Desktop\CompPhoto\GrabCut.avi

What GrabCut doesWhat GrabCut does What GrabCut doesWhat GrabCut does

User Input

Result

Magic Wand (198?)

Intelligent ScissorsMortensen and Barrett (1995)

GrabCut

Regions Boundary Regions & Boundary

GrabCut – Interactive Foreground Extraction 3

1. Assign Selection box or Lasso as background

2. Assign other pixels as unknown3. Learn Model of Fore/Background4. Assign Link Energy, GraphCut (Boykov &

Jolly)5. Use GraphCut results to assign

Fore/Background6. Goto Step 37. Allow user to add cleanup strokes

23

Graph CutsGraph Cuts Boykov and Jolly (2001)Boykov and Jolly (2001)

Graph CutsGraph Cuts Boykov and Jolly (2001)Boykov and Jolly (2001)

GrabCut – Interactive Foreground Extraction 5

ImageImage Min CutMin Cut

Cut: separating source and sink; Energy: collection of edges

Min Cut: Global minimal enegry in polynomial time

Foreground Foreground (source)(source)

BackgroundBackground(sink)(sink)

Iterated Graph CutIterated Graph Cut Iterated Graph CutIterated Graph Cut

User Initialisation

K-means for learning

colour distributions

Graph cuts to infer the

segmentation

?

GrabCut – Interactive Foreground Extraction 6

1 2 3 4

Iterated Graph CutsIterated Graph Cuts Iterated Graph CutsIterated Graph Cuts

GrabCut – Interactive Foreground Extraction 7

Energy after each IterationResult

Guaranteed to

converge

Colour ModelColour Model Colour ModelColour Model

Gaussian Mixture Model (typically 5-8 components)

Foreground &Background

Background

Foreground

BackgroundG

GrabCut – Interactive Foreground Extraction 8

R

G

RIterated graph cut

Moderately straightforward Moderately straightforward

examples examples

Moderately straightforward Moderately straightforward

examples examples

… GrabCut completes automatically

GrabCut – Interactive Foreground Extraction 10

Difficult ExamplesDifficult Examples Difficult ExamplesDifficult Examples

Camouflage &

Low ContrastNo telepathyFine structure

Initial Rectangle

InitialResult

GrabCut – Interactive Foreground Extraction 11

Comparison Comparison Comparison Comparison

GrabCutBoykov and Jolly (2001)

Error Rate: 0.72%Error Rate: 1.87%Error Rate: 1.81%Error Rate: 1.32%Error Rate: 1.25%Error Rate: 0.72%

GrabCut – Interactive Foreground Extraction 13

User Input

Result

Border Matting Border Matting Border Matting Border Matting

Hard Segmentation Automatic Trimap Soft Segmentation

GrabCut – Interactive Foreground Extraction 16

Solve

Mean Colour Foreground

Mean ColourBackground

GrabCut – Interactive Foreground Extraction 17

Natural Image MattingNatural Image MattingNatural Image MattingNatural Image Matting

Ruzon and Tomasi (2000): Alpha estimation in natural images

Noisy alpha-profile

Border MattingBorder MattingBorder MattingBorder Matting

GrabCut – Interactive Foreground Extraction 19

1

0

Foreground

Mix

Back-ground

Foreground Mix Background

Fit a smooth alpha-profile with parameters

ResultsResultsResultsResults

GrabCut – Interactive Foreground Extraction 21

SummarySummary SummarySummary

Magic Wand (198?)

Intelligent Scissors Mortensen and Barrett (1995)

GrabCutRother et al. (2004)

Graph Cuts Boykov and Jolly (2001)

LazySnappingLi et al. (2004)

GrabCut – Interactive Foreground Extraction 22

Boykov&Jolly Lazy Snapping GrabCut

Foreground & Background Model

Histogram of Intensities

K-means clustering, 64 representative colors

K-mean

Background Source

User Strokes User Strokes Initial Box (Lasso) + User Strokes

Foreground Source

User Strokes User Strokes None or User Strokes

Boundary Term

Similarity Penalty

Similarity Penalty

Similarity Penalty

Segmentation GraphCut Superpixels + GraphCut

Iterative GraphCut

Refinement User Strokes Boundary Editing

User Strokes

36

Texture Synthesis Revisited

37

Copy overlapping patches one at a times Find the seamwhich minimize change in

neighboring pixels Formulate as Graph Cut

38

Extends well into 3D

39