Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

39
Stephen J. Guy 1

Transcript of Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

Page 1: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

Stephen J. Guy

1

Page 2: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

PhotomontagePhotomontage PhotomontagePhotomontage

GrabCut – Interactive Foreground Extraction 1

Page 3: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

Background Segmentation Intro Graph Cut for Segmentation (Boykov & Jolly

‘01) Copying

Lazy Snapping (Li et al. ’04) GrabCut (Rother et al. ’04)

Other Texture & Video Synthesis (Kwatra et al. ’04)

3

Page 4: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

Goal Splitting Image into similar, meaningful(?)

regions E.g. Clustering, Edge Detection, *Graph Cuts*

X. Ren and J. Malik. Learning a classification model for segmentation.ICCV 2003.

“SuperPixels” GrabCutRother et al. 2004Rother et al. 2004

4

Page 5: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

A – Proposed Segmentation

E(A) – Overall Energy

R(A) – Degree to which pixels fits model

B(A) – Degree to which the cuts breaks up similar pixels

- Balance A() and B()

• Goal: Find Segmentation, A, which minimizes E(A) 5

Page 6: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

Image As Graph Each pixel is a node Create links between each pixel More nodes/links possible (e.g. foreground node)

Graph Cuts Assign energy for each link based on similarity Break the links with least energy, but Require each pixel is still connected to exactly one

foreground or background node Benefits

No Topological constraints on regions (can be concave, unconnected, etc.)

Fast (polynomial time) solution – Ford-Fulkerson Algorithm

6

Page 7: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

7

Page 8: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

Pixel links based on color/intensity similarities

Source/Target links based on histogram models of fore/background

8

Page 9: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

Effect of Recall:

9

Page 10: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

10

Page 11: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

Main Steps:1. Mark strokes as foreground & background2. Perform Boykov & Jolly style graphcut3. *Edit Boundary*

Familiar Formulation Minimize Gibbs Energy:

E1(X) – Cluster (K-means) known foreground/background colors to find representative colors

E2(X) – Similar to Boykov & Jolly 11

Page 12: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

Too Slow to allow for real-time refinement Solution?

Superpixels!

12

Page 13: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

13

Page 14: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

Graph Cut is only step 1 Allow user to edit boundary directly Formulate as Graph Cut Allow user to guide boundary regions w/

brush

14

Page 15: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

15

Page 16: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

Compared Lazy Snapping to Photoshop

16

Page 17: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

Results Easy of Use – Lazy Snapping 20% less mistakes Speed – Lazy Snapping 60% less time Accuracy – Lazy Snapping 60% less pixels

wrong Feedback

“Almost Magic” “Much Easier”

Suggestions Get rid of 2 step process Make it easy to switch between graph cut

strokes and boundary editing17

Page 18: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

18

Page 19: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

GrabCutGrabCut Interactive Foreground Extraction Interactive Foreground Extraction

using Iterated Graph Cutsusing Iterated Graph Cuts

Carsten RotherCarsten RotherVladimir Kolmogorov Vladimir Kolmogorov

Andrew BlakeAndrew Blake

Microsoft Research Cambridge-UKMicrosoft Research Cambridge-UK

GrabCutGrabCut Interactive Foreground Extraction Interactive Foreground Extraction

using Iterated Graph Cutsusing Iterated Graph Cuts

Carsten RotherCarsten RotherVladimir Kolmogorov Vladimir Kolmogorov

Andrew BlakeAndrew Blake

Microsoft Research Cambridge-UKMicrosoft Research Cambridge-UK

Page 20: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

New model of foreground & background pixels:GMM vs. K-mean colors vs.

Intensity Histogram More automation:

No need to specify foregroundIterative Graph Cuts

20

Page 22: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

What GrabCut doesWhat GrabCut does What GrabCut doesWhat GrabCut does

User Input

Result

Magic Wand (198?)

Intelligent ScissorsMortensen and Barrett (1995)

GrabCut

Regions Boundary Regions & Boundary

GrabCut – Interactive Foreground Extraction 3

Page 23: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

1. Assign Selection box or Lasso as background

2. Assign other pixels as unknown3. Learn Model of Fore/Background4. Assign Link Energy, GraphCut (Boykov &

Jolly)5. Use GraphCut results to assign

Fore/Background6. Goto Step 37. Allow user to add cleanup strokes

23

Page 24: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

Graph CutsGraph Cuts Boykov and Jolly (2001)Boykov and Jolly (2001)

Graph CutsGraph Cuts Boykov and Jolly (2001)Boykov and Jolly (2001)

GrabCut – Interactive Foreground Extraction 5

ImageImage Min CutMin Cut

Cut: separating source and sink; Energy: collection of edges

Min Cut: Global minimal enegry in polynomial time

Foreground Foreground (source)(source)

BackgroundBackground(sink)(sink)

Page 25: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

Iterated Graph CutIterated Graph Cut Iterated Graph CutIterated Graph Cut

User Initialisation

K-means for learning

colour distributions

Graph cuts to infer the

segmentation

?

GrabCut – Interactive Foreground Extraction 6

Page 26: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

1 2 3 4

Iterated Graph CutsIterated Graph Cuts Iterated Graph CutsIterated Graph Cuts

GrabCut – Interactive Foreground Extraction 7

Energy after each IterationResult

Guaranteed to

converge

Page 27: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

Colour ModelColour Model Colour ModelColour Model

Gaussian Mixture Model (typically 5-8 components)

Foreground &Background

Background

Foreground

BackgroundG

GrabCut – Interactive Foreground Extraction 8

R

G

RIterated graph cut

Page 28: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

Moderately straightforward Moderately straightforward

examples examples

Moderately straightforward Moderately straightforward

examples examples

… GrabCut completes automatically

GrabCut – Interactive Foreground Extraction 10

Page 29: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

Difficult ExamplesDifficult Examples Difficult ExamplesDifficult Examples

Camouflage &

Low ContrastNo telepathyFine structure

Initial Rectangle

InitialResult

GrabCut – Interactive Foreground Extraction 11

Page 30: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

Comparison Comparison Comparison Comparison

GrabCutBoykov and Jolly (2001)

Error Rate: 0.72%Error Rate: 1.87%Error Rate: 1.81%Error Rate: 1.32%Error Rate: 1.25%Error Rate: 0.72%

GrabCut – Interactive Foreground Extraction 13

User Input

Result

Page 31: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

Border Matting Border Matting Border Matting Border Matting

Hard Segmentation Automatic Trimap Soft Segmentation

GrabCut – Interactive Foreground Extraction 16

Page 32: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

Solve

Mean Colour Foreground

Mean ColourBackground

GrabCut – Interactive Foreground Extraction 17

Natural Image MattingNatural Image MattingNatural Image MattingNatural Image Matting

Ruzon and Tomasi (2000): Alpha estimation in natural images

Page 33: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

Noisy alpha-profile

Border MattingBorder MattingBorder MattingBorder Matting

GrabCut – Interactive Foreground Extraction 19

1

0

Foreground

Mix

Back-ground

Foreground Mix Background

Fit a smooth alpha-profile with parameters

Page 34: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

ResultsResultsResultsResults

GrabCut – Interactive Foreground Extraction 21

Page 35: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

SummarySummary SummarySummary

Magic Wand (198?)

Intelligent Scissors Mortensen and Barrett (1995)

GrabCutRother et al. (2004)

Graph Cuts Boykov and Jolly (2001)

LazySnappingLi et al. (2004)

GrabCut – Interactive Foreground Extraction 22

Page 36: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

Boykov&Jolly Lazy Snapping GrabCut

Foreground & Background Model

Histogram of Intensities

K-means clustering, 64 representative colors

K-mean

Background Source

User Strokes User Strokes Initial Box (Lasso) + User Strokes

Foreground Source

User Strokes User Strokes None or User Strokes

Boundary Term

Similarity Penalty

Similarity Penalty

Similarity Penalty

Segmentation GraphCut Superpixels + GraphCut

Iterative GraphCut

Refinement User Strokes Boundary Editing

User Strokes

36

Page 37: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

Texture Synthesis Revisited

37

Page 38: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

Copy overlapping patches one at a times Find the seamwhich minimize change in

neighboring pixels Formulate as Graph Cut

38

Page 39: Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

Extends well into 3D

39