Multiple View Object Cosegmentation using Appearance and Stereo Cues

1

Multiple View Object Cosegmentation using Appearance and Stereo Cues

ECCV 2012

2014/3/25 Yichong Bai

Multiple View Object Cosegmentation using Appearance and Stereo Cues

2

Problem

• Segmenting a rigid object in multiple views, using stereo cues• Input• unordered set of images

• Output• Segmentation• (Fine-grained depth map)• Object mask

2014/3/25 Yichong Bai

Multiple View Object Cosegmentation using Appearance and Stereo Cues

3

Overview

• Piecewise planar, layer-based depth map estimation based on stereo• Robust to unreliable stereo matching• Unsupervised• Workflow• Dense piecewise planar depth map, stereo cue, color• Segment object in each image• Fuse segmentation across multiple views

2014/3/25 Yichong Bai

Multiple View Object Cosegmentation using Appearance and Stereo Cues

4

Difference with ours

• Matching separated with segmentation• Stereo matching first• Not for matching

2014/3/25 Yichong Bai

Multiple View Object Cosegmentation using Appearance and Stereo Cues

5

Model

• Notations•

• L: labeling, pixelplanes

2014/3/25 Yichong Bai

Multiple View Object Cosegmentation using Appearance and Stereo Cues

6

Model

• Plane hypotheses• From semi-dense depth map, seed and grow• Depth map needed• Steps

• Fit surfel s_p at pixel p by least square• Fit plane by surfels• Seed and grow on graph where nodes represent pixels with valid surfels and

edges are 5-degree adjacencyLargest clusters are plane hypothesis

2014/3/25 Yichong Bai

Multiple View Object Cosegmentation using Appearance and Stereo Cues

7

Model

• Formulation• Multi-label MRF optimization problem (reference?)• Approximation by Energy Min.•

2014/3/25 Yichong Bai

Multiple View Object Cosegmentation using Appearance and Stereo Cues

8

Model

• Terms• Geometric unary term E^G

•

• Appearance unary term E^A• GMM in Lab color space

• Pairwise term E_{pq}• Contrast sensitive Potts model

2014/3/25 Yichong Bai

Multiple View Object Cosegmentation using Appearance and Stereo Cues

9

Solve

• Initialization• No appearance term

2014/3/25 Yichong Bai

Multiple View Object Cosegmentation using Appearance and Stereo Cues

10

Solve

• Iteration• GMM for appearance

• EM algorithm generating GMM model for surfel

2014/3/25 Yichong Bai

Multiple View Object Cosegmentation using Appearance and Stereo Cues

11

Model

• Surface segmentation• L segmentation

2014/3/25 Yichong Bai

Multiple View Object Cosegmentation using Appearance and Stereo Cues

12

Solve

• Fusion move• Initialize by assigning different regions to each pixel• Fuse coplanar polygons in a greedy way

2014/3/25 Yichong Bai

Multiple View Object Cosegmentation using Appearance and Stereo Cues

13

Model

• Object Segmentation• F: region labeling• Objectiveness

• Object’s pixels are visible in multi-views

2014/3/25 Yichong Bai

Multiple View Object Cosegmentation using Appearance and Stereo Cues

14

Evaluation

• Experiments• Use Grabcut (interactive) as groundtruth• Intersection over union metric

2014/3/25 Yichong Bai

Multiple View Object Cosegmentation using Appearance and Stereo Cues

15

Evaluation

• Results

2014/3/25 Yichong Bai