S

Constrained Parametric Min-Cuts for

Automatic Object Segmentation

SasiKanth BendapudiYogeshwar Nagaraj

What is a ‘Good Segmentation’?

http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/resources.html

“Geometric context from a single image”, Hoiem et al., ICCV 2005

“Using Multiple Segmentations to Discover Objects and their Extent in Image Collections”, Russel et al., CVPR 2006

“Improving Spatial Support for Objects via Multiple Segmentations”, Malisiewicz & Efros, BMVC 2007

“Towards Unsupervised Whole-Object Segmentation: Combining Automated Matting with Boundary Detection”, Stein & Hebert,

CVPR 2008

The Paper

Figure-Ground segmentation Solve CPMC by minimizing the objective function

using various seeds and parameters Reject redundancies and obvious negatives based

on segment energies and similarities Learn the characteristics of a ‘Figure’ segment to

qualitatively assess the remaining segments

Objective Function

Objective Function

Objective Function

Synthetic Example

Synthetic Example

Initialization

Foreground Regular 5x5 grid geometry Centroids of large N-Cuts regions Centroids of superpixels closest to grid positions

Background Full image boundary Horizontal boundaries Vertical boundaries All boundaries excluding the bottom one

Performance broadly invariant to different initializations

Fast RejectionLarge set of initial segmentations (~5500)

High Energy Low Energy

~2000 segments with the lowest energy

Cluster segments based on spatial overlap

Lowest energy member of each cluster (~154)

Segment Ranking

Model data using a host of features Graph partition properties Region properties Gestalt properties

Train a regressor with the largest overlap ground-truth segment using Random Forests

Diversify similar rankings using Maximal Marginal Relevance (MMR)

Graph Partition Properties

Cut – Sum of affinities along segment boundary Ratio Cut – Sum along boundary divided by the number Normalized Cut – Sum of cut and affinity in foreground

and background Unbalanced N-cut – N-cut divided by foreground affinity Thresholded boundary fraction of a cut

Region Properties

Area Perimeter Relative Centroid Bounding Box properties Fitting Ellipse properties Eccentricity Orientation

Convex Area

Euler Number

Diameter of Circle with the same area of the segment

Percentage of bounding box covered

Absolute distance to the center of the image

Gestalt Properties

Inter-region texton similarity

Intra-region texton similarity

Inter-region brightness similarity

Intra-region brightness similarity

Inter-region contour energy

Intra-region contour energy

Curvilinear continuity

Convexity – Ratio of foreground area to convex hull area

Feature Importance

Feature Importance

Feature Importance

What has been modeled?

Databases

Weizmann database F-measure criterion

MSR-Cambridge database & Pascal VOC2009 Segmentation covering

Performance

Test of the algorithm

Berkeley segmentation dataset Complete pool of images collected Ranked using the ranking methodology Top ranks evaluated to test the ranking procedure

How well does the algorithm perform?

Berkeley Database

Rank 269!

Berkeley Database

Rank 142!

Berkeley Database

Rank 98!

Berkeley Database

Compute the Segment Covering score for the top 40 segments of each image in the database

Database Segment Covering Score (Top 40)

BSDS 0.52MSR Cambridge 0.77

Pascal VOC 0.63Database Segment Covering Score

(All segments)BSDS 0.61

MSR Cambridge 0.85Pascal VOC 0.78

Conclusion

Does Constrained Parametric Min-Cuts work well? Yes

Does Fast Rejection work well? Yes

Does Segment Ranking work well? I don’t think so

Interesting follow up

Image Segmentation by Figure-Ground Composition into Maximal Clique, Ion, Carreira, Sminchisescu, ICCV 2011

Interesting follow up

Image Segmentation by Figure-Ground Composition into Maximal Clique, Ion, Carreira, Sminchisescu, ICCV 2011

Obtain pool of FG segmentations from CPMC Define tiling and a probabilistic model for the

same

Represent the probabilistic models using mid-level features

Compute and rank various tilings by implementing discrete searches from each of the nodes

Interesting follow up

Image Segmentation by Figure-Ground Composition into Maximal Clique, Ion, Carreira, Sminchisescu, ICCV 2011

Questions?