Detecting Curved Symmetric Parts using a Deformable Disc ModelTom Sie Ho Lee, University of Toronto Sanja Fidler, TTI Chicago Sven Dickinson, University of Toronto

OverviewMotivationMotivation

Robustness to Robustness to curvaturecurvature• Symmetric parts are often curved• We capture curvature explicitly and recover the

part in one piece

1. Representing symmetric parts

Object part Maximal discs

Superpixel approximation

Results

Deformation-invariant spaceDeformation-invariant space

• Evaluate symmetry in a warped space invariant to bending and tapering deformations

• Determine warp by fitting a deformable ellipse to region

• Extract spatial histogram of boundary edgels

• Extract interior color and texture features

Deformable discsDeformable discs

• Use compact superpixels as deformable disc hypotheses

• Superpixels from different scales compose a single part

2. Deformable Disc Affinity

3. Finding sequences

n

dddtdds

n

AP

n

i iii

n

i ii

1

1 111 1 ),,(),()(cost

Robustness to taperRobustness to taper

• Tapered parts vary in scale along the axis• We allow parts to be composed along the axis

from disc hypotheses of different scales

Symmetric part detection as sequence Symmetric part detection as sequence findingfinding• Good symmetry follows a curvilinear axis• We find high-affinity sequences of disc hypotheses• Optimal sequences are computed using dynamic programming• Our sequence formulation avoids branching clusters

• 81 images of horses

• Manually annotated symmetric parts as groundtruth regions [Levinshtein et al.]

• Count a hit when IoU-overlap > 40% between groundtruth and detected regions

Weizmann Horse Database Weizmann Horse Database (WHD)(WHD)

• Symmetric part detector trained on horse images generalizes to diverse objects

• Symmetry is a powerful and ubiquitous shape regularity

ConclusionsConclusions

• Source of images of diverse objects on cluttered backgrounds

• We manually annotated symmetric parts on 36 selected images

Berkeley Berkeley Segmentation Segmentation Database (BSDS)Database (BSDS)

Qualitative resultsQualitative results

• Perform best-first search using priority queue of candidate sequences

• Dequeue candidate sequences and consider possible extensions

• Repeat minimization to find multiple symmetric parts

Finding the optimal sequenceFinding the optimal sequence

Cost of a disc sequenceCost of a disc sequence

• Organize deformable disc hypotheses into graph

• Place edges between adjacent or overlapping discs

• Find disc sequences in the graph with high affinity

Disc hypothesis graphDisc hypothesis graph

Overview of affinityOverview of affinity

• Define affinity between adjacent disc hypotheses

• High affinity reflects non-accidental symmetry

• Adjacent discs occupy a region r on which to extract features

• Train affinity on region symmetry features

Affinity trainingAffinity training

• Learn to map a region r to its affinity σ(r)

• Generate positive and negative training regions from annotated dataset

• Extract features on each training region

• Fit logistic regressor σ(r) to training examples

• Recovering an object’s generic part structure is a key step in bottom-up object categorization

• Symmetry has formed the basis of many 2D and 3D generic part representations, e.g. skeletons, shock graphs, generalized cylinders, geons

• Our goal is to detect 2D symmetric parts in a cluttered image

• The medial axis transform [Blum] decomposes a shape into the locus of maximal inscribed discs

• We define a symmetric part as a sequence of deformable discs

• Discs deform to the shape’s boundary while remaining compact

Superpixel Superpixel approximationapproximation

Multi-scale composition

[Levinshtein et al.] [ours][Levinshtein et al.] [ours][Levinshtein et al.]

[ours]

Deformable ellipseDeformable ellipse

1)()( xWxW

)()()(),( tTbBaSpRW

2

1);(min wx

w

N

i ie

Our ellipse is parameterized by bending and tapering, axis scaling, and rigid transformations

Find parameters w that locally minimize non-linear least squares:

• Cost is normalized by number of discs in sequence• Growth term A favors longer sequences

Symmetric parts of objects detected by our method

Related workRelated work

• Classical skeletons [Blum ‘67; Brady ‘84] are inapplicable to cluttered scenes• Filter-based approaches require reliable templates• Contour-based approaches require quadratic grouping complexity• Region-based grouping [Levinshtein et al. ‘09] offers a good alternative• We demonstrate an improvement on [Levinshtein et al.] by capturing more

shape variability and applying an optimal algorithm

• cost(P) can be globally minimized using dynamic programming

• We use the algorithm of [Felzenszwalb & McAllester ‘06] to compute the global minimum P*

warp W spatial histogram

MethodAP on BSDS

AP on WHD

baseline [Levinshtein et al.]

10.9% 8.0%

baseline, with dynamic programming

16.2% 10.4%

ours, without ternary cost

21.2% 12.9%

ours 22.3% 14.2%

BSDS WHD

boundary edgels

• Score a sequence P = (d0, …, dn) in terms of local affinities σ(di-1,di) and σ(di-1,di,di+1)

• Affinities favor local grouping of adjacent discs

• Ternary affinities favor curvilinear axis (smoothness)

• Convert affinities into binary costs {si-1,i = 1 - σ(di-1,di)} and ternary costs {ti-1,i,i+1 = 1 - σ(di-1,di,di+1)}

Ellipse fitted to a region hypothesis

disc hypothesis graph

Candidate sequence extensionsunder consideration