New Learning-Based Contour Detection & Contour-Based Object...

1Learning-Based Contour Detection & Contour-Based Object DetectionLearning-based Contour Detection & Contour-based Object Detection

Iasonas Kokkinos

21 January, 2011Visual Geometry Group, Oxford

Galen GroupINRIA-Saclay

Department of Applied MathematicsEcole Centrale de Paris

2Learning-Based Contour Detection & Contour-Based Object Detection

Talk outlineBoundary Detection (35’)

Logistic regression and Anyboost

F-measure Boosting

MIL and boundary detection

Monte Carlo approximations for large-scale datasets

Object Detection (15’)


Appearance descriptors and boundary detection

Coarse-to-fine inference (parsing)

Model learning


Image ContoursObject/Surface Boundaries (edges)


Image ContoursSymmetry axes (ridges/valleys)


A biref anlaogy wtih txet

Waht mttares is waht hppaens on wrod bandouries

Mocpera iwht htsi

(compare with this)

Concrete evidence that our visual system employs boundary detection

Contour-based approaches: shape matching, segmentation, recognition,..


How can we detect boundaries?Filtering approaches

Canny (1984), Morrone and Owens (1987), Perona and Malik (1991),..

Scale-Space approaches

Tony Lindeberg `Edge Detection and Ridge Detection with Automatic Scale Selection.’,

IJCV, 30(2), 117-156, (1998)

Witkin, A. P. "Scale-space filtering", IJCAI (1983)

Variational approaches

V. Caselles, R. Kimmel, G. Sapiro: Geodesic Active Contours. IJCV22(1): 61-79 (1997)

K. Siddiqi, Y. Lauzière, A. Tannenbaum, S. Zucker: Area and length minimizing flows

for shape segmentation. IEEE TIP 7(3): 433-443 (1998)

Gestalt-based approaches

Agnès Desolneux, Lionel Moisan, Jean-Michel Morel: Meaningful

Alignments. International Journal of Computer Vision 40(1): 7-23 (2000)

M. Kass, A. Witkin and D. Terzopoulos, `Snakes: Active Contour Models’, ICCV (1987)


Learning-based approachesBoundary or non-boundary?

Use human-annotated segmentations

D. Martin, C. Fowlkes, J. Malik. "Learning to Detect Natural Image Boundaries Using Local Brightness, Color and Texture

Cues", IEEE PAMI, 2004

S. Konishi, A.Yuille, J. Coughlan, S.C. Zhu, “Statistical Edge Detection: Learning and Evaluating Edge Cues”, IEEE PAMI,

2003

Use human-annotated segmentations


Progress during the last 40 years

Canny+ Hysteresis

Berkeley PB, ‘04

Berkeley gPb, ‘08

Humans

Prewitt, 1965


θr

(x,y)

A closer look into gPb: featuresLocal features (Pb, 2004) Global features (gPb, 2008)

N-Cuts eigenvectors

In specific:


A closer look into gPb: classifierLogistic regression




F-measure Boosting







Model learning


Wanted: `simple’ that `works well’ on

Learning

Given: Training set of feature-label pairs

`simple’: quantified by VC dimension, curvature,…

`works well’: quantified by loss criterion


Logistic regression

Linear function:

Log-likelihood of training pair:

Loss function:

Optimization: Newton-Raphson (IRLS)


At each round, add optimal pair

Anyboost

Additive form:

See training cost as function of

Steepest descent direction:

Find `closest’ to

Adaboost: exponential loss

sign weight


Side-by-side

AnyboostLogistic regression

� Additive� Linear

� Summands: features � Summands: weak learners� Summands: features

� fixed

� Summands: weak learners

� added `on the fly’

� Cost: minus label log likelihood � Cost: exponential loss (Adaboost)

� : Coordinate descent� : Newton-Raphson

Connections: M. Collins, R. Schapire, Y. Singer `Logistic Regression, AdaBoost and Bregman Distances’ COLT (2000)


A compact combination

� Additive

� (linear part)

Goal: quick classification, using small (e.g. ) feature set.

� Remaining summands: weak learners (nonlinearities)

� Cost?

� : Newton-Raphson, at each iteration

� Slower, but off-line



Logistic regression and Boosting, Anyboost

F-measure Boosting







Model learning


Classifier

Loss

Cost function for training

Training set

additiveadditive

- but also potentially non-convex (local optimality)

- potentially better suited for the problem

non-additive: F-measure, Area Under Curve (AUC),…

M. Ranjbar, G. Mori and Y. Wang `Optimizing Complex Loss Functions in Structured Prediction’ ECCV, 2010

T. Joachims, `A Support Vector Method for Multivariate Performance Measures’, ICML, 2005

M. Jansche, `Maximum Expected F-Measure Training Of Logistic Regression Models’, EMNLP, 2005


F-measure

no reward for true negative decisions

Predicted label

Goal: deal with unbalanced datasets (many negative)

F-measure: geometric mean of precision and recall

false alarmstrue positives misses

precision recall


F-measure approximation

predicted label

differentiable approximation

approximate F-measure

M. Jansche, ‘Maximum Expected F-Measure Training Of Logistic Regression Models’, EMNLP, 2005


function of responses

Anyboost

F-measure optimization via Anyboost

Previous iteration

Loss

Anyboost

Newton-Raphson for coefficients: Jansche’s paper




F-measure Boosting







Model learning


mom’s keychain

Sneaking into the fun room

dad’s keychaingrandma’s keychain

We know that dad cannot enter the fun room, either

Which key should we try?

Slide Credit: B. Babenko/T. Dietterich


Multiple Instance Learning

Typical Learning Multiple Instance Learning

Slide Credit: K. Grauman

Typical Learning Multiple Instance Learning

Positive bag: at least one instance should be positiveNegative bag: no instance should be positive


Problem I: inconsistent orientation information


Problem II: inconsistent location information

rForm bag of image locations/orientations that can`support’ human boundary

Given orientation, location support:

Overall support for boundary at


Anyboost for F-measure boosting (previous section)

function of

Loss


function of

Loss

Anyboost for MIL & F-measure boosting




F-measure Boosting







Model learning


Weak learner selection

In all cases:


Gains so far

0.7

0.8

0.9

1Effect of Training

Pre

cisi

on

0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10.2

0.3

0.4

0.5

0.6

Recall

Pre

cisi

on

Global PB, F = 0.697MIL + Full training set, F = 0.704MIL + Full training set + Boosting, F = 0.711




F-measure Boosting







Model learning


Appearance Descriptors

Dense descriptors (DAISY-like)

Multi-scale Gaussian & Gabors, Infinite Impulse Response implementations

Goal: capture context for boundary detection


Discriminative dimensionality reduction

Squeeze discriminative information out of high-dimensional descriptor

LDA: only 1-D (2 class separation)

Large Margin Nearest Neighbors, Neighborhood Component Analysis, ...

iterative, work with


SAVEPCA

PCA vs SAVE (for SIFT features)

34

Projections from SAVE

Sca

le 1

Sca

le 2

Sca

le 3

Projections from PCA


Dense descriptor projections


Overall gains

0.7

0.8

0.9

1Effect of features

Pre

cisi

on

0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10.2

0.3

0.4

0.5

0.6

Recall

Pre

cisi

on

Global PB, F = 0.697MIL + Full training set + Boosting, F = 0.711DoG + LoG + Gabor, F = 0.719Context + DoG + LoG + Gabor, F = 0.726


PComparisons with gPb

Glo

bal

Pb

P>.

5




F-measure Boosting






Model learning



Where can contours be useful?

Recognition?

But contours are highly redundant(only junctions/corners/endings matter)

Attneave 1967

Contours carry most of the image information

But corners/blobs/junctions are hard to group post-hoc


Parts

Object

Hierarchical Compositional Models

Contours

Tokens

Iasonas Kokkinos and Alan Yuille,Inference and Learning with Hierarchical Shape Mode lsInt.l Journal of Computer Vision (IJCV), to appear


View production rules as composition rules

Build a parse tree for the object

Image ObjectParse Tree

Compositional Object Detection


Composition of the `back’ structure

Problem: Too many options!(Combinatorial explosion)


• A* Search

Exit

Cost so far

Cost to go

Heuristic cost

A* for object parsing

• How can we extend A* to parsing?– `The Generalized A* Architecture’, P. Felzenszwalb and D. McAllester, JAIR, 2007

• How can we apply A* parsing to object detection?– ‘HOP: Hierarchical Object Parsing’, I. Kokkinos and A. Yuille, CVPR 2009

43

Entry


Heuristics to Fine Level

Bottom-Up

Top-Down

Coarse-level parsing


Top-Down Guidance: Heuristic, Coarse Level

Fine-level parsing

Bottom-Up Composition, Fine level


• A* Parsing

Coarse Level

Front Part Middle Part Back Part Object Goal

A* vs Knuth’s Lightest Derivation (DP)

• KLD Parsing (only fine level)

Fine Level




F-measure Boosting





Coarse-to-fine inference(parsing)

Model learning



• Input: a set of unregistered images containing object

• Output: a hierarchical model and parsing cost criterion

Learning problem

• Learning pipeline– Contours– Parts– Cost


X S(X)

Deformable model

• Active Appearance Models

• Edges/ridges: throw away appearance variation


sT

M: UpdateE: Deform

Edges & RidgesInput Images

AAM Learning:

Learning deformable models

S

T

AAM Fit

I. Kokkinos and A. Yuille, Unsupervised Learning of Object Deformation Models, ICCV 2007


Recovering object contours


Recovering object contours- ETHZ Shapes


Recovering object parts

Perceptual grouping-based graph

Affinity propagation results


Recovering object parts – ETHZ Shapes


• Goal: learn cost that leads to accurate detection

Parts

Object

Discriminative cost training

– But, no manual annotations to train with– Sole information: Class labels

Parts

Contours

Tokens


Parses as hidden dataMIL-based formulationPositive bag Negative bag

57Learning-Based Contour Detection & Contour-Based Object DetectionImprovements in parsing

Round 2 Round 6

Improvement of cost function: better parsing

P. Gehler and O. Chapelle, Deterministic Annealing for Multiple Instance Learning, AISTATS, 2007


Improvement of cost function: better localization

Round 2 Round 6


Parsing and localization results


Benchmark results


Failure casesFront-end failures

Missing appearance information/poor shape model




F-measure Boosting







Model learning

Conclusions (1’)


Conclusion

It is not the same, indeed

Results: it is not too different

Future work: make it closer

combine contours and appearance descriptors

structured statistical models for shape

integrate segmentation (symmetry)


Thank you

Acknowledgements

M.Bronstein: slide template

New Learning-Based Contour Detection & Contour-Based Object...

Documents

Transcript of New Learning-Based Contour Detection & Contour-Based Object...