Associative Hierarchical CRFs for Object Class Image Segmentation Ľubor Ladický 1 1 Oxford Brookes...

Associative Hierarchical CRFs for Object Class Image

Segmentation

Ľubor Ladický1

1Oxford Brookes University 2Microsoft Research Cambridge

Based on the work done with :

Chris Russell1, Pushmeet Kohli1,2 , Paul Sturgess1 ,Karteek Alahari1 , Philip H.S. Torr1

Overview

Object-class Segmentation problem

Datasets

MSRC dataset

VOC2009 dataset

CamVid dataset

Standard approaches

Pairwise CRF over Pixels

Pairwise CRF over Segments

Detection-driven Segmentation

Model

Robust-PN model

Associative Hierarchical CRF

Detectors in AHCRF

Label co-occurrence in CRFs

Potential training

Inference

Results

Discussion

General overview Our approach


MSRC Image Our Result

Aims to assign a label for each pixel of an image

Classifier trained on the training set

Performance evaluated on never seen test set

Overview


Datasets

MSRC dataset

VOC2009 dataset

CamVid dataset

Standard approaches




Model

Robust-PN model


Detectors in AHCRF


Potential training

Inference

Results

Discussion


Datasets

MSRC Dataset591 images 320 x 213

21 classes

MSRC Image MSRC Image MSRC ImageGround Truth Ground Truth Ground Truth

Datasets

VOC2009 Dataset1499 training + 750 test images 500 x 375

20 foreground classes + 1 background class

VOC Image VOC Image VOC ImageGround Truth Ground Truth Ground Truth

Datasets

CamVid Dataset367 training + 233 test video frames 960 x 720

11 classes

CamVid Image CamVid Image CamVid ImageGround Truth Ground Truth Ground Truth

Overview


Datasets

MSRC dataset

VOC2009 dataset

CamVid dataset

Standard approaches




Model

Robust-PN model


Detectors in AHCRF


Potential training

Inference

Results

Discussion



CRF

Shotton ECCV06


No quantization errors

Lacks long range interactions

Results oversmoothed

Better performance on CamVid & MSRC datasets


Yang et al. CVPR07, Batra et al. CVPR08,

Shi, Malik PAMI2000, Comaniciu, Meer PAMI2002, Felzenschwalb, Huttenlocher, IJCV2004


Allows long range interactions

Better performance for VOC dataset

Can not recover from incorrect segmentation

Impossible to obtain perfect unsupervised segmentation


Larlus et al. CVPR08, Felzenszwalb et al. CVPR08, Vedaldi et al. ICCV09

GrabCut

Detector


The best performance for VOC dataset

Can not be used for background classes

Can not recover from incorrect detection

Overview


Datasets

MSRC dataset

VOC2009 dataset

CamVid dataset

Standard approaches




Model

Robust-PN model


Detectors in AHCRF


Potential training

Inference

Results

Discussion


Robust PN approach

Kohli, Ladický, Torr CVPR08

Robust PN approach

Segment consistency as a weak constraint

Robust to misleading segmentations

Allows multiple overlapping segmentations

General formulation not used in application

Unary and pairwise potentials only at the pixel level

Robust PN reformulation

Ladický, Russell, Kohli, Torr ICCV09

Old formulation of Robust PN potential

is equivalent to pairwise formulation

where

Robust PN reformulation



Allows unary potentials for region variables

Allows pairwise potentials for region variables

Allows multiple layers and multiple hierarchies

Zhu et al. NIPS2008, Lim et al. ICCV2009, Hinton 2002

Analysis of the new model


Let's have one segmentation and potentials only over segment level

Analysis of the new model

Let's have one segmentation and potentials only over segment level

Energy of two pixels from the same clique is symmetric and semi-metric

Minimum will be segment-consistent

The cost of every segment consistent labelling is the same as the cost of the pairwise CRF labelling over segments

Equivalent to pairwise CRF over segmentsLadický, Russell, Kohli, Torr ICCV09


Merges information over multiple scales

Allows multiple hierarchies

Allows long range interactions

Easy to learn weights

Interlayer connection limited(?) to associative relationship


Comparison to other methods


Robust PN Segment CRF Hierarchical CRF

Detectors inAssociative Hierarchical CRFs

Ladický, Alahari, Sturgess, Russell, Torr CVPR10 (submitted)

GrabCutDetector

Detector potential takes the form

where

Detectors inAssociative Hierarchical CRFs


Special 2-label case of Robust PN

Allows multiple overlapping detections

Can recover from incorrect detection

Same inference methods may be applied as for Associative Hierarchical CRFs

We can distinguish between different instances of objects in the final result


Ladický, Russell, Kohli, Torr CVPR10 (submitted)

Possible labelling ?

Why not ?



Our requirements

Global Energy function – no hard decisions

Invariance to number of pixels taking given label

Efficiency – should be tractable

Standard approaches

Hard decisions in the preprocessing step (Csurka et al. BMVC08)

Unary potential (Torralba et al. ICCV03)

Pairwise potential (Rabinovich et al. ICCV07, CVPR08)

Only solution : EC = C(L(x))

Overview


Datasets

MSRC dataset

VOC2009 dataset

CamVid dataset

Standard approaches




Model

Robust-PN model


Detectors in AHCRF


Potential training

Inference

Results

Discussion


Multi-feature BoostedUnary Pixel Potential

Ladický, Russell, Kohli, Torr ICCV09Sturgess, Alahari, Ladický, Torr BMVC09

Unary likelihoods based on spatial configuration (Shotton et al. ECCV06)

Classifier trained using boosting

Multi-feature BoostedUnary Segment Potential


Classifier trained using boosting

Other Potentials

Ladický, Russell, Kohli, Torr ICCV09Ladický, Russell, Kohli, Torr CVPR10 (submitted)


• Intensity dependent pairwise pixel potential

• Colour EMD-distance dependent pairwise segment potential

• Detector potentials based on off-the-shelf detectors (Felzenszwalb et al. CVPR08, Vedaldi et al. ICCV09)

• Generatively trained Co-occurence potential

Overview


Datasets

MSRC dataset

VOC2009 dataset

CamVid dataset

Standard approaches




Model

Robust-PN model


Detectors in AHCRF


Potential training

Inference

Results

Discussion


Inference over Hierarchical CRF

Russell, Ladický, Kohli, Torr AISTATS10 (submitted)


Problem is NP-hard

Any message passing algorithm (TRW-S, BP, ..) or ICM can be applied to pairwise model

αβ-swap (potentials must be semi-metric)

Ishikawa construction over (α-F-β transition)

αexpansion (potentials must be metric)

Reparametrization of interlayer connection to metric potential

Ishikawa construction over (α-F-old transition)

For more details read our technical report



Problem is NP-hard

Any message passing algorithm (TRW-S, BP, ..), ICM or LP relaxation can be applied to pairwise model

αβ-swap (potentials must be semi-metric)

Ishikawa construction over (α-F-β transition)







Problem is NP-hard


• αβ-swap (potentials must be semi-metric)

• Ishikawa construction over (α-F-β transition)







Problem is NP-hard


• αβ-swap (potentials must be semi-metric)

• Ishikawa construction over (α-F-β transition)

• α-expansion (potentials must be metric)

• Reparametrization of interlayer connection to metric potential

• Ishikawa construction over (α-F-old transition)


Inference for Co-occurence

Integer programming formulation possible using one variable for each subset of the label set

LP relaxation applicable by relaxing IP program

• αβ-swap possible if the cooccurence cost C(L(x)) is monotonically increasing with respect to L(x)

• α-expansion approximate move possible if the cooccurence cost C(L(x)) is monotonically increasing with respect to L(x)


Overview


Datasets

MSRC dataset

VOC2009 dataset

CamVid dataset

Standard approaches




Model

Robust-PN model


Detectors in AHCRF


Potential training

Inference

Results

Discussion


VOC2009 Results

MSRC Results

MSRC Results

Qualitative comparison of results without and with co-occurence


MSRC Image MSRC ImageWithout CO With CO With COWithout CO

CamVid Results

Sturgess, Alahari Ladický, Torr BMVC09Ladický, Alahari, Sturgess, Russell, Torr CVPR10 (submitted)

Our Result

CamVid Image

GroundTruth

Brostow et al.

Qualitative comparisons

Quantitative comparison of methods on VOC2009 dataset

Quantitative comparison of methods on MSRC dataset

Quantitative comparison of methods on CamVid dataset

Take home message

Take home message

Use our model

Take home message

Use our model

• Use your favourite potentials

Take home message

Use our model


• Use your friend's favourite potentials

Take home message

Use our model



• Use your friend's friend's favourite potentials

Take home message

Use our model




• Vision solved

Take home message

Use our model




• Vision solved (..almost)

Thank you

Questions?

Associative Hierarchical CRFs for Object Class Image Segmentation Ľubor Ladický 1 1 Oxford Brookes...

Documents

Transcript of Associative Hierarchical CRFs for Object Class Image Segmentation Ľubor Ladický 1 1 Oxford Brookes...