Global and Efficient Self-Similarity for Object Classification and Detection

CVPR 2010

Thomas Deselaers and Vittorio Ferrari

Conventional Image DescriptorsMeasure direct image properties

gradients

colors

2

Self-Similarity vs Conventional Descriptors

[Shechtman, Irani CVPR 07]

Assumption of conventional image descriptors• There is a direct visual property shared by images of objects of the same class

(e.g. colors, gradients, …).• This property can be used to compare images.

Self-similarity:• Indirect property: geometric layout of repeating patches within an image• More general property

3

Local Self-Similarity Descriptors

4[Shechtman, Irani CVPR 07]

Using Local Self-Similarity Descriptors

Applications: object recognition, image retrieval, action recognition• Ensemble matching [Shechtman CVPR 07]• Nearest neighbor matching [Boiman CVPR 08]• Bag of local self-similarities

[Gehler ICCV09, Vedaldi ICCV09, Hörster ACMM08, Lampert CVPR09, Chatfield ICCV09 WS]

1. Compute LSS descriptors for an image2. Assign the LSS descriptors to a codebook3. Represent the image as a histogram of LSS descriptors

5

Self-Similarity goes Global

Capture long-range self-similarities and their spatial arrangement

6

Self-Similarity goes Global

Capture long-range self-similarities and their spatial arrangement

7

compute self-similarity between all pairs of

pixels

compute self-similarity between all pairs of

pixels

Global Self-Similarity Tensor

4D self-similarity tensor

Note: local self-similarities included8

Problems with the GSS Tensor

• Computation time:• Memory requirement:

Aim: Reduce both9

1111

∼ 80GB ∼ 20h

Outline• Efficient global self-similarity tensor

• Global self-similarity descriptors– Bag of correlation surfaces– Self-similarity hypercubes

• Detection with self-similarity hypercubes– Efficient sliding window– Efficient subwindow search

• Experiments– Global self-similarity better than local self-similarity– Complementary to conventional descriptors– Object detection possible

10

Efficient Global Self-Similarity TensorFind an efficient approximation to

Quantize patches according to codebook

If two patches are assigned to the same prototype, they are similar

Reduces runtime to speedup: 11750

Efficient Global Self-Similarity

Two patches are only similar if they are assigned to the same prototype

Reduces memory to reduction: 12

Patch Prototype CodebooksRemember: Self-similarity encodes image content indirectly

Image-specific codebooks can be smaller than conventional ones

see paper for more generic codebooks and extensive evaluation 13

• Self-similarity hypercubes: now

• Bag of correlation surfaces: only in the paper

Global Self-Similarity DescriptorsSo far:• Compact GSS computed efficiently

Now: • Descriptors that can be used in machine learning classifiers• Fixed dimensionality• Compact representation

14

Self-Similarity Hybercubes

SSH of size

15

SSHs for Detection• Computing SSH naïvely requires operations

• Sliding windows has to evaluate many windows

16

operations

Efficient Computation of SSHs

Compute integral self-similarity tensor:

operations to compute SSHfor an image window

17

∼5000x speedup

160000

can be obtained using 16 lookups in

Efficient Subwindow Search for SSH

18

• Derive an upper bound on the score of a set of windows

• Section 5.2 in our paper

• Similar to [Lampert PAMI09]

Experiments: Object classificationPASCAL 07 objects– 9608 cropped images of objects from PASCAL 07 – 20 classes

Task: Classify each test image into one of 20 classesModel: Linear SVMTrain: train+val Test: test

19

Classification on the PASCAL 07 objects set

+ GSS outperform LSS+ Self-Similarity is truly complementary to conventional descriptors

20

clas

sific

ation

acc

urac

y [%

]

Experiments: Object detectionETHZ Shape Classes– 255 images– 5 classes (apple logos, bottles, giraffes, mugs, swans)

Task: Detect objects in imagesDetector: Linear SVM, sliding windows

21

e.g. [Ferrari CVPR07, Maji CVPR09]

Detection Results

+ SSH outperforms BOLSS+ it is possible to use GSS for detection with good results

22

BoLSS SSH

apple logos 10.0 80.0

bottles 10.7 96.4

giraffes 23.4 85.1

mugs 6.5 67.7

swans 17.6 70.6

Average 13.6 80.0

DR at FPPI 0.4

apple logos

bottlesgiraffes

mugs

swans

}

} SSH

BoLSS

FPPI 0.4

Comparison results (avg):[Ferrari CVPR07]: 71.9[Maji CVPR09]: 93.2

… many more

DR

at 0

.5 P

ASCA

L ov

erla

p

Runtimes for Computing Descriptors• 200x200 image• GSS tensor – directly: 5512s ( 1.5 hours)∼– using our method: 81s ( 1.5 minutes)∼

• Computing descriptors: few seconds• Our method: 70x speedup

• For Reference:– GIST: 0.4s– BOLSS: 0.7s

23

Runtimes for Detection

Given the prototype assignment map (80s) (once only)

SSH sliding window: 30s/img (once per class)

For Comparison– Computing direct GSS tensor for 25000 windows: 4 years/img

Speedup: ∼1 million ⇒ Using our methods, GSS can be used for object detection

For Reference:– Felzenszwalb PAMI 09: 5s.

24

June 2014

Global and Efficient Self-Similarity for Object Classification and Detection

CVPR 2010

Thomas Deselaers and Vittorio Ferrari

Feasible

Conclusion• self-similarity should be considered globally– Global self-similarity performs better than local self-similarity

• truly complementary to conventional descriptors

• global self-similarity is feasible – efficient computation of self-similarity– two descriptors based on self-similarity

• global self-similarity for detection

• code will be available soon

26

Thank you for your attention

Thank you for your attention