Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector •...

51
Descriptors II CSE 576 Ali Farhadi Many slides from Larry Zitnick, Steve Seitz

Transcript of Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector •...

Page 1: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

Descriptors II

CSE 576 Ali Farhadi

Many slides from Larry Zitnick, Steve Seitz

Page 2: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

How can we find corresponding points?

Page 3: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

How can we find correspondences?

Page 4: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

SIFT descriptorFull version

• Divide the 16x16 window into a 4x4 grid of cells (2x2 case shown below) • Compute an orientation histogram for each cell • 16 cells * 8 orientations = 128 dimensional descriptor

Adapted from slide by David Lowe

Page 5: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

Local Descriptors: Shape Context

Count the number of points inside each bin, e.g.:

Count = 4

Count = 10...

Log-polar binning: more precision for nearby points, more flexibility for farther points.

Belongie & Malik, ICCV 2001K. Grauman, B. Leibe

Page 6: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

Texture

• Texture is characterized by the repetition of basic elements or textons

• For stochastic textures, it is the identity of the textons, not their spatial arrangement, that matters

Julesz, 1981; Cula & Dana, 2001; Leung & Malik 2001; Mori, Belongie & Malik, 2001; Schmid 2001; Varma & Zisserman, 2002, 2003; Lazebnik, Schmid & Ponce, 2003

Page 7: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

Bag-of-words models• Orderless document representation: frequencies of words

from a dictionary Salton & McGill (1983)

Page 8: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

Bag-of-words models

US Presidential Speeches Tag Cloudhttp://chir.ag/phernalia/preztags/

• Orderless document representation: frequencies of words from a dictionary Salton & McGill (1983)

Page 9: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

Bag-of-words models

US Presidential Speeches Tag Cloudhttp://chir.ag/phernalia/preztags/

• Orderless document representation: frequencies of words from a dictionary Salton & McGill (1983)

Page 10: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

Bag-of-words models

US Presidential Speeches Tag Cloudhttp://chir.ag/phernalia/preztags/

• Orderless document representation: frequencies of words from a dictionary Salton & McGill (1983)

Page 11: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

Bags of features for image classification1. Extract features

Page 12: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

1. Extract features 2. Learn “visual vocabulary”

Bags of features for image classification

Page 13: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

1. Extract features 2. Learn “visual vocabulary” 3. Quantize features using visual vocabulary

Bags of features for image classification

Page 14: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

1. Extract features 2. Learn “visual vocabulary” 3. Quantize features using visual vocabulary 4. Represent images by frequencies of

“visual words”

Bags of features for image classification

Page 15: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

Texture representation

Universal texton dictionary

histogram

Julesz, 1981; Cula & Dana, 2001; Leung & Malik 2001; Mori, Belongie & Malik, 2001; Schmid 2001; Varma & Zisserman, 2002, 2003; Lazebnik, Schmid & Ponce, 2003

Page 16: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

• Regular grid • Vogel & Schiele, 2003 • Fei-Fei & Perona, 2005

• Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005

1. Feature extraction

Page 17: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

• Regular grid • Vogel & Schiele, 2003 • Fei-Fei & Perona, 2005

• Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005

• Other methods • Random sampling (Vidal-Naquet & Ullman, 2002) • Segmentation-based patches (Barnard et al. 2003)

1. Feature extraction

Page 18: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

Normalize patch

Detect patches [Mikojaczyk and Schmid ’02]

[Mata, Chum, Urban & Pajdla, ’02]

[Sivic & Zisserman, ’03]

Compute SIFT

descriptor [Lowe’99]

Slide credit: Josef Sivic

1. Feature extraction

Page 19: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

…

1. Feature extraction

Page 20: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

2. Discovering the visual vocabulary

…

Page 21: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

2. Discovering the visual vocabulary

Clustering

…

Slide credit: Josef Sivic

Page 22: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

2. Discovering the visual vocabulary

Clustering

…

Slide credit: Josef Sivic

Visual vocabulary

Page 23: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

Clustering and vector quantization• Clustering is a common method for learning a visual

vocabulary or codebook • Unsupervised learning process • Each cluster center produced by k-means becomes a

codevector • Codebook can be learned on separate training set • Provided the training set is sufficiently representative, the

codebook will be “universal”

• The codebook is used for quantizing features • A vector quantizer takes a feature vector and maps it to the

index of the nearest codevector in a codebook • Codebook = visual vocabulary • Codevector = visual word

Page 24: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

Example visual vocabulary

Fei-Fei et al. 2005

Page 25: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

Example codebook

…

Source: B. Leibe

Appearance codebook

Page 26: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

Another codebook

Appearance codebook…

………

…

Source: B. Leibe

Page 27: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

Visual vocabularies: Issues

• How to choose vocabulary size? • Too small: visual words not representative of all patches • Too large: quantization artifacts, overfitting

• Computational efficiency • Vocabulary trees

(Nister & Stewenius, 2006)

Page 28: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

3. Image representation

…..

freq

uenc

y

codewords

Page 29: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

Image classification• Given the bag-of-features representations of images

from different classes, learn a classifier using machine learning

Page 30: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

Another Representation: Filter bank

Page 31: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

Imag

e fro

m h

ttp://

ww

w.te

xase

xplo

rer.c

om/a

ustin

cap2

.jpg

Kristen Grauman

Page 32: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

Showing magnitude of responses

Kristen Grauman

Page 33: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

Kristen Grauman

Page 34: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

Kristen Grauman

Page 35: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

Kristen Grauman

Page 36: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

Kristen Grauman

Page 37: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

Kristen Grauman

Page 38: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

Kristen Grauman

Page 39: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

Kristen Grauman

Page 40: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

Kristen Grauman

Page 41: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

Kristen Grauman

Page 42: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

How can we represent texture?

• Measure responses of various filters at different orientations and scales

• Idea 1: Record simple statistics (e.g., mean, std.) of absolute filter responses

Page 43: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

Can you match the texture to the response?

Mean abs responses

FiltersA

B

C

1

2

3

Page 44: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

Representing texture by mean abs response

Mean abs responses

Filters

Page 45: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

Representing texture

• Idea 2: take vectors of filter responses at each pixel and cluster them, then take histograms

Page 46: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

Representing texture

clustering

Page 47: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

But what about layout?

All of these images have the same color histogram

Page 48: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

Spatial pyramid representation• Extension of a bag of features • Locally orderless representation at several levels of resolution

level 0

Lazebnik, Schmid & Ponce (CVPR 2006)

Page 49: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

level 0 level 1

Lazebnik, Schmid & Ponce (CVPR 2006)

Spatial pyramid representation• Extension of a bag of features • Locally orderless representation at several levels of resolution

Page 50: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

level 0 level 1 level 2

Lazebnik, Schmid & Ponce (CVPR 2006)

Spatial pyramid representation• Extension of a bag of features • Locally orderless representation at several levels of resolution

Page 51: Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

What about Scenes?


First Aid Christophe Perona

First Aid Christophe Perona

One Piece Perona and Nico Robin Cosplay

One Piece Perona and Nico Robin Cosplay

Kosmet Perona Pipi

Kosmet Perona Pipi

Lecture3 xing fei-fei

Lecture3 xing fei-fei

Recognition in Terra Incognita - Foundation · RecognitioninTerra Incognita Sara Beery, Grant Van Horn, and Pietro Perona Caltech {sbeery,gvanhorn,perona}@caltech.edu Abstract. It

Recognition in Terra Incognita - Foundation · RecognitioninTerra Incognita Sara Beery, Grant Van Horn, and Pietro Perona Caltech {sbeery,gvanhorn,perona}@caltech.edu Abstract. It

ANY XV NÚM. 56 Esteve Lucerón La Perona

ANY XV NÚM. 56 Esteve Lucerón La Perona

Mid Level Vision - unict.itbattiato/CVision1213/Parte 5 Bag_of_features.pdf · • Fei-Fei & Perona, 2005 • Sivic et al. 2005 Other methods • Random sampling (Vidal-Naquet & Ullman,

Mid Level Vision - unict.itbattiato/CVision1213/Parte 5 Bag_of_features.pdf · • Fei-Fei & Perona, 2005 • Sivic et al. 2005 Other methods • Random sampling (Vidal-Naquet & Ullman,

DELITOS CONTRA LA PROPIEDADportal.eco.unc.edu.ar/files/ief/workshops/2005/11oct05...DELITOS CONTRA LA PROPIEDAD-Un análisis de disparidad regional -Dra. Hada G. Juárez de Perona

DELITOS CONTRA LA PROPIEDADportal.eco.unc.edu.ar/files/ief/workshops/2005/11oct05...DELITOS CONTRA LA PROPIEDAD-Un análisis de disparidad regional -Dra. Hada G. Juárez de Perona

An Analysis of the Perona-Malik Schemeesedoglu/Papers_Preprints/pm_esedoglu.pdf · An Analysis of the Perona-Malik Scheme ... ANALYSIS OF PERONA-MALIK SCHEME 3 ... Some previous mathematical

An Analysis of the Perona-Malik Schemeesedoglu/Papers_Preprints/pm_esedoglu.pdf · An Analysis of the Perona-Malik Scheme ... ANALYSIS OF PERONA-MALIK SCHEME 3 ... Some previous mathematical

Perona Pipi (Rouge)

Perona Pipi (Rouge)

Lecture13 xing fei-fei

Lecture13 xing fei-fei

Lecture7 xing fei-fei

Lecture7 xing fei-fei

Cluster-specific Named Entity Transliteration Fei Huang HLT/EMNLP 2005.

Cluster-specific Named Entity Transliteration Fei Huang HLT/EMNLP 2005.

Li Fei-Fei, UIUC Rob Fergus, MIT Antonio Torralba, MIT Recognizing and Learning Object Categories ICCV 2005 Beijing, Short Course, Oct 15.

Li Fei-Fei, UIUC Rob Fergus, MIT Antonio Torralba, MIT Recognizing and Learning Object Categories ICCV 2005 Beijing, Short Course, Oct 15.

Fei Xiao Tong November 2, 1910 – April 24, 2005 Chinese anthropologist and public intellectual.

Fei Xiao Tong November 2, 1910 – April 24, 2005 Chinese anthropologist and public intellectual.

Understanding human motion · 2005. 5. 3. · Edge Filters Normalized derivatives of Gaussians (Lindeberg, Granlund and Knutsson, Perona, Freeman&Adelson, …) Edge filter response

Understanding human motion · 2005. 5. 3. · Edge Filters Normalized derivatives of Gaussians (Lindeberg, Granlund and Knutsson, Perona, Freeman&Adelson, …) Edge filter response

Transferring information using Bayesian priors on object categories Li Fei-Fei 1, Rob Fergus 2, Pietro Perona 1 1 California Institute of Technology, 2.

Transferring information using Bayesian priors on object categories Li Fei-Fei 1, Rob Fergus 2, Pietro Perona 1 1 California Institute of Technology, 2.

Diapositiva disolucion ,liquidacion y extincion de la perona juridica

Diapositiva disolucion ,liquidacion y extincion de la perona juridica

IBM Global Services IT Industry Trends & Directions Presented to FEI Regina Chapter March 9, 2005 Brian Fergusson Presented to FEI Regina Chapter March.

IBM Global Services IT Industry Trends & Directions Presented to FEI Regina Chapter March 9, 2005 Brian Fergusson Presented to FEI Regina Chapter March.

arXiv:2003.01455v3 [cs.CV] 10 Mar 2020Fedor Zhdanov Amazon fedor@amazon.com Pietro Perona Amazon perona@caltech.edu Krzysztof Chalupka Amazon chalupkk@amazon.com Abstract Trained on

arXiv:2003.01455v3 [cs.CV] 10 Mar 2020Fedor Zhdanov Amazon [email protected] Pietro Perona Amazon [email protected] Krzysztof Chalupka Amazon [email protected] Abstract Trained on