Lecture 7 - Fei-Fei Li

Lecture 7: Finding Features (part 2/2)

Dr. Juan Carlos Niebles Stanford AI Lab

Professor Fei-Fei Li Stanford Vision Lab

17-Oct-16 1

Lecture 7 - Fei-Fei Li

What we will learn today?

•  Local invariant features – MoOvaOon –  Requirements, invariances

•  Keypoint localizaOon – Harris corner detector

•  Scale invariant region selecOon – AutomaOc scale selecOon – Difference-of-Gaussian (DoG) detector

•  SIFT: an image region descriptor

17-Oct-16 2

Previous lecture (#6)

Some background reading: David Lowe, IJCV 2004

Lecture 7 - Fei-Fei Li

A quick review

•  Local invariant features – MoOvaOon – Requirements, invariances

•  Keypoint localizaOon – Harris corner detector

•  Scale invariant region selecOon – AutomaOc scale selecOon – Difference-of-Gaussian (DoG) detector

•  SIFT: an image region descriptor

17-Oct-16 3

Lecture 7 - Fei-Fei Li

General Approach N

p ix

el s

N pixels

Similarity measure A

f

e.g. color

Bf

e.g. color

B1 B2

B3 A1

A2 A3

Tffd BA

Lecture 7 - Fei-Fei Li

A quick review

•  Local invariant features – MoOvaOon –  Requirements, invariances

•  Keypoint localizaOon – Harris corner detector

•  Scale invariant region selecOon – AutomaOc scale selecOon – Difference-of-Gaussian (DoG) detector

•  SIFT: an image region descriptor

17-Oct-16 5

Lecture 7 - Fei-Fei Li

Quick review: Harris Corner Detector

17-Oct-16 6

“edge”: no change along the edge direction

“corner”: significant change in all directions

“flat” region: no change in all directions

Sl id

e cr

ed it

: Al

yo sh

a Ef

ro s

Lecture 7 - Fei-Fei Li

Quick review: Harris Corner Detector

17-Oct-16 7

•  Fast approximaOon –  Avoid compuOng the eigenvalues

–  α: constant (0.04 to 0.06)

θ = det(M )−α trace(M )2 = λ1λ2 −α(λ1 +λ2 ) 2

Sl id

e cr

ed it

: Kr

is te

n G

ra um

an

“Corner” θ > 0

“Edge” θ < 0

“Edge” θ < 0

“Flat” region

�1

Lecture 7 - Fei-Fei Li

Quick review: Harris Corner Detector

17-Oct-16 8

Sl id

e ad

ap te

d fr

om D

ar ya

F ro

lo va

, D

en is

S im

ak ov

Lecture 7 - Fei-Fei Li

Quick review: Harris Corner Detector

17-Oct-16 9

•  TranslaOon invariance •  RotaOon invariance •  Scale invariance?

Not invariant to image scale!

All points will be classified as edges!

Corner

Sl id

e cr

ed it

: Kr

is te

n G

ra um

an

Lecture 7 - Fei-Fei Li

What we will learn today?

•  Local invariant features – MoOvaOon –  Requirements, invariances

•  Keypoint localizaOon – Harris corner detector

•  Scale invariant region selecOon – AutomaOc scale selecOon – Difference-of-Gaussian (DoG) detector

•  SIFT: an image region descriptor

17-Oct-16 10

Lecture 7 - Fei-Fei Li

Scale Invariant DetecOon •  Consider regions (e.g. circles) of different sizes around a point

•  Regions of corresponding sizes will look the same in both images

17-Oct-16 11

Lecture 7 - Fei-Fei Li

Scale Invariant DetecOon

•  The problem: how do we choose corresponding circles independently in each image?

17-Oct-16 12

Lecture 7 - Fei-Fei Li

Scale Invariant DetecOon •  SoluOon:

–  Design a funcOon on the region (circle), which is “scale invariant” (the same for corresponding regions, even if they are at different scales)

Example: average intensity. For corresponding regions (even of different sizes) it will be the same.

scale = 1/2

–  For a point in one image, we can consider it as a funcOon of region size (circle radius)

f

region size

Image 1 f

region size

Image 2

17-Oct-16 13

Lecture 7 - Fei-Fei Li

Scale Invariant DetecOon •  Common approach:

scale = 1/2

f

region size

Image 1 f

region size

Image 2

Take a local maximum of this funcOon

•  ObservaOon: region size, for which the maximum is achieved, should be co-variant with image scale.

s1 s2

Important: this scale invariant region size is found in each image independently!

17-Oct-16 14

Lecture 7 - Fei-Fei Li

Scale Invariant DetecOon

•  A “good” funcOon for scale detecOon: has one stable sharp peak

f

region size

ba d

f

region size

bad

f

region size

Good !

•  For usual images: a good funcOon would be one which responds to contrast (sharp local intensity change)

17-Oct-16 15

Lecture 7 - Fei-Fei Li

Scale Invariant DetecOon •  FuncOons for determining scale

2 2

21 2 2

( , , ) x y

G x y e σ πσ

σ +−

=

( )2 ( , , ) ( , , )xx yyL G x y G x yσ σ σ= +

( , , ) ( , , )DoG G x y k G x yσ σ= −

Kernel Imagef = ∗ Kernels:

where Gaussian

Note: both kernels are invariant to scale and rotation

(Laplacian)

(Difference of Gaussians)

17-Oct-16 16

Lecture 7 - Fei-Fei Li

Scale Invariant Detectors •  Harris-Laplacian1

Find local maximum of: –  Harris corner detector in space (image coordinates)

–  Laplacian in scale

1 K.Mikolajczyk, C.Schmid. “Indexing Based on Scale Invariant Interest Points”. ICCV 2001 2 D.Lowe. “DisOncOve Image Features from Scale-Invariant Keypoints”. IJCV 2004

scale

x

y

� Harris �

� L

ap la

ci an

�

•  SIFT (Lowe)2 Find local maximum of: –  Difference of Gaussians in space

and scale

scale

x

y

� DoG �

� D

oG �

17-Oct-16 17

Lecture 7 - Fei-Fei Li

Scale Invariant Detectors

K.Mikolajczyk, C.Schmid. “Indexing Based on Scale Invariant Interest Points”. ICCV 2001

•  Experimental evaluaOon of detectors w.r.t. scale change

Repeatability rate: # correspondences # possible correspondences

17-Oct-16 18

Lecture 7 - Fei-Fei Li

Scale Invariant DetecOon: Summary

•  Given: two images of the same scene with a large scale difference between them

•  Goal: find the same interest points independently in each image

•  SoluOon: search for maxima of suitable funcOons in scale and in space (over the image)

Methods:

1.  Harris-Laplacian [Mikolajczyk, Schmid]: maximize Laplacian over scale, Harris’ measure of corner response over the image

2.  SIFT [Lowe]: maximize Difference of Gaussians over scale and space

17-Oct-16 19

Lecture 7 - Fei-Fei Li

What we will learn today?

•  Local invariant features – MoOvaOon –  Requirements, invariances

•  Keypoint localizaOon – Harris corner detector

•  Scale invariant region selecOon – AutomaOc scale selecOon – Difference-of-Gaussian (DoG) detector

•  SIFT: an image region descriptor

17-Oct-16 20

Lecture 7 - Fei-Fei Li

Local Descriptors •  We know how to detect points •  Next quesOon:

How to describe them for matching?

? Point descriptor should be:

1.   Invariant 2.   Distinctive Sli

de c

re di

t: K

ri st

en G

ra um

an

17-Oct-16 21

Lecture 7 - Fei-Fei Li

CVPR 2003 Tutorial

Recogni5on and Matching Based on Local Invariant Features

David Lowe Computer Science Department University of BriOsh Columbia

17-Oct-16 22

Lecture 7 -

Invariant Local Features

•  Image content is transformed into local feature coordinates that are invariant to translaOon, rotaOon, scale, and other imaging parameters

17-Oct-16 23

Lecture 7 - Fei-Fei Li

Advantages of invariant local features •  Locality: features are local, so robust to occlusion and clumer (no prior segmentaOon)

•  Dis5nc5veness: individual features can be matched to a large database of objects

•  Quan5ty: many features can be generated for even sma