#1

Lecture 8: Interest Point Detection

Saad J Bedrossbedros@umn.edu

#2

Last Lecture : Edge Detection• Preprocessing of image is desired to eliminate or

at least minimize noise effects• There is always tradeoff between sensitivity and

accuracy in edge detection • The parameters that we can set for edge detection

and labeling include the size of the edge detection mask and the value of the threshold

• A larger mask or a higher gray level threshold will tend to reduce noise effects, but may result in a loss of valid edge points

Example

original image (Lena)

Compute Gradients

X-Derivative of Gaussian Y-Derivative of Gaussian Gradient Magnitude

Get Orientation at Each Pixel

• Threshold at minimum level• Get orientation

theta = atan2(gy, gx)

#6

Edge Thinning: Non-maximum suppression for each orientation

At q, we have a maximum if the value is larger than those at both p and at r. Interpolate to get these values.

Today’s Lecture

Goal: Find features between multiple images taken from different position or time to be robustly matched

Approaches:• Patch/Template Matching• Interest Point Detection

Applications

Comparison Between 2 or more images• Image alignment • Image Stitching• 3D reconstruction• Object recognition ( matching ) • Indexing and database retrieval• Object tracking• Robot navigation

• What points would you choose?Known as: Template Matching Method, with option to use correlation as a metric

Correlation#11

Correlation#12

• Use Cross Correlation to find the template in an image

• Maximum indicate high similarity

Template/Patch Matching Options

• What do we do for different scales of the patch ?– Compute multiple Templates at different sizes– Match for different scales of the template

• What can we do to identify the shape of the pattern at different mean brightness levels – Remove the mean of the template

• Example in 1D

We need more robust feature descriptors for matching !!!!!

Interest Points

• Local features associated with a significant change of an image property of several properties simultaneously (e.g., intensity, color, texture).

Why Extract Interest Points?

• Corresponding points (or features) between images enable the estimation of parameters describing geometric transforms between the images.

What if we don’t know the correspondences?

• Need to compare feature descriptors of local patches surrounding interest points

( ) ( )=?feature

descriptorfeature

descriptor

?

Properties of good features• Local: features are local, robust to occlusion and

clutter (no prior segmentation!).• Accurate: precise localization.

• Invariant / Covariant• Robust: noise, blur, compression, etc.

do not have a big impact on the feature.

• Distinctive: individual features can be matched to a large database of objects.

• Efficient: close to real-time performance.

Repeatable

Invariant / Covariant

• A function f( ) is invariant under some transformation T( ) if its value does change when the transformation is applied to its argument:

if f(x) = y then f(T(x))=y

• A function f( ) is covariant when it commutes with the transformation T( ):

if f(x) = y then f(T(x))=T(f(x))=T(y)

Invariance• Features should be detected despite geometric or

photometric changes in the image.• Given two transformed versions of the same

image, features should be detected in corresponding locations.

Example: Panorama Stitching

• How do we combine these two images?

Panorama stitching (cont’d)

Step 1: extract featuresStep 2: match features

Panorama stitching (cont’d)

Step 1: extract featuresStep 2: match featuresStep 3: align images

What features should we use?

Use features with gradients in at least two (significantly) different orientations patches ? Corners ?

What features should we use? (cont’d)

(auto-correlation)

Corners vs Edges#27

Corners in Images#28

The Aperture Problem#29

#30

Corner Detection

#31

Corner Detection-Basic Idea

Corner Detector: Basic Idea

“flat” region:no change in all directions

“edge”:no change along the edge direction

“corner”:significant change in all directions

Corner Detection Using Intensity: Basic Idea

• Image gradient has two or more dominant directions near a corner.

• Shifting a window in any direction should give a large change in intensity.

“edge”: no change along the edge direction

“corner”: significant change in all directions

“flat” region:no change in all directions

Corner Detection Using Edge Detection?

• Edge detectors are not stable at corners.• Gradient is ambiguous at corner tip.• Discontinuity of gradient direction near

corner.

Corner Types

Example of L-junction, Y-junction, T-junction, Arrow-junction, and X-junction corner types

Mains Steps in Corner Detection

1. For each pixel in the input image, the corner operator is appliedto obtain a cornerness measure for this pixel.

2. Threshold cornerness map to eliminate weak corners.3. Apply non-maximal suppression to eliminate points whose

cornerness measure is not larger than the cornerness values of all points within a certain distance.

Mains Steps in Corner Detection (cont’d)

Moravec Detector (1977)• Measure intensity variation at (x,y) by shifting a

small window (3x3 or 5x5) by one pixel in each of the eight principle directions (horizontally, vertically, and four diagonals).

Moravec Detector (1977)• Calculate intensity variation by taking the sum of

squares of intensity differences of corresponding pixels in these two windows.

∆x, ∆y in {-1,0,1}

SW(-1,-1), SW(-1,0), ...SW(1,1)

8 directions

Moravec Detector (cont’d)• The “cornerness” of a pixel is the minimum intensity variation

found over the eight shift directions:

Cornerness(x,y) = min{SW(-1,-1), SW(-1,0), ...SW(1,1)}

CornernessMap

(normalized)

Note response to isolated points!

Moravec Detector (cont’d)

• Use Non-maximal suppression will yield the final corners.Process of Zero out all pixels that are not the maximum along the direction of the gradient (look at 1 pixel on each side)

Moravec Detector (cont’d)

• Does a reasonable job infinding the majority of true corners.

• Edge points not in one of the eight principle directions will be assigned a relatively large cornerness value.

Moravec Detector (cont’d)• The response is anisotropic (directionally sensitive)

as the intensity variation is only calculated at a discrete set of shifts (i.e., not rotationally invariant)

Corner Detection: Mathematics

2

,( , ) ( , ) ( , ) ( , )

x yE u v w x y I x u y v I x y

Change in appearance of window w(x,y) for the shift [u,v]:

I(x, y)E(u, v)

E(3,2)

w(x, y)

Corner Detection: Mathematics

2

,( , ) ( , ) ( , ) ( , )

x yE u v w x y I x u y v I x y

I(x, y)E(u, v)

E(0,0)

w(x, y)

Change in appearance of window w(x,y) for the shift [u,v]:

Harris Detector: Mathematics

2

,( , ) ( , ) ( , ) ( , )

x yE u v w x y I x u y v I x y

Change of intensity for the shift [u,v]:

IntensityShifted intensity

Window function

orWindow function w(x,y) =

Gaussian1 in window, 0 outside

#47

Harris Detector

• Improves the Moravec operator by avoiding the use of discrete directions and discrete shifts.

• Uses a Gaussian window instead of a square window.

Harris Detector (cont’d)

• Using first-order Taylor approximation:

Reminder: Taylor expansion ( )

2 1f´( ) f´´( ) f ( )( ) f( ) ( ) ( ) ( ) ( )1! 2! !

nn na a af x a x a x a x a O x

n

Harris Detector (cont’d)

Since

Harris Detector (cont’d)

2 x 2 matrix(auto-correlation or2nd order moment matrix)

2( , )( )i if x yx

2( , )( )i if x yy

AW(x,y)=

Harris Detector

2

,( , ) ( , ) ( , ) ( , )

i i

W i i i i i ix y

S x y w x y f x y f x x y y

default window function w(x,y) :

1 in window, 0 outside

• General case – use window function:

22

, ,2

,2

, ,

( , ) ( , )( , )

( , ) ( , )

x x yx y x y x x y

Wx y x y y

x y yx y x y

w x y f w x y f ff f f

A w x yf f f

w x y f f w x y f

Harris Detector (cont’d)

window function w(x,y) :Gaussian

• Harris uses a Gaussian window: w(x,y)=G(x,y,σI) where σIis called the “integration” scale

22

, ,2

,2

, ,

( , ) ( , )( , )

( , ) ( , )

x x yx y x y x x y

Wx y x y y

x y yx y x y

w x y f w x y f ff f f

A w x yf f f

w x y f f w x y f

Harris Detector

Describes the gradientdistribution (i.e., local structure)inside window!

Does not depend on

2

2,

x x yW

x y x y y

f f fA

f f f

,x y

Harris Detector (cont’d)

Since M is symmetric, we have: 11

2

00WA R R

We can visualize AW as an ellipse with axis lengths determined by the eigenvalues and orientation determined by R

(λmin)-1/2

(λmax)-1/2[ ] constW

xx y A

y

Ellipse equation:direction of the slowest change

direction of the fastest change

Harris Corner Detector (cont’d)

• The eigenvectors of AW encode direction of intensity change.

• The eigenvalues of AW encode strength of intensity change.

v1

v2

Harris Detector (cont’d)

• Eigenvectors encode edge direction

• Eigenvalues encode edge strength

(λmin)-1/2

(λmax)-1/2

direction of the fastest change

direction of the slowest change

Distribution of fx and fy

Distribution of fx and fy (cont’d)

Harris Detector (cont’d)

(assuming that λ1 > λ2)

2

%

Harris Detector

Measure of corner response:

(k – empirical constant, k = 0.04-0.06)

(Shi-Tomasi variation: use min(λ1,λ2) instead of R)

http

://w

ww

.wis

dom

.wei

zman

n.ac

.il/~

deni

ss/v

isio

n_sp

ring0

4/fil

es/In

varia

ntFe

atur

es.p

ptD

arya

Fro

lova

, Den

is S

imak

ov T

he W

eizm

ann

Inst

itute

of S

cien

ce

Harris Detector: Mathematics

“Corner”

“Edge”

“Edge”

“Flat”

• R depends only on eigenvalues of M

• R is large for a corner

• R is negative with large magnitude for an edge

• |R| is small for a flatregion

R > 0

R < 0

R < 0|R| small

Harris Corner Detector (cont’d)

1

2

“Corner”1 and 2 are large,1 ~ 2;intensity changes in all directions

1 and 2 are small;intensity is almost constant in all directions

“Edge” 1 >> 2

“Edge” 2 >> 1

“Flat” region

Classification of pixels using the eigenvalues of AW :

Harris Detector (cont’d)

( , ) ( , , )* ( , )i iD i i

f x y G x y f x yx x

( , ) ( , , )* ( , )i iD i i

f x y G x y f x yy y

σD is called the “differentiation” scale

Harris Detector

• The Algorithm:– Find points with large corner response function

R (R > threshold)– Take the points of local maxima of R

65

Harris corner detector algorithm• Compute image gradients Ix Iy for all pixels• For each pixel

– Compute

by looping over neighbors x,y– compute

• Find points with large corner response function R(R > threshold)

• Take the points of locally maximum R as the detected feature points (ie, pixels where R is bigger than for all the 4 or 8 neighbors).

65

http

://w

ww

.wis

dom

.wei

zman

n.ac

.il/~

deni

ss/v

isio

n_sp

ring0

4/fil

es/In

varia

ntFe

atur

es.p

ptD

arya

Fro

lova

, Den

is S

imak

ov T

he W

eizm

ann

Inst

itute

of S

cien

ce

Harris Corner Detector (cont’d)• To avoid computing the eigenvalues

explicitly, the Harris detector uses the following function:

R(AW) = det(AW) – α trace2(AW)

which is equal to:

R(AW) = λ1 λ2- α (λ1+ λ2)2

α: is a const

Harris Corner Detector (cont’d)

“Corner”R > 0

“Edge” R < 0

“Edge” R < 0

“Flat” region

|R| smallα: is usually between 0.04 and 0.06

R(AW) = det(AW) – α trace2(AW)

Classification of image

points using R(AW):

Harris Corner Detector (cont’d)

• Other functions:

2 1

1 2

1 2

det

a

AtrA

Harris Corner Detector - Steps

( , , )* ( , )x D i if G x y f x yx

( , , )* ( , )y D i if G x y f x yy

σD is called the “differentiation” scale

1. Compute the horizontal and vertical (Gaussian) derivatives

2. Compute the three images involved in AW :

2

2,

( , ) x x yW

x W y W x y y

f f fA w x y

f f f

Harris Detector - Steps

R(AW) = det(AW) – α trace2(AW)

3. Convolve each of the three images with a larger Gaussian

w(x,y) :Gaussian

σI is called the “integration” scale

4. Determine cornerness:

5. Find local maxima

Harris Detector - Example

Harris Detector - Example

Harris Detector - ExampleCompute corner response R

Harris Detector - ExampleFind points with large corner response: R>threshold

Harris Detector - ExampleTake only the points of local maxima of R

Harris Detector - ExampleMap corners on the original image

Harris Detector – Scale Parameters

• The Harris detector requires two scale parameters:

(i) a differentiation scale σD for smoothing prior to the computation of image derivatives,

&(ii) an integration scale σI for defining the size of the Gaussian window (i.e., integrating derivative responses). AW(x,y) AW(x,y,σI,σD)

• Typically, σI=γσD

Invariance to Geometric/Photometric Changes

• Is the Harris detector invariant to geometric and photometric changes?

• Geometric

– Rotation

– Scale

– Affine

• Photometric– Affine intensity change: I(x,y) a I(x,y) + b

Harris Detector: Rotation Invariance

• Rotation

Ellipse rotates but its shape (i.e. eigenvalues) remains the same

Corner response R is invariant to image rotation

Harris Detector: Rotation Invariance (cont’d)

Harris Detector: Photometric Changes

• Affine intensity change Only derivatives are used => invariance to intensity shift I(x,y) I (x,y) + b

Intensity scale: I(x,y) a I(x,y)

R

x (image coordinate)

threshold

R

x (image coordinate)

Partially invariant to affine intensity change

Harris Detector: Scale Invariance

• Scaling

All points will be classified as edges

Corner

Not invariant to scaling (and affine transforms)

Harris Detector: Repeatability

• In a comparative study of different interest point detectors, Harris was shown to be the most repeatable and most informative.

C. Schmid, R. Mohr, and C. Bauckhage, "Evaluation of Interest Point Detectors", International Journal of Computer Vision 37(2), 151.172, 2000.

best results:σI=1, σD=2

Repeatability = 1 2

#min( , )

correspondencesm m

Harris Detector: Disadvantages

• Sensitive to:– Scale change– Significant viewpoint change– Significant contrast change

How to handle scale changes?

• AW must be adapted to scale changes.

• If the scale change is known, we can adapt the Harris detector to the scale change (i.e., set properly σI ,σD).

• What if the scale change is unknown?

Multi-scale Harris Detector

scale

x

y

Harris

• Detects interest points at varying scales.R(AW) = det(AW(x,y,σI,σD)) – α trace2(AW(x,y,σI,σD))

σnσD= σnσI=γσD

σn=knσ

How to cope with transformations?

• Exhaustive search• Invariance• Robustness

Exhaustive search• Multi-scale approach

Slide from T. Tuytelaars ECCV 2006 tutorial

Exhaustive search• Multi-scale approach

Exhaustive search

• Multi-scale approach

Exhaustive search• Multi-scale approach

How to handle scale changes?

• Not a good idea!– There will be many points representing the same

structure, complicating matching! – Note that point locations shift as scale increases.

The size of the circle corresponds to the scale at which the point was detected

How to handle scale changes? (cont’d)

• How do we choose corresponding circles independently in each image?

How to handle scale changes? (cont’d)

• Alternatively, use scale selection to find the characteristic scale of each feature.

• Characteristic scale depends on the feature’s spatial extent (i.e., local neighborhood of pixels).

scale selection scale selection

How to handle scale changes?

The size of the circles corresponds to the scale at which the point was selected.

• Only a subset of the points computed in scale space are selected!

Automatic Scale Selection• Design a function F(x,σn) which provides some local

measure. • Select points at which F(x,σn) is maximal over σn.

T. Lindeberg, "Feature detection with automatic scale selection" International Journal of Computer Vision, vol. 30, no. 2, pp 77-116, 1998.

max of F(x,σn)corresponds tocharacteristic scale!

σn

F(x,σn)

Invariance

• Extract patch from each image individually

Automatic scale selection• Solution:

– Design a function on the region, which is “scale invariant” (the same for corresponding regions, even if they are at different scales)

Example: average intensity. For corresponding regions (even of different sizes) it will be the same.

scale = 1/2

– For a point in one image, we can consider it as a function of region size (patch width)

f

region size

Image 1 f

region size

Image 2

Automatic scale selection• Common approach:

scale = 1/2f

region size

Image 1 f

region size

Image 2

Take a local maximum of this function

Observation: region size, for which the maximum is achieved, should be invariant to image scale.

s1 s2

Important: this scale invariant region size is found in each image independently!

Automatic Scale Selection

100

)),(( )),((11

xIfxIfmm iiii

Same operator responses if the patch contains the same image up to scale factor.

Automatic Scale Selection• Function responses for increasing scale (scale signature)

101)),((

1xIf

mii )),((1

xIfmii

Automatic Scale Selection• Function responses for increasing scale (scale signature)

102)),((

1xIf

mii )),((1

xIfmii

Automatic Scale Selection• Function responses for increasing scale (scale signature)

103)),((

1xIf

mii )),((1

xIfmii

Automatic Scale Selection• Function responses for increasing scale (scale signature)

104)),((

1xIf

mii )),((1

xIfmii

Automatic Scale Selection• Function responses for increasing scale (scale signature)

105)),((

1xIf

mii )),((1

xIfmii

Automatic Scale Selection• Function responses for increasing scale (scale signature)

106)),((

1xIf

mii )),((1

xIfmii

Scale selection• Use the scale determined by detector to compute

descriptor in a normalized frame

Automatic Scale Selection (cont’d)

• Characteristic scale is relatively independentof the image scale.

• The ratio of the scale values corresponding to the max values, is equal to the scale factor between the images.

Scale selection allows for finding spatial extend that is covariant with the image transformation.

Scale factor: 2.5

Automatic Scale Selection

• What local measures should we use? – Should be rotation invariant– Should have one stable sharp peak

How should we choose F(x,σn) ?

• Typically, F(x,σn) is defined using derivatives, e.g.:

2 2 2

2

12

: ( ( , ) ( , ))

: | ( ( , ) ( , )) |

:| ( )* ( ) ( )* ( ) |

: det( ) ( )

x y

xx yy

n n

W W

Square gradient L x L x

LoG L x L x

DoG I x G I x G

Harris function A trace A

C. Schmid, R. Mohr, and C. Bauckhage, "Evaluation of Interest Point Detectors", International Journal of Computer Vision, 37(2), pp. 151-172, 2000.

Review#111• Patch/Template Matching

• Interest Points• Corner Detection: Harris Detector

• Properties of Harris Detector• Scaling approach