Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 1

Scale-space image processing

Corresponding image features can appear at different scales

Like shift-invariance, scale-invariance of image processing algorithms

is often desirable.

Scale-space representation is useful to process an image in a manner

that is both shift-invariant and scale-invariant

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 2

Scale-space image processing

Scale-space theory

Laplacian of Gaussian (LoG) and Difference of Gaussian (DoG)

Scale-space edge detection

Scale-space keypoint detection

Harris-Laplacian

SIFT detector

SURF detector

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 3

Scale-space representation of a signal

Successive smoothingwith a Gaussian filter

Zero-crossings of 2nd derivative Fewer edges at coarser scales

Parametric family of signals f t (x) where fine-scale information is successively attenuated

f x( ) = f 0 x( )

f t x( ) scale t

f t x

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 4

Scale-space representation of images

Parametric family of images smoothed by Gaussian filter

Shift-invariance

Rotation-invariance

2 2

2 2

1, , * , ; t 0 with , exp

2 2

, , , with , exp2

t t t

t t t

x y x y x y x y x y

x yf x y g x y f x y g x y

t t

tF G F G

, , * , t tf x x y y g x y f x x y y

cos sin , sin cos , * cos sin , sin cost tf x y x y g x y f x y x y

Original

image f (x,y)

Coarser

scalest

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 5

Scale-space representation of images (cont.)

Commutative semigroup property

Separability

2 2 2 2

2 2 2 2

1 1 1, exp exp exp

2 2 2 22 2

, exp exp exp2 2 2

t

t

x y x y x y

x y x yg x y

t t t tt t

t t tG

1 2 1 2 2 1

1 2

, , * , , * ,

, * , * ,

t t t t t t

t t

f x y g x y f x y g x y f x y

g x y g x y f x y

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 6

Scale-space representation of images (cont.)

Non-creation of local extrema (for f (x,y) and all of its partial derivatives)

since and unimodal.

Solution to diffusion equation (heat equation)

, 0tg x y

2 2

2 2 2 2

2 2

, , ,

exp ,2

1exp ,

2 2

1,

2

t t

x y x y x y

x y x y

x y x y x y

t

x y x y

F G Ft t

tF

t

tF

F

21, ,

2

t tf x y f x yt

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 7

21, ,

2

t tf x y f x yt

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 8

LoG vs. DoG

-4

-2

0

2

4

-4

-2

0

2

4

-0.25

-0.2

-0.15

-0.1

-0.05

0

0.05

XY -4

-2

0

2

4

-4

-2

0

2

4

-0.25

-0.2

-0.15

-0.1

-0.05

0

0.05

XY

2 2

2 2

22

1, 1

2

x y

tx y

LoG x y et t

21

, ( , ) ( , )1

k t tDoG x y g x y g x yk t

Laplacian of Gaussian Difference of Gaussians

t = σ2 = 1 t = σ2 = 1, k = 1.1 21

, ,2

t tf x y f x yt

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 9

LoG vs. DoG (cont.)

Laplacian of Gaussian Difference of Gaussians

-2

0

2

-2

0

2

0

0.2

0.4

0.6

0.8

x

y

|H|

-2

0

2

-2

0

2

0

0.2

0.4

0.6

0.8

x

y

|H|

2 2, ,t

x y x y x yH G

21

, , ,1

k t t

x y x y x yH G Gk t

t = σ2 = 1 t = σ2 = 1, k = 1.1

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 10

Scale space: Laplacian images

t = 1 t = 4 t = 16 t = 64

,tf x y

2 ,tt f x y

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 11

Scale space: Binarized Laplacian images

t = 1 t = 4 t = 16 t = 64

,tf x y

2 ,tsign t f x y

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 12

Scale space: edge detection

Zero crossings of Laplacian images

Low-gradient-magnitude edges removed

t = 1 t = 4 t = 16 t = 64

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 13

Laplacian zero-crossings

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 14

Scale-space representation provides all scales;

which scale is best for keypoint detection?

Harris-Laplacian

1. Detect Harris corners at some initial scale

2. For each Harris corner

detect characteristic scale

3. Apply Harris detector in a spatial neighborhood

at scale to refine keypoint location

4. Repeat 2. and 3. until convergence

2arg max ,t

h h ht

t t f x y

,h hx y

ht ,h hx y

scale t

x

y Harris

Harris

Harris

Keypoint detection with automatic scale selection

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 15

Harris-Laplacian example (150 strongest peaks)

Keypoint detection with automatic scale selection

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 16

Keypoint detection with automatic scale selection

Harris-Laplacian example (200 strongest peaks)

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 17

SIFT - Scale-Invariant Feature Transform

Decompose image into DoG scale-space representation

Detect minima and maxima locally and across scales

Fit 3-d quadratic function to localize extrema with sub-pixel/sub-scale accuracy [Brown, Lowe, 2002]

Eliminate edge responses based on Hessian

t 1

t 2

t 2

t 2 2

t 4

t 4

t 4 2

t 8

t 8 2

t 16

Gaussian

Difference of

Gaussian (DoG)

Scale

(first

octave)

Scale

(next

octave)

Scale

…SIFT keypoint detection

[Lowe, 1999, 2004]

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 18

SIFT scale space pyramid: octave 1

-

-

-

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 19

SIFT scale space pyramid: octave 2

-

-

-

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 20

SIFT scale space pyramid: octave 3

-

-

-

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 21

SIFT scale space pyramid: octave 4

-

-

-

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 22

SIFT scale space pyramid: octave 5

-

-

-

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 23

SIFT keypoints

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 24

SIFT keypoints

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 25

Robustness against scaling

[Mikolajczyk, Schmid, 2001]

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 26

Hessian keypoints in scale space

( , )g x y

( , )g x yx

( , )g x y

y

2

2( , )g x y

x

2

2( , )g x y

y

2

( , )g x yx y

2 2

2

2 2

2

,

, ,

, ,

t

t t

t t

x y

f x y f x yx x y

f x y f x yx y y

H

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 27

SURF keypoint detection

SURF – Speeded Up Robust Features [Bay, Tuytelaars, Van Gool, ECCV 2006]

No subsampling – all resolution levels at full spatial resolution

Simple approximation of scale space Gaussian derivatives using integral images

Determinant of Hessian

Non-maximum suppression in 3x3x3 [x,y,t] neighborhood

Interpolation of maximum of det(H) in image space x,y and scale t

t

yyD t

xyD

2

det 0.9t t t t

xx yy xyD D D H

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 28

SURF keypoints

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 29

SIFT keypoints

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 30

SURF keypoints

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 31

SIFT keypoints