Lecture 6: Finding Features (part 1/2)vision.stanford.edu/.../lectures/lecture6_detectors...Fei-Fei...

Lecture 6 - Fei-Fei Li

Lecture6:FindingFeatures(part1/2)

Dr.JuanCarlosNieblesStanfordAILab

ProfessorFei-FeiLiStanfordVisionLab

10-Oct-161


Whatwewilllearntoday?

•  Localinvariantfeatures– MoOvaOon–  Requirements,invariances

•  KeypointlocalizaOon– Harriscornerdetector

•  ScaleinvariantregionselecOon– AutomaOcscaleselecOon– Difference-of-Gaussian(DoG)detector

•  SIFT:animageregiondescriptor

10-Oct-162

Nextlecture(#7)


Whatwewilllearntoday?





10-Oct-163

Somebackgroundreading:RickSzeliski,Chapter4.1.1;DavidLowe,IJCV2004


Imagematching:achallengingproblem

10-Oct-164


by Diva Sian

by swashford

Slid

e cr

edit

: St

eve

Seit

z

10-Oct-165

Imagematching:achallengingproblem


HarderCase

by Diva Sian by scgbt

Slid

e cr

edit

: St

eve

Seit

z

10-Oct-166


HarderSOll?

NASA Mars Rover images

Slid

e cr

edit

: St

eve

Seit

z

10-Oct-167


AnswerBelow(LookforOnycoloredsquares)

NASAMarsRoverimageswithSIFTfeaturematches(FigurebyNoahSnavely) Sl

ide

cred

it:

Stev

e Se

itz

10-Oct-168


MoOvaOonforusinglocalfeatures•  GlobalrepresentaOonshavemajorlimitaOons•  Instead,describeandmatchonlylocalregions•  Increasedrobustnessto

–  Occlusions

–  ArOculaOon

–  Intra-categoryvariaOons

θqφ

dq

φ

θ

d

10-Oct-169


GeneralApproachN

pix

els

N pixels

Similarity measure Af

e.g. color

Bf

e.g. color

B1B2

B3A1

A2 A3

Tffd BA <),(

1. Find a set of distinctive key- points

3. Extract and normalize the region content

2. Define a region around each keypoint

4. Compute a local descriptor from the normalized region

5. Match local descriptors

Slid

e cr

edit

: Ba

stia

n Le

ibe

10-Oct-1610


CommonRequirements•  Problem1:

–  Detectthesamepointindependentlyinbothimages

No chance to match!

We need a repeatable detector! Slid

e cr

edit

: D

arya

Fro

lova

, D

enis

Sim

akov

10-Oct-1611


CommonRequirements•  Problem1:

–  Detectthesamepointindependentlyinbothimages

•  Problem2:–  Foreachpointcorrectlyrecognizethecorrespondingone

We need a reliable and distinctive descriptor!

?

Slid

e cr

edit

: D

arya

Fro

lova

, D

enis

Sim

akov

10-Oct-1612


Invariance:GeometricTransformaOons

Slid

e cr

edit

: St

eve

Seit

z

10-Oct-1613


LevelsofGeometricInvariance

Slid

e cr

edit

: Ba

stia

n Le

ibe

10-Oct-1614

CS131 CS231a


Invariance:PhotometricTransformaOons

•  OfenmodeledasalineartransformaOon:–  Scaling+Offset

Slid

e cr

edit

: Ti

nne

Tuyt

elaa

rs

10-Oct-1615


Requirements•  RegionextracOonneedstoberepeatableandaccurate

–  InvarianttotranslaOon,rotaOon,scalechanges–  Robustorcovarianttoout-of-plane(�affine)transformaOons–  RobusttolighOngvariaOons,noise,blur,quanOzaOon

•  Locality:Featuresarelocal,thereforerobusttoocclusionandcluier.

•  QuanOty:Weneedasufficientnumberofregionstocovertheobject.

•  DisOncOveness:Theregionsshouldcontain“interesOng”structure.

•  Efficiency:Closetoreal-Omeperformance.

Slid

e cr

edit

: Ba

stia

n Le

ibe

10-Oct-1616


ManyExisOngDetectorsAvailable

•  Hessian&Harris [Beaudet‘78],[Harris‘88]•  Laplacian,DoG [Lindeberg‘98],[Lowe‘99]•  Harris-/Hessian-Laplace [Mikolajczyk&Schmid‘01]•  Harris-/Hessian-Affine [Mikolajczyk&Schmid‘04]•  EBRandIBR [Tuytelaars&VanGool‘04]•  MSER [Matas‘02]•  SalientRegions [Kadir&Brady‘01]•  Others…

•  Thosedetectorshavebecomeabasicbuildingblockformanyrecentapplica8onsinComputerVision.

Slid

e cr

edit

: Ba

stia

n Le

ibe

10-Oct-1617


KeypointLocalizaOon

•  Goals:

–  RepeatabledetecOon–  PreciselocalizaOon–  InteresOngcontent

�Lookfortwo-dimensionalsignalchanges

Slid

e cr

edit

: Ba

stia

n Le

ibe

10-Oct-1618


FindingCorners

•  Keyproperty:–  Intheregionaroundacorner,imagegradienthastwoormoredominantdirecOons

•  Cornersarerepeatableanddis8nc8ve

C.Harris and M.Stephens. "A Combined Corner and Edge Detector.“ Proceedings of the 4th Alvey Vision Conference, 1988.

Slid

e cr

edit

: Sv

etla

na L

azeb

nik

10-Oct-1619


CornersasDisOncOveInterestPoints•  Designcriteria

– Weshouldeasilyrecognizethepointbylookingthroughasmallwindow(locality)

–  Shifingthewindowinanydirec8onshouldgivealargechangeinintensity(goodlocaliza8on)

“edge”: no change along the edge direction

“corner”: significant change in all directions

“flat” region: no change in all directions Sl

ide

cred

it:

Alyo

sha

Efro

s

10-Oct-1620


HarrisDetectorFormulaOon

•  Changeofintensityfortheshif[u,v]:E(u,v) = w(x, y) I (x +u, y + v)− I (x, y)"# $%

2

x ,y∑

Intensity Shifted intensity

Window function

or Window function w(x,y) =

Gaussian 1 in window, 0 outside

Slid

e cr

edit

: Ri

ck S

zelis

ki

10-Oct-1621


HarrisDetectorFormulaOon•  Thismeasureofchangecanbeapproximatedby:

whereMisa2�2matrixcomputedfromimagederivaOves:

⎥⎦

⎤⎢⎣

⎡≈

vu

MvuvuE ][),(

M

Sumoverimageregion–theareawearecheckingforcorner

Gradient with respect to x, times gradient with respect to y

2

2,( , ) x x y

x y x y y

I I IM w x y

I I I⎡ ⎤

= ⎢ ⎥⎢ ⎥⎣ ⎦

∑

Slid

e cr

edit

: Ri

ck S

zelis

ki

10-Oct-1622


HarrisDetectorFormulaOon

Ix Image I IxIy Iy

whereMisa2�2matrixcomputedfromimagederivaOves:

M

Sumoverimageregion–theareawearecheckingforcorner

Gradient with respect to x, times gradient with respect to y

2

2,( , ) x x y

x y x y y

I I IM w x y

I I I⎡ ⎤

= ⎢ ⎥⎢ ⎥⎣ ⎦

∑

Slid

e cr

edit

: Ri

ck S

zelis

ki

10-Oct-1623


WhatDoesThisMatrixReveal?•  First,let’sconsideranaxis-alignedcorner:

•  Thismeans:–  DominantgradientdirecOonsalignwithxoryaxis–  Ifeitherλiscloseto0,thenthisisnotacorner,solookforlocaOonswherebotharelarge.

•  Whatifwehaveacornerthatisnotalignedwiththeimageaxes?

⎥⎦

⎤⎢⎣

⎡=

⎥⎥⎦

⎤

⎢⎢⎣

⎡=

∑∑∑∑

2

12

2

00λ

λ

yyx

yxx

IIIIII

M

Slid

e cr

edit

: D

avid

Jac

obs

10-Oct-1624


WhatDoesThisMatrixReveal?•  First,let’sconsideranaxis-alignedcorner:

•  Thismeans:–  DominantgradientdirecOonsalignwithxoryaxis–  Ifeitherλiscloseto0,thenthisisnotacorner,solookforlocaOonswherebotharelarge.

•  Whatifwehaveacornerthatisnotalignedwiththeimageaxes?

⎥⎦

⎤⎢⎣

⎡=

⎥⎥⎦

⎤

⎢⎢⎣

⎡=

∑∑∑∑

2

12

2

00λ

λ

yyx

yxx

IIIIII

M

Slid

e cr

edit

: D

avid

Jac

obs

10-Oct-1625


GeneralCase

•  SinceMissymmetric,wehave

•  WecanvisualizeMasanellipsewithaxislengthsdeterminedbytheeigenvaluesandorientaOondeterminedbyR

RRM ⎥⎦

⎤⎢⎣

⎡= −

2

11

00λ

λ

Direction of the slowest change

Direction of the fastest change

(�max)-1/2

(�min)-1/2

adap

ted

from

Dar

ya F

rolo

va,

Den

is S

imak

ov

(Eigenvalue decomposition)

10-Oct-1626


InterpreOngtheEigenvalues•  ClassificaOonofimagepointsusingeigenvaluesofM:

�1

“Corner” �1and�2arelarge,�1 ~ �2;EincreasesinalldirecOons

�1and�2aresmall;EisalmostconstantinalldirecOons “Edge”

�1 >> �2

“Edge” �2 >> �1

“Flat”region

Slid

e cr

edit

: Kr

iste

n G

raum

an

�2

10-Oct-1627


CornerResponseFuncOon

•  FastapproximaOon–  AvoidcompuOngtheeigenvalues

–  α:constant(0.04to0.06)

�2

“Corner” θ > 0

“Edge” θ < 0

“Edge” θ < 0

“Flat”region

θ = det(M )−α trace(M )2 = λ1λ2 −α(λ1 +λ2 )2

�1 Slid

e cr

edit

: Kr

iste

n G

raum

an

10-Oct-1628


WindowFuncOonw(x,y)

•  OpOon1:uniformwindow–  Sumoversquarewindow

–  Problem:notrotaOoninvariant

•  OpOon2:SmoothwithGaussian–  Gaussianalreadyperformsweightedsum

–  ResultisrotaOoninvariant

1 in window, 0 outside

2

2,( , ) x x y

x y x y y

I I IM w x y

I I I⎡ ⎤

= ⎢ ⎥⎢ ⎥⎣ ⎦

∑

Gaussian

2

2,

x x y

x y x y y

I I IM

I I I⎡ ⎤

= ⎢ ⎥⎢ ⎥⎣ ⎦

∑

2

2( ) x x y

x y y

I I IM g

I I Iσ

⎡ ⎤= ∗⎢ ⎥

⎢ ⎥⎣ ⎦

Slid

e cr

edit

: Ba

stia

n Le

ibe

10-Oct-1629


Summary:HarrisDetector[Harris88]•  Computesecondmomentmatrix

(autocorrelaOonmatrix)1. Image derivatives

Ix Iy

2. Square of derivatives

Ix2 Iy2 IxIy

3. Gaussian filter g(sI) g(Ix2) g(Iy2) g(IxIy)

R

2

2

( ) ( )( , ) ( )

( ) ( )x D x y D

I D Ix y D y D

I I IM g

I I Iσ σ

σ σ σσ σ

⎡ ⎤= ∗⎢ ⎥

⎢ ⎥⎣ ⎦

2 2 2 2 2 2( ) ( ) [ ( )] [ ( ) ( )]x y x y x yg I g I g I I g I g Iα= − − +θ = det[M (σ I ,σ D )]−α[trace(M (σ I ,σ D ))]

2

4. Cornerness function – two strong eigenvalues

5. Perform non-maximum suppression

Slid

e cr

edit

: Kr

ysti

an M

ikol

ajcz

yk

10-Oct-1630


HarrisDetector:Workflow

Slid

e ad

apte

d fr

om D

arya

Fro

lova

, D

enis

Sim

akov

10-Oct-1631


HarrisDetector:Workflow-computercornerresponsesθ

Slid

e ad

apte

d fr

om D

arya

Fro

lova

, D

enis

Sim

akov

10-Oct-1632


Slid

e ad

apte

d fr

om D

arya

Fro

lova

, D

enis

Sim

akov

HarrisDetector:Workflow-Takeonlythelocalmaximaofθ,whereθ>threshold

10-Oct-1633


HarrisDetector:Workflow-ResulOngHarrispoints

Slid

e ad

apte

d fr

om D

arya

Fro

lova

, D

enis

Sim

akov

10-Oct-1634


HarrisDetector–Responses[Harris88]

Effect: A very precise corner detector.

Slid

e cr

edit

: Kr

ysti

an M

ikol

ajcz

yk

10-Oct-1635



Slid

e cr

edit

: Kr

ysti

an M

ikol

ajcz

yk

10-Oct-1636



Slid

e cr

edit

: Kr

iste

n G

raum

an

•  Resultsarewellsuitedforfindingstereocorrespondences

10-Oct-1637


HarrisDetector:ProperOes

•  TranslaOoninvariance?

Slid

e cr

edit

: Kr

iste

n G

raum

an

10-Oct-1638



•  TranslaOoninvariance•  RotaOoninvariance?

Ellipse rotates but its shape (i.e. eigenvalues) remains the same

Corner response θ is invariant to image rotation

Slid

e cr

edit

: Kr

iste

n G

raum

an

10-Oct-1639



•  TranslaOoninvariance•  RotaOoninvariance•  Scaleinvariance?

Not invariant to image scale!

All points will be classified as edges!

Corner

Slid

e cr

edit

: Kr

iste

n G

raum

an

10-Oct-1640


Whatwehavelearnedtoday?





10-Oct-1641

Nextlecture(#7)

Somebackgroundreading:RickSzeliski,Chapter14.1.1;DavidLowe,IJCV2004

Lecture 6 - Fei-Fei Li 10-Oct-1642


ApplicaOon:ImageSOtching

Slid

e cr

edit

: D

arya

Fro

lova

, D

enis

Sim

akov

10-Oct-1643



Slid

e cr

edit

: D

arya

Fro

lova

, D

enis

Sim

akov

10-Oct-1644

•  Procedure:– Detectfeaturepointsinbothimages–  Findcorrespondingpairs– Usethesepairstoaligntheimages



Slid

e cr

edit

: D

arya

Fro

lova

, D

enis

Sim

akov

10-Oct-1645





Slid

e cr

edit

: D

arya

Fro

lova

, D

enis

Sim

akov

10-Oct-1646

Lecture 6: Finding Features (part 1/2)vision.stanford.edu/.../lectures/lecture6_detectors...Fei-Fei...

Documents

Transcript of Lecture 6: Finding Features (part 1/2)vision.stanford.edu/.../lectures/lecture6_detectors...Fei-Fei...