Lecture 7: Finding Features (part 2/2)vision.stanford.edu/teaching/cs131_fall1617/... · Fei-Fei Li...

Lecture 7 - Fei-Fei Li

Lecture7:FindingFeatures(part2/2)

Dr.JuanCarlosNieblesStanfordAILab

ProfessorFei-FeiLiStanfordVisionLab

17-Oct-161


Whatwewilllearntoday?

•  Localinvariantfeatures– MoOvaOon–  Requirements,invariances

•  KeypointlocalizaOon– Harriscornerdetector

•  ScaleinvariantregionselecOon– AutomaOcscaleselecOon– Difference-of-Gaussian(DoG)detector

•  SIFT:animageregiondescriptor

17-Oct-162

Previouslecture(#6)

Somebackgroundreading:DavidLowe,IJCV2004


Aquickreview

•  Localinvariantfeatures– MoOvaOon– Requirements,invariances




17-Oct-163


GeneralApproachN

pix

els

N pixels

Similarity measure Af

e.g. color

Bf

e.g. color

B1B2

B3A1

A2 A3

Tffd BA <),(

1. Find a set of distinctive key- points

3. Extract and normalize the region content

2. Define a region around each keypoint

4. Compute a local descriptor from the normalized region

5. Match local descriptors

Slid

e cr

edit

: Ba

stia

n Le

ibe

17-Oct-164


Aquickreview





17-Oct-165


Quickreview:HarrisCornerDetector

17-Oct-166

“edge”: no change along the edge direction

“corner”: significant change in all directions

“flat” region: no change in all directions

Slid

e cr

edit

: Al

yosh

a Ef

ros



17-Oct-167

•  FastapproximaOon–  AvoidcompuOngtheeigenvalues

–  α:constant(0.04to0.06)

θ = det(M )−α trace(M )2 = λ1λ2 −α(λ1 +λ2 )2

Slid

e cr

edit

: Kr

iste

n G

raum

an

“Corner” θ > 0

“Edge” θ < 0

“Edge” θ < 0

“Flat”region

�1



17-Oct-168

Slid

e ad

apte

d fr

om D

arya

Fro

lova

, D

enis

Sim

akov



17-Oct-169

•  TranslaOoninvariance•  RotaOoninvariance•  Scaleinvariance?

Not invariant to image scale!

All points will be classified as edges!

Corner

Slid

e cr

edit

: Kr

iste

n G

raum

an







17-Oct-1610


ScaleInvariantDetecOon•  Considerregions(e.g.circles)ofdifferentsizesaroundapoint

•  Regionsofcorrespondingsizeswilllookthesameinbothimages

17-Oct-1611


ScaleInvariantDetecOon

•  Theproblem:howdowechoosecorrespondingcirclesindependentlyineachimage?

17-Oct-1612


ScaleInvariantDetecOon•  SoluOon:

–  DesignafuncOonontheregion(circle),whichis“scaleinvariant”(thesameforcorrespondingregions,eveniftheyareatdifferentscales)

Example:averageintensity.Forcorrespondingregions(evenofdifferentsizes)itwillbethesame.

scale = 1/2

–  Forapointinoneimage,wecanconsideritasafuncOonofregionsize(circleradius)

f

region size

Image 1 f

region size

Image 2

17-Oct-1613


ScaleInvariantDetecOon•  Commonapproach:

scale = 1/2

f

region size

Image 1 f

region size

Image 2

TakealocalmaximumofthisfuncOon

•  ObservaOon:regionsize,forwhichthemaximumisachieved,shouldbeco-variantwithimagescale.

s1 s2

Important: this scale invariant region size is found in each image independently!

17-Oct-1614


ScaleInvariantDetecOon

•  A“good”funcOonforscaledetecOon:hasonestablesharppeak

f

region size

bad

f

region size

bad

f

region size

Good !

•  Forusualimages:agoodfuncOonwouldbeonewhichrespondstocontrast(sharplocalintensitychange)

17-Oct-1615


ScaleInvariantDetecOon•  FuncOonsfordeterminingscale

2 2

21 22

( , , )x y

G x y e σπσ

σ+−

=

( )2 ( , , ) ( , , )xx yyL G x y G x yσ σ σ= +

( , , ) ( , , )DoG G x y k G x yσ σ= −

Kernel Imagef = ∗Kernels:

where Gaussian

Note: both kernels are invariant to scale and rotation

(Laplacian)

(Difference of Gaussians)

17-Oct-1616


ScaleInvariantDetectors•  Harris-Laplacian1

Findlocalmaximumof:–  Harriscornerdetectorinspace(imagecoordinates)

–  Laplacianinscale

1K.Mikolajczyk,C.Schmid.“IndexingBasedonScaleInvariantInterestPoints”.ICCV20012D.Lowe.“DisOncOveImageFeaturesfromScale-InvariantKeypoints”.IJCV2004

scale

x

y

� Harris �

� L

apla

cian

�

•  SIFT(Lowe)2Findlocalmaximumof:–  DifferenceofGaussiansinspace

andscale

scale

x

y

� DoG �

� D

oG �

17-Oct-1617


ScaleInvariantDetectors

K.Mikolajczyk, C.Schmid. “Indexing Based on Scale Invariant Interest Points”. ICCV 2001

•  ExperimentalevaluaOonofdetectorsw.r.t.scalechange

Repeatability rate: # correspondences # possible correspondences

17-Oct-1618


ScaleInvariantDetecOon:Summary

•  Given:twoimagesofthesamescenewithalargescaledifferencebetweenthem

•  Goal:findthesameinterestpointsindependentlyineachimage

•  SoluOon:searchformaximaofsuitablefuncOonsinscaleandinspace(overtheimage)

Methods:

1.  Harris-Laplacian [Mikolajczyk, Schmid]: maximize Laplacian over scale, Harris’ measure of corner response over the image

2.  SIFT [Lowe]: maximize Difference of Gaussians over scale and space

17-Oct-1619







17-Oct-1620


LocalDescriptors•  Weknowhowtodetectpoints•  NextquesOon:

Howtodescribethemformatching?

? Point descriptor should be:

1.   Invariant 2.   Distinctive Sl

ide

cred

it:

Kris

ten

Gra

uman

17-Oct-1621


CVPR2003Tutorial

Recogni5onandMatchingBasedonLocalInvariantFeatures

DavidLoweComputerScienceDepartmentUniversityofBriOshColumbia

17-Oct-1622

Lecture 7 -

InvariantLocalFeatures

•  ImagecontentistransformedintolocalfeaturecoordinatesthatareinvarianttotranslaOon,rotaOon,scale,andotherimagingparameters

17-Oct-1623


Advantagesofinvariantlocalfeatures•  Locality:featuresarelocal,sorobusttoocclusionandclumer(nopriorsegmentaOon)

•  Dis5nc5veness:individualfeaturescanbematchedtoalargedatabaseofobjects

•  Quan5ty:manyfeaturescanbegeneratedforevensmallobjects

•  Efficiency:closetoreal-Omeperformance

•  Extensibility:caneasilybeextendedtowiderangeofdifferingfeaturetypes,witheachaddingrobustness

17-Oct-1624

Lecture 7 -

ScaleinvarianceRequiresamethodtorepeatablyselectpointsinloca5on

andscale:•  Theonlyreasonablescale-spacekernelisaGaussian

(Koenderink,1984;Lindeberg,1994)•  Anefficientchoiceistodetectpeaksinthedifferenceof

Gaussianpyramid(Burt&Adelson,1983;Crowley&Parker,1984–butexaminingmorescales)

•  Difference-of-GaussianwithconstantraOoofscalesisacloseapproximaOontoLindeberg’sscale-normalizedLaplacian(canbeshownfromtheheatdiffusionequaOon)

Blur

Resample

Subtract

Blur

Resample

Subtract

17-Oct-16


BecomingrotaOoninvariant

•  WearegivenakeypointanditsscalefromDoG

•  WewillselectacharacterisOcorientaOonforthekeypoint(basedonthemostprominentgradientthere;discussednextslide)

•  Wewilldescribeallfeaturesrela5vetothisorientaOon

•  CausesfeaturestoberotaOoninvariant!–  Ifthekeypointappearsrotatedin

anotherimage,thefeatureswillbethesame,becausethey’rerela5vetothecharacterisOcorientaOon

0 2π

17-Oct-1626


BecomingrotaOoninvariant•  ChoosingcharacterisOcorientaOon:•  Usetheblurredimageassociatedwith

thekeypoint’sscale.Lookatpixelsinasquarearoundit(say,size16x16)

•  ComputegradientdirecOonateachpixel(thisiseasy,usingverOcalandhorizontaledgefilters)

•  CreateahistogramoftheselocalgradientdirecOons

•  KeypointorientaOon=thepeakofthathistogram

•  Minordetails:we’llalsoweighteachpixel’shistogramcontribuOonbythemagnitudeofitsgradientandhowcloseitistothekeypoint

•  Now,eachkeypointhasstable2Dcoordinates(x,y,scale,orientaBon).Nowwemustgiveita“fingerprint.” 0 2π

17-Oct-1627


ExampleofkeypointdetecOonThresholdonvalueatDOGpeakandonraOoofprinciplecurvatures(Harrisapproach)

(a)233x189image(b)832DOGextrema(c)729letaterpeakvaluethreshold(d)536letatertesOngraOoofprinciplecurvatures

17-Oct-1628

Vectorsindicatescale,orientaOonandlocaOon.


SIFTdescriptorformaOon

•  Usetheblurredimageassociatedwiththekeypoint’sscale•  Takeimagegradientsoverthekeypointneighborhood.•  TobecomerotaOoninvariant,rotatethegradientdirecOonsANDlocaOons

by(-keypointorientaOon)–  Nowwe’vecancelledoutrotaOonandhavegradientsexpressedatlocaOonsrela5ve

tokeypointorientaOonθ–  Wecouldalsohavejustrotatedthewholeimageby-θ,butthatwouldbeslower.

17-Oct-1629



•  UsingprecisegradientlocaOonsisfragile.We’dliketoallowsome“slop”intheimage,andsOllproduceaverysimilardescriptor

•  CreatearrayoforientaOonhistograms(a4x4arrayisshown)•  PuttherotatedgradientsintotheirlocalorientaOonhistograms

–  Agradients’scontribuOonisdividedamongthenearbyhistogramsbasedondistance.Ifit’shalfwaybetweentwohistogramlocaOons,itgivesahalfcontribuOontoboth.

–  Also,scaledowngradientcontribuOonsforgradientsfarfromthecenter•  TheSIFTauthorsfoundthatbestresultswerewith8orientaOonbinsper

histogram,anda4x4histogramarray.

17-Oct-1630

0 2 p



•  UsingprecisegradientlocaOonsisfragile.We’dliketoallowsome“slop”intheimage,andsOllproduceaverysimilardescriptor

•  CreatearrayoforientaOonhistograms(a4x4arrayisshown)•  PuttherotatedgradientsintotheirlocalorientaOonhistograms

–  Agradients’scontribuOonisdividedamongthenearbyhistogramsbasedondistance.Ifit’shalfwaybetweentwohistogramlocaOons,itgivesahalfcontribuOontoboth.

–  Also,scaledowngradientcontribuOonsforgradientsfarfromthecenter•  TheSIFTauthorsfoundthatbestresultswerewith8orientaOonbinsper

histogram,anda4x4histogramarray.

17-Oct-1631



•  8orientaOonbinsperhistogram,anda4x4histogramarray,yields8x4x4=128numbers.

•  SoaSIFTdescriptorisalength128vector,whichisinvarianttorotaOon(becausewerotatedthedescriptor)andscale(becauseweworkedwiththescaledimagefromDoG)

•  WecancompareeachvectorfromimageAtoeachvectorfromimageBtofindmatchingkeypoints!–  Euclidean“distance”betweendescriptorvectorsgivesagoodmeasure

ofkeypointsimilarity

17-Oct-1632



•  AddingrobustnesstoilluminaOonchanges:•  Rememberthatthedescriptorismadeofgradients(differences

betweenpixels),soit’salreadyinvarianttochangesinbrightness(e.g.adding10toallimagepixelsyieldstheexactsamedescriptor)

•  Ahigher-contrastphotowillincreasethemagnitudeofgradientslinearly.So,tocorrectforcontrastchanges,normalizethevector(scaletolength1.0)

•  Verylargeimagegradientsareusuallyfromunreliable3DilluminaOoneffects(glare,etc).So,toreducetheireffect,clampallvaluesinthevectortobe≤0.2(anexperimentallytunedvalue).Thennormalizethevectoragain.

•  ResultisavectorwhichisfairlyinvarianttoilluminaOonchanges.

17-Oct-1633


Repeatabilityvsnumberofscalessampledperoctave

DavidG.Lowe,"DisOncOveimagefeaturesfromscale-invariantkeypoints,"InternaOonalJournalofComputerVision,60,2(2004),pp.91-110

17-Oct-1634


SensiOvitytonumberofhistogramorientaOons

17-Oct-1635

DavidG.Lowe,"DisOncOveimagefeaturesfromscale-invariantkeypoints,"InternaOonalJournalofComputerVision,60,2(2004),pp.91-110


Featurestabilitytonoise•  Matchfeaturesaterrandomchangeinimagescale&

orientaOon,withdifferinglevelsofimagenoise•  Findnearestneighborindatabaseof30,000features

17-Oct-1636


Featurestabilitytoaffinechange•  Matchfeaturesaterrandomchangeinimagescale&

orientaOon,with2%imagenoise,andaffinedistorOon•  Findnearestneighborindatabaseof30,000features

17-Oct-1637


DisOncOvenessoffeatures•  Varysizeofdatabaseoffeatures,with30degreeaffinechange,

2%imagenoise•  Measure%correctforsinglenearestneighbormatch

17-Oct-1638


RaOoofdistancesreliableformatching

17-Oct-1639

Lecture 7 - Fei-Fei Li 17-Oct-1640

Lecture 7 - Fei-Fei Li 17-Oct-1641


NiceSIFTresources

•  VLFeattoolbox:– hmp://www.vlfeat.org/overview/sit.html

•  anonlinetutorial:hmp://www.aishack.in/2010/05/sit-scale-invariant-feature-transform/

•  Wikipedia:hmp://en.wikipedia.org/wiki/Scale-invariant_feature_transform

17-Oct-1642


ApplicaOonsoflocalinvariantfeatures

•  Widebaselinestereo•  MoOontracking•  Panoramas•  MobilerobotnavigaOon•  3DreconstrucOon•  RecogniOon•  …


AutomaOcmosaicing

hmp://www.cs.ubc.ca/~mbrown/autosOtch/autosOtch.html


Widebaselinestereo

[ImagefromT.TuytelaarsECCV2006tutorial]


RecogniOonofspecificobjects,scenes

Rothganger et al. 2003 Lowe 2002

Schmid and Mohr 1997 Sivic and Zisserman, 2003


Whatwehavelearnedthisweek?





17-Oct-1647

Previouslecture(#6)

today(#7)

Somebackgroundreading:R.Szeliski,Ch4.1.1;DavidLowe,IJCV2004

Lecture 7: Finding Features (part 2/2)vision.stanford.edu/teaching/cs131_fall1617/... · Fei-Fei Li...

Documents

Transcript of Lecture 7: Finding Features (part 2/2)vision.stanford.edu/teaching/cs131_fall1617/... · Fei-Fei Li...