Lecture 5&6 –Finding Features, Affine Invariance,...

CAP5415-ComputerVisionLecture5and6-FindingFeatures,

AffineInvariance,SIFT

[email protected]

Lecture5&6–FindingFeatures,AffineInvariance,SIFT

1

Outline

• ConceptofScale– Pyramids– Scale-spaceapproachesbriefly

• Scaleinvariantregionselection• SIFT:animageregiondescriptor

• DavidLowe,IJCV2004paper[Pleasereadit!]


2

Reminder:Motivation

• ImageMatching– Fundamentalaspectofmanyproblems

• ObjectRecognition• 3DStructures• StereoCorrespondence• MotionTracking• ….


3

Reminder:Motivation

• ImageMatching– Fundamentalaspectofmanyproblems

• ObjectRecognition• 3DStructures• StereoCorrespondence• MotionTracking• ….

Whatarethedesiredfeaturestoconductthesetasks?


4

Review oftheCornerDetectionLecture5&6–FindingFeatures,AffineInvariance,SIFT

55

rotation

translation

scaling

Cornersareinvariantto

✔

✔

✖

SCALE

• Theextrema inasignalanditsfirstafewderivatives provideausefulgeneralpurposedescriptionformanykindsofsignals– Edges– Corners


66

SCALE


• Inphysics,objectsliveonarangeofscales.


77

SCALE


• Inphysics,objectsliveonarangeofscales.• Neighborhoodoperationscanonlyextractlocalfeatures(atascaleofatmostafewpixels),however,imagescontaininformationatlargerscales


88

Scale-Space

• Anygivenimagecancontainobjectsthatexistatscalesdifferentfromotherobjectsinthesameimage


9

Scale-Space


• Or,thatevenexistatmultiplescalessimultaneously


10

Scale-Space


• Or,thatevenexistatmultiplescalessimultaneously


11

Multi-scaleRepresentationLecture5&6–FindingFeatures,AffineInvariance,SIFT

12

SmallScaleFeatures

SmallerFilterMasks

LargeScaleFeatures

LargerFilterMasks

Image

Multi-scaleRepresentationLecture5&6–FindingFeatures,AffineInvariance,SIFT

13

SmallScaleFeatures

SmallerFilterMasks

LargeScaleFeatures

LargerFilterMasks

Image

ComputationalCostincreases!

Doublingthescaleleadstofour-foldincreaseinthenumberofoperationsin2D.

Multi-ScaleRepresentation:Pyramid

• Pyramid isonewaytorepresentimagesinmulti-scale– Pyramidisbuiltbyusingmultiplecopiesofimage.– Eachlevelinthepyramidis1/4ofthesizeofpreviouslevel.

– Thelowestlevelisofthehighestresolution.– Thehighestlevelisofthelowestresolution.


14

Multi-ScaleRepresentation:PyramidLecture5&6–FindingFeatures,AffineInvariance,SIFT

15

Pyramidcancaptureglobalandlocalfeatures


16

Gaussian PyramidsLecture5&6–FindingFeatures,AffineInvariance,SIFT

17Source: Forsyth

Laplacian PyramidsLecture5&6–FindingFeatures,AffineInvariance,SIFT

18Source: Forsyth


19

1

1

-2 1 1-2

Laplacian ofGaussian(LoG):GaussianfirsttosmoothimagesthenperformLaplacian operation

52(G� ⇤ I) = I ⇤ 52G�


20


21

�G

�

�x

(x, y) = � x

2⇡�4e

�(x2+y

2)/(2�2)


22

�G

�

�x

(x, y) = � x

2⇡�4e

�(x2+y

2)/(2�2)

52G

�

(x, y) =1

2⇡�4

(x2 + y

2 � 2�2)

�

2e

�(x2+y

2)/(2�2)

Repeatsamederivativeforycomponent,andthentakesecondderivatives.

Laplacian PyramidConstructionLecture5&6–FindingFeatures,AffineInvariance,SIFT

23

Laplacian PyramidConstructionLecture5&6–FindingFeatures,AffineInvariance,SIFT

24

Decimation andInterpolationLecture5&6–FindingFeatures,AffineInvariance,SIFT

25

LowPass filter(i.e.,Gaussian)


26


y(n) = x(n) ⇤ h(n) =X

k

h(k)x(n� k)


27


y(n) = x(n) ⇤ h(n) =X

k

h(k)x(n� k)

z(n) = y(2n)


28


y(n) = x(n) ⇤ h(n) =X

k

h(k)x(n� k)

z(n) = y(2n)

z(n) =X

k

h(k)x(2n� k)

DifferenceofGaussian(DoG)

• ItisacommonapproximationofLoG (betterruntime).


29




30

D�,↵(x, y) = L(x, y,↵)� L(x, y,↵�)



• ItisthedifferencebetweenablurredcopyofimageIandanevenmoreblurredcopyofI.


31

D�,↵(x, y) = L(x, y,↵)� L(x, y,↵�)



• ItisthedifferencebetweenablurredcopyofimageIandanevenmoreblurredcopyofI.


32

D�,↵(x, y) = L(x, y,↵)� L(x, y,↵�)

5G�(x, y) ⇡G↵�(x, y)�G�(x, y)

(↵� 1)�2

ScaleSelection-AutomatedLecture5&6–FindingFeatures,AffineInvariance,SIFT

33

• Find scale that gives local maxima of some function f in both position and scale.

f

region size

Image 1f

region size

Image 2

s1 s2K. Grauman,

GoodFunction?Lecture5&6–FindingFeatures,AffineInvariance,SIFT

34

• A “good” function for scale detection:has one stable sharp peak

f

region size

bad

f

region size

bad

f

region size

Good !

• Forusualimages:agoodfunctionwouldbeaonewhichrespondstocontrast(sharplocalintensitychange)

PatchSizeCorrespondingtoScaleLecture5&6–FindingFeatures,AffineInvariance,SIFT

35

)),(( )),((11

ss ¢¢= xIfxIfmm iiii !!

How to find corresponding patch sizes?


36K.Grauman,B.Leibe)),((

1sxIf

mii ! )),((1

sxIfmii ¢!



1sxIf

mii ! )),((1

sxIfmii ¢!


41

• Functionresponsesforincreasingscale(scalesignature)

K.Grauman,B.Leibe)),((

1sxIf

mii ! )),((1

s ¢¢xIfmii !

NormalizationLecture5&6–FindingFeatures,AffineInvariance,SIFT

42

ScaleInvariantFeatureTransform(SIFT)

• Lowe.,D.2004,IJCV


43cited>43K


• Showsexistence,importance,andvalueofinvarianttypesofdetector

• Demonstratestherichnessthatfeaturedescriptorscanbringtofeaturematching

InsteadofusingLoG (Laplacian ofGaussian) likeinHarrisandHessian-basedoperators,SIFTuses

DoG (DifferenceofGaussian).


44


• Image content is transformed into local feature coordinates that are invariant to translation, rotation, scale, and other imaging parameters


45


• Image content is transformed into local feature coordinates that are invariant to translation, rotation, scale, and other imaging parameters


46

OverallProcedureataHighLevelLecture5&6–FindingFeatures,AffineInvariance,SIFT

47

Scale-SpaceExtremaDetection

Searchovermultiplescalesandimagelocations

KeyPointLocalization

Fitamodeltodeterminelocationandscale.SelectKeyPoints basedona

measureofstability.

OrientationAssignment

Computebestorientation(s)foreachkeyPoint region.

KeyPointDescription

Uselocalimagegradientsatselectedscaleandrotationtodescribeeach

keyPoint region.

SIFT DescriptorLecture5&6–FindingFeatures,AffineInvariance,SIFT

48

Basic idea:• Take n x n (i.e., n=16) square window (around a feature/interest point)• Divide them into m x m (i.e., m=4) cells• Compute gradient orientation for each cell• Create histogram over edge orientations weighted by magnitude

0 2anglehistogram


49

16histogramsx8orientations=128features


50

Full version• Divide the 16x16 window into a 4x4 grid of cells (2x2 case shown below)• Compute an orientation histogram for each cell• 16 cells * 8 orientations = 128 dimensional descriptor• Threshold normalize the descriptor:

suchthat:

0.2

PropertiesofSIFTLecture5&6–FindingFeatures,AffineInvariance,SIFT

51

Extraordinarilyrobustmatchingtechnique

– Canhandlechangesinviewpoint– Canhandlesignificantchangesinillumination– Efficient(realtime)– Lotsofcodeavailable

• http://people.csail.mit.edu/albert/ladypack/wiki/index.php/Known_implementations_of_SIFT

Revisit SIFTStepsLecture5&6–FindingFeatures,AffineInvariance,SIFT

52

(1) Scale-space extrema detection– Extract scale and rotation invariant interest points (i.e.,

keypoints). (2) KeyPoint localization

– Determine location and scale for each interest point.– Eliminate “weak” keypoints

(3) Orientation assignment– Assign one or more orientations to each keypoint.

(4) KeyPoint descriptor– Use local image gradients at the selected scale.

(1)Scale-SpaceExtrema DetectionLecture5&6–FindingFeatures,AffineInvariance,SIFT

53

Wewanttofindpointsthatgiveusinformationabouttheobjectsintheimage.

Theinformationabouttheobjectsisintheobject’sedges.

Wewillrepresenttheimageinawaythatgivesustheseedgesasthisrepresentationsextrema points.


54

• Harris-Laplace

Find local maxima of:– Harris detector in space – LoG in scale

• SIFT

Find local maxima of:– Hessian in space – DoG in scale

scale

x

y

Hessian

DoG

scale

x

y

¬ Harris ®

LoG


55


56

• Extractlocalextrema(i.e.,minimaormaxima)inDoG pyramid.

-Compareeachpointtoits8neighborsatthesamelevel,9neighborsinthelevelabove,and9neighborsinthelevelbelow(i.e.,26total).

( )sD

( )skD

( )s2kD


57

• Laplacian of Gaussian kernel– Scale normalized (x by scale2)– Proposed by Lindeberg

• Scale-space detection– Find local maxima across scale/space– A good “blob” detector

(2)KeyPoint Localization

• There are still a lot of points, some of them are not good enough.

• The locations of keypoints may be not accurate.

• Eliminating edge points.


58

(2)KeyPoint LocalizationLecture5&6–FindingFeatures,AffineInvariance,SIFT

� Reject (1) pointswithlowcontrast(flat)(2)poorlylocalizedalonganedge(edge)

(2)KeyPoint LocalizationLecture5&6–FindingFeatures,AffineInvariance,SIFT

� Reject (1) pointswithlowcontrast(flat)(2)poorlylocalizedalonganedge(edge)

if

Reject

WhereandpracticallySIFTusesr=10.

Tr(H)

|H| <(r + 1)2

r

r =�1

�2

InaccurateKeyPoint LocalizationLecture5&6–FindingFeatures,AffineInvariance,SIFT

61

xSampling

DetectedExtrema

TrueExtrema

Poorcontrast

InaccurateKeyPoint LocalizationLecture5&6–FindingFeatures,AffineInvariance,SIFT

62

TheSolution:

( ) xxDxx

xDDxD

TT

T !!

!!!

!2

2

21

¶¶

+¶¶

+=

Taylorexpansion:

Minimizetofindaccurateextrema:

xD

xDx !!

¶¶

¶¶

-=-1

2

2

ˆ

Brown&Lowe2002

Ifoffsetfromsamplingpointislargerthan0.5-Keypoint shouldbeinadifferentsamplingpoint.


63

Localextremas


64

Localextremas Removelowcontrastfeatures


65

Localextremas Removelowedges

SIFTDescriptorLecture5&6–FindingFeatures,AffineInvariance,SIFT

66

(3)OrientationAssignmentLecture5&6–FindingFeatures,AffineInvariance,SIFT

67

m(x, y) = (L(x +1, y)− L(x −1, y))2 + (L(x, y +1)− L(x, y −1))2

θ(x, y) = a tan2((L(x, y +1)− L(x, y −1)) / (L(x +1, y)− L(x −1, y)))

36 bins (i.e., 10o per bin)

• Histogramentriesare weighted by(i)gradientmagnitudeand(ii)aGaussianfunctionwithσ equalto1.5timesthescaleofthekeypoint.

0 2pi

L(x, y,σ ) = G(x, y,σ )* I(x, y)

Create histogram of gradient directions, within a region around the keyPoint, at selected scale:


68


69


70

(3)OrientationAssignment


71


72

(4)KeyPoint DescriptorLecture5&6–FindingFeatures,AffineInvariance,SIFT

73

• Partial Voting: distribute histogram entries into adjacent bins (i.e., additional robustness to shifts)– Each entry is added to all bins, multiplied by a weight of 1-d,

where d is the distance from the bin it belongs.

(4)KeyPoint DescriptorLecture5&6–FindingFeatures,AffineInvariance,SIFT

74

• Descriptordependsontwomainparameters:– (1)numberoforientations– nxnarrayoforientationhistograms

EvaluatingResultsLecture5&6–FindingFeatures,AffineInvariance,SIFT

75

0.7

Howcanwemeasuretheperformanceofafeaturematcher?

0 1

1

false positive rate

truepositive

rate# true positives matched

# true positives

0.1# false positives matched

# true negatives

features that really do have a match

the matcher correctlyfound a match

the matcher said yes when the right answer was no

EvaluatingResultsLecture5&6–FindingFeatures,AffineInvariance,SIFT

76

0.7

Howcanwemeasuretheperformanceofafeaturematcher?

0 1

1

false positive rate

truepositive

rate# true positives matched

# true positives

0.1# false positives matched

# true negatives

features that really do have a match

the matcher correctlyfound a match

the matcher said yes when the right answer was no

ROC curve (“Receiver Operator Characteristic”)

ChangeofIlluminationLecture5&6–FindingFeatures,AffineInvariance,SIFT

77

Change of brightness => doesn’t effect gradients (difference of pixels value).

Change of contrast => doesn’t effect gradients (up to normalization).Saturation (non-linear change of illumination) =>affects magnitudes much more than orientation.=> Threshold gradient magnitudes to 0.2 and

renormalize.

Illuminationinvariance?Lecture5&6–FindingFeatures,AffineInvariance,SIFT

78

1. Beforeyoustart,youcannormalizetheimages.2. Differencebasedmetricscanbeused(Haar,SIFT,…)


79


80

• Extractfeatures


81

• Extractfeatures• Computeputativematches


82

• Extractfeatures• Computeputativematches• Loop:

– Hypothesize transformationT(smallgroupofputativematchesthatarerelatedbyT)


83



– Verifytransformation(searchforothermatchesconsistentwithT)


84



– Verifytransformation(searchforothermatchesconsistentwithT)

ObjectRecognitionLecture5&6–FindingFeatures,AffineInvariance,SIFT

85

Fortrainingimages:ExtractingkeypointsbySIFT.Creatingdescriptorsdatabase.

Performingdetailedgeometricfitcheckforeachcluster.

Forqueryimages:ExtractingkeypointsbySIFT.Foreachdescriptor- findingnearestneighborinDB.

Findingclusterofat-least3keypoints.

ImageRegistrationLecture5&6–FindingFeatures,AffineInvariance,SIFT

86

Lecture 5&6 –Finding Features, Affine Invariance,...

Documents

Transcript of Lecture 5&6 –Finding Features, Affine Invariance,...