Lecture’13:’’ Tracking’mo3on’features’’...

Lecture 13 - !!!

Fei-Fei Li!

Lecture 13: Tracking mo3on features

– op3cal flow

Professor Fei-‐Fei Li Stanford Vision Lab

30-‐Oct-‐12 1

Lecture 13 - !!!

Fei-Fei Li!

What we will learn today?

•  Introduc3on •  Op3cal flow •  Feature tracking •  Applica3ons •  (Problem Set 3 (Q1))

30-‐Oct-‐12 2

Lecture 13 - !!!

Fei-Fei Li!

From images to videos •  A video is a sequence of frames captured over 3me •  Now our image data is a func3on of space (x, y) and 3me (t)

30-‐Oct-‐12 3

Lecture 13 - !!!

Fei-Fei Li!

Mo3on es3ma3on techniques

•  Op3cal flow –  Recover image mo3on at each pixel from spa3o-‐temporal

image brightness varia3ons (op3cal flow)

•  Feature-‐tracking –  Extract visual features (corners, textured areas) and “track”

them over mul3ple frames

30-‐Oct-‐12 4

Lecture 13 - !!!

Fei-Fei Li!

Picture courtesy of Selim Temizer -‐ Learning and Intelligent Systems (LIS) Group, MIT

Op3cal flow

Vector field func3on of the spa3o-‐temporal image brightness varia3ons

30-‐Oct-‐12 5

Lecture 13 - !!!

Fei-Fei Li!

Feature-‐tracking

Courtesy of Jean-‐Yves Bouguet – Vision Lab, California Ins3tute of Technology

30-‐Oct-‐12 6

Lecture 13 - !!!

Fei-Fei Li!

Feature-‐tracking

Courtesy of Jean-‐Yves Bouguet – Vision Lab, California Ins3tute of Technology

30-‐Oct-‐12 7

Lecture 13 - !!!

Fei-Fei Li!

Op3cal flow •  Defini3on: op3cal flow is the apparent mo3on of brightness paderns in the image

•  Note: apparent mo3on can be caused by ligh3ng changes without any actual mo3on –  Think of a uniform rota3ng sphere under fixed ligh3ng vs. a sta3onary sphere under moving illumina3on

GOAL: Recover image mo3on at each pixel from op3cal flow

Source: Silvio Savarese

30-‐Oct-‐12 8

Lecture 13 - !!!

Fei-Fei Li!

Es3ma3ng op3cal flow

•  Given two subsequent frames, es3mate the apparent mo3on field u(x,y), v(x,y) between them

•  Key assump3ons •  Brightness constancy: projec3on of the same point looks the same in

every frame •  Small mo7on: points do not move very far •  Spa7al coherence: points move like their neighbors

I(x,y,t–1) I(x,y,t)


30-‐Oct-‐12 9

Lecture 13 - !!!

Fei-Fei Li!

tyx IyxvIyxuItyxItuyuxI +⋅+⋅+−≈++ ),(),()1,,(),,(

•  Brightness Constancy Equa3on: ),()1,,( ),,(),( tyxyx vyuxItyxI ++=−

Linearizing the right side using Taylor expansion:

The brightness constancy constraint

I(x,y,t–1) I(x,y,t)

0≈+⋅+⋅ tyx IvIuIHence,

Image deriva3ve along x

[ ] 0IvuI tT =+⋅∇→

tyx IyxvIyxuItyxItuyuxI +⋅+⋅=−−++ ),(),()1,,(),,(


30-‐Oct-‐12 10

Lecture 13 - !!!

Fei-Fei Li!

The brightness constancy constraint

•  How many equa3ons and unknowns per pixel?

The component of the flow perpendicular to the gradient (i.e., parallel to the edge) cannot be measured

edge

(u,v)

(u’,v’)

gradient

(u+u’,v+v’)

If (u, v ) sa3sfies the equa3on, so does (u+u’, v+v’ ) if

• One equa3on (this is a scalar equa3on!), two unknowns (u,v)

[ ] 0IvuI tT =+⋅∇

[ ] 0'v'uI T =⋅∇

Can we use this equa3on to recover image mo3on (u,v) at each pixel?


30-‐Oct-‐12 11

Lecture 13 - !!!

Fei-Fei Li!

The aperture problem

Actual mo7on


30-‐Oct-‐12 12

Lecture 13 - !!!

Fei-Fei Li!

The aperture problem

Perceived mo7on Source: Silvio Savarese

30-‐Oct-‐12 13

Lecture 13 - !!!

Fei-Fei Li!

The barber pole illusion

hdp://en.wikipedia.org/wiki/Barberpole_illusion Source: Silvio Savarese

30-‐Oct-‐12 14

Lecture 13 - !!!

Fei-Fei Li!

The barber pole illusion

hdp://en.wikipedia.org/wiki/Barberpole_illusion Source: Silvio Savarese

30-‐Oct-‐12 15

Lecture 13 - !!!

Fei-Fei Li! 16

* From

Marc Po

llefeys COMP 256 20

03

Aperture problem cont’d

30-‐Oct-‐12

Lecture 13 - !!!

Fei-Fei Li!

Solving the ambiguity…

•  How to get more equa3ons for a pixel? •  Spa7al coherence constraint: •  Assume the pixel’s neighbors have the same (u,v)

–  If we use a 5x5 window, that gives us 25 equa3ons per pixel

B. Lucas and T. Kanade. An itera3ve image registra3on technique with an applica3on to stereo vision. In Proceedings of the Interna6onal Joint Conference on Ar6ficial Intelligence, pp. 674–679, 1981.


30-‐Oct-‐12 17

Lecture 13 - !!!

Fei-Fei Li!

•  Overconstrained linear system:

Lucas-‐Kanade flow


30-‐Oct-‐12 18

Lecture 13 - !!!

Fei-Fei Li!

Condi3ons for solvability •  When is this system solvable?

•  What if the window contains just a single straight edge?


30-‐Oct-‐12 19

Lecture 13 - !!!

Fei-Fei Li!

•  Overconstrained linear system

The summa3ons are over all pixels in the K x K window

Least squares solu3on for d given by


30-‐Oct-‐12 20

Lucas-‐Kanade flow

Lecture 13 - !!!

Fei-Fei Li!

Condi3ons for solvability – Op3mal (u, v) sa3sfies Lucas-‐Kanade equa3on

Does this remind anything to you?

When is This Solvable? •  ATA should be inver3ble •  ATA should not be too small due to noise

–  eigenvalues λ1 and λ 2 of ATA should not be too small •  ATA should be well-‐condi3oned

–  λ 1/ λ 2 should not be too large (λ 1 = larger eigenvalue)


30-‐Oct-‐12 21

Lecture 13 - !!!

Fei-Fei Li!

•  Eigenvectors and eigenvalues of ATA relate to edge direc3on and magnitude •  The eigenvector associated with the larger eigenvalue points in

the direc3on of fastest intensity change •  The other eigenvector is orthogonal to it

M = ATA is the second moment matrix ! (Harris corner detector…)


30-‐Oct-‐12 22

Lecture 13 - !!!

Fei-Fei Li!

Interpre3ng the eigenvalues

λ1

λ2

“Corner” λ1 and λ2 are large, λ1 ~ λ2

λ1 and λ2 are small “Edge” λ1 >> λ2

“Edge” λ2 >> λ1

“Flat” region

Classifica3on of image points using eigenvalues of the second moment matrix:


30-‐Oct-‐12 23

Lecture 13 - !!!

Fei-Fei Li!

Edge

–  gradients very large or very small –  large λ1, small λ2


30-‐Oct-‐12 24

Lecture 13 - !!!

Fei-Fei Li!

Low-‐texture region

–  gradients have small magnitude –  small λ1, small λ2


30-‐Oct-‐12 25

Lecture 13 - !!!

Fei-Fei Li!

High-‐texture region

–  gradients are different, large magnitudes –  large λ1, large λ2


30-‐Oct-‐12 26

Lecture 13 - !!!

Fei-Fei Li!

What are good features to track?

•  Can measure “quality” of features from just a single image

•  Hence: tracking Harris corners (or equivalent) guarantees small error sensi3vity!

à Implemented in Open CV


30-‐Oct-‐12 27

Lecture 13 - !!!

Fei-Fei Li!

•  Key assump3ons (Errors in Lucas-‐Kanade)

•  Small mo7on: points do not move very far

•  Brightness constancy: projec3on of the same point looks the same in every frame

•  Spa7al coherence: points move like their neighbors

Recap


30-‐Oct-‐12 28

Lecture 13 - !!!

Fei-Fei Li!

Revisi3ng the small mo3on assump3on

•  Is this mo3on small enough? –  Probably not—it’s much larger than one pixel (2nd order terms dominate) –  How might we solve this problem?

* Fr

om K

hurr

am H

assa

n-S

hafiq

ue C

AP

5415

Com

pute

r Vis

ion

2003

30-‐Oct-‐12 29

Lecture 13 - !!!

Fei-Fei Li!

Reduce the resolu3on!

* Fr

om K

hurr

am H

assa

n-S

hafiq

ue C

AP

5415

Com

pute

r Vis

ion

2003

30-‐Oct-‐12 30

Lecture 13 - !!!

Fei-Fei Li!

Mul3-‐resolu3on Lucas Kanade Algorithm


30-‐Oct-‐12 31

Lecture 13 - !!!

Fei-Fei Li!


image I image H

Gaussian pyramid of image 1 Gaussian pyramid of image 2

image 2 image 1 u=10 pixels

u=5 pixels

u=2.5 pixels

u=1.25 pixels

Coarse-‐to-‐fine op3cal flow es3ma3on

30-‐Oct-‐12 32

Lecture 13 - !!!

Fei-Fei Li! 33

Itera3ve Refinement

•  Iterative Lukas-Kanade Algorithm 1.  Estimate velocity at each pixel by solving Lucas-

Kanade equations 2.  Warp I(t-1) towards I(t) using the estimated flow field

- use image warping techniques

3.  Repeat until convergence

* Fr

om K

hurr

am H

assa

n-S

hafiq

ue C

AP

5415

Com

pute

r Vis

ion

2003

30-‐Oct-‐12

Lecture 13 - !!!

Fei-Fei Li!

image I image J

Gaussian pyramid of image 1 (t) Gaussian pyramid of image 2 (t+1)

image 2 image 1

Coarse-‐to-‐fine op3cal flow es3ma3on

run itera3ve L-‐K

run itera3ve L-‐K

warp & upsample

.

.

.


30-‐Oct-‐12 34

Lecture 13 - !!!

Fei-Fei Li!

Op3cal Flow Results

* Fr

om K

hurr

am H

assa

n-S

hafiq

ue C

AP

5415

Com

pute

r Vis

ion

2003

30-‐Oct-‐12 35

Lecture 13 - !!!

Fei-Fei Li!

Op3cal Flow Results

* Fr

om K

hurr

am H

assa

n-S

hafiq

ue C

AP

5415

Com

pute

r Vis

ion

2003

•  hdp://www.ces.clemson.edu/~stb/klt/ •  OpenCV

30-‐Oct-‐12 36

Lecture 13 - !!!

Fei-Fei Li!

•  Key assump3ons (Errors in Lucas-‐Kanade)

•  Small mo7on: points do not move very far

•  Brightness constancy: projec3on of the same point looks the same in every frame

•  Spa7al coherence: points move like their neighbors

Recap


30-‐Oct-‐12 37

Lecture 13 - !!!

Fei-Fei Li!

Mo3on segmenta3on •  How do we represent the mo3on in this scene?


30-‐Oct-‐12 38

Lecture 13 - !!!

Fei-Fei Li!

•  Break image sequence into “layers” each of which has a coherent (affine) mo3on

Mo3on segmenta3on J. Wang and E. Adelson. Layered Representa3on for Mo3on Analysis. CVPR 1993.


30-‐Oct-‐12 39

Lecture 13 - !!!

Fei-Fei Li!

What are layers? •  Each layer is defined by an alpha mask and an affine mo3on model

J. Wang and E. Adelson. Layered Representa3on for Mo3on Analysis. CVPR 1993. Source: Silvio Savarese

30-‐Oct-‐12 40

Lecture 13 - !!!

Fei-Fei Li!

•  Subs3tu3ng into the brightness constancy equa3on:

yaxaayxvyaxaayxu

654

321

),(),(

++=

++=

0≈+⋅+⋅ tyx IvIuI

Affine mo3on


30-‐Oct-‐12 41

Lecture 13 - !!!

Fei-Fei Li!

0)()( 654321 ≈++++++ tyx IyaxaaIyaxaaI

•  Subs3tu3ng into the brightness constancy equa3on:

yaxaayxvyaxaayxu

654

321

),(),(

++=

++=

•  Each pixel provides 1 linear constraint in 6 unknowns

[ ] 2∑ ++++++= tyx IyaxaaIyaxaaIaErr )()()( 654321

•  Least squares minimiza3on:

Affine mo3on


30-‐Oct-‐12 42

Lecture 13 - !!!

Fei-Fei Li!

How do we es3mate the layers? •  1. Obtain a set of ini3al affine mo3on hypotheses

–  Divide the image into blocks and es3mate affine mo3on parameters in each block by least squares •  Eliminate hypotheses with high residual error

•  Map into mo3on parameter space •  Perform k-‐means clustering on affine mo3on parameters

– Merge clusters that are close and retain the largest clusters to obtain a smaller set of hypotheses to describe all the mo3ons in the scene


30-‐Oct-‐12 43

Lecture 13 - !!!

Fei-Fei Li!





2. Iterate un3l convergence: • Assign each pixel to best hypothesis

– Pixels with high residual error remain unassigned


30-‐Oct-‐12 44

Lecture 13 - !!!

Fei-Fei Li!






30-‐Oct-‐12 45

Lecture 13 - !!!

Fei-Fei Li!





2. Iterate un3l convergence: • Assign each pixel to best hypothesis

– Pixels with high residual error remain unassigned

• Perform region filtering to enforce spa3al constraints • Re-‐es3mate affine mo3ons in each region


30-‐Oct-‐12 46

Lecture 13 - !!!

Fei-Fei Li!

Example result

J. Wang and E. Adelson. Layered Representation for Motion Analysis. CVPR 1993. Source: Silvio Savarese

30-‐Oct-‐12 47

Lecture 13 - !!!

Fei-Fei Li!

Tracking

Sources: Kristen Grauman, D

eva Ra

manan

30-‐Oct-‐12 48

Lecture 13 - !!!

Fei-Fei Li!


•  Introduc3on •  Op3cal flow •  Feature tracking •  Applica3ons

30-‐Oct-‐12 49

Lecture 13 - !!!

Fei-Fei Li!

Mo3on es3ma3on techniques

•  Op3cal flow –  Recover image mo3on at each pixel from spa3o-‐temporal image brightness varia3ons (op3cal flow)

•  Feature-‐tracking –  Extract visual features (corners, textured areas) and “track” them over mul3ple frames

•  Shi-‐Tomasi feature tracker •  Tracking with dynamics


30-‐Oct-‐12 50

Lecture 13 - !!!

Fei-Fei Li!

Feature tracking

•  So far, we have only considered op3cal flow es3ma3on in a pair of images

•  If we have more than two images, we can compute the op3cal flow from each frame to the next

•  Given a point in the first image, we can in principle reconstruct its path by simply “following the arrows”


30-‐Oct-‐12 51

Lecture 13 - !!!

Fei-Fei Li!

•  Ambiguity of op3cal flow –  Find good features to track

•  Large mo3ons –  Discrete search instead of Lucas-‐Kanade

•  Changes in shape, orienta3on, color –  Allow some matching flexibility

•  Occlusions, dis-‐occlusions –  Need mechanism for dele3ng, adding new features

•  Driy – errors may accumulate over 3me –  Need to know when to terminate a track

Tracking challenges


30-‐Oct-‐12 52

Lecture 13 - !!!

Fei-Fei Li!

Shi-‐Tomasi feature tracker

•  Find good features using eigenvalues of second-‐moment matrix

–  Key idea: “good” features to track are the ones that can be tracked reliably

•  From frame to frame, track with Lucas-‐Kanade and a pure transla6on model

–  More robust for small displacements, can be es3mated from smaller neighborhoods

•  Check consistency of tracks by affine registra3on to the first observed instance of the feature

–  Affine model is more accurate for larger displacements –  Comparing to the first frame helps to minimize driy

J. Shi and C. Tomasi. Good Features to Track. CVPR 1994.


30-‐Oct-‐12 53

Lecture 13 - !!!

Fei-Fei Li!

Tracking example


30-‐Oct-‐12 54

Lecture 13 - !!!

Fei-Fei Li!

Tracking with dynamics

•  Key idea: Given a model of expected mo3on, predict where objects will occur in next frame, even before seeing the image –  Restrict search for the object –  Improved es3mates since measurement noise is reduced by trajectory smoothness


30-‐Oct-‐12 55

Lecture 13 - !!!

Fei-Fei Li!

update initial position

x

y

x

y

prediction

x

y

measurement

x

y

Tracking with dynamics

The Kalman filter: •  Method for tracking linear dynamical models in Gaussian noise •  The predicted/corrected state distribu3ons are Gaussian

•  Need to maintain the mean and covariance •  Calcula3ons are easy (all the integrals can be done in closed form)


30-‐Oct-‐12 56

Lecture 13 - !!!

Fei-Fei Li!

2D Target tracking using Kalman filter in MATLAB by AliReza KashaniPour

hdp://www.mathworks.com/matlabcentral/fileexchange/14243


30-‐Oct-‐12 57

Lecture 13 - !!!

Fei-Fei Li!


•  Introduc3on •  Op3cal flow •  Feature tracking •  Applica3ons

30-‐Oct-‐12 58

Lecture 13 - !!!

Fei-Fei Li!

Uses of mo3on

•  Tracking features •  Segmen3ng objects based on mo3on cues •  Learning dynamical models •  Improving video quality

– Mo3on stabiliza3on –  Super resolu3on

•  Tracking objects •  Recognizing events and ac3vi3es

30-‐Oct-‐12 59

Lecture 13 - !!!

Fei-Fei Li!

Es3ma3ng 3D structure


30-‐Oct-‐12 60

Lecture 13 - !!!

Fei-Fei Li!

Segmen3ng objects based on mo3on cues

•  Background subtrac3on –  A sta3c camera is observing a scene –  Goal: separate the sta3c background from the moving foreground


30-‐Oct-‐12 61

Lecture 13 - !!!

Fei-Fei Li!

•  Mo3on segmenta3on –  Segment the video into mul3ple coherently moving objects

Segmen3ng objects based on mo3on cues

S. J. Pundlik and S. T. Birchfield, Mo3on Segmenta3on at Any Speed, Proceedings of the Bri3sh Machine Vision Conference (BMVC) 2006


30-‐Oct-‐12 62

Lecture 13 - !!!

Fei-Fei Li!

Z.Yin and R.Collins, "On-‐the-‐fly Object Modeling while Tracking," IEEE Computer Vision and PaJern Recogni6on (CVPR '07), Minneapolis, MN, June 2007.

Tracking objects


30-‐Oct-‐12 63

Lecture 13 - !!!

Fei-Fei Li!

Synthesizing dynamic textures

30-‐Oct-‐12 64

Lecture 13 - !!!

Fei-Fei Li! 65

Super-‐resolu3on

Example: A set of low quality images


30-‐Oct-‐12

Lecture 13 - !!!

Fei-Fei Li! 66

Super-‐resolu3on

Each of these images looks like this:


30-‐Oct-‐12

Lecture 13 - !!!

Fei-Fei Li! 67

Super-‐resolu3on

The recovery result:


30-‐Oct-‐12

Lecture 13 - !!!

Fei-Fei Li!

D. Ramanan, D. Forsyth, and A. Zisserman. Tracking People by Learning their Appearance. PAMI 2007.

Tracker

Recognizing events and ac3vi3es


30-‐Oct-‐12 68

Lecture 13 - !!!

Fei-Fei Li!

Juan Carlos Niebles, Hongcheng Wang and Li Fei-‐Fei, Unsupervised Learning of Human Ac7on Categories Using Spa7al-‐Temporal Words, (BMVC), Edinburgh, 2006.


30-‐Oct-‐12 69

Lecture 13 - !!!

Fei-Fei Li!

Crossing – Talking – Queuing – Dancing – jogging

W. Choi & K. Shahid & S. Savarese WMC 2010



30-‐Oct-‐12 70

Lecture 13 - !!!

Fei-Fei Li!

W. Choi, K. Shahid, S. Savarese, "What are they doing? : Collec3ve Ac3vity Classifica3on Using Spa3o-‐Temporal Rela3onship Among People", 9th Interna3onal Workshop on Visual Surveillance (VSWS09) in conjuc3on with ICCV 09

30-‐Oct-‐12 71

Lecture 13 - !!!

Fei-Fei Li!

Op3cal flow without mo3on!

30-‐Oct-‐12 72

Lecture 13 - !!!

Fei-Fei Li!

What we have learned today?

•  Introduc3on •  Op3cal flow •  Feature tracking •  Applica3ons •  (Problem Set 3 (Q1))

30-‐Oct-‐12 73

Lecture’13:’’ Tracking’mo3on’features’’...

Documents

Transcript of Lecture’13:’’ Tracking’mo3on’features’’...