Optical Flow, Motion Field, Horn Schunck, Lucas Kanade, Aperture Problem, Optical Flow Failure,...

7/27/2019 Optical Flow, Motion Field, Horn Schunck, Lucas Kanade, Aperture Problem, Optical Flow Failure, Pyramids, Interpolation, 3d Rigid Motion, Displacement, Instantaneous Velocity, Affine Transformation

http://slidepdf.com/reader/full/optical-flow-motion-field-horn-schunck-lucas-kanade-aperture-problem-optical 1/169

Optical Flow

http://www2.ece.ohio-state.edu/~gotardop/







Where did each pixel in image 1 go to in image 2



General Scenario

Possibilities:

• camera moving, stationary scene

• camera stationary, moving objects

• both camera and scene moving



Scene Interpretation

• How is the camera moving?

• How many moving objects are there?

• Which directions are they moving in?

• How fast are they moving?

• Can we recognize their type of motion (e.g.

walking, running, etc.)?



Motion Field & Optical Flow Field

• Motion Field = Real world 3D motion

• Optical Flow Field = Projection of the

motion field onto the 2d image3D motion vector

2D optical flow

vector

vu,u

CCD



Optical Flow

• Goal:

Find for each pixel a velocity vector

which says: – How quickly is the pixel moving across the image

– In which direction it is moving

vu,u



Estimating Optical Flow

• Assume the image intensity is constant

t y x I ,, dt t dy ydx x I ,,

I Time = t Time = t+dt

dy ydx x ,

y x,



Sobel Example



Assume constant (u,v) in small neighborhood



Time t+dt

Aperture problem

Time t

?

Time t+dt

• Where did the blue point move to?



Aperture problem

Sometimes enlarging the aperture can help



Aperture problem in Real Life



Local Patch Analysis

• How certain are the motion estimates?



Edge



Low texture region



High textured region



When does it break?

The screen is

stationary yet

displays motion

Homogeneous

objects

generate zero

optical flow.

Fixed sphere.

Changing light

source.

Non-rigid

texture motion



33



Verbal Assignment



Examples of Motion fields

Forward

motion

Rotation Horizontal

translation

Closer objects

appear to move

faster!!



Optical Flow for an Overhead View of a

Dragonfly

• Travelling forward

• Optical flow is faster on the

left & right than in the front

or back

• Rotating to the right

• Optical flow is to the left in all

directions

• If flying a curved path?



What is it useful for? - Flying

Hovering: keep the optic flow zero

everywhere

Avoiding obstacles: the focus of expansion

(FOE) is inside the rock, and the optic flow is

expanding rapidly.

Speed control through dense environments:

by reducing optical flow



Optical Flow as Seen from an Aircraft



Block-based motion estimation

• Break image up into square blocks

• Estimate translation for each block



Pyramids



???

….



Low-Res.

High-Res.

Interpolation



1-D Interpolation

Given any two points, )(,,)(, 1100 x f x x f x



1-D Interpolation

The line that interpolates the two points is:

Example :

Find a polynomial that interpolates (1,2) and (2,4).

001

0101

)()()()( x x

x x

x f x f x f x f

x x x f 2112

242)(1



Low-Res.

High-Res.

2-D Interpolation

l



1

1.5

2

1

1.5

2

0

5

10

15

2-D Interpolation



Assignment

• At least 10 Problems around us/in our daily

life you see the Optical Flow can solve?



3D Rigid Motion



I M i M d l



Image Motion Models

• Displacement Model

• Instantaneous Velocity Model



Perspective Effects:



Perspective EffectsFunny things happen…



3D Transformations



3D Transformations• Right-handed / left-handed systems

3D Transformations



3D Transformations …

• Positive rotation angles for right-handed

systems: (counter-clockwise rotations)



Perspective Effects:



Funny things happen…



Mapping between planes



Ax + By + Cz + D = 0

Assignment:

Given an equation of the line. How would you determine a point is on the line or not?



Geometric Transformations




Coordinate Mapping



Coordinate Mapping

• Transformation: ( x’, y’ ) = T {( x, y)}

f(x, y) g(x’, y’ )

T(x, y)

x'

y'

x

y

Rotation



Rotation

• Rotates points by an angle θ about origin

(θ >0: counterclockwise rotation)

• From ABP triangle:

• From ACP ’ triangle: A

BC

Rotation …



Rotation …

y

x

y

x*

cossin

sincos

y x, y x ,

x

y

0

Scaling



Scaling

y s

x s

y

x

s

s

y

x

y

x

y

x*

0

0

x

y

0

Scaling …



Scaling …

• Uniform vs non-uniform scaling

• Effect of scale factors:

Shear



Shear

• along X-axis

y

dy x

y

xd

y

x

10

1

x

y

0

Shear …



Shear …

•along Y-axis

dx y

x

y

x

d y

x

1

01

x

y

0



Summary



ummary

1

*

100

00

00

1

y

x

s

s

y

x

y

x

1

*

100

0cossin0sincos

1

y x

y x

1

*

100

01

01

1

y x

sh sh

y x

y

x

1

*

100

10

01

1

y

x

t

t

y

x

y

x

Translation Scaling

ShearingRotation

Affine Transformations



ff f m

• Perceptually, Affine Transformations are linear combinations of




ff f

11001

''

2221

1211

y x

t aat aa

y x

y

x

• How to put together

– Translation

– Rotation

– Scaling

– Shearing

Affine Transformations …



ff f

• Approximates viewpoint changes for roughly

planar objects and roughly orthographic cameras




center at

infinity



Vanishing



Vanishing

Point

Vanishing

Point Horizon

Line

Assignment



g

• How to put together successive:

– Translation

– Rotation – Scaling

– Shearing



Coordinate Mapping



pp gT(x, y)

x

y

x'

y'

f(x, y) g(x’, y’ )

Transformation: ( x’, y’ ) = T {( x,




Rotation …



y

x

y

x*

cossin

sincos

y x, y x ,

x

y

0

Scaling



y s

x s

y

x

s

s

y

x

y

x

y

x*

0

0

x

y

0

Scaling …



g

• Uniform vs non-uniform scaling

• Effect of scale factors:



Shear



• along X-axis

y

dy x

y

xd

y

x

10

1

x

y

0

Shear …



•

along Y-axis

dx y

x

y

x

d y

x

1

01

x

y

0

Translation



• Moves a point to a new location by adding

translation amounts to the coordinates of the

point.

or

or



Summary



1

*

100

00

00

1

y

x

s

s

y

x

y

x

1

*

100

0cossin

0sincos

1

y

x

y

x

1

*

100

01

01

1

y

x

sh

sh

y

x

y

x

1

*

100

10

01

1

y

x

t

t

y

x

y

x

Translation Scaling

ShearingRotation




• Perceptually, Affine Transformations are linear combinations of




11001

''

2221

1211

y x

t aat aa

y x

y

x

• How to put together

– Translation

– Rotation

– Scaling

– Shearing




center at

infinity

Perspective Effects:Funny things happen



Funny things happen…



Translation …



• e.g., stereo pairs, video sequence assumingsmall camera motion




• Approximates viewpoint changes for roughly

planar objects and roughly orthographic cameras

Overview transformations



1002221

1211

y

x

t aa

t aa

100

2221

1211

y

x

t sr sr

t sr sr

333231

232221

131211

hhh

hhhhhh

100

2221

1211

y

x

t r r

t r r

Projective

8dof

Affine

6dof

Similarity

4dof

Euclidean

3dof

Concurrency, collinearity,order of contact (intersection,

tangency, inflection, etc.),

cross ratio

Parallellism, ratio of areas,

ratio of lengths on parallel

lines (e.g midpoints), linear combinations of vectors

(centroids).

Ratios of lengths, angles.

Angles, lengths, areas.



Assignment



• How to put together successive:

– Translation

– Rotation

– Scaling

– Shearing

Think of This

Optical Flow, Motion Field, Horn Schunck, Lucas Kanade, Aperture Problem, Optical Flow Failure,...

Documents

Transcript of Optical Flow, Motion Field, Horn Schunck, Lucas Kanade, Aperture Problem, Optical Flow Failure,...