DR. J. Shanbehzadeh Shanbehzadeh@gmail Sara Omid

DR. J. [email protected]

Sara Omid

Science and Research Branch of Azad University

Lecture 06Computing Optical Flow

Machine Vision

R. C

. Gon

zale

z, a

nd R

. E. W

oods

, Dig

ital I

mag

e P

roce

ssin

g, N

ew J

erse

y: P

rent

ice

Hal

l, 3rd

edi

tion,

200

8.

Table of Contents2

Optical Flow ExamplesOptical FlowHorn & Schunck Optical Flow

• Interpretation Of Optical Flow eq• Horn & Schunck• Derivative Masks• Laplacian• Algorithm-1• Synthetic Images• Horn & Schunck Results

Lucas & KanadeComments

3

Optical Flow Examples

R. C

. Gon

zale

z, a

nd R

. E. W

oods

, Dig

ital I

mag

e P

roce

ssin

g, N

ew J

erse

y: P

rent

ice

Hal

l, 3rd

edi

tion,

200

8.

Optical Flow Examples4


Optical Flow

Horn & Schunck Optical Flow

Interpretation Of Optical Flow eq

Horn & Schunck

Derivative Masks

Laplacian

Algorithm-1

Synthetic Images

Horn & Schunck Results

Lucas & Kanade

Comments

R. C

. Gon

zale

z, a

nd R

. E. W

oods

, Dig

ital I

mag

e P

roce

ssin

g, N

ew J

erse

y: P

rent

ice

Hal

l, 3rd

edi

tion,

200

8.



Optical Flow



Horn & Schunck

Derivative Masks

Laplacian

Algorithm-1

Synthetic Images


Lucas & Kanade

Comments

14

Optical Flow

R. C

. Gon

zale

z, a

nd R

. E. W

oods

, Dig

ital I

mag

e P

roce

ssin

g, N

ew J

erse

y: P

rent

ice

Hal

l, 3rd

edi

tion,

200

8.

Optical Flow15


Optical Flow



Horn & Schunck

Derivative Masks

Laplacian

Algorithm-1

Synthetic Images


Lucas & Kanade

Comments

Applications

• Motion based segmentation

• Structure from Motion(3D shape and Motion)

• Alignment (Global motion compensation) Camcorder video stabilization UAV Video Analysis

• Video Compression

16

Horn & Schunck

R. C

. Gon

zale

z, a

nd R

. E. W

oods

, Dig

ital I

mag

e P

roce

ssin

g, N

ew J

erse

y: P

rent

ice

Hal

l, 3rd

edi

tion,

200

8.

Horn & Schunck Optical Flow17


Optical Flow



Horn & Schunck

Derivative Masks

Laplacian

Algorithm-1

Synthetic Images


Lucas & Kanade

Comments

Brightness constancy assumption𝑓 (𝑥 , 𝑦 , 𝑡 )= 𝑓 (𝑥+𝑑𝑥 , 𝑦+𝑑𝑦 , 𝑡+𝑑𝑡 )

𝑓 (𝑥 , 𝑦 , 𝑡 )= 𝑓 (𝑥 , 𝑦 ,𝑡 )+𝜕 𝑓𝜕𝑥 𝑑𝑥+𝜕 𝑓𝜕 𝑦 𝑑𝑦+

𝜕 𝑓𝜕𝑡 𝑑𝑡

𝑓 𝑥𝑑𝑥+ 𝑓 𝑦𝑑𝑦+ 𝑓 𝑡 𝑑𝑡=0

𝑓 𝑥𝑢+ 𝑓 𝑦𝑣+ 𝑓 𝑡=0

Taylor Series

R. C

. Gon

zale

z, a

nd R

. E. W

oods

, Dig

ital I

mag

e P

roce

ssin

g, N

ew J

erse

y: P

rent

ice

Hal

l, 3rd

edi

tion,

200

8.

Interpretation Of Optical Flow eq18


Optical Flow



Horn & Schunck

Derivative Masks

Laplacian

Algorithm-1

Synthetic Images


Lucas & Kanade

Comments

𝑓 𝑥𝑢+ 𝑓 𝑦𝑣+ 𝑓 𝑡=0

d = normal flowP = parallel flow𝑑=

𝑓 𝑡√ 𝑓 𝑥

2+ 𝑓 𝑦2

𝑣=−𝑓 𝑥

𝑓 𝑦𝑢−

𝑓 𝑡𝑓 𝑦

R. C

. Gon

zale

z, a

nd R

. E. W

oods

, Dig

ital I

mag

e P

roce

ssin

g, N

ew J

erse

y: P

rent

ice

Hal

l, 3rd

edi

tion,

200

8.

Horn & Schunck19


Optical Flow



Horn & Schunck

Derivative Masks

Laplacian

Algorithm-1

Synthetic Images


Lucas & Kanade

Comments

∬ { ( 𝑓 𝑥𝑢+ 𝑓 𝑦𝑣+ 𝑓 𝑡 )2+λ (𝑢𝑥2+𝑢𝑦

2 +𝑣𝑥2+𝑣𝑦

2 ) }𝑑𝑥𝑑𝑦Brightness Constancy

Smoothness Constraint

min

R. C

. Gon

zale

z, a

nd R

. E. W

oods

, Dig

ital I

mag

e P

roce

ssin

g, N

ew J

erse

y: P

rent

ice

Hal

l, 3rd

edi

tion,

200

8.

Horn & Schunck20


Optical Flow



Horn & Schunck

Derivative Masks

Laplacian

Algorithm-1

Synthetic Images


Lucas & Kanade

Comments

Laplacian

=

=

R. C

. Gon

zale

z, a

nd R

. E. W

oods

, Dig

ital I

mag

e P

roce

ssin

g, N

ew J

erse

y: P

rent

ice

Hal

l, 3rd

edi

tion,

200

8.

Derivative Masks21


Optical Flow



Horn & Schunck

Derivative Masks

Laplacian

Algorithm-1

Synthetic Images


Lucas & Kanade

Comments

fx fy

ft

R. C

. Gon

zale

z, a

nd R

. E. W

oods

, Dig

ital I

mag

e P

roce

ssin

g, N

ew J

erse

y: P

rent

ice

Hal

l, 3rd

edi

tion,

200

8.

Derivative Masks (Roberts)22


Optical Flow



Horn & Schunck

Derivative Masks

Laplacian

Algorithm-1

Synthetic Images


Lucas & Kanade

Comments

Apply first mask to 1st image

Second mask to 2nd image

Add the responses to get fx, fy, ft

R. C

. Gon

zale

z, a

nd R

. E. W

oods

, Dig

ital I

mag

e P

roce

ssin

g, N

ew J

erse

y: P

rent

ice

Hal

l, 3rd

edi

tion,

200

8.

Laplacian23


Optical Flow



Horn & Schunck

Derivative Masks

Laplacian

Algorithm-1

Synthetic Images


Lucas & Kanade

Comments

R. C

. Gon

zale

z, a

nd R

. E. W

oods

, Dig

ital I

mag

e P

roce

ssin

g, N

ew J

erse

y: P

rent

ice

Hal

l, 3rd

edi

tion,

200

8.

Laplacian24


Optical Flow



Horn & Schunck

Derivative Masks

Laplacian

Algorithm-1

Synthetic Images


Lucas & Kanade

Comments

Discrete version

R. C

. Gon

zale

z, a

nd R

. E. W

oods

, Dig

ital I

mag

e P

roce

ssin

g, N

ew J

erse

y: P

rent

ice

Hal

l, 3rd

edi

tion,

200

8.

Laplacian25


Optical Flow



Horn & Schunck

Derivative Masks

Laplacian

Algorithm-1

Synthetic Images


Lucas & Kanade

Comments

variational calculus

R. C

. Gon

zale

z, a

nd R

. E. W

oods

, Dig

ital I

mag

e P

roce

ssin

g, N

ew J

erse

y: P

rent

ice

Hal

l, 3rd

edi

tion,

200

8.

Algorithm-126


Optical Flow



Horn & Schunck

Derivative Masks

Laplacian

Algorithm-1

Synthetic Images


Lucas & Kanade

Comments

• k=0

• Initialize uk and vk

• Repeat until some error measure is satisfied(converges)

R. C

. Gon

zale

z, a

nd R

. E. W

oods

, Dig

ital I

mag

e P

roce

ssin

g, N

ew J

erse

y: P

rent

ice

Hal

l, 3rd

edi

tion,

200

8.

Synthetic Images27


Optical Flow



Horn & Schunck

Derivative Masks

Laplacian

Algorithm-1

Synthetic Images


Lucas & Kanade

Comments

R. C

. Gon

zale

z, a

nd R

. E. W

oods

, Dig

ital I

mag

e P

roce

ssin

g, N

ew J

erse

y: P

rent

ice

Hal

l, 3rd

edi

tion,

200

8.

Horn & Schunck Results28


Optical Flow



Horn & Schunck

Derivative Masks

Laplacian

Algorithm-1

Synthetic Images


Lucas & Kanade

Comments

29

Lucas & Kanade

R. C

. Gon

zale

z, a

nd R

. E. W

oods

, Dig

ital I

mag

e P

roce

ssin

g, N

ew J

erse

y: P

rent

ice

Hal

l, 3rd

edi

tion,

200

8.

Lucas & Kanade30


Optical Flow



Horn & Schunck

Derivative Masks

Laplacian

Algorithm-1

Synthetic Images


Lucas & Kanade

Comments

• Optical Flow eq

A u = ft

𝑓 𝑥𝑢+ 𝑓 𝑦𝑣=− 𝑓 𝑡

• Consider 3 by 3 window

. . .

R. C

. Gon

zale

z, a

nd R

. E. W

oods

, Dig

ital I

mag

e P

roce

ssin

g, N

ew J

erse

y: P

rent

ice

Hal

l, 3rd

edi

tion,

200

8.

Lucas & Kanade31


Optical Flow



Horn & Schunck

Derivative Masks

Laplacian

Algorithm-1

Synthetic Images


Lucas & Kanade

Comments

A U = ft

AT A U = AT ft

U = (ATA)-1 AT ft

Pseudo Inverse

Least Squares Fit

R. C

. Gon

zale

z, a

nd R

. E. W

oods

, Dig

ital I

mag

e P

roce

ssin

g, N

ew J

erse

y: P

rent

ice

Hal

l, 3rd

edi

tion,

200

8.

Lucas & Kanade32


Optical Flow



Horn & Schunck

Derivative Masks

Laplacian

Algorithm-1

Synthetic Images


Lucas & Kanade

Comments

R. C

. Gon

zale

z, a

nd R

. E. W

oods

, Dig

ital I

mag

e P

roce

ssin

g, N

ew J

erse

y: P

rent

ice

Hal

l, 3rd

edi

tion,

200

8.

Lucas & Kanade33


Optical Flow



Horn & Schunck

Derivative Masks

Laplacian

Algorithm-1

Synthetic Images


Lucas & Kanade

Comments

[ ∑ 𝑓 𝑥𝑖2 ∑ 𝑓 𝑥𝑖 𝑓 𝑦𝑖

∑ 𝑓 𝑥𝑖 𝑓 𝑦𝑖 ∑ 𝑓 𝑦𝑖2 ][𝑢𝑣 ]=[−∑ 𝑓 𝑥𝑖 𝑓 𝑡𝑖

−∑ 𝑓 𝑦𝑖 𝑓 𝑡𝑖]

R. C

. Gon

zale

z, a

nd R

. E. W

oods

, Dig

ital I

mag

e P

roce

ssin

g, N

ew J

erse

y: P

rent

ice

Hal

l, 3rd

edi

tion,

200

8.

Lucas & Kanade34


Optical Flow



Horn & Schunck

Derivative Masks

Laplacian

Algorithm-1

Synthetic Images


Lucas & Kanade

Comments

R. C

. Gon

zale

z, a

nd R

. E. W

oods

, Dig

ital I

mag

e P

roce

ssin

g, N

ew J

erse

y: P

rent

ice

Hal

l, 3rd

edi

tion,

200

8.

Lucas & Kanade35


Optical Flow



Horn & Schunck

Derivative Masks

Laplacian

Algorithm-1

Synthetic Images


Lucas & Kanade

Comments

R. C

. Gon

zale

z, a

nd R

. E. W

oods

, Dig

ital I

mag

e P

roce

ssin

g, N

ew J

erse

y: P

rent

ice

Hal

l, 3rd

edi

tion,

200

8.

Lucas & Kanade36


Optical Flow



Horn & Schunck

Derivative Masks

Laplacian

Algorithm-1

Synthetic Images


Lucas & Kanade

Comments

R. C

. Gon

zale

z, a

nd R

. E. W

oods

, Dig

ital I

mag

e P

roce

ssin

g, N

ew J

erse

y: P

rent

ice

Hal

l, 3rd

edi

tion,

200

8.

Lucas & Kanade37


Optical Flow



Horn & Schunck

Derivative Masks

Laplacian

Algorithm-1

Synthetic Images


Lucas & Kanade

Comments

38

Comments

R. C

. Gon

zale

z, a

nd R

. E. W

oods

, Dig

ital I

mag

e P

roce

ssin

g, N

ew J

erse

y: P

rent

ice

Hal

l, 3rd

edi

tion,

200

8.

Comments39


Optical Flow



Horn & Schunck

Derivative Masks

Laplacian

Algorithm-1

Synthetic Images


Lucas & Kanade

Comments

• Horn‐Schunck and Lucas‐Kanade optical methods work only for small motion.

• If object moves faster, the brightness changes rapidly,

• 2x2 or 3x3 masks fail to estimate spatiotemporal derivatives.

• Pyramids can be used to compute large optical flow vectors.

DR. J. Shanbehzadeh Shanbehzadeh@gmail Sara Omid

Documents

Transcript of DR. J. Shanbehzadeh Shanbehzadeh@gmail Sara Omid