Lecture-4Computing Optical Flow

Hamburg Taxi seq

Horn&Schunck Optical Flow

df (x, y, t)dt

=∂f∂x

dxdt

+∂f∂y

dydt

+∂f∂t

= 0

Sequence Image ),,( tyxf

Horn&Schunck Optical Flow

Taylor Series

f x y t f x y tfx

dxfy

dyft

dt( , , ) ( , , )= + + +∂∂

∂∂

∂∂

),,(),,( dttdyydxxftyxf +++=

0=++ dtfdyfdxf tyx

Interpretation of optical flow eq

d=normal flowp=parallel flow

Equation of st.line

0=++ tyx fvfuf

y

t

y

x

ffu

ffv −−=

df

f ft

x y

=+2 2

Horn&Schunck (contd)

variational calculus

discrete version

min

{( ) ( )}∫∫ + + + + + +f u f v f u u v v dxdyx y t x y x y2 2 2 2 2λ

( ) ( )

( ) (( )

f u f v f f u

f u f v f f vx y t x

x y t y

+ + + =

+ + + =

λ

λ

∆

∆

2

2

0

0

( ) ( )

( ) (( )

f u f v f f u u

f u f v f f v vx y t x av

x y t y av

+ + + − =

+ + + − =

λ

λ

0

0

u u fPD

v v fPD

av x

av y

= −

= −

P f u f v f

D f fx av y av t

x y

= + +

= + +λ 2 2

yyxx uuu +=∆2

Algorithm-1

k=0Initialize

• Repeat until some error measure is satisfied(converges)

u vK K

22yx

tavyavx

ffD

fvfufP

++=

++=

λu u f

PD

v v fPD

av x

av y

= −

= −

Derivatives

Derivative: Rate of change of some quantity Speed is a rate of change of a distanceAcceleration is a rate of change of speed

Derivative

speed

acceleration

xx fxfx

xxfxfdxdf

=′=∆

∆−−= →∆ )()()(lim 0

dtdva

dtds

v

=

=

Examples

x

x

exdxdy

exy

−

−

−+=

+=

)1(cos

sin3

42

42 xxdxdy

xxy

+=

+=

Second Derivative

xxx fxf

dxdf

=′′= )(

22

2

3

42

122

42

xdx

yd

xxdxdy

xxy

+=

+=

+=

Discrete Derivative

(Finite Difference)

)()()(lim 0 xfx

xxfxfdxdf

x ′=∆

∆−−= →∆

)(1

)1()( xfxfxfdxdf ′=

−−=

)()1()( xfxfxfdxdf ′=−−=

Discrete Derivative

Left difference

Right difference

Center difference

)()1()( xfxfxfdxdf ′=−−=

)()1()( xfxfxfdxdf ′=+−=

)()1()1( xfxfxfdxdf ′=−−+=

Example

F(x)=10 10 10 10 20 20 20F’(x)=0 0 0 0 10 0 0F’’(x)=0 0 0 0 10 -10 0

-1 1 left difference1 -1 right difference-1 0 1 center difference

Left difference

Derivatives in Two Dimensions

(partial Derivatives)

yyyxfyxff

yf

xyxxfyxff

xf

yxf

yy

xx

∆∆−−

==∂∂

∆∆−−

==∂∂

→∆

→∆

),(),(lim

),(),(lim

),(

0

0

x

y

yx

yx

ff

ff

ff

1

22

tandirection

)(magnitude

VectorGradient ),(

−==

+=

θ

Laplacian2 =+=∆ yyxx fff

Derivatives of an Image

Derivative & average

Prewit

xf 101101101

−

−−

yf 111000111 −−−

⎥⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢⎢

⎣

⎡

=

20202010102020201010202020101020202010102020201010

),( yxI

⎥⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢⎢

⎣

⎡

=

0000000303000030300003030000000

xI

Derivatives of an Image

⎥⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢⎢

⎣

⎡

=

0000000000000000000000000

yI

⎥⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢⎢

⎣

⎡

=

20202010102020201010202020101020202010102020201010

),( yxI

Laplacian

0 −14

0

−14

1 −14

0 −14

0

fxx + fyy

Convolution

Convolution (contd)

),(*),(),(

),(),(),(1

1

1

1

yxgyxfyxh

jigjyixfyxhi j

=

++= ∑ ∑−= −=

h(x, y) = f (x −1,y −1)g(−1,−1) + f (x, y −1)g(0,−1) + f (x +1,y −1)g(1,−1) + f (x −1, y)g(−1,0) + f (x, y)g(0,0) + f (x +1,y)g(1,0) + f (x −1, y +1)g(−1,1) + f (x, y +1)g(0,1) + f (x +1, y +1)g(1,1)

Derivative Masks

Apply first mask to 1st imageSecond mask to 2nd imageAdd the responses to get f_x, f_y, f_t

xf

image second1111

imagefirst 1111

⎥⎦

⎤⎢⎣

⎡−−

⎥⎦

⎤⎢⎣

⎡−−

yf

image second1111

imagefirst 1111

⎥⎦

⎤⎢⎣

⎡ −−

⎥⎦

⎤⎢⎣

⎡ −−

tf

image second1111

imagefirst 1111

⎥⎦

⎤⎢⎣

⎡

⎥⎦

⎤⎢⎣

⎡−−−−

Synthetic Images

Horn & Schunck Results

One iteration 10 iterations

4=λ

Lucas & Kanade (Least Squares)

Optical flow eq

• Consider 3 by 3 windowtyx fvfuf −=+

999

111

tyx

tyx

fvfuf

fvfuf

−=+

−=+

M

⎥⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢⎢

⎣

⎡

−

−

=⎥⎦

⎤⎢⎣

⎡

⎥⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢⎢

⎣

⎡

9

1

9

1

9

1

t

t

y

y

x

x

f

f

vu

f

f

f

fMMM

tfAu =

Lucas & Kanade

tTT

tTT

t

fAAAu

fAAuA

fAu

1)( −=

=

=

2)(min tyixi fvfuf ++∑

Lucas & Kanade2)(min tyixi fvfuf ++∑

0)(

0)(

=++

=++

∑

∑

yitiyixi

xitiyixi

ffvfuf

ffvfuf

fxi2∑ u + fxi∑ fyiv = − fxi∑ f ti

fxi fyi∑ u + fyi2∑ v = − fyi∑ f ti

fxi2∑ fxi fyi∑

fxi fyi∑ fyi2∑

⎡

⎣

⎢ ⎢ ⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥ ⎥ ⎥

uv

⎡

⎣ ⎢

⎤

⎦ ⎥ =

− fxi f ti∑

− fyi f ti∑

⎡

⎣

⎢ ⎢ ⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥ ⎥ ⎥

( fxi∑ u + fyiv + f ti) f xi = 0

( fxi∑ u + fyiv + f ti) f yi = 0

Lucas & Kanade

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

−

−

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

=⎥⎦

⎤⎢⎣

⎡

∑

∑

∑∑

∑∑−

tiyi

tixi

yiyixi

yixixi

ff

ff

fff

fff

vu

1

2

2

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

−

−

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

−

−

−=⎥

⎦

⎤⎢⎣

⎡

∑

∑

∑∑

∑∑

∑ ∑∑ tiyi

tixi

xiyixi

yix iyi

yixiyixi ff

ff

fff

fff

ffffvu

2

2

222 )(

1

∑ ∑∑

∑ ∑ ∑∑

∑ ∑∑

∑ ∑ ∑∑

−

−=

−

+−=

222

2

222

2

)(

)(

yixiyixi

tiy ix iyixitix i

y ix iyixi

tiyiyixitix iyi

ffff

fffffffv

ffff

fffffffu

Lucas-Kanadewithout pyramids

Fails in areas of large motion

Lucas-Kanade with Pyramids

Comments

Horn-Schunck and Lucas-Kanade optical method works only for small motion.If object moves faster, the brightness changes rapidly, 2x2 or 3x3 masks fail to estimate spatiotemporal derivatives.Pyramids can be used to compute large optical flow vectors.