Post on 02-Aug-2020
Lecture-4Computing Optical Flow
Hamburg Taxi seq
Horn&Schunck Optical Flow
df (x, y, t)dt
=∂f∂x
dxdt
+∂f∂y
dydt
+∂f∂t
= 0
Sequence Image ),,( tyxf
Horn&Schunck Optical Flow
Taylor Series
f x y t f x y tfx
dxfy
dyft
dt( , , ) ( , , )= + + +∂∂
∂∂
∂∂
),,(),,( dttdyydxxftyxf +++=
0=++ dtfdyfdxf tyx
Interpretation of optical flow eq
d=normal flowp=parallel flow
Equation of st.line
0=++ tyx fvfuf
y
t
y
x
ffu
ffv −−=
df
f ft
x y
=+2 2
Horn&Schunck (contd)
variational calculus
discrete version
min
{( ) ( )}∫∫ + + + + + +f u f v f u u v v dxdyx y t x y x y2 2 2 2 2λ
( ) ( )
( ) (( )
f u f v f f u
f u f v f f vx y t x
x y t y
+ + + =
+ + + =
λ
λ
∆
∆
2
2
0
0
( ) ( )
( ) (( )
f u f v f f u u
f u f v f f v vx y t x av
x y t y av
+ + + − =
+ + + − =
λ
λ
0
0
u u fPD
v v fPD
av x
av y
= −
= −
P f u f v f
D f fx av y av t
x y
= + +
= + +λ 2 2
yyxx uuu +=∆2
Algorithm-1
k=0Initialize
• Repeat until some error measure is satisfied(converges)
u vK K
22yx
tavyavx
ffD
fvfufP
++=
++=
λu u f
PD
v v fPD
av x
av y
= −
= −
Derivatives
Derivative: Rate of change of some quantity Speed is a rate of change of a distanceAcceleration is a rate of change of speed
Derivative
speed
acceleration
xx fxfx
xxfxfdxdf
=′=∆
∆−−= →∆ )()()(lim 0
dtdva
dtds
v
=
=
Examples
x
x
exdxdy
exy
−
−
−+=
+=
)1(cos
sin3
42
42 xxdxdy
xxy
+=
+=
Second Derivative
xxx fxf
dxdf
=′′= )(
22
2
3
42
122
42
xdx
yd
xxdxdy
xxy
+=
+=
+=
Discrete Derivative
(Finite Difference)
)()()(lim 0 xfx
xxfxfdxdf
x ′=∆
∆−−= →∆
)(1
)1()( xfxfxfdxdf ′=
−−=
)()1()( xfxfxfdxdf ′=−−=
Discrete Derivative
Left difference
Right difference
Center difference
)()1()( xfxfxfdxdf ′=−−=
)()1()( xfxfxfdxdf ′=+−=
)()1()1( xfxfxfdxdf ′=−−+=
Example
F(x)=10 10 10 10 20 20 20F’(x)=0 0 0 0 10 0 0F’’(x)=0 0 0 0 10 -10 0
-1 1 left difference1 -1 right difference-1 0 1 center difference
Left difference
Derivatives in Two Dimensions
(partial Derivatives)
yyyxfyxff
yf
xyxxfyxff
xf
yxf
yy
xx
∆∆−−
==∂∂
∆∆−−
==∂∂
→∆
→∆
),(),(lim
),(),(lim
),(
0
0
x
y
yx
yx
ff
ff
ff
1
22
tandirection
)(magnitude
VectorGradient ),(
−==
+=
θ
Laplacian2 =+=∆ yyxx fff
Derivatives of an Image
Derivative & average
Prewit
xf 101101101
−
−−
yf 111000111 −−−
⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢
⎣
⎡
=
20202010102020201010202020101020202010102020201010
),( yxI
⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢
⎣
⎡
=
0000000303000030300003030000000
xI
Derivatives of an Image
⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢
⎣
⎡
=
0000000000000000000000000
yI
⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢
⎣
⎡
=
20202010102020201010202020101020202010102020201010
),( yxI
Laplacian
0 −14
0
−14
1 −14
0 −14
0
fxx + fyy
Convolution
Convolution (contd)
),(*),(),(
),(),(),(1
1
1
1
yxgyxfyxh
jigjyixfyxhi j
=
++= ∑ ∑−= −=
h(x, y) = f (x −1,y −1)g(−1,−1) + f (x, y −1)g(0,−1) + f (x +1,y −1)g(1,−1) + f (x −1, y)g(−1,0) + f (x, y)g(0,0) + f (x +1,y)g(1,0) + f (x −1, y +1)g(−1,1) + f (x, y +1)g(0,1) + f (x +1, y +1)g(1,1)
Derivative Masks
Apply first mask to 1st imageSecond mask to 2nd imageAdd the responses to get f_x, f_y, f_t
xf
image second1111
imagefirst 1111
⎥⎦
⎤⎢⎣
⎡−−
⎥⎦
⎤⎢⎣
⎡−−
yf
image second1111
imagefirst 1111
⎥⎦
⎤⎢⎣
⎡ −−
⎥⎦
⎤⎢⎣
⎡ −−
tf
image second1111
imagefirst 1111
⎥⎦
⎤⎢⎣
⎡
⎥⎦
⎤⎢⎣
⎡−−−−
Synthetic Images
Horn & Schunck Results
One iteration 10 iterations
4=λ
Lucas & Kanade (Least Squares)
Optical flow eq
• Consider 3 by 3 windowtyx fvfuf −=+
999
111
tyx
tyx
fvfuf
fvfuf
−=+
−=+
M
⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢
⎣
⎡
−
−
=⎥⎦
⎤⎢⎣
⎡
⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢
⎣
⎡
9
1
9
1
9
1
t
t
y
y
x
x
f
f
vu
f
f
f
fMMM
tfAu =
Lucas & Kanade
tTT
tTT
t
fAAAu
fAAuA
fAu
1)( −=
=
=
2)(min tyixi fvfuf ++∑
Lucas & Kanade2)(min tyixi fvfuf ++∑
0)(
0)(
=++
=++
∑
∑
yitiyixi
xitiyixi
ffvfuf
ffvfuf
fxi2∑ u + fxi∑ fyiv = − fxi∑ f ti
fxi fyi∑ u + fyi2∑ v = − fyi∑ f ti
fxi2∑ fxi fyi∑
fxi fyi∑ fyi2∑
⎡
⎣
⎢ ⎢ ⎢ ⎢ ⎢
⎤
⎦
⎥ ⎥ ⎥ ⎥ ⎥
uv
⎡
⎣ ⎢
⎤
⎦ ⎥ =
− fxi f ti∑
− fyi f ti∑
⎡
⎣
⎢ ⎢ ⎢ ⎢ ⎢
⎤
⎦
⎥ ⎥ ⎥ ⎥ ⎥
( fxi∑ u + fyiv + f ti) f xi = 0
( fxi∑ u + fyiv + f ti) f yi = 0
Lucas & Kanade
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
−
−
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=⎥⎦
⎤⎢⎣
⎡
∑
∑
∑∑
∑∑−
tiyi
tixi
yiyixi
yixixi
ff
ff
fff
fff
vu
1
2
2
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
−
−
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
−
−
−=⎥
⎦
⎤⎢⎣
⎡
∑
∑
∑∑
∑∑
∑ ∑∑ tiyi
tixi
xiyixi
yix iyi
yixiyixi ff
ff
fff
fff
ffffvu
2
2
222 )(
1
∑ ∑∑
∑ ∑ ∑∑
∑ ∑∑
∑ ∑ ∑∑
−
−=
−
+−=
222
2
222
2
)(
)(
yixiyixi
tiy ix iyixitix i
y ix iyixi
tiyiyixitix iyi
ffff
fffffffv
ffff
fffffffu
Lucas-Kanadewithout pyramids
Fails in areas of large motion
Lucas-Kanade with Pyramids
Comments
Horn-Schunck and Lucas-Kanade optical method works only for small motion.If object moves faster, the brightness changes rapidly, 2x2 or 3x3 masks fail to estimate spatiotemporal derivatives.Pyramids can be used to compute large optical flow vectors.