CPSC 425: Computer Vision
Transcript of CPSC 425: Computer Vision
Lecture 17: Optical Flow (cont.)
CPSC 425: Computer Vision
!1
Menu for Today (March 11, 2020)Topics:
— Optical Flow (cont) — Classification
Redings: — Today’s Lecture: Forsyth & Ponce (2nd ed.) 15.1, 15.2
— Next Lecture: Forsyth & Ponce (2nd ed.) 16.1.3, 16.1.4, 16.1.9
Reminders: — Assignment 4: Local Invariant Features and RANSAC due Tuesday — Midterm graded. Grades will be released soon.
— Naive Bayes Classifier — Bayes' Risk
!3
Today’s “fun” Example: Visual Microphone
Follow-up work to previous lecture’s example of Eulerian video magnification
!3
Today’s “fun” Example: Visual Microphone
Follow-up work to previous lecture’s example of Eulerian video magnification
!4
Lecture 16: Re-cap
Optical flow is the apparent motion of brightness patterns in the image
Applications — image and video stabilization in digital cameras, camcorders — motion-compensated video compression schemes such as MPEG — image registration for medical imaging, remote sensing — action recognition — motion segmentation
!5Figure credit: M. Srinivasan
Lecture 16: Re-cap
Aperture Problem
!6
In which direction is the line moving?
Image Credit: Ioannis (Yannis) Gkioulekas (CMU)
Aperture Problem
!7
In which direction is the line moving?
Image Credit: Ioannis (Yannis) Gkioulekas (CMU)
Aperture Problem
!8 Image Credit: Ioannis (Yannis) Gkioulekas (CMU)
Aperture Problem
!9 Image Credit: Ioannis (Yannis) Gkioulekas (CMU)
Aperture Problem
!10 Image Credit: Ioannis (Yannis) Gkioulekas (CMU)
Aperture Problem
!11 Image Credit: Ioannis (Yannis) Gkioulekas (CMU)
Aperture Problem
— Without distinct features to track, the true visual motion is ambiguous
— Locally, one can compute only the component of the visual motion in the direction perpendicular to the contour
!12
Aperture Problem
— Without distinct features to track, the true visual motion is ambiguous
— Locally, one can compute only the component of the visual motion in the direction perpendicular to the contour
!13
Visual Motion
Visual motion is determined when there are distinct features to track, provided: — the features can be detected and localized accurately; and — the features can be correctly matched over time
!14
Motion as Matching
!15
Consider image intensity also to be a function of time, . We write
Optical Flow Constraint Equation
!16
I(x, y, t)
t
Consider image intensity also to be a function of time, . We write
Applying the chain rule for differentiation, we obtain
where subscripts denote partial differentiation
Optical Flow Constraint Equation
!17
I(x, y, t)
t
dI(x, y, t)
dt
= I
x
dx
dt
+ I
y
dy
dt
+ I
t
Consider image intensity also to be a function of time, . We write
Applying the chain rule for differentiation, we obtain
where subscripts denote partial differentiation
Define . and . Then is the 2-D motion and the space of all
such and is the 2-D velocity space
Optical Flow Constraint Equation
!18
I(x, y, t)
t
dI(x, y, t)
dt
= I
x
dx
dt
+ I
y
dy
dt
+ I
t
v =dy
dtu =
dx
dt
[u, v]
u v
Consider image intensity also to be a function of time, . We write
Applying the chain rule for differentiation, we obtain
where subscripts denote partial differentiation
Define . and . Then is the 2-D motion and the space of all
such and is the 2-D velocity space
Suppose . Then we obtain the (classic) optical flow constraint equation
Optical Flow Constraint Equation
!19
I(x, y, t)
t
dI(x, y, t)
dt
= I
x
dx
dt
+ I
y
dy
dt
+ I
t
v =dy
dtu =
dx
dt
dI(x, y, t)
dt
= 0
Ix
u+ Iy
v + It
= 0
[u, v]
u v
Consider image intensity also to be a function of time, . We write
Applying the chain rule for differentiation, we obtain
where subscripts denote partial differentiation
Define . and . Then is the 2-D motion and the space of all
such and is the 2-D velocity space
Suppose . Then we obtain the (classic) optical flow constraint equation
Optical Flow Constraint Equation
!20
I(x, y, t)
t
dI(x, y, t)
dt
= I
x
dx
dt
+ I
y
dy
dt
+ I
t
v =dy
dtu =
dx
dt
dI(x, y, t)
dt
= 0
Ix
u+ Iy
v + It
= 0
[u, v]
u v
Consider image intensity also to be a function of time, . We write
Applying the chain rule for differentiation, we obtain
where subscripts denote partial differentiation
Define . and . Then is the 2-D motion and the space of all
such and is the 2-D velocity space
Suppose . Then we obtain the (classic) optical flow constraint equation
Optical Flow Constraint Equation
!21
dI(x, y, t)
dt
= 0
Ix
u+ Iy
v + It
= 0
What does this mean, and why is it reasonable?
Consider image intensity also to be a function of time, . We write
Applying the chain rule for differentiation, we obtain
where subscripts denote partial differentiation
Define . and . Then is the 2-D motion and the space of all
such and is the 2-D velocity space
Suppose . Then we obtain the (classic) optical flow constraint equation
Optical Flow Constraint Equation
!22
dI(x, y, t)
dt
= 0
Ix
u+ Iy
v + It
= 0
What does this mean, and why is it reasonable?
Scene point moving through image sequence
Image Credit: Ioannis (Yannis) Gkioulekas (CMU)
Consider image intensity also to be a function of time, . We write
Applying the chain rule for differentiation, we obtain
where subscripts denote partial differentiation
Define . and . Then is the 2-D motion and the space of all
such and is the 2-D velocity space
Suppose . Then we obtain the (classic) optical flow constraint equation
Optical Flow Constraint Equation
!23
dI(x, y, t)
dt
= 0
Ix
u+ Iy
v + It
= 0
What does this mean, and why is it reasonable?
Scene point moving through image sequence
Image Credit: Ioannis (Yannis) Gkioulekas (CMU)
Consider image intensity also to be a function of time, . We write
Applying the chain rule for differentiation, we obtain
where subscripts denote partial differentiation
Define . and . Then is the 2-D motion and the space of all
such and is the 2-D velocity space
Suppose . Then we obtain the (classic) optical flow constraint equation
Optical Flow Constraint Equation
!24
dI(x, y, t)
dt
= 0
Ix
u+ Iy
v + It
= 0
What does this mean, and why is it reasonable?
constant
Brightness Constancy Assumption: Brightness of the point remains the same
Image Credit: Ioannis (Yannis) Gkioulekas (CMU)
Aside: Derivation of Optical Flow Constraint
!25
For small space-time step, brightness of a point is the same
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
),( yx
),( tvytux δδ ++
ttime tttime δ+
),( yx
Aside: Derivation of Optical Flow Constraint
!26
For small space-time step, brightness of a point is the same
Insight: If the time step is really small,
we can linearize the intensity function
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
Aside: Derivation of Optical Flow Constraint
!27
Multivariable Taylor Series Expansion (First order approximation, two variables)
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
Aside: Derivation of Optical Flow Constraint
!28
Multivariable Taylor Series Expansion (First order approximation, two variables)
assuming small motion
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
Aside: Derivation of Optical Flow Constraint
!29
Multivariable Taylor Series Expansion (First order approximation, two variables)
assuming small motionfixed point
partial derivative
cancel terms
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
Aside: Derivation of Optical Flow Constraint
!30
Multivariable Taylor Series Expansion (First order approximation, two variables)
assuming small motion
cancel terms
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
Aside: Derivation of Optical Flow Constraint
!31
Multivariable Taylor Series Expansion (First order approximation, two variables)
assuming small motion
divide by take limit
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
Aside: Derivation of Optical Flow Constraint
!32
Multivariable Taylor Series Expansion (First order approximation, two variables)
assuming small motion
divide by take limit
Brightness Constancy Equation
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
How do we compute …
!33 Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
How do we compute …
!34
spatial derivative
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
How do we compute …
!35
spatial derivative
Forward difference Sobel filter Scharr filter
…
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
How do we compute …
!36
spatial derivative
Forward difference Sobel filter Scharr filter
…
temporal derivative
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
How do we compute …
!37
spatial derivative
Forward difference Sobel filter Scharr filter
…
temporal derivative
Frame differencing
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
Frame Differencing: Example
!38
1 1 1 1 11 1 1 1 11 10 10 10 101 10 10 10 101 10 10 10 101 10 10 10 10
1 1 1 1 11 1 1 1 11 1 1 1 11 1 10 10 101 1 10 10 101 1 10 10 10
0 0 0 0 00 0 0 0 00 -9 -9 -9 -90 -9 0 0 00 -9 0 0 00 -9 0 0 0
- =
(example of a forward temporal difference)
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
!39
1 1 1 1 11 1 1 1 11 10 10 10 101 10 10 10 101 10 10 10 101 10 10 10 10
1 1 1 1 11 1 1 1 11 1 1 1 11 1 10 10 101 1 10 10 101 1 10 10 10
- 0 0 0 -- 0 0 0 -- 9 0 0 -- 9 0 0 -- 9 0 0 -- 9 0 0 -
- - - - -0 0 0 0 00 9 9 9 90 0 0 0 00 0 0 0 0- - - - -
0 0 0 0 00 0 0 0 00 -9 -9 -9 -90 -9 0 0 00 -9 0 0 00 -9 0 0 0
y
x
-1 0 1
-1 0 1
y
x
y
x
y
x
y
x
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
How do we compute …
!40
spatial derivative optical flow
Forward difference Sobel filter Scharr filter
…
temporal derivative
Frame differencingHow do you compute this?
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
How do we compute …
!41
spatial derivative optical flow
Forward difference Sobel filter Scharr filter
…
temporal derivative
Frame differencingWe need to solve for this! (this is the unknown in the
optical flow problem)
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
How do we compute …
!42
spatial derivative optical flow
Forward difference Sobel filter Scharr filter
…
temporal derivative
Frame differencingSolution lies on a line
Cannot be found uniquely with a single constraint
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
!43
Equation determines a straight line in velocity space
many combinations of u and v will satisfy the equality
Optical Flow Constraint Equation
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
Lucas-Kanade
Observations: 1. The 2-D motion, , at a given point, , has two degrees-of-freedom 2. The partial derivatives, , provide one constraint 3. The 2-D motion, , cannot be determined locally from alone
!44
[u, v] [x, y]
Ix
, Iy
, It
[u, v] Ix
, Iy
, It
Lucas-Kanade
Observations: 1. The 2-D motion, , at a given point, , has two degrees-of-freedom 2. The partial derivatives, , provide one constraint 3. The 2-D motion, , cannot be determined locally from alone
Lucas–Kanade Idea: Obtain additional local constraint by computing the partial derivatives, , in a window centered at the given
!45
[u, v] [x, y]
Ix
, Iy
, It
[u, v] Ix
, Iy
, It
Ix
, Iy
, It
[x, y]
Lucas-Kanade
Observations: 1. The 2-D motion, , at a given point, , has two degrees-of-freedom 2. The partial derivatives, , provide one constraint 3. The 2-D motion, , cannot be determined locally from alone
Lucas–Kanade Idea: Obtain additional local constraint by computing the partial derivatives, , in a window centered at the given
Constant Flow Assumption: nearby pixels will likely have same optical flow
!46
[u, v] [x, y]
Ix
, Iy
, It
[u, v] Ix
, Iy
, It
Ix
, Iy
, It
[x, y]
Suppose is the (original) center point in the window. Let be any other point in the window. This gives us two equations that we can write
and that can be solved locally for and as
provided that and are the same in both equations and provided that the required matrix inverse exists.
!47
[x1, y1] = [x, y] [x2, y2]
x
I
x1u+ I
y1v = �I
t1
I
x2u+ I
y2v = �I
t2
u v
uv
�= �
Ix1 I
y1
Ix2 I
y2
��1 It1
It2
�
u v
Lucas-Kanade Ix
u+ Iy
v + It
= 0Optical Flow Constraint Equation:
Considering all n points in the window, one obtains
which can be written as the matrix equation
where , and
!48
Ix1u+ I
y1v = �It1
Ix2u+ I
y2v = �It2
...Ixnu+ I
ynv = �Itn
A =
2
6664
Ix1 I
y1
Ix2 I
y2
......
Ixn I
yn
3
7775
b = �
2
6664
It1
It2
...Itn
3
7775
Av = b
v = [u, v]T
Lucas-Kanade
A =
2
6664
Ix1 I
y1
Ix2 I
y2
......
Ixn I
yn
3
7775
b = �
2
6664
It1
It2
...Itn
3
7775
Ix
u+ Iy
v + It
= 0Optical Flow Constraint Equation:
The standard least squares solution, , to is
again provided that and are the same in all equations and provided that the rank of is 2 (so that the required inverse exists)
!49
v̄ = (ATA)�1ATb
ATA
v̄ = (ATA)�1ATb
u v
Lucas-Kanade
The standard least squares solution, , to is
again provided that and are the same in all equations and provided that the rank of is 2 (so that the required inverse exists)
!50
v̄ = (ATA)�1ATb
ATA
v̄ = (ATA)�1ATb
u v
Lucas-Kanade
A =
2
6664
Ix1 I
y1
Ix2 I
y2
......
Ixn I
yn
3
7775
b = �
2
6664
It1
It2
...Itn
3
7775
A=
2 6 6 6 4
I x1
I y1
I x2
I y2
. . .. . .
I xn
I yn
3 7 7 7 5
b=
�
2 6 6 6 4
I t1
I t2 . . . I tn
3 7 7 7 5
Note that we can explicitly write down an expression for as
which is identical to the matrix that we saw in the context of Harris corner detection
!51
ATA =
PI2x
PIx
IyP
Ix
Iy
I2y
�
ATA
C
Lucas-Kanade
Note that we can explicitly write down an expression for as
which is identical to the matrix that we saw in the context of Harris corner detection
!52
ATA =
PI2x
PIx
IyP
Ix
Iy
I2y
�
ATA
C
Lucas-Kanade
What does that mean?
A dense method to compute motion, , at every location in an image
Key Assumptions:
1. Motion is slow enough and smooth enough that differential methods apply (i.e., that the partial derivatives, , are well-defined)
2. The optical flow constraint equation holds (i.e., )
3. A window size is chosen so that motion, , is constant in the window
4. A window size is chosen so that the rank of is 2 for the window
!53
Lucas-Kanade Summary
[u, v]
Ix
, Iy
, It
[u, v]
ATA
dI(x, y, t)
dt
= 0
Aside: Optical Flow Smoothness Constraint
Many methods trade off a ‘departure from the optical flow constraint’ cost with a ‘departure from smoothness’ cost.
The optimization objective to minimize becomes
where is a weighing parameter.
!54
E =
Z Z(I
x
u+ Iy
v + It
)2 + �(||5 u||2 + ||5 v||2)
�
!55
smoothness brightness constancy
weight
Horn-Schunck Optical Flow
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
Horn-Schunck Optical Flow
!56
Brightness constancy
Smoothness
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
SummaryMotion, like binocular stereo, can be formulated as a matching problem. That is, given a scene point located at in an image acquired at time , what is its position, , in an image acquired at time ?
Assuming image intensity does not change as a consequence of motion, we obtain the (classic) optical flow constraint equation
where , is the 2-D motion at a given point, , and are the partial derivatives of intensity with respect to , , and
Lucas–Kanade is a dense method to compute the motion, , at every location in an image
!57
(x0, y0) t0(x1, y1) t1
Ix
u+ Iy
v + It
= 0
[u, v] [x, y] Ix
, Iy
, It
x y t
[u, v]