An Introduction to the Optical Flow

Sylvain LOBRY

Contents

1 Introduction 2

2 Definition of the optical flow 2

3 Overview of the existing methods 33.1 Local methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 33.2 Global methods . . . . . . . . . . . . . . . . . . . . . . . . . . 4

4 Further ideas 5

1

1 Introduction

Optical flow is defined as the apparent motion of the brightness patterns inimage. Computing it leads to many applications, including video compressionand motion estimation. It is also the first step of many video inpaintingalgorithms.

Even though the first attempts of computing the optical flow has beenmade in 1981 by [3], the subject is the object of many recent studies leadingto many improvements.

In this introduction to the optical flow computation, we will start bygiving a formal definition of the problem. The existing methods will be thenoverviewed and classified. Finally, we will specify a roadmap for this study.

2 Definition of the optical flow

In the following, we will only handle the 2D case with 1D values (i.e. grayimages).

Let I(x, y, t) be the value of the (x, y) pixel at frame t and (u(x, y, t), v(x, y, t))the flow at the (x, y) pixel at frame t.

The main idea of most algorithms for computing optical flow is to use theBrightness Constancy : from a frame to an other, the brightness of a pixeldoes not change according to the flow. This can be written as :

I(x, y, t) = I(x+ u, y + v, t+ 1) (1)

Equation 1 can be approximate with Taylor series to

I(x, y, t) = I(x, y, t) + u∂I

∂x+ v

∂I

∂y+∂I

∂t(2)

Introducing the abbreviations Ex = ∂I∂x

, Ey = ∂I∂y

and Et = ∂I∂t

, the followingholds :

uEx + vEy + Et = 0 (3)

Equation 3 is often denoted as the Optical Flow Contraint. We have a singleequation with two unknowns. This ill-posed problem is often called theApperture Problem illustrated by figure 1.

2

(a) First frame (b) Second frame

Figure 1: We can not tell in which way the line is moving with only those 2frames.

To compute optical flow from equation 3, we must add additional con-straints to obtain a system with more equations. Finding those constraintsis the main challenge adressed by articles about optical flow.

3 Overview of the existing methods

Methods for computing optical flow can be classified into two categories. Thefirst one is known as local methods whereas the other take a global approachto the problem.

3.1 Local methods

Local methods are based on the approximation that the flow does not varyinside small regions. The first method using this assumption is [3].

Following this simple idea, we can obtain, given a window of size n × n

3

and by denoting pn the nth pixel inside :

Ex(p1)u+ Ey(p1)v + Et(p1) = 0

Ex(p2)u+ Ey(p2)v + Et(p2) = 0

. . .

Ex(pn2)u+ Ey(pn2)v + Et(pn2) = 0

Thus, this assumption allows us to have n2 equations for 2 unknowns,giving a simple system to solve.

Further developpements of this method includes a coarse-to-fine imple-mentation by pyramidal representation of the image. This have two advan-tages :

• We can take into account larger displacements.

• It can speed-up computations.

This coarse-to-fine implementations seem to be the most used optical flowcomputation algorithm into video inpainting solutions. On the opposite, ithas not been the object of many improvements during the recent years foroptical flow algorithms. Indeed, the global methods seems to have morepotential.

3.2 Global methods

Global methods generally works by favoring small first-order derivates of theflow field. The first method to follow this idea is [2] which considers the opti-cal flow computation by defining the measure of departure from smoothnessin the flow using an L2 norm :

E2s =

(∂u

∂x

)2

+

(∂u

∂y

)2

+

(∂v

∂x

)2

+

(∂v

∂y

)2

(4)

Using equations 3 (denoted Ec = uEx + vEy +Et) and 4 and introducingα2 to balance both terms, we can define the total error to be minimized :

Err2 =

∫∫(α2E2

s + E2c ) dx dy (5)

4

We can note that having a high value for α2 will lead to smoother flows. Thisfactor is application dependant.

Other approaches include weightening Es along with the gradients (|∇I|)to give less importance to areas having a high value for |∇I|. Indeed, itis more likely to find flow discontinuities at an edge than elsewhere. Theequation to be minimized become :

Err2 =

∫∫(α2w(|∇I|)E2

s + E2c ) dx dy (6)

Equation 6 can be improved by adopting an anisotropic approach such as in[5]. Some methods also look at the second-order derivate ([4]).

A method which might have a lot of potential is [1]. It considers severalenergies including ones given by high level descriptors such as SIFT. We willsee in section 4 an idea to improve it.

4 Further ideas

Understanding existing methods implies implementing a certain amount ofthem. After doing that, some idea must be considered :

• We can make a better use (than in [1]) of high level descriptors. Whereasin [1] the descriptor is used only fot the computation of the optical flowwhere it is defined, we want to use it to guide our computation. Indeed,those descriptors gives much more robust information than the opticalflow we could compute gives.

• We can use the fact that the optical flow is temporally smooth. Insection 3.2, we have seen how to use the fact that the optical flowis smooth inside a frame. We could apply this idea to many framesand trying to find the smoothest flow at each point. This idea couldbe combined with the traditionals global methods in order to improvetheir results.

Trying those ideas will be the main challenge of this semester.

5

References

[1] Thomas Brox and Jitendra Malik. Large displacement optical flow: De-scriptor matching in variational motion estimation. IEEE Trans. PatternAnal. Mach. Intell., 33(3):500–513, March 2011.

[2] Berthold K. P. Horn and Brian G. Schunck. Determining optical flow.ARTIFICAL INTELLIGENCE, 17:185–203, 1981.

[3] Bruce D. Lucas and Takeo Kanade. An iterative image registration tech-nique with an application to stereo vision. In Proceedings of the 7thinternational joint conference on Artificial intelligence - Volume 2, pages674–679, San Francisco, CA, USA, 1981. Morgan Kaufmann PublishersInc.

[4] Werner Trobin and Daniel Cremers. An unbiased second-order prior forhigh-accuracy motion estimation ?, 2008.

[5] Manuel Werlberger, Werner Trobin, Thomas Pock, Andreas Wedel,Daniel Cremers, and Horst Bischof. Anisotropic huber-l1 optical flow.In Proceedings of the British Machine Vision Conference (BMVC), Lon-don, UK, September 2009. to appear.

6