1

Tzu ming Su

Advisor ： S.J.Wang

MOTION DETAIL PRESERVING OPTICAL FLOW ESTIMATION

2013/1/28

L. Xu, J. Jia, and Y. Matsushita. Motion detail preserving

optical flow estimation. In CVPR, 2010.

2

OUTLINE

2013/1/28

• Previous Work

• Optical flow

• Conventional optical flow estimation

3

MOTION FIELD

2013/1/28

• Definition ： an ideal representation of 3D motion as it is projected onto a camera image.

3D motion vector

2D optical flow vector

CCD

4

MOTION FIELD

2013/1/28

• Applications ：• Video enhancement ： stabilization, denoising, super resolution

• 3D reconstruction ： structure from motion (SFM)

• Video segmentation

• Tracking/recognition

• Advanced video editing (label propagation)

5

MOTION FIELD ESTIMATION

2013/1/28

• Optical flowRecover image motion at each pixel from spatio-temporal

image brightness variations

• Feature-trackingExtract visual features (corners, textured areas) and “track”

them over multiple frames

6

OPTICAL FLOW

2013/1/28

• Definition ： the apparent motion of brightness patterns in the images

• Map flow vector to color

• Magnitude: saturation

• Orientation: hue

7

OPTICAL FLOW

2013/1/28

• Key assumptions

• Brightness constancy

• Small motion

• Spatial coherence

Remark ： Brightness constancy is often violatedÞ Use gradient constancy for addition , both of them are called data constraint

8

BRIGHTNESS CONSISTENCY

2013/1/28

• 1-D case

Ix v It

9

BRIGHTNESS CONSISTENCY

x

)1,( txII ( x , t )

p?v

• 1-D case

2013/1/28

10

BRIGHTNESS CONSISTENCY

x

)1,( txII ( x , t )

p

xI

Spatial derivative

Temporal derivative v

• 1-D case

2013/1/28

11

BRIGHTNESS CONSISTENCY• 1-D case

• 2-D case

One equation, two velocity (u,v) unknowns…

u v

2013/1/28

12

APERTURE PROBLEM

2013/1/28

13

APERTURE PROBLEM

2013/1/28

14

APERTURE PROBLEM

2013/1/28

Time t

?Time t+dt

• We know the movement parallel to the direction of gradient , but not the movement orthogonal to the gradient

• We need additional constraints

15

CONVENTIONAL ESTIMATION

2013/1/28

• Use data consistency & additional constraint to estimate optical flow

• Horn-Schunck

• Minimize energy function with smoothness term

• Lucas-Kanade

• Minimize least square error function with local region coherence

16

HORN-SCHUNCK ESTIMATION

2013/1/28

• Imposing spatial smoothness to the flow field

• Adjacent pixels should move together as much as possible

• Horn & Schunck equation

17

HORN-SCHUNCK ESTIMATION

2013/1/28

• Use 2D Euler Lagrange

• Can be iteratively solved

18

COARSE TO FINE ESTIMATION

2013/1/28

• Optical flow is assumed to be small motions , but in fact most motions are not

• Solved by coarse to fine resolution

19

image Iimage It-1

COARSE TO FINE ESTIMATION

run iteratively

run iteratively...

2013/1/28

20

OUTLINE

2013/1/28

• Previous Work

• Contributions

• Extended Flow Initialization

• Selective data term

• Efficient optimization solver

• Experimental result

• Conclusion

21

OUTLINE

2013/1/28

• Previous Work

• Contributions

• Extended Flow Initialization

• Selective data term

• Efficient optimization solver

• Experimental result

• Conclusion

22

MUTI-SCALE PROBLEM

2013/1/28

• Conventional coarse to fine estimation can’t deal with large displacement.

• With different motion scales between foreground & background , even small motions can be miss detected.

23

MUTI-SCALE PROBLEM

2013/1/28

24

Ground truth

… Estimate EstimateEstimate

Ground truthGround truth

2013/1/28

25

MUTI-SCALE PROBLEM

2013/1/28

• Large discrepancy between initial values and optimal motion vectors

• Solution ： Improve flow initialization to reduce the reliance on the initialization from coarser levels

262013/1/28

Sparse feature matching

Fusion

Dense nearest-neighbor patch matching

Selection

EXTENDED FLOW INITIALIZATION• Sparse feature matching for each level

28

EXTENDED FLOW INITIALIZATION• Identify missing motion vectors

2013/1/28

29

EXTENDED FLOW INITIALIZATION• Identify missing motion vectors

2013/1/28

30

EXTENDED FLOW INITIALIZATION

…

2013/1/28

31

…

EXTENDED FLOW INITIALIZATION

Fuse

2013/1/28

322013/1/28

Sparse feature matching

Fusion

Dense nearest-neighbor patch matching

Selection

33

OUTLINE

2013/1/28

• Previous Work

• Contributions

• Extended Flow Initialization

• Selective data term

• Efficient optimization solver

• Experimental result

• Conclusion

34

CONSTRAINTS

2013/1/28

• Brightness consistency

• Gradient consistency

• Average

I 2 1(u, x) (x u) (x)I ID

I 2 1(u, x) (x u) (I I x)D

Ix

I

1 1(u, x) (u, x) (u, x)

2 2DE D D

35

• Pixels moving out of shadow

CONSTRAINTS

pI 1 1p(u , ) 6.63D

• Color constancy is violated

I Ip1 1 p1 1

1(u , ) (u , ) = 3.48

2p pD D

• Average:

p1u : ground truth motion of p1

• Gradient constancy holdsp 1I 1 p(u , ) 0.32D

2013/1/28

36

• Pixels undergoing rotational motion

CONSTRAINTS

• Color constancy holds

• Gradient constancy is violatedp2u : ground truth motion of p2

p 2I 2 p(u , ) 4.20D • Average:

I Ip2 2 p2 2

1(u , ) (u , ) = 2.24

2p pD D

pI 2 2p(u , ) 0.29D

2013/1/28

37

SELECTIVE DATA TERM• Selectively combine the constraints

where 2(x) : {0,1}

I Ix

(u, ) (u,x(x) 1) ( ) (u,x)(x)DE D D

2013/1/28

38

SELECTIVE DATA TERM

RubberWhale Urban22

2.5

3

3.5

4

4.5

5

colorgradientaverageours

AAE

selective

2013/1/28

39

OUTLINE

2013/1/28

• Previous Work

• Contributions

• Extended Flow Initialization

• Selective data term

• Efficient optimization solver

• Experimental result

• Conclusion

40

DISCRETE-OPTIMIZATION

2013/1/28

• Minimizing energy including discrete α & continuous u ：

• Try to separate α & u

• For α

• Probability of a particular state of MRF system

Ix

I(x) (u, x) (1 (x)) (u, x)(u, ) ( u, x)E SD D

(u, )1(u, ) EP e

Z

41

DISCRETE-OPTIMIZATION

2013/1/28

• Partition function

• Sum over all possible values of α

(u, )

{u} { 0,1}

EZ e

I Ix

( u,x) (u, x) ((x) (x)

{ 0,

1 ) (u, x)

{u 1}}

x

S D D

e e

(u, x)(u, x) II

x

1{ ( u,x) ln( )}

{u}

DDS e e

e

• Optimal condition (Euler-Lagrange equations)

• It decomposes to

II (u, x)(u, x)

x

1(u) ( u,x) ln( )DDeffE S e e

I I( (u,x) (u,x))

1(x)

1 D De

u I u I u(x) (u, x) (1 (x)) (u, x) div( ( u,x)) 0D D S

I I

I II I

(u,x) (u,x)

(u,x) (u,u I u I

u

x)(u,x) (u,x)(u, x) (u, x)

div( ( u,x)) 0

D D

D DD D

e e

e e e eD D

S

( )x 1 ( )x

DISCRETE-OPTIMIZATION

• Minimization – Update α– Compute flow field

43

CONTINUOUS-OPTIMIZATION

2013/1/28

• Energy function

• Variable splitting

44

CONTINUOUS-OPTIMIZATION

2013/1/28

• Fix u , estimate w,p

• Fix w,p , estimate u

• The Euler-Lagrange equation Is linear.

45

OUTLINE

2013/1/28

• Previous Work

• Contributions

• Extended Flow Initialization

• Selective data term

• Efficient optimization solver

• Experimental result

• Conclusion

46

SELECTIVE DATA TERM

Averaging Selective

Difference

2013/1/28

47

EXPERIMENTAL RESULTS

2013/1/28

48

RESULTS FROM DIFFERENT STEPS

Coarse-to-fine

Extended coarse-to-fine2013/1/28

492013/1/28

50

LARGE DISPLACEMENT

Overlaid Input 2013/1/28

51

LARGE DISPLACEMENT • Motion Estimates

Coarse-to-fine Result Warping Result2013/1/28

52

LARGE DISPLACEMENT • Motion Magnitude Maps

LDOP [Brox et al. 09 ] [Steinbrucker et al. 09]] Result2013/1/28

53

OVERLAID INPUT

2013/1/28

54Conventional Coarse-to-fine Result2013/1/28

55

EXPERIMENTAL RESULTS

Overlaid Input2013/1/28

56Coarse-to-fine Result2013/1/28

57

OUTLINE

2013/1/28

• Previous Work

• Contributions

• Extended Flow Initialization

• Selective data term

• Efficient optimization solver

• Experimental result

• Conclusion

58

CONCLUSION

2013/1/28

• To solve the coarse-to-fine problem , it seems more easier to make a correctness in every level.

• Using optical flow for small motion & other tracking skill for large displacement seems reasonable.

• It takes 40s ~ 3mins to compute an optical flow field respect to the amount of missing parts. Tradeoff problem.

592013/1/28

Thank you for your listening

60

FEATURE MATCHING

2013/1/28

• Feature ：” interesting “ , ” unique” part of image

• Two components of feature ：

Test image Detector: where are the local features?

Descriptor: how to describe them?

61

FEATURE MATCHING

2013/1/28

• Local measure of feature uniqueness Shifting the window in any direction causes a big change

“flat” :no change in all directions

“edge”: no change along the edge direction

“corner”:significant change in all directions

62

SIFT FEATURE MATCHING

2013/1/28

• SIFT ： Scale Invariant Feature Transform

• Problem： non-invariant between image scales

All points will be classified as edges

Corner

63

SIFT FEATURE MATCHING

2013/1/28

• Find scale that gives local maxima of some function f in both position and scale

64

SIFT FEATURE MATCHING

2013/1/28

• Function f ： Laplacian-of-Gaussian

65

SIFT FEATURE MATCHING

2013/1/28

• We define the characteristic scale as the scale that produces peak of Laplacian response

66

ALGORITHM• Scale-space extrema detection

• Keypoint localization

• Orientation assignment

• Keypoint descriptor

( )local descriptor

detector

descriptor

67

ALGORITHM• Scale-space extrema detection

• Keypoint localization

• Orientation assignment

• Keypoint descriptor

( )local descriptor

detector

descriptor

68

DETECTOR

2013/1/28

69

SCALE-SPACE EXTREMA DETECTION

2013/1/28

• Use Difference of Gaussian instead of LOG

• More efficient

DOG & LOG

70

KEYPOINT LOCALIZATION

2013/1/28

• X is selected if it is larger or smaller than all 26 neighbors

• Eliminating edge responses

71

ALGORITHM• Scale-space extrema detection

• Keypoint localization

• Orientation assignment

• Keypoint descriptor

( )local descriptor

detector

descriptor

72

ORIENTATION ASSIGNMENT

2013/1/28

• Use orientation histogram in the window to vote for total orientation

• Rotation-Invariant

73

KEYPOINT DESCRIPTOR

2013/1/28

• Describe the orientation histogram in 8x8 window near the pixel

• Illumination-robust

Back

74

PATCH MATCHING

2013/1/28

• SIFT still lose information about objects lacking features

• Using “patch” as a unit , minimizing

• Without smoothness term , it can detect large replacement , but also produce errors . Errors can be eliminate by fusion step.

75

PATCH MATCHING

2013/1/28

• Randomized Correspondence Algorithm

• Idea ： Coherent matches with neighbors

• Algorithm

• Initialization

• Propagation

• Search

76

PATCH MATCHING

2013/1/28Back

77

GRAPHIC CUT

2013/1/28

• Regard every pixel of image as a random variable , then the image is a “ random field .”

• Every pixel is only related to its neighbors , the filed is a “ Markov random field. ”(MRF)

• MRF can be viewed as a graph.

78

GRAPHIC CUT

2013/1/28

• Regard the optical field as a MRF.

• The value of a pixel is chosen within optical flow frames produced previously , it’s a “ labeling problem. ”

• The edges between pixels are smoothness relation.

• Cut the graph with minimum energy.

79

GRAPHIC CUT

2013/1/28

• Minimize the energy function

• Multi-labeling problem

• expansion move algorithm ： expanse the label which can decrease the energy

V. Lempitsky, S. Roth, and C. Rother, “Fusionflow: Discrete-Continuous Optimization for Optical Flow Estimation,”Proc. IEEE Conf. Computer Vision and Pattern Recognition,2008Back

80

FEATURE MATCHING

2013/1/28

• Find corners

Change of intensity for the shift [u,v]:

IntensityShifted intensity

Window function

orWindow function w(x,y) =

Gaussian1 in window, 0 outside

81

FEATURE MATCHING

2013/1/28

• For small shifts [u,v] we have a bilinear approximation:

where M is a 22 matrix computed from image derivatives:

82

FEATURE MATCHING

2013/1/28

2

“Corner”1 and 2 are large, 1 ~ 2;E increases in all directions

1 and 2 are small;E is almost constant in all directions

“Edge” 1 >> 2

“Edge” 2 >> 1

“Flat” region

Classification of image points using eigenvalues of M: