Post on 12-Jul-2015
Video Stabilization with L1-L2
Optimization
Hui Qu, Li Song
Institute of Image Communication and Network Engineering
Shanghai Jiao Tong University
17 September 2013
Outline
Introduction
Benchmark work
Our algorithm
L1-L2 mixed optimization model
Online video stabilization scheme
Experimental results
Conclusion
Introduction
Video is often shaky
Introduction
Video stabilization steps
Original camera path estimation
shaky
Introduction
Video stabilization steps
Smooth camera path computation
smooth
Introduction
Video stabilization steps
Synthesizing the stabilized video
shaky stabilized
Introduction
Video Stabilization methods
Original camera path estimation
2D camera path——2D linear model
3D camera path——Structure from Motion (SfM)
……
Smooth camera path computation
Filtering
Optimization
……
Synthesizing the stabilized video
Cropping——keep the central part
Inpainting——full frame
……
Grundmann et al. 2011
L1 camera path optimization
Integrated into Google ‘s YouTube Editor
Benchmark work
L1 Camera Path Optimization method
Original camera path: 𝐶
Optimal camera path: 𝑃
Objective function:
D means derivative operator
𝜔1, 𝜔2, 𝜔3 are empirical weights
2 3
1 2 31 1 1( ) ( ) ( ) ( )P D P D P D P
Benchmark work
L1 Camera Path Optimization method
constant path
static camera
Objective function:
2 3
1 2 31 1 1( ) ( ) ( ) ( )P D P D P D P
Benchmark work
L1 Camera Path Optimization method
path with constant velocity
panning or dolly shot
Objective function:
2 3
1 2 31 1 1( ) ( ) ( ) ( )P D P D P D P
Benchmark work
L1 Camera Path Optimization method
path with constant acceleration
ease in and out transition
Benchmark work
L1 Camera Path Optimization method
Inclusion constraint:
stabilized frame
Benchmark work
Problem of L1 Path Method
The method discards information due to cropping
not suitable for videos with important information near the boundary.
Benchmark work
Problem of L1 Path Method
𝜔1, 𝜔2, 𝜔3 are empirically set
hard to be adaptable to different kinds of videos
sequence 1 sequence 2
Our algorithm
L1-L2 mixed optimization model
Objective function:
2 3
1 1 1 2( ) ( ) ( ) ( )P D P D P D P P C
L1 part, ensure smoothness
L2 part, ensure proximity to original path
weight, adjust the degree of smoothness and fidelity
Our algorithm
L1-L2 mixed optimization model
Objective function:
2 3
1 1 1 2( ) ( ) ( ) ( )P D P D P D P P C
Our algorithm
L1-L2 mixed optimization model
Objective function:
2 3
1 1 1 2( ) ( ) ( ) ( )P D P D P D P P C
Our algorithm
L1-L2 mixed optimization model
𝜆 = 0.1 𝜆 = 0.5
𝜆 = 1.0 𝜆 = 2.0
Our algorithm
L1-L2 mixed optimization model
𝜆 can be relatively small 𝜆 should be relatively large
Our algorithm
Online processing scheme
…… … ……
time
segment 1:N frames
……
segment 2:N frames
K overlapped frames
Our algorithm
Online processing scheme
segment 1: N=150 frames
segment 2: N=150 frames
K=30 overlapped frames
Our algorithm
Online processing scheme
Optimal path for the overlapped frames
𝑃𝑡1
: optimal path of previous segment
𝑃𝑡2
: optimal path of current segment
𝜐𝑖: weights
1 21t i t i tP P P
0.5 cos 1 0.5i
i
K
Experimental results
L1 path vs. L1-L2 path
Input shaky video
Experimental results
L1 path vs. L1-L2 path
Grundmann et al’s result our result
Experimental results
L1 path vs. L1-L2 path
Grundmann et al’s result our result
Experimental results
Different values of 𝜆
Input shaky video
Experimental results
Different values of 𝜆
𝜆 = 0.1 𝜆 = 0.5
𝜆 = 1.0 𝜆 = 2.0
Experimental results
Different values of 𝜆
𝜆 = 0.1 80% crop
𝜆 = 0.5 85% crop
𝜆 = 1.0 90% crop
𝜆 = 2.0 95% crop
Experimental results
Speed
Platform
OS: Windows 7
CPU: Inter Core i5 & 3.1 GHz
Execution time
About 20 fps ( resolution: 640*360)
Conclusion
Video stabilization method by mixed L1-L2 optimization
Stabilize & preserve video content
Adjust the degree of stabilization according to different demands
Able to handle online stabilization and unlimited length videos
Thank you! Q&A
Insert inconsecutive frames of another video
original
stabilized
Transition between different scene
original
stabilized
Insert some consecutive frames of few features
original
stabilized