Improving Frame Interpolation With Spatial Motion...
Transcript of Improving Frame Interpolation With Spatial Motion...
![Page 1: Improving Frame Interpolation With Spatial Motion ...amalia.img.lx.it.pt/~jmda/Eurasip_presentation.pdf · Improving Frame Interpolation With Spatial Motion Smoothing For Pixel Domain](https://reader030.fdocuments.net/reader030/viewer/2022013007/5c63d1b109d3f2241d8bd024/html5/thumbnails/1.jpg)
Improving Frame Interpolation With Spatial Motion Smoothing For Pixel Domain
Distributed Video Coding
João [email protected]
Instituto Superior de Engenharia de Lisboa Instituto de Telecomunicações
Portugal
![Page 2: Improving Frame Interpolation With Spatial Motion ...amalia.img.lx.it.pt/~jmda/Eurasip_presentation.pdf · Improving Frame Interpolation With Spatial Motion Smoothing For Pixel Domain](https://reader030.fdocuments.net/reader030/viewer/2022013007/5c63d1b109d3f2241d8bd024/html5/thumbnails/2.jpg)
EC–SIP–M 200501/07/05
Summary
1. Context and Applications
2. Objectives
3. IST-PDWZ Video Codec
4. Frame Interpolation
5. Experimental Results
6. Conclusions and Future Work
![Page 3: Improving Frame Interpolation With Spatial Motion ...amalia.img.lx.it.pt/~jmda/Eurasip_presentation.pdf · Improving Frame Interpolation With Spatial Motion Smoothing For Pixel Domain](https://reader030.fdocuments.net/reader030/viewer/2022013007/5c63d1b109d3f2241d8bd024/html5/thumbnails/3.jpg)
EC–SIP–M 200501/07/05
Traditional Video Coding
– Predictive coding framework:– Encoder is 5 to 10 times more complex than decoder mainly due to
the motion estimation/compensation tools.– Well-suited for “one-to-many” topologies:
– Broadcasting or video-on-demand applications.
![Page 4: Improving Frame Interpolation With Spatial Motion ...amalia.img.lx.it.pt/~jmda/Eurasip_presentation.pdf · Improving Frame Interpolation With Spatial Motion Smoothing For Pixel Domain](https://reader030.fdocuments.net/reader030/viewer/2022013007/5c63d1b109d3f2241d8bd024/html5/thumbnails/4.jpg)
EC–SIP–M 200501/07/05
Emerging Applications, New Requirements
– Applications:– Mobile video.– Multimedia sensor networks.– Wireless video surveillance.– Multi-view acquisition.– …
– Encoding requirements:– Low complexity.– Low power-consumption.– Low cost.– ...
![Page 5: Improving Frame Interpolation With Spatial Motion ...amalia.img.lx.it.pt/~jmda/Eurasip_presentation.pdf · Improving Frame Interpolation With Spatial Motion Smoothing For Pixel Domain](https://reader030.fdocuments.net/reader030/viewer/2022013007/5c63d1b109d3f2241d8bd024/html5/thumbnails/5.jpg)
EC–SIP–M 200501/07/05
Need for a New Coding Paradigm
– Coding configuration with low-complexity and low-power encoder at the expense of a high-complexity decoder– Without compromising coding efficiency.
New video coding paradigm:
Distributed Video Coding (DVC) Distributed Video Coding (DVC) Exploits video statistics, partially or totally, at the decoder.
![Page 6: Improving Frame Interpolation With Spatial Motion ...amalia.img.lx.it.pt/~jmda/Eurasip_presentation.pdf · Improving Frame Interpolation With Spatial Motion Smoothing For Pixel Domain](https://reader030.fdocuments.net/reader030/viewer/2022013007/5c63d1b109d3f2241d8bd024/html5/thumbnails/6.jpg)
EC–SIP–M 200501/07/05
Summary
1. Context and Applications
2. Objectives
3. IST-PDWZ Video Codec
4. Frame Interpolation
5. Experimental Results
6. Conclusions and Future Work
![Page 7: Improving Frame Interpolation With Spatial Motion ...amalia.img.lx.it.pt/~jmda/Eurasip_presentation.pdf · Improving Frame Interpolation With Spatial Motion Smoothing For Pixel Domain](https://reader030.fdocuments.net/reader030/viewer/2022013007/5c63d1b109d3f2241d8bd024/html5/thumbnails/7.jpg)
EC–SIP–M 200501/07/05
Main Goal
The challenge is:
How to generate the best side information (a frame) as close as How to generate the best side information (a frame) as close as
possible to the current frame to be decoded ? possible to the current frame to be decoded ?
Distributed videodecoder
Lightweight encoder
Side Information
The rate-distortion (RD) performance of a distributed video coding scheme is highly dependent on the quality of the side information.
![Page 8: Improving Frame Interpolation With Spatial Motion ...amalia.img.lx.it.pt/~jmda/Eurasip_presentation.pdf · Improving Frame Interpolation With Spatial Motion Smoothing For Pixel Domain](https://reader030.fdocuments.net/reader030/viewer/2022013007/5c63d1b109d3f2241d8bd024/html5/thumbnails/8.jpg)
EC–SIP–M 200501/07/05
Summary
1. Context and Applications
2. Objectives
3. IST-PDWZ Video Codec
4. Frame Interpolation
5. Experimental Results
6. Conclusions and Future Work
![Page 9: Improving Frame Interpolation With Spatial Motion ...amalia.img.lx.it.pt/~jmda/Eurasip_presentation.pdf · Improving Frame Interpolation With Spatial Motion Smoothing For Pixel Domain](https://reader030.fdocuments.net/reader030/viewer/2022013007/5c63d1b109d3f2241d8bd024/html5/thumbnails/9.jpg)
EC–SIP–M 200501/07/05
IST Pixel-Domain Wyner-Ziv Video Codec
– Based on the Wyner-Ziv coding scenario.– Intra-frame encoder and inter-frame decoder.– X source: even frames of the video sequence.– Y source corresponds to the Side Information:
– Generated using the odd frames at the decoder – Y is an estimate of the current frame X and is available at the decoder.
– Channel coding of X allows to improve the quality of Y.
Inter-framedecoder
Intra-frame encoder
Virtual channeldependence
Side Information
![Page 10: Improving Frame Interpolation With Spatial Motion ...amalia.img.lx.it.pt/~jmda/Eurasip_presentation.pdf · Improving Frame Interpolation With Spatial Motion Smoothing For Pixel Domain](https://reader030.fdocuments.net/reader030/viewer/2022013007/5c63d1b109d3f2241d8bd024/html5/thumbnails/10.jpg)
EC–SIP–M 200501/07/05
IST Pixel-Domain Wyner-Ziv Video Codec
– Turbo code based Slepian-Wolf codec.– Each bitplane is independently turbo coded.– Two frame types:
– Key-frames (odd frames) and Wyner-Ziv frames (even frames).– Side information (Y2i) is obtained by frame interpolation.
Wyner-Ziv Encoder Wyner-Ziv Decoder
Slepian-Wolf Encoder Slepian-Wolf Decoder
2N- Level Uniform
Quantizer
X2i ..
..
Bitplane 1X'2i
Y2i
X2i - 1
X2i + 1
Wyner-Ziv Bitstream
Turbo Encoder
Turbo Decoder Reconstruction
FrameInterpolation
Buffer
Feedback Channel
Bitplane N
![Page 11: Improving Frame Interpolation With Spatial Motion ...amalia.img.lx.it.pt/~jmda/Eurasip_presentation.pdf · Improving Frame Interpolation With Spatial Motion Smoothing For Pixel Domain](https://reader030.fdocuments.net/reader030/viewer/2022013007/5c63d1b109d3f2241d8bd024/html5/thumbnails/11.jpg)
EC–SIP–M 200501/07/05
Slepian-Wolf Codec
– Turbo encoder:– Two identical recursive systematic
convolutional (RSC) encoders.– Pseudo-random interleaver.
– Each RSC encoder outputs the systematic and the parity streams:– Systematic stream is discarded.– Parity stream is stored in the buffer
and punctured.– Decoder request parity bits until
successful decoding.
– Iterative turbo decoder:– Two SISO decoders. – Maximum A Posteriori (MAP) algorithm.– Laplacian distribution to model the correlation between X and Y.
![Page 12: Improving Frame Interpolation With Spatial Motion ...amalia.img.lx.it.pt/~jmda/Eurasip_presentation.pdf · Improving Frame Interpolation With Spatial Motion Smoothing For Pixel Domain](https://reader030.fdocuments.net/reader030/viewer/2022013007/5c63d1b109d3f2241d8bd024/html5/thumbnails/12.jpg)
EC–SIP–M 200501/07/05
Summary
1. Context and Applications
2. Objectives
3. IST-PDWZ Video Codec
4. Frame Interpolation
5. Experimental Results
6. Conclusions and Future Work
![Page 13: Improving Frame Interpolation With Spatial Motion ...amalia.img.lx.it.pt/~jmda/Eurasip_presentation.pdf · Improving Frame Interpolation With Spatial Motion Smoothing For Pixel Domain](https://reader030.fdocuments.net/reader030/viewer/2022013007/5c63d1b109d3f2241d8bd024/html5/thumbnails/13.jpg)
EC–SIP–M 200501/07/05
Frame Interpolation: Architecture
– Block-based motion compensated interpolation (5 steps):– 1) Low pass filter to improve the motion vectors reliability.– 2) Forward motion estimation.– 3) Bi-directional motion refines initial motion vector estimate.– 4) Spatial smoothing of motion vectors.– 5) Bi-directional MC to fill the interpolated frame.
![Page 14: Improving Frame Interpolation With Spatial Motion ...amalia.img.lx.it.pt/~jmda/Eurasip_presentation.pdf · Improving Frame Interpolation With Spatial Motion Smoothing For Pixel Domain](https://reader030.fdocuments.net/reader030/viewer/2022013007/5c63d1b109d3f2241d8bd024/html5/thumbnails/14.jpg)
EC–SIP–M 200501/07/05
Initial MV & Bi-directional ME
– Motion vectors obtained by FME are candidates for each non-overlapped block in the interpolation frame:– Selected the MV that intercepts the interpolated frame closest to the
center of block.– Find linear trajectory (symmetric MVs) between key frames passing at the
center of the block in the interpolated frame:– Small displacement around the initial block position.
![Page 15: Improving Frame Interpolation With Spatial Motion ...amalia.img.lx.it.pt/~jmda/Eurasip_presentation.pdf · Improving Frame Interpolation With Spatial Motion Smoothing For Pixel Domain](https://reader030.fdocuments.net/reader030/viewer/2022013007/5c63d1b109d3f2241d8bd024/html5/thumbnails/15.jpg)
EC–SIP–M 200501/07/05
Spatial Motion Smoothing
– Spatial smoothing algorithms target: – Reduction on the number of false motion vectors when compared to
the true motion field.– Better spatial homogeneity of the resulting motion field.
– Weighted vector median filters are proposed:– Filter smoothing strength depends on the prediction MSE.– Low value weights when MSE for the candidate vector is high.– High value weights when MSE for the candidate vector is low.
1 1
N N
j wvmf j j i jL Lj j
w x x w x x= =
− ≤ −∑ ∑ ( )( )
,,
cj
j
MSE x Bw
MSE x B=
![Page 16: Improving Frame Interpolation With Spatial Motion ...amalia.img.lx.it.pt/~jmda/Eurasip_presentation.pdf · Improving Frame Interpolation With Spatial Motion Smoothing For Pixel Domain](https://reader030.fdocuments.net/reader030/viewer/2022013007/5c63d1b109d3f2241d8bd024/html5/thumbnails/16.jpg)
EC–SIP–M 200501/07/05
Spatial Motion Smoothing
– Without motion vector smoothing filter
![Page 17: Improving Frame Interpolation With Spatial Motion ...amalia.img.lx.it.pt/~jmda/Eurasip_presentation.pdf · Improving Frame Interpolation With Spatial Motion Smoothing For Pixel Domain](https://reader030.fdocuments.net/reader030/viewer/2022013007/5c63d1b109d3f2241d8bd024/html5/thumbnails/17.jpg)
EC–SIP–M 200501/07/05
Spatial Motion Smoothing
– With motion vector smoothing filter
![Page 18: Improving Frame Interpolation With Spatial Motion ...amalia.img.lx.it.pt/~jmda/Eurasip_presentation.pdf · Improving Frame Interpolation With Spatial Motion Smoothing For Pixel Domain](https://reader030.fdocuments.net/reader030/viewer/2022013007/5c63d1b109d3f2241d8bd024/html5/thumbnails/18.jpg)
EC–SIP–M 200501/07/05
Summary
1. Context and Applications
2. Objectives
3. IST-PDWZ Video Codec
4. Frame Interpolation
5. Experimental Results
6. Conclusions and Future Work
![Page 19: Improving Frame Interpolation With Spatial Motion ...amalia.img.lx.it.pt/~jmda/Eurasip_presentation.pdf · Improving Frame Interpolation With Spatial Motion Smoothing For Pixel Domain](https://reader030.fdocuments.net/reader030/viewer/2022013007/5c63d1b109d3f2241d8bd024/html5/thumbnails/19.jpg)
EC–SIP–M 200501/07/05
Coastguard Sequence
Coastguard sequence
26.5
28.5
30.5
32.5
34.5
36.5
38.5
40.5
42.5
0 50 100 150 200 250 300 350 400 450 500 550 600Rate of even frames (kbps)
PSN
R o
f eve
n fr
ames
(dB
)
H.263+ Intra
H.263+ I-B-I-B
PDWZ Average
PDWZ FME
PDWZ BiME
PDWZ SS
11.5 dB
![Page 20: Improving Frame Interpolation With Spatial Motion ...amalia.img.lx.it.pt/~jmda/Eurasip_presentation.pdf · Improving Frame Interpolation With Spatial Motion Smoothing For Pixel Domain](https://reader030.fdocuments.net/reader030/viewer/2022013007/5c63d1b109d3f2241d8bd024/html5/thumbnails/20.jpg)
EC–SIP–M 200501/07/05
Foreman Sequence
Foreman sequence
27.0
29.0
31.0
33.0
35.0
37.0
39.0
41.0
43.0
0 50 100 150 200 250 300 350 400 450 500 550 600 650Rate of even frames (kbps)
PSN
R o
f eve
n fr
ames
(dB
)
H.263+ IntraH.263+ I-B-I-BPDWZ AveragePDWZ FMEPDWZ BiMEPDWZ SS
10 dB
![Page 21: Improving Frame Interpolation With Spatial Motion ...amalia.img.lx.it.pt/~jmda/Eurasip_presentation.pdf · Improving Frame Interpolation With Spatial Motion Smoothing For Pixel Domain](https://reader030.fdocuments.net/reader030/viewer/2022013007/5c63d1b109d3f2241d8bd024/html5/thumbnails/21.jpg)
EC–SIP–M 200501/07/05
Rate-Distortion Comparison
Foreman QCIF Sequence @ 30fps
35.0
36.0
37.0
38.0
39.0
40.0
41.0
42.0
43.0
0 50 100 150 200 250 300 350 400
Rate of even frames [kbps]
PS
NR
of e
ven
fram
es [d
B]
H.263+ I-B-I-BSpatial Motion SmoothingStanford
2.6 dB
2 dB
![Page 22: Improving Frame Interpolation With Spatial Motion ...amalia.img.lx.it.pt/~jmda/Eurasip_presentation.pdf · Improving Frame Interpolation With Spatial Motion Smoothing For Pixel Domain](https://reader030.fdocuments.net/reader030/viewer/2022013007/5c63d1b109d3f2241d8bd024/html5/thumbnails/22.jpg)
EC–SIP–M 200501/07/05
Summary
1. Context and Applications
2. Objectives
3. IST-PDWZ Video Codec
4. Frame Interpolation
5. Experimental Results
6. Conclusions and Future Work
![Page 23: Improving Frame Interpolation With Spatial Motion ...amalia.img.lx.it.pt/~jmda/Eurasip_presentation.pdf · Improving Frame Interpolation With Spatial Motion Smoothing For Pixel Domain](https://reader030.fdocuments.net/reader030/viewer/2022013007/5c63d1b109d3f2241d8bd024/html5/thumbnails/23.jpg)
EC–SIP–M 200501/07/05
Conclusions & Future Work
– New motion compensated frame interpolation tools are proposed and compared in a DVC framework.
– Major gains in RD performance:– Essentially due to spatial motion smoothing.
– Future Work:– Adaptation to longer GOPs.– Improve performance when large camera movements occur.– Iterative motion refinement approach using decoded image and
keyframes.
![Page 24: Improving Frame Interpolation With Spatial Motion ...amalia.img.lx.it.pt/~jmda/Eurasip_presentation.pdf · Improving Frame Interpolation With Spatial Motion Smoothing For Pixel Domain](https://reader030.fdocuments.net/reader030/viewer/2022013007/5c63d1b109d3f2241d8bd024/html5/thumbnails/24.jpg)
Thanks for your attention !