Technical University of Berlin Communication Systems Group Director: Prof. Thomas Sikora Carsten...
-
Upload
eric-adkins -
Category
Documents
-
view
220 -
download
0
Transcript of Technical University of Berlin Communication Systems Group Director: Prof. Thomas Sikora Carsten...
![Page 1: Technical University of Berlin Communication Systems Group Director: Prof. Thomas Sikora Carsten Clemens Clemens@nue.tu-berlin.de Error Concealment for.](https://reader033.fdocuments.net/reader033/viewer/2022051613/55146f93550346414e8b5ff9/html5/thumbnails/1.jpg)
Technical University of BerlinCommunication Systems Group
Director: Prof. Thomas Sikora
Carsten [email protected]
Error Concealment for Stereoscopic Sequences
ITG Fachausschusstagung 3.2, Juni 2006
![Page 2: Technical University of Berlin Communication Systems Group Director: Prof. Thomas Sikora Carsten Clemens Clemens@nue.tu-berlin.de Error Concealment for.](https://reader033.fdocuments.net/reader033/viewer/2022051613/55146f93550346414e8b5ff9/html5/thumbnails/2.jpg)
Error Concealment for Stereoscopic Sequences 2Communication Systems GroupC. Clemens
Technische Universität Berlin
• Monoscopic Error Concealment strategies are not well suited for stereoscopic scenario
• Assumtions
– independently coded views of a stereo image pair
• remaining redundancies between the channels, which can be utilized for error concealment
– block based coding (16x16 blocks)
Introduction
![Page 3: Technical University of Berlin Communication Systems Group Director: Prof. Thomas Sikora Carsten Clemens Clemens@nue.tu-berlin.de Error Concealment for.](https://reader033.fdocuments.net/reader033/viewer/2022051613/55146f93550346414e8b5ff9/html5/thumbnails/3.jpg)
Error Concealment for Stereoscopic Sequences 3Communication Systems GroupC. Clemens
Technische Universität Berlin
Algorithm Overview
Identification of corresponding region– feature extraction– feature matching along epipolar line– selection of matches (M-estimator/RANSAC)
Projective Transformation– initial parameter set from matches– optimization by Newton Method
Smoothing– only in case of discontinuities of depth
![Page 4: Technical University of Berlin Communication Systems Group Director: Prof. Thomas Sikora Carsten Clemens Clemens@nue.tu-berlin.de Error Concealment for.](https://reader033.fdocuments.net/reader033/viewer/2022051613/55146f93550346414e8b5ff9/html5/thumbnails/4.jpg)
Error Concealment for Stereoscopic Sequences 4Communication Systems GroupC. Clemens
Technische Universität Berlin
Matching and Transformation
projective transformation:
![Page 5: Technical University of Berlin Communication Systems Group Director: Prof. Thomas Sikora Carsten Clemens Clemens@nue.tu-berlin.de Error Concealment for.](https://reader033.fdocuments.net/reader033/viewer/2022051613/55146f93550346414e8b5ff9/html5/thumbnails/5.jpg)
Error Concealment for Stereoscopic Sequences 5Communication Systems GroupC. Clemens
Technische Universität Berlin
Selection of feature matches
• M-estimator– uses all matches with different weights– In some cases the transformation fails, because pixels from
outside the image were warped into the erroneous burst.
• RANSAC (random sample consensus)– uses a number of subsamples (four feature matches) – minimize the sum of squared residues of the boundary region:
RANSAC yields better results than M-estimator
![Page 6: Technical University of Berlin Communication Systems Group Director: Prof. Thomas Sikora Carsten Clemens Clemens@nue.tu-berlin.de Error Concealment for.](https://reader033.fdocuments.net/reader033/viewer/2022051613/55146f93550346414e8b5ff9/html5/thumbnails/6.jpg)
Error Concealment for Stereoscopic Sequences 6Communication Systems GroupC. Clemens
Technische Universität Berlin
Adapted Newton Method
• Find the optimal transformation parameter by minimizing a cost function C(k):
• b is the Border Region of the erroneous block burst
T (k ;(xl, yl))
left view right view
b
![Page 7: Technical University of Berlin Communication Systems Group Director: Prof. Thomas Sikora Carsten Clemens Clemens@nue.tu-berlin.de Error Concealment for.](https://reader033.fdocuments.net/reader033/viewer/2022051613/55146f93550346414e8b5ff9/html5/thumbnails/7.jpg)
Error Concealment for Stereoscopic Sequences 7Communication Systems GroupC. Clemens
Technische Universität Berlin
Adapted Newton Method
• Iteration step:
• Problem I: Local minimum solution
• Initial Parameter set is of prime importance
Cost function C(k) over horizontal and vertical translation parameter
![Page 8: Technical University of Berlin Communication Systems Group Director: Prof. Thomas Sikora Carsten Clemens Clemens@nue.tu-berlin.de Error Concealment for.](https://reader033.fdocuments.net/reader033/viewer/2022051613/55146f93550346414e8b5ff9/html5/thumbnails/8.jpg)
Error Concealment for Stereoscopic Sequences 8Communication Systems GroupC. Clemens
Technische Universität Berlin
Adapted Newton Method
• Problem II: Convergence of Newton method
• Successivly decreasing of border pixel size L after every minimum search
Speed of convergence of the Newthon algorithm for different border sizes L
![Page 9: Technical University of Berlin Communication Systems Group Director: Prof. Thomas Sikora Carsten Clemens Clemens@nue.tu-berlin.de Error Concealment for.](https://reader033.fdocuments.net/reader033/viewer/2022051613/55146f93550346414e8b5ff9/html5/thumbnails/9.jpg)
Error Concealment for Stereoscopic Sequences 9Communication Systems GroupC. Clemens
Technische Universität Berlin
3D Block Smoothing
• In case of great discontinuities in depth (variation of disparity) we perform a linear smoothing algorithm towards the surrounding pixel region (3D-BS)
• Minimization of the intersample variance between neighboring samples and to the block borders
![Page 10: Technical University of Berlin Communication Systems Group Director: Prof. Thomas Sikora Carsten Clemens Clemens@nue.tu-berlin.de Error Concealment for.](https://reader033.fdocuments.net/reader033/viewer/2022051613/55146f93550346414e8b5ff9/html5/thumbnails/10.jpg)
Error Concealment for Stereoscopic Sequences 10Communication Systems GroupC. Clemens
Technische Universität Berlin
HQ EC: Results / Example
![Page 11: Technical University of Berlin Communication Systems Group Director: Prof. Thomas Sikora Carsten Clemens Clemens@nue.tu-berlin.de Error Concealment for.](https://reader033.fdocuments.net/reader033/viewer/2022051613/55146f93550346414e8b5ff9/html5/thumbnails/11.jpg)
Error Concealment for Stereoscopic Sequences 11Communication Systems GroupC. Clemens
Technische Universität Berlin
Results / Subjective Evaluation
• Subjective Simulation Results:
– Double Stimulus Continuous Quality Scale Method (DSCQS) as phsychovisual test with 15 subjects
– Shutterglasses (StereoGraphics)
DSCQS method
-0,5
0,5
1,5
2,5
3,5
4,5
5,5
6,5
hall castle tower over all
monoscopic NEWTburst NEWT 3D-BS
DMOS
![Page 12: Technical University of Berlin Communication Systems Group Director: Prof. Thomas Sikora Carsten Clemens Clemens@nue.tu-berlin.de Error Concealment for.](https://reader033.fdocuments.net/reader033/viewer/2022051613/55146f93550346414e8b5ff9/html5/thumbnails/12.jpg)
Error Concealment for Stereoscopic Sequences 12Communication Systems GroupC. Clemens
Technische Universität Berlin
Fast EC: Algorithm Overview
• Block Search– Directional Diamond Search– SAD
• Side Match Distortion ||1 out
jinjsm bb
Nd
![Page 13: Technical University of Berlin Communication Systems Group Director: Prof. Thomas Sikora Carsten Clemens Clemens@nue.tu-berlin.de Error Concealment for.](https://reader033.fdocuments.net/reader033/viewer/2022051613/55146f93550346414e8b5ff9/html5/thumbnails/13.jpg)
Error Concealment for Stereoscopic Sequences 13Communication Systems GroupC. Clemens
Technische Universität Berlin
Fast EC: Matching Example „Hall“
• SMD determined for each possible position• Position with minimum SMD selected• Block used for reconstruction
![Page 14: Technical University of Berlin Communication Systems Group Director: Prof. Thomas Sikora Carsten Clemens Clemens@nue.tu-berlin.de Error Concealment for.](https://reader033.fdocuments.net/reader033/viewer/2022051613/55146f93550346414e8b5ff9/html5/thumbnails/14.jpg)
Error Concealment for Stereoscopic Sequences 14Communication Systems GroupC. Clemens
Technische Universität Berlin
Fast EC: Subjective Evaluation
BALLOON DMOS S.D. conf. interval
Interpolation 5,69 1,19 ±0,65
SBS 4,39 1,12 ±0,62
TBS 2,52 2,64 ±1,45
TSBS 2,49 1,37 ±0,75
![Page 15: Technical University of Berlin Communication Systems Group Director: Prof. Thomas Sikora Carsten Clemens Clemens@nue.tu-berlin.de Error Concealment for.](https://reader033.fdocuments.net/reader033/viewer/2022051613/55146f93550346414e8b5ff9/html5/thumbnails/15.jpg)
Error Concealment for Stereoscopic Sequences 15Communication Systems GroupC. Clemens
Technische Universität Berlin
Simple Matching - Example
• ‘Balloons’ 720x480, corrupted frame and features
![Page 16: Technical University of Berlin Communication Systems Group Director: Prof. Thomas Sikora Carsten Clemens Clemens@nue.tu-berlin.de Error Concealment for.](https://reader033.fdocuments.net/reader033/viewer/2022051613/55146f93550346414e8b5ff9/html5/thumbnails/16.jpg)
Error Concealment for Stereoscopic Sequences 16Communication Systems GroupC. Clemens
Technische Universität Berlin
Simple Matching - Example
• Reference frame (temporal) with features
![Page 17: Technical University of Berlin Communication Systems Group Director: Prof. Thomas Sikora Carsten Clemens Clemens@nue.tu-berlin.de Error Concealment for.](https://reader033.fdocuments.net/reader033/viewer/2022051613/55146f93550346414e8b5ff9/html5/thumbnails/17.jpg)
Error Concealment for Stereoscopic Sequences 17Communication Systems GroupC. Clemens
Technische Universität Berlin
Simple Matching - Example
• Reference blocks
![Page 18: Technical University of Berlin Communication Systems Group Director: Prof. Thomas Sikora Carsten Clemens Clemens@nue.tu-berlin.de Error Concealment for.](https://reader033.fdocuments.net/reader033/viewer/2022051613/55146f93550346414e8b5ff9/html5/thumbnails/18.jpg)
Error Concealment for Stereoscopic Sequences 18Communication Systems GroupC. Clemens
Technische Universität Berlin
Simple Matching - Example
• Reconstructed frame
![Page 19: Technical University of Berlin Communication Systems Group Director: Prof. Thomas Sikora Carsten Clemens Clemens@nue.tu-berlin.de Error Concealment for.](https://reader033.fdocuments.net/reader033/viewer/2022051613/55146f93550346414e8b5ff9/html5/thumbnails/19.jpg)
Error Concealment for Stereoscopic Sequences 19Communication Systems GroupC. Clemens
Technische Universität Berlin
Publications
• K. Günther, C. Clemens, and T. Sikora
A Fast Displacement-Estimation Based Approach For Stereoscopic Error Concealment
PCS 2004, San Francisco
• C. Clemens, M. Kunter, S. Knorr, and T. Sikora:
A hybrid approach for error concealment in stereoscopic images
WIAMIS '04, Lissabon
• M. Kunter, S. Knorr, C. Clemens, and T. Sikora:
A gradient based approach for stereoscopic error concealment
ICIP '04, Singapore
• S. Knorr, C. Clemens, M. Kunter, and T. SikoraRobust Concealment for Erroneous Block Bursts in Stereoscopic Images3D Data Processing, Visualization, and Transmission (3DPVT'04), Thessaloniki, Greece
• K. Günther, C. Clemens, and T. Sikora
A Fast Displacement-Estimation Based Approach For Stereoscopic Error Concealment
PCS 2004, San Francisco
• C. Clemens, M. Kunter, S. Knorr, and T. Sikora:
A hybrid approach for error concealment in stereoscopic images
WIAMIS '04, Lissabon
• M. Kunter, S. Knorr, C. Clemens, and T. Sikora:
A gradient based approach for stereoscopic error concealment
ICIP '04, Singapore
• S. Knorr, C. Clemens, M. Kunter, and T. SikoraRobust Concealment for Erroneous Block Bursts in Stereoscopic Images3D Data Processing, Visualization, and Transmission (3DPVT'04), Thessaloniki, Greece