Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays
description
Transcript of Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays
![Page 1: Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays](https://reader035.fdocuments.net/reader035/viewer/2022070417/568153f7550346895dc1f5b0/html5/thumbnails/1.jpg)
Reconstruction with Depth and Color cameras for 3D
Autostereoscopic Consumer Displays
SAIT – INRIA collaborationPeriod: 15 July 2012 / 15 January 2013
Date: 3-4 December 2012
![Page 2: Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays](https://reader035.fdocuments.net/reader035/viewer/2022070417/568153f7550346895dc1f5b0/html5/thumbnails/2.jpg)
INRIA team
• Georgios Evangelidis, postdoc, 100%• Michel Amat, development engineer, 100%• Soraya Arias, senior development engineer, 20% • Jan Cech, posdoc, 20%• Radu Horaud, 10%
![Page 3: Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays](https://reader035.fdocuments.net/reader035/viewer/2022070417/568153f7550346895dc1f5b0/html5/thumbnails/3.jpg)
Past achievements
• A method and software for aligning TOF data with a stereoscopic camera pair
• Extension to the calibration of several TOF-stereo units
• 3D texture-based rendering of the TOF data using the color-image information
![Page 4: Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays](https://reader035.fdocuments.net/reader035/viewer/2022070417/568153f7550346895dc1f5b0/html5/thumbnails/4.jpg)
Publications
• One CVPR 2011 paper• A tutorial at ICIP 2011• One Springer Briefs book
just published• The two teams published
several other papers
![Page 5: Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays](https://reader035.fdocuments.net/reader035/viewer/2022070417/568153f7550346895dc1f5b0/html5/thumbnails/5.jpg)
Current achievements
• Finalization of the calibration & rectification methods/software
• TOF to stereo-pair mapping with filtering• TOF + texture in live mode• Disparity map initialization• Stereo correspondence based on seed-growing• Final high-resolution depth map with gap filling• A paper submitted to CVPR’13
![Page 6: Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays](https://reader035.fdocuments.net/reader035/viewer/2022070417/568153f7550346895dc1f5b0/html5/thumbnails/6.jpg)
Improved Calibration• New calibration board– mat sticker glued to a
rigid plane– plane attached to a tripod
• Refined Calibration algorithm– TOF-Stereo Calibration error: <1.5 pixel
• Improved Rectification– Rectification error:
<0.25 pixel
![Page 7: Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays](https://reader035.fdocuments.net/reader035/viewer/2022070417/568153f7550346895dc1f5b0/html5/thumbnails/7.jpg)
Given a calibrated TOF-Stereo system
• Each TOF point PT defines a correspondence between PL and PR
![Page 8: Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays](https://reader035.fdocuments.net/reader035/viewer/2022070417/568153f7550346895dc1f5b0/html5/thumbnails/8.jpg)
Correspondences (samples) obtained by using the calibration parameters
• each correspondence comes from a TOF point• different color -> different depth
![Page 9: Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays](https://reader035.fdocuments.net/reader035/viewer/2022070417/568153f7550346895dc1f5b0/html5/thumbnails/9.jpg)
Correspondences (samples) obtained by using the calibration parameters
• each correspondence comes from a TOF point• different color -> different depth
![Page 10: Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays](https://reader035.fdocuments.net/reader035/viewer/2022070417/568153f7550346895dc1f5b0/html5/thumbnails/10.jpg)
TOF-to-Left Mapping
• We use the left image as reference
![Page 11: Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays](https://reader035.fdocuments.net/reader035/viewer/2022070417/568153f7550346895dc1f5b0/html5/thumbnails/11.jpg)
TOF-to-Left Mapping is not perfect
Resolution mismatch
![Page 12: Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays](https://reader035.fdocuments.net/reader035/viewer/2022070417/568153f7550346895dc1f5b0/html5/thumbnails/12.jpg)
Left-to-Tof Occlusions
TOF-to-Left Mapping is not perfect
Left-to-Tof Occlusions: the depth decreases from left to right
![Page 13: Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays](https://reader035.fdocuments.net/reader035/viewer/2022070417/568153f7550346895dc1f5b0/html5/thumbnails/13.jpg)
Tof-to-Left Occlusions
TOF-to-Left Mapping is not perfect
Tof-to-Left Occlusions: the depth increases from left to right
![Page 14: Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays](https://reader035.fdocuments.net/reader035/viewer/2022070417/568153f7550346895dc1f5b0/html5/thumbnails/14.jpg)
Point Cloud filtering• We reject points in left-to-tof occluded area• We keep the minimum-depth points in case of
overlap (due to Tof-to-left occlusions)
![Page 15: Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays](https://reader035.fdocuments.net/reader035/viewer/2022070417/568153f7550346895dc1f5b0/html5/thumbnails/15.jpg)
Disparity Map: Initialization• Run Delauney-Triangulation on low-resolution point
cloud
![Page 16: Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays](https://reader035.fdocuments.net/reader035/viewer/2022070417/568153f7550346895dc1f5b0/html5/thumbnails/16.jpg)
Disparity Map: Initialization• Run Delauney-Triangulation on low-resolution point
cloud…• …and initialize the stereo disparity map It looks good,
but it’s noisy and non-accurate!
![Page 17: Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays](https://reader035.fdocuments.net/reader035/viewer/2022070417/568153f7550346895dc1f5b0/html5/thumbnails/17.jpg)
Seed-Growing Idea• Start from points with known disparities
(seeds) and propagate the disparity to neighboring points (video?)
• Main issues:– What are our seeds?– What is the visiting order of seeds?– How do I propagate the message?– How the stereo and depth data are fused within
this framework?
![Page 18: Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays](https://reader035.fdocuments.net/reader035/viewer/2022070417/568153f7550346895dc1f5b0/html5/thumbnails/18.jpg)
Depth-Color Fusion • Built on the seed-growing idea– A:Depth data, S: Stereo data, dN : neighbor of d– For each pixel (node), find its disparity value that
maximizes the posterior probability (MAP)
SS
AA
d N
dInput data
Pixel with unknown disparity
Range-search constraintPenalize the choice
wrt to depth information
Penalize the choice wrt to color information
Pixel with known disparityA represents the initial estimation of d (obtained by the previous interpolation)S represents the color matching cost that corresponds to d
![Page 19: Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays](https://reader035.fdocuments.net/reader035/viewer/2022070417/568153f7550346895dc1f5b0/html5/thumbnails/19.jpg)
Depth-Color Fusion• Bayes rule translates each posterior into a
likelihood
• If likelihood terms are chosen from the exponential family, the “-log”-ness translates MAP into an energy minimization scheme
SS
AA
d N
d
Input data
Pixel with unknown disparity
Pixel with known disparity
We are currently working on these
terms!
Because of the uniform distribution
Because of theBayes rule
![Page 20: Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays](https://reader035.fdocuments.net/reader035/viewer/2022070417/568153f7550346895dc1f5b0/html5/thumbnails/20.jpg)
Seed-Growing Idea (revisited)• For each pixel, an energy function is defined and we
look for its minimizer (disparity)• Main issues:– What are our seeds?
• the points from Tof-to-Left mapping after refinement – What is the visiting order of seeds?
• First visit reliable seeds (points with low energy value)– How do we propagate the message?
• Given the disparity of a seed, bound the disparity-range for its neighbor
– How the stereo and depth data are fused within this framework?• Described above
![Page 21: Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays](https://reader035.fdocuments.net/reader035/viewer/2022070417/568153f7550346895dc1f5b0/html5/thumbnails/21.jpg)
Examples
White areas: unreliable matches
Black areas: Occlusions
![Page 22: Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays](https://reader035.fdocuments.net/reader035/viewer/2022070417/568153f7550346895dc1f5b0/html5/thumbnails/22.jpg)
Examples (with gap filling)
![Page 23: Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays](https://reader035.fdocuments.net/reader035/viewer/2022070417/568153f7550346895dc1f5b0/html5/thumbnails/23.jpg)
Examples (with gap filling)
![Page 24: Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays](https://reader035.fdocuments.net/reader035/viewer/2022070417/568153f7550346895dc1f5b0/html5/thumbnails/24.jpg)
Examples (with gap filling)
![Page 25: Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays](https://reader035.fdocuments.net/reader035/viewer/2022070417/568153f7550346895dc1f5b0/html5/thumbnails/25.jpg)
Examples (with gap filling)
![Page 26: Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays](https://reader035.fdocuments.net/reader035/viewer/2022070417/568153f7550346895dc1f5b0/html5/thumbnails/26.jpg)
Paper Submission
• Stereo-Depth Fusion for High-Resolution Disparity Maps. G. Evangelidis, R. Horaud, M. Amat, and S. Lee – submitted to CVPR 2013.
• An extended version of the CVPR submission is under preparation and it will be submitted to IEEE TPAMI in January/February 2013.
![Page 27: Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays](https://reader035.fdocuments.net/reader035/viewer/2022070417/568153f7550346895dc1f5b0/html5/thumbnails/27.jpg)
Work during the remaining month
• Improve the accuracy of the matching by better exploiting the color/texture information
• Currently the software implementation runs in offline-mode: We will provide a live-mode version at approximatively 1-2 frames/second
• An updated version will be available at the end of the period (~15 January 2013)
![Page 28: Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays](https://reader035.fdocuments.net/reader035/viewer/2022070417/568153f7550346895dc1f5b0/html5/thumbnails/28.jpg)
Prospects for the next collaboration(1 February 2013 – 31 January 2014)
• Finalize the TOF-stereo seed-growing algorithm, in particular improve the performance in non-textured areas
• Depth disambiguation using TOF-TOF and TOF-stereo • Combine depth disambiguation with the seed-growing
algorithm• Perform full 3D realistic rendering with four TOF-
stereo units• Perform continuous 3D reconstruction with a moving
TOF-stereo unit