Post on 12-Jul-2020
Detailed 3D Reconstruction of Static Scenes from a Multi-camera SystemQIANG DUAN, COMP8755
SUPERVISOR: DR. YUCHAO DAI, DR. BEN SWIFT
THE RESEARCH SCHOOL OF COMPUTER SCIENCE
AUSTRALIAN NATIONAL UNIVERSITY
OCTOBER 2017
Introduction
* The image shown here is indicative only. The actual equipment may differ.[1] Image achieved from: https://www.bhphotovideo.com/c/buy/professional-360-video-cameras/ci/36033/N/3527143157
Reconstruct 3D scenes from 360 degree multi-camera system
Development tool:OPENCV and MATLAB
MotivationDevelop research and learning abilities for advanced technologies in computer science.
Have better understanding of multi-view geometry.
Design, implement an algorithm for 3D reconstruction of static scenes captured from multi-camera system.
1. Camera calibration
Estimates the parameters of a lens and image sensor of an image or video camera.
Three parameters:◦ Intrinsic parameter
◦Extrinsic parameter
◦Distortion coefficients
[2] image achieved from https://au.mathworks.com/help/vision/ug/camera-calibration.html
Pinhole camera model:intrinsic parameters
[2] image achieved from https://au.mathworks.com/help/vision/ug/camera-calibration.html
Extrinsic parametersdenote the coordinate system transformations from 3D world coordinates to 3D camera coordinates
[3] image achieved from: https://www.researchgate.net/figure/312376910_fig20_Figure-42-The-extrinsic-parameters-consisting-of-a-translation-vector-t-and-a-rotation
2. Image undistortRefer to OpenCV: undistort() function.
3. Feature matching using SIFT descriptor
4. Epipolar geometry and Structure-from-motion (SFM) algorithm
Epipolar constraint:1. If the projection point xL is known,
then the epipolar line eR––xR is knownand the point X projects into the rightimage, on a point xR which must lie onthis particular epipolar line
2. If the points xL and xR are known, theirprojection lines are also known. If thetwo image points correspond to thesame 3D point X the projection linesmust intersect precisely at X.
[4] image achieved from :https://en.wikipedia.org/wiki/File:Epipolar_geometry.svg
Result: Sparse 3D points
[5] triangulation process refers to MATLAB triangulate() function
5. Superpixel
[6] Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk, SLIC Superpixels Compared to State-of-the-art Superpixel Methods, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, num. 11, p. 2274 - 2282, May 2012.
Applying SLIC algorithm. Assume every superpixel is flat and pixels located on same plane.
6. HomographyDecompose homography matrix for each superpixel to obtain the normal vector and distance of its plane
Plane equation: 𝑛𝑇 ∗ 𝑋 + 𝑑 = 0
[7] image and equations are achieved from :Dingfu ZHOU, Yuchao Dai and Hongdong Li. Reliable Scale Estimation and Correction for Monocular Visual Odometry. In Intelligent Vehicles Symposium, 2016 Proceedings (IV 2016)
Overall process
Feature matching using SIFT sparse correspondence
Triangulation sparse 3D point cloud
Apply SLIC algorithm superpixel
Calculate and decompose homography matrix normal vector and distance
Construct plane according to normal vector, matched points and distance 3D planes
Optimizing assemble and glue planes to obotain dense 3D scene
Future works
Applying to panoramic scene reconstruction
A better feature matching algorithm can help
Apply dense feature matching instead of sparse matching
Thanks!
Any Question?
Intrinsic parameters illustration