3D-R2N2: A Unified Approach for Single and Multi-view 3D Object ReconstructionComputational Vision& Geometry Lab

Christopher B. Choy, Danfei Xu*, JunYoung Gwak*, Kevin Chen, Silvio Savarese

Stanford University* Indicates equal contribution

Motivation

- Goal: 3D reconstruction from sparse viewpoint

- Assumptions of previous works:- Dense Viewpoints- Lambertian and non uniform albedo- Shape Prior

- Single-view [Kar et al., Aubry et al.]- Multi-view [Bao et al.]

- General lighting condition- Dense viewpoints

Recurrent Neural Network

Recurrent Reconstruction Neural Network

- Long Short-Term Memory

- Gated Recurrent Unit

- Memory

- 3D Recurrent Reconstruction Neural Network

- Encoder- 3D-Convolutional LSTM- Decoder

3D-Convolutional LSTM

- Attention for viewpoint specific update: - LSTM cell assigned for each 3D region

- 3D Convolutional LSTM & GRU Single-view: PASCAL 3D+[4]

Multi-view Evaluations

Multi-view Stereo vs. Ours

Multi-view: ShapeNet[1] and real images

Acknowledgement

References

Deep Residual GRU/LSTM Network

GRU Input Gate Analysis

- NSF CAREER grant N.1054127 - Toyota Award #122282 - Korea Foundation for Advanced Studies- NSF GRFP

[1] Chang, A.X., et al., ShapeNet: An Information Rich 3D Model Repository, ArXiv 2015[2] Kar, A., et al., Category specific object reconstruction from a single image, CVPR 2015[3] Openmvs: open multi-view stereo reconstruction library[4] Xiang, Y., et al., Beyond pascal: A benchmark for 3d object detection in the wild

- 3D Convolution: Regularization

Side view Back view

- http://colah.github.io/posts/2015-08-Understanding-LSTMs/

- Contributions- Reconstruction free from assumptions above- Unifying single- and multi-view reconstruction- 3D convolutional LSTM

- Attention for view specific update

Rec

onst

ruct

ion

sam

ples

Fai

led

Rec

onst

ruct

ion

sam

ples

Fai

led