Visualizing BYOD - Get the most outof personal devices in the workforce
Out-of-focus: Learning Depth from...
1
… … Image 1 Image i Image N Depth Image … … 640 x 480 x 3 512 x 424 x 1 224 x 224 x 3 224 x 224 x 3 224 x 224 x 3 24 x 24 x 1 640 x 480 x 3 640 x 480 x 3 Raw Data … … 470 x 470 x 3 291 x 291 x 1 470 x 470 x 3 470 x 470 x 3 Filterd / Aligned Training Out-of-focus: Learning Depth from Image Bokeh for Robotic Perception Eric Cristofalo and Zijian Wang, {ecristof, zjwang}@stanford.edu MSL INTRODUCTION DATA ACQUISITION CONVOLUTIONAL NEURAL NETWORK ACKNOWLEDGEMENTS REFERENCES We would like to acknowledge Professor Mac Schwager and the Multi-robot Systems Lab (MSL) in the department of Aeronautics and Astronautics at Stanford University. [1] Liu, Fayao, et al. "Learning depth from single monocular images using deep convolutional neural fields,” 2015. [2] S. Choi, D. Min, B. Ham, Y. Kim, C. Oh, and K. Sohn, “Depth analogy: Data-driven approach for single image depth estimation using gradient samples,” IEEE Transactions on Image Processing, vol. 24, no. 12, pp. 5953–5966, 2015. [3] K. Lai, L. Bo, X. Ren, and D. Fox, “A large-scale hierarchical multi-view rgb-d object dataset,” ICRA 2011 [4] S. Pertuz, D. Puig, and M. A. Garcia, “Analysis of focus measure operators for shape-from-focus,” Pattern Recognition, 2013. [5] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, “Indoor segmentation and support inference from rgbd images,” ECCV, 2012. • VGG-like Convolutional Neural Network (Figure 3) • Input: 224x224 single or stacked RGB image(s). For N focal stack images input, we superpose them together and make the depth 3*N • 6 convolution layers with stride 3x3 using ReLU activation • 4 max pooling layers • 2 fully connected layers • Output: 24x24 (flattened) depth estimation image • Loss function: mean square error for all pixel depth values • Implemented in Keras with TensorFlow backend • Data are split into ~ 85% training examples and 15% test examples • Images are shuffled before training • [Exp1] Training with NYU Depth V2 dataset [5]. 11670 examples (single RGB input to the neural net) in total. Took 6 hours for 30-epoch training on a 12-core, 64-gb memory workstation. • [Exp2] Training with our own dataset. However, the input is just single RGB image, for the purpose of comparison with our focal stack method. Took 3.5 hours for 30-epoch training. • [Exp3] Training with our own dataset (3674 examples). Each example contains 10 RGB images input taken under different focal lengths (36740 images used in total). • Same network structure for fair comparison (see Table 1) [Exp1] Testing prediction: depth from single RGB images in NYU dataset [5] CONCLUSION AND FUTURE WORK • Showed CNN with focus data in images yields lower training and testing error than single RGB images • Experimentally acquired novel data set of labeled, out-of-focus images • Future work: Collect varied data in other environments (e.g. outdoors) with different RGB optics that contain more depth information at larger distances in the Kinect’s workspace – [0.4,4] meters CS 229 Final Project Poster Session December 13, 2016 RESULTS • Autonomous robotic systems require detailed environmental perception (e.g. 3D reconstructions) to complete challenging tasks such as object recognition, obstacle avoidance, planning, and manipulation • Pixel depth estimation typically comes from triangulating image features, learning from single RGB images [1,2,3], or using laser rangefinders • Instead, we employ the out-of-focus aesthetic – bokeh – of an image to determine a pixel-wise depth map via supervised learning using out-of- focus images from a camera with labeled RGB-D sensor information • Current Depth from Focus (DfF) methods quantify the focus measure of out-of-focus images using specific focus descriptors [4] • Project Goal: Learn pixel-wise depth mapping from labeled focal- stack (sequence of images with varying focal-depth ) using Convolutional Neural Networks (CNN) • Acquired 344 labeled data examples of office scenes include 52 image focal-stack and RGB-D image • Hardware (Figure 1): 1) Logitech C920 USB webcam for varying focus images 2) Microsoft Kinect V2 for ground truth depth images • Acquire data using ROS and OpenCV in C++ • Performed data filtering, alignment, and scaling in MATLAB using OpenCV (Figure 2) • Replicated data using random 3D perspective transforms to yield 3674 data examples for the network Figure 3. Convolutional Neural Network architecture using multiple input images from focal stack 224 x 224 x 3 x N 224 x 224 x 3 x N 112 x 112 x 16 56 x 56 x 32 28 x 28 x 64 14 x 14 x 128 Convolution + ReLU Max Pooling Fully Connected + ReLU Softmax 1 x 1 x 1024 1 x 1 x 576 Figure 2. Example data from training set. Raw focal stack (N images with varying focal depth) and corresponding depth map (top row). Filtered and aligned data (middle row). Training images and labeled data (bottom row). Figure 1. Mobile experimental data collection setup. [Exp2] Testing prediction: depth from single RGB images in our dataset [Exp3] Testing prediction: depth from focal-stack images in our dataset Learning Curves on training data for all three data sets IMPLEMENTATION & EXPERIMENTS 5 10 15 20 25 30 0 10 20 30 40 50 60 70 Epochs Mean Squared Error (pixels 2 ) Single RGB Image − NYU Single RGB Image RGB Focal Stack 5 10 15 20 25 30 0 1000 2000 3000 4000 5000 6000 7000 Epochs Loss (pixels) Single RGB Image − NYU Single RGB Image RGB Focal Stack Ground Truth Ground Truth Prediction Prediction Error: 5911 Error: 7180 Figure 4. Learning curves with training data showing the mse loss (left) and mean absolute error (middle) per pixel. Note that the final testing error for the focal stacks is slightly less than the single image data. Link to all testing results for our data set (right). Table 1. Training and testing error results for all three data sets Ground Truth Prediction Link for all testing results
Transcript of Out-of-focus: Learning Depth from...
… …
Image 1 Image i Image N Depth Image
… …
640 x 480 x 3! 512 x 424 x 1!
224 x 224 x 3! 224 x 224 x 3! 224 x 224 x 3! 24 x 24 x 1!
640 x 480 x 3! 640 x 480 x 3!
Raw
Dat
a
… …
470 x 470 x 3! 291 x 291 x 1!470 x 470 x 3! 470 x 470 x 3!
Filte
rd /
Alig
ned
Trai
ning
Out-of-focus: Learning Depth from Image Bokeh for Robotic Perception
Eric Cristofalo and Zijian Wang, {ecristof, zjwang}@stanford.edu MSLINTRODUCTION
DATA ACQUISITION
CONVOLUTIONAL NEURAL NETWORK
ACKNOWLEDGEMENTS
REFERENCES
We would like to acknowledge Professor Mac Schwager and the Multi-robot Systems Lab (MSL) in the department of Aeronautics and Astronautics at Stanford University.
[1] Liu, Fayao, et al. "Learning depth from single monocular images using deep convolutional neural fields,” 2015.[2] S. Choi, D. Min, B. Ham, Y. Kim, C. Oh, and K. Sohn, “Depth analogy: Data-driven approach for single image depth estimation using gradient samples,” IEEE Transactions on Image Processing, vol. 24, no. 12, pp. 5953–5966, 2015.[3] K. Lai, L. Bo, X. Ren, and D. Fox, “A large-scale hierarchical multi-view rgb-d object dataset,” ICRA 2011[4] S. Pertuz, D. Puig, and M. A. Garcia, “Analysis of focus measureoperators for shape-from-focus,” Pattern Recognition, 2013.[5] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, “Indoor segmentation and support inference from rgbd images,” ECCV, 2012.
• VGG-like Convolutional Neural Network (Figure 3)• Input: 224x224 single or stacked RGB image(s). For N focal stack images
input, we superpose them together and make the depth 3*N• 6 convolution layers with stride 3x3 using ReLU activation• 4 max pooling layers• 2 fully connected layers• Output: 24x24 (flattened) depth estimation image• Loss function: mean square error for all pixel depth values
• Implemented in Keras with TensorFlow backend• Data are split into ~ 85% training examples and 15% test examples• Images are shuffled before training• [Exp1] Training with NYU Depth V2 dataset [5]. 11670 examples (single
RGB input to the neural net) in total. Took 6 hours for 30-epoch training on a 12-core, 64-gb memory workstation.
• [Exp2] Training with our own dataset. However, the input is just single RGB image, for the purpose of comparison with our focal stack method. Took 3.5 hours for 30-epoch training.
• [Exp3] Training with our own dataset (3674 examples). Each example contains 10 RGB images input taken under different focal lengths (36740 images used in total).
• Same network structure for fair comparison (see Table 1)
[Exp1] Testing prediction: depth from single RGB images in NYU dataset [5]
CONCLUSION AND FUTURE WORK• Showed CNN with focus data in images yields lower training and testing
error than single RGB images• Experimentally acquired novel data set of labeled, out-of-focus images• Future work: Collect varied data in other environments (e.g. outdoors) with
different RGB optics that contain more depth information at larger distances in the Kinect’s workspace – [0.4,4] meters
CS 229 Final Project Poster Session December 13, 2016
RESULTS• Autonomous robotic systems require detailed environmental perception
(e.g. 3D reconstructions) to complete challenging tasks such as object recognition, obstacle avoidance, planning, and manipulation
• Pixel depth estimation typically comes from triangulating image features, learning from single RGB images [1,2,3], or using laser rangefinders
• Instead, we employ the out-of-focus aesthetic – bokeh – of an image to determine a pixel-wise depth map via supervised learning using out-of-focus images from a camera with labeled RGB-D sensor information
• Current Depth from Focus (DfF) methods quantify the focus measure of out-of-focus images using specific focus descriptors [4]
• Project Goal: Learn pixel-wise depth mapping from labeled focal-stack (sequence of images with varying focal-depth) using Convolutional Neural Networks (CNN)
• Acquired 344 labeled data examples of office scenes include 52 image focal-stack and RGB-D image
• Hardware (Figure 1): 1) Logitech C920 USB webcam for varying focus
images2) Microsoft Kinect V2 for ground truth depth images• Acquire data using ROS and OpenCV in C++• Performed data filtering, alignment, and scaling in
MATLAB using OpenCV (Figure 2)• Replicated data using random 3D perspective
transforms to yield 3674 data examples for the network
Figure 3. Convolutional Neural Network architecture using multiple input images from focal stack
224 x 224 x 3 x N! 224 x 224 x 3 x N!
112 x 112 x 16!
56 x 56 x 32!
28 x 28 x 64!14 x 14 x 128!
Convolution + ReLU!
Max Pooling!
Fully Connected + ReLU !
Softmax!
1 x 1 x 1024!
1 x 1 x 576!
Figure 2. Example data from training set. Raw focal stack (N images with varying focal depth) and corresponding depth map (top row). Filtered and aligned data (middle row). Training images and labeled data (bottom row).
Figure 1. Mobile experimental data collection setup.
[Exp2] Testing prediction: depth from single RGB images in our dataset
[Exp3] Testing prediction: depth from focal-stack images in our dataset
Learning Curves on training data for all three data sets
IMPLEMENTATION & EXPERIMENTS5 10 15 20 25 30
0
10
20
30
40
50
60
70
Epochs
Mea
n Sq
uare
d Er
ror (
pixe
ls2 )
Single RGB Image − NYUSingle RGB ImageRGB Focal Stack
5 10 15 20 25 300
1000
2000
3000
4000
5000
6000
7000
Epochs
Loss
(pix
els)
Single RGB Image − NYUSingle RGB ImageRGB Focal Stack
Gro
und
Trut
hG
roun
d Tr
uth
Pred
ictio
nPr
edic
tion
Erro
r: 59
11Er
ror:
7180
Figure 4. Learning curves with training data showing the mse loss (left) and mean absolute error (middle) per pixel. Note that the final testing error for the focal stacks is slightly less than the single image data. Link to all testing results for our data set (right).
Table 1. Training and testing error results for all three data sets
Gro
und
Trut
h
Pred
ictio
n
Link for all testing results
![1] HOW I GOT IN AND WHY I CAME OUTOF THE COLD · title: 1] how i got in and why i came outof the cold: subject: 1] how i got in and why i came outof the cold: keywords](https://static.fdocuments.net/doc/165x107/5e2cf6ecea126f65e82364c9/1-how-i-got-in-and-why-i-came-outof-the-cold-title-1-how-i-got-in-and-why-i-came.jpg)
![functions are exactly solvable? Boros Hammer [1965], Kolmogorov Zabih [ECCV 2002, PAMI 2004] , Ishikawa [PAMI 2003], Schlesinger …](https://static.fdocuments.net/doc/165x107/5b31e06c7f8b9a744a8c2b1a/functions-are-exactly-solvable-boros-hammer-1965-kolmogorov-zabih-eccv-2002.jpg)
