PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf ·...
Transcript of PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf ·...
![Page 1: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/1.jpg)
PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
Charles R. Qi*Hao Su*Kaichun MoLeonidas J. Guibas
![Page 2: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/2.jpg)
Big Data + Deep Representation Learning
Robot Perception Augmented Reality Shape Design
source: Scott J Grunewald source: Google Tango source: solidsolutions
Emerging 3D Applications
![Page 3: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/3.jpg)
Big Data + Deep Representation Learning
Robot Perception Augmented Reality Shape Design
source: Scott J Grunewald source: Google Tango source: solidsolutions
Need for 3D Deep Learning!
![Page 4: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/4.jpg)
3D Representations
Projected ViewPoint Cloud Mesh Volumetric
…
RGB(D)
![Page 5: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/5.jpg)
3D Representation: Point Cloud
LiDAR
Depth Sensor
Point cloud is close to raw sensor data
Point Cloud
![Page 6: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/6.jpg)
3D Representation: Point Cloud
LiDAR
Depth Sensor
Point cloud is close to raw sensor data
Point Cloud
Point cloud is canonicalMesh
Volumetric
Depth Map
![Page 7: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/7.jpg)
Previous Works
Most existing point cloud features are handcrafted towards specific tasks
Source: https://github.com/PointCloudLibrary/pcl/wiki/Overview-and-Comparison-of-Features
![Page 8: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/8.jpg)
Previous Works
Point cloud is converted to other representations before it’s fed to a deep neural network
Conversion Deep Net
Voxelization 3D CNN
Projection/Rendering 2D CNN
Feature extraction Fully Connected
![Page 9: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/9.jpg)
Research Question:
Can we achieve effective feature learning directly on point clouds?
![Page 10: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/10.jpg)
Our Work: PointNet
End-to-end learning for scattered, unordered point data
PointNet
![Page 11: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/11.jpg)
Our Work: PointNet
End-to-end learning for scattered, unordered point data
Unified framework for various tasks
Object Classification
Object Part Segmentation
Semantic Scene Parsing
...
PointNet
![Page 12: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/12.jpg)
Our Work: PointNet
End-to-end learning for scattered, unordered point data
Unified framework for various tasks
![Page 13: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/13.jpg)
Challenges
Unordered point set as input
Model needs to be invariant to N! permutations.
Invariance under geometric transformations
Point cloud rotations should not alter classification results.
![Page 14: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/14.jpg)
Challenges
Unordered point set as input
Model needs to be invariant to N! permutations.
Invariance under geometric transformations
Point cloud rotations should not alter classification results.
![Page 15: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/15.jpg)
Unordered Input
Point cloud: N orderless points, each represented by a D dim vector
N
D
![Page 16: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/16.jpg)
Unordered Input
Point cloud: N orderless points, each represented by a D dim vector
N
D
N
D
represents the same set as
![Page 17: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/17.jpg)
Unordered Input
Point cloud: N orderless points, each represented by a D dim vector
N
D
N
D
represents the same set as
Model needs to be invariant to N! permutations
![Page 18: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/18.jpg)
Permutation Invariance: Symmetric Function
f (x1, x2,…, xn ) ≡ f (xπ1 , xπ2 ,…, xπn ) xi ∈!D,
![Page 19: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/19.jpg)
Permutation Invariance: Symmetric Function
Examples:
…
f (x1, x2,…, xn ) = max{x1, x2,…, xn}f (x1, x2,…, xn ) = x1 + x2 +…+ xn
f (x1, x2,…, xn ) ≡ f (xπ1 , xπ2 ,…, xπn ) xi ∈!D,
![Page 20: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/20.jpg)
Permutation Invariance: Symmetric Function
Examples:
…
f (x1, x2,…, xn ) = max{x1, x2,…, xn}f (x1, x2,…, xn ) = x1 + x2 +…+ xn
f (x1, x2,…, xn ) ≡ f (xπ1 , xπ2 ,…, xπn ) xi ∈!D,
How can we construct a family of symmetric functions by neural networks?
![Page 21: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/21.jpg)
Permutation Invariance: Symmetric FunctionObserve:f (x1, x2,…, xn ) = γ ! g(h(x1),…,h(xn )) is symmetric if is symmetricg
![Page 22: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/22.jpg)
Permutation Invariance: Symmetric FunctionObserve:f (x1, x2,…, xn ) = γ ! g(h(x1),…,h(xn )) is symmetric if is symmetricg
(1,2,3)
(1,1,1)
(2,3,2)
(2,3,4)
h
…
![Page 23: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/23.jpg)
Permutation Invariance: Symmetric Function
f (x1, x2,…, xn ) = γ ! g(h(x1),…,h(xn )) is symmetric if is symmetricg
(1,2,3)
(1,1,1)
(2,3,2)
(2,3,4)
hsimple symmetric function
g
…
Observe:
![Page 24: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/24.jpg)
Permutation Invariance: Symmetric Function
(1,2,3)
(1,1,1)
(2,3,2)
(2,3,4)
hsimple symmetric function
g γ
f (x1, x2,…, xn ) = γ ! g(h(x1),…,h(xn )) is symmetric if is symmetricg
…
Observe:
PointNet (vanilla)
![Page 25: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/25.jpg)
Permutation Invariance: Symmetric Function
PointNet (vanilla)
Symmetric functions
What symmetric functions can be constructed by PointNet?
![Page 26: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/26.jpg)
Universal Set Function Approximator
Theorem:A Hausdorff continuous symmetric function can be arbitrarily approximated by PointNet.
f :2X → !
S ⊆ !dPointNet (vanilla)
![Page 27: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/27.jpg)
Basic PointNet Architecture
Empirically, we use multi-layer perceptron (MLP) and max pooling:
(1,2,3)
(1,1,1)
(2,3,2)
(2,3,4)
h
g
MLP
γ
MLP
MLP
MLP
MLP
max …
PointNet (vanilla)
![Page 28: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/28.jpg)
Challenges
Unordered point set as input
Model needs to be invariant to N! permutations.
Invariance under geometric transformations
Point cloud rotations should not alter classification results.
![Page 29: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/29.jpg)
Input Alignment by Transformer Network
Idea: Data dependent transformation for automatic alignment
T-Net
Transform
transform params
Data Transformed Data
N
3 3
N
![Page 30: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/30.jpg)
Input Alignment by Transformer Network
Idea: Data dependent transformation for automatic alignment
T-Net
Transform
transform params
Data Transformed Data
N
3 3
N
![Page 31: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/31.jpg)
Input Alignment by Transformer Network
Idea: Data dependent transformation for automatic alignment
T-Net
Transform
transform params
Data Transformed Data
N
3 3
N
![Page 32: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/32.jpg)
Input Alignment by Transformer Network
T-Net
Matrix Mult.
transform params: 3x3
Data Transformed Data
N
3 3
The transformation is just matrix multiplication!
![Page 33: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/33.jpg)
T-Net
Embedding Space Alignment
Matrix Mult.
transform params: 64x64
Input embeddings:
Nx64
Transformed embeddings:
Nx64
![Page 34: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/34.jpg)
T-Net
Embedding Space Alignment
Matrix Mult.
transform params: 64x64
Input embeddings:
Nx64
Transformed embeddings:
Nx64
Regularization:
Transform matrix A 64x64 close to orthogonal:
![Page 35: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/35.jpg)
PointNet Classification Network
![Page 36: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/36.jpg)
PointNet Classification Network
![Page 37: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/37.jpg)
PointNet Classification Network
![Page 38: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/38.jpg)
PointNet Classification Network
![Page 39: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/39.jpg)
PointNet Classification Network
![Page 40: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/40.jpg)
PointNet Classification Network
![Page 41: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/41.jpg)
PointNet Classification Network
![Page 42: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/42.jpg)
Extension to PointNet Segmentation Network
local embedding global feature
![Page 43: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/43.jpg)
Extension to PointNet Segmentation Network
local embedding global feature
![Page 44: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/44.jpg)
Results
![Page 45: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/45.jpg)
Results on Object Classification
dataset: ModelNet40; metric: 40-class classification accuracy (%)
3D CNNs
![Page 46: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/46.jpg)
Results on Object Part Segmentation
![Page 47: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/47.jpg)
Results on Object Part Segmentation
dataset: ShapeNetPart; metric: mean IoU (%)
![Page 48: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/48.jpg)
Results on Semantic Scene Parsing
Input
Output
dataset: Stanford 2D-3D-S (Matterport scans)
![Page 49: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/49.jpg)
Robustness to Data Corruption
dataset: ModelNet40; metric: 40-class classification accuracy (%)
![Page 50: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/50.jpg)
Robustness to Data Corruption
Less than 2% accuracy drop with 50% missing data
dataset: ModelNet40; metric: 40-class classification accuracy (%)
![Page 51: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/51.jpg)
Robustness to Data Corruption
dataset: ModelNet40; metric: 40-class classification accuracy (%)
![Page 52: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/52.jpg)
Robustness to Data Corruption
Why is PointNet so robust to missing data?
3D CNN
![Page 53: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/53.jpg)
Visualizing Global Point Cloud Features
maxpool
shared n
MLP
global feature
3 1024
Which input points are contributing to the global feature? (critical points)
![Page 54: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/54.jpg)
Visualizing Global Point Cloud Features
Original Shape:
Critical Point Sets:
![Page 55: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/55.jpg)
Visualizing Global Point Cloud Features
maxpool
shared n
MLP
global feature
3 1024
Which points won’t affect the global feature?
![Page 56: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/56.jpg)
Visualizing Global Point Cloud Features
Original Shape:
Critical Point Set:
Upper bound set:
![Page 57: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/57.jpg)
Visualizing Global Point Cloud Features (OOS)
Original Shape:
Critical Point Set:
Upper bound Set:
![Page 58: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/58.jpg)
Conclusion
• PointNet is a novel deep neural network that directly consumes point cloud.
• A unified approach to various 3D recognition tasks. • Rich theoretical analysis and experimental results.
Code & Data Available!http://stanford.edu/~rqi/pointnet
See you at Poster 9!
![Page 59: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/59.jpg)
Thank you!
![Page 60: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/60.jpg)
THE END
![Page 61: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/61.jpg)
Speed and Model Size
Inference time 11.6ms, 25.3ms GTX1080, batch size 8
![Page 62: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/62.jpg)
Permutation Invariance: How about Sorting?
(1,2,3) (1,1,1) (2,3,2) (2,3,4)
(1,1,1) (1,2,3) (2,3,2) (2,3,4)
MLP
lexsorted
“Sort” the points before feeding them into a network.
Unfortunately, there is no canonical order in high dim space.
![Page 63: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/63.jpg)
Permutation Invariance: How about Sorting?
Accuracy
Unordered Input 12%
Lexsorted Input 40%
PointNet (vanilla) 87%
Multi-Layer Perceptron (ModelNet shape classification)
(1,2,3) (1,1,1) (2,3,2) (2,3,4)
(1,1,1) (1,2,3) (2,3,2) (2,3,4)
MLP
lexsorted
“Sort” the points before feeding them into a network.
Unfortunately, there is no canonical order in high dim space.
![Page 64: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/64.jpg)
Permutation Invariance: How about RNNs?
(1,2,3) (1,1,1) (2,3,2) (2,3,4)
LSTM
MLP
LSTM
MLP
LSTM
MLP
LSTM
MLP…
…
Train RNN with permutation augmentation.
However, RNN forgets and order matters.
![Page 65: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/65.jpg)
Permutation Invariance: How about RNNs?
Train RNN with permutation augmentation.
However, RNN forgets and order matters.
Accuracy
LSTM 75%
PointNet (vanilla) 87%
LSTM Network (ModelNet shape classification)
(1,2,3) (1,1,1) (2,3,2) (2,3,4)
LSTM
MLP
LSTM
MLP
LSTM
MLP
LSTM
MLP…
…
![Page 66: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/66.jpg)
PointNet Classification Network
ModelNet40 Accuracy
PointNet (vanilla) 87.1%
+ input 3x3 87.9%
+ feature 64x64 86.9%
+ feature 64x64 + reg 87.4%
+ both 89.2%
![Page 67: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/67.jpg)
Find the top-K points in a dense volumetric grid that activates neuron X.
Visualizing Point Functions
FCs1x3 1x1024
FC1x3 1x1024
FC FC FC64 64 64
Expanded View:
Compact View:
Which input point will activate neuron X?
FC128
![Page 68: PointNet: Deep Learning on Point Sets for 3D ...rqi/pointnet/docs/cvpr17_pointnet_slides.pdf · PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles](https://reader030.fdocuments.net/reader030/viewer/2022040201/5e607906f544af2405511eb4/html5/thumbnails/68.jpg)
Visualizing Point Functions