視覚情報処理論 (Visual Information Processing )G. Turk, M. Levoy, Proc. SIGGRAPH ‘94, In...
Transcript of 視覚情報処理論 (Visual Information Processing )G. Turk, M. Levoy, Proc. SIGGRAPH ‘94, In...
視覚情報処理論(Visual Information Processing )
開講所属: 学際情報学府
水(Wed)5 [16:50-18:35]
3D data processing 2
Modeling procedures
1. Data acquisition3. Merging
2. Alignment
3. Merging
Merging methods�Point-based
• Directly generate mesh from point cloud�Mesh-based
• Connect the boundaries of range data�Volume-based
• Convert range data to volume data (implicit function) and re-convert to mesh data
Point-based method[Chang et al. ‘07]
Delaunay Triangulation
• every triangle has an empty circumscribing circle
• find and remove all triangles with non-empty circumcircles
• triangulate by connecting new point
Streaming Computation of 2D Delaunay Triangulations�streaming, out-of-core, I/O and memory efficient Delaunay
triangulation in 2D and 3D
[Isenburg et al. TOG 2006]
Streaming Delaunay in 3D
Ball-Pivoting Algorithm [Bernardini et al. TVCG 99]
Surface Reconstruction from Point Clouds by Transforming the Medial Scaffold
• Relate the sampled shape with the underlying (unknown) surface by a sequence of shape deformations (growing from samples).
• Represent (2D) shapes by their medial “shock graphs”. [Kimia et al.]
• Handle shock transitions across different shock topologies to recover gaps.
[CVIU 2009]
Costa minimum surface (courtesy of H. Hoppe)Mobius strip
Result: Videos on Meshing Algebraic Surfaces
Result:
The rocker arm data courtesy of Cyberware.Flat smooth regions are meshed prior to the ridges/corners.
Dealing with sampling qualityInput of non-uniform and low-density sampling:
Response to additive noise:
50% 100% 150%
Mesh-based method
Range data 1Range data 2
Zippered Polygon Meshes from Range ImagesG. Turk, M. Levoy, Proc. SIGGRAPH ‘94, In Computer Graphics Proc., Annual Conf. Series, pp.311-318, 1994.
Zippering polygon meshes
Range data 1Range data 2
Removal ofOverlapping area
Zippering
Procedures of Zipper method
Process at boundary
Refinement of data
�Estimate surface normal vectors�Compute corresponding points along the
surface normal�Move points to weighted average positions
Merging result 1
Merging result 2
Volumetric method
Signed Distance
FieldAlignment result
Connect highestprobability points
Merging result
Compute probabilityof each voxel
A Volumetric method for building complex modelsfrom range imagesB. Curless, M. Levoy, Proc. SIGGRAPH'96. ACM, pp.303-312, 1996.
Volumetric merging�Represent a space by voxels�Computer distance from voxel and represent as implicit
function (Signed Distance Field)
Example of SDF
Computation of SDF
[Curless et al. ‘96]
Space Curving
29
Inside
Outside
Mesh generation• Marching Cubes algorithm [Lorensen ’87]
32
Experimental result (Hole filling)
Zipper vs. Volume
Zipper Volume
Consensus surfaces for modeling 3d objects from multiple range imagesM. D. Wheeler, Y. Sato and K. Ikeuchi, International Conference on Computer Vision (ICCV), pp.917-924,1998.
Signed Distance Field (SDF)
35
Signed distance Field
Marching Cubes [Lorensen et al., 1987]
Computation of SDF• Naïve algorithm
• Value: distance to nearest point on the surface• Sign: inside or outside of the surface
f(x’)
f(x)
Issues of conventional method�Bumpy surface is generated�Instability of signs: in-out-in transit
Consensus surface�Grouping by surface normal and distance between
corresponding points (consensus group)
f(x)f(x’)
( )( )( )
≥⋅∧≤−
=
otherwiseFalse
cosTrue
,,,eSameSurfac
1010
1100
nd nnpp θδ ))
))
npnp
Experimental result
39
Closest Signed DistanceConsensus Surfaces
Volume-based Merging�Merging methods
• Curless et al., 1996• Wheeler et al., 1998• Masuda et al., 2003• Turk et al., 1999• Rocchini et al., 2001�Practical ideas & methods
• Sagawa et al., 2002• Nakao et al., 2008
41
Registration and Integration of Multiple Range Images byMatching Signed Distance Fieldsfor Object Shape ModelingT. Masuda, Computer Vision and Image Understanding, 2002
43
Matching Signed Distance [Masuda et al., 2003]
[ ] [ ]( )( ) [ ] [ ]( )
( )∫∫ +−+
=pp
pppp
xx
xxpxpx
pp
dw
dBAw
BAd2
2
,SD,SD
,SDF,,SDF
[ ][ ] ( )[ ]( )∑ ∑
∑ ∑≤≤
≤≤=
p
pp
p
Tpp
T
N
N
SIdw
wIS
α
α
ααα
ααα
1
1
2
1
,SDF,,SDF
,,;E
Error function
T: affine matrixw: weightI: merged model
Alignment result
Merging result
The Marching Intersections algorithm for merging range imagesClaudioRocchini, PaoloCignoni, FabioGanovelli, ClaudioMontani, Paolo Pingi, Roberto Scopigno, Visual Computer 2004.
Marching Intersections [Rocchini et al. 2001]
47
Surface Reconstruction
48
Experimental result
Robust and Adaptive Integration of Multiple Range Images with Photometric AttributesR. Sagawa K. Nishino K. Ikeuchi, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2001
Adaptive Integration [Sagawa et al., 2002]�Splitting voxel by Octree
51
Geometric featureOptical feature
More adaptive
Adaptive merging�Geometric feature
�Photometric feature
52
( )( )nn
nii
TN >
<⋅ δnnarccosmax
( )( )cc
cjiji
TN
cc
>
< δ,Distancemax,
Experimental results
53
Only geometric feature With optical feature
Result of adaptive merging
Variational implicit surfacesG. Turk. J. F. O'Brien, 1999
Variational implicit functions
four zero-valued and one positive constraint
(left) + three new zero-valuedconstraints
Variational Implicit Surfaces [Turk et al., 1999]
57
0)()()(1
=−+= ∑=
N
iiivf xxxx φλ
zcycxccv 3210)( +++=x
( ) ( ) ( )( )∑
∑=
xw
xdxwxD
i
ii fi(x): VIF of the ith imageVariational Implicit Function
Low-degree polynomial Radial basis
Variational Implicit Functions
58
kihf ii ,,2,1 ,)( K==p
=
0
0
0
0
0000
0000
0000
0000111
1
1
1
2
1
3
2
1
0
2
1
21
21
21
21
22222221
11111211
kk
k
k
k
kkkkkkk
k
k
h
h
h
c
c
c
c
zzz
yyy
xxx
zyx
zyx
zyx
MM
L
L
L
L
L
MMMMMMM
L
L
λ
λλ
φφφ
φφφφφφ
zcycxccfN
iii 3210
1
)()( ++++−= ∑=
xxx φλConstraint pointsRange image
Surfaces by VIF
Surface Reconstruction from Point Clouds by Transforming the Medial ScaffoldM. –C. Chang*, F. F. Leymarie † and B. B. Kimia, Computer Vision and Image Understanding (CVIU), 2009
4. Online and Dynamic Modeling
Surround Structured Lighting for Full Object ScanningD. Lanman, D. Crispell, and G. Taubin, 3D Digital Imaging and Modeling (3DIM), 2007
Data acquisition
Multiple scans and alignment operations are required
System architecture�Multi-view: digital camera + planar mirrors�Orthographic: DLP projector + Fresnel lens
Mechanical Alignment ProcedureManual Mirror Alignment�Parallel to the orthographic illumination rays�Gray code patterns assist adjusting
Step 1: Alignment using a Flat Surface�Cover each mirror with a blank surface�Adjust the uncovered mirror so that the reflected and projected stripes coincide
Step 2: Alignment using a Cylinder�Place a blank cylindrical object in the center of the scanning volume�Adjust both mirrors until the reflected stripes coincide on the cylinder surface
Gray Code Sequence
Reconstruction Algorithm
Step 1: Recover Projector Rows� Project Gray code image sequence�Recover projector scanline illuminating each pixel� Post-process using image morphology
Step 2: Recover 3D point cloud�Reconstruct using ray-plane intersection�Consider each real/virtual camera separately�Assign per-point color using ambient image
Recovered Projector Rows
Real and Virtual Cameras
Camera Centers Optical Rays
Experimental reconstruction results
Ambient Illumination Gray Code Sequence Recovered Projector Rows
In-Hand Scanning with Online Loop ClosureThibaut Weise (EZH), Thomas Wismer (EZH), Bastian Leibe (RWTH Aachen), Luc Van Gool (EZH), ICCV Workshop on 3D Digital Imaging and Modeling (3DIM), 2009
Real-Time 3D model acquisition
[Rusinkiewicz et al. ’02]
Time-Coded Light Patterns�Assign each stripe a unique illumination code over time
[Posdamer 82]
Space
Time
Illumination history = (WB),(BW),(WB)
Code
Codes for Moving Scenes�Assign time codes to stripe boundaries�Perform frame-to-frame tracking of corresponding boundaries�Propagate illumination history
[Hall-Holt & Rusinkiewicz, ICCV 2001]
ICP in the Real-Time Pipeline�Potential problem with ICP: local minima
• In this pipeline, scans close together• Very likely to converge to correct (global) minimum�Basic ICP algorithm too slow (~ seconds)
• Point-to-plane minimization• Projection-based matching• With these tweaks, running time ~ milliseconds
[Rusinkiewicz & Levoy, 3DIM 2001]
Issues on Online Modeling�Loop closure problem
• Accumulation of alignment errors�Sensor noise
Approach�Fast ICP�Surfel representation with topology graph�As-rigid-as-possible deformation
Scanning system�Structured Light(Phase shifting)
Registration�Fast ICP
• Correspondence search by projection• Point-surface distance• Rigid transformation
Fast ICP
�Point-to-Plane error metric
( )∑≠
+−+⋅=kji
iikijijkjikitR
txRtyRnR,
2
,)}(){(minˆ
rr
rrr
ε
)( yxn −⋅y y
nyx
Surfel representation
Surfel
Hexagonal surfel forefficient rendering
Surfel visibilityconfidence
Integration�Surfel Update
�Surfel Addition�Surfel Removal�Surface Growing
• All surfels having low confidences deleted• Lost parts are rescanned
Topology graph�A surfel has at least one node attached in range: attached
nodes are closer than a threshold
Topology graph update�Surfel Addition�Node Addtion�Node Deletion�Surfel / Node Update
Online loop closure
Only registering and integrating new input data with one of the ’meeting’ bordersThe model is closed by an as-rigid-as-possible deformation
Loop closure procedure1. Determine the currently visible nodes2. Intersect the visible nodes with the topology graph and
decompose into connected components3. Register each component individually to the input scan4. Deform the model surface
Loop closure procedure
As-rigid-as-possible Deformation�Deformation is represented by a collection
of rigid transformations�Each graph node gj is assigned a
transformation
�Deformed position pi of surfel i is computed as weighted sum (surfel belongs to multiple nodes)
Estimation of rigid transformations
Position term (distance between corresponding points)
Regularization term (smoothness)
Energy function
Result
Other results
Elastic Convolved ICP for the Registration of Deformable ObjectsRyusuke Sagawa, Kyotaka Akasaka, Yasushi Yagi, Henning Hamer, Luc Van Gool
Registration of deformable object
Rigid object
Deformable object
Approach�Iterative manner (ICP-based)�Point-to-Plane error metric�Search along Line-of-sight�Convolution of the differences
Basic error metric�Point-to-Plane
)( yxn −⋅yy
x yn
T: Rigid transformation
Correspondence search
Similar to window matching
Convolved error
: Neighboring vertices of i
Data term Smoothness term
Weighting SchemesSmoothness (distance between neighboring points)
Error (distance between corresponding points)
Gaussian
Tukey
Linealization
is small
Solving by Least Square method
Smoothness weight
To avoid the shrinkage of the surface
KinectFusion
https://msdn.microsoft.com/ja-jp/library/dn188670.aspx
[Newcombe et al. ISMAR 2011]
Frame to model tracking
DynamicFusion
DynamicFusion [Newcombe et al. CVPR 2015]
Summary (3D Data Processing)�3D geometric modeling pipeline
• Data acquisition• Alignment• Merging
• Mesh-based• Volume-based• Point-based
�Online and Dynamic Modeling