Shape from Contours and Multiple Stereo A Hierarchical, Mesh-Based Approach
description
Transcript of Shape from Contours and Multiple Stereo A Hierarchical, Mesh-Based Approach
![Page 1: Shape from Contours and Multiple Stereo A Hierarchical, Mesh-Based Approach](https://reader031.fdocuments.net/reader031/viewer/2022013012/56815a88550346895dc7f999/html5/thumbnails/1.jpg)
Shape from Contours and Multiple Stereo
A Hierarchical, Mesh-Based Approach
Hendrik Kück, Wolfgang Heidrich, Christian Vogelgsang
![Page 2: Shape from Contours and Multiple Stereo A Hierarchical, Mesh-Based Approach](https://reader031.fdocuments.net/reader031/viewer/2022013012/56815a88550346895dc7f999/html5/thumbnails/2.jpg)
The goalThe goal
![Page 3: Shape from Contours and Multiple Stereo A Hierarchical, Mesh-Based Approach](https://reader031.fdocuments.net/reader031/viewer/2022013012/56815a88550346895dc7f999/html5/thumbnails/3.jpg)
The goalThe goal
![Page 4: Shape from Contours and Multiple Stereo A Hierarchical, Mesh-Based Approach](https://reader031.fdocuments.net/reader031/viewer/2022013012/56815a88550346895dc7f999/html5/thumbnails/4.jpg)
Our approachOur approach
• Perform reconstruction using• Color• Object’s silhouettes
1. Create initial approximation based on silhouettes (Visual Hull)
2. Improve mesh using color information within an optimization approach
• Perform reconstruction using• Color• Object’s silhouettes
1. Create initial approximation based on silhouettes (Visual Hull)
2. Improve mesh using color information within an optimization approach
![Page 5: Shape from Contours and Multiple Stereo A Hierarchical, Mesh-Based Approach](https://reader031.fdocuments.net/reader031/viewer/2022013012/56815a88550346895dc7f999/html5/thumbnails/5.jpg)
Talk outlineTalk outline
• Image data & preprocessing• Visual Hull
• Definition• Computation as a triangle mesh
• Image based mesh optimization• Results
• Image data & preprocessing• Visual Hull
• Definition• Computation as a triangle mesh
• Image based mesh optimization• Results
![Page 6: Shape from Contours and Multiple Stereo A Hierarchical, Mesh-Based Approach](https://reader031.fdocuments.net/reader031/viewer/2022013012/56815a88550346895dc7f999/html5/thumbnails/6.jpg)
Image data & preprocessingImage data & preprocessing
• Calibrated images
• Foreground/ background segmentation
• Calibrated images
• Foreground/ background segmentation
![Page 7: Shape from Contours and Multiple Stereo A Hierarchical, Mesh-Based Approach](https://reader031.fdocuments.net/reader031/viewer/2022013012/56815a88550346895dc7f999/html5/thumbnails/7.jpg)
Shape from SilhouettesShape from Silhouettes
• Silhouette + camera information: Silhouette Cone
• Completely contains real object
• Silhouette + camera information: Silhouette Cone
• Completely contains real object
![Page 8: Shape from Contours and Multiple Stereo A Hierarchical, Mesh-Based Approach](https://reader031.fdocuments.net/reader031/viewer/2022013012/56815a88550346895dc7f999/html5/thumbnails/8.jpg)
Shape from SilhouettesShape from Silhouettes
• Silhouette + camera information: Silhouette Cone
• Completely contains real object
• Silhouettes can have holes
• Silhouette + camera information: Silhouette Cone
• Completely contains real object
• Silhouettes can have holes
![Page 9: Shape from Contours and Multiple Stereo A Hierarchical, Mesh-Based Approach](https://reader031.fdocuments.net/reader031/viewer/2022013012/56815a88550346895dc7f999/html5/thumbnails/9.jpg)
Visual HullVisual Hull
• Definition: Largest volume that produces the same silhouettes as the object
• Construction:Intersection of the silhouette cones
• Definition: Largest volume that produces the same silhouettes as the object
• Construction:Intersection of the silhouette cones
![Page 10: Shape from Contours and Multiple Stereo A Hierarchical, Mesh-Based Approach](https://reader031.fdocuments.net/reader031/viewer/2022013012/56815a88550346895dc7f999/html5/thumbnails/10.jpg)
Visual HullVisual Hull
![Page 11: Shape from Contours and Multiple Stereo A Hierarchical, Mesh-Based Approach](https://reader031.fdocuments.net/reader031/viewer/2022013012/56815a88550346895dc7f999/html5/thumbnails/11.jpg)
Computing the Visual HullComputing the Visual Hull
• Extract Visual Hull using Extended Marching Cubes algorithm(Kobbelt, Botsch, Schwanecke, Seidel, 2001)
• Vertices lie exactly on isosurface• Can preserve sharp discontinuities
• Requires signed directed distance functionD(VH,x,d)• Distance from x to VH surface along direction d• Positive, if x outside VH, negative if inside
• Extract Visual Hull using Extended Marching Cubes algorithm(Kobbelt, Botsch, Schwanecke, Seidel, 2001)
• Vertices lie exactly on isosurface• Can preserve sharp discontinuities
• Requires signed directed distance functionD(VH,x,d)• Distance from x to VH surface along direction d• Positive, if x outside VH, negative if inside
![Page 12: Shape from Contours and Multiple Stereo A Hierarchical, Mesh-Based Approach](https://reader031.fdocuments.net/reader031/viewer/2022013012/56815a88550346895dc7f999/html5/thumbnails/12.jpg)
Computing the Visual HullComputing the Visual Hull
• D(VH,x,d) = D( SCi ,x,d) = maxi D(SCi ,x,d)
• D(SCi ,x,d) can be efficiently computed in image space
• D(VH,x,d) = D( SCi ,x,d) = maxi D(SCi ,x,d)
• D(SCi ,x,d) can be efficiently computed in image space
![Page 13: Shape from Contours and Multiple Stereo A Hierarchical, Mesh-Based Approach](https://reader031.fdocuments.net/reader031/viewer/2022013012/56815a88550346895dc7f999/html5/thumbnails/13.jpg)
Computing the Visual HullComputing the Visual Hull
14000 triangles 1400 triangles
![Page 14: Shape from Contours and Multiple Stereo A Hierarchical, Mesh-Based Approach](https://reader031.fdocuments.net/reader031/viewer/2022013012/56815a88550346895dc7f999/html5/thumbnails/14.jpg)
The Optimization StageThe Optimization Stage
• Evolve triangle mesh into a shape that is• Consistent with color in the images• Consistent with silhouettes in images• Free from self-intersections• Smooth (low curvature)• Composed of well shaped triangles
• Evolve triangle mesh into a shape that is• Consistent with color in the images• Consistent with silhouettes in images• Free from self-intersections• Smooth (low curvature)• Composed of well shaped triangles
![Page 15: Shape from Contours and Multiple Stereo A Hierarchical, Mesh-Based Approach](https://reader031.fdocuments.net/reader031/viewer/2022013012/56815a88550346895dc7f999/html5/thumbnails/15.jpg)
OptimizationOptimization
• Only geometry is optimized, not topology
• 3 Nv degrees of freedom
• Global optimization hopeless Use local per-vertex optimizations
• Locally minimize energy function E(vi) using 3D Simplex Method
• Iterate over vertices
• Only geometry is optimized, not topology
• 3 Nv degrees of freedom
• Global optimization hopeless Use local per-vertex optimizations
• Locally minimize energy function E(vi) using 3D Simplex Method
• Iterate over vertices
![Page 16: Shape from Contours and Multiple Stereo A Hierarchical, Mesh-Based Approach](https://reader031.fdocuments.net/reader031/viewer/2022013012/56815a88550346895dc7f999/html5/thumbnails/16.jpg)
Per Vertex Energy FunctionPer Vertex Energy Function
color
silhouette
triangle shape
local curvature
self penetration
![Page 17: Shape from Contours and Multiple Stereo A Hierarchical, Mesh-Based Approach](https://reader031.fdocuments.net/reader031/viewer/2022013012/56815a88550346895dc7f999/html5/thumbnails/17.jpg)
Color consistencyColor consistency
• Assumption: Lambertian reflectance• Surface points appear the same from all
viewing directions• Points on the real surface will project
onto pixels of the same color in all images that see them
• Projecting images onto the mesh• If surface is consistent, the color from
different images will match• Color cost term color mismatch (L2 norm)
• Assumption: Lambertian reflectance• Surface points appear the same from all
viewing directions• Points on the real surface will project
onto pixels of the same color in all images that see them
• Projecting images onto the mesh• If surface is consistent, the color from
different images will match• Color cost term color mismatch (L2 norm)
![Page 18: Shape from Contours and Multiple Stereo A Hierarchical, Mesh-Based Approach](https://reader031.fdocuments.net/reader031/viewer/2022013012/56815a88550346895dc7f999/html5/thumbnails/18.jpg)
Color consistencyColor consistency
• Use OpenGL for color projection• Projective texture
mapping
• And for determining visibility• Shadow mapping
• Use OpenGL for color projection• Projective texture
mapping
• And for determining visibility• Shadow mapping
v v
v v
![Page 19: Shape from Contours and Multiple Stereo A Hierarchical, Mesh-Based Approach](https://reader031.fdocuments.net/reader031/viewer/2022013012/56815a88550346895dc7f999/html5/thumbnails/19.jpg)
Color consistencyColor consistency
• Set up orthographic view of triangle fan around vertex v• Choose scale
according to sampling rate in the images
• Render fan to get samples of color and occlusion, once for each (relevant) image
• Set up orthographic view of triangle fan around vertex v• Choose scale
according to sampling rate in the images
• Render fan to get samples of color and occlusion, once for each (relevant) image
![Page 20: Shape from Contours and Multiple Stereo A Hierarchical, Mesh-Based Approach](https://reader031.fdocuments.net/reader031/viewer/2022013012/56815a88550346895dc7f999/html5/thumbnails/20.jpg)
Silhouette consistencySilhouette consistency
• No part of the geometry may project outside any silhouette (must stay inside the visual hull)
Strongly penalize distance outside Visual Hull¼ maxi ( distance outside silhouettes in image i )
• Geometry may be smaller than Visual Hull• Where color does not provide enough
information, use Visual Hull as fallback solution Slightly penalize distance inside Visual Hull
¼ mini ( distance inside silhouettes in image i )
• No part of the geometry may project outside any silhouette (must stay inside the visual hull)
Strongly penalize distance outside Visual Hull¼ maxi ( distance outside silhouettes in image i )
• Geometry may be smaller than Visual Hull• Where color does not provide enough
information, use Visual Hull as fallback solution Slightly penalize distance inside Visual Hull
¼ mini ( distance inside silhouettes in image i )
![Page 21: Shape from Contours and Multiple Stereo A Hierarchical, Mesh-Based Approach](https://reader031.fdocuments.net/reader031/viewer/2022013012/56815a88550346895dc7f999/html5/thumbnails/21.jpg)
Silhouette consistencySilhouette consistency
• Encode distance from silhouette in alpha channel of OpenGL textures• Project onto triangle fan along with
color & visibility
• Encode distance from silhouette in alpha channel of OpenGL textures• Project onto triangle fan along with
color & visibility
![Page 22: Shape from Contours and Multiple Stereo A Hierarchical, Mesh-Based Approach](https://reader031.fdocuments.net/reader031/viewer/2022013012/56815a88550346895dc7f999/html5/thumbnails/22.jpg)
Multi-Resolution OptimizationMulti-Resolution Optimization
• Local minima are a problem,especially when• Triangle size is small compared to
geometric error• Texture frequencies are high
compared to geometric error
• Solution: Perform optimization at multiple resolutions
• Local minima are a problem,especially when• Triangle size is small compared to
geometric error• Texture frequencies are high
compared to geometric error
• Solution: Perform optimization at multiple resolutions
![Page 23: Shape from Contours and Multiple Stereo A Hierarchical, Mesh-Based Approach](https://reader031.fdocuments.net/reader031/viewer/2022013012/56815a88550346895dc7f999/html5/thumbnails/23.jpg)
Multi-Resolution OptimizationMulti-Resolution Optimization
• Start with low resolution Visual Hull mesh
• Start with low resolution Visual Hull mesh
![Page 24: Shape from Contours and Multiple Stereo A Hierarchical, Mesh-Based Approach](https://reader031.fdocuments.net/reader031/viewer/2022013012/56815a88550346895dc7f999/html5/thumbnails/24.jpg)
Multi-Resolution OptimizationMulti-Resolution Optimization
• Start with low resolution Visual Hull mesh
• Optimize until convergence
• Start with low resolution Visual Hull mesh
• Optimize until convergence
![Page 25: Shape from Contours and Multiple Stereo A Hierarchical, Mesh-Based Approach](https://reader031.fdocuments.net/reader031/viewer/2022013012/56815a88550346895dc7f999/html5/thumbnails/25.jpg)
Multi-Resolution OptimizationMulti-Resolution Optimization
• Start with low resolution Visual Hull mesh
• Optimize until convergence
• Subdivide & optimize more
• Start with low resolution Visual Hull mesh
• Optimize until convergence
• Subdivide & optimize more
![Page 26: Shape from Contours and Multiple Stereo A Hierarchical, Mesh-Based Approach](https://reader031.fdocuments.net/reader031/viewer/2022013012/56815a88550346895dc7f999/html5/thumbnails/26.jpg)
Multi-Resolution OptimizationMulti-Resolution Optimization
• Start with low resolution Visual Hull mesh
• Optimize until convergence
• Subdivide & optimize more
• Do it again, …
• Start with low resolution Visual Hull mesh
• Optimize until convergence
• Subdivide & optimize more
• Do it again, …
![Page 27: Shape from Contours and Multiple Stereo A Hierarchical, Mesh-Based Approach](https://reader031.fdocuments.net/reader031/viewer/2022013012/56815a88550346895dc7f999/html5/thumbnails/27.jpg)
Final ResultsFinal Results
before
![Page 28: Shape from Contours and Multiple Stereo A Hierarchical, Mesh-Based Approach](https://reader031.fdocuments.net/reader031/viewer/2022013012/56815a88550346895dc7f999/html5/thumbnails/28.jpg)
Final ResultsFinal Results
before after
![Page 29: Shape from Contours and Multiple Stereo A Hierarchical, Mesh-Based Approach](https://reader031.fdocuments.net/reader031/viewer/2022013012/56815a88550346895dc7f999/html5/thumbnails/29.jpg)
Final ResultsFinal Results
before after
![Page 30: Shape from Contours and Multiple Stereo A Hierarchical, Mesh-Based Approach](https://reader031.fdocuments.net/reader031/viewer/2022013012/56815a88550346895dc7f999/html5/thumbnails/30.jpg)
Thank you