Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active...
Transcript of Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active...
![Page 1: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/1.jpg)
Lecture 7 -Silvio Savarese 12-Feb-18
• Activestereo• Structuredlighting• Depthsensing
• Volumetricstereo:• Spacecarving• Shadowcarving• Voxelcoloring
Lecture8Activestereo &Volumetricstereo
Reading:[Szelisky]Chapter11“Multi-viewstereo”
Seitz, S. M., & Dyer, C. R. (1999). Photorealistic scene reconstruction by voxel coloring.International Journal of Computer Vision, 35(2), 151-173.
S. Savarese, M. Andreetto, H. Rushmeier, F. Bernardini and P. Perona, 3D Reconstruction by Shadow Carving: Theory and Practical Evaluation, International Journal of Computer Vision (IJCV) , 71(3), 305-336, 2006
![Page 2: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/2.jpg)
Traditional stereo
What’s the main problem in traditional stereo?We need to find correspondences!
P
Camera O2
p’
O1
p
Camera
![Page 3: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/3.jpg)
Active stereo (point)
Replace one of the two cameras by a projector- Projector geometry calibrated- What’s the advantage of having the projector?
P
O2
p’
Correspondence problem solved!
Camera
p
Projector (e.g. DLP)
image
projector virtual image
Any limitation??
![Page 4: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/4.jpg)
Active stereo (stripe)
Projector (e.g. DLP)O
- Projector and camera are parallel- Correspondence problem is solved!
Camera
p’p
P
projector virtual image image
s
S
s’
l’
![Page 5: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/5.jpg)
Calibrating the system
Projector (e.g. DLP)O
- Use calibration rig to calibrate camera and localize rig in 3D- Project patterns on rig and calibrate projector
Camera
projector virtual image image
![Page 6: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/6.jpg)
Laser scanning
• Optical triangulation– Project a single stripe of laser light– Scan it across the surface of the object– This is a very precise version of structured light scanning
Digital Michelangelo Project (1990)http://graphics.stanford.edu/projects/mich/
Source: S. Seitz
![Page 7: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/7.jpg)
The Digital Michelangelo Project, Levoy et al.
Laser scanning
Source: S. Seitz
![Page 8: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/8.jpg)
Light source O
Active stereo (shadows)J. Bouguet & P. Perona, 99
- 1 camera,1 light source- very cheap setup- calibrated the light source
![Page 9: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/9.jpg)
Active stereo (shadows)
![Page 10: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/10.jpg)
Limitations of Laser scanning
• Slow• Cannot capture deformations in time
Source: S. Seitz
![Page 11: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/11.jpg)
Active stereo (color-coded stripes)
- Dense reconstruction - Correspondence problem again- Get around it by using color codes
projectorO
L. Zhang, B. Curless, and S. M. Seitz 2002S. Rusinkiewicz & Levoy 2002
LCD or DLP displays
![Page 12: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/12.jpg)
L. Zhang, B. Curless, and S. M. Seitz. Rapid Shape Acquisition Using Color Structured Light and Multi-pass Dynamic Programming. 3DPVT 2002
Rapid shape acquisition: Projector + stereo cameras
Active stereo (color-coded stripes)
![Page 13: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/13.jpg)
Pattern of projected infrared points to generate a dense 3D image
Active stereo – the kinect sensor
• Infrared laser projector combined with a CMOS sensor• Captures video data in 3D under any ambient light conditions.
Depth map
Source: wikipedia
![Page 14: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/14.jpg)
Lecture 7 -Silvio Savarese 12-Feb-18
Lecture8Activestereo &Volumetricstereo• Activestereo
• Structuredlighting• Depthsensing
• Volumetricstereo:• Spacecarving• Shadowcarving• Voxelcoloring
![Page 15: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/15.jpg)
“Traditional” Stereo
Goal: estimate the position of P given the observation ofP from two view pointsAssumptions: known camera parameters and position (K, R, T)
p’p
O1 O2
P
![Page 16: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/16.jpg)
Subgoals: 1. Solve the correspondence problem2. Use corresponding observations to triangulate
“Traditional” Stereo
p’p
O1 O2
P
![Page 17: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/17.jpg)
Volumetric stereo
1. Hypothesis: pick up a point within the volume
O1O2
Working volume
2. Project this point into 2 (or more) images
Assumptions: known camera parameters and position (K, R, T)
3. Validation: are the observations consistent?
P
![Page 18: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/18.jpg)
Consistency based on cues such as:
- Contours/silhouettes à Space carving- Shadows à Shadow carving- Colors à Voxel coloring
![Page 19: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/19.jpg)
Lecture 7 -Silvio Savarese 12-Feb-18
Lecture8Activestereo &Volumetricstereo
Reading:[Szelisky]Chapter11“Multi-viewstereo”
• Activestereo• Structuredlighting• Depthsensing
• Volumetricstereo:• Spacecarving• Shadowcarving• Voxelcoloring
![Page 20: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/20.jpg)
Contours/silhouettes
• Contoursarearichsourceofgeometricinformation
![Page 21: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/21.jpg)
ApparentContours• Projectionofthelocusofpointsontheobjectsurfacewhich
separatethevisibleandoccludedpartsonthesurface[sato &cipolla]
Image
Camera
apparent contour
![Page 22: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/22.jpg)
Silhouettes
Image
Camera
silhouette
Asilhouetteisdefinedastheareaenclosedbytheapparentcontours
![Page 23: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/23.jpg)
Detectingsilhouettes
Image
Camera
Object
Easy contour segmentation
![Page 24: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/24.jpg)
Detectingsilhouettes
![Page 25: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/25.jpg)
Howcanweusecontours?
Image
Camera
Object
Objectapparent contour
Visual cone
Camera Center
Silhouette
![Page 26: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/26.jpg)
Image Camera
Object
Objectapparent contour
Visual cone
2D slice:
Camera
Image line
Object
Visual cone
Object’s silhouette
Howcanweusecontours?
Silhouette
![Page 27: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/27.jpg)
View point 1
Object
View point 2
Object
Object estimate(visual hull)
• Camera intrinsics are known
• We know location and pose of the camera as it moves around the object
Howcanweusecontours?
![Page 28: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/28.jpg)
Howtoperformvisualconesintersection?• Decompose visual cone in polygonal surfaces
(among others: Reed and Allen ‘99)
![Page 29: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/29.jpg)
Spacecarving
• Using contours/silhouettes in volumetric stereo, also calledspace carving
[ Martin and Aggarwal (1983) ]
Voxel à empty or full
![Page 30: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/30.jpg)
ComputingVisualHullin2D
![Page 31: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/31.jpg)
ComputingVisualHullin2D
![Page 32: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/32.jpg)
ComputingVisualHullin2D
![Page 33: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/33.jpg)
Visualhull:anupperboundestimate
Consistency:A voxel must be projected into a silhouette in each image
ComputingVisualHullin2D
![Page 34: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/34.jpg)
Spacecarvinghascomplexity…
O(N3)
N
N
N
Anysuggestionforspeedingthisprocessup?
Octrees!(Szeliski ’93)
![Page 35: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/35.jpg)
ComplexityReduction:Octrees
1
1
1
![Page 36: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/36.jpg)
• Subdiving volume in sub-volumes of progressive smaller size
2
2
2
ComplexityReduction:Octrees
![Page 37: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/37.jpg)
22
2
ComplexityReduction:Octrees
![Page 38: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/38.jpg)
ComplexityReduction:Octrees
![Page 39: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/39.jpg)
Complexity reduction: 2D example
4 elements analyzed in this step
![Page 40: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/40.jpg)
16 elements analyzed in this step
Complexity reduction: 2D example
![Page 41: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/41.jpg)
52 elements are analyzed in this step
Complexity reduction: 2D example
![Page 42: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/42.jpg)
1+ 4 + 16 + 52 + 544 = 617 voxels have been analyzed in total(rather than 32x32 = 1024)
16x34=544 voxels are analyzed in this step
Complexity reduction: 2D example
![Page 43: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/43.jpg)
AdvantagesofSpacecarving
• Robust and simple• No need to solve for correspondences
![Page 44: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/44.jpg)
LimitationsofSpacecarving
• Accuracy function of numberof views
Not a good estimate
What else?
![Page 45: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/45.jpg)
• Concavities are not modeled Concavity
For 3D objects:Are all types of concavities
problematic?
LimitationsofSpacecarving
![Page 46: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/46.jpg)
• Concavities are not modeled Concavity
(hyperbolic regions are ok)
Visual hull
-- see Laurentini (1995)
LimitationsofSpacecarving
![Page 47: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/47.jpg)
Spacecarving:AClassicSetup
Camera
Object
Turntable
Courtesy of Seitz & Dyer
![Page 48: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/48.jpg)
Spacecarving:AClassicSetup
![Page 49: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/49.jpg)
30 cm
24 poses (15O)voxel size = 1mm
Spacecarving:Experiments
![Page 50: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/50.jpg)
Spacecarving:Experiments
10cm
24 poses (15O)
voxel size = 2mm
6cm
![Page 51: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/51.jpg)
Spacecarving:Conclusions
• Robust• Produce conservative estimates• Concavities can be a problem• Low-end commercial 3D scanners
blue backdrop
turntable
![Page 52: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/52.jpg)
Lecture 7 -Silvio Savarese 12-Feb-18
Lecture8Activestereo &Volumetricstereo• Activestereo
• Structuredlighting• Depthsensing
• Volumetricstereo:• Spacecarving• Shadowcarving• Voxelcoloring
![Page 53: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/53.jpg)
ShapefromShadows• Self-shadows are visual cues for shape recovery
Self-shadows indicate concavities (no modeled by contours)
![Page 54: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/54.jpg)
Shadowcarving:TheSetup
Camera
Turntable
Object
Array of lights
![Page 55: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/55.jpg)
Array of lights
Camera
Turntable
Object
Shadowcarving:TheSetup
![Page 56: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/56.jpg)
Shadowcarving:TheSetup
Array of lights
Camera
Turntable
Object
[Savarese et al ’01]
![Page 57: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/57.jpg)
Shadowcarving
+
Object’s upper bound
Self-shadows
[Savarese et al. 2001]
![Page 58: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/58.jpg)
Algorithm:Stepk
Object
Lightsource
Camera
Imageline
Imageshadow
Upper-bound from step k-1
Image
virtual light image
![Page 59: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/59.jpg)
Image
Object
Lightsource
Camera
Imageline
Imageshadow
Upper-bound from step k-1
Lightsource
virtual image shadow
?
Algorithm:Stepk
Theorem:A voxel that projects into an image shadow AND an virtual image shadow cannot belong to the object.
projection of the shadow points in the image to the upper bound estimate of the object
virtual light image
![Page 60: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/60.jpg)
Image
Lightsource
Imageline
Imageshadow
Upper-bound from step k-1
Object
Lightsource
Camera
Virtual image shadow
No further volumecan be removedCarving process always conservative
Proof of correctness Consistency:
In order for a voxel to be removed it must project into both image shadow and virtual image shadow
Properties: virtual light image
Algorithm:Stepk
![Page 61: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/61.jpg)
Image
Lightsource
Imageline
Imageshadow
Upper-bound from step k-1
Object
Lightsource
Camera
Complexity?O(2N3)
Algorithm:Stepk
Virtual image shadow
virtual light image
![Page 62: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/62.jpg)
SimulatingtheSystem
- 24 positions- 4 lights
- 72 positions- 8 lights
![Page 63: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/63.jpg)
- 16 positions- 4 lights
Results
Space carving
Shadow carving
![Page 64: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/64.jpg)
Results
Space carving Shadow carving
![Page 65: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/65.jpg)
Shadowcarving:Summary• Produces a conservative volume estimate• Accuracy depending on view point and light source number• Limitations with reflective & low albedo regions
![Page 66: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/66.jpg)
Lecture 7 -Silvio Savarese 12-Feb-18
Lecture8Activestereo &Volumetricstereo• Activestereo
• Structuredlighting• Depthsensing
• Volumetricstereo:• Spacecarving• Shadowcarving• Voxelcoloring
![Page 67: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/67.jpg)
VoxelColoring[Seitz & Dyer (‘97)]
[R. Collins (Space Sweep, ’96)]
• Color/photo-consistency• Jointlymodelstructure
andappearance
![Page 68: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/68.jpg)
BasicIdea
View1View3
View2
Is this voxel in or out?
![Page 69: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/69.jpg)
View1View3
View2
BasicIdea
![Page 70: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/70.jpg)
Uniqueness
![Page 71: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/71.jpg)
Uniqueness
• Multipleconsistentscenes
How to fix this?Need to use a visibilityconstraint
![Page 72: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/72.jpg)
Algorithmforenforcingvisibilityconstraints
C
![Page 73: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/73.jpg)
Thepreviousambiguityisremoved!
![Page 74: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/74.jpg)
Thepreviousambiguityisremoved!
![Page 75: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/75.jpg)
AlgorithmComplexity• Voxel coloring visits each N3 voxels only once• Project each voxel into L images
à O(L N3)
NOTE:notfunctionofthenumberofcolors
![Page 76: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/76.jpg)
ACriticalAssumption:Lambertian Surfaces
![Page 77: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/77.jpg)
NonLambertian Surfaces
![Page 78: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/78.jpg)
Photoconsistency Test
If C > l= threshold è voxel consistent
C = corr ( , )
w w’
C = (w−w)( "w − "w )(w−w) ( "w − "w )
w =mean(w)!w =mean(w ')
![Page 79: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/79.jpg)
ExperimentalResults
Dinosaur 72 k voxels colored7.6 M voxels tested7 min to compute on a 250MHz
Imagesource:http://www.cs.cmu.edu/~seitz/vcolor.html
![Page 80: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/80.jpg)
Flower 70 k voxels colored7.6 M voxels tested7 min to compute on a 250MHz
Imagesource:http://www.cs.cmu.edu/~seitz/vcolor.html
ExperimentalResults
![Page 81: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/81.jpg)
Imagesource:http://www.cs.cmu.edu/~seitz/vcolor.html
Room + weird people
ExperimentalResults
![Page 82: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/82.jpg)
VoxelColoring:Conclusions
• Good things– Model intrinsic scene colors and texture– No assumptions on scene topology
• Limitations:– Constrained camera positions– Lambertian assumption
![Page 83: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/83.jpg)
FurtherContributions
• A Theory of Space carving– Voxel coloring in more general framework– No restrictions on camera position
• Probabilistic Space carving
[Kutulakos & Seitz ’99]
[Broadhurst & Cipolla, ICCV 2001][Bhotika, Kutulakos et. al, ECCV 2002]
![Page 84: Lecture 8 Active stereo& - Stanford UniversitySilvio Savarese Lecture 7 - 12-Feb-18 Lecture 8 Active stereo& Volumetric stereo Reading: [Szelisky] Chapter 11 “Multi-view stereo”](https://reader031.fdocuments.net/reader031/viewer/2022041110/5f0f7f2f7e708231d444745e/html5/thumbnails/84.jpg)
Next lecture…
Intro to visual recognition