Image-Based Modeling and Rendering
Transcript of Image-Based Modeling and Rendering
![Page 1: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/1.jpg)
Image-Based Modeling
and Rendering
Richard Szeliski
Microsoft Research
IPAM Graduate Summer School:
Computer Vision
July 26, 2013
![Page 2: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/2.jpg)
How far have we come?
![Page 3: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/3.jpg)
Light Fields / Lumigraph - 1996
Richard Szeliski Image-Based Rendering and Modeling 3
![Page 4: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/4.jpg)
Environment matting - 2001
Richard Szeliski Image-Based Rendering and Modeling 4
![Page 5: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/5.jpg)
Photo Tourism - 2006
Richard Szeliski Image-Based Rendering and Modeling 5
![Page 6: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/6.jpg)
Ambient Point Clouds - 2010
Richard Szeliski Image-Based Rendering and Modeling 6
![Page 7: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/7.jpg)
Photo Tours - 2012
Richard Szeliski Image-Based Rendering and Modeling 7[Kushal et al., 3DIMPVT 2012]
![Page 8: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/8.jpg)
The Visual Turing Test
Video
Richard Szeliski Image-Based Rendering and Modeling 8[Shan et al., 3DV 2013]
![Page 9: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/9.jpg)
Where are we going?
![Page 10: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/10.jpg)
A personal retrospective
• Panoramas and 360° Video Walkthroughs
• Light Fields and Lumigraphs
• LDIs, Sprites, and Layered Video
• Image-Based Modeling
• Environment Mattes and Matting
• Photo Tourism and Photosynth
• Point-Based Rendering and NPR
• Reflections and Transparency [2012]
Richard Szeliski Image-Based Rendering and Modeling 10
![Page 11: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/11.jpg)
Pre-history (?): view interpolation
• Depth maps and images: view extrapolation
[Matthies,Szeliski,Kanade’89]
input depth image novel view
• View interpolation: warp and dissolve
[Beier & Neely ‘92; Chen & Williams ‘93]
Richard Szeliski Image-Based Rendering and Modeling 11
![Page 12: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/12.jpg)
Panoramic image stitching
Convert image sequence into a cylindrical image
+ + … + =
Richard Szeliski Image-Based Rendering and Modeling 12
![Page 13: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/13.jpg)
Panoramic image stitching
Key technical breakthroughs:
interactive viewing [(Lippman’80), Chen ‘95]
general camera motions [1997…]
parallax compensation [1998…]
fully automated feature detection and
matching [Brown & Lowe ‘03]
seam selection [1975 (1998), 2001…]
Laplacian and Poisson blending [1984, 2003]
Early example of computational photographyRichard Szeliski Image-Based Rendering and Modeling 13
![Page 14: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/14.jpg)
“VR” today: Gigapan
Richard Szeliski Image-Based Rendering and Modeling 14
![Page 15: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/15.jpg)
“VR” today: 360 cities
Richard Szeliski Image-Based Rendering and Modeling 15
![Page 16: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/16.jpg)
“VR” today: Photosynth
Richard Szeliski Image-Based Rendering and Modeling 16
![Page 17: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/17.jpg)
Moving beyond single (or linked)
panoramas…
![Page 18: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/18.jpg)
Panoramic Video-Based Tours
[Uyttendaele et al. 2004]
2D → “3D Light Field” (?)
![Page 19: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/19.jpg)
Video-Based Walkthroughs
Move camera along a rail (“dolly track”) and
play back a 360 video
Applications:
• Homes and architecture
• Outdoor locations
(tourist destinations)
Richard Szeliski Image-Based Rendering and Modeling 19
![Page 20: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/20.jpg)
Surround video acquisition system
OmniCam (six-camera head)
[ Point Grey Ladybug ]
Richard Szeliski Image-Based Rendering and Modeling 20
![Page 21: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/21.jpg)
Acquisition platforms (then)
Robotic cart
Wearable
Richard Szeliski Image-Based Rendering and Modeling 22
![Page 22: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/22.jpg)
Acquisition platforms (today)
Richard Szeliski Image-Based Rendering and Modeling 23
![Page 23: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/23.jpg)
Acquisition platforms (future?)
Richard Szeliski Image-Based Rendering and Modeling 24
![Page 24: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/24.jpg)
Acquisition platforms (now)
• 10 camera panoramic rig
• One bubble every 2 meters
• Reproject into cube maps
Richard Szeliski Image-Based Rendering and Modeling 25
![Page 25: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/25.jpg)
Lightfields and Lumigraphs
“True 4D”
(with lots of slides from Michael Cohen)
![Page 26: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/26.jpg)
Modeling light
How do we generate new scenes and
animations from existing ones?
Classic “3D Vision + Graphics”:
• take (lots of) pictures
• recover camera pose
• build 3D model
• extract texture maps / BRDFs
• synthesize new views
• … tons of great work and demos as 3DIMPVT
Richard Szeliski Image-Based Rendering and Modeling 27
![Page 27: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/27.jpg)
Real Scene
Computer Vision
Real Cameras
Model
Output
Richard Szeliski Image-Based Rendering and Modeling 28
![Page 28: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/28.jpg)
Computer Graphics
Image
Output
ModelSynthetic
Camera
Richard Szeliski Image-Based Rendering and Modeling 29
![Page 29: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/29.jpg)
Combined
Model Real Scene
Real Cameras
Image
Output
Synthetic
Camera
Richard Szeliski Image-Based Rendering and Modeling 30
![Page 30: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/30.jpg)
But, vision technology falls short
ModelReal Scene
Real Cameras
Image
Output
Synthetic
Camera
Richard Szeliski Image-Based Rendering and Modeling 31
![Page 31: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/31.jpg)
… and so does graphics.
ModelReal Scene
Real Cameras
Image
Output
Synthetic
Camera
Richard Szeliski Image-Based Rendering and Modeling 32
![Page 32: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/32.jpg)
Image Based Rendering
Images+ModelReal Scene
Real Cameras
-or-
Expensive Image Synthesis
Image
Output
Synthetic
Camera
Richard Szeliski Image-Based Rendering and Modeling 33
![Page 33: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/33.jpg)
Lumigraph / Lightfield
Outside convex space
4D
StuffEmpty
Richard Szeliski Image-Based Rendering and Modeling 34
![Page 34: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/34.jpg)
2D position
2D position
2 plane parameterization
su
Lumigraph - Organization
Richard Szeliski Image-Based Rendering and Modeling 35
![Page 35: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/35.jpg)
2D position
2D position
2 plane parameterization
us
t s,t
u,v
v
s,t
u,v
Lumigraph - Organization
Richard Szeliski Image-Based Rendering and Modeling 36
![Page 36: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/36.jpg)
For each output pixel
• determine s,t,u,v
• either
– find closest discrete RGB
– interpolate near valuess,t u,v
Lumigraph - Rendering
Richard Szeliski Image-Based Rendering and Modeling 37
![Page 37: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/37.jpg)
For each output pixel
• determine s,t,u,v
• either
• use closest discrete RGB
• interpolate near values
Lumigraph - Rendering
s uRichard Szeliski Image-Based Rendering and Modeling 38
![Page 38: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/38.jpg)
Lumigraph - Rendering
Nearest
• closest s
• closest u
• draw it
Blend 16 nearest
• quadrilinear interpolation
s uRichard Szeliski Image-Based Rendering and Modeling 39
![Page 39: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/39.jpg)
Lumigraph - Rendering
Depth Correction
• quadralinear interpolation
• new “closest”
• like focus
s uRichard Szeliski Image-Based Rendering and Modeling 40
![Page 40: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/40.jpg)
Lumigraph – Image Effects
Image effects:
• parallax
• occlusion
• transparency
• highlights
Richard Szeliski Image-Based Rendering and Modeling 43
![Page 41: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/41.jpg)
Unstructured Lumigraph
What if the images
aren’t sampled on
a regular 2D grid?
• can still re-sample
rays
• ray weighting
becomes more
complex[Buehler et al.,
SIGGRAPH’2000]
Richard Szeliski Image-Based Rendering and Modeling 45
![Page 42: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/42.jpg)
Surface Light Fields
[Wood et al, SIGGRAPH 2000]
Turn 4D parameterization around:
image @ every
surface pt.
Leverage coherence:
compress radiance fn
(BRDF * illumination)
after rotation by n
Richard Szeliski Image-Based Rendering and Modeling 46
![Page 43: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/43.jpg)
Surface Light Fields
[Wood et al, SIGGRAPH 2000]
Richard Szeliski Image-Based Rendering and Modeling 47
![Page 44: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/44.jpg)
2.5D Representations
Images (and panoramas) are 2D
Lumigraph is 4D
3D• 3D Lumigraph subsets (360 video tours)
• Concentric mosaics
2.5D• Layered Depth Images
• Sprites with Depth (impostors)
• Layered (Virtual Viewpoint) Video
• View Dependent Surfaces (see Façade)
Richard Szeliski Image-Based Rendering and Modeling 49
![Page 45: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/45.jpg)
Layered Depth Image
ImageDepthLayered
2.5 D ?
Richard Szeliski Image-Based Rendering and Modeling 50
![Page 46: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/46.jpg)
Layered Depth Image
Rendering from LDI[Shade et al., SIGGRAPH’98]
• Incremental in LDI X and Y
• Guaranteed to be in back-to-front order
Richard Szeliski Image-Based Rendering and Modeling 51
![Page 47: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/47.jpg)
Sprites with Depth
Represent scene as collection of cutouts with
depth (planes + parallax)
Render back to front with fwd/inverse warping [Shade et al., SIGGRAPH’98]
Basis of Virtual
Viewpoint Video
[Zitnick et al. 2004]
Richard Szeliski Image-Based Rendering and Modeling 53
![Page 48: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/48.jpg)
Virtual Viewpoint (3D) Video
[Zitnick et al., SIGGRAPH 2004]
![Page 49: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/49.jpg)
Scenarios for 3D video
Sports events, e.g., CMU’s 30-camera
“EyeVision” system at
SuperBowl XXXV)
Concert performances,
plays, circus acts
Games
Instructional video,
e.g., golf, skating, martial arts
Interactive (Internet) video
Richard Szeliski Image-Based Rendering and Modeling 55
![Page 50: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/50.jpg)
3D video: Challenges
Capture: record multiple synchronized
camera streams
Processing and representation:
ensure photorealism
Compression: exploit redundancy
Rendering: provide
interactive viewpoint control
Richard Szeliski Image-Based Rendering and Modeling 56
![Page 51: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/51.jpg)
concentrators
hard
disks
cameras
controlling
laptop
Richard Szeliski Image-Based Rendering and Modeling 58
![Page 52: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/52.jpg)
Matching pixels
Easy
Richard Szeliski Image-Based Rendering and Modeling 59
![Page 53: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/53.jpg)
Harder
Richard Szeliski Image-Based Rendering and Modeling 60
![Page 54: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/54.jpg)
Really Really Hard
Richard Szeliski Image-Based Rendering and Modeling 61
Occlusion
![Page 55: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/55.jpg)
Matting
Some pixels
get
influence
for multiple
surfaces.
Background Surface
Foreground Surface
Image
Camera
Close up of real image:
Multiple colors and
depths at boundary
pixels…Richard Szeliski Image-Based Rendering and Modeling 62
![Page 56: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/56.jpg)
Find matting information:
1. Find boundary
strips using depth.
2. Within boundary strips compute the colors and
depths of the foreground and background
object.
Background
ForegroundStrip
WidthRichard Szeliski Image-Based Rendering and Modeling 63
![Page 57: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/57.jpg)
Why matting is important
MattingNo Matting
Richard Szeliski Image-Based Rendering and Modeling 64
![Page 58: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/58.jpg)
Representation
Main Layer: Boundary Layer:
Color Color
Depth Depth
Alpha
Main
Boundary
Strip
Width
Richard Szeliski Image-Based Rendering and Modeling 65
![Page 59: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/59.jpg)
Richard Szeliski Image-Based Rendering and Modeling 66
![Page 60: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/60.jpg)
Outline
• Panoramas and 360° Video Walkthroughs
• Light Fields and Lumigraphs
• LDIs, Sprites, and Layered Video
Image-Based Modeling
• Environment Mattes and Matting
• Photo Tourism and Photosynth
• Point-Based Rendering and NPR
• Reflections and Transparency [2012]
Richard Szeliski Image-Based Rendering and Modeling 67
![Page 61: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/61.jpg)
Image-Based Modeling
![Page 62: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/62.jpg)
Image-Based Modeling
Richard Szeliski Image-Based Rendering and Modeling 69
![Page 63: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/63.jpg)
Façade
1. Select building blocks
2. Align them in each image
3. Solve for camera pose
and block parameters
(using constraints)
Richard Szeliski Image-Based Rendering and Modeling 71
![Page 64: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/64.jpg)
View-dependent texture mapping
1. Determine visible cameras for each surface
element
2. Blend textures (images) depending on
distance between original camera and novel
viewpoint
Richard Szeliski Image-Based Rendering and Modeling 72
![Page 65: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/65.jpg)
Graz reconstruction
Fully automated multi-view stereo, ca. 2009
Richard Szeliski Image-Based Rendering and Modeling 74
![Page 66: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/66.jpg)
Non-photorealistic experiences
![Page 67: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/67.jpg)
Photo Tourism
Computed 3D structureImages on the Internet
[Snavely, Seitz, Szeliski, SIGGRAPH 2006]Richard Szeliski Image-Based Rendering and Modeling 76
![Page 68: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/68.jpg)
Photo Tourism
Richard Szeliski Image-Based Rendering and Modeling 77
![Page 69: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/69.jpg)
Internet Computer Vision
Use the Internet as an image and/or annotation
source to solve challenging vision problems.
Richard Szeliski Image-Based Rendering and Modeling 78
![Page 70: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/70.jpg)
Internet Computer Vision
Richard Szeliski Image-Based Rendering and Modeling 79
![Page 71: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/71.jpg)
Internet Computer Vision
Richard Szeliski Image-Based Rendering and Modeling 80
![Page 72: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/72.jpg)
Recent research
![Page 73: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/73.jpg)
Piecewise planar proxies
[Sinha, Steedly, Szeliski ICCV’09]
Richard Szeliski Image-Based Rendering and Modeling 82
![Page 74: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/74.jpg)
Building Interiors
[ Furukawa et al., ICCV’09]Richard Szeliski Image-Based Rendering and Modeling 84
![Page 75: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/75.jpg)
Internet-Scale Stereo
[ Furukawa et al., CVPR’10]Richard Szeliski Image-Based Rendering and Modeling 85
![Page 76: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/76.jpg)
What’s missing?
Reflections
Gloss
…?
Richard Szeliski Image-Based Rendering and Modeling 86
![Page 77: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/77.jpg)
Reflections
Richard Szeliski Image-Based Rendering and Modeling 87
![Page 78: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/78.jpg)
Image-Based Rendering for
Scenes with Reflections
Sudipta N. Sinha
Johannes Kopf
Michael Goesele
Daniel Scharstein
Richard Szeliski
![Page 79: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/79.jpg)
Wrapping up…
![Page 80: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/80.jpg)
Graphics/Imaging Continuum
Lumigraph Light field
Geometry centric Image centric
Warping InterpolationPolygon rendering +
texture mapping
Fixed
geometry
View-
dependent
geometry
View-
dependent
texture
Concentric mosaics
Sprites
with
depth
LDI
Richard Szeliski Image-Based Rendering and Modeling 90
![Page 81: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/81.jpg)
What works? What doesn’t?
Automatic 3D pose estimation
Aerial (+ active ground level) modeling
• Accurate boundaries & matting
• Reflections and transparency
• User-generated content
• Casually acquired photos and videos
New sensors
• Active sensing (Kinect…); Plenoptic cameras
• Integration with recognitionRichard Szeliski Image-Based Rendering and Modeling 91
![Page 82: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/82.jpg)
Richard Szeliski Image-Based Rendering and Modeling 92
What about Computational Photography?
![Page 83: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/83.jpg)
Image-Based Rendering
• Panoramas and 360° Video Walkthroughs
• Light Fields and Lumigraphs
• Layered Representations and Matting
• Image-Based Modeling
• Environment Mattes and Matting
• Photo Tourism and Photosynth
• Point-Based Rendering and NPR
• Reflections and Transparency
• Where next?
• How to use in your applications?
Richard Szeliski Image-Based Rendering and Modeling 93
![Page 84: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/84.jpg)
Graphics/Imaging Continuum
Lumigraph Light field
Geometry centric Image centric
Warping InterpolationPolygon rendering +
texture mapping
Fixed
geometry
View-
dependent
geometry
View-
dependent
texture
Concentric mosaics
Sprites
with
depth
LDI
Richard Szeliski Image-Based Rendering and Modeling 94
![Page 85: Image-Based Modeling and Rendering](https://reader031.fdocuments.net/reader031/viewer/2022020706/61fc81038d33c02b785df0eb/html5/thumbnails/85.jpg)
( Questions )
[ The End ]