Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for...
Transcript of Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for...
![Page 1: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/1.jpg)
Image-based modeling
Digital Visual EffectsYung-Yu Chuang
with slides by Richard Szeliski, Steve Seitz and Alexei Efros
![Page 2: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/2.jpg)
Outline
• Models from multiple (sparse) images– Structure from motion– Facade
• Models from single images– Tour into pictures– Single view metrology– Other approaches
![Page 3: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/3.jpg)
Models from multiple images(Façade, Debevec et. al. 1996)
![Page 4: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/4.jpg)
Facade
• Use a sparse set of images• Calibrated camera (intrinsic only)• Designed specifically for modeling architecture• Use a set of blocks to approximate architecture
• Three components:– geometry reconstruction– texture mapping – model refinement
![Page 5: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/5.jpg)
Idea
![Page 6: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/6.jpg)
Idea
![Page 7: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/7.jpg)
Geometric modeling
A block is a geometric primitive with a small set of parameters
Hierarchical modeling for a scene
Rotation and translation could be constrained
![Page 8: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/8.jpg)
Reasons for block modeling
• Architectural scenes are well modeled by geometric primitives.
• Blocks provide a high level abstraction, easier to manage and add constraints.
• No need to infer surfaces from discrete features; blocks essentially provide prior models for architectures.
• Hierarchical block modeling effectively reduces the number of parameters for robustness and efficiency.
![Page 9: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/9.jpg)
Reconstruction
minimize
![Page 10: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/10.jpg)
Reconstruction
![Page 11: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/11.jpg)
Reconstruction
nonlinear w.r.t. camera and model
![Page 12: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/12.jpg)
Results3 of 12 photographs
![Page 13: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/13.jpg)
Results
![Page 14: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/14.jpg)
![Page 15: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/15.jpg)
Texture mapping
![Page 16: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/16.jpg)
Texture mapping in real world
Demo movieMichael Naimark,San Francisco Museum of Modern Art, 1984
![Page 17: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/17.jpg)
Texture mapping
![Page 18: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/18.jpg)
Texture mapping
![Page 19: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/19.jpg)
View-dependent texture mapping
![Page 20: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/20.jpg)
View-dependent texture mapping
model VDTM
VDTMsingle
texturemap
![Page 21: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/21.jpg)
View-dependent texture mapping
![Page 22: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/22.jpg)
Model-based stereo
• Use stereo to refine the geometry
knowncamera
viewpoints
![Page 23: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/23.jpg)
Stereo
scene point
optical center
image plane
![Page 24: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/24.jpg)
Stereo
• Basic Principle: Triangulation– Gives reconstruction as intersection of two rays– Requires
• calibration• point correspondence
![Page 25: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/25.jpg)
Stereo correspondence
• Determine Pixel Correspondence– Pairs of points that correspond to same scene point
• Epipolar Constraint– Reduces correspondence problem to 1D search along
conjugate epipolar lines
epipolar plane epipolar lineepipolar line
![Page 26: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/26.jpg)
Finding correspondences
• apply feature matching criterion (e.g., correlation or Lucas-Kanade) at all pixels simultaneously
• search only over epipolar lines (much fewer candidate positions)
![Page 27: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/27.jpg)
Image registration (revisited)
• How do we determine correspondences?
– block matching or SSD (sum squared differences)
d is the disparity (horizontal motion)
• How big should the neighborhood be?
![Page 28: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/28.jpg)
Neighborhood size
• Smaller neighborhood: more details• Larger neighborhood: fewer isolated mistakes
w = 3 w = 20
![Page 29: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/29.jpg)
Depth from disparity
f
x x’
baseline
z
C C’
X
f
input image (1 of 2)[Szeliski & Kang ‘95]
depth map 3D rendering
![Page 30: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/30.jpg)
– Camera calibration errors– Poor image resolution– Occlusions– Violations of brightness constancy (specular reflections)– Large motions– Low-contrast image regions
Stereo reconstruction pipeline• Steps
– Calibrate cameras– Rectify images– Compute disparity– Estimate depth
• What will cause errors?
![Page 31: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/31.jpg)
Model-based stereokey image
offset image
warped offset image
![Page 32: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/32.jpg)
Results
![Page 33: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/33.jpg)
Comparisons
single texture, flat VDTM, flat
VDTM, model-based stereo
![Page 34: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/34.jpg)
Final results
Kite photography
![Page 35: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/35.jpg)
Final results
![Page 36: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/36.jpg)
![Page 37: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/37.jpg)
Results
![Page 38: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/38.jpg)
Results
![Page 39: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/39.jpg)
Commercial packages
• Autodesk REALVIZ ImageModeler
![Page 40: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/40.jpg)
The Matrix
Cinefex #79, October 1999.
![Page 41: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/41.jpg)
The Matrix
• Academy Awards for Scientific and Technical achievement for 2000To George Borshukov, Kim Libreri and Dan Piponi for the development of a system for image-based rendering allowing choreographed camera movements through computer graphic reconstructed sets.
This was used in The Matrix and Mission Impossible II; See The Matrix Disc #2 for more details
![Page 42: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/42.jpg)
Models from single images
![Page 43: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/43.jpg)
Vanishing points
• Vanishing point– projection of a point at infinity
image plane
cameracenter
ground plane
vanishing point
![Page 44: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/44.jpg)
Vanishing points (2D)
image plane
cameracenter
line on ground plane
vanishing point
![Page 45: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/45.jpg)
Vanishing points
• Properties– Any two parallel lines have the same vanishing point
v– The ray from C through v is parallel to the lines– An image may have more than one vanishing point
image plane
cameracenterC
line on ground plane
vanishing point V
line on ground plane
![Page 46: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/46.jpg)
Vanishing lines
• Multiple Vanishing Points– Any set of parallel lines on the plane define a vanishing
point– The union of all of these vanishing points is the horizon line
• also called vanishing line– Note that different planes define different vanishing lines
v1 v2
![Page 47: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/47.jpg)
Computing vanishing points
• Properties– P is a point at infinity, v is its projection– They depend only on line direction– Parallel lines P0 + tD, P1 + tD intersect at P
V
DPP t 0
0/1///
1Z
Y
X
ZZ
YY
XX
ZZ
YY
XX
t DDD
t
tDtPDtPDtP
tDPtDPtDP
PP
ΠPv
P0
D
![Page 48: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/48.jpg)
Tour into pictures
• Create a 3D “theatre stage” of five billboards
• Specify foreground objects through bounding polygons
• Use camera transformations to navigate through the scene
![Page 49: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/49.jpg)
Tour into pictures
![Page 50: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/50.jpg)
The idea
• Many scenes (especially paintings), can be represented as an axis-aligned box volume (i.e. a stage)
• Key assumptions:– All walls of volume are orthogonal– Camera view plane is parallel to back of volume– Camera up is normal to volume bottom– Volume bottom is y=0
• Can use the vanishing point to fit the box to the particular Scene!
![Page 51: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/51.jpg)
Fitting the box volume
• User controls the inner box and the vanishing point placement (6 DOF)
![Page 52: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/52.jpg)
Foreground Objects• Use separate
billboard for each
• For this to work, three separate images used:– Original image.– Mask to isolate
desired foreground images.
– Background with objects removed
![Page 53: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/53.jpg)
Foreground Objects
• Add vertical rectangles for each foreground object
• Can compute 3D coordinates P0, P1 since they are on known plane.
• P2, P3 can be computed as before (similar triangles)
![Page 54: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/54.jpg)
Example
![Page 55: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/55.jpg)
Example
![Page 56: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/56.jpg)
glTip• http://www.cs.ust.hk/~cpegnel/glTIP/
![Page 57: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/57.jpg)
3D Truth in Old Masters
![Page 58: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/58.jpg)
Criminisi et al. ICCV 1999
1. Find world coordinates (X,Y,Z) for a few points2. Connect the points with planes to model geometry
– Texture map the planes
![Page 59: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/59.jpg)
1 2 3 4
1
2
3
4
Measurements on planes
Approach: unwarp then measureWhat kind of warp is this?
![Page 60: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/60.jpg)
Image rectification
To unwarp (rectify) an image• solve for homography H given p and p’• solve equations of the form: wp’ = Hp
– linear in unknowns: w and coefficients of H– H is defined up to an arbitrary scale factor– how many points are necessary to solve for H?
pp’
![Page 61: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/61.jpg)
Solving for homographies
![Page 62: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/62.jpg)
Solving for homographies
A h 0
• Defines a least squares problem:
2n × 9 9 2n
– Since h is only defined up to scale, solve for unit vector ĥ
– Works with 4 or more points
![Page 63: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/63.jpg)
Finding world coordinates (X,Y,Z)
1. Define the ground plane (Z=0)2. Compute points (X,Y,0) on that plane3. Compute the heights Z of all other points
![Page 64: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/64.jpg)
Measuring height
1
2
3
4
55.4
2.83.3
Camera height
![Page 65: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/65.jpg)
q1
Computing vanishing points
• Intersect p1q1 with p2q2
v
p1
p2
q2
• Least squares version– Better to use more than two lines and compute the “closest”
point of intersection– See notes by Bob Collins for one good way of doing this:
• http://www-2.cs.cmu.edu/~ph/869/www/notes/vanishing.txt
![Page 66: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/66.jpg)
Criminisi et al., ICCV 99• Load in an image• Click on lines parallel to X axis
– repeat for Y, Z axes• Compute vanishing points
![Page 67: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/67.jpg)
Vanishingpoint
Vanishingline
Vanishingpoint
Vertical vanishingpoint
(at infinity)
Criminisi et al., ICCV 99
![Page 68: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/68.jpg)
Criminisi et al., ICCV 99• Load in an image• Click on lines parallel to X axis
– repeat for Y, Z axes• Compute vanishing points• Specify 3D and 2D positions of 4 points on reference
plane• Compute homography H• Specify a reference height • Compute 3D positions of several points• Create a 3D model from these points• Extract texture maps• Output a VRML model
![Page 69: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/69.jpg)
Results
![Page 70: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/70.jpg)
Zhang et. al. CVPR 2001
![Page 71: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/71.jpg)
Zhang et. al. CVPR 2001
![Page 72: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/72.jpg)
Oh et. al. SIGGRAPH 2001
![Page 73: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/73.jpg)
Oh et. al. SIGGRAPH 2001
video
![Page 74: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/74.jpg)
Automatic popup
Input
Ground
Vertical
Sky
Geometric Labels Cut’n’Fold 3D Model
Image
Learned Models
![Page 75: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/75.jpg)
Geometric cuesColor
Location
Texture
Perspective
![Page 76: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/76.jpg)
Automatic popup
![Page 77: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/77.jpg)
Results
Automatic Photo Pop-upInput Images
![Page 78: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/78.jpg)
Results
This approach works roughly for 35% of images.
![Page 79: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/79.jpg)
Failures
Labeling Errors
![Page 80: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/80.jpg)
FailuresForeground Objects
![Page 81: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/81.jpg)
3-sweep
![Page 82: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/82.jpg)
The Avengers• video
![Page 83: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/21...The Matrix • Academy Awards for Scientific and Technical achievement for 2000 To George Borshukov, Kim Libreri and Dan Piponi](https://reader036.fdocuments.net/reader036/viewer/2022071511/612fb4201ecc515869439f0b/html5/thumbnails/83.jpg)
References• P. Debevec, C. Taylor and J. Malik. Modeling and Rendering
Architecture from Photographs: A Hybrid Geometry- and Image-Based Approach, SIGGRAPH 1996.
• Y. Horry, K. Anjyo and K. Arai. Tour Into the Picture: Using a Spidery Mesh Interface to Make Animation from a Single Image, SIGGRAPH 1997.
• A. Criminisi, I. Reid and A. Zisserman. Single View Metrology, ICCV 1999.
• L. Zhang, G. Dugas-Phocion, J.-S. Samson and S. Seitz. Single View Modeling of Free-Form Scenes, CVPR 2001.
• B. Oh, M. Chen, J. Dorsey and F. Durand. Image-Based Modeling and Photo Editing, SIGGRAPH 2001.
• D. Hoiem, A. Efros and M. Hebert. Automatic Photo Pop-up, SIGGRAPH 2005.