Course 10 Realistic Materials in Computer...
Transcript of Course 10 Realistic Materials in Computer...
1
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Course 10Realistic Materials in Computer Graphics
Course 10Realistic Materials in Computer Graphics
Environment Matting
Yung-Yu Chuang
National Taiwan University
Environment Matting
Yung-Yu Chuang
National Taiwan University
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
input image alpha composite
Blue screen mattingBlue screen matting
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
alpha compositesource image
Problem: blue foregroundProblem: blue foreground
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Two-screen mattingTwo-screen matting
alpha compositeCourse 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Problem: refractive objectProblem: refractive object
alpha composite
2
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
alpha composite photograph
Problem: refractive objectProblem: refractive object
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Refracted image of a single pixelRefracted image of a single pixel
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Environment matting frameworkEnvironment matting framework
backgroundC
composite
T
F
foreground
Ccomposite
Tbackground
F
foreground
Ccomposite
Tbackground
1-αB
3
F
foreground
Ccomposite
Tbackground
W
weightingfunction
0
C Tweightingfunction
F
foreground
W
backgroundcomposite
C Tweightingfunction
W
backgroundcomposite
Arbitrary weighting function
Arbitrary weighting function
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Environment matting algorithmsEnvironment matting algorithms• A model to approximate the environment
compositing equation
• A method to estimate model parameters
• Complexity of the model determines the complexity of the captured effects
4
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Environment Matting and CompositingEnvironment Matting and Compositing
Douglas Zongker Dawn WernerBrian Curless David Salesin
SIGGRAPH 1999 F
foreground
Ccomposite
Tbackground
1-αB
AR
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Acquisition setupAcquisition setup
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Hierarchical backgroundsHierarchical backgrounds
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Hierarchical backgroundsHierarchical backgrounds
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Searching for α and ASearching for α and A
2)(computedobserved∑ −= ACCE
hypothesized A
5
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
),( rl ),( bt
Separate x and y extent searchesSeparate x and y extent searches
foreground alpha(coverage)
background
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Environment matte compositeEnvironment matte composite
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
alpha mattecomposite
environment mattecomposite
Results and comparisonsResults and comparisons
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
photographenvironment mattecomposite
Results and comparisonsResults and comparisons
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
environment mattecomposite
ResultsResults
environment mattecomposite
6
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Problem: glossy surfaceProblem: glossy surface
environment mattecomposite photograph
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Problem: multiple mappingsProblem: multiple mappings
environment mattecomposite photograph
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Problem: color dispersionProblem: color dispersion
environment mattecomposite photograph
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Problem: many photographs Problem: many photographs
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Problem: many photographs Problem: many photographs
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Environment Matting Extensions: Toward Higher Accuracy and Real-Time Capture
Environment Matting Extensions: Toward Higher Accuracy and Real-Time Capture
Yung-Yu Chuang Douglas Zongker Joel HindorffBrian Curless David Salesin Richard Szeliski
SIGGRAPH 2000
7
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
O(log n)
higheraccuracy
fewerimages
Environment mattingEnvironment matting
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
O(log n) 1
fewerimages
higheraccuracy
Towards real-time captureTowards real-time capture
F
foreground
Ccomposite
Tbackground
1-αB
W
weightingfunction
AR
3
3 observations
3 1 4
11 variables
• A, R• α• F
3
4
7 variables
• A, R• α• F
33
3 observations
4
5 variables
• A, R• α• F
13
3 observations
A, ρcolorless
8
3 variables
• A, R• α• F
13
3 observations
A, ρcolorless
T(cx , cy)11
cx , cy , ρspecularlyrefractive
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
T
M(T , A)A
x
y
(cx, cy)
≈ T(cx, cy)
Stimulus functionStimulus function
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
x
y
T(x,y)
Ideal plane in RGB cubeIdeal plane in RGB cube
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
T(x,y)
x
y
(cx ,cy)C′
Calibrated manifold in RGB cubeCalibrated manifold in RGB cube
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
T
backgroundmanifold
K
W
C′
C′ → (cx ,cy)
Cρρ = KC
KC′
Estimate cx ,cy and ρEstimate cx ,cy and ρ
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Problem: noisy matteProblem: noisy matte
9
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
with filteringwithout filtering
Edge-preserving filteringEdge-preserving filtering
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Input imageInput image
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Difference thresholdingDifference thresholding
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Morphological operationMorphological operation
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
FeatheringFeathering
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Heuristics for specular highlightsHeuristics for specular highlights
T
backgroundmanifold
K
W
Cρ >1+ε C′
10
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Heuristics for specular highlightsHeuristics for specular highlights
C=ρT(cx, cy)
ρ cx cy
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
=
input estimation foreground(highlights)
Heuristics for specular highlightsHeuristics for specular highlights
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Composite with highlightsComposite with highlights
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
O(log n) 1
fewerimages
higheraccuracy
Towards real-time captureTowards real-time capture
11
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
O(log n)O(n)
fewerimages
higheraccuracy
Towards higher accuracyTowards higher accuracy
1
Arbitrary weighting function
Multimodal oriented Gaussian Multimodal oriented Gaussian
σu
σv θ
• Better BRDF approximation
• Multiple mappings
• Wavelength-coupled variation
Multimodal oriented Gaussian Unimodal axis-aligned Gaussian
backgroundcomposite
12
Unimodal axis-aligned Gaussian
σu backgroundcomposite
Unimodal axis-aligned Gaussian
Unimodal axis-aligned Gaussianσs
Tbackground
composite
Unimodal axis-aligned Gaussian
Unimodal axis-aligned Gaussian
cu
cv c
Unimodal axis-aligned Gaussian
13
cσv
σu
Unimodal axis-aligned Gaussian Unimodal oriented Gaussian
θσv
σu
Multimodal oriented Gaussian Multimodal oriented Gaussian
Multimodal oriented Gaussian
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
14
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Glossy surfaceGlossy surface
SIGGRAPH 99 photograph
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Glossy surfaceGlossy surface
higher accuracyalgorithm
photograph
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Oriented GaussianOriented Gaussian
without orientation photograph
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Oriented GaussianOriented Gaussian
with orientation photograph
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Multiple mappingsMultiple mappings
SIGGRAPH 99 photograph
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Multiple mappingsMultiple mappings
higher accuracyalgorithm
photograph
15
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Color dispersionColor dispersion
SIGGRAPH 99 photograph
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Color dispersionColor dispersion
higher accuracyalgorithm
photograph
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Frequency-based Environment MattingFrequency-based Environment Matting
Jiayuan Zhu Yee-Hong Yang
Pacific Graphics 2004
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Frequency-based environment mattingFrequency-based environment matting
spectrum of C
C
B(x1, t)
B(x2, t)
C(t)
x1x2
t
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Frequency-based environment mattingFrequency-based environment matting
)()(),( yWxWyxW yx ⋅=
t
xttxB π2sin),( =
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Results: refractionResults: refraction
frequency-based environment matting
photograph
16
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Results: color dispersionResults: color dispersion
frequency-based environment matting
photograph
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Results: orientedResults: oriented
frequency-based environment matting
photograph
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Wavelet Environment MattingWavelet Environment Matting
Pieter Peers Philip DutréEurographics Symposium on Rendering 2003
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Wavelet environment mattingWavelet environment matting
∫= WTC ∑=i
iiTaT
∫ ∑ ⎟⎠
⎞⎜⎝
⎛=
iiiTaW
∑ ∫=i
ii TWa
∑=i
iiCa ii TWC ∫=
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Wavelet environment mattingWavelet environment matting
a1 a2 a3 ak ak+1
iTT
ii TWC ∫=CCourse 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Wavelet environment mattingWavelet environment matting
a1 a2 a3 ak ak+1
17
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Error treeError tree
W
W
W
+
+
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuangreference image
Results: number of basis imagesResults: number of basis images
100 basis images300 basis images600 basis images900 basis images1200 basis images
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Results: number of basis imagesResults: number of basis images
reference image 100 basis images300 basis images600 basis images900 basis images1200 basis images
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
reference image
1000 Haar patterns
1000 Daubechies(9,7) patterns
Results: wavelet patternsResults: wavelet patterns
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Results: wavelet patternsResults: wavelet patterns
reference image
1000 Haar patterns
1000 Daubechies (9,7) patterns
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Results: wavelet patternsResults: wavelet patterns
reference image
1000 Haar patterns
1000 Daubechies (9,7) patterns
18
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Results: diffuse materialsResults: diffuse materials
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Image-based Environment MattingImage-based Environment Matting
Yonatan Wexler Andrew Fitzgibbon Andrew Zisserman
Eurographics Workshop on Rendering 2002
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Image-based environment mattingImage-based environment matting
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Image-based environment mattingImage-based environment matting
p
q
2)()()( qBpCeqP −−=
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Image-based environment mattingImage-based environment matting
p
q
2)()()( qBpCeqP −−=
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Image-based environment mattingImage-based environment matting
19
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Image-based environment mattingImage-based environment matting
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
ResultsResults
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
ResultsResults
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
SummarySummary
number of images
complexity of visual effects
1 logn n
HEM
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
SummarySummary
number of images
complexity of visual effects
1 logn n
HEM
RTEM
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
SummarySummary
number of images
complexity of visual effects
1 logn n
HEM
RTEM
GEM
20
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
SummarySummary
number of images
complexity of visual effects
1 logn n
HEM
RTEM
GEMFBEM
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
SummarySummary
number of images
complexity of visual effects
1 logn n
HEM
RTEM
GEMFBEM
WEM
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
SummarySummary
number of images
complexity of visual effects
1 logn n
HEM
RTEM
GEMFBEM
WEM
IBEM
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Acquisition and Rendering ofTransparent and Refractive Objects
Acquisition and Rendering ofTransparent and Refractive Objects
Wojciech Matusik Hanspeter PfisterRemo Ziegler Addy Ngan Leonard McMillan
Eurographics Workshop on Rendering 2002
21
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
The systemThe system
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Geometry - opacity hullGeometry - opacity hull
• Visual hull augmented with view-dependent opacity
• Store the opacity of each observation at each point on the visual hull.
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
ResultsResults
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
InterpolationInterpolation
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
ComparisonsComparisons
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
ReferenceReference
• D. Zongker, D. M. Werner, B. Curless, and D. H. Salesin. Environment Matting and Compositing, SIGGRAPH 1999, pp205-214.
• Yung-Yu Chuang, Douglas E. Zongker, Joel Hindorff, Brian Curless, David H. Salesin, Richard Szeliski, Environment Matting Extensions: Towards Higher Accuracy and Real-Time Capture, SIGGRAPH 2000.
• P. Peers and P. Dutre. Wavelet Environment Matting, EGSR 2003.
• J. Zhu and Y.-H. Yang. Frequency-Based Environment Matting, Pacific Graphics 2004.
• Y. Wexler, A. Fitzgibbon and A. Zisserman. Image-Based Environment Matting, EGWR 2002, pp279-289.
22
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Future workFuture work
• More general materials (Specular + diffuse)
• Principled Bayesian approach
• Advanced rendering
• Editing
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
ConclusionConclusion
• No restriction on material properties
• Linearly combining weighted basis images
• Hierarchical algorithm– Wavelets
– Feedback loop
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Future WorkFuture Work
• Extend to full Image Based Relighting technique
• Better stop-criterion
• Environment matting with moving camera
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
On Refractive Optical FlowOn Refractive Optical Flow
Sameer Agarwal Satya MallickDavid Kriegman Serge Belongie
ECCV 2004
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Refractive optical flowRefractive optical flow
refractive optical flow equation
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
ResultsResults
without water
with water
23
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
AcknowledgementsAcknowledgements
Douglas Zongker, Dawn Werner, Brian Curless, David Salesin, Joel Hindorff, Richard Szeliski, Jiayuan Zhu, Yee-Hong Yang, Pieter Peers, Philip Dutre, Yonatan Wexler, Andrew Fitzgibbon, Andrew Zisserman, SameerAgarwal, Satya Mallick, David Kriegman, Serge Belongie, Wojciech Matusik, HanspeterPfister, Remo Ziegler, Addy Ngan, Leonard McMillan
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Results: orientedResults: oriented
frequency-based environment matting
photograph
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Results: orientedResults: oriented
frequency-based environment matting
photograph
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Refracted image of a single pixelRefracted image of a single pixel
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Refracted image of a single pixelRefracted image of a single pixel
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Refracted image of a single pixelRefracted image of a single pixel
24
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Refracted image of a single pixelRefracted image of a single pixel
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
isotropic filter anisotropic filter
Edge-preserving filteringEdge-preserving filtering
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Depth correctionDepth correction
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Capturing at a single depthCapturing at a single depth
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Capturing a second depthCapturing a second depth
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Constructing the 3D beamConstructing the 3D beam
25
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Rendering at novel depthsRendering at novel depths
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Rendering at novel depthsRendering at novel depths
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Wavelet environment mattingWavelet environment matting
∫= WTC ∑=i
iiTaT
∫ ∑ ⎟⎠
⎞⎜⎝
⎛=
iiiTaW
∑ ∫=i
ii TWa
∑=i
iiCa
ii TWC ∫=
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
c
A
wx
wy
Estimate wx ,wyEstimate wx ,wy
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Reflective objectsReflective objects
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Many rays not capturedMany rays not captured
26
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Add sidedrops to capture these raysAdd sidedrops to capture these rays
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Capturing multiple sidesCapturing multiple sides
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Contributions from multiple sidesContributions from multiple sides
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
Computing R and F from alphaComputing R and F from alpha
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
alpha mattecomposite
environment mattecomposite
Results and comparisonsResults and comparisons
27
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
photographenvironment mattecomposite
Results and comparisonsResults and comparisons
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
alpha mattecomposite
environment mattecomposite
Results and comparisonsResults and comparisons
Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang
photographenvironment mattecomposite
Results and comparisonsResults and comparisons