\\Server01\S Homes$\SchüLer\Toita Abdulaeva\It\Pack And Co PräSentation\Flintstones
MIT Media Lab - Motivation TensorTextures: …maov/classes/comp_photo...Flintstones Bird Title...
9
TensorTextures: Multilinear Image-Based Rendering TensorTextures: Multilinear Image-Based Rendering Computer Graphics • Goal: Generation of photorealistic virtual environments • Classical Computer Graphics: Model – based Rendering – From object models to images – Model specifies geometry of a scene and surface properties – Images are generated by projecting 3D model onto an image plane and computing surface shading • Photorealism requires complex models – Difficult – Time consuming Motivation Motivation • World is modeled by a collection of images (and possibly some coarse geometry) • These images are used to synthesize novel images representing the scene from arbitrary viewpoints and illuminations • Advantages: – Rendering is decoupled from the scene complexity – Photorealism is improved Image-Based Rendering [Gortler et al. 1996, Levoy & Hanrahan 1996, Debevec, Taylor & Malik 1996, …] Image-Based Rendering [Gortler et al. 1996, Levoy & Hanrahan 1996, Debevec, Taylor & Malik 1996, …] Our Contribution Our Contribution • We introduce a tensor framework for image-based rendering (IBR) – Specifically, rendering of 3D textured surfaces • Surface appearance is determined by the complex interaction of multiple factors: – Scene geometry – Illumination – Imaging Bidirectional Texture Function Bidirectional Texture Function • BTF: Captures the appearance of extended textured surfaces with – Spatially varying reflectance – Surface mesostructure (3D texture) – Subsurface scattering – Etc. • Generalization of BRDF, which accounts only for surface microstructure at a point Plaster Pebbles Concrete BTF Texture Mapping [Dana et al. 1999] BTF Texture Mapping [Dana et al. 1999] Standard Texture Mapping BTF Texture Mapping
Transcript of MIT Media Lab - Motivation TensorTextures: …maov/classes/comp_photo...Flintstones Bird Title...
1
TensorTextures: Multilinear Image-Based Rendering
TensorTextures: Multilinear Image-Based Rendering
Computer Graphics
• Goal: Generation of photorealistic virtual environments
• Classical Computer Graphics: Model – based Rendering– From object models to images– Model specifies geometry of a scene and surface properties– Images are generated by projecting 3D model onto an image
plane and computing surface shading
• Photorealism requires complex models– Difficult – Time consuming
MotivationMotivation
• World is modeled by a collection of images (and possibly some coarse geometry)
• These images are used to synthesize novel images representing the scene from arbitrary viewpoints and illuminations
• Advantages:– Rendering is decoupled from the scene complexity– Photorealism is improved
Image-Based Rendering[Gortler et al. 1996, Levoy & Hanrahan 1996,
Debevec, Taylor & Malik 1996, …]
Image-Based Rendering[Gortler et al. 1996, Levoy & Hanrahan 1996,
Debevec, Taylor & Malik 1996, …]
Our ContributionOur Contribution
• We introduce a tensor framework for image-based rendering (IBR)– Specifically, rendering of 3D textured surfaces
• Surface appearance is determined by the complex interaction of multiple factors:– Scene geometry– Illumination– Imaging
Bidirectional Texture FunctionBidirectional Texture Function
• BTF: Captures the appearance of extended textured surfaces with– Spatially varying reflectance– Surface mesostructure (3D texture)– Subsurface scattering– Etc.
• Generalization of BRDF, which accounts only for surface microstructure at a point
PlasterPebblesConcrete
BTF Texture Mapping[Dana et al. 1999]
BTF Texture Mapping[Dana et al. 1999]
StandardTexture Mapping
BTFTexture Mapping
2
BTFBTFBTF
• Reflectance as a function of position on surface, view direction, and illumination direction
• The BTF captures shading and mesostructuralself-shadowing, self-occlusion, interreflection
),,,,,( iivvBTF yxf φθφθposition on surface
view direction
illumination direction
photometric angles
TensorTexture MappingTensorTexture Mapping
+
StandardTexture Mapping
TensorTextureMapping
TextureGeometry
TensorTexturesTensorTextures: Learns BTFs from ensembles of sample imagesNonlinear generative BTF model
• BTF introduced by Dana et al. [1999]
• BTF acquisition devices[Debevec et al. 2000][Dana 2001] [Furukawa et al. 2002] [Han & Perlin 2003] (BTF Kaleidoscope)
• BTF based rendering methods– Polynomial texture maps
[Malzbender et al. 2001]– Synthesis of BTFs for curved surfaces
[Liu et al. 2001] [Tong et al. 2002]
• BTF introduced by Dana et al. [1999]
• BTF acquisition devices[Debevec et al. 2000][Dana 2001] [Furukawa et al. 2002] [Han & Perlin 2003] (BTF Kaleidoscope)
• BTF based rendering methods– Polynomial texture maps
[Malzbender et al. 2001]– Synthesis of BTFs for curved surfaces
[Liu et al. 2001] [Tong et al. 2002]
BackgroundBackground TensorTextures OverviewTensorTextures Overview
1. Mathematical foundations: Eigentextures– Linear Analysis / Principal Components Analysis
• fixed viewpoint, changing illumination
• changing viewpoint and illumination
2. TensorTextures– Nonlinear (multilinear) Analysis / Tensor decomposition
3. Experiments and results
1. Mathematical foundations: Eigentextures– Linear Analysis / Principal Components Analysis
• fixed viewpoint, changing illumination
• changing viewpoint and illumination
2. TensorTextures– Nonlinear (multilinear) Analysis / Tensor decomposition
3. Experiments and results
THE FOLLOWING PREVIEW HAS BEEN APPROVED FOR
ALL AUDIENCES
BY THE MOTION PICTURE DISASSOCIATION OF AMERICA
THE FOLLOWINGTHE FOLLOWING PREVIEW PREVIEW HAS BEEN APPROVED FORHAS BEEN APPROVED FOR
ALL AUDIENCESALL AUDIENCES
BY THE MOTION PICTURE DISASSOCIATION OF AMERICABY THE MOTION PICTURE DISASSOCIATION OF AMERICA
3
“Eigentextures” – PCA(Matrix Algebra)
“Eigentextures” – PCA(Matrix Algebra)
Simple Data Acquisition: Fixed Viewpoint, Varying Illumination
Simple Data Acquisition: Fixed Viewpoint, Varying Illumination
Sample images are points in “pixel space”
Sample images are points in “pixel space”
Images
Texe
ls
IRNxM
texel 1
texe
lNM
texel 2
IRNMx1
D
• Eigentextures – captures variation across illuminations• Eigentextures – captures variation across illuminations
The 1-Mode Case(fixed viewpoint, varying illumination)
Images
Pixe
ls
D
texel 1
texe
lNM
texel 2
Principal Components Analysis (PCA) -Eigentextures
Principal Components Analysis (PCA) -Eigentextures
• Eigentextures – captures variation across illuminations• Eigentextures – captures variation across illuminations • Note: This is a linear representation• Note: This is a linear representation
texel 1
texe
lNM
texel 2
Image Representation using PCA Image Representation using PCA
c1c2
+ c1 + c2c0 + ck
Its PCA RepresentationAn arbitrary image
Ucd =d
4
• This poses a 3-mode BTF estimation problem– Viewpoint, illumination, and pixel modes
• This poses a 3-mode BTF estimation problem– Viewpoint, illumination, and pixel modes
Sampling Multiple Viewpoints and Illuminations
Sampling Multiple Viewpoints and Illuminations
Image Rectification
Rectify
Rectifying HomographyRectifying Homography
1 2
34
12
34
• Image unwarping– H can be computed given at least 4 fiducials p’ & p
• Image unwarping– H can be computed given at least 4 fiducials p’ & p
Hpp ='
Rectify
Rectify
Applying PCAPixe
ls
Images
• Eigentextures – variation across views and illuminations
PCA ReconstructionPCA Reconstruction
111 basis vectors
33 basis vectors
original
TensorTextures(Tensor Algebra)
TensorTextures(Tensor Algebra)
5
System DiagramSystem Diagram
Image Acquisition, Pre-processing
&Organization
D
D
• This leads to a multilinear BTF learning method • This leads to a multilinear BTF learning method
TensorTextures: 3-Mode Data TensorTensorTextures: 3-Mode Data Tensor
Texe
ls
ImagesIlluminations
Views
D
Texe
ls
Illuminations
Views
TensorTexture: 3-Mode Data TensorTensorTexture: 3-Mode Data Tensor System DiagramSystem Diagram
Image Acquisition, Pre-processing
&Organization
Tensor Decomposition &
Dimensionality Reduction Uviews
Uillums
T
D
D
T
Background on Tensor DecompositionBackground on Tensor Decomposition• Factor Analysis:
– Psychometrics, Econometrics, Chemometrics,…
• SVD:– [Beltrani, 1873] (Giornalle di Matematiche 11) “Sulle funzioni bilineari“
– [Eckart and Young, 1936] (Psychometrika)“The approximation of one matrix by another of lower rank“
• 3-Way Factor Analysis:– [Tucker,1966] (Psychometrika)
“Some mathematical notes on three mode factor analysis“
– [Kroonenberg and De Leeuw, 1980] – 3-mode ALS
• N-Way Factor Analysis:– [Kapteyn, Neudecker, and Wansbeek, 1986] – N-way ALS factor analysis– [Franc, 1992] – tensor algebra– [Denis & Dhorne, 1989]– [de Lathauwer, 1997] 2UU SD x1x 2 1 =
• A matrix has a column and row space
• SVD orthogonalizes these spaces and decomposes
• Rewrite in terms of mode-n products
Matrix Decomposition - SVDMatrix Decomposition - SVD
2xI1IIR∈D
T2SUUD 1=
Tex
els
ImagesD
( contains the “eigentextures” )1U
D
6
Tensor DecompositionTensor Decomposition• D is a n-dimensional matrix, comprising N-spaces
• N-mode SVD is the natural generalization of SVD
• N-mode SVD orthogonalizes these spaces & decomposes
D as the mode-n product of N-orthogonal spaces
• Z core tensor; governs interaction between mode matrices
• mode-n matrix, is the column space of nU)(nD
N3 21 N432 1 UUUU ... ×××××= ΖD
T2SUUD 1= 21 2 1 UUS D ××=
viewsxillumsxtexelsx 321 UUU ZD =
D
Texe
ls
Illuminations
Views
TensorTexture DecompositionTensorTexture Decomposition
Tensor DecompositionTensor Decomposition
D viewsillums.texels 32 1 UUU ×××=Z
323
32121
,.,1,
3
1
2
1
1
1rviewsrillumsrtexels
rrrr
rruuu oo∑∑∑=
===
RRR
σ
( ) ( ) ( )ZD vectexelsillumsviewsvec UUU ⊗⊗=
U1
U2
U3
D Z
N-Mode SVD AlgorithmN-Mode SVD Algorithm
1. For n=1,…, N, compute matrix by computing the SVD of the flattened matrix and setting to be the left matrix of the SVD.
2. Solve for the core tensor as follows
)(nDnU
nU
TNN
TT UUU x xx L2211 DZ =
Computing UviewsComputing Uviews
• D(views) - flatten D along the view point dimension
• Uviews – orthogonalize the column space of D(views)
Illuminations
Texe
ls
Views
ImagesD(views)
• D(illums) - flatten D along the illumination dimension
• Uillums – orthogonalizes the column space of D(illums)
Texe
ls
Illuminations
Views
Images
Illums
D(illums)
Computing UillumsComputing Uillums
7
Texe
ls
ImagesIlluminations
Views
D(texels)
• D(texels) - flatten D along the pixel dimension
• Utexels – orthogonal column space of D(texels)
– eigenimages
Computing UtexelsComputing Utexels N-Mode SVD AlgorithmN-Mode SVD Algorithm
1. For n=1,…,N, compute matrix by computing the SVD of the flattened matrix and setting to be the left matrix of the SVD.
2. Solve for the core tensor as follows
)(nDnU
nU
TNN
TT UUU x xx L2211 DZ =
• Mode-n product is a generalization of the product of two matrices• It is the product of a tensor with a matrix
• Mode-n product of and
( ) ∑ +−=
+−ni
ninjNinininiiiijiin maNnnn
............A
111111
Mx
Nn IIIIR x...xx...x1∈A
I1
I2
I3
I2
J2
J2
I2
J2
I2
I1
I2
I3
= x2x1
Nnnn IIJIIIR x..xxx.x...x 111 +−∈B
nn IJIR x=M
Mnx AB =
Mode-N ProductMode-N Product
I1
J1
AB M x3
J3
I3
I3
J3
Mode-N ProductMode-N Product
N-th order tensor NIII L××ℜ∈ 21A
nn IJ ×ℜ∈Mmatrix (2-nd order tensor)
mode-n product:
Mn×= AB )()( nn AMB =where
TensorTextures:TensorTextures: T = Z x3Upixels
TensorTextures:explicitly representcovariance acrossfactors
Variation in Illuminations
Varia
tion in
View
ing D
irection
Texe
ls
TensorTextures:TensorTextures: T = Z x3Upixels
TensorTextures:explicitly representcovariance acrossfactors
Variation in Illuminations
Variat
ion
in V
iewi
ng D
irec
tion
8
TensorTextures vs. PCATensorTextures vs. PCA
• Multilinear Analysis / TensorTextures:
viewsxillums.xtexelsx 321 UUU ZD =
(texels) Z illums.U⊗( )TviewsU=321
matrix data(texels)D
43421matrix basis
texelsU
• Linear Analysis :
• TensorTextures subsumes PCA / Eigentextures
44444 344444 21matrix tcoefficien
Strategic Dimensionality ReductionStrategic Dimensionality Reduction
Strategic Dimensionality Reduction
37
111
TensorTexturesPCA - Eigentextures#basis
System DiagramSystem Diagram
Image Acquisition, Pre-processing
&Organization
Tensor Decomposition &
Dimensionality Reduction RenderingAlgorithm
Uviews
Uillums
T
D
D
T
?Viewpoint
Illumination
Geometry
Computing vnew : Homogeneous Barycentric BlendComputing vnew : Homogeneous Barycentric Blend
Vi
Vj
Vk
Vnew
kkjjiinew ttt ∆∆∆ ++= VVVV
?
jt
it
kt
ViVi
VjVj
VkVk
1=∆∆∆ ++ kji ttt ( ) ( )( )( ) ( )( )ikij
newknewjit VVVV
VVVV−×−
−×−=∆
9
Synthesis Algorithm / Texture RepresentationSynthesis Algorithm / Texture Representation
Tnew
Tnew vld 32 ××= T
viewpoint
illumination
Rendered Texture for a Planar SurfaceRendered Texture for a Planar Surface
Rendered Textures for CylinderRendered Textures for Cylinder Rendering on Arbitrary GeometryRendering on Arbitrary GeometryBonn natural BTF datasets
=
view
illum.te
xels T
vTj l
ld
j
vT1 1
Tj
T1
TensorTextures renderings
Treasure Chest
VideoVideo
Scarecrows’ Quarterly
Flintstones Bird
