Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.
-
Upload
francine-berry -
Category
Documents
-
view
222 -
download
1
Transcript of Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.
![Page 1: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/1.jpg)
Texture Optimization for Example-based Synthesis
Vivek Kwatra
Irfan EssaAaron BobickNipun Kwatra
![Page 2: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/2.jpg)
Image vs. Texture
![Page 3: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/3.jpg)
Example-based Texture Synthesis
Input Example
![Page 4: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/4.jpg)
Related Work
• Local region-growing method
-Pixel-based
-Patch-based
• Global optimization-based method Heeger and Bergen sig95,Pyramid-based texture synthesis
Paget and Longstaff IEEE Tran… 98,Texture synthesis via a noncausal nonparametric multiscale markov random field
…..
• Physical Simulation et al
![Page 5: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/5.jpg)
Example-based Texture Synthesis
1. Dynamic texturing (animation/video)
2. Greater control: external knobs, e.g., flow,
shape, illumination
3. Texture quality evaluation and refinement
![Page 6: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/6.jpg)
Motivation
+
InputTexture
InputFlow Field
OutputSequence
![Page 7: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/7.jpg)
Limitations of previous work
• Pixel-based
can not maintain global structure of the texture
inconsistencies
• Patch-based
have no control over individual pixel values
• Global methods:
pixel-to-pixel interactions
insufficient to capture large scale structures
complex formulations difficult to optimize
![Page 8: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/8.jpg)
Texture Energy
• Pixel-by-pixel comparison of source and target not possible
• Compare texture “elements”– Local pixel neighborhoods
• Want each target neighborhood to be similar to some source neighborhood
![Page 9: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/9.jpg)
Texture Energy
X(target frame)
Z(source)
![Page 10: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/10.jpg)
Energy for Single Neighborhood
p (pixel)Z(source)
X(target frame)
![Page 11: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/11.jpg)
Energy for Single Neighborhood
Xp (neighborhood)Z(source)
X(target frame)
![Page 12: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/12.jpg)
Energy for Single Neighborhood
Xp (neighborhood)Z(source)
X(target frame)
![Page 13: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/13.jpg)
Energy for Single Neighborhood
(nearest neighbor) Zp
Xp (neighborhood)Z(source)
X(target frame)
![Page 14: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/14.jpg)
Energy for Single Neighborhood
X
Z
(nearest neighbor) Zp
Xp (neighborhood)
Texture Energy(single neighborhood)
2
t P P PE X X Z
![Page 15: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/15.jpg)
Texture Energy for Entire Image
X
Z
![Page 16: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/16.jpg)
Texture Energy for Entire Image
X
Z
individual neighborhood energy
2;t P P P
P X X
E X Z X Z
![Page 17: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/17.jpg)
4.Repeat iteratively until set {ZP} stops changing
3.Update X to be texture that minimizes the energy
Texture Synthesis
1.Initialize texture X
2
t P P PE X X Z
2.Find closest input neighborhood ZP corresponding to each output neighborhood XP
2;t P P P
P X X
E X Z X Z
![Page 18: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/18.jpg)
Texture Synthesis
![Page 19: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/19.jpg)
Gradient-based Energy
D is the differentiation operator and µ is a relative weighting coefficient
![Page 20: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/20.jpg)
Muti-level Synthesis
• Multi-resolution
• Multi-scale/multiple neighborhood sizes
Reasons a good initialization of the texture avoid undesirable local minima avoid inconsistencies capture large scale structures
![Page 21: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/21.jpg)
• Random Initialization
• Multiple– Resolution Levels– Neighborhood
Sizes
• Progressively refined output
Texture Energy Graph
Level 1
Level 2
Level 3
Level 4
Level 5
Level 6
Level 7
0 5 10 15 20 25 30 35 40 45 500
50
100
150
200
250
300
350
400
Number of Iterations
![Page 22: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/22.jpg)
Controllable Synthesis
2; c c
ck
E X X X k X k
; ; ct P cE X E X Z E X X
![Page 23: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/23.jpg)
• Flow consistency– Perceived motion similar to flow
• Texture similarity– Shape, size, orientation of texture
elements similar to source
Flow-guided Texture Animation
Target FlowFlowing Target
Source Texture
TextureSimilarity
FlowConsistency
![Page 24: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/24.jpg)
Naive Approach
Target Flow
• Ignore Texture Similarity
• Warp each frame via flow field
X1 Xn
TargetFrames X0
Warp
Flowing Target
FlowConsistency
Source Texture
TextureSimilarity
Source Texture
TextureSimilarity
![Page 25: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/25.jpg)
Naive Approach
Texture structure not maintained
Target FlowFlowing Target
FlowConsistency
Source Texture
TextureSimilarity
![Page 26: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/26.jpg)
Warp
Instead of…
WarpXnX0 X1
TargetFrames
![Page 27: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/27.jpg)
Warp
Instead of…
WarpXnX0 X1
TargetFrames
![Page 28: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/28.jpg)
Warp + Correct
“Correct” Operation
TargetFrames XX–
Warp
W
Correct
WarpedFrame
SynthesizedFrame
![Page 29: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/29.jpg)
Link to original goals
“Correct” Operation
XTargetFrames X–
W
Correct
Warp
![Page 30: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/30.jpg)
Link to original goals
“Correct” Operation
XTargetFrames X–
W
Correct
Flow ConsistencyWarp
![Page 31: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/31.jpg)
Link to original goals
“Correct” Operation
XTargetFrames X–
W
Correct
Source
Flow Consistency
Texture Similarity
Warp
![Page 32: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/32.jpg)
Energy = Flow Energy + Texture Energy
Optimization of Energy
XTargetFrames X–
W
Correct
Source
Flow Consistency
Texture Similarity
Warp
![Page 33: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/33.jpg)
Flow Energy
1 2 1, , , Lf f f f
2D flow field
:if frame i frame i+1
pixel (p,i) pixel (q,i+1)
2
1: 1 ,
; , , 1 ,i
c ii L p i X
E X f X p i X q i q f p
![Page 34: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/34.jpg)
Results: Texture Synthesis
![Page 35: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/35.jpg)
![Page 36: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/36.jpg)
Results: Texture Synthesis
Input Wei-Levoy[Wei’00]
Image Quilting[Efros’01]
Graph-cuts[Kwatra’03]
Texture Optimization
[Kwatra’05]
![Page 37: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/37.jpg)
Time Cost
• Static texture synthesis– 5-10 iterations per resolution level
• Flowing texture synthesis– 20-60 seconds per frame
Resolution One iteration Total time
64x64 2.5 sec 20 sec
128x128 10 sec 2 min
256x256 25 sec 8.5 min
![Page 38: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/38.jpg)
Limitations
• Optimization finds local minima
• Blurry / misaligned texture elements
• Texture scale should be compatible
with flow field
![Page 39: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/39.jpg)
Thanks a lot!
![Page 40: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/40.jpg)
Pixel-based Methods
• Compare local causal neighbourhoods Efros and Leung (ICCV ’99)
Wei and Levoy (Siggraph 2000)Ashikhmin (I3D 2001)
Input Output
![Page 41: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/41.jpg)
Patch-based Methods
• Copy patches of pixels rather than single pixels
Chaos Mosaic, Xu et al, 1997
Patch-Based Sampling, Liang et al(ACM 2001)
Image Quilting, Efros and Williams(Siggraph 2001)
![Page 42: Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra.](https://reader034.fdocuments.net/reader034/viewer/2022052603/56649ede5503460f94beeb00/html5/thumbnails/42.jpg)
Back