Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1...
Transcript of Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1...
![Page 1: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/1.jpg)
1
Graphcut TexturesImage and Video Synthesis Using
Graph Cuts
Vivek KwatraArno Schödl
Irfan EssaGreg Turk
Aaron Bobick
Presented by Benjamin Sigg
![Page 2: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/2.jpg)
2
Topics
• Whats the goal?• Graph Cut• Patch placement• Extensions, Video• Authors conclusions• My conclusions
![Page 3: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/3.jpg)
3
Creating Textures
• Input is a small image• Output is a large image
InputOutput
![Page 4: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/4.jpg)
4
Idea behind
• Take a patch from the input• Place the patch somewhere on the output
such that no seams are visible
InputOutput
![Page 5: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/5.jpg)
5
Idea behind
• Iterate over– Step 1: Patch matching & placement– Step 2: Graph cut
1
2
![Page 6: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/6.jpg)
6
Graph Cut
• Cut two images, such that they can be seamless merged
• Minimize a costfunction for cut and merge
![Page 7: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/7.jpg)
7
Graph Cut
• A cut is more than a line
+ =
![Page 8: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/8.jpg)
8
Graph Cut
• Input– One graph– One costfunction
• Output– The cut with minimal costs
![Page 9: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/9.jpg)
9
Graph Cut: Input
• Two images seen as graphs
![Page 10: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/10.jpg)
10
Graph Cut: Input
• But only one graph required• Use an “embracing” graph
![Page 11: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/11.jpg)
11
Graph Cut: Input
• Costfunction
M A , B , s , tA s B s A t B t
AB
s t
![Page 12: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/12.jpg)
12
Graph Cut - Input
M A , B , s , tA s B s A t B t
AB
s t
AB
s t
AB
s t
![Page 13: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/13.jpg)
13
Graph Cut - Output
• The minimal cut of the graph
![Page 14: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/14.jpg)
14
Graph Cut - Output
• Merge the two images along the cut• (Hopefully) no visible seams
![Page 15: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/15.jpg)
15
Graph Cut: Example
• Two merged images• No seams are visible
![Page 16: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/16.jpg)
16
Graph Cut
• Algorithms are well known• Many possible Costfunctions• What about many images?
![Page 17: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/17.jpg)
17
Many Images
• Merge first two images• Merge resulting image with a third
image...
+ =+ +
![Page 18: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/18.jpg)
18
Many Images
• Merge with a third image• Result is very bad• Don't throw away information!
+ =
![Page 19: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/19.jpg)
19
Many Images
• Store old seam costs
![Page 20: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/20.jpg)
20
Many Images
• Add the old seam costs to the new cut
old cut new cut
cost costnew costold
![Page 21: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/21.jpg)
21
Patch Placement
• Place input somewhere on the output
• Cut out a patch
• Random placement• Entire patch matching• Sub-patch matching
t
![Page 22: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/22.jpg)
22
Random placement
• Place the input randomatically on the output
• Graph cut decides, which part of the input will be seen
• The fastest approach• Results are good for random textures
![Page 23: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/23.jpg)
23
Entire patch matching
• Place the input such that it matches the output best
• Costfunction for Placement (SSD):
• Results are good for (semi-) structured textures
C t 1At p At
I p O p t 2I : InputO : OutputA: Intersection I Op : Pointt : Translation
![Page 24: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/24.jpg)
24
Sub patch matching
• Choose a small patch in the output• Search a patch in the input that fits well• Same Costfunction as “entire patch
matching”
• Most general technique• Good for videos
![Page 25: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/25.jpg)
25
Results
![Page 26: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/26.jpg)
26
Results
• Input, Output and Error
![Page 27: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/27.jpg)
27
Results
• Not every image is a good input
![Page 28: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/28.jpg)
28
Extensions
• Other costfunctions• Blending techniques to omit seams• FFT for patch placing, dramatically fastens
the algorithm• Use not only translations, use rotations,
mirroring and scaling.
![Page 29: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/29.jpg)
29
Extensions
• Perspective by scaling the input• Additional Constraints for scaling
![Page 30: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/30.jpg)
30
Extensions
• Interactive Merging and Blending
1-2 hours of work
![Page 31: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/31.jpg)
31
Video
• Can be used to generate infinite videos• Just see a video as 3-dimensional graph
with axis: x, y, t.• Loop the video
![Page 32: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/32.jpg)
32
Video Transition
• Search two good matching frames• Make the cut through this two frames• Cut is a plane
![Page 33: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/33.jpg)
33
Random Temporal offset
• For small input• No periodicity• Requires continous part in time
![Page 34: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/34.jpg)
34
Spatio-Temporally Stationary Videos
• For videos with a one-directional movement (e.x. Smoke)
• Pixels moved in time and space
![Page 35: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/35.jpg)
35
Temporal Constraints
• Moving pixels in time and place causes problems, continuity is lost
• First and last n frames fixed by input• Last n frames removed when output
created
![Page 36: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/36.jpg)
36
Authors conclusions
• Algorithm is fast• No restrictions for the shape• Method for adding constraints• Easely generalized
![Page 37: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/37.jpg)
37
My conclusions
• Very flexible technique:– User interaction is possible– Different costfunctions applicable– Different pach placing/matching algorithms
available• Good for random images• Still problems with videos
![Page 38: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno](https://reader031.fdocuments.net/reader031/viewer/2022022021/5b9f3d7909d3f2ab0b8d1ee0/html5/thumbnails/38.jpg)
38
The End