Introduction and course overview - Computer...
Transcript of Introduction and course overview - Computer...
![Page 1: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/1.jpg)
Image compositing
15-463, 15-663, 15-862Computational Photography
Fall 2018, Lecture 8http://graphics.cs.cmu.edu/courses/15-463
![Page 2: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/2.jpg)
Course announcements
• If you haven’t started Homework 2 yet, you should do that ASAP.
• Today we will post more details about the final project.
• Guest lecture on Monday: Ravi Teja Mullapudi, “High-performance image processing.”
• RI Seminar on Friday: Anat Levin, “Light-sensitive displays.”
![Page 3: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/3.jpg)
Overview of today’s lecture
• Leftover from color lecture.
• Some motivational examples.
• Cut-and-paste.
• Alpha (linear) blending.
• Multi-band blending.
• Seam stitching.
![Page 4: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/4.jpg)
Slide credits
Most of these slides were adapted from:
• Kris Kitani (15-463, Fall 2016).
Some slides were inspired or taken from:
• Fredo Durand (MIT).• James Hays (Georgia Tech).
![Page 5: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/5.jpg)
Some motivational examples
![Page 6: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/6.jpg)
Gangster, Frankie Yale, killed by a drive-by in Brooklyn in 1928.
![Page 7: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/7.jpg)
A tragic photo from 1959 after three-year-old Martha Cartagena was killed while riding her tricycle in Brooklyn
![Page 8: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/8.jpg)
In 1958 there was a fatal fire at the Elkins Paper & Twine Co. on Wooster Street in SoHo. The building burned to the ground.
![Page 9: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/9.jpg)
Berlin 1945/2010
![Page 10: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/10.jpg)
Berlin 65 years later
![Page 11: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/11.jpg)
Berlin, 1945/2010, Mehringdamm
![Page 12: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/12.jpg)
Forrest Gump (1994)
![Page 13: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/13.jpg)
• Cut-and-paste.
• Alpha (linear) blending.
• Multi-band blending.
• Poisson blending.
• Seam stitching.
Techniques for compositing
![Page 14: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/14.jpg)
Cut-and-paste
![Page 15: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/15.jpg)
1. Extract Sprites (e.g., using Intelligent Scissors in Photoshop)
2. Blend them into the composite (in the right order)
Cut and paste procedure
You may have also heard it as collaging
![Page 16: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/16.jpg)
Sometimes it produces visually
compelling results.
Cut and paste
![Page 17: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/17.jpg)
What is wrong with this composite?
Cut and paste
Other times, not so much.
![Page 18: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/18.jpg)
Alpha (linear) blending
![Page 19: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/19.jpg)
output = foreground * mask + background * (1-mask)
mask outputforeground
background
Alpha blending
a.k.a. alpha matte or alpha composite
![Page 20: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/20.jpg)
Does this look unnatural?
Binary alpha mask
![Page 21: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/21.jpg)
Does this look unnatural?
Binary alpha mask
How can we fix it?
![Page 22: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/22.jpg)
feathering (smoothed alpha)
How would you implement feathering?
binary alpha mask
Non-binary alpha mask
![Page 23: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/23.jpg)
How would you create a binary alpha mask for these two images?
Setting the alpha mask: center seam
![Page 24: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/24.jpg)
Setting the alpha mask: center seam
Step 1: Compute their distance transform (bwdist)
![Page 25: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/25.jpg)
alpha = logical(dtrans1>dtrans2)
Setting the alpha mask: center seam
Step 2: set mask
![Page 26: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/26.jpg)
Setting the alpha mask: center seam
Anything wrong with this alpha matte?
![Page 27: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/27.jpg)
alpha = blur(alpha)
Setting the alpha mask: center seam
Step 3: blur the mask
![Page 28: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/28.jpg)
Setting the alpha mask: center seam
Still doesn’t look terribly good
![Page 29: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/29.jpg)
alpha = 0.5 in overlap region
Setting the alpha mask: center seam
Step 4: go beyond blurring for non-binary
![Page 30: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/30.jpg)
Setting the alpha mask: center seam
Still not OK
![Page 31: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/31.jpg)
alpha = dtrans1 / (dtrans1+dtrans2)
Setting the alpha mask: center seam
Step 5: more elaborate non-binary
![Page 32: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/32.jpg)
Setting the alpha mask: center seam
Looks better but some dangers remain.
![Page 33: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/33.jpg)
Let’s blend these two images...
left side right side
What kind of mask would you use?
Another blending example
![Page 34: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/34.jpg)
Another blending example
How would you select this window?
![Page 35: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/35.jpg)
Effects of different windows
Bad window: non-smooth seam.
Bad windows: ghosting.Good window: smooth
but no ghosting.
![Page 36: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/36.jpg)
To avoid discontinuities:window = size of largest
prominent feature
To avoid ghosting:window <= 2*size of
smallest prominent feature
What is a good window size?
![Page 37: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/37.jpg)
Fourier domain interpretation:
To avoid discontinuities:window = size of largest
prominent feature
To avoid ghosting:window <= 2*size of
smallest prominent feature
linear blending should work when:image frequency content occupies
roughly one “octave” (power of two)
linear blending should work when:largest frequency <= 2*size of smallest
frequency
What is a good window size?
What if the frequency spread is too wide?
![Page 38: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/38.jpg)
Fourier domain interpretation:
To avoid discontinuities:window = size of largest
prominent feature
To avoid ghosting:window <= 2*size of
smallest prominent feature
linear blending should work when:image frequency content occupies
roughly one “octave” (power of two)
linear blending should work when:largest frequency <= 2*size of smallest
frequency
What is a good window size?
Most natural images have a very wide frequency spread. What do we do then?
![Page 39: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/39.jpg)
Multi-band blending
![Page 40: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/40.jpg)
Time to use pyramids again
left image right imagealpha mask
At low frequencies, blend slowly to avoid seamsAt high frequencies, blend quickly to avoid ghosts
?
?
?
Which mask goes where?
![Page 41: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/41.jpg)
Time to use pyramids again
left image right imagealpha mask
At low frequencies, blend slowly to avoid seamsAt high frequencies, blend quickly to avoid ghosts
![Page 42: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/42.jpg)
Remember our two types of pyramids
Gaussian pyramid Laplacian pyramid
![Page 43: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/43.jpg)
1. Build Laplacian pyramids for each image
image 1 at level i
3. Collapse the pyramid to get the final blended image
region mask at level i
How large should the blending region be at each level?
Remember our two types of pyramids
2. Blend each level of pyramid using region mask
image 2at level i
![Page 44: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/44.jpg)
1. Build Laplacian pyramids for each image
image 1 at level i
3. Collapse the pyramid to get the final blended image
region mask at level i
How large should the blending region be at each level?
Remember our two types of pyramids
2. Blend each level of pyramid using region mask
image 2at level i
About the size of that level’s blur
![Page 45: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/45.jpg)
Multi-band blending using the Laplacian pyramid
Laplacian level 0
Laplacian level 2
Laplacian level 4
left pyramid right pyramid blended pyramid
![Page 46: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/46.jpg)
A famous result (for its time)
![Page 47: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/47.jpg)
A famous result (for its time)
![Page 48: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/48.jpg)
A creepier result
Can we get the same result with less computation?
![Page 49: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/49.jpg)
Two-band blending
left image right imagealpha mask
do not use
Only use two bands: high frequency and low frequency• Blend low frequency with smooth alpha• Blend high frequency with binary alpha
![Page 50: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/50.jpg)
Example: blending panoramas
original collage
blended collage
![Page 51: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/51.jpg)
Example: blending panoramas
low frequency
blend
high frequency
blend
![Page 52: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/52.jpg)
Linear blending
![Page 53: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/53.jpg)
Two-band blending
![Page 54: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/54.jpg)
One more comparison
copy-paste linear two-band
![Page 55: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/55.jpg)
Why do these images look weirdly cropped?
They were warped using homographies before being aligned.
Homework 6: autostitching
![Page 56: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/56.jpg)
Seam stitching
![Page 57: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/57.jpg)
1. Extract Sprites
Two different ways to think about the same thing:
• Finding seams (i.e., finding the pixels where to cut an image)
• Segmentation (i.e., splitting the image into “foreground” and “background”)
I will be using the two terms interchangeable
Cut and paste procedure
How do we do this?
![Page 58: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/58.jpg)
When blending multiple images of the same scene, moving objects become ghosts!
Question about blending
What can we do instead of blending?
![Page 59: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/59.jpg)
Question about blending
Instead of blending the images, cut them and stitch them together!
When blending multiple images of the same scene, moving objects become ghosts!
![Page 60: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/60.jpg)
Image as a graph
![Page 61: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/61.jpg)
Nodes: pixels
Edges: Constraints between neighboring pixels
Fundamental theme of today’s lecture
Images can be viewed as graphs
![Page 62: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/62.jpg)
Graph-view of segmentation problem
Segmentation is node-labeling
Given pixel values and neighborhoods, decide:• which nodes to label as
foreground/backgroundor• which nodes to label as seamsusing graph algorithms
Nodes: pixels
Edges: Constraints between neighboring pixels
![Page 63: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/63.jpg)
Method Labeling problem Algorithm Intuition
Intelligent scissors
label pixels as seamsDijkstra’s shortest
path (dynamic programming)
short path is a good boundary
GrabCutlabel pixels as
foreground/backgroundmax-flow/min-cut
(graph cutting)good region has low cutting cost
Today we will cover:
Graph-view of segmentation problem
![Page 64: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/64.jpg)
Shortest graph paths and intelligent scissors
![Page 65: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/65.jpg)
Mortenson and Barrett (SIGGRAPH 1995)(you can tell it’s old from the paper’s low quality teaser figure)
Intelligent scissors
Problem statement:Given two seed points, find a good boundary connecting them
Challenges:• Make this real-time for interaction• Define what makes a good boundary
![Page 66: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/66.jpg)
Nodes: pixels
Edges: Constraints between neighboring pixels
Graph-view of this problem
Images can be viewed as graphs
![Page 67: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/67.jpg)
Graph-view of this problem
Graph-view of intelligent scissors:
1 2 1
41
6
9
13
1
4
113
2
3
5
1. Assign weights (costs) to edges
![Page 68: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/68.jpg)
Graph-view of this problem
Graph-view of intelligent scissors:
1 2 1
41
6
9
13
1
4
113
2
3
5
End
Start1. Assign weights (costs) to edges
2. Select the seed nodes
![Page 69: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/69.jpg)
Graph-view of this problem
Graph-view of intelligent scissors:
1 2 1
41
6
9
13
1
4
113
2
3
5
End
Start1. Assign weights (costs) to edges
2. Select the seed nodes
3. Find shortest path between them
![Page 70: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/70.jpg)
Graph-view of this problem
Graph-view of intelligent scissors:
1 2 1
41
6
9
13
1
4
113
2
3
5
End
Start1. Assign weights (costs) to edges
2. Select the seed nodes
3. Find shortest path between them
What algorithm can we use to find the shortest path?
![Page 71: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/71.jpg)
Graph-view of this problem
Graph-view of intelligent scissors:
1 2 1
41
6
9
13
1
4
113
2
3
5
End
Start1. Assign weights (costs) to edges
2. Select the seed nodes
3. Find shortest path between them
What algorithm can we use to find the shortest path?• Dijkstra’s algorithm (dynamic
programming)
![Page 72: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/72.jpg)
Initialize, given seed s (pixel ID):
• cost(s) = 0 % total cost from seed to this point
• cost(!s) = big
• A = {all pixels} % set to be expanded
• prev(s)=undefined % pointer to pixel that leads to q=s
Precompute cost2(q, r) % cost between q to neighboring pixel r
Loop while A is not empty
1.q = pixel in A with lowest cost
2.Remove q from A
3.For each pixel r in neighborhood of q that is in A
a)cost_tmp = cost(q) + cost2(q,r) %this updates the costs
b)if (cost_tmp < cost(r))
i.cost(r) = cost_tmp
ii. prev(r) = q
Dijkstra’s shortest path algorithm
![Page 73: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/73.jpg)
Graph-view of this problem
Graph-view of intelligent scissors:
1 2 1
41
6
9
13
1
4
113
2
3
5
End
Start1. Assign weights (costs) to edges
2. Select the seed nodes
3. Find shortest path between them
What algorithm can we use to find the shortest path?• Dijkstra’s algorithm (dynamic
programming)
How should we select the edge weights to get good boundaries?
![Page 74: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/74.jpg)
Selecting edge weights
Define boundary cost between neighboring pixels:
1. Lower if an image edge is present (e.g., as found by Sobel filtering).
2. Lower if the gradient magnitude at that point is strong.
3. Lower if gradient is similar in boundary direction.
![Page 75: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/75.jpg)
Gradient magnitude
Edge image
Pixel-wise cost
75
Selecting edge weights
![Page 76: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/76.jpg)
1.Use cursor as the “end” seed, and always connect start seed to that
2.Every time the user clicks, use that point as a new starting seed and repeat
Making it more interactive
![Page 77: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/77.jpg)
htt
p:/
/yo
utu
.be/
X_d
Z_7
xAcI
M
![Page 78: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/78.jpg)
Examples
![Page 79: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/79.jpg)
Seam collaging
Another use for image seam selection
Kwatra et al., Graphcut Textures: Image and Video Synthesis using Graph Cuts, SIGGRAPH 2003
![Page 80: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/80.jpg)
Graph-view of this problem
Graph-view of image collaging:
1 2 1
41
6
9
13
1
4
113
2
3
5
End
Start1. Assign weights (costs) to edges
2. Select the seed nodes
3. Find shortest path between them
What edge weights would you use for collaging?
![Page 81: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/81.jpg)
Good places to cut:• similar color in both images• high gradient in both images
Selecting edge weights for seam collaging
![Page 82: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/82.jpg)
Avidan and Shamir, Seam Carving for Content-Aware Image Resizing, SIGGRAPH 2007
Another use for image seam selection
Seam carving
![Page 83: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/83.jpg)
![Page 84: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/84.jpg)
Another use for image seam selection: • instead of blending the images, cut them and stitch them together
Seam stitching
![Page 85: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/85.jpg)
alpha blending
AutoStitch
Seam stitching
![Page 86: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/86.jpg)
Where will intelligent scissors work well, or have problems?
Examples
![Page 87: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/87.jpg)
Some notes about cutting-and-pasting
![Page 88: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/88.jpg)
Real or composite, and why?
![Page 89: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/89.jpg)
Real: Lahaina noon (or noon at subsolar point)
![Page 90: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/90.jpg)
Real or composite, and why?
![Page 91: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/91.jpg)
Composite: Inconsistent shadows
![Page 92: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/92.jpg)
Composite: Inconsistent shadows
Kee et al., “Exposing Photo Manipulation with Inconsistent Shadows,” ToG 2014
![Page 93: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/93.jpg)
Photorealistic compositing
Karsch et al., “Rendering Synthetic Objects into Legacy Photographs,” SIGGRAPH Asia 2011
![Page 94: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/94.jpg)
Photorealistic compositing
Karsch et al., “Rendering Synthetic Objects into Legacy Photographs,” SIGGRAPH Asia 2011
![Page 95: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/95.jpg)
Photorealistic compositing
Karsch et al., “Rendering Synthetic Objects into Legacy Photographs,” SIGGRAPH Asia 2011
![Page 96: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/96.jpg)
Photorealistic compositing
Karsch et al., “Rendering Synthetic Objects into Legacy Photographs,” SIGGRAPH Asia 2011
![Page 97: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/97.jpg)
Photorealistic compositing
Karsch et al., “Rendering Synthetic Objects into Legacy Photographs,” SIGGRAPH Asia 2011
![Page 98: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/98.jpg)
Photorealistic compositing
Karsch et al., “Rendering Synthetic Objects into Legacy Photographs,” SIGGRAPH Asia 2011
![Page 99: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/99.jpg)
Photorealistic compositing
How would you do this?
![Page 100: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/100.jpg)
References
Basic reading:• Szeliski textbook, Sections 3.13, 3.5.5, 9.3.4, 10.4.3.
Additional reading:• Pérez et al., “Poisson Image Editing,” SIGGRAPH 2003.
the original Poisson blending paper.• Georgiev, “Covariant Derivatives and Vision,” ECCV 2006.
a paper from Adobe describing the version of Poisson blending implemented in Photoshop’s “healing brush”.
• Elder and Goldberg, “Image editing in the contour domain”, PAMI 2001.• Bhat et al., “GradientShop: A Gradient-Domain Optimization Framework for Image and Video Filtering,” ToG
2010.• Agrawal and Raskar, “Gradient Domain Manipulation Techniques in Vision and Graphics,” ICCV 2007 course,
http://www.amitkagrawal.com/ICCV2007Course/the above references provide an overview of gradient-domain processing as a general image processing paradigm, which can be used for a broad set of applications beyond blending, including tone-mapping, colorization, converting to grayscale, edge enhancement, image abstraction and non-photorealistic rendering.
![Page 101: Introduction and course overview - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2018_fall/lectures/... · 2019-04-22 · Blend each level of pyramid using region mask image](https://reader033.fdocuments.net/reader033/viewer/2022042018/5e767a1dd9f54204c24d13ba/html5/thumbnails/101.jpg)
References
Basic reading:• Szeliski textbook, Sections 3.13, 3.5.5, 5.1.3, 5.3.1, 9.3.2, 9.3.3, 9.3.4, 10.4.3.• Mortensen and Barrett, “Intelligent scissors for image composition,” SIGGRAPH 1995.
The intelligent scissors paper.• Kwatra et al., Graphcut Textures: Image and Video Synthesis using Graph Cuts, SIGGRAPH 2003.
The seam collaging paper.• Avidan and Shamir, “Seam Carving for Content-aware Image Resizing,” SIGGRAPH 2007.
The seam carving paper.
Additional reading:• Felzenszwalb and Zabih, “Dynamic Programming and Graph Algorithms in Computer Vision,” PAMI 2010.
A great review of graph-based techniques, including shortest path and graph-cut, in computer vision.• Kee et al., “Exposing photo manipulation with inconsistent shadows,” ToG 2013.
The paper demonstrating how image forgeries can be detecting by reasoning about the physical accuracy of shadows in an image.
• Karsch et al., “Rendering synthetic objects into legacy photographs”, SIGGRAPH 2011.The paper where the photorealistic compositing examples came from.