Lecture 24: Segmentation
description
Transcript of Lecture 24: Segmentation
![Page 1: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/1.jpg)
Lecture 24: Segmentation
CS6670: Computer VisionNoah Snavely
From Sandlot Science
![Page 2: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/2.jpg)
Announcements
• Final project presentations– Wednesday, December 16th, 2-4:45pm, Upson 315– Volunteers to present on Tuesday the 15th?
• Final quiz this Thursday
![Page 3: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/3.jpg)
Deblurring Application: Hubble Space Telescope
• Launched with flawed mirror
• Initially used deconvolution to correct images before corrective optics installed Image of star
![Page 4: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/4.jpg)
Fast Separation of Direct and Global
Images
Using High Frequency IlluminationShree K. Nayar
Gurunandan G. Krishnan
Columbia University
SIGGRAPH 2006
Michael D. Grossberg
City College of New York
Ramesh Raskar
MERL
![Page 5: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/5.jpg)
source
surface
P
Direct and Global Illumination
A
A : Direct
B
B : Interrelection
C
C : Subsurface
D
participating medium
D : Volumetric translucent surface
E
E : Diffusion
camera
![Page 6: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/6.jpg)
],[],[],[ icLicLicL gd
direct globalradiance
Direct and Global Components: Interreflections
surface
i
camera
source
P
g jiLjiAicL ],[],[],[
j
BRDF and geometry
![Page 7: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/7.jpg)
High Frequency Illumination Pattern
surface
camera
source
fraction of activated source elements
],[],[],[ icLicLicL gd +
i
![Page 8: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/8.jpg)
High Frequency Illumination Pattern
surface
fraction of activated source elements
camera
source
],[],[],[ icLicLicL gd + ],[],[ icLicL g )1( -
i
![Page 9: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/9.jpg)
:2
1 min2LLg
Separation from Two Images
direct global
,minmax LLLd
![Page 10: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/10.jpg)
Other Global Effects: Subsurface Scattering
translucent surface
camera
source
i
j
![Page 11: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/11.jpg)
Other Global Effects: Volumetric Scattering
surface
camera
source
participating medium
i
j
![Page 12: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/12.jpg)
Diffuse Interreflections
SpecularInterreflections
Volumetric Scattering Subsurface
Scattering
Diffusion
![Page 13: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/13.jpg)
Scene
![Page 14: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/14.jpg)
Scene
Direct Global
![Page 15: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/15.jpg)
Real World Examples:
![Page 16: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/16.jpg)
Eggs: Diffuse Interreflections
Direct Global
![Page 17: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/17.jpg)
Wooden Blocks: Specular Interreflections
Direct Global
![Page 18: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/18.jpg)
Kitchen Sink: Volumetric Scattering
Volumetric Scattering:
Chandrasekar 50, Ishimaru 78
Direct Global
![Page 19: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/19.jpg)
Peppers: Subsurface Scattering
Direct Global
![Page 20: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/20.jpg)
Hand
Direct Global
Skin: Hanrahan and Krueger
93,
Uchida 96, Haro 01, Jensen et
al. 01,
Cula and Dana 02, Igarashi et
al. 05, Weyrich et al. 05
![Page 21: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/21.jpg)
Face: Without and With Makeup
GlobalDirect
GlobalDirectWithout Makeup
With Makeup
![Page 22: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/22.jpg)
Blonde Hair
Hair Scattering: Stamm et al. 77,
Bustard and Smith 91, Lu et al. 00
Marschner et al. 03
Direct Global
![Page 23: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/23.jpg)
Photometric Stereo using Direct Images
Bowl
Shape
Source 1 Source 2 Source 3
Direct
Global
Nayar et al., 1991
![Page 24: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/24.jpg)
www.cs.columbia.edu/CAVE
![Page 25: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/25.jpg)
Questions?
![Page 26: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/26.jpg)
From images to objects
What defines an object?• Subjective problem, but has been well-studied• Gestalt Laws seek to formalize this
– proximity, similarity, continuation, closure, common fate
– see notes by Steve Joordens, U. Toronto
![Page 27: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/27.jpg)
Extracting objects
How could we do this automatically (or at least semi-automatically)?
![Page 28: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/28.jpg)
The Gestalt school• Grouping is key to visual perception• Elements in a collection can have properties that result
from relationships • “The whole is greater than the sum of its parts”
subjective contours occlusion
familiar configuration
http://en.wikipedia.org/wiki/Gestalt_psychology Slide from S.Lazebnik
![Page 29: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/29.jpg)
The ultimate Gestalt?
Slide from S.Lazebnik
![Page 30: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/30.jpg)
Gestalt factors
• These factors make intuitive sense, but are very difficult to translate into algorithms
Slide from S.Lazebnik
![Page 31: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/31.jpg)
Semi-automatic binary segmentation
![Page 32: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/32.jpg)
Intelligent Scissors (demo)
![Page 33: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/33.jpg)
Intelligent Scissors [Mortensen 95]• Approach answers a basic question
– Q: how to find a path from seed to mouse that follows object boundary as closely as possible?
![Page 34: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/34.jpg)
GrabCutGrabcut [Rother et al., SIGGRAPH 2004]
![Page 35: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/35.jpg)
Is user-input required?Our visual system is proof that automatic methods are
possible• classical image segmentation methods are automatic
Argument for user-directed methods?• only user knows desired scale/object of interest
![Page 36: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/36.jpg)
q
Automatic graph cut [Shi & Malik]
Fully-connected graph• node for every pixel• link between every pair of pixels, p,q
• cost cpq for each link
– cpq measures similarity
» similarity is inversely proportional to difference in color and position
p
Cpqc
![Page 37: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/37.jpg)
Segmentation by Graph Cuts
Break Graph into Segments• Delete links that cross between segments• Easiest to break links that have low cost (similarity)
– similar pixels should be in the same segments
– dissimilar pixels should be in different segments
w
A B C
![Page 38: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/38.jpg)
Cuts in a graph
Link Cut• set of links whose removal makes a graph disconnected• cost of a cut:
A B
Find minimum cut• gives you a segmentation
![Page 39: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/39.jpg)
But min cut is not always the best cut...
![Page 40: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/40.jpg)
Cuts in a graph
A B
Normalized Cut• a cut penalizes large segments• fix by normalizing for size of segments
• volume(A) = sum of costs of all edges that touch A
![Page 41: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/41.jpg)
Interpretation as a Dynamical System
Treat the links as springs and shake the system• elasticity proportional to cost• vibration “modes” correspond to segments
– can compute these by solving an eigenvector problem– http://www.cis.upenn.edu/~jshi/papers/pami_ncut.pdf
![Page 42: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/42.jpg)
Interpretation as a Dynamical System
Treat the links as springs and shake the system• elasticity proportional to cost• vibration “modes” correspond to segments
– can compute these by solving an eigenvector problem– http://www.cis.upenn.edu/~jshi/papers/pami_ncut.pdf
![Page 43: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/43.jpg)
Color Image Segmentation
![Page 44: Lecture 24: Segmentation](https://reader035.fdocuments.net/reader035/viewer/2022062723/56814023550346895dab82a8/html5/thumbnails/44.jpg)
Extension to Soft Segmentation• Each pixel is convex combination of segments.
Levin et al. 2006- compute mattes by solving eigenvector problem