Contour-Based Joint Clustering of Multiple Segmentations Daniel Glasner * 1 Shiv N. Vitaladevuni * 2...
-
Upload
ambrose-preston -
Category
Documents
-
view
215 -
download
0
Transcript of Contour-Based Joint Clustering of Multiple Segmentations Daniel Glasner * 1 Shiv N. Vitaladevuni * 2...
![Page 1: Contour-Based Joint Clustering of Multiple Segmentations Daniel Glasner * 1 Shiv N. Vitaladevuni * 2 Ronen Basri 1 * equal contribution authors 1 2.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649cf95503460f949c9fb1/html5/thumbnails/1.jpg)
Contour-Based Joint Clustering of Multiple Segmentations
Daniel Glasner *1 Shiv N. Vitaladevuni *2
Ronen Basri 1
* equal contribution authors
1 2
![Page 2: Contour-Based Joint Clustering of Multiple Segmentations Daniel Glasner * 1 Shiv N. Vitaladevuni * 2 Ronen Basri 1 * equal contribution authors 1 2.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649cf95503460f949c9fb1/html5/thumbnails/2.jpg)
1
2 3
1
2 3
1
2 3
1
2 3
Joint-Clustering of Image SegmentsObjective: 1. Similar shape across frames 2. Coherent regions within frames
Segments /Super-pixels
Ourclusteringresult
Inputframe
Object contours can be matchedOversegmentation artifacts can not
![Page 3: Contour-Based Joint Clustering of Multiple Segmentations Daniel Glasner * 1 Shiv N. Vitaladevuni * 2 Ronen Basri 1 * equal contribution authors 1 2.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649cf95503460f949c9fb1/html5/thumbnails/3.jpg)
• Large deformation histograms are similar
• Single object
• Inter image similarity:• Overlap• Color
Related Work This WorkCo-segmentation
[Rother et al.06, Bagon et al.08, Hochbaum et al.09, Joulin et al.10]
Co-clustering of image segments[Vitaladevuni and Basri
2010]• Small deformation
shapes are similar• Full image segmentation
# objects unknown• Inter image similarity:
• Shape• Intra Image similarity:
• Color / optical flow etc.
![Page 4: Contour-Based Joint Clustering of Multiple Segmentations Daniel Glasner * 1 Shiv N. Vitaladevuni * 2 Ronen Basri 1 * equal contribution authors 1 2.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649cf95503460f949c9fb1/html5/thumbnails/4.jpg)
Different lighting conditions
Some ApplicationsVideo segmentation
EM slices[Vitaladevuni &Basri CVPR10]
![Page 5: Contour-Based Joint Clustering of Multiple Segmentations Daniel Glasner * 1 Shiv N. Vitaladevuni * 2 Ronen Basri 1 * equal contribution authors 1 2.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649cf95503460f949c9fb1/html5/thumbnails/5.jpg)
( )
+ Coherency + Coherency( )
F(union) = Shape-sim( , )
Shape similarityacross frames ×Coherencywithin frame ✓
✓
Searching for a Good ClusterConvex functional of unions of segments 1. Bounding contours match - across frames2. Coherent regions - within frameExterior bounding contours match
![Page 6: Contour-Based Joint Clustering of Multiple Segmentations Daniel Glasner * 1 Shiv N. Vitaladevuni * 2 Ronen Basri 1 * equal contribution authors 1 2.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649cf95503460f949c9fb1/html5/thumbnails/6.jpg)
Incorporating Shape
Input frames Shape-basedjoint clustering
IntersectionShape-basedsimilarity
![Page 7: Contour-Based Joint Clustering of Multiple Segmentations Daniel Glasner * 1 Shiv N. Vitaladevuni * 2 Ronen Basri 1 * equal contribution authors 1 2.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649cf95503460f949c9fb1/html5/thumbnails/7.jpg)
Descriptor (segment) = bounding contour
A Novel Contour Descriptor
3
dimension =
# contour samples
in image 1
Image 1
0ei(θ+)
0eiα
ei
eiθ
3
3
![Page 8: Contour-Based Joint Clustering of Multiple Segmentations Daniel Glasner * 1 Shiv N. Vitaladevuni * 2 Ronen Basri 1 * equal contribution authors 1 2.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649cf95503460f949c9fb1/html5/thumbnails/8.jpg)
Descriptor (union) = its bounding contour
Additive Descriptor & Contour Cancellation
ei
eiθ
0
ei(θ+)
union
ei
0+ =
dimension =
#contour samples
![Page 9: Contour-Based Joint Clustering of Multiple Segmentations Daniel Glasner * 1 Shiv N. Vitaladevuni * 2 Ronen Basri 1 * equal contribution authors 1 2.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649cf95503460f949c9fb1/html5/thumbnails/9.jpg)
Comparing Shapes Across Images
B2B1
0
0eiθ
# contour elements image 2
# segments image 2
# contour elements image 1
# segments image 1
Image 2Image 1
![Page 10: Contour-Based Joint Clustering of Multiple Segmentations Daniel Glasner * 1 Shiv N. Vitaladevuni * 2 Ronen Basri 1 * equal contribution authors 1 2.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649cf95503460f949c9fb1/html5/thumbnails/10.jpg)
Comparing Shapes Across Images
B2x2B1
= =
Contour descriptorin image 2
Contour descriptorin image 1
x1
Image 2Image 1
Binary indicator of segmentsin image 1
Binary indicator of segmentsin image 2
![Page 11: Contour-Based Joint Clustering of Multiple Segmentations Daniel Glasner * 1 Shiv N. Vitaladevuni * 2 Ronen Basri 1 * equal contribution authors 1 2.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649cf95503460f949c9fb1/html5/thumbnails/11.jpg)
B2x2B1
= =x1
Shape-sim( , )
= ?
Correspondence
Matrix
W1,2
B1Hx1
T
?
# contour elements image1X
# contour elements image2
Comparing Shapes Across Images
![Page 12: Contour-Based Joint Clustering of Multiple Segmentations Daniel Glasner * 1 Shiv N. Vitaladevuni * 2 Ronen Basri 1 * equal contribution authors 1 2.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649cf95503460f949c9fb1/html5/thumbnails/12.jpg)
Comparing Shapes Across Images
B2 x2
W1,2
x1T B1
H
Contour descriptor of shape in image 1
Contour descriptor of shape in image 2
Q1,2 =
# segments image1X
# segments image2
# contour elements image1X
# contour elements image2
![Page 13: Contour-Based Joint Clustering of Multiple Segmentations Daniel Glasner * 1 Shiv N. Vitaladevuni * 2 Ronen Basri 1 * equal contribution authors 1 2.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649cf95503460f949c9fb1/html5/thumbnails/13.jpg)
1. Similar shape across frames2. Coherent regions within frames
2 wkl cos(k,l external contour
k l )=
= Shape-sim( , )
xT
x
Q1,2
Q1,2H
xTQx =
For arbitrary selection of segments
x x1
x2
Q1,1
Q2,2
( )
+ Coherency + Coherency( )
![Page 14: Contour-Based Joint Clustering of Multiple Segmentations Daniel Glasner * 1 Shiv N. Vitaladevuni * 2 Ronen Basri 1 * equal contribution authors 1 2.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649cf95503460f949c9fb1/html5/thumbnails/14.jpg)
Optimization
maxX x 1 ... x c ,c
x jTQx j
j1
c
s.t. x ij {0,1}
x ij2
j =1
c
1 i
Vitaladevuni and Basri CVPR 2010
NP-hard [Garey and Johnson 1979]
![Page 15: Contour-Based Joint Clustering of Multiple Segmentations Daniel Glasner * 1 Shiv N. Vitaladevuni * 2 Ronen Basri 1 * equal contribution authors 1 2.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649cf95503460f949c9fb1/html5/thumbnails/15.jpg)
Convex Relaxation
• Efficient linear programming relaxation[Vitaladevuni & Basri 2010, Charikar et al. 2003]
• Q is hermitian objective is real
![Page 16: Contour-Based Joint Clustering of Multiple Segmentations Daniel Glasner * 1 Shiv N. Vitaladevuni * 2 Ronen Basri 1 * equal contribution authors 1 2.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649cf95503460f949c9fb1/html5/thumbnails/16.jpg)
Video Dataset for Occlusion/Object Boundary Detection
A. Stein and M. Hebert. Occlusion boundaries from motion: low-level detection and mid-level reasoning. IJCV, 82(3), 2009. 2http://www.cs.cmu.edu/~stein/occlusion_data/
referenceframe
ground truth
![Page 17: Contour-Based Joint Clustering of Multiple Segmentations Daniel Glasner * 1 Shiv N. Vitaladevuni * 2 Ronen Basri 1 * equal contribution authors 1 2.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649cf95503460f949c9fb1/html5/thumbnails/17.jpg)
Qualitative ResultsReference frame:
Our result:
![Page 18: Contour-Based Joint Clustering of Multiple Segmentations Daniel Glasner * 1 Shiv N. Vitaladevuni * 2 Ronen Basri 1 * equal contribution authors 1 2.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649cf95503460f949c9fb1/html5/thumbnails/18.jpg)
Qualitative ResultsReference frame:
Our result:
![Page 19: Contour-Based Joint Clustering of Multiple Segmentations Daniel Glasner * 1 Shiv N. Vitaladevuni * 2 Ronen Basri 1 * equal contribution authors 1 2.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649cf95503460f949c9fb1/html5/thumbnails/19.jpg)
Summary
• Joint segmentation of closely related images• Additive contour representation
– Ignores internal contours of unions of segments• Efficient convex optimization to find subsets
of segments with a similar shape• Combine inter and intra image similarity
![Page 20: Contour-Based Joint Clustering of Multiple Segmentations Daniel Glasner * 1 Shiv N. Vitaladevuni * 2 Ronen Basri 1 * equal contribution authors 1 2.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649cf95503460f949c9fb1/html5/thumbnails/20.jpg)
Thank you!