Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother...
Transcript of Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother...
![Page 1: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/1.jpg)
Contour Detection and Hierarchical Image Segmentation
β Some Experiments
CS395T β Visual Recognition
Presented by Elad Liebman
![Page 2: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/2.jpg)
Overview
β’ Understanding the method better: β The importance of thresholding the ouput β Understanding the inputs better β Understanding the UCM
β’ Pushing for boundaries: β Types of difficult inputs β Difficult input examples
![Page 3: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/3.jpg)
Warm-up: choosing the right threshold
π = 0.4
![Page 4: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/4.jpg)
Warm-up: choosing the right threshold
π = 0.1
![Page 5: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/5.jpg)
Warm-up: choosing the right threshold
π = 0.8
![Page 6: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/6.jpg)
So what goes into the OWT?
β’ From Contours to Regions: An Empirical Evaluation. P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik. CVPR 2009.
β’ Contour Detection and Hierarchical Image Segmentation. P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik. PAMI 2011
![Page 7: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/7.jpg)
The Features
β’ Difference in feature channels on the two halves of a disc of radius π and orientation π.
β’ Feature channels in our case: β Color gradients β Brightness gradients β Texture gradients
β’ Comparison between the two disc halves using π2 distance.
![Page 8: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/8.jpg)
The basic signals
β’ Color/brightness channels based on Lab color space.
β’ Repeatedly generated at different scales (different π radii values - π
2,π, 2π).
β’ All in all we get 13 different inputs.
![Page 9: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/9.jpg)
Illustration n. 1: Color gradient π (red-green scale)
π =π2
π = βπ4
π =
π2
maxπ
{πππ}
![Page 10: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/10.jpg)
Illustration n. 2: Color gradient π, different π value
maxπ
{πππ}
π = 0
π =π4
π = βπ4
![Page 11: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/11.jpg)
Illustration 3: Brightness gradient (light-dark scale)
maxπ
{πππ}
π =π2
π = βπ4
π =
π2
![Page 12: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/12.jpg)
Illustration 4: Texton Map
![Page 13: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/13.jpg)
Multiscale ππ and Spectral/Global ππ β’ The features are linearly combined to create a
unified signal, the multiscale Pb input. β’ They are then βglobalizedβ spectrally. β’ An affinity matrix π€ is constructed with πππ
being the maximal value of πππ along the line connecting π and π.
β’ Generalized eigenvectors are then extracted and combined (after processing) with the local ππ data to create the final input for the OWT.
![Page 14: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/14.jpg)
πππ vs. πππ
![Page 15: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/15.jpg)
(In practice we sample 8 orientationsβ¦)
πππ = maxπ
{ }
π = [0,π8
,π4
,3π8
, β¦ ,7π8
]
![Page 16: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/16.jpg)
(In practice we sample 8 orientationsβ¦)
πππ =
![Page 17: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/17.jpg)
Playing with the orientations a bitβ¦
![Page 18: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/18.jpg)
Playing with the orientations a bitβ¦
![Page 19: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/19.jpg)
Using less than 8 orientations
2 orientations
![Page 20: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/20.jpg)
Using less than 8 orientations
4 orientations (in this case itβs enough)
![Page 21: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/21.jpg)
The Ultrametric Contour Map
β’ The basic idea is to organize and merge sub-regions hierarchically and iteratively.
β’ At each point we merge two regions with the weakest boundary between them.
β’ The result is a dendrogram of nested regions.
![Page 22: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/22.jpg)
Example 1
![Page 23: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/23.jpg)
Example 2
![Page 24: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/24.jpg)
Potentially Difficult Input
β’ We would like to test the OWT-UCM method against problematic input.
β’ See where it might break.
![Page 25: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/25.jpg)
warming up - blurring
![Page 26: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/26.jpg)
Potentially Difficult Input II
β’ Blurring isnβt very interesting β doesnβt reflect the kind of challenges the algorithm is likely to face.
β’ What realistic problems exist? β Difficult patches β Complex, contour-rich images β Not enough color/texture information
![Page 27: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/27.jpg)
Difficult Input β abstract art
π = 0.4
![Page 28: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/28.jpg)
Difficult Input β abstract art
π = 0.1
![Page 29: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/29.jpg)
Difficult(er) Input β abstract(er?) art
π = 0.4
![Page 30: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/30.jpg)
Difficult(er) Input β abstract(er?) art
π = 0.1
![Page 31: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/31.jpg)
Another Difficult Example β’ Occlusion β’ Similarity of objects in image:
β Similar textures β Similar colors β Contour lines blend
![Page 32: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/32.jpg)
Another Difficult Example
π = 0.4
![Page 33: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/33.jpg)
Another Difficult Example
π = 0.1
![Page 34: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/34.jpg)
Another Difficult Example
π = 0.2
![Page 35: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/35.jpg)
Sneaking a look under the hood
β’ Possibly we need to weight elements differently in certain casesβ¦
CGA1 CGB1
Textons
BG1
![Page 36: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/36.jpg)
Summary β’ Studied some of the technical aspects of the
method: β Threshold selection β Understanding the input data β Illustration of UCM in action
β’ Tested the method against difficult input: β Problematic contours β Complex images β Cases where features arenβt informative enough β Many similar items occluding one another
![Page 37: Contour Detection and Hierarchical Image Segmentation ...cv-fall2012/slides/elad-expt.pdfAnother Difficult Example π= 0.2. Sneaking a look under the hood β’ Possibly we need to](https://reader035.fdocuments.net/reader035/viewer/2022070107/6023884d04f5c44b0b231cd5/html5/thumbnails/37.jpg)
References β’ From Contours to Regions: An Empirical Evaluation. P.
Arbelaez, M. Maire, C. Fowlkes, and J. Malik. CVPR 2009. β’ Contour Detection and Hierarchical Image Segmentation.
P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik. PAMI 2011
β’ Using Contours to Detect and Localize Junctions in Natural Images. M. Maire, P. Arbelaez, C. Fowlkes, and J. Malik. CVPR 2008.
β’ What are Textons? S. Zhu, C. Guo, Y. Want Z. Xu, International Journal of Computer Vision 2005.
β’ Class notes for 16-721: Learning-Based Methods in Vision, taught by Prof. Alexei Efros, CMU.
β’ Class notes for CS 143: Introduction to Computer Vision, taught by Prof. James Hayes, Brown.