1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with...
-
Upload
martha-randall -
Category
Documents
-
view
216 -
download
0
Transcript of 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with...
![Page 1: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/1.jpg)
1
Contours and Junctions in Natural
Images
Jitendra MalikUniversity of California at Berkeley
(with Jianbo Shi, Thomas Leung, Serge Belongie, Charless Fowlkes, David Martin, Xiaofeng Ren, Michael Maire, Pablo Arbelaez)
![Page 2: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/2.jpg)
2
From Pixels to Perception
TigerGrass
Water
Sand
outdoorwildlife
Tiger
tail
eye
legs
head
back
shadow
mouth
![Page 3: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/3.jpg)
3
I stand at the window and see a house, trees, sky. Theoretically I might say there were 327 brightnesses and nuances of colour.
Do I have "327"?
No. I have sky, house, and trees.
---- Max Wertheimer, 1923
![Page 4: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/4.jpg)
4
Perceptual Organization
Grouping Figure/Ground
![Page 5: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/5.jpg)
5
Key Research Questions in Perceptual Organization
• Predictive power– Factors for complex, natural stimuli ?
– How do they interact ?
• Functional significance– Why should these be useful or confer some
evolutionary advantage to a visual organism?
• Brain mechanisms– How are these factors implemented given what we
know about V1 and higher visual areas?
![Page 6: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/6.jpg)
6
Attneave’s Cat (1954)Line drawings convey most of the
information
![Page 7: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/7.jpg)
7
Contours and junctions are fundamental…
• Key to recognition, inference of 3D scene properties, visually- guided manipulation and locomotion…
• This goes beyond local, V1-like, edge-detection. Contours are the result of perceptual organization, grouping and figure/ground processing
![Page 8: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/8.jpg)
8
Some computer vision history…
• Local Edge Detection was much studied in the 1970s and early 80s (Sobel, Rosenfeld, Binford-Horn, Marr-Hildreth, Canny …)
• Edge linking exploiting curvilinear continuity was studied as well (Rosenfeld, Zucker, Horn, Ullman …)
• In the 1980s, several authors argued for perceptual organization as a precursor to recognition (Binford, Witkin and Tennebaum, Lowe, Jacobs …)
![Page 9: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/9.jpg)
9
However in the 90s …
1. We realized that there was more to images than edges• Biologically inspired filtering approaches (Bergen & Adelson,
Malik & Perona..)
• Pixel based representations for recognition (Turk & Pentland, Murase & Nayar, LeCun …)
2. We lost faith in the ability of bottom-up vision• Do minimal bottom up processing , e.g. tiled orientation
histograms don’t even assume that linked contours or junctions can be extracted
• Matching with memory of previously seen objects then becomes the primary engine for parsing an image.
√
?
![Page 10: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/10.jpg)
10
At Berkeley, we took a contrary view…
1. Collect Data Set of Human segmented images
2. Learn Local Boundary Model for combining brightness, color and texture
3. Global framework to capture closure, continuity
4. Detect and localize junctions
5. Integrate low, mid and high-level information for grouping and figure-ground segmentation
![Page 11: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/11.jpg)
11D. Martin, C. Fowlkes, D. Tal, J. Malik. "A Database of Human Segmented Natural Images and its
Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics", ICCV, 2001
Berkeley Segmentation DataSet [BSDS]
![Page 12: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/12.jpg)
12
![Page 13: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/13.jpg)
13
Contour detection ~1970
13
![Page 14: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/14.jpg)
14
Contour detection ~1990
14
![Page 15: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/15.jpg)
15
Contour detection ~2004
15
![Page 16: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/16.jpg)
16
Contour detection ~2008 (gray)
16
![Page 17: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/17.jpg)
17
Contour detection ~2008 (color)
17
![Page 18: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/18.jpg)
18
Outline
1. Collect Data Set of Human segmented images
2. Learn Local Boundary Model for combining brightness, color and texture
3. Global framework to capture closure, continuity
4. Detect and localize junctions
5. Integrate low, mid and high-level information for grouping and figure-ground segmentation
![Page 19: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/19.jpg)
19
Contours can be defined by any of a number of cues (P. Cavanagh)
![Page 20: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/20.jpg)
20Grill-Spector et al. , Neuron 1998
Objects from disparityObjects from texture
Objects from luminance
Cue-Invariant Representations
Line drawings
Gray level photographs
Objects from motion
![Page 21: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/21.jpg)
21
Martin, Fowlkes, Malik PAMI 04Image
Boundary Cues
Model
Pb
Brightness
Color
Texture
Challenges: texture cue, cue combination Goal: learn the posterior probability of a boundary Pb(x,y,) from local information only
Cue Combination
![Page 22: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/22.jpg)
22
Individual Features
• 1976 CIE L*a*b* colorspace• Brightness Gradient BG(x,y,r,)
– Difference of L* distributions
• Color Gradient CG(x,y,r,)– Difference of a*b* distributions
• Texture Gradient TG(x,y,r,)– Difference of distributions of
V1-like filter responses
r(x,y)
These are combined using logistic regression
![Page 23: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/23.jpg)
23
Various Cue Combinations
![Page 24: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/24.jpg)
24
Outline
1. Collect Data Set of Human segmented images
2. Learn Local Boundary Model for combining brightness, color and texture
3. Global framework to capture closure, continuity
4. Detect and localize junctions
5. Integrate low, mid and high-level information for grouping and figure-ground segmentation
![Page 25: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/25.jpg)
25
Exploiting global constraints:Image Segmentation as Graph Partitioning
Build a weighted graph G=(V,E) from image
V: image pixels
E: connections between pairs of nearby pixels
Partition graph so that similarity within group is large and similarity between groups is small -- Normalized Cuts [Shi & Malik 97]
![Page 26: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/26.jpg)
26
Wij small when intervening contour strong, small when weak..
Cij = max Pb(x,y) for (x,y) on line segment ij; Wij = exp ( - Cij /
![Page 27: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/27.jpg)
27
Normalized Cuts as a Spring-Mass system
• Each pixel is a point mass; each connection is a spring:
• Fundamental modes are generalized eigenvectors of
(D - W) x = Dx
DyyWD =− )(
QuickTime™ and aYUV420 codec decompressor
are needed to see this picture.
![Page 28: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/28.jpg)
28
Eigenvectors carry contour information
![Page 29: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/29.jpg)
29
![Page 30: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/30.jpg)
30
We do not try to find regions from the eigenvectors, so we avoid the “broken sky” artifacts of Ncuts ..
![Page 31: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/31.jpg)
31
The Benefits of GlobalizationMaire, Arbelaez, Fowlkes, Malik, CVPR 08
![Page 32: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/32.jpg)
32
Comparison to other approaches
![Page 33: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/33.jpg)
33
![Page 34: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/34.jpg)
34
Outline
1. Collect Data Set of Human segmented images
2. Learn Local Boundary Model for combining brightness, color and texture
3. Global framework to capture closure, continuity
4. Detect and localize junctions
5. Integrate low, mid and high-level information for grouping and figure-ground segmentation
![Page 35: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/35.jpg)
35
Detecting Junctions
![Page 36: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/36.jpg)
36
![Page 37: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/37.jpg)
37
Benchmarking corner detection
![Page 38: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/38.jpg)
38
![Page 39: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/39.jpg)
39
Better object recognition using previous version of Pb
• Ferrari, Fevrier, Jurie and Schmid (PAMI 08)
• Shotton, Blake and Cipolla (PAMI 08)
![Page 40: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/40.jpg)
40
Outline
1. Collect Data Set of Human segmented images2. Learn Local Boundary Model for combining
brightness, color and texture3. Global framework to capture closure, continuity4. Detect and localize junctions5. Integrate low, mid and high-level cues for
grouping and figure-ground segmentation1. Ren, Fowlkes, Malik, IJCV ‘082. Fowlkes, Martin, Malik, JOV ‘073. Ren, Fowlkes, Malik, ECCV ‘06
![Page 41: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/41.jpg)
41
Power laws for contour lengths
![Page 42: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/42.jpg)
42
Convexity(p) = log(ConvF / ConvG)
Convexity [Metzger 1953, Kanizsa and Gerbino 1976]
ConvG = percentage of straight lines that lie completely within region G
pG F
![Page 43: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/43.jpg)
43
Figural regions tend to be convex
![Page 44: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/44.jpg)
44
LowerRegion(p) = θG
Lower Region[Vecera, Vogel & Woodman 2002]
θ
p
center of mass
![Page 45: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/45.jpg)
45
Figural regions tend to lie below ground regions
![Page 46: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/46.jpg)
46
Ren, Fowlkes, Malik ECCV ‘06
• Human subjects label groundtruth figure/ground assignments in natural images.
• Shapemes encode high-level knowledge in a generic way, capturing local figure/ground cues.
• A conditional random field incorporates junction cues and enforces global consistency.
Grouping /Segmentation
Figure/GroundOrganization
Object and Scene Recognition
![Page 47: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/47.jpg)
47
Forty years of contour detection
47
Roberts(1965)
Sobel(1968)
Prewitt(1970)
MarrHildreth(1980)
Canny(1986)
PeronaMalik(1990)
MartinFowlkesMalik(2004)
MaireArbelaezFowlkesMalik(2008)
![Page 48: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/48.jpg)
48
Forty years of contour detection
48
???(2013)
Roberts(1965)
Sobel(1968)
Prewitt(1970)
MarrHildreth(1980)
Canny(1986)
PeronaMalik(1990)
MartinFowlkesMalik(2004)
MaireArbelaezFowlkesMalik(2008)
![Page 49: 1 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless.](https://reader033.fdocuments.net/reader033/viewer/2022051618/56649d1c5503460f949f18de/html5/thumbnails/49.jpg)
49
Curvilinear Grouping
• Boundaries are smooth in nature!
• A number of associated visual phenomena
Good continuation Visual completion Illusory contours