Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning...
-
Upload
quinton-wyke -
Category
Documents
-
view
217 -
download
1
Transcript of Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning...
![Page 1: Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.net/reader036/viewer/2022081520/56649cab5503460f9496c7ce/html5/thumbnails/1.jpg)
Richard G. Baraniuk Chinmay Hegde
Manifold Learning in the WildA New Manifold Modeling and Learning Framework for Image Ensembles
Aswin C. SankaranarayananRice University
![Page 2: Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.net/reader036/viewer/2022081520/56649cab5503460f9496c7ce/html5/thumbnails/2.jpg)
Sensor Data Deluge
![Page 3: Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.net/reader036/viewer/2022081520/56649cab5503460f9496c7ce/html5/thumbnails/3.jpg)
Internet Scale Databases
• Tremendous size of corpus of available data– Google Image Search of “Notre Dame Cathedral”
yields 3m results 3Tb of data
![Page 4: Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.net/reader036/viewer/2022081520/56649cab5503460f9496c7ce/html5/thumbnails/4.jpg)
Concise Models
• Efficient processing / compression requires concise representation
• Our interest in this talk: Collections of images
![Page 5: Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.net/reader036/viewer/2022081520/56649cab5503460f9496c7ce/html5/thumbnails/5.jpg)
Concise Models
• Our interest in this talk: Collections of image
parameterized by q \in Q
– translations of an object q: x-offset and y-offset
– rotations of a 3D object: q pitch, roll, yaw
– wedgeletsq: orientation and offset
![Page 6: Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.net/reader036/viewer/2022081520/56649cab5503460f9496c7ce/html5/thumbnails/6.jpg)
Concise Models
• Our interest in this talk: Collections of image
parameterized by q \in Q
– translations of an object q: x-offset and y-offset
– rotations of a 3D object: q pitch, roll, yaw
– wedgeletsq: orientation and offset
• Image articulation manifold
![Page 7: Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.net/reader036/viewer/2022081520/56649cab5503460f9496c7ce/html5/thumbnails/7.jpg)
Image Articulation Manifold• N-pixel images:
• K-dimensional articulation space
• Thenis a K-dimensional manifoldin the ambient space
• Very concise model– Can be learnt using Non-linear dim. reduction
articulation parameter space
![Page 8: Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.net/reader036/viewer/2022081520/56649cab5503460f9496c7ce/html5/thumbnails/8.jpg)
Ex: Manifold Learning
LLEISOMAPLEHEDiff. Geo …
• K=1rotation
![Page 9: Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.net/reader036/viewer/2022081520/56649cab5503460f9496c7ce/html5/thumbnails/9.jpg)
Ex: Manifold Learning
• K=2rotation and scale
![Page 10: Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.net/reader036/viewer/2022081520/56649cab5503460f9496c7ce/html5/thumbnails/10.jpg)
Smooth IAMs• N-pixel images:
• Local isometry image distance parameter space distance
• Linear tangent spacesare close approximationlocally
• Low dimensional articulation space
articulation parameter space
![Page 11: Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.net/reader036/viewer/2022081520/56649cab5503460f9496c7ce/html5/thumbnails/11.jpg)
Smooth IAMs
articulation parameter space
• N-pixel images:
• Local isometry image distance parameter space distance
• Linear tangent spacesare close approximationlocally
• Low dimensional articulation space
![Page 12: Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.net/reader036/viewer/2022081520/56649cab5503460f9496c7ce/html5/thumbnails/12.jpg)
Smooth IAMs
articulation parameter space
• N-pixel images:
• Local isometry image distance parameter space distance
• Linear tangent spacesare close approximationlocally
• Low dimensional articulation space
![Page 13: Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.net/reader036/viewer/2022081520/56649cab5503460f9496c7ce/html5/thumbnails/13.jpg)
• Ex: translation manifold
all blue images are equidistant from the red image
• Local isometry
– satisfied only when sampling is dense
0 20 40 60 80 1000
0.5
1
1.5
2
2.5
3
3.5
Translation in [px]
Euc
lidea
n di
stan
ce
Theory/Practice Disconnect Isometry
![Page 14: Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.net/reader036/viewer/2022081520/56649cab5503460f9496c7ce/html5/thumbnails/14.jpg)
Theory/Practice DisconnectNuisance articulations
• Unsupervised data, invariably, has additional undesired articulations– Illumination– Background clutter, occlusions, …
• Image ensemble is no longer low-dimensional
![Page 15: Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.net/reader036/viewer/2022081520/56649cab5503460f9496c7ce/html5/thumbnails/15.jpg)
Image representations
• Conventional representation for an image– A vector of pixels– Inadequate!
pixel image
![Page 16: Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.net/reader036/viewer/2022081520/56649cab5503460f9496c7ce/html5/thumbnails/16.jpg)
Image representations• Replace vector of pixels with an abstract
bag of features
– Ex: SIFT (Scale Invariant Feature Transform) selects keypoint locations in an image and computes keypoint descriptors for each keypoint
– Very popular in many many vision problems
![Page 17: Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.net/reader036/viewer/2022081520/56649cab5503460f9496c7ce/html5/thumbnails/17.jpg)
Image representations• Replace vector of pixels with an abstract
bag of features
– Ex: SIFT (Scale Invariant Feature Transform) selects keypoint locations in an image and computes keypoint descriptors for each keypoint
– Keypoint descriptors are local; it is very easy to make them robust to nuisance imaging parameters
![Page 18: Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.net/reader036/viewer/2022081520/56649cab5503460f9496c7ce/html5/thumbnails/18.jpg)
Loss of Geometrical Info
• Bag of features representations hide potentially useful image geometry
• Goal: make salient image geometrical info more explicit for exploitation
Image space
Keypoint space
![Page 19: Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.net/reader036/viewer/2022081520/56649cab5503460f9496c7ce/html5/thumbnails/19.jpg)
Key idea
• Keypoint space can be endowed with a rich low-dimensional structure in many situations
![Page 20: Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.net/reader036/viewer/2022081520/56649cab5503460f9496c7ce/html5/thumbnails/20.jpg)
Key idea
• Keypoint space can be endowed with a rich low-dimensional structure in many situations
• Mechanism: define kernels , between keypoint locations, keypoint descriptors
![Page 21: Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.net/reader036/viewer/2022081520/56649cab5503460f9496c7ce/html5/thumbnails/21.jpg)
Keypoint Kernel
• Keypoint space can be endowed with a rich low-dimensional structure in many situations
• Mechanism: define kernels , between keypoint locations, keypoint descriptors
• Joint keypoint kernel between two images
is given by
![Page 22: Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.net/reader036/viewer/2022081520/56649cab5503460f9496c7ce/html5/thumbnails/22.jpg)
Many Possible Kernels
• Euclidean kernel
• Gaussian kernel
• Polynomial kernel
• Pyramid match kernel [Grauman et al. ’07]
• Many others
![Page 23: Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.net/reader036/viewer/2022081520/56649cab5503460f9496c7ce/html5/thumbnails/23.jpg)
Keypoint Kernel
• Joint keypoint kernel between two images
is given by
• Using Euclidean/Gaussian (E/G) combination yields
![Page 24: Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.net/reader036/viewer/2022081520/56649cab5503460f9496c7ce/html5/thumbnails/24.jpg)
From Kernel to Metric
Lemma: The E/G keypoint kernel is a Mercer kernel
– enables algorithms such as SVM
Lemma: The E/G keypoint kernel induces a metric on the space of images
– alternative to conventional L2 distance between images– keypoint metric robust to nuisance imaging parameters,
occlusion, clutter, etc.
![Page 25: Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.net/reader036/viewer/2022081520/56649cab5503460f9496c7ce/html5/thumbnails/25.jpg)
Keypoint Geometry
Theorem: Under the metric induced by the kernel
certain ensembles of articulating images formsmooth, isometric manifolds
• Keypoint representation compact, efficient, and …
• Robust to illumination variations, non-stationary backgrounds, clutter, occlusions
![Page 26: Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.net/reader036/viewer/2022081520/56649cab5503460f9496c7ce/html5/thumbnails/26.jpg)
Keypoint Geometry
Theorem: Under the metric induced by the kernel
certain ensembles of articulating images formsmooth, isometric manifolds
• In contrast: conventional approach to image fusion via image articulation manifolds (IAMs) fraught with non-differentiability (due to sharp image edges)– not smooth– not isometric
![Page 27: Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.net/reader036/viewer/2022081520/56649cab5503460f9496c7ce/html5/thumbnails/27.jpg)
Application: Manifold Learning
2D Translation
![Page 28: Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.net/reader036/viewer/2022081520/56649cab5503460f9496c7ce/html5/thumbnails/28.jpg)
Application: Manifold Learning
2D Translation IAM KAM
![Page 29: Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.net/reader036/viewer/2022081520/56649cab5503460f9496c7ce/html5/thumbnails/29.jpg)
Manifold Learning in the Wild
• Rice University’s Duncan Hall Lobby– 158 images– 360° panorama using handheld camera– Varying brightness, clutter
![Page 30: Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.net/reader036/viewer/2022081520/56649cab5503460f9496c7ce/html5/thumbnails/30.jpg)
• Duncan Hall Lobby• Ground truth using state of the art
structure-from-motion software
Manifold Learning in the Wild
Ground truth IAM KAM
![Page 31: Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.net/reader036/viewer/2022081520/56649cab5503460f9496c7ce/html5/thumbnails/31.jpg)
Manifold Learning in the Wild
• Rice University’s Brochstein Pavilion– 400 outdoor images of a building– occlusions, movement in foreground, varying background
![Page 32: Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.net/reader036/viewer/2022081520/56649cab5503460f9496c7ce/html5/thumbnails/32.jpg)
Manifold Learning in the Wild
• Brochstein Pavilion– 400 outdoor images of a building– occlusions, movement in foreground, background
IAM KAM
![Page 33: Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.net/reader036/viewer/2022081520/56649cab5503460f9496c7ce/html5/thumbnails/33.jpg)
Internet scale imagery
• Notre-dame cathedral – 738 images
– Collected from Flickr
– Large variations in illumination (night/day/saturations), clutter (people, decorations), camera parameters (focal length, fov, …)
– Non-uniform sampling of the space
![Page 34: Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.net/reader036/viewer/2022081520/56649cab5503460f9496c7ce/html5/thumbnails/34.jpg)
Organization
• k-nearest neighbors
![Page 35: Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.net/reader036/viewer/2022081520/56649cab5503460f9496c7ce/html5/thumbnails/35.jpg)
Organization
• “geodesics’
3D rotation
“Walk-closer”
“zoom-out”
![Page 36: Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.](https://reader036.fdocuments.net/reader036/viewer/2022081520/56649cab5503460f9496c7ce/html5/thumbnails/36.jpg)
Summary
• Challenges for manifold learning in the wild are both theoretical and practical
• Need for novel image representations– Sparse features
Robustness to outliers, nuisance articulations, etc. Learning in the wild: unsupervised imagery
• Promise lies in fast methods that exploit only neighborhood properties– No complex optimization required