Nonlinear Dimensionality Reduction Frameworks
description
Transcript of Nonlinear Dimensionality Reduction Frameworks
Nonlinear Dimensionality Reduction Frameworks
Rong Xu
Chan su Lee
Outline
• Intuition of Nonlinear Dimensionality Reduction(NLDR)
• ISOMAP
• LLE
• NLDR in Gait Analysis
Intuition: how does your brain store these pictures?
Brain Representation
Brain Representation
• Every pixel?
• Or perceptually meaningful structure?– Up-down pose
– Left-right pose
– Lighting direction
So, your brain successfully reduced the high-dimensional inputs to an intrinsically 3-dimensional manifold!
Data for Faces
Data for Handwritten “2”
Data for Hands
Manifold Learning
• A manifold is a topological space which is locally Euclidean
• An example of nonlinear manifold:
Manifold Learning• Discover low dimensional
representations (smooth manifold) for data in high dimension.
• Linear approaches(PCA, MDS) vs Non-linear approaches (ISOMAP, LLE)
d
i Ry Y
XN
i Rx
latent
observed
Linear Approach- PCA
• PCA Finds subspace linear projections of input data.
Linear Approach- MDS
• MDS takes a matrix of pairwise distances and gives a mapping to Rd. It finds an embedding that preserves the interpoint distances, equivalent to PCA when those distance are Euclidean.
• BUT! PCA and MDS both fail to do embedding with nonlinear data, like swiss roll.
Nonlinear Approaches- ISOMAP
• Constructing neighbourhood graph G• For each pair of points in G, Computing shortest
path distances ---- geodesic distances.• Use Classical MDS with geodesic distances.
Euclidean distance Geodesic distance
Josh. Tenenbaum, Vin de Silva, John langford 2000
Sample points with Swiss Roll
• Altogether there are 20,000 points in the “Swiss roll” data set. We sample 1000 out of 20,000.
Construct neighborhood graph G
K- nearest neighborhood (K=7)
DG is 1000 by 1000 (Euclidean) distance matrix of two neighbors (figure A)
Compute all-points shortest path in G
Now DG is 1000 by 1000 geodesic distance matrix of two arbitrary points along the manifold(figure B)
Find a d-dimensional Euclidean space Y(Figure c) to minimize the cost function:
Use MDS to embed graph in Rd
22
222
22
2
2
)(min||||min
||)(5.0||min||)()(||min
1*11
2
1)(
2
1)(
YYXXtraceYYXX
HDDHDD
NIH
DD
HHDD
HHDD
TTTT
GYG
ijG
YY
GG
Y
Theorem: For any squared distance matrix ,there exists of points xi and,xj, such that
So
2GD
)()'(2
jijiij xxxxd
XXD TG 2
Linear Approach-classical MDS
Solution
dxNdddxN
N
v
v
v
yyyY
22
11
21
|
|
|
|
|
|
VY dY2/1
)(
Y lies in Rd and consists of N points correspondent to the N original points in input space.
PCA, MD vs ISOMAP
• Nonlinear
• Globally optimal• Still produces globally optimal low-dimensional Euclidean
representation even though input space is highly folded,
twisted, or curved.
• Guarantee asymptotically to recover the true dimensionality.
Isomap: Advantages
• May not be stable, dependent on topology of data
• Guaranteed asymptotically to recover geometric structure of nonlinear manifolds– As N increases, pairwise distances provide better
approximations to geodesics, but cost more computation
– If N is small, geodesic distances will be very inaccurate.
Isomap: Disadvantages