Learning Low-rank Transformations: Algorithms and...
Transcript of Learning Low-rank Transformations: Algorithms and...
![Page 1: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/1.jpg)
Learning Low-rank Transformations:
Algorithms and Applications
Qiang Qiu
Guillermo Sapiro
![Page 2: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/2.jpg)
Motivation
![Page 3: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/3.jpg)
Outline
Low-rank transform - algorithms and theories
Applications
Subspace clustering
Classification
Hashing/indexing
3
![Page 4: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/4.jpg)
Algorithms and Theories
![Page 5: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/5.jpg)
A Toy Formulation
A toy formulation
Notation Yc denotes d-dim points in the c-th class (arranged as columns).
Y=[Y1, Y2, …, YC], points from all C classes.
T is a learned d×d transformation matrix.
Theorem Non-negative.
But zero for independent matrices.
Intra-class Inter-class Non-trivial solution
![Page 6: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/6.jpg)
Low-rank Transformation
Basic formulation
|A|* denotes the nuclear norm of the matrix A: The sum of the singular values of A.
A good approximation to the matrix rank.
Theorem: Non-negative
Zero for orthogonal subspaces Not true for rank and other popular norms
Works on-line.
Works with compressing transform matrix.
![Page 7: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/7.jpg)
Low-rank Transformation
rank(A)=1, rank(B)=1, rank([A B]) =2
![Page 8: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/8.jpg)
Low-rank Transformation
![Page 9: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/9.jpg)
Theorem 1
![Page 10: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/10.jpg)
Theorem 2
![Page 11: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/11.jpg)
Other Popular Norms?
![Page 12: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/12.jpg)
Kernelized Transform
12
![Page 13: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/13.jpg)
Transform-based Dimension
Reduction
13
Extended YaleB face dataset
![Page 14: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/14.jpg)
Subspace Clustering using Low-rank
Transform
![Page 15: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/15.jpg)
Subspace clustering
15
![Page 16: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/16.jpg)
Robust Sparse Subspace Clustering
(R-SSC) For the transformed points, we first recover their low-
rank representation L
Each transformed point Tyi is then represented using
its KNN in L, denoted as Li
Let ,
Perform spectral clustering on sparse representation
matrix (|X|+|X’|)
16
![Page 17: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/17.jpg)
17
![Page 18: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/18.jpg)
18
Misclassification rate (e%) on clustering different subjects.
![Page 19: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/19.jpg)
Classification using Low-rank
Transform
![Page 20: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/20.jpg)
Basic Scheme
For the c-th class, we first recover its low-rank
representation Lc
Each testing point y is assigned to Lc that gives the
minimal reconstruction error
20
![Page 21: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/21.jpg)
21
Face recognition across illumination
![Page 22: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/22.jpg)
22
Face recognition across pose and illumination
![Page 23: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/23.jpg)
Classification using Transform Forest
![Page 24: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/24.jpg)
Transform Forest
![Page 25: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/25.jpg)
Transform learner
Learn T at each split node
Kernelized version
25
![Page 26: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/26.jpg)
Transform Learner
Random Grouping: Randomly partition training classes arriving at
each split node into two groups.
Learn a pair of dictionaries D±, for each of the two
groups by minimizing
The split function is evaluated using the reconstruction error,
where
26
![Page 27: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/27.jpg)
Results
![Page 28: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/28.jpg)
Quantitative Results
Extended YaleB face dataset
![Page 29: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/29.jpg)
On the Number of Trees
![Page 30: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/30.jpg)
Hashing using Transform Forest
![Page 31: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/31.jpg)
Motivation
Hash/binary codes are needed to deal with big data
Storage
Retrieval
31
![Page 32: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/32.jpg)
ForestHash
Challenge 1: Create consistent hash codes in each tree.
Low-rank transform.
Challenge 2: Merging trees for unique codes per class.
A mutual information based technique for near-optimal
code aggregation.
32
We simply set ‘1’ for the visited
nodes, and ‘0’ for the rest,
obtaining a (2d−2)-bit hash code.
![Page 33: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/33.jpg)
Challenge 1: Consistent Codes
Transform learner: learn T at each split node
Random Grouping: Randomly partition training classes arriving at
each split node into two groups.
Each tree enforces consistent but non-unique codes for a class.
But each class shares codes with different classes in different trees.
33
![Page 34: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/34.jpg)
Challenge 2: Code Aggregation
Hash codes from a random forest consisting M trees
for N training samples
Our objective is to select k code blocks B∗,
k ≤ L/(2d − 2)
Unsupervised code aggregation,
Supervised code aggregation (class labels C ),
Semi-supervised code aggregation,
34
![Page 35: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/35.jpg)
Example 1: Image Retrieval
35
HDML and DeepNet are deep learning based hashing methods.
![Page 36: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/36.jpg)
Example 1: Image Retrieval
36
Cifar-10 dataset
![Page 37: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/37.jpg)
Example 2: Cross-modality
37
The Wikipedia dataset
![Page 38: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/38.jpg)
Example 3: Document Retrieval
38
Reuters21578 dataset
![Page 39: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/39.jpg)
Example 4: Faces
39
… FaceHash
Each face is indexed by a 48-bit hash code.
~30 microseconds to index a face.
~20 milliseconds to scan one million faces.
![Page 40: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/40.jpg)
40
The blacklist/whitelist scenario
- Enrolling in a list 200 subjects.
- 5,992 face queries
~24 seconds to index all 0.7M faces.
~16 milliseconds to query 0.7M faces.
Query over a 73K database containing 37,007 unseen
faces from 200 known subjects and 35,902 unseen faces
from 27,859 unknown subjects.
Query over a 0.7M database containing 37,007 unseen
faces (200 known subjects) and 0.7M unseen faces (
29,392 unknown subjects).
![Page 41: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/41.jpg)
Example 4: Faces
41
![Page 42: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/42.jpg)
Example 5: Cross-modality Faces
42
Face attributes
![Page 43: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/43.jpg)
Example 5: Cross-modality Faces
43
![Page 44: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/44.jpg)
44
![Page 45: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/45.jpg)
Thank you!
45
![Page 46: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/46.jpg)
Reference
Qiang Qiu, Guillermo Sapiro, "Learning Transformations for Clustering and
Classification", Journal of Machine Learning Research (JMLR),
16(Feb):187−225, 2015
Qiang Qiu, Guillermo Sapiro, Alex Bronstein, "Random Forests Can Hash",
International Conference on Learning Representations (ICLR) Workshop,
2015
Qiang Qiu and Guillermo Sapiro, "Learning Transformations for
Classification Forests", International Conference on Learning
Representations (ICLR), 2014
Qiang Qiu and Guillermo Sapiro, "Learning Compressed Image Classification
Features", International Conference on Image Processing, 2014
46
![Page 47: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/47.jpg)
The Concave-Convex Procedure
Difference of convex function
A simple projected subgradient method
47
![Page 48: Learning Low-rank Transformations: Algorithms and Applicationspeople.duke.edu/~qq3/pub/learnT_cvpr15.pdf · 2015-10-26 · A Toy Formulation A toy formulation Notation Y c denotes](https://reader034.fdocuments.net/reader034/viewer/2022050522/5fa5a65df96155350c555b22/html5/thumbnails/48.jpg)
A subgradient of matrix nuclear
norm
48