Approximation algorithms for large-scale kernel methods Taher Dameh School of Computing Science...
-
Upload
virginia-ursula-cunningham -
Category
Documents
-
view
246 -
download
2
Transcript of Approximation algorithms for large-scale kernel methods Taher Dameh School of Computing Science...
Approximation algorithms for large-scale kernel methods
Taher DamehSchool of Computing Science
Simon Fraser [email protected] 29th, 2010
Joint work with F. Gao, M. Hefeeda and W. Abdel-Majeed
1
Outline
Introduction Motivation Local Sensitive Hashing Z and H Curves Affinity propagation Results
2
Introduction
Machine learning Kernel-based methods require O(N2) time and space complexities to compute and store non-sparse Gram matrices.
We are developing methods to approximate the Gram matrix with a band matrix
N pointsN*N Gram Matrix
3
Motivation
Exact vs. Approximate Answer Approximate might be good-enough and much-
faster Time-quality and memory trade-off As machine learning point of view; we can live
with bounded (controlled) error as long as we can run on large scale data where in normal ways we cann’t at all due to the memory usage.
4
Ideas of approximation
To construct the approximated band matrix we evaluate the kernel function only between a fixed neighborhood around each point.
This low rank method depends on the observation that the eigen-spectrum of the kernel function is a Radial Basis Function (real-valued function whose value depends only on the Euclidean distance )
(The most information is stored in the first of eigen vectors)
5
How to choose this neighborhood window? Since kernel function is monotonically decreasing with
the Euclidian distance between the input points, so we can compute the kernel function only between close points.
We should find a fast and reliable technique to order the points.
Space filling Curves: Z-Curve and H-Curve Locality Sensitive Hashing
6
LSH: Motivation
Similarity Search over large scale High-Dimensional Data
Exact vs. Approximate Answer Approximate might be good-enough and much-faster Time-quality trade-off
7
LSH: Key idea
Hash the data-point using several LSH functions so that probability of collision is higher for closer objects
Algorithm:• Input
− Set of N points { p1 , …….. pn }− L ( number of hash tables )
• Output
− Hash tables Ti , i = 1 , 2, …. L
• Foreach i = 1 , 2, …. L
− Initialize Ti with a random hash function gi(.)
• Foreach i = 1 , 2, …. L Foreach j = 1 , 2, …. N Store point pj on bucket gi(pj) of hash table Ti
8
LSH: Algorithm
g1(pi) g2(pi) gL(pi)
TLT2T1
pi
P
9
LSH: Analysis
• Family H of (r1, r2, p1, p2)-sensitive functions, {hi(.)}
− dist(p,q) < r1 ProbH [h(q) = h(p)] p1
− dist(p,q) r2 ProbH [h(q) = h(p)] p2
− p1 > p2 and r1 < r2
• LSH functions: gi(.) = { h1(.) …hk(.) }
10
Our approach
N points Hash the points using LSH functions family
Compute the kernel function only between the points in same bucket (0 between points on different buckets)
Using “m” size hash table we can achieve as best case O (N2/m) memory and computation
11
Validation Methods
Low Level (matrix Level) Frobenius Norm Eigen spectrum
High Level (Application Level) Affinity Propagation Support Vector Machines
12
Example
i,k 0 1 2 3 4 5
0 -1 -1 -4 -25 -36 -49
1 -1 -1 -1 -26 -37 -50
2 -4 -1 -1 -29 -40 -53
3 -25 -26 -29 -1 -1 -4
4 -36 -37 -40 -1 -1 -1
5 -49 -50 -53 -4 -1 -1
i,k 0 1 2
0 -1 -1 -4
1 -1 -1 -1
2 -4 -1 -1
i,k 3 4 5
3 -1 -1 -4
4 -1 -1 -1
5 -4 -1 -1
S (i,k)
S0 (i,k) S1 (i,k)
1
2
5 6 7P0P1P2
P3 P4 P5
0 P0 P1 P2
1 P3 P4 P5
LSH
FrobNorm (S) = 230.469
FrobNorm ( [S0 S1] ) = 217.853
13
ResultsMemory Usage
All data Z512 Z1024 Lsh3000 Lsh5000
64 K 4 G 32 M 64 M 19 M 18 M
128 K 16 G 64 M 128 M 77 M 76 M
256 K 64 G 128 M 256 M 309 M 304 M
512 K 256 G 256 M 512 M 1244 M 1231 M
14
15
References
[1] M. Hussein and W. Abd-Almageed, “Efficient band approximation of gram matrices for large scale kernel methods on gpus,” in Proc. of the International Conference for High Performance Computing, Networking, Storage, and Analysis (SuperComputing’09), Portland, OR, November 2009.
[2] A. Andoni and P. Indyk, “Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions,” Communications of the ACM, vol. 51, no. 1, pp. 117–122, January 2008.
[3] B. J. Frey and D. Dueck, “Clustering by passing messages between data points,” Science, vol. 315, no. 5814, pp. 972–976, 2007.
16
AP : Motivation
No priori on the number of clusters Independence on initialization Processing time to achieve good
performance
17
Affinity Propagation
Take each data point as a node in the network
Consider all data points as potential cluster centers
Start the clustering with a similarity between pairs of data points
Exchange messages between data points until the good cluster centers are found
18
Terminology and Notation
Similarity s(i,k): single evidence of data k to be the exemplar for data I
(Kernel function for the Gram matrix)
19
Responsibility r (i,k ): accumulated evidence of data k to be the exemplar for data i
20
Availability a ( i,k) : accumulated evidence of data i pick data k as the exemplar
21
22
flow chart
23