EIGENSYSTEMS, SVD, PCA

Big Data Seminar, Dedi Gadot, December 14th, 2014

EIGVALS AND EIGVECS

Eigvals + Eigvecs• An eigenvector of a square matrix A is a non-zero vector

V that when multiplied with A yields a scalar multiplication of itself by LAMBDA (the eigenvalue)

• If A is a square, diagonalizable matrix –

Eigvecs – Toy Example

Geometric Transformations

SVD

SVD• Singular Value Decomposition

• A factorization of a given matrix to its components:

M = UΣV∗

• When:• M – an m x n real or complex matrix• U – an m x m unitary matrix, called the left singular vectors• V – an n x n unitary matrix, called the right singular vectors• Σ – an m x n rectangular diagonal matrix, called the singular values

Applications and Intuition• If M is a real, square matrix –

• U,V can be referred to as rotation matrices and Σ as a scaling matrix

M = UΣV∗

Applications and Intuition• The columns of U and V are orthonormal bases

• Singular vectors (of a square matrix) can be interpreted as the semiaxes of an ellipsoid in n-dimensional space

• SVD can be used to solve homogeneous linear equations• Ax=0, A is a square matrix x is the right singular vector which corresponds

to a singular value of A which is zero

• Low rank matrix approximation• Take Σ of M and leave only the r largest singular values, rebuild the matrix

using U,V and you’ll get a low rank approximation of M

• …

SVD and Eigenvalues• Given an SVD of M the following two relations hold:

• The columns of V are eigenvectors of M*M• The columns of U are eigenvectors of MM*• The non-zero elements of Σ are the square roots of the non-

zero eigenvalues of M*M or MM*

PCA

PCA• Principal Components Analysis

• PCA can be thought as fitting an n-dimensional ellipsoid to the data, such that each axis of the ellipsoid represents a principal component, i.e. an axis of maximal variance

PCA

X1

X2

PCA – the algorithm• Step A – subtract the mean of each data dimension, thus

move all data-points to be centered around the origin

• Step B – calculate the covariance matrix of the data

PCA – the algorithm• Step C – calculate the eigenvectors and the eigenvalues of

the covariance matrix

• The eigenvectors of the covariance matrix are orthonormal (see below)• The eigenvalues tell us the ‘amount of variance’ of the data along each

specific new dimension/axis (eigenvector)

PCA – the algorithm• Step D – sort the eigenvalues in descending order

• Eigvec #1, which is correlated with Eigval #1, is the 1st principal component – i.e. the (new) axis with highest variance

• Step E (optional) – take only ‘strong’ Principal Components

• Step F – project the original data on the newly created base (the PCs, the eigenvectors) to get a rotated, translated coordinate system correlated with highest variance per each axis

PCA – the algorithm• For dimensionality reduction – take only some of the new

principal components to represent the data, accountable for the highest amount of variance (hence, data)