Tensor’dataanalysis - Georgia Institute of...
Transcript of Tensor’dataanalysis - Georgia Institute of...
![Page 1: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/1.jpg)
Tensor data analysis
Machine Learning II: Advanced Topics CSE 8803ML, Spring 2012
Mariya Ishteva
![Page 2: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/2.jpg)
Scalars
2
![Page 3: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/3.jpg)
Vectors
3
![Page 4: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/4.jpg)
Matrices
4
![Page 5: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/5.jpg)
Tensors
5
![Page 6: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/6.jpg)
Some clarificaFons
! Nth order tensor
! DefiniFon ! An element of the tensor product of N vector spaces
! When the choice of basis is implicit we think of a tensor as its representaFon as an N-‐way array
! Difficult to visualize ! We will talk mainly about 3rd order tensors ! Results are extendable to higher orders
! NotaFon ! Not standardized
6
![Page 7: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/7.jpg)
ApplicaFons ! ProbabiliFes
! matrix: joint/cond. probability table of two variables ,
! tensor: joint/cond. probability table of a set of variables
! Text mining: ! matrix: document – term
! tensor: document – term – year – author
! Social networks: ! matrix: find communiFes
! tensor: monitor the change of the community over Fme
! CollaboraFve filtering ! matrix: user – item
! tensor: user – item – Fme
! Signal processing: Example 1 & 3
! Chemometrics: Example 2
! Etc. 7
![Page 8: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/8.jpg)
Example 1: EpilepFc seizure onset localizaFon
! Electrodes
8
! Electroencephalogram
![Page 9: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/9.jpg)
EpilepFc seizure onset localizaFon: CP model
9
! CP model / Canonical decomposiFon / Parafac
![Page 10: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/10.jpg)
Example 2: Chemometrics
! [R. Bro, KVL, Denmark] 10
![Page 11: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/11.jpg)
Fluorescence spectroscopy
! Demo: ! R. Bro, KVL, Denmark ! hbp://www.models.kvl.dk
11
![Page 12: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/12.jpg)
Example 3: Parameter esFmaFon
12 ! [More details later]
![Page 13: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/13.jpg)
Outline
! MoFvaFon ! Basic concepts
! Basic tensor decomposiFons
Next lecture:
! Other useful decomposiFons ! Local minima
! Tensors and graphical models
13
![Page 14: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/14.jpg)
Outline
! MoFvaFon ! Basic concepts
! Rank and mulFlinear rank
! Matrix representaFons ! Tensor – matrix mulFplicaFon
! Basic tensor decomposiFons
14
![Page 15: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/15.jpg)
Matrix rank
! # linearly independent rows
! # linearly independent columns ! # rank-‐1 terms
! Singular value decomposiFon (SVD)
15
![Page 16: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/16.jpg)
Tensor ranks
16
![Page 17: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/17.jpg)
Matrix representaFons of a tensor
! mulFlinear rank: (rank(A(1)), rank(A(2)), rank(A(3))) 17
![Page 18: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/18.jpg)
Tensor-‐matrix mulFplicaFon
18
![Page 19: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/19.jpg)
Useful matrix operaFons
! Kronecker product
! Khatri-‐Rao product ! Column-‐wise Kronecker product
! Let
19
![Page 20: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/20.jpg)
Scalar product, Frobenius norm, contracFon
! Scalar product
! Frobenius norm
! ContracFon
20
4th order tensor
![Page 21: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/21.jpg)
Outline
! MoFvaFon ! Basic concepts
! Basic tensor decomposiFons ! CP / CANDECOMP / PARAFAC
! MulFlinear SVD
21
![Page 22: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/22.jpg)
Basic decomposiFons
22
![Page 23: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/23.jpg)
CP / Canonical decomposiFon / PARAFAC
23
! Polyadic form (Hitchcock, 1927)
! CANDECOMP = Canonical DecomposiFon (Carroll & Chang, 1970) ! PARAFAC = Parallel Factors (Harshman, 1970)
! Vectors are not necessarily orthogonal
![Page 24: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/24.jpg)
CP: uniqueness
24
! Kruskal rank k(A): max k, s.t. any k columns are linearly independent
! Uniqueness
! Up to permutaFon of the terms
! Up to scaling of the factors
! Sufficient condiFon:
! Note: matrix factorizaFons are not unique
€
A = [a1 a2 a R]B = [b1 b2 b R]C = [c1 c2 c R]
![Page 25: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/25.jpg)
ProperFes of tensor rank ! Not bounded by the dimensions of the tensor
! CompuFng R: NP-‐hard problem
! Maximum rank, typical rank
! Best rank approximaFon: ill-‐posed problem
! Rank over ≤ rank over ! Example
25
T. Kolda & B. Bader, Tensor decomposiFons and applicaFons SIAM Review, V. 51, # 3, 2009
![Page 26: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/26.jpg)
CompuFng CP
! Many algorithms
! AlternaFng least squares: ! Repeat unFl convergence:
! OpFmize A
! OpFmize B ! OpFmize C
26
![Page 27: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/27.jpg)
[Example 1]: EpilepFc seizure onset localizaFon
27
! CP model / Canonical decomposiFon / Parafac
![Page 28: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/28.jpg)
[Example 2]: Chemometrics
! [R. Bro, KVL, Denmark] 28
![Page 29: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/29.jpg)
Outline
! MoFvaFon ! Basic concepts
! Basic tensor decomposiFons ! CP / CANDECOMP / PARAFAC
! MulFlinear SVD
29
![Page 30: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/30.jpg)
MLSVD decomposiFon
! 3MFA/Tucker3 = Three-‐mode factor analysis (Tucker, 1966)
! MLSVD = MulFlinear SVD (De Lathauwer, 2000)
! normalized Tucker decomposiFon
! U(n): orthogonal ! All-‐orthogonality:
! Ordering:
30
![Page 31: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/31.jpg)
MLSVD decomposiFon
! Not unique
! ComputaFon
! SVDs of the matrix representaFons A(1), A(2), A(3) U(1),U(2),U(3)
! A, U(1),U(2),U(3) S
31
€
Α = S •1U(1) •2U
(2) •3U(3)
= (S •1 X−1 •2 Y
−1 •3 Z−1) •1U
(1)X •2U(2)Y •3U
(3)Z
€
S = A •1 (U(1))T •2 (U
(2))T •3 (U(3))T
![Page 32: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/32.jpg)
Low mulFlinear rank approximaFon
32
![Page 33: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/33.jpg)
Low mulFlinear rank approximaFon
33
![Page 34: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/34.jpg)
Higher-‐order orthogonal iteraFon
34
![Page 35: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/35.jpg)
Best rank-‐(R1,R2,R3) approx.: applicaFons
! ApplicaFon areas ! Chemometrics ! Biomedical signal processing
! TelecommunicaFons
35
! Tool for ! Dimensionality reducFon ! Signal subspace esFmaFon
![Page 36: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/36.jpg)
Dimensionality reducFon
36
![Page 37: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/37.jpg)
[Example 3]: Parameter esFmaFon
37
![Page 38: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/38.jpg)
[Example 3]: Parameter esFmaFon
! HO-‐HTLSstack algorithm
38
![Page 39: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/39.jpg)
[Example 3]: Parameter esFmaFon
! Tensors can be ill-‐condiFoned in one mode but well-‐condiFoned in other modes. Not possible in matrix case
39
![Page 40: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/40.jpg)
Matlab toolboxes
! Tensor Toolbox ! hbp://csmr.ca.sandia.gov/~tgkolda/TensorToolbox/ ! B. Bader, T. Kolda and others
! N-‐way toolbox ! hbp://www.models.life.ku.dk/nwaytoolbox ! R. Bro and C. Andersson
40
![Page 41: Tensor’dataanalysis - Georgia Institute of Technologylsong/teaching/8803ML/lecture_tensors1.pdf · Tensor’dataanalysis ’ Machine’Learning’II:’Advanced’Topics’ CSE’8803ML,’Spring’2012’](https://reader033.fdocuments.net/reader033/viewer/2022041421/5e1f26f2e9fb9f34bc649f03/html5/thumbnails/41.jpg)
Outline
! MoFvaFon ! Basic concepts
! Basic tensor decomposiFons
Next lecture:
! Other useful decomposiFons ! Local minima
! Tensors and graphical models
41