Augmented Tensor Decomposition with Stochastic Alternating ...
tensor-decomposition
-
Upload
kenta-oono -
Category
Technology
-
view
30.524 -
download
0
Transcript of tensor-decomposition
.
Introduction to TensorsPFI Seminar
Kenta OONO
2012/07/11
Kenta OONOIntroduction to Tensors
Self Introduction
• Kenta OONO (Twitter:@delta2323 )• PSS(Professional and Support Service) Division• Jubatus Project
• msgpack-idl, generation of clients• Survey and Prototyping Team• 2010 PFI Summer Intern → 2011 Engineer
Kenta OONOIntroduction to Tensors
Agenda
• What is Tensor?• Decomposition of Matrices and Tensors• Symmetry Parametrized by Young Diagram
Kenta OONOIntroduction to Tensors
Agenda
• What is Tensor?• Decomposition of Matrices and Tensors• Symmetry Parametrized by Young Diagram
Kenta OONOIntroduction to Tensors
Vector as a Special Case of Tensor
Consider a n dimensional vector (Below is the case n = 3)
v =
749
(1)
When we write this vector as v = (vi), we have v1 = 7, v2 = 4,and v3 = 9. As v has One index, this vector is regarded as Rank 1Tensor.
Note• We use 1-index notation, not 0-index.• We somtimes use rank in a different meaning, (also in the
context of tensors).
Kenta OONOIntroduction to Tensors
Matrix as a Special Case of Tensor
Consider a n × n Matrix (Below is the case n = 3).
A =
7 10 34 −1 −29 4 5
(2)
We write this matrix as A = (aij)1≤i ,j≤3.• e.g. a11 = 7, a23 = −2, a32 = 4.
A has Two indices (namely i and j) and these indeices run through1 to 3. A can be regarded as Rank 2 Tensor.
→ What happens if the number of indices are not just 1 or 2 ?
Kenta OONOIntroduction to Tensors
An Example of Rank 3 Tensor
Consider the set of real numbers X = (xijk)1≤i ,j,k≤3. As i , j , k runthrouth 1 to 3 independently, this set contains 3 × 3 × 3 = 27elements.If we fix the value of i , we can write this set in a matrix form.
x1∗∗ =
7 10 34 −1 −29 4 5
, x2∗∗ =
10 2 13 −1 08 1 4
, x3∗∗ =
5 −10 18−4 1 −29 −4 −10
(3)
Note• In some field, we concatenate these matrix and name it X(1)
• This is called Flattening or Matricing of X .
Kenta OONOIntroduction to Tensors
Is This a Multidimensional Array?
We can realize a tensor as a multiple dimensional array in mostprogramming languages.
PseudoCode(C like):int[3][3][3] x;x[1][1][2] = -1;
Kenta OONOIntroduction to Tensors
Inner Product of Tensors
Let X = (xijk), Y = (yijk) be two rank 3 tensors and G = (g ij) bea symmetric (i.e. g i j = g j i) and positive definite matrix. We canintroduce an inner product of X and Y by:
< X , Y >=n∑
a,b,c,i ,j,k=1gaigbjgckxabc yi jk (4)
Note:• We can similarly define an inner product of two arbitrary rank
tensor• X and Y must have same rank.
Kenta OONOIntroduction to Tensors
An Example of Inner Product (Vector)
Let’s consider the case of rank 1 tensor ( = vector). We can writeX = (xi), Y = (yi). And we set G = (g ij) an identity matrix:
G =
1 · · · 0... 1
...0 · · · 1
(5)
In this case, this definition of inner product reduce to ordinal innerproduct of two vectors, namely,
< X , Y >=∑a,i
gaixayi =∑
ixiyi (6)
Kenta OONOIntroduction to Tensors
An Example of TensorSuppose we have a (smooth) function f : R3 → R. We can derivetensors of arbitrary rank from this function.Taking Gradient, we obtain rank 1 tensor. (Please replace (1, 2,3) with (x , y , z) and vice versa.
∇f = (∇i f ) =
∂f∂x∂f∂y∂f∂z
(7)
Similarly, Hessian is an example of rank 2 tensor.
H(f ) = (Hij(f )) =
∂2f
∂x∂x∂2f
∂x∂y∂2f
∂x∂z∂2f
∂y∂x∂2f
∂y∂y∂2f
∂y∂z∂2f
∂z∂x∂2f
∂z∂y∂2f
∂z∂z
(8)
Kenta OONOIntroduction to Tensors
Supersymmetry Property
By taking derivative, we can create an arbitrary rank of tensors.These tensors are (Super)Symmetric in the sense thatinterchange of any two indices remains the tensor identical. Forexample,
fxyz = fxzy = fyxz = fyzx = fzxy = fzyx . (9)
Note:• Afterward, we write this symmetry as .
Kenta OONOIntroduction to Tensors
Agenda
• What is Tensor?• Decomposition of Matrices and Tensors• Symmetry Parametrized by Young Diagram
Kenta OONOIntroduction to Tensors
Decompose!
It is fundamental in mathematics to decompose some object intosimpler, easier-to-handle objects.
• Fourier Expansion : f =∑n∈Z
an exp(√
−1nx)
• Legendre Polynomial : P =∑
kakPk
• where Pk(x) =1
k!2kdk
dxk (x2 − 1)k
• Jordan-Holder : e = G0 ◁ G1 ◁ G2 ◁ · · · ◁ Gn−1 ◁ Gn = G
Kenta OONOIntroduction to Tensors
Decomposition of Matrix
There are many theories regarding the decomposition (or,factorization) of matrices.
• SVD• LU• QR• Cholesky• Jordan• Diagonalization ...
Although we have theories of factorization of Tensors (e.g. TuckerDecomposition, higher-order SVD etc.) We do NOT go thisdirection. Instead, we consider decomposition of matrix intoSummation of matrix.
Kenta OONOIntroduction to Tensors
Example: Projection to Axis
Kenta OONOIntroduction to Tensors
Example of Decomposition of Matrix
We can decompose matrix into Symmetric part andAntisymmetric part.
Example:7 10 34 −1 −29 4 5
=
7 7 67 −1 16 1 5
+
0 3 −3−3 0 −33 3 0
Symmetric Part Antisymmetric Part
A Asym Aanti
(10)
Kenta OONOIntroduction to Tensors
How to Calculate Sym. and Antisym. Part
We can calculate the symmetic and antisymmetric part by simplecalculation (Exercise!).
Asym =12
7 10 3
4 −1 −29 4 5
+
7 4 910 −1 43 −2 5
asymij =
12(ai j + aj i)
(11)
Aanti =12
7 10 3
4 −1 −29 4 5
−
7 4 910 −1 43 −2 5
aantiij =
12(ai j − aj i)
(12)
Kenta OONOIntroduction to Tensors
Picture of Projection to Sym. and Antisym. Part
Kenta OONOIntroduction to Tensors
Notation Using Young Diagram
We can symbolize symmetry and antisymmetry with YoungDiagram
7 10 34 −1 −29 4 5
=
7 7 67 −1 16 1 5
+
0 3 −3−3 0 −33 3 0
A Asym Aanti
(13)
The number of boxes indicates rank of the tensor (one box forvector and two boxes for matrix).
Kenta OONOIntroduction to Tensors
Agenda
• What is Tensor?• Decomposition of Matrices and Tensors• Symmetry Parametrized by Young Diagram
Note:• From now on, we concentrate on Rank 3 Tensors (i.e.
k = 3).• And we assume that n = 3, that is, indices run from 1 to 3.
Kenta OONOIntroduction to Tensors
Symmetrizing Operator
We consider the transformation of tensor:
T : X = (xijk) 7→ X ′ = (x ′ijk), (14)
where x ′ijk = 1
6 (xi jk + xikj + xj ik + xjki + xkij + xkji)
= x(i jk).(15)
Kenta OONOIntroduction to Tensors
Young Diagram and Symmetry of Tensor (Sym. Case)
Let X = (x123) be a tensor of rank 3, we call X Has a Symmetryof , if interchange of any of two indices doesn’t change eachentry of X .
Example:• If X has a symmetry of , then
x112 = x121 = x211x123 = x132 = x213 = x231 = x312 = x321 etc...
(16)
• Symmetric matrices have a symmetry of
Kenta OONOIntroduction to Tensors
Property of T
• For all X , T (X ) has symmetry of .• If X has symmetry of , then T (X ) = X .• For all X , T (T (X )) = T (X ).
Kenta OONOIntroduction to Tensors
Antisymmetrizing OperatorNext, we consider Antisymmetrization of tensors:
T : X = (xijk) 7→ X ′ = (x ′ijk), (17)
where x ′ijk = 1
6 (xi jk − xikj − xj ik + xjki + xkij − xkji)
= x[i jk].(18)
Kenta OONOIntroduction to Tensors
Young Diagram and Symmetry of Tensor (Antisym. Case)
Let X = (x123) be a tensor of rank k, we call X has a symmetry of
if interchange of any of two indices change only the sign ofeach entry of X .
Example.
• If X has a symmetry of , then• x123 = −x132 = −x213 = x231 = x312 = −x321• x112 = 0, because x112 must be equal to −x112.
• Antisymmetric matrix has a symmetry of
Kenta OONOIntroduction to Tensors
Orthogonality of ProjectionT and T are orthogonal in the sense
T ◦ T (X ) = T ◦ T (X ) = 0. (19)
Kenta OONOIntroduction to Tensors
Counting Dimension
How many dimension do the vector space consisting of tensors ...
• with symmetry have ?
• with symmetry have ?
Kenta OONOIntroduction to Tensors
Dimensions of each subvector spaces.If n = 3 (that is, i , j and k runs through 1 to 3), then eachdimension is as follows.
But dimension of total space is 3 × 3 × 3 = 27. This is larger than1 + 10 = 11.
Kenta OONOIntroduction to Tensors
Young Diagram
Young Diagram is an arrangent of boxes of the same size so that:• two adjacent boxes share one of their side.• the number of boxes in each row is non-increasing.• the number of boxes in each column is non-increasing.
Here is an example of Young Diagram with 10 boxes.
English Notation French Notation
(20)
Kenta OONOIntroduction to Tensors
Decomposition of Tensor (of Rank 3)
We have three types of Young Diagram which have three boxes,namely,
, , andSymmetric Antisymmetric ???
(21)
→ What symmetry does represent?
Kenta OONOIntroduction to Tensors
Coefficients of Symmetrizers
We list coneffients of each operator in a tabular form.
Coefficient of xijki jk ikj j ik jki ki j kji
1/6 1/6 1/6 1/6 1/6 1/6
1/6 -1/6 -1/6 1/6 1/6 -1/6
Sum 1
Kenta OONOIntroduction to Tensors
Coefficients of Symmetrizers
Coefficient of xijki jk ikj j ik jki ki j kji
1/6 1/6 1/6 1/6 1/6 1/6
1/6 -1/6 -1/6 1/6 1/6 -1/6
4/6 -2/6 -2/6
Sum 1
Kenta OONOIntroduction to Tensors
Symmetrizer T
We consider the transformation of tensor:
T : X = (xijk) 7→ X ′ = (x ′ijk), (22)
where x ′ijk = 1
3 (2xijk − xjki − xkij).
Kenta OONOIntroduction to Tensors
Property of T
• For all X , T (X ) has symmetry of .
• If X has symmetry of , then T (X ) = X .
• For all X , T(
T (X )
)= T (X ).
Kenta OONOIntroduction to Tensors
Tensors with Symmetry of
Let X = (xijk) be a tensor of rank 3. We say X Has a Symmetryof , if
xijk =13 (2xijk − xjki − xkij) (23)
for all 1 ≤ i , j , k ≤ n.
Kenta OONOIntroduction to Tensors
Where This Equation ”Symmetric” ?
Remember that X has a symmetry of , if
xijk =13 (2xijk − xjki − xkij) (24)
for all 1 ≤ i , j , k ≤ n.
This equation can be transormed as follows :
xijk =13 (2xijk − xjki − xkij)
⇔ 3xijk = 2xijk − xjki − xkij⇔ xi jk + xjki + xkij = 0
(25)
Kenta OONOIntroduction to Tensors
Projection to Each Symmetrized Tensors
Kenta OONOIntroduction to Tensors
Decomposition of Higher Rank Tensors
Higher rank tensors are also decompose into symmetric tensorsparametrized by Young Diagram.
Rn×n×n×n×n =⊕ ⊕
⊕ ⊕ ⊕ ⊕(26)
Kenta OONOIntroduction to Tensors
Summary
• Tensors as a Generalization of Vectors and Matrices.• Decomposition of Matrices and Tensors• Symmetry of Tensors Parametrized by Young Diagrams.
Kenta OONOIntroduction to Tensors
References
• T G. Kolda and B W. Bader, Tensor Decompositions andApplications
• 2009, SIAM Review Vol. 51, No 3, pp 455-500• D S. Watkins, Fundamentals of Matrix Computations 3rd. ed.
• 2010, A Wiley Series of Texts, Monographs and Tracts• W. Fulton and J Harris, Representation Theory
• 1991, Graduate Texts in Mathematics Readins in Mathematics.• And Google ”Representation Theory”, ”Symmetric Group”,
”Young Diagram” and so on.
Kenta OONOIntroduction to Tensors