From Matrix to Tensor:The Transition to Numerical Multilinear Algebra
Lecture 3. Transpositions, KroneckerProducts, and Contractions
Charles F. Van Loan
Cornell University
The Gene Golub SIAM Summer School 2010Selva di Fasano, Brindisi, Italy
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Where We Are
Lecture 1. Introduction to Tensor Computations
Lecture 2. Tensor Unfoldings
Lecture 3. Transpositions, Kronecker Products, Contractions
Lecture 4. Tensor-Related Singular Value Decompositions
Lecture 5. The CP Representation and Rank
Lecture 6. The Tucker Representation
Lecture 7. Other Decompositions and Nearness Problems
Lecture 8. Multilinear Rayleigh Quotients
Lecture 9. The Curse of Dimensionality
Lecture 10. Special Topics
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
What is this Lecture About?
Preview
Now we turn our attention to the engagement of thesemultidimensional data objects in two elementary operations.
Transposition. While there is only one way transpose a matrixthere are an exponential number of ways to transpose an order-dtensor. This will require (surprise surprise) more facility withmulti-indexing.
Contraction. The contraction of two tensors can be regarded as ageneralization of matrix-matrix multiplication. Analogous to howthe serious business of matrix decompositions builds on matrixmultiplication, we will find that tensor contractions have afoundational role to play in tensor computations.
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
What is this Lecture About?
More “Bridging the Gap”
The Kronecker product helps bridge the gap between matrixcomputations and tensor computations. For example, thecontraction between two tensors can sometimes be “reshaped” intoa matrix computation that involves Kronecker products.
So in advance of our introduction to tensor contractions, we willget familiar with this all-important matrix operation and some ofits nearby “cousins.”
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Tensor Transposition
Example: A(1:4, 1:5, 1:6) → B(1:5, 1:6, 1:4)
A = randn([ 4 5 6]);B = zeros([5 6 4]);for i1=1:4
for i2 = 1:5for i3 = 1:6
B(i2,i3,i1) = A(i1,i2,i3);end
endend
How to say this: B = A< [ 2 3 1 ] >.
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Tensor Transposition: The 3rd Order Case
Six possibilities...
If A ∈ IRn1×n2×n3 , then there are 6 = 3! possible transpositionsidentified by the notation A< [i j k] > where [i j k] is a permutationof [1 2 3]:
B =
A< [1 2 3] >
A< [1 3 2] >
A< [2 1 3] >
A< [2 3 1] >
A< [3 1 2] >
A< [3 2 1] >
=⇒
bijk
bikj
bjik
bjki
bkij
bkji
= aijk
for i = 1:n1, j = 1:n2, k = 1:n3.
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Tensor Transposition: General Case
Definition
If A ∈ IRn1×···×nd and p = [p1, . . . , pd ] is a permutation of theindex vector 1:d , then A<p> ∈ IRnp1×···×npd denotes thep-transpose of A and is defined by
A<p>(jp1 , . . . , jpd) = A(j1, . . . , jd)
where 1 ≤ jk ≤ nk for k = 1:d .
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Matlab Tensor Toolbox: Transposition
% If A is an order -d tensor and
% p is a permutation of 1:d, then
% the following assigns the p-transpose
% of A to B...
B = permute(A,p)
Problem 3.1. Any square matrix can be written as the sum of a symmetricmatrix and a skew-symmetric matrix:
A =
„A + AT
2
«+
„A− AT
2
«Formulate a generalization of this result for order-3 tensors and write aMatlab function that embodies your idea.
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Tensor Notation: Subscript Vectors (Review)
Reference
If A ∈ IRn1×···×nd and i = (i1, . . . , id) with 1 ≤ ik ≤ nk for k = 1:d ,then
A(i) ≡ A(i1, . . . , ik)
We say that i is a subscript vector. Bold font will be useddesignate subscript vectors.
Bounds
If L and R are subscript vectors having the same dimension, thenL ≤ R means that Lk ≤ Rk for all k.
Special Cases
A subscript vector of all ones is denoted by 1. (Dimension clearfrom context.) If N is an integer, then N = N · 1.
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Tensor Transposition: General Case
Definition
If A ∈ IRn1×···×nd and p = [p1, . . . , pd ] is a permutation of theindex vector 1:d , then A<p> ∈ IRnp1×···×npd denotes thep-transpose of A and is defined by
A<p>(j(p)) = A(j) 1 ≤ j ≤ n.
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Problem 3.2. Verify that if p and q are both permutations of 1:d , then`A<p>´<q> = A<p(q)>
where A ∈ IRn1×···×nd .
Problem 3.3. Suppose A ∈ IRn1×···×nd and that B = A[ p ] where p is apermutation of 1:d . Specify a permutation matrix P so thatB(k) = A(p(k))P.
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
What is a Kronecker Product?
Definition
B ⊗ C is a block matrix whose ij-th block is bijC .
An Example...[b11 b12
b21 b22
]⊗ C =
[b11C b12C
b21C b22C
]
Replicated Block Structure
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
What Is a Kronecker Product?
There are three ways to regard A = B ⊗ C ...
If
A =
b11 · · · b1n...
. . ....
bm1 · · · bmn
⊗
c11 · · · c1q...
. . ....
cp1 · · · cpq
then
(i). A is an m-by-n block matrix with p-by-q blocks.
(ii). A is an mp-by-nq matrix of scalars.
(iii). A is an unfolded 4-th order tensor A ∈ IRp×q×m×n:
A(i1, i2, i3, i4) = B(i3, i4)C (i1, i2)
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
All Possible Entry-Entry Products
4× 9 = 36
[b11 b12
b21 b22
]⊗
c11 c12 c13
c21 c22 c23
c31 c32 c33
=
b11c11 b11c12 b11c13 b12c11 b12c12 b12c13
b11c21 b11c22 b11c23 b12c21 b12c22 b12c23
b11c31 b11c32 b11c33 b12c31 b12c32 b12c33
b21c11 b21c12 b21c13 b22c11 b22c12 b22c13
b21c21 b21c22 b21c23 b22c21 b22c22 b22c23
b21c31 b21c32 b21c33 b22c31 b22c32 b22c33
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Hierarchical
Kronecker Products of Kronecker Products
B ⊗ C ⊗ D=[
b11 b12
b21 b22
]⊗
c11 c12 c13 c14
c21 c22 c23 c24
c31 c32 c33 c34
c41 c42 c43 c44
⊗
d11 d12 d13
d21 d22 d23
d31 d32 d33
A 2-by-2 block matrix whose entries are 4-by-4 block matriceswhose entries are 3-by-3 matrices.
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Kronecker Product Properties
Some Basic Facts...
(B ⊗ C )T = BT ⊗ CT
(B ⊗ C )−1 = B−1 ⊗ C−1
(B ⊗ C )(D ⊗ F ) = BD ⊗ CF
B ⊗ (C ⊗ D) = (B ⊗ C )⊗ D
Note that B ⊗ C 6= C ⊗ B. However, it can be shown that
C ⊗ B = PT (B ⊗ C )Q
where P and Q are perfect shuffle permutations. More later.
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Kronecker Products and Structure
Structured Factors Produce a Structured Product...
B and C B ⊗ C
NonsingularLower(Upper) triangularBandedSymmetricPositive DefiniteStochasticToeplitzPermutationOrthogonal
NonsingularLower(Upper) TriangularBlock BandedSymmetricPositive DefiniteStochasticBlock ToeplitzPermutationOrthogonal
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Kronecker Products and Data Sparsity
Example 1.
If B,C ∈ IRm×m andA ≈ B ⊗ C
then we are using O(m2) numbers ( the bij and cij) torepresent/approximate O(m4) data (the aij .)
Example 2.
If Bij ∈ IRm×m and
A ≈r∑
k=1
Bk,1 ⊗ Bk,2 ⊗ · · · ⊗ Bk,d
then we are using rdm2 numbers to represent/approximate amatrix with m2d entries.
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Kronecker Products and Fast Matrix-Vector Multiplication
A Basic Reshaping Result...
Y = CXBT ⇔ vec(Y ) = (B ⊗ C )vec(X )
In Practice...
function y = KronTimesVector(B,C,y)% y = kron(B,C)*x[mb,nb] = size(B);[mc,nc] = size(C);Y = C*reshape(x,nc,nb)*B’;y = reshape(Y,mc*mb,1);
If both matrices are n-by-n, then work is O(n3). Ordinarily, an n2-by-n2
matrix-vector product would be O(n4).
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Problem 3.4. How would you compute the matrix-vector producty = (Ip ⊗ A⊗ Ir )x where A ∈ IRq×q and x ∈ IRpqr? Assess the amount ofwork.
Problem 3.5. How would you compute the matrix-vector product
y = (A1 ⊗ A2 ⊗ A3)x where Ai ∈ IRn×n and x ∈ IRn3
?. Hint:
(A1 ⊗ A2 ⊗ A3) = (A1 ⊗ In2)(In ⊗ A2 ⊗ In)(In2 ⊗ A3)
Assess the amount of work.
Problem 3.6. How would you compute the matrix-vector product
y = (A1 ⊗ · · · ⊗ Ad)x where Ai ∈ IRn×n and x ∈ IRn3
?. Assess the amountof work.
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Kronecker Products and Matrix Factorizations
Just take the Kronecker Product of the Factors...
LU: (PB ⊗ PC )(B ⊗ C ) = (LB ⊗ LC )(UB ⊗ UC )
Cholesky: B ⊗ C = (LB ⊗ LC )(LB ⊗ LC )T
Schur: (QB ⊗ QC )T (B ⊗ C )(QB ⊗ QC ) = TB ⊗ TC
QR: B ⊗ C = (QB ⊗ QC )(RB ⊗ RC )
SVD: B ⊗ C = (UB ⊗ UC )(ΣB ⊗ ΣC )(VB ⊗ VC )T
There are some annoying details if the matrices are rectangular, e.g.,
(ΣB ⊗ ΣC ) needs to be permuted to obtain a true diagonal form.
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Problem 3.7. How would you minimize ‖ (A1 ⊗ · · · ⊗ Ad)x − b ‖2 giventhat each A1, . . . , Ad ∈ IRm×n has rank ri < n?
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Perfect Shuffle Permutations
Transposition is a Permutation
This: [b11 b12 b13
b21 b22 b23
]=
a11 a12
a21 a22
a31 a32
T
Is the same as this:
b11
b21
b12
b22
b13
b23
=
1 0 0 0 0 00 0 0 1 0 00 1 0 0 0 00 0 0 0 1 00 0 1 0 0 00 0 0 0 0 1
a11
a21
a31
a12
a22
a32
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Perfect Shuffle Permutations
Formal Definition
If n = pr then the mod-p perfect shuffle permutation Sp,r isdefined by
Sp,rx =
x(1:r :n)x(2:r :n)
...x(r :r :n)
where x ∈ IRn.
Implication for High Performance Computing: Matrix transpose
involves non-unit stride access to matrix elements.
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Problem 3.8. (a) Think about the perfect shuffling of a 52-card deck andexplain why we use the term “perfect shuffle” when referring to Sp,r . (b)Show that Sr,p = ST
p,r . (c) Explain why Sp,r is aptly called the “mod-p sortpermutation.”
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Perfect Shuffle Permutations
The Connection to Matrix Transpose...
If X ∈ IRp×r and Y = XT , then
vec(Y ) = Sp,r · vec(X )
The Connection to Kronecker Products...
If A1 ∈ IRm1×n1 and A2 ∈ IRm2×n2 then
A2 ⊗ A1 = STm1,m2
· (A1 ⊗ A2) · Sn1,n2
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Perfect Shuffle Permutations
The Connection to Tensor Transpose
If A ∈ IRn1×n2×n3 , then
B =
A< [1 2 3] >
A< [1 3 2] >
A< [2 1 3] >
A< [2 3 1] >
A< [3 1 2] >
A< [3 2 1] >
=⇒ vec(B) =
In1n2n3
Sn2,n3⊗ In1
In3⊗ Sn1,n2
Sn1,n2n3
Sn1n2,n3
(Sn1,n2⊗ In3)Sn1n2,n3
·vec(A)
Implication for High Performance Computing: Tensor transpose
involves non-unit stride access to matrix elements.
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Problem 3.9. Suppose A ∈ IRn1×···×nd , N = n1 · · · nd , and that p is apermutation of 1:d that involves swapping a single pair of indices, e.g.,[1 4 3 2 5]. Determine a permutation matrix P ∈ IRN×N so that ifB = A< p >, then
vec(B) = P · vec(A).
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Tensor Contractions
Definition
A contraction between two tensors produces a third tensor througha summation process.
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Linear Algebra Operations in Tensor Contraction Terms
The Contraction of an Order-1 Tensor with an Order-1 Tensor
c =n∑
k=1
a(k)·b(k) (Dot Product)
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Linear Algebra Operations in Tensor Contraction Terms
The Contraction of an Order-2 Tensor and an Order-1 Tensor
c(i) =n∑
k=1
A(i , k)·b(k) (Matrix-Vector Product)
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Linear Algebra Operations in Tensor Contraction Terms
The Contraction of an Order-2 Tensor and an Order-2 Tensor
C (i , j) =n∑
k=1
A(i , k)·B(k, j) (Matrix-Matrix Product)
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Tensor Contractions: Subscript Level
Example 1. (A Single-Index Contraction)
C(i , j , α3, α4, β3, β4, β5) =∑k
A(i , k, α3, α4) · B(k, j , β3, β4, β5)
⇑7th
Order
⇑4th
Order
⇑5th
Order
A = A(1:n1, 1:n2, 1:n3, 1:n4)
B = B(1:m1, 1:m2, 1:m3, 1:m4, 1:m5)
⇒ C = C(1:n1, 1:m2, 1:n3, 1:n4, 1:m3, 1:m4, 1:m5)
Order(C) = Order(A) + Order(B) - 2
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Tensor Contractions: Subscript Level
Example 2. (A Double-Index Contraction)
C(i , j , p, q) =∑k
∑s
A(i , s, p, k) ∗ B(s, j , k, q)
⇑4th
Order
⇑4th
Order
⇑4th
Order
Order(C) = Order(A) + Order(B) - 2 - 2
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Tensor Contractions: Summation Notation
Summation
If L = (L1, . . . , Lk) and R = (R1, . . . ,Rk) then
R∑i=L
≡R1∑
i1=L1
R2∑i2=L2
· · ·Rk∑
ik=Lk
Example: Frobenius Norm of A ∈ IRn1×···×nd
‖ A ‖F =
√√√√ n∑i=1
A(i)2
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Tensor Contractions: Subscript Level
Example 3. (A 4-Index Contraction)
C(γ1, γ2, γ3, γ4)
=m∑
i=1
A(i1, i2, i3, i4)·W (i1, γ1)·X (i2, γ2)·Y (i3, γ3)·Z (i4, γ4)
⇑4th
Order
⇑2nd
Order
⇑2nd
Order
⇑2nd
Order
⇑2nd
Order
Order(C) = Order(A) + Order(W) + Order(X )+ Order(Y)+Order(Z) - 2 - 2 - 2 - 2
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Towards a High-Performance Contraction Framework
In Reality...
Tensor computations are matrix computations. A contraction canbe regarded as a generalization of matrix-matrix multiplication.
The Challenge...
Expose the underlying matrix multiplications with good notation.
Two Reasons:
1. To facilitate tensor-level thinking.2. To facilitate level-3 BLA exploitation
Let us revisit the five examples rephrasing them to expose the underlyingmatrix products.
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Towards a High-Performance Contraction Framework
The Block Vec Unfolding
Blockvec(A) =
266666666666664
A( : , : , 1, 1, 1)
A( : , : , 2, 1, 1)
A( : , : , 1, 2, 1)
A( : , : , 2, 2, 1)
A( : , : , 1, 1, 2)
A( : , : , 2, 1, 2)
A( : , : , 1, 2, 2)
A( : , : , 2, 2, 2)
377777777777775A ∈ IRn1×n2×2×2×2
The Block-Vector Product
»F1
F2
–⊗
24 G1
G2
G3
35 =
26666664F1G1
F1G2
F1G3
F2G1
F2G2
F2G3
37777775⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Tensor Contractions: Matrix Level
Example 1.
C(i , j , α3, α4, β3, β4, β5) =∑k
A(i , k, α3, α4)B(k, j , β3, β4, β5)
As a collection of matrix-matrix products...
C( : , : , α3, α4, β3, β4, β5) = A( : , : , α3, α4)B( : , : , β3, β4, β5)
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Tensor Contractions: Matrix Level
As a Block Vector Product...
C( : , : , 1, 1, 1, 1, 1)C( : , : , 1, 1, 2, 1, 1)C( : , : , 1, 1, 1, 2, 1)
...C( : , : , 2, 2, 1, 2, 2)C( : , : , 2, 2, 2, 2, 2)
=
A( : , : , 1, 1)A( : , : , 2, 1)A( : , : , 1, 2)A( : , : , 2, 2)
⊗
B( : , : , 1, 1, 1)B( : , : , 2, 1, 1)B( : , : , 1, 2, 1)B( : , : , 2, 2, 1)B( : , : , 1, 1, 2)B( : , : , 2, 1, 2)B( : , : , 1, 2, 2)B( : , : , 2, 2, 2)
32 matrix-matrix products
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Problem 3.10. Suppose A ∈ IRn1×···×nd and B ∈ IRm1×···×me withn2 = m1. If
C(α1, β2, α3, . . . , αd , β3, . . . , βe) =
n2Xk=1
A(α1, k, α3, . . . , αd) · B(k, β2, . . . , βe)
then we refer to C as the canonical contraction of A and B. Write aMatlab function C = CanonContract(A,B) that computes this tensorgiven that A and B are tensors.
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Towards a High-Performance Contraction Framework
Block Matrix Unfolding for A = A(1:n1, 1:n2, 1:n3, 1:n4)
BlockMat(A) =
A( : , : , 1, 1) A( : , : , 1, 2) · · · A( : , : , 1, n4)
A( : , : , 2, 1) A( : , : , 2, 2) · · · A( : , : , 2, n4)...
.... . .
...
A( : , : , n3, 1) A( : , : , n3, 2) · · · A( : , : , n3, n4)
A(i , j , k, `) is the (i , j) entry of block (k, `)
Note that tenmat(A,[1 3],[2 4]) produces the BlockMatunfolding.
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Tensor Contractions: Matrix Level
Example 2.
C(i , j , p, q) =∑k
∑s
A(i , s, p, k) · B(s, j , k, q)
As a collection of matrix products and sums...
C( : , : , p, q) =∑k
A( : , : , p, k) · B( : , : , k, q)
As a product of two block matrices...
BlockMat(C) = BlockMat(A) · BlockMat(B)
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Problem 3.11. Write an efficient Matlab function C = Eg2(A,B) thattakes two order-4 tensors A and B and produces the order-4 tensor definedin Example 2.
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Tensor Contractions: Matrix Level
Example 3.
C(m1,m2,m3,m4)
=m∑
i=1
A(i1, i2, i3, i4)·W (i1,m1)·X (i2,m2)·Y (i3,m3)·Z (i4,m4)
As a block-matrix product...
BlockMat(C) = (W ⊗ Y ) · BlockMat(A) · (X ⊗ Z )
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Problem 3.12. Write an efficient Matlab function C = Eg5(A,W,X,Y,Z)
that takes an order-4 tensor A and four suitably dimensioned matrices W ,X , Y , and Z and produces the order-4 tensor C defined in Example 5.
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
High Performance Tensor Computations
There are Locality Issues...
If the B( : , : , p, r) and C( : , : , r , q) are stored continguously, then
A(i , j , p, q) =∑k
∑s
B(i , s, p, k) ∗ C(s, j , k, q)
is nice because the participating “slices” are laid out nicely:
A( : , : , p, q) =∑
r
B( : , : , p, r) ∗ C ( : , : , r , q)
Unfriendly:
A(i , j , p, q) =∑k
∑s
B(i , p, s, k) ∗ C(k, s, j , q)
LAPACK has it easy: BC , BCT , BTC , BTCT . Tensor BLAS?
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
High Performance Tensor Computations
There are Data Structure/Layout Issues...
SupposeA = A(1:n1, . . . , 1:n7).
Can regard A as a 4-th order tensor with 3rd order entries. Storethe 3rd order subtensors contiguously.
Can regard A as an aggregation of “little” 7th order tensors. (Likea block matrix.)
The what-fits-in-cache inequality:
m7 ≤ Ncache
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Summary of Lecture 3.
Key Words
A transposition of an order-d tensor is defined by apermutation p of 1:d . Element i is mapped to element p(i).
The Kronecker product of two matrices is a highly structuredblock matrix.
Perfect shuffle permutations are a handy way to describetensor transposition.
A contraction between two tensors is a summation processthat produces a new tensor.
⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions
Top Related