I. Approaches to bounding the exponent of matrix ... · • Ambainis-Filmus 2014: N-th tensor power...

I. Approaches to

bounding the exponent of

matrix multiplication

Chris Umans

Caltech

Based on joint work with Noga Alon, Henry Cohn, Bobby

Kleinberg, Amir Shpilka, Balazs Szegedy

Modern Applications of Representation Theory, IMA, Chicago July 2014

Introduction

• Standard method: O(n3) operations

• Strassen (1969): O(n2.81) operations

X = A B C

July 31, 2014 2

July 31, 2014 3

Strassen’s Algorithm

• linear combos

of A, B entries

• 7 mults

• linear combos

of results

X = a1, 1 a1, 2

a2, 2 a2, 1

b1, 1 b1, 2

b2, 2 b2, 1

c1, 1 c1, 2

c2, 2 c2, 1

(a1,1 - a1,2) (b2,1 + b2,2)

(a1,1 + a2,2) (b1,1 + b2,2)

(a1,1 - a2,1) (b1,1 + b1,2)

(a1,1 + a1,2) (b2,2)

(a1,1) (b1,2 + b2,2)

(a2,2) (b2,1 - b1,1)

(a2,1 - a2,2) (b1,1)

P1 = (a1,1 - a1,2) × (b2,1 + b2,2)

P2 = (a1,1 + a2,2) × (b1,1 + b2,2)

P3 = (a1,1 - a2,1) × (b1,1 + b1,2)

P4 = (a1,1 + a1,2) × (b2,2)

P5 = (a1,1) × (b1,2 + b2,2)

P6 = (a2,2) × (b2,1 - b1,1)

P7 = (a2,1 - a2,2) × (b1,1)

c1,1 = P1 + P2 - P4 + P6

c1,2 = P4 + P5

c2,1 = P6 + P7

c2,2 = P2 - P3 + P5 - P7

July 31, 2014 4

Strassen’s Algorithm

• linear combos

of A, B entries

• 7 mults

• linear combos

of results

X = a1, 1 a1, 2

a2, 2 a2, 1

b1, 1 b1, 2

b2, 2 b2, 1

c1, 1 c1, 2

c2, 2 c2, 1

T(n) = # operations to

multiply n x n

matrices:

T(n) ≤ 7T(n/2) + O(n2)

T(n) ≤ O(nlog27)

Introduction

• Standard method: O(n3) operations

• Strassen (1969): O(n2.81) operations

X = A B C

The exponent of matrix multiplication:

smallest number such that for all >0

O(n + ) operations suffice July 31, 2014 5

History • Standard algorithm ≤ 3

• Strassen (1969) < 2.81

• Pan (1978) < 2.79

• Bini; Bini et al. (1979) < 2.78

• Schönhage (1981) < 2.55

• Pan; Romani; Coppersmith

+ Winograd (1981-1982) < 2.50

• Strassen (1987) < 2.48

• Coppersmith + Winograd (1987) < 2.375

• Stothers (2010) < 2.3737

• Williams (2011) < 2.3729

• Le Gall (2014) < 2.37286

July 31, 2014 6

July 31, 2014 7

Introduction

• Other important problems have same

algorithmic complexity as matrix

multiplication (via reductions)

– determinants

– LUP decompositions

– matrix inversion

– solving systems of linear equations

– basic graph problems …

Outline

1. crash course on main ideas from

Strassen 1969 through Le Gall 2014

2. conjectures implying ! = 2

Two more lectures:

II. group-theoretic approach

III. extending to coherent configurations

July 31, 2014 8

Bilinear algorithms & tensor rank

• Bilinear computation of complexity m:

– m products of form (L.C. of A) x (L.C. of B)

– result matrix entries = L.C.’s of these products

• equivalent: rank of matrix multiplication

tensor “<n,n,n>” is at most m

• Strassen: “bilinear w.l.o.g.”

! = inf {¿ : n x n matmult in O(n¿) size}

= inf {¿ : rank(<n,n,n>) · O(n¿)}

July 31, 2014 9

Tensors

• tensor: 3-D array of

complex numbers

• rank 1 tensor: (i,j,k) entry

= xiyjzk

• rank r tensor: sum of r

rank 1 tensors

X1

x2

x3

x4

y1 y2 y3 y4

equivalent: slices all L.C.’s of r rank 1 matrices

July 31, 2014 10

Tensors

• tensor: 3-D array of complex numbers

• tensor T with entries Ti,j,k

– equivalent: trilinear form

i,j,k

Ti,j,k

Xi Yj Zk

July 31, 2014 11

The matrix multiplication tensor

1

1

1

1

1

1

1

1

b11 b12 b21 b22

a2

2 a

12 a

21 a

11

a11 a12

a21 a22

b11 b12

b21 b22 x =

c11 c12

c21 c22

<n,n,n> is a n2 x n2 x n2 tensor described

by trilinear form i,j,kXi,jYj,kZk,i

July 31, 2014 12


1

1

1

1

1

1

1

1

b11 b12 b21 b22

a2

2 a

12 a

21 a

11

a11 a12

a21 a22

b11 b12

b21 b22 x =

c11 c12

c21 c22



July 31, 2014 13


1

1

1

1

1

1

1

1

b11 b12 b21 b22

a2

2 a

12 a

21 a

11

a11 a12

a21 a22

b11 b12

b21 b22 x =

c11 c12

c21 c22



July 31, 2014 14


1

1

1

1

1

1

1

1

b11 b12 b21 b22

a2

2 a

12 a

21 a

11

a11 a12

a21 a22

b11 b12

b21 b22 x =

c11 c12

c21 c22



July 31, 2014 15


1

1

1

1

1

1

1

1

b11 b12 b21 b22

a2

2 a

12 a

21 a

11

a11 a12

a21 a22

b11 b12

b21 b22 x =

c11 c12

c21 c22



July 31, 2014 16


<n,m,p> is a nm £ mp £ pn tensor

described by trilinear form i,j,kXi,jYj,kZk,i

X = A B C n

m

p

m n

p

1

1

Each of

np slices of

<n,m,p>:

July 31, 2014 17

Strategies

for upper bounding the rank

of the

matrix multiplication tensor

Upper bounds on rank

• Observation: <n,n,n>i = <ni, ni, ni>

) R(<ni, ni, ni>) · R(<n,n,n>)i

• Strategy I: bound rank for small n by hand

– R(<2,2,2>) = 7 ! < 2.81

– R(<3,3,3>) 2 [19..23] (worse bound)

– even computer search infeasible…

July 31, 2014 19

Strassen’s example

1

1

1

1

1

1

1

1

1 1

1 1

1 -

1 1

1

1

1 -

1 1

1 1

-

1

-

1 -

1

-

1

1 1 July 31, 2014 20


• Border rank = rank of sequence of tensors

approaching target tensor entrywise

• Strategy II: bound border rank for small n

– Lemma: R(<n,n,n>) < r ) ! < logn r

Idea: t-th tensor power is degree O(t) polynomial in ²;

interpolate to recover coefficient on ²0

– R(<2,2,3>) · 10 ! < 2.79

1 1

1

²-1 1

1 ²

1

rank = 3

border rank = 2: ²

July 31, 2014 21


• Strategy III: bound (border) rank of direct

sums of small matrix multiplication tensors

– R(<4,1,3> © <1,6,1>) · 13 ! < 2.55

• Direct sum of tensors

<n,n,n> © <m,m,m>

<n,n,n>

<m,m,m>

(multiple matrix multiplications in parallel)

R(<n1,n1,n1> © … © <nk,nk,nk>) < r ) ini! < r

“Asymptotic Sum Inequality” and

example (Schönhage 1981)

July 31, 2014 22

• Strategy IV: Strassen “laser method”

– tensor with “course structure” of MM and “fine

structure” components isomorphic to MM

(many independent MMs in high tensor powers)

1

1


1

1

1

1

…

scalar x row vector

col vector x scalar

q

<1,2,1> coarse structure fine =

July 31, 2014 23

• Strategy IV: Strassen “laser method”

– tensor with “course structure” of MM and “fine

structure” components isomorphic to MM

(many independent MMs in high tensor powers)

border rank = q + 1; q = 5 yields ! < 2.48

1

1


1

1

1

1

…

q

July 31, 2014 24


• Strategy V : (C-W) border rank of this tensor

is q+2:

– zero-out variables leaving many independent

MMs in high tensor power (sophisticated proof)

– q = 6 yields ! < 2.41

i=1…q X0YiZi + XiY0Zi + XiYiZ0

July 31, 2014 25


• Strategy VI: (C-W) border rank of this tensor

is q+2:

– 6 “pieces”: target proportions in high tensor

power affect # and size of independent MMs

– optimize by hand

– q = 6 yields ! < 2.388

i=1…q X0YiZi + XiY0Zi + XiYiZ0 +

X0Y0Zq+1 + X0Yq+1Z0 + Xq+1Y0Z0

July 31, 2014 26


• Strategy VII: border rank of tensor powers

of T:

T = i=1…q X0YiZi + XiY0Zi + XiYiZ0 +

X0Y0Zq+1 + X0Yq+1Z0 + Xq+1Y0Z0

Tensor power # pieces bound reference

2 36 2.375 C-W

4 1296 2.3737 Stothers

8 1679616 2.3729 Williams

16 2.82 x 10^12 2.3728640 Le Gall

32 7.95 x 10^24 2.3728639 Le Gall

July 31, 2014 27


• Strategy VII: border rank of tensor powers

• Ambainis-Filmus 2014: N-th tensor power

cannot beat bound of 2.3078

Tensor power # pieces bound reference

2 36 2.375 C-W

4 1296 2.3737 Stothers

8 1679616 2.3729 Williams

16 2.82 x 10^12 2.3728640 Le Gall

32 7.95 x 10^24 2.3728639 Le Gall

July 31, 2014 28

Conjectures implying

! = 2

“Asymptotic Rank” conjecture [CW90]

– border rank = 4

– asymptotic rank of T = lim n ! 1 R(Tn)1/n

1

1

1

1

1

1 T slices

T = i=1,2 X0YiZi + XiY0Zi + XiYiZ0

conjecture: asymptotic rank of T = 3 July 31, 2014 30

“no 3 disjoint equivoluminous

subsets” conjecture [CW90]

one way to achieve asymptotic rank 3:

• m1, m2, …, mn 2 H (abelian group)

• for all disjoint S, T, U 2 [n] the sums

i 2 S mi i 2 U mi i 2 T mi

are not all equal

conjecture: can take |H| · 2o(n)

July 31, 2014 31

Strong Uniquely Solvable Puzzle

Conjecture [CKSU05]

1

1

1

1

1

1 T

1

1

1

1

1

1

1

1

1

T’

rank = 4

rank = 3

Strong Uniquely Solvable Puzzle:

gives a way to zero-out variables

leaving many independent MMs in high

tensor power of T’ (instead of T)

July 31, 2014 32

Strong Uniquely Solvable Puzzle

Conjecture [CKSU05]

Uniquely Solvable Puzzle:

every unintended way of

assembling pieces has overlap

of 2 or 3 in some cell

Strong Uniquely Solvable Puzzle:

every unintended way of

assembling pieces has overlap

of exactly 2 in some cell

0 0 1 1 1 2

0 1 0 1 2 1

0 1 1 1 2 2

1 0 0 2 1 1

1 0 1 2 1 2

1 1 0 2 2 1

Ã w !

N rows

conjecture: Strong USPs exist with

N = (w choose w/3)1 - o(1) rows

“Two Families” conjecture [CKSU05]

• subsets A1, A2, …, An, B1, B2, …, Bn of

Abelian group H

1. |Ai + Bi| = |Ai|¢|Bi|

2. (Ai + Bi) Å (Aj + Bk) = ;

if j k

A1 A2 An

all

distinct

all

distinct

all

distinct

B1

B2

Bn

conjecture: can achieve

• n = |H|1/2 – o(1)

• |Ai| = |Bi| = |H|1/2 – o(1)

July 31, 2014 34

Status of conj’s implying ! = 2

• asymptotic rank(T) = R(Tn)1/n ! 3?

• no 3 disjoint equivoluminous subsets [CW90]

• strong uniquely solvable puzzle [CKSU05]

• two families [CKSU05]

1

1

1

1

1

1 T slices

sunflower conj. #1 false

)


)

[CW90]

July 31, 2014 35

Sunflower conjecture #1

• s!2s known

• conjecture widely believed

3-sunflower

Erdos-Rado conjecture: every collection of

consts s-subsets contains a 3-sunflower

July 31, 2014 36






1

1

1

1

1

1 T slices


)


)

[CW90]

July 31, 2014 37

Sunflower conjecture #2

• 50% chance of being false?

• 60% chance of being false?

Every collection of 3n(1- const) vectors in Z3n

contains u,v,w such that u + v + w = 0

Every 3-colored collection* of 3n(1- const)

vectors in Z3n contains u,v,w s.t. u+v+w=0

July 31, 2014 38






1

1

1

1

1

1 T slices


)


)

[CW90]

July 31, 2014 39

A different approach

• So far...

– bound border rank of small tensor (by hand)

– asymptotic bound from high tensor powers

• Disadvantages

– limited universe of “starting” tensors

– high tensor powers hard to analyze

• Next: matrix multiplication via groups

July 31, 2014 40

I. Approaches to bounding the exponent of matrix ... · • Ambainis-Filmus 2014: N-th tensor power...

Documents

Transcript of I. Approaches to bounding the exponent of matrix ... · • Ambainis-Filmus 2014: N-th tensor power...