Parameterized Complexity of Matrix Factorization
David P. Woodruff
IBM Almaden
. .
. .
Based on joint works with Ilya Razenshteyn, Zhao Song, Peilin Zhong
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 1 / 50
Outline
Frobenius norm factorization can be done in polynomial time
Intractable matrix problems
I Nonnegative matrix factorization(NMF)
F Arora-Ge-Kannan-Moitra’12, Moitra’13
I Weighted low rank approximation(WLRA)
F Razenshteyn-Song-W’16
I `1 low rank approximation(`1LRA)
F Song-W-Zhong’16
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 2 / 50
Outline
Frobenius norm factorization can be done in polynomial timeIntractable matrix problems
I Nonnegative matrix factorization(NMF)
F Arora-Ge-Kannan-Moitra’12, Moitra’13
I Weighted low rank approximation(WLRA)
F Razenshteyn-Song-W’16
I `1 low rank approximation(`1LRA)
F Song-W-Zhong’16
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 2 / 50
Outline
Frobenius norm factorization can be done in polynomial timeIntractable matrix problems
I Nonnegative matrix factorization(NMF)
F Arora-Ge-Kannan-Moitra’12, Moitra’13I Weighted low rank approximation(WLRA)
F Razenshteyn-Song-W’16
I `1 low rank approximation(`1LRA)
F Song-W-Zhong’16
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 2 / 50
Outline
Frobenius norm factorization can be done in polynomial timeIntractable matrix problems
I Nonnegative matrix factorization(NMF)F Arora-Ge-Kannan-Moitra’12, Moitra’13
I Weighted low rank approximation(WLRA)
F Razenshteyn-Song-W’16
I `1 low rank approximation(`1LRA)
F Song-W-Zhong’16
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 2 / 50
Outline
Frobenius norm factorization can be done in polynomial timeIntractable matrix problems
I Nonnegative matrix factorization(NMF)F Arora-Ge-Kannan-Moitra’12, Moitra’13
I Weighted low rank approximation(WLRA)
F Razenshteyn-Song-W’16I `1 low rank approximation(`1LRA)
F Song-W-Zhong’16
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 2 / 50
Outline
Frobenius norm factorization can be done in polynomial timeIntractable matrix problems
I Nonnegative matrix factorization(NMF)F Arora-Ge-Kannan-Moitra’12, Moitra’13
I Weighted low rank approximation(WLRA)F Razenshteyn-Song-W’16
I `1 low rank approximation(`1LRA)
F Song-W-Zhong’16
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 2 / 50
Outline
Frobenius norm factorization can be done in polynomial timeIntractable matrix problems
I Nonnegative matrix factorization(NMF)F Arora-Ge-Kannan-Moitra’12, Moitra’13
I Weighted low rank approximation(WLRA)F Razenshteyn-Song-W’16
I `1 low rank approximation(`1LRA)
F Song-W-Zhong’16
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 2 / 50
Outline
Frobenius norm factorization can be done in polynomial timeIntractable matrix problems
I Nonnegative matrix factorization(NMF)F Arora-Ge-Kannan-Moitra’12, Moitra’13
I Weighted low rank approximation(WLRA)F Razenshteyn-Song-W’16
I `1 low rank approximation(`1LRA)F Song-W-Zhong’16
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 2 / 50
Frobenius Norm Matrix Factorization
QuestionGiven a matrix A ∈ Rn×n and k > 1, the goal is to output two matricesU ∈ Rn×k ,V ∈ Rk×n such that
‖UV − A‖2F 6 (1 + ε) minrank−k A ′
‖A ′ − A‖2F ,
where ‖A‖2F = (∑n
i=1∑n
j=1 A2i,j)
2, ε ∈ [0,1)
Useful for data compression, easier to store factorizations U,VCan be solved using the singular value decomposition (SVD) inmatrix multiplication n2.376... timeFails if you want a nonnegative factorizationNot robust to noise, outliers, or missing entries
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 3 / 50
Frobenius Norm Matrix Factorization
QuestionGiven a matrix A ∈ Rn×n and k > 1, the goal is to output two matricesU ∈ Rn×k ,V ∈ Rk×n such that
‖UV − A‖2F 6 (1 + ε) minrank−k A ′
‖A ′ − A‖2F ,
where ‖A‖2F = (∑n
i=1∑n
j=1 A2i,j)
2, ε ∈ [0,1)
Useful for data compression, easier to store factorizations U,V
Can be solved using the singular value decomposition (SVD) inmatrix multiplication n2.376... timeFails if you want a nonnegative factorizationNot robust to noise, outliers, or missing entries
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 3 / 50
Frobenius Norm Matrix Factorization
QuestionGiven a matrix A ∈ Rn×n and k > 1, the goal is to output two matricesU ∈ Rn×k ,V ∈ Rk×n such that
‖UV − A‖2F 6 (1 + ε) minrank−k A ′
‖A ′ − A‖2F ,
where ‖A‖2F = (∑n
i=1∑n
j=1 A2i,j)
2, ε ∈ [0,1)
Useful for data compression, easier to store factorizations U,VCan be solved using the singular value decomposition (SVD) inmatrix multiplication n2.376... time
Fails if you want a nonnegative factorizationNot robust to noise, outliers, or missing entries
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 3 / 50
Frobenius Norm Matrix Factorization
QuestionGiven a matrix A ∈ Rn×n and k > 1, the goal is to output two matricesU ∈ Rn×k ,V ∈ Rk×n such that
‖UV − A‖2F 6 (1 + ε) minrank−k A ′
‖A ′ − A‖2F ,
where ‖A‖2F = (∑n
i=1∑n
j=1 A2i,j)
2, ε ∈ [0,1)
Useful for data compression, easier to store factorizations U,VCan be solved using the singular value decomposition (SVD) inmatrix multiplication n2.376... timeFails if you want a nonnegative factorization
Not robust to noise, outliers, or missing entries
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 3 / 50
Frobenius Norm Matrix Factorization
QuestionGiven a matrix A ∈ Rn×n and k > 1, the goal is to output two matricesU ∈ Rn×k ,V ∈ Rk×n such that
‖UV − A‖2F 6 (1 + ε) minrank−k A ′
‖A ′ − A‖2F ,
where ‖A‖2F = (∑n
i=1∑n
j=1 A2i,j)
2, ε ∈ [0,1)
Useful for data compression, easier to store factorizations U,VCan be solved using the singular value decomposition (SVD) inmatrix multiplication n2.376... timeFails if you want a nonnegative factorizationNot robust to noise, outliers, or missing entries
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 3 / 50
. .
. .
Nonnegative Matrix Factorization
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 4 / 50
Example of NMF. .
. .Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 5 / 50
Example of NMF. .
. .
Given : A ∈ Rn×n, n = 4, k = 2
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 5 / 50
Example of NMF. .
. .
Given : A ∈ Rn×n, n = 4, k = 2
A =
0 1 1 11 2 1 12 4 2 23 5 2 2
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 5 / 50
Example of NMF. .
. .
Given : A ∈ Rn×n, n = 4, k = 2
A =
0 1 1 11 2 1 12 4 2 23 5 2 2
Question : Are there two matrices U,V> ∈ Rn×k , such thatUV = A,U > 0,V > 0
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 5 / 50
Example of NMF. .
. .
Given : A ∈ Rn×n, n = 4, k = 2
A =
0 1 1 11 2 1 12 4 2 23 5 2 2
Question : Are there two matrices U,V> ∈ Rn×k , such thatUV = A,U > 0,V > 0
=
0 11 12 23 2
· [1 1 0 00 1 1 1
]
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 5 / 50
Example of NMF. .
. .
Given : A ∈ Rn×n, n = 4, k = 2
A =
0 1 1 11 2 1 12 4 2 23 5 2 2
Question : Are there two matrices U,V> ∈ Rn×k , such thatUV = A,U > 0,V > 0
=
0 11 12 23 2
· [1 1 0 00 1 1 1
]
= UV
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 5 / 50
Main Question, Nonnegative Matrix Factorization
QuestionGiven matrix A ∈ Rn×n and k > 1, is there an algorithm that candetermine if there exist two matrices U,V> ∈ Rn×k ,
UV = A,U > 0,V > 0.
Or, are there any hardness results?
Equivalent to computing the nonnegative rank of A, rank+(A)Fundamental question in machine learningApplications
I Text mining, Spectral data analysis, Scalable Internet distanceprediction, Non-stationary speech denoising, Bioinformatics,Nuclear imaging, etc.
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 6 / 50
Main Question, Nonnegative Matrix Factorization
QuestionGiven matrix A ∈ Rn×n and k > 1, is there an algorithm that candetermine if there exist two matrices U,V> ∈ Rn×k ,
UV = A,U > 0,V > 0.
Or, are there any hardness results?
Equivalent to computing the nonnegative rank of A, rank+(A)
Fundamental question in machine learningApplications
I Text mining, Spectral data analysis, Scalable Internet distanceprediction, Non-stationary speech denoising, Bioinformatics,Nuclear imaging, etc.
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 6 / 50
Main Question, Nonnegative Matrix Factorization
QuestionGiven matrix A ∈ Rn×n and k > 1, is there an algorithm that candetermine if there exist two matrices U,V> ∈ Rn×k ,
UV = A,U > 0,V > 0.
Or, are there any hardness results?
Equivalent to computing the nonnegative rank of A, rank+(A)Fundamental question in machine learning
Applications
I Text mining, Spectral data analysis, Scalable Internet distanceprediction, Non-stationary speech denoising, Bioinformatics,Nuclear imaging, etc.
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 6 / 50
Main Question, Nonnegative Matrix Factorization
QuestionGiven matrix A ∈ Rn×n and k > 1, is there an algorithm that candetermine if there exist two matrices U,V> ∈ Rn×k ,
UV = A,U > 0,V > 0.
Or, are there any hardness results?
Equivalent to computing the nonnegative rank of A, rank+(A)Fundamental question in machine learningApplications
I Text mining, Spectral data analysis, Scalable Internet distanceprediction, Non-stationary speech denoising, Bioinformatics,Nuclear imaging, etc.
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 6 / 50
Main Question, Nonnegative Matrix Factorization
QuestionGiven matrix A ∈ Rn×n and k > 1, is there an algorithm that candetermine if there exist two matrices U,V> ∈ Rn×k ,
UV = A,U > 0,V > 0.
Or, are there any hardness results?
Equivalent to computing the nonnegative rank of A, rank+(A)Fundamental question in machine learningApplications
I Text mining, Spectral data analysis, Scalable Internet distanceprediction, Non-stationary speech denoising, Bioinformatics,Nuclear imaging, etc.
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 6 / 50
Rank vs. Nonnegative Rankrank(A) is the rank and rank+(A) is the nonnegative rank
I rank(A) 6 rank+(A)I Vavasis’09, determining whether rank+(A) = rank(A) is NP-hard.
.
.
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 7 / 50
Rank vs. Nonnegative Rankrank(A) is the rank and rank+(A) is the nonnegative rank
I rank(A) 6 rank+(A)
I Vavasis’09, determining whether rank+(A) = rank(A) is NP-hard.
.
.
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 7 / 50
Rank vs. Nonnegative Rankrank(A) is the rank and rank+(A) is the nonnegative rank
I rank(A) 6 rank+(A)I Vavasis’09, determining whether rank+(A) = rank(A) is NP-hard.
.
.
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 7 / 50
Rank vs. Nonnegative Rankrank(A) is the rank and rank+(A) is the nonnegative rank
I rank(A) 6 rank+(A)I Vavasis’09, determining whether rank+(A) = rank(A) is NP-hard.
.
.
A =
1 1 0 01 0 1 00 1 0 10 0 1 1
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 7 / 50
Rank vs. Nonnegative Rankrank(A) is the rank and rank+(A) is the nonnegative rank
I rank(A) 6 rank+(A)I Vavasis’09, determining whether rank+(A) = rank(A) is NP-hard.
.
.
A =
1 1 0 01 0 1 00 1 0 10 0 1 1
=
1 1 01 0 10 1 00 0 1
·1 0 0 −1
0 1 0 10 0 1 1
rank(A) = 3,
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 7 / 50
Rank vs. Nonnegative Rankrank(A) is the rank and rank+(A) is the nonnegative rank
I rank(A) 6 rank+(A)I Vavasis’09, determining whether rank+(A) = rank(A) is NP-hard.
.
.
A =
1 1 0 01 0 1 00 1 0 10 0 1 1
=
1 1 01 0 10 1 00 0 1
·1 0 0 −1
0 1 0 10 0 1 1
rank(A) = 3, but rank+(A) = 4
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 7 / 50
Polynomial System Verifier
. .
. .
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 8 / 50
Polynomial System Verifier
. .
. .[Renegar’92, Basu-Pollack-Roy’96]
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 8 / 50
Polynomial System Verifier
. .
. .[Renegar’92, Basu-Pollack-Roy’96]
Given : a polynomial system P(x) over the real numbers
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 8 / 50
Polynomial System Verifier
. .
. .[Renegar’92, Basu-Pollack-Roy’96]
Given : a polynomial system P(x) over the real numbersv : #variables, x = (x1, x2, · · · , xv )
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 8 / 50
Polynomial System Verifier
. .
. .[Renegar’92, Basu-Pollack-Roy’96]
Given : a polynomial system P(x) over the real numbersv : #variables, x = (x1, x2, · · · , xv )
m : #polynomial constraints fi(x) > 0, ∀i ∈ [m]
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 8 / 50
Polynomial System Verifier
. .
. .[Renegar’92, Basu-Pollack-Roy’96]
Given : a polynomial system P(x) over the real numbersv : #variables, x = (x1, x2, · · · , xv )
m : #polynomial constraints fi(x) > 0, ∀i ∈ [m]
d : maximum degree of all polynomial constraints
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 8 / 50
Polynomial System Verifier
. .
. .[Renegar’92, Basu-Pollack-Roy’96]
Given : a polynomial system P(x) over the real numbersv : #variables, x = (x1, x2, · · · , xv )
m : #polynomial constraints fi(x) > 0, ∀i ∈ [m]
d : maximum degree of all polynomial constraintsH : the bitsizes of the coefficients of the polynomials
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 8 / 50
Polynomial System Verifier
. .
. .[Renegar’92, Basu-Pollack-Roy’96]
Given : a polynomial system P(x) over the real numbersv : #variables, x = (x1, x2, · · · , xv )
m : #polynomial constraints fi(x) > 0, ∀i ∈ [m]
d : maximum degree of all polynomial constraintsH : the bitsizes of the coefficients of the polynomials
In (md)O(v) poly(H) time, candecide if there exists a solution to polynomial system P
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 8 / 50
Main Idea
. .
. .
1. Write minU,V>∈Rn×k ,U,V>0
‖UV − A‖2F as a polynomial system
that has poly(k) variables and poly(n) constraints and degree
2. Use polynomial system verifier to solve it
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 9 / 50
Algorithm. .
. .Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 10 / 50
Algorithm. .
. .
Given : A ∈ Rn×n, k ∈ N
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 10 / 50
Algorithm. .
. .
Given : A ∈ Rn×n, k ∈ N
Question :
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 10 / 50
Algorithm. .
. .
Given : A ∈ Rn×n, k ∈ N
Question : Are there matrices U,V> ∈ Rn×k such that
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 10 / 50
Algorithm. .
. .
Given : A ∈ Rn×n, k ∈ N
Question : Are there matrices U,V> ∈ Rn×k such thatUV = A
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 10 / 50
Algorithm. .
. .
Given : A ∈ Rn×n, k ∈ N
Question : Are there matrices U,V> ∈ Rn×k such thatUV = AU > 0, V > 0
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 10 / 50
Algorithm. .
. .
Given : A ∈ Rn×n, k ∈ N
Question : Are there matrices U,V> ∈ Rn×k such thatUV = AU > 0, V > 0
Output :
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 10 / 50
Algorithm. .
. .
Given : A ∈ Rn×n, k ∈ N
Question : Are there matrices U,V> ∈ Rn×k such thatUV = AU > 0, V > 0
Output : Yes or No
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 10 / 50
Algorithm. .
. .
Given : A ∈ Rn×n, k ∈ N
Question : Are there matrices U,V> ∈ Rn×k such thatUV = AU > 0, V > 0
Output : Yes or Noin n2O(k)
time Arora-Ge-Kannan-Moitra’12
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 10 / 50
Algorithm. .
. .
Given : A ∈ Rn×n, k ∈ N
Question : Are there matrices U,V> ∈ Rn×k such thatUV = AU > 0, V > 0
Output : Yes or Noin n2O(k)
time Arora-Ge-Kannan-Moitra’12in 2O(k3)nO(k2) time Moitra’13
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 10 / 50
k -SUM. .
. .Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 11 / 50
k -SUM. .
. .
k -SUM
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 11 / 50
k -SUM. .
. .
k -SUM
Given: a set of n values s1, s2, · · · , sn each in the range [0,1]
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 11 / 50
k -SUM. .
. .
k -SUM
Given: a set of n values s1, s2, · · · , sn each in the range [0,1]
Determine:
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 11 / 50
k -SUM. .
. .
k -SUM
Given: a set of n values s1, s2, · · · , sn each in the range [0,1]
Determine: if there is a set of k numbers that sum to exactly k/2
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 11 / 50
k -SUM. .
. .
k -SUM
Given: a set of n values s1, s2, · · · , sn each in the range [0,1]
Determine: if there is a set of k numbers that sum to exactly k/2
k -SUM hardness [Patrascu-Williams’10]
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 11 / 50
k -SUM. .
. .
k -SUM
Given: a set of n values s1, s2, · · · , sn each in the range [0,1]
Determine: if there is a set of k numbers that sum to exactly k/2
k -SUM hardness [Patrascu-Williams’10]
Assume: 3-SAT on n variables cannot be solved in 2o(n) time
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 11 / 50
k -SUM. .
. .
k -SUM
Given: a set of n values s1, s2, · · · , sn each in the range [0,1]
Determine: if there is a set of k numbers that sum to exactly k/2
k -SUM hardness [Patrascu-Williams’10]
Assume: 3-SAT on n variables cannot be solved in 2o(n) time
then k -SUM cannot be solved in no(k) time
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 11 / 50
Hardness - Under Exponential Time Hypothesis. .
. .Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 12 / 50
Hardness - Under Exponential Time Hypothesis. .
. .
Given : A ∈ Rn×n, k ∈ N
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 12 / 50
Hardness - Under Exponential Time Hypothesis. .
. .
Given : A ∈ Rn×n, k ∈ N
Output : Are there two matrices U,V> ∈ Rn×k , such thatUV = AU > 0,V > 0
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 12 / 50
Hardness - Under Exponential Time Hypothesis. .
. .
Given : A ∈ Rn×n, k ∈ N
Output : Are there two matrices U,V> ∈ Rn×k , such thatUV = AU > 0,V > 0
Assume : Exponential Time Hypothesis[Impagliazzo-Paturi-Zane’98]
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 12 / 50
Hardness - Under Exponential Time Hypothesis. .
. .
Given : A ∈ Rn×n, k ∈ N
Output : Are there two matrices U,V> ∈ Rn×k , such thatUV = AU > 0,V > 0
Assume : Exponential Time Hypothesis[Impagliazzo-Paturi-Zane’98]
Requires : nΩ(k) time
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 12 / 50
Open Problems
The upper bound is nO(k2) while the lower bound is nΩ(k) - what isthe right answer?
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 13 / 50
. .
. .
Weighted Low Rank Approximation
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 14 / 50
Weighted Low Rank Approximation. .
. .Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 15 / 50
Weighted Low Rank Approximation. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, k ∈ N, ε > 0
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 15 / 50
Weighted Low Rank Approximation. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, k ∈ N, ε > 0
Output :
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 15 / 50
Weighted Low Rank Approximation. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, k ∈ N, ε > 0
Output : rank-k A ∈ Rn×n s.t.
AW A )− (
2
F
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 15 / 50
Weighted Low Rank Approximation. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, k ∈ N, ε > 0
Output : rank-k A ∈ Rn×n s.t.‖W (A − A)‖2F 6 (1 + ε) min
rank−k A ′‖W (A ′ − A)‖2F
AW A )− (
2
F
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 15 / 50
Weighted Low Rank Approximation. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, k ∈ N, ε > 0
Output : rank-k A ∈ Rn×n s.t.‖W (A − A)‖2F 6 (1 + ε) min
rank−k A ′‖W (A ′ − A)‖2F
.
∑i,j
W 2i,j(A
′i,j − Ai,j)
2
AW A )− (
2
F
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 15 / 50
Weighted Low Rank Approximation. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, k ∈ N, ε > 0
Output : rank-k A ∈ Rn×n s.t.‖W (A − A)‖2F 6 (1 + ε) min
rank−k A ′‖W (A ′ − A)‖2F
.
∑i,j
W 2i,j(A
′i,j − Ai,j)
2
.OPT
AW A )− (
2
F
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 15 / 50
Weighted Low Rank Approximation. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, k ∈ N, ε > 0
Output :
..
U,V> ∈ Rn×k s.t.‖W (UV − A)‖2F 6 (1 + ε)OPT
U
V
W A )− (
2
F
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 15 / 50
Motivation. .
. .Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 16 / 50
Motivation. .
. .Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 16 / 50
Motivation. .
. .Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 16 / 50
Motivation. .
. .
Action Comedy Historical Cartoon Magical
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 16 / 50
Motivation. .
. .
Action Comedy Historical Cartoon Magical
.
9 8
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 16 / 50
Motivation. .
. .
Action Comedy Historical Cartoon Magical
..
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 16 / 50
Motivation. .
. .
Action Comedy Historical Cartoon Magical
...
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 16 / 50
Motivation. .
. .
Action Comedy Historical Cartoon Magical
....
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 16 / 50
Motivation. .
. .
Action Comedy Historical Cartoon Magical
.....
15
15
11
11
11
11
15
15
11
11
14
14
16
16
11
11
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 16 / 50
Motivation. .
. .
Action Comedy Historical Cartoon Magical
.....
15
15
11
11
11
11
15
15
11
11
14
14
16
16
11
11
.
11
11
11
11
11
11
16
16
13
13
12
12
12
12
14
14
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 16 / 50
Motivation. .
. .
Action Comedy Historical Cartoon Magical
.....
15
15
11
11
11
11
15
15
11
11
14
14
16
16
11
11
.
11
11
11
11
11
11
16
16
13
13
12
12
12
12
14
14
.
13
13
16
16
14
14
12
12
17
17
15
15
14
14
11
11
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 16 / 50
Motivation. .
. .
Action Comedy Historical Cartoon Magical
.....
15
15
11
11
11
11
15
15
11
11
14
14
16
16
11
11
.
11
11
11
11
11
11
16
16
13
13
12
12
12
12
14
14
.
13
13
16
16
14
14
12
12
17
17
15
15
14
14
11
11
.
12
12
11
11
12
12
11
11
15
15
11
11
13
13
14
14
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 16 / 50
Motivation. .
. .
Action Comedy Historical Cartoon Magical
.....
15
15
11
11
11
11
15
15
11
11
14
14
16
16
11
11
.
11
11
11
11
11
11
16
16
13
13
12
12
12
12
14
14
.
13
13
16
16
14
14
12
12
17
17
15
15
14
14
11
11
.
12
12
11
11
12
12
11
11
15
15
11
11
13
13
14
14
.
12
12
12
12
16
16
14
14
12
12
13
13
16
16
12
12
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 16 / 50
Motivation. .
. .
Action Comedy Historical Cartoon Magical
..........
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 16 / 50
Motivation. .
. .
Action Comedy Historical Cartoon Magical
...........
8 8
8 8
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 16 / 50
Motivation. .
. .
Action Comedy Historical Cartoon Magical
...........
8 8
8 8
.
4 4
4 4
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 16 / 50
Motivation. .
. .
Action Comedy Historical Cartoon Magical
...........
8 8
8 8
.
4 4
4 4
.
1 1
1 1
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 16 / 50
Motivation. .
. .
Action Comedy Historical Cartoon Magical
...........
8 8
8 8
.
4 4
4 4
.
1 1
1 1
.
0 0
0 0
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 16 / 50
Motivation. .
. .
Action Comedy Historical Cartoon Magical
...........
8 8
8 8
.
4 4
4 4
.
1 1
1 1
.
0 0
0 0
.
8 8
8 8
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 16 / 50
Motivation. .
. .
Action Comedy Historical Cartoon Magical
...........
8 8
8 8
.
4 4
4 4
.
1 1
1 1
.
0 0
0 0
.
8 8
8 8
.
4 4
4 4
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 16 / 50
Motivation. .
. .
Action Comedy Historical Cartoon Magical
...........
8 8
8 8
.
4 4
4 4
.
1 1
1 1
.
0 0
0 0
.
8 8
8 8
.
4 4
4 4
.
2 2
2 2
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 16 / 50
Motivation. .
. .
Action Comedy Historical Cartoon Magical
...........
8 8
8 8
.
4 4
4 4
.
1 1
1 1
.
0 0
0 0
.
8 8
8 8
.
4 4
4 4
.
2 2
2 2
.
1 1
1 1
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 16 / 50
Matrix Completion. .
. .Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 17 / 50
Matrix Completion. .
. .
Given : A ∈ R, ?n×n, Ω ⊂ [n]× [n] and k ∈ R
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 17 / 50
Matrix Completion. .
. .
Given : A ∈ R, ?n×n, Ω ⊂ [n]× [n] and k ∈ R
A =
1 ? ? 0? 1 0 ?? 0 1 ?0 ? ? 1
, k = 2
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 17 / 50
Matrix Completion. .
. .
Given : A ∈ R, ?n×n, Ω ⊂ [n]× [n] and k ∈ R
A =
1 ? ? 0? 1 0 ?? 0 1 ?0 ? ? 1
, k = 2
Output : AΩ s.t. rank(A) = k
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 17 / 50
Matrix Completion. .
. .
Given : A ∈ R, ?n×n, Ω ⊂ [n]× [n] and k ∈ R
A =
1 ? ? 0? 1 0 ?? 0 1 ?0 ? ? 1
, k = 2
Output : AΩ s.t. rank(A) = k
.
A =
1 1 0 01 1 0 00 0 1 10 0 1 1
, A =
1 0 1 00 1 0 11 0 1 00 1 0 1
.
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 17 / 50
Matrix Completion. .
. .
Given : A ∈ R, ?n×n, Ω ⊂ [n]× [n] and k ∈ R
A =
1 ? ? 0? 1 0 ?? 0 1 ?0 ? ? 1
, k = 2
Output : AΩ s.t. rank(A) = k
.
Algorithms : Candes-Recht’09, Candes-Recht’10, Keshavan’12Hardt-Wootters’14, Jain-Netrapalli-Sanghavi’13Hardt’15, Sun-Luo’15
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 17 / 50
Matrix Completion. .
. .
Given : A ∈ R, ?n×n, Ω ⊂ [n]× [n] and k ∈ R
A =
1 ? ? 0? 1 0 ?? 0 1 ?0 ? ? 1
, k = 2
Output : AΩ s.t. rank(A) = k
.
Algorithms : Candes-Recht’09, Candes-Recht’10, Keshavan’12Hardt-Wootters’14, Jain-Netrapalli-Sanghavi’13Hardt’15, Sun-Luo’15
Hardness : Peeters’96, Gillis-Glineur’11Hardt-Meka-Raghavendra-Weitz’14
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 17 / 50
More Real-life Datasets. .
. .
RateBeer
Documents
Amazon
Biology
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 18 / 50
More Real-life Datasets. .
. .
RateBeer
Tasters Beers Ratings
Documents
Amazon
Biology
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 18 / 50
More Real-life Datasets. .
. .
RateBeer
Tasters Beers Ratings
Documents
Words Books Frequency
Amazon
Biology
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 18 / 50
More Real-life Datasets. .
. .
RateBeer
Tasters Beers Ratings
Documents
Words Books Frequency
Amazon
Users Products Ratings
Biology
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 18 / 50
More Real-life Datasets. .
. .
RateBeer
Tasters Beers Ratings
Documents
Words Books Frequency
Amazon
Users Products Ratings
Biology
Genes Cells Levels
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 18 / 50
Main Results - Summary
Algorithm for Weighted low rank approximation(WLRA) problem
I W has r distinct rows and columnsI W has r distinct columnsI W has rank at most r
Hardness for Weighted low rank approximation(WLRA) problem
.
.
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 19 / 50
Main Results - Summary
Algorithm for Weighted low rank approximation(WLRA) problemI W has r distinct rows and columns
I W has r distinct columnsI W has rank at most r
Hardness for Weighted low rank approximation(WLRA) problem
.
.
W =
1 1 1 3 3 41 1 1 3 3 42 2 2 5 5 02 2 2 5 5 02 2 2 5 5 07 7 7 4 4 6
, r = 3
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 19 / 50
Main Results - Summary
Algorithm for Weighted low rank approximation(WLRA) problemI W has r distinct rows and columnsI W has r distinct columns
I W has rank at most r
Hardness for Weighted low rank approximation(WLRA) problem
.
.
W =
1 1 1 1 1 12 2 2 1 1 33 3 3 1 1 54 4 4 1 1 15 5 5 1 1 36 6 6 1 1 5
, r = 3
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 19 / 50
Main Results - Summary
Algorithm for Weighted low rank approximation(WLRA) problemI W has r distinct rows and columnsI W has r distinct columnsI W has rank at most r
Hardness for Weighted low rank approximation(WLRA) problem
.
.
W =
1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6
, r = 3
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 19 / 50
Main Results - Summary
Algorithm for Weighted low rank approximation(WLRA) problemI W has r distinct rows and columnsI W has r distinct columnsI W has rank at most r
Hardness for Weighted low rank approximation(WLRA) problem.
.
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 19 / 50
r Distinct Rows and Columns. .
. .Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 20 / 50
r Distinct Rows and Columns. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 20 / 50
r Distinct Rows and Columns. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
W =
1 1 1 3 3 41 1 1 3 3 42 2 2 5 5 02 2 2 5 5 02 2 2 5 5 07 7 7 4 4 6
, r = 3
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 20 / 50
r Distinct Rows and Columns. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
W =
1 1 1 3 3 41 1 1 3 3 42 2 2 5 5 02 2 2 5 5 02 2 2 5 5 07 7 7 4 4 6
, r = 3
Output :
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 20 / 50
r Distinct Rows and Columns. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
W =
1 1 1 3 3 41 1 1 3 3 42 2 2 5 5 02 2 2 5 5 02 2 2 5 5 07 7 7 4 4 6
, r = 3
Output : rank-k A ∈ Rn×n s.t.
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 20 / 50
r Distinct Rows and Columns. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
W =
1 1 1 3 3 41 1 1 3 3 42 2 2 5 5 02 2 2 5 5 02 2 2 5 5 07 7 7 4 4 6
, r = 3
Output : rank-k A ∈ Rn×n s.t.‖W (A − A)‖2F 6 (1 + ε) min
rank−k A ′‖W (A − A ′)‖2F
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 20 / 50
r Distinct Rows and Columns. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
W =
1 1 1 3 3 41 1 1 3 3 42 2 2 5 5 02 2 2 5 5 02 2 2 5 5 07 7 7 4 4 6
, r = 3
Output : rank-k A ∈ Rn×n s.t.‖W (A − A)‖2F 6 (1 + ε) min
rank−k A ′
.
∑i,j
W 2i,j(Ai,j − A ′i,j)
2
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 20 / 50
r Distinct Rows and Columns. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
W =
1 1 1 3 3 41 1 1 3 3 42 2 2 5 5 02 2 2 5 5 02 2 2 5 5 07 7 7 4 4 6
, r = 3
Output : rank-k A ∈ Rn×n s.t.‖W (A − A)‖2F 6 (1 + ε)
..
OPT
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 20 / 50
r Distinct Rows and Columns. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
W =
1 1 1 3 3 41 1 1 3 3 42 2 2 5 5 02 2 2 5 5 02 2 2 5 5 07 7 7 4 4 6
, r = 3
Output : rank-k A ∈ Rn×n s.t.‖W (A − A)‖2F 6 (1 + ε)
..
OPTwith prob. 9/10
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 20 / 50
r Distinct Rows and Columns. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
W =
1 1 1 3 3 41 1 1 3 3 42 2 2 5 5 02 2 2 5 5 02 2 2 5 5 07 7 7 4 4 6
, r = 3
Output : rank-k A ∈ Rn×n s.t.‖W (A − A)‖2F 6 (1 + ε)
..
OPTwith prob. 9/10in O((nnz(A) + nnz(W )) · nγ) + n · 2O(k2r/ε) time
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 20 / 50
r Distinct Rows and Columns. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
W =
1 1 1 3 3 41 1 1 3 3 42 2 2 5 5 02 2 2 5 5 02 2 2 5 5 07 7 7 4 4 6
, r = 3
Output : rank-k A ∈ Rn×n s.t.‖W (A − A)‖2F 6 (1 + ε)
..
OPTwith prob. 9/10in O((nnz(A) + nnz(W )) · nγ) + n · 2O(k2r/ε) timefor an arbitrarily small constant γ > 0
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 20 / 50
r Distinct Rows and Columns. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
W =
1 1 1 3 3 41 1 1 3 3 42 2 2 5 5 02 2 2 5 5 02 2 2 5 5 07 7 7 4 4 6
, r = 3
Output : rank-k A ∈ Rn×n s.t.‖W (A − A)‖2F 6 (1 + ε)
..
OPTwith prob. 9/10in O((nnz(A) + nnz(W )) · nγ) + n · 2O(k2r/ε) timefor an arbitrarily small constant γ > 0fixed parameter tractable
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 20 / 50
r Distinct Columns. .
. .Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 21 / 50
r Distinct Columns. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 21 / 50
r Distinct Columns. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
W =
1 1 1 1 1 12 2 2 1 1 33 3 3 1 1 54 4 4 1 1 15 5 5 1 1 36 6 6 1 1 5
, r = 3
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 21 / 50
r Distinct Columns. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
W =
1 1 1 1 1 12 2 2 1 1 33 3 3 1 1 54 4 4 1 1 15 5 5 1 1 36 6 6 1 1 5
, r = 3
Output :
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 21 / 50
r Distinct Columns. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
W =
1 1 1 1 1 12 2 2 1 1 33 3 3 1 1 54 4 4 1 1 15 5 5 1 1 36 6 6 1 1 5
, r = 3
Output : rank-k A ∈ Rn×n s.t.
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 21 / 50
r Distinct Columns. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
W =
1 1 1 1 1 12 2 2 1 1 33 3 3 1 1 54 4 4 1 1 15 5 5 1 1 36 6 6 1 1 5
, r = 3
Output : rank-k A ∈ Rn×n s.t.‖W (A − A)‖2F 6 (1 + ε) min
rank−k A ′‖W (A − A ′)‖2F
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 21 / 50
r Distinct Columns. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
W =
1 1 1 1 1 12 2 2 1 1 33 3 3 1 1 54 4 4 1 1 15 5 5 1 1 36 6 6 1 1 5
, r = 3
Output : rank-k A ∈ Rn×n s.t.‖W (A − A)‖2F 6 (1 + ε) min
rank−k A ′
.
∑i,j
W 2i,j(Ai,j − A ′i,j)
2
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 21 / 50
r Distinct Columns. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
W =
1 1 1 1 1 12 2 2 1 1 33 3 3 1 1 54 4 4 1 1 15 5 5 1 1 36 6 6 1 1 5
, r = 3
Output : rank-k A ∈ Rn×n s.t.‖W (A − A)‖2F 6 (1 + ε)
..
OPT
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 21 / 50
r Distinct Columns. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
W =
1 1 1 1 1 12 2 2 1 1 33 3 3 1 1 54 4 4 1 1 15 5 5 1 1 36 6 6 1 1 5
, r = 3
Output : rank-k A ∈ Rn×n s.t.‖W (A − A)‖2F 6 (1 + ε)
..
OPTwith prob. 9/10
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 21 / 50
r Distinct Columns. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
W =
1 1 1 1 1 12 2 2 1 1 33 3 3 1 1 54 4 4 1 1 15 5 5 1 1 36 6 6 1 1 5
, r = 3
Output : rank-k A ∈ Rn×n s.t.‖W (A − A)‖2F 6 (1 + ε)
..
OPTwith prob. 9/10in O((nnz(A) + nnz(W )) · nγ) + n · 2O(k2r2/ε) time
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 21 / 50
r Distinct Columns. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
W =
1 1 1 1 1 12 2 2 1 1 33 3 3 1 1 54 4 4 1 1 15 5 5 1 1 36 6 6 1 1 5
, r = 3
Output : rank-k A ∈ Rn×n s.t.‖W (A − A)‖2F 6 (1 + ε)
..
OPTwith prob. 9/10in O((nnz(A) + nnz(W )) · nγ) + n · 2O(k2r2/ε) timefor an arbitrarily small constant γ > 0
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 21 / 50
r Distinct Columns. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
W =
1 1 1 1 1 12 2 2 1 1 33 3 3 1 1 54 4 4 1 1 15 5 5 1 1 36 6 6 1 1 5
, r = 3
Output : rank-k A ∈ Rn×n s.t.‖W (A − A)‖2F 6 (1 + ε)
..
OPTwith prob. 9/10in O((nnz(A) + nnz(W )) · nγ) + n · 2O(k2r2/ε) timefor an arbitrarily small constant γ > 0fixed parameter tractable
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 21 / 50
Rank r. .
. .Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 22 / 50
Rank r. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 22 / 50
Rank r. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
W =
1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6
, r = 3
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 22 / 50
Rank r. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
W =
1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6
, r = 3
Output :
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 22 / 50
Rank r. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
W =
1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6
, r = 3
Output : rank-k A ∈ Rn×n s.t.
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 22 / 50
Rank r. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
W =
1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6
, r = 3
Output : rank-k A ∈ Rn×n s.t.‖W (A − A)‖2F 6 (1 + ε) min
rank−k A ′‖W (A − A ′)‖2F
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 22 / 50
Rank r. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
W =
1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6
, r = 3
Output : rank-k A ∈ Rn×n s.t.‖W (A − A)‖2F 6 (1 + ε) min
rank−k A ′
.
∑i,j
W 2i,j(Ai,j − A ′i,j)
2
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 22 / 50
Rank r. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
W =
1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6
, r = 3
Output : rank-k A ∈ Rn×n s.t.‖W (A − A)‖2F 6 (1 + ε)
..
OPT
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 22 / 50
Rank r. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
W =
1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6
, r = 3
Output : rank-k A ∈ Rn×n s.t.‖W (A − A)‖2F 6 (1 + ε)
..
OPTwith prob. 9/10
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 22 / 50
Rank r. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
W =
1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6
, r = 3
Output : rank-k A ∈ Rn×n s.t.‖W (A − A)‖2F 6 (1 + ε)
..
OPTwith prob. 9/10
in nO(k2r/ε) time
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 22 / 50
Rank r. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
W =
1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6
, r = 3
Output : rank-k A ∈ Rn×n s.t.‖W (A − A)‖2F 6 (1 + ε)
..
OPTwith prob. 9/10
in nO(k2r/ε) timepreviously only r=1 was known to be in polynomial time
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 22 / 50
Random-4SAT Hypothesis. .
. .Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 23 / 50
Random-4SAT Hypothesis. .
. .
[Feige’02, Goerdt-Lanka’04]
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 23 / 50
Random-4SAT Hypothesis. .
. .
[Feige’02, Goerdt-Lanka’04]
Given : random 4-SAT formula S on n variables
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 23 / 50
Random-4SAT Hypothesis. .
. .
[Feige’02, Goerdt-Lanka’04]
Given : random 4-SAT formula S on n variableseach clause has 4 literals
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 23 / 50
Random-4SAT Hypothesis. .
. .
[Feige’02, Goerdt-Lanka’04]
Given : random 4-SAT formula S on n variableseach clause has 4 literalseach of Θ(n4) clauses is picked ind. with prob. Θ(1/n3)m = Θ(n) is the number of clauses
(x4 ∨ x32 ∨ x9 ∨ x20)S1
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 23 / 50
Random-4SAT Hypothesis. .
. .
[Feige’02, Goerdt-Lanka’04]
Given : random 4-SAT formula S on n variableseach clause has 4 literalseach of Θ(n4) clauses is picked ind. with prob. Θ(1/n3)m = Θ(n) is the number of clauses
Any algorithm that :
(x4 ∨ x32 ∨ x9 ∨ x20)S1
(x2 ∨ x13 ∨ x5 ∨ x25)S2
· · · · · ·(x11 ∨ x19 ∨ x24 ∨ x6)Sm
S = S1 ∧ S2 ∧ · · ·∧ Sm
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 23 / 50
Random-4SAT Hypothesis. .
. .
[Feige’02, Goerdt-Lanka’04]
Given : random 4-SAT formula S on n variableseach clause has 4 literalseach of Θ(n4) clauses is picked ind. with prob. Θ(1/n3)m = Θ(n) is the number of clauses
Any algorithm that :outputs 1 with prob. 1 when S is satisfiableoutputs 0 with prob. > 1/2probability is over the input instances
(x4 ∨ x32 ∨ x9 ∨ x20)S1
(x2 ∨ x13 ∨ x5 ∨ x25)S2
· · · · · ·(x11 ∨ x19 ∨ x24 ∨ x6)Sm
S = S1 ∧ S2 ∧ · · ·∧ Sm
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 23 / 50
Random-4SAT Hypothesis. .
. .
[Feige’02, Goerdt-Lanka’04]
Given : random 4-SAT formula S on n variableseach clause has 4 literalseach of Θ(n4) clauses is picked ind. with prob. Θ(1/n3)m = Θ(n) is the number of clauses
Any algorithm that :outputs 1 with prob. 1 when S is satisfiableoutputs 0 with prob. > 1/2probability is over the input instances
Requires : 2Ω(n) time
(x4 ∨ x32 ∨ x9 ∨ x20)S1
(x2 ∨ x13 ∨ x5 ∨ x25)S2
· · · · · ·(x11 ∨ x19 ∨ x24 ∨ x6)Sm
S = S1 ∧ S2 ∧ · · ·∧ Sm
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 23 / 50
Maximum Edge Biclique(MEB). .
. .Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 24 / 50
Maximum Edge Biclique(MEB). .
. .
Given : A bipartite graph G = (U,V ,E) with |U | = |V | = n
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 24 / 50
Maximum Edge Biclique(MEB). .
. .
Given : A bipartite graph G = (U,V ,E) with |U | = |V | = n
Output : A k1-by-k2 complete bipartite subgraph of G
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 24 / 50
Maximum Edge Biclique(MEB). .
. .
Given : A bipartite graph G = (U,V ,E) with |U | = |V | = n
Output : A k1-by-k2 complete bipartite subgraph of Gs.t. k1 · k2 is maximized
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 24 / 50
Maximum Edge Biclique(MEB). .
. .
Given : A bipartite graph G = (U,V ,E) with |U | = |V | = n
Output : A k1-by-k2 complete bipartite subgraph of Gs.t. k1 · k2 is maximized
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 24 / 50
Maximum Edge Biclique(MEB). .
. .
Given : A bipartite graph G = (U,V ,E) with |U | = |V | = n
Output : A k1-by-k2 complete bipartite subgraph of Gs.t. k1 · k2 is maximized
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 24 / 50
Maximum Edge Biclique(MEB). .
. .
Given : A bipartite graph G = (U,V ,E) with |U | = |V | = n
Output : A k1-by-k2 complete bipartite subgraph of Gs.t. k1 · k2 is maximized
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 24 / 50
Maximum Edge Biclique(MEB). .
. .
Given : A bipartite graph G = (U,V ,E) with |U | = |V | = n
Output : A k1-by-k2 complete bipartite subgraph of Gs.t. k1 · k2 is maximized
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 24 / 50
Maximum Edge Biclique(MEB). .
. .
[Goerdt-Lanka’04]
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 25 / 50
Maximum Edge Biclique(MEB). .
. .
[Goerdt-Lanka’04]
Assume : Random 4-SAT hardness hypothesis
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 25 / 50
Maximum Edge Biclique(MEB). .
. .
[Goerdt-Lanka’04]
Assume : Random 4-SAT hardness hypothesis
there exist two constants ε1 > ε2 > 0 such that
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 25 / 50
Maximum Edge Biclique(MEB). .
. .
[Goerdt-Lanka’04]
Assume : Random 4-SAT hardness hypothesis
there exist two constants ε1 > ε2 > 0 such that
Any algorithm that : distinguishes between
bipartite graphs G = (U,V ,E) with |U | = |V | = n in two cases
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 25 / 50
Maximum Edge Biclique(MEB). .
. .
[Goerdt-Lanka’04]
Assume : Random 4-SAT hardness hypothesis
there exist two constants ε1 > ε2 > 0 such that
Any algorithm that : distinguishes between
bipartite graphs G = (U,V ,E) with |U | = |V | = n in two cases
1. there is a clique of size > (n/16)2(1 + ε1)
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 25 / 50
Maximum Edge Biclique(MEB). .
. .
[Goerdt-Lanka’04]
Assume : Random 4-SAT hardness hypothesis
there exist two constants ε1 > ε2 > 0 such that
Any algorithm that : distinguishes between
bipartite graphs G = (U,V ,E) with |U | = |V | = n in two cases
1. there is a clique of size > (n/16)2(1 + ε1)
2. all bipartite cliques are of size 6 (n/16)2(1 + ε2)
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 25 / 50
Maximum Edge Biclique(MEB). .
. .
[Goerdt-Lanka’04]
Assume : Random 4-SAT hardness hypothesis
there exist two constants ε1 > ε2 > 0 such that
Any algorithm that : distinguishes between
bipartite graphs G = (U,V ,E) with |U | = |V | = n in two cases
1. there is a clique of size > (n/16)2(1 + ε1)
2. all bipartite cliques are of size 6 (n/16)2(1 + ε2)
Requires :
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 25 / 50
Maximum Edge Biclique(MEB). .
. .
[Goerdt-Lanka’04]
Assume : Random 4-SAT hardness hypothesis
there exist two constants ε1 > ε2 > 0 such that
Any algorithm that : distinguishes between
bipartite graphs G = (U,V ,E) with |U | = |V | = n in two cases
1. there is a clique of size > (n/16)2(1 + ε1)
2. all bipartite cliques are of size 6 (n/16)2(1 + ε2)
Requires : 2Ω(n) time
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 25 / 50
Reduction. .
. .Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 26 / 50
Reduction. .
. .
MEB
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 26 / 50
Reduction. .
. .
MEB Given : U,V with |U | = |V | = n
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 26 / 50
Reduction. .
. .
MEB Given : U,V with |U | = |V | = nOutput : a biclique s.t. k1k2 is maximized
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 26 / 50
Reduction. .
. .
MEB Given : U,V with |U | = |V | = nOutput : a biclique s.t. k1k2 is maximized
( complement ) |E |− k1k2 is minimized
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 26 / 50
Reduction. .
. .
MEB Given : U,V with |U | = |V | = nOutput : a biclique s.t. k1k2 is maximized
( complement ) |E |− k1k2 is minimized
WLRA
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 26 / 50
Reduction. .
. .
MEB Given : U,V with |U | = |V | = nOutput : a biclique s.t. k1k2 is maximized
( complement ) |E |− k1k2 is minimized
WLRA Given : A ∈ 0,1n×n,W ∈ Rn×n
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 26 / 50
Reduction. .
. .
MEB Given : U,V with |U | = |V | = nOutput : a biclique s.t. k1k2 is maximized
( complement ) |E |− k1k2 is minimized
WLRA Given : A ∈ 0,1n×n,W ∈ Rn×n
Output : u, v ∈ Rn s.t. ‖W (uv> − A)‖2F is minimized
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 26 / 50
Reduction. .
. .
MEB Given : U,V with |U | = |V | = nOutput : a biclique s.t. k1k2 is maximized
( complement ) |E |− k1k2 is minimized
WLRA Given : A ∈ 0,1n×n,W ∈ Rn×n
Output : u, v ∈ Rn s.t. ‖W (uv> − A)‖2F is minimized
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 26 / 50
Reduction. .
. .
MEB Given : U,V with |U | = |V | = nOutput : a biclique s.t. k1k2 is maximized
( complement ) |E |− k1k2 is minimized
WLRA Given : A ∈ 0,1n×n,W ∈ Rn×n
Output : u, v ∈ Rn s.t. ‖W (uv> − A)‖2F is minimized
Ai,j =
10
if edge (Ui ,Vj) exists
otherwise
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 26 / 50
Reduction. .
. .
MEB Given : U,V with |U | = |V | = nOutput : a biclique s.t. k1k2 is maximized
( complement ) |E |− k1k2 is minimized
WLRA Given : A ∈ 0,1n×n,W ∈ Rn×n
Output : u, v ∈ Rn s.t. ‖W (uv> − A)‖2F is minimized
Ai,j =
10
if edge (Ui ,Vj) exists
otherwiseW i,j =
1n6
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 26 / 50
Main Results - Hardness. .
. .Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 27 / 50
Main Results - Hardness. .
. .
Weighted Low Rank Approximation Hardness
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 27 / 50
Main Results - Hardness. .
. .
Weighted Low Rank Approximation Hardness
Given : A ∈ Rn×n, distinct r columns W ∈ Rn×n
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 27 / 50
Main Results - Hardness. .
. .
Weighted Low Rank Approximation Hardness
Given : A ∈ Rn×n, distinct r columns W ∈ Rn×n
k ∈ N, ε > 0,W ij ∈ 0,1,2, · · · ,poly(n)
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 27 / 50
Main Results - Hardness. .
. .
Weighted Low Rank Approximation Hardness
Given : A ∈ Rn×n, distinct r columns W ∈ Rn×n
k ∈ N, ε > 0,W ij ∈ 0,1,2, · · · ,poly(n)
Output : rank-k A s.t. ‖W (A − A)‖2F 6 (1 + ε)OPTwith prob. 9/10
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 27 / 50
Main Results - Hardness. .
. .
Weighted Low Rank Approximation Hardness
Given : A ∈ Rn×n, distinct r columns W ∈ Rn×n
k ∈ N, ε > 0,W ij ∈ 0,1,2, · · · ,poly(n)
Output : rank-k A s.t. ‖W (A − A)‖2F 6 (1 + ε)OPTwith prob. 9/10
Assume : Random-4SAT Hypothesis[Feige’02, Goerdt-Lanka’04]
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 27 / 50
Main Results - Hardness. .
. .
Weighted Low Rank Approximation Hardness
Given : A ∈ Rn×n, distinct r columns W ∈ Rn×n
k ∈ N, ε > 0,W ij ∈ 0,1,2, · · · ,poly(n)
Output : rank-k A s.t. ‖W (A − A)‖2F 6 (1 + ε)OPTwith prob. 9/10
Assume : Random-4SAT Hypothesis[Feige’02, Goerdt-Lanka’04]
∃ε0 > 0, for any algorithm with ε < ε0 and k > 1
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 27 / 50
Main Results - Hardness. .
. .
Weighted Low Rank Approximation Hardness
Given : A ∈ Rn×n, distinct r columns W ∈ Rn×n
k ∈ N, ε > 0,W ij ∈ 0,1,2, · · · ,poly(n)
Output : rank-k A s.t. ‖W (A − A)‖2F 6 (1 + ε)OPTwith prob. 9/10
Assume : Random-4SAT Hypothesis[Feige’02, Goerdt-Lanka’04]
∃ε0 > 0, for any algorithm with ε < ε0 and k > 1Requires : 2Ω(r) time
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 27 / 50
Algorithmic Techniques
Tools
I Polynomial system verifier [Renegar’92, Basu-Pollack-Roy’96]I Lower bound on the cost [Jeronimo-Perrucci-Tsigaridas’13]I Multiple regression sketch
Algorithm, assuming a rank r weight matrix W
I Warmup, inefficient WLRA algorithmI “Guessing a sketch”
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 28 / 50
Algorithmic Techniques
ToolsI Polynomial system verifier [Renegar’92, Basu-Pollack-Roy’96]
I Lower bound on the cost [Jeronimo-Perrucci-Tsigaridas’13]I Multiple regression sketch
Algorithm, assuming a rank r weight matrix W
I Warmup, inefficient WLRA algorithmI “Guessing a sketch”
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 28 / 50
Algorithmic Techniques
ToolsI Polynomial system verifier [Renegar’92, Basu-Pollack-Roy’96]I Lower bound on the cost [Jeronimo-Perrucci-Tsigaridas’13]
I Multiple regression sketchAlgorithm, assuming a rank r weight matrix W
I Warmup, inefficient WLRA algorithmI “Guessing a sketch”
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 28 / 50
Algorithmic Techniques
ToolsI Polynomial system verifier [Renegar’92, Basu-Pollack-Roy’96]I Lower bound on the cost [Jeronimo-Perrucci-Tsigaridas’13]I Multiple regression sketch
Algorithm, assuming a rank r weight matrix W
I Warmup, inefficient WLRA algorithmI “Guessing a sketch”
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 28 / 50
Algorithmic Techniques
ToolsI Polynomial system verifier [Renegar’92, Basu-Pollack-Roy’96]I Lower bound on the cost [Jeronimo-Perrucci-Tsigaridas’13]I Multiple regression sketch
Algorithm, assuming a rank r weight matrix W
I Warmup, inefficient WLRA algorithmI “Guessing a sketch”
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 28 / 50
Algorithmic Techniques
ToolsI Polynomial system verifier [Renegar’92, Basu-Pollack-Roy’96]I Lower bound on the cost [Jeronimo-Perrucci-Tsigaridas’13]I Multiple regression sketch
Algorithm, assuming a rank r weight matrix WI Warmup, inefficient WLRA algorithm
I “Guessing a sketch”
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 28 / 50
Algorithmic Techniques
ToolsI Polynomial system verifier [Renegar’92, Basu-Pollack-Roy’96]I Lower bound on the cost [Jeronimo-Perrucci-Tsigaridas’13]I Multiple regression sketch
Algorithm, assuming a rank r weight matrix WI Warmup, inefficient WLRA algorithmI “Guessing a sketch”
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 28 / 50
Polynomial System Verifier (Recall)
. .
. .
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 29 / 50
Polynomial System Verifier (Recall)
. .
. .[Renegar’92, Basu-Pollack-Roy’96]
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 29 / 50
Polynomial System Verifier (Recall)
. .
. .[Renegar’92, Basu-Pollack-Roy’96]
Given : a real polynomial system P(x)
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 29 / 50
Polynomial System Verifier (Recall)
. .
. .[Renegar’92, Basu-Pollack-Roy’96]
Given : a real polynomial system P(x)v : #variables, x = (x1, x2, · · · , xv )
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 29 / 50
Polynomial System Verifier (Recall)
. .
. .[Renegar’92, Basu-Pollack-Roy’96]
Given : a real polynomial system P(x)v : #variables, x = (x1, x2, · · · , xv )
m : #polynomial constraints fi(x) > 0, ∀i ∈ [m]
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 29 / 50
Polynomial System Verifier (Recall)
. .
. .[Renegar’92, Basu-Pollack-Roy’96]
Given : a real polynomial system P(x)v : #variables, x = (x1, x2, · · · , xv )
m : #polynomial constraints fi(x) > 0, ∀i ∈ [m]
d : maximum degree of all polynomial constraints
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 29 / 50
Polynomial System Verifier (Recall)
. .
. .[Renegar’92, Basu-Pollack-Roy’96]
Given : a real polynomial system P(x)v : #variables, x = (x1, x2, · · · , xv )
m : #polynomial constraints fi(x) > 0, ∀i ∈ [m]
d : maximum degree of all polynomial constraintsH : the bitsizes of the coefficients of the polynomials
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 29 / 50
Polynomial System Verifier (Recall)
. .
. .[Renegar’92, Basu-Pollack-Roy’96]
Given : a real polynomial system P(x)v : #variables, x = (x1, x2, · · · , xv )
m : #polynomial constraints fi(x) > 0, ∀i ∈ [m]
d : maximum degree of all polynomial constraintsH : the bitsizes of the coefficients of the polynomials
It takes (md)O(v) poly(H) time todecide if there exists a solution to polynomial system P
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 29 / 50
Lower Bound on the Cost
. .
. .
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 30 / 50
Lower Bound on the Cost
. .
. .[Jeronimo-Perrucci-Tsigaridas’13]
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 30 / 50
Lower Bound on the Cost
. .
. .[Jeronimo-Perrucci-Tsigaridas’13]
Given: a real polynomial system P(x)
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 30 / 50
Lower Bound on the Cost
. .
. .[Jeronimo-Perrucci-Tsigaridas’13]
Given: a real polynomial system P(x)
v : #variables, x = (x1, x2, · · · , xv )
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 30 / 50
Lower Bound on the Cost
. .
. .[Jeronimo-Perrucci-Tsigaridas’13]
Given: a real polynomial system P(x)
v : #variables, x = (x1, x2, · · · , xv )
m : #polynomials constraints fi(x) > 0, ∀i ∈ [m]
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 30 / 50
Lower Bound on the Cost
. .
. .[Jeronimo-Perrucci-Tsigaridas’13]
Given: a real polynomial system P(x)
v : #variables, x = (x1, x2, · · · , xv )
m : #polynomials constraints fi(x) > 0, ∀i ∈ [m]
d : maximum degree of all polynomials
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 30 / 50
Lower Bound on the Cost
. .
. .[Jeronimo-Perrucci-Tsigaridas’13]
Given: a real polynomial system P(x)
v : #variables, x = (x1, x2, · · · , xv )
m : #polynomials constraints fi(x) > 0, ∀i ∈ [m]
d : maximum degree of all polynomialsH : the bitsizes of the coefficients of the polynomials
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 30 / 50
Lower Bound on the Cost
. .
. .[Jeronimo-Perrucci-Tsigaridas’13]
Given: a real polynomial system P(x)
v : #variables, x = (x1, x2, · · · , xv )
m : #polynomials constraints fi(x) > 0, ∀i ∈ [m]
d : maximum degree of all polynomialsH : the bitsizes of the coefficients of the polynomials
.
Rv
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 30 / 50
Lower Bound on the Cost
. .
. .[Jeronimo-Perrucci-Tsigaridas’13]
Given: a real polynomial system P(x)
v : #variables, x = (x1, x2, · · · , xv )
m : #polynomials constraints fi(x) > 0, ∀i ∈ [m]
d : maximum degree of all polynomialsH : the bitsizes of the coefficients of the polynomials
.
Rv
.
T : x ∈ Rv |f1(x) > 0, · · · , fm(x) > 0
T
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 30 / 50
Lower Bound on the Cost
. .
. .[Jeronimo-Perrucci-Tsigaridas’13]
Given: a real polynomial system P(x)
v : #variables, x = (x1, x2, · · · , xv )
m : #polynomials constraints fi(x) > 0, ∀i ∈ [m]
d : maximum degree of all polynomialsH : the bitsizes of the coefficients of the polynomials
.
Rv
.
T : x ∈ Rv |f1(x) > 0, · · · , fm(x) > 0
T
.
minimum value that nonnegative g takes over T ∩ Ball is
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 30 / 50
Lower Bound on the Cost
. .
. .[Jeronimo-Perrucci-Tsigaridas’13]
Given: a real polynomial system P(x)
v : #variables, x = (x1, x2, · · · , xv )
m : #polynomials constraints fi(x) > 0, ∀i ∈ [m]
d : maximum degree of all polynomialsH : the bitsizes of the coefficients of the polynomials
.
Rv
.
T : x ∈ Rv |f1(x) > 0, · · · , fm(x) > 0
T
.
minimum value that nonnegative g takes over T ∩ Ball is
Ball
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 30 / 50
Lower Bound on the Cost
. .
. .[Jeronimo-Perrucci-Tsigaridas’13]
Given: a real polynomial system P(x)
v : #variables, x = (x1, x2, · · · , xv )
m : #polynomials constraints fi(x) > 0, ∀i ∈ [m]
d : maximum degree of all polynomialsH : the bitsizes of the coefficients of the polynomials
.
Rv
.
T : x ∈ Rv |f1(x) > 0, · · · , fm(x) > 0
T
.
minimum value that nonnegative g takes over T ∩ Ball is
Ball
either 0 or > (2H + m)−dv
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 30 / 50
Lower Bound on the Cost
. .
. .[Jeronimo-Perrucci-Tsigaridas’13]
Given: a real polynomial system P(x)
v : #variables, x = (x1, x2, · · · , xv )
m : #polynomials constraints fi(x) > 0, ∀i ∈ [m]
d : maximum degree of all polynomialsH : the bitsizes of the coefficients of the polynomials
.
Rv
.
T : x ∈ Rv |f1(x) > 0, · · · , fm(x) > 0
T
.
minimum value that nonnegative g takes over T ∩ Ball is
Ball
either 0 or > (2H + m)−dv
g = (x1x2 − 1)2 + x22
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 30 / 50
Multiple Regression Sketch. .
. .Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 31 / 50
Multiple Regression Sketch. .
. .
Given :
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 31 / 50
Multiple Regression Sketch. .
. .
Given : A(1),A(2), · · · ,A(m) ∈ Rn×k
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 31 / 50
Multiple Regression Sketch. .
. .
Given : A(1),A(2), · · · ,A(m) ∈ Rn×k
b(1),b(2), · · · ,b(m) ∈ Rn×1
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 31 / 50
Multiple Regression Sketch. .
. .
Given : A(1),A(2), · · · ,A(m) ∈ Rn×k
b(1),b(2), · · · ,b(m) ∈ Rn×1
Let x(j) = arg minx∈Rk×1
‖A(j)x − b(j)‖,∀j ∈ [m]
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 31 / 50
Multiple Regression Sketch. .
. .
Given : A(1),A(2), · · · ,A(m) ∈ Rn×k
b(1),b(2), · · · ,b(m) ∈ Rn×1
Let x(j) = arg minx∈Rk×1
‖A(j)x − b(j)‖,∀j ∈ [m]
Choose : S to be a random Gaussian matrix
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 31 / 50
Multiple Regression Sketch. .
. .
Given : A(1),A(2), · · · ,A(m) ∈ Rn×k
b(1),b(2), · · · ,b(m) ∈ Rn×1
Let x(j) = arg minx∈Rk×1
‖A(j)x − b(j)‖,∀j ∈ [m]
Choose : S to be a random Gaussian matrixDenote y(j) = arg min
y∈Rk×1‖SA(j)y − Sb(j)‖, ∀j ∈ [m]
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 31 / 50
Multiple Regression Sketch. .
. .
Given : A(1),A(2), · · · ,A(m) ∈ Rn×k
b(1),b(2), · · · ,b(m) ∈ Rn×1
Let x(j) = arg minx∈Rk×1
‖A(j)x − b(j)‖,∀j ∈ [m]
Choose : S to be a random Gaussian matrixDenote y(j) = arg min
y∈Rk×1‖SA(j)y − Sb(j)‖, ∀j ∈ [m]
Gaurantee :
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 31 / 50
Multiple Regression Sketch. .
. .
Given : A(1),A(2), · · · ,A(m) ∈ Rn×k
b(1),b(2), · · · ,b(m) ∈ Rn×1
Let x(j) = arg minx∈Rk×1
‖A(j)x − b(j)‖,∀j ∈ [m]
Choose : S to be a random Gaussian matrixDenote y(j) = arg min
y∈Rk×1‖SA(j)y − Sb(j)‖, ∀j ∈ [m]
Gaurantee : For all ε ∈ (0,1/2), one can set t = O(k/ε) such that
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 31 / 50
Multiple Regression Sketch. .
. .
Given : A(1),A(2), · · · ,A(m) ∈ Rn×k
b(1),b(2), · · · ,b(m) ∈ Rn×1
Let x(j) = arg minx∈Rk×1
‖A(j)x − b(j)‖,∀j ∈ [m]
Choose : S to be a random Gaussian matrixDenote y(j) = arg min
y∈Rk×1‖SA(j)y − Sb(j)‖, ∀j ∈ [m]
Gaurantee : For all ε ∈ (0,1/2), one can set t = O(k/ε) such thatm∑
j=1‖A(j)y(j) − b(j)‖22 6 (1 + ε)
m∑j=1‖A(j)x(j) − b(j)‖22
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 31 / 50
Multiple Regression Sketch. .
. .
Given : A(1),A(2), · · · ,A(m) ∈ Rn×k
b(1),b(2), · · · ,b(m) ∈ Rn×1
Let x(j) = arg minx∈Rk×1
‖A(j)x − b(j)‖,∀j ∈ [m]
Choose : S to be a random Gaussian matrixDenote y(j) = arg min
y∈Rk×1‖SA(j)y − Sb(j)‖, ∀j ∈ [m]
Gaurantee : For all ε ∈ (0,1/2), one can set t = O(k/ε) such thatm∑
j=1‖A(j)y(j) − b(j)‖22 6 (1 + ε)
m∑j=1‖A(j)x(j) − b(j)‖22
with prob. 9/10
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 31 / 50
Multiple Regression Sketch. .
. .
Given : A(1),A(2), · · · ,A(m) ∈ Rn×k
b(1),b(2), · · · ,b(m) ∈ Rn×1
Let x(j) = arg minx∈Rk×1
‖A(j)x − b(j)‖,∀j ∈ [m]
Choose : S to be a random Gaussian matrixDenote y(j) = arg min
y∈Rk×1‖SA(j)y − Sb(j)‖, ∀j ∈ [m]
Gaurantee : For all ε ∈ (0,1/2), one can set t = O(k/ε) such thatm∑
j=1‖A(j)y(j) − b(j)‖22 6 (1 + ε)
m∑j=1‖A(j)x(j) − b(j)‖22
with prob. 9/10
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 31 / 50
Warmup, inefficient WLRA Algorithm. .
. .Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 32 / 50
Warmup, inefficient WLRA Algorithm. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 32 / 50
Warmup, inefficient WLRA Algorithm. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
.
Aij ∈ 0,±1,±2, · · · ,±∆
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 32 / 50
Warmup, inefficient WLRA Algorithm. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
W ij ∈ 0,1,2, · · · ,∆
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 32 / 50
Warmup, inefficient WLRA Algorithm. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : rank-k A s.t. ‖W (A − A)‖2F 6 (1 + ε)OPT
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 32 / 50
Warmup, inefficient WLRA Algorithm. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : rank-k A s.t. ‖W (A − A)‖2F 6 (1 + ε)OPTAlgorithm :
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 32 / 50
Warmup, inefficient WLRA Algorithm. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : rank-k A s.t. ‖W (A − A)‖2F 6 (1 + ε)OPTAlgorithm :
1. create 2nk variables for U,V> ∈ Rn×k
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 32 / 50
Warmup, inefficient WLRA Algorithm. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : rank-k A s.t. ‖W (A − A)‖2F 6 (1 + ε)OPTAlgorithm :
1. create 2nk variables for U,V> ∈ Rn×k
2. write polynomial g(x1, · · · , x2nk ) = ‖W (A − UV )‖2F
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 32 / 50
Warmup, inefficient WLRA Algorithm. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : rank-k A s.t. ‖W (A − A)‖2F 6 (1 + ε)OPTAlgorithm :
1. create 2nk variables for U,V> ∈ Rn×k
2. write polynomial g(x1, · · · , x2nk ) = ‖W (A − UV )‖2F3. pick C ∈ [L−,L+], run polynomial verifier g(x) 6 C
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 32 / 50
Warmup, inefficient WLRA Algorithm. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : rank-k A s.t. ‖W (A − A)‖2F 6 (1 + ε)OPTAlgorithm :
1. create 2nk variables for U,V> ∈ Rn×k
2. write polynomial g(x1, · · · , x2nk ) = ‖W (A − UV )‖2F3. pick C ∈ [L−,L+], run polynomial verifier g(x) 6 C4. optimize C by binary search over [L−,L+]
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 32 / 50
Warmup, inefficient WLRA Algorithm. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : rank-k A s.t. ‖W (A − A)‖2F 6 (1 + ε)OPTAlgorithm :
1. create 2nk variables for U,V> ∈ Rn×k
2. write polynomial g(x1, · · · , x2nk ) = ‖W (A − UV )‖2F3. pick C ∈ [L−,L+], run polynomial verifier g(x) 6 C4. optimize C by binary search over [L−,L+]
Time : 2Ω(nk)
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 32 / 50
Warmup, inefficient WLRA Algorithm. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : rank-k A s.t. ‖W (A − A)‖2F 6 (1 + ε)OPTAlgorithm :
1. create 2nk variables for U,V> ∈ Rn×k
2. write polynomial g(x1, · · · , x2nk ) = ‖W (A − UV )‖2F3. pick C ∈ [L−,L+], run polynomial verifier g(x) 6 C4. optimize C by binary search over [L−,L+]
Time : 2Ω(nk)
How can we do better?
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 32 / 50
Warmup, inefficient WLRA Algorithm. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : rank-k A s.t. ‖W (A − A)‖2F 6 (1 + ε)OPTAlgorithm :
1. create 2nk variables for U,V> ∈ Rn×k
Time : 2Ω(nk)
How can we do better?
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 32 / 50
Warmup, inefficient WLRA Algorithm. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : rank-k A s.t. ‖W (A − A)‖2F 6 (1 + ε)OPTAlgorithm :
1. create 2nk variables for U,V> ∈ Rn×k
Time : 2Ω(nk)
How can we do better?polynomial verifier runs in (# constraints · degree)O(# variables)
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 32 / 50
Warmup, inefficient WLRA Algorithm. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : rank-k A s.t. ‖W (A − A)‖2F 6 (1 + ε)OPTAlgorithm :
1. create 2nk variables for U,V> ∈ Rn×k
Time : 2Ω(nk)
How can we do better?polynomial verifier runs in (# constraints · degree)O(# variables)
lower bound on cost (# constraints)−degreeO(# variables)
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 32 / 50
Warmup, inefficient WLRA Algorithm. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : rank-k A s.t. ‖W (A − A)‖2F 6 (1 + ε)OPTAlgorithm :
1. create 2nk variables for U,V> ∈ Rn×k
Time : 2Ω(nk)
How can we do better?polynomial verifier runs in (# constraints · degree)O(# variables)
lower bound on cost (# constraints)−degreeO(# variables)
write a polynomial with few #variables, i.e. poly(kr/ε)
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 32 / 50
Warmup, inefficient WLRA Algorithm. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : rank-k A s.t. ‖W (A − A)‖2F 6 (1 + ε)OPTAlgorithm :
1. create 2nk variables for U,V> ∈ Rn×k
Time : 2Ω(nk)
How can we do better?polynomial verifier runs in (# constraints · degree)O(# variables)
lower bound on cost (# constraints)−degreeO(# variables)
write a polynomial with few #variables, i.e. poly(kr/ε)without blowing up degree and #constraints too much
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 32 / 50
Main Idea
. .
. .
To reduce the number of variables to poly(kr/ε) :
1. Multiple regression sketch with O(k/ε) rows
2. Weight matrix W has rank at most r
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 33 / 50
Guess a Sketch. .
. .Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 34 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 34 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
.
Aij ∈ 0,±1,±2, · · · ,±∆
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 34 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
W ij ∈ 0,1,2, · · · ,∆
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 34 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 34 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm : W j be j th column of WDW j be diagonal matrix with vector W j
W =
1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 34 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm : W j be j th column of WDW j be diagonal matrix with vector W j
.
W =
1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6
DW 1 =
10
10
10
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 34 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm : W j be j th column of WDW j be diagonal matrix with vector W j
..
W =
1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6
DW 2 =
11
10
00
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 34 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm : W j be j th column of WDW j be diagonal matrix with vector W j
...
W =
1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6
DW 3 =
12
34
56
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 34 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm : W j be j th column of WDW j be diagonal matrix with vector W j
....
W =
1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6
DW 4 =
21
20
10
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 34 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm : W j be j th column of WDW j be diagonal matrix with vector W j
.....
W =
1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6
DW 5 =
22
44
66
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 34 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm : W j be j th column of WDW j be diagonal matrix with vector W j
......
W =
1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6
DW 6 =
23
44
56
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 34 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm :
......
.
U
V
W A )− (
2
F
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 34 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm :
......
.
U
V
W A )− (
2
F
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 34 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm :
......
.n∑j=1‖DW j UV j − DW j Aj‖22
U
V
W A )− (
2
F
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 34 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm :
......
.n∑j=1‖DW j UV j − DW j Aj‖22
U
V
W A )− (
2
F
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 34 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm :
......
.n∑j=1‖DW j UV j − DW j Aj‖22
n∑i=1‖U iVDW i − AiDW i‖22
U
V
W A )− (
2
F
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 34 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm :
......
.n∑j=1‖DW j UV j − DW j Aj‖22
n∑i=1‖U iVDW i − AiDW i‖22
Sketch by Gaussian S,T> ∈ Rt×n
S T
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 34 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm :
......
.n∑j=1‖DW j UV j − DW j Aj‖22
n∑i=1‖U iVDW i − AiDW i‖22
Sketch by Gaussian S,T> ∈ Rt×nn∑
j=1‖SDW j UV j − SDW j Aj‖22
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 34 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm :
......
.n∑j=1‖DW j UV j − DW j Aj‖22
n∑i=1‖U iVDW i − AiDW i‖22
Sketch by Gaussian S,T> ∈ Rt×nn∑
j=1‖SDW j UV j − SDW j Aj‖22
n∑i=1‖U iVDW i T − AiDW i T‖22
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 34 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm :
......
.n∑j=1‖DW j UV j − DW j Aj‖22
n∑i=1‖U iVDW i − AiDW i‖22
Sketch by Gaussian S,T> ∈ Rt×nn∑
j=1‖SDW j UV j − SDW j Aj‖22
n∑i=1‖U iVDW i T − AiDW i T‖22
Guess SDW j U ∈ Rt×k and VDW i T ∈ Rk×t
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 34 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm :
......
.n∑j=1‖DW j UV j − DW j Aj‖22
n∑i=1‖U iVDW i − AiDW i‖22
Sketch by Gaussian S,T> ∈ Rt×nn∑
j=1‖SDW j UV j − SDW j Aj‖22
n∑i=1‖U iVDW i T − AiDW i T‖22
Guess SDW j U ∈ Rt×k and VDW i T ∈ Rk×t
.
SDW 1U = S
U
10
10
10
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 34 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm :
......
.n∑j=1‖DW j UV j − DW j Aj‖22
n∑i=1‖U iVDW i − AiDW i‖22
Sketch by Gaussian S,T> ∈ Rt×nn∑
j=1‖SDW j UV j − SDW j Aj‖22
n∑i=1‖U iVDW i T − AiDW i T‖22
Guess SDW j U ∈ Rt×k and VDW i T ∈ Rk×t
..
SDW 2U = S
U
11
10
00
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 34 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm :
......
.n∑j=1‖DW j UV j − DW j Aj‖22
n∑i=1‖U iVDW i − AiDW i‖22
Sketch by Gaussian S,T> ∈ Rt×nn∑
j=1‖SDW j UV j − SDW j Aj‖22
n∑i=1‖U iVDW i T − AiDW i T‖22
Guess SDW j U ∈ Rt×k and VDW i T ∈ Rk×t
...
SDW 3U = S
U
12
34
56
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 34 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm :
......
.n∑j=1‖DW j UV j − DW j Aj‖22
n∑i=1‖U iVDW i − AiDW i‖22
Sketch by Gaussian S,T> ∈ Rt×nn∑
j=1‖SDW j UV j − SDW j Aj‖22
n∑i=1‖U iVDW i T − AiDW i T‖22
Guess SDW j U ∈ Rt×k and VDW i T ∈ Rk×t
....
SDW 4U = S
U
21
20
10
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 34 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm :
......
.n∑j=1‖DW j UV j − DW j Aj‖22
n∑i=1‖U iVDW i − AiDW i‖22
Sketch by Gaussian S,T> ∈ Rt×nn∑
j=1‖SDW j UV j − SDW j Aj‖22
n∑i=1‖U iVDW i T − AiDW i T‖22
Guess SDW j U ∈ Rt×k and VDW i T ∈ Rk×t
.....
SDW 5U = S
U
22
44
66
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 34 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm :
......
.n∑j=1‖DW j UV j − DW j Aj‖22
n∑i=1‖U iVDW i − AiDW i‖22
Sketch by Gaussian S,T> ∈ Rt×nn∑
j=1‖SDW j UV j − SDW j Aj‖22
n∑i=1‖U iVDW i T − AiDW i T‖22
Guess SDW j U ∈ Rt×k and VDW i T ∈ Rk×t
......
SDW 6U = S
U
23
44
56
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 34 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm :
......
.n∑j=1‖DW j UV j − DW j Aj‖22
n∑i=1‖U iVDW i − AiDW i‖22
Sketch by Gaussian S,T> ∈ Rt×nn∑
j=1‖SDW j UV j − SDW j Aj‖22
n∑i=1‖U iVDW i T − AiDW i T‖22
Guess SDW j U ∈ Rt×k and VDW i T ∈ Rk×t
......
Create t × k variables for each of n SDW j U s?
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 34 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm :
......
.n∑j=1‖DW j UV j − DW j Aj‖22
n∑i=1‖U iVDW i − AiDW i‖22
Sketch by Gaussian S,T> ∈ Rt×nn∑
j=1‖SDW j UV j − SDW j Aj‖22
n∑i=1‖U iVDW i T − AiDW i T‖22
Guess SDW j U ∈ Rt×k and VDW i T ∈ Rk×t
......
Create t × k variables for each of n SDW j U s?
.
×t × kn
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 34 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm :
......
.n∑j=1‖DW j UV j − DW j Aj‖22
n∑i=1‖U iVDW i − AiDW i‖22
Sketch by Gaussian S,T> ∈ Rt×nn∑
j=1‖SDW j UV j − SDW j Aj‖22
n∑i=1‖U iVDW i T − AiDW i T‖22
Guess SDW j U ∈ Rt×k and VDW i T ∈ Rk×t
......
Create t × k variables for each of n SDW j U s?
.
×t × k
.
?
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 34 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
..
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm :
......
.n∑j=1‖DW j UV j − DW j Aj‖22
n∑i=1‖U iVDW i − AiDW i‖22
Sketch by Gaussian S,T> ∈ Rt×nn∑
j=1‖SDW j UV j − SDW j Aj‖22
n∑i=1‖U iVDW i T − AiDW i T‖22
Guess SDW j U ∈ Rt×k and VDW i T ∈ Rk×t
......
Create t × k variables for each of n SDW j U s?
.
×t × k
..
r
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 34 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm : W j be j th column of W
W =
1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 35 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm : W j be j th column of W.
W =
1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6
1 1 10 1 21 1 30 0 41 0 50 0 6
column span of W
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 35 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm : W j be j th column of W..
W =
1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6
101010
=
101010
W 1 = W 1
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 35 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm : W j be j th column of W...
W =
1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6
111000
=
111000
W 2 = W 2
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 35 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm : W j be j th column of W....
W =
1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6
123456
=
123456
W 3 = W 3
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 35 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm : W j be j th column of W.....
W =
1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6
212010
= +
101010
111000
W 4 = W 1 + W 2
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 35 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm : W j be j th column of W......
W =
1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6
224466
= +
101010
123456
W 5 = W 1 + W 3
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 35 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm : W j be j th column of W.......
W =
1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6
234456
= +
111000
123456
W 6 = W 2 + W 3
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 35 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm :....... n∑
j=1‖DW j UV j − DW j Aj‖22
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 35 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm :....... n∑
j=1‖DW j UV j − DW j Aj‖22
sketchn∑
j=1‖SDW j UV j − SDW j Aj‖22
1 + ε
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 35 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm :....... n∑
j=1‖DW j UV j − DW j Aj‖22
sketchn∑
j=1‖SDW j UV j − SDW j Aj‖22
1 + ε
create variables for SDW j U, ∀j ∈ [r ]
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 35 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm :....... n∑
j=1‖DW j UV j − DW j Aj‖22
sketchn∑
j=1‖SDW j UV j − SDW j Aj‖22
1 + ε
create variables for SDW j U, ∀j ∈ [r ]
.
S
U
10
10
10
create t × k variables for SDW 1U
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 35 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm :....... n∑
j=1‖DW j UV j − DW j Aj‖22
sketchn∑
j=1‖SDW j UV j − SDW j Aj‖22
1 + ε
create variables for SDW j U, ∀j ∈ [r ]
..
S
U
11
10
00
create t × k variables for SDW 2U
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 35 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm :....... n∑
j=1‖DW j UV j − DW j Aj‖22
sketchn∑
j=1‖SDW j UV j − SDW j Aj‖22
1 + ε
create variables for SDW j U, ∀j ∈ [r ]
...
S
U
12
34
56
create t × k variables for SDW 3U
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 35 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm :....... n∑
j=1‖DW j UV j − DW j Aj‖22
sketchn∑
j=1‖SDW j UV j − SDW j Aj‖22
1 + ε
create variables for SDW j U, ∀j ∈ [r ]
....
S
U
21
20
10
write SDW 4U as SDW 1U + SDW 2U
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 35 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm :....... n∑
j=1‖DW j UV j − DW j Aj‖22
sketchn∑
j=1‖SDW j UV j − SDW j Aj‖22
1 + ε
create variables for SDW j U, ∀j ∈ [r ]
.....
S
U
1 + 10 + 1
1 + 10 + 0
1 + 00 + 0
write SDW 4U as SDW 1U + SDW 2U
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 35 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm :....... n∑
j=1‖DW j UV j − DW j Aj‖22
sketchn∑
j=1‖SDW j UV j − SDW j Aj‖22
1 + ε
create variables for SDW j U, ∀j ∈ [r ]
......
S
U(
10
10
10
+
11
10
00
)
write SDW 4U as SDW 1U + SDW 2U
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 35 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm :....... n∑
j=1‖DW j UV j − DW j Aj‖22
sketchn∑
j=1‖SDW j UV j − SDW j Aj‖22
1 + ε
create variables for SDW j U, ∀j ∈ [r ]
.......
S
U
22
44
66
write SDW 5U as SDW 1U + SDW 3U
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 35 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm :....... n∑
j=1‖DW j UV j − DW j Aj‖22
sketchn∑
j=1‖SDW j UV j − SDW j Aj‖22
1 + ε
create variables for SDW j U, ∀j ∈ [r ]
........
S
U
1 + 10 + 2
1 + 30 + 4
1 + 50 + 6
write SDW 5U as SDW 1U + SDW 3U
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 35 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm :....... n∑
j=1‖DW j UV j − DW j Aj‖22
sketchn∑
j=1‖SDW j UV j − SDW j Aj‖22
1 + ε
create variables for SDW j U, ∀j ∈ [r ]
.........
S
U(
10
10
10
+
12
34
56
)
write SDW 5U as SDW 1U + SDW 3U
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 35 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm :....... n∑
j=1‖DW j UV j − DW j Aj‖22
sketchn∑
j=1‖SDW j UV j − SDW j Aj‖22
1 + ε
create variables for SDW j U, ∀j ∈ [r ]
.........
S
U
23
44
56
write SDW 6U as SDW 2U + SDW 3U
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 35 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm :....... n∑
j=1‖DW j UV j − DW j Aj‖22
sketchn∑
j=1‖SDW j UV j − SDW j Aj‖22
1 + ε
create variables for SDW j U, ∀j ∈ [r ]
.........
S
U
1 + 11 + 2
1 + 30 + 4
0 + 50 + 6
write SDW 6U as SDW 2U + SDW 3U
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 35 / 50
Guess a Sketch. .
. .
Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0
Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT
Algorithm :....... n∑
j=1‖DW j UV j − DW j Aj‖22
sketchn∑
j=1‖SDW j UV j − SDW j Aj‖22
1 + ε
create variables for SDW j U, ∀j ∈ [r ]
.........
S
U(
11
10
00
+
12
34
56
)
write SDW 6U as SDW 2U + SDW 3U
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 35 / 50
Wrapping Up
Can use an explicit formula for the regression solution in terms ofthe variables created
The regression solution is a rational function, but can use tricks toclear the denominatorMultiple Regression Sketch + Bounded Rank Weight Matrix implya small number of variablesTime : nO(rk2/ε)
(# constraints · degree)O(# variables) = (O(n) ·O(nk))O(krt)
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 36 / 50
Wrapping Up
Can use an explicit formula for the regression solution in terms ofthe variables createdThe regression solution is a rational function, but can use tricks toclear the denominator
Multiple Regression Sketch + Bounded Rank Weight Matrix implya small number of variablesTime : nO(rk2/ε)
(# constraints · degree)O(# variables) = (O(n) ·O(nk))O(krt)
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 36 / 50
Wrapping Up
Can use an explicit formula for the regression solution in terms ofthe variables createdThe regression solution is a rational function, but can use tricks toclear the denominatorMultiple Regression Sketch + Bounded Rank Weight Matrix implya small number of variables
Time : nO(rk2/ε)
(# constraints · degree)O(# variables) = (O(n) ·O(nk))O(krt)
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 36 / 50
Wrapping Up
Can use an explicit formula for the regression solution in terms ofthe variables createdThe regression solution is a rational function, but can use tricks toclear the denominatorMultiple Regression Sketch + Bounded Rank Weight Matrix implya small number of variablesTime : nO(rk2/ε)
(# constraints · degree)O(# variables) = (O(n) ·O(nk))O(krt)
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 36 / 50
Wrapping Up
Can use an explicit formula for the regression solution in terms ofthe variables createdThe regression solution is a rational function, but can use tricks toclear the denominatorMultiple Regression Sketch + Bounded Rank Weight Matrix implya small number of variablesTime : nO(rk2/ε)
(# constraints · degree)O(# variables) = (O(n) ·O(nk))O(krt)
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 36 / 50
Open Problems
For a rank-r weight matrix W , the upper bound is nO(k2r/ε) but thelower bound is only 2Ω(r) - can we close this gap?
Can we prove a hardness result with respect to the parameter k ,e.g., a 2Ω(k) lower bound for WLRA problem?
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 37 / 50
Open Problems
For a rank-r weight matrix W , the upper bound is nO(k2r/ε) but thelower bound is only 2Ω(r) - can we close this gap?Can we prove a hardness result with respect to the parameter k ,e.g., a 2Ω(k) lower bound for WLRA problem?
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 37 / 50
. .
. .
`1 Low Rank Approximation
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 38 / 50
`1-Low Rank Approximation. .
. .Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 39 / 50
`1-Low Rank Approximation. .
. .
Given : A ∈ Rn×n, k ∈ N, α > 1
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 39 / 50
`1-Low Rank Approximation. .
. .
Given : A ∈ Rn×n, k ∈ N, α > 1
Output :
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 39 / 50
`1-Low Rank Approximation. .
. .
Given : A ∈ Rn×n, k ∈ N, α > 1
Output : rank-k A ∈ Rn×n s.t.
A A−
1
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 39 / 50
`1-Low Rank Approximation. .
. .
Given : A ∈ Rn×n, k ∈ N, α > 1
Output : rank-k A ∈ Rn×n s.t.‖A − A‖1 6 α · min
rank−k A ′‖A ′ − A‖1
A A−
1
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 39 / 50
`1-Low Rank Approximation. .
. .
Given : A ∈ Rn×n, k ∈ N, α > 1
Output : rank-k A ∈ Rn×n s.t.‖A − A‖1 6 α · min
rank−k A ′‖A ′ − A‖1
.
∑i,j|A ′i,j − Ai,j |
A A−
1
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 39 / 50
`1-Low Rank Approximation. .
. .
Given : A ∈ Rn×n, k ∈ N, α > 1
Output : rank-k A ∈ Rn×n s.t.‖A − A‖1 6 α · min
rank−k A ′‖A ′ − A‖1
.
∑i,j|A ′i,j − Ai,j |
.OPT
A A−
1
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 39 / 50
`1-Low Rank Approximation. .
. .
Given : A ∈ Rn×n, k ∈ N, α > 1
Output :
..
U ∈ Rn×k ,V ∈ Rk×n s.t.‖UV − A‖1 6 α · OPT
U
V
A−
1
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 39 / 50
Why is `1-low Rank Approximation Interesting?
The problem was shown to be NP-hard by Gillis-Vavasis’15 and anumber of heuristics have been proposed
Asked in several places if there are any approximation algorithms
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 40 / 50
Why is `1-low Rank Approximation Interesting?
The problem was shown to be NP-hard by Gillis-Vavasis’15 and anumber of heuristics have been proposedAsked in several places if there are any approximation algorithms
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 40 / 50
Thought Experiments. .
. .Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 41 / 50
Thought Experiments. .
. .
Given : A set of points in 2-dimensions
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 41 / 50
Thought Experiments. .
. .
Given : A set of points in 2-dimensions
Goal : Find a line to fit those points
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 41 / 50
Thought Experiments. .
. .
Given : A set of points in 2-dimensions
Goal : Find a line to fit those points
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 41 / 50
Thought Experiments. .
. .
Given : A set of points in 2-dimensions
Goal : Find a line to fit those points
.
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 41 / 50
Thought Experiments. .
. .
Given : A set of points in 2-dimensions
Goal : Find a line to fit those points
.
Output : `2 minimizer line,
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 41 / 50
Thought Experiments. .
. .
Given : A set of points in 2-dimensions
Goal : Find a line to fit those points
.
Output : `2 minimizer line, `1 minimizer line
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 41 / 50
Thought Experiments. .
. .Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 42 / 50
Thought Experiments. .
. .
Given : k = 1 and A =
4 0 0 00 1 1 10 1 1 10 1 1 1
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 42 / 50
Thought Experiments. .
. .
Given : k = 1 and A =
4 0 0 00 1 1 10 1 1 10 1 1 1
Output : rank-1 ‖‖F−solution A =
4 0 0 00 0 0 00 0 0 00 0 0 0
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 42 / 50
Thought Experiments. .
. .
Given : k = 1 and A =
4 0 0 00 1 1 10 1 1 10 1 1 1
Output : rank-1 ‖‖F−solution A =
4 0 0 00 0 0 00 0 0 00 0 0 0
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 42 / 50
Thought Experiments. .
. .
Given : k = 1 and A =
4 0 0 00 1 1 10 1 1 10 1 1 1
Output : rank-1 ‖‖F−solution A =
4 0 0 00 0 0 00 0 0 00 0 0 0
Output : rank-1 ‖‖1−solution A =
0 0 0 00 1 1 10 1 1 10 1 1 1
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 42 / 50
Thought Experiments. .
. .
Given : k = 1 and A =
4 0 0 00 1 1 10 1 1 10 1 1 1
Output : rank-1 ‖‖F−solution A =
4 0 0 00 0 0 00 0 0 00 0 0 0
Output : rank-1 ‖‖1−solution A =
0 0 0 00 1 1 10 1 1 10 1 1 1
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 42 / 50
Heuristics and Nearly `1 Setting
`1-setting
I Heuristic algorithms Ke-Kanade’05, Ding-Zhou-He-Zha’06,Kwak’08, Brooks-Dula-Boone’13
I Robust PCA Candes-Li-Ma-Wright’11,Wright-Ganesh-Rao-Peng-Ma’09,Netrapalli-Niranjan-Sanghavi-Anandkumar-Jain’14,Yi-Park-Chen-Caramanis’16
None of them can achieve better than poly(n) approximation ratio
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 43 / 50
Heuristics and Nearly `1 Setting
`1-settingI Heuristic algorithms Ke-Kanade’05, Ding-Zhou-He-Zha’06,
Kwak’08, Brooks-Dula-Boone’13
I Robust PCA Candes-Li-Ma-Wright’11,Wright-Ganesh-Rao-Peng-Ma’09,Netrapalli-Niranjan-Sanghavi-Anandkumar-Jain’14,Yi-Park-Chen-Caramanis’16
None of them can achieve better than poly(n) approximation ratio
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 43 / 50
Heuristics and Nearly `1 Setting
`1-settingI Heuristic algorithms Ke-Kanade’05, Ding-Zhou-He-Zha’06,
Kwak’08, Brooks-Dula-Boone’13I Robust PCA Candes-Li-Ma-Wright’11,
Wright-Ganesh-Rao-Peng-Ma’09,Netrapalli-Niranjan-Sanghavi-Anandkumar-Jain’14,Yi-Park-Chen-Caramanis’16
None of them can achieve better than poly(n) approximation ratio
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 43 / 50
Heuristics and Nearly `1 Setting
`1-settingI Heuristic algorithms Ke-Kanade’05, Ding-Zhou-He-Zha’06,
Kwak’08, Brooks-Dula-Boone’13I Robust PCA Candes-Li-Ma-Wright’11,
Wright-Ganesh-Rao-Peng-Ma’09,Netrapalli-Niranjan-Sanghavi-Anandkumar-Jain’14,Yi-Park-Chen-Caramanis’16
None of them can achieve better than poly(n) approximation ratio
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 43 / 50
Main Question, `1 Low Rank Approximation
QuestionGiven matrix A ∈ Rn×n, is there an algorithm that is able to output arank-k matrix A such that
‖A − A‖1 6 α · minrank−k A ′
‖A ′ − A‖1?
Or, are there any inapproximability results?
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 44 / 50
Main Results - Algorithms
Upper bound (Algorithm) for an arbitrary n × n matrix A
I poly(k) log n-approximation, poly(n) timeI O(k)-approximation, nO(k)2O(k2) time.I Bicriteria algorithm
F rank-2k O(1)-approximation, nO(k2) running time
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 45 / 50
Main Results - Algorithms
Upper bound (Algorithm) for an arbitrary n × n matrix AI poly(k) log n-approximation, poly(n) time
I O(k)-approximation, nO(k)2O(k2) time.I Bicriteria algorithm
F rank-2k O(1)-approximation, nO(k2) running time
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 45 / 50
Main Results - Algorithms
Upper bound (Algorithm) for an arbitrary n × n matrix AI poly(k) log n-approximation, poly(n) timeI O(k)-approximation, nO(k)2O(k2) time.
I Bicriteria algorithm
F rank-2k O(1)-approximation, nO(k2) running time
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 45 / 50
Main Results - Algorithms
Upper bound (Algorithm) for an arbitrary n × n matrix AI poly(k) log n-approximation, poly(n) timeI O(k)-approximation, nO(k)2O(k2) time.I Bicriteria algorithm
F rank-2k O(1)-approximation, nO(k2) running time
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 45 / 50
Main Results - Algorithms
Upper bound (Algorithm) for an arbitrary n × n matrix AI poly(k) log n-approximation, poly(n) timeI O(k)-approximation, nO(k)2O(k2) time.I Bicriteria algorithm
F rank-2k O(1)-approximation, nO(k2) running time
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 45 / 50
Hardness - Under Exponential Time Hypothesis. .
. .Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 46 / 50
Hardness - Under Exponential Time Hypothesis. .
. .
`1-Low Rank Approximation Hardness
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 46 / 50
Hardness - Under Exponential Time Hypothesis. .
. .
`1-Low Rank Approximation Hardness
Given : A ∈ Rn×n, k ∈ N, α > 1
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 46 / 50
Hardness - Under Exponential Time Hypothesis. .
. .
`1-Low Rank Approximation Hardness
Given : A ∈ Rn×n, k ∈ N, α > 1
Output : U ∈ Rn×k , V ∈ Rk×n s.t. ‖UV − A‖1 6 α ·OPTwith prob. 9/10
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 46 / 50
Hardness - Under Exponential Time Hypothesis. .
. .
`1-Low Rank Approximation Hardness
Given : A ∈ Rn×n, k ∈ N, α > 1
Output : U ∈ Rn×k , V ∈ Rk×n s.t. ‖UV − A‖1 6 α ·OPTwith prob. 9/10
Assume : Exponential Time Hypothesis(ETH)[Impagliazzo-Paturi-Zane’98]
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 46 / 50
Hardness - Under Exponential Time Hypothesis. .
. .
`1-Low Rank Approximation Hardness
Given : A ∈ Rn×n, k ∈ N, α > 1
Output : U ∈ Rn×k , V ∈ Rk×n s.t. ‖UV − A‖1 6 α ·OPTwith prob. 9/10
Assume : Exponential Time Hypothesis(ETH)[Impagliazzo-Paturi-Zane’98]
for arbitrarily small constant γ > 0
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 46 / 50
Hardness - Under Exponential Time Hypothesis. .
. .
`1-Low Rank Approximation Hardness
Given : A ∈ Rn×n, k ∈ N, α > 1
Output : U ∈ Rn×k , V ∈ Rk×n s.t. ‖UV − A‖1 6 α ·OPTwith prob. 9/10
Assume : Exponential Time Hypothesis(ETH)[Impagliazzo-Paturi-Zane’98]
for arbitrarily small constant γ > 0for any algorithm running in nO(1) time
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 46 / 50
Hardness - Under Exponential Time Hypothesis. .
. .
`1-Low Rank Approximation Hardness
Given : A ∈ Rn×n, k ∈ N, α > 1
Output : U ∈ Rn×k , V ∈ Rk×n s.t. ‖UV − A‖1 6 α ·OPTwith prob. 9/10
Assume : Exponential Time Hypothesis(ETH)[Impagliazzo-Paturi-Zane’98]
for arbitrarily small constant γ > 0for any algorithm running in nO(1) time
Requires : α > (1 + 1log1+γ n
)
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 46 / 50
Hardness - Under Exponential Time Hypothesis. .
. .
`1-Low Rank Approximation Hardness
Given : A ∈ Rn×n, k ∈ N, α > 1
Output : U ∈ Rn×k , V ∈ Rk×n s.t. ‖UV − A‖1 6 α ·OPTwith prob. 9/10
Assume : Exponential Time Hypothesis(ETH)[Impagliazzo-Paturi-Zane’98]
for arbitrarily small constant γ > 0for any algorithm running in nO(1) time
Requires : α > (1 + 1log1+γ n
)
Implies a 2Ω(1/ε1−γ) Hardness
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 46 / 50
Open Problems
The hardness result is for (1 + 1/ poly(log n)) approximation, canwe obtain a hardness result for O(1) approximation ?
We have an nO(k)-time O(k)-approximation algorithm, can eitherthe approximation or the time be improved?We have a poly(n)-time O(log n) · poly(k)-approximationalgorithm. Can the approximation be improved?
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 47 / 50
Open Problems
The hardness result is for (1 + 1/ poly(log n)) approximation, canwe obtain a hardness result for O(1) approximation ?
We have an nO(k)-time O(k)-approximation algorithm, can eitherthe approximation or the time be improved?
We have a poly(n)-time O(log n) · poly(k)-approximationalgorithm. Can the approximation be improved?
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 47 / 50
Open Problems
The hardness result is for (1 + 1/ poly(log n)) approximation, canwe obtain a hardness result for O(1) approximation ?
We have an nO(k)-time O(k)-approximation algorithm, can eitherthe approximation or the time be improved?We have a poly(n)-time O(log n) · poly(k)-approximationalgorithm. Can the approximation be improved?
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 47 / 50
Conclusions
Studied intractable matrix factorization problems through the lensof parameterized complexity
I nonnegative matrix factorizationI weighted low rank approximationI `1-low rank approximation
Algorithm techniques
I polynomial system verifierI random matrices for dimensionality reduction to reduce the number
of variables
Hardness techniques
I random 4-SAT, ETH, etc.
Parameterized Complexity gives a way of coping with intractabilityfor emerging machine learning problems
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 48 / 50
Conclusions
Studied intractable matrix factorization problems through the lensof parameterized complexity
I nonnegative matrix factorization
I weighted low rank approximationI `1-low rank approximation
Algorithm techniques
I polynomial system verifierI random matrices for dimensionality reduction to reduce the number
of variables
Hardness techniques
I random 4-SAT, ETH, etc.
Parameterized Complexity gives a way of coping with intractabilityfor emerging machine learning problems
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 48 / 50
Conclusions
Studied intractable matrix factorization problems through the lensof parameterized complexity
I nonnegative matrix factorizationI weighted low rank approximation
I `1-low rank approximationAlgorithm techniques
I polynomial system verifierI random matrices for dimensionality reduction to reduce the number
of variables
Hardness techniques
I random 4-SAT, ETH, etc.
Parameterized Complexity gives a way of coping with intractabilityfor emerging machine learning problems
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 48 / 50
Conclusions
Studied intractable matrix factorization problems through the lensof parameterized complexity
I nonnegative matrix factorizationI weighted low rank approximationI `1-low rank approximation
Algorithm techniques
I polynomial system verifierI random matrices for dimensionality reduction to reduce the number
of variables
Hardness techniques
I random 4-SAT, ETH, etc.
Parameterized Complexity gives a way of coping with intractabilityfor emerging machine learning problems
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 48 / 50
Conclusions
Studied intractable matrix factorization problems through the lensof parameterized complexity
I nonnegative matrix factorizationI weighted low rank approximationI `1-low rank approximation
Algorithm techniques
I polynomial system verifierI random matrices for dimensionality reduction to reduce the number
of variablesHardness techniques
I random 4-SAT, ETH, etc.
Parameterized Complexity gives a way of coping with intractabilityfor emerging machine learning problems
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 48 / 50
Conclusions
Studied intractable matrix factorization problems through the lensof parameterized complexity
I nonnegative matrix factorizationI weighted low rank approximationI `1-low rank approximation
Algorithm techniquesI polynomial system verifier
I random matrices for dimensionality reduction to reduce the numberof variables
Hardness techniques
I random 4-SAT, ETH, etc.
Parameterized Complexity gives a way of coping with intractabilityfor emerging machine learning problems
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 48 / 50
Conclusions
Studied intractable matrix factorization problems through the lensof parameterized complexity
I nonnegative matrix factorizationI weighted low rank approximationI `1-low rank approximation
Algorithm techniquesI polynomial system verifierI random matrices for dimensionality reduction to reduce the number
of variables
Hardness techniques
I random 4-SAT, ETH, etc.
Parameterized Complexity gives a way of coping with intractabilityfor emerging machine learning problems
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 48 / 50
Conclusions
Studied intractable matrix factorization problems through the lensof parameterized complexity
I nonnegative matrix factorizationI weighted low rank approximationI `1-low rank approximation
Algorithm techniquesI polynomial system verifierI random matrices for dimensionality reduction to reduce the number
of variablesHardness techniques
I random 4-SAT, ETH, etc.
Parameterized Complexity gives a way of coping with intractabilityfor emerging machine learning problems
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 48 / 50
Conclusions
Studied intractable matrix factorization problems through the lensof parameterized complexity
I nonnegative matrix factorizationI weighted low rank approximationI `1-low rank approximation
Algorithm techniquesI polynomial system verifierI random matrices for dimensionality reduction to reduce the number
of variablesHardness techniques
I random 4-SAT, ETH, etc.
Parameterized Complexity gives a way of coping with intractabilityfor emerging machine learning problems
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 48 / 50
Conclusions
Studied intractable matrix factorization problems through the lensof parameterized complexity
I nonnegative matrix factorizationI weighted low rank approximationI `1-low rank approximation
Algorithm techniquesI polynomial system verifierI random matrices for dimensionality reduction to reduce the number
of variablesHardness techniques
I random 4-SAT, ETH, etc.
Parameterized Complexity gives a way of coping with intractabilityfor emerging machine learning problems
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 48 / 50
Thank you!. .
. .
Questions?
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 49 / 50
Cauchy Distribution. .
. .
Cauchy distribution C(µ,γ), pdf f (x) = 1πγ
γ2
(x−µ)2+γ
µ
1π
For any three independent random variablesX ,Y ,Z drawn from Cauchy distribution.For any a,b ∈ R, aX + bY ∼ (|a|+ |b|)Z
Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 50 / 50
Top Related