Polynomial Embeddings for Quadratic Algebraic Equations

Radu Balan

University of Maryland, College Park, MD 20742

Math-CS Joint Lecture, Drexel University Monday April 23, 2012

Overview

1. Introduction: Motivation and statement of problem

2. Invertibility Results in the Real Case3. The Algebraic Approach:

a. Quasi-Linear Embeddingsb. Hierarchical Embeddingsc. Numerical Analysis

4. Modified Least Square Estimator5. Theoretical Bounds: CRLB6. Performance Analysis

2/50

1. Introduction: Motivation

Byx BA=Identity

Question: What if |y| is known instead (that is, one looses the phase information) ?

?|| xAxz

Where is important: X-Ray Crystallography, Speech Processing

knowns?

Inversion of Nonlinear Transformations

y = A x

3/50

1. Introduction: Statement of the Problem

Reconstruction from magnitudes of frame coefficients

a complete set of vectors (frame) for the n-dimensional Hilbert space H (H=Cn or H=Rn).

Equivalence relation: x,yH, x~y iff there is a scalar z, |z|=1 so that x=zy(real case: x=y ; complex case: x=eiy). Let .

Define

Problems: 1. Is an injective map?2. If it is, how to invert it efficiently?

4/50

H

Rm

x(x)

f1

f2

fm

5/50

Real Case: K=RTheorem [R.B.,Casazza, Edidin, ACHA(2006)] is injective iff for any subset JF either J or F\J

spans Rn.Corollaries [2006]• if m 2n-1, and a generic frame set F, then is

injective;• if m2n-2 then for any set F, cannot be injective;• if any n-element subset of F is linearly independent,

then is injective; for m=2n-1 this is a necessary and sufficient condition.

2. Invertibility Results: Real Case (1)

6/50

Invertibility Results: Real Case (2)Real Case: K=R

Theorem [R.B.(2012)] is injective iff any one of the following equivalent conditions holds true:

• For any x,yRn, x≠0, y≠0,

• There is a constant a>0 so that for all x, RI

7/50

Invertibility Results: Real Case (3) complete set of vectors (frame) for H=Rn.

One would expect that if is injective and m>2n-1 then there is a strict subsetJ{1,2,…,m} so that π is injective, where π:RmR|J| is the restriction to J index.

However the next example shows this is not the case.

Example. Consider n=3, m=6, and F the set of columns of the following matrix

F =

Note that for any subset J of 3 columns, either J or F\J is linearly dependent.Thus is injective but removing any column makes π not injective.

8/50

3. The Algebraic Approach3.1 Quasi-Linear EmbeddingsExample (a)Consider the real case: n=3 , m=6.Frame vectors:

Need to solve a system of the form:

9/50

[1 2 2 1 2 11 4 −2 4 −2 11 −2 −4 1 4 41 −4 −6 4 12 91 2 −14 1 −14 491 0 2 0 0 1

] [𝑦1

𝑦2

𝑦3

𝑦 4

𝑦5

𝑦6

]=[4949

1964

]𝑠 𝑜𝑙𝑣𝑒 𝑓𝑜𝑟 𝑦→𝑦=[

1−121−24

]Then factor:

Thus, we obtain:

10/50

Summary of this approach:

where

11/50

Summary of this approach:

where

12/50

Example (b)Consider the real case: n=3 , m=5.Frame vectors:

Need to solve the system:

Is it possible? How?

X

13/50

|𝑥1+𝑥2+𝑥3|=2

|𝑥1+2 𝑥2−𝑥3|=3|𝑥1−𝑥2−2𝑥3|=2

|𝑥1−2𝑥2−3𝑥3|=3

|𝑥1+𝑥2−7 𝑥3|=14

→

𝑥12+2𝑥1𝑥2+2𝑥1𝑥3+𝑥2

2+2𝑥2𝑥3+𝑥32 ¿ 4

𝑥12+4 𝑥1𝑥2−2 𝑥1𝑥3+4 𝑥2

2−2𝑥2𝑥3+𝑥32 ¿ 9

𝑥12−2 𝑥1 𝑥2−4 𝑥1 𝑥3+𝑥2

2+4 𝑥2 𝑥3+4 𝑥32 ¿ 4

𝑥12−4 𝑥1𝑥2−6 𝑥1𝑥3+4 𝑥2

2+12𝑥2𝑥3+9 𝑥32 ¿ 9

𝑥12+2 𝑥1𝑥2−14 𝑥1𝑥3+𝑥2

2−14 𝑥2𝑥3+49𝑥32 ¿ 196

Let’s square again:

We obtained 5 linear equations with 6 unknown monomials.

Idea: Let’s multiply again these equations (square and cross)

New equations: : 15 equations

New variables (monomials): : 15 unknowns

14/50

Summary of this approach:

where

15/50

3.2 Hierarchical Embeddings

Primary data:

Level d embedding: ,

Identify:

Then

a homogeneous polynomial of total degree 2d.

16/50

How many monomials?

Real case: number of monomials of degree 2d in n variables:

Complex case: …Number of degree (d,d) in n variables:

Define redundancy at level d:

(R.B. [SampTA2009])

17/50

Real case:

18/50

Real case:

19/50

Real case:

20/50

Complex case:

21/50

Complex case:

22/50

Complex case:

23/50

Fundamental question: How many equations are linearly independent?

Recall: , Note:

and

The matrix is not canonical, and so is * for d>1.However its range is basis independent.

We are going to compute this range in terms of a canonical matrix.

24/50

Theorem The following hold true:1. (as a quadratic form)

2. Rows of are linearly independent iff

where is the mdxmd matrix given by

Real case:Complex case:

25/50

Let denote the Gram matrixAnd for integer p.

Theorem In either real or complex case:

Hence for d=1, the number of independent quadratics is given by:

Theorem For d=2,

Remark Note the k1=k2,l1=l2 submatrix of is 26/50

3.3. Numerical analysisResults for the complex case: random frames

n=3,m=627/50

n=3,m=6 28/50

n=16m=64

29/50

n=32m=128

30/50

n=4m=16

31/50

n=4m=14

Note 6 zero eigenvalues of instead of 5.32/50

n=4m=15

Number of zero eigenvalues = 20 =120-100, as expected.33/50

4. Modified Least Square Error Estimator

• Model: mifxd iii 1 , ,2

“Vanilla” Least-Square-Estimator (LSE):

m

iiixLSE fxdx

1

22,minargˆ

Tm

i

Tiiiii xxXffFdXFtr

, ,

:asly equivalentcriterion theRewrite

1

2

We modify this criterion in two ways:1. Replace X by a rank r positive matrix Y2. Regularize the criterion by adding a norm of Y

34/50

)ˆ(.)ˆ(.ˆ

)()(minargˆ1

2)(,0

QeigvectprincYeigvalprincx

YtrdYFtrYm

iiirYrankY

Use a rank-r factorization of Y to account for constraint:

)1(:,ˆˆ :ionfactorizat SDV

minargˆ

1,1

1

2

UxVUL

LLtrdLFLtrL Tm

iii

TRL rn

35/50

Optimization procedure

LLtrdLFLtrL Tm

iii

TRL rn

1

2minargˆ

Our approach: 1. Start with a large and decrease its value over

time;2. Replace one L in the inner quadratic term by a

previous estimate3. Penalize large successive variations.

36/50

KKtrKLKLtrLLtrdKFLtrKLJ Tt

m

i

Tt

Ttii

Tt

1

2),(

1

,minarg )()1(

ttLLJL t

tLt

Algorithm (Part I)Step I: Initialization

Step II: Iteration

Step III: Factorization

1

00)0(

,for policy adaptationan Choose0,,, Initialize

ttt

tL

)1(:,ˆ 1,1

)(

UxVULSVD

How to initialize?How to adapt?

Convergence?

37/50

Initialization

m

iii

m

ii

Tm

i

Tii

T

FdQ

dLQLtrLLtrdLFLtrLJ

L

1

1

2

1

2

:where

22)(

: small and largeFor

21 , : of eigenpairs be ,Let eeveQvQve kkkkk

rr vvvL ||| 2211)0(

Set:

m

i

r

k ik

krfv

rree1 1

20

10,2

,/... 38/50

Convergence

KKtrKLKLtrLLtrdKFLtrKLJ Tt

m

i

Tt

Ttii

Tt

1

2),(

Consider the iterative process:

)()1(

)()1(

,:

,minarg:tt

tt

ttL

t

LLJj

LLJL

Theorem Assume (t)t,(µt)t are monotonically decreasing non-negative sequences. Then (jt)t≥0 is a monotonically decreasing convergent sequence.

LKJKLJ tt ,, :Note

39/50

5. Theoretical bounds: CRLB

m

iii

iiii

mmdfxxdpoodLoglikelih

diiNmifxdModel

1

22

2

22

)log()2log(2

,2

1)|(log :

...),,0(~ ; 1 , , :

jk

jk xxxdpEI )|(log: :Matrixn InformatioFisher

2

,

m

i

Tiii fffxRRI

1

2

2 , where, 4

12

1

4CO : RItimatorUnbiasedEsVARCRLB

𝐸 [‖𝑥− �̂�‖2 ]≥ 𝜎2

4𝑡𝑟 (𝑅−1 )

𝐸 [‖𝑥 𝑥𝑇− �̂� �̂�𝑇‖2 ]≥ 𝜎2

2[‖𝑥‖2 𝑡𝑟 (𝑅−1 )+𝑥𝑇 𝑅− 1𝑥 ] 40/50

The LSE/MLE is a biased estimator. Modified CRLB for biased estimators:

xxBiasxEx

LBModifiedCRBiasBiasx

Ix

xxxxE TT

T

)( , ˆ)(

:ˆˆ 1

Asymptotically (for high SNR):

,

,41 ,

2

1

112

Idx

fRfRffxBiasm

iii

Tii

).(..44

6114

12

10

tohRRRMSE

MSE

T

MSE

41/50

6. Performance Analysis

n=3 , m=9 , d = 1 (dlevel) , r = 2 and , decrease by 5% every step w/ saturation (subspace)

Mintrace algorithm: Candes, Strohmer, Voroninski (2011)

42/50

n=3 , m=9, d=1, r=243/50

n=3 , m=9, d=1, r=244/50

n=8 , m=24, d=3, r=245/50

n=8 , m=24, d=3, r=246/50

n=8 , m=24, d=3, r=247/50

n=9 , m=27, d=3, r=248/50

n=9 , m=27, d=3, r=249/50

n=9 , m=27, d=3, r=250/50