Departemen Statistika

FMIPA – IPB

Bogor

ALJABAR MATRIKS

Eigenvalues and Eigenvectors

For a square matrix A, let I be a conformable identity matrix. Then the scalars satisfying the polynomial equation |A - lI| = 0 are called the eigenvalues (or characteristic roots) of A.

The equation |A - lI| = 0 is called the characteristic equation or the determinantal equation.

Eigenvalues and Eigenvectors

For example, if we have a matrix A:

2 44 -4

A

then

l l

l l l l

l l 2

2 4 1 04 -4 0 1

2 - 4 2 4 16 04 -4 -

or

2 24 0

A I

which implies there are two roots or eigenvalues -- l=-6 and l=4.

Eigenvalues and Eigenvectors

For a matrix A with eigenvectors l, a nonzero vector x such that Ax = lx is called an eigenvector (or characteristic vector) of A associated with l.

Eigenvalues and Eigenvectors

For example, if we have a matrix A:

2 44 -4

A

l

1 1

2 2

1 2 1 1 2

1 2 2 1 2

x x2 4 64 -4 x x

2x 4x 6x 8x 4x 0

and

4x 4x 6x 4x 2x 0

Ax x

Fixing x1=1 yields a solution for x2 of –2.

with eigenvalues l = -6 and l = 4, the eigenvector of A associated with l = -6 is

Eigenvalues and Eigenvectors

Note that eigenvectors are usually normalized so they have unit length, i.e.,

Thus our arbitrary choice to fix x1=1 has no impact on the eigenvector associated with l = -6.

For our previous example we have:

x

ex'x

1 1 1 -2 -2 5-25 11 -2 5-2

xe

x'x

Eigenvalues and Eigenvectors

For matrix A and eigenvalue l = 4, we have

l

1 1

2 2

1 2 1 1 2

1 2 2 1 2

x x2 4 44 -4 x x

2x 4x 4x 2x 4x 0

and

4x 4x 4x 4x 8x 0

Ax x

We again arbitrarily fix x1=1, which now yields a solution for x2 of 1/2.

Eigenvalues and Eigenvectors

Normalization to unit length yields

Again our arbitrary choice to fix x1=1 has no impact on the eigenvector associated with l = 4.

1 1 121 1 1

2 2 2 515 5 1

1 541 2122

xe

x'x

Quadratic Forms

A Quadratic From is a function

Q(x) = x’Ax

in k variables x1,…,xk where

and A is a k x k symmetric matrix.

x

1

2

k

xx

x

Quadratic Forms

Note that a quadratic form has only squared terms and crossproducts, and so can be written

then

x A1

2

x 1 4 and x 0 2

x x'Ax 2 21 1 2 2Q( ) = = x + 4x x - 2x

Suppose we have

xk k

ij i ji 1 j 1

Q a x x

Spectral Decomposition and

Quadratic Forms Any k x k symmetric matrix can be expressed in

terms of its k eigenvalue-eigenvector pairs (li, ei) as

This is referred to as the spectral decomposition of A.

lk

'i i i

i 1

A ee

Spectral Decomposition and

Quadratic Forms For our previous example on eigenvalues and

eigenvectors we showed that

2 4 4 4

A

has eigenvalues l1 = -6 and l2 = -4, with corresponding (normalized) eigenvectors

1 2

1 2 5 5, ,

-2 15 5

e e

Spectral Decomposition and

Quadratic Forms Can we reconstruct A?

l

k

'i i i

i 1

1 2 5 51 -2 2 1 6 +4

-2 15 5 5 55 5

1 -2 4 2 2 45 5 5 5 6 4-2 4 2 1 4 4

5 5 5 5

A ee

A

Spectral Decomposition and

Quadratic Forms Spectral decomposition can be used to

develop/illustrate many statistical results/ concepts. We start with a few basic concepts:

- Nonnegative Definite Matrix – when any k x k matrix A such that

0 x’Ax x’ =[x1, x2, …, xk]

the matrix A and the quadratic form are said to be nonnegative definite.

Spectral Decomposition and

Quadratic Forms - Positive Definite Matrix – when any k x k matrix A

such that

0 < x’Ax x’ =[x1, x2, …, xk] [0, 0, …, 0]

the matrix A and the quadratic form are said to be positive definite.

Spectral Decomposition and

Quadratic Forms Example - Show that the following quadratic form

is positive definite:

2 21 2 1 26x + 4x - 4 2x x

We first rewrite the quadratic form in matrix notation:

11 2

2

x6 -2 2Q( ) = x x = 'x-2 2 4

x x Ax

Spectral Decomposition and

Quadratic Forms Now identify the eigenvalues of the resulting

matrix A (they are l1 = 2 and l2 = 8).

l l

l l l l

l l l l 2

1 06 -2 20 1-2 2 4

6 - -2 2 6 4 -2 2 -2 2 0-2 2 4 -

or

10 16 2 8 0

A I

Spectral Decomposition and

Quadratic Forms Next, using spectral decomposition we can write:

l l l k

' ' ' ' 'i i i 1 1 1 2 2 2 1 1 2 2

i 1

2 8A ee e e e e e e e e

where again, the vectors ei are the normalized and orthogonal eigenvectors associated with the eigenvalues l1 = 2 and l2 = 8.

l

k

'i i i

i 1

1 23 31 2 2 -1 2 +8

3 -1 32 3 33 3

1 22 2 63 3 2 23 3 2 82 2 42 22 1

3 3 3 3

A ee

A

Sidebar - Note again that we can recreate the original matrix A from the spectral decomposition:

Spectral Decomposition and

Quadratic Forms

Because l1 and l2 are scalars, premultiplication and postmultiplication by x’ and x, respectively, yield:

Spectral Decomposition and

Quadratic Forms

' ' ' ' ' 2 21 1 2 2 1 2 2 8 2y +8y 0x Ax x e e x x e e x

where

At this point it is obvious that x’Ax is at least nonnegative definite!

' ' ' '1 1 1 1 2 2 2 2y and yx e e x x e e x

We now show that x’Ax is positive definite, i.e.

Spectral Decomposition and

Quadratic Forms

' 2 21 2 2y +8y 0x Ax

From our definitions of y1 and y2 we have

'1 1 1

'2 22

y or

ye x

y Exxe

Since E is an orthogonal matrix, E’ exists.

Thus,

Spectral Decomposition and

Quadratic Forms

'x E y

But 0 x = E’y implies y 0 .

At this point it is obvious that x’Ax is positive definite!

This suggests rules for determining if a k x k symmetric matrix A (or equivalently, its quadratic form x’Ax) is nonegative definite or positive definite:

- A is a nonegative definite matrix iff li 0, i = 1,…,rank(A)

- A is a positive definite matrix iff li > 0, i = 1,…,rank(A)

Spectral Decomposition and

Quadratic Forms