Stats 443.3 & 851.3

Summary

The Woodbury Theorem

11 1 1 1 1 1A BCD A A B C DA B DA

where the inverses11 1 1 1, and exist.A C C DA B

11 12

21 22

q

n m n qp m p

A AA

A A

Block Matrices

Let the n × m matrix

be partitioned into sub-matrices A11, A12, A21, A22,

11 12

21 22

p

m k m pl k l

B BB

B B

Similarly partition the m × k matrix

11 12 11 12

21 22 21 22

A A B BA B

A A B B

Product of Blocked Matrices

Then

11 11 12 21 11 12 12 22

21 11 22 21 21 12 22 22

A B A B A B A BA B A B A B A B

11 12

21 22

p

n n n pp n p

A AA

A A

The Inverse of Blocked Matrices

Let the n × n matrix

be partitioned into sub-matrices A11, A12, A21, A22,

11 12

21 22

p

n n n pp n p

B BB

B B

Similarly partition the n × n matrix

Suppose that B = A-1

11 12

21 22

p

n n n pp n p

A AA

A A

Summarizing

Let

11 12

21 22

p

n pp n p

B BB B

Suppose that A-1 = B

then

11 1 121 22 21 11 22 21 11 12 22 21B A A B A A A A A A

11 1 112 11 12 22 11 12 22 21 11 12B A A B A A A A A A

1 11 1 1 1 111 11 12 22 21 11 11 12 22 21 11 12 21 11B A A A A A A A A A A A A A

1 11 1 1 1 1

22 22 21 11 12 22 22 21 11 12 22 21 12 22B A A A A A A A A A A A A A

Symmetric Matrices

• An n × n matrix, A, is said to be symmetric if

Note:AA

11

111

AA

ABAB

ABAB

The trace and the determinant of a square matrix

11 12 1

21 22 2

1 2

n

nij

n n nn

a a aa a a

A a

a a a

Let A denote then n × n matrix

Then

1

n

iii

tr A a

11 12 1

21 22 2

1 2

det the determinant of

n

n

n n nn

a a aa a a

A A

a a a

also

where1

n

ij ijj

a A

cofactor of ij ijA a

the determinant of the matrix

after deleting row and col.th thi j

11 1211 22 12 21

21 22

deta a

a a a aa a

ji1

1. 1, I tr I n

Some properties

2. , AB A B tr AB tr BA

1 13. AA

122 11 12 22 2111 12

121 22 11 22 21 11 12

4. A A A A AA A

AA A A A A A A

22 11 12 21 if 0 or 0A A A A

Special Types of Matrices

1. Orthogonal matrices– A matrix is orthogonal if PˊP = PPˊ = I– In this cases P-1=Pˊ .– Also the rows (columns) of P have length 1 and

are orthogonal to each other

Special Types of Matrices(continued)

2. Positive definite matrices– A symmetric matrix, A, is called positive definite

if:

– A symmetric matrix, A, is called positive semi definite if:

022 112211222

111 nnnnn xxaxxaxaxaxAx

0 allfor

x

0 xAx

0 allfor

x

Theorem The matrix A is positive definite if0,,0,0,0 321 nAAAA

nnnn

n

n

n

aaa

aaaaaa

AA

aaaaaaaaa

Aaaaa

AaA

21

22212

11211

332313

232212

131211

32212

12112111

and

,,,

where

Special Types of Matrices(continued)

3. Idempotent matrices– A symmetric matrix, E, is called idempotent if:

– Idempotent matrices project vectors onto a linear subspace

EEE

xExEE

xE

x

Eigenvectors, Eigenvalues of a matrix

DefinitionLet A be an n × n matrixLet and be such thatx

with 0Ax x x

then is called an eigenvalue of A andand is called an eigenvector of A andx

Note:

0A I x

1If 0 then 0 0A I x A I

thus 0 A I

is the condition for an eigenvalue.

11 1

1

det = 0n

n nn

a aA I

a a

= polynomial of degree n in .

Hence there are n possible eigenvalues 1, … , n

Thereom If the matrix A is symmetric with distinct eigenvalues, 1, … , n, with corresponding eigenvectors

1 1 1then n n nA x x x x

1, , nx x

Assume 1 i ix x

1 1

1

0, ,

0n

n n

xx x

x

PDP

The Generalized Inverse of a matrix

DefinitionB (denoted by A-) is called the generalized inverse (Moore – Penrose inverse) of A if

1. ABA = A2. BAB = B3. (AB)' = AB4. (BA)' = BA

Note: A- is unique

Hence B1 = B1AB1 = B1AB2AB1 = B1 (AB2)'(AB1) '

= B1B2'A'B1

'A'= B1B2'A' = B1AB2 = B1AB2AB2

= (B1A)(B2A)B2 = (B1A)'(B2A)'B2 = A'B1'A'B2

'B2

= A'B2'B2= (B2A)'B2

= B2AB2 = B2

The general solution of a system of Equations

Ax b

x A b I A A z

The general solution

x A b I A A z

where is arbitrary

1 1then C B BB A A A

Let C be a p×q matrix of rank k < min(p,q),

then C = AB where A is a p×k matrix of rank k and B is a k×q matrix of rank k

The General Linear Model

npnn

p

p

pn xxx

xxxxxx

y

yy

21

22221

11211

2

1

2

1

,,Let Xβy

ondistributi , a has where 2I0εεβXy N

Geometrical interpretation of the General Linear Model

p

npnn

p

p

pn xxx

xxxxxx

y

yy

xxXβy

1

21

22221

11211

2

1

2

1

,,Let

ondistributi , a has where 2I0εεβXy N

X

xxxβXyμ

of columns by the spanned spacelinear in the lies

221 ppE

Estimation

The General Linear Model

n

iiip

n

iiip

n

iii

n

ii yxxxxxx

11

112

1121

1

21

n

iiip

n

iiip

n

ii

n

iii yxxxxxx

12

122

1

221

121

n

iiipp

n

iip

n

iipi

n

iipi yxxxxxx

11

22

121

11

the Normal Equations

yXβXX

The Normal Equations

npnn

p

p

n xxx

xxxxxx

y

yy

21

22221

11211

2

1

, where Xy

Solution to the normal equations

yXβXX ˆ

. of is matrix theIf rank fullX

yXXXβ 1ˆ

Estimate of 2

βXyβXy ˆˆ1ˆ 2

n

yXXXXIy 1

n1

βXyyy ˆ1

n

Properties of The Maximum Likelihood Estimates

Unbiasedness, Minimum Variance

yXXXβ

1ˆ EE

ββXXXXyXXX

11 E

βcβcβc

ˆˆ EE

22ˆ n

pnE

βXyβXy ˆˆ1ˆ 22

pnpn

ns

s2 is an unbiased estimator of 2.

Unbiasedness

Distributional Properties

Least square Estimates (Maximum Likelidood estimates)

The General Linear Model

and yXXXXIy 1

pns 1 2. 2

XXXAyAyXXXβ 11 whereˆ 1.

yBy

yXXXXIy 1

1 Now 22

2

spnU

XXXXIB 1 2

1 where

IβXy 2, ~

nN

The Estimates

Theorem

.0 with ,~ 2. 22

2

pnspnU

12, ~ ˆ 1. XXββ

pN

tindependen are and ˆ 3. 2sβ

The General Linear Model

with an intercept

npnn

p

p

pn xxx

xxxxxx

y

yy

21

22221

11211

2

1

0

2

1

1

11

,,Let Xβy

ondistributi , a has i.e. 2IβXy

N

ondistributi , a has where 2I0εεβXy

N

The matrix formulation (intercept included)

Then the model becomes

Thus to include an intercept add an extra column of 1’s in the design matrix X and include the intercept in the parameter vector

The Gauss-Markov Theorem

An important result in the theory of Linear models

Proves optimality of Least squares estimates in a more general setting

The Gauss-Markov TheoremAssume

IyβXy 2var and E

Consider the least squares estimate of

ˆ 1 yXXXβ

β

nn yayaya

2211

1

ˆ

yayXXXcβc

, an unbiased linear estimator of βc

and

Let nn ybybyb

2211ybdenote any other unbiased linear estimator of βc

βcyb ˆvarvarthen

Hypothesis testing for the GLM

The General Linear Hypothesis

Testing the General Linear Hypotheses

The General Linear Hypothesis H0: h111 + h122 + h133 +... + h1pp = h1

h211 + h222 + h233 +... + h2pp = h2

...hq11 + hq22 + hq33 +... + hqpp = hq

where h11h12, h13, ... , hqp and h1h2, h3, ... , hq are known coefficients. In matrix notation

11

qppqhβH

Testing hβH

:0H

βXyβXy

hβHHXXHhβH

ˆˆ

ˆˆ

statistictest 1

111

pn

qF

pnqFFH , if Reject 0

An Alternative form of the F statistic

pnRSS

qRSSRSSF H

0

0 assuming Squares of Sum Residual 0

HRSSH

0 assuming Squares of Sum Residual HRSS not

Confidence intervals, Prediction intervals, Confidence Regions

General Linear Model

cXXcβc 12

ˆ st pn

One at a time (1 – )100 % confidence interval for βc

(1 – )100 % confidence interval for 2 and .

2

2/1

2

22/

2

to

spnspn

22/1

22/

to

pnspns

Multiple Confidence Intervals associated with the test hβH

:0H

Theorem: Let H be a q × p matrix of rank q.

then

ccHXXHcβHc

allfor ,ˆ 1spnqqF

form a set of (1 – )100 % simultaneous confidence interval for cβHc

allfor

cβHcc

allfor Consider