Lecture Notes 1: Matrix Algebra Part C: Pivoting and ...hammond/matrixAlgC.pdfLecture Notes 1:...

Lecture Notes 1: Matrix AlgebraPart C: Pivoting and Reduced Row Echelon Form

Peter J. Hammond

revised 2019 September 8th

University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 1 of 90

Lecture Outline

Pivoting to Reach the Reduced Row Echelon FormExampleThe Row Echelon FormThe Reduced Row Echelon FormDeterminants and Inverses

Properties of DeterminantsEight Basic Rules for DeterminantsVerifying the Product RuleCofactor ExpansionExpansion by Alien Cofactors and the Adjugate MatrixInvertible Matrices

Dimensions, Rank, and MinorsColumn and Row RankSolutions to Linear Equation SystemsMinor Determinants and Determinantal Rank


Outline





Three Simultaneous EquationsConsider the following systemof three simultaneous equations in three unknowns,which depends upon two “exogenous” constants a and b:

x + y − z = 1x − y + 2z = 2x + 2y + az = b

It can be expressed, using an augmented 3× 4 matrix, as :

1 1 −1 11 −1 2 21 2 a b

Perhaps even more useful is the doubly augmented 3× 7 matrix:

1 1 −1 1 1 0 01 −1 2 2 0 1 01 2 a b 0 0 1

whose last 3 columns are those of the 3× 3 identity matrix I3.University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 4 of 90

The First Pivot StepStart with the doubly augmented 3× 7 matrix:

1 1 −1 1 1 0 01 −1 2 2 0 1 01 2 a b 0 0 1

First, we pivot about the element in row 1 and column 1to eliminate or “zeroize” the other elements of column 1.

This elementary row operation requires us to subtract row 1from both rows 2 and 3. It is equivalent to multiplying

by the lower triangular matrix E1 =

1 0 0−1 1 0−1 0 1

.

Note: this is the result of applying the same row operations to I3.

The resulting 3× 7 matrix is:

1 1 −1 1 1 0 00 −2 3 1 −1 1 00 1 a + 1 b − 1 −1 0 1


The Second Pivot StepAfter augmenting again by the identity matrix, we have:

1 1 −1 1 1 0 0 1 0 00 −2 3 1 −1 1 0 0 1 00 1 a + 1 b − 1 −1 0 1 0 0 1

Next, we pivot about the element in row 2 and column 2.Specifically, multiply the second row by −1

2 ,then subtract the new second row from the third to obtain:

1 1 −1 1 1 0 0 1 0 00 1 −3

2 −12

12 −1

2 0 0 −12 0

0 0 a + 52 b − 1

2 −32

12 1 0 1

2 1

Again, the pivot operation is equivalent to multiplying

by the lower triangular matrix E2 =

1 0 00 −1

2 00 1

2 1

,

which is the result of applying the same row operation to I3.University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 6 of 90

Case 1: Dependent Equations

In case 1, when a + 52 = 0, the equation system reduces to:

x + y − z = 1y − 3

2z = −12

0 = b − 12

In case 1A, when b 6= 12 , neither the last equation,

nor the system as a whole, has any solution.

In case 1B, when b = 12 , the third equation is redundant.

In this case, the first two equations have a general solutionwith y = 3

2z −12 and x = z + 1− y = z + 1− 3

2z + 12 = 3

2 −12z ,

where z is an arbitrary scalar.

In particular, there is a one-dimensional set of solutionsalong the unique straight line in R3 that passes through both:(i) ( 3

2 ,−12 , 0), when z = 0; (ii) (1, 1, 1), when z = 1.


Case 2: Three Independent Equations

1 1 −1 1 1 0 0 1 0 00 1 −3

2 −12

12 −1

2 0 0 −12 0

0 0 a + 52 b − 1

2 −32

12 1 0 −1

2 1

Case 2 occurs when a + 52 6= 0,

and so the reciprocal c := 1/(a + 52 ) is well defined.

Now divide the last row by a + 52 , or multiply by c , to obtain:

1 1 −1 1 1 0 00 1 −3

2 −12

12 −1

2 00 0 1 (b − 1

2 )c −32c

32c

12c

The system has been reduced to row echelon formin which the leading zeroes of each successive row formthe steps (in French, echelons, meaning rungs)of a ladder (or echelle in French)which descends steadily as one goes from left to right.University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 8 of 90

Case 2: Three Independent Equations, Third Pivot

1 1 −1 1 1 0 00 1 −3

2 −12

12 −1

2 00 0 1 (b − 1

2 )c −32c

32c

12c

Next, we zeroize the elements in the third column above row 3.

To do so, pivot about the element in row 3 and column 3.

This requires adding 1 times the last row to the first,and 3

2 times the last row to the second.

In effect, one multiplies

by the upper triangular matrix E3 :=

1 1 10 1 3

20 0 1

The first three columns of the result are

1 1 00 1 00 0 1


Case 2: Three Independent Equations, Final Pivot

As already remarked, the first three columns of the matrixwe are left with are

1 1 00 1 00 0 1

The final pivoting operation involves subtracting the second rowfrom the first, so the first three columns become the identity matrix

1 0 00 1 00 0 1

This is a matrix in reduced row echelon form because,given the leading non-zero element of any row (if there is one),all elements above this element are zero.


Final Exercise

Exercise

1. Find the last 4 columns of each 3× 7 matrixproduced by these last two pivoting steps.

2. Check that the fourth columnsolves the original system of 3 simultaneous equations.

3. Check that the last 3 columnsform the inverse of the original coefficient matrix.


Outline





Definition of Row Echelon Form

DefinitionAn m × n matrix A is in row echelon form just in case:

1. The first r ≤ m rows i ∈ Nr

each have a non-zero leading entry ai ,ì in column ìsuch that aij = 0 for all j < ì .

2. Each successive leading entry is in a column to the rightof the leading entry in the previous row.

That is, given the leading element ai ,ì 6= 0 of row i ,one has ahj = 0 for all h > i and all j ≤ ì .

3. If r < m, then any row i ∈ {r + 1, . . . ,m} = Nm \ Nr

has no leading entry, because all its elements are zero.

This row without a leading entrymust be below any row with a leading entry.


Examples

Assuming that α, β, γ ∈ R \ {0},here are three examples of matrices in row echelon form:

A =

α 2 0 00 0 β 00 0 0 γ

; B =

α 2 0 00 0 β 00 0 0 γ0 0 0 0

; C =

α 00 β0 0

Here are three examples of matricesthat are not in row echelon form

D =

0 11 00 0

; E =

1 20 10 1

; F =

1 00 00 1


Pivoting to Reach a Generalized Row Echelon Form

Any m × n matrix A can be transformed into row echelon formby applying a series of determinant preserving row operationsinvolving non-zero pivot elements.

1. Look for the first or leading non-zero column `1 in the matrix.

2. Find within column `1 an element ai1`1 6= 0with a large absolute value |ai1`1 |; this will be the first pivot.

3. Interchange rows 1 and i1, moving the pivot to the top row.

4. To preserve the determinant,change the sign of either row 1 or row i1 (not both)by multiplying that entire row by −1.

5. Subtract ai`1/a1`1 times the new row 1from each new row i > 1.

This first pivot operation will eliminate all the elementsof the pivot column `1 that lie below the new row 1.


The Intermediate Matrices and Pivot StepsAfter k − 1 pivoting operations have been completed,and column `k−1 (with `k−1 ≥ k − 1) was the last to be used:

1. The first or “top” k − 1 rows of the m × n matrixform a (k − 1)× n submatrix in row echelon form.

2. The last or “bottom” m − k + 1 rows of the m × n matrixform an (m − k + 1)× n submatrixwhose first `k−1 columns are all zero.

3. Find the first column `kthat has at least one non-zero element below row k − 1.

4. Choose as the kth pivot element the aik`k with ik ≥ kwhich has the large absolute value |aik`k |.

5. Interchange rows k and ik , moving the pivot up to row k ,and change the sign of just one of these rows.

6. Subtract ai`k/ak`k times the new row kfrom each new row i > k .

This kth pivot operation will eliminate all the elementsof the pivot column `k that lie below the new row k .


Ending the Pivoting Process

1. Continue pivoting about successive pivot elements aik`k 6= 0,moving row ik ≥ k up to row k at each stage k,while leaving all rows above k unchanged.

2. Stop after r steps when either r = m,or else all elements in the remaining m − r rows are zero,so no further pivoting is possible.


Outline





Definition of Reduced Row Echelon Form

DefinitionAn m × n matrix A is in reduced row echelon formjust in case it is in row echelon form,and the leading entry ai ,`i 6= 0 in each row iis the only non-zero entry in its column.

That is, aij = 0 for all j 6= `i .

When m = n, it is obvious that any diagonal matrixin which all diagonal elements are non-zerois in reduced row echelon form.

Assuming that α, β, γ ∈ R \ {0}, here are three more examplesof matrices in reduced row echelon form:

A =

α 2 0 00 0 β 00 0 0 γ

; B =

α 2 0 00 0 β 00 0 0 γ0 0 0 0

; C =

α 00 β0 0


Reaching a Reduced Row Echelon Form

Consider an m × n matrix C that is already in row echelon form.

Suppose it has r leading non-zero elements ck,`kin rows k = 1, 2, . . . , r , where `k is increasing in k.

Starting at the pivot element cr ,`r 6= 0 in the last pivot row r ,zeroize all the elements in column `r above this elementby subtracting from each row k above rthe multiple ck,`r /cr ,`r of row r of the matrix C,while leaving row r itself unchanged.

Repeat this pivoting operation for each of the pivot elements ck,`k ,working from cr−1,`r−1 all the way back and up to c1,`1 .


Permuting the Columns

We have shown how to transform a general m × n matrix Ainto a matrix C = RA in reduced row echelon formby applying the row operation R that equals the productof several determinant preserving row operations.

Denote the leading non-zero elements in the first r rows of Cby ck`k , where `k is increasing in k for k = 1, 2, . . . , r .

Finally, post multiply C by an n × n permutation matrix Pthat moves column `k to column k , for k = 1, 2, . . . , r .

It also partitions the matrix columns into two sets:

1. first, a complete set of r columns containing all the r pivots,with one pivot in each row and one in each column;

2. then second, the remaining n − r columns without any pivots.

So the resulting matrix CP = RAP has a diagonal sub-matrix Dr×rin its top left-hand corner; the diagonal elements of Dr×rare the pivots, all of which must be non-zero, by construction.


A Maximally Diagonalized Matrix

Our constructions have led to the equality

RAP =

(Dr×r Br×(n−r)

0(m−r)×r 0(m−r)×(n−r)

)The right-hand side is a partitioned m × n matrix,whose four sub-matrices have the indicated dimensions.

Because the diagonal matarix Dr×r is as large as possible,we may call it a maximally diagonalized matrix.

Because the diagonal of Dr×r = diag(d1, d2, . . . , dr )consists of all the non-zero pivots,the inverse D−1

r×r = diag(1/d1, 1/d2, . . . , 1/dr ) exists.

Provided that the non-negative integer r ≤ m is unique,independent of what pivots are chosen,we may want to call r the pivot rank of the matrix A.


Decomposing an m × n Matrix

Premultiplying the equality

RAP =

(Dr×r Br×(n−r)

0(m−r)×r 0(m−r)×(n−r)

)by the inverse matrix R−1, which certainly exists, gives

AP = R−1

(Dr×r Br×(n−r)

0(m−r)×r 0(m−r)×(n−r)

)Postmultiplying the result by P−1 leads to

A = R−1

(Dr×r Br×(n−r)

0(m−r)×r 0(m−r)×(n−r)

)P−1

This is a decomposition of A into the productof three matrices that are much easier to manipulate.


Three Special Cases

So far we have been writing out full partitioned matrices,as is required when the number of pivots satisfies r < min{m, n}.

There are three other special cases when r = min{m, n}.In these three cases, the maximally diagonalized m × n matrix

RAP =

(Dr×r Br×(n−r)

0(m−r)×r 0(m−r)×(n−r)

)reduces to:

1. Dn×n in case r = m = n, so m − r = n − r = 0;

2.(Dm×m Bm×(n−m)

)in case r = m < n, so m − r = 0;

3.

(Dn×n

0(m−n)×n

)in case r = n < m, so n − r = 0.


Outline





Finding the Determinant of a Square MatrixIn the case of an n × n matrix A, our earlier equality becomes

RAP =

(Dr×r Br×(n−r)

0(n−r)×r 0(n−r)×(n−r)

)The determinant of this upper triangular matrix is clearly 0except in the special case when r = n.

When r = n, there is a complete set of n pivots.

There are no missing columns, so no need to permute the columnsby applying the permutation matrix P.

Instead, we have the complete diagonalization RA = D.

Because R is determinant preserving,one has |RA| = |A| = |D| =

∏ni=1 di .

So, to calculate the determinant when r = n, it is enough:

1. to pivot to reduce A to row echelon form or diagonal form;

2. then multiply the diagonal elements.


Inverting a Square Matrix: Necessary Condition

Suppose that A is n × n, with maximally diagonal form RAP.

Consider the equation system AX = In.

Premultiplying this system by R gives

RAX = RAPP−1X =

(Dr×r Br×(n−r)

0(n−r)×r 0(n−r)×(n−r)

)P−1X = R

This has a solution only if the last n − r rows of R are all zero.

But R is determinant preserving, so |R| = 1, implying that r = n.

That is, a necessary condition for A to be invertibleis that r = n, implying that A has a full set of n pivots.


Inverting a Square Matrix: Sufficient Condition

Conversely, if r = n, then there is a complete set of pivots,so one can take P = I.

Then the maximally diagonalized matrix RAP is fully diagonalized,so AX = I is equivalent to RAX = DX = R.

The unique solution is X = A−1 = D−1R.

In this case pivoting does virtually all the work of matrix inversion.

This is because all that is left to do is:

1. invert the resulting diagonal matrix D;

2. postmultiply D−1 by the matrix R,which represents the product of all the pivoting operations.


Outline





Eight Basic Rules (Rules A–H of EMEA, Section 16.4)Let |A| denote the determinant of any n × n matrix A.

1. |A| = 0 if all the elements in a row (or column) of A are 0.

2. |A>| = |A|, where A> is the transpose of A.

3. If all the elements in a single row (or column) of Aare multiplied by a scalar α, so is its determinant.

4. If two rows (or two columns) of A are interchanged,the determinant changes sign, but not its absolute value.

5. If two of the rows (or columns) of A are proportional,then |A| = 0.

6. The value of the determinant of A is unchangedif any multiple of one row (or one column)is added to a different row (or column) of A.

7. The determinant of the product |AB| of two n × n matricesequals the product |A| · |B| of their determinants.

8. If α is any scalar, then |αA| = αn|A|.University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 30 of 90

Verifying the Transpose Rule 2The transpose rule 2 is very useful: it implies thatfor any statement S about how |A| depends on the rows of A,there is an equivalent “transpose” statement S>about how |A| depends on the columns of A.

ExerciseVerify Rule 2 directly for 2× 2 and then for 3× 3 matrices.

Proof of Rule 2 The expansion formula implies that

|A| =∑

π∈Πsgn(π)

∏n

i=1aiπ(i) =

∑π∈Π

sgn(π)∏n

j=1aπ−1(j)j

But we proved earlier that sgn(π−1) = sgn(π).

Also aπ−1(j)j = a>jπ−1(j) by definition of transpose.

Hence, because π ↔ π−1 is a bijection on the set Π,the expansion formula with π replaced by π−1

implies that |A| =∑

π−1∈Π sgn(π−1)∏n

j=1 a>jπ−1(j) = |A>|.


Verifying the Alternation Rule 4Recall the notation τr ,s for the transposition of r , s ∈ Nn.

Let Ar↔s denote the matrix that results from applying τr ,sto the rows of the matrix A — i.e., interchanging rows r and s.

TheoremGiven any n × n matrix A and any transposition τr ,s ,one has det Ar↔s = − det A.

Proof.Write τ for τr ,s . Then, because π ↔ τ−1 ◦ π is a bijection on Πn

and sgn(τ−1 ◦ π) = − sgn(π) for all π ∈ Πn, we have

det Ar↔s =∑

π∈Πnsgn(π)

∏ni=1 aτ(i),π(i)

=∑

π∈Πnsgn(π)

∏ni=1 ai ,(τ−1◦π)(i)

= −∑

π∈Πnsgn(τ−1 ◦ π)

∏ni=1 ai ,(τ−1◦π)(i)

= −∑

π∈Πnsgn(π)

∏ni=1 ai ,π(i) = − det A


The Duplication Rule, and Rule 8The following duplication rule is a special case of Rule 5.

Proposition

If two different rows r and s of A are equal, then |A| = 0.

Proof.Suppose that rows r and s of A are equal.

Then Ar↔s = A, and so |Ar↔s | = |A|.Yet the alternation Rule 4 implies that |Ar↔s | = −|A|.Hence |A| = −|A|, implying that |A| = 0.

Rule 8: |αA| = αn|A| for any α ∈ R.

Proof.The expansion formula implies that

|αA| =∑

π∈Π sgn(π)∏n

i=1(αaiπ(i))

= αn∑

π∈Π sgn(π)∏n

i=1 aiπ(i)) = αn|A|


First Implications of Multilinearity: Rules 1 and 3

Recall the notation A/b>r for the matrix that resultsafter the r th row a>r of A has been replaced by b>r .

With this notation, the matrix A/αa>r is the resultof replacing the rth row a>r of A by αa>r .

That is, it is the result of multiplying the rth row a>r of Aby the scalar α.

Rule 3: If all the elements in a single row of Aare multiplied by a scalar α, so is its determinant.

Proof.By multilinearity one has |A/αa>r | = α|A/a>r | = α|A|.

Rule 1: |A| = 0 if all the elements in a row of A are 0.

Proof.This follows from putting α = 0 in Rule 3.


More Implications of Multilinearity: Rules 5 and 6Rule 5: If two rows of A are proportional, then |A| = 0.

Proof.Suppose that a>r = αa>s where r 6= s.

Then |A| = |A/(αa>s )r | = α|A/(a>s )r | = 0 by duplication.

Rule 6: |A| is unchanged if any multiple of one row is addedto a different row of A.

Proof.For the matrix A/(a>r + αa>s )r , where α times row s of Ahas been added to row r , row multilinearity implies that

|A/(a>r + αa>s )r | = |A/(a>r )r |+ α|A/(a>s )r |

But A/(a>r )r = A and A/(a>s )r has a copy of row s in row r .

By the duplication rule, it follows that

|A/(a>r + αa>s )r | = |A/(a>r )r |+ α|A/(a>s )r | = |A|+ 0 = |A|


Outline





Verification of the Product Rule 7: Diagonal CaseRecall that Rule 7 is the product rule stating that |AB| = |A| · |B|.First we consider the special casewhen A is the n × n diagonal matrix D = diag(d1, d2, . . . , dn).

Proposition

For any n× n matrix B, one has |DB| = |D| · |B| = (∏n

k=1 dk) |B|.

Proof.First, note that (DB)i ,j =

∑nk=1 diδikbkj = dibij for all (i , j) ∈ N2

n.

Then applying the expansion formula thrice implies that

|D| =∑

π∈Πsgn(π)

∏n

i=1diδi ,π(i) =

∏n

i=1diδii =

∏n

i=1di

because the only non-zero term comes when π = ι, and also

|DB| =∑

π∈Π sgn(π)∏n

i=1 dibi ,π(i)

= (∏n

k=1 dk)∑

π∈Π sgn(π)∏n

i=1 bi ,π(i) = |D| · |B|


Determinant Preserving Row Operations: Definition

DefinitionLet Mm×n denote the family of all m × n matrices.

Then any m ×m matrix R induces, for every n ∈ N,a row operation Mm×n 3 A 7→ RA ∈Mm×n.

The row operation represented by the m ×m matrix Ris determinant preserving just in case,given any m ×m matrix X, one has |RX| = |X|.

LemmaIf the m ×m matrix R is determinant preserving, then |R| = 1.

Proof.Putting X = I in the definition gives |R| = |RI| = |I| = 1.


Determinant Preserving Row Operations: Examples

Let X denote an arbitrary n × n matrix.

Recall the notation Er+αq and Er+αqX for the matriceswhich result from applying to I and X respectivelythe elementary row operation of adding α times row q to row r .

Recall too that Tr↔s denotes the elementary row operationof interchanging rows r and s; we know that |Tr↔s | = −1.

To preserve determinants, define T∗r→s→r− as Tr↔s

followed by changing the sign of the new row r (but not row s).

Evidently, for any m ×m matrix Xwe have |Er+αqX| = |T∗r→s→r−X| = |X|.So the row operations Er+αq and T∗r→s→r−

are all determinant preserving.

This implies that their inverses all exist,with E−1

r+αq = Er−αq and (T∗r→s→r−)−1 = T∗s→r→s− .


The Subgroup of Determinant Preserving Row Operations

The set of all non-singular m ×m matrices forms a group Gmunder matrix multiplication, with identity I and matrix inversion.

The set Rm of all determinant preserving row operationson m × n matrices is a subgroup of Gm because:

1. if the m ×m matrix R is determinant preserving,then it is non-singular because |R| = |RI| = |I| = 1;

2. if the two m ×m matrices R and Sare both determinant preserving,then for every m ×m matrix X one has

|(RS)X| = |R(SX)| = |SX| = |X|

implying that RS is also determinant preserving.


Verification of the Product Rule 7: Non-Singular Case

Proposition

For any two n × n matrices A and B where |A| 6= 0,one has |AB| = |A| · |B|.

Proof.Because |A| 6= 0, there exist a non-singular diagonal matrix Dand a sequence of determinant preserving row operations 〈Rk〉mk=1

such that RA = D where R =∏m

q=1 Rq.

Because the family of all determinant preserving row operationsis a subgroup, and so closed under matrix multiplication,the matrix R, as well as its inverse R−1,are also determinant preserving row operations.

It follows that |A| = |R−1D| = |D|and also |AB| = |R−1DB| = |DB| = |D| · |B|.Hence |AB| = |D| · |B| = |A| · |B|.


Verification of the Product Rule 7: Singular Case

In case the n × n matrix A satisfies |A| = 0,pivoting reaches a maximally diagonalized matrix of the form

RAP =

(Dr×r Cr×(n−r)

0(n−r)×r 0(n−r)×(n−r)

)where n − r ≥ 1, while P is a n × n permutation matrix,and the m ×m matrix R is determinant preserving.

So there exist matrices S,T,U,V of suitable dimensionsuch that RAB = (RAP)P−1B takes the form(

Dr×r Cr×(n−r)

0(n−r)×r 0(n−r)×(n−r)

)(Sr×r Tr×(n−r)

U(n−r)×r V(n−r)×(n−r)

)

Hence |AB| = |RAB| =

∣∣∣∣DS + CU DT + CV0(n−r)×r 0(n−r)×(n−r)

∣∣∣∣ = 0 = |A| · |B|

also in this case when |A| = 0.


Verification of the Product Rule 7: Summary

Finally, therefore, in view of the previous proposition when |A| 6= 0,we have proved:

TheoremFor any n × n matrices A and B, one has |AB| = |A| · |B|.


Outline





Cofactor Expansion: Theorem

DefinitionGiven any element ars of the matrix n × n matrix A,the associated (r , s)-cofactor |Crs |is the determinant of the (n − 1)× (n − 1) matrix Crs

obtained by omitting row r and column s from A.

The cofactor expansions of |A| along any row r or column sare respectively

∑nj=1(−1)r+jarj |Crj | and

∑ni=1(−1)i+sais |Cis |.

TheoremFor every row r and column s of any n × n matrix A,these cofactor expansions are valid — i.e., one has

|A| =∑n

j=1(−1)r+jarj |Crj | =

∑n

i=1(−1)i+sais |Cis |

The proof of this theorem will occupy the next 6 slides.


Cofactor Expansion: Proof, Part 1

Later we will prove the row expansion formula.

If it is valid, then applying it to the transpose matrix A> gives

|A>| =∑n

j=1(−1)r+ja>rj |C>rj |

Taking transposes throughout gives

|A| =∑n

j=1(−1)r+jajr |Cjr |

Replacing j by i and r by s, then s + i by i + s, one obtains

|A| =∑n

i=1(−1)i+sais |Cis |

This is the column expansion formula.

So we have proved that the column expansion formula is impliedby the row expansion formula, leaving us to prove the latter.



Consider the bordered n × n matrix Crj =

(Crj (aj)−r0> 1

)whose:

1. top left hand corneris the (n − 1)× (n − 1) cofactor matrix Crj ;

2. top right hand border is the column vector (aj)−r ∈ Rn−1

that is constructed by dropping the r th elementfrom the jth column aj of the original matrix A;

3. bottom left hand borderis the n − 1-dimensional row vector 0> of zeros;

4. bottom right hand corner is the number 1.

Three lemmas will be used to show that, for each j ∈ Nn:(i) the permutations πr↗n and πj↗n

with their associated permutation matrices Pr↗n and Pj↗n

together satisfy Crj = Pr↗nAPj↗n;(ii) |A/e>j | = (−1)r+j |Crj |; and (iii) |Crj | = |Crj |.This will complete the proof that |A/(e>j )r | = (−1)r+j |Crj |.University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 48 of 90


Given k ≤ ` ≤ n, recall that πk↗` ∈ Πn moves k to `,and then moves each q ∈ {k + 1, . . . , `} to q − 1.

Let Pk↗` denote the corresponding permutation Pπk↗`.

LemmaFor each r , j ∈ Nn, one has

Crj =

(Crj (aj)−r0> 1

)= Pr↗n [A/(e>j )r ] Pj↗n

Proof.Premultiplying by Pr↗n applies πr↗n to the rows,whereas postmultiplying by Pj↗n applies πj↗n the columns.

Now the result follows immediately from the definitions of:(i) the matrix Crj ; (ii) the permutations πr↗n and πj↗n;(iii) the associated permutation matrices Pr↗n and Pj↗n.



Lemma

For each j ∈ Nn one has |Crj | =

∣∣∣∣Crj (aj)−r0> 1

∣∣∣∣ = |Crj | .

Proof.Note that (Crj)n,π(n) = δn,π(n), so the expansion formula yields

|Crj | =∑

π∈Πn

∏n

i=1(Crj)i ,π(i) =

∑π∈Πn−1

∏n−1

i=1(Crj)i ,π(i)

because all other terms are equal to zero.

But then the definition of the bordered matrix Crj implies that

|Crj | =∑

π∈Πn−1

∏n−1

i=1(Crj)i ,π(i) = |Crj |

This completes all the parts of the proofthat the row r cofactor expansion of |A| is valid.University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 51 of 90

Outline





Expansion by Alien Cofactors

Expanding along either row r or column s gives

|A| =∑n

j=1arj |Crj | =

∑n

i=1ais |Cis |

when one uses matching cofactors.

Expanding by alien cofactors, however,from either the wrong row i 6= ror the wrong column j 6= s, gives

0 =∑n

j=1arj |Cij | =

∑n

i=1ais |Cij |

This is because the answer will be the determinantof an alternative matrix in which:

I either row i has been duplicated and put in row r ;

I or column j has been duplicated and put in column s.


The Adjugate Matrix

DefinitionThe adjugate (or “(classical) adjoint”) adj Aof an order n square matrix Ahas elements given by (adj A)ij = |Cji |.

It is therefore the transpose (C+)> of the cofactor matrix C+

whose elements (C+)ij = |Cij | are the respective cofactors of A.


Main Property of the Adjugate Matrix

TheoremFor every n × n square matrix A one has

(adj A)A = A(adj A) = |A|InProof.The (i , j) elements of the two product matrices are respectively

[(adj A) A]ij =∑n

k=1|Cki |akj and [A(adj A)]ij =

∑n

k=1aik |Cjk |

These are both cofactor expansions, which are expansions by:

I alien cofactors in case i 6= j , implying that both equal 0;

I matching cofactors in case i = j , implying that both equal |A|.Hence for each pair (i , j) one has

[(adj A) A]ij = [A (adj A)]ij = |A|δij = |A|(In)ij


Outline





Existence of the Inverse Matrix

TheoremAn n × n matrix A has an inverse if and only if |A| 6= 0,which holds if and only if at least oneof the two matrix equations AX = In and XA = In has a solution.

Proof.Provided that |A| 6= 0, the identity (adj A)A = A(adj A) = |A|Inshows that the matrix X := (1/|A|) adj A is well definedand satisfies XA = AX = In, so X is the inverse A−1.

Conversely, if XA = In has a solution,then the product rule for determinants impliesthat 1 = |In| = |XA| = |X||A|.Similarly if AX = In has a solution.

In either case one has |A| 6= 0.

The rest follows from the paragraph above.


Singularity versus Invertibility

So A−1 exists if and only if |A| 6= 0.

Definition

1. In case |A| = 0,the matrix A is said to be singular;

2. In case |A| 6= 0,the matrix A is said to be non-singular or invertible.


Example and Application to Simultaneous Equations

Exercise

Verify that A =

(1 1

1 −1

)=⇒ A−1 = C :=

(12

12

12 −1

2

)by using direct multiplication to show that AC = CA = I2.

Example

Suppose that a system of n simultaneous equations in n unknownsis expressed in matrix notation as Ax = b.

Of course, A must be an n × n matrix.

Suppose A has an inverse A−1.

Premultiplying both sides of the equation Ax = b by this inversegives A−1Ax = A−1b, which simplifies to Ix = A−1b.

Hence the unique solution of the equation is x = A−1b.


Inverting Triangular Matrices

TheoremIf the inverse U−1 of an upper triangular matrix U exists,then it is upper triangular.

Taking transposes leads immediately to:

Corollary

If the inverse L−1 of a lower triangular matrix L exists,then it is lower triangular.


Inverting Triangular Matrices: Proofs

Recall the (n − 1)× (n − 1) cofactor matrix Crs

that results from omitting row r and column s of U = (uij).

When it exists, U−1 = (1/|U|) adj U, so it is enough to provethat the n × n matrix (|Crs |) of cofactor determinants,whose transpose (|Crs |)> is the adjugate, is lower triangular.

In case r < s, every element below the diagonal of the matrix Crs

is also below the diagonal of U, so must equal 0.

Hence Crs is upper triangular,with determinant equal to the product of its diagonal elements.

Yet s − r of these diagonal elements are ui+1,i for i = r , . . . , s − 1.These elements are from below the diagonal of U, so equal zero.

Hence r < s implies that |Crs | = 0, so the n × n matrix (|Crs |)of cofactor determinants is indeed lower triangular, as required.


Cramer’s Rule: Statement

NotationGiven any m × n matrix A,let [A−j ,b] denote the new m × n matrixin which column j has been replaced by the column vector b.

Evidently [A−j , aj ] = A.

TheoremProvided that the n × n matrix A is invertible,the simultaneous equation system Ax = bhas a unique solution x = A−1b whose ith componentis given by the ratio of determinants xi = |[A−i ,b]|/|A|.This result is known as Cramer’s rule.


Cramer’s Rule: Proof

Proof.Given the equation Ax = b, each cofactor |Cij | of the coefficientmatrix A is formed by dropping row i and column j of A.

It therefore equals the (i , j) cofactor of the matrix |[A−j ,b]|.

Expanding the determinant by cofactors along column jtherefore gives

|[A−j ,b]| =∑n

i=1bi |Cij | =

∑n

i=1(adj A)jibi

by definition of the adjugate matrix.

Hence the unique solution to the equation system has components

xi = (A−1b)i =1

|A|∑n

i=1(adj A)jibi =

1

|A||[A−i ,b]|

for i = 1, 2, . . . , n.


Outline





Definition of Dimension

The dimension of a linear space is the number of elementsin the largest linearly independent subset.

TheoremThe dimension of Rm is m.

To prove this, we first constructa linearly independent set of m vectors.

Indeed, consider the list (ej)mj=1 of m unit column vectors in Rm

with each ej equal to jth column of the m ×m identity matrix Im.

Obviously 0 = Imx implies that x = 0,so this list does form a linearly independent set.


Too Many Vectors Are Linearly Dependent

To complete the proof that Rm has dimension m,consider any list (yj)

nj=1 of n > m vectors in Rm.

These n vectors form the columns of an m × n matrix Y.

After applying enough suitable pivoting operations,the matrix equation Yx = 0 reduces to

RYPz =

(Dr×r Br×(n−r)

0(m−r)×r 0(m−r)×(n−r)

)(z1

z2

)= 0

where r ≤ m < n and z = P−1x.

This equation system has many non-trivial solutionsof the form z1 = −D−1Bz2 where z2 ∈ Rn−r \ {0} is arbitrary.

It follows that the list (yj)nj=1 of n vectors is linearly dependent.


Linear Independence of Matrix Columns

The n column vectors of the m × n matrix Aare linearly independent just in casethe vector equation 0m =

∑nj=1 ξjaj in Rm

implies that ξj = 0 for each j = 1, 2, . . . , n.

Or equivalently, just in case the only solution of 0m = Axis the trivial solution x = 0n.


Spanning

DefinitionGiven any finite set S = {xj ∈ Rn | j ∈ Nm} of m vectors in Rn,the set of vectors spanned by S , or the span of S , is the set

spS := {z ∈ Rn | ∀j ∈ Nm; ∃yj ∈ R : z =∑m

j=1yj xj}

Note that any vector z ∈ spSis a linear combination of the vectors in S .

ExerciseVerify that sp A is a linear subspace of Rn

— i.e., it satisfies the vector space axioms.


The Column and Row Spaces

In case the set S = {a1, . . . , an} ⊂ Rm

consists of the n columns of the m × n matrix A, one has

sp({a1, . . . , an}) = {y ∈ Rm | ∃x ∈ Rn | y = Ax}

This is the column space of A; the row space spanned by its rows,which equals the column space of A>, is given by

sp({a>1 , . . . , a>m}) = {w> ∈ Rn | ∃z> ∈ Rm | w> = z>A}


Column and Row Rank

DefinitionThe column rank of the m × n matrix Ais the dimension rC ≤ m of its column space,which is the maximum number of linearly independent columns.

The row rank of the m × n matrix Ais the dimension rR ≤ n of its row space,which is the maximum number of linearly independent rows.

Obviously, the row rank of Aequals the column rank of the transpose A>.


The Column Rank of a Maximally Diagonalized Matrix

TheoremThe maximally diagonalized m × n matrix(

Dr×r Br×(n−r)

0(m−r)×r 0(m−r)×(n−r)

)where Dr×r is invertible, has column rank r .

Proof.Given an arbitrary z ∈ Rr and w ∈ Rm−r , the vector equation(

Dr×r Br×(n−r)

0(m−r)×r 0(m−r)×(n−r)

)(xy

)=

(zw

)has a solution given by x = D−1(z− By) ∈ Rr iff w = 0m−r .

Hence the column space is Rr × {0m−r}.It is isomorphic to Rr , whose dimension is r ,the number of pivots.University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 71 of 90

The Row Rank of a Maximally Diagonalized Matrix

TheoremThe maximally diagonalized m × n matrix(

Dr×r Br×(n−r)

0(m−r)×r 0(m−r)×(n−r)

)where Dr×r is invertible has row rank r .

Proof.Given an arbitrary row vector (z>,w>) ∈ Rr × Rm−r , the equation

(x>, y>)

(Dr×r Br×(n−r)

0(m−r)×r 0(m−r)×(n−r)

)= (z>,w>)

has a solution given by (x>, y>) = (z>D−1, 0>m−r )if and only if w> = z>D−1B.

Hence the row space is {(z>,w>) ∈ Rr ×Rm−r | w> = z>D−1B}.It is isomorphic to Rr , whose dimension is r .University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 72 of 90

Invariance of Row Space

TheoremLet A be any m × n matrixand R any determinant preserving row operation.

Then A and RA have the same row space.

Proof.Suppose that w> ∈ Rn is in the row space of A,with w> = z>A where z> ∈ Rm.

Then w> = (z>R−1)RA,so w> ∈ Rn is in the row space of RA.

Conversely, suppose w> ∈ Rn is in the row space of RA,with w> = z>RA where z> ∈ Rm.

Then w> = (z>R)A,so w> ∈ Rn is in the row space of A.


Isomorphism of Column Spaces

TheoremLet A be any m × n matrixand R any determinant preserving row operation.

Then A and RA have isomorphic column spaces.

Proof.Suppose that y ∈ Rm is in the column space of A,with y = Ax where x ∈ Rn.

Then Ry = (RA)x, so Ry is in the column space of RA.

Conversely, suppose Ry is in the column space of RA,with Ry = (RA)x where x ∈ Rn.

Because R is determinant preserving, it is invertible.

Then y = R−1(RA)x = Ax, so y is in the column space of A.

It follows that y↔ Ry is a linear bijectionbetween the column spaces of A and RA.


Column Rank Equals Row Rank

TheoremSuppose the m × n matrix A can be maximally diagonalized

as RAP =

(Dr×r Br×(n−r)

0(m−r)×r 0(m−r)×(n−r)

)where D−1

r×r exists,

while R is determinant preserving, and P is a permutation.

Then both the column and row rank of A are equal to r .

Proof.Because permuting the columns of a matrix makes no differenceto its row or column rank, the row and column ranks of RAare equal to those of RAP, both of which equal r .

By the previous theorems, the two matrices A and RAhave isomorphic row and column spaces, with equal dimensions.

So the row and column ranks of A are equalto the row and column ranks of RA, both of which are r .


Outline





Rank Condition for Existence of a Solution, I

TheoremLet A be an m × n matrix, and b a column m-vector.

Then the equation Ax = b has a solution x ∈ Rn

if and only the rank of the m × (n + 1) augmented matrix (A,b)equals the rank of A.

Proof.Necessity: Suppose that Ax = b has a solution x = (xj)

nj=1.

Now apply to (A,b) the compound column operationof successively subtracting from its last columnthe multiple xj of each column j .

This converts (A,b) to (A, 0) while preserving the column rank.

Hence the ranks of (A,b) and (A, 0) are equal,with both equal to the rank of A.


Rank Condition for Existence of a Solution, II

Proof.Sufficiency: Suppose the ranks of A and (A,b) are both r .

Then there is an r × n submatrix Aconsisting of r linearly independent columns of A.

Because the rank of (A,b) equals r , and not r + 1,the r + 1 columns of (A,b) must be linearly independent.

This can only be true because there exists an r -vector xsuch that b = Ax.

By augmenting x with n − k appropriately placed zero elements,one can construct x ∈ Rn to satisfy Ax = b.

ExerciseLet A and B be m × n and m × k matrices.

Prove that the matrix equation AX = Bhas one or more solutions for the n × k matrix X if and only ifboth A and the augmented matrix (A,B) have the same rank.


Superfluous Equations and Degrees of Freedom, I

TheoremLet A be an m × n matrix, and b a column m-vector.

Suppose A and the augmented matrix (A,b) have both rank r .

1. If r < m, then Ax = b has m − r superfluous equations.

2. If r < n, then there are n − r degrees of freedomin the solution to Ax = b.

In the following proof, we assume that the m × n matrix Acan be maximally diagonalized

as RAP =

(Dr×r Br×(n−r)

0(m−r)×r 0(m−r)×(n−r)

)where D−1

r×r exists,

while R is determinant preserving, and P is a permutation.


Superfluous Equations and Degrees of Freedom, II

Proof.Under the previous assumption, the equation system Ax = bis equivalent to RAPz = w where z = P−1x and w = Rb.

This system can be written as(Dr×r Br×(n−r)

0(m−r)×r 0(m−r)×(n−r)

)(z1r

z2n−r

)=

(w1

r

w2m−r

)Here the rank of (RAP,w) is r iff w2

m−r = 0m−r ,in which case the last m − r equations are superfluous.

Then, for each z2n−r ∈ Rn−r there is a unique solution

given by z1r = D−1

r×r (w1r − Br×(n−r)z

2n−r ).

Hence there are n − r degrees of freedom.


Equation Systems: Existence of a Solution

Consider again the matrix equation AX = Y in its equivalent form

RAX = RAPP−1X =

(Dr×r Br×(n−r)

0(m−r)×r 0(m−r)×(n−r)

)P−1X = RY

Introduce the partitioned matrix

(Z1

Z2

)as notation for Z = P−1X,

where the r × p matrix Z1 consists of the first r rows of Z,and the (n − r)× p matrix Z2 consists of the other n − r rows.

The equation system takes the form(Dr×r Br×(n−r)

0(m−r)×r 0(m−r)×(n−r)

)(Z1

Z2

)= RY =

(Vr×p

W(m−r)×p

)Because the matrix Dr×r of pivots is invertible,and the last m − r rows of the left-hand side matrix are all zero,a solution exists if and only if W(m−r)×p = 0(m−r)×p.


Equation Systems: The Solution Space

The necessary and sufficient condition for solutions to existis W(m−r)×p = 0(m−r)×p.

In case this is met, the system reduces to DZ1 + BZ2 = RY1.

The general solution is Z1 = D−1(RY1 − BZ2).

Because the (n − r)× p matrix Z2 can be chosen arbitrarily,there are n − r degrees of freedom in each equation system.

The first r rows of the matrix P−1X with permuted columnshave been expressed as a linear function of Yand of these last arbitrary n − r rows of P−1X.

The remaining m − r equations are redundant.


Outline





Minors and Determinantal Rank

DefinitionGiven any m × n matrix A, a minor (determinant) of order kis the determinant |Ai1i2...ik , j1j2...jk | of a k × k submatrix (aij),whose row numbers satisfy 1 ≤ i1 < i2 < . . . < ik ≤ mand whose column numbers satisfy 1 ≤ j1 < j2 < . . . < jk ≤ n.

The matrix Ai1i2...ik , j1j2...jk , also denoted by AI×J , is formedby selecting in the right order all the elements that lie in both:

I one of the k chosen rows in the ordered set I := 〈ir 〉kr=1;

I one of the k chosen columns in the ordered set J := 〈js〉ks=1.

DefinitionThe determinantal or minor rank of a matrixis the dimension of its largest non-zero minor determinant.


Minors: Some Examples

Example

1. In case A is an n × n matrix:I the whole determinant |A| is the only minor of order n;I each of the n2 cofactors Cij is a minor of order n − 1.

2. In case A is an m × n matrix:I each element of the mn elements of the matrix

is a minor of order 1;I the number of minors of order k is(

m

k

)·(n

k

)=

m!

k!(m − k)!

n!

k!(n − k)!

ExerciseVerify that the set of elements that make upthe minor |Ai1i2...ik , j1j2...jk | of order k is completely determinedby its k diagonal elements aih,jh (h = 1, 2, . . . , k).(These need not be diagonal elements of A).


Principal and Leading Principal Minors

DefinitionIf A is an n × n matrix,the minor |Ai1i2...ik , j1j2...jk | of order k is:

I a principal minor if ih = jh for h = 1, 2, . . . , k ,implying that its diagonal elements aihjhare all on the (principal) diagonal of A;

I a leading principal minor if its diagonal elementsare the leading elementsof the (principal) diagonal of ahh (h = 1, 2, . . . , k).

ExerciseExplain why an n × n determinant has:

1. 2n − 1 principal minors;

2. n leading principal minors.


A First Lemma

LemmaGiven the m × n matrix A,suppose that |AI×J | is any non-zero minor of order k.

Then both the set {a>i | i ∈ I} of rows of Aand the set {aj | j ∈ J} of columns of A are linearly independent.

Corollary

Let r denote the row rank of the m × n matrix A,which equals its column rank.

Let d denote the determinantal rank of the m × n matrix A.

Then r ≥ d .

Proof.There is a non-zero minor |AI×J | of order d , so #I = d .

But then the same d rows of A are linearly dependent,so the row rank r ≥ d .


Proof of First Lemma

Proof.Suppose that the linear combination

∑i∈I ξi a>i

of the set of rows {a>i | i ∈ I} equals 0n.

Then∑

i∈I ξi aij = 0 for every column j ∈ Nn.

In particular,∑

i∈I ξi aij = 0 for every column j ∈ J.

So the linear combination∑

i∈I ξi a>iof the rows of the k × k matrix A = AI×J is zero.

Since |AI×J | 6= 0, these rows are linearly independent.

Hence∑

i∈I ξi a>i = 0n implies that ξi = 0 for all i ∈ I .

Finally, this implies that the set {a>i | i ∈ I}of rows of A is linearly independent.

To prove the corresponding result for columns,consider the transpose of each matrix.


A Second Lemma

LemmaSuppose that the m × n matrix A has row rank r .

Then there exist subsets I ⊆ Nm consisting of r rowsand J ⊆ Nn consisting of r columnssuch that the minor |AI×J | of order r is non-zero.

Proof.If A has row rank r ,then there exists a set I ⊆ Nm of r linearly independent rows.

These form an r × n submatrix AI×Nn whose row rank is r .

Because row and column rank are equal,it follows that AI×Nn has column rank r , where r ≤ n.

So AI×Nn has a subset J ⊆ Nn of r linearly independent columns.

These form an r × r submatrix AI×J whose rank is r .

So |AI×J | is a non-zero minor of order r .


Determinantal Rank: Theorem and Proof

TheoremThe determinantal rank d of any m × n matrix Aequals both its row and column rank r .

Proof.By the corollary to the first lemma, one has r ≥ d .

But the second lemma implies that d ≥ r .

Hence d = r .


Lecture Notes 1: Matrix Algebra Part C: Pivoting and ...hammond/matrixAlgC.pdfLecture Notes 1:...

Documents

Transcript of Lecture Notes 1: Matrix Algebra Part C: Pivoting and ...hammond/matrixAlgC.pdfLecture Notes 1:...