Direct Methods for Solving Linear Systemssmpun/MATH417/Chapter-6.pdf · 94 CHAPTER 6. DIRECT...

Chapter 6

Direct Methods for Solving LinearSystems

Linear systems of equations are associated with many problems in engineering and science, aswell as with applications of mathematics to the social sciences and the quantitative study ofbusiness and economic problems.

In this chapter we consider direct methods for solving a linear system of n equations in n variables.Such a system has the form:

a11x1 ` a12x2 ` ¨ ¨ ¨ ` a1nxn “ b1,

a21x1 ` a22x2 ` ¨ ¨ ¨ ` a2nxn “ b2,

...

an1x1 ` an2x2 ` ¨ ¨ ¨ ` a1nxn “ bn.

(6.1)

In this system, the coe�cients aij for each i, j “ 1, 2, ¨ ¨ ¨ , n, and bi for each i “ 1, 2, ¨ ¨ ¨ , n, aregiven. We need to determine the unknowns x1, x2, ¨ ¨ ¨ , xn.The linear system (6.1) can be written in the form Ax “ b, where A is an invertible n ˆ nmatrix, x and b are two n ˆ 1 vectors. Direct methods theoretically give the exact solution tothe system in a finite number of steps. In principle, using row operations, one transfer the Amatrix in (6.1) into upper triangular of the following form:

¨

˚˚˚

a11 a12 ¨ ¨ ¨ a1,n´1 a1na21 a22 ¨ ¨ ¨ a2,n´1 a2n...

.... . .

......

an´1,1 an´1,2 ¨ ¨ ¨ an´1,n´1 an´1,n

an1 an2 ¨ ¨ ¨ an,n´1 ann

˛

‹‹‹‹‹‚ùñ

¨

˚˚˚

m11 m12 ¨ ¨ ¨ m1,n´1 m1n

0 m22 ¨ ¨ ¨ m2,n´1 m2n...

.... . .

......

0 0 ¨ ¨ ¨ mn´1,n´1 mn´1,n

0 0 ¨ ¨ ¨ 0 mnn

˛

‹‹‹‹‹‚.

The process of transforming A into upper triangular form is called elimination. In terms of thematrix, by doing elimination, we find a matrix M such that MA is of upper triangular form.After that, we solve the equation

MAx “ Mb,

by doing back-substitution. In actual calculations, one does not derive explicitly the matrix M ,but only the matrix MA and the vector Mb.

In practice, of course, the solution obtained will be contaminated by the round-o↵ error thatis involved with the arithmetic being used. Analyzing the e↵ect of this round-o↵ error anddetermining ways to keep it under control will be a major component of this chapter.

93

94 CHAPTER 6. DIRECT METHODS FOR SOLVING LINEAR SYSTEMS

6.1 Review of Linear Algebra

A linear system is often replaced by a matrix, which contains all the information about thesystem that is necessary to determine its solution, but in a compact way, and one that is easilyrepresented in a computer. In this section, we recall some definitions and results of linear algebrathat will be used in this Chapter.

Definition 6.1.1 (Matrix). An n ˆ m matrix A is a rectangular array of elements with n rowsand m columns in which not only is the value of an element important, but also its position inthe array. That is,

A “

¨

˚˚

a11 a12 ¨ ¨ ¨ a1,m´1 a1ma21 a22 ¨ ¨ ¨ a2,m´1 a2m...

.... . .

......

an1 an2 ¨ ¨ ¨ an,m´1 anm

˛

‹‹‹‚.

We sometimes denote A “ paijq. If n “ m, then the matrix is called a square matrix of order n.

Definition 6.1.2 (Transpose). Given an n ˆ m matrix A “ paijq, we define its transpose as anm ˆ n matrix, denoted AT , as follows:

AT “

¨

˚˚˚

a11 a21 ¨ ¨ ¨ an1a12 a22 ¨ ¨ ¨ an2...

.... . .

...a1,m´1 a2,m´1 ¨ ¨ ¨ an,m´1

a1m a2m ¨ ¨ ¨ anm

˛

‹‹‹‹‹‚.

Definition 6.1.3 (Equivalence of matrices). Two matrices A and B are equal if they have thesame number of rows and columns, say n ˆ m, and if aij “ bij , for each i “ 1, ¨ ¨ ¨ , n, andj “ 1, ¨ ¨ ¨ ,m.

Example 6.1.4. The following matrix

A “ˆ

2 ´1 73 1 0

˙

has two rows and three columns, so it is of size 2 ˆ 3. Its entries are described by a11 “ 2,a12 “ ´1, a13 “ 7, a21 “ 3, a22 “ 1, and a23 “ 0. In this case, the transpose of A is

AT “¨

˝2 3

´1 17 0

˛

‚.

Note that AT is of size 3 ˆ 2. Based on Definition 6.1.3, we have

A “ˆ

2 ´1 73 1 0

˙‰

¨

˝2 3

´1 17 0

˛

‚“ AT

because they di↵er in dimension.

In general, a matrix and its transpose are of same size only if this matrix is a square matrix.We have the following definition if the matrix itself is equal to its transpose.

Definition 6.1.5 (Symmetric). A matrix M is called symmetric if MT “ M .

6.1. REVIEW OF LINEAR ALGEBRA 95

Example 6.1.6. The following matrix

M “¨

˝4 ´1 0

´1 4 ´10 ´1 4

˛

‚

is symmetric.

Two important operations performed on matrices are the sum of two matrices and the multipli-cation of a matrix by a real number (scalar).

Definition 6.1.7 (Sum of two matrices). If A and B are both nˆm matrices, then the sum ofA and B, denoted A ` B, is the n ˆ m matrix whose entries are aij ` bij for each i “ 1, ¨ ¨ ¨ , n,and j “ 1, ¨ ¨ ¨ ,m.

Definition 6.1.8 (Scalar multiplication). If A is an nˆm matrix and � is a real number, thenthe scalar multiplication of � and A, denoted �A, is the nˆm matrix whose entries are �aij foreach i “ 1, ¨ ¨ ¨ , n, and j “ 1, ¨ ¨ ¨ ,m.

Example 6.1.9. Determine A ` B and �A when

A “ˆ

2 ´1 73 1 0

˙, B “

ˆ4 2 ´80 1 6

˙, and � “ ´2.

Solution. We have

A ` B “ˆ

2 ` 4 ´1 ` 2 7 ´ 83 ` 0 1 ` 1 0 ` 6

˙“

ˆ6 1 ´13 2 6

˙

and

�A “ˆ ´2p2q ´2p´1q ´2p7q

´2p3q ´2p1q ´2p0q

˙“

ˆ ´4 2 ´14´6 ´2 0

˙.

We have the following general properties for matrix addition and scalar multiplication. Theseproperties are su�cient to classify the set of all n ˆ m matrices with real entries as a vectorspace over the field of real numbers. We let O denote a matrix all of whose entries are 0 andÁ denote the matrix whose entries are áij .

Theorem 6.1.10. Let A, B, C be nˆm matrices and � and µ be real numbers. The followingproperties of addition and scalar multiplication hold:

(i) A ` B “ B ` A; (ii) pA ` Bq ` C “ A ` pB ` Cq;(iii) A ` O “ O ` A “ A; (iv) A ` pÁq “ Á ` A “ O;(v) �pA ` Bq “ �A ` �B; (vi) p� ` µqA “ �A ` µA;(vii) �pµAq “ �µA; (viii) 1A “ A.

All these properties follow from similar results concerning the real numbers.

We introduce the notion of row and column vectors.


Definition 6.1.11 (Vector). The 1 ˆ n matrix

x “`x1 x2 ¨ ¨ ¨ xn

˘

is called an n-dimensional row vector, and the n ˆ 1 matrix

y “

¨

˚˚

y1y2...yn

˛

‹‹‹‚

is called an n-dimensional column vector. We sometimes denote column vector as y “ py1, y2, ¨ ¨ ¨ , ynqT .A n-dimensional (column or row) vector 0n with all entries being zero is called n-dimensionalzero vector. We denote Rn the collection of all n-dimensional column vectors.

Example 6.1.12. Any n ˆ n matrix A can be understood as an n ˆ n array, or a collection ofn-dimensional column vectors:

A “

¨

˚˚

a11 a12 ¨ ¨ ¨ a1na21 a22 ¨ ¨ ¨ a2n...

... ¨ ¨ ¨ ...an1 an2 ¨ ¨ ¨ ann

˛

‹‹‹‚.

Each column of the matrix A is a n-dimensional vector.

Definition 6.1.13. The set of n-dimensional column vectors teiuni“1:

e1 “

¨

˚˚

10...0

˛

‹‹‹‚, e2 “

¨

˚˚

01...0

˛

‹‹‹‚, ¨ ¨ ¨ , and en “

¨

˚˚

00...1

˛

‹‹‹‚

is called a natural basis of Rn. We remark that any n-dimensional column vector x “ pxiqni“1can be written in the linear combination of the basis vectors teiuni“1 as follows:

x “

¨

˚˚

x1x2...xn

˛

‹‹‹‚“ x1

¨

˚˚

10...0

˛

‹‹‹‚` x2

¨

˚˚

01...0

˛

‹‹‹‚` ¨ ¨ ¨ ` xn

¨

˚˚

00...1

˛

‹‹‹‚“ x1e1 ` x2e2 ` ¨ ¨ ¨xnen.

Definition 6.1.14 (Identity matrix of order n). We define

In :“ pe1 e2 ¨ ¨ ¨ enq “

¨

˚˚

1 0 ¨ ¨ ¨ 00 1 ¨ ¨ ¨ 0...

.... . .

...0 0 ¨ ¨ ¨ 1

˛

‹‹‹‚

to be the identity matrix of order n.

Definition 6.1.15 (Matrix-vector multiplication). Given an n ˆ m matrix A “ paijq and anm-dimensional column vector y “ py1, y2, ¨ ¨ ¨ , ymqT , the n-dimensional column vector is definedto be their product x “ Ay: for each i “ 1, ¨ ¨ ¨ , n,

x “ Ay “

¨

˚˚

x1x2...xn

˛

‹‹‹‚, where xi “mÿ

j“1

aijyj .


One can understand the matrix-vector multiplication from another point of view: the new vectorx “ Ay is a linear combination of the columns of the matrix A. That is,

x “ Ay “

¨

˚˚

a11 a12 ¨ ¨ ¨ a1ma21 a22 ¨ ¨ ¨ a2m...

.... . .

...an1 an2 ¨ ¨ ¨ anm

˛

‹‹‹‚

¨

˚˚

y1y2...ym

˛

‹‹‹‚“ y1

¨

˚˚

a11a21...

an1

˛

‹‹‹‚` y2

¨

˚˚

a12a22...

an2

˛

‹‹‹‚` ¨ ¨ ¨ ` ym

¨

˚˚

a1ma2m...

anm

˛

‹‹‹‚.

Remark. One may easily check that Inx “ x for any n-dimensional vector x P Rn.

Example 6.1.16. Let A be a 2 ˆ 3 matrix as in Example 6.1.4 and y “ p1, 2, 3qT . Then, theirproduct is

x “ Ay “ˆ

2 ´1 73 1 0

˙ ¨

˝123

˛

‚“ˆ

2 ´ 1 ¨ 2 ` 7 ¨ 33 ` 1 ¨ 2 ` 0 ¨ 3

˙“

ˆ215

˙.

Moreover, we can understand this operation from another point of view:

x “ Ay “ 1 ¨ˆ

23

˙` 2 ¨

ˆ ´11

˙` 3 ¨

ˆ70

˙“

ˆ215

˙.

We can use this matrix-vector multiplication to define general matrix-matrix multiplication.

Definition 6.1.17 (Matrix-matrix multiplication). Let A be an n ˆ m matrix and B an m ˆ pmatrix. The matrix product of A and B, denoted AB, is an nˆ p matrix C whose entries cijare

cij “mÿ

k“1

aikbkj “ ai1b1j ` ai2b2j ` ¨ ¨ ¨ ` aimbmj ,

for each i “ 1, 2, ¨ ¨ ¨ , n and j “ 1, 2, ¨ ¨ ¨ , p.

The computation of cij can be viewed as the multiplication of the entries of the i-th row of Awith corresponding entries in the j-th column of B, followed by a summation; that is,

cij “ pai1, ai2, ¨ ¨ ¨ , aimq

¨

˚˚

b1jb2j...

bmj

˛

‹‹‹‚.

This explains why the number of columns of A must equal the number of rows of B for theproduct AB to be defined. Moreover, one can also understand matrix-matrix multiplication inthe following sense:

AB “ A

¨

˚˚

b11 b12 ¨ ¨ ¨ b1pb21 b22 ¨ ¨ ¨ b2p...

... ¨ ¨ ¨ ...bm1 bm2 ¨ ¨ ¨ bmp

˛

‹‹‹‚“ A pb1 b2 ¨ ¨ ¨ bpq “ pAb1 Ab2 ¨ ¨ ¨ Abpq ,

where bj is the j-th column of the matrix B for j “ 1, 2, ¨ ¨ ¨ , p. The following example shouldserve to clarify the matrix multiplication process.


Example 6.1.18. Determine all possible produces of the matrices:

A “¨

˝3 2

´1 11 4

˛

‚, B “ˆ

2 1 ´13 1 2

˙,

C “¨

˝2 1 0 1

´1 3 2 11 1 2 0

˛

‚, and D “ˆ

1 ´12 ´1

˙.

Solution. The size of the matrices are

A : 3 ˆ 2, B : 2 ˆ 3, C : 3 ˆ 4, and D : 2 ˆ 2.

The product that can be defined, and their dimensions, are:

AB : 3 ˆ 3, BA : 2 ˆ 2, AD : 3 ˆ 2, BC : 2 ˆ 4, DB : 2 ˆ 3, and DD “ D2 : 2 ˆ 2.

These produces are

AB “¨

˝12 5 11 0 314 5 7

˛

‚, BA “ˆ

4 110 15

˙, AD “

¨

˝7 ´51 09 ´5

˛

‚,

BC “ˆ

2 4 0 37 8 6 4

˙, DB “

ˆ ´1 0 ´31 1 ´4

˙, and D2 “

ˆ ´1 00 ´1

˙.

Notice that although the matrix products AB and BA are both defined, their results are verydi↵erent: they do not even have the same dimension. We say that the matrix product operationis not commutative, that is, products in reverse order can di↵er. This is the case even whenboth products are defined and are of the same dimension. Almost any example will show this,for example,

ˆ1 11 0

˙ ˆ0 11 1

˙“

ˆ1 20 1

˙whereas

ˆ0 11 1

˙ ˆ1 11 0

˙“

ˆ1 02 1

˙.

Certain important operations involving matrix product do hold, however, as indicated in thefollowing result.

Theorem 6.1.19. Let A be n ˆ m matrix, B be an m ˆ k matrix, C be a k ˆ p matrix, D bean m ˆ k matrix, and � be a real number. The following properties hold:

• ApBCq “ pABqC;

• ApB ` Dq “ AB ` AD;

• �pABq “ Ap�Bq.

Recall that matrices that have the same number of rows as columns are called square matrixand they are important in applications. We summarize the terminology used in this Chapter.

Definition 6.1.20. • A square matrix has the same number of rows as columns.


• A diagonal matrix D “ pdijq is a square matrix with dij “ 0 if i ‰ j.

• The identity matrix of order n, is a diagonal matrix whose diagonal entries are all 1s.When the size of In is clear, this matrix is generally written simply as I. (See Definition6.1.14)

Example 6.1.21. Consider the identity matrix of order three,

I3 “¨

˝1 0 00 1 00 0 1

˛

‚.

If A is any 3 ˆ 3 matrix, then

AI3 “¨

˝a11 a12 a13a21 a22 a23a31 a32 a33

˛

‚

¨

˝1 0 00 1 00 0 1

˛

‚“¨

˝a11 a12 a13a21 a22 a23a31 a32 a33

˛

‚“ A.

One can also check that I3A “ A. The identity matrix In commutes with any n ˆ n matrix A;that is, the order of multiplication does not matter,

InA “ AIn “ A.

Keep in mind that this property is not true in general, even for square matrices.

Definition 6.1.22 (Upper and lower triangular). An upper-triangular nˆn matrix U “ puijqhas, for each j “ 1, 2, ¨ ¨ ¨ , n, the entries

uij “ 0, for each i “ j ` 1, j ` 2, ¨ ¨ ¨ , n;

and a lower-triangular n ˆ n matrix L “ p`ijq has, for each j “ 1, 2, ¨ ¨ ¨ , n, the entries

`ij “ 0, for each i “ 1, 2, ¨ ¨ ¨ , j ´ 1.

We remark that a diagonal matrix is both upper triangular and lower triangular because its onlynonzero entries must lie on the main diagonal.

Definition 6.1.23 (Invertible matrix). Assume that a square matrix A of order n is given. Ifthere exist a set of n-dimensional vectors tviuni“1 such that

Avi “ ei for all i “ 1, 2, ¨ ¨ ¨ , n,

where ei’s are the natural basis of Rn. Then, the matrix A is called invertible (or nonsingular)and the matrix containing all vectors tviuni“1 is called the inverse of A, denoting A´1. That is,

A´1 “ pv1 v2 ¨ ¨ ¨ vnq .

Moreover, we have AA´1 “ A´1A “ In. A matrix without an inverse is called singular (ornoninvertible).

The following properties regarding matrix inverses follow from the definition above.

Theorem 6.1.24. For any invertible n ˆ n matrix A:

• A´1 is unique;


• A´1 is nonsingular and pA´1q´1 “ A;

• If B is also a nonsingular n ˆ n matrix, then pABq´1 “ B´1A´1.

Example 6.1.25. Let

A “¨

˝1 2 ´12 1 0

´1 1 2

˛

‚ and B “ 1

9

¨

˝´2 5 ´14 ´1 2

´3 3 3

˛

‚.

One can check that B “ A´1.

Recall the transpose of a given matrix A, denoted AT . The proof of the next result followsdirectly from the definition of the transpose.

Theorem 6.1.26. The following operations involving the transpose of a matrix hold wheneverthe operation is possible:

• pAT qT “ A;

• pABqT “ BTAT ;

• pA ` BqT “ AT ` BT ;

• If A´1 exists, then pA´1qT “ pAT q´1.

6.2 Linear Systems of Equations

Using the notations of matrix and vector, the linear system (6.1) can be written in the matrix-vector-multiplication form: Ax “ b, where

A “

¨

˚˚

a11 a12 ¨ ¨ ¨ a1na21 a22 ¨ ¨ ¨ a2n...

.... . .

...an1 an2 ¨ ¨ ¨ ann

˛

‹‹‹‚, x “

¨

˚˚

x1x2...xn

˛

‹‹‹‚, and b “

¨

˚˚

b1b2...bn

˛

‹‹‹‚. (6.2)

Remark. In actual computation, the solution of a linear system Ax “ b, with A an invertible(square) matrix of order n , is not obtained by calculating the matrix A´1 and then calculatingx “ A´1b. Calculating the matrix A´1 is e↵ectively equivalent to solving the n linear systems

Avi “ ei, for all i “ 1, 2, ¨ ¨ ¨ , n,

with ej ’s are the natural basis of Rn. In order words, by such a method one would be replacingthe solution of n linear systems, followed by the multiplication of A´1 by the vector b.

Back-substitution

The methods which we study in this chapter are based on the following observation: If theinvertible matrix A is upper triangular (or lower triangular), then, the numerical solution of a

6.2. LINEAR SYSTEMS OF EQUATIONS 101

linear system Ax “ b is immediate; in fact, it can be written as

a11x1 ` a12x2 ` ¨ ¨ ¨ ` a1,n´1xn´1 ` a1nxn “ b1,

a22x2 ` ¨ ¨ ¨ a2,n´1xn´1 ` a2nxn “ b2,

...

an´1,n´1xn´1 ` an´1,nxn “ bn´1,

annxn “ bn,

and, since a11a22 ¨ ¨ ¨ ann “ detpAq ‰ 0, the system is solved by calculating xn from the lastequation, then xn´1 from the last but one, etc., giving

xn “ a´1nnbn,

xn´1 “ ann´1,n´1pbn´1 ´ an´1,nxnq,...

x1 “ a´111 pb1 ´ a12x2 ´ ¨ ¨ ¨ ´ a1,n´1xn´1 ´ a1nxnq.

(6.3)

The method of calculation given above, called back-substitution (or backward-substitution), re-quires a total of

$’’’’’’’&

’’’’’’’%

1 ` 2 ` ¨ ¨ ¨ ` pn ´ 1q “ npn ´ 1q2

additions/subtraction,

1 ` 2 ` ¨ ¨ ¨ ` pn ´ 1q “ npn ´ 1q2

multiplications, and

n divisions.

Note that each component xi appears as a linear combination of bi, bi`1, ¨ ¨ ¨ , bn, this shows thatthe inverse of a triangular matrix is (still) a triangular matrix of the same type (upper or lower).

Gaussian-elimination

We briefly discuss the process of elimination, transforming a general matrix to the form of uppertriangular. Recall the linear system (6.1) and we refer to the i-th row as equation Ei. We usethree operations to do elimination (without changing the solution):

1. Equation Ei can be multiplied by any nonzero constant � ‰ 0 with the resulting equationused in place of Ei. This operation is denoted p�Eiq Ñ pEiq.

2. Equation Ej can be multiplied by any constant � and added to equation Ei with theresulting equation used in place of Ei. This operation is denoted pEi ` �Ejq Ñ pEiq.

3. Equations Ei and Ej can be transposed in order. This operation is denoted pEiq Ø pEjq.

By a sequence of these operations, a linear system will be systematically transformed into a newlinear system that is upper triangular. The upper triangular system is more easily solved andhas the same solutions. The sequence of operations is illustrated in the following example.

Example 6.2.1. Consider the following linear system:

E1 : x1 `x2 `3x4 “ 4,E2 : 2x1 `x2 ´x3 `x4 “ 1,E3 : 3x1 ´x2 ´x3 `2x4 “ ´3,E4 : ´x1 `2x2 `3x3 ´x4 “ 4.

(6.4)


We will solve for x1, x2, x3, and x4. First use equation E1 to eliminate the unknown x1 fromequations E2, E3, and E4 by performing pE2´2E1q Ñ pE2q, pE3´3E1q Ñ pE3q, and pE4`E1q ÑpE4q. For example, in the second equation, we perform pE2 ´ 2E1q Ñ pE2q produces

p2x1 ` x2 ´ x3 ` x4q ´ 2px1 ` x2 ` 3x4q “ 1 ´ 2p4q ùñ ´x2 ´ x3 ´ 5x4 “ ´7.

Then, the linear system becomes

E1 : x1 `x2 `3x4 “ 4,E2 : ´x2 ´x3 ´5x4 “ ´7,E3 : ´4x2 ´x3 ´7x4 “ ´15,E4 : 3x2 `3x3 `2x4 “ 8.

(6.5)

For simplicity, the new equations are again labeled E1, E2, E3, and E4. In the new system (6.5),E2 is used to eliminate the unknown x2 from E3 and E4 by performing pE3 ´ 4E2q Ñ pE3q andpE4 ` 3E2q Ñ pE4q. This results in

E1 : x1 `x2 `3x4 “ 4,E2 : ´x2 ´x3 ´5x4 “ ´7,E3 : 3x3 `13x4 “ 13,E4 : ´13x4 “ ´13.

(6.6)

The linear system (6.6) is now in upper triangular form and can be solved for the unknowns bya back-substitution. Since E4 in (6.6) implies x4 “ 1, we can solve E3 for x3 to give

x3 “ 1

3p13 ´ 13x4q “ 0.

Continuing, E2 gives

x2 “ ´p´7 ` 5x4 ` x3q “ ´p´7 ` 5 ` 0q “ 2,

and E1 givesx1 “ 4 ´ 3x4 ´ x2 “ 4 ´ 3 ´ 2 “ ´1.

The solution to linear system (6.4) is therefore x1 “ ´1, x2 “ 2, x3 “ 0, and x4 “ 1.

Notice that one would not need to write out the full equations at each step or to carry thevariables through the calculations, if they always remained in the same column. The onlyvariation from system to system occurs in the coe�cients of the unknowns and in the values onthe right side of the equations. For this reason, instead of the original equation (6.4), one canconsider the augmented matrix of (6.4):

¨

˚˝

1 1 0 3 42 1 ´1 1 13 ´1 ´1 2 ´3

´1 2 3 ´1 4

˛

‹‹‚.

Performing the operations as described in the example produces the augmented matrices

¨

˚˝

1 1 0 3 40 ´1 ´1 ´5 ´70 ´4 ´1 ´7 ´150 3 3 2 8

˛

‹‹‚ and

¨

˚˝

1 1 0 3 40 1 ´1 1 10 0 3 13 130 0 0 ´13 ´13

˛

‹‹‚.


One can perform back-substitution to obtain x1, x2, x3, and x4. This procedure is called Gaus-sian elimination with back substitution.

The general Gaussian elimination procedure applied to the linear system

E1 : a11x1 ` a12x2 ` ¨ ¨ ¨ ` a1nxn “ b1,

E2 : a21x1 ` a22x2 ` ¨ ¨ ¨ ` a2nxn “ b2,

...

En : an1x1 ` an2x2 ` ¨ ¨ ¨ ` a1nxn “ bn.

(6.7)

is handled in a similar manner. First, form the augmented matrix:

¨

˚˚

a11 a12 ¨ ¨ ¨ a1n b1a21 a22 ¨ ¨ ¨ a2n b2...

.... . .

......

an1 an2 ¨ ¨ ¨ ann bn

˛

‹‹‹‚. (6.8)

Provided a11 ‰ 0, we perform the operations corresponding to

pEj ´ paj1{a11qE1q Ñ pEjq for each j “ 2, 3, ¨ ¨ ¨ , n

to eliminate the coe�cient of x1 in each of these rows. The augmented matrix reads:¨

˚˚˝

a11 a12 ¨ ¨ ¨ a1n b10 ap1q

22 ¨ ¨ ¨ ap1q2n bp1q

2...

.... . .

......

0 ap1qn2 ¨ ¨ ¨ ap1q

nn bp1qn

˛

‹‹‹‹‚.

The superscript p1q means the elimination process has been done once, and so on. We thenfollow a sequential procedure for i “ 2, 3, ¨ ¨ ¨ , n ´ 1 and perform the operation

pEj ´ papi´1qji {api´1q

ii qEiq Ñ pEjq for each j “ i ` 1, i ` 2, ¨ ¨ ¨ , n

provided api´1qii ‰ 0. This eliminates xi in each row below the i-th for all i “ 1, 2, ¨ ¨ ¨ , n ´ 1.

The resulting matrix has the form of upper triangular and perform back-substitution (6.3) toobtain the solution x1, x2, ¨ ¨ ¨ , xn.

Example 6.2.2. Represent the linear system

E1 : x1 ´x2 `2x3 ´x4 “ ´8,E2 : 2x1 ´2x2 `3x3 ´3x4 “ ´20,E3 : x1 `x2 `x3 “ ´2,E4 : x1 ´x2 `4x3 `3x4 “ 4,

(6.9)

as an augmented matrix and use Gaussian elimination to find its solution.

Solution. The augmented matrix is

A “ Ap1q “

¨

˚˝

1 ´1 2 ´1 ´82 ´2 3 ´3 ´201 1 1 0 ´21 ´1 4 3 4

˛

‹‹‚.


Performing the operations pE2 ´ 2E1q Ñ pE2q, pE3 ´ E1q Ñ pE3q, and pE4 ´ E1q Ñ pE4q gives

Ap2q “

¨

˚˝

1 ´1 2 ´1 ´80 0 ´1 ´1 ´40 2 ´1 1 60 0 2 4 12

˛

‹‹‚.

The diagonal entry ap1q22 , called the pivot element, is 0, so the procedure cannot continue in its

present form. But operations pEiq Ø pEjq are permitted, so a search is made of the elements

ap1q32 and ap1q

42 for the first nonzero element. Since ap1q32 “ 2 ‰ 0, the operation pE2q Ø pE3q is

performed to obtain a new matrix,

Ap3q “

¨

˚˝

1 ´1 2 ´1 ´80 2 ´1 1 60 0 ´1 ´1 ´40 0 2 4 12

˛

‹‹‚.

Since x2 is already eliminated from E3 and E4, and the computations continue with the operationpE4 ` 2E3q Ñ pE4q, giving

Ap4q “

¨

˚˝

1 ´1 2 ´1 ´80 2 ´1 1 60 0 ´1 ´1 ´40 0 0 2 4

˛

‹‹‚.

Finally, the back substitution is applied to obtain the solution:

x4 “ 4

2“ 2,

x3 “ p´4 ´ p´1qx4q´1

“ 2,

x2 “ p6 ´ x4 ´ p´1qx3q2

“ 3,

x1 “ p´8 ´ p´1qx4 ´ 2x3 ´ p´1qx2q1

“ ´7.

This example illustrate what is done if one of the pivot elements is zero. The k-th column of

Apkq from the k-th row to the n-th row is searched for the first nonzero entry. If apk´1qjk ‰ 0

for some j with k ` 1 § j § n, then the operation pEkq Ø pEjq is performed to obtain the

augmented matrix. The procedure can then be continued. If apk´1qjk “ 0 for each j, it can be

shown that the linear system does not have a unique solution and the procedure stops. Finally,

if apn´1qnn “ 0, the linear system does not have a unique solution, and again the procedure stops.

Finally, let us count the number of elementary operations required in Gaussian elimination withback substitution.

(i) Elimination: in advancing from Apkq to the augmented matrix Apk`1q, 1 § k § n ´ 1,n ´ k divisions are carried out and pn ´ k ` 2qpn ´ kq “ pn ´ kq2 ` 2pn ´ kq additions and


multiplications, or a total of

$’’’’’’’’&

’’’’’’’’%

npn ´ 1qp2n ` 5q6


npn ´ 1qp2n ` 5q6


npn ´ 1q2

divisions.

(ii) The back-substitution step requires$’’’’’’’&

’’’’’’’%

npn ´ 1q2


npn ´ 1q2


n divisions.

In the final count, Gaussian elimination with back substitution requires operations of the orderof $

’’’’’’’’’&

’’’’’’’’’%

n3

3additions/subtraction,

n3

3multiplications, and

n2

2divisions.

It is very instructive to compare the number of elementary operations needed for Gaussianelimination with the number of elementary operations required in applying Cramer’s rule:

xi “ detpBiqdetpAq , where Bi “

¨

˚˚

a11 ¨ ¨ ¨ a1,i´1 b1 a1,i`1 ¨ ¨ ¨ a1na21 ¨ ¨ ¨ a2,i´1 b2 a2,i`1 ¨ ¨ ¨ a2n...

......

......

an1 ¨ ¨ ¨ an,i´1 bn an,i`1 ¨ ¨ ¨ ann

˛

‹‹‹‚,

for which it is necessary to evaluate n ` 1 determinants and to carry out n divisions. Now thebrute-force calculation of a determinant requires n! ´ 1 additions and pn ´ 1qn! multiplications,so that the use of Cramer’s rule requires something of the order of

$&

%

pn ` 1q! additions,pn ` 2q! multiplications,n divisions.

For n “ 10, for example, this gives approximately"

700 operations for Gaussian elimination,400, 000, 000 operations for Cramer’s rule.

It should be remembered that Gaussian elimination is the method most frequently used to solvelinear systems whose matrices do not have any special properties. In particular, this method isemployed for system with full matrices.


6.3 Pivoting Strategies

In Example 6.2.2, we see that a row interchange was needed when one of the pivot elements

apk´1qkk “ 0. This row interchange has the form pEkq Ø pEjq, where j is the smallest integer

greater than k with apk´1qjk ‰ 0. Moreover, to reduce round-o↵ error, it is often necessary to

perform row interchanges even when the pivot elements are not zero.

If the pivot element apk´1qkk is small in magnitude compared to apk´1q

jk , then the magnitude of

the multiplier mjk “ apk´1qjk {apk´1q

kk will be much larger than 1. Round-o↵ error introduced in

the computation of one of the terms apk´1qkl is multiplied by mjk when computing apkq

jl , whichcompounds the original error. Also, when performing the back substitution for

xk “bpk´1qk ´ ∞n

j“k`1 apk´1qkj

apk´1qkk

,

with a small value of apk´1qkk , any error in the numerator can be dramatically increased because

of the division by apk´1qkk . In the next example, we see that even for small systems, round-o↵

error can dominate the calculations.

Example 6.3.1. Apply Gaussian elimination to the system

E1 : 0.003x1 `59.14x2 “ 59.17,E2 : 5.291x1 ´6.130x2 “ 46.78,

(6.10)

using four-digit arithmetic with rounding, and compare the results to the exact solution x1 “10.00 and x2 “ 1.000.

Solution. The first pivot element, a11 “ ap0q11 “ 0.003, is small, and its associated multiplier,

m21 “ 5.291

0.003“ 1763.6 96,

rounds to the large number 1764. Performing pE2´m21E1q Ñ pE2q and the appropriate roundinggives the system

E1 : 0.003x1 `59.14x2 « 59.17,E2 : ´104300x2 « ´104400,

instead of the exact system, which is

E1 : 0.003x1 `59.14x2 “ 59.17,E2 : ´104309.37 96x2 “ ´104309.37 96.

The disparity in the magnitudes of m21b1 and b2 has introduced round-o↵ error, but the round-o↵ error has not yet been propagated. Back substitution yields x2 « 1.001, which is a closeapproximation to the actual value, x2 “ 1.000. However, because of the small pivot a11 “ 0.003,it yields

x1 « 59.17 ´ p59.14qp1.001q0.003

“ 59.170 ´ 59.200

0.003“ ´10.00.

This ruins the approximation to the actual value x1 “ 10.00.

6.3. PIVOTING STRATEGIES 107

Partial pivoting

Example 6.3.1 shows how di�culties can raise when the pivot element is small relative to therest of the entries in the column. To avoid this problem, pivoting is performed by selecting an

element apk´1qjk with a larger magnitude as the pivot, and interchanging the k-th and j-th rows

for j • k. The simplest strategy is to select a smallest integer j • k such that

��apk´1qjk

�� “ maxk§i§n

��apk´1qik

��

and perform pEkq Ø pEjq. This technique just described is called partial pivoting.

Example 6.3.2. Consider the linear system (6.10) and apply Gaussian elimination using partialpivoting and four-digit arithmetic with rounding, and compare the results to the exact solutionx1 “ 10.00 and x2 “ 1.000.

Solution. The partial-pivoting procedure first requires finding

max!��ap0q

11

�� ,��ap0q

21

��)

“ |5.291| “��ap0q

21

�� .

This requires that the operation pE2q Ø pE1q be performed to produce the equivalent system

E1 : 5.291x1 ´6.130x2 “ 46.78,E2 : 0.003x1 `59.14x2 “ 59.17.

The multiplier for this system ism21 “ ap0q21 {ap0sq

11 “ 0.0005670, and the operation pE2´m21E1q ÑpE2q reduces the system to

E1 : 5.291x1 ´6.130x2 « 46.78,E2 : 59.14x2 « 59.12.

The four-digit rounding answers resulting from the back substitution are the correct valuesx1 “ 10.00 and x2 “ 1.000.

Using the strategy of partial pivoting, each multiplier mji has magnitude less than or equal to1. Although this strategy is su�cient for many linear systems, situations do arise when it isinadequate.

Example 6.3.3. The linear system

E1 : 30.00x1 `591400x2 “ 591700,E2 : 5.291x1 ´6.130x2 “ 46.78.

is the same as that in Example 6.3.1 except that all the entries in the first equation have beenmultiplied by 104. The partial pivoting procedure with four-digit arithmetic leads to the sameresults as obtained in Example 6.3.1. The maximal value in the first column is 30.00, and themultiplier m21 “ 5.291{30.00 “ 0.1764 leads to the system

E1 : 30.00x1 `591400x2 « 591700,E2 : ´104300x2 « ´104400,

which has the same inaccurate solutions as in Example 6.3.1: x1 « ´10.00 and x2 « 1.001.


Scaled partial pivoting

We introduce one more strategy for pivoting: Scaled partial pivoting. It places the elementin the pivot position that is largest relative to the entries in its row. The first step in thisprocedure is to define a scale factor si for each row:

si :“ max1§j§n

|aij | .

If we have si “ 0 for some i, then the system has no unique solution since all entries in the i-throw are 0. Assuming that this is not the case, the appropriate row interchange to place zeros inthe first column is determined by choosing the least integer p with

|ap1|sp

“ max1§k§n

|ak1|sk

and performing pE1q Ø pEpq. The e↵ect of scaling is to ensure that the largest element in eachrow has a relative magnitude of 1 before the comparison for row interchange is performed.

In a similar manner, before eliminating the variable xi using the operations pEk ´ mkiEiq fork “ i ` 1, ¨ ¨ ¨ , n, we select the smallest integer p • i with

|api|sp

“ max1§k§n

|aki|sk

and perform the row interchange pEiq Ø pEpq if i ‰ p. The scale factors s1, ¨ ¨ ¨ , sn are com-puted only once, at the start of the procedure. They are row dependent, so they must also beinterchanged when row in interchanges are performed.

Example 6.3.4. Consider the linear system

E1 : 30.00x1 `591400x2 “ 591700,E2 : 5.291x1 ´6.130x2 “ 46.78.

We apply scaled partial pivoting to this system. First, compute

s1 “ maxt|30.00| , |591400|u “ 591400 and s2 “ maxt|5.291| , |´6.130|u “ 6.130.

Consequently,

|a11|s1

“ 30.00

591400“ 0.5073 ˆ 10´4 and

|a21|s2

“ 5.291

6.130“ 0.8631,

and the interchange pE1q Ø pE2q is made. Applying Gaussian elimination to the new system

E1 : 5.291x1 ´6.130x2 “ 46.78,E2 : 30.00x1 `591400x2 “ 591700,

produces the correct results: x1 “ 10.00 and x2 “ 1.000.

The first additional computations required for scaled partial pivoting result from the determi-nation of the scale factors; there are pn ´ 1q comparison for each of the n rows, for a total ofnpn ´ 1q comparisons.

To determine the correct first interchange, n division are performed, followed by n ´ 1 compar-isons. Hence, the first interchange determination adds n divisions and pn ´ 1q comparisons.

6.4. THE DETERMINANT OF A MATRIX 109

The scaling factors are computed only once, so the second step requires: pn ´ 1q divisions andpn ´ 2q comparisons. We proceed in a similar manner until there are zeros below in the maindiagonal in all but the n-th row. The final step requires that we perform 2 divisions and 1comparison. As a consequence, scaled partial pivoting adds a total of

npn ´ 1q `n´1ÿ

k“1

k “ npn ´ 1q ` npn ´ 1q2

“ 3

2npn ´ 1q comparisons,

andnÿ

k“2

k “˜

nÿ

k“1

k

¸´ 1 “ npn ` 1q

2´ 1 “ 1

2pn ´ 1qpn ` 2q divisons

to the Gaussian elimination procedure. The time required to perform a comparison is about thesame as an addition or subtraction. Since the total time to perform the basic Gaussian elimina-tion procedure is Opn3q, scaled partial pivoting does not add significantly to the computationaltime required to solve a system for large values of n.

6.4 The Determinant of a Matrix

The determinant of a matrix provides existence and uniqueness results for linear systems havingthe same number of equations and unknowns. We will denote the determinant of a square matrixA by detpAq, but it is also common to use the notation |A|.

Definition 6.4.1 (Determinant). Suppose that A is a square matrix.

• If A “ paq is a 1 ˆ 1 matrix, then detpAq “ a.

• If A is an nˆnmatrix, with n ° 1, theminorMij is the determinant of the pn´1qˆpn´1qsubmatrix of A obtained by deleting the i-th row and j-th column of the matrix A.

• The cofactor Aij associated with Mij is defined by Aij “ p´1qi`jMij .

• The determinant of the n ˆ n matrix A, when n ° 1, is given either by

detpAq “nÿ

j“1

aijAij , for any i “ 1, 2, ¨ ¨ ¨ , n,

or

detpAq “nÿ

i“1

aijAij , for any j “ 1, 2, ¨ ¨ ¨ , n.

It can be shown that to calculate the determinant of a general n ˆ n matrix by this definitionrequires Opn!q multiplications/divisions and addition/subtractions. Even for relatively smallvalues of n, the number of calculations becomes unwieldy.

Although it appears that there are 2n di↵erent definition of detpAq, depending on which row orcolumn is chosen, all definitions give the same numerical result. The flexibility in the definitionis used in the following example. It is most convenient to compute detpAq across the row ordown the column with the most zeros.


Example 6.4.2. Find the determinant of the matrix

A “

¨

˚˝

2 ´1 3 04 ´2 7 0

´3 ´4 1 56 ´6 8 0

˛

‹‹‚

using the row or column with the most zero entries.

Solution. To compute detpAq, it is easiest to use the fourth column:

detpAq “ a14A14 ` a24A24 ` a34A34 ` a44A44 “ 5A34 “ ´5M34.

Eliminating the third row and the fourth column gives

detpAq “ ´5 det

¨

˝2 ´1 34 ´2 76 ´6 8

˛

‚“ ´30.

The following properties are useful in relating linear systems and Gaussian elimination to de-terminants.

Theorem 6.4.3. Suppose that A is an n ˆ n matrix.

• If any row or column of A has only zero entries, then detpAq “ 0.

• If A has two rows or two columns the same, then detpAq “ 0.

• If A is obtained from A by the operation pEiq Ø pEjq, with i ‰ j, then detpAq “ ´detpAq.• If A is obtained from A by the operation p�Eiq Ø pEiq, then detpAq “ � detpAq.• If A is obtained from A by the operation pEi ` �Ejq Ø pEiq, with i ‰ j, then detpAq “

detpAq.• If B is also an n ˆ n matrix, then detpABq “ detpAqdetpBq.• detpAT q “ detpAq.• When A´1 exists, detpA´1q “ pdetpAqq´1.

• If A is an upper triangular, lower triangular, or diagonal matrix, then detpAq “ ±ni“1 aii.

The determinant of a triangular matrix is simply the product of its diagonal elements. Byemploying the row operations we can reduce a given square matrix to triangular form to find itsdeterminant.

Example 6.4.4. One can compute the determinant of the matrix

A “

¨

˚˝

2 1 ´1 11 1 0 3

´1 2 3 ´13 ´1 ´1 2

˛

‹‹‚

and find out that detpAq “ 39 using the properties of determinant and row operations.

6.5. MATRIX FACTORIZATION 111

The key result relating non-singularity, Gaussian elimination, linear systems, and determinantsis that the following statements are equivalent.

Theorem 6.4.5. The following statements are equivalent for any n ˆ n matrix A.

• The equation Ax “ 0 has the unique solution x “ 0.

• The system Ax “ b has a unique solution for any n-dimensional column vector b.

• The matrix A is nonsingular; that is, A´1 exists.

• detpAq ‰ 0.

• Gaussian elimination with row interchanges can be performed on the system Ax “ b forany n-dimensional column vector b.

6.5 Matrix Factorization

Gaussian elimination is the principal direct technique to solve linear systems of equations. Inthis section, we see that the steps used to solve a system of the form Ax “ b can be used tofactor a matrix. The factorization is particularly useful when it has the form A “ LU , where Lis lower triangular and U is upper triangular.

Previously, we found that Gaussian elimination applied to an arbitrary linear system requiresOpn3{3q arithmetic operations to determine its solution. However, to solve a linear system thatinvolves an upper-triangular system requires only backward substitution, which takes Opn2qoperations. The number of operations required to solve a lower-triangular system is similar.

Suppose that A has been factored into the triangular form A “ LU , where L is lower triangularand U is upper triangular. Then, we can solve for x more easily by using a two-step process:

• First, let y “ Ux and solve the lower triangular system Ly “ b for y. Since L is lower-triangular, determining y from this equation requires only Opn2q operations.

• Once y is known, the upper triangular system Ux “ y requires only an additional Opn2qoperations to determine the solution x.

Therefore, solving a linear system Ax “ b in factored form means that the number of operationsneeded to solve the system is reduced from Opn3{3q to Op2n2q. Not surprisingly, the reductionsfrom the factorization come at a cost; determining the specific matrices L and U requires Opn3{3qoperations. But once the factorization is determined, systems involving the matrix A can besolved in this simplified manner for any number of vectors b.

To see which matrices have an LU factorization and to find how it is determined, first supposethat Gaussian elimination can be performed on the system Ax “ b without row interchanges.

This is equivalent to having nonzero pivot elements apiqii for each i “ 1, 2, ¨ ¨ ¨ , n.

The first step in the Gaussian elimination process consists of performing, for each j “ 2, 3, ¨ ¨ ¨ , n,the operations

pEj ´ mj,1E1q Ñ pEjq, where mj1 “ aj1a11

. (6.11)

These operations transform the system into one in which all the entries in the first column belowthe diagonal are zero. The system of operations in (6.11) can be viewed in another way. It is


simultaneously accomplished by multiplying the original matrix A on the left by the matrix

M p1q :“

¨

˚˚˚˝

1 0 0 ¨ ¨ ¨ 0´m21 1 0 ¨ ¨ ¨ 0

´m31 0 1. . . 0

......

. . .. . .

...´mn1 0 0 ¨ ¨ ¨ 1

˛

‹‹‹‹‹‹‚.

This matrix is called the first Gaussian transformation matrix. We denote Ap0q ” A,Ap1q “ M p1qAp0q, and

Ap1qx “ M p1qAp0qx “ M p1qAx “ M p1qb “: bp1q.

In a similar manner, we construct M p2q, the identity matrix with the entries below the diagonalin the second column replaced by the negatives of the multipliers

mj2 “ap1qj2

ap1q22

.

We denote Ap2q “ M p2qAp1q and bp2q “ M p2qbp1q. Then, we have Ap2qx “ bp2q. In general, withApkqx “ bpkq already formed, multiply by the k-th Gaussian transformation matrix

M pkq “

¨

˚˚˚˚˚˚˚˚

1 0 ¨ ¨ ¨ ¨ ¨ ¨ ¨ ¨ ¨ ¨ ¨ ¨ ¨ ¨ ¨ 0

0 1. . .

... 0...

. . .. . .

. . ....

0 ¨ ¨ ¨ 0 1. . . ¨ ¨ ¨ ¨ ¨ ¨ 0

...... ´mk`1,k

. . .. . .

......

...... 0

. . .. . .

......

......

.... . . 1 0

0 ¨ ¨ ¨ 0 ´mnk 0 ¨ ¨ ¨ 0 1

˛

‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‚

,

to obtain

Apkqx “ M pkqApk´1qx “ M pkq ¨ ¨ ¨M p1qAx “ M pkqbpk´1q “ bpkq “ M pkq ¨ ¨ ¨M p1qb. (6.12)

The process ends with the formation of Apnqx “ bpnq, where Apnq is the upper triangular matrix

Apnq “

¨

˚˚˝

a11 a12 ¨ ¨ ¨ a1n0 ap1q

22 ¨ ¨ ¨ ap1q2n

......

. . ....

0 0 ¨ ¨ ¨ apn´1qnn

˛

‹‹‹‹‚

given byApnq “ M pn´1qM pn´2q ¨ ¨ ¨M p1qA.

This process forms the U “ Apnq portion of the matrix factorization A “ LU . To determine thecomplementary lower triangular matrix L, first recall the multiplication of Apkqx “ bpkq by theGaussian transformation of M pkq used to obtain (6.12):

Apkqx “ M pkqApk´1qx “ M pkqbpk´1q “ bpkq,


where M pkq generates the row operations

pEj ´ mjkEkq Ñ pEjq for j “ k ` 1, ¨ ¨ ¨ , n.To reverse the e↵ects of this transformation and return to Apkq requires that the operationspEj ` mjkEkq Ñ pEjq be performed for each j “ k ` 1, ¨ ¨ ¨ , n. This is equivalent to multiplyingby the inverse of the matrix M pkq, the matrix

Lpkq “ pM pkqq´1 “

¨

˚˚˚˚˚˚˚˚

1 0 ¨ ¨ ¨ ¨ ¨ ¨ ¨ ¨ ¨ ¨ ¨ ¨ ¨ ¨ ¨ 0

0 1. . . 0

.... . .

. . .. . .

...... 0

. . .. . .

......

... mk`1,k. . .

. . ....

......

... 0. . .

. . ....

......

......

. . . 1 00 ¨ ¨ ¨ 0 mn,k 0 ¨ ¨ ¨ 0 1

˛

‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‚

.

The lower triangular matrix L in the factorization of A is the product of the matrices Lpkq:

L “ Lp1qLp2q ¨ ¨ ¨Lpn´1q “

¨

˚˚

1 0 ¨ ¨ ¨ 0m21 1 ¨ ¨ ¨ 0...

.... . .

...mn1 mn2 ¨ ¨ ¨ 1

˛

‹‹‹‚,

since the product of L with the upper triangular matrix U “ M pn´1qM pn´2q ¨ ¨ ¨M p1qA gives

LU “ Lp1qLp2q ¨ ¨ ¨Lpn´1qM pn´1qM pn´2q ¨ ¨ ¨M p1qA “ A.

We summarize the result above in the following theorem.

Theorem 6.5.1 (LU -factorization). If Gaussian elimination can be performed on the linearsystem Ax “ b without row interchanges, then the matrix A can be factored into the productof a lower triangular matrix L and an upper triangular matrix U , that is, A “ LU , with

mji “ apiqji {apiq

ii ,

U “

¨

˚˚˝

a11 a12 ¨ ¨ ¨ a1n0 ap1q

22 ¨ ¨ ¨ ap1q2n

......

. . ....

0 0 ¨ ¨ ¨ apn´1qnn

˛

‹‹‹‹‚and L “

¨

˚˚

1 0 ¨ ¨ ¨ 0m21 1 ¨ ¨ ¨ 0...

.... . .

...mn1 mn,2 ¨ ¨ ¨ 1

˛

‹‹‹‚.

Example 6.5.2. (a) Determine the LU factorization for matrix A, where

A “

¨

˚˝

1 1 0 32 1 ´1 13 ´1 ´1 2

´1 2 3 ´1

˛

‹‹‚.

(b) Use the factorization to solve Ax “ b with

b “

¨

˚˝

8714´7

˛

‹‹‚.


Solution. (a) We see that the sequence of operations with the multipliers m21 “ 2, m31 “ 3,m41 “ ´1, m32 “ 4, m42 “ ´3, and m43 “ 0,

pE2 ´ 2E1q Ñ pE2q,pE3 ´ 3E1q Ñ pE3q,

pE4 ´ p´1qE1q Ñ pE4q,pE3 ´ 4E2q Ñ pE3q,

pE4 ´ p´3qE2q Ñ pE4q,

converts the matrix A to the upper triangular one:

U “

¨

˚˝

1 1 0 30 ´1 ´1 ´50 0 3 130 0 0 ´13

˛

‹‹‚.

The multipliers mij and the upper triangular matrix produce the factorization:

A “

¨

˚˝

1 1 0 32 1 ´1 13 ´1 ´1 2

´1 2 3 ´1

˛

‹‹‚“

¨

˚˝

1 0 0 02 1 0 03 4 1 0

´1 ´3 0 1

˛

‹‹‚

¨

˚˝

1 1 0 30 ´1 ´1 ´50 0 3 130 0 0 ´13

˛

‹‹‚“ LU.

(b) To solve the systemAx “ LUx “ b,

we first introduce the substitution y “ Ux. Then b “ LpUxq “ Ly. That is,

Ly “

¨

˚˝

1 0 0 02 1 0 03 4 1 0

´1 ´3 0 1

˛

‹‹‚

¨

˚˝

y1y2y3y4

˛

‹‹‚“ b “

¨

˚˝

8714´7

˛

‹‹‚.

This system is solved for y by a simple forward substitution process:

y1 “ 8;

y2 “ 7 ´ 2y1 “ ´9;

y3 “ 14 ´ 3y1 ´ 4y2 “ 26;

y4 “ ´7 ` y1 ` 3y2 “ ´26.

We then solve Ux “ y for x, the solution of the original system; that is,

¨

˚˝

1 1 0 30 ´1 ´1 ´50 0 3 130 0 0 ´13

˛

‹‹‚

¨

˚˝

x1x2x3x4

˛

‹‹‚“

¨

˚˝

8´926

´26

˛

‹‹‚.

Using back substitution we obtain x4 “ 2, x3 “ 0, x2 “ ´1, and x1 “ 3.

The factorization used in the example above is called Doolittle’s method and requires that 1’s beon the diagonal of L, which results in the factorization described above. Later on, we consider


Crout’s method, a factorization which requires that 1’s be on the diagonal elements of U , andCholesky’s method, which requires that ìi “ uii for each i.

Once the matrix factorization is complete, the solution to a linear system of the form Ax “LUx “ b is found by first letting y “ Ux and solving Ly “ b for y. Since L is lower triangular,we have

y1 “ b1`11

and yi “ 1

ìi

˜bi ´

i´1ÿ

j“1

ìjyj

¸for i “ 2, 3, ¨ ¨ ¨ , n.

After y is found by this forward substitution process, the upper triangular system Ux “ y issolved for x by back substitution using the equations

xn “ ynunn

and xi “ 1

uii

˜yi ´

nÿ

j“i`1

uijxj

¸for i “ n ´ 1, n ´ 1, ¨ ¨ ¨ , 1.

Permutation matrices

Previously, we assumed that Ax “ b can be solved using Gaussian elimination without rowinterchanges. For a practical standpoint, this factorization is useful only when row interchangesare not required to control the round-o↵ error resulting from the use of finite-digit arithmetic.We begin the discussion with the introduction of a class of matrices that are used to rearrange,or permute, rows of a given matrix.

An n ˆ n permutation matrix P “ ppijq is a matrix obtained by rearranging the rows of theidentity matrix In of order n. This gives a matrix with precisely one nonzero entry in each rowand in each column, and each nonzero entry is a 1.

Example 6.5.3. The matrix

P “¨

˝1 0 00 0 10 1 0

˛

‚

is a 3 ˆ 3 permutation matrix. For any 3 ˆ 3 matrix A, multiplying on the left by P has thee↵ect of interchanging the second and third rows of A:

PA “¨

˝1 0 00 0 10 1 0

˛

‚

¨

˝a11 a12 a13a21 a22 a23a31 a32 a33

˛

‚“¨

˝a11 a12 a13a31 a32 a33a21 a22 a23

˛

‚.

Similarly, multiplying A on the right by P interchanges the second and third columns of A.

Two useful properties of permutation matrices relate to Gaussian elimination, the first of whichis illustrated in the previous example. Suppose k1, ¨ ¨ ¨ , kn is a permutation of the integers1, ¨ ¨ ¨ , n and the permutation matrix P “ ppijq is defined by

pij “"

1, if j “ ki,0, otherwise.

Then, (i) PA permutes the rows of A; that is,

PA “

¨

˚˚

ak11 ak12 ¨ ¨ ¨ ak1nak21 ak22 ¨ ¨ ¨ ak2n...

.... . .

...akn1 akn2 ¨ ¨ ¨ aknn

˛

‹‹‹‚,


and P´1 exists and P´1 “ P T . For any nonsingular matrix A, the linear system Ax “ bcan be solved by Gaussian elimination, with the possibility of row interchanges. If we knewthe row interchanges that were required to solve the system by Gaussian elimination, we couldarrange the original equations in an order that would ensure that now row interchanges areneeded. Hence, there is a rearrangement of the equations in the system that permits Gaussianelimination to proceed without row interchanges. This implies that for any nonsingular matrixA, a permutation matrix P exists for which the system

PAx “ Pb

can be solved without row interchanges. As a consequence, the matrix PA can be factored intoPA “ LU , where L is lower triangular and U is upper triangular. Because P´1 “ P T , thisproduces the factorization

A “ P´1LU “ P TLU.

The matrix U is still upper triangular, but P TL is not lower triangular unless P “ In.

Example 6.5.4. Determine a factorization in the form A “ P TLU for the matrix

A “

¨

˚˝

0 0 ´1 11 1 ´1 2

´1 ´1 2 01 2 0 2

˛

‹‹‚.

Solution. The matrix A cannot have an LU factorization since a11 “ 0. However, using the rowinterchange pE1q Ø pE2q, followed by pE3 ´ p´1qE1q Ñ pE3q and pE4 ´ E1q Ñ pE4q produces

¨

˚˝

1 1 ´1 20 0 ´1 10 0 1 20 1 1 0

˛

‹‹‚.

Then the row interchange pE2q Ø pE4q, followed by pE4 ´ p´1qE3q Ñ pE4q, give the matrix

U “

¨

˚˝

1 1 ´1 20 1 1 00 0 1 20 0 0 3

˛

‹‹‚.

The permutation matrix associated with the row interchanges pE1q Ø pE2q and pE2q Ø pE4q is

P “

¨

˚˝

0 1 0 00 0 0 10 0 1 01 0 0 0

˛

‹‹‚

and

PA “

¨

˚˝

1 1 ´1 21 2 0 2

´1 ´1 2 00 0 ´1 1

˛

‹‹‚.

Gaussian elimination is performed on PA using the same operations as on A, except withoutthe row interchanges. That is, pE2 ´ E1q Ñ pE2q, pE3 ´ p´1qE1q Ñ pE3q, followed by pE4 ´p´1qE3q Ñ pE4q. The nonzero multipliers for PA are consequently,

m21 “ 1, m31 “ ´1, and m43 “ ´1,

6.6. SPECIAL TYPES OF MATRICES 117

and the LU factorization of PA is

PA “

¨

˚˝

1 0 0 01 1 0 0

´1 0 1 00 0 ´1 1

˛

‹‹‚

¨

˚˝

1 1 ´1 20 1 1 00 0 1 20 0 0 3

˛

‹‹‚“ LU.

Multiplying by P´1 “ P T produces the factorization

A “ P´1LU “ P TLU “

¨

˚˝

0 0 ´1 11 0 0 0

´1 0 1 01 1 0 0

˛

‹‹‚

¨

˚˝

1 1 ´1 20 1 1 00 0 1 20 0 0 3

˛

‹‹‚.

6.6 Special Types of Matrices

We now turn attention to two classes of matrices for which Gaussian elimination can be per-formed e↵ectively without row interchanges.

Diagonally Dominant Matrices

The first class is described in the following definition.

Definition 6.6.1. The n ˆ n matrix A is said to be diagonally dominant when

|aii| •nÿ

j“1, j‰i

|aij | hold for each i “ 1, 2, ¨ ¨ ¨ , n.

A diagonally dominant matrix is said to be strictly diagonally dominant when the inequalityabove is strict for each n, that is,

|aii| °nÿ

j“1, j‰i

|aij | hold for each i “ 1, 2, ¨ ¨ ¨ , n.

Example 6.6.2. Consider the matrices

A “¨

˝7 2 03 5 ´10 5 ´6

˛

‚ and B “¨

˝6 4 ´34 ´2 0

´3 0 1

˛

‚.

The non-symmetric matrix A is strictly diagonally dominant because

|7| ° |2| ` |0| , |5| ° |3| ` |´1| , and |´6| ° |0| ` |5| .

The symmetric matrix B is not strictly diagonally dominant because, for example, in the firstrow the absolute value of the diagonal element is |6| † |4| ` |´3| “ 7. It is interesting to notethat AT is not strictly diagonally dominant, because the middle row of AT is p2 5 5q, nor, ofcourse, is BT because BT “ B.

We have the following result for strictly diagonally dominant matrices.

Theorem 6.6.3. A strictly diagonally dominant matrix A is nonsingular. Moreover, in thiscase, Gaussian elimination can be performed on any linear system of the form Ax “ b to obtainits unique solution without row or column interchanges.


Positive Definite Matrices

The next special class of matrices is called positive definite.

Definition 6.6.4 (Positive definite). A square matrix A of order n is called positive definite ifxTAx ° 0 for every n-dimensional vector x ‰ 0. To be precise, it specifies that the 1ˆ 1 matrixgenerated by the operation xTAx has a positive value for its only entry since the operation isperformed as follows:

xTAx “ px1 x2 ¨ ¨ ¨ xnq

¨

˚˚

a11 a12 ¨ ¨ ¨ a1na21 a22 ¨ ¨ ¨ a2n...

.... . .

...an1 an2 ¨ ¨ ¨ ann

˛

‹‹‹‚

¨

˚˚

x1x2...xn

˛

‹‹‹‚

“ px1 x2 ¨ ¨ ¨ xnq

¨

˚˚

∞nj“1 a1jxj∞nj“1 a2jxj

...∞nj“1 anjxj

˛

‹‹‹‚“˜

nÿ

i“1

nÿ

j“1

aijxixj

¸.

Example 6.6.5. The matrix

A “¨

˝2 ´1 0

´1 2 ´10 ´1 2

˛

‚

is positive definite. Suppose x is any three-dimensional column vector. Then

xTAx “ px1 x2 x3q¨

˝2 ´1 0

´1 2 ´10 ´1 2

˛

‚

¨

˝x1x2x3

˛

‚

“ px1 x2 x3q¨

˝2x1 ´ x2

´x1 ` 2x2 ´ x3´x2 ` 2x3

˛

‚

“ 2x21 ´ 2x1x2 ` 2x22 ´ 2x2x3 ` 2x23.

Rearranging the terms gives

xTAx “ x21 ` px21 ´ 2x1x2 ` x22q ` px22 ´ 2x2x3 ` x23q ` x23

“ x21 ` px1 ´ x2q2 ` px2 ´ x3q2 ` x23,

which implies that xTAx ° 0 unless x1 “ x2 “ x3 “ 0. Moreover, it is symmetric.

It should be clear from the example that using the definition to determine if a matrix is positivedefinite can be di�cult. The next result provides some necessary conditions that can be usedto eliminate certain matrices from consideration.

Theorem 6.6.6. If A is an n ˆ n positive definite matrix, then

(i) A has an inverse;

(ii) aii ° 0 for each i “ 1, 2, ¨ ¨ ¨ , n.(iii) max1§k, j§n |akj | § max1§i§n |aii|;

(iv) paijq2 † aiiajj for each i ‰ j.


The theorem above provides some important conditions that must be true of positive definitematrices, it does not ensure that a matrix satisfying these conditions is positive definite. Thefollowing notion will be used to provide a necessary and su�cient condition.

Definition 6.6.7. A leading principal submatrix of a matrix A is a matrix of the form

Ak “

¨

˚˚

a11 a12 ¨ ¨ ¨ a1ka21 a22 ¨ ¨ ¨ a2k...

.... . .

...ak1 ak2 ¨ ¨ ¨ akk

˛

‹‹‹‚

for some 1 § k § n.

We have the following result.

Theorem 6.6.8. A symmetric matrix A is positive definite if and only if each of its leadingprincipal submatrices has a positive determinant.

Example 6.6.9. In Example 6.6.5, we used the definition to show that the symmetric matrix

A “¨

˝2 ´1 0

´1 2 ´10 ´1 2

˛

‚

is positive definite. Confirm this using Theorem 6.6.8.

Solution. Note that detpA1q “ 2 ° 0,

detpA2q “ det

ˆ2 ´1

´1 2

˙“ 4 ´ 1 “ 3 ° 0,

and

detpA3q “ det

¨

˝2 ´1 0

´1 2 ´10 ´1 2

˛

‚“ 2 det

ˆ2 ´1

´1 2

˙´ p´1qdet

ˆ ´1 ´10 2

˙

“ 2p4 ´ 1q ` p´2 ` 0q “ 4 ° 0.

It is in agreement with Theorem 6.6.8.

Theorem 6.6.10. A symmetric matrix A is positive definite if and only if Gaussian eliminationwithout row interchanges can be performed on the linear system Ax “ b with all pivot elementspositive. Moreover, in this case, the computations are stable with respect to the growth ofround-o↵ errors.

Some interesting facts that are uncovered in the constructing the proof of Theorem 6.6.10 arepresented in the following corollaries.

Corollary 6.6.11. The matrix A is positive definite if and only if A can be factored in the formA “ LDLT , where L is lower triangular with 1’s on its diagonal and D is a diagonal matrixwith positive diagonal entries.

Corollary 6.6.12 (Cholesky factorization). The matrix A is positive definite if and only if Acan be factored in the form A “ LLT , where L is lower triangular with nonzero diagonal entries.


Note that the matrix L in two corollaries are not equal. Moreover, we have the following resultwhen A is symmetric (may not necessarily be positive definite).

Corollary 6.6.13. Let A be a symmetric n ˆ n matrix for which Gaussian elimination can beapplied without row interchanges. Then A can be factored into the form of A “ LDLT , where

L is lower triangular with 1’s on its diagonal and D is the diagonal matrix with ap1q11 , ¨ ¨ ¨ , ap1q

nn onits diagonal.

Example 6.6.14. Determine the LDLT factorization of the positive definite matrix:

A “¨

˝4 ´1 1

´1 4.25 2.751 2.75 3.5

˛

‚.

Solution. The LDLT factorization has 1’s on the diagonal of the lower triangular matrix L sowe need to have

A “¨

˝1 0 0`21 1 0`31 `32 1

˛

‚

¨

˝d1 0 00 d2 00 0 d3

˛

‚

¨

˝1 `21 `310 1 `320 0 1

˛

‚

“¨

˝d1 d1`21 d1`31

d1`21 d2 ` d1`221 d2`32 ` d1`21`31d1`31 d2`32 ` d1`21`31 d1`231 ` d2`232 ` d3

˛

‚.

Thus, we have

a11 : 4 “ d1 ùñ d1 “ 4, a21 : ´1 “ d1`21 ùñ `21 “ ´0.25,

a31 : 1 “ d1`31 ùñ `31 “ 0.25, a22 : 4.25 “ d2 ` d1`221 ùñ d2 “ 4,

a32 : 2.75 “ d2`32 ` d1`21`31 ùñ `32 “ 0.75, a33 : 3.5 “ d1`231 ` d2`

232 ` d3 ùñ d3 “ 1.

Therefore, we have

A “¨

˝1 0 0

´0.25 1 00.25 0.75 1

˛

‚

¨

˝4 0 00 4 00 0 1

˛

‚

¨

˝1 ´0.25 0.250 1 0.750 0 1

˛

‚.

Example 6.6.15. Determine the Cholesky LLT factorization of the positive definite matrix:

A “¨

˝4 ´1 1

´1 4.25 2.751 2.75 3.5

˛

‚.

Solution. The LLT factorization does not necessarily has 1’s on the diagonal of the lower trian-gular matrix L so we need to have

A “¨

˝`11 0 0`21 `22 0`31 `32 `33

˛

‚

¨

˝`11 `21 `310 `22 `320 0 `33

˛

‚

“¨

˝`211 `11`21 `11`31

`11`21 `221 ` `222 `22`32 ` `21`31`11`31 `22`32 ` `21`31 `231 ` `232 ` `233

˛

‚.


Thus, we have

a11 : 4 “ `211 ùñ `11 “ 2, a21 : ´1 “ `11`21 ùñ `21 “ ´0.5,

a31 : 1 “ `11`31 ùñ `31 “ 0.5, a22 : 4.25 “ `222 ` `221 ùñ `22 “ 2,

a32 : 2.75 “ `22`32 ` `21`31 ùñ `32 “ 1.5, a33 : 3.5 “ `231 ` `232 ` `233 ùñ `33 “ 1

and

A “¨

˝2 0 0

´0.5 2 00.5 1.5 1

˛

‚

¨

˝2 ´0.5 0.50 2 1.50 0 1

˛

‚.

Tridiagonal matrices

The last class of matrices considered are tridiagonal matrices having the form

A “

¨

˚˚˚˚˚˝

a11 a12 0 ¨ ¨ ¨ ¨ ¨ ¨ 0

a21 a22 a23. . .

...

0 a32 a33 a34. . .

......

. . .. . .

. . .. . . 0

.... . .

. . .. . . an´1,n

0 ¨ ¨ ¨ ¨ ¨ ¨ 0 an,n´1 ann

˛

‹‹‹‹‹‹‹‹‹‹‚

.

Tridiagonal matrices are also considered in connection with the study of piecewise linear approx-imations to boundary-value problems. The factorizations can be simplified considerably in thecase of tridiagonal matrices because a large number of zeros appear in these matrices in regularpatterns.

To illustrate the situation, suppose a tridiagonal matrix A can be factored into the triangularmatrices L and U . Then A has at most 3n ´ 2 nonzero entries. Then there are only 3n ´ 2conditions to be applied to determine the entries of L and U , provided that the zero entries ofA are also obtained.

Suppose that the matrices L and U also have tridiagonal form, that is,

L “

¨

˚˚˚˚˚˝

`11 0 0 ¨ ¨ ¨ ¨ ¨ ¨ 0

`21 `22 0. . .

...

0 `32 `33 0. . .

......

. . .. . .

. . .. . . 0

.... . .

. . .. . . 0

0 ¨ ¨ ¨ ¨ ¨ ¨ 0 `n,n´1 `nn

˛

‹‹‹‹‹‹‹‹‹‹‚

and U “

¨

˚˚˚˚˚˝

1 u12 0 ¨ ¨ ¨ ¨ ¨ ¨ 0

0 1 u23. . .

...

0 0 1 u34. . .

......

. . .. . .

. . .. . . 0

.... . .

. . .. . . un´1,n

0 ¨ ¨ ¨ ¨ ¨ ¨ 0 0 1

˛

‹‹‹‹‹‹‹‹‹‹‚

.

There are 2n ´ 1 undetermined entries of L and n ´ 1 undetermined entries of U , which totals3n-2, the number of possible nonzero entries of A. The 0 entries of A are obtained automatically.

The multiplication involved with A “ LU gives, in addition to the 0 entries,

a11 “ `11;

ai,i´1 “ ì,i´1, for i “ 2, 3, ¨ ¨ ¨ , n;aii “ ì,i´1ui´1,i ` ìi, for i “ 2, 3, ¨ ¨ ¨ , n;

ai,i`1 “ ìiui,i`1, for i “ 1, 2, ¨ ¨ ¨ , n ´ 1.

(6.13)


This is called the Crout factorization of the tridiagonal matrix.

Example 6.6.16. Determine the Crout factorization of the (symmetric) tridiagonal matrix

A “

¨

˚˝

2 ´1 0 0´1 2 ´1 00 ´1 2 ´10 0 ´1 2

˛

‹‹‚.

Use this factorization to solve the linear system Ax “ b with

b “

¨

˚˝

1001

˛

‹‹‚.

Solution. The Crout factorization of A has the form

A “

¨

˚˝

a11 a12 0 0a21 a22 a23 00 a32 a33 a340 0 a43 a44

˛

‹‹‚“

¨

˚˝

`11 0 0 0`21 `22 0 00 `32 `33 00 0 `43 `44

˛

‹‹‚

¨

˚˝

1 u12 0 00 1 u23 00 0 1 u340 0 0 1

˛

‹‹‚

“

¨

˚˝

`11 `11u12 0 0`21 `22 ` `21u12 `22u23 00 `32 `33 ` `32u23 `33u340 0 `43 `44 ` `43u34

˛

‹‹‚

Thus, we have

a11 : 2 “ `11 ùñ `11 “ 2, a12 : ´1 “ `11u12 ùñ u12 “ ´1

2,

a21 : ´1 “ `21 ùñ `21 “ ´1, a22 : 2 “ `22 ` `21u12 ùñ `22 “ ´3

2,

a23 : ´1 “ `22u23 ùñ u23 “ ´2

3, a32 : ´1 “ `32 ùñ `32 “ ´1,

a33 : 2 “ `33 ` `32u23 ùñ `33 “ 4

3, a34 : ´1 “ `33u34 ùñ u34 “ ´3

4,

a43 : ´1 “ `43 ùñ `43 “ ´1, a44 : 2 “ `44 ` `43u34 ùñ `44 “ ´5

4.

This gives the Crout factorization

A “

¨

˚˝

2 0 0 0´1 3

2 0 00 ´1 4

3 00 0 ´1 5

4

˛

‹‹‚

¨

˚˝

1 ´12 0 0

0 1 ´23 0

0 0 1 ´34

0 0 0 1

˛

‹‹‚“ LU.

Solving the system

Lz “

¨

˚˝

2 0 0 0´1 3

2 0 00 ´1 4

3 00 0 ´1 5

4

˛

‹‹‚

¨

˚˝

z1z2z3z4

˛

‹‹‚“

¨

˚˝

1001

˛

‹‹‚ gives

¨

˚˝

z1z2z3z4

˛

‹‹‚“

¨

˚˝

1213141

˛

‹‹‚,

6.7. CONDITION OF A LINEAR SYSTEM 123

and then solving

Ux “

¨

˚˝

1 ´12 0 0

0 1 ´23 0

0 0 1 ´34

0 0 0 1

˛

‹‹‚

¨

˚˝

x1x2x3x4

˛

‹‹‚“

¨

˚˝

1213141

˛

‹‹‚ gives

¨

˚˝

x1x2x3x4

˛

‹‹‚“

¨

˚˝

1111

˛

‹‹‚.

6.7 Condition of a linear system

Recall that Rn denote the set of all n-dimensional column vectors with real-number components.To define a distance in Rn we use the notion of a norm, which is the generalization of the absoluteon R, the set of real numbers.

Definition 6.7.1. A vector norm on Rn is a function }¨} from Rn into R with the followingproperties:

(i) }x} • 0 for all x P Rn,

(ii) }x} “ 0 if and only if x “ 0,

(iii) }↵x} “ |↵| }x} for all ↵ P R and x P Rn,

(iv) }x ` y} § }x} ` }y} for all x, y P Rn.

Definition 6.7.2. The `2 and `8 norms for vector x “ px1, x2, ¨ ¨ ¨ , xnqT P Rn are defined by

}x}2 “˜

nÿ

i“1

x2i

¸1{2and }x}8 “ max

1§i§n|xi| .

Note that each of these norms reduces to the absolute value in the case n “ 1. The `2 normis called the Euclidean norm of the vector because it represents the usual notion of distancefrom the origin. For example, the `2 norm of the vector x “ px1, x2, x3qT gives the length of thestraight line joining the points p0, 0, 0q and px1, x2, x3q.Example 6.7.3. Determine the `2 norm and the `8 norm of the vector x “ p´1, 1,´2qT .

Solution. The vector x “ p´1, 1,´2qT in R3 has norms

}x}2 “a

p´1q2 ` 12 ` p´2q2 “?6

and}x}8 “ maxt|´1| , |1| , |´2|u “ 2.

It is easy to show that the properties in Definition 6.7.1 hold for the `8 norm because they followfrom similar results for absolute values. The only property that requires much demonstration is(iv), and in this case if x “ px1, x2, ¨ ¨ ¨ , xnqT and y “ py1, y2, ¨ ¨ ¨ , ynqT , then

}x ` y}8 “ max1§i§n

|xi ` yi| § max1§i§n

p|xi| ` |yiq| max1§i§n

|xi| ` max1§i§n

|yi| “ }x}8 ` }y}8 .

The first three conditions in Definition 6.7.1 also are easy to show for the `2 norm. But to show(iv), we need a famous inequality.


Theorem 6.7.4 (Cauchy-Schwarz Inequality). For each x “ px1, x2, ¨ ¨ ¨ , xnqT and y “ py1, y2, ¨ ¨ ¨ , ynqTin Rn,

xT y “nÿ

i“1

xiyi §˜

nÿ

i“1

x2i

¸1{2 ˜nÿ

i“1

y2i

¸1{2“ }x}2 ¨ }y}2 .

With the Cauchy-Schwarz Inequality, we see that for x, y P Rn, we have

}x ` y}22 “nÿ

i“1

pxi ` yiq2 “nÿ

i“1

x2i ` 2nÿ

i“1

xiyi `nÿ

i“1

y2i § }x}22 ` 2 }x}2 ¨ }y}2 ` }y}22 ,

which gives norm property (iv):

}x ` y}2 § p}x}22 ` 2 }x}2 ¨ }y}2 ` }y}22q1{2 “ }x}2 ` }y}2 .

Theorem 6.7.5. For each x P Rn, we have

}x}8 § }x}2 § ?n }x}8 .

Proof. Let xj be a coordinate of x such that }x}8 “ max1§i§n |xi| “ |xj |. Then,

}x}28 “ |xj |2 §nÿ

i“1

x2i “ }x}22 ùñ }x}8 § }x}2 .

On the other hand,

}x}22 “nÿ

i“1

x2i §nÿ

i“1

x2j “ nx2j “ n }x}28 ùñ }x}2 § ?n }x}8 .

Remark: It can be shown that all norms on Rn are equivalent with respect to convergence;that is, if }¨} and }¨}1 are any two norms on Rn and txpkqu8

k“1 has the limit x with respect to }¨},then the sequence also has the same limit with respect to another norm.

Matrix norms and distances

We need methods for determining the distance between nˆ n matrices. This again requires theuse of norm. We first define the norm of matrix.

Definition 6.7.6 (Matrix norm). A matrix norm on the set of all n ˆ n matrices is a real-valued function, }¨}, defined on this set, satisfying for all n ˆ n matrices A and B and all realnumbers ↵:

(i) }A} • 0;

(ii) }A} “ 0 if and only if A “ O, the matrix with all 0 entries;

(iii) }↵A} “ |↵| }A};

(iv) }A ` B} § }A} ` }B}:

(v) }AB} § }A} ¨ }B}.


Although matrix norms can be obtained in various ways, the norms considered most frequentlyare those that are natrual consequences of the vector norms `2 and `8. These norms are definedusing the following results.

Theorem 6.7.7. If }¨}v is a vector norm on Rn, we define }¨}M as follows:

}A}M “ max}x}v“1

}Ax}v .

Then, }¨}M is a matrix norm.

Matrix norms defined by vector norms are called the natural, or induced, matrix normassociated with the vector norm. In this text, all matrix norms will be assumed to be naturalmatrix norms unless specified otherwise. For any z ‰ 0, the vector x “ z{ }z}v is a unit vectorwith respect to the vector norm }¨}v. Hence,

max}x}v“1

}Ax}v “ maxz‰0

››››Aˆ

z

}z}v

˙››››v

“ maxz‰0

}Az}v}z}v

,

and we can alternatively write the matrix norm }¨}M as

}A}M “ maxz‰0

}Az}v}z}v

.

The following result follows from this representation of }A}M .

Theorem 6.7.8. For any vector z ‰ 0, matrix A, and any natural matrix norm }¨}M inducedby the vector norm }¨}v, we have

}Az}v § }A}M ¨ }z}v .

The measure given to a matrix under a natural norm describes how the matrix stretches unitvectors relative to that norm. The maximum stretch is the norm of the matrix. The matrixnorms we will consider have the forms

}A}8 “ max}x}8“1

}Ax}8 , the `8 norm,

and}A}2 “ max

}x}2“1}Ax}2 , the `2 norm.

The `8 norm of a matrix can be easily computed from the entries of the matrix.

Theorem 6.7.9. If A “ paijq is an n ˆ n matrix, then

}A}8 “ max1§i§n

nÿ

j“1

|aij | .

Example 6.7.10. Determine }A}8 for the matrix

A “¨

˝1 2 ´10 3 ´15 ´1 1

˛

‚.


Solution. We have

3ÿ

j“1

|a1j | “ |1| ` |2| ` |´1| “ 4,3ÿ

j“1

|a2j | “ |0| ` |3| ` |´1| “ 4,

and3ÿ

j“1

|a3j | “ |5| ` |´1| ` |1| “ 7.

It implies that }A}8 “ maxt4, 4, 7u “ 7.

Perturbating a linear system

Consider the following linear system

¨

˚˝

10 7 8 77 5 6 58 6 10 97 5 9 10

˛

‹‹‚

¨

˚˝

x1x2x3x4

˛

‹‹‚“

¨

˚˝

32233331

˛

‹‹‚ with solution

¨

˚˝

1111

˛

‹‹‚

and consider the perturbed system, where the right-hand side has been very slightly modified,the matrix staying unchanged,

¨

˚˝

10 7 8 77 5 6 58 6 10 97 5 9 10

˛

‹‹‚

¨

˚˝

x1 ` �x1x2 ` �x2x3 ` �x3x4 ` �x4

˛

‹‹‚“

¨

˚˝

32.122.933.130.9

˛

‹‹‚ with solution

¨

˚˝

9.2´12.6

4.5´1.1

˛

‹‹‚.

In order words, a relative error of the order of 1{200 in the data (here, the components of theright-hand side) produces a relative error of the order of 10{1 in the result (the solution of thelinear system), which represents an amplification of the relative errors of the order of 2000.

Let us now analyze this kind of phenomenon. Assume that A is invertible, and a comparisonneeds to be made between the two exact solutions x and x ` �x of the systems:

Ax “ b,

Apx ` �xq “ b ` �b.

Let the same symbol }¨} denote any vector norm and its associated matrix norm. From theequalities b “ Ax and �x “ A´1�b, using Theorem 6.7.8, we conclude

}�x} §››A´1

›› }�b} and }b} § }A} }u} ,

so that the relative error in the result, measured by the ratio }�u} { }u}, is bounded in terms ofthe relative error }�b} { }b} in the datum b, as follows:

}�x}}x} § }A} ¨

››A´1›› }�b}

}b} . (6.14)

That is, the relative error in the result is bounded by the relative error in the data, multiplied bythe number }A}

››A´1››. In order words, for a given relative error in the data, the relative error in

the corresponding result may be larger by a factor proportional to this number; in actual fact,it can be shown that this number is optimal, and the inequalities above being the best possible.These considerations lead to the following definition.


Definition 6.7.11 (Condition number). Let }¨} be a matrix norm associated with a vector normand let A be invertible. The number

condpAq :“ }A}››A´1

››

is called the condition number of the matrix A, relative to the given matrix norm.

The inequality (6.14) shows that the number condpAq measures the sensitivity of the solution ofthe linear system to variations in the data; a feature which is referred to as the condition of thelinear system in question. The preceding, therefore, gives sense to a statement such as a linearsystem is well-conditioned or ill-conditioned, according as the condition number of its matrix issmall or large.

Example 6.7.12. Consider the following matrix

A “

¨

˚˝

10 7 8 77 5 6 58 6 10 97 5 9 10

˛

‹‹‚.

Calculate the condition number of A with respect to the `8 (vector) norm.

Solution. Recall Theorem 6.7.9 that

}A}8 “ max1§i§n

nÿ

j“1

|aij | .

Then, we have

4ÿ

j“1

|a1j | “ 10 ` 7 ` 8 ` 7 “ 32,4ÿ

j“1

|a2j | “ 7 ` 5 ` 6 ` 5 “ 23,

4ÿ

j“1

|a3j | “ 8 ` 6 ` 10 ` 9 “ 33,4ÿ

j“1

|a4j | “ 7 ` 5 ` 9 ` 10 “ 31.

It implies that }A}8 “ 33. On the other hand, we have

A´1 “

¨

˚˝

25 ´41 10 ´6´41 68 ´17 1010 ´17 5 ´3´6 10 ´3 2

˛

‹‹‚ and››A´1

››8 “ 136.

Therefore, the condition number of A with respect to `8-norm is condpAq “ 136 ¨ 33 “ 4488.Next, we recall the linear systems at the beginning of the section. Let

x “

¨

˚˝

1111

˛

‹‹‚, �x “

¨

˚˝

8.2´13.6

3.5´2.1

˛

‹‹‚, b “

¨

˚˝

32233331

˛

‹‹‚, and �b “

¨

˚˝

0.1´0.10.1

´0.1

˛

‹‹‚.

We compute the ratios:

}�x}8}x}8

“ 13.6,}�b}8}b}8

“ 0.1

33ùñ condpAq}�b}8

}b}8“ 136 ¨ 0.1 “ 13.6


and we have}�x}8}x}8

“ condpAq}�b}8}b}8

.

In this case, the inequality (6.14) becomes an equality when the associated vector norm is`8-norm.

Direct Methods for Solving Linear Systemssmpun/MATH417/Chapter-6.pdf · 94 CHAPTER 6. DIRECT...

Documents

Transcript of Direct Methods for Solving Linear Systemssmpun/MATH417/Chapter-6.pdf · 94 CHAPTER 6. DIRECT...