Ch 1. Applied Linear Algebra - Yonsei University · 2014-12-30 · Ch 1. Applied Linear Algebra 2...

Ch 1. Applied Linear Algebra

Yoon Mo Jung

Computational Science and Engineering

Yonsei University

Ch 1. Applied Linear Algebra 2

§0 Prelude: Structure of Computational Science andEngineering

• Overview of Computational Science and Engineering

laws

rules

constraints

−−−−→modeling

Mathematical

problem:

equations

−−−−−−−−−−−−→Numerical procedures

Linear

equations

Ax = bin engineering and physics

Remark 1. Designing and validating numerical procedures is called “Nu-

merical Analysis”.

2. How to solve Ax = b is called Numerical Linear Algebra, which is the

heart of the Scientific Computing. Strang said “Its importance is

now recognized.”

• Four Simplifications

1. Nonlinear becomes linear.

ex) Bending of beam

u′′

(1 + (u′)2)3/2≈ u′′ if u′ is small.

2. Continuous becomes discrete.

ex) f ′(x) = limh→0

f(x + h)− f(x)

h≈

f(xn+1)− f(xn)

∆x,

if ∆x is small.

3. Multidimensional becomes one-dimensional.

ex) ut = uxx (heat equation)

Computational Science & Engineering (CSE) Yoon Mo Jung


Let u(t, x) = T (t)X(x): separation of variables

4. Variable coefficients becomes constant coefficients.

ex) inhomogeneous heat equation: ut = (c(x)ux)x

with conductivity c(x).

If c(x) ≈ c, one may use Fourier transform or FFT.



§1 Applied Linear Algebra

§1.1 Four Special Matrices

• Two important problems

1. Solving linear systems: Ax = b

x: cause or input, b: result or output.

ex) x: displacements, pressures, voltages, concentrations, ......

2. Eigenvalue problem: Ax = λx

eigen means prime.

• Four special matrices: Kn, Cn, Tn, Bn

K2 =

[2 −1−1 2

],K3 =

2 −1 0

−1 2 −10 −1 2

,K4 =

2 −1 0 0

−1 2 −1 0

0 −1 2 −10 0 −1 2

Remark (Good) Properties

1. Symmetric (Kij = Kji or K = KT ).

2. Sparse (lots of zeros if n≫ 1).

3. Tridiagonal, banded.

4. Constant diagonals: called Toeplitz matrix.

- Something is not changing when we move in space or time. Shift-

invariant or time-invariant.

5. Invertible (∃K−1 s.t. KK−1 = K−1K = I).

-K−1 also symmetric but full.

- Important: We don’t want or need K−1 to find u = K−1f . All we

compute is the solution x.



6. Positive definite (xTAx > 0 if x = 0).

Circulant matrix C4 =

2 −1 0 −1−1 2 −1 0

0 −1 2 −1−1 0 −1 2

Remark 1. Singular, not invertible (C4[1, 1, 1, 1]

T = 0).

2. Positive semidefinite (xTAx ≥ 0).

T2 =

[1 −1−1 2

], T3 =

1 −1 0

−1 2 −10 −1 2

Gaussian elimination:

T =

1 −1 0

−1 2 −10 −1 2

−−−→Step 1

1 −1 0

0 1 −10 −1 2

−−−→Step 2

1 −1 0

0 1 −10 0 1

= U

U−1 =

1 −1 0

0 1 −10 0 1

−1

=

1 1 1

0 1 1

0 0 1

: The inverse of “difference matrix” is a “sum matrix”. 1 −1 0

0 1 −10 0 1

u1

u2

u3

=

u1 − u2

u2 − u3

u3 − 0

1 1 1

0 1 1

0 0 1

u1 − u2

u2 − u3

u3 − 0

=

u1

u2

u3



Remark: The inverse of triangular matrix is also triangular.

B2 =

[1 −1−1 1

], B3 =

1 −1 0

−1 2 −10 −1 1

It is positive semidefinite.

B =

1 −1 0

−1 2 −10 −1 1

−−−→Step 1

1 −1 0

0 1 −10 −1 1

−−−→Step 2

1 −1 0

0 1 −10 0 0

= U

1 −1 0

0 1 −10 0 0

1

1

1

=

0

0

0

and

1 −1 0

−1 2 −10 −1 1

1

1

1

=

0

0

0

• Summary

1. Kn and Tn are invertible and positive definite.

2. Cn and Bn are singular and positive semidefinite.

The nullspace(kernel) is the constant vector u = [c, c, · · · , c].

Remark: Bu = f is solvable when f is perpendicular to e = [1, 1, · · · , 1].

f = Bu =

1 −1 0

−1 2 −10 −1 1

u1

u2

u3

u4

=

u1 − u2

−u12u2 − u3

−u2 + 2u3 − u4

−u3 + u4

f1 + f2 + f3 + f4 = fTe = f · e = ⟨f, e⟩ = 0.



Figure 1: Finite Differences

§1.2 Differences, Derivatives, Boundary Conditions

Observation: -1, 2, -1 produces a second difference.

Kn, Cn, Tn, Bn are all involved in approximating the equation

−d2u

dx2= f(x)

with boundary conditions at x = 0 and x = 1.

Part I: Finite Differences

-want to approximate dudx.

dudx≈ △u△x if △ x is small.

ex) Choose test function u(x) = x2.

Forward difference △+f(x) = u(x+h)−u(x)h

ex) (x+h)2−x2

h= 2x + h

Backward difference △−f(x) = u(x)−u(x−h)h

ex) x2−(x−h)2h

= 2x− h

Centered difference △0f(x) = u(x+h)−u(x−h)2h

ex) (x+h)2−(x−h)2h

= 2x

Taylor series: series in h



u(x + h) = u(x) + hu′(x) + h2/2u′′(x) + h3/3!u′′′(x) + · · ·u(x− h) = u(x)− hu′(x) + h2/2u′′(x)− h3/3!u′′′(x) + · · ·

u(x + h)− u(x)

h= u′(x) + h/2u′′(x) + · · ·

: It is first order accurate.

u(x + h)− u(x− h)

2h= u′(x) + h2/3!u′′′(x) + · · ·

: Centered is second order.

Centered difference matrix:

△0 =

. . .

−1 0 1

−1 0 1. . .

ui−1ui

ui+1

ui+2

=

...

ui+1 − ui−1ui+2 − ui

...

Rmk: △T

0 = −△0: antisymmetric (skew symmetric).

The centered difference is the average of forward and backward.

• Second Differences from First Differences

△2ui =△−△+ui = 1/h

[(ui+1 − ui

h

)−

(ui − ui−1

h

)]=ui+1 − 2ui + ui−1

h2

△2u(x) = u(x+h)−2u(x)+u(x−h)h2 = u′′(x) + 2h2/4!u(4)(x) + · · ·

: second order accuracy

• The Important Multiplications

1. For constant and linear vectors, the second difference are zero:



△2(constant)

. . .

1 −2 1

1 −2 1. . .

1

1

1

1

=

...

0

0...

△2(linear)

. . .

1 −2 1

1 −2 1. . .

1

2

3

4

=

...

0

0...

For squares, the second difference is constant:

△2(squares)

. . .

1 −2 1

1 −2 1. . .

1

4

9

16

=

...

2

2...

2. Second difference of the ramp vector produce the delta vector:

△2(ramp)

. . .

1 −2 1

1 −2 1. . .

0

0

1

2

=

0

1

0

0

= delta

3. Second difference of the sine and cosine and exponential produce 2 cos t−2 times those vectors.

△2(sines)

. . .

1 −2 1

1 −2 1. . .

sin t

sin 2t

sin 3t

sin 4t

= (2cost− 2)

sin t

sin 2t

sin 3t

sin 4t

△2(cosines)

. . .

1 −2 1

1 −2 1. . .

cos t

cos 2t

cos 3t

cos 4t

= (2 cos t−2)

cos t

cos 2t

cos 3t

cos 4t



△2(exponentials)

. . .

1 −2 1

1 −2 1. . .

eit

e2it

e3it

e4it

= (2 cos t−2)

eit

e2it

e3it

e4it

Remark: sines or cosines or exponentials are eigenvectors of

K,T,B,C with the right boundary conditions.

Part II: Finite Difference Equations

−d2udx2 = f with boundary conditions u(0) = 0 and u(1) = 0.

Divide the interval [0, 1] into equal pieces of length h = △x.

unknown u =

u(h)

u(2h)...

u(nh)

=

u1

u2...

un

where h = 1n+1

.

Finite difference equation: −ui+1 − 2ui + ui−1

h2= fi

The first and last (i = 1, i = n) involves u0, un+1.

Ex) Solve −d2udx2 = 1 with u(0) = 0 and u(1) = 0,

−ui+1−2ui+ui−1h2 = 1, u0 = 0 and un+1 = 0.

Sol) Complete solution: ucomplete = uparticular + unullspace

Particular solution: −d2udx2 = 1 is solved by uparticular = −1

2x2.

Nullspace solution: −d2udx2 = 0 is solved by unullspace = Cx + D.

u(x) = −12x2 + Cx + D.

u(0) = 0: D = 0.



u(1) = 0: −12+ C = 0⇒ C = 1

2.

u(x) = +12x− 1

2x2.

This is special: the differential and difference equation have the same so-

lutions!

u(x) = −12x2 + 1

2x and ui =

12(ih− i2h2).

With h = 1/4,

Ku = f leads 16

2 −1 0

−1 2 −10 −1 2

u1

u2

u3

=

1

1

1

⇒ 3/32

4/32

3/32

• A Different Boundary Condition

Ex) Solve −d2udx2 = 1 with du

dx(0) = 0 (free end) and u(1) = 0.

−ui+1−2ui+ui−1h2 = 1, u1−u0

h= 0 and un+1 = 0.

Sol) u(x) = 12(1− x2).

We expect a O(h) error because of the forward difference u1−u0h

.

For n = 3, h = 1/4,

1/h2

1 −1 0

−1 2 −10 −1 2

u1

u2

u3

=

1

1

1

gives h2

6

5

3

To have O(h2) accuracy, see the worked examples 1.2.A in the book.

99% of the difficulties with DE’s occurs at the boundary.



§1.3 elimination leads to K = LDLT

Two theme of the book:

1. How to understand equations.

2. how to solve them. - This section’s topic.

Ku = f :

u = K−1f theorectically, not computationally:

u = inv(K) ∗ f in MATLAB.

Solved by Gaussian elimination - LU decomposition.

If symmetric, K = LDLT (related to Cholesky factorization).

u = K\f in MATLAB. For multiple f ’s, [L, U] = lu(K).

Ex) Ku = f

2 −1 0

−1 2 −10 −1 2

u1

u2

u3

=

f1

f2

f3

2 −1 0

0 3/2 −10 −1 2

u1

u2

u3

=

f1

f2 + 1/2f1

f3

by 2nd row +1/2× 1st row,

2 −1 0

0 3/2 −10 0 4/3

u1

u2

u3

=

f1

f2 + 1/2f1

f3 + 2/3f2 + 1/3f1

by 3rd row + 1

3/22st row.

2u1 − u2 =f1,

3/2u2 −u3 =f2 + 1/2f1,

4/3u3 =f3 + 2/3f2 + 1/3f1.Solution by backsubstitution.



Matrix-vector multiplication Ku as a combination of the columns of K: 2 −1 0

−1 2 −10 −1 2

u1

u2

u3

= u1

2

−10

+ u2

−12−1

+ u3

0

−12

Solving a system Ku = f is exactly the same as finding a combination of

the column of K that produces the vector f .

Multiplier lij = entry to eliminatepivot

(in row i)(in row j)

The convention: Subtract lij times the pivot row j from row i. Then the

i, j entry is 0.

• Elimination Produces K = LU

K = LU

2 −1 0

−1 2 −10 −1 2

=

1 0 0

−1/2 1 0

0 −3/2 1

2 −1 0

0 3/2 −10 0 4/3

L reverses the elimination steps.

Suppose the forward elimination uses the multipliers in L to change the rows

of K in to the rows of U (Upper triangular). Then K is factored into L

times U .

Ku = f : LUu = f ⇒ u = U−1L−1f

Lc = f ⇒ c = L−1f (forward substitution)

Uu = c⇒ u = U−1c (back substitution)

• Singular Systems



Ex) Circulant C =

2 −1 −1−1 2 −1−1 −1 2

2 −1 −10 3/2 −3/20 −3/2 3/2

2 −1 −10 3/2 −3/20 0 0

= U : The rows are linearly dependent

An invertible matrix has a full set of pivots.

No row exchange to get n pivots: A is invertible and A = LU .

Row exchange by P to get n pivots: A is invertible and PA = LU .

No way to find n pivots: A is singular, there is no inverse matrix A−1.

Pivoting matrix P : i row! j row

Composition of In−2×n−2 except i, j rows and columns and

[1

1

]for

i, j rows and columns.

Ex) 2 row! 3 row:

1

1

1

1

• Symmetry Converts K = LU to K = LDLT

K︸︷︷︸symmetric

= L︸︷︷︸lower triangular

U︸︷︷︸upper triangular

: not symmetric

K =

2 −1 0

−1 2 −10 −1 2

= LU =

1 0 0

−1/2 1 0

0 −3/2 1

2 −1 0

0 3/2 −10 0 4/3



=

1 0 0

−1/2 1 0

0 −3/2 1

2 0 0

0 3/2 0

0 0 4/3

1 −1/2 0

0 1 −3/20 0 1

= LDLT : symmetric factorization

Remark 1. ATCA is symmetric if C is symmetric.

2. DA: rowwise multiplication by the diagonal entry in D.

3. AD: columnwise multiplication by the diagonal entry in D.

4. For any rectangular matrix A, the product ATA is square and symmet-

ric.

5. LDLT = L√D√DLT = (L

√D)(L

√D)T := LLT is called

Cholesky factorization.

• The Determinant Kn

detK =by definition of determinant ∼ n! : computationally useless!

=detLU = detL︸︷︷︸=1

detU

=product of diagonal entries of U

=2 · 3/2 · 4/3 · · · (n + 1)/n = n + 1

The LU decomposition is also a quick way to compute determinant.

Remark 1. If a matrix is tridiagonal, then L and U are bidiagonal.

2. If a row/column of K starts with p/q zeros (no elimination needed

there), then that low of L/ column of U also starts with p/q zeros.

3. Zeros inside the band can unfortunately be “filled in” by elimination -

It leads to fundamental problem of reordering the rows and columns to

make the p’s and q’s are as large as possible.



Figure 2: Left: The sparsity pattern of K for 2 dim. Right: The sparsity pattern of the Choleskyfactor of K.

• Positive Pivots and Positive Determinant

If is positive definite (xTAx > 0 if x = 0) if all pivots are positive.[a b

b c

]=

[1

b/a 1

] [a

(ac− b2)/a

] [1 b/a

1

][a b

b c

]is positive definite iff a > 0 and ac− b2 > 0.

• Operation Counts

LU decomposition ∼ 2/3n3 in general, 1/3n3 if symmetric.

Operation Count Full Banded Tridiagonal

Factor: Find L and U ≈ 2/3n3 2w2n + wn 3n

Solve: Forward and back on f 2n2 4wn + n 5n



§1.4 Inverse and Delta Functions

: Want to solve for f = point load.

Ku = δj =

0...

1...

0

(j th entry) = j th column of I.

−u′′ = δ(x− a): Green’s function u.

Delta function δ(x):

δ(x) = 0 if x = 0,∫∞−∞ δ(x)dx = 1 .

: not “true function”. “spike”, “point load”, “impulsive”concentrated at

x = 0, “infinitely tall and infinitely thin”

A sequence of functions generating or approximating Dirac delta:

fk(x) =

1

2k, if − k ≤ x ≤ k,

0, otherwise.∫∞−∞ fk(x) dx = 1 and ‘fk −−→

k→0δ ’.

Note that

K[u1|u2| · · · |un] = [δ1|δ2| · · · |δn]⇐⇒ KK−1 = I.

uj = column j of K−1. We are solving KK−1 = I column by column.

If we know the Green’s function for all point load δ(x − a), we can solve

−u′′ = f for any load f(x).

So, it is the “discrete Green’s function”.

K−1ij = the solution at point i from a load at point j .



0 0.2 0.4 0.6 0.8 1

0

0.2

0.4

0.6

0.8

1

Figure 3: Green’s function for fixed-fixed case

• Concentrated load

Ex) −d2udx2 = δ(x− a) with

{fixed u(0) = 0 and fixed u(1) = 0

free u′(0) = 0 and fixed u(1) = 0.

Sol)∫ right

left −d2udx2 dx =

∫ right

left δ(x− a) dx

=⇒ −(dudx

)right

+(dudx

)left

= 1: The slope drops by 1.

Since u′′ = 0 except x = a,

u =

{Ax + B if x < a,

Cx + D if x > a.

Boundary Conditions Jump/No Jump Conditions at x = a

fixed u(0) = 0 : B = 0 No jump in u : Aa + B = Ca + D

fixed u(1) = 0: C + D = 0 Drop by 1 in u′ : A = C + 1

=⇒ u(x)︸︷︷︸u(x;a)

=

{(1− a)x if x < a,

a(1− x) if x > a.

Remark u(x; a) is symmetric w.r.t. x and a. Note that its discrete version

K−1 is also symmetric (since K is symmetric).

• Delta Function and Green’ function

Delta ft δ(x)

δ(x)=dSdx←−−−−

−−−−−−→∫ x−∞ δ(y)dy

Step ft S(x)

S(x)=dRdx←−−−−−

−−−−−−→∫ x−∞ S(y)dy

Ramp ft R(x)



where S(x) =

{1 if x ≥ 0

0 if x < 0and R(x) =

{x if x ≥ 0

0 if x < 0.

The first derivative of the ramp function R(x) jumps by 1 at 0 and the

second derivative is a delta function.

Complete solution −d2u

dx2= δ(x− a) is solved by

u(x) = −R(x− a)︸︷︷︸particular solution

+ Cx + D︸︷︷︸null space u′′=0

0 = u(0) = −R(0− a) + C · 0 + D =⇒ D = 0.

0 = u(1) = −R(1− a) + C +D = a− 1 + C =⇒ C = 1− a.

u(x)︸︷︷︸u(x,a)

= −R(x− a) + (1− a)x =

{(1− a)x if x ≤ a,

(1− x)a if x ≥ a.

The response at x to a load at a equals the response at a to a load at x:

symmetric.

cf) (K−1)ij = (K−1)jiFree-Fixed: u′(0) = 0, u(1) = 1

u(x)︸︷︷︸u(x,a)

= −R(x− a) + (1− a)x =

{1− a if x ≤ a,

1− x if x ≥ a.

• Discrete Vectors: Load and Step and Lamp

The delta vector δ: δ = (· · · , 0, 0, 1, 0, 0, · · · )The step vector S: S = (· · · , 0, 0, 1, 1, 1, · · · )The Lamp vector R: R = (· · · , 0, 0, 0, 1, 2, · · · )Note that △−S = δ but △+R = S

△2 = △−△+ so,

△2R = △−△+R = △−S = δ



△2(ramp)

. . .

1 −2 1

1 −2 1. . .

0

0

1

2

=

0

1

0

0

= delta

The solution to △2u = 0 are “linear vectors” with ui = Ci + D.

The complete solution to △2u = δ is

ui = Ri︸︷︷︸uparticular

+Ci + D︸︷︷︸unullspace

cf) u(x) = Rx + Cx + D

Sampling the ramp u(x) at equally space points without any error.

• The Discrete Equations Ku = δj and Tu = δj

−△2u = δj: ui = −Ri−j + Ci + D

u0 = −R0−j + C · 0 + D = 0 =⇒ D = 0

un+1 = −Rn+1−j + C(n + 1) + 0 = 0

=⇒ C = n+1−jn+1

= 1−j

n + 1︸︷︷︸1−a

Fixed ends: ui = −Ri−j + Ci =

(n + 1− j

n + 1

)i if i ≤ j,(

n + 1− i

n + 1

)j if i ≥ j.

Note: K−1n is symmetric. cf) u(x) =

{1− a if x ≤ a,

1− x if x ≥ a.

Free-Fixed: ui = −Ri−j + (n+1− j) =

{n + 1− j if i ≤ j,

n + 1− i if i ≥ j.

cf) u(x) =

{1− a if x ≤ a,

1− x if x ≥ a.



Green’s Function and Inverse Matrix

f =

f1

f2

f3

= f1

1

0

0

+ f2

0

1

0

+ f3

0

0

1

:A combination of n point loads

⇒

K−1f = f1 (column 1 of K−1)︸︷︷︸K−1δ1

+ f2 (column 2 of K−1)︸︷︷︸K−1δ2

+ f3 (column 3 of K−1)︸︷︷︸K−1δ3

The load f(x) is an integral of point load f(a)δ(x− a).

−u′′ = f(x) =

∫ 1

0

f(a)δ(x− a)da =⇒ u(x) =

∫ 1

0

f(a)u(x, a)da

The Green’s function u(x, a) corresponds to “row x and column a of

continuous K−1.



§1.5 Eigenvalues Eigenvectors

Part I: Ax = λx and Akx = λkx and Diagonalizing A

• Matrix as a Linear Transformation

A matrix A is considered as a linear transformation .

A :Rn →Rm

x 7→Ax

Definition of linear : A(rx + sy) = rAx + sAy

’Superposition Principle’

Example 1 A =

[1 0

0 1

]: identity, A =

[2 0

0 1

]: dilation, A =[

cos θ − sin θ

sin θ cos θ

]: rotation, A =

[1 0

0 −1

]: reflection.

• We may regard linear transformation as a composition of those dilations,

rotations, and reflections, etc.

• Eigenvalues and Eigenvectors

Definition 2 λ: eigenvalue and x: eigenvector if

Ax = λx, x = 0 .

Geometrically, along the eigen-direction, there is only scaling

or dilation.

More specifically, the special vector x lies along the same line as Ax. The

eigenvalue λ tells whether the vector x is stretched or shrunk or reversed or

left unchanged.

eigen: prime in German

Why are we interested in eigenvalues and eigenvectors?



Figure 4: Geometric interpretation of eigenvalue and eigenvector

- It reveals the ‘innate or invariant structure’ of A.

Especially, eigenvalues are invariant under change of basis.

We can understand the matrix A easily by observing the eigenstructure of

A, called spectrum.

The ‘easiest’ matrix:

I =

1. . .

1

: leave every entry unchanged.

The ‘second easiest’ matrix:

D =

d1. . .

dn

D

x1...

xn

=

d1x1...

dnxn

: cooridinatewise multiplication.

Dx = b can be easily solved by division.

D is positive definite⇔ di > 0 for all i.

xTDx =∑n

i=1 dix2i = d1x

21 + · · ·+ dnx

2n.

A ∼ D : A is ‘similar’ to D?

Can we treat or understand A as a diagonal matrix?



Definition 3 A is similar to B if there exists a nonsingular X such that

X−1AX = B.

Remark 1. If A is similar to a diagonal matrix D, then X−1AX = D,

called diagonalization.

2. Since AX = XD, X is the eigenvectors of A with eigenvalues D.

3. X is nonsingular means the columns of X consists of a basis.

4. Diagonalization is equivalent to finding n eigenvalues and eigenvectors.

• Diagonalizing a matrix

A is a n by n matrix with n independent eigenvectors x1, · · ·xn with

eigenvalues λ1, · · ·λn, respectively.

AX =A[x1| · · · |xn] = [Ax1| · · · |Axn] = [λ1x1| · · · |λnxn]

=[x1| · · · |xn]

λ1. . .

λn

= XΛ

Now X−1AX = Λ. then A = XΛX−1: Consider Ay = XΛX−1y.

X−1: express v w.r.t the eigenbasis X.

Λ: dilation along each eigendirection.

X: send back to the original basis.

Thus, the role of X is change of basis.

When square matrices are diagonalizable?

1. In general, it is not guaranteed because there may not be n independent

eigenvectors.

Other decompositions: Jordan canonical form, Schur’s canonical form,

Singular Value decomposition (SVD).

2. If A is symmetric (for real-valued) or Hermite (for complex-valued), OK.



3. The weakest condition is possibly AAT = ATA, called normal matrix.

• Symmetric Matrices and Orthonormal Eigenvectors

If A is symmetric, there exist n independent eigenvectors x1, · · · , xn with

n real-valued eigenvalues λ1, · · · , λn. Furthermore, they are orthogonal:

AU = UΛ and UTU = I

A = UΛUT or UTAU = Λ

Definition 4 The vectors ui, · · ·un are orthonormal if

⟨ui, uj⟩ = uTi uj = δij =

{1 if i = j (normality),

0 if i = j (orthogonality).

Definition 5 The square matrix U is orthogonal if UTU = I. i.e.

UTU =

uT1

uT2...

uTn

u1 u2 · · · un

=

1. . .

1

= I

• Finding eigenvalues and eigenvectors

Finding x and λ satisfying Ax = λx is n equations with n+1 unknown.

We first try to find λ and next find x:

λ : Ax = λx, x = 0⇔(A− λI)x = 0, x = 0

⇔A− λI is singluar

⇔ det(A− λI) = 0: charateristic equation

det(A− λI) =cnλn + cn−1λ

n−1 · · ·+ c0

=cn(λ− λ1)(λ− λ2) · · · (λ− λn)

by the Fundamental Theorem of Algebra.

Remark :

1. det(A− λI) ∼ O(n!), not useful computationally.



2. Furthermore, it is sensitive perturbation and rounding error. Also finding

roots is not easy.

3. Numerically power method or some other numerical algorithms are used.

Ex) eig(A) in MATLAB

4. One great success of numerical linear algebra is the development of fast

and stable algorithm to compute eigenvalues, especially for the symmet-

ric case.

Example 6 Consider the symmetric case K =

[2 −1−1 2

].

Sol)

det(K − λI) =

∣∣∣∣ 2− λ −1−1 2− λ

∣∣∣∣ = (2− λ)2 − 1

=λ2 − 4λ + 3 = (λ− 3)(λ− 1)

For λ = 1 : K − I =

[1 −1−1 1

](K − I)x = (K − I)[u; v] = 0⇒ u− v = 0

x1 =

[1

1

]or 1√

2

[1

1

].

For λ = 3 : K − 3I =

[−1 −1−1 −1

](K − 3I)x = (K − 3I)[u; v] = 0⇒ u + v = 0

x2 =

[1

−1

]or 1√

2

[1

−1

].

Now

K =

[2 −1−1 2

]=

1√2

[1 1

1 −1

] [1 0

0 3

]1√2

[1 1

1 −1

]T

Note that K is a discrete approximation of

−d2udx2 with u(0) = 0 and u(1) = 0.



−d2u

dx2= λu⇔ Ku = λu with Ku = −

d2u

dx2

and

K is linear . −d2u

dx2= λu,

u(0) = 0 and u(1) = 0.

Eigenfunctions: sin kπx, k = 1, 2, · · · .With h = 1/3, x0 = 0, x1 = 1/3, x2 = 2/3, x3 = 1,

sinπx ∼ [sinπx1; sinπx2] = [sin 1/3π; sin 2/3π] = 1√2[1; 1]

and

sin 2πx ∼ [sin 2πx1; sin 2πx2] = [sin 2/3π; sin 4/3π] = 1√2[1;−1].

K = UΛUT , U =1√2

[1 1

1 −1

], Λ =

[1 0

0 3

].

K2 = (UΛUT )(UΛUT ) = UΛ2UT .

Kn = UΛnUT = U

[1 0

0 3n

]UT : Kn grows like 3n.

The product of n eigenvalues equals the determinant of A.

detA =n∏

i=1

λi

Proof) ‘determinant of products is product of determinants’.detA = det(UΛU−1) = detU detΛ detU−1

= det(UU−1) det Λ = detΛ.

Remark It is the constant term w.r.t. λ in det(A− λI).

The sum of the n eigenvalues equal the sum of the n diagonal entries.

tr A =n∑

i=1

λi



Remark It is the coefficient of (−λ)n−1 in det(A− λI).

If A = XΛX−1 with no nonzero λi, then A is invertible and

A−1 = XΛ−1X−1 = X

1/λ1. . .

1/λn

X−1 ,

which is an eigenvalue decomposition of A−1.

• The Power of a Matrix

-Eigenvalues have their greatest importance in dynamic problems.

Example 7 Population problem: u(t + 1) = Au(t) each year where[u1(t + 1)

u2(t + 1)

]=

[0.8 0.3

0.2 0.7

] [u1(t)

u2(t)

].

The column sums are always 1, which means nobody is created or destroyed.

Furthermore, populations stay positive because has no negative entries.

This type of matrices are called Markov matrix which expresses probability

transition matrix.

Let u(0) =

[1000

0

]. The matrixA has eigenvectors

[600

400

],

[400

−400

]with eigenvalues 1, 1/2 respectively.

If Ax = λx, A2x = λ2x, · · · , Akx = λkx.

Similarly, Ak(α1x1 + · · ·+ αnxn) = α1λk1x1 + · · ·+ αnλ

knxn.

u(t) = At

[600

400

]=At

([600

400

]+

[400

−400

])=1t

[600

400

]︸︷︷︸steady state

+

(1

2

)t [400

−400

]︸︷︷︸

transient

−−−−→t→∞

[600

400

].

• Three steps to find uk = Aku0 from eigenvalues and eigen-

vectors



Step 1 : Write u0 as a combination of the eigenvectors:

u0 = α1x1 + · · ·+ αnxn

Step 2 : Multiply each number αj by (λj)k.

Step 3 : Recombine the eigenvectors into

uk = α1λk1x1 + · · ·+ αnλ

knxn

In matrix,

Step 1 : u0 = [x1| · · · |xn]

α1...

αn

= Xa, α = X−1u0.

Step 2 : Multiply

λk1

. . .

λkn

α1

...

αn

= Λkα, ΛkX−1u0.

Step 3 : Recombine uk = [x1| · · · |xn]

λk1α1...

λknαn

= XΛkα

uk = XΛkX−1u0.

Remark

Ay = XΛX−1y

X−1y expresses y w.r.t. the basis X = [x1| · · · |xn].

Λ multiplies eigenvalues.

X recombines.

• Application to Vector Differential equations

Example 8 dy

dt= ay general solution y(t) = Ceat.

y(0) = y0 It determines C.



The solution y(t) = y0eat decays if a < 0: stability.

The solution y(t) = y0eat grows if a > 0: instability.

When a is a complex number, its real part determines the growth or decay,

the imaginary part gives oscillatory factor since

eiwt = coswt + i sinwt (Euler formula)

Vectorial case or system of equations

dy

du= Au, u(0) = u0

Example 9

dy

dt=2y − z

dz

dt=− y + 2z

, ddt

[y

z

]=

[2 −1−1 2

]︸︷︷︸

K2

[y

z

]

Sol) Assume u(t) = eλtx = eλt[y

z

].

dudt

= λeλtx = LHS = RHS = Ku = eλtKx

Kx = λx : eigenvalue problem

x =

[1

1

],

[1

−1

]with λ = 1, 3 respectively.

General solution:

u(t) =c1eλ1tx1 + c2e

λ2tx2

=c1et

[1

1

]+ c2e

3t

[1

−1

]

u0 = u(0) = c1

[1

1

]+ c2

[1

1

]=

[1 1

1 −1

]︸︷︷︸

X

[c1c2

]Three steps for powers apply here too:

Expand u0 = Xα: α = X−1u0.



Multiply each αj by eλjt:

eλ1t

. . .

eλnt

a = eΛtX−1u0.

Recombine into u(t) = XeΛtX−1u0.

Part II: Eigenvectors for Derivatives and Differences

−d2u

dx2= λu is solved by y = cosωx, y = sinωx with λ = ω2.

Analog to Kn: fixed-fixed case y(0) = 0, y(1) = 0.

y(x) = sin kπx with λ = k2π2, k = 1, 2, · · ·Sol) y = a cosωx + b sinωx

0 = y(0) = a

0 = y(1) = b sinω ⇒ ω = kπ: determined by boundary condition!

Analog to Bn: free-free case y′(0) = 0, y′(1) = 0.

y(x) = cos kπx with λ = k2π2, k = 0, 1, 2, · · ·

Analog to Cn: periodic case y(0) = y(1), y′(0) = y′(1).

y(x) = cos 2kπx, sin 2kπx with λ = 4k2π2, k = 0, 1, 2, · · · ,k = 1, 2, · · ·

Analog to Tn: free-fixed case y′(0) = 0, y(1) = 0.

y(x) = cos(k + 1/2)πx with λ = k2π2, k = 0, 1, 2, · · ·

• Eigenvectors of Kn: Discrete Sines

−[sin(j − 1)θ

cos(j − 1)θ

]+2

[sin jθ

cos jθ

]−

[sin(j + 1)θ

cos(j + 1)θ

]=(2− 2 cos θ)

[sin jθ

cos jθ

]Computational Science & Engineering (CSE) Yoon Mo Jung


These are the imaginary and real parts of

−ei(j−1)θ + 2eijθ − ei(j+1)θ = (2− e−iθ − eiθ)eijθ .

The boundary rows decide θ everything!

For sin jθ,

From the first row,

2 sin θ − sin 2θ = (2− 2 cos θ) sin θ, it is true for any θ.

From the last row,

− sin(n− 1)θ − 2 sinnθ = (2− cos θ) sinnθ,

−(sinnθ cos θ−cosnθ sin θ)+2 sinnθ = 2 sinnθ−2 sinnθ cos θ

sinnθ cos θ + cos θ sinnθ = sinn(θ + 1) = 0

⇒ θ = kn+1

π, k = 1, 2, · · ·For cos jθ,

From the first row,

2 cos θ − cos 2θ = (2− 2 cos θ) cos θ

cos2 θ − 1 = 2 cos2 θ

⇒ −1 = 0: There is no such θ.

The first eigenvector y1 will sample the first eigenfunction y(x) = sinπx

at n meshpoint with h = 1n+1

:

First eigenvector = dicrete sine y1 = (sinπh, sin 2πh, · · · , sinnπh)

First eigenvalue of Kn:

λ1 = 2− 2 cosπh = 2− 2(1− π2h2

2+ · · · ) ≈ π2h2

To match differences with derivatives, divide K by h2 = (△x)2.

eigenvectors = discrete sines yk = (sin kπh, · · · , sinnkπh)

eigenvalues of Kn : 2− 2 cos kπh, k = 1, · · ·n.



Discrete sine transform

DST =

sin π4

sin 2π4

sin 3π4

sin 2π4

sin 4π4

sin 6π4

sin 3π4

sin 6π4

sin 9π4

=

1√2

1 1√2

1 0 −11√2−1 1√

2

Q = 1√

2DST is orthogonal, i.e. QTQ = I, Q−1 = QT .

Remark∫ 1

0 sinnπx sinmπxdx = 0 if n = m.

• Eigenvectors of Bn: Discrete Cosines

eigenvalues of Bn : 2− 2 coskπ

n, k = 0, · · ·n− 1

eigenvectors: yk =

(cos

1

2

kπ

n, cos

3

2,kπ

n· · · , cos

(n−

1

2

)kπ

n

)Eigenvalues of B sample cos kπx at the n midpoints x = (j − 1

2)/n.

y′(0) = 0 ∼ y(x1)− y( x0︸︷︷︸ghost grid

) = 0

y′(1) = 0 ∼ y( xn+1︸︷︷︸ghost grid

)− y(xn) = 0

Since the cosine is even, those vectors have zero slope at the ends:

cos−12kπn

= cos 12kπn

and cos(n− 1

2

)kπn

= cos(n + 1

2

)kπn

: The reason for choosing midpoints as gridpoints.

Note that k = 0 gives the all-ones eigenvector y0 = (1, 1, · · · , 1) with

λ = 0: DC vector with zero frequency.

Discrete cosine transform

DCT =

cos 0 cos 12π3

cos 122π3

cos 0 cos 32π3

cos 322π3

cos 0 cos 52π3

cos 522π3

=

1 12

√3 1

2

1 0 −11 −1

2

√3 1

2



• Eigenvectors of Cn: Powers of ω = e2πin

Eigenvectors of Cn: Both sine and cosine

(Euler formula) eiθ = cos θ + i sin θ

Circulant matrix (periodic) C4 =

2 −1 0 −1−1 2 −1 0

0 −1 2 −1−1 0 −1 2

It has constant diagonals with wrap-around.

The k th eigenvector of Cn comes from sampling yk(x) = ei2πkx at the

n meshpoints x = j./n, j = 0, · · · , n− 1.

j th component of yk: ei2πk(j/n) = ωjk where ω = ei2π/n = n th root

of 1.

eigenvalues of Cn : 2− ωk − ω−k = 2− 2 coskπ

n, k = 0, · · ·n− 1

eigenvectors: yk = (1, ωk, ω2k · · · , ω(n−1)k)

• The Fourier Matrix

Discrete Fourier transform (DFT)

F4 =

1 1 1 1

1 i i2 i3

1 i2 i4 i6

1 i3 i6 i9

, (Fn)jk = ωjk = ei2πjk/n

The columns are orthogonal in C: ⟨x, y⟩ = x∗y = xTy.

F4TF4 = 4I so that F−14 = 1

4F4

T.



In general,

FnTFn = nI and F−1n =

1

nFn

T=

1

nF ∗n

.

Un = 1√nFn: The normalized Fourier matrix is unitary.

Columns are ’orthonormal’ in C: UnTUn = 1√

nFn

T 1√nFn = I.

Unitary matrix (Q∗Q = QTQ = I) is the complex analog of orthogonal

matrix (ATA = I).



Figure 5: Quadratic function in 1-d example: x2 − x− 2

§1.6 Positive Definite Matrix

What is ‘positive definite’?

3 basic facts

1. K = ATA is symmetric and positive definite (or at least semidefinite).

xTATAx = (Ax)TAx

2. If K1 and K2 are positive definite, then so is K1 + K2.

3. All pivots and all eigenvalues of a positive definite matrix is positive.

Why do we want to consider positive definite matrices?

- It is closely related to the concept of energy as the quadratic form 12uTKu

and we are interested in its minimum.

Example 10 1 dimensional example (See Fig. 5)

f(x) = 12ax2 − bx + c︸︷︷︸

=0

= 12a(x− b

a)2 − 1

2b2

a.

Optimization: For its minimum,

The first necessary condition: f ′(x) = ax− b.

The second sufficient condition: f ′′(x) = a > 0.

If K is positive definite,

the minimum of P (u) =1

2uTKu− uTf is Pmin = −

1

2fK−1f

when Ku = f.



Example 11 2 dimensional examples (See Fig. 6)

A =

[1 0

0 2

], B =

[1 0

0 0

], C =

[1 0

0 −2

], D =

[−1 0

0 −2

].

xTAx = [x y]

[1 0

0 2

] [x

y

]= x2 + 2y2 : positive definite (elliptic).

Similarly,

xTBx = x2: semipositive definite (parabolic),

xTCx = x2 − 2y2: indefinite (hyperbolic),

xTDx = −x2 − 2y2 = −(x2 + 2y2): negative definite.

• Examples and Energy-based Definition

Quadratic function:

uTSu = [u1, u2]

[u1

u2

]= au2

1 + 2bu1u2 + cu2.

Example 12 Sum of squares examples (See also Fig. 6)Positive definite Semipositive definite Indefinite

K =

[2 −1−1 2

]B =

[1 −1−1 1

]M =

[1 −3−3 1

]2u2

1 − 2u1u2 + 2u22 u2

1 − 2u1u2 + u22 2u2

1 − 6u1u2 + 2u22

Always positive Positive or zero Positive or negative

2u21 − 2u1u2 + 2u2

2

=u21 + (u1 − u2)

2 + u22 : ATA

=2(u1 −1

2u2)

2 +3

2u22 : LDLT .

K =

[1 −1 0

0 1 −1

] 1 0

−1 1

0 1

= ATA

=

[1 0

−12

1

] [2 0

0 32

] [1 −1

2

0 1

]= LDLT .



Figure 6: Positive definite, Indefinite, Semidefinite functions in 2 dim

u21 − 2u1u2 + u2

2 = (u1 − u2)2.

2u21 − 6u1u2 + 2u2

2 = (u1 − 3u2)2 − 8u2

2.

• Positive definiteness from ATA,ATCA,LDLT , QΛQT

1. K = ATA is symmetric positive definite iffA has independent columns:

uTKu = uTATAu = (Au)TAu = ∥Au∥ > 0 for x = 0 if A

has full rank.

2. K = ATCA is symmetric positive definite iff A has independent

columns and C is symmetric positive definite:

uTKu = uTATCAu = (Au)TCAu > 0 for x = 0.

3. If symmetric K has a full set of positive pivots, it is positive definite:

K = LDLT , the diagonal pivot matrix D is positive definite and LT

has independent columns.

4. If a symmetric K has all positive eigenvalues in Λ, it is positive definite:

K = QΛQT , Q−1 = QT .

• Minimum Problem in n Dimensions

Very often, 12uTKu is the “internal energy” in the system.



P (u) =1

2uTKu− uTf : total energy

∇P = Ku− f = 0: the first necessary condition

H = K > 0, positive definite: the second sufficient conditon

P (K−1f) = 12(K−1f)TK(K−1f)− (K−1f)Tf = −1

2fTK−1f

P (u)− P (K−1f) =1

2uTKu− uTf − (−

1

2fTK−1f)

=1

2(u−K−1f)TK(u−K−1f) ≥ 0.

• Test for a minimum: Positive Definite Second Derivatives

Test for 1 dimensional function:

f(x) = f(a)︸︷︷︸const

+ f ′(a)︸︷︷︸slope = 0

(x− a) +1

2f ′′(a)︸︷︷︸

concavity >0

(x− a)2 + · · ·

by Taylor series.

Test for n dimensional function:

P (u) = P (u∗)+(u−u∗)T∇P (u∗)+1

2(u−u∗)TH(u∗)(u−u∗)+· · ·

again by Taylor series.

For minimum,

(1st derivative vector: slope) ∇P (u∗) =

∂P∂u1...∂P∂un

= 0

and

(2nd derivative matrix: concavity) Hij =∂2P

∂ui∂uj

=∂2P

∂uj∂ui

= Hji > 0

i.e. positive definite.

• Newton method



Approximation by by quadratic form:

P (u) ≈P (u∗)︸︷︷︸=0

+(u− u∗)T︸︷︷︸uT

∇P (u∗)︸︷︷︸−f

+1

2(u− u∗)T︸︷︷︸

uT

H(u∗)︸︷︷︸K

(u− u∗)︸︷︷︸u

:=1

2uTKu− uTf

⇒ Ku = f.

H(u∗)(u− u∗) = −∇P (u∗).

Newton’s method: H(ui)(ui+1 − ui) = −∇P (ui)

If ui hits exactly a minimum u∗ (not too likely)∇P (u∗) = 0, so ui+1−ui = 0, no more steps.

It is an iterative method to solve a minimum problem.



§1.7 Numerical Linear Algebra: LU, QR, SVD

Ex) Ku = f or Kx = λx or Mu′′ + Ku = 0

Crucial properties of K: symmetric? banded? sparse? well-conditioned?

• Three Essential Factorization

1. A = LU = lower triangle matrix × upper triangle matrix

by Gaussian elimination.

2. A = QR = Orthogonal matrix × upper triangle matrix

by Gram-Schmidt orthogonalization or Householder transformation.

3. A = UΣV T = orthonormal columns × singular values × orthonor-

mal lows

by singular value decomposition.

It is a ‘generalized eigenvalue decomposition’. cf) QΛQT .

• Orthogonal Matrices

⟨qi, qj⟩ = qTi qj = 0 if i = j (orthogonality)

⟨qi, qi⟩ = qTi qi = 1 (normalization to unit vector)

Let Q = [q1|q2| · · · |qn].

QTQ =

qT1

qT2...

qTn

q1 q2 · · · qn

=

1. . .

1

= I

The inverse is its transpose: Q−1 = QT .

Length preserving (also angle preserving): ∥Qx∥ = ∥x∥



Ex) permutations, rotations, reflections.

Example 13 Permutation: the same rows as I, in a different order. 0 1 0

0 0 1

1 0 0

x

y

z

=

y

z

x

P TP = I

Example 14 Rotation

Rotation matrix in the 1− 3 plane:

cos θ 0 − sin θ

0 1 0

sin θ 0 cos θ

Example 15 Reflection: The reflection takes v to Hv on the other side

of a plane mirror. The unit vector u perpendicular to the mirror is reversed

into Hu = −u.Reflection matrix u = (cos θ, 0, sin θ):

H =I − 2uuT =

1 0 0

0 1 0

0 0 1

− 2

cos θ

0

sin θ

[cos θ, 0, sin θ]

=

1− cos2 θ 0 −2 sin θ cos θ

0 1 0

−2 sin θ cos θ 0 1− sin2 θ

=

− cos 2θ 0 − sin 2θ

0 1 0

− sin 2θ 0 cos 2θ

detH = −1, Hu = (I − 2uuT )u = u− 2u = −u.It is a popular method for QR decomposition.

• Orthogonalization A = QR

1. Gram-Schmidt algorithm

Am×n =

a1 · · · an



Figure 7: Householder transformation

Figure 8: Gram-Schmidt orthogonalization

rank n, independent n vectors, consists of a basis for the column space of

A

q1 := a1∥a1∥

, a1 = r11q1 with r11 =: ∥a1∥.B = a2 − (qT

1 a2)q1 is orthogonal to q1.

qT1 a2: projection in the q1 direction := r12.

q2 = B∥B∥ = r22.

Gram-Schmidt a1 an

=

q1 q2

[r11 r120 r22

]i.e.

a1 = r11q1

a2 = r12q1 + r22q2

2. Householder algorithm : I − 2uuT

- used in MATLAB and popular numerical linear algebra package.

The great virtue of Q is its stability.

Qx = b is perfectly conditioned since ∥x∥ = ∥b∥ and an error △b

produces an error △x of the same size:

Q(x +△x) = b +△b gives Q(△x) = △b and ∥△x∥ = ∥△b∥.



Singular Value Decomposition

Motivation: If Am×n is symmetric positive definite, QΛQT .

If a full-rank matrix A is not symmetric, furthermore, not square i.e. general

m× n, what can we do?

A︸︷︷︸m×n

= U︸︷︷︸m×m

Σ︸︷︷︸m×n

V T︸︷︷︸n×n

with UTU = I and V TV = I.

Note that ATA is symmetric positive definite.ATA︸︷︷︸n×n

=(UΣV T )T (UΣV T ) = V ΣTUTUΣV T

= V︸︷︷︸n×n

ΣTΣ︸︷︷︸n×n

V T︸︷︷︸n×n

= V

σ21

. . .

σ2n

V T

:=QΛQT = Q

λ. . .

λ

QT with Q = V, λi = σ2.

It is an eigenvalue decomposition of ATA.

From AV = UΣ Avi = σiui ⇒ ui = Avi/σi.

Since AAT = (UΣV T )(UΣV T )T = UΣΣTUT ,

ui are orthonormal eigenvectors of AAT .

Reduced SVD :

(rank r case)

Am×n = Um×r︸︷︷︸left singular vector

Σr×r UTr×n︸︷︷︸

right singular vector

=

u1 · · · ur

σ1

. . .

σr

vT

1...

vTr

with singular values σ1 ≥ σ2 ≥ · · ·σr > 0.

MATLAB command: svd(A, 0).

To complete v’s, add any orthogonal basis vr+1, · · · , vn for nullspace of

A.



Figure 9: Reduced SVD of the full-rank A

To complete u’s, add any orthogonal basis ur+1, · · · , um for nullspace of

AT .

To complete Σ to an m by n matrix, add zeros.

Full SVD :

Am×n = Um×mΣm×nUTn×n

=

u1 · · · ur ur+1 · · · um

σ1

. . .

σr

vT1...

vTr

vTr+1...

vTn

MATLAB command: svd(A).

Figure 10: Full SVD of the full-rank A

A = u1σ1v1 + u2σ2v2 + · · ·urσrvr

Avj =

{σjuj for j ≤ r,

0 for j > r.ATuj =

{σjvj for j ≤ r,

0 for j > r.



Figure 11: The transformation A in terms of SVD: ‘Fundamental Theorem of Linear Algebra’ byStrang.

Example 16 Find the SVD for A =

[1 1

7 7

].

Sol) ATA =

[1 7

1 7

] [1 1

7 7

]=

[50 50

50 50

].

det(ATA− λI) =

∣∣∣∣ 50− λ 50

50 50− λ

∣∣∣∣ = (50− λ)2 − 502

=λ2 − 100λ = λ(λ− 100).

λ = 100, 0 with eigenvectors [v1, v2] =1√2

[1 1

1 −1

].

Now σ = 10, 0:

u1 = Av1/σ1 = 110

[1 7

1 7

]1√2

[1

1

]= 1

10√2

[2

14

]= 1

5√2

[1

7

].

u2 = 1

5√2

[−71

]since uT

2u1 = 0 and ∥u2∥ = 1.

A = UΣV T =1

5√2

[1 −77 1

] [10 0

0 0

]1√2

[1 1

1 −1

](full)

=1

5√2

[1

7

]10

1√2

[1 1

](reduced)

.



Example 17 SVD of the n + 1 by n backward difference matrix △−.vk and uk are DST and DCT matrices. i.e. △− = (DCT)Σ(DST)T .

Thus △−(DST)T = (DCT)Σ.

cf)(sinnπx)′ = nπ cosnπx.

• The Pseudoinverse

If A = QΛQT = Q

λ1. . .

λn

QT with full lank, AQ = QΛ.

A−1 = QΛ−1QT = Q

1/λ1. . .

1/λn

QT , A−1Q = QΛ−1.

If Aqi = λqi, A−1qi = 1/λiqi.

Similarly, Avi = σiui, then A−1ui = 1/σivi.

For a square and invertible A,

if A = UΣV T then A−1 = V Σ−1UT .

Now, if A is nonsquare or singular?

Pseudoinverse A+ = V Σ+UT , A+ui =

vi

σi

for i ≤ r

0 for i > r

Example 18 Find the pseudoinverse A+ of A =

[1 1

7 7

].

Sol)



A+ = V Σ+UT =1√2

[1 −11 1

] [1/10 0

0 0

]1

5√2

[1 7

−7 1

](full)

=1√2

[1

1

]1/10

1

5√2

[1 7

](reduced)

=1/100

[1 7

1 7

] .

• The Condition Number and Norm

The condition number c(K) = λmaxλmin

for symmetric positive definite K.

It measures the “sensitivity” of the linear system Ku = f .

△f : measurement error, roundoff etc.

Ku = f K(u +△u) = f +△f.

The error equation: K△u = △f ⇒△u = K−1△f

∥△u∥ ≤ λmax(K−1)∥△f∥ =

1

λmin(K)∥△f∥,

since it is maximized when △f = qmin. i.e. K−1qmin =

1λmin(K)

qmin.

The λmin indicates how close K to a singular matrix.

With c >> 1, λmin(cK) = cλmin(K), it is far away from singular.

But if we multiply K by 1000 for examle, then u and△u should be divided

by 1000. That rescaling to make K less singular and λmin larger cannot

change the reality of the problem.

The relative error ∥△u∥∥u∥ stays the same.

∥△u∥ ≤∥△f∥λmin(K)

.

∥f∥ = ∥Ku∥ ≤ λmax(K)∥u∥ ⇒1

∥u∥≤

λmax(K)

∥f∥.



∥△u∥∥u∥

≤λmax(K)

λmin(K)

∥△f∥∥f∥

.

Condition number for

symmetric positive definite Kc(K) =

λmax(K)

λmin(K)

Definition 19 Matrix norm (induced)

∥A∥ = maxx=0

∥Ax∥∥x∥

= max∥x∥=1

∥Ax∥

Remark The norm of a matrix measures the maximum stretching the matrix

does to any vector.

Figure 12: Matrix norm

Definition 20 Condition number

c(A) = ∥A∥∥A−1∥

∥Ax∥∥x∥

≤ ∥A∥ for all x = 0⇒ ∥Ax∥ ≤ ∥A∥∥x∥.

∥AB∥ ≤ ∥A∥∥B∥ and ∥A + B∥ ≤ ∥A∥+ ∥B∥.



∥A∥2 = maxx =0

∥Ax∥2

∥x∥2= max

x=0

xTATAx

xTx︸︷︷︸Raleigh quotient

= λmax(ATA) = σ2

max.

c(A) = ∥A∥∥A−1∥ =σmax

σmin

.

1 = ∥I∥ = ∥AA−1∥ ≤ ∥A∥∥A−1∥.λmax(A) ≤ σmax(A).

Example 21 A =

[0 2

0 0

].

|A− λI| =∣∣∣∣ −λ 2

0 −λ

∣∣∣∣ = λ2 = 0, λmax = 0.

ATA =

[0 0

2 0

] [0 2

0 0

]=

[0 0

0 4

], σmax = 2 = ∥A∥.


Ch 1. Applied Linear Algebra - Yonsei University · 2014-12-30 · Ch 1. Applied Linear Algebra 2...

Documents

Transcript of Ch 1. Applied Linear Algebra - Yonsei University · 2014-12-30 · Ch 1. Applied Linear Algebra 2...