Post on 10-Sep-2019
Topics
Review of lecture 2/11Error, Residual and Condition Number
Review of lecture 2/16Backward Error AnalysisThe General Case
1 / 22
Theorem (Calculation of 2 norm of a symmetric matrix)If A = At is symmetric then ‖A‖2 is given by
‖A‖2 = max{|λ| : λ is an eigenvalue of A}.
Theorem (Matrix 1-norm and∞-norm)We have
‖A‖∞ = max1≤i≤N
N∑j=1
|aij |,
‖A‖1 = max1≤j≤N
N∑i=1
|aij |.
2 / 22
Topics
Review of lecture 2/11Error, Residual and Condition Number
Review of lecture 2/16Backward Error AnalysisThe General Case
3 / 22
? Approximate solution x̂ of Ax = b
error : = e = x − x̂ ,residual : = r = b − Ax̂ .
? Small residual need not imply small errorthe point P = (0.5,0.7) and the 2× 2 linear system
x − y = 0−0.8x + y = 1/2.
? Plot is two almost-parallel lines with x̂ between them, far from thesolution.
4 / 22
Justification of picture
I Normalize each equation in the system so that√
a211 + a2
12 = 1
I Normal distance from x̂ to the line is r1 = b1 − a11x̂1 − a12x̂2
I Hence ‖r‖2 =square root of sum of squares of normal distancefrom x̂ to lines.
I Same for 3D and higher dimensions
5 / 22
Recall qualitative facts
I If A is invertible then r = 0 if and only if e = 0.I If A is well conditioned then ||r || and ||e|| are comparable in size.I If A is ill conditioned then ||r || can be small while ||e|| is large.I det(A) is not the right way to quantify this connection.
6 / 22
Condition number
DefinitionLet ‖ · ‖ be a matrix norm. Then the condition number of the matrix Ainduced by the vector norm ‖ · ‖ is
cond‖·‖(A) := ‖A−1‖‖A‖.
? Usually use κ
κ(A) = cond(A) = condition number of A.
7 / 22
Matlab
I cond(A) or cond(A,p)I condest(A)I rcond(A) = reciprocal condition number
8 / 22
Theorem (Relative Error ≤ cond(A)×Relative Residual)Let Ax = b and let x̂ be an approximation to the solution x. Withr = b − Ax̂
‖x − x̂‖‖x‖
≤ cond(A)‖r‖‖b‖
.
9 / 22
Proof.I Ae = r so e = A−1r
‖e‖ = ‖A−1r‖ ≤ ‖A−1‖‖r‖. (1)
I Ax = b so ‖b‖ = ‖Ax‖ ≤ ‖A‖‖x‖.I Dividing the smaller side of (1) by the larger quantity and the
larger side of (1) by the smaller gives
‖e‖‖A‖‖x‖
≤ ‖A−1‖‖r‖‖b‖
.
I Rearrangement proves the theorem.
10 / 22
I ‖I‖ = 1 so cond(I) = 1 in any induced matrix norm. Similarly foran orthogonal matrix cond2(O) = 1.
I Example: error = (cond(A))*residual
11 / 22
Topics
Review of lecture 2/11Error, Residual and Condition Number
Review of lecture 2/16Backward Error AnalysisThe General Case
12 / 22
Example (The Hilbert Matrix)The N × N Hilbert matrix HN×N is the matrix with entries
Hij =1
i + j − 1, 1 ≤ i , j ≤ n.
This matrix is extremely ill conditioned even for quite moderate valueson n.
1 12
13
14
12
13
14
15
13
14
15
16
14
15
16
17
13 / 22
Topics
Review of lecture 2/11Error, Residual and Condition Number
Review of lecture 2/16Backward Error AnalysisThe General Case
14 / 22
[Basic Result of Backward Error Analysis] The result x̂ ofsolving Ax = b by Gaussian elimination in finite precisionarithmetic subject to rounding errors is precisely the same asthe exact solution of a perturbed problem
(A + E)x̂ = b + f (2)
where‖E‖||A||
,‖f‖||b||
= O(machine precision).
15 / 22
Theorem (Effect of Storage errors in A)Let A be an N × N matrix. Suppose Ax = b and (A + E)x̂ = b. Then,
‖x − x̂‖‖x̂‖
≤ cond(A)‖E‖‖A‖
.
Theorem (Effect of Storage Errors in b)Let Ax = b and Ax̂ = b + f . Then
‖x − x̂‖‖x‖
≤ cond(A)‖f‖‖b‖
.
16 / 22
Sketch proof Theorem 145Proof.Step 1: By subtraction get an equation for the error driven by theperturbation:
Ax = b(A + E)x̂ = b
___subtract____A(x − x̂) = Ex̂
x − x̂ = A−1Ex̂
Step 2: Take norms
||x − x̂ || = ||A−1Ex̂ || ≤ ||A−1|| · ||E || · ||x̂ || ‖A‖‖A‖
Step 3: rearrange using def of cond no
‖x − x̂‖‖x̂‖
≤ cond(A)‖E‖‖A‖
.
17 / 22
Sketch proof Theorem 146
Proof.I Since Ax̂ = Ax + f , x − x̂ = −A−1fI ‖x − x̂‖ ≤ ‖A−1‖‖f‖ = cond(A) ‖f‖‖A‖I ‖x − x̂‖ ≤ cond(A) ‖f‖‖b‖‖x‖ because ‖b‖ ≤ ‖A‖‖x‖.
18 / 22
Facts about condition number
I When E is due to roundoff errors ‖E‖/‖A‖ = O(machineprecision). cond(A) tells you how many significant digits arelost (worst case) when solving Ax = b
I cond(A) ≥ 1 and cond(I) = 1.I Scaling A does not influence cond(A):I cond(A) depends on the norm chosen but usually it is of the
same order of magnitude for different norms.I If A is symmetric then
cond2(A) = |λ|max/|λ|min.
I If A is symmetric, positive definite and ‖ · ‖ = ‖ · ‖2, then cond(A)equals the spectral condition number, λmax/λmin
cond2(A) = λmax/λmin.
I cond(A) = cond(A−1).
19 / 22
Topics
Review of lecture 2/11Error, Residual and Condition Number
Review of lecture 2/16Backward Error AnalysisThe General Case
20 / 22
Working toward the general case
Lemma (Spectral localization)For any N × N matrix B and ‖ · ‖ any matrix norm:
|λ(B)| ≤ ‖B‖.
Proof.For an eigenpair (λ, φ), Bφ = λφ.Thus |λ|‖φ‖ = ‖Bφ‖ ≤ ‖B‖‖φ‖.
21 / 22
Theorem 149
Theorem
limn→∞
Bn = 0 if and only if there exists a norm ‖ · ‖ with ‖B‖ < 1.
Proof.I We prove ‖B‖ < 1⇒ ‖Bn‖ ≤ ‖B‖n → 0I Start the induction with n = 2,
‖B2‖ = ‖B · B‖ ≤ ‖B‖ · ‖B‖ ≤ ‖B‖2.
I Assume ‖Bn‖ ≤ ‖B‖n, then
‖Bn+1‖ = ‖BnB‖ ≤ ‖Bn‖‖B‖ ≤ ‖B‖n+1
I ‖Bn‖ ≤ ‖B‖n → 0 as n→∞.
22 / 22