Code: 101MAT4–101MT4B Today’s topics Normed linear (vector) space Symmetric matrix … · 2020....
Transcript of Code: 101MAT4–101MT4B Today’s topics Normed linear (vector) space Symmetric matrix … · 2020....
Code: 101MAT4–101MT4B
Today’s topics◮ Linear (vector) space◮ Normed linear (vector) space◮ Matrix and vector norms◮ Symmetric matrix, positive definite matrix◮ Gaussian elimination, Cholesky factorization, condition
number
1 / 29
Vector (linear) space V : For all u, v ∈ V and all α, β ∈ R,the linear combination αu + βv belongs to V . Next,
x + y = y + x ∀x , y ∈ V , (1)
x + (y + z) = (x + y) + z ∀x , y , z ∈ V , (2)
∃! 0̃ ∈ V x + 0̃ = x , ∀x ∈ V , (3)
∀x ∈ V ∃! − x ∈ V x + (−x) = 0̃, (4)
∀α ∈ R ∀x ∈ V αx ∈ V , (5)
1x = x , (6)
∀α, β ∈ R α(βx) = (αβ)x , (7)
α(x + y) = αx + αy , (8)
(α+ β)x = αx + βx . (9)
Examples: Ordered n-tuples, matrices of the same size m × n,polynomials of the degree n, functions, etc.
Remark: Vector space over C: α, β ∈ C.
2 / 29
Normed vector space XA real vector space X is called a normed vector (linear) space,if a real number ‖x‖, called the norm, is assigned to eachx ∈ X in such a way that
‖x‖≥ 0 ∀x ∈ X , (10)
‖x + y‖≤ ‖x‖+ ‖y‖ ∀x , y ∈ X , (11)
‖αx‖= |α| ‖x‖ ∀x ∈ X , ∀α ∈ R, (12)
‖x‖= 0 ⇒ x = 0̃. (13)
The inequality (11) is equivalent to
‖u − v‖ ≤ ‖u − w‖+ ‖w − v‖ ∀u, v ,w ∈ X . (14)
The inequality (11) (and (14)) is called the triangle inequality.
Remark: If X is a space over the complex numbers, thedefinition remains unchanged except for α ∈ C.
3 / 29
Vector norms x = (x1, x2, . . . , xn) ∈ Rn, or ∈ C
n
‖x‖1 =n∑
i=1
|xi | (taxicab norm, Manhattan norm),
‖x‖2 =
(n∑
i=1
|xi |2)1/2
(Euclidean norm),
‖x‖p =
(n∑
i=1
|xi |p)1/p
(p-norm), p ≥ 1,
‖x‖∞ = maxi∈{1,2,...,n}
|xi | (max-norm).
4 / 29
We limit ourselves to real vectors and matrices.(Nevertheless, the statemets are valid for complex matrices,too.)
The norm of an m × n matrix A induced by the vector norms:
‖A‖YmXn = max{x∈Xn: x 6=0}
‖Ax‖Ym
‖x‖Xn
, (15)
where Xn ⊂ Rn and Ym ⊂ R
m are vector spaces.
If y = Ax , where x ∈ Xn and y ∈ Ym,then ‖y‖Ym ≤ ‖A‖YmXn ‖x‖Xn .
If the norm ‖ · ‖ξ , where ξ stands for 1, 2, p, or ∞, is used inboth spaces, then the induced norm is denoted by ‖A‖ξ.In special cases, the calculation of ‖A‖ξ is simpler than in (15).
5 / 29
In particular,
‖A‖1 = maxk∈{1,2,...,n}
m∑
i=1
|aik |, ‖A‖∞ = maxi∈{1,2,...,m}
n∑
k=1
|aik |,
‖A‖2 = (̺(ATA))1/2 (spectral norm).
If A is real and symmetric, then
‖A‖2 =(̺(ATA))1/2 = (̺(A2))1/2 = ̺(A).
Frobenius norm (is not induced!) ‖A‖F =
(m∑
i=1
n∑
k=1
|aik |2)1/2
.
It holds ̺(A) ≤ ‖A‖ for the Frobenius and all the inducednorms. Moreover, for the n × n identity matrix I, ‖I‖ = 1 and‖I‖F =
√n, where ‖ · ‖ is an arbitrary induced norm.
In the vector space with a finite dimension, all the norms areequivalent, that is, for instance,
∃c1, c2 > 0 ∀x ∈ Rn c1‖x‖1 ≤ ‖x‖2 ≤ c2‖x‖1.
6 / 29
Scalar (inner) product in a vector space V (over R)The map (·, ·) : V × V → R that maps pairs of the elements ofV into R is called scalar (inner) product on V if for all x , y ∈ Vand all α ∈ R the following statements are valid
(y , x) = (x , y), (16)
(x + z, y) = (x , y) + (z, y), (17)
(αx , y) = α(x , y), (x , αy) = α(x , y), (18)
(x , x) ≥ 0, (19)
(x , x) = 0 ⇔ x = 0. (20)
Moreover, ‖x‖2 =√
(x , x) defines a norm on V .
Remark: The inner product has to be defined in a partlydifferent way for the vector space over the complex numbers.
7 / 29
Inner product of real vectors
(x , y) =n∑
k=1
xkyk , (21)
where x = (x1, . . . , xn) ∈ Rn and y = (y1, . . . , yn) ∈ R
n.
The Euclidean norm on Rn is defined through ‖x‖2 =
√(x , x).
Remark: The inner product of functions(u, v) =
∫ ba u(x)v(x)dx , where u, v ∈ C([a,b]), also comply
with (16)-(20).
8 / 29
Schwarz (Cauchy) inequality
|(x , y)| ≤ ‖x‖2 ‖y‖2 ∀x , y ∈ V .
Proof: y = 0 ⇒ the statement holds. If y 6= 0, then
0 ≤∥∥∥∥∥x − (x , y)
‖y‖22
y
∥∥∥∥∥
2
2
=
(x − (x , y)
‖y‖22
y , x − (x , y)‖y‖2
2
y
)
= ‖x‖22 − 2
(x , y)2
‖y‖22
+(x , y)2
‖y‖22
= ‖x‖22 − (x , y)2
‖y‖22
.
Remark: The proof is general because it uses only theproperties (17)-(18), not a particular definition of the innerproduct! The inequality is valid for the inner product offunctions, too.
9 / 29
Positive definite matricesA symmetric matrix A = (aij) ∈ R
n×n is called positive definite,if
(Ax , x) > 0, i.e.,n∑
i=1
n∑
j=1
aijxixj > 0 (22)
for all 0 6= x ∈ Rn (if (Ax , x) ≥ 0 . . . positive semidefinite;
if > 0, < 0 for some vectors . . . indefinite)
Characterization: A matrix A is positive definite, i.e, A is s.p.d.
⇔ all the eigenvalues of A are positive
⇔ expression (22) is an inner product
⇔ the leading principal minors∗ of A are all positive(Sylvester’s criterion)
⇔ A has a unique Cholesky decomposition A = LLT
(see later), where the diagonal elements of L arepositive
∗ The k th leading principal minor of a matrix A is the determinant ofits upper-left k × k sub-matrix.
10 / 29
Examples:
A =
(9 66 5
), B =
(9 66 4
), C =
(9 66 3
),
xTAx = 9x21 + 12x1x2 + 5x2
2 = (3x1 + 2x2)2 + x2
2 , A is s.p.d.xTBx = 9x2
1 + 12x1x2 + 4x22 = (3x1 + 2x2)
2, B is positivesemidefinite, B is not s.p.d.xTCx = 9x2
1 +12x1x2 +3x22 = (3x1 +2x2)
2 − x22 , C is indefinite.
The leading principal minors of A are 9 an 9, of B are 9 and 0,of C are 9 and -9.
11 / 29
Relationship between a linear system of equations and theminimization of a quadratic function
A = (aij) ∈ Rn×n is s.p.d., b ∈ R
n is a column vector:
Let g(x) =12(Ax , x)− (b, x) and let the minimum of g is
attained at x̂ , then it follows from (22) that
x̂ = arg minx∈Rn
g(x) ⇐⇒ grad g(x̂) = 0 ⇐⇒ Ax̂ = b
12 / 29
Gauss elimination and Cholesky decomposition
The original system Ax = b is transformed to an equivalentsystem Ux = b̂, where U is the upper triangular matrix.
The transformation can be represented by a lower triangularmatrix L, then A = LU (LU factorization), that is, LUx = b. Bydefining b̂ ≡ Ux , we easily solve Lb̂ = b and Ux = b̂ byapplying the back substitution twice.
The decomposition is not that simple in practice: pivotingneeded ⇒ permutation matrices involved.
If A is positive definite, aii 6= 0 because aii = eTi Aei > 0.
If A is s.p.d., then a unique Cholesky decomposition(factorization) A = LLT exists, where L is a lower triangularmatrix.
Sparse matrix: Not more than ≈ 5% nonzero entries.
Fill-in. Band matrix. Number of operations.
13 / 29
Condition number
Let ‖ · ‖ be an induced norm and let A is nonsingular. The value
κ(A) = ‖A‖ ‖A−1‖
is called the condition number of the matrix A with respect tothe norm ‖ · ‖.
If A is symmetric and positive definite and if we use ‖ · ‖2, thenκ(A) = λmax/λmin.
Let Ax0 = b0 and Ax1 = b1, where b0 6= b1, then
‖x1 − x0‖‖x0‖
≤ κ(A)‖b1 − b0‖
‖b0‖. (23)
Moreover, b0 6= 0 and b0 6= b1 exist such that (23) becomes theequality.
14 / 29
If the condition number is large, we say that the matrix isill-conditioned.
Ill-conditioned systems might be (and often are) difficult to solvewith the desired accuracy.
15 / 29
Hilbert matrix H(n)
H(n) = (aij), where aij =1
i + j − 1, i , j = 1,2, . . . ,n.
As an example take H(3) =
1 1/2 1/31/2 1/3 1/41/3 1/4 1/5
.
Let us solve H(n)x =
1...1
for n = 2,3, . . . ,14.
16 / 29
1 2−2
0
2
4
6
Exact solution. ||Ax−b||∞ = 0.00E+00
1 2
10−15.3
10−15.1
|exact solutioni − numer. solution
i| Cond. numb. = 1.9E+01 ||Ax−b||
∞ = 2.22E−16
17 / 29
1 2 3−40
−20
0
20
40
Exact solution. ||Ax−b||∞ = 0.00E+00
1 2 310
−15
10−14
10−13
|exact solutioni − numer. solution
i| Cond. numb. = 5.2E+02 ||Ax−b||
∞ = 8.88E−16
18 / 29
1 2 3 4−200
−100
0
100
200
Exact solution. ||Ax−b||∞ = 0.00E+00
1 2 3 410
−14
10−12
10−10
|exact solutioni − numer. solution
i| Cond. numb. = 1.6E+04 ||Ax−b||
∞ = 3.55E−15
19 / 29
1 2 3 4 5−2000
−1000
0
1000
Exact solution. ||Ax−b||∞ = 0.00E+00
1 2 3 4 510
−12
10−10
10−8
|exact solutioni − numer. solution
i| Cond. numb. = 4.8E+05 ||Ax−b||
∞ = 2.84E−14
20 / 29
1 2 3 4 5 6−1
−0.5
0
0.5
1x 10
4 Exact solution. ||Ax−b||∞ = 0.00E+00
1 2 3 4 5 610
−10
10−8
10−6
|exact solutioni − numer. solution
i| Cond. numb. = 1.5E+07 ||Ax−b||
∞ = 8.53E−14
21 / 29
1 2 3 4 5 6 7−4
−2
0
2
4x 10
4 Exact solution. ||Ax−b||∞ = 0.00E+00
1 2 3 4 5 6 710
−10
10−5
100
|exact solutioni − numer. solution
i| Cond. numb. = 4.8E+08 ||Ax−b||
∞ = 1.25E−12
22 / 29
1 2 3 4 5 6 7 8−2
0
2
4x 10
5Exact solution. ||Ax−b||
∞ = 0.00E+00
1 2 3 4 5 6 7 810
−10
10−5
100
|exact solutioni − numer. solution
i| Cond. numb. = 1.5E+10 ||Ax−b||
∞ = 3.64E−12
23 / 29
1 2 3 4 5 6 7 8 9−2
−1
0
1
2x 10
6Exact solution. ||Ax−b||
∞ = 0.00E+00
1 2 3 4 5 6 7 8 910
−5
100
105
|exact solutioni − numer. solution
i| Cond. numb. = 4.9E+11 ||Ax−b||
∞ = 1.82E−11
24 / 29
1 2 3 4 5 6 7 8 9 10−1
−0.5
0
0.5
1x 10
7Exact solution. ||Ax−b||
∞ = 0.00E+00
1 2 3 4 5 6 7 8 9 1010
−5
100
105
|exact solutioni − numer. solution
i| Cond. numb. = 1.6E+13 ||Ax−b||
∞ = 1.02E−10
25 / 29
1 2 3 4 5 6 7 8 9 10 11−5
0
5x 10
7Exact solution. ||Ax−b||
∞ = 0.00E+00
1 2 3 4 5 6 7 8 9 10 1110
−5
100
105
|exact solutioni − numer. solution
i| Cond. numb. = 5.2E+14 ||Ax−b||
∞ = 9.31E−10
26 / 29
1 2 3 4 5 6 7 8 9 10 11 12−4
−2
0
2x 10
8Exact solution. ||Ax−b||
∞ = 0.00E+00
1 2 3 4 5 6 7 8 9 10 11 1210
0
105
1010
|exact solutioni − numer. solution
i| Cond. numb. = 1.7E+16 ||Ax−b||
∞ = 2.79E−09
27 / 29
1 2 3 4 5 6 7 8 9 10 11 12 13−2
−1
0
1
2x 10
9Exact solution. ||Ax−b||
∞ = 0.00E+00
1 2 3 4 5 6 7 8 9 10 11 12 1310
0
105
1010
|exact solutioni − numer. solution
i| Cond. numb. = 1.8E+18 ||Ax−b||
∞ = 2.51E−08
28 / 29
1 2 3 4 5 6 7 8 9 10 11 12 13 14−1
−0.5
0
0.5
1x 10
10Exact solution. ||Ax−b||
∞ = 0.00E+00
1 2 3 4 5 6 7 8 9 10 11 12 13 1410
0
105
1010
|exact solutioni − numer. solution
i| Cond. numb. = 3.1E+17 ||Ax−b||
∞ = 1.34E−07
29 / 29