Linear algebra & Numerical Analysisdom033/predmety/parisLA/02_direct...Elementary operations...
Transcript of Linear algebra & Numerical Analysisdom033/predmety/parisLA/02_direct...Elementary operations...
![Page 1: Linear algebra & Numerical Analysisdom033/predmety/parisLA/02_direct...Elementary operations DEFINITION 1. interchange of 2 equations, say “interchange the i-th and j-th equations"](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e81a839ff55da6d293ebbfb/html5/thumbnails/1.jpg)
Linear algebra
& Numerical Analysis
Direct methods
Marta Jarošová http://homel.vsb.cz/~dom033/
![Page 2: Linear algebra & Numerical Analysisdom033/predmety/parisLA/02_direct...Elementary operations DEFINITION 1. interchange of 2 equations, say “interchange the i-th and j-th equations"](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e81a839ff55da6d293ebbfb/html5/thumbnails/2.jpg)
Outline
Gaussian elimination
LU factorization
Pivoting
Cholesky factorization
Solving the systems of linear
equations
![Page 3: Linear algebra & Numerical Analysisdom033/predmety/parisLA/02_direct...Elementary operations DEFINITION 1. interchange of 2 equations, say “interchange the i-th and j-th equations"](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e81a839ff55da6d293ebbfb/html5/thumbnails/3.jpg)
System of linear equations
![Page 4: Linear algebra & Numerical Analysisdom033/predmety/parisLA/02_direct...Elementary operations DEFINITION 1. interchange of 2 equations, say “interchange the i-th and j-th equations"](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e81a839ff55da6d293ebbfb/html5/thumbnails/4.jpg)
Elementary operations
DEFINITION
1. interchange of 2 equations,
say “interchange the i-th and j-th equations"
2. multiply an equation by a non-zero constant,
say “multiply the k-th equation by c≠0 “
3. add a constant multiple of one equation to
another equation,
say “add c time multiple of the i-th equation to j-th
equation”
![Page 5: Linear algebra & Numerical Analysisdom033/predmety/parisLA/02_direct...Elementary operations DEFINITION 1. interchange of 2 equations, say “interchange the i-th and j-th equations"](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e81a839ff55da6d293ebbfb/html5/thumbnails/5.jpg)
Elementary row operations
DEFINITION
1. interchange of 2 rows,
say “interchange the i-th and j-th rows"
2. multiply a row by a non-zero constant,
say “multiply the k-th row by c≠0 “
3. add a constant multiple of one row to another row,
say “add c time multiple of the i-th row to j-th row”
![Page 6: Linear algebra & Numerical Analysisdom033/predmety/parisLA/02_direct...Elementary operations DEFINITION 1. interchange of 2 equations, say “interchange the i-th and j-th equations"](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e81a839ff55da6d293ebbfb/html5/thumbnails/6.jpg)
Gaussian elimination
Two steps:
1. Forward elimination – reduces a given system to
either triangular or echelon form
2. Back substitution – finds the solution of the
system above
Echelon matrix
![Page 7: Linear algebra & Numerical Analysisdom033/predmety/parisLA/02_direct...Elementary operations DEFINITION 1. interchange of 2 equations, say “interchange the i-th and j-th equations"](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e81a839ff55da6d293ebbfb/html5/thumbnails/7.jpg)
Gaussian elimination – example
Example:
Solution:
augmented matrix
![Page 8: Linear algebra & Numerical Analysisdom033/predmety/parisLA/02_direct...Elementary operations DEFINITION 1. interchange of 2 equations, say “interchange the i-th and j-th equations"](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e81a839ff55da6d293ebbfb/html5/thumbnails/8.jpg)
Gaussian elimination – example
Forward elimination
![Page 9: Linear algebra & Numerical Analysisdom033/predmety/parisLA/02_direct...Elementary operations DEFINITION 1. interchange of 2 equations, say “interchange the i-th and j-th equations"](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e81a839ff55da6d293ebbfb/html5/thumbnails/9.jpg)
Gaussian elimination – example
Back substitution
![Page 10: Linear algebra & Numerical Analysisdom033/predmety/parisLA/02_direct...Elementary operations DEFINITION 1. interchange of 2 equations, say “interchange the i-th and j-th equations"](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e81a839ff55da6d293ebbfb/html5/thumbnails/10.jpg)
LU factorization
Let us consider the factorization
with an upper triangular matrix and a lower triangular
matrix with a unit diagonal.
1 1 1 1
0 1 2 3
0 2 5 9
0 3 9 19
1 1 1 1
0 1 2 3
0 0 1 3
0 0 3 10
k-th column
![Page 11: Linear algebra & Numerical Analysisdom033/predmety/parisLA/02_direct...Elementary operations DEFINITION 1. interchange of 2 equations, say “interchange the i-th and j-th equations"](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e81a839ff55da6d293ebbfb/html5/thumbnails/11.jpg)
LU factorization
multipliers
is k-th vector of standard basis in
![Page 12: Linear algebra & Numerical Analysisdom033/predmety/parisLA/02_direct...Elementary operations DEFINITION 1. interchange of 2 equations, say “interchange the i-th and j-th equations"](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e81a839ff55da6d293ebbfb/html5/thumbnails/12.jpg)
LU factorization
for
for
pivots
![Page 13: Linear algebra & Numerical Analysisdom033/predmety/parisLA/02_direct...Elementary operations DEFINITION 1. interchange of 2 equations, say “interchange the i-th and j-th equations"](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e81a839ff55da6d293ebbfb/html5/thumbnails/13.jpg)
LU factorization
Example:
1 1 1 1
0 1 2 3
0 2 5 9
0 3 9 19
1 1 1 1
0 1 2 3
0 0 1 3
0 0 3 10
k-th column
1 0 0 0
0 1 0 0
0 -2 1 0
0 -3 0 1
=
![Page 14: Linear algebra & Numerical Analysisdom033/predmety/parisLA/02_direct...Elementary operations DEFINITION 1. interchange of 2 equations, say “interchange the i-th and j-th equations"](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e81a839ff55da6d293ebbfb/html5/thumbnails/14.jpg)
LU factorization
And what about computing the inverse matrices of ?
Isn’t it expensive?
![Page 15: Linear algebra & Numerical Analysisdom033/predmety/parisLA/02_direct...Elementary operations DEFINITION 1. interchange of 2 equations, say “interchange the i-th and j-th equations"](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e81a839ff55da6d293ebbfb/html5/thumbnails/15.jpg)
LU factorization
To obtain it is enough to multiply off-diagonal elements by -1.
So the matrix L can be constructed directly since
![Page 16: Linear algebra & Numerical Analysisdom033/predmety/parisLA/02_direct...Elementary operations DEFINITION 1. interchange of 2 equations, say “interchange the i-th and j-th equations"](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e81a839ff55da6d293ebbfb/html5/thumbnails/16.jpg)
Example
![Page 17: Linear algebra & Numerical Analysisdom033/predmety/parisLA/02_direct...Elementary operations DEFINITION 1. interchange of 2 equations, say “interchange the i-th and j-th equations"](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e81a839ff55da6d293ebbfb/html5/thumbnails/17.jpg)
Algorithm
function [L,U] = my_lu(A)
n = size(A, 1); I = eye(n); L = I; U = A;
for k=1:n-1
L(k+1:n,k) = U(k+1:n,k)/U(k,k); %multipliers
for j = k+1:n
U(j,k:n) = U(j,k:n)-L(j,k)*U(k,k:n); %rows
end
end
function [L,U] = my_lu2(A)
n = size(A, 1); I = eye(n); L = I; U = A;
for k=1:n-1
L(k+1:n,k) = U(k+1:n,k)/U(k,k); %multipliers
U(k+1:n,k:n) = U(k+1:n,k:n)-L(k+1:n,k)*U(k,k:n); %block
end
![Page 18: Linear algebra & Numerical Analysisdom033/predmety/parisLA/02_direct...Elementary operations DEFINITION 1. interchange of 2 equations, say “interchange the i-th and j-th equations"](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e81a839ff55da6d293ebbfb/html5/thumbnails/18.jpg)
Algorithm
function [L,U] = my_lu3(A)
n = size(A, 1); I = eye(n); O = zeros(n);
L = I; U = O;
for k = 1:n
if k == 1
v(k:n) = A(k:n,k);
else
z = L(1:k-1,1:k-1)\A(1:k-1,k);
U(1:k-1,k) = z;
v(k:n) = A(k:n,k)-L(k:n,1:k-1)*z;
end
if k < n, L(k+1:n,k) = v(k+1:n)/v(k); end
U(k,k) = v(k);
end
![Page 19: Linear algebra & Numerical Analysisdom033/predmety/parisLA/02_direct...Elementary operations DEFINITION 1. interchange of 2 equations, say “interchange the i-th and j-th equations"](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e81a839ff55da6d293ebbfb/html5/thumbnails/19.jpg)
Which implementation is more efficient?
tic/toc functions work together to measure elapsed time
A = pascal(1000);
tic; lu(sparse(A),0); toc 0.7754 sec
time in sec my_lu(A) my_lu2(A) my_lu3(A)
11.4791
11.1275
11.1100
11.1042
11.1608
5.9897
5.9620
5.9940
5.9959
6.0212
2.8146
2.8878
2.9164
2.9275
2.9194
aver 11.1963 5.9926 2.8931
![Page 20: Linear algebra & Numerical Analysisdom033/predmety/parisLA/02_direct...Elementary operations DEFINITION 1. interchange of 2 equations, say “interchange the i-th and j-th equations"](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e81a839ff55da6d293ebbfb/html5/thumbnails/20.jpg)
Alternative implementation
Since L11 = 1 we obtain U1k from (Eq1).
From (Eq2)
Since Lkk = 1 we have
(Eq1)
(Eq2)
and
![Page 21: Linear algebra & Numerical Analysisdom033/predmety/parisLA/02_direct...Elementary operations DEFINITION 1. interchange of 2 equations, say “interchange the i-th and j-th equations"](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e81a839ff55da6d293ebbfb/html5/thumbnails/21.jpg)
Computational complexity
System of n equations and n unknowns
1. Forward elimination: in k-th step: (n-k) divisions to evaluate
multipliers and (n-k)(n-k+1) operations (+) and (*) to evaluate
elements aij
2. Back substitution – to evaluate k-th unknown needs one
division and (n-k) operations (+) and (*)
For large n dominates the term
(+) additions and subtractions
(*) multiplications
![Page 22: Linear algebra & Numerical Analysisdom033/predmety/parisLA/02_direct...Elementary operations DEFINITION 1. interchange of 2 equations, say “interchange the i-th and j-th equations"](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e81a839ff55da6d293ebbfb/html5/thumbnails/22.jpg)
Pivoting
In the case of a zero pivot element interchanging rows or
columns is necessary.
The strategy which place a "good" element in the diagonal
position (k,k) prior to a particular operation is called
partial pivoting - the interchanging of rows
full pivoting - the interchanging of both rows and columns
![Page 23: Linear algebra & Numerical Analysisdom033/predmety/parisLA/02_direct...Elementary operations DEFINITION 1. interchange of 2 equations, say “interchange the i-th and j-th equations"](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e81a839ff55da6d293ebbfb/html5/thumbnails/23.jpg)
Partial pivoting
We consider the LU factorization in the form
PA = LU
where P is permutation matrix.
to place the element (p,k) to the position (k,k) we have to
interchange the rows k and p
p is row index of the element which is maximal in absolute
value of
![Page 24: Linear algebra & Numerical Analysisdom033/predmety/parisLA/02_direct...Elementary operations DEFINITION 1. interchange of 2 equations, say “interchange the i-th and j-th equations"](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e81a839ff55da6d293ebbfb/html5/thumbnails/24.jpg)
Algorithm: LU with partial pivoting
function [L,U,P] = my_lu_piv(A)
n = size(A,1); I = eye(n); O = zeros(n); L = I; U = O; P = I;
function change_rows(k,p) …
function change_L(k,p) …
for k = 1:n
if k == 1, v(k:n) = A(k:n,k);
else
z = L(1:k-1,1:k -1)\ A(1:k-1,k);
U(1:k-1,k) = z;
v(k:n) = A(k:n,k)-L(k:n,1:k-1)*z;
end
if k<n
x = v(k:n); p = (k-1)+find(abs(x) == max(abs(x))); % find index p
change_rows(k,p);
L(k+1:n,k) = v(k+1:n)/v(k);
if k > 1, change_L(k,p); end
end
U(k,k) = v(k);
end
end
![Page 25: Linear algebra & Numerical Analysisdom033/predmety/parisLA/02_direct...Elementary operations DEFINITION 1. interchange of 2 equations, say “interchange the i-th and j-th equations"](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e81a839ff55da6d293ebbfb/html5/thumbnails/25.jpg)
function change_rows(k,p)
x = P(k,:); P(k,:) = P(p,:); P(p,:) = x;
x = A(k,:); A(k,:) = A(p,:); A(p,:) = x;
x = v(k); v(k) = v(p); v(p) = x;
end
function change_L(k,p)
x = L(k,1:k-1); L(k,1:k-1) = L(p,1:k-1);
L(p,1:k-1) = x;
end
![Page 26: Linear algebra & Numerical Analysisdom033/predmety/parisLA/02_direct...Elementary operations DEFINITION 1. interchange of 2 equations, say “interchange the i-th and j-th equations"](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e81a839ff55da6d293ebbfb/html5/thumbnails/26.jpg)
Full pivoting
We consider the LU factorization in the form
PAQ = LU
where P and Q are permutation matrices.
to place the element (p,q) to the position (k,k) we have to
interchange the rows k and p and columns k and q
p/q are row/column indices of the element which is maximal
in absolute value of
![Page 27: Linear algebra & Numerical Analysisdom033/predmety/parisLA/02_direct...Elementary operations DEFINITION 1. interchange of 2 equations, say “interchange the i-th and j-th equations"](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e81a839ff55da6d293ebbfb/html5/thumbnails/27.jpg)
MATLAB function: lu
[L,U]=lu(A) stores an upper triangular matrix in U and a
"psychologically lower triangular matrix" (i.e. a product of
lower triangular and permutation matrices) in L, so that A =
L*U. A can be rectangular.
[L,U,P]=lu(A) returns unit lower triangular matrix L,
upper triangular matrix U, and permutation matrix P so that
P*A = L*U.
[L,U]=lu(sparse(A),0) no pivoting
For more details see: help lu
![Page 28: Linear algebra & Numerical Analysisdom033/predmety/parisLA/02_direct...Elementary operations DEFINITION 1. interchange of 2 equations, say “interchange the i-th and j-th equations"](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e81a839ff55da6d293ebbfb/html5/thumbnails/28.jpg)
Cholesky factorization
We consider the factorization of a positive definite matrix A
with a lower triangular matrix .
Cholesky factorization can be generalized for positive semi-definite matrices.
![Page 29: Linear algebra & Numerical Analysisdom033/predmety/parisLA/02_direct...Elementary operations DEFINITION 1. interchange of 2 equations, say “interchange the i-th and j-th equations"](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e81a839ff55da6d293ebbfb/html5/thumbnails/29.jpg)
Algorithm
function L = my_chol(A)
n = size(A,1); O = zeros(n);
L = O;
for k = 1:n
if k == 1
L(k,k) = sqrt(A(k,k));
L(k+1:n,k) = A(k+1:n,k)/L(k,k);
else
v = L(k,1:k-1)'; %L_k1'
L(k,k) = sqrt(A(k,k)-v'*v);
L(k+1:n,k) = (A(k+1:n,k)-L(k+1:n,1:k-1)*v)/L(k,k);
end
end
Cholesky factorization can be considered also with pivoting: PAPT = LLT.
![Page 30: Linear algebra & Numerical Analysisdom033/predmety/parisLA/02_direct...Elementary operations DEFINITION 1. interchange of 2 equations, say “interchange the i-th and j-th equations"](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e81a839ff55da6d293ebbfb/html5/thumbnails/30.jpg)
MATLAB function: chol
chol(A) uses only the diagonal and upper triangle of A. The lower
triangle is assumed to be the (complex conjugate) transpose of the upper triangle. If A is positive definite, then R = chol(A) produces an upper
triangular R so that R'*R = A. If A is not positive definite, an error message
is printed.
L = chol(A,'lower') uses only the diagonal and the lower triangle
of A to produce a lower triangular L so that L*L' = A. If A is not positive
definite, an error message is printed. When A is sparse, this syntax of
CHOL is typically faster.
[R,p] = chol(A) with two output arguments function never produces
an error message.
For more details see: help chol
![Page 31: Linear algebra & Numerical Analysisdom033/predmety/parisLA/02_direct...Elementary operations DEFINITION 1. interchange of 2 equations, say “interchange the i-th and j-th equations"](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e81a839ff55da6d293ebbfb/html5/thumbnails/31.jpg)
Solving the systems of linear equations
LU factorization To find the solution of the system Ax = b we will solve two systems with triangular matrices.
Ax = b
L (LT (Ux)) = b
L(Ux) = Pb
d = Pb, Lz = d, Ux = z
In Matlab: x = U\(L\(P*b))
PA = LU
A = PTLU
d z
![Page 32: Linear algebra & Numerical Analysisdom033/predmety/parisLA/02_direct...Elementary operations DEFINITION 1. interchange of 2 equations, say “interchange the i-th and j-th equations"](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e81a839ff55da6d293ebbfb/html5/thumbnails/32.jpg)
Solving the systems of linear equations
Cholesky factorization
The situation is similar to previous one.
Ax = b
L (LTx) = b
Lz = b, Ux = z
In Matlab: x = L'\(L\b)
A = LLT
z
![Page 33: Linear algebra & Numerical Analysisdom033/predmety/parisLA/02_direct...Elementary operations DEFINITION 1. interchange of 2 equations, say “interchange the i-th and j-th equations"](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e81a839ff55da6d293ebbfb/html5/thumbnails/33.jpg)
References
Gilbert W. Stewart, Matrix Algorithms: Basic
decompositions (available on Google books)
Lloyd Nicholas Trefethen, David Bau, Numerical linear
algebra (available on Google books)
Kozubek, Brzobohatý, Hapla, Jarošová, Markopoulos:
LINEÁRNÍ ALGEBRA S MATLABEM, http://mi21.vsb.cz/
(in Czech)
MATLAB Help