Topic 3a -- Review of Linear Algebra - University of Texas …emlab.utep.edu/ee4386_5301cmee/Topic...
Transcript of Topic 3a -- Review of Linear Algebra - University of Texas …emlab.utep.edu/ee4386_5301cmee/Topic...
2/12/2018
1
Topic 3a –Review of Linear Algebra
EE 4386/5301 Computational Methods in EE
Course InstructorDr. Raymond C. RumpfOffice: A‐337Phone: (915) 747‐6958E‐Mail: [email protected]
Outline
• Solving systems of equations
• Matrix terminology and special matrices
• Matrix operations
• Common linear algebra problems
2Topic 3a ‐‐ Review of Linear Algebra
2/12/2018
2
Solving Systems of Equations
3Topic 3a ‐‐ Review of Linear Algebra
Systems of Linear Equations
Very often in science and engineering, problems can be reduced to a system of linear equations.
11 1 12 2 1 1
21 1 22 2 2 2
1 1 2 2
n n
n n
n n nn n n
a x a x a x b
a x a x a x b
a x a x a x b
constant coefficient (usually known)
unknown values
constants (usually excitation)
ij
i
i
a
x
b
4Topic 3a ‐‐ Review of Linear Algebra
2/12/2018
3
Direct Analytical Solution
Step 1 – Solve first equation for x1.
Suppose we wish to solve the following system of equations
11 1 12 2 13 3 1
21 1 22 2 23 3 2
31 1 32 2 33 3 3
a x a x a x b
a x a x a x b
a x a x a x b
131 1211 1 12 2 13 3 1 1 2 3
11 11 11
21 1 22 2 23 3 2
31 1 32 2 33 3 3
ab a
a x a x a x b x x xa a a
a x a x a x b
a x a x a x b
5Topic 3a ‐‐ Review of Linear Algebra
Direct Analytical Solution
Step 2 – Forward Substitution – Substitute this new equation into 2nd
and 3rd equations to eliminate x1.
11 1 12 2 13 3 1
22 2 23 3 2
32 2 33 3 3
a x a x a x b
a x a x b
a x a x b
21 1321 12 21 122 22 23 23 2 2
11 11 11
31 12 31 13 31 132 32 33 33 3 3
11 11 11
a aa a a ba a a a b b
a a a
a a a a a ba a a a b b
a a a
6Topic 3a ‐‐ Review of Linear Algebra
2/12/2018
4
Direct Analytical Solution
Step 3 – Solve second equation for x2.
11 1 12 2 13 3 1
23222 2 23 3 2 2 3
22 22
32 2 33 3 3
a x a x a x b
aba x a x b x x
a a
a x a x b
7Topic 3a ‐‐ Review of Linear Algebra
Direct Analytical Solution
Step 4 – Forward Substitution – Substitute this new equation into 3rd
equation to eliminate x2.
11 1 12 2 13 3 1
22 2 23 3 2
33 3 3
a x a x a x b
a x a x b
a x b
32 23 32 233 33 3 3
22 22
a a a b
a a b ba a
8Topic 3a ‐‐ Review of Linear Algebra
2/12/2018
5
Direct Analytical Solution
Step 5 – Solve third equation for x3. Since this is the last equation, we get the final answer for x3.
33
33
bx
a
9Topic 3a ‐‐ Review of Linear Algebra
Direct Analytical Solution
Step 6 – Backward Substitution – Given x3, calculate x2 using equation from Step 3.
2 23 32
22
b a xx
a
10Topic 3a ‐‐ Review of Linear Algebra
2/12/2018
6
Direct Analytical Solution
Step 7 – Backward Substitution – Given x2 and x3, calculate x1 using equation from Step 1.
1 12 2 13 31
11
b a x a xx
a
11Topic 3a ‐‐ Review of Linear Algebra
Matrix Terminology& Special Matrices
12Topic 3a ‐‐ Review of Linear Algebra
2/12/2018
7
Systems of Linear Equations
Systems of equations can be written in matrix form.
11 1 12 2 1 1
21 1 22 2 2 2
1 1 2 2
n n
n n
n n nn n n
a x a x a x b
a x a x a x b
a x a x a x b
11 12 1 1 1
21 22 2 2 2
1 2
n
n
n n nn n n
a a a x b
a a a x b
a a a x b
or
A
A
or
x
x
or
b
b
13Topic 3a ‐‐ Review of Linear Algebra
Rows, Columns, and Diagonals
The center diagonal is usually just called the diagonal.
The elements along the diagonal are sometimes called the pivot elements.
14Topic 3a ‐‐ Review of Linear Algebra
2/12/2018
8
Special Matrices (1 of 2)
Symmetric Matrix
1 2 9 4
2 6 5 8
9 5 7 0
4 8 0 3
A
Diagonal Matrix
1 0 0 0
0 6 0 0
0 0 7 0
0 0 0 3
A
Identity Matrix
1 0 0 0
0 1 0 0
0 0 1 0
0 0 0 1
I
Zero Matrix
0 0 0 0
0 0 0 00
0 0 0 0
0 0 0 0
15Topic 3a ‐‐ Review of Linear Algebra
Bandwidth of 3
Special Matrices (2 of 2)
Upper Triangular Matrix
1 2 9 4
0 6 5 8
0 0 7 5
0 0 0 3
A
Lower Triangular Matrix
1 0 0 0
2 6 0 0
9 5 7 0
4 8 1 3
A
Banded Matrix
1 2 0 0
4 6 5 0
0 8 7 5
0 0 10 3
A
16Topic 3a ‐‐ Review of Linear Algebra
Vandermonde Matrix2
1 1 12
2 2 22
3 3 3
21 1 1
1
1
1
1
N
N
N
NN N N
x x x
x x x
x x x
x x x
Arises when curve fitting to polynomials.Usually ill‐conditioned for large matrices.
Triangular matrices can be thought of as “almost” solved matrices. They are very fast to solve.
2/12/2018
9
Block Matrices
Block matrices are “matrices of matrices.”
11 12 11 12 11 12 11 12
21 22 21 22 21 22 21 22
a a b b c c d d
A B C Da a b b c c d d
11 12 11 12
21 22 21 22
11 12 11 12
21 22 21 22
a a b b
a a b bA BF
C D c c d d
c c d d
17Topic 3a ‐‐ Review of Linear Algebra
Sparse Matrices
Many matrices contain 99.9% zeros.
It is not efficient use of memory to store all these zeros. Instead, we store only the non‐zero elements along with their positions in the matrix.
The opposite of a “sparse” matrix is a “full” matrix.
18Topic 3a ‐‐ Review of Linear Algebra
2/12/2018
10
Matrix Problem Size
# Equations > # Unknowns # Equations = # Unknowns # Equations < # Unknowns
Usually occurs when the equations are derived from samples.
Solution is obtained as a best fit and is not exact.
Applications• Curve fitting
Most usual case.
Many standard algorithms exist to obtain an exact solution.
Applications• Circuit theory• Solving ODEs
Usually occurs when little is known about the problem or solution.
Solution is obtained by optimization and is not exact.
Applications• Topology optimization
19Topic 3a ‐‐ Review of Linear Algebra
Health of a Matrix (1 of 3)
Is this system of equations solvable?
2 8 1 2 1 8
2 8 1 2 1 8
3 4 3 1 1 4
x y z x
x y z y
x y z z
No!The 1st and 2nd equations are the same. The 2nd equation does not provide any new information to the problem.
2 8 1 2 1 8
2 4 2 16 2 4 2 16
3 4 3 1 1 4
x y z x
x y z y
x y z z
2 8 1 2 1 8
4 2 12 4 1 2 12
3 4 3 1 1 4
x y z x
x y z y
x y z z
No!The 2nd equation is just 2× the 1st equation. The 2nd equation is still not providing any new information.
No!The 2nd equation is the sum of the 1st and 3rd equation, thus the 2nd equation still does not provide any new information.
20Topic 3a ‐‐ Review of Linear Algebra
2/12/2018
11
Health of a Matrix (2 of 3)
Is this system of equations solvable?
8 1 0 1 8
2 7 1 0 2 7
3 4 3 0 1 4
x z x
x z y
x z z
No!None of these equations contain any information about y.
So how do we know if a problem is solvable?
• All rows must be linearly independent – this ensures they provide new information to the problem.
• No rows can be all zeros – This would not provide any information.• No columns can be all zeros – This would be ignoring information from one of the unknowns.
is solvable if det 0A x b A
21Topic 3a ‐‐ Review of Linear Algebra
Health of a Matrix (3 of 3)
Is the following system of equations solvable?
Condition Number of a Matrix
max
min
AA
A
2 8 1 2 1 8
1.0001 2 8.0001 1.0001 2 1 8.0001
3 4 3 1 1 4
x y z x
x y z y
x y z z
Technically yes, but we would expect the solution to be somewhat “touchy” and unstable. This is an ill-conditionedmatrix.
The condition number (A) of matrix A is a measure of how numerically “stable” it is.
Matrices with high condition numbers are less stable. Small changes in the element values of A will result in large changes in the elements of b.
22Topic 3a ‐‐ Review of Linear Algebra
min
max
smallest singular value of
largest singular value of
A A
A A
2/12/2018
12
Example: Condition Number
What is the condition number?
23Topic 3a ‐‐ Review of Linear Algebra
1 2 1
1 2 1
3 1 1
A 165.84 10 A
1 2 1
1 0 1
3 1 1
A 7.76 A
1 2 1
1.0001 2 1
3 1 1
A 51.4 10 A
1 2 1
1.01 2 1
3 1 1
A 31.4 10 A
Matrix Operations
24Topic 3a ‐‐ Review of Linear Algebra
2/12/2018
13
Matrix Math (1 of 4)
Addition:
11 12 13 11 12 13 11 11 12 12 13 13
21 22 23 21 22 23 21 21 22 22 23 23
31 32 33 31 32 33 31 31 32 32 33 33
a a a b b b a b a b a b
A B a a a b b b a b a b a b
a a a b b b a b a b a b
11 12 13 11 12 13 11 11 12 12 13 13
21 22 23 21 22 23 21 21 22 22 23 23
31 32 33 31 32 33 31 31 32 32 33 33
a a a b b b a b a b a b
A B a a a b b b a b a b a b
a a a b b b a b a b a b
Subtraction:
25Topic 3a ‐‐ Review of Linear Algebra
Matrix Math (2 of 4)
Multiplication by a Scalar:
11 12 13 11 12 13
21 22 23 21 22 23
31 32 33 31 32 33
a a a sa sa sa
s A s a a a sa sa sa
a a a sa sa sa
11 12 13 11 12 13 11 11 12 21 13 31
21 22 23 21 22 23
31 32 33 31 32 33
11 12 13 1 11 1 12 2 13 3
21 22 23 2 21 1 22
31 32 33 3
# #
# # #
# # #
a a a b b b a b a b a b
A B a a a b b b
a a a b b b
a a a x a x a x a x
A x a a a x a x a x
a a a x
2 23 3
31 1 32 2 33 3
a x
a x a x a x
Multiplication by a Matrix
26Topic 3a ‐‐ Review of Linear Algebra
2/12/2018
14
Matrix Math (3 of 4)
Matrix Transpose:
11 12 13 11
21 22 23 22
31 32 3
21 31
12 32
13 2 333 3
T
T
a a a a
A a a a a
a a a a
a a
a a
a a
* *21 31
* *1
**
11 12 13 11* *
21 22 23 222 32* *1
*31 32 33 333 23
T
H
ij ji
a a
a a
a a
a a a a
A a a a a a a
a a a a
Hermitian Transpose:
27Topic 3a ‐‐ Review of Linear Algebra
ij jia a
Animation of Transpose Operation
Matrix Math (4 of 4)
Determinants:
det A Think of this as the “magnitude” or “volume” of a matrix.
Matrix Inverse:
1A A I
Matrix Division:
1
1
predivide
postdivide
A B
B A
28Topic 3a ‐‐ Review of Linear Algebra
Matrix Multiplication:
premultiplies
postmultiplies
A B A B
B A A B
A\B
B/AWhile both expressions divide by [A], these do not give the same answer.
2/12/2018
15
Matrix Algebra (1 of 3)
Commutative Laws
A B B A
A B B A
Associative Laws
A B C A B C
A B C A B C
Distributive Laws
A B C A C B C
A B C A B A C
Matrix Inverses and Transposes
1 1
11
1 1 1
1 1
TT
T T T
A A A A I
A A
A B B A
A A
A B A B
TT
T T T
A A
A B B A
29Topic 3a ‐‐ Review of Linear Algebra
[A][B] = [B][A] when [A] and [B]are diagonal matrices.
Matrix Algebra (2 of 3)
Addition with a Scalar
11 12 1
21 22 2
1 2
doesn't make sense
n
n
n n nn
A
a a a
a a aI A
a a a
Multiplication with a Scalar
A B A B
A B A B A B
30Topic 3a ‐‐ Review of Linear Algebra
2/12/2018
16
Matrix Algebra (3 of 3)
Operations with Special Matrices
0 0 0
0 0
0
A A
I A A I A
A A A
A A
31Topic 3a ‐‐ Review of Linear Algebra
Example of Matrix Algebra
Simplify the Following Equation
11 C A D BC D
Step 1 – Subtract D from both sides
11
11
11
C A D D BC D D
C A 0 BC 0
C A BC
Step 2 – Inverse both sides
1 111
1 1 1
C A
C A C B
BC
Step 3 – Premultiply both sides by C.1 1 1
1
1
CC A CC B
IA IB
A B
32Topic 3a ‐‐ Review of Linear Algebra
2/12/2018
17
Common Linear Algebra Problems
33Topic 3a ‐‐ Review of Linear Algebra
[A][x] = [b]
This problem arises when a problem [A] is given some excitation [b]and produces a solution [x].
Examples: (1) waves scattering from an object, (2) heat through a device, (3) solving currents and voltages in a circuit.
It produces a single solution.
Step 1 – Differential equation2
2
d f dff b
dx dx
Step 2 – ODE is converted to system of equations using finite‐differences, finite elements, etc.
11 1 12 2 1 1
21 1 22 2 2 2
1 1 2 2
n n
n n
n n nn n n
a f a f a f b
a f a f a f b
a f a f a f b
Step 3 – System of equations is put into matrix form.
11 12 1 1 1
21 22 2 2 2
1 2
n
n
n n nn n n
a a a f b
a a a f b
a a a f b
Step 4 – Matrix problem is solved for [f]
1f A b
Step 5 – [f] is post processed to learn something.
34Topic 3a ‐‐ Review of Linear Algebra
2/12/2018
18
Eigen‐Value Problems
Eigen‐value problems arise when multiple solutions exist. No excitation is needed.
Examples: (1) resonating modes on a string, (2) electromagnetic modes in a waveguide, (3) electronic bands in a semiconductor.
Standard eigen-value problem
Generalized eigen-value problem
A x x
A x B x
is the linear operation
is the unknown (eigen-vector)
is the eigen-value and is just a scalar number
is potentially another part of the linear operation
A
x
B
35Topic 3a ‐‐ Review of Linear Algebra
DeterminantsThe determinant is an important number associated with square matrices.
It is sort of a magnitude or volume.
Unique solutions to systems of equations do not exist when the determinant is zero.
11 1211 22 12 21
21 22
deta a
A a a a aa a
3×3 Matrices
2×2 Matrices
11 12 13
22 23 21 23 21 2221 22 23 11 12 13
32 33 31 33 31 3231 32 33
det
a a aa a a a a a
A a a a a a aa a a a a a
a a a
This can be calculated by walking across any of the rows. 36Topic 3a ‐‐ Review of Linear Algebra
2/12/2018
19
Cramer’s Rule
Cramer’s rule provides a methodical approach for calculating the unknowns of a system of equations.
11 12 13 1 1
21 22 23 2 2
31 32 33 3 3
a a a x b
a a a x b
a a a x b
1 12 13
1 2 22 23
3 32 33
1b a a
x b a aD
b a a
11 1 13
2 21 2 23
31 3 33
1a b a
x a b aD
a b a
11 12 1
3 21 22 2
31 32 3
1a a b
x a a bD
a a b
11 12 13 11 12 13
21 22 23 21 22 23
31 32 33 31 32 33
det
a a a a a a
D a a a a a a
a a a a a a
37Topic 3a ‐‐ Review of Linear Algebra