Linear SystemsLinear Systems
COS 323COS 323
Linear SystemsLinear Systems
3333232131
2323222121
1313212111
bxaxaxa
bxaxaxa
bxaxaxa
3333232131
2323222121
1313212111
bxaxaxa
bxaxaxa
bxaxaxa
3
2
1
3
2
1
333231
232221
131211
b
b
b
x
x
x
aaa
aaa
aaa
3
2
1
3
2
1
333231
232221
131211
b
b
b
x
x
x
aaa
aaa
aaa
Linear SystemsLinear Systems
• Solve Ax=b, where A is an Solve Ax=b, where A is an nnnn matrix and matrix andb is an b is an nn1 column vector1 column vector
• Can also talk about non-square systems Can also talk about non-square systems wherewhereA is A is mmnn, b is , b is mm1, and x is 1, and x is nn11– OverdeterminedOverdetermined if if mm>>nn::
“more equations than unknowns”“more equations than unknowns”
– UnderdeterminedUnderdetermined if if nn>>mm::“more unknowns than equations”“more unknowns than equations”Can look for best solution using least squaresCan look for best solution using least squares
Singular SystemsSingular Systems
• A is singular if some row isA is singular if some row islinear combination of other rowslinear combination of other rows
• Singular systems can be Singular systems can be underdetermined:underdetermined:
or inconsistent:or inconsistent:
1064
532
21
21
xx
xx
1064
532
21
21
xx
xx
1164
532
21
21
xx
xx
1164
532
21
21
xx
xx
Inverting a MatrixInverting a Matrix
• Usually not a good idea to compute x=AUsually not a good idea to compute x=A-1-1bb– InefficientInefficient
– Prone to roundoff errorProne to roundoff error
• In fact, compute inverse using linear solverIn fact, compute inverse using linear solver– Solve AxSolve Axii=b=bii where b where bii are columns of identity, are columns of identity,
xxii are columns of inverse are columns of inverse
– Many solvers can solve several R.H.S. at onceMany solvers can solve several R.H.S. at once
Gauss-Jordan EliminationGauss-Jordan Elimination
• Fundamental operations:Fundamental operations:1.1. Replace one equation with linear combinationReplace one equation with linear combination
of other equationsof other equations
2.2. Interchange two equationsInterchange two equations
3.3. Re-label two variablesRe-label two variables
• Combine to reduce to trivial systemCombine to reduce to trivial system
• Simplest variant only uses #1 operations,Simplest variant only uses #1 operations,but get better stability by addingbut get better stability by adding#2 (partial pivoting) or #2 and #3 (full #2 (partial pivoting) or #2 and #3 (full pivoting)pivoting)
Gauss-Jordan EliminationGauss-Jordan Elimination
• Solve:Solve:
• Only care about numbers – form Only care about numbers – form “tableau” or “augmented matrix”:“tableau” or “augmented matrix”:
1354
732
21
21
xx
xx
1354
732
21
21
xx
xx
13
7
54
32
13
7
54
32
Gauss-Jordan EliminationGauss-Jordan Elimination
• Given:Given:
• Goal: reduce this to trivial systemGoal: reduce this to trivial system
and read off answer from right columnand read off answer from right column
13
7
54
32
13
7
54
32
?
?
10
01
?
?
10
01
Gauss-Jordan EliminationGauss-Jordan Elimination
• Basic operation 1: replace any row byBasic operation 1: replace any row bylinear combination with any other rowlinear combination with any other row
• Here, replace row1 with Here, replace row1 with 11//22 * row1 + 0 * * row1 + 0 *
row2row2
13
7
54
32
13
7
54
32
1354
1 27
23
1354
1 27
23
Gauss-Jordan EliminationGauss-Jordan Elimination
• Replace row2 with row2 – 4 * row1Replace row2 with row2 – 4 * row1
• Negate row2Negate row2
1354
1 27
23
1354
1 27
23
110
1 27
23
110
1 27
23
110
1 27
23
110
1 27
23
Gauss-Jordan EliminationGauss-Jordan Elimination
• Replace row1 with row1 – Replace row1 with row1 – 33//22 * row2 * row2
• Read off solution: xRead off solution: x1 1 = 2, x= 2, x2 2 = 1= 1
110
1 27
23
110
1 27
23
1
2
10
01
1
2
10
01
Gauss-Jordan EliminationGauss-Jordan Elimination
• For each row i:For each row i:– Multiply row i by 1/aMultiply row i by 1/aiiii
– For each other row j:For each other row j:
• Add –aAdd –ajiji times row i to row j times row i to row j
• At the end, left part of matrix is identity,At the end, left part of matrix is identity,answer in right partanswer in right part
• Can solve any number of R.H.S. Can solve any number of R.H.S. simultaneouslysimultaneously
PivotingPivoting
• Consider this system:Consider this system:
• Immediately run into problem:Immediately run into problem:algorithm wants us to divide by zero!algorithm wants us to divide by zero!
• More subtle version:More subtle version:
8
2
32
10
8
2
32
10
8
2
32
1001.0
8
2
32
1001.0
PivotingPivoting
• Conclusion: small diagonal elements Conclusion: small diagonal elements badbad
• Remedy: swap in larger element from Remedy: swap in larger element from somewhere elsesomewhere else
Partial PivotingPartial Pivoting
• Swap rows 1 and 2:Swap rows 1 and 2:
• Now continue:Now continue:
8
2
32
10
8
2
32
10
2
8
10
32
2
8
10
32
2
1
10
01
2
4
10
1 23
2
1
10
01
2
4
10
1 23
Full PivotingFull Pivoting
• Swap largest element onto diagonal by Swap largest element onto diagonal by swapping rows 1 and 2 and columns 1 swapping rows 1 and 2 and columns 1 and 2:and 2:
• Critical: when swapping columns, must Critical: when swapping columns, must remember to swap results!remember to swap results!
8
2
32
10
8
2
32
10
2
8
01
23
2
8
01
23
Full PivotingFull Pivoting
• Full pivoting more stable, but only Full pivoting more stable, but only slightlyslightly
2
8
01
23
2
8
01
23
32
38
32
32
0
1
32
38
32
32
0
1
1
2
10
01
1
2
10
01
** Swap results Swap results 1 and 2 1 and 2
Operation CountOperation Count
• For one R.H.S., how many operations?For one R.H.S., how many operations?
• For each of n rows:For each of n rows:– Do n times:Do n times:
• For each of n+1 columns:For each of n+1 columns:
– One add, one multiplyOne add, one multiply
• Total = nTotal = n33+n+n22 multiplies, same # of adds multiplies, same # of adds
• Asymptotic behavior: when n is large, Asymptotic behavior: when n is large, dominated by ndominated by n33
Faster AlgorithmsFaster Algorithms
• Our goal is an algorithm that does this Our goal is an algorithm that does this inin11//33 nn33 operations, and does not require operations, and does not require
all R.H.S. to be known at beginningall R.H.S. to be known at beginning
• Before we see that, let’s look at a fewBefore we see that, let’s look at a fewspecial cases that are even fasterspecial cases that are even faster
Tridiagonal SystemsTridiagonal Systems
• Common special case:Common special case:
• Only main diagonal + 1 above and 1 Only main diagonal + 1 above and 1 belowbelow
4
3
2
1
4443
343332
232221
1211
00
0
0
00
b
b
b
b
aa
aaa
aaa
aa
4
3
2
1
4443
343332
232221
1211
00
0
0
00
b
b
b
b
aa
aaa
aaa
aa
Solving Tridiagonal SystemsSolving Tridiagonal Systems
• When solving using Gauss-Jordan:When solving using Gauss-Jordan:– Constant # of multiplies/adds in each rowConstant # of multiplies/adds in each row
– Each row only affects 2 othersEach row only affects 2 others
4
3
2
1
4443
343332
232221
1211
00
0
0
00
b
b
b
b
aa
aaa
aaa
aa
4
3
2
1
4443
343332
232221
1211
00
0
0
00
b
b
b
b
aa
aaa
aaa
aa
Running TimeRunning Time
• 2n loops, 4 multiply/adds per loop2n loops, 4 multiply/adds per loop(assuming correct bookkeeping)(assuming correct bookkeeping)
• This running time has a fundamentally This running time has a fundamentally different dependence on n: linear different dependence on n: linear instead of cubicinstead of cubic– Can say that tridiagonal algorithm is O(n) Can say that tridiagonal algorithm is O(n)
whilewhileGauss-Jordan is O(nGauss-Jordan is O(n33))
Big-O NotationBig-O Notation
• Informally, O(nInformally, O(n33) means that the dominant ) means that the dominant term for large n is cubicterm for large n is cubic
• More precisely, there exist a c and nMore precisely, there exist a c and n00 such such
that that running time running time c n c n33
ifif n > n n > n00
• This type of This type of asymptotic analysisasymptotic analysis is often is often usedusedto characterize different algorithmsto characterize different algorithms
Triangular SystemsTriangular Systems
• Another special case: A is lower-Another special case: A is lower-triangulartriangular
4
3
2
1
44434241
333231
2221
11
0
00
000
b
b
b
b
aaaa
aaa
aa
a
4
3
2
1
44434241
333231
2221
11
0
00
000
b
b
b
b
aaaa
aaa
aa
a
Triangular SystemsTriangular Systems
• Solve by forward substitutionSolve by forward substitution
4
3
2
1
44434241
333231
2221
11
0
00
000
b
b
b
b
aaaa
aaa
aa
a
4
3
2
1
44434241
333231
2221
11
0
00
000
b
b
b
b
aaaa
aaa
aa
a
11
11 a
bx
11
11 a
bx
Triangular SystemsTriangular Systems
• Solve by forward substitutionSolve by forward substitution
4
3
2
1
44434241
333231
2221
11
0
00
000
b
b
b
b
aaaa
aaa
aa
a
4
3
2
1
44434241
333231
2221
11
0
00
000
b
b
b
b
aaaa
aaa
aa
a
22
12122 a
xabx
22
12122 a
xabx
Triangular SystemsTriangular Systems
• Solve by forward substitutionSolve by forward substitution
4
3
2
1
44434241
333231
2221
11
0
00
000
b
b
b
b
aaaa
aaa
aa
a
4
3
2
1
44434241
333231
2221
11
0
00
000
b
b
b
b
aaaa
aaa
aa
a
33
23213133 a
xaxabx
33
23213133 a
xaxabx
Triangular SystemsTriangular Systems
• If A is upper triangular, solve by If A is upper triangular, solve by backsubstitutionbacksubstitution
5
4
3
2
1
55
4544
353433
25242322
1514131211
0000
000
00
0
b
b
b
b
b
a
aa
aaa
aaaa
aaaaa
5
4
3
2
1
55
4544
353433
25242322
1514131211
0000
000
00
0
b
b
b
b
b
a
aa
aaa
aaaa
aaaaa
55
55 a
bx
55
55 a
bx
Triangular SystemsTriangular Systems
• If A is upper triangular, solve by If A is upper triangular, solve by backsubstitutionbacksubstitution
5
4
3
2
1
55
4544
353433
25242322
1514131211
0000
000
00
0
b
b
b
b
b
a
aa
aaa
aaaa
aaaaa
5
4
3
2
1
55
4544
353433
25242322
1514131211
0000
000
00
0
b
b
b
b
b
a
aa
aaa
aaaa
aaaaa
44
54544 a
xabx
44
54544 a
xabx
Triangular SystemsTriangular Systems
• Both of these special cases can be solved Both of these special cases can be solved ininO(nO(n22) time) time
• This motivates a factorization approach to This motivates a factorization approach to solving arbitrary systems:solving arbitrary systems:– Find a way of writing A as LU, where L and U are Find a way of writing A as LU, where L and U are
both triangularboth triangular
– Ax=b Ax=b LUx=b LUx=b Ly=b Ly=b Ux=y Ux=y
– Time for factoring matrix dominates Time for factoring matrix dominates computationcomputation
Cholesky DecompositionCholesky Decomposition
• For symmetric matrices, choose U=LFor symmetric matrices, choose U=LTT
• Perform decompositionPerform decomposition
• Ax=b Ax=b LL LLTTx=b x=b Ly=b Ly=b LLTTx=yx=y
33
3222
312111
333231
2221
11
332313
232212
131211
00
00
00
l
ll
lll
lll
ll
l
aaa
aaa
aaa
33
3222
312111
333231
2221
11
332313
232212
131211
00
00
00
l
ll
lll
lll
ll
l
aaa
aaa
aaa
Cholesky DecompositionCholesky Decomposition
22
312123322332223121
221222222
222
221
11
1331133111
11
1221122111
1111112
11
l
llalallll
lalall
l
alall
l
alall
alal
22
312123322332223121
221222222
222
221
11
1331133111
11
1221122111
1111112
11
l
llalallll
lalall
l
alall
l
alall
alal
33
3222
312111
333231
2221
11
332313
232212
131211
00
00
00
l
ll
lll
lll
ll
l
aaa
aaa
aaa
33
3222
312111
333231
2221
11
332313
232212
131211
00
00
00
l
ll
lll
lll
ll
l
aaa
aaa
aaa
Cholesky DecompositionCholesky Decomposition
ii
i
kjkikij
ji
i
kikiiii
l
llal
lal
1
1
1
1
2
ii
i
kjkikij
ji
i
kikiiii
l
llal
lal
1
1
1
1
2
33
3222
312111
333231
2221
11
332313
232212
131211
00
00
00
l
ll
lll
lll
ll
l
aaa
aaa
aaa
33
3222
312111
333231
2221
11
332313
232212
131211
00
00
00
l
ll
lll
lll
ll
l
aaa
aaa
aaa
Cholesky DecompositionCholesky Decomposition
• This fails if it requires taking square root This fails if it requires taking square root of a negative numberof a negative number
• Need another condition on A: positive Need another condition on A: positive definitedefinite
For any v, vFor any v, vT T A v > 0A v > 0
(Equivalently, all positive eigenvalues)(Equivalently, all positive eigenvalues)
Cholesky DecompositionCholesky Decomposition
• Running time turns out to be Running time turns out to be 11//66nn33
– Still cubic, but much lower constantStill cubic, but much lower constant
• Result: this is preferred method for Result: this is preferred method for solvingsolvingsymmetric positive definite systemssymmetric positive definite systems
LU DecompositionLU Decomposition
• Again, factor A into LU, whereAgain, factor A into LU, whereL is lower triangular and U is upper L is lower triangular and U is upper triangulartriangular
• Last 2 steps in O(nLast 2 steps in O(n22) time, so total time ) time, so total time dominated by decompositiondominated by decomposition
Ax=bAx=bLUx=bLUx=bLy=bLy=bUx=yUx=y
Crout’s MethodCrout’s Method
• More unknowns than equations!More unknowns than equations!
• Let all lLet all liiii=1=1
33
2322
131211
333231
2221
11
333231
232221
131211
00
00
00
u
uu
uuu
lll
ll
l
aaa
aaa
aaa
33
2322
131211
333231
2221
11
333231
232221
131211
00
00
00
u
uu
uuu
lll
ll
l
aaa
aaa
aaa
Crout’s MethodCrout’s Method
33
2322
131211
3231
21
333231
232221
131211
00
0
1
01
001
u
uu
uuu
ll
l
aaa
aaa
aaa
33
2322
131211
3231
21
333231
232221
131211
00
0
1
01
001
u
uu
uuu
ll
l
aaa
aaa
aaa
22
123132323222321231
1221222222221221
1212
11
3131311131
11
2121211121
1111
u
ulalaulul
ulauauul
au
u
alaul
u
alaul
au
22
123132323222321231
1221222222221221
1212
11
3131311131
11
2121211121
1111
u
ulalaulul
ulauauul
au
u
alaul
u
alaul
au
Crout’s MethodCrout’s Method
• For i = 1..nFor i = 1..n– For j = 1..iFor j = 1..i
– For j = i+1..nFor j = i+1..n
33
2322
131211
3231
21
333231
232221
131211
00
0
1
01
001
u
uu
uuu
ll
l
aaa
aaa
aaa
33
2322
131211
3231
21
333231
232221
131211
00
0
1
01
001
u
uu
uuu
ll
l
aaa
aaa
aaa
1
1
j
kkijkjiji ulau
1
1
j
kkijkjiji ulau
ii
i
kkijkji
ji u
ulal
1
1
ii
i
kkijkji
ji u
ulal
1
1
Crout’s MethodCrout’s Method
• Interesting note: # of outputs = # of inputs,Interesting note: # of outputs = # of inputs,algorithm only refers to elements not output algorithm only refers to elements not output yetyet– Can do this in-place!Can do this in-place!
– Algorithm replaces A with matrixAlgorithm replaces A with matrixof l and u values, 1s are impliedof l and u values, 1s are implied
– Resulting matrix must be interpreted in a special Resulting matrix must be interpreted in a special way: not a regular matrixway: not a regular matrix
– Can rewrite forward/backsubstitution routines to Can rewrite forward/backsubstitution routines to use this “packed” l-u matrixuse this “packed” l-u matrix
333231
232221
131211
ull
uul
uuu
333231
232221
131211
ull
uul
uuu
LU DecompositionLU Decomposition
• Running time is Running time is 11//33nn33
– Only a factor of 2 slower than symmetric Only a factor of 2 slower than symmetric casecase
– This is the preferred general method forThis is the preferred general method forsolving linear equationssolving linear equations
• Pivoting very importantPivoting very important– Partial pivoting is sufficient, and widely Partial pivoting is sufficient, and widely
implementedimplemented
Top Related