Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina...
Transcript of Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina...
![Page 1: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/1.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
Inner-outer iterative methods for large sparse
Eigenvalue computations
Melina Freitag
Department of Mathematical Sciences
University of Bath
Informal Postgraduate SeminarUniversity of Bath21st March 2006
![Page 2: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/2.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
1 Motivation
2 Definitions
3 Characteristic Polynomial
4 Eigenvalue algorithms based on similarity transformation
5 Iterative methods
6 Inner-outer iterative methods
![Page 3: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/3.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
Outline
1 Motivation
2 Definitions
3 Characteristic Polynomial
4 Eigenvalue algorithms based on similarity transformation
5 Iterative methods
6 Inner-outer iterative methods
![Page 4: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/4.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
What is an eigenvalue?
eigenvalue comes from the German word Eigenwert (likeliverwurst, only half of it has been translated).
arises after simplification/discretisation/linearisation of aproblem
can be meaningless intermediate values of a computationmethod in order to find the solution of a problem
sometimes the values have a meaning for the problem(stability analysis)
![Page 5: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/5.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
What is an eigenvalue?
eigenvalue comes from the German word Eigenwert (likeliverwurst, only half of it has been translated).
arises after simplification/discretisation/linearisation of aproblem
can be meaningless intermediate values of a computationmethod in order to find the solution of a problem
sometimes the values have a meaning for the problem(stability analysis)
![Page 6: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/6.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
Millenium Footbridge
On its opening day, the Millenium footbridge in London startedto wobble under the weight of 100s of people, who, in turn alsostruggled to keep their balance. The bridge had to be closed.After fitting of 37 fluid-viscous dampers and 1 year and £5mlater the problem was fixed.
![Page 7: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/7.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
Millenium Footbridge and some more applications
What had happened?
Some of the frequencies of the bridge were similar to thecomponents of the pedestrians footsteps, causing vibrationamplification. Finding these natural frequencies amounts tosolving an eigenvalue problem.
Structural dynamics
Quantum Chemistry/Chemical Reactions
Markov chain techniques/Google
Stability analysis of dynamical systems/solution ofdifferential equations
![Page 8: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/8.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
Outline
1 Motivation
2 Definitions
3 Characteristic Polynomial
4 Eigenvalue algorithms based on similarity transformation
5 Iterative methods
6 Inner-outer iterative methods
![Page 9: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/9.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
A few definitions
Eigenvalues of A ∈ Rn,n are roots of det(A − λI)
Eigenvectors are x 6= 0 such that Ax = λx for someeigenvalue λ ∈ C
Schur Decomposition: There exists an orthogonal matrixQ ∈ C
n,n (QT Q = I) s.t.
QT AQ =
d11 · · · d1n
0 d11 · · · d2n
0 0. . .
...0 0 · · · dnn
![Page 10: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/10.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
A few definitions
Eigenvalues of A ∈ Rn,n are roots of det(A − λI)
Eigenvectors are x 6= 0 such that Ax = λx for someeigenvalue λ ∈ C
Schur Decomposition: There exists an orthogonal matrixQ ∈ C
n,n (QT Q = I) s.t.
QT AQ =
d11 · · · d1n
0 d11 · · · d2n
0 0. . .
...0 0 · · · dnn
![Page 11: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/11.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
Outline
1 Motivation
2 Definitions
3 Characteristic Polynomial
4 Eigenvalue algorithms based on similarity transformation
5 Iterative methods
6 Inner-outer iterative methods
![Page 12: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/12.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
Why not calculating the roots of the characteristic
polynomial?
det(A) =∑
σ∈Sn
(
sgn(σ)n∏
i=1
ai,σ(i)
)
σ . . . , permutation
contains n! summands, not very handy
there exists no formula for calculation the roots of apolynomial of degree larger then 5
calculating the roots numerically is unstable
Back to matrices!
![Page 13: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/13.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
Why not calculating the roots of the characteristic
polynomial?
det(A) =∑
σ∈Sn
(
sgn(σ)n∏
i=1
ai,σ(i)
)
σ . . . , permutation
contains n! summands, not very handy
there exists no formula for calculation the roots of apolynomial of degree larger then 5
calculating the roots numerically is unstable
Back to matrices!
![Page 14: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/14.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
Why not calculating the roots of the characteristic
polynomial?
det(A) =∑
σ∈Sn
(
sgn(σ)n∏
i=1
ai,σ(i)
)
σ . . . , permutation
contains n! summands, not very handy
there exists no formula for calculation the roots of apolynomial of degree larger then 5
calculating the roots numerically is unstable
Back to matrices!
![Page 15: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/15.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
Why not calculating the roots of the characteristic
polynomial?
det(A) =∑
σ∈Sn
(
sgn(σ)n∏
i=1
ai,σ(i)
)
σ . . . , permutation
contains n! summands, not very handy
there exists no formula for calculation the roots of apolynomial of degree larger then 5
calculating the roots numerically is unstable
Back to matrices!
![Page 16: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/16.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
Why not calculating the roots of the characteristic
polynomial?
det(A) =∑
σ∈Sn
(
sgn(σ)n∏
i=1
ai,σ(i)
)
σ . . . , permutation
contains n! summands, not very handy
there exists no formula for calculation the roots of apolynomial of degree larger then 5
calculating the roots numerically is unstable
Back to matrices!
![Page 17: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/17.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
Outline
1 Motivation
2 Definitions
3 Characteristic Polynomial
4 Eigenvalue algorithms based on similarity transformation
5 Iterative methods
6 Inner-outer iterative methods
![Page 18: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/18.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
Algorithms based on the idea of calculating the
Schur Form
Schur Form:
QT AQ =
d11 · · · d1n
0 d11 · · · d2n
0 0. . .
...0 0 · · · dnn
Similarity transformation does not change the eigenvalues:
det(QT AQ − λI) = det(QT AQ − QT λQ)
= det(QT )det(A − λI)det(Q)
= det(A − λI)
QR Algorithm
![Page 19: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/19.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
Algorithms based on the idea of calculating the
Schur Form
Schur Form:
QT AQ =
d11 · · · d1n
0 d11 · · · d2n
0 0. . .
...0 0 · · · dnn
Similarity transformation does not change the eigenvalues:
det(QT AQ − λI) = det(QT AQ − QT λQ)
= det(QT )det(A − λI)det(Q)
= det(A − λI)
QR Algorithm
![Page 20: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/20.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
Algorithms based on the idea of calculating the
Schur Form
Schur Form:
QT AQ =
d11 · · · d1n
0 d11 · · · d2n
0 0. . .
...0 0 · · · dnn
Similarity transformation does not change the eigenvalues:
det(QT AQ − λI) = det(QT AQ − QT λQ)
= det(QT )det(A − λI)det(Q)
= det(A − λI)
QR Algorithm
![Page 21: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/21.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
QR-Algorithm I
Basic QR-iteration: given A0 := A compute
Factor Ai = QiRi (QR decomposition)
Ai+1 = RiQi
Ai and A have the same eigenvalues and we hope that
Ai → R =
d11 · · · d1n
0 d11 · · · d2n
0 0. . .
...0 0 · · · dnn
Work for calculating all the eigenvalues and eigenvectorsof a matrix in O(n3) operations
![Page 22: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/22.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
QR-Algorithm I
Basic QR-iteration: given A0 := A compute
Factor Ai = QiRi (QR decomposition)
Ai+1 = RiQi
Ai and A have the same eigenvalues and we hope that
Ai → R =
d11 · · · d1n
0 d11 · · · d2n
0 0. . .
...0 0 · · · dnn
Work for calculating all the eigenvalues and eigenvectorsof a matrix in O(n3) operations
![Page 23: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/23.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
QR-Algorithm I
Basic QR-iteration: given A0 := A compute
Factor Ai = QiRi (QR decomposition)
Ai+1 = RiQi
Ai and A have the same eigenvalues and we hope that
Ai → R =
d11 · · · d1n
0 d11 · · · d2n
0 0. . .
...0 0 · · · dnn
Work for calculating all the eigenvalues and eigenvectorsof a matrix in O(n3) operations
![Page 24: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/24.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
QR-Algorithm II
Problem Fill-in for large sparse matrices
0 100 200 300 400
0
50
100
150
200
250
300
350
400
450
nz = 1887
Figure: Sparse matrix
0 100 200 300 400
0
50
100
150
200
250
300
350
400
450
nz = 147114
Figure: One step of QR
![Page 25: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/25.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
Outline
1 Motivation
2 Definitions
3 Characteristic Polynomial
4 Eigenvalue algorithms based on similarity transformation
5 Iterative methods
6 Inner-outer iterative methods
![Page 26: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/26.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
Large sparse matrices
Only a few eigenvalues (largest, smallest) are required
QR algorithm is too expensive in terms of storage andcomputation time
need iterative methods!
Examples: Power method, Inverse iteration, RayleighQuotient Iteration, Subspace iteration, Lanczos method,Arnoldi’s method
![Page 27: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/27.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
Large sparse matrices
Only a few eigenvalues (largest, smallest) are required
QR algorithm is too expensive in terms of storage andcomputation time
need iterative methods!
Examples: Power method, Inverse iteration, RayleighQuotient Iteration, Subspace iteration, Lanczos method,Arnoldi’s method
![Page 28: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/28.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
Large sparse matrices
Only a few eigenvalues (largest, smallest) are required
QR algorithm is too expensive in terms of storage andcomputation time
need iterative methods!
Examples: Power method, Inverse iteration, RayleighQuotient Iteration, Subspace iteration, Lanczos method,Arnoldi’s method
![Page 29: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/29.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
Concepts of the Power method
Given x0
yi+1 = Axi
xi+1 = yi+1/‖yi+1‖
λi+1 = xTi+1Axi+1
convergence to λ1 with linear rate|λ2|
|λ1|< 1
only needs one matrix-vector multiplication at each step
Inverse Iteration is power method applied to (A − σI)−1
Rayleigh Quotient iteration with special shiftσi+1 = xT
i+1Axi+1 (quadratic/cubic convergence)
![Page 30: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/30.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
Concepts of the Power method
Given x0
yi+1 = Axi
xi+1 = yi+1/‖yi+1‖
λi+1 = xTi+1Axi+1
convergence to λ1 with linear rate|λ2|
|λ1|< 1
only needs one matrix-vector multiplication at each step
Inverse Iteration is power method applied to (A − σI)−1
Rayleigh Quotient iteration with special shiftσi+1 = xT
i+1Axi+1 (quadratic/cubic convergence)
![Page 31: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/31.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
Concepts of the Power method
Given x0
yi+1 = Axi
xi+1 = yi+1/‖yi+1‖
λi+1 = xTi+1Axi+1
convergence to λ1 with linear rate|λ2|
|λ1|< 1
only needs one matrix-vector multiplication at each step
Inverse Iteration is power method applied to (A − σI)−1
Rayleigh Quotient iteration with special shiftσi+1 = xT
i+1Axi+1 (quadratic/cubic convergence)
![Page 32: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/32.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
Concepts of the Power method
Given x0
yi+1 = Axi
xi+1 = yi+1/‖yi+1‖
λi+1 = xTi+1Axi+1
convergence to λ1 with linear rate|λ2|
|λ1|< 1
only needs one matrix-vector multiplication at each step
Inverse Iteration is power method applied to (A − σI)−1
Rayleigh Quotient iteration with special shiftσi+1 = xT
i+1Axi+1 (quadratic/cubic convergence)
![Page 33: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/33.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
Concepts of the Power method
Given x0
yi+1 = Axi
xi+1 = yi+1/‖yi+1‖
λi+1 = xTi+1Axi+1
convergence to λ1 with linear rate|λ2|
|λ1|< 1
only needs one matrix-vector multiplication at each step
Inverse Iteration is power method applied to (A − σI)−1
Rayleigh Quotient iteration with special shiftσi+1 = xT
i+1Axi+1 (quadratic/cubic convergence)
![Page 34: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/34.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
Concepts of Lanczos/ Arnoldi’s method I
Power method with initial vector q computesq,Aq, . . . , Akq
idea of Arnoldi’s method: retain past information: after ksteps we have k + 1 vectors q,Aq, . . . , Akq
Given q1, ‖q1‖2 = 1 On subsequent steps k = 1, 2, . . . ,mtake
q̃k+1 = Aqk −
k∑
j=1
qjhjk
where hjk is the Gram-Schmidt coefficienthjk =< Aqk, qj >. Normalise
qk+1 =q̃k+1
hk+1,k
where hk+1,k = ‖q̃k+1‖2
![Page 35: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/35.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
Concepts of Lanczos/ Arnoldi’s method I
Power method with initial vector q computesq,Aq, . . . , Akq
idea of Arnoldi’s method: retain past information: after ksteps we have k + 1 vectors q,Aq, . . . , Akq
Given q1, ‖q1‖2 = 1 On subsequent steps k = 1, 2, . . . ,mtake
q̃k+1 = Aqk −
k∑
j=1
qjhjk
where hjk is the Gram-Schmidt coefficienthjk =< Aqk, qj >. Normalise
qk+1 =q̃k+1
hk+1,k
where hk+1,k = ‖q̃k+1‖2
![Page 36: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/36.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
Concepts of Lanczos/ Arnoldi’s method II
Definition
For any j the space span{q,Aq, . . . , Aj−1q} is called the jthKrylov subspace associated with A and q and is denoted byKj(A, q).
Matrix representation
The Arnoldi process can be written in the form
AQm = QmHm + qm+1hm+1,meTm
where Hm is square upper Hessenberg.
![Page 37: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/37.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
Concepts of Lanczos/ Arnoldi’s method III
For the Lanczos process: Hm is tridiagonal (three termrecurrence)
Approximate eigenvalues and eigenvectors of A can befound from eigenvalues and eigenvectors of much smallermatrix Hm:
Let Qm, Hm and hm+1,m be generated by the Arnoldi process.Let µ be an eigenvalue of Hm with associated eigenvector xnormalised so that ‖x‖2=1. Let y = Qmx ∈ C
n. Then
‖Ay − µy‖2 = |hm+1,m||xm|,
where xm denotes the mth (and last) component of x.
![Page 38: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/38.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
Concepts of Lanczos/ Arnoldi’s method III
For the Lanczos process: Hm is tridiagonal (three termrecurrence)
Approximate eigenvalues and eigenvectors of A can befound from eigenvalues and eigenvectors of much smallermatrix Hm:
Let Qm, Hm and hm+1,m be generated by the Arnoldi process.Let µ be an eigenvalue of Hm with associated eigenvector xnormalised so that ‖x‖2=1. Let y = Qmx ∈ C
n. Then
‖Ay − µy‖2 = |hm+1,m||xm|,
where xm denotes the mth (and last) component of x.
![Page 39: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/39.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
Concepts of Lanczos/ Arnoldi’s method III
For the Lanczos process: Hm is tridiagonal (three termrecurrence)
Approximate eigenvalues and eigenvectors of A can befound from eigenvalues and eigenvectors of much smallermatrix Hm:
Let Qm, Hm and hm+1,m be generated by the Arnoldi process.Let µ be an eigenvalue of Hm with associated eigenvector xnormalised so that ‖x‖2=1. Let y = Qmx ∈ C
n. Then
‖Ay − µy‖2 = |hm+1,m||xm|,
where xm denotes the mth (and last) component of x.
![Page 40: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/40.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
Example
random complex matrix of dimension n = 144 generated inMatlab:
−4 −3 −2 −1 0 1 2 3 4
−3
−2
−1
0
1
2
3
eigenvalues of A
approximation of outer eigenvalues first!
![Page 41: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/41.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
after 5 Arnoldi steps
−4 −3 −2 −1 0 1 2 3 4
−3
−2
−1
0
1
2
3
Arnoldi after 5 steps
![Page 42: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/42.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
after 10 Arnoldi steps
−4 −3 −2 −1 0 1 2 3 4
−3
−2
−1
0
1
2
3
Arnoldi after 10 steps
![Page 43: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/43.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
after 15 Arnoldi steps
−4 −3 −2 −1 0 1 2 3 4
−3
−2
−1
0
1
2
3
Arnoldi after 15 steps
![Page 44: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/44.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
after 20 Arnoldi steps
−4 −3 −2 −1 0 1 2 3 4
−3
−2
−1
0
1
2
3
Arnoldi after 20 steps
![Page 45: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/45.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
after 25 Arnoldi steps
−4 −3 −2 −1 0 1 2 3 4
−3
−2
−1
0
1
2
3
Arnoldi after 25 steps
![Page 46: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/46.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
after 30 Arnoldi steps
−4 −3 −2 −1 0 1 2 3 4
−3
−2
−1
0
1
2
3
Arnoldi after 30 steps
![Page 47: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/47.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
after 35 Arnoldi steps
−4 −3 −2 −1 0 1 2 3 4
−3
−2
−1
0
1
2
3
Arnoldi after 35 steps
![Page 48: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/48.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
after 40 Arnoldi steps
−4 −3 −2 −1 0 1 2 3 4
−3
−2
−1
0
1
2
3
Arnoldi after 40 steps
![Page 49: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/49.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
Outline
1 Motivation
2 Definitions
3 Characteristic Polynomial
4 Eigenvalue algorithms based on similarity transformation
5 Iterative methods
6 Inner-outer iterative methods
![Page 50: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/50.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
Large sparse matrices
interior eigenvalues: Shift-Invert Transformation
Ax = λx
(A − σI)−1x =1
λ − σx
”outer” eigenvalues are the one closest to σ
Problem: requires a solution of a linear system at eachstep:
(A − σI)u = b
Solve is done iteratively (since A is large and sparse) usingMINRES, GMRES, other iterative methods
![Page 51: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/51.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
Large sparse matrices
interior eigenvalues: Shift-Invert Transformation
Ax = λx
(A − σI)−1x =1
λ − σx
”outer” eigenvalues are the one closest to σ
Problem: requires a solution of a linear system at eachstep:
(A − σI)u = b
Solve is done iteratively (since A is large and sparse) usingMINRES, GMRES, other iterative methods
![Page 52: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/52.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
Large sparse matrices
The solve will be inexact
How does the inexact inner solve influence theconvergence property of the outer solve?
For inexact inverse iteration:
(A − σI)yi = xi + resi
If residual resi is chosen to decrease in the same manneras the eigenvalue residual decreases the convergence ratefrom exact solves is recovered
Ongoing research: Situation for Krylov methods
![Page 53: Inner-outer iterative methods for large sparse Eigenvalue ...mamamf/talks/pissbath.pdf · Melina Freitag Outline Motivation De nitions Characteristic Polynomial Eigenvalue algorithms](https://reader034.fdocuments.net/reader034/viewer/2022050214/5f60247518a15f7cd8703df0/html5/thumbnails/53.jpg)
Large sparse
Eigenvalue
computations
Melina Freitag
Outline
Motivation
Definitions
Characteristic
Polynomial
Eigenvalue
algorithms
based on
similarity
transformation
Iterative
methods
Inner-outer
iterative
methods
Large sparse matrices
The solve will be inexact
How does the inexact inner solve influence theconvergence property of the outer solve?
For inexact inverse iteration:
(A − σI)yi = xi + resi
If residual resi is chosen to decrease in the same manneras the eigenvalue residual decreases the convergence ratefrom exact solves is recovered
Ongoing research: Situation for Krylov methods