Krylov subspace methods for functions of matrices - Bergische

Classical and extendedKrylov subspace methods for

matrix functions

DiplomarbeitPaul Zingler

Betreuer: Prof. Dr. Andreas Frommer

Bergische Universitat WuppertalFachbereich Mathematik und Naturwissenschaften

http://www.uni-wuppertal.de

http://www.math.uni-wuppertal.de

Preface

This Diploma Thesis is about computing the propagation operator f(A)b,that is the action a matrix function has to a vector. Many applications donot afford an explicitly knowledge of f(A) and furthermore, the matrices Aare possibly large and sparse. The more general computation of f(A) cannot be feasible since storing f(A) exceeds the memory capacity or takes toomuch time. The size of the matrices and the complexity of computing f(A)let us seek for methods which reduce the problem size. Our main tool areKrylov subspaces methods. They provide a compression of the given matrixA and furthermore the Krylov subspaces contain the images of the givenvector b under certain polynomials in A. This is of particular interest sinceit will turn out that functions of matrices and polynomials in a matrix areclosely related.

The absence of a residual is a critical point in the approximation of f(A)b.We are therefore interested in indicators for the error of the approximationwhich rather arise from theory than from practice.

When the inverse of the matrix A is given or easily obtained, extendedKrylov subspaces can be used to approximate f(A)b. The property of alsocontaining information about the images of b under polynomials in A−1 can,for a given subspace dimension, be exploited to generate an approximationto f(A)b which is more precise than the standard Krylov subspace approxi-mation.

Structure of the Thesis. In Chapter 1, we explain how the concept ofscalar functions can be extended to functions of matrices, and the definitionof a matrix function will motivate Krylov subspaces.

In Chapter 2, we present algorithms that compute Krylov subspaces andthe associated compressions of a given matrix. Furthermore, we motivatestandard Krylov subspace methods for the approximation of f(A)b.

Chapter 3 derives error representations for the standard Krylov subspaceapproximations, which can be used to obtain error indicators.

Extended Krylov subspaces, their properties and the associated methods

i

for approximating f(A)b are treated in Chapter 4.Applications from which functions of matrices arise, implementations of

the Krylov subspace methods and the Regensburg method form the contentof Chapter 5.

Since in theory, there is no difference between theory and practice, butin practice there is, we try to verify the results from the earlier chaptersconcerning Krylov subspace processes and apply Krylov subspace methodsto matrices arising from discretizations of partial differential equations andlattice QCD.

Acknowledgements. First of all, I would like to thank my advisor Prof.Dr. Andreas Frommer for his help and discussions during the developmentof this thesis. Furthermore, I am deeply indebted to Sebastian Birk, who wasa brick when implementing the MATLAB code and provided me with thenewest results computed on the workgroup’s computers, even if it was Sundayevening. Furthermore, I am grateful to Dr. Kathrin Schafer who found a 64

lattice QCD matrix in diagonal form and ran tests for me on friday evenings.I thank Lukas Kramer for being a reliable friend, new insights in the qualityof my orthography and being precise in the mathematical context.

I also like to thank Valeria Simoncini and Stefan Guttel for providingMATLAB code for the extended Krylov subspace process and the error in-dicators.

Last but not least, I would like to thank my family and friends for theirsupport, the never empty fridge an endless supply with coffee and warmwords.

Wuppertal, first of December 2009 Paul Zingler

ii

Contents

Preface i

Contents ii

1 Introduction 11.1 What is a Matrix Function? . . . . . . . . . . . . . . . . . . . 11.2 Three Definitions of a Matrix Function . . . . . . . . . . . . . 31.3 Krylov Subspaces . . . . . . . . . . . . . . . . . . . . . . . . . 6

2 Krylov Subspace Methods 82.1 The Arnoldi Process . . . . . . . . . . . . . . . . . . . . . . . 82.2 The Two-Sided Lanczos Process . . . . . . . . . . . . . . . . . 14

3 Error Analysis 183.1 Basic Error Representation . . . . . . . . . . . . . . . . . . . . 183.2 Adding Auxiliary Nodes . . . . . . . . . . . . . . . . . . . . . 20

4 Extended Krylov Subspaces 27

5 Application and Implementation 375.1 How Functions of Matrices Arise . . . . . . . . . . . . . . . . 37

5.1.1 Differential Equations . . . . . . . . . . . . . . . . . . 375.1.2 Lattice Quantum Chromo Dynamic . . . . . . . . . . . 38

5.2 The Complete Algorithm for SKSM . . . . . . . . . . . . . . . 385.3 Explicitly Computing f(A)? . . . . . . . . . . . . . . . . . . . 40

5.3.1 Taylor Series . . . . . . . . . . . . . . . . . . . . . . . 415.3.2 Rational Functions . . . . . . . . . . . . . . . . . . . . 41

5.4 The Complete Algorithm for EKSM . . . . . . . . . . . . . . . 425.4.1 The Regensburg Method . . . . . . . . . . . . . . . . . 42

iii

CONTENTS CONTENTS

6 Numerical Experiments 466.1 The Orthogonalization Processes . . . . . . . . . . . . . . . . 466.2 The Error Indicator . . . . . . . . . . . . . . . . . . . . . . . . 50

6.2.1 The Laplacians . . . . . . . . . . . . . . . . . . . . . . 526.2.2 The Lattice QCD Matrices . . . . . . . . . . . . . . . . 55

6.3 Extended Krylov Subspace Methods . . . . . . . . . . . . . . 576.3.1 The Lattice QCD matrices . . . . . . . . . . . . . . . . 586.3.2 The Laplacians . . . . . . . . . . . . . . . . . . . . . . 61

7 Conclusion 65

Bibliography 67

Index 69

iv

List of Algorithms

2.1 Arnoldi Process . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.7 Two-Sided Lanczos Process . . . . . . . . . . . . . . . . . . . . 154.4 Extended Krylov Subspace Process for EKm,m(A, b) . . . . . . . 305.1 Standard Krylov Subspace Method . . . . . . . . . . . . . . . . 395.3 Extended Krylov Subspace Method . . . . . . . . . . . . . . . . 425.6 Regensburg Method . . . . . . . . . . . . . . . . . . . . . . . . 455.7 EKS Equivalent to Regensburg Method . . . . . . . . . . . . . 45

v

List of Figures

6.1 Spectrum of A . . . . . . . . . . . . . . . . . . . . . . . . . . . 476.2 V H

m Vm gained with Arnoldi process, ‖Vm‖2 = 1.0000 . . . . . . 486.3 V H

mWm gained with two-sided Lanczos process,‖Vm‖2 = 6.8468, ‖Wm‖2 = 1.0351 · 103 . . . . . . . . . . . . . 49

6.4 V Hm,mVm,m gained with EKS process, ‖Vm,m‖2 = 1.0000 . . . . . 50

6.5 16× 16, expm, maxdim = 40, stepsize = 1 . . . . . . . . . . . 536.6 16× 16, sqrtm, maxdim = 40, stepsize = 1 . . . . . . . . . . . 536.7 32× 32, expm, maxdim = 80, stepsize = 2 . . . . . . . . . . . 546.8 32× 32, sqrtm, maxdim = 80, stepsize = 2 . . . . . . . . . . . 546.9 64× 64, expm, maxdim = 60, stepsize = 10 . . . . . . . . . . . 556.10 64× 64, sqrtm, maxdim = 240, stepsize = 10 . . . . . . . . . 556.11 128× 128, expm, maxdim = 120, stepsize = 40 . . . . . . . . . 566.12 128× 128, sqrtm, maxdim = 480, stepsize = 40 . . . . . . . . 566.13 44 lattice, conf = 5.0, signm, maxdim = 610, stepsize = 10 . . 576.14 44 lattice, conf = 6.0, signm, maxdim = 300, stepsize = 10 . . 576.15 44 lattice, conf = 5.0, dim level 1 = 1500 . . . . . . . . . . . . 586.16 44 lattice, conf = 6.0, dim level 1 = 1500 . . . . . . . . . . . . 596.17 44 lattice, conf = 5.0, dim level 1 = 450 . . . . . . . . . . . . . 596.18 44 lattice, conf = 6.0, dim level 1 = 200 . . . . . . . . . . . . . 606.19 64 lattice, dim level 1 = 1000, deflation . . . . . . . . . . . . . 616.20 64 lattice, dim level 1 = 500, deflation . . . . . . . . . . . . . . 616.21 64 lattice, dim level 1 = 1000, deflation . . . . . . . . . . . . . 626.22 16× 16 lattice . . . . . . . . . . . . . . . . . . . . . . . . . . . 626.23 32× 32 lattice . . . . . . . . . . . . . . . . . . . . . . . . . . . 636.24 64× 64 lattice, . . . . . . . . . . . . . . . . . . . . . . . . . . 636.25 128× 128 lattice . . . . . . . . . . . . . . . . . . . . . . . . . 64

vi

Chapter 1

Introduction

1.1 What is a Matrix Function?

The easiest functions one can think of are monomials, where one simplytakes the function argument z, multiplies it several times with itself andwith a constant a. The next step to make it more complex is to add severalmonomials what leads to the well known polynomials. All that was doneis multiplying with oneself or a constant and adding things up — thesearithmetic operations are also feasible with square matrices and so we do notwonder that substituting A for z leads to polynomials in a matrix A. Whenwe no longer look at polynomials but on the more general power series

∞∑k=0

akzk,

things get interesting. It is not possible to just substitute A for z any morejust as it is also not possible to insert any value for z in the power series. Theseries should converge to a finite value. This happens when z lies in a discaround zero whose radius is the radius of convergence of the given series.

We can derive a similar result for matrices, but therefore we must refreshour basic linear algebra knowledge, establish notation and go back to thepolynomials. What we know is:

Theorem 1.1 (Jordan Decomposition)Let A ∈ Cn×n. Then there are matrices V , J ∈ Cn×n such that

A = V JV −1,

1

1.1. WHAT IS A MATRIX FUNCTION? 2

where V is non-singular, J is block diagonal, J = diag(J1, . . . , Jp), with

(1.1) Jk = Jmk(λk) :=

λk 1

λk. . .. . . 1

λk

∈ Cmk×mk

and m1 + . . . + mp = n. The matrix J is unique up to the ordering of theJordan blocks Jk, the transforming matrix V is not unique.

Definition 1.2 (Index)For λ1, . . . , λs being the distinct eigenvalues of A, index (λi) is the order ofthe largest Jordan block in which λi appears.

Definition 1.3 (The Set of Polynomials)The set of polynomials of degree m over a ring R is denoted by PR

m .

We will restrict us to R = C.With the Jordan decomposition decomposition, a matrix A and a complexpolynomial p of degree m we get(1.2)

p(A) = p(V JV −1) =m∑j=0

aj(V JV−1)j = V (

m∑j=0

ajJj)V −1 = V p(J)V −1,

so all we need to know is what a polynomial does to a Jordan block. AJordan block Jmk

(λ) can also be written as Jmk(λ) = λEmk

+ N1mk

wherethe matrix N ν

mkdenotes the nilpotent matrix of size mk ×mk which is zero

except for the ν-th superdiagonal where it has ones as entries.With (N1

mk)m = Nm

mk, (N1

mk)m = 0mk

for m ≥ mk, the binomial theoremand the property λEm · A = A · λEm = λA we get

(Jmk(λ))m =

m∑j=0

(m

j

)λm−j(N1

mk)j =

m∑j=0

(m

j

)λm−jN j

mk,

and therefore

p(Jmk(λ)) =

m∑j=0

aj

j∑i=0

(j

i

)λj−iN i

mk.

We can extend this approach, at least formally, to power series where weobtain:

∞∑j=0

aj(Jmk(λ))j =

∞∑j=0

aj

j∑i=0

(j

i

)λj−iN i

mk.

1.2. THREE DEFINITIONS OF A MATRIX FUNCTION 3

We observe that all entries in a superdiagonal have the same value, and whenwe fix i we get as entry of the i-th superdiagonal

∞∑j=0

aj

(j

i

)λj−i =

∞∑j=0

1

i!aj · j · (j − 1) · . . . · (j − i+ 1)λj−i.

When λ lies in the area of convergence of our power series - which is thereC∞ and will now be named f - we get

∞∑j=0

1

i!aj · j · (j − 1) · . . . · (j − i+ 1)λj−i =

1

i!f (i)(λ).

This leads us to two important insights: When dealing with power series weneed f to be convergent on the spectrum of a given matrix A. But moreimportantly, we now have a first way to define an arbitrary matrix functionand what the function and a matrix it acts on, must fulfil.

Theorem 1.4 (Power Series of a Matrix)Let f(z) =

∑∞k=0 akz

k be a power series and A be in Cn×n with Jordandecomposition A = V JV −1. If spec(A) lies in the area of convergence of fthen f(A) exists and

(1.3) f(A) = V f(J)V −1 = V diag(f(Jk))V−1,

where

(1.4) f(Jk) =

f(λk) f ′(λk) . . . f (mk−1)(λk)

(mk−1)!

f(λk). . .

.... . . f ′(λk)

f(λk

.

1.2 Three Definitions of a Matrix Function

When looking at the last theorem, we see that if f is index(λk) − 1 timesdifferentiable at the eigenvalues λk, k = 1, . . . , s we still get a reasonabledefinition for what f(A) is. We want to use this as a definition of f beingdefined on the spectrum of A.

Definition 1.5Let A be a given matrix with spectrum spec(A) = {λ1, . . . , λs}, index(λk) =nk, k = 1, . . . , s and f be a given function. We say the function f is definedon the spectrum of A, if the values

f (j)(λk), j = 0, . . . , nk − 1, k = 1, . . . , s


exist.

Now we can state the first definition for f(A).

Definition 1.6 (Matrix Functions via Jordan Decomposition)Let f be defined on the spectrum of A. Then we have

(1.5) f(A) := V f(J)V −1 = V diag(f(Jk))V−1,

where

(1.6) f(Jk) =

f(λk) f ′(λk) . . . f (mk−1)(λk)

(mk−1)!

f(λk). . .

.... . . f ′(λk)

f(λk

.

When we look at the blocks f(Jk) we see that all we need are a few valuesof our function f and its derivatives. So the next idea for defining f(A) is totake a polynomial that interpolates f at the nodes f (j)(λk). This polynomialis called the interpolation polynomial in the Hermite sense.

Definition 1.7 (Matrix Functions via Hermite Interpolation)Let f be defined on the spectrum of A ∈ Cn×n an let ψ be the minimalpolynomial of A. Then f(A) := p(A) where p is a polynomial of degree lessthan ψ which fulfils the interpolation conditions

(1.7) p(j)(λk) = f (j)(λk) j = 0, . . . , nk − 1, k = 1, . . . , s

at the eigenvalues λ of A.

The polynomial p is unique and can be derived using the Lagrange-Hermiteformula

(1.8) p(t) =s∑i=1

ni−1∑

j=0

1

j!φ

(j)i (λi)(t− λi)j,

∏j 6=i

(t− λj)nj

where φi(t) = f(t)/

∏j 6=i(t− λj)nj . Another way to obtain p are the Newton

divided differences.

Definition 1.8 (Divided Differences)Let x0, x1, . . . , xn ∈ C be ordered so that identical points are numberedconsecutively. The divided differences of a function f at the points xk aredefined recursively by


f [xk] = f(xk),

f [xk, xk+1] =

{f(xk+1)−f(xk)

xk+1−xk, xk 6= xk+1

f ′(xk+1), xk = xk+1

,

f [x0, x1, . . . , xk+1] =

f [x1,x2,...,xk+1]−f [x0,x1,...,xk]

xk+1−x0, x0 6= xk+1

f (k+1)(xk+1)

(k+1)!, x0 = xk+1

.

By results of standard numerical analysis, e.g., [26], we then get the interpo-lation polynomial p by

(1.9) p(t) = f [x1]+f [x1, x2](t−x1)+. . .+f [x1, . . . , xm](t−x1) . . . (t−xm−1),

where m = deg ψ and {xi}mi=1 is the set of the s distinct eigenvalues λ1, . . . , λshaving index ni, see [13, p. 8]. We want to point out that the polynomialgained by this interpolation method highly depends on the eigenvalues of thematrix A and furthermore one cannot say that f now is the polynomial p forall matrices. This is only true if the matrix has the the same spectrum andthe same indices for all eigenvalues. If our matrix A is diagonalizable, weget that f(A) has the same eigenvectors as A and the eigenvalues are f(λk),k = 1, . . . , s.

There is an elegant third definition of what can be understood as a matrixfunction, but this time f needs to be analytic on and inside a closed pathwhich contains spec(A) in contrast to the earlier definitions where f onlyneeded to be a few times differentiable on the spectrum of A.

Definition 1.9 (Matrix Functions via Cauchy integral)Let A ∈ Cn×n and Γ be a closed contour that contains spec(A) and f afunction that is analytic in and on Γ. Then

(1.10) f(A) :=1

2πi

∫Γ

f(z)(zEn − A)−1dz.

Here the integral is to be understood entry for entry and (zEn−A)−1 alsois defined since z /∈ spec(A) and so (zEn − A) has full rank.

We will finish this section with a theorem that tells us that the threedifferent definitions of what a matrix function can be, are all equivalent:

Theorem 1.10 (Equivalence of the three Definitions of f(A))The definition of f(A) via Jordan decomposition is equivalent to the defini-tion via Hermite interpolation. Furthermore, if f is analytic, then Definition1.6, Definition 1.7 and Definition 1.9 are equivalent.

See [13, Theorem 1.12] for a proof.

1.3. KRYLOV SUBSPACES 6

1.3 Krylov Subspaces

Now that we know that a matrix function is more or less a polynomial we canthink of smarter methods than forming f(A) explicitly to compute the actionof a matrix function to a vector f(A)b. “Smarter” because an evaluation off(A) typically costs O(n3) flops which is too expensive or even not feasible.

Since f(A) is of the form p(A) for the Hermite interpolation polynomialfrom Definition 1.7 of degree m < deg ψ, we get that

f(A)b = p(A)b = (m∑j=0

ajAj)b =

m∑j=0

ajAjb.

This means that f(A)b is an element of the Krylov subspace spanned by Aand b.

Definition 1.11 (Standard Krylov Subspace (SKS))Given A ∈ Cn×n, b ∈ Cn and m ∈ N the subspace

Km(A, b) := span〈b, Ab,A2b, . . . , Am−1b〉

is called the m-th standard Krylov subspace spanned by A and b.

In the following two chapters we will disclaim the “standard” and simplyuse the term Krylov subspace. But when extended Krylov subspaces willbe introduced in Chapter 4, this specification will become necessary. Krylovsubspaces come with some nice properties, most of them do not need a specialproof.

Lemma 1.12Let A ∈ Cn×n, b ∈ Cn. Then the following properties hold:

1. Km(A, b) = {q(A)b : q is a polynomial of degree ≤ m− 1},

2. Km(A, b) ⊆ Km+1(A, b) and AKm(A, b) ⊆ Km+1(A, b),

3. There exists k ∈ N sucht that Km(A, b) = Km+1(A, b) for all m ≥ k.

From part 3 of the lemma we get that there exists a monic polynomialψA,b ∈ PC

k of minimal degree such that ψA,b(A)b = 0. From now on, thispolynomial will be referred to as the minimal polynomial of b with respectto A. Since it must divide the minimal polynomial of A, we see that ψA,b isgiven by(1.11)

ψA,b(λ) =s∏j=1

(λ− λj)nj ∈ PL, with L = L(A, b) =s∑j=1

nj = deg ψA,b(λ),

1.3. KRYLOV SUBSPACES 7

where s is the number of the distinct eigenvalues λk. Note that L was denotedk in Lemma 1.12, part 3.

With this and the definition of a matrix function we get for 2 polynomialsp, q and A ∈ Cn×n that p(A)b = q(A)b if and only if

p(j)(λi) = q(j)(λi), j = 0, . . . , ni − 1, i = 1, . . . , s,

where ni denotes the index of λi.We can now state a first result for the propagation operator f(A)b follow-

ing directly from Definition 1.7, ψA,b and the equality of polynomials:

Theorem 1.13Let A ∈ Cn×n, f be a function defined on the spectrum of A, b ∈ Cn andψA,b be the minimal polynomial of b with respect to A.

Then f(A)b = q(A)b, where q is the unique Hermite interpolating polyno-mial of degree less than L that satisfies the interpolating conditions

q(j)(λi) = f (j)(λi), j = 0, . . . , ni − 1, i = 1, . . . , s,

where ni denotes the multiplicity of λ1 as a root of ψA,b.

The theorem tells us that the dimension of the Krylov subspace that con-tains f(A)b depends on both, A and b, and is of dimension ≤ L.

Chapter 2

Krylov Subspace Methods

In the last chapter we saw that f(A)b is just a polynomial q(A)b and thereforean element of a Krylov subspace KL(A, b). The idea behind Krylov subspaceapproximations to f(A)b is projecting the matrix A onto the Krylov subspaceKm(A, b) for m � n, where m not necessarily reaches the degree of q. Theprojection of A and the generation of a basis of Km(A, b) are computed usingthe well known Arnoldi or two-sided Lanczos process. They provide betterbases than the power basis {b, Ab, . . . , Am−1} in which the vectors tend toloose their linear independence with increasing m, since they tend towardsan eigenvector belonging to the eigenvalue of biggest modulus.

In this chapter we will present the Arnoldi and two-sided Lanczos processesand explain how they can be used to approximate f(A)b.

2.1 The Arnoldi Process

Several motivations lead to the Arnoldi process, one includes finding coconutson a deserted island [28], but probably the main two are reducing an arbitrarymatrix A to Hessenberg form and generating an orthonormal basis of a Krylovsubspace Km(A, b).

For a given integer m, a matrix A ∈ Cn×n and an arbitrary vector b ∈ Cn,the following algorithm produces an orthonormal basis Vm = [v1, . . . , vm] ofKm(A, b) and an upper m×m Hessenberg matrix Hm .

8

http://www.youtube.com/watch?v=OfOACVCOJsM

http://www.youtube.com/watch?v=B1O2jcfOylU

2.1. THE ARNOLDI PROCESS 9

Algorithm 2.1: Arnoldi ProcessInput : A ∈ Cn×n, b ∈ Cn, an integer mOutput: Vm = [v1, . . . , vm] ∈ Cn×m , Hm = (hi,j) ∈ Cm×m

v1 := b/‖b‖1

for j = 1, . . . ,m do2

w := Avj3

for i = 1, . . . , j do4

/* modified Gram-Schmidt */

hij := wHvi5

w = w − hijvi6

hj+1,j := ‖w‖7

vj+1 := w/hj+1,j8

This is a very simplistic implementation of the Arnoldi process. E.g. letL = deg ψA,b, then we have hL+1,j = 0 and the algorithm should stop atthat point since we found the invariant subspace of A and b. This case isreferred to as a “lucky breakdown”. From now on we will assume m ≤ L.The orthogonalization process in lines 4 to 6 is implemented in the modifiedGram-Schmidt manner, making the algorithm more stable in contrast to thestandard Gram-Schmidt process. Nevertheless, in finite precision arithmetica loss of orthogonality will occur and can partly be avoided by reorthogonal-ization or using Householder transformations for the orthogonalization.

We now want to take a closer look on the generated matrices Hm andVm. First we see that v1 is the normalized vector b. In the j-th step thealgorithm produces the entry hj+1,j and a vector vj+1. So after the m-thstep, the algorithm already produced an orthonormal basis of Km+1(A, b)and a m ×m upper Hessenberg matrix Hm. The relations between A, Hm

and Vm might be derived the following way.When we look at Avk for some k ≤ m and rewrite lines 4 to 6 we get that

(2.1) Avj =

j+1∑i=1

hijvi,

so that with line 8 we obtain

hj+1,jvj+1 = Avj −j∑i=1

hijvi.

Writing this in a matrix manner, we obtain the Arnoldi decomposition ofKm(A, b),

(2.2) AVm = VmHm + hm+1,mvm+1eTm.


Herein, column j expresses the relation (2.1) for j = 1, . . . ,m. In the casem = L we get the even nicer result AVm = VmHm because vL+1 = 0. This isdue to the fact that vL+1 ∈ KL(A, b) and vL+1 is orthogonalized against allprevious basis vectors.

Having in mind that Vm has orthonormal columns, vm+1 is orthogonal tothe columns of Vm, and multiplying both sides of (2.2) with V H

m from the leftwe get

V Hm AVm = Hm.

This can be understood as a compression of A onto the Krylov subspaceKm(A, b).

In the following we need f to be defined in the field of values of A.

Definition 2.2 (Field of Values)For a given matrix A ∈ Cn×n, F (A) denotes the field of values of A, which isthe image of the unit sphere under the quadratic form induced by the matrix.

F (A) ={xHAx |x ∈ Cn, ‖x‖2 = 1

}.

More precisely, since we want f(Hm) to be defined, f has to exist at thepoints of the spectrum of Hm and derivatives of f have to exist according tothe index of the eigenvalue. Since Hm = V H

m AVm we have spec(A) ⊆ F (A),so it is safe to assume that f and its derivatives (as needed) are defined onF (A).

The following considerations will show that it is reasonable to try to ap-proximate f(A)b on Km(A, b) via the Arnoldi approximation

(2.3) fm := βVmf(Hm)e1, where β = ‖b‖.

Using Definition 1.7, we get that f(Hm) is equivalent to q(Hm) where q isthe Hermite interpolation polynomial fulfilling the interpolation conditionsat the Ritz values.

Definition 2.3 (Ritz Values)The eigenvalues of Hm are called Ritz values.

We can now write (2.3) as

(2.4) fm = βVmf(Hm)e1 = βVmq(Hm)e1.

The question is under which circumstances the approximation fm is exact.For f being a polynomial, the following result can be provided.


Lemma 2.4 (Exactness of Polynomials)Let A ∈ Cn×n be an arbitrary matrix and Vm, Hm be the result of m steps ofthe Arnoldi process. For any polynomial p ∈ PC

j with j ≤ m− 1 there holds

(2.5) p(A)v1 = Vmp(Hm)e1.

Proof. We follow the proof in [22]. It suffices to only verify the equation formonomials, so we need to show that Akv1 = VmH

kme1 for k = 0, . . . ,m − 1.

This will be done by induction. For j = 0 the equality clearly holds becausewe only multiply with a scalar. Suppose now that it is true for j ≤ m − 2.Let πm := VmV

Hm be the orthogonal projector onto Km(A, b). With Aj+1v1

and Ajv1 belonging to Km(A, b) we get

Aj+1v1 = πmAj+1v1 = πmAA

jv1 = πmAπmAjv1.

Using (2.1) gives us

πmAπm = VmVHm AVmV

Hm = VmHmV

Hm

and with the induction hypothesis Ajv1 = VmHjme1 it follows that

Aj+1v1 = VmHmVHm VmH

jme1 = VmH

j+1m e1.

�

From the lemma we see that (2.4) can now be written as

(2.6) fm = βVmq(Hm)e1 = q(A)b,

because q is of degree m− 1.Lemma 2.4 only made use of the first m vectors gained in the Arnoldi

process, although we already computed an orthonormal basis of Km+1(A, b).In [8, 7] a result for polynomials of degree m is given:

Corollary 2.5Let A ∈ Cn×n be an arbitrary matrix and Vm, Hm be the result of m stepsof the Arnoldi process. For any polynomial p ∈ PC

m, p(λ) =∑m

j=0 αjλj there

holds

(2.7) p(A)b = β(Vmp(Hm)e1 + αmγmvm+1),

where γm =∏m

j=1 hj+1,j, β = ‖b‖.In particular, when m ≥ deg ψA,b = L then

p(A)b = βVmp(Hm)e1 = βVLp(HL)e1.


Proof. The lemma only needs to be shown for monomials and makes use ofLemma 2.4 and its proof. We need to show that Amb = β(VmH

mme1+γmvm+1).

Amb = A Am−1b︸︷︷︸βVmH

m−1m e1

= β AVm︸︷︷︸(2.2)

Hm−1m e1

= β(VmHm + hm+1,mvm+1eTm)Hm−1

m e1

= β(VmHmme1 + hm+1,mvm+1e

TmH

m−1m e1)

= β(VmHmme1 + hm+1,me

TmH

m−1m e1vm+1).

It is left to show that the scalar hm+1,meTmH

m−1m e1 is equal to γm =

∏mj=1 hj+1,j.

The scalar eTmHm−1m e1 is equal to the entry in the lower left corner of Hm−1

m .We show by induction that eTkH

k−1m e1 =

∏k−1j=1 hj+1,j, 2 ≤ k ≤ m.

For k = 2 we get Hme1 = h11e1 + h21e2, so eT2Hme1 = h21.We are interested in the (k + 1)-th entry of the k-th image of e1 under Hm,that is Hk

me1. Since the matrix Hm is upper Hessenberg, the image of thevector ej, j = 1, . . . ,m− 1, i.e. Hmek, is a linear combination of the vectorse1, . . . , ej, ej+1. The factors in the linear combinations correspond to teh j-thcolumn of Hm. For k = 3 we get Hk−1

m e1 = H2me1 = Hm(h11e1 + h21e2). We

seeh21Hme2 = ∗e1 + ∗e2 + h21h32e3,

where ∗ are some numbers. The third entry is what we are looking for andhas the value h21h32.The induction step k → k + 1 is the task to obtain the (k + 1)-th entry ofthe k-th image of e1. We have

ek+1Hkme1 = ek+1HmH

k−1m e1 = ek+1Hm(?e1 + ?e2 + . . .+

k−1∏j=1

hj+1,jek),

where the ? are some numbers. The last step is to write

ek+1Hm(?e1 +?e2 + . . .+k−1∏j=1

hj+1,jek) = ek+1(♥e1 + . . .+♥ek+k∏j=1

hj+1,jek+1),

where again the ♥ are some numbers which are not of our interest. With theorthogonality of the ej we obtain

ek+1Hkme1 =

k∏j=1

hj+1,jek+1.


The extension of the lemma for m ≥ L directly follows from the maximaldimension of Km(A, b). �

Let us have a look at the vectors v1, . . . , vm+1. Supposing m < deg ψA,b − 1,which assures that Km(A, b) is of dimension m and vm+1 6= 0, for each of thevj there exists a unique polynomial wj−1 ∈ PC

j−1 such that

(2.8) vj = wj−1(A)b.

Substituting λ for A in (2.2) yields

(2.9) λ[w0(λ), . . . , wm−1(λ)] =

[w0(λ), . . . , wm−1(λ)]Hm + hm+1,m[0, . . . , 0, wm(λ)],

and we notice that if wm(λ) = 0, then λ must be an eigenvalue of Hm with aleft eigenvector. Note that the substitution is not self-evident or reasonablesince we tacitly dropped the vector b. In fact, relation (2.9) is obtainedwhen we understand equation (2.1) as a polynomial in several variables withcoefficients hi,j, arguments vi and use the nestedness the vectors vj possess.

The Hessenberg matrixHm is unreduced sincem does not exceed deg ψA,b−1, i.e. the subdiagonal elements satisfy hj+1,j 6= 0. From this we getdim(ker(Hm − λEm)) ≤ 1, so every eigenvalue of Hm has geometric multiplic-ity 1. By this it follows that for each eigenvalue λ of Hm, there is only oneJordan block. We want to use this property to prove the following lemma.

Lemma 2.6With the polynomials wj, j = 0, . . . ,m, as defined in (2.8) and the identity(2.9) there holds

(2.10) χHm(λ) =1

αmwm(λ),

where χHm(λ) is the characteristic polynomial of Hm and αm denotes theleading coefficient of the polynomial wm.

Proof. Let us suppose that of all eigenvalues λj, j = 1, . . . , k of Hm withindex (λj) = νj, wm possesses ` zeroes λj with multiplicity ρj, j = 1, . . . , `.For every eigenvalue of Hm we have only one Jordan block of size index (λj),so the index of every eigenvalue of Hm is equal to its algebraic multiplicity,leading us to

χHm(λ) =k∏j=1

(λ− λj)νj .

2.2. THE TWO-SIDED LANCZOS PROCESS 14

We know that the zeros of wm are eigenvalues, but we do not now whetherwm has every eigenvalue of Hm as a root and if so, which multiplicity theypossess. All in all we have to show that ` = k and ρj = νj, j = 1, . . . , k.

Without loss of generality, we may suppose that λ1 is a (multiple) zero ofwm and ρ1 ≥ ν1.Differentiating (2.9) with respect to λ, we obtain the equation

[w0(λ), . . . , wm−1(λ)] + λ[w′0(λ), . . . , w′m−1(λ)] =

[w′0(λ), . . . , w′m−1(λ)]Hm + hm+1,m[0, . . . , 0, w′m(λ)],

inserting λ1 leads to

(2.11) [w0(λ1), . . . , wm−1(λ1)] + λ1[w′0(λ1), . . . , w′m−1(λ1)] =

[w′0(λ1), . . . , w′m−1(λ1)]Hm,

because λ1 is a multiple root of wm and so is a zero of w′m. With λ1 being aneigenvalue of Hm we know that [w0(λ1), . . . , wm−1(λ1)] is an eigenvector. Inaddition, [w0(λ1), . . . , wm−1(λ1)] is the only true eigenvector with eigenvalueλ1, because Hm has only one Jordan block for every eigenvalue. Furthermore,in (2.11) we see that [w′0(λ1), . . . , w′m−1(λ1)] is a generalized eigenvector ofHm. We may differentiate (2.9) ν1 − 1 times and still the generalized eigen-

vector property of [w(j)0 (λ1), . . . , w

(j)m−1(λ1)] holds. For the ν-th derivative,

the property can no longer hold, because otherwise the size of the Jordanblock of λ1 would be larger than ν1. It follows that ρ1 = ν1. The sameargumentation holds true for every other multiple root of wm with ρj ≥ νj.On the other hand, wm can not possess zeros with multiplicity smaller thantheir multiplicity in χHm because wm is of degree m.Together with the property that every root of wm is an eigenvalue and wm isnot monic, the lemma follows. �

Using the lemma, equation (2.7) leads to wm(A)b = βαmγmvm+1 becausewm(Hm) = 0, but on the other hand we have wm(A)b = vm+1 by definition.So we obtain

(2.12) βαmγm = 1.

2.2 The Two-Sided Lanczos Process

One drawback of the Arnoldi approximation is that we always have to storethe basis Vm of the Krylov subspace Km(A, b) explicitly. This becomes a cru-cial point when the storage space is limited and results in a natural bound


for the problem size n or the Krylov subspace size m. Further, the compu-tational work increases linearly with m since we have to orthogonalize everynew basis vector against all precedent basis vectors.

If the matrix A is hermitian the Arnoldi process reduces to the Lanczosprocess and results in a three term recurrence for computing an orthonormalbasis of Km(A, b). We could use this to generate the Arnoldi approximationon the fly without explicitly storing the whole basis of Km(A, b).

For arbitrary matrices A, a similar method exists which is called the two-sided Lanczos or bi-Lanczos process and generates two bases Vm, Wm for theKrylov subspaces Km(A, b) and Km(AH , b) respectively. Here the columns ofthe matrices Vm, Wm can be computed in a 6 term recurrence, so this onlyrequires the storage of six vectors instead of an n×m matrix in the Arnoldicase. Furthermore, the compression of A gained in the two-sided Lanczosprocess is only tridiagonal instead of upper Hessenberg, and therefore needsless storage, in fact only 3m− 2 instead of m2

2+ 3

2m− 1 numbers.

Algorithm 2.7: Two-Sided Lanczos ProcessInput : A ∈ Cn×n, b ∈ Cn, an integer mOutput: Vm = [v1, . . . , vm] ∈ Cn×m, Wm = [w1, . . . , wm] ∈ Cn×m ,

Tm ∈ Cm×m

v1 := b/‖b‖1

Put w1 = v1 .2

Set β0 = γ0 = 0 and v0 = w0 = 0.3

for j = 1, . . .,m do4

v := Av15

w := AHw16

α(j) := wH1 v7

v = v − α(j)v1 − β(j − 1)v08

w = w − α(j)w1 − γ(j − 1)w09

γ(j) := ‖v‖10

v0 = v111

v1 = v/γ(j)12

V (:, j + 1) := v113

β(j) := wHvj14

w0 = w115

w1 = w/β(j)16

W (:, j + 1) := w117

Tm = diag(α(1 : m)) + diag(γ(1 : m− 1),−1) + diag(β(1 : m− 1), 1)18

Note that the algorithm does save the matrices Vm,Wm in contrast to the


proposed procedure.We want to make the section of the two-sided Lanczos process short and

just present some major results. For a more detailed analysis see [11, 23].In the algorithm we chose w1 = v1, which is equivalent to choosing b = b.

In general it suffices to choose w1, and therefore b, such that wH1 v1 = 1, see[11]. Whenever A is hermitian, the two-sided Lanczos process reduces to thehermitian Lanczos Process if one takes w1 = v1.

The matrix Tm is tridiagonal and has the form

Tm =

α1 β1

γ1. . . . . .. . . . . . βm−1

γm−1 αm

.The matrices Vm and Wm are biorthogonal, i.e. V H

mWm = WHm Vm = Em.

Although the columns of Vm = [v1, . . . , vm] and Wm = [w1, . . . , wm] are nor-malized, they are not orthogonal in general. This leads to possibly largenorms of Vm or Wm.

We get two decompositions similar to the Arnoldi decomposition (2.2)

AVm = VmTm + γmvm+1eTm(2.13)

AHWm = WmTHm + βmwm+1e

Tm.(2.14)

This property is a gem; we are able to use the results shown for polynomialsand Arnoldi decompositions, e.g. the exactness of polynomials of degree m−1in Lemma (2.4) or the exactness for polynomials of degree m in Corollary(2.5). The only thing we have to change in the argumentation of the proofis the projector πm. For the two-sided Lanczos process we have to substituteπm = VmV

Hm with ωm := VmW

Hm . The rest is the same word for word. We

are also able to define a reasonable Lanczos approximation

(2.15) fm = βVmf(Tm)e1, where β = ‖b‖.

Note that we do not need Wm for the approximation. One downside howeveris that the computation of the matrices Vm, Wm and Tm might be highlyaffected by rounding errors, so in practice results may be gained which arefar from the identity above.

The Unlucky Breakdown

Unlike the Arnoldi process, in the two-sided Lanczos process an unluckybreakdown may occur. Generally, a breakdown might occur in line 14 when-ever wHvj = 0. If one of the vectors w, vj vanishes we have something related


to the lucky breakdown, and an invariant subspace, either Vj or Wj, is found.A downside though might be that we found the wrong one, namely Wj, theone associated to AH and b. Then nothing can be said about the approxi-mation for f(A)b and the breakdown can therefore be called unlucky.

If we found the wanted invariant subspace, that is when vj = 0, then theapproximation is exact, at least in exact arithmetic.

The last reason why wHvj can become zero was called serious breakdownby Wilkinson [29]. This occurs whenever w and vj are orthogonal and w 6=0 6= vj. Then the algorithm is simply not applicable to our choice of b. Ofcourse, this breakdown is also unlucky.

There are strategies for solving linear systems called look ahead Lanczoswhich aim at circumventing serious or unlucky breakdowns, see [23] since wewill not get into this subject.

Conclusion

The ability of both, the Arnoldi and the two-sided Lanczos process, to ex-actly approximate polynomials up to a specific degree leads us to not longerdistinguish between the two methods. We are able to treat the matricesgained in each of the two processes in the same manner.

In the following we will therefore use the same unified notation — Hm

for the compression of A, Vm for the matrix containing the basis vectors forKm(A, b) — unless we mention it explicitly.

http://www.youtube.com/watch?v=l9DPmPxIZhc

Chapter 3

Error Analysis

Our approximations fm (2.6) to f(A)b was motivated via the polynomialinterpolation property on the spectrum of the reduced matrix Hm. Thoughthis seems intuitively a reasonable approach, it is very desirable to have acloser look at the error of this approximation and to derive for estimatingor bounding this error. This is the subject of the present chapter, which isbased on work from [1].

3.1 Basic Error Representation

Let us suppose, we know the polynomial q ∈ PCm−1 that interpolates f in the

Hermite sense at the eigenvalues of Hm gained by m steps of the Arnoldiprocess for a matrix A ∈ Cn×n and a vector b ∈ Cn.

Using results and notation from [4, 7] , we define the following functionfor scalars in C

(3.1) ∆χf :=f − qχ

to be the mth order divided difference of f with respect to the not necessarilydistinct roots ϑ1, . . . , ϑm of the polynomial χ ∈ PC

m and q ∈ PCm−1. Here, the

polynomial χ shall denote the characteristic polynomial of Hm, so its rootsare the Ritz values of Hm. It is also possible to take any polynomial as χ aslong as its roots are the Hermite interpolation points of q.

Note that it is reasonable to call ∆ divided difference; adding the variablet in Definition 1.8 to the nodes θk yields exactly ∆χf(t) = f(t)−q(t)

χ(t).

It is not clear whether ∆ exists or is defined in a proper way since we have

18

3.1. BASIC ERROR REPRESENTATION 19

a singularity at the interpolation points. Luckily, the following lemma showsits existence for scalars.

Lemma 3.1Let the above notation hold. Then for any t ∈ C the value

[∆χf ](t) =f(t)− q(t)

χ(t)

exists.

Proof. For t /∈ {ϑ1, . . . , ϑm} the quotient exists since χ(t) 6= 0.Now let t ∈ {ϑ1, . . . , ϑm} have multiplicity k as a zero of χ.For j = 0, . . . , k − 1 we have χ(j)(t) = 0 and f (j)(t) − q(j)(t) = 0, butχ(k)(t) 6= 0 and f (k)(t) − q(k)(t) 6= 0 because of the Hermite interpolationproperty of q. So applying l’Hospital’s rule k times leads us to

[∆χf ](t) =f(t)− q(t)

χ(t)=f (k)(t)− q(k)(t)

χ(k)(t),

and we therefore obtain a reasonable value for [∆χf ](t). �

In the proof of the lemma, we tacitly assumed that f be k times differ-entiable at the eigenvalues of A with multiplicity k. That is not necessarilythe case for any f for which f(A) is defined, because f only needs to bek − 1-times differentiable at these values, but it is essential for the proof ofthe existence of [∆χf ](t). The extension of this lemma to matrix argumentsfollows from the definition of matrix functions and its dependence on theeigenvalues. Consequently, the existence of [∆χf ](A) affords the existence ofdj

dtj[∆χf ](λ) for λ being an eigenvalue of A with index(λ) = j, which can be

assured if f is sufficiently smooth. Nevertheless, the request that f and someof its derivatives are defined on F (A) is far weaker than claiming that f isanalytic in a domain containing F (A). In the following we will assume thatf fulfils the above demand. Now that we showed that [∆χf ](z) is definedfor all z ∈ C, and ∆ also exists for matrix arguments, we are able to state afirst result for the quality of our approximation fm.

Lemma 3.2 (Basic Error Representation)Let A ∈ Cn×n, b ∈ Cn, β = ‖b‖, and f be a function that fulfil our demandsin F (A). After m steps of the Arnoldi process, the approximations fm as in(2.3) fulfil

(3.2) f(A)b− fm = βγm[∆χHmf ](A)vm+1

with γm =∏m

j=1 hj+1,j, and χHmis the characteristic polynomial of Hm.

3.2. ADDING AUXILIARY NODES 20

Proof. We can convert the definition of ∆ from (3.1) for scalars to

f(t)− q(t) = [∆χHmf ](t)χHm

(t),

where q denotes the interpolation polynomial of f in the Hermite sense at theroots of χHm

, which at the same time are the Ritz values of Hm. Inserting Aas a matrix argument and multiplying both sides from the right with b thenyields

f(A)b− q(A)b = [∆χHmf ](A)χHm

(A)b.

Using Corollary 2.5 for χHm(A)b , Lemma 2.4 for q(A)b and equation (2.6)

we obtain

f(A)b− fm = [∆χHmf ](A)(β(VmχHm

(Hm)e1 + αmγmvm+1)).

Fortunately, χHm(Hm) = 0 and χHm

is monic, so αm = 1, we arrive at

f(A)b− fm = βγm[∆χHmf ](A)vm+1.

�

3.2 Adding Auxiliary Nodes

It was Saad [22], who stated a posteriori error estimates for the matrix ex-ponential propagation operator by adding auxiliary nodes in addition to theRitz values. A power series that represents the error can be deduced and atruncation of it then offers an error indicator. At the auxiliary nodes, thevalues for f = exp were known — the nodes were chosen to be zero, the valueof f therefore is 1.

We want to generalize this idea from the exponential to arbitrary functionsand appropriate auxiliary nodes.

Let θ1, . . . , θm ∈ C be the auxiliary nodes such that f(θj) is defined forevery j in the Hermite sense. Θj(λ) denotes the jth nodal polynomial, thatis the unique monic polynomial of degree j with the first j nodes as roots.

(3.3) Θ0(λ) := 1, Θj(λ) =

j∏k=1

(λ− θk), for j = 1, . . . , m.

To distinguish between interpolation at the Ritz values and interpolation atauxiliary nodes and to simplify notation, we denote the divided differencesby

(3.4) φ0(λ) := f(λ), φj(λ) = [∆Θjf ](λ), for j = 1, . . . , m.


Of course, the interpolation identities for divided differences and the usualrecurrences still hold:

φj(λ) =φj−1(λ)− φj−1(θj)

λ− θj, for j = 1, . . . , m

and for the interpolation polynomial qj+1 of f of degree j at the nodes θk,k = 1, . . . , j + 1 we have

qj+1 = qj + φjΘj.

Now, we want to combine the results gained by the Arnoldi approximationand the auxiliary node idea.

After m steps of the Arnoldi process, we produced the vector vm+1. From(3.3) we get the relation

Θj+1(λ) = Θj(λ) · (λ− θj+1) = λΘj(λ)− θj+1Θj(λ)

which can also be stated as

λΘj(λ) = Θj+1(λ) + θj+1Θj(λ).

So, looking at the vectors AΘj(A)vm+1 we get

AΘj(A)vm+1 = Θj+1(A)vm+1 + θj+1Θj(A)vm+1,

and for the matrices Wm := [Θ0(A)vm+1, . . . ,Θm−1(A)vm+1] and the bidiag-onal matrix

Bm :=

θ1

1 θ2

. . . . . .

1 θm

∈ Cm×m

we obtain the familiar looking relation

(3.5) AWm = WmBm + Θm(A)vm+1eTm.

We note that the vectors Θj(A)vm+1 belong to the Krylov subspaceKm+1+j(A, b).When we now combine the Arnoldi decomposition (2.2) and (3.5) via the

block matrix H, which is in upper Hessenberg form,

(3.6) H =

[Hm 0

hm+1,me1eTm Bm

], where e1 ∈ Cm

we may write

(3.7) A[VmWm] = [VmWm]H + Θm(A)vm+1eTm+m.


Analogously to (2.3) we may now approximate f(A)b by

fm+m := β[VmWm]f(H)e1.

The question, whether this is reasonable or not, though still remains to beanswered. First of all, our interest lies therefore in finding a result equivalentto Lemma 3.2, which now makes use of the matrix H.Looking at the proof of Lemma 3.2, we notice that the ability of Hm to ex-actly approximate polynomials of degree m− 1 is substantial. With provinga result similar to Lemma 2.4 with H instead of Hm, we therefore would havemade a large step towards an error representation involving H.Let us first discuss this aspect. We know that b = β[VmWm]e1. This comesnaturally from the Arnoldi process because the first column of Vm is b nor-malized. In the following, we want to assume that ‖b‖ = β = 1.

Together with (3.7), we get

Ab = A[VmWm]e1

= [VmWm]He1 + Θm(A)vm+1eTm+me1,

and since em+m and e1 are orthogonal, we get

(3.8) Ab = [VmWm]He1.

For A2b we get

A2b = A(Ab) = A([VmWm]He1)

= ([VmWm]H + Θm(A)vm+1eTm+m)He1

= [VmWm]H2e1 + Θm(A)vm+1eTm+mHe1,

and again we see that em+m and He1 are orthogonal if m+ m ≥ 3, so

A2b = [VmWm]H2e1.

At this point, we need to assume that m+m is greater than the degree of thepolynomial for which we want to make a statement. Proceeding in the aboveway and using results from the proof of Corollary 2.5 concerning the imagesof the first unit vector under powers of a Hessenberg matrix, the relation

q(A)b = [VmWm]q(H)e1

holds for all polynomials q ∈ PCm+m−1, because the last term in the sum, that

is Θm(A)vm+1eTm+mH

je1, j < m+ m− 1, is zero due to the orthogonality of

em+m and Hje1.


This orthogonality gets lost when deg(q) = m + m, but again the proof

of the corollary gives us the hint that eTm+mHm+me1 is the product of the

subdiagonal entries of H. For polynomials q ∈ PCm+m we may now write

q(A)b = [VmWm]q(H)e1 + αm+mγm+mΘm(A)vm+1.

In a nutshell, we may say that Lemma 2.4 and Corollary 2.5 also hold for H.We now use χ eH(λ) = Θm(λ)χHm

(λ), the monicity of characteristic poly-nomials, the conversions applied to ∆ in the proof of Lemma 3.2 and thefact that the subdiagonal entries of Bm are all of value 1, to get the followingerror representation for fm+m, involving H and the corresponding character-istic polynomial.

Corollary 3.3Let A ∈ Cn×n, b ∈ Cn, β = ‖b‖, θ1, . . . , θm ∈ C be the auxiliary nodes and fbe a function such that f(A), f(θj), 1 ≤ j ≤ m are defined. After m stepsof the Arnoldi process, the approximations fm+m from (2.3) fulfil

(3.9) f(A)b− fm+m = βγ[∆χeHf ](A)Θm(A)vm+1,

with γ = γm =∏m

j=1 hj+1,j, and χ eH the characteristic polynomial of H.

For the evaluation of fm+m, we need the first column of f(H). The followinglemma states two results, one concerning the evaluation of f(Bm) and itsrelation to divided differences [19], and the other one being a special case ofthe lemma itself [21] .

Lemma 3.4 (f(H))For H as in (3.6) and an appropriate function f , there holds

(3.10) f(H) =

[f(Hm) 0Fm,m f(Bm)

]

with(3.11)

f(Bm) =

f(θ1)f [θ1, θ2] f(θ2)

... f [θ2, θ3] f(θ3)

...... f [θ3, θ4]

. . .

f [θ1, . . . , θm] f [θ2, . . . , θm] f [θ3, . . . , θm] . . . f(θm)

∈ Cm×m,


where the f [. . .] are the divided differences as in Definition 1.8.Furthermore, Fm,m ∈ Cm×m has rows

(3.12) eTj Fm,m = hm+1,meTmφj(Hm), j = 1, . . . , m.

Proof. The structure of f(H) is clear since a polynomial does not affect thestructure of a bloc triangular matrix.

The equality (3.11) goes back to Opitz, who called the Bm “Steigungsma-trizen” and showed the connection between computing the divided differencesat the nodes θj and evaluating f(Bm) which is literally the same. The proofcan be found in [19].

The main work now, which is proving equation (3.12), follows the prooffor the special case f = exp and θj = 0 for all j, which is due to Philippeand Sidje and can be found in [21].

One has f(H)H = Hf(H) since f(H) = q(H) for q being the Hermiteinterpolation polynomial, and for all polynomials p and all square matricesA there holds Ap(A) = p(A)A.

Looking at f(H)H and Hf(H), we get

f(H)H =

[f(Hm) 0Fm,m f(Bm)

][Hm 0

hm+1,me1eTm Bm

]and

Hf(H) =

[Hm 0

hm+1,me1eTm Bm

][f(Hm) 0Fm,m f(Bm)

],

respectively.Now comparing the (2, 1) blocks of the products yields

Fm,mHm + f(Bm)hm+1,me1eTm = hm+1,me1e

Tmf(Hm) +BmFm,m

or equivalently

(3.13) Fm,mHm −BmFm,m = hm+1,m

(e1e

Tmf(Hm)− f(Bm)e1e

Tm

).

Multiplying the difference (3.13) with eT1 from the left offers

eT1 (Fm,mHm −BmFm,m) = hm+1,m(eTmf(Hm)− f(θ1)eTm︸︷︷︸=eT

mf(θ1)Em

),

soeT1 (Fm,mHm −BmFm,m) = hm+1,me

Tm(f(Hm)− f(θ1)Em).


Using eT1Bm = θ1eT1 and multiplying with (Hm − θ1Em)−1from the right,

which seems questionable if θ1 is a Ritz value of Hm but is justified byLemma 3.1 and its consequences on matrices, we have

eT1 Fm,m(Hm − θ1Em) = hm+1,meTm(f(Hm)− f(θ1)Em),

and

eT1 Fm,m = hm+1,meTm

f(Hm)− f(θ1)EmHm − θ1Em︸︷︷︸

=φ1(Hm)

,

so finally for the first row of Fm,m we get

eT1 Fm,m = hm+1,meTmφ1(Hm).

An induction over j will show the hypothesis to be true for the remainingrows 2, . . . , m.

We look at equation (3.13) again and multiply it with eTj from the left:

eTj Fm,mHm − eTj Bm︸︷︷︸=eT

j−1+θjeTj

Fm,m = hm+1,m(eTj e1eTm︸︷︷︸

=0

f(Hm)− eTj f(Bm)e1eTm),

and together with θjeTj Fm,m = eTj Fm,m(θjEm) we obtain

eTj Fm,m(Hm − θjEm)− eTj−1Fm,m = −hm+1,meTj f(Bm)e1e

Tm),

which can be written as

eTj Fm,m(Hm − (θjEm)) = eTj−1Fm,m − hm+1,meTj f(Bm)e1e

Tm).

The induction hypothesis for eTj−1Fm,m yields hm+1,meTmφj−1(Hm) and eTj f(Bm)e1

is simply φj−1(θj), so we may write

eTj Fm,m(Hm − θjEm) = hm+1,meTm(φj−1(Hm)− φj−1(θj)Em),

and the definition of φj and the dubious, yet justified division by (Hm−θjEm)finally leads us to

eTj Fm,m = hm+1,meTmφj(Hm).

�

Now we may reap what we sowed: from Lemma 3.4 we get an identity forfm+m which can be used as an error indicator for the approximation fm,

(3.14) fm+m = Vmf(Hm)e1 + hm+1,m

m∑j=1

[eTmφj(Hm)e1]Θj−1(A)vm+1.

This, together with Corollary 3.3 yields the following theorem.


Theorem 3.5 (Error Representation with Auxiliary Nodes)Let A ∈ Cn×n, b ∈ Cn, θ1, . . . , θm ∈ C. After m steps of the Arnoldi process,the approximations fm to the function f , which should be defined on thespectrum of A, the spectrum of Hm and the values θj satisfy

(3.15) f(A)b− fm = hm+1,m

m∑j=1

[eTmφj(Hm)e1]Θj−1(A)vm+1

+ βγ[∆χeHf ](A)Θm(A)vm+1,

where β = ‖b‖, γ = γm =∏m

j=1 hj+1,j and χ eH is the characteristic polynomial

of H.

Chapter 4

Extended Krylov Subspaces

It comes naturally that the Hessenberg matrices Hm contain informationmainly about the eigenvalues of A with largest modulus. This stems fromthe fact that the construction of the basis of the Krylov subspace Km(A, b) isclosely related to the power method which reveals eigenvectors correspondingto large eigenvalues. Some functions, like the exponential, mainly depend onthe largest modulus eigenvalues, so the Arnoldi approximations yield in theright direction. Other functions, like the sign function, behave differently.

For a scalar z ∈ C, the sign function is defined as

(4.1) sign(z) =

{1, Re(z) > 0−1, Re(z) < 0

.

Note that the sign function is not defined for values z which lie on the imag-inary axis.

We apply the sign function to a matrix A whose eigenvalues fulfil theabove restriction. The Jordan decomposition approach yields

sign(A) = V sign(J)V −1.

Let the Jordan matrix J be arranged such that the p eigenvalues from theright open half-plane are followed by the q eigenvalues from the left half-plane, where p+ q = n is the size of J . Since the sign function is analytic inthe open left and right half-plane with all derivatives being zero, we obtain

sign(J) =

[Ep−Eq

].

As a consequence, each eigenvalue of A carries the same amount of informa-tion into sign(A), no matter of which modulus it is.

27

CHAPTER 4. EXTENDED KRYLOV SUBSPACES 28

That is a problem; even the smallest eigenvalues have a big impact on theresult of sign(A).

When using a standard Krylov subspace method (SKSM) to approximatethe propagation operator of the sign function, the representation of smalleigenvalues in the Krylov subspace tends to vanish. The approximationsonly vaguely represent the true result and a drastic increase of the dimensionof the Krylov subspace becomes inevitable.

One remedy has been proposed by Druskin and Knizhnerman [6]. Insteadof approximating a solution on the standard Krylov subspace spanned byb, Ab,A2b, . . ., they also added the vectors A−1b, A−2b, . . . to the basis.

Definition 4.1 (The Extended Krylov Subspace (EKS))Given a non-singular matrix A ∈ Cn×n, a vector b ∈ Cn and two integers`,m ∈ N, the subspace

EK`,m(A, b) := span〈A−`b, A−`+1b, . . . , A−1b, b, Ab, . . . , Am−1b〉

is called the (`,m)-th extended Krylov subspace spanned by A and b.

The compression of A onto this subspace contains good information aboutthe smallest eigenvalues of A because the vectors A−1b, A−2b, . . . are closelyrelated to the inverse iteration, a method which computes small eigenvaluesand the corresponding eigenvectors of A well. Solving a linear system in eachstep of the generation of a basis seems exaggerated. But Druskin and Knizh-nerman [6] were dealing with special matrices arising from a finite differenceapproximation of a 2-D elliptic operator. Here, an LU decomposition onlyhas computational costs of O(n

32 ) flops and is done only once. Furthermore,

each action of the inverse to a vector can be computed in O(n log n) flops,which is only slightly more than the O(n) flops for an action of such a matrixto a vector.

We need to introduce a new class of functions.

Definition 4.2 (Rational Functions)The set of rational functions of degree `,m over a field F is denoted as RF

`,m.

Further any element f of RF`,m can be represented in the form

f(x) =p(x)

q(x), where p ∈ PF

m, q ∈ PF` .

The function f is an element of SRF`,m, if f can be written as

f(z) = p(z) + q(z−1), for p ∈ PFm, q ∈ PF

` .


Note that SRF`,m ⊂ RF

`,m+`. Further, we restrict us to F = C.We intuitively assumed, that the subspace EK`,m(A, b) offers a better abil-

ity to approximate f(A)b. The following result confirms this.

Lemma 4.3Given a non-singular matrix A ∈ Cn×n, a vector b ∈ Cn and a rational

function f ∈ SRC`,m−1 of the form

m−1∑j=−`

αjzj,

the space EK`,m(A, b) contains the value f(A)b .

The proof follows directly from the definition of EK`,m(A, b). We have topoint out that the functions f above are special rational functions. For fbeing an arbitrary rational function the above does not hold.

We may also look at the extended Krylov subspaces in a different way.We have

EK`,m(A, b) = K`+m(A,A−`b),

so the results we derived for the standard Krylov subspaces and the incorpo-rated Arnoldi process can be applied to the matrix A and the vector A−`b.Further, the error representations and indicators hold for the accordinglymodified subspace K`+m(A,A−`b) and the approximations.

It is also possible to look at EK`,m(A, b) in the following way.

EK`,m(A, b) = EKm−1,`+1(A−1, b).

What is our Goal? We seek a procedure which generates an appropriatebasis for the extended Krylov subspace EK`,m(A, b). The basis should haveuseful properties like orthonormality. The compression of A should also beof a nice form, i.e., it should be structured in some way, so we can exploitits structure when computing with it. Here identities like the ones derivedin the Arnoldi process would be optimal.

We will now present an algorithm, which is a modified version of an algo-rithm originally published by Simoncini and Knizhnerman [15]. It generatesan orthonormal basis of the EKS EKm,m(A, b) stored in the matrix Vm,mand the compression Hm,m of a non-singular matrix A. The algorithm iskept very general; specifications concerning the computation of Hm,m are notgiven explicitly. The possibility of choosing ` 6= m is also not covered by the

http://www.youtube.com/watch?v=NDNZK5MiQ1A


algorithm but can be implemented in practice.

Algorithm 4.4: Extended Krylov Subspace Process for EKm,m(A, b)

Input : A ∈ Cn×n, b ∈ Cn, integer mOutput: Vm,m , Hm,m

v1 := b/‖b‖1

V0 := ∅2

V1 := gram-schmidt([v1, A−1v1])3

for j = 1, . . .,m do4

Vj,j := [Vj−1,j−1,Vj]5

Hj,j := V Hj,jAVj,j6

Vj+1 := [AVje1, A−1Vje2]7

Vj+1 = orthogonalise Vj+1 with respect to Vj,j8

Vj+1 = gram-schmidt(Vj+1)9

Similar to the Arnoldi process, the Algorithm already generates an orthonor-mal basis for EKm+1,m+1(A, b) and the first column of Vm,m is the vector b

‖b‖ .There are smarter methods for updating the matrix Hm,m in line 6 than usingmatrix matrix multiplications, see [25].

Of course, the subspaces EKm,m(A, b) are nested, i.e.,

EKm+1,m+1(A, b) ⊃ EKm,m(A, b).

When an error bound is given and an error indicator is available, one couldestimate the approximation after line 6 and check whether convergence oc-curred or not. In the latter case, one could raise the dimension of the ap-proximation space and use the previously computed subspaces. In analogyto (2.2) the following crucial identity holds for the matrices Vj+1,Vj,j, Hj,j,j = 1, . . . ,m

(4.2) AVj,j = Vj,jHj,j + Vj+1τj+1,j[e2j−1, e2j]T ,

where τj+1,j = VHj+1AVj.We now want to prove (4.2).Let us suppose we have an orthonormal basis for the EKS EK`+1,m+1(A, b).

The basis vectors form the columns of the matrix V`+1,m+1 and shall beordered in a nested way. For ` = m we want an order according to Algorithm4.4. For m > `, we want the first 2 min(`,m) columns to be ordered in theway according to Algorithm 4.4 and the remaining columns shall be orderedin a way according to their contribution to the vectors Ajb, Aj+1b, . . . , Amb.For m < ` we order the first 2 min(`,m) columns in the way according to


Algorithm 4.4 and the last columns in the way according to their contributionto the vectors A−jb, A−(j+1)b, . . . , A−(`+1)b.

In the case `+1 = m+1, the matrix Vj,j is the simple restriction of V`+1,m+1

to the first 2j columns. This correlates to the case treated in Algorithm 4.4.We now take a look at AVj,j. We have

AVj,j = Vj,jQj,j + AVj[e2j−1, e2j]T ,

where Qj,j is some 2j× 2j matrix and Vj is a n× 2 matrix , containing as itscolumns the basis vectors associated with Aj−1b and A−jb. We allocate allthe shares that the vectors AVj have with the vectors in Vj,j, that is Vj,jW ,where W is some 2j × 2 matrix, and add them to Qj,j. We get

(4.3) AVj,j = Vj,jTj,j + Vj+1 · c · [e2j−1, e2j]T ,

where Tj,j is of the same size as Q, T = Q+Vj,jW [e2j−1, e2j]T , Vj+1 as above

and c is some 2 × 2 matrix. The remaining shares of AVj with respect toVj+1 are represented in c. Multiplying the equation with V H

j,j from the leftyields

V Hj,jAVj,j = Tj,j,

since V Hj,jVj,j = Ej and V H

j,jVj+1 = 0. This proves Tj,j = Hj,j.Multiplying (4.3) with VHj+1 from the left yields

VHj+1AVj,j = c · [e2j−1, e2j]T ,

since VHj+1Vj,j = 0 and VHj+1Vj+1 = E2. Since AVj,je2j is contained in the EKSEKj,j(A, b), the second row of c is zero. We look more carefully at VHj+1AVj,j;

VHj+1AVj,j = VHj+1[AVj−1,j−1, AVj] = VHj+1[Vj,jK,AVj] = [0,VHj+1AVj],

where K is some 2j × 2(j − 1) matrix. This completes the proof to showc = VHj+1AVj = τj+1,j.

It is also possible to obtain a decomposition similar to (4.2), but with A−1

instead of A.

(4.4) A−1Vj,j = Vj,jH?j,j + Vj+1τ

?j+1,j[e2j−1, e2j]

T ,

where H?j,j = V H

j,jA−1Vj,j, τ

?j+1,j = VHj+1A

−1Vj. The proof is the same as theproof for A, except that we have to replace A by A−1 in the argumentation.A difference though exists; the matrices τ ?j+1,j now have their first row zeroand 2 non-zero entries in the second row.


In addition, the matrices Hj,j, H?j,j are in upper block Hessenberg form

with block size 2. We show this for Hj,j, the proof for H?j,j is the same. We

have

Hj,j = V Hj,jAVj,j = [Vj−1,j−1,Vj]HA[Vj−1,j−1,Vj]

= [Vj−1,j−1,Vj]H [AVj−1,j−1, AVj]

=

[V Hj−1,j−1

VHj

] [AVj−1,j−1 AVj

]

=

V Hj−1,j−1AVj−1,j−1 V H

j−1,j−1AVj

VHj AVj−1,j−1 VHj AVj

=

Hj−1,j−1 V Hj−1,j−1AVj

VHj AVj−1,j−1 VHj AVj

.We need to look at VHj AVj−1,j−1 more carefully. We have

VHj AVj−1,j−1 = VHj (Vj−1,j−1Hj−1,j−1 + Vjτj,j−1[e2j−3, e2j−2]T ).

The columns of Vj are orthonormal and orthogonal to the columns of Vj−1,j−1,so

VHj AVj−1,j−1 = VHj Vjτj,j−1[e2j−3, e2j−2]T = τj,j−1[e2j−3, e2j−2]T

has the desired form of a zero block except for its upper right 2 × 2 cornerwhich is τj,j−1.

The case m 6= ` is treated in the same way as above for the first min(`,m)steps. When min(`,m) is exceeded, the argumentation of the simple Arnoldidecomposition (2.2) can be applied to the Arnoldi process for A and b orA−1 and b, respectively. The matrices Tj,j then have 2 × 2 block upperHessenberg form first, followed by true (non-block) upper Hessenberg formwhen j exceeds min(`,m). The remainder term c is the τj+1,j from abovefirst, and when j exceeds min(`,m), it becomes the hm+1,m from the Arnoldidecomposition.

The results are summarized in the following theorem.

Theorem 4.5 (Extended Krylov Subspace Decomposition)Given a non-singular matrix A ∈ Cn×n and a vector b ∈ Cn, the matricesH`,m, V`,m as above fulfil

AVj,j = Vj,jHj,j + Vj+1τj+1,j[e2j−1, e2j]T ,


andAV`,m = V`,mH`,m + h`+m+1,`+mv`+m+1e

T`+m+1,

if ` 6= m.

We now want to try to derive similar results to Lemma 2.4 and Corollary2.5 for the matrices V`,m, H`,m.

Lemma 4.6 (Exactness of Rational Functions)Given a non-singular matrix A ∈ Cn×n, a vector b ∈ Cn, ‖b‖ = 1 and a

rational function f ∈ SRC`,m−1, f(z) =

∑m−1j=−` αjz

j for which f(A) is defined,the following property holds:

(4.5) f(A)b = V`,mf(H`,m)e1,

where V`,m, H`,m are the matrices gained in an extended Krylov subspaceprocess for EK`,m(A, b).

Proof. From Lemma 4.3 it is directly seen that f(A)b is contained inEK`,m(A, b). Still left to prove is that the matrices V`,m, H`,m fulfil the aboveequation.

We have H`,m = V H`,mAV`,m, so

f(H`,m) =m−1∑j=−`

αjHj`,m =

m−1∑j=−`

aj(VH`,mAV`,m)j.

We follow the proof of Lemma 2.4 and show that equation (4.5) holds formonomials of the form zj, −` ≤ j < m. For j = 0, the equality clearly holds.We want to treat the case j ≥ 0 first.Looking at the orthogonal projector π`,m = V`,mV

H`,m onto the EKS EK`,m(A, b),

we suppose the equality to hold for j < m − 1. With Aj+1b, Ajb belongingto EK`,m(A, b), we get

Aj+1b = π`,mAj+1b = π`,mAπ`,mA

jb.

Now,π`,mAπ`,m = V`,m V

H`,mAV`,m︸︷︷︸H`,m

V H`,m = V`,mH`,mV

H`,m,

and with Ajb = V`,mHj`,me1, it follows that

Aj+1b = V`,mH`,mVH`,mV`,mH

j`,me1 = V`,mH

j+1`,m e1.


Higher powers than m − 1 of A may not be treated in this way, since ingeneral they are not contained in EK`,m(A, b).

We now consider the case of negative j. For j = −1 we have to showA−1b = V`,mH

−1`,me1.

We know that A−1b is an element of EK`,m(A, b) and as such, it is invariantunder π`,m. With this it follows that A−1b = π`,mA

−1b. Multiplying theequation with A from the left yields

b = Aπ`,mA−1b.

Another multiplication with V H`,m from the left and extending π`,m to V`,mV

H`,m

givesV H`,mb︸︷︷︸e1

= V H`,mAV`,m︸︷︷︸H`,m

V H`,mA

−1b,

which, multiplied by H−1`,m and V`,m, finally gives

V`,mH−1`,me1 = A−1b.

Suppose we proved the lemma to be true for j < `. We have A−(j+1)b ∈EK`,m(A, b), and so A−(j+1)b = π`,mA

−(j+1)b. Multiplying both sides with Ayields

A−jb = Aπ`,mA−(j+1)b,

and using the induction hypothesis we obtain

V`,mH−j`,me1 = Aπ`,mA

−(j+1)b.

Extending π`,m and multiplying with V H`,m yields

H−`+1`,m e1 = V H

`,mAV`,m︸︷︷︸H`,m

V H`,mA

−(j+1)b,

so after the division with H`,m and multiplication with V`,m we proved

A−(j+1)b = V`,mH−(j+1)`,m e1.

Analogously to the case for positive powers of A, we can not guarantee highernegative powers of A times b to be contained in EK`,m(A, b). �

As a consequence of the results collected so far, when given a rational approx-imation f to a function g, where f is the sum of two polynomials in z and


z−1 respectively, f(A)b can be computed exactly in an extended Krylov sub-space of appropriate size with (block) Hessenberg matrices which are smalland easy to invert.

The question now arises whether we can also extend Corollary 2.5 to theextended Krylov subspace method.

We follow the proof of Corollary 2.5 and look at the symmetric EKSEKm,m(A, b).

Let Vm,m, Hm,m be the matrices obtained in the extended Krylov subspaceprocess for EKm,m(A, b) an let b have norm 1. We want to know how amonomial of degree m in A times b behaves with respect to the associatedextended Krylov subspace approximation. We have

Amb = AAm−1b

= AVm,mHm−1m,m e1

= (Vm,mHm,m + Vm+1τm+1,m[e2m−1, e2m]T )Hm−1m,m e1

= Vm,mHmm,me1 + Vm+1τm+1,m[e2m−1, e2m]THm−1

m,m e1.

The first term is nice. The second term seems a bit bulky but we can derivea similar formula as in the corresponding corollary for SKS. To that purpose,we treat the 2m×2m block Hessenberg matrix Hm,m with block size 2 like anm×m Hessenberg matrix of size m. And we know the lower left corner entryof the (m− 1)-th power of such a matrix from the proof of Corollary 2.5. Itis just the product of all subdiagonal entries. In our case, it is the productof the subdiagonal 2× 2 blocks, which were shown to be the matrices τj+1,j,j = 0, . . . ,m − 1. These matrices also had a special form; their second rowwas zero. Products of this type of matrices result in a matrix of the sametype. Consequently, we get that τm+1,m[e2m−1, e2m]THm−1

m,m e1 is the vector oflength 2 with last entry zero and the first entry is equal to the product ofthe upper left entries in the matrices τj+1,j, j = 0, . . . ,m, i.e.

[e2m−1, e2m]THm−1m,m e1 = (

m−1∏j=0

τj+1,j)e1,

so

Amb = Vm,mHmm,me1 + Vm+1(

m∏j=0

τj+1,j)e1.

Note that the product∏m

j=0 τj+1,j needs to be evaluated in the correct order,

so we have to recursively compute∏m

j=0 τj+1,j = τm+1,m

∏m−1j=0 τj+1,j .

For A−(m+1)b we make use of the alternative decomposition (4.4). Theresult of Lemma 4.6 also holds for the inverse decomposition and the above


train of thoughts for positive powers of A can be be transferred to positivepowers of A−1. We summarize this in the following corollary.

Corollary 4.7Given a non-singular matrix A ∈ Cn×n, a vector b ∈ Cn, ‖b‖ = 1 and a

rational function f ∈ SRCm+1,m for which f(A) is defined and let f have the

form f(z) =∑m

j=−(m+1) αjzj = q(z−1) + p(z), where p ∈ PC

m, q ∈ PCm+1. Let

Vm,m, Hm,m, Vm+1 be the matrices gained in an extended Krylov subspaceprocess for EKm,m(A, b). Then the following property holds:

(4.6) f(A)b = Vm,m

(p(Hm,m) + q(H?

m,m))e1

+ Vm+1

αm m∏j=0

τj+1,j + α−(m+1)

m∏j=0

τ ?j+1,j

e1.

Chapter 5

Application andImplementation

In this chapter we will present applications in which matrix functions arise.We will also present complete algorithms for the evaluation of f(A)b, in-cluding error control. Methods for computing f(A) explicitly will also bepresented. A procedure equivalent to the “Regensburg” approach for evalu-ating the sign function, which makes use of the extended Krylov subspaces,will complete this chapter.

5.1 How Functions of Matrices Arise

5.1.1 Differential Equations

Consider the initial value problem for a linear system of ODEs

dy

dt= Ay, y(0) = c, y ∈ Cn, A ∈ Cn×n.

Its solution is y(t) = exp(At)c. The function y(t) can directly be obtainedusing the matrix function exp which is defined for any matrix A ∈ Cn×n.Obviously, this explains why we are interested in the propagation operatorf(A)b and do not need to know f(A) explicitly in this case. So in princi-ple, the solution of the ODE is the exponential of a scaled matrix A timesan initial vector c. Practically, this matrix exponential method is not com-petitive. Other methods for solving systems of ODEs numerically providecheaper results and even error estimates are known explicitly.

Besides the disadvantages when solving a homogeneous system of ODEswith matrix functions, there exist applications where matrix functions arehighly efficient. Exponential integrators based on matrix functions for the

37

5.2. THE COMPLETE ALGORITHM FOR SKSM 38

solution of large stiff systems of differential equations are state of the art, see[14].

Justifications for further matrix functions arise from systems of differentialequations of second order. The System

d2y

d2t+ Ay = 0, y(0) = y0, y

′(0) = y′0 y0, y′0 ∈ Cn, A ∈ Cn×n,

has solutiony(t) = cos(

√At)y0 + (

√A)−1 sin(

√At)y′0.

5.1.2 Lattice Quantum Chromo Dynamic

Quantum chromodynamics (QCD) is an area in physics which deals withthe strong interactions between quarks. Lattice QCD uses four dimensionalspace-time lattices to formulate the theory. The numerical simulation onthese lattices are of our interest. Our main focus lies on the overlap-Diracoperator of Neuberger [18]. Here, we have to solve linear systems of the form

(G− sign(Q))x = b.

The matrix G is diagonal and has ±1 as entries, the matrix Q is sparse,complex and the size of the system is very large. This makes a direct com-putation of sign(Q) and therefore a direct solution of the system impossible.Krylov subspace methods are used to compute a solution x and therefore itis reasonable to use Krylov subspace approximations to sign(Q)v for a givenvector v in the iterative solution process.

5.2 The Complete Algorithm for

Standard Krylov Subspace Methods

The algorithm for standard Krylov subspace methods contains the Arnoldiprocess, the evaluation of the function f on the compression of the matrix A,i.e.,f(Hm), and checking whether convergence has occurred. If convergence isattained in the sense that an error indicator is small enough, the final solutionis computed. If not, the dimension of the Krylov subspace is augmented.

The algorithm thus reads as follows.

5.2. THE COMPLETE ALGORITHM FOR SKSM 39

Algorithm 5.1: Standard Krylov Subspace MethodInput : A ∈ Cn×n, b ∈ Cn, ‖b‖ = 1, proper function f , step size k,

error bound εOutput: x, an Approximation to f(A)b

m := 01

while εerrind > ε do2

m = m+ k3

[Vm, Hm] =arnoldi(A, b,m)4

Choose j auxiliary nodes and create matrix H5

Evaluate x := f(H)e16

Compute the error indicator εerrind following Theorem 3.57

x := Vm[x(1 : m)]8

In line 4, arnoldi(·, ·, ·) denotes the Arnoldi process. The previously com-puted Hessenberg matrices Hm−k and Arnoldi vectors Vm−k should be used,so in every loop the computational effort consists of extending the basis ma-trix Vm−k by k vectors, extending the Hessenberg matrix Hm−k by k rows andcolumns of length m, where the k rows mostly contain zeroes, and evaluatingf(H)e1.

Of course, it is also possible to use the two-sided Lanczos process insteadof the Arnoldi process, line 4 then reads bi-lanczos(A, b,m). Disregardingline 8, the two-sided Lanczos process also allows us to not explicitly store thewhole matrix Vm. We remember that the columns of Vm are created usinga 6 term recurrence. This means we only have to store the last 4 vectorsof the basis — the 2 last columns of Vm and Wm — plus the 2 current newbasis vectors. In every loop, only these are needed and used to computethe next columns of Vm+1, Wm+1 and the enlarged matrix Hm+1. Whenconvergence occurred, we could iteratively create the lift of the compressedsolution x to the whole space Cn in line 8. Note that this means we haveto run bi-lanczos again. Although this results in less storage and we savesome orthogonalization costs compared to the Arnoldi process, we need morecomputational time for the generation of the final approximation. All of thisseems like a very good idea, but the bi-Lanczos process is highly affected byrounding errors, and the theoretical properties like WH

m Vm = Em are far fromreality. This will be seen in numerical experiments.

The task of evaluating x is often done by explicitly computing f(H),

but it is also possible to recursively evaluate f(H)e1 using a Krylov subspacemethod. This approach is referred to as the nested Krylov subspace approach.For f = sign, and with a little manipulation of the matrix Hm, we call this

5.3. EXPLICITLY COMPUTING F (A)? 40

the Regensburg method. It will be discussed in a later section with specialemphasis on EKSM, see Section 5.4.1 .

5.3 Explicitly Computing f (A)?

We have come to a point where we have to evaluate f(Hm). In general, thematrices Hm are much smaller than the original matrix A. In most cases, mwill not exceed 20 percent of the original problem size n.

In the following, A and f denote some matrix and some function, notnecessarily connected to the previous context.

The first way to define a matrix function was the Jordan decompositionapproach. Whenever A is not diagonalisable, the computerised computationof Jordan decompositions is highly unstable, often the similarity transforma-tion matrices are very ill conditioned, leading to completely unusable results.This implies that even if we own a fast computer and a powerful mathemat-ical software, such as MATLAB, the Jordan approach is not maintainable.

Using the Hermite interpolating polynomial is also not recommendable.This approach assumes knowledge about the characteristic polynomial of agiven matrix, hence, we need to know every single eigenvalue and its multi-plicity. If we had a procedure that offered these insights at reasonable costand precision, we still would have to deal with the evaluation of a polynomialin the matrix A. Although there exist methods, like the Paterson-Stockmeyerapproach [20], which can compute polynomials in a matrix fast, the error inthe computed result can be large [13] and the costs are high, approximatelyO(n4) flops for f(A) if f is given as a Hermite interpolation polynomial.This is not acceptable, since most other methods, e.g., iterative methods,cost O(n3) flops.

Another possibility is to use the above mentioned iterative methods, butthis often requires knowledge about the spectrum of A or the norm ‖A‖.These iterative methods are specific to every function. Established methodsexist for the exponential, the sign, the inverse square root and others.

Another approach is not to deal with the function f itself, but with anapproximation to f . The approximation should fulfil several criteria, perhapsthe two most important are being easy to apply to a square matrix and havinggood approximation quality on the given matrix. We said polynomials in amatrix are a bad idea, but we have to state this more precisely. Polynomialsof degree ≈ n are a bad idea. Polynomials of degree � n are acceptable andwell suited for our goal.

5.3. EXPLICITLY COMPUTING F (A)? 41

5.3.1 Taylor Series

The simplest approximation polynomials we can think of are truncated Taylorseries, the Taylor polynomials. Well known, easy to obtain and even an errorbound is included. The problem though is that they only approximate fand its derivatives well in a possibly small neighbourhood of a single value.An arbitrary matrix has more than one eigenvalue, and so the quality ofthe approximation on the spectrum may become very bad. Nevertheless,we want to give an error bound for the approximation quality of truncatedTaylor series.

Theorem 5.2 (Error Bound for truncated Taylor Series)Suppose the function f has a Taylor series expansion of the form

f(z) =∞∑k=0

f (k)(α)

k!(z − α)k

and the eigenvalues of the matrix A ∈ Cn×n lie in the area of convergence ofthe Taylor series.

Then f(A) is defined, and the truncated series fulfils

‖f(A)−j−1∑k=0

f (k)(α)

k!(A−αEn)k‖ ≤ 1

j!max0≤t≤1

‖(A−αEn)jf (j)(αEn+t(A−αEn))‖

for any matrix norm and any j ≥ 1.

A proof can be found in [16].

5.3.2 Rational Functions

In Chapter 4, we treated a special class of rational functions, namely theclass SRC

`,m ⊂ RC`,m+`. The only poles functions of SRC

`,m possess, are locatedat zero. In many cases though, rational approximations to f are given inpartial fraction form and the approximations have different poles than zero.

The possibly most prominent rational approximants are the Pade approx-imations. These approximations fulfil the relation

f(z)− rm,n = O(zm+n+1) for z → 0,

where f is a given scalar function, rm,n = pm,n

qm,n∈ RC

m,n and qm,n(0) = 1. This

behaviour origins in the desire to find a rational function that interpolates fand its first m + n derivatives in zero. The rational function rm,n is called

5.4. THE COMPLETE ALGORITHM FOR EKSM 42

the [m,n] Pade approximant. If an [m,n] Pade approximant exists, then itis unique. From this it follows, that once such an approximant is found, itscoefficients are written down once and forever.

We will not immerse ourselves in the theory of Pade approximations andsimply use them where necessary. For a serious discussion, see [2].

5.4 The Complete Algorithm for

Extended Krylov Subspace Methods

We will now give an algorithm for extended Krylov subspace methods. It isclosely related to the algorithm for SKSM and was originally published bySimoncini and Knizhnerman [15]. One drawback though is the absence of atruly approved error indicator.

Algorithm 5.3: Extended Krylov Subspace MethodInput : A ∈ Cn×n, b ∈ Cn, proper function f , error bound εOutput: x, an Approximation to f(A)b

m := 01

while εerrind > ε do2

m = m+ 13

[Vm,m, Hm,m] =eks(A, b,m)4

Evaluate x := f(Hm,m)e15

Compute the error indicator εerrind6

x := Vm,mx7

The subroutine eks(A, b,m) is Algorithm 4.4 and assigns an orthonormalbasis of EKm,m(A, b) and the corresponding compression of A. Like in Algo-rithm 5.1, previously computed basis and compressions may be used in orderto minimize computational costs.

The error indicator is a crucial point, Knizhnerman and Simoncini proposeto use ‖τm+1,m[e2m−1, e2m]f(Hm,m)e1‖, see [15], which is closely related tothe error indicator we proposed in chapter 3.1 but has not been proved to beapplicable. Nevertheless, it can be computed at low costs, since all quantitiesare available.

5.4.1 The Regensburg Method

The Regensburg method was proposed by Heybrock and Bloch [3] from theUniversity of Regensburg and is mainly based on two ideas. One idea is


to amplify small eigenvalues so they have a bigger impact on the Krylovsubspace approximation. This is done by transferring sign(z) = sign(z+z−1)to matrix arguments. The approximation then reads

sign(A)b ≈ Vm sign(Hm)e1 = Vm sign(Hm +H−1m )e1.

The second idea is to approximate sign(Hm + H−1m )e1 by a standard Krylov

subspace method, so we get

Vm sign(Hm +H−1m )e1 ≈ Vm(Vm sign(V H

m (Hm +H−1m )Vm︸︷︷︸

Hm

)V Hm e1.

The sizes of the matrix Vm is m× m, the size of Hm is m× m and e1 is thefirst unit vector of Cm. The matrix Hm is the projection of Hm +H−1

m ontothe Krylov subspace spanned by the columns of Vm, or written in a matrixmanner Hm = V H

m (Hm + H−1m )Vm. Since V H

m e1 = e1, where the first e1 is oflength m and the latter has length m, we obtain

sign(A)b ≈ Vm sign(Hm)e1 = Vm sign(Hm +H−1m )e1 ≈ VmVm sign(Hm)e1.

Note that the approximation of sign(Hm+H−1m )e1 is necessary for the method

to be economic. A direct evaluation would afford an explicit inversion of thematrix Hm. The compression to Hm on the other hand only needs solvingm systems with a Hessenberg matrix. A QR-decomposition of an m × mHessenberg matrix can be obtained in O(m2) flops with Givens rotations andthe solution of such a system can be computed in O(m2) flops. A matrix-vector product also costs O(m2) flops, but when we go into details and dropthe O-notation, we see that solving such a system costs slightly more. Thecomputational cost can be further reduced when we use the two-sided Lanczosprocess instead of Arnoldi. The matrices Hm then only have tridiagonal formand the systems may be solved in O(5m) flops.

Moreover, evaluating sign(Hm) is cheaper since we are dealing with amatrix argument of size m× m.

So far, the method seems promising. But with extended Krylov subspaces,we are able to approximate the sign function even better. This can be seenin the following way.

Building the matrix Vm with the matrix (Hm +H−1m ) and equivalently the

Krylov subspace Km((Hm + H−1m , e1), we look at the elements contained is

this space in the wayv = p(Hm +H−1

m )e1,

where p is a polynomial of degree less than m, which was proposed in Lemma1.12. Looking at monomials in (H + H−1), we see that they have a special


form:

(Hm +H−1m )j =

j∑k=0

(j

k

)HkmH

−(j−k)m =

j∑k=0

(j

k

)H−j+2km .

Only every second power of Hm and H−1m is involved. If we extend the view to

polynomials, the elements in Km((Hm +H−1m ), e1) are still“inflexible” in the

sense that they are only linear combinations of polynomials in (Hm +H−1m ).

If we used an extended Krylov subspace to evaluate sign(Hm)e1 this draw-back is eliminated. We give an example.The Krylov subspace K2((Hm +H−1

m ), e1) is the span of the vectorse1, (Hm +H−1

m )e1 and its computation affords — assuming Hm +H−1m is not

known exlicitly — once the solution of a system with Hm, a matrix vectorproduct and a vector vector addition. For simplicity we do not consider theorthonormalization process. The extended Krylov subspace EK1,2(Hm, e1) isspanned by the vectors e1, Hme1, H

−1m e1 which affords once the solution of a

system with a Hessenberg matrix and one matrix vector product. In additionto the standard Krylov subspace, we need one more vector to be stored, butthis can be neglected.

Where is the difference? The computational effort is more or less thesame, but we get K2((Hm +H−1

m ), e1) ⊂ EK1,2(Hm, e1) and furthermore it isH−1m e1, Hme1 /∈ K2((Hm + H−1

m ), e1) so the EKS shows a better approxima-tion behaviour especially for polynomials. Since matrix functions are matrixdepending polynomials, we see that this is essential.

We want to summarize this in a lemma.

Lemma 5.4For a non-singular, matrix H with H ·H 6= En and an integer m > 1, thereholds

Km((H +H−1), e1) ⊆ EKm−1,m(H, e1).

Furthermore, the costs for the construction of a basis of Km((H +H−1), e1)and EKm−1,m(H, e1) are the same when no orthogonalization process is in-volved and (H +H−1) is not known explicitly.

Proof. The subspace inclusion is clear. For the computational cost wesee that an increase of the dimension of the subspace Kj((H + H−1), e1) byone affords the same amount of work as the increase of EKj−1,j(H, e1) toEKj,j+1(H, e1) �

Note that we did not consider the vector vector addition in the SKS and theincreased amount of storage needed for the EKS approach.


Remark 5.5When Km((H +H−1), e1) is no invariant subspace of (H +H−1), there holds

Km((H +H−1), e1) ( EKm−1,m(H, e1).

When an orthogonalization process, such as the Arnoldi process, is used forthe basis, the extra costs for the orthogonalization are doubled in the EKSapproach, making O((2m)m2) flops in contrast to O(mm2). But since thesubspaces’ dimensions are chosen small, this is negligible.

We now present the algorithms realizing the Regensburg method and theextended Krylov subspace equivalent.

Algorithm 5.6: Regensburg MethodInput : A ∈ Cn×n, b ∈ Cn, ‖b‖ = 1, integers m, mOutput: x, an Approximation to sign(A)b


[Vm, Hm] =arnoldi(Hm +H−1m , e1, m)2

x = VmVm sign(Hm)e13

Algorithm 5.7: Extended Krylov Subspace Equivalent to RegensburgMethod

Input : A ∈ Cn×n, b ∈ Cn, ‖b‖ = 1, integers m, mOutput: x, an Approximation to sign(A)b


[Vm,m, Hm,m] =eks(Hm, e1, m)2

x = VmVm,m sign(Hm,m)e13

However, the improved approximation abilities of extended Krylov sub-spaces also afford a larger matrix argument. The matrix Hm,m has size2m× 2m, so if given a O(n3)-flop method for computing the sign of an n×nmatrix, the evaluation of sign(Hm,m) affords 8 times as long as sign(Hm).

Chapter 6

Numerical Experiments

This chapter is devoted to experiments. What happens, when we apply thetheoretically derived error indicator or extended Krylov subspace methodsto rounding error afflicted computing machines?

Throughout this chapter, our main focus lies on the quality of the ap-proximations with respect to the size of the Krylov subspace. Therefore alot of space for optimization of the respective code is available. Since weimplement all the experiments in MATLAB, a more direct implementation ,e.g., in Fortran, also offers possibilities for tuning the algorithms.

6.1 The Orthogonalization Processes

In theory, the algorithms 2.1 and 2.7 compute a basis for Km(A, b) using theArnoldi or two-sided Lanczos process. The Arnoldi process computes an or-thonormal basis, the two-sided Lanczos process produces bases for Km(A, b)and Km(AH , b) , which are biorthogonal and normalised. Algorithm 4.4 pro-vides an orthonormal basis for EKm,m(A, b). Furthermore, any of the abovementioned basis generating processes computes compressions of the matrixA onto the corresponding Krylov subspaces.

We want to observe their behaviour when they are implemented in MAT-LAB and applied to a matrix A and a vector b. With this we want assurethat the matrices we produced using one of the algorithms are appropriatefor our needs and able to offer Krylov subspace approximations to f(A)b ina stable, well conditioned and suitable way.

We use a matrix A of size 500 × 500 whose eigenvalues are randomlydistributed in two circles with radius 10 and middle points 1, −1 in thecomplex plane, see figure (6.1). The vector b has ones as entries.

We build the Krylov subspace K450(A, b), which has 90% of the original

46

6.1. THE ORTHOGONALIZATION PROCESSES 47

−20 −15 −10 −5 0 5 10 15 20−10

−8

−6

−4

−2

0

2

4

6

8

10

Figure 6.1: Spectrum of A

spaces dimension, with the Arnoldi and two-sided Lanczos process. Buildingsubspaces spaces larger than 20% of the original space for approximatingf(A)b is uncommon in practice, but we want to show something about thebehaviour of the processes.

In the Arnoldi case, we think that a basis is “good” when it is orthonormal,i.e., V H

m Vm = Em, and the MATLAB computed 2-norm of Vm is close to one.In the two-sided Lanczos case, we also check the 2-norm of the generatedbases, but test what V H

mWm has as entries. Theoretically, the diagonal shouldhave ones as entries and every other entry should be close to εmachine.

The computations for the Arnoldi process generate a matrix Vm with‖Vm‖2 = 1.0000, which is the value we desired. For dimensions up to 300,the vectors generated within the Arnoldi process are “very orthogonal” andthe non-diagonal entries of V H

m Vm are close to εmachine. With increasing di-mensions, this property slowly fades. Still, the largest entries in the matrixV Hm Vm do not exceed 10−8. The results can be seen in figure 6.2.

The two-sided Lanczos process and the associated subspace approxima-tion was very nice in theory. It needs less storage and computational effort


concerning orthogonalization, since it is based on a 6 term recurrence andthe compression of the matrix A is only tridiagonal instead of upper Hessen-berg in the Arnoldi case. But figure 6.3 shows what we have to pay for thisadvantage. The first 25 × 25 block of V H

mWm shows the wanted behaviour,but increasing the dimension of the Krylov subspace worsens the result dras-tically. For a subspace of dimension 100, the entries in the matrix exceed1. A further increase of the dimension worsens the result and the qualityof the matrices Vm,Wm does not improve. This is natural and stems fromthe 6 term recurrence. The orthogonalization process is only carried outwith respect to the 2 last vectors and the share earlier basis vectors have arenot eliminated. In practice, the two-sided Lanczos process can be used andfurther improved with a reorthogonalization of the basis vectors. If the di-mension of the chosen Krylov subspace is small enough, one could even thinkabout using the two-sided Lanczos process without reorthogonalization.

50 100 150 200 250 300 350 400 450

50

100

150

200

250

300

350

400

450−18

−16

−14

−12

−10

−8

−6

−4

−2

0

Figure 6.2: V Hm Vm gained with Arnoldi process, ‖Vm‖2 = 1.0000

We now take a look at the basis gained in the extended Krylov subspaceprocess, namely algorithm 4.4. The Code was provided by Valeria Simonciniand I am happy to be allowed to use it. The Matrix Hm,m is generated

http://www.youtube.com/watch?v=4hdrcDDqRHk


50 100 150 200 250 300 350 400 450

50

100

150

200

250

300

350

400

450−16

−14

−12

−10

−8

−6

−4

−2

0

Figure 6.3: V HmWm gained with two-sided Lanczos process,‖Vm‖2 = 6.8468, ‖Wm‖2 = 1.0351 · 103

differently from the way proposed in algorithm 4.4. In every step, onlythe new entries are updated. Like in the Arnoldi case, the norm of thematrix Vm,m is computed to be 1.0000. The entries in V H

m,mVm,m now form ahomogeneous picture of ones on the diagonal and values very close to εmachineeverywhere else. This makes the basis very suitable for our demands. Theresults can be seen in figure 6.4.

In the following experiments, we restrict ourselves to the Arnoldi and EKSprocess.

6.2. THE ERROR INDICATOR 50

50 100 150 200 250 300 350 400 450

50

100

150

200

250

300

350

400

450−18

−16

−14

−12

−10

−8

−6

−4

−2

0

Figure 6.4: V Hm,mVm,m gained with EKS process, ‖Vm,m‖2 = 1.0000

6.2 The Error Indicator

We recall the error indicator affiliated in Chapter 3, Theorem 3.5.

f(A)b− fm = hm+1,m

m∑j=1

[eTmφj(Hm)e1]Θj−1(A)vm+1

+ βγ[∆χeHf ](A)Θm(A)vm+1,

where β = ‖b‖, γ = γm =∏m

j=1 hj+1,j and χ eH is the characteristic polynomial

of H.The computation of the error might seem a bit futile since it contains the

evaluation of some divided differences and still we need f(A). Computingthe true error in the above sense in practice is not feasible. But we mayjust compute the leading sum, take this as an error indicator and hope thatthe remainder term βγ[∆χeHf ](A)Θm(A)vm+1 does not have a big impact onthe error. Still we need to evaluate some divided differences, but Lemma3.4 directly offers a solution. Instead of computing f(Hm) we may compute


f(H), where the size of H is only slightly larger than the size of H, leadingto negligible additional computational expenses. The values [eTmφj(Hm)e1]

are the corresponding entries in the first column of f(H).One crucial point when using auxiliary nodes are the nodes themselves.

Unfortunately, there are no results which propose values that should be cho-sen to guarantee an optimal error indicator for arbitrary functions. Goluband Meurant [9] state that for some functions, like f = exp, the choice of thelargest and smallest eigenvalue results in an upper bound for the error, andwhen only choosing the largest eigenvalue, one obtains a lower bound for theerror.

The drawback of not having a justification for which auxiliary nodes tochoose, might be compensated with the following: we can say somethingabout the number of nodes. The error indicator is the sum of the divideddifferences φj multiplied with a polynomial Θj−1 in A times the vector vm+1.Choosing only one extra auxiliary node means no extra costs concerning ma-trix vector multiplications since Θ0 = 1. This means choosing one auxiliarynode comes for free. Furthermore, the vectors generated by Θj−1(A)vm+1

can be used for extending the Krylov subspace in a further Arnoldi processif convergence did not occur.

In our experiments we restrict ourselves to the use of 2 auxiliary nodes.This choice might not be optimal, since we do not make use of the spe-cific functions and MATLAB has to solve an eigenproblem. But the extracomputational costs of these two values are relatively small compared to thecomputation of f(Hm). On the one hand, like proposed in [1], we use theeigenvalues of largest and smallest modulus of Hm. On the other hand, wewill use the largest and second largest eigenvalue as auxiliary node. The firstcase will be called minmax, the latter case will be called maxmax.

Test-Vector, -Matrices, and -Functions The test vector b is alwayschosen to be the vector with ones as entries. As test-matrices we choose ma-trices resulting from a 9-point stencil finite element discretization of a LaplacePDE. These matrices are used as an argument for the functions f = exp andf =√

. Furthermore we use matrices resulting from lattice QCD and takethem as an argument for f = sign. An implementation of the exponentialis available in MATLAB as expm and makes use of a scaling and squaringmethod, see [17] for details. Note that the scaling depends on the givenmatrix and therefore expm can not be treated as “a” function. With this,the routine expm stands for a whole class of functions which approximate theexponential for a given matrix argument. An implementation for

√can

also be found in MATLAB as a routine called sqrtm. The implementation of


the sign function, signm, can be found in the misc section on the homepageof Nicholas Higham [12]. The routine signm is the straight forward imple-mentation of the Schur method for the sign function, see Algorithm 5.5 in[13].

The cases signm and sqrtm both afford a Schur decomposition of thematrix. This explains why we do not use sign(z) = z√

z2to evaluate the

sign function in a matrix. The extra costs for forming A2 and solving thesystem

√A2x = b exceed the costs for the recurrence in the Schur method

for computing sign(A) of O(23n3) flops.

The error indicator is chosen to be

‖m∑j=1

[eTmφj(Hm)e1]Θj−1(A)vm+1‖2.

When I first implemented the error indicator, I made the mistake of simplyforming √√√√ m∑

j=1

[eTmφj(Hm)e1]2

instead of ‖∑m

j=1[eTmφj(Hm)e1]Θj−1(A)vm+1‖2. The mistake is equivalent toforming the 2-norm of the corresponding last entries in the first column off(H). In practice this was discovered to be a cheaper way to obtain an errorindicator than forming matrix vector multiplications in addition, so we getanother error indicator called the cheap error indicator.

6.2.1 The Laplacians

We use four lattices of size 16× 16, 32× 32, 64× 64 and 128× 128 resultingin matrices of size 256, 1024, 4096 and 16384 respectively. The matrices aresparse with not more than 9 entries per row. For illustrative reasons, we firstcompute the “true ” solution f(A)b and then start an Arnoldi process upto a reasonable size depending on the matrix. After this we use the nested-ness of the basis generated by the Arnoldi process and successively computeVmf(H)e1 depending on a given stepsize and the corresponding eigenvalues.We point out that this is only done for pictures and insight. Computing thetrue solution alone makes any Krylov subspace method unnecessary.

We want to start with some pictures to illustrate the behaviour of thefunctions and error indicators. Let us consider figures 6.5 to 6.12.

We observe, that less than 10% of the dimension of the complete spacesuffice to approximate the true solution. With rising lattice size, we also

http://www.maths.manchester.ac.uk/~higham/mftoolbox/


0 5 10 15 20 25 30 35 40 4510

−40

10−30

10−20

10−10

100

1010

Dimension of Krylov subspace

Nor

m o

f err

or

||f(A)b − β V

m f(H

m) e

1||

2

True Error IndicatorCheap Error Indicator

minmax

0 5 10 15 20 25 30 35 40 4510

−40

10−30

10−20

10−10

100

1010


Nor

m o

f err

or

||f(A)b − β V

m f(H

m) e

1||

2


maxmax

Figure 6.5: 16× 16, expm, maxdim = 40, stepsize = 1

0 5 10 15 20 25 30 35 40 4510

−20

10−15

10−10

10−5

100

105


Nor

m o

f err

or

||f(A)b − β V

m f(H

m) e

1||

2


minmax

0 5 10 15 20 25 30 35 40 4510

−40

10−30

10−20

10−10

100

1010


Nor

m o

f err

or

Norm of f(A)b − β V

m f(H

m) e

1


maxmax

Figure 6.6: 16× 16, sqrtm, maxdim = 40, stepsize = 1

see that really small subspaces are able to approximate the true solution.Especially the function expm shows this behaviour. The 128 × 128 latticein figure 6.11 only needs a Krylov subspace dimension of 50 to approximatethe true solution. Compared to the complete space of size C16384, this is lessthan 0.5%. The approximations for sqrtm do not converge as fast as theapproximations for expm, but we can also observe that with a growing latticesize, a smaller sufficient subspace dimension comes in hand. For the 64× 64lattice, a subspace dimension of 90 is sufficient whereas the 128× 128 latticeneeds a dimension of 200 to produce an exact approximation. Indeed, 200is a larger number than 90, but seen in relation, 1.22% is about the half of2.19%.


0 10 20 30 40 50 60 70 80 9010

−100

10−80

10−60

10−40

10−20

100

1020


Nor

m o

f err

or

||f(A)b − β V

m f(H

m) e

1||

2


minmax

0 10 20 30 40 50 60 70 80 9010

−100

10−80

10−60

10−40

10−20

100

1020


Nor

m o

f err

or

||f(A)b − β V

m f(H

m) e

1||

2


maxmax


0 10 20 30 40 50 60 70 80 9010

−20

10−15

10−10

10−5

100


Nor

m o

f err

or

||f(A)b − β V

m f(H

m) e

1||

2


minmax

0 10 20 30 40 50 60 70 80 9010

−20

10−15

10−10

10−5

100


Nor

m o

f err

or

||f(A)b − β V

m f(H

m) e

1||

2


maxmax


We take a look at the error indicators. The true error indicators showthe wanted behaviour of forming and upper and lower bound for the error.Especially the upper bound is important. When given an error bound andthe minmax indicator undercuts the error bound, we can be sure that thecomputed approximation is sufficiently close to the true solution. In theexponential case, some of the pictures show no difference between the cheapan true error indicator, especially when the lattice sizes, and therefore thematrix sizes, are large. This can make a further evaluation of a polynomialin a large matrix times a long vector unnecessary.


0 10 20 30 40 50 6010

−50

10−40

10−30

10−20

10−10

100

1010


Nor

m o

f err

or

||f(A)b − β V

m f(H

m) e

1||

2


minmax

0 10 20 30 40 50 6010

−50

10−40

10−30

10−20

10−10

100

1010


Nor

m o

f err

or

||f(A)b − β V

m f(H

m) e

1||

2


maxmax


0 50 100 150 200 25010

−20

10−15

10−10

10−5

100


Nor

m o

f err

or

||f(A)b − β V

m f(H

m) e

1||

2


minmax

0 50 100 150 200 25010

−20

10−15

10−10

10−5

100


Nor

m o

f err

or

||f(A)b − β V

m f(H

m) e

1||

2


maxmax


6.2.2 The Lattice QCD Matrices

We want to test the error indicator on the sign function and matrices whicharise from lattice QCD. We use the matrices Q arising from the packagesCONF5.0-00L4X4-2600 and CONF6.0-00L4X4-3000 which are available atMatrix Market1. The matrices Q are of the form Q = γ5(En − 0.95κcD)where κc = 0.21070 in the case CONF5.0-00L4X4-2600 and κc = 0.16453in the case CONF6.0-00L4X4-3000. Further, the matrices are sparse, com-plex, hermitian and represent a lattice of 44 points, so the dimensions of thecomplex valued matrices are 12 · 44 = 3072.

1http://math.nist.gov/MatrixMarket/

http://math.nist.gov/MatrixMarket/


0 10 20 30 40 50 60 70 80 9010

−100

10−80

10−60

10−40

10−20

100

1020


Nor

m o

f err

or

||f(A)b − β V

m f(H

m) e

1||

2


minmax

0 10 20 30 40 50 60 70 80 9010

−80

10−60

10−40

10−20

100

1020


Nor

m o

f err

or

||f(A)b − β V

m f(H

m) e

1||

2


maxmax


0 50 100 150 200 250 300 350 400 45010

−20

10−15

10−10

10−5

100


Nor

m o

f err

or

||f(A)b − β V

m f(H

m) e

1||

2


minmax

0 50 100 150 200 250 300 350 400 45010

−16

10−14

10−12

10−10

10−8

10−6

10−4

10−2

100


Nor

m o

f err

or

||f(A)b − β V

m f(H

m) e

1||

2


maxmax


When we look at the figures 6.13 and 6.14 , we see that the cheap errorindicator is closer to the error than the true error indicator. Furthermore,the minmax indicator does not show the wanted behaviour of forming anupper bound.

6.3. EXTENDED KRYLOV SUBSPACE METHODS 57

0 100 200 300 400 500 600 70010

−20

10−15

10−10

10−5

100


Nor

m o

f err

or

||f(A)b − β V

m f(H

m) e

1||

2


minmax

0 100 200 300 400 500 600 70010

−20

10−15

10−10

10−5

100


Nor

m o

f err

or

||f(A)b − β V

m f(H

m) e

1||

2


maxmax

Figure 6.13: 44 lattice, conf = 5.0, signm, maxdim = 610, stepsize = 10

0 50 100 150 200 250 30010

−20

10−15

10−10

10−5

100


Nor

m o

f err

or

||f(A)b − β V

m f(H

m) e

1||

2


minmax

0 50 100 150 200 250 30010

−16

10−14

10−12

10−10

10−8

10−6

10−4

10−2

100


Nor

m o

f err

or

||f(A)b − β V

m f(H

m) e

1||

2


maxmax

Figure 6.14: 44 lattice, conf = 6.0, signm, maxdim = 300, stepsize = 10

6.3 Extended Krylov Subspace Methods

This section deals with the nested extended Krylov subspace method andits comparison to standard Krylov subspace methods and the Regensburgmethod.

We use matrices arising from lattice QCD to observe the difference betweenthe Regensburg method and its EKS equivalent. Further, we use the Laplacematrices to observe how a nested Krylov subspace method and the EKSequivalent behave when we are interested in exp(−A)b.


6.3.1 The Lattice QCD matrices

In this subsection, we try to compare the Regensburg method proposed byHeybrock and Bloch with a nested extended Krylov subspace method forcomputing the sign function of matrices arising from lattice QCD. In Chap-ter 5, we stated that the extended Krylov subspace shows a better approx-imation ability than a standard Krylov subspace generated with H + H−1

if we measure “better” in terms of the number of solved linear systems, seeLemma 5.4.

We implemented a routine that realizes Algorithm 5.6 and Algorithm5.7 and ran it with the matrices Q from above. Furthermore, we deflatedthe 30 smallest eigenvalues, so the test-vector was the original test-vectorb without the share of the eigenvectors belonging to the 30 smallest eigen-values. For Q arising from CONF5.0-00L4X4-2600, the moduli of the 30smallest eigenvalues lie in the interval [0.0271, 0.1219] and for Q associatedwith CONF6.0-00L4X4-3000, the moduli of the smallest eigenvalues lie in[0.1369, 0.2686]. The dimension of the standard Krylov subspace on level 1was set to m = 1500 and the dimension of the inner Krylov subspaces onlevel 2 were also chosen to have this dimension, so m = 1500.

We plot the errors of the approximations on level 2 with respect to thetrue solutions on level 0 and level 1 for the first 750 steps and obtain thefigures 6.15 and 6.16.

0 100 200 300 400 500 600 700 80010

−3

10−2

10−1

100


Nor

m o

f err

or

Norm of KSM Error Level 0Norm of EKSM Error Level 0

Level 0

0 100 200 300 400 500 600 700 80010

−3

10−2

10−1

100


Nor

m o

f err

or


Level 1

Figure 6.15: 44 lattice, conf = 5.0, dim level 1 = 1500

The Convergence does occur in the EKSM case also as fast as we expectit to, but the error concerning level 1 is to large. At least the exact solutionof sign(Hm)e1 on level 1 should be approximated up to an error smaller than10−3.


0 100 200 300 400 500 600 700 80010

−3

10−2

10−1

100


Nor

m o

f err

or


Level 0

0 100 200 300 400 500 600 700 80010

−3

10−2

10−1

100


Nor

m o

f err

or


Level 1


Using smaller dimensions on level 1 shows a different behaviour. Afterdetermining which dimension of level 1 generates an error smaller than 10−13

with respect to level 1, and adjusting the dimensions of level 2 accordantly,we obtain the figures 6.17 and 6.18.

0 50 100 150 200 25010

−14

10−12

10−10

10−8

10−6

10−4

10−2

100


Nor

m o

f err

or


Level 0

0 50 100 150 200 25010

−14

10−12

10−10

10−8

10−6

10−4

10−2

100


Nor

m o

f err

or


Level 1


The bad approximation quality in the first plots can be explained with thesubspace methods on level 2. The rounding errors occurring when solvingthe linear systems involving the matrix Hm of size 1500 × 1500 worsen theresults drastically.

The similar shape of the graphs is explained by the bases generated withthe Arnoldi and EKS process — they are highly orthonormal.


0 10 20 30 40 50 60 70 80 90 10010

−14

10−12

10−10

10−8

10−6

10−4

10−2

100


Nor

m o

f err

or


Level 0

0 10 20 30 40 50 60 70 80 90 10010

−15

10−10

10−5

100


Nor

m o

f err

or


Level 1


When we count the number of linear systems that have to be solved untilconvergence occurs, we see that the EKSM approach needs less than theRegensburg method. For the matrix arising from CONF5.0-00L4X4-2600,EKSM affords 55 systems whereas the Regensburg method needs 70 systemsto be solved. For the matrix arising from CONF6.0-00L4X4-3000, EKSMsolves 35 systems and the Regensburg approach solves 45 systems.

We also tested the Regensburg method and its extended Krylov subspaceequivalent with a matrix arising from a 64 lattice. The matrix was given indiagonal form, that means we knew all of its eigenvalues and eigenvectors.The test-vector was a randomly generated complex, normalized vector whoseentries lie in the unit circle. We ran three tests. The first with deflation sothe test-vector did not contain shares of eigenvectors to eigenvalues withmodulus smaller than 0.5 and the dimension of level 1 was set to 1000. Inthe second test, the size of level 1 was set to 500 and we used the samedeflation constraints. In the third test, we did not deflate the test-vector andthe size of level 1 and level 2 was set to 1000. The results can be seen infigures 6.19 to 6.21.

First of all, we notice that the figures show now observable differencebetween the error of the Regensburg method and the EKS equivalent onlevel 0. Although the error is — compared to the previous experiments —relatively large, we can see that already small dimensions on level 2 sufficeto approximate the solution as close as the space of level 1 permits.

Further, we see that when deflation is used, the subspaces on level 2 ofthe same dimension generate solutions which are nearly the same on level 1and 0. This means that the ESKM should be preferred over the Regensburg


0 50 100 150 200 250 300 350 400 450 50010

−3

10−2

10−1

100

101


Nor

m o

f err

or


Level 0

0 50 100 150 200 250 300 350 400 450 50010

−14

10−12

10−10

10−8

10−6

10−4

10−2

100

102


Nor

m o

f err

or


Level 1

Figure 6.19: 64 lattice, dim level 1 = 1000, deflation

0 50 100 150 200 250 300 350 400 450 50010

−3

10−2

10−1

100

101


Nor

m o

f err

or


Level 0

0 50 100 150 200 250 300 350 400 450 50010

−14

10−12

10−10

10−8

10−6

10−4

10−2

100

102


Nor

m o

f err

or


Level 1


method because less systems need to be solved in this approach and thecorresponding compressions of Q have the same size.

However, when we do not deflate the test-vector, the solution is not ac-ceptable and does not converge to a acceptable value.

6.3.2 The Laplacians

Similar to the approach above, we test the EKS method in a nested way anda nested SKS method to compute exp(−A)b.

We take the same lattices as above and compute Krylov subspaces on twolevels. The first one is computed with the Arnoldi process and has dimension


0 100 200 300 400 500 600 700 800 900 100010

0

101

102

103

104


Nor

m o

f err

or


Level 0

0 100 200 300 400 500 600 700 800 900 100010

−6

10−4

10−2

100

102

104


Nor

m o

f err

or


Level 1


200 in all cases. As we have seen in the section dealing with error indicators,these small dimension suffice to approximate the true solution up to εmachine.The two different subspaces on level 2 are computed using the Arnoldi andEKS process and have dimension 120. The results can be seen in the figures6.22 - 6.25.

0 5 10 15 20 25 3010

−15

10−10

10−5

100


Nor

m o

f err

or


Level 0

0 5 10 15 20 25 3010

−25

10−20

10−15

10−10

10−5

100


Nor

m o

f err

or


Level 1

Figure 6.22: 16× 16 lattice

We observe that the extended Krylov subspace method converges slowerthat the standard Krylov method. The approximations for a fixed dimensionsize obtained by the SKSM are closer to the true solution than the approxi-mations by the EKSM. Furthermore, the solutions generated by the EKSMare more expensive since we have to solve a linear system in every second


0 5 10 15 20 25 30 35 40 45 5010

−15

10−10

10−5

100


Nor

m o

f err

or


Level 0

0 5 10 15 20 25 30 35 40 45 5010

−16

10−14

10−12

10−10

10−8

10−6

10−4

10−2

100


Nor

m o

f err

or


Level 1


0 10 20 30 40 50 6010

−15

10−10

10−5

100


Nor

m o

f err

or


Level 0

0 10 20 30 40 50 6010

−40

10−35

10−30

10−25

10−20

10−15

10−10

10−5

100


Nor

m o

f err

or


Level 1

Figure 6.24: 64× 64 lattice,

step. We have to say that for this problem, the EKSM has no advantageover the SKSM.


0 10 20 30 40 50 60 70 80 9010

−15

10−10

10−5

100


Nor

m o

f err

or


Level 0

0 10 20 30 40 50 60 70 80 9010

−16

10−14

10−12

10−10

10−8

10−6

10−4

10−2

100


Nor

m o

f err

or


Level 1


Chapter 7

Conclusion

The subject of computing f(A)b is a challenging task in numerical linearalgebra. The variety of functions and matrices and their special properties,which we exploited only for the sign function and Hessenberg matrices, leavespace for a more precise analysis.

The error indicators were shown to be well suited for the estimation ofthe true error and can be easily obtained. A rigorous analysis for arbitraryfunctions and the choice of the number and values of the auxiliary nodesremains and is desirable. Further, an error representation for the extendedKrylov subspace methods remains to be derived.

In our experiments we saw that the advantage of extended Krylov sub-spaces to approximate sign(A)b better than the Regensburg method for agiven number of inversions was excelled by the size of the corresponding ma-trix for which the sign function had to be evaluated. A not nested approachwhere the information of the matrix A is directly contained in the EKS andnot filtered by a first Krylov subspace, is not possible for lattice QCD matri-ces. Therefore we must say that the Regensburg approach is more promisingthan its EKS equivalent. However, when a different kind of matrices is in-volved, and they are easily inverted or a linear system can be solved with alittle more computational effort than building a matrix vector product, thismethod is a serious alternative to standard Krylov subspace methods.

In a talk held at the University of Wuppertal, Valeria Simoncini proposedthe possibility to compute an extended-Krylov-like-subspace, where the ele-ments corresponding to A−kb were successively approximated by 5 steps of aGMRES algorithm. This could clear the way for a more general applicationof EKSM.

65

Appendix

Notation

We use capital letters for matrices, lower case letters for vectors and greek let-ters for scalars. A matrix A ∈ Cn×m consists of entries aij where i = 1, . . . , n,j = 1, . . . ,m, with En we denote the n× n identity matrix, sometimes drop-ping the index n, 0n is the n × n zero matrix and ej is the j-th canonicalunit vector. Here the length of the vector should be clear from the context.We will sometimes use a MATLAB-style notation [27], especially for partsof vectors or matrices and in algorithms. We then have aij = A(i, j) orej = E(:, j). For complex vectors and matrices bH and AH denotes the com-plex conjugated transpose of the corresponding vector or matrix. For realarguments we use bT and AT , which shall sometimes point the element outto be real. With ‖b‖ for a vector b ∈ Cn, we denote the standard euclidean2-norm of a vector, i.e. ‖b‖ =

√bH · b.

Special matrices

A diagonal matrix is a square matrix whose only possible nonzero entries lieon the diagonal, we write a diagonal n×n matrix A as A = diag(λ1, . . . , λn).An upper triangular matrix A is a square matrix such that

i > j ⇒ A(i, j) = 0.

A matrix A is upper Hessenberg when

i+ 1 > j ⇒ A(i, j) = 0

and is called proper or unreduced when every subdiagonal entry is nonzero.

66

Bibliography

[1] M. Afanasjew, M. Eiermann, O. Ernst, and S. Guettel, Im-plementation of a restarted Krylov subspace method for the evaluationof matrix functions, Linear Algebra Appl., 429 (2008), pp. 2293–2314.

[2] G. A. Baker and P. R. Graves-Morris, Pade approximants,Addison-Wesley, Advanced Book Program, Reading, Mass. :, 1981.

[3] J. C. R. Bloch and S. Heybrock, A nested Krylov subspace methodfor the overlap operator, The XXVII International Symposium on Lat-tice Field Theory, (2009). http://arxiv.org/abs/0910.2918v1.

[4] C. de Boor, Divided Differences, Surveys in Approximation Theory, 1(2005), pp. 46–69.

[5] J. W. Demmel, Applied Numerical Linear Algebra, Society for Indus-trial and Applied Mathematics, Philadelphia, PA, 1997.

[6] V. Druskin and L. Knizhnerman., Extended Krylov Subspaces: Ap-proximation of the Matrix Square Root and Related Functions, SIAM J.Matrix Anal. Appl., 19 (1998), pp. 755 – 771.

[7] M. Eiermann and O. Ernst, A restarted Krylov subspace method forthe evaluation of matrix functions, SIAM J. Numer. Anal., 44 (2006),pp. 2481 – 2504.

[8] R. Freund and M. Hochbruck, Gauss quadratures associated withthe Arnoldi process and the Lanczos algorithm, in Linear Algebra forLarge Scale and Real-Time Applications, Kluwer Academic Publishers,1993.

[9] G. H. Golub and G. Meurant, Matrices, moments and quadrature,Pitman Research Notes in Mathematics, 303 (1994), pp. 105 – 156.

[10] G. H. Golub and C. F. Van Loan, Matrix Computations, JohnsHopkins University Press, Baltimore, MD, third ed., 1996.

67

http://arxiv.org/abs/0910.2918v1

BIBLIOGRAPHY 68

[11] A. Greenbaum, Iterative Methods for Solving Linear Systems, Societyfor Industrial and Applied Mathematics, 1997.

[12] N. J. Higham, The Matrix Function Toolbox. http://www.ma.man.

ac.uk/~higham/mftoolbox.

[13] , Functions of Matrices: Theory and Computation, Society for In-dustrial and Applied Mathematics, 2008.

[14] M. Hochbruck, C. Lubich, and H. Selhofer, Exponential inte-grators for large systems of differential equations, SIAM J. Sc. Comp.,19 (1998), pp. 1552–1574.

[15] L. Knizhnerman and V. Simoncini, A new investigation of the ex-tended Krylov subspace method for matrix functions, Numer. Linear Al-gebra Appl., (To appear), pp. 1 – 17.

[16] R. Mathias, Approximation of matrix-valued functions, SIAM J. Ma-trix Anal. Appl., 14(4) (1993), pp. 1061 – 1063.

[17] C. Moler and C. Van Loan, Nineteen dubious ways to compute theexponential of a matrix, SIAM Rev., 20(4) (1978), pp. 801 – 836.

[18] H. Neuberger, Exactly massless quarks on the lattice, Phys. Lett. B,417(1-2) (1998), pp. 141 – 144.

[19] G. Opitz, Steigungsmatrizen, ZAMM, 44 (1964), pp. T52–T54.

[20] M. Paterson and L. J. Stockmeyer, On the number of nonscalarmultiplications necessary to evaluate polynomials, SIAM J. Comput., 2(1973), pp. 60–66.

[21] B. Philippe and R. B. Sidje, Transient Solutions of Markov Pro-cesses by Krylov Subspaces, Research Report INRIA, 1989 (1995),pp. 95–119.

[22] Y. Saad, Analysis of some Krylov subspace approximations to the ma-trix exponential operator, SIAM J. Numer. Anal., 29 (1992), pp. 209–228.

[23] , Iterative Methods for Sparse Linear Systems, PWS PublishingCompany, Boston, MA, 1996.

[24] V. Simoncini, Extended Krylov Subspace for Parameter Dependent Sys-tems, to appear.

http://www.ma.man.ac.uk/~higham/mftoolbox

http://www.ma.man.ac.uk/~higham/mftoolbox

BIBLIOGRAPHY 69

[25] V. Simoncini, A new iterative method for solving large-scale Lyapunovmatrix equations, SIAM J. Sc. Comp., 29 (2007), pp. 1268 – 1288.

[26] J. Stoer and R. Burlisch, Introduction to Numerical Analysis,Springer-Verlag, 2002.

[27] The MathWorks, Inc., MATLAB, Version 7, 2004.

[28] L. N. Trefethen and D. Bau, III, Numerical Linear Algebra, Soci-ety for Industrial and Applied Mathematics, Philadelphia, PA, 1997.

[29] J. Wilkinson, The Algebraic Eigenvalue Problem, Clarendon Press,Oxford, 1965.

URLs checked: December 1, 2009

Erklarung

Hiermit versichere ich, die vorliegende Arbeit selbststandig und unter aus-schließlicher Verwendung der angegebenen Literatur und Hilfsmittel erstelltzu haben.

Die Arbeit wurde bisher in gleicher oder ahnlicher Form keiner anderenPrufungsbehorde vorgelegt und auch nicht veroffentlicht.

Wuppertal, . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .(Paul Zingler)

70

Krylov subspace methods for functions of matrices - Bergische

Documents

Transcript of Krylov subspace methods for functions of matrices - Bergische