Index

A-optimality, 439

absolute error, 484, 494, 526ACM Transactions on Mathematical

Software, 552, 617

adj(·), 69

adjacency matrix, 331–334, 390Exercise 8.20:, 396

augmented, 390

adjoint (see also conjugate transpose),59

adjoint, classical (see also adjugate), 69

adjugate, 69, 598

adjugate and inverse, 118

affine group, 115, 179affine space, 43

affine transformation, 228

Aitken’s integral, 217

AL(·) (affine group), 115algebraic multiplicity, 144

algorithm, 264, 499–513

batch, 512

definition, 509

direct, 264divide and conquer, 506

greedy, 506

iterative, 264, 508–511, 564

reverse communication, 564online, 512

out-of-core, 512

real-time, 512

algorithmically singular, 121Anaconda, 553, 556, 569

angle(·, ·), 37

angle between matrices, 168angle between vectors, 37, 229, 357ANSI (standards), 475, 562, 563Applied Statistics algorithms, 617approximation and estimation, 431approximation of a matrix, 174, 257,

339, 437, 533approximation of a vector, 41–43arithmetic mean, 35, 37Arnoldi method, 318artificial ill-conditioning, 269ASCII code, 460association matrix, 328–338, 357, 365,

366, 369–371adjacency matrix, 331–332, 390connectivity matrix, 331–332, 390dissimilarity matrix, 369distance matrix, 369incidence matrix, 331–332, 390similarity matrix, 369

ATLAS (Automatically Tuned LinearAlgebra Software), 555

augmented adjacency matrix, 390augmented connectivity matrix, 390Automatically Tuned Linear Algebra

Software (ATLAS), 555autoregressive process, 447–450axpy, 12, 50, 83, 554, 555axpy elementary operator matrix, 83

back substitution, 274, 406backward error analysis, 495, 500Banach space, 33Banachiewicz factorization, 255

632 Index

banded matrix, 58inverse, 121

Bartlett decomposition, 346base, 467base point, 466basis, 21–23

Exercise 2.6:, 52orthonormal, 40–41

batch algorithm, 512Bauer-Fike theorem, 307Beowulf (cluster computing), 559bias, in exponent of floating-point

number, 468big endian, 480big integer, 466, 492big O (order), 497, 503, 591big omega (order), 497bilinear form, 91, 134bit, 460bitmap, 461BLACS (software), 557, 558BLAS (software), 553–556

CUDA, 560PBLAS, 558PLASMA, 559PSBLAS, 558

block diagonal matrix, 62determinant of, 70inverse of, 121multiplication, 78

BMvN distribution, 219, 548Bolzano-Weierstrass theorem for

orthogonal matrices, 133Boolean matrix, 390Box M statistic, 368bra·ket notation, 24byte, 460

C (programming language), 474, 489,568–569

C++ (programming language), 475,568–569

CALGO (Collected Algorithms of the

ACM), 617cancellation error, 487, 500canonical form, equivalent, 110canonical form, similar, 149canonical singular value factorization,

162

Cartesian geometry, 35, 73catastrophic cancellation, 486Cauchy matrix, 389Cauchy-Schwarz inequality, 24, 98Cauchy-Schwarz inequality for matrices,

98, 178Cayley multiplication, 75, 94Cayley-Hamilton theorem, 138CDF (Common Data Format), 463centered matrix, 288, 363centered vector, 49chaining of operations, 485character data, 461character string, 461characteristic equation, 138characteristic polynomial, 138characteristic value (see also eigenvalue),

135characteristic vector (see also eigenvec-

tor), 135chasing, 317Chebyshev norm, 28chi-squared distribution, 400

noncentral, 400PDF, equation (9.3), 400

Cholesky decomposition, 253–256, 345,436

computing, 558, 574root-free, 255

circulant matrix, 383classification, 389cluster analysis, 389cluster computing, 559coarray (Fortran construct), 563Cochran’s theorem, 353–356, 401cofactor, 68, 598Collected Algorithms of the ACM

(CALGO), 617collinearity, 265, 406, 430column rank, 99column space, 55, 89, 104, 105column-major, 522, 539, 545column-sum norm, 165Common Data Format (CDF), 463companion matrix, 139, 305compatible linear systems, 106compensated summation, 486complementary projection matrix, 356complementary vector spaces, 19

Index 633

complete graph, 329

complete pivoting, 275

complete space, 33

completing the Gramian, 177

complex data type, 491

complex vectors/matrices, 33, 132, 387

Conda, 553

condition (problem or data), 499

condition number, 265, 270, 290, 426,427, 499, 502, 523, 533

computing the number, 533, 574

inverse of matrix, 266, 270

nonfull rank matrices, 290

nonsquare matrices, 290

sample standard deviation, 502

conditional inverse, 128

cone, 43–46, 327

Exercise 2.18:, 53

convex cone, 44, 346

of nonnegative definite matrices, 346

of nonnegative matrices, 371

of positive definite matrices, 349

of positive matrices, 371

conference matrix, 381

configuration matrix, 369

conjugate gradient method, 279–283preconditioning, 282

conjugate norm, 93

conjugate transpose, 59, 132

conjugate vectors, 93, 134

connected vertices, 329, 334

connectivity matrix, 331–334, 390

Exercise 8.20:, 396

augmented, 390

consistency property of matrix norms,164

consistency test, 527, 551

consistent system of equations, 105,272, 277

constrained least squares, equalityconstraints, 413

Exercise 9.4d:, 451

continuous function, 186

contrast, 410

convergence criterion, 508

convergence of a sequence of matrices,134, 152, 171

convergence of a sequence of vectors, 32

convergence of powers of a matrix, 171,375

convergence rate, 509convex combination, 12convex cone, 44–46, 346, 349, 371convex function, 26, 197convex optimization, 346convex set, 12convexity, 26coordinate, 4Cor(·, ·), 51correlation, 51, 90correlation matrix, 365, 422

Exercise 8.8:, 394positive definite approximation, 436pseudo-correlation matrix, 437sample, 422

cost matrix, 369Cov(·, ·), 50covariance, 50covariance matrix, see variance-

covariance matrixCRAN (Comprehensive R Archive

Network), 552, 576cross product of vectors, 47

Exercise 2.19:, 54cross products matrix, 256, 358cross products, computing sum of

Exercise 10.18c:, 519Crout method, 245cuBLAS, 560CUDA, 559curl, 192curse of dimensionality, 511cuSPARSE, 560

D-optimality, 439–440, 533daxpy, 12decomposable matrix, 373decomposition, see also factorization of

a matrixadditive, 353, 354Bartlett decomposition, 346multiplicative, 108nonnegative matrix factorization,

257, 337singular value decomposition,

161–164, 320, 336, 425, 532spectral decomposition, 155

634 Index

defective (deficient) matrix, 149, 150deficient (defective) matrix, 149, 150deflation, 308–310degrees of freedom, 360, 362, 407, 430del, 192derivative with respect to a matrix, 195derivative with respect to a vector,

189–195det(·), 66determinant, 66–74

as criterion for optimal design, 439computing, 533derivative of, 195Jacobian, 217of block diagonal matrix, 70of Cayley product, 88of diagonal matrix, 70of elementary operator matrix, 86of inverse, 117of Kronecker product, 96of nonnegative definite matrix, 345of partitioned matrix, 70, 122of permutation matrix, 87of positive definite matrix, 347of transpose, 70of triangular matrix, 70relation to eigenvalues, 141relation to geometric volume, 73, 217

diag(·), 56, 60, 596with matrix arguments, 62

diagonal element, 56diagonal expansion, 73diagonal factorization, 148, 152diagonal matrix, 57

determinant of, 70inverse of, 120multiplication, 78

diagonalizable matrix, 148–152, 306,343

orthogonally, 154unitarily, 147, 343, 387

diagonally dominant matrix, 57, 62,100, 348

differential, 188differentiation of vectors and matrices,

183–220digraph, 332

of a matrix, 333dim(·), 14

dimension of vector space, 14dimension reduction, 20, 356, 425direct method for solving linear systems,

272–277direct product (of matrices), 95direct product (of sets), 5direct product (of vector spaces), 20

basis for, 23direct sum decomposition of a vector

space, 19direct sum of matrices, 62direct sum of vector spaces, 18–20, 64

basis for, 22direct sum decomposition, 19

directed dissimilarity matrix, 369direction cosines, 38, 231discrete Fourier transform, 384discrete Legendre polynomials, 380discretization error, 498, 509dissimilarity matrix, 369distance, 32

between matrices, 175between vectors, 32

distance matrix, 369distributed computing, 463, 508, 544distributed linear algebra machine, 558distribution vector, 377div, 192divergence, 192divide and conquer, 506document-term matrix, 336dominant eigenvalue, 142Doolittle method, 245dot product of matrices, 97dot product of vectors, 23, 91double cone, 43double precision, 472, 480doubly stochastic matrix, 377Drazin inverse, 129–130dual cone, 44

Epq, E(π), Ep(a), Epq(a) (elementaryoperator matrices), 84

E(·) (expectation operator), 216E-optimality, 439echelon form, 111edge of a graph, 329EDP (exact dot product), 493effective degrees of freedom, 362, 430

Index 635

efficiency, computational, 503–508eigenpair, 134eigenspace, 144eigenvalue, 134–164, 166, 305–322

computing, 306–318, 558, 574Jacobi method, 313–316Krylov methods, 318power method, 311–313QR method, 316–318

of a graph, 391of a polynomial Exercise 3.26:, 181relation to singular value, 163upper bound on, 142, 145, 306

eigenvector, 134–164, 305–322left eigenvector, 135, 158

eigenvectors, linear independence of,143

EISPACK, 556elementary operation, 79elementary operator matrix, 79–87, 101,

242, 273eigenvalues, 137

elliptic metric, 94elliptic norm, 93endian, 480equivalence of norms, 29, 33, 170equivalence relation, 444equivalent canonical factorization, 112equivalent canonical form, 110, 112equivalent matrices, 110error bound, 496error in computations

cancellation, 487, 500error-free computations, 493measures of, 484, 494–497, 526rounding, 487, 495Exercise 10.10:, 517

error of approximation, 498discretization, 498truncation, 498

error, measures of, 272, 484, 494–497,526

error-free computations, 493errors-in-variables, 405essentially disjoint vector spaces, 15, 64estimable combinations of parameters,

409estimation and approximation, 431Euclidean distance, 32, 369

Euclidean distance matrix, 369Euclidean matrix norm (see also

Frobenius norm), 167Euclidean vector norm, 27Euler’s constant Exercise 10.2:, 516Euler’s integral, 593Euler’s rotation theorem, 231exact computations, 493exact dot product (EDP), 493exception, in computer operations, 483,

487expectation, 212–220exponent, 467exponential order, 503exponential, matrix, 153, 184extended precision, 472extrapolation, 509

factorization of a matrix, 108, 112, 147,148, 161, 225–227, 239–259, 272,274

Banachiewicz factorization, 255Bartlett decomposition, 346canonical singular value factorization,

162Cholesky factorization, 253–256diagonal factorization, 148equivalent canonical factorization,

112full rank factorization, 108, 112Gaussian elimination, 272LQ factorization, 247LU or LDU factorization, 240–246nonnegative matrix factorization,

257, 337orthogonally diagonal factorization,

147QL factorization, 247QR factorization, 246–252root-free Cholesky, 255RQ factorization, 247Schur factorization, 147singular value factorization, 161–164,

320, 336, 425, 532square root factorization, 160unitarily diagonal factorization, 147

fan-in algorithm, 485, 506fast Fourier transform (FFT), 386fast Givens rotation, 239, 525

636 Index

fill-in, 259, 526Fisher information, 205fixed-point representation, 465flat, 43floating-point representation, 466FLOP, or flop, 505FLOPS, or flops, 505Fortran, 475–478, 505, 546, 566–568Fourier coefficient, 41, 42, 99, 157, 163,

169Fourier expansion, 36, 41, 98, 157, 163,

169Fourier matrix, 384Frobenius norm, 167–169, 171, 175, 314,

340, 369Frobenius p norm, 169

full precision, 480full rank, 100, 101, 104, 111–113full rank factorization, 108

symmetric matrix, 112full rank partitioning, 104, 122

g1 inverse, 128, 130g2 inverse, 128g4 inverse (see also Moore-Penrose

inverse), 128gamma function, 220, 593GAMS (Guide to Available Mathemati-

cal Software), 539Gauss (software), 570Gauss-Markov theorem, 411Gauss-Newton method, 203Gauss-Seidel method, 277Gaussian elimination, 84, 272, 317Gaussian matrix, 84, 241gemm (general matrix-matrix), 556gemv (general matrix-vector), 556general linear group, 114, 133generalized eigenvalue, 160, 319generalized inverse, 124–125, 127–131,

249, 359relation to QR factorization, 249

generalized least squares, 414generalized least squares with equality

constraints Exercise 9.4d:, 451generalized variance, 366generating set, 14, 21

of a cone, 44generation of random numbers, 440

geometric multiplicity, 144geometry, 35, 73, 227, 231Gershgorin disks, 145GitHub, 539

Exercise 12.3:, 581Givens transformation (rotation),

236–239, 317QR factorization, 251

GL(·) (general linear group), 114GMP (software library), 466, 491, 492

Exercise 10.5:, 516GMRES, 282GNU Scientific Library (GSL), 556GPU (graphical processing unit), 559graceful underflow, 470gradient, 189

projected gradient, 207reduced gradient, 207

gradient descent, 198, 199gradient of a function, 190, 191gradual underflow, 470, 487Gram-Schmidt transformation, 39, 40,

524linear least squares, 289QR factorization, 252

Gramian matrix, 115, 117, 256, 289,358–359

completing the Gramian, 177graph of a matrix, 332graph theory, 8, 329–336, 389graphical processing unit (GPU), 559greedy algorithm, 506group, 114, 133GSL (GNU Scientific Library), 556guard digit, 485

Haar distribution, 219, 549Exercise 4.10:, 221Exercise 8.8:, 394

Haar invariant measure, 220Hadamard matrix, 380Hadamard multiplication, 94Hadamard’s inequality Exercise 5.5:,

260, 606Hadoop, 464, 544Hadoop Distributed File System

(HDFS), 464, 514half precision, 480Hankel matrix, 388

Index 637

Hankel norm, 389hat matrix, 360, 408HDF, HDF5 (Hierarchical Data

Format), 463HDFS (Hadoop Distributed File

System), 464, 514Helmert matrix, 378, 410Hemes formula, 286, 415Hermite form, 111Hermitian matrix, 56, 60Hessenberg matrix, 59, 316Hessian matrix, 193

projected Hessian, 207reduced Hessian, 207

Hessian of a function, 193hidden bit, 468Hierarchical Data Format (HDF), 463high-performance computing, 507Hilbert matrix, 548Hilbert space, 33, 168Hilbert-Schmidt norm (see also

Frobenius norm), 167Hoffman-Wielandt theorem, 339Holder norm, 27Holder’s inequality Exercise 2.11a:, 52hollow matrix, 57, 369homogeneous coordinates, 232

in graphics applications, Exercise 5.2:,259

homogeneous system of equations, 43,123

Horner’s method, 512Householder transformation (reflection),

233–236, 250, 317hyperplane, 43hypothesis testing, 408

idempotent matrix, 350–357identity matrix, 60, 76IDL (software), 6, 570IEC standards, 464IEEE standards, 464, 487

Standard 754, 472, 480, 487, 494Standard P1788, 493

IFIP Working Group 2.5, 460, 494ill-conditioned (problem or data), 264,

426, 499, 523artificial, 269stiff data, 502

ill-posed problem, 121image data, 461IMSL Libraries, 556, 560–562incidence matrix, 331–334, 390incomplete data, 435–438incomplete factorization, 258, 526independence, linear, see linear

independenceindependent vertices, 390index-index-value (sparse matrices), 548induced matrix norm, 165infinity, floating-point representation,

473, 487, 488infix operator, 489inner product, 23, 245

inner product of matrices, 96–99inner product space, 24

inner pseudoinverse, 128integer representation, 465integration and expectation, 212–220integration of vectors and matrices, 213Intel Math Kernel Library (MKL), 555,

578intersection graph, 334intersection of vector spaces, 18interval arithmetic, 492, 493invariance property, 227invariant distribution, 445invariant vector (eigenvector), 135inverse of a matrix, 107

determinant of, 117Drazin inverse, 129–130generalized inverse, 124–125, 127–131

Drazin inverse, 129–130Moore-Penrose inverse, 127–129pseudoinverse, 128

Kronecker product, 118left inverse, 108Moore-Penrose inverse, 127–129partitioned matrix, 122products or sums of matrices, 118pseudoinverse, 128right inverse, 108transpose, 107triangular matrix, 121

inverse of a vector, 31IRLS (iteratively reweighted least

squares), 297irreducible Markov chain, 444

638 Index

irreducible matrix, 311, 335–336,372–377, 445

is.na, 473isnan, is.nan, 473ISO (standards), 475, 562, 563isometric matrix, 167isometric transformation, 227isotropic transformation, 228iterative method, 277, 284, 305,

508–511, 525for solving linear systems, 277–283iterative refinement, 284

iteratively reweighted least squares, 297

Jacobi method for eigenvalues, 313–316Jacobi transformation (rotation), 236Jacobian, 191, 217Jordan block, 77, 139Jordan decomposition, 151Jordan form, 77, 111

of nilpotent matrix, 77

Kalman filter, 502Kantorovich inequality, 350Karush-Kuhn-Tucker conditions, 210kind (for data types), 476Kronecker multiplication, 94–96

inverse, 118properties, 95symmetric matrices, 96, 156

diagonalization, 156Kronecker structure, 219, 419Krylov method, 281, 318Krylov space, 281Kuhn-Tucker conditions, 210Kulisch accumulator, 493Kullback-Leibler divergence, 175

L1, L2, and L∞ normsof a matrix, 165of a symmetric matrix, 167of a vector, 27relations among, 170

L2 norm of a matrix (see also spectralnorm), 166

Lagrange multiplier, 208, 414Exercise 9.4a:, 450

Lagrangian function, 208Lanczos method, 318

LAPACK, 276, 534, 556, 558LAPACK95, 556

Laplace expansion, 68Laplace operator (∇2), 599Laplace operator (∇2), 192Laplacian matrix, 391lasso regression, 430latent root (see also eigenvalue), 135LAV (least absolute values), 295LDU factorization, 240–246leading principal submatrix, 62, 348least absolute values, 295least squares, 200–204, 256, 286–294

constrained, 206–211nonlinear, 202–204

least squares regression, 201left eigenvector, 135, 158left inverse, 108length of a vector, 4, 27, 31Leslie matrix, 378, 446

Exercise 8.10:, 395Exercise 9.22:, 455

Levenberg-Marquardt method, 204leverage, 408

Exercise 9.6:, 451life table, 447likelihood function, 204line, 43linear convergence, 509linear estimator, 409linear independence, 12, 99linear independence of eigenvectors, 143linear programming, 346linear regression, 401–422, 426–431

variable selection, 427LINPACK, 276, 533, 556Lisp-Stat (software), 570little endian, 480little o (order), 497, 591little omega (order), 497log order, 503log-likelihood function, 205Longley data Exercise 9.10:, 453loop unrolling, 565Lorentz cone, 44lower triangular matrix, 57Lp norm

of a matrix, 165of a vector, 27–28, 186

Index 639

LQ factorization, 247LR method, 306LU factorization, 240–246

computing, 558, 574

M-matrix, 393MACHAR, 478

Exercise 10.3(d)i:, 516machine epsilon, 470Mahalanobis distance, 94, 365Manhattan norm, 27manifold of a matrix, 55Maple (software), 491, 570MapReduce, 463, 513, 531, 544Markov chain, 443–445Markov chain Monte Carlo (MCMC),

445Mathematica (software), 491, 570Matlab (software), 546, 578–580matrix, 5matrix derivative, 183–220matrix exponential, 153, 184matrix factorization, 108, 112, 147, 148,

161, 225–227, 239–259, 272, 274matrix function, 152matrix gradient, 191matrix inverse, 107matrix multiplication, 75–99, 528

Cayley, 75, 94CUDA, 560Hadamard, 94inner product, 96–99Kronecker, 94–96MapReduce, 531Strassen algorithm, 529–531

matrix norm, 164–171orthogonally invariant, 164

matrix normal distribution, 218matrix of type 2, 58, 383matrix pencil, 161matrix polynomial, 78

Exercise 3.26:, 181matrix random variable, 218–220matrix storage mode, 546–548Matrix Template Library, 569max norm, 28maximal linearly independent subset,

12maximum likelihood, 204–206

MCMC (Markov chain Monte Carlo),445

mean, 35, 37mean vector, 35message passing, 557Message Passing Library, 557metric, 32, 175metric space, 32Microsoft R Open, 578MIL-STD-1753 standard, 478Minkowski inequality, 27

Exercise 2.11b:, 52Minkowski norm, 27minor, 67, 597missing data, 435–438, 571

representation of, 462, 473MKL (Intel Math Kernel Library), 555,

578mobile Jacobi scheme, 315modified Cholesky decomposition, 436“modified” Gauss-Newton, 203“modified” Gram-Schmidt (see also

Gram-Schmidt transformation),40

Moore-Penrose inverse, 127–129, 248,249, 291

relation to QR factorization, 249MPI (message passing interface), 557,

559MPL (Message Passing Library), 557multicollinearity, 265, 406multigrid method, 283multiple precision, 466, 491multiplicity of an eigenvalue, 144multivariate gamma function, 220multivariate linear regression, 418–422multivariate normal distribution,

217–219, 399, 441singular, 217, 432

multivariate random variable, 215–220

N (·), 126NA (“Not Available”), 462, 473, 571nabla (∇), 190, 191Nag Libraries, 556NaN (“Not-a-Number”), 473, 488NetCDF, 463netlib, xiv, 617network, 329–336, 391

640 Index

Newton’s method, 198nilpotent matrix, 77, 174NMF (nonnegative matrix factoriza-

tion), 257, 337noncentral chi-squared distribution, 400

PDF, equation (9.3), 400noncentral Wishart distribution

Exercise 4.12:, 222nonlinear regression, 201nonnegative definite matrix, 91, 159,

253, 344–350summary of properties, 344–345

nonnegative matrix, 258, 369nonnegative matrix factorization, 257,

337nonsingular matrix, 100, 111norm, 25–30

convexity, 26equivalence of norms, 29, 33, 170of a matrix, 164–171

orthogonally invariant, 164of a vector, 27–31weighted, 28, 93

normal distribution, 217–219matrix, 218multivariate, 217–219

normal equations, 256, 289, 404, 420normal matrix, 342

Exercise 8.1:, 394circulant matrix, 384

normal vector, 34normalized floating-point numbers, 468normalized generalized inverse (see also

Moore-Penrose inverse), 128normalized vector, 31normed space, 25not-a-number (“NaN”), 473NP-complete problem, 504nuclear norm, 169null space, 126, 127, 144nullity, 126numpy, 556, 569Nvidia, 559

O(·), 497, 503, 591o(·), 497, 591oblique projection, 356Octave (software), 578OLS (ordinary least squares), 288

one vector, 16, 34online algorithm, 512online processing, 512open-source, 538OpenMP, 557, 559operator matrix, 80, 273operator norm, 165optimal design, 438–440optimization of vector/matrix functions,

196–212constrained, 206–211least squares, 200–204, 206–211

order of a graph, 329order of a vector, 4order of a vector space, 15order of computations, 503order of convergence, 497order of error, 497ordinal relations among matrices, 92,

348ordinal relations among vectors, 16orthogonal array, 380orthogonal basis, 40–41orthogonal complement, 34, 126, 131orthogonal distance regression, 299–302,

405orthogonal group, 133, 220orthogonal matrices, binary relation-

ship, 98orthogonal matrix, 131–134, 228orthogonal residuals, 299–302, 405orthogonal transformation, 228orthogonal vector spaces, 34, 131orthogonal vectors, 33

Exercise 2.6:, 52orthogonalization (Gram-Schmidt

transformations), 38, 252, 524orthogonally diagonalizable, 147, 154,

338, 343, 423orthogonally invariant norm, 164, 167,

168orthogonally similar, 146, 154, 164, 168,

269, 343orthonormal vectors, 33out-of-core algorithm, 512outer product, 90, 245

Exercise 3.14:, 179outer product for matrix multiplication,

529

Index 641

outer pseudoinverse, 128, 129outer/inner products matrix, 357overdetermined linear system, 124, 255,

286overfitting, 298, 429overflow, in computer operations, 483,

487Overleaf, 539overloading, 11, 63, 165, 479, 489

p-inverse (see also Moore-Penroseinverse), 128

paging, 565parallel processing, 507, 508, 528, 530,

544, 557parallelogram equality, 27parallelotope, 74Parseval’s identity, 41, 169partial ordering, 16, 92, 348

Exercise 8.2a:, 394partial pivoting, 275partitioned matrix, 61, 79, 131

determinant, 70, 122sum of squares, 361, 413, 420

partitioned matrix, inverse, 122, 131partitioning sum of squares, 361, 413,

420PBLAS (parallel BLAS), 558pencil, 161permutation, 66permutation matrix, 80, 86, 273, 377Perron root, 371, 374Perron theorem, 371Perron vector, 371, 375, 445Perron-Frobenius theorem, 374pivoting, 84, 244, 249, 275PLASMA, 559polar cone, 45polynomial in a matrix, 78

Exercise 3.26:, 181polynomial order, 503polynomial regression, 379polynomial, evaluation of, 512pooled variance-covariance matrix, 368population model, 445portability, 481, 494, 540positive definite matrix, 91, 101,

159–160, 253, 346–350, 422summary of properties, 346–348

positive matrix, 258, 369positive semidefinite matrix, 91positive stable, 159, 393power method for eigenvalues, 311–313precision, 472–480, 491

arbitrary, 466double, 472, 480extended, 472half precision, 480infinite, 466multiple, 466, 491single, 472, 480

preconditioning, 282, 310, 525for eigenvalue computations, 310in the conjugate gradient method,

282primitive Markov chain, 445primitive matrix, 375, 445principal axis, 36principal components, 422–426principal components regression, 428principal diagonal, 56principal minor, 72, 104, 598principal submatrix, 62, 104, 241, 344,

347leading, 62, 348

probabilistic error bound, 497programming model, 463, 544projected gradient, 207projected Hessian, 207projection (of a vector), 20, 36projection matrix, 356–357, 408projective transformation, 228proper value (see also eigenvalue), 135PSBLAS (parallel sparse BLAS), 558pseudo-correlation matrix, 437pseudoinverse (see also Moore-Penrose

inverse), 128PV-Wave (software), 6, 570Pythagorean theorem, 27Python, 478, 546, 569, 570

Q-convergence, 509QL factorization, 247QR factorization, 246–252

and a generalized inverse, 249computing, 558, 574matrix rank, 250skinny, 247

642 Index

QR method for eigenvalues, 316–318quadratic convergence, 509quadratic form, 91, 93quasi-Newton method, 199quotient space, 115

R (software), 546, 570–578Microsoft R Open, 578Rcpp, 577RcppArmadillo, 577roxygen, 577RPy, 578RStudio, 578Spotfire S+ (software), 578

radix, 467random graph, 337random matrix, 218–220

BMvN distribution, 219computer generation Exercise 4.10:,

221correlation matrix, 441Haar distribution, 219Exercise 4.10:, 221

normal, 218orthogonal Exercise 4.10:, 221rank Exercise 4.11:, 222Wishart, 122, 346, 421, 432Exercise 4.12:, 222

random number generation, 440–442random matrices Exercise 4.10:, 221

random variable, 215range of a matrix, 55rank deficiency, 100, 144rank determination, 532rank of a matrix, 99–122, 250, 431, 532

of idempotent matrix, 351rank-revealing QR, 250, 431statistical tests, 431–435

rank of an array, 5rank reduction, 533rank(·), 99rank, linear independence, 99, 532rank, number of dimensions, 5rank-one decomposition, 164rank-one update, 234, 285rank-revealing QR, 250, 431, 532rate constant, 509rate of convergence, 509rational fraction, 491

Rayleigh quotient, 90, 157, 209, 392Rcpp, 577RcppBlaze, 559RCR (Replicated Computational

Results), 552real numbers, 466real-time algorithm, 512recursion, 511reduced gradient, 207reduced Hessian, 207reduced rank regression problem, 432reducibility, 311, 334–336, 372

Markov chains, 444reflection, 231–233reflector, 233reflexive generalized inverse, 128register, in computer processor, 485regression, 401–422, 426–431regression variable selection, 427regression, nonlinear, 201regular graph, 329regular matrix (see also diagonalizable

matrix), 149regularization, 298, 405, 429relative error, 484, 494, 526relative spacing, 470Reliable Computing, 493Replicated Computational Results

(RCR), 552Reproducible R Toolkit, 578reproducible research, 551–552, 578residue arithmetic, 493restarting, 525reverse communication, 564ρ(·) (spectral radius), 142Richardson extrapolation, 510ridge regression, 270, 362, 405, 418, 429

Exercise 9.11a:, 453right direct product, 95right inverse, 108robustness (algorithm or software), 500root of a function, 487root-free Cholesky, 255Rosser test matrix, 550rotation, 229–232, 236rounding, 473

rounding error, 487, 495row echelon form, 111row rank, 99

Index 643

row space, 56

row-major, 522, 539, 545row-sum norm, 166

roxygen, 577

RPy, 578RQ factorization, 247

RStudio, 578

S (software), 570

Samelson inverse, 31sample variance, computing, 501

saxpy, 12scalability, 506

ScaLAPACK, 558

scalar, 11scalar product, 23

scaled matrix, 365

scaled vector, 49scaling of a vector or matrix, 269

scaling of an algorithm, 503, 506Schatten p norm, 169

Schur complement, 121, 412, 421

Schur factorization, 147–148Schur norm (see also Frobenius norm),

167

SDP (semidefinite programming), 346Seidel adjacency matrix, 332

self-adjoint matrix (see also Hermitianmatrix), 56

semidefinite programming (SDP), 346

seminorm, 25semisimple eigenvalue, 144, 149

sequences of matrices, 171

sequences of vectors, 32shape of matrix, 5

shearing transformation, 228Sherman-Morrison formula, 285, 415

shifting eigenvalues, 310

shrinkage, 405side effect, 554

σ(·) (sign of permutation), 66, 86

σ(·) (spectrum of matrix), 141sign bit, 465

sign(·), 16significand, 467

similar canonical form, 149

similar matrices, 146similarity matrix, 369

similarity transformation, 146–148, 313,317

simple eigenvalue, 144simple graph, 329simple matrix (see also diagonalizable

matrix), 149single precision, 472, 480singular matrix, 100singular multivariate normal distribu-

tion, 217, 432singular value, 162, 425, 532

relation to eigenvalue, 163singular value decomposition, 161–164,

320, 336, 425, 532uniqueness, 163

skew diagonal element, 57skew diagonal matrix, 57skew symmetric matrix, 56, 60skew upper triangular matrix, 58, 388skinny QR factorization, 247smoothing matrix, 361, 418software testing, 548–551SOR (method), 279span(·), 14, 21, 55spanning set, 14, 21

of a cone, 44Spark (software system), 544sparse matrix, 59, 259, 277, 523, 526,

556, 557index-index-value, 548software, 557

CUDA, 560storage mode, 548

spectral circle, 142spectral condition number, 268, 269,

290spectral decomposition, 155, 163spectral norm, 166, 169spectral projector, 155spectral radius, 142, 166, 170, 278spectrum of a graph, 392spectrum of a matrix, 141–145splitting extrapolation, 511Spotfire S+ (software), 578square root matrix, 160, 252, 254, 345stability, 276, 500standard deviation, 49, 364

computing the standard deviation,501

644 Index

Standard Template Library, 569standards (see also specific standard),

460stationary point of vector/matrix

functions, 198statistical reference datasets (StRD),

551statlib, xiv, 617steepest descent, 198, 199Stiefel manifold, 133stiff data, 502stochastic matrix, 377stochastic process, 442–450stopping criterion, 508storage mode, for matrices, 546–548storage unit, 464, 467, 480Strassen algorithm, 529–531StRD (statistical reference datasets),

551stride, 522, 539, 554string, character, 461strongly connected graph, 334submatrix, 61, 79subspace of vector space, 17successive overrelaxation, 279summation, 485, 486summing vector, 34Sun ONE Studio Fortran 95, 493superlinear convergence, 509SVD (singular value decomposition),

161–164, 320, 336, 425, 532uniqueness, 163

sweep operator, 413Sylvester’s law of nullity, 117symmetric matrix, 56, 60, 112, 153–160,

338–343eingenvalues/vectors, 153–160equivalent forms, 112inverse of, 120summary of properties, 338

symmetric pair, 319symmetric storage mode, 61, 547

Taylor series, 188, 198Template Numerical Toolkit, 569tensor, 5term-document matrix, 336test problems for algorithms or software,

527, 548–551

Exercise 3.24:, 180consistency test, 527, 551Ericksen matrix, 549Exercise 12.8:, 582

Hilbert matrix, 548Matrix Market, 550randomly generated data, 549Exercise 11.5:, 536

Rosser matrix, 550StRD (statistical reference datasets),

551Wilkinson matrix, 550Exercise 12.9:, 582

Wilkinson’s polynomial, 499testable hypothesis, 410testing software, 548–551thread, 544Tikhonov regularization, 299, 429time series, 447–450

variance-covariance matrix, 383, 449Toeplitz matrix, 382, 449

circulant matrix, 383inverse of, 383

Exercise 8.12:, 395Exercise 12.12:, 583

total least squares, 299, 405tr(·), 65trace, 65

derivative of, 195of Cayley product, 87of idempotent matrix, 351of inner product, 92of Kronecker product, 96of matrix inner product, 97of outer product, 92relation to eigenvalues, 141

trace norm, 169transition matrix, 443translation transformation, 232transpose, 59

determinant of, 70generalized inverse of, 124inverse of, 107norm of, 164of Cayley product of matrices, 76of Kronecker product, 95of partitioned matrices, 63of sum of matrices, 63trace of, 65

Index 645

trapezoidal matrix, 58, 241, 246triangle inequality, 25, 164

Exercise 2.11b:, 52triangular matrix, 57, 84, 240

determinant of, 70inverse of, 121multiplication, 78

tridiagonal matrix, 58triple scalar product Exercise 2.19c:, 54triple vector product Exercise 2.19d:, 54truncation error, 42, 99, 498twos-complement representation, 465,

483type 2 matrix, 58, 383

ulp (“unit in the last place”), 471underdetermined linear system, 123underflow, in computer operations, 470,

487Unicode, 461union of vector spaces, 18unit in the last place (ulp), 471unit roundoff, 470unit vector, 16, 22, 36, 76unitarily diagonalizable, 147, 343, 387unitarily similar, 146unitary matrix, 132unrolling do-loop, 565updating a solution, 285, 293, 415–417

regression computations, 415–417upper Hessenberg form, 59, 316upper triangular matrix, 57usual norm (see also Frobenius norm),

167

V(·) (variance operator), 49, 216V(·) (vector space), 21, 55Vandermonde matrix, 379

Fourier matrix, 384variable metric method, 199variable selection, 427variance, computing, 501variance-covariance matrix, 216, 219

Kronecker structure, 219, 419positive definite approximation, 436sample, 365, 422

vec(·), 61vec-permutation matrix, 82vecdiag(·), 56

vech(·), 61

vector, 4centered vector, 48

mean vector, 35

normal vector, 34normalized vector, 31

null vector, 15one vector, 15, 34

“row vector”, 89

scaled vector, 49sign vector, 16

summing vector, 15, 34

unit vector, 16zero vector, 15

vector derivative, 183–220vector processing, 557

vector space, 13–15, 17–23, 55, 64, 126,127

basis, 21–23

definition, 13

dimension, 14direct product, 20

direct sum, 18–20direct sum decomposition, 19

essentially disjoint, 15

intersection, 18null vector space, 13

of matrices, 64

order, 15set operations, 17

subspace, 17union, 18

vector subspace, 17

vectorized processor, 508vertex of a graph, 329

volume as a determinant, 73, 217

weighted graph, 329

weighted least squares, 414with equality constraints Exer-

cise 9.4d:, 451

weighted norm, 28, 93Wilk’s Λ, 421

Wilkinson matrix, 550Wishart distribution, 122, 346

Exercise 4.12:, 222

Woodbury formula, 285, 415word, computer, 464, 467, 480

646 Index

XDR (external data representation),481

Yule-Walker equations, 449

Z-matrix, 393zero matrix, 76, 99zero of a function, 487zero vector, 15