Matrix Lec01 Intro

8/11/2019 Matrix Lec01 Intro

1/38

CSCI 5304 Fall 2014

COMPUTATIONAL ASPECTS OF MATRIX THEORY

Class time : 2:30 3:45 pm MW

Room : MechE 212

Instructor : Daniel Boley

URL : www-users.itlabs.umn.edu/classes/Fall-2014/csci5304/

September 3, 2014

Thanks to Yousef Saad for the use of his class notes.

CSci 5304: header 0-1


2/38

GENERAL INTRODUCTION

Linear algebra and numerical linear algebra Types of problems to be seen in this course

Mathematical background - matrices, eigenvalues, ... Quick review of Matlab

Vector norms, matrix norms

CSci 5304: BG 1-1


3/38

Introduction

This course is about Matrix algorithms or matrix com-

putations

It involves: algorithms for standard matrix computations(e.g. solving linear systems) - and their analysis (e.g., their

cost, numerical behavior, ..)

Matrix algorithms pervade most areas of science andengineering.

In computer science: recent increase of interest in matrix

algorithms for data mining, information retrieval, searchengines, pattern recognition, graphics, ...

CSci 5304: BG 1-2


4/38

Examples

Ancient Chinese Problem (3rd cent BC) [Babylonians

had similar problem]:

There are three types of corn, of which three bundles ofthe first, two of the second, and one of the third make 38

measures. Two of the first, three of the second and one of

the third make 33 measures. And one of the first, two of

the second and three of the third make 26 measures. How

many measures of corn are contained in one bundle of each

type?

CSci 5304: BG 1-3


5/38

Pagerank of Webpages (21st cent AD)

If one were to do a random walk from web page to webpage, following each link on a given web page at random

with equal likelihood, which are the pages to be encoun-

tered this way most often?

CSci 5304: BG 1-3a


6/38

Vibrations in mechanical systems. See:

www.cs.umn.edu/~saad/eig_book_2ndEd.pdf

Problem: Determine the vibration

modes of the mechanical system [to

avoid resonance]. See details inChapter 10 (sec. 10.2) of above

reference.

m1

m2

l1

l2

k1

k2

CSci 5304: BG 1-4
http://www.cs.umn.edu/~saad/eig_book_2ndEd.pdfhttp://www.cs.umn.edu/~saad/eig_book_2ndEd.pdfhttp://www.cs.umn.edu/~saad/eig_book_2ndEd.pdfhttp://www.cs.umn.edu/~saad/eig_book_2ndEd.pdf


7/38

Examples (cont.)

Method of least-squares (inspired by first use of least

squares ever, by Gauss around 1801)

A planet follows an elliptical orbit according toay2

+bxy

+cx+ dy+e = x2 in cartesian coordinates. Given a set of

noisy observations of(x, y)positions, compute a, b, c, d, e,

and use to predict future positions of the planet. This

least squares problem is nearly rank-deficient and hence

very sensitive to perturbations in the observations.

Read Wikipedias article on planet ceres:http://en.wikipedia.org/wiki/Ceres_(dwarf_planet)

CSci 5304: BG 1-5
http://en.wikipedia.org/wiki/Ceres_(dwarf_planet)http://en.wikipedia.org/wiki/Ceres_(dwarf_planet)


8/38

Background in linear algebra

Review vector spaces. Read section 1.1 of text on vector

notation.

A vector subspace of R

n

is a subset of R

n

that is alsoa real vector space. The set of all linear combinations of

a set of vectors G = {a1, a2, . . . , aq} of Rn is a vector

subspace called the linear span ofG,

If the ais are linearly independent, then each vector of

span{G} admits a unique expression as a linear combina-

tion of the ais. The set G is then called a basis.

Recommended reading: Sections 1.1 1.6 of

www.cs.umn.edu/~saad/eig_book_2ndEd.pdf

CSci 5304: BG 1-6
http://www.cs.umn.edu/~saad/eig_book_2ndEd.pdfhttp://www.cs.umn.edu/~saad/eig_book_2ndEd.pdfhttp://www.cs.umn.edu/~saad/eig_book_2ndEd.pdfhttp://www.cs.umn.edu/~saad/eig_book_2ndEd.pdf


9/38

Matrices

A realmnmatrixAis anmnarray of real numbers

aij, i = 1, . . . , m, j = 1, . . . , n .

The set of all mnmatrices is a real vector space denotedby Rmn.

Complex matrices defined similarly.

A matrix represents a linear mapping between two vector

spaces of finite dimension n and m.

CSci 5304: BG 1-7


10/38

Operations:

Addition: C= A+ B, where A, B, C Rmn and

cij = aij+bij, i = 1, 2, . . . m, j = 1, 2, . . . n .

Multiplication by a scalar: C= A, where

cij = aij, i= 1, 2, . . . m, j = 1, 2, . . . n .

CSci 5304: BG 1-7a


11/38

Multiplication by another matrix:

C= AB,

where A Rmn, B Rnp, C Rmp, and

cij =

nk=1

aikbkj.

CSci 5304: BG 1-8


12/38

Transposition: IfA Rmn then its transpose is a matrix

C Rnm with entries

cij = aji, i= 1, . . . , n, j = 1, . . . , m

Notation : AT.

CSci 5304: BG 1-8a


13/38

Transpose Conjugate: for complex matrices, the trans-

pose conjugate matrix denoted by AH is more relevant:

AH = AT = AT.

CSci 5304: BG 1-9


14/38

Review: Matrix-matrix and Matrix-vector producs

Recall definition ofC= A B.

Recall what C represents [in terms of mappings]..

Can do the product column-wise [Matlab notation used]:

C:,j =n

k=1

bkjA:,k

Can do it row-wise:

Ci,: =n

k=1

aikBk,:

CSci 5304: BG 1-10


15/38

Can do it as a sum of outer-product matrices:

C=

nk=1

A:,kBk,:

Verify (prove) all 3 formulas above..

Complexity? [number of multiplications and additions]

What happen to these 3 different approaches when B

has one column (p = 1)?

CSci 5304: BG 1-11


16/38

Square matrices, matrix inversion, eigenvalues

Square matrix: n= m - so A Rnn

Identity matrix: square matrix with

aij = 1 ifi = j

0 otherwise

Notation: I.

Property: AI= IA = A

Inverse ofA (when it exists) is a matrix Csuch that

AC= CA= I

Notation: A1

CSci 5304: BG 1-12


17/38

Eigenvalues and eigenvectors

A complex scalar is an eigenvalue of the square matrixA if a nonzero vector u ofCn exists such that

Au= u.

The vector u is an eigenvector of A associated with .

The set of all the eigenvalues of A is the spectrum of A.

Notation: (A).

is an eigenvalue ofA if and only ifdet(A I) = 0

pA() = det(A I) is a polynomial of degree n in = characteristic polynomial ofA.

(A)if and only if is a root of the characteristic

polynomial pA().CSci 5304: BG 1-13


18/38

Spectral radius = The maximum modulus of the eigen-

values

(A) = max(A) ||.

Trace ofA = sum of diagonal elements ofA.

tr(A) = ni=1 aii. tr(A) = sum of all the eigenvalues of A counted with

their multiplicities.

CSci 5304: BG 1-13a


19/38

Recall that det(A) = product of all the eigenvalues of

A counted with their multiplicities.

Example: Trace, spectral radius, and determinant of

A = 2 13 0 .

CSci 5304: BG 1-14


20/38

For two n n matrices A and B are the eigenvalues

of AB and BA the same?

IfA is nonsingular what are the eigenvalues/eigenvectors

of A1?

What are the eigenvalues/eigenvectors ofAk

for a giveninteger power k?

What are the eigenvalues/eigenvectors of p(A) for a

polynomial p?

Review the Jordan canonical form. Define the eigen-

values, and eigenvectors from the Jordan form.

CSci 5304: BG 1-15


21/38

Range and null space

Range: Ran(A) ={Ax | x Rn} Rm

Null Space: Null(A) ={x Rn | Ax= 0 } Rn

Range = linear span of the columns ofA

Rank of a matrix rank(A) = dim(Ran(A)) n

CSci 5304: BG 1-15a


22/38

Ran(A) Rm rank (A) m

rank (A)min{m, n}

rank (A) = number of linearly independent columns of

A = number of linearly independent rows ofA

A is of full rank ifrank(A) = min{m, n}. Otherwiseit isrank-deficient.

CSci 5304: BG 1-16


23/38

Rank+Nullity theorem for an m n matrix:

dim(Ran(A)) +dim(Null(A)) = n

Apply toAT: dim(Ran(AT))+dim(Null(AT)) = m

rank(A) +dim(Null(AT)) = m

Terminology:

dim(Null(A)) is theNullityof A

n rank(A) is called theco-rankof A

so: co-rank ofA equals nullity ofA.

CSci 5304: BG 1-17


24/38

Show that A Rnn is of rank one iff [if and only if]

there exist two nonzero vectors u and v such that

A = uvT.

What are the eigenvalues and eigenvectors ofA?

Is it true that

rank(A) = rank(A) = rank(AT) = rank(AH) ?

Matlab exercise: explore the matlab function rank.

Find the range and null space of the matrix

1 1 0

1 2 3

1 2 1

2 1 1

Verify your result with matlab.CSci 5304: BG 1-18


25/38

CSci 5304: BG 1-19


26/38

Types of matrices (square)

Symmetric AT = A. Skew-symmetric AT =A.

Hermitian AH = A. Skew-Hermitian AH =A.

Normal AHA= AAH.

Nonnegative aij 0, i , j = 1, . . . , n

Similarly for nonpositive, positive, and negative matrices

Unitary QHQ= I.

CSci 5304: matrices 1-20 GvL (2.1);Heath 2.1; TB 1-2


27/38

[Note: Common useage restricts this definintion to com-

plex matrices an orthogonal matrix is a unitary real matrix

not very natural ]

Orthogonal QTQ= D [orthogonal columns]

The term orthonormal matrix is never used.



28/38

What is the inverse of a unitary matrix?

What can you say about the diagonal entries of a skew-

symmetric (real) matrix?

What can you say about the diagonal entries of a

Hermitian (complex) matrix? What can you say about the diagonal entries of a skew-

Hermitian (complex) matrix?

The following types of matrices are normal [true-false]:

real symmetric, real skew-symmetric, complex Hermitian,

complex skew-Hermitian.

Find all real 2 2 matrices that are normal.

Show that any triangular matrix that is normal is diag-

onalCSci 5304: matrices 1-22 GvL (2.1);Heath 2.1; TB 1-2


29/38

Matrices with structure

Diagonal aij = 0 for j = i. Notation :

A = diag (a11, a22, . . . , ann) .

Upper triangular aij = 0 for i > j.

Lower triangular aij = 0 for i < j.

Upper bidiagonal aij = 0 for j = i or j = i+ 1.

Lower bidiagonal aij = 0 for j = i or j = i 1.

Tridiagonal aij = 0 when |i j| >1.



30/38

Banded aij = 0 only when i ml j i + mu,

Bandwidth = ml+mu+ 1.

Upper Hessenberg aij = 0 when i > j + 1. Lower

Hessenberg matrices can be defined similarly.

Outer productA= uvT

, where bothuandvare vectors. Block tridiagonal generalizes tridiagonal matrices by

replacing each nonzero entry by a square matrix.



31/38

Special matrices

Vandermonde:

Given a column of entries[x0, x1, , xn]T put its(com-

ponent-wise)powers into the columns of a matrix V:

V =

1 x0 x

20 x

n0

1 x1 x21 x

21

... ... ... ...

1 xn x2n x

nn

Try the matlab function vander

What does the matrix-vector product V a represent?

Interpret the solution of the linear system V a = y

where a is the unknown. Sketch a fast solution method

based on this.CSci 5304: matrices 1-25 GvL (2.1);Heath 2.1; TB 1-2


32/38


T li


33/38

Toeplitz:

Entries are constant along diagonals, i.e., aij = rji. Determined by m+ n 1 values rji.

T =

r0 r1 r2 r3 r4

r1 r0 r1 r2 r3

r2 r1 r0 r1 r2

r3 r2 r1 r0 r1r4 r3 r2 r1 r0

T oeplitz

CSci 5304: matrices 1-26a GvL (2.1);Heath 2.1; TB 1-2

T lit t ( ) b l i O( 2)


34/38

Toeplitz systems (m= n) can be solver in O(n2) ops.

The whole inverse (!) can be determined inO

(n2

) ops. Explore toeplitz(c,r) in matlab.



35/38

Hankel: Entries are

constant along anti-

diagonals, i.e., aij =

hj+i1.

Determined bym+n

1 values hj+i1.

H=

h1 h2 h3 h4 h5

h2 h3 h4 h5 h6h3 h4 h5 h6 h7

h4 h5 h6 h7 h8

h5

h6

h7

h8

h9

Hankel



36/38

Circulant : Entries

in a row are cyclicallyright-shifted to form

next row. Determined

by n values.

C=

v1 v2 v3 v4 v5

v5 v1 v2 v3 v4v4 v5 v1 v2 v3

v3 v4 v5 v1 v2

v2

v3

v4

v5

v1

Circulant


Explore hankel(c r) in matlab


37/38

Explore hankel(c,r) in matlab.

How can you generate a circulant matrix in matlab?


S i


38/38

Sparse matrices

Matrices with very few nonzero entries so few thatthis can be exploited.

Many of the large matrices encountered in applications

are sparse.

Main idea of sparse matrix techniques is not to rep-

resent the zeros.

This will be covered in some detail at the end of the

course.


Matrix Lec01 Intro

Documents

Transcript of Matrix Lec01 Intro