Linear Algebra - Part IIrgrosse/courses/csc411_f18/... · 2018-10-01 · 2 1 1 2 & The...
Transcript of Linear Algebra - Part IIrgrosse/courses/csc411_f18/... · 2018-10-01 · 2 1 1 2 & The...
![Page 1: Linear Algebra - Part IIrgrosse/courses/csc411_f18/... · 2018-10-01 · 2 1 1 2 & The characteristic polynomial is: det(A−λI) = det % 2−λ 1 1 2−λ & = 3−4λ+λ2 = 0 It](https://reader033.fdocuments.net/reader033/viewer/2022042418/5f33d6055bec6b65e409df57/html5/thumbnails/1.jpg)
Linear Algebra - Part IIProjection Eigendecomposition SVD
Punit Shah
(Adapted from Sargur Sriharirsquos slides)
Brief Review from Part 1
Symmetric MatrixA = AT
Orthogonal Matrix
ATA = AAT = I and Aminus1 = AT
L2 Norm
||x||2 =
983158983131
i
x2i
Linear Algebra Part II 220
Angle Between Vectors
Dot product of two vectors can be written in terms oftheir L2 norms and the angle θ between them
aTb = ||a||2||b||2 cos(θ)
Linear Algebra Part II 320
Cosine Similarity
Cosine between two vectors is a measure of theirsimilarity
cos(θ) =a middot b
||a|| ||b||
Orthogonal Vectors Two vectors a and b areorthogonal to each other if a middot b = 0
Linear Algebra Part II 420
Vector Projection Given two vectors a and b let b = b
||b|| be the unit vectorin the direction of b
Then a1 = a1b is the orthogonal projection of a onto astraight line parallel to b where
a1 = ||a|| cos(θ) = a middot b = a middot b||b||
Image taken from wikipedia
Linear Algebra Part II 520
Diagonal Matrix
Diagonal matrix has mostly zeros with non-zero entriesonly in the diagonal eg identity matrix
A square diagonal matrix with diagonal elements given byentries of vector v is denoted
diag(v)
Multiplying vector x by a diagonal matrix is efficient
diag(v)x = v ⊙ x
Inverting a square diagonal matrix is efficient
diag(v)minus1 = diag983059[1v1
1vn]T983060
Linear Algebra Part II 620
Determinant
Determinant of a square matrix is a mapping to a scalar
det(A) or |A|
Measures how much multiplication by the matrix expandsor contracts the space
Determinant of product is the product of determinants
det(AB) = det(A)det(B)
Image taken from wikipedia
Linear Algebra Part II 720
List of Equivalencies
The following are all equivalent
A is invertible ie Aminus1 exists
Ax = b has a unique solution
Columns of A are linearly independent
det(A) ∕= 0
Ax = 0 has a unique trivial solution x = 0
Linear Algebra Part II 820
Zero Determinant
If det(A) = 0 then
A is linearly dependent
Ax = b has no solution or infinitely many solutions
Ax = 0 has a non-zero solution
Linear Algebra Part II 920
Matrix Decomposition
We can decompose an integer into its prime factors eg12 = 2 times 2 times 3
Similarly matrices can be decomposed into factors tolearn universal properties
A = Vdiag(λ)Vminus1
Linear Algebra Part II 1020
Eigenvectors
An eigenvector of a square matrix A is a nonzero vector vsuch that multiplication by A only changes the scale of v
Av = λv
The scalar λ is known as the eigenvalue
If v is an eigenvector of A so is any rescaled vector svMoreover sv still has the same eigenvalue Thus weconstrain the eigenvector to be of unit length
||v|| = 1
Linear Algebra Part II 1120
Characteristic Polynomial
Eigenvalue equation of matrix A
Av = λvAv minus λv = 0
(A minus λI)v = 0
If nonzero solution for v exists then it must be the casethat
det(A minus λI) = 0
Unpacking the determinant as a function of λ we get
|A minus λI| = (λ1 minus λ)(λ2 minus λ) (λn minus λ) = 0
The λ1λ2 λn are roots of the characteristicpolynomial and are eigenvalues of A
Linear Algebra Part II 1220
Example
Consider the matrix
A =
9830632 11 2
983064
The characteristic polynomial is
det(A minus λI) = det9830632 minus λ 1
1 2 minus λ
983064= 3 minus 4λ+ λ2 = 0
It has roots λ = 1 and λ = 3 which are the twoeigenvalues of A
We can then solve for eigenvectors using Av = λv
vλ=1 = [1minus1]T and vλ=3 = [1 1]T
Linear Algebra Part II 1320
Eigendecomposition
Suppose that n times n matrix A has n linearly independenteigenvectors v(1) v(n) with eigenvaluesλ1 λn
Concatenate eigenvectors to form matrix V
Concatenate eigenvalues to form vectorλ = [λ1 λn]
T
The eigendecomposition of A is given by
A = Vdiag(λ)Vminus1
Linear Algebra Part II 1420
Symmetric Matrices
Every real symmetric matrix A can be decomposed intoreal-valued eigenvectors and eigenvalues
A = QΛQT
Q is an orthogonal matrix of the eigenvectors of A andΛ is a diagonal matrix of eigenvalues
We can think of A as scaling space by λi in direction v(i)
Linear Algebra Part II 1520
Eigendecomposition is not Unique
Decomposition is not unique when two eigenvalues arethe same
By convention order entries of Λ in descending orderThen eigendecomposition is unique if all eigenvalues areunique
If any eigenvalue is zero then the matrix is singular
Linear Algebra Part II 1620
Positive Definite Matrix
A matrix whose eigenvalues are all positive is calledpositive definite
If eigenvalues are positive or zero then matrix is calledpositive semidefinite
Positive definite matrices guarantee that
xTAx gt 0 for any nonzero vector x
Similarly positive semidefinite guarantees xTAx ge 0
Linear Algebra Part II 1720
Singular Value Decomposition (SVD)
If A is not square eigendecomposition is undefined
SVD is a decomposition of the form
A = UDVT
SVD is more general than eigendecomposition
Every real matrix has a SVD
Linear Algebra Part II 1820
SVD Definition (1)
Write A as a product of three matrices A = UDVT
If A is m times n then U is m timesm D is m times n and V isn times n
U and V are orthogonal matrices and D is a diagonalmatrix (not necessarily square)
Diagonal entries of D are called singular values of A
Columns of U are the left singular vectors andcolumns of V are the right singular vectors
Linear Algebra Part II 1920
SVD Definition (2)
SVD can be interpreted in terms of eigendecompostion
Left singular vectors of A are the eigenvectors of AAT
Right singular vectors of A are the eigenvectors of ATA
Nonzero singular values of A are square roots ofeigenvalues of ATA and AAT
Linear Algebra Part II 2020
![Page 2: Linear Algebra - Part IIrgrosse/courses/csc411_f18/... · 2018-10-01 · 2 1 1 2 & The characteristic polynomial is: det(A−λI) = det % 2−λ 1 1 2−λ & = 3−4λ+λ2 = 0 It](https://reader033.fdocuments.net/reader033/viewer/2022042418/5f33d6055bec6b65e409df57/html5/thumbnails/2.jpg)
Brief Review from Part 1
Symmetric MatrixA = AT
Orthogonal Matrix
ATA = AAT = I and Aminus1 = AT
L2 Norm
||x||2 =
983158983131
i
x2i
Linear Algebra Part II 220
Angle Between Vectors
Dot product of two vectors can be written in terms oftheir L2 norms and the angle θ between them
aTb = ||a||2||b||2 cos(θ)
Linear Algebra Part II 320
Cosine Similarity
Cosine between two vectors is a measure of theirsimilarity
cos(θ) =a middot b
||a|| ||b||
Orthogonal Vectors Two vectors a and b areorthogonal to each other if a middot b = 0
Linear Algebra Part II 420
Vector Projection Given two vectors a and b let b = b
||b|| be the unit vectorin the direction of b
Then a1 = a1b is the orthogonal projection of a onto astraight line parallel to b where
a1 = ||a|| cos(θ) = a middot b = a middot b||b||
Image taken from wikipedia
Linear Algebra Part II 520
Diagonal Matrix
Diagonal matrix has mostly zeros with non-zero entriesonly in the diagonal eg identity matrix
A square diagonal matrix with diagonal elements given byentries of vector v is denoted
diag(v)
Multiplying vector x by a diagonal matrix is efficient
diag(v)x = v ⊙ x
Inverting a square diagonal matrix is efficient
diag(v)minus1 = diag983059[1v1
1vn]T983060
Linear Algebra Part II 620
Determinant
Determinant of a square matrix is a mapping to a scalar
det(A) or |A|
Measures how much multiplication by the matrix expandsor contracts the space
Determinant of product is the product of determinants
det(AB) = det(A)det(B)
Image taken from wikipedia
Linear Algebra Part II 720
List of Equivalencies
The following are all equivalent
A is invertible ie Aminus1 exists
Ax = b has a unique solution
Columns of A are linearly independent
det(A) ∕= 0
Ax = 0 has a unique trivial solution x = 0
Linear Algebra Part II 820
Zero Determinant
If det(A) = 0 then
A is linearly dependent
Ax = b has no solution or infinitely many solutions
Ax = 0 has a non-zero solution
Linear Algebra Part II 920
Matrix Decomposition
We can decompose an integer into its prime factors eg12 = 2 times 2 times 3
Similarly matrices can be decomposed into factors tolearn universal properties
A = Vdiag(λ)Vminus1
Linear Algebra Part II 1020
Eigenvectors
An eigenvector of a square matrix A is a nonzero vector vsuch that multiplication by A only changes the scale of v
Av = λv
The scalar λ is known as the eigenvalue
If v is an eigenvector of A so is any rescaled vector svMoreover sv still has the same eigenvalue Thus weconstrain the eigenvector to be of unit length
||v|| = 1
Linear Algebra Part II 1120
Characteristic Polynomial
Eigenvalue equation of matrix A
Av = λvAv minus λv = 0
(A minus λI)v = 0
If nonzero solution for v exists then it must be the casethat
det(A minus λI) = 0
Unpacking the determinant as a function of λ we get
|A minus λI| = (λ1 minus λ)(λ2 minus λ) (λn minus λ) = 0
The λ1λ2 λn are roots of the characteristicpolynomial and are eigenvalues of A
Linear Algebra Part II 1220
Example
Consider the matrix
A =
9830632 11 2
983064
The characteristic polynomial is
det(A minus λI) = det9830632 minus λ 1
1 2 minus λ
983064= 3 minus 4λ+ λ2 = 0
It has roots λ = 1 and λ = 3 which are the twoeigenvalues of A
We can then solve for eigenvectors using Av = λv
vλ=1 = [1minus1]T and vλ=3 = [1 1]T
Linear Algebra Part II 1320
Eigendecomposition
Suppose that n times n matrix A has n linearly independenteigenvectors v(1) v(n) with eigenvaluesλ1 λn
Concatenate eigenvectors to form matrix V
Concatenate eigenvalues to form vectorλ = [λ1 λn]
T
The eigendecomposition of A is given by
A = Vdiag(λ)Vminus1
Linear Algebra Part II 1420
Symmetric Matrices
Every real symmetric matrix A can be decomposed intoreal-valued eigenvectors and eigenvalues
A = QΛQT
Q is an orthogonal matrix of the eigenvectors of A andΛ is a diagonal matrix of eigenvalues
We can think of A as scaling space by λi in direction v(i)
Linear Algebra Part II 1520
Eigendecomposition is not Unique
Decomposition is not unique when two eigenvalues arethe same
By convention order entries of Λ in descending orderThen eigendecomposition is unique if all eigenvalues areunique
If any eigenvalue is zero then the matrix is singular
Linear Algebra Part II 1620
Positive Definite Matrix
A matrix whose eigenvalues are all positive is calledpositive definite
If eigenvalues are positive or zero then matrix is calledpositive semidefinite
Positive definite matrices guarantee that
xTAx gt 0 for any nonzero vector x
Similarly positive semidefinite guarantees xTAx ge 0
Linear Algebra Part II 1720
Singular Value Decomposition (SVD)
If A is not square eigendecomposition is undefined
SVD is a decomposition of the form
A = UDVT
SVD is more general than eigendecomposition
Every real matrix has a SVD
Linear Algebra Part II 1820
SVD Definition (1)
Write A as a product of three matrices A = UDVT
If A is m times n then U is m timesm D is m times n and V isn times n
U and V are orthogonal matrices and D is a diagonalmatrix (not necessarily square)
Diagonal entries of D are called singular values of A
Columns of U are the left singular vectors andcolumns of V are the right singular vectors
Linear Algebra Part II 1920
SVD Definition (2)
SVD can be interpreted in terms of eigendecompostion
Left singular vectors of A are the eigenvectors of AAT
Right singular vectors of A are the eigenvectors of ATA
Nonzero singular values of A are square roots ofeigenvalues of ATA and AAT
Linear Algebra Part II 2020
![Page 3: Linear Algebra - Part IIrgrosse/courses/csc411_f18/... · 2018-10-01 · 2 1 1 2 & The characteristic polynomial is: det(A−λI) = det % 2−λ 1 1 2−λ & = 3−4λ+λ2 = 0 It](https://reader033.fdocuments.net/reader033/viewer/2022042418/5f33d6055bec6b65e409df57/html5/thumbnails/3.jpg)
Angle Between Vectors
Dot product of two vectors can be written in terms oftheir L2 norms and the angle θ between them
aTb = ||a||2||b||2 cos(θ)
Linear Algebra Part II 320
Cosine Similarity
Cosine between two vectors is a measure of theirsimilarity
cos(θ) =a middot b
||a|| ||b||
Orthogonal Vectors Two vectors a and b areorthogonal to each other if a middot b = 0
Linear Algebra Part II 420
Vector Projection Given two vectors a and b let b = b
||b|| be the unit vectorin the direction of b
Then a1 = a1b is the orthogonal projection of a onto astraight line parallel to b where
a1 = ||a|| cos(θ) = a middot b = a middot b||b||
Image taken from wikipedia
Linear Algebra Part II 520
Diagonal Matrix
Diagonal matrix has mostly zeros with non-zero entriesonly in the diagonal eg identity matrix
A square diagonal matrix with diagonal elements given byentries of vector v is denoted
diag(v)
Multiplying vector x by a diagonal matrix is efficient
diag(v)x = v ⊙ x
Inverting a square diagonal matrix is efficient
diag(v)minus1 = diag983059[1v1
1vn]T983060
Linear Algebra Part II 620
Determinant
Determinant of a square matrix is a mapping to a scalar
det(A) or |A|
Measures how much multiplication by the matrix expandsor contracts the space
Determinant of product is the product of determinants
det(AB) = det(A)det(B)
Image taken from wikipedia
Linear Algebra Part II 720
List of Equivalencies
The following are all equivalent
A is invertible ie Aminus1 exists
Ax = b has a unique solution
Columns of A are linearly independent
det(A) ∕= 0
Ax = 0 has a unique trivial solution x = 0
Linear Algebra Part II 820
Zero Determinant
If det(A) = 0 then
A is linearly dependent
Ax = b has no solution or infinitely many solutions
Ax = 0 has a non-zero solution
Linear Algebra Part II 920
Matrix Decomposition
We can decompose an integer into its prime factors eg12 = 2 times 2 times 3
Similarly matrices can be decomposed into factors tolearn universal properties
A = Vdiag(λ)Vminus1
Linear Algebra Part II 1020
Eigenvectors
An eigenvector of a square matrix A is a nonzero vector vsuch that multiplication by A only changes the scale of v
Av = λv
The scalar λ is known as the eigenvalue
If v is an eigenvector of A so is any rescaled vector svMoreover sv still has the same eigenvalue Thus weconstrain the eigenvector to be of unit length
||v|| = 1
Linear Algebra Part II 1120
Characteristic Polynomial
Eigenvalue equation of matrix A
Av = λvAv minus λv = 0
(A minus λI)v = 0
If nonzero solution for v exists then it must be the casethat
det(A minus λI) = 0
Unpacking the determinant as a function of λ we get
|A minus λI| = (λ1 minus λ)(λ2 minus λ) (λn minus λ) = 0
The λ1λ2 λn are roots of the characteristicpolynomial and are eigenvalues of A
Linear Algebra Part II 1220
Example
Consider the matrix
A =
9830632 11 2
983064
The characteristic polynomial is
det(A minus λI) = det9830632 minus λ 1
1 2 minus λ
983064= 3 minus 4λ+ λ2 = 0
It has roots λ = 1 and λ = 3 which are the twoeigenvalues of A
We can then solve for eigenvectors using Av = λv
vλ=1 = [1minus1]T and vλ=3 = [1 1]T
Linear Algebra Part II 1320
Eigendecomposition
Suppose that n times n matrix A has n linearly independenteigenvectors v(1) v(n) with eigenvaluesλ1 λn
Concatenate eigenvectors to form matrix V
Concatenate eigenvalues to form vectorλ = [λ1 λn]
T
The eigendecomposition of A is given by
A = Vdiag(λ)Vminus1
Linear Algebra Part II 1420
Symmetric Matrices
Every real symmetric matrix A can be decomposed intoreal-valued eigenvectors and eigenvalues
A = QΛQT
Q is an orthogonal matrix of the eigenvectors of A andΛ is a diagonal matrix of eigenvalues
We can think of A as scaling space by λi in direction v(i)
Linear Algebra Part II 1520
Eigendecomposition is not Unique
Decomposition is not unique when two eigenvalues arethe same
By convention order entries of Λ in descending orderThen eigendecomposition is unique if all eigenvalues areunique
If any eigenvalue is zero then the matrix is singular
Linear Algebra Part II 1620
Positive Definite Matrix
A matrix whose eigenvalues are all positive is calledpositive definite
If eigenvalues are positive or zero then matrix is calledpositive semidefinite
Positive definite matrices guarantee that
xTAx gt 0 for any nonzero vector x
Similarly positive semidefinite guarantees xTAx ge 0
Linear Algebra Part II 1720
Singular Value Decomposition (SVD)
If A is not square eigendecomposition is undefined
SVD is a decomposition of the form
A = UDVT
SVD is more general than eigendecomposition
Every real matrix has a SVD
Linear Algebra Part II 1820
SVD Definition (1)
Write A as a product of three matrices A = UDVT
If A is m times n then U is m timesm D is m times n and V isn times n
U and V are orthogonal matrices and D is a diagonalmatrix (not necessarily square)
Diagonal entries of D are called singular values of A
Columns of U are the left singular vectors andcolumns of V are the right singular vectors
Linear Algebra Part II 1920
SVD Definition (2)
SVD can be interpreted in terms of eigendecompostion
Left singular vectors of A are the eigenvectors of AAT
Right singular vectors of A are the eigenvectors of ATA
Nonzero singular values of A are square roots ofeigenvalues of ATA and AAT
Linear Algebra Part II 2020
![Page 4: Linear Algebra - Part IIrgrosse/courses/csc411_f18/... · 2018-10-01 · 2 1 1 2 & The characteristic polynomial is: det(A−λI) = det % 2−λ 1 1 2−λ & = 3−4λ+λ2 = 0 It](https://reader033.fdocuments.net/reader033/viewer/2022042418/5f33d6055bec6b65e409df57/html5/thumbnails/4.jpg)
Cosine Similarity
Cosine between two vectors is a measure of theirsimilarity
cos(θ) =a middot b
||a|| ||b||
Orthogonal Vectors Two vectors a and b areorthogonal to each other if a middot b = 0
Linear Algebra Part II 420
Vector Projection Given two vectors a and b let b = b
||b|| be the unit vectorin the direction of b
Then a1 = a1b is the orthogonal projection of a onto astraight line parallel to b where
a1 = ||a|| cos(θ) = a middot b = a middot b||b||
Image taken from wikipedia
Linear Algebra Part II 520
Diagonal Matrix
Diagonal matrix has mostly zeros with non-zero entriesonly in the diagonal eg identity matrix
A square diagonal matrix with diagonal elements given byentries of vector v is denoted
diag(v)
Multiplying vector x by a diagonal matrix is efficient
diag(v)x = v ⊙ x
Inverting a square diagonal matrix is efficient
diag(v)minus1 = diag983059[1v1
1vn]T983060
Linear Algebra Part II 620
Determinant
Determinant of a square matrix is a mapping to a scalar
det(A) or |A|
Measures how much multiplication by the matrix expandsor contracts the space
Determinant of product is the product of determinants
det(AB) = det(A)det(B)
Image taken from wikipedia
Linear Algebra Part II 720
List of Equivalencies
The following are all equivalent
A is invertible ie Aminus1 exists
Ax = b has a unique solution
Columns of A are linearly independent
det(A) ∕= 0
Ax = 0 has a unique trivial solution x = 0
Linear Algebra Part II 820
Zero Determinant
If det(A) = 0 then
A is linearly dependent
Ax = b has no solution or infinitely many solutions
Ax = 0 has a non-zero solution
Linear Algebra Part II 920
Matrix Decomposition
We can decompose an integer into its prime factors eg12 = 2 times 2 times 3
Similarly matrices can be decomposed into factors tolearn universal properties
A = Vdiag(λ)Vminus1
Linear Algebra Part II 1020
Eigenvectors
An eigenvector of a square matrix A is a nonzero vector vsuch that multiplication by A only changes the scale of v
Av = λv
The scalar λ is known as the eigenvalue
If v is an eigenvector of A so is any rescaled vector svMoreover sv still has the same eigenvalue Thus weconstrain the eigenvector to be of unit length
||v|| = 1
Linear Algebra Part II 1120
Characteristic Polynomial
Eigenvalue equation of matrix A
Av = λvAv minus λv = 0
(A minus λI)v = 0
If nonzero solution for v exists then it must be the casethat
det(A minus λI) = 0
Unpacking the determinant as a function of λ we get
|A minus λI| = (λ1 minus λ)(λ2 minus λ) (λn minus λ) = 0
The λ1λ2 λn are roots of the characteristicpolynomial and are eigenvalues of A
Linear Algebra Part II 1220
Example
Consider the matrix
A =
9830632 11 2
983064
The characteristic polynomial is
det(A minus λI) = det9830632 minus λ 1
1 2 minus λ
983064= 3 minus 4λ+ λ2 = 0
It has roots λ = 1 and λ = 3 which are the twoeigenvalues of A
We can then solve for eigenvectors using Av = λv
vλ=1 = [1minus1]T and vλ=3 = [1 1]T
Linear Algebra Part II 1320
Eigendecomposition
Suppose that n times n matrix A has n linearly independenteigenvectors v(1) v(n) with eigenvaluesλ1 λn
Concatenate eigenvectors to form matrix V
Concatenate eigenvalues to form vectorλ = [λ1 λn]
T
The eigendecomposition of A is given by
A = Vdiag(λ)Vminus1
Linear Algebra Part II 1420
Symmetric Matrices
Every real symmetric matrix A can be decomposed intoreal-valued eigenvectors and eigenvalues
A = QΛQT
Q is an orthogonal matrix of the eigenvectors of A andΛ is a diagonal matrix of eigenvalues
We can think of A as scaling space by λi in direction v(i)
Linear Algebra Part II 1520
Eigendecomposition is not Unique
Decomposition is not unique when two eigenvalues arethe same
By convention order entries of Λ in descending orderThen eigendecomposition is unique if all eigenvalues areunique
If any eigenvalue is zero then the matrix is singular
Linear Algebra Part II 1620
Positive Definite Matrix
A matrix whose eigenvalues are all positive is calledpositive definite
If eigenvalues are positive or zero then matrix is calledpositive semidefinite
Positive definite matrices guarantee that
xTAx gt 0 for any nonzero vector x
Similarly positive semidefinite guarantees xTAx ge 0
Linear Algebra Part II 1720
Singular Value Decomposition (SVD)
If A is not square eigendecomposition is undefined
SVD is a decomposition of the form
A = UDVT
SVD is more general than eigendecomposition
Every real matrix has a SVD
Linear Algebra Part II 1820
SVD Definition (1)
Write A as a product of three matrices A = UDVT
If A is m times n then U is m timesm D is m times n and V isn times n
U and V are orthogonal matrices and D is a diagonalmatrix (not necessarily square)
Diagonal entries of D are called singular values of A
Columns of U are the left singular vectors andcolumns of V are the right singular vectors
Linear Algebra Part II 1920
SVD Definition (2)
SVD can be interpreted in terms of eigendecompostion
Left singular vectors of A are the eigenvectors of AAT
Right singular vectors of A are the eigenvectors of ATA
Nonzero singular values of A are square roots ofeigenvalues of ATA and AAT
Linear Algebra Part II 2020
![Page 5: Linear Algebra - Part IIrgrosse/courses/csc411_f18/... · 2018-10-01 · 2 1 1 2 & The characteristic polynomial is: det(A−λI) = det % 2−λ 1 1 2−λ & = 3−4λ+λ2 = 0 It](https://reader033.fdocuments.net/reader033/viewer/2022042418/5f33d6055bec6b65e409df57/html5/thumbnails/5.jpg)
Vector Projection Given two vectors a and b let b = b
||b|| be the unit vectorin the direction of b
Then a1 = a1b is the orthogonal projection of a onto astraight line parallel to b where
a1 = ||a|| cos(θ) = a middot b = a middot b||b||
Image taken from wikipedia
Linear Algebra Part II 520
Diagonal Matrix
Diagonal matrix has mostly zeros with non-zero entriesonly in the diagonal eg identity matrix
A square diagonal matrix with diagonal elements given byentries of vector v is denoted
diag(v)
Multiplying vector x by a diagonal matrix is efficient
diag(v)x = v ⊙ x
Inverting a square diagonal matrix is efficient
diag(v)minus1 = diag983059[1v1
1vn]T983060
Linear Algebra Part II 620
Determinant
Determinant of a square matrix is a mapping to a scalar
det(A) or |A|
Measures how much multiplication by the matrix expandsor contracts the space
Determinant of product is the product of determinants
det(AB) = det(A)det(B)
Image taken from wikipedia
Linear Algebra Part II 720
List of Equivalencies
The following are all equivalent
A is invertible ie Aminus1 exists
Ax = b has a unique solution
Columns of A are linearly independent
det(A) ∕= 0
Ax = 0 has a unique trivial solution x = 0
Linear Algebra Part II 820
Zero Determinant
If det(A) = 0 then
A is linearly dependent
Ax = b has no solution or infinitely many solutions
Ax = 0 has a non-zero solution
Linear Algebra Part II 920
Matrix Decomposition
We can decompose an integer into its prime factors eg12 = 2 times 2 times 3
Similarly matrices can be decomposed into factors tolearn universal properties
A = Vdiag(λ)Vminus1
Linear Algebra Part II 1020
Eigenvectors
An eigenvector of a square matrix A is a nonzero vector vsuch that multiplication by A only changes the scale of v
Av = λv
The scalar λ is known as the eigenvalue
If v is an eigenvector of A so is any rescaled vector svMoreover sv still has the same eigenvalue Thus weconstrain the eigenvector to be of unit length
||v|| = 1
Linear Algebra Part II 1120
Characteristic Polynomial
Eigenvalue equation of matrix A
Av = λvAv minus λv = 0
(A minus λI)v = 0
If nonzero solution for v exists then it must be the casethat
det(A minus λI) = 0
Unpacking the determinant as a function of λ we get
|A minus λI| = (λ1 minus λ)(λ2 minus λ) (λn minus λ) = 0
The λ1λ2 λn are roots of the characteristicpolynomial and are eigenvalues of A
Linear Algebra Part II 1220
Example
Consider the matrix
A =
9830632 11 2
983064
The characteristic polynomial is
det(A minus λI) = det9830632 minus λ 1
1 2 minus λ
983064= 3 minus 4λ+ λ2 = 0
It has roots λ = 1 and λ = 3 which are the twoeigenvalues of A
We can then solve for eigenvectors using Av = λv
vλ=1 = [1minus1]T and vλ=3 = [1 1]T
Linear Algebra Part II 1320
Eigendecomposition
Suppose that n times n matrix A has n linearly independenteigenvectors v(1) v(n) with eigenvaluesλ1 λn
Concatenate eigenvectors to form matrix V
Concatenate eigenvalues to form vectorλ = [λ1 λn]
T
The eigendecomposition of A is given by
A = Vdiag(λ)Vminus1
Linear Algebra Part II 1420
Symmetric Matrices
Every real symmetric matrix A can be decomposed intoreal-valued eigenvectors and eigenvalues
A = QΛQT
Q is an orthogonal matrix of the eigenvectors of A andΛ is a diagonal matrix of eigenvalues
We can think of A as scaling space by λi in direction v(i)
Linear Algebra Part II 1520
Eigendecomposition is not Unique
Decomposition is not unique when two eigenvalues arethe same
By convention order entries of Λ in descending orderThen eigendecomposition is unique if all eigenvalues areunique
If any eigenvalue is zero then the matrix is singular
Linear Algebra Part II 1620
Positive Definite Matrix
A matrix whose eigenvalues are all positive is calledpositive definite
If eigenvalues are positive or zero then matrix is calledpositive semidefinite
Positive definite matrices guarantee that
xTAx gt 0 for any nonzero vector x
Similarly positive semidefinite guarantees xTAx ge 0
Linear Algebra Part II 1720
Singular Value Decomposition (SVD)
If A is not square eigendecomposition is undefined
SVD is a decomposition of the form
A = UDVT
SVD is more general than eigendecomposition
Every real matrix has a SVD
Linear Algebra Part II 1820
SVD Definition (1)
Write A as a product of three matrices A = UDVT
If A is m times n then U is m timesm D is m times n and V isn times n
U and V are orthogonal matrices and D is a diagonalmatrix (not necessarily square)
Diagonal entries of D are called singular values of A
Columns of U are the left singular vectors andcolumns of V are the right singular vectors
Linear Algebra Part II 1920
SVD Definition (2)
SVD can be interpreted in terms of eigendecompostion
Left singular vectors of A are the eigenvectors of AAT
Right singular vectors of A are the eigenvectors of ATA
Nonzero singular values of A are square roots ofeigenvalues of ATA and AAT
Linear Algebra Part II 2020
![Page 6: Linear Algebra - Part IIrgrosse/courses/csc411_f18/... · 2018-10-01 · 2 1 1 2 & The characteristic polynomial is: det(A−λI) = det % 2−λ 1 1 2−λ & = 3−4λ+λ2 = 0 It](https://reader033.fdocuments.net/reader033/viewer/2022042418/5f33d6055bec6b65e409df57/html5/thumbnails/6.jpg)
Diagonal Matrix
Diagonal matrix has mostly zeros with non-zero entriesonly in the diagonal eg identity matrix
A square diagonal matrix with diagonal elements given byentries of vector v is denoted
diag(v)
Multiplying vector x by a diagonal matrix is efficient
diag(v)x = v ⊙ x
Inverting a square diagonal matrix is efficient
diag(v)minus1 = diag983059[1v1
1vn]T983060
Linear Algebra Part II 620
Determinant
Determinant of a square matrix is a mapping to a scalar
det(A) or |A|
Measures how much multiplication by the matrix expandsor contracts the space
Determinant of product is the product of determinants
det(AB) = det(A)det(B)
Image taken from wikipedia
Linear Algebra Part II 720
List of Equivalencies
The following are all equivalent
A is invertible ie Aminus1 exists
Ax = b has a unique solution
Columns of A are linearly independent
det(A) ∕= 0
Ax = 0 has a unique trivial solution x = 0
Linear Algebra Part II 820
Zero Determinant
If det(A) = 0 then
A is linearly dependent
Ax = b has no solution or infinitely many solutions
Ax = 0 has a non-zero solution
Linear Algebra Part II 920
Matrix Decomposition
We can decompose an integer into its prime factors eg12 = 2 times 2 times 3
Similarly matrices can be decomposed into factors tolearn universal properties
A = Vdiag(λ)Vminus1
Linear Algebra Part II 1020
Eigenvectors
An eigenvector of a square matrix A is a nonzero vector vsuch that multiplication by A only changes the scale of v
Av = λv
The scalar λ is known as the eigenvalue
If v is an eigenvector of A so is any rescaled vector svMoreover sv still has the same eigenvalue Thus weconstrain the eigenvector to be of unit length
||v|| = 1
Linear Algebra Part II 1120
Characteristic Polynomial
Eigenvalue equation of matrix A
Av = λvAv minus λv = 0
(A minus λI)v = 0
If nonzero solution for v exists then it must be the casethat
det(A minus λI) = 0
Unpacking the determinant as a function of λ we get
|A minus λI| = (λ1 minus λ)(λ2 minus λ) (λn minus λ) = 0
The λ1λ2 λn are roots of the characteristicpolynomial and are eigenvalues of A
Linear Algebra Part II 1220
Example
Consider the matrix
A =
9830632 11 2
983064
The characteristic polynomial is
det(A minus λI) = det9830632 minus λ 1
1 2 minus λ
983064= 3 minus 4λ+ λ2 = 0
It has roots λ = 1 and λ = 3 which are the twoeigenvalues of A
We can then solve for eigenvectors using Av = λv
vλ=1 = [1minus1]T and vλ=3 = [1 1]T
Linear Algebra Part II 1320
Eigendecomposition
Suppose that n times n matrix A has n linearly independenteigenvectors v(1) v(n) with eigenvaluesλ1 λn
Concatenate eigenvectors to form matrix V
Concatenate eigenvalues to form vectorλ = [λ1 λn]
T
The eigendecomposition of A is given by
A = Vdiag(λ)Vminus1
Linear Algebra Part II 1420
Symmetric Matrices
Every real symmetric matrix A can be decomposed intoreal-valued eigenvectors and eigenvalues
A = QΛQT
Q is an orthogonal matrix of the eigenvectors of A andΛ is a diagonal matrix of eigenvalues
We can think of A as scaling space by λi in direction v(i)
Linear Algebra Part II 1520
Eigendecomposition is not Unique
Decomposition is not unique when two eigenvalues arethe same
By convention order entries of Λ in descending orderThen eigendecomposition is unique if all eigenvalues areunique
If any eigenvalue is zero then the matrix is singular
Linear Algebra Part II 1620
Positive Definite Matrix
A matrix whose eigenvalues are all positive is calledpositive definite
If eigenvalues are positive or zero then matrix is calledpositive semidefinite
Positive definite matrices guarantee that
xTAx gt 0 for any nonzero vector x
Similarly positive semidefinite guarantees xTAx ge 0
Linear Algebra Part II 1720
Singular Value Decomposition (SVD)
If A is not square eigendecomposition is undefined
SVD is a decomposition of the form
A = UDVT
SVD is more general than eigendecomposition
Every real matrix has a SVD
Linear Algebra Part II 1820
SVD Definition (1)
Write A as a product of three matrices A = UDVT
If A is m times n then U is m timesm D is m times n and V isn times n
U and V are orthogonal matrices and D is a diagonalmatrix (not necessarily square)
Diagonal entries of D are called singular values of A
Columns of U are the left singular vectors andcolumns of V are the right singular vectors
Linear Algebra Part II 1920
SVD Definition (2)
SVD can be interpreted in terms of eigendecompostion
Left singular vectors of A are the eigenvectors of AAT
Right singular vectors of A are the eigenvectors of ATA
Nonzero singular values of A are square roots ofeigenvalues of ATA and AAT
Linear Algebra Part II 2020
![Page 7: Linear Algebra - Part IIrgrosse/courses/csc411_f18/... · 2018-10-01 · 2 1 1 2 & The characteristic polynomial is: det(A−λI) = det % 2−λ 1 1 2−λ & = 3−4λ+λ2 = 0 It](https://reader033.fdocuments.net/reader033/viewer/2022042418/5f33d6055bec6b65e409df57/html5/thumbnails/7.jpg)
Determinant
Determinant of a square matrix is a mapping to a scalar
det(A) or |A|
Measures how much multiplication by the matrix expandsor contracts the space
Determinant of product is the product of determinants
det(AB) = det(A)det(B)
Image taken from wikipedia
Linear Algebra Part II 720
List of Equivalencies
The following are all equivalent
A is invertible ie Aminus1 exists
Ax = b has a unique solution
Columns of A are linearly independent
det(A) ∕= 0
Ax = 0 has a unique trivial solution x = 0
Linear Algebra Part II 820
Zero Determinant
If det(A) = 0 then
A is linearly dependent
Ax = b has no solution or infinitely many solutions
Ax = 0 has a non-zero solution
Linear Algebra Part II 920
Matrix Decomposition
We can decompose an integer into its prime factors eg12 = 2 times 2 times 3
Similarly matrices can be decomposed into factors tolearn universal properties
A = Vdiag(λ)Vminus1
Linear Algebra Part II 1020
Eigenvectors
An eigenvector of a square matrix A is a nonzero vector vsuch that multiplication by A only changes the scale of v
Av = λv
The scalar λ is known as the eigenvalue
If v is an eigenvector of A so is any rescaled vector svMoreover sv still has the same eigenvalue Thus weconstrain the eigenvector to be of unit length
||v|| = 1
Linear Algebra Part II 1120
Characteristic Polynomial
Eigenvalue equation of matrix A
Av = λvAv minus λv = 0
(A minus λI)v = 0
If nonzero solution for v exists then it must be the casethat
det(A minus λI) = 0
Unpacking the determinant as a function of λ we get
|A minus λI| = (λ1 minus λ)(λ2 minus λ) (λn minus λ) = 0
The λ1λ2 λn are roots of the characteristicpolynomial and are eigenvalues of A
Linear Algebra Part II 1220
Example
Consider the matrix
A =
9830632 11 2
983064
The characteristic polynomial is
det(A minus λI) = det9830632 minus λ 1
1 2 minus λ
983064= 3 minus 4λ+ λ2 = 0
It has roots λ = 1 and λ = 3 which are the twoeigenvalues of A
We can then solve for eigenvectors using Av = λv
vλ=1 = [1minus1]T and vλ=3 = [1 1]T
Linear Algebra Part II 1320
Eigendecomposition
Suppose that n times n matrix A has n linearly independenteigenvectors v(1) v(n) with eigenvaluesλ1 λn
Concatenate eigenvectors to form matrix V
Concatenate eigenvalues to form vectorλ = [λ1 λn]
T
The eigendecomposition of A is given by
A = Vdiag(λ)Vminus1
Linear Algebra Part II 1420
Symmetric Matrices
Every real symmetric matrix A can be decomposed intoreal-valued eigenvectors and eigenvalues
A = QΛQT
Q is an orthogonal matrix of the eigenvectors of A andΛ is a diagonal matrix of eigenvalues
We can think of A as scaling space by λi in direction v(i)
Linear Algebra Part II 1520
Eigendecomposition is not Unique
Decomposition is not unique when two eigenvalues arethe same
By convention order entries of Λ in descending orderThen eigendecomposition is unique if all eigenvalues areunique
If any eigenvalue is zero then the matrix is singular
Linear Algebra Part II 1620
Positive Definite Matrix
A matrix whose eigenvalues are all positive is calledpositive definite
If eigenvalues are positive or zero then matrix is calledpositive semidefinite
Positive definite matrices guarantee that
xTAx gt 0 for any nonzero vector x
Similarly positive semidefinite guarantees xTAx ge 0
Linear Algebra Part II 1720
Singular Value Decomposition (SVD)
If A is not square eigendecomposition is undefined
SVD is a decomposition of the form
A = UDVT
SVD is more general than eigendecomposition
Every real matrix has a SVD
Linear Algebra Part II 1820
SVD Definition (1)
Write A as a product of three matrices A = UDVT
If A is m times n then U is m timesm D is m times n and V isn times n
U and V are orthogonal matrices and D is a diagonalmatrix (not necessarily square)
Diagonal entries of D are called singular values of A
Columns of U are the left singular vectors andcolumns of V are the right singular vectors
Linear Algebra Part II 1920
SVD Definition (2)
SVD can be interpreted in terms of eigendecompostion
Left singular vectors of A are the eigenvectors of AAT
Right singular vectors of A are the eigenvectors of ATA
Nonzero singular values of A are square roots ofeigenvalues of ATA and AAT
Linear Algebra Part II 2020
![Page 8: Linear Algebra - Part IIrgrosse/courses/csc411_f18/... · 2018-10-01 · 2 1 1 2 & The characteristic polynomial is: det(A−λI) = det % 2−λ 1 1 2−λ & = 3−4λ+λ2 = 0 It](https://reader033.fdocuments.net/reader033/viewer/2022042418/5f33d6055bec6b65e409df57/html5/thumbnails/8.jpg)
List of Equivalencies
The following are all equivalent
A is invertible ie Aminus1 exists
Ax = b has a unique solution
Columns of A are linearly independent
det(A) ∕= 0
Ax = 0 has a unique trivial solution x = 0
Linear Algebra Part II 820
Zero Determinant
If det(A) = 0 then
A is linearly dependent
Ax = b has no solution or infinitely many solutions
Ax = 0 has a non-zero solution
Linear Algebra Part II 920
Matrix Decomposition
We can decompose an integer into its prime factors eg12 = 2 times 2 times 3
Similarly matrices can be decomposed into factors tolearn universal properties
A = Vdiag(λ)Vminus1
Linear Algebra Part II 1020
Eigenvectors
An eigenvector of a square matrix A is a nonzero vector vsuch that multiplication by A only changes the scale of v
Av = λv
The scalar λ is known as the eigenvalue
If v is an eigenvector of A so is any rescaled vector svMoreover sv still has the same eigenvalue Thus weconstrain the eigenvector to be of unit length
||v|| = 1
Linear Algebra Part II 1120
Characteristic Polynomial
Eigenvalue equation of matrix A
Av = λvAv minus λv = 0
(A minus λI)v = 0
If nonzero solution for v exists then it must be the casethat
det(A minus λI) = 0
Unpacking the determinant as a function of λ we get
|A minus λI| = (λ1 minus λ)(λ2 minus λ) (λn minus λ) = 0
The λ1λ2 λn are roots of the characteristicpolynomial and are eigenvalues of A
Linear Algebra Part II 1220
Example
Consider the matrix
A =
9830632 11 2
983064
The characteristic polynomial is
det(A minus λI) = det9830632 minus λ 1
1 2 minus λ
983064= 3 minus 4λ+ λ2 = 0
It has roots λ = 1 and λ = 3 which are the twoeigenvalues of A
We can then solve for eigenvectors using Av = λv
vλ=1 = [1minus1]T and vλ=3 = [1 1]T
Linear Algebra Part II 1320
Eigendecomposition
Suppose that n times n matrix A has n linearly independenteigenvectors v(1) v(n) with eigenvaluesλ1 λn
Concatenate eigenvectors to form matrix V
Concatenate eigenvalues to form vectorλ = [λ1 λn]
T
The eigendecomposition of A is given by
A = Vdiag(λ)Vminus1
Linear Algebra Part II 1420
Symmetric Matrices
Every real symmetric matrix A can be decomposed intoreal-valued eigenvectors and eigenvalues
A = QΛQT
Q is an orthogonal matrix of the eigenvectors of A andΛ is a diagonal matrix of eigenvalues
We can think of A as scaling space by λi in direction v(i)
Linear Algebra Part II 1520
Eigendecomposition is not Unique
Decomposition is not unique when two eigenvalues arethe same
By convention order entries of Λ in descending orderThen eigendecomposition is unique if all eigenvalues areunique
If any eigenvalue is zero then the matrix is singular
Linear Algebra Part II 1620
Positive Definite Matrix
A matrix whose eigenvalues are all positive is calledpositive definite
If eigenvalues are positive or zero then matrix is calledpositive semidefinite
Positive definite matrices guarantee that
xTAx gt 0 for any nonzero vector x
Similarly positive semidefinite guarantees xTAx ge 0
Linear Algebra Part II 1720
Singular Value Decomposition (SVD)
If A is not square eigendecomposition is undefined
SVD is a decomposition of the form
A = UDVT
SVD is more general than eigendecomposition
Every real matrix has a SVD
Linear Algebra Part II 1820
SVD Definition (1)
Write A as a product of three matrices A = UDVT
If A is m times n then U is m timesm D is m times n and V isn times n
U and V are orthogonal matrices and D is a diagonalmatrix (not necessarily square)
Diagonal entries of D are called singular values of A
Columns of U are the left singular vectors andcolumns of V are the right singular vectors
Linear Algebra Part II 1920
SVD Definition (2)
SVD can be interpreted in terms of eigendecompostion
Left singular vectors of A are the eigenvectors of AAT
Right singular vectors of A are the eigenvectors of ATA
Nonzero singular values of A are square roots ofeigenvalues of ATA and AAT
Linear Algebra Part II 2020
![Page 9: Linear Algebra - Part IIrgrosse/courses/csc411_f18/... · 2018-10-01 · 2 1 1 2 & The characteristic polynomial is: det(A−λI) = det % 2−λ 1 1 2−λ & = 3−4λ+λ2 = 0 It](https://reader033.fdocuments.net/reader033/viewer/2022042418/5f33d6055bec6b65e409df57/html5/thumbnails/9.jpg)
Zero Determinant
If det(A) = 0 then
A is linearly dependent
Ax = b has no solution or infinitely many solutions
Ax = 0 has a non-zero solution
Linear Algebra Part II 920
Matrix Decomposition
We can decompose an integer into its prime factors eg12 = 2 times 2 times 3
Similarly matrices can be decomposed into factors tolearn universal properties
A = Vdiag(λ)Vminus1
Linear Algebra Part II 1020
Eigenvectors
An eigenvector of a square matrix A is a nonzero vector vsuch that multiplication by A only changes the scale of v
Av = λv
The scalar λ is known as the eigenvalue
If v is an eigenvector of A so is any rescaled vector svMoreover sv still has the same eigenvalue Thus weconstrain the eigenvector to be of unit length
||v|| = 1
Linear Algebra Part II 1120
Characteristic Polynomial
Eigenvalue equation of matrix A
Av = λvAv minus λv = 0
(A minus λI)v = 0
If nonzero solution for v exists then it must be the casethat
det(A minus λI) = 0
Unpacking the determinant as a function of λ we get
|A minus λI| = (λ1 minus λ)(λ2 minus λ) (λn minus λ) = 0
The λ1λ2 λn are roots of the characteristicpolynomial and are eigenvalues of A
Linear Algebra Part II 1220
Example
Consider the matrix
A =
9830632 11 2
983064
The characteristic polynomial is
det(A minus λI) = det9830632 minus λ 1
1 2 minus λ
983064= 3 minus 4λ+ λ2 = 0
It has roots λ = 1 and λ = 3 which are the twoeigenvalues of A
We can then solve for eigenvectors using Av = λv
vλ=1 = [1minus1]T and vλ=3 = [1 1]T
Linear Algebra Part II 1320
Eigendecomposition
Suppose that n times n matrix A has n linearly independenteigenvectors v(1) v(n) with eigenvaluesλ1 λn
Concatenate eigenvectors to form matrix V
Concatenate eigenvalues to form vectorλ = [λ1 λn]
T
The eigendecomposition of A is given by
A = Vdiag(λ)Vminus1
Linear Algebra Part II 1420
Symmetric Matrices
Every real symmetric matrix A can be decomposed intoreal-valued eigenvectors and eigenvalues
A = QΛQT
Q is an orthogonal matrix of the eigenvectors of A andΛ is a diagonal matrix of eigenvalues
We can think of A as scaling space by λi in direction v(i)
Linear Algebra Part II 1520
Eigendecomposition is not Unique
Decomposition is not unique when two eigenvalues arethe same
By convention order entries of Λ in descending orderThen eigendecomposition is unique if all eigenvalues areunique
If any eigenvalue is zero then the matrix is singular
Linear Algebra Part II 1620
Positive Definite Matrix
A matrix whose eigenvalues are all positive is calledpositive definite
If eigenvalues are positive or zero then matrix is calledpositive semidefinite
Positive definite matrices guarantee that
xTAx gt 0 for any nonzero vector x
Similarly positive semidefinite guarantees xTAx ge 0
Linear Algebra Part II 1720
Singular Value Decomposition (SVD)
If A is not square eigendecomposition is undefined
SVD is a decomposition of the form
A = UDVT
SVD is more general than eigendecomposition
Every real matrix has a SVD
Linear Algebra Part II 1820
SVD Definition (1)
Write A as a product of three matrices A = UDVT
If A is m times n then U is m timesm D is m times n and V isn times n
U and V are orthogonal matrices and D is a diagonalmatrix (not necessarily square)
Diagonal entries of D are called singular values of A
Columns of U are the left singular vectors andcolumns of V are the right singular vectors
Linear Algebra Part II 1920
SVD Definition (2)
SVD can be interpreted in terms of eigendecompostion
Left singular vectors of A are the eigenvectors of AAT
Right singular vectors of A are the eigenvectors of ATA
Nonzero singular values of A are square roots ofeigenvalues of ATA and AAT
Linear Algebra Part II 2020
![Page 10: Linear Algebra - Part IIrgrosse/courses/csc411_f18/... · 2018-10-01 · 2 1 1 2 & The characteristic polynomial is: det(A−λI) = det % 2−λ 1 1 2−λ & = 3−4λ+λ2 = 0 It](https://reader033.fdocuments.net/reader033/viewer/2022042418/5f33d6055bec6b65e409df57/html5/thumbnails/10.jpg)
Matrix Decomposition
We can decompose an integer into its prime factors eg12 = 2 times 2 times 3
Similarly matrices can be decomposed into factors tolearn universal properties
A = Vdiag(λ)Vminus1
Linear Algebra Part II 1020
Eigenvectors
An eigenvector of a square matrix A is a nonzero vector vsuch that multiplication by A only changes the scale of v
Av = λv
The scalar λ is known as the eigenvalue
If v is an eigenvector of A so is any rescaled vector svMoreover sv still has the same eigenvalue Thus weconstrain the eigenvector to be of unit length
||v|| = 1
Linear Algebra Part II 1120
Characteristic Polynomial
Eigenvalue equation of matrix A
Av = λvAv minus λv = 0
(A minus λI)v = 0
If nonzero solution for v exists then it must be the casethat
det(A minus λI) = 0
Unpacking the determinant as a function of λ we get
|A minus λI| = (λ1 minus λ)(λ2 minus λ) (λn minus λ) = 0
The λ1λ2 λn are roots of the characteristicpolynomial and are eigenvalues of A
Linear Algebra Part II 1220
Example
Consider the matrix
A =
9830632 11 2
983064
The characteristic polynomial is
det(A minus λI) = det9830632 minus λ 1
1 2 minus λ
983064= 3 minus 4λ+ λ2 = 0
It has roots λ = 1 and λ = 3 which are the twoeigenvalues of A
We can then solve for eigenvectors using Av = λv
vλ=1 = [1minus1]T and vλ=3 = [1 1]T
Linear Algebra Part II 1320
Eigendecomposition
Suppose that n times n matrix A has n linearly independenteigenvectors v(1) v(n) with eigenvaluesλ1 λn
Concatenate eigenvectors to form matrix V
Concatenate eigenvalues to form vectorλ = [λ1 λn]
T
The eigendecomposition of A is given by
A = Vdiag(λ)Vminus1
Linear Algebra Part II 1420
Symmetric Matrices
Every real symmetric matrix A can be decomposed intoreal-valued eigenvectors and eigenvalues
A = QΛQT
Q is an orthogonal matrix of the eigenvectors of A andΛ is a diagonal matrix of eigenvalues
We can think of A as scaling space by λi in direction v(i)
Linear Algebra Part II 1520
Eigendecomposition is not Unique
Decomposition is not unique when two eigenvalues arethe same
By convention order entries of Λ in descending orderThen eigendecomposition is unique if all eigenvalues areunique
If any eigenvalue is zero then the matrix is singular
Linear Algebra Part II 1620
Positive Definite Matrix
A matrix whose eigenvalues are all positive is calledpositive definite
If eigenvalues are positive or zero then matrix is calledpositive semidefinite
Positive definite matrices guarantee that
xTAx gt 0 for any nonzero vector x
Similarly positive semidefinite guarantees xTAx ge 0
Linear Algebra Part II 1720
Singular Value Decomposition (SVD)
If A is not square eigendecomposition is undefined
SVD is a decomposition of the form
A = UDVT
SVD is more general than eigendecomposition
Every real matrix has a SVD
Linear Algebra Part II 1820
SVD Definition (1)
Write A as a product of three matrices A = UDVT
If A is m times n then U is m timesm D is m times n and V isn times n
U and V are orthogonal matrices and D is a diagonalmatrix (not necessarily square)
Diagonal entries of D are called singular values of A
Columns of U are the left singular vectors andcolumns of V are the right singular vectors
Linear Algebra Part II 1920
SVD Definition (2)
SVD can be interpreted in terms of eigendecompostion
Left singular vectors of A are the eigenvectors of AAT
Right singular vectors of A are the eigenvectors of ATA
Nonzero singular values of A are square roots ofeigenvalues of ATA and AAT
Linear Algebra Part II 2020
![Page 11: Linear Algebra - Part IIrgrosse/courses/csc411_f18/... · 2018-10-01 · 2 1 1 2 & The characteristic polynomial is: det(A−λI) = det % 2−λ 1 1 2−λ & = 3−4λ+λ2 = 0 It](https://reader033.fdocuments.net/reader033/viewer/2022042418/5f33d6055bec6b65e409df57/html5/thumbnails/11.jpg)
Eigenvectors
An eigenvector of a square matrix A is a nonzero vector vsuch that multiplication by A only changes the scale of v
Av = λv
The scalar λ is known as the eigenvalue
If v is an eigenvector of A so is any rescaled vector svMoreover sv still has the same eigenvalue Thus weconstrain the eigenvector to be of unit length
||v|| = 1
Linear Algebra Part II 1120
Characteristic Polynomial
Eigenvalue equation of matrix A
Av = λvAv minus λv = 0
(A minus λI)v = 0
If nonzero solution for v exists then it must be the casethat
det(A minus λI) = 0
Unpacking the determinant as a function of λ we get
|A minus λI| = (λ1 minus λ)(λ2 minus λ) (λn minus λ) = 0
The λ1λ2 λn are roots of the characteristicpolynomial and are eigenvalues of A
Linear Algebra Part II 1220
Example
Consider the matrix
A =
9830632 11 2
983064
The characteristic polynomial is
det(A minus λI) = det9830632 minus λ 1
1 2 minus λ
983064= 3 minus 4λ+ λ2 = 0
It has roots λ = 1 and λ = 3 which are the twoeigenvalues of A
We can then solve for eigenvectors using Av = λv
vλ=1 = [1minus1]T and vλ=3 = [1 1]T
Linear Algebra Part II 1320
Eigendecomposition
Suppose that n times n matrix A has n linearly independenteigenvectors v(1) v(n) with eigenvaluesλ1 λn
Concatenate eigenvectors to form matrix V
Concatenate eigenvalues to form vectorλ = [λ1 λn]
T
The eigendecomposition of A is given by
A = Vdiag(λ)Vminus1
Linear Algebra Part II 1420
Symmetric Matrices
Every real symmetric matrix A can be decomposed intoreal-valued eigenvectors and eigenvalues
A = QΛQT
Q is an orthogonal matrix of the eigenvectors of A andΛ is a diagonal matrix of eigenvalues
We can think of A as scaling space by λi in direction v(i)
Linear Algebra Part II 1520
Eigendecomposition is not Unique
Decomposition is not unique when two eigenvalues arethe same
By convention order entries of Λ in descending orderThen eigendecomposition is unique if all eigenvalues areunique
If any eigenvalue is zero then the matrix is singular
Linear Algebra Part II 1620
Positive Definite Matrix
A matrix whose eigenvalues are all positive is calledpositive definite
If eigenvalues are positive or zero then matrix is calledpositive semidefinite
Positive definite matrices guarantee that
xTAx gt 0 for any nonzero vector x
Similarly positive semidefinite guarantees xTAx ge 0
Linear Algebra Part II 1720
Singular Value Decomposition (SVD)
If A is not square eigendecomposition is undefined
SVD is a decomposition of the form
A = UDVT
SVD is more general than eigendecomposition
Every real matrix has a SVD
Linear Algebra Part II 1820
SVD Definition (1)
Write A as a product of three matrices A = UDVT
If A is m times n then U is m timesm D is m times n and V isn times n
U and V are orthogonal matrices and D is a diagonalmatrix (not necessarily square)
Diagonal entries of D are called singular values of A
Columns of U are the left singular vectors andcolumns of V are the right singular vectors
Linear Algebra Part II 1920
SVD Definition (2)
SVD can be interpreted in terms of eigendecompostion
Left singular vectors of A are the eigenvectors of AAT
Right singular vectors of A are the eigenvectors of ATA
Nonzero singular values of A are square roots ofeigenvalues of ATA and AAT
Linear Algebra Part II 2020
![Page 12: Linear Algebra - Part IIrgrosse/courses/csc411_f18/... · 2018-10-01 · 2 1 1 2 & The characteristic polynomial is: det(A−λI) = det % 2−λ 1 1 2−λ & = 3−4λ+λ2 = 0 It](https://reader033.fdocuments.net/reader033/viewer/2022042418/5f33d6055bec6b65e409df57/html5/thumbnails/12.jpg)
Characteristic Polynomial
Eigenvalue equation of matrix A
Av = λvAv minus λv = 0
(A minus λI)v = 0
If nonzero solution for v exists then it must be the casethat
det(A minus λI) = 0
Unpacking the determinant as a function of λ we get
|A minus λI| = (λ1 minus λ)(λ2 minus λ) (λn minus λ) = 0
The λ1λ2 λn are roots of the characteristicpolynomial and are eigenvalues of A
Linear Algebra Part II 1220
Example
Consider the matrix
A =
9830632 11 2
983064
The characteristic polynomial is
det(A minus λI) = det9830632 minus λ 1
1 2 minus λ
983064= 3 minus 4λ+ λ2 = 0
It has roots λ = 1 and λ = 3 which are the twoeigenvalues of A
We can then solve for eigenvectors using Av = λv
vλ=1 = [1minus1]T and vλ=3 = [1 1]T
Linear Algebra Part II 1320
Eigendecomposition
Suppose that n times n matrix A has n linearly independenteigenvectors v(1) v(n) with eigenvaluesλ1 λn
Concatenate eigenvectors to form matrix V
Concatenate eigenvalues to form vectorλ = [λ1 λn]
T
The eigendecomposition of A is given by
A = Vdiag(λ)Vminus1
Linear Algebra Part II 1420
Symmetric Matrices
Every real symmetric matrix A can be decomposed intoreal-valued eigenvectors and eigenvalues
A = QΛQT
Q is an orthogonal matrix of the eigenvectors of A andΛ is a diagonal matrix of eigenvalues
We can think of A as scaling space by λi in direction v(i)
Linear Algebra Part II 1520
Eigendecomposition is not Unique
Decomposition is not unique when two eigenvalues arethe same
By convention order entries of Λ in descending orderThen eigendecomposition is unique if all eigenvalues areunique
If any eigenvalue is zero then the matrix is singular
Linear Algebra Part II 1620
Positive Definite Matrix
A matrix whose eigenvalues are all positive is calledpositive definite
If eigenvalues are positive or zero then matrix is calledpositive semidefinite
Positive definite matrices guarantee that
xTAx gt 0 for any nonzero vector x
Similarly positive semidefinite guarantees xTAx ge 0
Linear Algebra Part II 1720
Singular Value Decomposition (SVD)
If A is not square eigendecomposition is undefined
SVD is a decomposition of the form
A = UDVT
SVD is more general than eigendecomposition
Every real matrix has a SVD
Linear Algebra Part II 1820
SVD Definition (1)
Write A as a product of three matrices A = UDVT
If A is m times n then U is m timesm D is m times n and V isn times n
U and V are orthogonal matrices and D is a diagonalmatrix (not necessarily square)
Diagonal entries of D are called singular values of A
Columns of U are the left singular vectors andcolumns of V are the right singular vectors
Linear Algebra Part II 1920
SVD Definition (2)
SVD can be interpreted in terms of eigendecompostion
Left singular vectors of A are the eigenvectors of AAT
Right singular vectors of A are the eigenvectors of ATA
Nonzero singular values of A are square roots ofeigenvalues of ATA and AAT
Linear Algebra Part II 2020
![Page 13: Linear Algebra - Part IIrgrosse/courses/csc411_f18/... · 2018-10-01 · 2 1 1 2 & The characteristic polynomial is: det(A−λI) = det % 2−λ 1 1 2−λ & = 3−4λ+λ2 = 0 It](https://reader033.fdocuments.net/reader033/viewer/2022042418/5f33d6055bec6b65e409df57/html5/thumbnails/13.jpg)
Example
Consider the matrix
A =
9830632 11 2
983064
The characteristic polynomial is
det(A minus λI) = det9830632 minus λ 1
1 2 minus λ
983064= 3 minus 4λ+ λ2 = 0
It has roots λ = 1 and λ = 3 which are the twoeigenvalues of A
We can then solve for eigenvectors using Av = λv
vλ=1 = [1minus1]T and vλ=3 = [1 1]T
Linear Algebra Part II 1320
Eigendecomposition
Suppose that n times n matrix A has n linearly independenteigenvectors v(1) v(n) with eigenvaluesλ1 λn
Concatenate eigenvectors to form matrix V
Concatenate eigenvalues to form vectorλ = [λ1 λn]
T
The eigendecomposition of A is given by
A = Vdiag(λ)Vminus1
Linear Algebra Part II 1420
Symmetric Matrices
Every real symmetric matrix A can be decomposed intoreal-valued eigenvectors and eigenvalues
A = QΛQT
Q is an orthogonal matrix of the eigenvectors of A andΛ is a diagonal matrix of eigenvalues
We can think of A as scaling space by λi in direction v(i)
Linear Algebra Part II 1520
Eigendecomposition is not Unique
Decomposition is not unique when two eigenvalues arethe same
By convention order entries of Λ in descending orderThen eigendecomposition is unique if all eigenvalues areunique
If any eigenvalue is zero then the matrix is singular
Linear Algebra Part II 1620
Positive Definite Matrix
A matrix whose eigenvalues are all positive is calledpositive definite
If eigenvalues are positive or zero then matrix is calledpositive semidefinite
Positive definite matrices guarantee that
xTAx gt 0 for any nonzero vector x
Similarly positive semidefinite guarantees xTAx ge 0
Linear Algebra Part II 1720
Singular Value Decomposition (SVD)
If A is not square eigendecomposition is undefined
SVD is a decomposition of the form
A = UDVT
SVD is more general than eigendecomposition
Every real matrix has a SVD
Linear Algebra Part II 1820
SVD Definition (1)
Write A as a product of three matrices A = UDVT
If A is m times n then U is m timesm D is m times n and V isn times n
U and V are orthogonal matrices and D is a diagonalmatrix (not necessarily square)
Diagonal entries of D are called singular values of A
Columns of U are the left singular vectors andcolumns of V are the right singular vectors
Linear Algebra Part II 1920
SVD Definition (2)
SVD can be interpreted in terms of eigendecompostion
Left singular vectors of A are the eigenvectors of AAT
Right singular vectors of A are the eigenvectors of ATA
Nonzero singular values of A are square roots ofeigenvalues of ATA and AAT
Linear Algebra Part II 2020
![Page 14: Linear Algebra - Part IIrgrosse/courses/csc411_f18/... · 2018-10-01 · 2 1 1 2 & The characteristic polynomial is: det(A−λI) = det % 2−λ 1 1 2−λ & = 3−4λ+λ2 = 0 It](https://reader033.fdocuments.net/reader033/viewer/2022042418/5f33d6055bec6b65e409df57/html5/thumbnails/14.jpg)
Eigendecomposition
Suppose that n times n matrix A has n linearly independenteigenvectors v(1) v(n) with eigenvaluesλ1 λn
Concatenate eigenvectors to form matrix V
Concatenate eigenvalues to form vectorλ = [λ1 λn]
T
The eigendecomposition of A is given by
A = Vdiag(λ)Vminus1
Linear Algebra Part II 1420
Symmetric Matrices
Every real symmetric matrix A can be decomposed intoreal-valued eigenvectors and eigenvalues
A = QΛQT
Q is an orthogonal matrix of the eigenvectors of A andΛ is a diagonal matrix of eigenvalues
We can think of A as scaling space by λi in direction v(i)
Linear Algebra Part II 1520
Eigendecomposition is not Unique
Decomposition is not unique when two eigenvalues arethe same
By convention order entries of Λ in descending orderThen eigendecomposition is unique if all eigenvalues areunique
If any eigenvalue is zero then the matrix is singular
Linear Algebra Part II 1620
Positive Definite Matrix
A matrix whose eigenvalues are all positive is calledpositive definite
If eigenvalues are positive or zero then matrix is calledpositive semidefinite
Positive definite matrices guarantee that
xTAx gt 0 for any nonzero vector x
Similarly positive semidefinite guarantees xTAx ge 0
Linear Algebra Part II 1720
Singular Value Decomposition (SVD)
If A is not square eigendecomposition is undefined
SVD is a decomposition of the form
A = UDVT
SVD is more general than eigendecomposition
Every real matrix has a SVD
Linear Algebra Part II 1820
SVD Definition (1)
Write A as a product of three matrices A = UDVT
If A is m times n then U is m timesm D is m times n and V isn times n
U and V are orthogonal matrices and D is a diagonalmatrix (not necessarily square)
Diagonal entries of D are called singular values of A
Columns of U are the left singular vectors andcolumns of V are the right singular vectors
Linear Algebra Part II 1920
SVD Definition (2)
SVD can be interpreted in terms of eigendecompostion
Left singular vectors of A are the eigenvectors of AAT
Right singular vectors of A are the eigenvectors of ATA
Nonzero singular values of A are square roots ofeigenvalues of ATA and AAT
Linear Algebra Part II 2020
![Page 15: Linear Algebra - Part IIrgrosse/courses/csc411_f18/... · 2018-10-01 · 2 1 1 2 & The characteristic polynomial is: det(A−λI) = det % 2−λ 1 1 2−λ & = 3−4λ+λ2 = 0 It](https://reader033.fdocuments.net/reader033/viewer/2022042418/5f33d6055bec6b65e409df57/html5/thumbnails/15.jpg)
Symmetric Matrices
Every real symmetric matrix A can be decomposed intoreal-valued eigenvectors and eigenvalues
A = QΛQT
Q is an orthogonal matrix of the eigenvectors of A andΛ is a diagonal matrix of eigenvalues
We can think of A as scaling space by λi in direction v(i)
Linear Algebra Part II 1520
Eigendecomposition is not Unique
Decomposition is not unique when two eigenvalues arethe same
By convention order entries of Λ in descending orderThen eigendecomposition is unique if all eigenvalues areunique
If any eigenvalue is zero then the matrix is singular
Linear Algebra Part II 1620
Positive Definite Matrix
A matrix whose eigenvalues are all positive is calledpositive definite
If eigenvalues are positive or zero then matrix is calledpositive semidefinite
Positive definite matrices guarantee that
xTAx gt 0 for any nonzero vector x
Similarly positive semidefinite guarantees xTAx ge 0
Linear Algebra Part II 1720
Singular Value Decomposition (SVD)
If A is not square eigendecomposition is undefined
SVD is a decomposition of the form
A = UDVT
SVD is more general than eigendecomposition
Every real matrix has a SVD
Linear Algebra Part II 1820
SVD Definition (1)
Write A as a product of three matrices A = UDVT
If A is m times n then U is m timesm D is m times n and V isn times n
U and V are orthogonal matrices and D is a diagonalmatrix (not necessarily square)
Diagonal entries of D are called singular values of A
Columns of U are the left singular vectors andcolumns of V are the right singular vectors
Linear Algebra Part II 1920
SVD Definition (2)
SVD can be interpreted in terms of eigendecompostion
Left singular vectors of A are the eigenvectors of AAT
Right singular vectors of A are the eigenvectors of ATA
Nonzero singular values of A are square roots ofeigenvalues of ATA and AAT
Linear Algebra Part II 2020
![Page 16: Linear Algebra - Part IIrgrosse/courses/csc411_f18/... · 2018-10-01 · 2 1 1 2 & The characteristic polynomial is: det(A−λI) = det % 2−λ 1 1 2−λ & = 3−4λ+λ2 = 0 It](https://reader033.fdocuments.net/reader033/viewer/2022042418/5f33d6055bec6b65e409df57/html5/thumbnails/16.jpg)
Eigendecomposition is not Unique
Decomposition is not unique when two eigenvalues arethe same
By convention order entries of Λ in descending orderThen eigendecomposition is unique if all eigenvalues areunique
If any eigenvalue is zero then the matrix is singular
Linear Algebra Part II 1620
Positive Definite Matrix
A matrix whose eigenvalues are all positive is calledpositive definite
If eigenvalues are positive or zero then matrix is calledpositive semidefinite
Positive definite matrices guarantee that
xTAx gt 0 for any nonzero vector x
Similarly positive semidefinite guarantees xTAx ge 0
Linear Algebra Part II 1720
Singular Value Decomposition (SVD)
If A is not square eigendecomposition is undefined
SVD is a decomposition of the form
A = UDVT
SVD is more general than eigendecomposition
Every real matrix has a SVD
Linear Algebra Part II 1820
SVD Definition (1)
Write A as a product of three matrices A = UDVT
If A is m times n then U is m timesm D is m times n and V isn times n
U and V are orthogonal matrices and D is a diagonalmatrix (not necessarily square)
Diagonal entries of D are called singular values of A
Columns of U are the left singular vectors andcolumns of V are the right singular vectors
Linear Algebra Part II 1920
SVD Definition (2)
SVD can be interpreted in terms of eigendecompostion
Left singular vectors of A are the eigenvectors of AAT
Right singular vectors of A are the eigenvectors of ATA
Nonzero singular values of A are square roots ofeigenvalues of ATA and AAT
Linear Algebra Part II 2020
![Page 17: Linear Algebra - Part IIrgrosse/courses/csc411_f18/... · 2018-10-01 · 2 1 1 2 & The characteristic polynomial is: det(A−λI) = det % 2−λ 1 1 2−λ & = 3−4λ+λ2 = 0 It](https://reader033.fdocuments.net/reader033/viewer/2022042418/5f33d6055bec6b65e409df57/html5/thumbnails/17.jpg)
Positive Definite Matrix
A matrix whose eigenvalues are all positive is calledpositive definite
If eigenvalues are positive or zero then matrix is calledpositive semidefinite
Positive definite matrices guarantee that
xTAx gt 0 for any nonzero vector x
Similarly positive semidefinite guarantees xTAx ge 0
Linear Algebra Part II 1720
Singular Value Decomposition (SVD)
If A is not square eigendecomposition is undefined
SVD is a decomposition of the form
A = UDVT
SVD is more general than eigendecomposition
Every real matrix has a SVD
Linear Algebra Part II 1820
SVD Definition (1)
Write A as a product of three matrices A = UDVT
If A is m times n then U is m timesm D is m times n and V isn times n
U and V are orthogonal matrices and D is a diagonalmatrix (not necessarily square)
Diagonal entries of D are called singular values of A
Columns of U are the left singular vectors andcolumns of V are the right singular vectors
Linear Algebra Part II 1920
SVD Definition (2)
SVD can be interpreted in terms of eigendecompostion
Left singular vectors of A are the eigenvectors of AAT
Right singular vectors of A are the eigenvectors of ATA
Nonzero singular values of A are square roots ofeigenvalues of ATA and AAT
Linear Algebra Part II 2020
![Page 18: Linear Algebra - Part IIrgrosse/courses/csc411_f18/... · 2018-10-01 · 2 1 1 2 & The characteristic polynomial is: det(A−λI) = det % 2−λ 1 1 2−λ & = 3−4λ+λ2 = 0 It](https://reader033.fdocuments.net/reader033/viewer/2022042418/5f33d6055bec6b65e409df57/html5/thumbnails/18.jpg)
Singular Value Decomposition (SVD)
If A is not square eigendecomposition is undefined
SVD is a decomposition of the form
A = UDVT
SVD is more general than eigendecomposition
Every real matrix has a SVD
Linear Algebra Part II 1820
SVD Definition (1)
Write A as a product of three matrices A = UDVT
If A is m times n then U is m timesm D is m times n and V isn times n
U and V are orthogonal matrices and D is a diagonalmatrix (not necessarily square)
Diagonal entries of D are called singular values of A
Columns of U are the left singular vectors andcolumns of V are the right singular vectors
Linear Algebra Part II 1920
SVD Definition (2)
SVD can be interpreted in terms of eigendecompostion
Left singular vectors of A are the eigenvectors of AAT
Right singular vectors of A are the eigenvectors of ATA
Nonzero singular values of A are square roots ofeigenvalues of ATA and AAT
Linear Algebra Part II 2020
![Page 19: Linear Algebra - Part IIrgrosse/courses/csc411_f18/... · 2018-10-01 · 2 1 1 2 & The characteristic polynomial is: det(A−λI) = det % 2−λ 1 1 2−λ & = 3−4λ+λ2 = 0 It](https://reader033.fdocuments.net/reader033/viewer/2022042418/5f33d6055bec6b65e409df57/html5/thumbnails/19.jpg)
SVD Definition (1)
Write A as a product of three matrices A = UDVT
If A is m times n then U is m timesm D is m times n and V isn times n
U and V are orthogonal matrices and D is a diagonalmatrix (not necessarily square)
Diagonal entries of D are called singular values of A
Columns of U are the left singular vectors andcolumns of V are the right singular vectors
Linear Algebra Part II 1920
SVD Definition (2)
SVD can be interpreted in terms of eigendecompostion
Left singular vectors of A are the eigenvectors of AAT
Right singular vectors of A are the eigenvectors of ATA
Nonzero singular values of A are square roots ofeigenvalues of ATA and AAT
Linear Algebra Part II 2020
![Page 20: Linear Algebra - Part IIrgrosse/courses/csc411_f18/... · 2018-10-01 · 2 1 1 2 & The characteristic polynomial is: det(A−λI) = det % 2−λ 1 1 2−λ & = 3−4λ+λ2 = 0 It](https://reader033.fdocuments.net/reader033/viewer/2022042418/5f33d6055bec6b65e409df57/html5/thumbnails/20.jpg)
SVD Definition (2)
SVD can be interpreted in terms of eigendecompostion
Left singular vectors of A are the eigenvectors of AAT
Right singular vectors of A are the eigenvectors of ATA
Nonzero singular values of A are square roots ofeigenvalues of ATA and AAT
Linear Algebra Part II 2020