Linear Algebra and Probability Theory

8/3/2019 Linear Algebra and Probability Theory

http://slidepdf.com/reader/full/linear-algebra-and-probability-theory 1/47

Quick Tour of Basic Linear Algebra and Probability Theory

Quick Tour of Basic Linear Algebra and

Probability Theory

CS246: Mining Massive Data Sets

Winter 2011

Basic Linear Algebra

Outline

1 Basic Linear Algebra

2 Basic Probability Theory

Matrices and Vectors

Matrix: A rectangular array of numbers, e.g., A ∈ Rm ×n :

a 11 a 12 . . . a 1n a 21 a 22 . . . a 2n

... ... ... ...

a m 1 a m 2 . . . a mn

Matrices and Vectors

Matrix: A rectangular array of numbers, e.g., A ∈ Rm ×n :

a 11 a 12 . . . a 1n a 21 a 22 . . . a 2n

... ... ... ...

a m 1 a m 2 . . . a mn

Vector: A matrix consisting of only one column (default) or

one row, e.g., x ∈Rn

x 1x 2...

Matrix Multiplication

If A ∈ Rm ×n , B ∈ Rn ×p , C = AB , then C ∈ Rm ×p :

C ij =

n k =1

Aik B kj

Matrix Multiplication

If A ∈ Rm ×n , B ∈ Rn ×p , C = AB , then C ∈ Rm ×p :

C ij =

n k =1

Aik B kj

Special cases: Matrix-vector product, inner product of two

vectors. e.g., with x , y

∈Rn :

x T y =n

x i y i ∈ R

Q i k T f B i Li Al b d P b bili Th

Properties of Matrix Multiplication

Associative: (AB )C = A(BC )

Q i k T f B i Li Al b d P b bilit Th

Distributive: A(B + C ) = AB + AC

Non-commutative: AB = BA

Block multiplication: If A = [Aik ], B = [B kj ], where Aik ’s and

B kj ’s are matrix blocks, and the number of columns in Aik is

equal to the number of rows in B kj , then C = AB = [C ij ]where C ij =

k Aik B kj

Block multiplication: If A = [Aik ], B = [B kj ], where Aik ’s and

B kj ’s are matrix blocks, and the number of columns in Aik is

equal to the number of rows in B kj , then C = AB = [C ij ]where C ij =

k Aik B kj

Example: If−→x ∈ Rn and A = [

−→a 1| −→a 2| . . . | −→a n ] ∈ Rm ×n ,

B = [−→b 1| −→b 2| . . . | −→b p ] ∈ Rn ×p

A−→x =

n i =1

x i −→a i

AB = [A−→b 1|A−→b 2| . . . |A−→b p ]

Operators and properties

Transpose: A ∈ Rm ×n , then AT ∈ Rn ×m : (AT )ij = A ji

Transpose: A ∈ Rm ×n , then AT ∈ Rn ×m : (AT )ij = A ji Properties:

= A(AB )T = B T AT

(A + B )T = AT + B T

= A(AB )T = B T AT

(A + B )T = AT + B T

Trace: A ∈ Rn ×n , then: tr (A) =n

i =1 Aii

= A(AB )T = B T AT

(A + B )T = AT + B T

Trace: A ∈ Rn ×n , then: tr (A) =n

i =1 Aii

Properties:

tr (A) = tr (AT )tr (A + B ) = tr (A) + tr (B )tr (λA) = λtr (A)If AB is a square matrix, tr (AB ) = tr (BA)

Special types of matrices

Identity matrix: I = I n ∈ Rn ×n :

I ij =

1 i=j,

0 otherwise.

∀A ∈ Rm ×n : AI n = I m A = A

I ij =

1 i=j,

0 otherwise.

∀A ∈ Rm ×n : AI n = I m A = A

Diagonal matrix: D = diag (d 1,d 2, . . . , d n ):

D ij =d i j=i,

0 otherwise.

I ij =

1 i=j,

0 otherwise.

∀A ∈ Rm ×n : AI n = I m A = A

D ij =d i j=i,

0 otherwise.

Symmetric matrices: A ∈ Rn ×n is symmetric if A = AT .

I ij =

1 i=j,

0 otherwise.

∀A ∈ Rm ×n : AI n = I m A = A

D ij =d i j=i,

0 otherwise.

Symmetric matrices: A ∈ Rn ×n is symmetric if A = AT .

Orthogonal matrices: U

∈Rn ×n is orthogonal if

UU T = I = U T U

Linear Independence and Rank

A set of vectors x 1, . . . , x n is linearly independent if

α1, . . . , αn

i =1 αi x i = 0

α1, . . . , αn

i =1 αi x i = 0

Rank: A ∈ Rm ×n , then rank (A) is the maximum number of

linearly independent columns (or equivalently, rows)

α1, . . . , αn

i =1 αi x i = 0

Rank: A ∈ Rm ×n , then rank (A) is the maximum number of

linearly independent columns (or equivalently, rows)

Properties:

rank (A) ≤ minm ,n rank (A) = rank (A

)rank (AB ) ≤ minrank (A), rank (B )rank (A + B ) ≤ rank (A) + rank (B )

Matrix Inversion

If A∈Rn ×n , rank (A) = n , then the inverse of A, denoted

A−1 is the matrix that: AA−1 = A−1A = I

Properties:

(A−1)−1 = A(AB )−1 = B −1A−1

B i Li Al b

Range and Nullspace of a Matrix