K236: Basis of Data ScienceLecture 4. Review of linear algebra

Lecturer: Tu Bao Ho and Hieu Chi DamTA: Moharasan Gandhimathi

and Nuttapong Sanglerdsinlapachai

2

Schedule of K236

1. Introduction to data science ���*"�3 6/9

2. Introduction to data science ���*"�3 6/13

3. Data and databases ��������� 6/16

4. Review of univariate statistics !2+. 6/20

5. Review of linear algebra ,#�$ 6/23

6. Data mining software ��������� ��� 6/27

7. Data preprocessing �����' 6/30

8. Classification and prediction (1) �5��& (1) 7/4

9. Knowledge evaluation )1/� 7/7

10. Classification and prediction (2) �5��& (2) 7/11

11. Classification and prediction (3) �5��& (3) 7/14

12. Mining association rules (1) (4����-% 7/18

13. Mining association rules (2) (4����-% 7/21

14. Cluster analysis ����-% 7/25

15. Review and Examination �����06 (the data is not fixed) 7/27

Outline

1. Vector and matrix2. Operations on vectors and matrices3. Linear transformation4. Other topics

3

Linear)algebra)is)the)math)of)vectors)and)matrices.)

Application of matrices• To represent linear transformations

The rotation of vectors in 3D space is a linear transformation, which can be represented by a rotation matrix !: if " is a column vector describing the position of a point in space, the product !" is a column vector describing the position of that point after a rotation.

• To represent the composition of two linear transformations by the product of two transformation matrices.

• To solve systems of linear equations.

• Insight into the geometry of a linear transformation is obtainable (along with other information) from the matrix's eigenvalues and eigenvectors.

4https://en.wikipedia.org/wiki/Matrix_(mathematics)

Vectors and matrices

• Let # be a positive integer and let ℝ denote the set of real numbers, then ℝ% is the set of all #-tuples of real numbers.

• A vector &'⃗ ∈ ℝ% is an #-tuple of real numbers.

• A matrix * ∈ ℝ+×% is a rectangular array of real numbers with -rows and # columns. For example, a 3 × 2 matrix looks like this:

• Which math operations can perform on vectors and matrices?

5

Vectors and matrices

6

., 0 Dentry:)123#&567: &-

#&96:;-#: &#

-1<5.=&>.?@: &-&=&#

.-th row vector: 52 = 12B, 12C, … , 12% , 0-column vector: 93 =

1B31C3

⋮1+3

If * = 123 %×%Then Trace of A, F5 * = 1BB + 1CC + ⋯+ 1%%&&

Properties: F5 * ± J = F5 * ± F5 J , F5 *J = F5(J*)

* = 123 =

1BB1CB⋮

12B⋮

1+B

1BC1CC⋮

12C⋮

1+C

⋯

1B31C3

⋮

123

⋮

1+3

⋯

1B%1C%⋮

12%⋮

1+%+×%

Vector operations

• Operations can perform on vectors ; &= ;B, ;C, ;M &and '⃗ = 'B, 'C, 'M

are:

• ;. '⃗ = ; & '⃗ 96>O, O is the angle between two vectors.• ; and '⃗ are orthogonal if O = 90R. In this case ;. '⃗ = 0.

7

Addition

Subtraction

Scaling

norm (length))

dot)product

cross)product

Linear independence

8

Matrix operations

• Matrix addition:

ST&* = & 123 +×%, J = & U23 +×%

&&&&&&Fℎ@#&* + J = 123 +×%+& U23 +×%

= 123 + U23 +×%

• Matrix subtraction:

* − J = * + −1 J

• Scalar multiplication:

ST&* = & 123 +×%&1#Y&9&.>&1&>91:15,&

Fℎ@#&&9* = 9123 +×%

9

Matrix operations

• Matrix-vector product

• Matrix multiplication

• Matrix inverse (denoted *ZB)

• Determinant (denoted det * 65& * &)

• Matrix transpose (denoted *^)

10

Property:) *J ^ = J^*^

Matrix multiplication

11

pnijnmij bBaA !! == ][ ,][ If

pmijpnijnmij cbaAB !!! == ][][][Then

njin

n

kjijikjikij babababac +++==!

=

!1

2211where

Properties: (1) A+B = B+A, (2) BAAB !

Size of AB

!!!!

"

#

$$$$

%

&

=

!!!!!

"

#

$$$$$

%

&

!!!!!!

"

#

$$$$$$

%

&

inijii

nnnjn

nj

nj

nnnn

inii

n

ccccbbb

bbbbbb

aaa

aaa

aaa

!!!!

""""!"!!

"!

""!

"""!

21

1

2221

1111

21

21

11211

Matrix-vector product

• Two interpretations: ! In column picture (C), the multiplication of the matrix * by the vector =⃗

produces a linear combination of the columns of the matrix: _⃗ = *=⃗ =

=B*[:,B] + =C*[:,C], where *[:,B]&and *[:,C]&are the first and second columns of the matrix *.

! In the row picture, (R), multiplication of the matrix A by the vector =⃗produces a column vector with coefficients equal to the dot products of rows of the matrix with the vector =⃗.

12

Linear transformations

• A mapping Fb:ℝ% ⟶ ℝ+ is called linear transformation if for any two vectors ;, ' ∈ ℝ% and any scalar 9, we have Fb ; + ' = Fb ; + Fb(')

and Fb 9; = 9Fb(;).

• Given a matrix * ∈ ℝ+×%&and&= ∈ ℝ%, the function Fb = = *= describes the linear transformation Fb that takes #-vectors as inputs and produces --vectors as outputs:

Fb:ℝ% ⟶ ℝ+.&

• We simply write *=⃗ = U.

• Applying the linear transformation Fb to the vector =⃗ corresponds to the product of the matrix * and the column vector =⃗. We say Fb is represented by the matrix *.

13

Linear transformations

14

!!"

!!#

$

=+++

=+++=+++

mnmnmm

nn

nn

bxaxaxa

bxaxaxabxaxaxa

!"!!

2211

22222121

11212111

= = =

A x b

equationslinear �m

equationmatrix Single

bx A = 1 !! nnm 1!m!

!!!

"

#

$$$$

%

&

=

!!!!

"

#

$$$$

%

&

!!!!

"

#

$$$$

%

&

mnmnmm

n

n

b

bb

x

xx

aaa

aaaaaa

!!"

!!!!""

2

1

2

1

21

22221

11211

Matrix inverse

• Consider matrix *%×%. If matrix J%×% such that *J = J* = S%

Then (1) * is invertible (or nonsingular), (2) J is the inverse of *.

• The inverse matrix *ZB&undoes the effects of the matrix *

*ZB* = S% ≡

1 0

⋱

0 1

• To solve for h in h* = J, multiple both sides by *ZB&from the right,

we have h = J*ZB.

• To solve for h in *Jihj = k, multiple both sides by jZB&on the right,

and *ZB, JZB, iZB from the left, we have

h = iZBJZB*ZBkjZB.

15

Systems of equations as matrix equations

• Consider the matrix equation had the form *=⃗ = U, where * is

a 2 × 2 matrix, =⃗&is the vector of unknowns, and U is a vector of

constants.

• We can solve for =⃗ by multiplying both sides of the equation by the

matrix inverse *ZB:

• But how did we know what the inverse matrix *ZB is?

16

Find the inverse of a matrix by

Gauss-Jordan elimination method

• l1;>> − m65Y1#&@:.-.#1<.6#:& *&|&S → S&|&*B

17

!"

#$%

&''

=3141

ASolution *h = S

!"

#$%

&=!

"

#$%

&!"

#$%

&'' 10

013141

2221

1211

xxxx

!"

#$%

&=!

"

#$%

&''''++

1001

3344

22122111

22122111

xxxxxxxx

(2) 1304

(1) 0314

2212

2212

2111

2111

=!!=+=!!=+"

xxxxxxxx

18

Find the inverse of a matrix by Gauss-Jordan elimination method

!"#

$%& '((( )(!"

#$%&

''*'

110301

031141)1(

)4(21

)1(12 ,

!!

!! rr

!"#

$%& '((( )(!"

#$%&

''*'

110401

131041)2(

)4(21

)1(12 ,

!!

!! rr

1 ,3 2111 =!=" xx

1 ,4 2212 =!=" xx

)( 1143 1-

2221

12111 AAIAXxxxx

AX ==!"

#$%

& ''=!

"

#$%

&== '

Thus

11104301

10310141

1

inationJordanElimGauss

, )4(21

)1(12

!

!

"#

$%&

' !!((( )("

#

$%&

'!!

!

AIIA

rr !!

!!

If * can’t be row reduced to S, then * is singular.

Fundamental vector spaces

• A vector space consists of a set of vectors and all linear combinations of

these vectors. For example the vector space p = >q1#{'⃗B, '⃗C} consists of all

vectors of the form '⃗ = t'⃗B + u'⃗C, where t, u ∈ ℝ.

• The column space v * &of a matrix * is the set of vectors that can be

produced as linear combinations of the columns of the matrix *:

v * ≡ & _⃗ ∈ ℝ+|&_⃗ = *=⃗&&for&some&x ∈ ℝ%

• The null space }(*)&of * ∈ ℝ+×%&is&} * ≡ x ∈ ℝ%&|&*=⃗ = 0 .

• The row space ℛ * &of a matrix *&is the set of linear combinations of the

rows of *. The row space ℛ * &is the orthogonal complement of the null

space } * , .. @. ,&for all vectors '⃗ ∈ ℛ * and all vectors 7 ∈ } * ,&we

have '⃗. 7 = 0&and&ℝ% = } * ⊕ ℛ * where ⊕ stands for orthogonal

direct sum.

19

Basis

• A basis is any set of vectors that can be used as a coordinate system

for a vector space.

• The standard basis for the =_-plane, ' described by coordinate pair

'Å, 'Ç with respect to these axes, or equivalently '⃗ = 'ÅÉ̂ + 'ÇÖ̂,&&

where É̂ = 1,0 &1#Y&Ö̂ = 0,1 are unit vectors that point along the

&=-axis and _-axis respectively.

• Definition (Basis). A basis for a #-dimensional vector space Ü is any

set of # linearly independent vectors that are part of Ü.

• Any set of two linearly independent vectors eáB, eáC can serve as a

basis for ℝC. We can write any vector ' ∈ ℝC as a linear combination

of these basis vectors '⃗ = 'B@̂B + 'C@̂C.

20

Basis

• Note the same vector '⃗&&corresponds to different coordinate pairs

depending on the basis used: '⃗ = 'Å, 'Ç in the standard basis

Jà ≡ É,̂ Ö̂ , and '⃗ = 'B, 'C in the basis Jâ ≡ @̂B, @̂C

• Itis important to keep in mind the basis with respect to which the

coefficients are taken, and if necessary specify the basis as a

subscript, e.g., 'Å, 'Ç äã&or 'B, 'C äå

.

• Converting a coordinate vector from the basis Jâ to the basis Jà is

essence of data transformation, and performed as a multiplication by

a change of basis matrix:

• The change of basis from the Bs-basis to the Be-basis is accomplished

using the inverse matrix

21

Dimension and bases for vector spaces

• The dimension of a vector space is defined as the number of

vectors in a basis for that vector space.

• There is a general procedure for finding a basis for a vector space.

• Suppose you are given a description of a vector space in terms of

- vectors ç = >q1# '⃗B, '⃗C, … , '⃗+ and you are asked to find a basis

for ç and the dimension of ç.

• To find a basis for ç, you must find a set of linearly independent

vectors that span ç.&We can use the Gauss–Jordan elimination

procedure to accomplish this task.

22

Determinants

• The determinant of a matrix, denoted det&(*)&or |*|, is a special

way to combine the entries of a matrix that serves to check if a

matrix is invertible or not. The determinant formulas for 2 × 2 and

3 × 3 matrices are

23

Rank of a matrix

• Rank of a matrix&*+×%, denoted by 51#è(*), &is the maximum number

of its linearly independent rows (it is proven that number is equal

the maximum number of its linearly independent columns).

• We have 0 ≤ 51#è(*) ≤ min&(-, #)

24

Determinant)|ë|• Real+number• Defined+to+square+matrix+only• * ≠ 0& ⟹&∃*ZB

• * = 0,+we+only+know+that+all+the+columns+(or+rows)+together+are+linearly+dependent,+but+don’t+know+which+subset+of+columns++(or+rows)+which+are+linearly+independent.

!ïñó(ë)

• Integer• Defined+to+any+rectangular+matrix

• When+A+is+a&&#!#&square+matrix,+51#è(*) = 0& ⟹&∃*ZB

Eigenvalues and eigenvectors

• The vector " is an eigenvector of matrix A and ò is an eigenvalue of

A if:

*' = ò' (assume non-zero ')

• Interpretation: the linear transformation implied by A cannot

change the direction of the eigenvectors ', only their magnitude.

• To find the eigenvalues ò of a matrix *, find the roots of the

characteristic polynomial det * − òS = 0.

• Example:

23

Eigenvalues and Eigenvectors

• Eigenvalues and eigenvectors are only defined for square matrices

(i.e., -& = &#)

• Eigenvectors are not unique (e.g., if ' is an eigenvector, so is è')

• Suppose ò1, ò2, … , ò#&are the eigenvalues of *, then:

26

Homework for K236-L4

Based on the key issues mentioned in the class, choose

and use your suitable documents to study or recover what

you have learnt about linear algebra.

Recommended site

http://www.statisticshowto.com/matrices-and-matrix-

algebra/

Note: You don’t have to submit the report of this

homework.

27