Lecture 10: Vector Algebra: Orthogonal Basis

• Orthogonal Basis of a subspace

• Computing an orthogonal basis for a subspace using Gram-Schmidt Orthogonalization Process

1

Orthogonal Set

• Any set of vectors that are mutually orthogonal, is a an orthogonal set.

Orthonormal Set

• Any set of unit vectors that are mutually orthogonal, is a an orthonormal set.

• In other words, any orthogonal set is an orthonormal set if all the vectors in the set are unit vectors.

• Example: ෞ𝒖1, ෞ𝒖2, ෞ𝒖3 is an orthonormal set, where,

ෞ𝒖1 =

3

111

111

11

, ෞ𝒖2 =

−1

62

61

6

, ෞ𝒖3 =

−1

66

−4

667

66

An orthogonal set is Linearly Independent

Projection of vector 𝒃 on vector 𝒂

• 𝒂 ∙ 𝒃 = 𝒂 𝒃 cos 𝜃

• Vector 𝒄 is the image/perpendicular projection of 𝒃 on 𝒂

• Direction of 𝒄 is the same as 𝒂

• Magnitude of 𝒄 is 𝒄 = 𝒃 cos 𝜃 =𝒂∙𝒃

𝒂

𝒄 = ෝ𝒂 ∙ 𝒃

• If ෝ𝒂 is the unit vector of 𝒂, then

• vector 𝒄 =𝒂∙𝒃

𝒂ෝ𝒂 =

𝒂∙𝒃

𝒂

𝒂

𝒂=

𝒂∙𝒃

𝒂∙𝒂𝒂 𝒃 cos 𝜃

𝒃 sin 𝜽𝒃

Orthogonal Basis

• An orthogonal basis for a subspace 𝑊 of 𝑅𝑛 is a basis for 𝑊 that is also an orthogonal set.

• Example: 100

,010

,001

is basically the 𝑥, 𝑦, and 𝑧 axis. It is an orthogonal basis in ℝ3,

and it spans the whole ℝ3 space. It is also an orthogonal set.

Orthogonal Basis

• We know that given a basis of a subspace, any vector in that subspace will be a linear combination of the basis vectors.

• For example, if 𝒖 𝑎𝑛𝑑 𝒗 are linearly independent and form the basis for a subspace S, then any vector 𝒚 in S can be expressed as:

𝒚 = 𝑐1𝒖 + 𝑐2𝒗

• But computing 𝑐1 and 𝑐2 is not straight forward.

• On the other hand, if 𝒖 𝑎𝑛𝑑 𝒗 form an orthogonal basis, then

𝑐1 =𝒚 ∙ 𝒖

𝒖=𝒚 ∙ 𝒖

𝒖 ∙ 𝒖and 𝑐2 =

𝒚 ∙ 𝒗

𝒗=𝒚 ∙ 𝒗

𝒗 ∙ 𝒗

𝒗

𝒖

𝒚

𝑐2

𝑐1

Does not work if it is not Orthogonal basis

• 𝒚 = 𝑐1𝒖 + 𝑐2𝒗

• But computing 𝑐1 and 𝑐2 is not straight forward (yet).

What is computed is,

• 𝑑1 =𝒚∙𝒖

𝒖=

𝒚∙𝒖

𝒖∙𝒖

• 𝑑2 =𝒚∙𝒗

𝒗=

𝒚∙𝒗

𝒗∙𝒗

8

𝒗

𝒖

𝒚

𝑐2

𝑐1𝑑1

𝑑2

Orthogonal Decomposition Theorem

Orthogonal Decomposition Theorem

Orthogonal Decomposition Theorem

Orthogonal Decomposition Theorem

Example

Projection of a vector on a Subspace

• 𝒖 and 𝒗 are orthogonal 3D vectors.

• They span a plane (green plane) in 3D

• 𝒚 is an arbitrary 3D vector out of the plane.

• 𝒚′ is the projection of 𝒚 onto the plane.

• 𝒚′ =𝒚∙𝒖

𝒖∙𝒖𝒖 +

𝒚∙𝒗

𝒗∙𝒗𝒗

• The “point” 𝒚′ is also

the closest point to 𝒚 on the plane.

𝒚 − 𝒚′is perpendicular to 𝒚′, Span{𝒖, 𝒗}, and hence 𝒖 𝑎𝑛𝑑 𝒗

𝒗

𝒖

𝒚

Perpendicular to 𝒚′, 𝒖, 𝒗

𝒚′

Closest point of a vector to a span does not depend on the basis of that span𝒚 is an arbitrary 3D vector out of the plane.• 𝒚′ is the projection of 𝒚 onto the plane.

• 𝒚′ = 𝑥1𝒖𝟏 + 𝑥2𝒖𝟐

• 𝒚′ = 𝑥3𝒖𝟑 + 𝑥4𝒖𝟒

• 𝒚′ = 𝑥5𝒖𝟏 + 𝑥6𝒖𝟑

• …• Coordinates of 𝒚′ change if the basis changes. • But the vector 𝒚′ itself does not change. • Hence the the closest point to 𝒚 on the plane does not change.Even here, 𝒚 − 𝒚′is perpendicular to 𝒚′ and Span{., . }

𝒖𝟏

𝒚

Perpendicular to 𝒚′and the

span

𝒚′𝒖𝟐

𝒖𝟑

𝒖𝟒

Closest point of a vector to a span does not depend on the basis of that span

• FINDING THE CLOSEST POINT OF A VECTOR TO A SPAN means :

“Finding the coordinates of the projection of the vector”

• So, if you want to compute the closest point of a vector to a span, then find an appropriate basis with which you can compute the coordinates of the projection easily.

What would be that basis? Answer: An orthogonal basis!

𝒖𝟏

𝒚

Perpendicular to 𝒚′and the

span

𝒚′𝒖𝟐

𝒖𝟑

𝒖𝟒

Projection on a span of non-orthogonal vectors• How to find projection of any arbitrary 3D vector onto

the span of two non-orthogonal, linearly independent vectors?

• 𝒖1and 𝒖2 are not orthogonal, but linearly independent vectors in 3D.

• 𝒚 is an arbitrary 3D vector.• Find the projection of 𝒚 in the space spanned by 𝒖1and 𝒖2.• a) First, find the orthogonal set of vectors 𝒗1and 𝒗2that span

the same subspace as 𝒖1and 𝒖2 . In other words, find an orthogonal basis.

• b) Project 𝒚 onto the space spanned by orthogonal 𝒗1and 𝒗2vectors, as we earlier.

𝒖𝟏

𝒚

𝒚′

𝒗𝟏

𝒖𝟐

𝒗𝟐

How to find an orthogonal basis?

How to find an orthogonal basis?

• Assume that the first vector 𝒖1 is in the orthogonal basis. Other vector(s) of the basis are computed that are perpendicular to 𝒖1

• Let 𝒗1 = 𝒖1• Let 𝒗2 = 𝒖2 −

𝒖2𝒗1

𝒗1𝒗1𝒗1

• We know that 𝒗2is perpendicular to 𝒗1.

• 𝒗2 is in the Span{𝒖1, 𝒖2} (Why?)

• So Span{𝒖1, 𝒖2} = Span{𝒗1, 𝒗2}

• And {𝒗1, 𝒗2} is an orthogonal basis

• Projection of 𝒚 on to the Span{𝒖1, 𝒖2}

𝒚′ =𝒚∙𝒗1

𝒗1∙𝒗1𝒗1 +

𝒚∙𝒗2

𝒗2∙𝒗2𝒗2

𝒖1

𝒚

𝒚′

𝒖2𝒗2

𝒗1

Gram-Schmidt Orthogonalization Process

Gram-Schmidt Orthogonalization Process

Example:

Solving Inconsistent Systems

Solving Inconsistent Systems

Solving Inconsistent Systems

Example 1:

A trader buys and/or sells tomatoes and potatoes. (Negative number means buys, positive number means sells.) In the process, he either makes profit (positive number) or loss (negative number). A week’s transaction is shown; find the approximate cost of tomatoes and potatoes.

Tomatoes(tons)

Potatoes(tons)

Profit/Loss (in thousands)

1 -6 -1

1 -2 2

1 1 1

1 7 6

Solving Inconsistent Systems

• 𝟏𝒕 - 6𝒑 = −𝟏

• 𝟏𝒕 - 2𝒑 = 𝟐

• 𝟏𝒕 + 1𝒑 = 𝟏

• 𝟏𝒕 + 7𝒑 = 𝟔

•

1111

𝑡 +

−6−217

p =

−1216

The above equation might not have a solution (values of t and p that would satisfy that equation). So the best we can do is to find the values of t and p that would result in a vector on the right hand side that is as close as possible to the desired right hand side vector.

Solving Inconsistent Systems

Example 2: