Numerical Algs

Inner products, Matrix Multiplication,

LU-Decomposition (factorization)Simplex Method,

Iterative methods (sqrt)

Optimization

• Many problems in computing are of the form:– Find the values

that minimize (or maximizes) some function

subject to constraints

Example (unconstrained)• Find mean and standard deviation of a normal distribution

that maximizes the probability of samples

where the probability is

Example (unconstrained)• Find a quadratic polynomial

where we have made measurements

so that the mean squared error in our measurements is minimized.

Example

• (Help me out here)

Gradient Descent

Newton Iteration

Matrices• This REALLY should be review for you….

• A matrix is a 2D array of numbers (real or complex) – The number of rows is usually called – The number of columns is usually called

• Matrices are represented by upper case letters

• Matrix elements two subscripts – is the element in row and column of matrix – Indices start at 1, not zero– Sometimes elements are written lowercase, sometime uppercase. (sloppy)

• Matrix multiplication is special

Matrix + Matrix• Addition/Subtraction is done element by element

Matrix * Scalar

• Done in parallel• Symmetric ()

Matrix Transpose

• Use superscript to indicate transpose.• Exchange the order of subscripts (rows

become columns)

Slicing• In this class I will use slicing to indicate a sub-array.• My notation is something like Matlab’s.

• Concatenation– I will use the | operator to concatenate matrices: , , or

Vectors• A vector is a single-column matrix.• We write them with lowercase boldface letters.

• A row-vector is a single-row matrix. We always write them as to indicate they are the transpose of a column matrix.

Inner Product• Defined for two vectors

Other notations for inner product:, ,

Also called the ‘dot’ product.

Types of Matrix• Zero • Identity • Symmetric • Lower Triangular • Strict Lower Triangular • Upper & Strict Upper • Banded

Accessing Matrix Elements

If we know the dimensions of the array ahead of time we can use a fixed-size 2D array.

But usually the dimensions are decided at runtime.I cannot pass this as a parameter to a function that operates on arrays with a variable size

Writing functions for 2D arrays

• We need to do some extra work to be able to pass matrices arount

Accessing Matrix Elements

• The layout is ‘contiguous’ in memory.• The is row major order. • The difference between items on the same row and different columns is

_______• The difference between items on the same column and different rows is

_______• The index of the item at row and column is

________

Accessing Matrix Elements

• Other API’s use different layout (Fortran, OpenGL, MatLAB, …)• This is column major order. • The difference between items on the same row and different columns is

_______• The difference between items on the same column and different rows is

_______• The index of the item at row and column is

________• LaPACK is the most commonly used library for math.• It is implemented in Fortran, and called from c / c++.

[0 2 41 3 5 ]→[0,1 ,2 ,3 ,4 ,5 ]

Representing a slice (MatLAB style)Let denote the numbers

So

Consider int A[] = Row 1 of has indices 0:1:2. Column 1 of A has indices 0:3:9.

Basic idea of a slice

• Keep track of the start index (or pointer)

• Keep track of the number of values (or end index)

• Keep track of the difference between consecutive values (stepsize)

C++ Slice

• std::slice– Start– Size (number of items)– Step (difference between consecutive items)

• std::slice(0, 3, 2) => 0, 2, 4Valarray<double> x;…..x[slice(0,3,2)] => x[0], x[2], x[4]

• I don’t use std::slice often (I write my own matrix class or use another library)

• The idea is very important

Matrix Views & Submatrices• What if we want a submatrix that is not a single row or

column?

How can we represent ?• We need a 2D version of a slice • Keep track of…

– Linear index of in this case

– Offset between consecutive rows is , in this case 3– Offset between consecutive columns is – Keep track of the sizes (in this case )

Matrix View• Imagine that library code & functions always operate on views

of some other underlying data. • A vector view might need the following:

– A pointer to the memory – Offset to the first item– Offset between consecutive items– The number of items in the vector.

• A matrix view might need the following:– Pointer to the memory– Offset to the first item– Offset between consecutive rows– Offset between consecutive items– Number of rows– Number of columns

In the STL these are called std::slice_array but do not seem to be used often. There are many custom implementations of this idea though.

In the STL these are called std::gslice_array but do not seem to be used often. There are many custom implementations of this idea though.

Matrix MultiplicationIf then

Dimensions must agree:The number of columns in A must match the number of rows in B

• Given what is the cost of MATRIX-MULTIPLY?

Inner Products

Vector Norms

Outer Products

Valarrays

Matrix Transpose

Inverse matrix (2 x 2)

LU Decomposition