Intro to Analysis of Algorithms

Algorithm

• “A sequence of unambiguous instructions for solving a problem, i.e., for obtaining a required output for any legitimate input in a finite amount of time.”

• Named for Al Khwarizmi, who laid out basic procedures for arithmetic functions. (Read about him!)

Analysis of Algorithms

• Correctness• Generality• Optimality• Simplicity• Time Efficiency• Space Efficiency

Measuring Efficiency

• What is basic unit to measure input size? (n)• What is basic unit of resource?

– Time: basic unit operation– Space: memory units

• Best, worst, or average case?• Find its efficiency class

Why do we care?

• Let’s look at Fibonacci numbers:1, 1, 2, 3, 5, 8, 13, …

Fibonacci Sequence

• We want to compute the nth number in the sequence. (F3 = 2, for example.)

• This definition can be translated directly into code – a recursive method.

• How many additions does it take to compute Fn?

Which is better?

function fib2(n)create an array f[0...n]f[0] = 0, f[1] = 1for i = 2...n:

f[i] = f[i-1] + f[i-2]return f[n]

function fib1(n)if n = 0: return 0if n = 1: return 1return fib1(n-1) + fib1(n-2)

• Consider calculating F200. The fib1 method takes over 2138 steps.

• Computers can do several billion instructions per second.

• Suppose we have a supercomputer that does 40 trillion instructions per second.

• Consider calculating F200. The fib1 method takes over 2138 steps.

• Computers can do several billion instructions per second.

• Suppose we have a supercomputer that does 40 trillion instructions per second.

• Even on this machine, fib1(200) takes at least 292 seconds, or 1018 centuries, long after the expected end of our sun!!!

• Consider calculating F200. The fib1 method takes over 2138 steps.

• Computers can do several billion instructions per second.

• Suppose we have a supercomputer that does 40 trillion instructions per second.

• Even on this machine, fib1(200) takes at least 292 seconds, or 1018 centuries, long after the expected end of our sun!!!

• But, fib2(200) would take less than a billionth of a second to compute!!!

106 instructions/sec, runtimes

N O(log N) O(N) O(N log N)

O(N2)

10 0.000003 0.00001 0.000033 0.0001

100 0.000007 0.00010 0.000664 0.1000

1,000 0.000010 0.00100 0.010000 1.0

10,000 0.000013 0.01000 0.132900 1.7 min

100,000 0.000017 0.10000 1.661000 2.78 hr

1,000,000 0.000020 1.0 19.9 11.6 day

1,000,000,000

0.000030 16.7 min 18.3 hr 318 centuries

Some helpful mathematics

• 1 + 2 + 3 + 4 + … + N– N(N+1)/2 = N2/2 + N/2 is O(N2)

• N + N + N + …. + N (total of N times)– N*N = N2 which is O(N2)

• 1 + 2 + 4 + … + 2N – 2N+1 – 1 = 2 x 2N – 1 which is O(2N )

Basics of Efficiency

• Big-oh – upper bound on the efficiency class• Efficiency classes don’t worry with constants• A cubic is worse than a quadratic, quartic

worse than cubic…• Getting big-oh analysis of non-recursive code

is pretty easy

• Maximum(A[1..n]) max <- A[0] for i<- 1 to n

if A[i] > max max <- A[i]

return max

1. What is input size?2. What is unit of time?3. What is big-oh analysis?

• Maximum(A[1..n]) max <- A[0] for i<- 1 to n

if A[i] > max max <- A[i]

return max

1 assignmentn times:

1 comparemaybe 1 assignment1 addition to i

1 return______________________1 + n(3) + 1 = O(3n+2)

• The algorithm is O(3n+2), which is O(n).• We only care about the efficiency class. Why?

• The algorithm is O(3n+2), which is O(n).• We only care about the efficiency class. Why?

• At some point, every parabola (n2) overtakes any line (n). We only really care about large input.

Efficiency classes

• So we really just care about the leading term, which determines the shape of the graph.

• This means for non-recursive algorithms, what’s important is the loops.

Analyze this…

AllUnique(A[1..n])for i<-1 to n

for j<- 1 to nif A[i] = A[j] and i ≠ j return false

return true

Best case?Worst case?Average case?

Analyze this…

AllUnique(A[1..n])for i<-1 to n

for j<- 1 to nif A[i] = A[j] and i ≠ j return false

return trueAverage case?Quit halfway through, O(n*n/2)Still O(n2)Often, Average case = Worst case.

AllUnique(A[1..n])for i<-1 to n

for j<- i to nif A[i] = A[j] return false

return true

AllUnique(A[1..n])for i<-1 to n

for j<- i to nif A[i] = A[j] return false

return true

n + (n-1) + (n-2) + … + 3 + 2 + 1 = n(n+1)/2

MatrixMultiply(A[nxn], B[nxn])Initialize empty C[nxn]for i<- 1 to n

for j <- 1 to nC[i,j] <- 0for k<- 1 to n

C[i,j] <- C[i,j] + A[i,k]*B[k,j]return C

• MethodA(n)– answer <- 1– for i<- 1 to n/2:

• answer <- answer + i• for i<- 1 to n

– answer <- answer + 1

– return answer• MethodB(n)

– answer <- 1– for i <- 1 to lg n:

• answer = answer + MethodA(n)

return answer

Big-oh Definition

• Let f(n) and g(n) be functions from positive integers to positive reals. We say f = O(g) if there exists some constant c > 0 such that

f(n) ≤ cg(n) for all n.

Big-oh Definition

• Let f(n) and g(n) be functions from positive integers to positive reals. We say f = O(g) if there exists some constant c > 0 such that

f(n) ≤ cg(n) for all n.

Say what?

Big-O• f(n) = O(g(n), think: f(n) ≤ g(n)n = O(n2) because I can find a big number to multiply the parabola by that makes it lie completely above the line. I can’t do that in reverse.

Know the shapes

constant

logarithmic

linear

quadraticexponential

Theta Notation

• Theta = tight bound

• Multiply constants by a function, results are always (asymptotically) above and below.

• What it means: If you have a parabola, you can always find parabolas that stay above and below the given parabola. Same for lines, and other curves. Just tells us the efficiency category.

Practice – True or False?• ½n(n-1) = Ө(n)• if f(n) = Ө g(n), then g(n) =

Ө(f(n))• -n – 100000 = Ө(n)

O - notation• f = O(g) if f grows slower than or the

same as g• That is, asymptotically, g is not below f• g is an upper bound• Think: O is “≤”

Practice – True or False?

• n = O(n2)• n3 = O(n2)• .00000001n3 = O(n2)• 100n + 5 = O(n2)• ½ n(n-1) = O(n2)• n4 +n+1= O(n2)

-Notation

• Opposite of big-oh• f = (g) means f is always above or

equal to g.• It gives an asymptotic lower bound.• Think: is “≥”

Practice – True or False?

• n3 = (n2)• ½ n(n-1) = (n2)• 100n + 5 = (n2)

o and ω

• f = O(g) means g is above or equal to f

• f = o(g) means g is strictly above f• Also means – there is a better,

tighter bound • ω is analogous for - there is a

tighter bound available

Function Growth Rates

Say that f is Mean that f is Write

“big oh of g” No faster than g, ≤

f = O(g)

“theta g” About as fast as g, =

f = Θ(g)

“omega g” No slower than g, ≥

f = Ω(g)

Polynomials are Easy – What about other functions?

• constant (1): Very few examples of algorithms that don’t grow with input size

• logarithmic (lg n): Usually result of cutting input size by a constant factor each time through the loop

• linear (n): Look at each input element a constant number of times

• nlgn: Divide and conquer• quadratic (n2): Two embedded loops• cubic (n3): Three embedded loops• exponential (2n): Generate all subsets of the input

elements• factorial (n!): Generate all permutations of the

input• nn : Generate all permutations of input, allowing

repetitions

Other standard functions

• Polylog – log3n = (logn)3

• Any log grows slower than any polynomial:logan = o(nb), for any a,b > 0

• Any polynomial grows slower than any exponential with base c > 1.nb = o(cn) for any c > 1

• n! = o(nn)• n! = ω(2n)• lg(n!) = Ө(nlgn)

Relative Growths –O, , or Ө? lg2n 2n

n4+3n 2n

√n n2/3

4n

4n/2

n+2n 3n

100n + lgn n+(lgn) 2

lg(n2) lg(n3)