Algorithm Analysis

What is an algorithm?

An algorithm is a step-by-step procedure for accomplishing some

end.

Why do we want to analyze it?

To be able to draw conclusions about how the implementation of

that algorithm will perform in general.

Problem

Cannot make determination from a run on a single machine

What can we analyze?

• The running time of a program as a function of its

inputs

• The total or maximum memory space needed for program data

• The total size of the program code

• Whether the program correctly computes the desired result

• The complexity of the program - e.g how easy is it to read,

understand and modify

1

• The robustness of the program - e.g how well does it deal with

unexpected or erroneous inputs?

Performance of a Computer

Determined by

• the hardware - processor, memory, disk available

• programming language used

• compiler/interpreter used

• operating system

Problem

• difficult

• time-consuming

• technology changes

Detailed Model of the Computer

• Independent of the underlying hardware and software

• General model

Basic Axioms

Axiom 1

The time required to fetch an integer operand from memory is a

constant, Tfetch, and the time required to store an integer result

in memory is a constant, Tstore.

2

y=x;

running time: Tfetch + Tstore

Axiom 2

The times required to perform elementary operations on integers,

such as addition, subtraction, multiplication, division, and com-

parison, are all constants. These are denoted as T+, T−, T×, T÷,

and T<.

y=y+1;

running time = 2Tfetch + T+ + Tstore

Axiom 3

The time required to call a function is a constant, Tcall, and the

time to return from a function is a constant, Treturn.

Axiom 4

The time required to pass an integer argument to a function is the

same as the time required to store an integer in memory, Tstore.

y=f(x)

running time = Tfetch + 2Tstore + Tcall + Tf(x)

where Tf(x) is the running time of function f(x) for input x.

Example

Analyze the running time of a program to compute the following

arithmetic summationn

∑

i=1

i

3

1 i n t Sum( in t n)2 3 i n t r e s u l t = 0 ;4 f o r ( i n t i =0; i<=n ; ++i )5 r e s u l t += i ;6 return r e s u l t ;7

Table 1: Computing Running TimeStatement Time Code

3 Tfetch + Tstore result = 0

4a Tfetch + Tstore i = 1

4b (2Tfetch + T<) × (n + 1) i ≤ n

4c (2Tfetch + T+ + Tstore) × n ++i

5 (2Tfetch + T+ + Tstore) × n result += i

6 Tfetch + Treturn return result

TOTAL = (6Tfetch + 2Tstore + T< + 2T+) × n

= +(5Tfetch + 2Tstore + T< + Treturn)

T (n) = t1 + t2n

Axiom 5

The time required for the address calculation implied by an array

subscripting operation, e.g. a[i], is a constant, T[.]. This time does

not include the time to compute the subscript expression, nor does

it include the time to access (i.e. fetch or store) the array element.

y = a[i]

running time = 3Tfetch + Tstore

Example

Finding the largest element in an array

max0≤i<n

ai

4

1 i n t FindMaximum( in t a [ ] , i n t n )2 3 i n t r e s u l t = a [ 0 ] ;4 f o r ( i n t i = 1 ; i < n ; ++i )5 i f ( a [ i ] > r e s u l t )6 r e s u l t = a [ i ] ;7 return r e s u l t ;8

Table 2: Computing Running TimeStatement Time

3 3Tfetch + T[.] + Tstore

4a Tfetch + Tstore

4b (2Tfetch + T<) × n4c (2Tfetch + T+ + Tstore) × (n − 1)5 (4Tfetch + T[.] + T<) × (n − 1)6 3Tfetch + T[.] + Tstore×?7 Tfetch + Tstore

Line 6 executed only if

ai > (max0≤j<iaj)

? - depends on the actual elements of the array, a0, a1, ..., an−1

T (n, a0, a1, ..., an−1) = t1 + t2n +∑n−1

i=1 t3

where

t1 = 2Tstore − Tfetch − T+ − T<

t2 = 8Tfetch + 2T< + T[.] + T+ + Tstore

t3 = 3Tfetch + T[.] + Tstore

Average Running Times

If a program is run a large number of times on a selection of ran-

dom inputs of length n, what will the average running time be?

5

Taverage(n) = t1 + t2n +∑n−1

i=1 pit3

pi is the probability that line 6 is executed.

If the array is ordered smallest to largest, probability is 1.

If the array is ordered largest to smallest, probability is 0.

If the order is unknown, line 6 is only executed if ai is the largest

of the i + 1 values a0, a1, ..., ai

pi = 1i+1

Taverage(n) = t1 + t2n +n−1∑

i=1

pit3

= t1 + t2n + t3

n−1∑

i=1

1

i + 1

= t1 + t2n + t3(

n∑

i=1

1

i− 1)

= t1 + t2n + t3(Hn − 1)

where Hn =∑n

i=11i is the nth harmonic number

Hn − Hn−1 = 1n

Hn−1 is approximately ln n for n > 1

Error generated by the approximation is called Euler’s constant,

γ.

Hn ≈ ln n + γ

6

Taverage(n) = t1 + t2n + t3(Hn − 1)

≈ t1 + t2n + t3(ln n + γ − 1)

≈ (t1 + t3(γ − 1)) + t2n + t3(ln n)

Best and Worst-Case Running Time

Tworst(n) = t1 + t2n +n−1∑

i=1

pit3

= t1 + t2n + t3

n−1∑

i=1

1

= t1 + t2n + t3(n − 1)

= (t1 − t3) + (t2 + t3) × n

Tbest(n) = t1 + t2n +

n−1∑

i=1

pit3

= t1 + t2n + t3 × 0

= t1 + t2n

Simplified Model

Let T be the clock cycle of a machine. e.g Tfetch = kT

Assume

• All timing parameters expressed in units of clock cycles. Thus

T = 1

7

• k is assumed to be the same for all parameters, thus k = 1

8

Example

Compute the running time of a geometric series summation

n∑

i=0

xi.

1 i n t GeometricSeriesSum ( in t x , i n t n )2 3 i n t sum = 0 ;4 f o r ( i n t i = 0 ; i <= n ; ++i )5 6 i n t prod = 1 ;7 f o r ( i n t j = 0 ; j< i ; ++j )8 prod ∗= x ;9 sum += prod ;

10 11 return sum ;12

Table 3: Computing Running TimeStatement Time

3 24a 24b 3(n + 2)4c 4(n + 1)6 2(n + 1)7a 2(n + 1)7b 3

∑n

i=0(i + 1)7c 4

∑n

i=0 i8 4

∑n

i=0 i9 4(n + 1)11 2

Total 112n2 + 47

2n + 27

Sn =

n∑

i=1

i =n(n + 1)

2

1 i n t GeometricSeriesSum ( in t x , i n t n )

9

2 3 i n t sum = 0 ;4 f o r ( i n t i = 0 ; i <= n ; ++i )5 sum = sum ∗ x + 1 ;6 return sum ;7

Table 4: Computing Running TimeStatement Time

3 24a 24b 3(n + 2)4c 4(n + 1)5 6(n + 1)6 2

Total 13n + 22

10

1 i n t GeometricSeriesSum ( in t x , i n t n )2 3 return (Power (x , n+1) − 1) /(x − 1)4

1 i n t Power ( i n t x , i n t n )2 3 i f (n == 0)4 return 1 ;5 e l s e i f (n % 2 == 0)6 return Power ( x∗x , n /2 ) ;7 e l s e8 return x∗ Power (x∗x , n /2 ) ;9

Running Time: 19(⌊log2(n + 1)⌋ + 1) + 18

0

200

400

600

800

1000

0 20 40 60 80 100

T(n

)

n

13*x + 225.5*x*x + 23.5*x + 27

19*(floor(log(x+1)+1)) + 18

Asymptotic Notation

What does it mean to say one function, TA(n) is better thananother, TB(n)?

If the size of the problem is known. i.e. n0, then if TA(n)

If the problem size is not known, if it can be shown that ∀n ≥ 0 :

11

TA(n) ≤ TB(n) then A is better than B

Consider the asymptotic behavior of the two functions instead.

Asymptotic Upper Bound - Big Oh

Definition

Consider a function f(n) which is non-negative for all integers n ≥

0. We say that “f(n) is big oh g(n),” which we write f(n) =

O(g(n)), if there exists an integer n0 and a constant c > 0 such

that for all integers n ≥ n0, f(n) ≤ cg(n).

Example

Consider the function f(n) = 8n + 128, show that f(n) = O(n2).

We need to find an integer n0 and a constant c > 0 such that for

all integers n ≥ n0, f(n) ≤ cn2.

Let c = 1 then

f(n) ≤ cn2 → 8n2 + 128 ≤ n2

→ 0 ≤ n2 − 8n − 128

→ 0 ≤ (n − 16)(n + 8)

Since (n + 8) > 0 for all values n ≥ 0, we can conclude that

(n0 − 16) ≥ 0, i.e. n0 = 16.

Thus we have c = 1, n0 = 16, f(n) ≤ cn2 for all integers n ≥ n0,

hence f9n) = O(n2).

Solutions for c = 1, 2, 4 and n0 = 16, 10.2, 6.7

12

0

100

200

300

400

500

0 5 10 15 20 25

y=f(

n)

n

4*x*x2*x*x

x*x8*x + 128

Big Oh Fallacies

1. Given that f1(n) = O(g(n)) and f2(n) = O(g(n)) then

f1(n) = f2(n).

e.g f1(n) = n and f1(n) = n2, both are O(n2), but not equal

2. If f(n) = O(g(n)), then g(n) = O−1(f(n)).

O−1 has no meaning since Big Oh is not a mathematical func-

tion

Properties of Big Oh

Theorem 1

If f1(n) = O(g1(n)) and f2(n) = O(g2(n)) then f1(n) + f2(n) =

O(max(g1(n), g2(n)).

13

Example

Let g1(n) = 1 and g2(n) = 2cos2(nπ/2) then

h(n) = max(g1(n), g2(n))

= max(1, 2cos2(nπ/2)

h(n) =

1 n is even

2 n is odd

Theorem 2

If f(n) = f1(n) + f2(n) in which f1(n) and f2(n) are both non-

negative for all integers n ≥ 0 such that limn→∞f2(n)/f1(n) = L

for some limit L ≥ 0, then f(n) = O(f1(n)).

Example

Consider f1(n) = n3 and f2(n) = n2

limn→∞f2(n)

f1(n)= limn→∞

n2

n3

= limn→∞1

n= 0

Thus f1(n) + f2(n) = n3 + n2 = O(n3).

Conclusion: The sum of a series of powers of n is

O(nm) where m is the largest power of n in the sum-

mation.

14

Theorem 3

If f1(n) = O(g1(n)) and f2(n) = O(g2(n)), then

f1(n) × f2(n) = O(g1(n) × g2(n))

Example

Consider f1(n) = n3 +n2 +n+1 and f2(n) = n2 +n+1, then by

Theorem 2 we know that f1(n) = O(n3) and f2(n) = O(n2) thus

f1(n) × f2(n) = O(n3 × n2) = O(n5)

Theorem 4

If f1(n) = O(g1(n)) and g2(n) is a function whose value is non-

negative for integers n ≥ 0, then

f1(n) × g2(n) = O(g1(n) × g2(n))

Theorem 5

If f(n) = O(g(n)) and g(n) = O(h(n)) then f(n) = O(h(n)).

Theorem 6

Consider a polynomial in n of the form

f(n) =m

∑

i=0

aini

= amnm + am−1nm−1 + ... + a2n

2 + a1n + a − 0

where am > 0. Then f(n) = O(nm).

15

Theorem 7

For every integer k ≥ 1, logkn = O(n). log n diverges as n gets

large. log n < n for all integers n ≥ 0. Hence log n = O(n).

0

5

10

15

20

25

0 5 10 15 20 25

y=f(

n)

n

log(x)x

Tight Big Oh Bounds

Consider a function f(n) = O(g(n)). If for every function h(n)

such that f(n) = O(h(n)) it is also true that g(n) = O(h(n)),

then we say that g(n) is a tight asymptotic bound on f(n).

Example

Consider f(n) = 8n + 128, we showed earlier f(n) = O(n2) but

we know from earlier theorems that f(n) = O(n). Thus O(n) is a

tighter bound on the asymptotic behavior of f(n) than is O(n2).

16

Expression NameO(1) constantO(logn) logarithmicO(log2n) log squaredO(n) linearO(nlogn) nlognO(n2) quadraticO(n3) cubicO(2n) exponential

Conventions for Writing Big Oh Expressions

• Drop all but the most significant terms. Thus instead of

O(n2 + nlogn + n) we write O(n2).

• Drop constant coefficients. Instead of O(3n2) we write O(n2).

• Use a tight asymptotic bound. Instead of f(n) = n = O(n3),

write f(n) = O(n).

0

5

10

15

20

25

0 5 10 15 20 25

y=f(

n)

n

1log(x)

log(x)*log(x)x

x*log(x)x*x

x*x*x2**x

17

Asymptotic Lower Bound - Omega

Consider a function f(n) which is non-negative for all integers

n ≥ 0. We say that “f(n) is omega g(n)” which we write f(n) =

Ω(g(n)), if there exists an integer n0 and a constant c > 0 such

that for all integers n ≥ n0, f(n) ≥ cg(n).

Example

Consider the function f(x) = 5n2−64n+256, f(n) is non-negative

for all integers n ≥ 0. Show that f(n) = Ω(n2). Need to find an

integer n0 and a constant c > 0 such that for all integers n ≥ n0,

f(n) ≥ cn2.

Suppose c = 1,

f(n) ≥ cn2 → 5n2 − 64n + 256 ≥ n2

→ 4n2 − 64n + 256 ≥ 0

→ 4(n − 8)2 ≥ 0

Since (n − 8)2 > 0 for all values n ≥ 0, we conclude n0 = 0.

Theorem 8

Consider a polynomial in n of the form

f(n) =m

∑

i=0

aini

= amnm + am−1nm−1 + ... + a2n

2 + a1n + a0

where am > 0. Then f(n) = Ω(nm).

18

0

200

400

600

800

1000

1200

1400

0 5 10 15 20 25

y=f(

n)

n

5*x*x-64*x+2562*x*x

x*x

Theta

Consider a function f(n) which is non-negative for all integers

n ≥ 0. We say that “f(n) is theta g(n)” which we write f(n) =

Θ(g(n)), if and only if f(n) is O(g(n)) and f(n) is Ω(g(n)).

Example

Previously showed that for a polynomial, f(n) = amnm+am−1nm−1+

... + a2n2 + a1n + a0, is both O(nm) and Ω(nm). Thus we can say

f(n) = Θ(nm).

Little Oh

Consider a function f(n) which is non-negative for all integers

n ≥ 0. We say that “f(n) is little oh g(n)” which we write

f(n) = o(g(n)), if and only if f(n) is O(g(n)) but f(n) is not

Θ(g(n)).

19

Example

Consider the function f(n) = n + 1. Clearly f(n) = O(n2). Also

f(n) 6= Ω(n2) since no matter what c is chosen, for large enough

n, cn2 ≥ n + 1. Thus f(n) = n + 1 = o(n2).

1 i n t Horner ( i n t a [ ] , i n t n , i n t x )2 3 i n t r e s u l t = a [ n ] ;4 f o r ( i n t i = n−1; i >= 0 ; −− i )5 r e s u l t = r e s u l t ∗ x + a [ i ] ;6 return r e s u l t ;7

Statement Detailed Model Simple Model Big Oh3 3Tfetch + T[.] + Tstore 5 O(1)4a 2Tfetch + T− + Tstore 4 O(1)4b (2Tfetch + T<) × (n + 1) 3n+3 O(n)4c (2Tfetch + T− + Tstore) × n 4n O(n)5 (5Tfetch + T[.] + T+ + T× + Tstore) × n 9n O(n)6 Tfetch + Treturn 2 O(1)Total Ttotal 16n + 14 O(n)

Ttotal = (9Tfetch + T[.] + T+ + T< + T− + T× + 2Tstore) × n +

= 8Tfetch + T[.] + T< + T− + Treturn + 2Tstore

20

Rules for Big Oh Analysis of Running Time

Rule 1: Sequential Composition

The worst case running time of a sequence of C++ statements

such as

S1;

S2;

.

.

.

Sm;

is O(max(T1(n), T2(n), ..., Tm(n))), where the running time of

Si, the ith statement in the sequence, is O(Ti(n)).

Rule 2: Iteration

The worst case running time of a C++ for loop such as

for (S1; S2; S3)

S4;

is O(max(T1(n), T2(n)×(I(n)+1), T3(n)×I(n), T4(n)×I(n))),

where the running time of a statement Si is O(Ti(n)), for i =

1, 2, 3, 4 and I(n) is the number of iterations executed in the worst

case.

Example

1 f o r ( i n t i = 0 ; i <n ; i++)2 S 4 ;

21

S1 is int i = 0, so the running time is constant T1(n) = 1.

S2 is i < n, so the running time is constant T2(n) = 1.

S3 is + + 1, so its running time is a constant T3(n) = 1.

The number of iterations is I(n) = n.

According to Rule 2, the running time of this is O(max(1, 1× (n+

1), 1 × n, T4(n) × n)), which simplifies to O(max(n, T4(n) × n)).

If the loop does anything at all, its running time must be T4(n) =

Ω(1).

Loop body dominates and the running time is O(n × T4(n).

If number of iterations is unknown, the rule also applies if an upper

bound on the number of iterations is known, i.e I(n) = O(f(n)).

Running time would be O(max(T1(n), T2(n)× (f(n)+1), T3(n)×

f(n), T4(n) × f(n))).

Rule 3:Conditional Execution

The worst case running time of a C++ if-then-else such as

if(S1)

S2;

else

S3;

is O(max(T1(n), T2(n), T3(n))) the running time of statement

Si is O(Ti(n)), for i = 1, 2, 3.

22

Reality Check

Table 5: Actual Lower Bounds assuming a 100Mhz Clock, c = 1 Cycle and n0 = 0n = 1 n = 8 n = 1K n = 1024K

Ω(1) 10ns 10ns 10ns 10nsΩ(logn) 10ns 30ns 100ns 200nsΩ(n) 10ns 80ns 10.2µs 10msΩ(nlogn) 10ns 240ns 10.2µs 210msΩ(n2) 10ns 640ns 102µs 3.05hrsΩ(n3) 10ns 5.12µs 10.7s 365yrs

Ω(2n) 10ns 2.56µs 10293yrs 10105yrs

23