CSC401 – Analysis of AlgorithmsCSC401 – Analysis of Algorithms Lecture Notes 2Lecture Notes 2

Asymptotic Analysis Asymptotic Analysis

Objectives:Objectives:

Mathematics foundation for algorithm analysisMathematics foundation for algorithm analysisAmortization analysis techniquesAmortization analysis techniquesCase studiesCase studiesAsymptotic notions: Big-Oh, Big-Omega, Little-Asymptotic notions: Big-Oh, Big-Omega, Little-oh, little-omega, Big-Thetaoh, little-omega, Big-Theta

22

Summation:Summation:

Geometric summation:Geometric summation:

Natural summation:Natural summation:

Harmonic number:Harmonic number:

Split summation:Split summation:

SummationsSummations

a

aaaaa

nn

n

i

i

1

11

12

0

2

)1(321

0

nnni

n

i

)()2()1()()( bfafafafifb

ai

nnii

ln1

3

1

2

11

1

0

n

ki

k

i

n

i

ififif110

)()()(

aaaaa n

i

i

1

11 2

0

33

LogarithmsLogarithms– properties of logarithms:properties of logarithms:

loglogbb(xy) = log(xy) = logbbx + logx + logbbyy

loglogbb (x/y) = log (x/y) = logbbx - logx - logbbyy

loglogbbxa = alogxa = alogbbxx

loglogbba = loga = logxxa/loga/logxxbb

ExponentsExponents– properties of exponentialsproperties of exponentials::

aa(b+c)(b+c) = a = abba a cc

aabcbc = (a = (abb))cc

aabb /a /acc = a = a(b-c)(b-c)

b = a b = a loglogaabb

bbcc = a = a c*logc*logaabb

Floor and ceiling functionsFloor and ceiling functions– the largest integer less than or equal to x– the smallest integer greater than or equal to x

Logarithms and Exponents Logarithms and Exponents

x x

44

By Example -- counterexampleBy Example -- counterexampleContrapositiveContrapositive– Principle: To justify “if p is true, then q is true”, show “if Principle: To justify “if p is true, then q is true”, show “if

q is not true, then p is not true”.q is not true, then p is not true”.– Example: To justify “if ab is odd, a is odd or b is even”, Example: To justify “if ab is odd, a is odd or b is even”,

assume “’a is odd or b is odd’ is not true”, then “a is assume “’a is odd or b is odd’ is not true”, then “a is even and b is odd”, then “ab is even”, then “’ab is odd’ even and b is odd”, then “ab is even”, then “’ab is odd’ is not true”is not true”

ContradictionContradiction– Principle: To justify “p is true”, show “if p is not true, Principle: To justify “p is true”, show “if p is not true,

then there exists a contradiction”.then there exists a contradiction”.– Example: To justify “if ab is odd, then a is odd or b is Example: To justify “if ab is odd, then a is odd or b is

even”, let “ab is odd”, and assume “’a is odd or b is even”, let “ab is odd”, and assume “’a is odd or b is even’ is not true”, then “a is even and b is odd”, thus even’ is not true”, then “a is even and b is odd”, thus “ab is even” which is a contradiction to “ab is odd”.“ab is even” which is a contradiction to “ab is odd”.

Proof TechniquesProof Techniques

55

Induction -- To justify S(n) for n>=n0Induction -- To justify S(n) for n>=n0– PrinciplePrinciple

Base cases: Justify S(n) is true for n0<=n<=n1Base cases: Justify S(n) is true for n0<=n<=n1Assumption: Assume S(n) is true for n=N>=n1;Assumption: Assume S(n) is true for n=N>=n1;

or Assume S(n) is true for or Assume S(n) is true for n1<=n<=Nn1<=n<=NInduction: Justify S(n) is true for n=N+1Induction: Justify S(n) is true for n=N+1

– Example: Example: Question: Fibonacci sequence is defined as F(1)=1, Question: Fibonacci sequence is defined as F(1)=1, F(2)=1, and F(n)=F(n-1)+F(n-2) for n>2. Justify F(2)=1, and F(n)=F(n-1)+F(n-2) for n>2. Justify F(n)<2^n for all n>=1F(n)<2^n for all n>=1Proof by induction where n0=1. Let n1=2.Proof by induction where n0=1. Let n1=2.

– Base cases: For n0<=n<=n1, n=1 or n=2. If n=1, Base cases: For n0<=n<=n1, n=1 or n=2. If n=1, F(1)=1<2=2^1. If n=2, F(2)=2<4=2^2. It holds.F(1)=1<2=2^1. If n=2, F(2)=2<4=2^2. It holds.

– Assumption: Assume F(n)<2^n for n=N>=2.Assumption: Assume F(n)<2^n for n=N>=2.– Induction: For n=N+1, F(N+1)=F(N)+F(N-Induction: For n=N+1, F(N+1)=F(N)+F(N-

1)<2^N+2^(N-1)<2 2^N=2^(N+1). It holds.1)<2^N+2^(N-1)<2 2^N=2^(N+1). It holds.

Proof by InductionProof by Induction

66

Principle:Principle:– To prove a statement S about a loop is correct, define S in To prove a statement S about a loop is correct, define S in

terms of a series of smaller statements S0, S1, …, Sk, terms of a series of smaller statements S0, S1, …, Sk, wherewhere

The initial claim S0 is true before the loop beginsThe initial claim S0 is true before the loop beginsIf Si-1 is true before iteration i begins, then show that Si If Si-1 is true before iteration i begins, then show that Si is true after iteration i is overis true after iteration i is overThe final statement Sk is trueThe final statement Sk is true

Example: Consider the algorithm Example: Consider the algorithm arrayMaxarrayMax– Statement S: max is the maximum number when finishedStatement S: max is the maximum number when finished– A series of smaller statements:A series of smaller statements:

Si: max is the maximum Si: max is the maximum in the first i+1 elements in the first i+1 elements of the arrayof the array

S0 is true before the loopS0 is true before the loopIf Si is true, then easy to If Si is true, then easy to

show Si+1 is trueshow Si+1 is trueS=Sn-1 is also trueS=Sn-1 is also true

Proof by Loop InvariantsProof by Loop Invariants

AlgorithmAlgorithm arrayMaxarrayMax((AA, , nn))maxmax AA[0] [0] forfor ii 11 toto nn 1 1 dodo

ifif AA[[ii] ] maxmax thenthenmaxmax AA[[ii]]

returnreturn maxmax

77

Sample space: Sample space: the set of all possible outcomes from the set of all possible outcomes from some experimentsome experiment

Probability space: Probability space: a sample space S together with a a sample space S together with a probability function that maps subsets of S to real numbers probability function that maps subsets of S to real numbers between 0 and 1between 0 and 1

Event: Each subset of A of S called an Event: Each subset of A of S called an eventeventProperties of the probability function PrProperties of the probability function Pr– Pr(Ø)=0Pr(Ø)=0– Pr(S)=1Pr(S)=1– 0<=Pr(A)<=1 for any subset A of S0<=Pr(A)<=1 for any subset A of S– If A and B are subsets of S and AIf A and B are subsets of S and AB=B=, then , then

Pr(APr(AB)=Pr(A)+Pr(B)B)=Pr(A)+Pr(B)

IndependenceIndependence– A and B are independent if Pr(AA and B are independent if Pr(AB)=Pr(A)Pr(B)B)=Pr(A)Pr(B)– AA11, A, A22, …, A, …, Ann are mutually independent if Pr(A are mutually independent if Pr(A11AA22… …

AAnn)=Pr(A)=Pr(A11)Pr(A)Pr(A22)…Pr(A)…Pr(Ann) )

Basic ProbabilityBasic Probability

88

Conditional probabilityConditional probability– The conditional probability that A occurs, given B, is The conditional probability that A occurs, given B, is

defined as: Pr(A|B)=Pr(Adefined as: Pr(A|B)=Pr(AB)/Pr(B), assuming Pr(B)>0B)/Pr(B), assuming Pr(B)>0

Random variablesRandom variables– Intuitively, Variables whose values depend on the Intuitively, Variables whose values depend on the

outcomes of some experimentoutcomes of some experiment– Formally, a function X that maps outcomes from some Formally, a function X that maps outcomes from some

sample space S to real numberssample space S to real numbers– Indicator random variable: a variable that maps outcomes Indicator random variable: a variable that maps outcomes

to either 0 or 1to either 0 or 1

Expectation:a random variable has random valuesExpectation:a random variable has random values– Intuitively, average value of a random variableIntuitively, average value of a random variable– Formally, expected value of a random variable X is Formally, expected value of a random variable X is

defined as E(X)=defined as E(X)=xxxPr(X=x)xPr(X=x)– Properties:Properties:

Linearity: E(X+Y) = E(X) + E(Y)Linearity: E(X+Y) = E(X) + E(Y)Independence: If X and Y are independent, that is, Independence: If X and Y are independent, that is, Pr(X=x|Y=y)=Pr(X=x), then E(XY)=E(X)E(Y)Pr(X=x|Y=y)=Pr(X=x), then E(XY)=E(X)E(Y)

Basic ProbabilityBasic Probability

99

Computing Prefix AveragesComputing Prefix AveragesWe further illustrate We further illustrate asymptotic analysis with asymptotic analysis with two algorithms for prefix two algorithms for prefix averagesaveragesThe The ii-th prefix average of -th prefix average of an array an array XX is average of is average of the first the first ((ii 1) 1) elements of elements of XX::AA[[ii]] XX[0] [0] XX[1] [1] … …

XX[[ii])/(])/(ii+1)+1)

Computing the array Computing the array AA of of prefix averages of prefix averages of another array another array XX has has applications to financial applications to financial analysisanalysis

0

5

10

15

20

25

30

35

1 2 3 4 5 6 7

X

A

1010

Prefix Averages (Quadratic)Prefix Averages (Quadratic)The following algorithm computes prefix The following algorithm computes prefix averages in quadratic time by applying the averages in quadratic time by applying the definitiondefinition

AlgorithmAlgorithm prefixAverages1prefixAverages1((X, nX, n))InputInput array array XX of of nn integers integersOutputOutput array array AA of prefix averages of of prefix averages of XX #operations#operations

AA new array of new array of nn integers integers nnforfor ii 00 toto nn 1 1 dodo nn

ss XX[0] [0] nnforfor jj 11 toto ii dodo 1 1 2 2 …… ( (nn 1) 1)

ss ss XX[[jj]] 1 1 2 2 …… ( (nn 1) 1)AA[[ii]] ss ( (ii 1) 1) nn

returnreturn A A 11

1111

Arithmetic ProgressionArithmetic ProgressionThe running time of The running time of prefixAverages1 prefixAverages1 isisOO(1 (1 2 2 ……nn))

The sum of the first The sum of the first nn integers is integers is nn((nn 1) 1) 22– There is a simple visual There is a simple visual

proof of this factproof of this fact

Thus, algorithm Thus, algorithm prefixAverages1 prefixAverages1 runs in runs in OO((nn22) ) time time 0

1

2

3

4

5

6

7

1 2 3 4 5 6

1212

Prefix Averages (Linear)Prefix Averages (Linear)The following algorithm computes prefix The following algorithm computes prefix averages in linear time by keeping a running averages in linear time by keeping a running sumsum

AlgorithmAlgorithm prefixAverages2prefixAverages2((X, nX, n))InputInput array array XX of of nn integers integersOutputOutput array array AA of prefix averages of of prefix averages of XX #operations#operations

AA new array of new array of nn integers integers nnss 0 0 11forfor ii 00 toto nn 1 1 dodo nn

ss ss XX[[ii]] nnAA[[ii]] ss ( (ii 1) 1) nn

returnreturn A A 11Algorithm Algorithm prefixAverages2 prefixAverages2 runs in runs in OO((nn) ) time time

1313

AmortizationAmortizationAmortizationAmortization– Typical data structure supports a wide variety of Typical data structure supports a wide variety of

operations for accessing and updating the elementsoperations for accessing and updating the elements– Each operation takes a varying amount of running timeEach operation takes a varying amount of running time– Rather than focusing on each operationRather than focusing on each operation– Consider the interactions between all the operations by Consider the interactions between all the operations by

studying the running time of a series of these operationsstudying the running time of a series of these operations

– Average the operations’ running timeAverage the operations’ running time Amortized running timeAmortized running time– The amortized running time of an operation within a series The amortized running time of an operation within a series

of operations is defined as the worst-case running time of of operations is defined as the worst-case running time of the series of operations divided by the number of the series of operations divided by the number of operationsoperations

– Some operations may have much higher actual running Some operations may have much higher actual running time than its amortized running time, while some others time than its amortized running time, while some others have much lowerhave much lower

1414

The Clearable Table Data StructureThe Clearable Table Data Structure The clearable table The clearable table – An ADT An ADT

Storing a table of elements Storing a table of elements

Being accessing by their index in the tableBeing accessing by their index in the table

– Two methods:Two methods:add(e)add(e) -- add an element -- add an element ee to the next available cell to the next available cell in the tablein the table

clear()clear() -- empty the table by removing all elements -- empty the table by removing all elements

Consider a series of operations (Consider a series of operations (addadd and and clearclear) performed on a clearable table ) performed on a clearable table SS– Each Each addadd takes O(1) takes O(1)– Each Each clearclear takes O(n) takes O(n)– Thus, a series of operations takes O(nThus, a series of operations takes O(n22), ),

because it may consist of only because it may consist of only clearclearss

1515

Amortization AnalysisAmortization AnalysisTheorem: Theorem: – A series of n operations on an initially empty clearable A series of n operations on an initially empty clearable

table implemented with an array takes O(n) timetable implemented with an array takes O(n) time

Proof:Proof:– Let MLet M00, M, M11, …, M, …, Mn-1n-1 be the series of operations performed be the series of operations performed

on S, where k operations are on S, where k operations are clearclear

– Let MLet Mi0i0, M, Mi1i1, …, M, …, Miik-1k-1 be the k be the k clearclear operations within the operations within the

series, and others be the (n-k) series, and others be the (n-k) addadd operations operations – Define iDefine i-1-1=-1,Mij takes i=-1,Mij takes ijj-i-ij-1j-1, because at most i, because at most ijj-i-ij-1j-1-1 -1

elements are added by add operations between Melements are added by add operations between M iij-1j-1 and M and Miijj

– The total time of the series is: The total time of the series is:

(n-k) + ∑(n-k) + ∑k-1k-1j=0j=0(i(ijj-i-ij-1j-1) = n-k + (i) = n-k + (ik-1k-1 - i - i-1-1) <= 2n-k) <= 2n-k

Total time is O(n)Total time is O(n)

Amortized time is O(1)Amortized time is O(1)

1616

Accounting MethodAccounting MethodThe methodThe method– Use a scheme of credits and debits: each Use a scheme of credits and debits: each

operation pays an amount of operation pays an amount of cyber-dollarcyber-dollar– Some operations overpay --> creditsSome operations overpay --> credits– Some operations underpay --> debitsSome operations underpay --> debits– Keep the balance at any time at least 0Keep the balance at any time at least 0

Example: the clearable tableExample: the clearable table– Each operation pays two cyber-dollarsEach operation pays two cyber-dollars– addadd always overpays one dollar -- one credit always overpays one dollar -- one credit– clearclear may underpay a variety of dollars may underpay a variety of dollars

the underpaid amount equals the number of the underpaid amount equals the number of addadd operations since last operations since last clear clear - 2 - 2

– Thus, the balance is at least 0Thus, the balance is at least 0– So the total cost is 2n -- may have creditsSo the total cost is 2n -- may have credits

1717

Potential FunctionsPotential FunctionsBased on energy modelBased on energy model– associate a value with the structure, Ø, associate a value with the structure, Ø,

representing the current energy staterepresenting the current energy state– each operation contributes to Ø a certain amount each operation contributes to Ø a certain amount

of energy t’ and consumes a varying amount of of energy t’ and consumes a varying amount of energy tenergy t

– ØØ00 -- the initial energy, Ø -- the initial energy, Øii -- the energy after the -- the energy after the i-th operationi-th operation

– for the i-th operationfor the i-th operationttii -- the actual running time -- the actual running time

t’t’ii -- the amortized running time -- the amortized running time

ttii = t’ = t’ii + Ø + Øi-1i-1 - Ø - Øii – Overall: T=∑(tOverall: T=∑(tii), T’=∑(t’), T’=∑(t’ii))– The total actual time T = T’ + ØThe total actual time T = T’ + Ø00 - Ø - Ønn – As long as ØAs long as Ø00 <= Ø <= Ønn, T<=T’ , T<=T’

1818

Potential FunctionsPotential FunctionsExample -- The clearable tableExample -- The clearable table– the current energy state Ø is defined as the the current energy state Ø is defined as the

number of elements in the table, thus Ø>=0number of elements in the table, thus Ø>=0– each operation contributes to Ø t’=2each operation contributes to Ø t’=2

– ØØ00 = 0 = 0

– ØØii = Ø = Øi-1i-1 + 1, if the i-th operation is + 1, if the i-th operation is addaddadd: t = t’+ Øadd: t = t’+ Øi-1i-1 - Ø - Øii = 2 - 1 = 1 = 2 - 1 = 1

– ØØii = 0, if the i-th operation is = 0, if the i-th operation is clearclearclear: t =t’+ Øclear: t =t’+ Øii-1 = 2 + Ø-1 = 2 + Øii-1-1

– Overall: T=∑ (t), T’=∑(t’)Overall: T=∑ (t), T’=∑(t’)

– The total actual time T = T’ + ØThe total actual time T = T’ + Ø00 - Ø - Ønn – Because ØBecause Ø00 = 0 <= Ø = 0 <= Ønn, T<=T’ =2n, T<=T’ =2n

1919

Extendable ArrayExtendable ArrayADT: ADT: Extendable array is an array with extendable Extendable array is an array with extendable size. One of its methods is size. One of its methods is addadd to add an element to to add an element to the array. If the array is not full, the element is the array. If the array is not full, the element is added to the first available cell. When the array is added to the first available cell. When the array is full, the full, the addadd method performs the following method performs the following– Allocate a new array with double sizeAllocate a new array with double size– Copy elements from the old array to the new arrayCopy elements from the old array to the new array– Add the element to the first available cell in the new arrayAdd the element to the first available cell in the new array– Replace the old array with the new arrayReplace the old array with the new array

Question: Question: What is the amortization time of What is the amortization time of addadd??– Two situations of Two situations of addadd: :

add -- O(1)add -- O(1)Extend and add -- (n)Extend and add -- (n)

– Amortization analysis:Amortization analysis:Each Each add add deposits 3 dollars and spends 1 dollardeposits 3 dollars and spends 1 dollarEach extension spends k dollars from the size k to 2k Each extension spends k dollars from the size k to 2k (copy)(copy)Amortization time of Amortization time of addadd is O(1) is O(1)

2020

Relatives of Big-OhRelatives of Big-Ohbig-Omegabig-Omega– f(n) is f(n) is (g(n)) if there is a constant c > 0 (g(n)) if there is a constant c > 0

and an integer constant nand an integer constant n00 1 such that 1 such that

f(n) f(n) c c••g(n) for n g(n) for n n n00

big-Thetabig-Theta– f(n) is f(n) is (g(n)) if there are constants c’ > 0 and c’’ > 0 and (g(n)) if there are constants c’ > 0 and c’’ > 0 and

an integer constant nan integer constant n00 1 such that c’ 1 such that c’••g(n) g(n) f(n) f(n) c’’c’’••g(n) for n g(n) for n n n00

little-ohlittle-oh– f(n) is o(g(n)) if, for any constant c > 0, there is an integer f(n) is o(g(n)) if, for any constant c > 0, there is an integer

constant nconstant n00 0 such that f(n) 0 such that f(n) c c••g(n) for n g(n) for n n n00

little-omegalittle-omega– f(n) is f(n) is (g(n)) if, for any constant c > 0, there is an integer (g(n)) if, for any constant c > 0, there is an integer

constant nconstant n00 0 such that f(n) 0 such that f(n) c c••g(n) for n g(n) for n n n00

2121

Intuition for Asymptotic NotationIntuition for Asymptotic NotationBig-OhBig-Oh– f(n) is O(g(n)) if f(n) is asymptotically f(n) is O(g(n)) if f(n) is asymptotically less than or less than or

equalequal to g(n) to g(n)

big-Omegabig-Omega– f(n) is f(n) is (g(n)) if f(n) is asymptotically (g(n)) if f(n) is asymptotically greater than greater than

or equalor equal to g(n) to g(n)big-Thetabig-Theta– f(n) is f(n) is (g(n)) if f(n) is asymptotically (g(n)) if f(n) is asymptotically equalequal to g(n) to g(n)

little-ohlittle-oh– f(n) is o(g(n)) if f(n) is asymptotically f(n) is o(g(n)) if f(n) is asymptotically strictly lessstrictly less

than g(n)than g(n)little-omegalittle-omega– f(n) is f(n) is (g(n)) if is asymptotically (g(n)) if is asymptotically strictly greaterstrictly greater

than g(n)than g(n)

2222

Example Uses of the Relatives of Big-OhExample Uses of the Relatives of Big-Oh 5n2 is (n2)

f(n) is (g(n)) if there is a constant c > 0 and an integer constant n0 1 such that f(n) c•g(n) for n n0 let c = 5 and n0 = 1

5n2 is (n)f(n) is (g(n)) if there is a constant c > 0 and an integer constant n0 1 such that f(n) c•g(n) for n n0 let c = 1 and n0 = 1

5n2 is (n)f(n) is (g(n)) if, for any constant c > 0, there is an integer constant n0 0 such that f(n) c•g(n) for n n0 need 5n02 c•n0 given c, the n0 that satisfies this is n0 c/5 0