COMP3121 E-Commerce Technologies Richard Henson University of Worcester November 2010.
COMP3121 2 Basic Tools for Analysis of Algorithms
Transcript of COMP3121 2 Basic Tools for Analysis of Algorithms
-
8/12/2019 COMP3121 2 Basic Tools for Analysis of Algorithms
1/23
Topic 1:
Basic tools for analysis of algorithms
Assignment: please read pages: 41-90(23-76 in the first edition)
Recurrences
Recurrences are recursive definitions, i.e.
definitions that specify values of a functionin terms of values of the same function for
smaller arguments, like:
(n+1)! = (n+1)n!
f(n+1) = f(n) + f(n-1)f(n) = f( n/2) + n
Recurrences naturally arise when estimating
the run time of recursive algorithms. Forexample, recall that theMergeSorthas the
following pseudo-code:
1
-
8/12/2019 COMP3121 2 Basic Tools for Analysis of Algorithms
2/23
Merge-Sort(A,p,r)
1if p < r
2 then q
(p+r)/23 Merge-Sort(A,p,q)4 Merge-Sort(A,q+1,r)
5 Merge(A,p,q,r)
TheMergeprocedure is running in linear
time, i.e., in time k nfor some k.
Thus, as we have shown, the run time of
MergeSortsatisfies recurrence formula
T(n)=2T(n/2)+ C n
To make things a bit simpler we will assumen is a power of two; in the CLRS textbook it
is shown that thats OK. Thus we get the
recurrence formula:
T(n)=2T(n/2)+ C n
There are ways to solve some recurrences
explicitly, but they all tend to be somewhat
tricky.
2
-
8/12/2019 COMP3121 2 Basic Tools for Analysis of Algorithms
3/23
However, in estimating the efficiency of an
algorithm wedo notneed the solution of arecurrence formula, but we only need to find
the growth rateof the solutioni.e., its
asymptotic behavior.
The master method providesan estimate of
the growth rateof the solution forrecurrences of the form
T(n) = a T(n/b) + f(n)
for certain classes of a, band overhead
functions f(n), without having to solve
these recurrences. This is what the Master
Theorem is all about.
3
-
8/12/2019 COMP3121 2 Basic Tools for Analysis of Algorithms
4/23
If T(n) is interpreted as the number of steps
needed to execute an algorithm for an input
of length n, this recurrence corresponds to adivide and conquermethod, in which a
problem of size nis divided into aproblems
of size n/b, where a, bare positive constants:
T(n) = a T(n/b) + f(n)
number of sub-problem price for dividing the
sub-problems size problem and combining
the solutions
(overhead)
Clearly, in order to have a recursion at all, we
must have n/b< n; i.e., b > 1.
Also, since acorresponds to the number of
problems of smaller size, we are interested
only in cases with a r1.
Function f(n) is the overhead, i.e. the cost of
dividing the problem of size ninto asub-
problems of size n/band then putting the
results of sub-problems together.
4
-
8/12/2019 COMP3121 2 Basic Tools for Analysis of Algorithms
5/23
The Master Theorem:
Let a, bbe constants such that a r1 and
b >1, letf(n)be a function and let T(n)be
defined on natural numbers by the
recurrence T(n)=a T(n/b) + f(n)Then T(n) can be bounded asymptotically asfollows :
1) If f(n)=O(nlogba-e) for somee > 0,
then: T(n)=Q(nlogba)
Thus, iff(n)does not grow faster than the
polynomial nlogb a-e of degree strictly
smallerthan logba then the sollution T(n)
grows as the polynomial nlogb a.
5
-
8/12/2019 COMP3121 2 Basic Tools for Analysis of Algorithms
6/23
2) If f(n)=Q(nlogba)
then : T(n)=Q(nlogbalogn)
Thus, iff(n) grows exactly as fast asthe
polynomial nlogb a then the solution T(n)
grows exactly as fast as nlogb alog n.
6
-
8/12/2019 COMP3121 2 Basic Tools for Analysis of Algorithms
7/23
3) If f(n) = W(nlogba+e) for somee > 0,
and if af(n/b)bc f(n)for some constant c
-
8/12/2019 COMP3121 2 Basic Tools for Analysis of Algorithms
8/23
Informally speaking,
i) iff(n)growssubstantially slowerthan
the polynomial nlogb a then the solution
T(n) grows exactly as nlogb a ;
ii) iff(n) growsas fast as the polynomial
nlogb a then the sollution T(n) growsexactly as n
logb alogn;
iii) finally, if f(n)growssubstantially
faster than the polynomial nlogb a and the
extraconditiona f(n/b)bc f(n)is satisfied
for some c
-
8/12/2019 COMP3121 2 Basic Tools for Analysis of Algorithms
9/23
Dumb Examples (better ones later):
1. Let T(n) = 9 T(n/3) + n. Then a = 9 andb = 3, so nlog
39=n
2. Since f(n) = n =
O(n2-.5
) = O(n1.5
)the first case applies
with e= .5 and we conclude that T(n) =
n2.
2. If T(n) = T(2/3n) + 1 then a = 1 and b= 3/ 2, so n
log3/2
1=n
0=1. Thus, both
f(n)=1 and nlog
ba=1 which implies
f(n)=Q(nlogb a) and so the second case
applies; consequently:
T(n)=Q(nlogb a logn)=Q(n
0logn)=Q(logn)
3. Consider now T(n) = 3T(n/4)+ n.Then
a=3and b=4and so nlog
ba= n
log43n
0.79.
Thus,f(n)= n = W(n.79+.1
)W(n0.89
). Also,
a f(n/b )=3 f(n/4 ) = 3 n/4=3/4 n=3/4 f(n)=.75n < .8 n = .8 f(n)
Consequently by case 3, T(n)=Q(n).
9
-
8/12/2019 COMP3121 2 Basic Tools for Analysis of Algorithms
10/23
4. Finally, consider recurrence
T(n)=2T(n/2)+n log n
In this case a=2, b=2,f(n) = nlogn. Thus,
nlogb a=n
log2 2= n. While f(n)= n log n >n,
but for no e f(n)=n log n >n1+e
for
sufficiently large n, so the third case doesnot apply and thus wecannot use the
master theorem.
Homework: Use basic calculus to show that
for every e there isN such that for all n > N
ne>log n.
10
-
8/12/2019 COMP3121 2 Basic Tools for Analysis of Algorithms
11/23
Real examples:
Analysis of Merge Sort
As we saw, Merge Sort running time is
given by the recurrence relation
T(n)=2T(n/2)+ C n
Thus, for the Master Theorem we have:
a=2, b=2, f(n)=Cn
Since
nlogba = n
log22 = n1= n
we get that the second case holds because
f(n)=Cn =(n)=(n
logba)
Thus, the solution of the recurrence satisfies
T(n)=(nlogbalog n)=(n
log n)
11
-
8/12/2019 COMP3121 2 Basic Tools for Analysis of Algorithms
12/23
Real examples:
Analysis of the Karatsuba algorithm for
integer multiplication
The Karatsuba algorithmrunning time is
given by the recurrence relation
T(n)=3T(n/2)+ C n
Thus, for the Master Theorem we have:
a=3, b=2, f(n)=Cn
Since
nlogba = n
log23 = n1.58
we get that the first case holds, because
f(n)=Cn =O(n1.48
)=O(n
1.58 - .1)
=
O (n
log23-)
Thus, the solution of the recurrence satisfies
T(n)=(nlogba)= (n
log23)=(n
1.58)
12
-
8/12/2019 COMP3121 2 Basic Tools for Analysis of Algorithms
13/23
Determining the running time of an
algorithm seems to be hard because it
requires knowing difficult theorems. Whybother? Why not just write a correct
program and not worry how fast it is?
n 16 256 65536
nlogn 64 2048 1048576n
2256 65536 4294967296
n3
4096 16777216 281474976710656
2n
65536 1077
1019728
The above table shows that how fastan
algorithm is, often determines in practice if
an algorithm will run at all or notbecause it
would take too much time to finish!!!
Why bother proving the master theoremand not just take it on faith?
13
-
8/12/2019 COMP3121 2 Basic Tools for Analysis of Algorithms
14/23
For a practical reason :
We areinterested intechniquesused in thisproof : we want to acquire skills inhow to
countthe total number of steps of a
recursive procedure,how to do sumationsof
various sums which will often pop up in our
analyses of algorithms etc. Themethodsof
the proof will keep appearing on manyoccasions.
Thus, by mastering the proof we will get
used to concepts involved and learn the
machinery used in estimating the run time of
manyalgorithms.
The same machinery can be used to estimate
the run time of recursive algorithms which
you might be designing yourselves!
14
-
8/12/2019 COMP3121 2 Basic Tools for Analysis of Algorithms
15/23
Proof of the Master Theorem
We will prove the Master Theorem only for numbers which
are powers of b. The proof for an arbitrary number followsfrom the properties of the floor and ceiling functions; recall
2.99 = 2 and 2.01 = 3. It does not add anythinginteresting from the prospective of algorithms.
Proof : (concentrate on the method, NOT on calculations !!)
Step 1.(unwinding recursion):
Substitute m in T(m) = aT(m/b) + f(m)by
n/bjfor all 1ji where i = logb n (for
simplicity we assume nis a power of b.
Thus, assume that T(m) = aT(m/b) + f(m)
for all m.Then by substituting m in this
equality by n,n/b, n/b2and so on up ton/b
i-1,
we get : T(n) = aT(n/b) + f(n)
T(n/b) = aT(n/b2) + f(n/b)
T(n/b2
) = aT(n/b3
) + f(n/b2
).
.
.
T(n/bi-1) = aT(n/b
i) + f(n/b
i-1)
15
-
8/12/2019 COMP3121 2 Basic Tools for Analysis of Algorithms
16/23
Step 2.Sucessive eliminationsof T(n/bk) in
order to replace the instances T(n/b
k
) of therecursively definedfunction T(n)by the
instancesf(n/bk) of the explicitely defined,
simpler functionf(n).
By consecutive replacements of T(n/bk) by
aT(n/bk+1) + f(n/bk) using equationsobtained at Step 1 we get :
T(n) = aT(n/b)+ f(n)=
a(aT(n/b2) + f(n/b)) + f(n)=
a2T(n/b
2)+a f(n/b) + f(n)=
a2(aT(n/b
3) + f(n/b
2)) + a f(n/b)+ f(n)=
a3T(n/b
3)+ a
2f(n/b
2)+a f(n/b)+ f(n)=
.
.
aiT(n/b
i)+a
i-1f(n/b
i-1)+a
i-2f(n/b
i-2)++ f(n)
Here we essentially unwind the recursion
untill we hit the base case T(n/bi)=T(1),
(because bi= n, i.e. i=logbn).
16
-
8/12/2019 COMP3121 2 Basic Tools for Analysis of Algorithms
17/23
Thus,
T(n)=aiT(1) + a
i-1f(n/b
i-1)+a
i-2f(n/b
i-2)++ f(n)=
i summands
=
=
+=+1)(log
0
log1
0
)()1()()1(n
jb
njni
jb
njib
jb
j faTafaTa
Step 3:simplifying the coefficient in frontof T(1).
Since alogbn=n
logba we get:
=
+=1)(log
0
log)()1()(
n
jb
njab
j
b faTnnT
Thus, to estimate the behavior of T(n)we
have to estimate the behavior of the sum
=
=1log
0
)()(n
jb
njb
jfanS
Notice that in this sum )( jbnjfa is a product of members of an increasing
geometric progression aj
and values off(n/bj
) with inputsn/bj
that form adecreasing geometric progression. Thus, the behavior of the sum depends on
whether ajgrows faster thanf(n/b
j) decreases!! (we are interested only inf(n)
that is monotonically increasing, becausef(x) will stand for the overhead of
divide-and-conquer algorithms. ) This is the crux of Master Theorem!. We
now consider the growth rate off(n) accordingly, i.e., we replacef(n/bj)
with its asymptotic estimate.
17
-
8/12/2019 COMP3121 2 Basic Tools for Analysis of Algorithms
18/23
Step 4:Replacing f(n/bj) by asymptotic
estimates. We proceed according to our
three cases.
Case 1: Assumef(m)=O(mlogba-e). Then
using properties of the big-oh, O(which are
easy to show) and by substitutingmin the
above byn/bi
we get f(n/bi)=O((n/b
i)
logb a-e),
and so
( )( ) ( )
( )
=
=
=
=
===
=
=
=
=
=
=
=
1log
0
log
1log
0
log1log
0
log
1log
0
log1log
0
log
1log
0
log1log
0
log
log)(
))(()()(
n
j
ja
n
j
j
aaba
n
j
j
b
aba
n
j
j
b
aan
j
a
b
nj
n
j
a
b
njn
jb
nj
b
b
b
b
b
ab
b
b
ab
b
b
bj
b
bj
b
j
bnO
nOnO
nOaO
OafanS
18
-
8/12/2019 COMP3121 2 Basic Tools for Analysis of Algorithms
19/23
By using
nbq
qqqq nn
n b==++++
+
log
1
2 and11...1
we sum the geometric progression
and get
( )
=
1log
0
n
j
jb
b
=
=
1
1)(
loglog
b
bnOnS
na
b
b
( )aaa
a bbb
b nO
b
nnO
b
nnO
logloglog
log
11
1=
=
Consequently,
( )aan
j
b
nja bbb
jb nOTnfaTnnT
loglog1log
0
log)1()()1()( +=+=
=
19
-
8/12/2019 COMP3121 2 Basic Tools for Analysis of Algorithms
20/23
However, adding to nlog
baT(1) something
that grows not faster than n
logba
results insomething that grows as fast as nlog
ba, i.e.
aaa bbb nnOTnnTlogloglog
)1()( =+=
This proves the first case.
20
-
8/12/2019 COMP3121 2 Basic Tools for Analysis of Algorithms
21/23
Case 2: Assumef(n)=(nlog
ba). Then as in
the first case, we first estimate
( )
( )
( )nn
nn
na
afanS
b
a
n
j
an
j
j
aaa
n
j
j
b
aan
j
a
b
nj
n
j
a
b
njn
jb
nj
b
bb
bb
b
ab
b
b
bj
b
bj
b
j
log
1
)(
))(()()(
log
1log
0
log1log
0
log
1log
0
log1log
0
log
1log
0
log1log
0
log
=
=
=
=
===
=
=
=
=
=
=
Since logbn= logb2log2n we get
nnnSab
2
loglog)( =
Consequently, denoting log2nby lg n, we
get
( ) ( )nnnnTn
faTnnT
aaa
n
jbnja
bbb
b
jb
lglg)1(
)()1()(
logloglog
1log
0
log
=+
=+=
=
21
-
8/12/2019 COMP3121 2 Basic Tools for Analysis of Algorithms
22/23
Case 3: In theassumption that
f(n/b)bc/a f(n) we substitute nby n/bk-1
to
getf(n/bk)bc/a f(n/bk-1) . Thus, we get:f(n/b
j)bc/a f(n/b
j-1)
f(n/bj-1
)bc/a f(n/bj-2
)
f(n/bj-2
)bc/a f(n/bj-3
)
...f(n)bc/a f(n/b)
By substitution, chaining these inequalities,
we get
f(n/bj)bc/a f(n/b
j-1)b(c/a)
2f(n/b
j-2) b
...b(c/a)jf(n),i.e. we get
f(n/bj)bc
j/a
jf(n)
22
-
8/12/2019 COMP3121 2 Basic Tools for Analysis of Algorithms
23/23
Thus, we can estimate S(n)as follows
))((1
)()()()(
)()()(
0
1log
0
1log
0
1log
0
1log
0
nfOc
nfcnfcnfnfc
nfacafanS
j
jn
j
jn
j
j
n
jj
j
j
n
jbnj
bb
bb
j
=
==
==
=
=
=
=
=
Consequently, we get:
))(()1(
)()1()(
log
1log
0
log
nfOTn
faTnnT
a
n
jb
nja
b
b
j
b
+
=+=
=
By assumption in the third case
f(n) = W(nlogb a+e) for somee > 0, and so,
since f(n) grows much faster than nlog
ba,
))(())(()1()(log
nfOnfOTnnTab =+=
This finishes the proof for the third case and
thus we are done !