Project 2 due …

Chapter 2

Divide and Conquer

Recurrence Relations

Equation or an inequality that describes a function by its values on smaller inputs.

Recurrence relations arise when we analyze the running time of iterative or recursive algorithms. Ex: Divide and Conquer.

T(n) = O(1) if n cT(n) = a T(n/b) + D(nd) otherwise

Solution Methods Substitution Method. Recursion-tree Method. Master Method.

Divide and Conquer Strategy

How does Divide and Conquer Strategy work?

Examples?

Another Example: Merge Sort

Sorting Problem: Sort a sequence or n elements into non-decreasing order.

Divide: Divide the n-element sequence to be sorted into two subsequences of n/2 elements each

Conquer: Sort the two subsequences recursively using merge sort.

Combine: Merge the two sorted subsequences to produce the sorted answer.

Merge Sort – Example

43 15 9 1 22 26 19 55 37 43 99 2

18 26 32 6 43 15 9 1 22 26 19 55 37 43 99 2

18 26 32 6 43 15 9 1 22 26 19 55 37 43 99 2

18 26 32 6 43 15 9 1 22 26 19 55 37 43 99 2

18 26 32 6 43 15 9 1 22 26 19 55 37 43 99 2

18 26 32 6

Merge Sort – Example

18 26 32 6 43 15 9 1

18 26 32 6 43 15 9 1

18 26 32 6 43 15 9 1

2618 6 32 1543 1 9

18 26 32 6 43 15 9 1

18 26 32 6 43 15 9 1

18 26 32 6 15 43 1 9

6 18 26 32 1 9 15 43

1 6 9 15 18 26 32 43

18 26

18 26

18 26

32

32

6

6

32 6

18 26 32 6

43

43

15

15

43 15

9

9

1

1

9 1

43 15 9 1

18 26 32 6 43 15 9 1

18 26 6 32

6 26 3218

1543 1 9

1 9 15 43

1 6 9 1518 26 32 43

Original Sequence Sorted Sequence

Analysis of Merge Sort

Running time T(n) of Merge Sort: Divide: computing the middle takes O(1) Conquer: solving 2 sub-problems takes

2T(n/2) Combine: merging n elements takes O(n) Total:

T(n) = O(1) if n = 1T(n) = 2T(n/2) + O(n) if n > 1

T(n) = ?

Binary searchAlgorithm:

if (no element in the array), return –1; else

check K against the middle value of the array

if (same), return position;else if (K is smaller)

binary search the 1st half of the array;

elsebinary search the 2st half of

the array;

T(n) = T(n/2) + 1;T(1) = 1;

? T(n) = ?

Multiplication

x=10011010, y = 10101011 x*y = ? – divide and conquer?

Running Time

Recurrence: T(n) = O(1) if n = 1 T(n) = 3T(n/2) + O(n)

if n > 1

T(n) = ?

Quicksort

http://en.wikipedia.org/wiki/Quicksort

Recurrence Relations

Equation or an inequality that describes a function by its values on smaller inputs.

Recurrence relations arise when we analyze the running time of iterative or recursive algorithms. Ex: Divide and Conquer.

T(n) = O(1) if n cT(n) = a T(n/b) + D(nd) otherwise

Solution Methods Substitution Method. Recursion-tree Method. Master Method.

Substitution Method

Guess the form of the solution, then use mathematical induction to show it correct.

Substitute guessed answer for the function when the inductive hypothesis is applied to smaller values.

Works well when the solution is easy to guess.

No general way to guess the correct solution.

Example –AsymptoticsTo Solve: T(n) = 2T(n/2) + n Guess: T(n) = O(n lg n) Need to prove: T(n) cn lg n, for some c > 0. Hypothesis: T(k) ck lg k, for all k < n. Calculate:

T(n) 2c n/2 lg n/2 + n

c n lg (n/2) + n

= c n lg n – c n lg2 + n

= c n lg n – n (c lg 2 – 1)

c n lg n

(The last step is true for c 1 / lg2.)

Exercises

Solution of T(n) = 3T(n/3) + n is

Solution of T(n) = T(n/3) + 1 is

Solve T(n) = 2T(n/2) + 1

O(nlogn)

O(lgn)

O(n)

Running Time

Recurrence: T(n) = O(1) if n = 1 T(n) = 3T(n/2) + O(n)

if n > 1

T(n) = ?

Recursion-tree Method Making a good guess is sometimes difficult

with the substitution method. Use recursion trees to devise good

guesses. Recursion Trees

Show successive expansions of recurrences using trees.

Keep track of the time spent on the subproblems of a divide and conquer algorithm.

Help organize the algebraic bookkeeping necessary to solve a recurrence.

Recursion Tree – Example

Running time of Merge Sort:T(n) = (1) if n = 1T(n) = 2T(n/2) + (n) if n > 1

Rewrite the recurrence asT(n) = c if n = 1 T(n) 2T(n/2) + cn if n > 1

c > 0: Running time for the base case and

time per array element for the divide and

combine steps.

Recursion Tree for Merge Sort

For the original problem, we have a cost of cn, plus two subproblems each of size (n/2) and running time T(n/2).

cn

T(n/2) T(n/2)

Each of the size n/2 problems has a cost of cn/2 plus two subproblems, each costing T(n/4).

cn

cn/2 cn/2

T(n/4) T(n/4) T(n/4) T(n/4)

Cost of divide and merge.

Cost of sorting subproblems.

Recursion Tree for Merge Sort

Continue expanding until the problem size reduces to 1.cn

cn/2 cn/2

cn/4 cn/4 cn/4 cn/4

c c c cc c

lg n

cn

cn

cn

cn

Total : cnlgn+cn

Other Examples

Use the recursion-tree method to determine a guess for the recurrences

T(n) = 4T(n/4)+4 T(n) = T(n/2) + 2T(n/3) + n

The Master Method

Based on the Master theorem. “Cookbook” approach for solving

recurrences of the form T(n) = aT(n/b) + f(nd)

a 1, b > 1 are constants. f(n) is asymptotically positive. n/b does not have to be an integer, but we

ignore floors and ceilings. Why?

Requires memorization of three cases.

Master Theorem

Master Theorem

Let a 1, b > 1 be constants, f(n) be a function. Let T(n) be defined on nonnegative integers by T(n) = aT(n/b) + f(n), where we can replace n/b by n/b or n/b.

Then T(n) can be bounded asymptotically in three cases:

1. If f(n) = O(nlogba–) for some > 0, then T(n) = (nlogba).

2. If f(n) = (nlogba), then T(n) = (nlogbalg n).

3. If f(n) = (nlogba+) for some constant > 0, and if, for some constant c < 1 and all sufficiently

large n, we have af(n/b) c f(n), then T(n) = (f(n)).

Master Method – Examples

T(n) = 16T(n/4)+n a = 16, b = 4, nlogba = nlog416 = n2. f(n) = n = O(nlogba-) = O(n2- ), where =

1 Case 1. Hence, T(n) = (nlogba ) = (n2).

T(n) = T(3n/7) + 1 a = 1, b=7/3, and nlogba = nlog 7/3 1 = n0 =

1 f(n) = 1 = (nlogba) Case 2. Therefore, T(n) = (nlogba lg n) = (lg n)

Master Method – Examples

T(n) = 3T(n/2) + n ?

Exercise

Which method? T(n) = 4T(n/3)+4 T(n) = 2T(n/2 + 17) + n T(n) = T(n/2) + 2T(n/4) + n T(n) = T(n-1)+n

Matrix Multiplications

Matrix multiplications

Example: AB=C

Let A,B,C be nn matrices. What’s the cost to obtain C ?

(assuming n is a power of 2)

ut

sr

hg

fe

dc

ba

r = ae + bgs = af + bht = ce + dgu = cf + dh

Cost of the direct method

It takes T(n/2) to obtain each of ae, bf, …dh.

It takes n/2n/2 additions to obtain r, s, t, or u.

Therefore:T(n) = 8T(n/2)+(n2)

= 8T(n/2)+cn2

= 8( 8T(n/22)+c(n/2)2 )+cn2

= ?

Cost of the direct method

It takes T(n/2) to obtain each of ae, bf, …dh.

It takes n/2n/2 additions to obtain r, s, t, or u.

Therefore:T(n) = 8T(n/2)+(n2)

= (n3)

Strassen’s Algorithm Strassen’s Algorithm: an undetermined

coefficient method. It’s based on that fact that A+B is much cheaper to calculate than AB. Outline of the proof: Let Pi = AiBi, i=1,…,7

where Ai = (i1a+i2b+i3c+i4d), Bi = (i1e+i2f+i3g+i4h), ij, ij {-1,0,1}.

Try to determine ij, ij such that r, s, t, u = ijPi, ij {-1,0,1}

Note: The hidden coefficients in front of nlog2

7

is larger than the one in front of n3.

Strassen’s Algorithm

Determine the coefficients ij, ij: A1 = a, A2 = a+b, A3 = c+d, A4 = d,

A5 = a+d, A6 = b d, A7 = a c B1 = f h, B2 = h, B3 = e, B4 = g e,

B5 = e+h, B6 = g+h, B7 = e+f

r = P5+P4-P2+P6

s = P1+P2

t = P3+P4

u = P5+P1P3 P7

Verify u=…

ut

sr

hg

fe

dc

ba

Cost for n=8 ?

Strassen’s Algorithm

Strassen’s Algorithm:T(n) = 7T(n/2)+(n2) → T(n)= (nlog

27) =

O(n2.8)

Strassen’s Algorithm

n2.8

n3

Insertion sort and selection sort

Recurrence: T(n) = O(1) if n = 1T(n) = T(n-1) + O(n) if n > 1

T(n) = ?

Complexity?No divide and conquer algorithms